WO2016157802A1

WO2016157802A1 - Information processing apparatus, information processing system, information processing method, and storage medium

Info

Publication number: WO2016157802A1
Application number: PCT/JP2016/001595
Authority: WO
Inventors: 高橋　徹; 哲夫井下; 大輔西脇; 吉洋服部; 達哉安部
Original assignee: 日本電気株式会社
Priority date: 2015-03-27
Filing date: 2016-03-18
Publication date: 2016-10-06
Also published as: JP6683195B2; JPWO2016157802A1

Abstract

In order to calculate a coordinate transformation between coordinates of a camera coordinate system and coordinates of a world coordinate system, regardless of target regions, an information processing apparatus according to the present invention includes: a small region grouping means for dividing small regions, which are formed on the basis of points having three-dimensional information about a location in a terrain corresponding to a region included in captured image data captured by an image capture device, into small region groups, which are groups of a small region, on the basis of the normal direction and altitude of the small region; and a coordinate transformation table creation means for creating a coordinate transformation table which contains information for the coordinate transformation between the coordinates of the camera coordinate system, which are coordinates in the captured image data, and coordinates of the world coordinate system, which are coordinates in a small region that corresponds to the captured image data, using the small region groups and candidate information which is related to the coordinate transformation.

Description

Information processing apparatus, information processing system, information processing method, and recording medium

The present invention relates to information processing, and more particularly, to an information processing apparatus, an information processing system, an information processing method, and a recording medium that process position information.

A wide area monitoring system generally detects a vehicle and / or a person from a captured image and grasps the position of the vehicle and / or the person.

In order to accurately determine the position of the vehicle included in the image captured by the camera, the wide area monitoring system preliminarily calculates the coordinates of each pixel in the camera coordinate system in which the vehicle is captured, the coordinates in the world coordinate system, and Must be associated with each other. That is, the wide area monitoring system needs to calculate the coordinate conversion between the coordinates of each pixel in the camera coordinate system and the coordinates of the world coordinate system.

However, in general, the road surface on which the vehicle travels is often not a simple plane. In such a case, the road surface shape projected on the camera coordinate system is complicatedly distorted. Therefore, the coordinate transformation in this case cannot be expressed using a simple linear transformation. Therefore, accurate association between the camera coordinate system and the world coordinate system is not easy.

Therefore, for example, when a new camera or the like for monitoring a traffic vehicle is installed on the roadside of the road, first, a correspondence table that accurately associates the position (coordinates) of the camera coordinate system with the position (coordinates) of the world coordinate system. It is important to create Here, the correspondence table is a table for converting a two-dimensional coordinate table that is a camera coordinate system into a three-dimensional coordinate table that is a world coordinate system. Generally, methods 1 to 3 described below are mainly conceivable as a method for performing this association.

(Method 1)
The method described below is referred to as “Method 1”.

Method 1 determines the position of the main target (hereinafter referred to as “Landmark”) on the camera field of view in the world coordinate system based on general maps, aerial photographs or road drawings. To derive. An example of a landmark is an illumination column. Then, the method 1 compares the position of the derived landmark in the world coordinate system (coordinates such as latitude, longitude, and altitude) with the position on the captured image (that is, coordinates in the camera coordinate system). And associate the two. In Method 1, a continuous correspondence table is created by interpolating the coordinates between the landmarks.

(Method 2)
The method described below is referred to as “Method 2”.

Method 2 derives the position (coordinates) of the landmark on the camera field of view in the world coordinate system using road surveying equipment such as Total Station. Then, in the method 2, the position (coordinate) of the derived landmark in the world coordinate system is compared with the position on the photographed image (that is, sitting in the camera coordinate system), and the two are associated with each other. Method 2 creates a continuous correspondence table by interpolating the coordinates between the landmarks.

(Method 3)
The method described below is referred to as “Method 3”.

Method 3 uses a survey vehicle equipped with a GPS (Global Positioning System) / inertial navigation system, stereo camera, laser radar (Laser radar) or the like. Method 3 derives three-dimensional continuous road shape data (map data) as a position (coordinates) in the world coordinate system. Then, in the method 3, the map data values of the landmarks on the camera field of view are compared with the positions on the image obtained by photographing them (that is, the coordinates in the camera coordinate system), and the two are associated with each other. Method 3 does not use interpolation or the like for the coordinates between the landmarks. Method 3 creates a continuous correspondence table using the map data for the coordinates between the landmarks.

Further, Patent Document 1 discloses the following method. The method described in Patent Document 1 records the coordinates of the world coordinate system recorded by a survey vehicle equipped with GPS and the frame of a moving image captured by the camera in complete synchronization. This method creates a continuous correspondence table between the coordinates of the world coordinate system and the coordinates of the camera coordinate system without performing the road surface shape surveying based on the above record.

Further, Patent Document 2 discloses the following method. The method described in Patent Document 2 analyzes a three-dimensional point group representing the shape of a plurality of features and the like based on statistical processing as three-dimensional topographic data. In this method, the coordinate axis used to associate the three-dimensional terrain data with the coordinates of the world coordinate system is estimated based on the analysis result. Japanese Patent Application Laid-Open No. H10-228707 describes clustering point groups that are relatively close to each other. Further, Patent Document 2 describes that a point group is separated for each side surface based on the distribution of normal vectors on the side surface of the building.

Japanese Patent Application Laid-Open No. 2010-236891 JP 2014-186565 A

However, in the method 1 and the method 2, the area associated with the position (coordinates) in the camera coordinate system is limited to a city area where a landmark such as a high-rise building or a lighting column exists, or a place where a road drawing exists. It will be. Therefore, when a place where a landmark does not exist (for example, a remote island, a port, or a mountainous area) is monitored in a wide area, Method 1 and Method 2 can associate the coordinates of the camera coordinate system with the coordinates of the world coordinate system. Absent. That is, in the above method, the method 1 and the method 2 cannot obtain the coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. Furthermore, when there are undulations between landmarks, method 1 and method 2 need to interpolate the coordinate transformation of the point according to the undulations. Therefore, when a simple linear transformation is used, the estimation accuracy deteriorates in Method 1 and Method 2.

In the method described in Method 3 and Patent Document 1, since a surveying vehicle is used, there is no problem with the landmark. However, in order to associate the coordinates of the camera coordinate system and the coordinates of the world coordinate system with respect to the entire field of view of the camera, it is necessary to perform shooting using the camera in a situation where the survey vehicle is running over a wide area. Therefore, the work cost is increased. Furthermore, in the method described in Method 3 and Patent Document 1, map data is derived in an area where the survey vehicle can travel. For this reason, in the method described in Method 3 and Patent Document 1, there is a possibility that the region in which the coordinates of the camera coordinate system and the coordinates of the world coordinate system are associated with each other may be limited.

Thus, in the methods 1 to 3 and the method described in Patent Document 1, the area in which the camera field of view (camera coordinate system) and the real coordinates (world coordinate system) can be associated with each other is limited. Specifically, in the methods 1 to 3 and the method described in Patent Document 1, the associated area (area where coordinate transformation can be obtained) is an area where a landmark exists, or a terrain area with little undulation. Or, it is limited to the area where the surveying vehicle can travel. Therefore, when monitoring a wide area outside, in the area where there are no landmarks, the area of terrain with little undulations, or the area where the surveying vehicle cannot run, the position in the world coordinate system of the object shown in the image taken by the camera ( It was difficult to accurately obtain the coordinates. As described above, the methods 1 to 3 and the method described in Patent Document 1 have a problem in that the coordinates of the camera coordinate system and the coordinates of the world coordinate system cannot be associated with each other in the region. That is, the methods 1 to 3 and the method described in Patent Document 1 have a problem that coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system cannot be obtained in the region. Note that the method described in Patent Document 2 is a method of determining a gravity method from the outer shape of a building and cannot solve the above problem.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-mentioned problems, and an information processing apparatus, an information processing system, and an information processing method capable of calculating coordinate transformation between the coordinates of the camera coordinate system and the coordinates of the world coordinate system regardless of the target region And providing a recording medium.

An information processing apparatus according to an aspect of the present invention includes a small area formed on the basis of a point having three-dimensional information regarding a position on the terrain corresponding to an area included in captured image data captured by the imaging apparatus. Cameras that are coordinates in captured image data using small region grouping means that divides into small region groups that are groups of small regions based on the normal direction and elevation, and candidate information that is information regarding small region groups and coordinate transformation Coordinate conversion table creating means for creating a coordinate conversion table that is information on coordinate conversion between coordinates in the coordinate system and coordinates in the world coordinate system that is coordinates in a small area corresponding to the captured image data.

An information processing system according to an aspect of the present invention includes the above information processing apparatus, display means for displaying three-dimensional information and / or measurement point candidates received from the information processing apparatus, three-dimensional information and / or Input means for acquiring candidate information in the measurement point candidates.

In addition, an information processing method according to an embodiment of the present invention includes a small area formed on the basis of a point having three-dimensional information regarding a position on the terrain corresponding to an area included in captured image data captured by the imaging apparatus. Based on the normal direction and altitude of the area, it is divided into small area groups that are groups of small areas, and using the small area group and candidate information that is information related to coordinate transformation, the coordinates of the camera coordinate system that is the coordinates in the captured image data A coordinate conversion table that is information on coordinate conversion between the coordinates and coordinates in the world coordinate system that is coordinates in a small area corresponding to the captured image data is created.

According to another aspect of the present invention, there is provided a recording medium including a small area formed based on a point having three-dimensional information regarding a position on the terrain corresponding to the area included in the captured image data captured by the imaging device. The camera coordinate system that is the coordinates in the captured image data using the process of dividing into small area groups that are groups of small areas based on the normal direction and elevation of the image, and candidate information that is information related to the small area groups and coordinate transformation A computer readable recording process for creating a coordinate conversion table, which is information of coordinate conversion between the coordinates of the image and the coordinates in the world coordinate system, which are the coordinates in the small area corresponding to the captured image data, and the program to be executed by the computer are recorded.

According to the present invention, it is possible to obtain an effect that coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system can be calculated regardless of the target region.

FIG. 1 is a block diagram illustrating an example of a configuration of a coordinate association system including a coordinate conversion table creation device according to a first embodiment of the present invention. FIG. 2 is a schematic diagram illustrating an example of 3D terrain data stored in the 3D terrain database storage unit. FIG. 3 is a flowchart illustrating an example of the operation in the coordinate conversion table creating apparatus according to the first embodiment. FIG. 4 is a schematic diagram illustrating an example of a small region group according to the first embodiment. FIG. 5 is a block diagram showing an example of the configuration of the object position measuring apparatus according to the first embodiment. FIG. 6 is a schematic diagram illustrating a situation in which coordinates in the camera coordinate system of an object such as a suspicious vehicle in the object position measurement apparatus according to the first embodiment are converted into coordinates in the world coordinate system. FIG. 7 is a block diagram illustrating an example of a configuration of a coordinate association system according to the second embodiment. FIG. 8 is a flowchart illustrating an example of the operation of the coordinate conversion table creation device according to the second embodiment. FIG. 9 is a schematic diagram illustrating an example of a group obtained as a result of step S203 in FIG. FIG. 10 is a block diagram illustrating an example of a configuration of a coordinate association system according to the third embodiment. FIG. 11 is a flowchart illustrating an example of the operation of the coordinate conversion table creating apparatus according to the third embodiment. FIG. 12 is a schematic diagram for explaining a method of extracting measurement point candidates by the measurement point extraction unit according to the third embodiment. FIG. 13 is a block diagram illustrating an example of a configuration of a coordinate association system according to the fourth embodiment. FIG. 14 is a flowchart illustrating an example of the operation of the coordinate conversion table creation device according to the fourth embodiment. FIG. 15 is a schematic diagram illustrating an example of extraction of measurement point candidates in the fourth embodiment. FIG. 16 is a block diagram illustrating an example of a configuration of a coordinate association system according to the sixth embodiment. FIG. 17 is a flowchart illustrating an example of the operation of the coordinate conversion table creation device according to the sixth embodiment. FIG. 18 is an explanatory diagram illustrating an example of a measurement point candidate and a calculated shortest path in the sixth embodiment. FIG. 19 is a block diagram illustrating an example of a configuration of a coordinate association system according to the seventh embodiment. FIG. 20 is a flowchart illustrating an example of the operation of the coordinate conversion table creation device according to the seventh embodiment. FIG. 21 is a schematic diagram for explaining the operation in step S606 in the seventh embodiment. FIG. 22 is a block diagram illustrating an example of a configuration of a coordinate association system according to the eighth embodiment. FIG. 23 is a flowchart illustrating an example of the operation of the coordinate conversion table creation device according to the eighth embodiment. FIG. 24 is a schematic diagram for explaining the operation in step S708 in FIG. FIG. 25 is a block diagram illustrating an example of a hardware configuration of the coordinate conversion table creation device according to each embodiment of the present invention. FIG. 26 is a block diagram showing an example of the configuration of the information processing apparatus according to the embodiment of the present invention. FIG. 27 is a block diagram showing an example of the configuration of the information processing system according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Each drawing explains an embodiment of the present invention. However, the present invention is not limited to the description of each drawing. In addition, repeated description of the same configuration in each drawing may be omitted. Further, in the drawings used for the following description, the description of the configuration of the part not related to the description of the present invention is omitted, and there are cases where it is not illustrated.

The information processing apparatus according to the embodiment of the present invention creates information related to coordinate transformation (correspondence) between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. The information processing apparatus is included in, for example, a predetermined information processing system (for example, a coordinate association system). The information processing system performs an operation related to coordinate transformation (correspondence) created by the information processing apparatus. Therefore, in the following description, as an example of the information processing apparatus, coordinates for creating a coordinate conversion table (correspondence relationship table) that is information on coordinate conversion (correspondence relationship) between coordinates in the camera coordinate system and coordinates in the world coordinate system. This will be described using a conversion table creation device. Further, as an example of the information processing system, a description will be given using a coordinate association system including a device that transmits information used for the operation of the coordinate conversion table creation device. However, the information processing apparatus according to each embodiment is not limited to the coordinate conversion table creation apparatus. Further, the information processing system according to each embodiment is not limited to the coordinate correspondence system. In addition, the following description is demonstrated using a coordinate conversion table. However, the coordinate conversion table may be called a correspondence table.

<First Embodiment>
FIG. 1 is a block diagram showing an example of a configuration of a coordinate association system 100 including a coordinate conversion table creation device 1 according to the first embodiment of the present invention. A coordinate association system 100 according to the first embodiment includes an imaging device 101, a three-dimensional terrain database storage unit 102, a coordinate conversion table creation device 1, a display device 104, a coordinate input device 105, and a storage device 107. including. The coordinate conversion table creation device 1 includes a small area grouping unit 103 and a coordinate conversion table creation unit 106.

The imaging apparatus 101 is an apparatus (for example, a camera) that is installed at a monitoring target point and images a monitoring target area. An image photographed by the photographing apparatus 101 is referred to as “photographed image data”.

The 3D terrain database storage unit 102 is a storage device that stores terrain data in the monitoring target area (hereinafter referred to as “3D terrain data”) in advance. The three-dimensional terrain data is a small area (for example, three points) formed based on point data including three-dimensional information (for example, latitude, longitude, and elevation values) and a predetermined number of points. Data). However, the 3D terrain data stored in the 3D terrain database storage unit 102 is not limited to the above. The data format of the three-dimensional terrain data is not particularly limited. The three-dimensional landform database storage unit 102 may store, for example, three-dimensional landform data using a database format.

FIG. 2 is a schematic diagram showing an example of 3D terrain data stored in the 3D terrain database storage unit 102. For example, the three-dimensional terrain database storage unit 102 may store a three-dimensional map database owned by the Geographical Survey Institute or the like, or three-dimensional terrain data created based on the topographic survey of the monitoring target area. Note that the 3D terrain database storage unit 102 includes at least data corresponding to a region (location) included in an image (captured image data) captured by the imaging apparatus 101 as 3D terrain data.

The small area grouping unit 103 divides small areas included in the 3D terrain data into groups in the 3D terrain data corresponding to the locations included in the captured image data. That is, the small area grouping unit 103 creates a group in the small area included in the three-dimensional terrain data corresponding to the captured image data. Hereinafter, the created small area group may be simply referred to as a “small area group”.

The display device 104 displays 3D terrain data grouped into small area groups.

The coordinate input device 105 acquires information (candidate information) related to coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system regarding the three-dimensional terrain data displayed on the display device 104. For example, the coordinate input device 105 may include an input device (not shown) and acquire an operator's input operation as information related to coordinate conversion. The coordinate input device 105 acquires information (candidate information) regarding coordinate conversion between the camera coordinate system and the world coordinate system for at least four arbitrary points within the small region for each small region group.

The coordinate conversion table creation unit 106 converts the coordinates of the camera coordinate system, which are the coordinates in the captured image data, into the coordinates of the world coordinate system, which is the coordinates of the three-dimensional terrain data, using the small area group and information related to the coordinate conversion. A coordinate conversion table that is information to be created is created. That is, the coordinate conversion table is a table of information related to coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system in the captured image data.

In this way, the coordinate conversion table creation device 1 creates a coordinate conversion table.

The storage device 107 stores the created coordinate conversion table.

The small area grouping unit 103 and the coordinate conversion table creating unit 106 are realized by using, for example, a CPU (Central Processing Unit) of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. Here, the elements are the small region grouping unit 103 and the coordinate conversion table creation unit 106. The computer will be described in detail later.

Next, the operation of the first embodiment will be described.

FIG. 3 is a flowchart showing an example of the operation in the coordinate conversion table creating apparatus 1 according to the first embodiment. Note that the image capturing apparatus 101 captures the monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 106.

The small area grouping unit 103 acquires the 3D terrain data corresponding to the location included in the captured image data from the 3D terrain database storage unit 102 (step S101).

The small area grouping unit 103 performs grouping (grouping) on the small areas of the acquired three-dimensional terrain data based on the normal direction and elevation information in each small area included in the acquired three-dimensional terrain data ( Step S102).

FIG. 4 is a schematic diagram illustrating an example of a small region group according to the first embodiment.

4 is an example when the small region grouping unit 103 performs grouping using the following method. First, the small area grouping unit 103 obtains normal vectors (for example, normalized normal vectors) of all small areas included in the target three-dimensional terrain data and the average elevation of the small areas. Then, the small area grouping unit 103 divides all small areas into pairs according to a predetermined rule. Then, the small region grouping unit 103 calculates the inner product of the normal vectors and the elevation difference (the above-described difference in average elevation) for each pair of all the small regions. The small area grouping unit 103 uses a pair of small areas in which the inner product and the altitude difference are included in a certain range of values as grouping targets. In detail, the small area grouping unit 103 selects a small area pair whose normal direction is substantially the same using the inner product value, and further uses a difference in altitude to form a pair of small areas having the same altitude. Select. In other words, the small area grouping unit 103 selects a pair of small areas that have substantially the same elevation in substantially the same direction. Then, the small area grouping unit 103 groups the small area pairs using the altitude. Note that the small area grouping unit 103 may select a small area whose normal direction is a predetermined direction (for example, a vertical direction) as a processing target, and execute the grouping. For example, as illustrated in FIG. 4, the small region grouping unit 103 may select a substantially horizontal small region (small region pair) and divide it into groups. In FIG. 4, a set of small areas with low elevation is a set of small areas included in group 1. A set of small areas whose altitude is higher than a certain value than the altitude of group 1 is a set of small areas included in group 2.

Note that the method of grouping small areas is not limited to the above method. The small area grouping method may be any grouping technique that uses the normal direction and elevation information of each small area, and may be an existing grouping technique or clustering technique.

The display device 104 displays 3D terrain data grouped into small area groups.

The coordinate input device 105 acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system for each small region group with respect to the three-dimensional terrain data displayed on the display device 104 (step). S103). For example, the coordinate input device 105 may include an input device (not shown) and acquire an operator's input operation as information related to coordinate conversion. The coordinate input device 105 acquires information related to coordinate conversion between the camera coordinate system and the world coordinate system for at least four arbitrary points within the small region for each small region group.

Next, for each small region group, the coordinate conversion table creation unit 106 coordinates the coordinates of the camera coordinate system, which is the coordinate system of the imaging apparatus 101, and the coordinates of the world coordinate system (for example, latitude, longitude, and altitude). Ask for conversion. Specifically, the coordinate conversion table creation unit 106 obtains coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system for all pixels on which the ground surface in the captured image is displayed. Then, the coordinate conversion table creation unit 106 creates a coordinate conversion table including the coordinate conversion of all the small area groups based on the calculated coordinate conversion (step S104).

The coordinate conversion table creation unit 106 may obtain coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system as follows, for example. The coordinate conversion table creation unit 106 uses information regarding coordinate conversion between the camera coordinate system and the world coordinate system of each plane acquired by the coordinate input device 105 and the plane that approximates the small region group. For example, the coordinate conversion table creation unit 106 obtains a plane projective conversion parameter for performing plane projective conversion to the coordinates of a plane that approximates the coordinates of the camera coordinate system for each small region group. Then, the coordinate conversion table creation unit 106 performs planar projection corresponding to the small region group with respect to the position of the pixel corresponding to the ground surface in the captured image data in which the location corresponding to the small region group is captured (the coordinates in the camera coordinate system). A coordinate transformation using the transformation parameter is calculated. As described above, the coordinate conversion table creation unit 106 creates a coordinate conversion table using coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system.

Note that the parameters related to the coordinate conversion of the small area group are not limited to the parameters of the planar projective conversion. The parameter relating to coordinate transformation may be a parameter that is an approximation of affine transformation or other linear transformation. In the case of complex terrain, the coordinate conversion table creation unit 106 obtains continuous coordinate conversion for pixels in each small region group using nonlinear conversion such as spline interpolation or TPS (ThinＳPlate Spline) interpolation. May be. In addition, when applying each transformation other than the planar projective transformation, the coordinate input device 105 acquires information related to the coordinate transformation between the camera coordinate system and the world coordinate system regarding the number of points necessary and sufficient to estimate each transformation parameter. Assuming that

The coordinate conversion table creation unit 106 stores the created coordinate conversion table in the storage device 107.

As described above, the small area grouping unit 103 divides the small areas into groups (small area groups) that can be approximated to a plane based on the normal direction and the elevation information. Then, the coordinate conversion table creation unit 106 uses the plane that approximates the small region group to convert the coordinates of the camera coordinate system of the imaging device 101 arranged at the monitoring target point and the coordinates of the world coordinate system. Create a conversion table. As described above, the coordinate conversion table creation device 1 can create a coordinate conversion table using conversion using a plane (for example, plane projection conversion) regardless of the topography of the monitoring target region. For this reason, the coordinate conversion table creation device 1 can determine the coordinates of the camera coordinate system and the world even if the monitoring target area is an area where no landmark is present, a complex terrain with undulations, or an area where a survey vehicle cannot travel. Coordinate conversion with the coordinates of the coordinate system can be realized.

Various types of usage methods are assumed for the coordinate conversion table stored in the storage device 107.

For example, the coordinate conversion table can be used to measure the position coordinates of an object such as a suspicious vehicle.

FIG. 5 is a block diagram illustrating an example of a configuration of the object position measurement device 2001 and the coordinate association system 100 according to the first embodiment. The object position measuring apparatus 2001 illustrated in FIG. 5 is suspicious in the captured image data captured by the imaging apparatus 101 using the coordinate conversion table created by the coordinate conversion table creating apparatus 1 and stored in the storage device 107. Measure the position coordinates of an object such as a vehicle. Therefore, the object position measurement device 2001 includes an object detection device 2002 and an object position coordinate conversion device 2003. The object detection device 2002 detects an object in the captured image data captured by the imaging device 101. The object position coordinate conversion device 2003 uses the coordinate conversion table to convert object coordinates (camera coordinate system coordinates) in the captured image data into world coordinate system coordinates. As described above, the object position measuring apparatus 2001 measures the position of the object (for example, coordinates such as latitude, longitude, and altitude) in the captured image data captured by the image capturing apparatus 101.

FIG. 6 illustrates a situation in which coordinates in the camera coordinate system (position in the captured image data) of an object such as a suspicious vehicle in the object position measurement apparatus 2001 according to the first embodiment are converted into coordinates in the world coordinate system. It is a schematic diagram. The coordinates of the camera coordinate system of each car included in the photographed image data shown on the left side of FIG. 6 are converted into the coordinates of the world coordinate system shown on the right side of FIG.

Note that the coordinate conversion table stored in the storage device 107 includes a setting change of the photographing apparatus 101 (a change in resolution, a change in focus or angle of view accompanying a lens change, or a change in position or orientation of the photographing apparatus). It does not need to be updated unless it occurs. Therefore, when the coordinate conversion table is stored in the storage device 107, the object detection device 2002 can operate using the coordinate conversion table unless the photographing apparatus 101 is changed. That is, the object detection device 2002 uses the captured image data obtained from the imaging device 101 even after the coordinate conversion table creation device 1 does not operate unless the imaging device 101 is changed after the creation of the coordinate conversion table. The coordinates of the contained object in the world coordinate system can be obtained. Here, the object is, for example, a suspicious person or a suspicious vehicle.

[Description of effects]
The effect of the first embodiment will be described.

The coordinate conversion table creation device 1 according to the first embodiment can achieve the effect of calculating coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system regardless of the target region.

The reason is as follows.

The small area grouping unit 103 divides the small areas in the 3D terrain data corresponding to the captured image data into groups based on the 3D terrain data stored in the 3D terrain database storage unit 102. More specifically, the small area grouping unit 103 divides the small areas into groups using the normal direction and the altitude of the small areas. Then, the coordinate conversion table creation unit 106 calculates coordinate conversion (coordinate conversion table) for converting the coordinates of the camera coordinate system into the coordinates of the world coordinate system for each group of small regions. Here, the small area grouping unit 103 divides the small areas into small area groups so as to be substantially flat based on the normal direction and the altitude. Therefore, the coordinate conversion table creation unit 106 can create a coordinate conversion table for converting the coordinates of the camera coordinate system to the coordinates of the world coordinate system using a plane that approximates each small region group. Thus, the coordinate conversion table creation device 1 is for creating a coordinate conversion table using plane approximation regardless of the target region.

<Second Embodiment>
FIG. 7 is a block diagram illustrating an example of a configuration of the coordinate association system 200 according to the second embodiment. The coordinate association system 200 includes an imaging device 201, a three-dimensional landform database storage unit 202, a coordinate conversion table creation device 2, a display device 205, a coordinate input device 206, and a storage device 208. The coordinate conversion table creation device 2 includes a position / orientation acquisition unit 203, a small area grouping unit 204, and a coordinate conversion table creation unit 207.

The imaging device 201, the three-dimensional terrain database storage unit 202, and the storage device 208 are the same as the imaging device 101, the three-dimensional terrain database storage unit 102, and the storage device 107, respectively, according to the first embodiment. However, the imaging device 201 outputs information regarding the imaging device 201 to the coordinate conversion table creation device 2 in addition to the captured image data. The information related to the imaging apparatus 201 includes information related to the position of the imaging apparatus 201 (position information) and information related to the orientation of the imaging apparatus 201. The position information is position information (for example, latitude, longitude, and altitude) in the world coordinate system of the photographing apparatus 201. Further, the information related to the orientation of the image capturing apparatus 201 is information related to the image capturing direction of the image capturing apparatus 201 (orientation information).

The position / orientation acquisition unit 203 acquires position information and direction information from the photographing apparatus 201.

The small area grouping unit 204 generates a small area of the 3D terrain data based on the 3D terrain data corresponding to the location included in the captured image data and the distance from the imaging device 201 to the location (3D terrain data). Create a group.

The display device 205 displays 3D terrain data grouped into small area groups.

The coordinate input device 206 acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system regarding the three-dimensional terrain data displayed on the display device 205. For example, the coordinate input device 206 may include an input device (not shown) and acquire an operator's input operation as information related to coordinate conversion. The coordinate input device 206 acquires information related to coordinate conversion between the camera coordinate system and the world coordinate system for at least four arbitrary points within the small region for each small region group.

The coordinate conversion table creation unit 207 creates a coordinate conversion table for converting the coordinates of the camera coordinate system in the captured image data into the coordinates of the world coordinate system using the small area group and the information related to the coordinate conversion.

The position / orientation acquisition unit 203, the small area grouping unit 204, and the coordinate conversion table creation unit 207 are realized using, for example, a CPU of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The computer will be described in detail later.

Next, the operation of the second embodiment will be described. Note that detailed description of operations similar to those of the first embodiment is omitted as appropriate.

FIG. 8 is a flowchart showing an example of the operation of the coordinate conversion table creation device 2 according to the second embodiment. Note that the image capturing apparatus 201 captures the monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 207.

The position / orientation acquisition unit 203 acquires position information and direction information of the photographing apparatus 201. Similar to the small region grouping unit 103 of the first embodiment, the small region grouping unit 204 acquires three-dimensional landform data corresponding to the location of the captured image data from the three-dimensional landform database storage unit 202 (step S201). .

The position / orientation acquisition unit 203 may acquire the position information of the image capturing apparatus 201 using, for example, Exif (Exchangeable image file format) information of the captured image data. Further, the position / orientation acquisition unit 203 may acquire position information of the image capturing apparatus 201 from a GPS attached to the image capturing apparatus 201. Further, the position / orientation acquisition unit 203 may acquire the orientation information of the imaging apparatus 201 from a magnetic compass attached to the imaging apparatus 201. However, the method by which the position / orientation acquisition unit 203 acquires the position information and the direction information of the imaging apparatus 201 is not limited to the above. The position / orientation acquisition unit 203 may acquire the position information and the direction information of the photographing apparatus 201 from other sensors (not shown). Further, the position / orientation acquisition unit 203 may acquire the position information and the direction information of the photographing apparatus 201 based on an operation of an operator on an input device (not shown).

Similar to the small region grouping unit 103 of the first embodiment, the small region grouping unit 204 groups small regions of the acquired three-dimensional terrain data based on the normal direction and elevation information of each small region (step S202).

Next, the small area grouping unit 204 obtains a distance from the photographing apparatus 201 to the small area for each small area based on the position information and the azimuth information of the photographing apparatus 201 acquired by the position / orientation acquisition unit 203. The small area grouping unit 204 groups the small areas again based on the distance from the photographing apparatus 201 to the small area (step S203). In other words, the small area grouping unit 204 further subdivides the group created in step S202 according to the distance from the imaging device 201 to the small area. Thus, the small region grouping unit 204 divides at least a part of the groups into a plurality of groups based on the distance.

FIG. 9 is a schematic diagram showing an example of a group obtained as a result of step S203 in FIG. 9, in FIG. 9, a set of small areas in a short distance in a small area having a low altitude is group 1. In a small area with a low altitude, a group of small areas at a long distance is group 2. In a small area with a high altitude, a set of small areas at a short distance (left side in FIG. 9) is Group 3. A group of small areas at a long distance (on the right side in FIG. 9) in the small area having a high altitude is group 4. In

groups

1 and 2, the normal vector and the elevation are the same. However, group 1 and group 2 are divided into two groups based on the distance from the photographing apparatus 101.

The display device 205 displays the three-dimensional terrain data grouped in the subdivided small area groups.

The coordinate input device 206 acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system for each small region group with respect to the three-dimensional terrain data displayed on the display device 205 (step). S204). For example, the coordinate input device 206 may include an input device (not shown) and acquire an operator's input operation as information related to coordinate conversion. The coordinate input device 206 acquires information related to coordinate conversion between the camera coordinate system and the world coordinate system for at least four arbitrary points within the small region for each small region group.

The coordinate conversion table creation unit 207 creates a coordinate conversion table based on the small area group, similarly to the coordinate conversion table creation unit 106 of the first embodiment (step S205).

The coordinate conversion table creation unit 207 stores the created coordinate conversion table in the storage device 208.

[Description of effects]
The effect of the second embodiment will be described. In the second embodiment, in addition to the effects of the first embodiment, conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system can be obtained with higher accuracy. For example, even when an object far from the image capturing device 201 is detected in the captured image data, the coordinate conversion table creating device 2 can obtain the position coordinates of the object in the world coordinate system with high accuracy. The reason is as follows.

The small area grouping unit 204 groups small areas using the distance. That is, the small area group of the second embodiment includes small areas having the same distance from the imaging device 201. When the range of coordinates to be subjected to each process of coordinate conversion is wide, the accuracy of the coordinate conversion obtained by the coordinate conversion table creation unit 207 may be reduced. However, the coordinate conversion table creation unit 207 creates a coordinate conversion table using the small area groups grouped based on the distance. Therefore, the coordinate conversion table creation device 2 can create a coordinate conversion table with high accuracy. Therefore, even when an object far from the photographing apparatus 201 is detected in the photographed image data, the coordinate conversion table creating apparatus 2 creates a coordinate conversion table using a small area group that is substantially the same distance as the object. is doing. Therefore, the coordinates of the object in the world coordinate system are calculated with high accuracy.

In the following description of each embodiment, a case where small areas are grouped using a distance from the photographing apparatus to the small area will be described. However, each embodiment described below may be configured not to execute this operation. That is, in each embodiment described below, the small areas may be grouped without using the distance from the imaging device to the small areas.

<Third Embodiment>
FIG. 10 is a block diagram illustrating an example of a configuration of a coordinate association system 300 according to the third embodiment. A coordinate association system 300 according to the third embodiment includes an imaging device 301, a three-dimensional landform database storage unit 302, a coordinate conversion table creation device 3, a display device 306, a coordinate input device 307, and a storage device 309. including. The coordinate conversion table creation device 3 includes a position / orientation acquisition unit 303, a small area grouping unit 304, a measurement point extraction unit 305, and a coordinate conversion table creation unit 308.

The photographing device 301, the three-dimensional landform database storage unit 302, and the storage device 309 are the same as the photographing device 201, the three-dimensional landform database storage unit 202, and the storage device 208 in the second embodiment. The position / orientation acquisition unit 303 and the small region grouping unit 304 are the same as the position / orientation acquisition unit 203 and the small region grouping unit 204 in the second embodiment.

The measurement point extraction unit 305 extracts measurement point candidates in each small region group based on the small region grouping result in the small region grouping unit 304. A measurement point candidate is a point from which information (candidate information) regarding coordinate transformation between the coordinates of the camera coordinate system and the corresponding coordinates (position) of the world coordinate system is acquired. A measurement point candidate may be simply referred to as a “measurement point”. Note that the measurement point candidates are displayed on the display device 306, as will be described later.

Display device 306 displays measurement point candidates. For example, the display device 306 indicates a measurement point candidate to the worker.

The coordinate input device 307 is a device that acquires candidate information on measurement point candidates. For example, the coordinate input device 307 is a device for acquiring the candidate information based on the operation of the worker. Here, the operator's operation specifically includes information (coordinate conversion information) about a pair of coordinates (the coordinates of the camera coordinate system and the coordinates of the world coordinate system) in the measurement point candidate displayed on the display device 306. ) Input. That is, the coordinate input device 307 acquires information (candidate information) related to coordinate conversion in a coordinate pair of the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the measurement point candidate.

The coordinate conversion table creation unit 308 acquires candidate information related to the measurement point candidate from the coordinate input device 307, and based on the acquired candidate information, coordinate conversion related to coordinate conversion from the coordinates of the camera coordinate system to the coordinates of the world coordinate system. Create a table.

The position / orientation acquisition unit 303, the small area grouping unit 304, the measurement point extraction unit 305, and the coordinate conversion table creation unit 308 are realized using, for example, a CPU of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The CPU executes an operation of transmitting information (candidate for measurement point) to the display device 306 and an operation of acquiring information (candidate information) from the coordinate input device 307 based on the coordinate conversion program. This also applies to other embodiments described later. The computer will be described in detail later.

Next, the operation of the third embodiment will be described. Note that detailed description of operations similar to those in the first and second embodiments is omitted as appropriate.

FIG. 11 is a flowchart showing an example of the operation of the coordinate conversion table creation device 3 according to the third embodiment. Note that the imaging apparatus 301 captures the monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 308.

The position / orientation acquisition unit 303 acquires the position information and the direction information in the world coordinate system of the image capturing apparatus 301 in the same manner as the position / orientation acquisition unit 203 of the second embodiment. Further, the small area grouping unit 304 acquires the three-dimensional terrain data corresponding to the location of the captured image data from the three-dimensional terrain database storage unit 302, similarly to the small region grouping unit 204 of the second embodiment (step S301). ).

Similar to the small region grouping unit 204 of the second embodiment, the small region grouping unit 304 divides the small regions of the acquired three-dimensional terrain data into groups based on the normal direction and elevation information of each small region ( Step S302).

Next, as in the second embodiment, the small region grouping unit 304 is a place corresponding to a small region from the imaging device 301 based on the position information and orientation information of the imaging device 301 acquired by the position / orientation acquisition unit 303. Find the distance to. The small area grouping unit 304 groups the small area groups again based on the distance from the photographing apparatus 301 to the place corresponding to the small area (step S303).

The measurement point extraction unit 305 extracts a measurement point candidate for each small area group (step S304).

FIG. 12 is a schematic diagram for explaining a method of extracting measurement point candidates by the measurement point extraction unit 305 in the third embodiment. However, FIG. 12 shows the measurement point candidates by using the group shown in FIG. 4 instead of the group regrouped based on the distance shown in FIG. 9 for easy understanding of the operation. It was.

The measurement point extraction unit 305 calculates the position (coordinates) of the center of gravity for each small area group. Next, the measurement point extraction unit 305 selects a plurality of points whose distance from the center of gravity is a certain value or more. Then, the measurement point extraction unit 305 extracts, as measurement point candidates, a predetermined number of points at which the angle formed by all line segments connecting each point and the center of gravity is equal to or greater than a predetermined angle from the selected points. That is, the measurement point extraction unit 305 extracts the measurement point candidate points so that they are separated from the center of gravity of the small region group by a predetermined distance or more and are not adjacent to each other. As described above, the measurement point extraction unit 305 extracts a plurality of measurement point candidates for each small region group. The predetermined number of measurement point candidates extracted here is a predetermined number or more, or a predetermined number of ranges. Then, the measurement point extraction unit 305 transmits the measurement point candidate to the display device 306.

The display device 306 displays the acquired measurement point candidates. At this time, the display device 306 may display the measurement point candidates together with the three-dimensional terrain data, as shown in FIG.

The coordinate input device 307 acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system in the measurement point candidates displayed on the display device 306 (step S305). For example, the coordinate input device 307 may include an input device (not shown) and acquire an operator's input operation as information related to coordinate conversion.

For example, the worker confirms the captured image data displayed on the display device 306 and the map data or the aerial image. Then, the operator may input information (candidate information) related to coordinate transformation between the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the position corresponding to the measurement point candidate to the coordinate input device 307. In addition, the worker has a sensor (for example, GPS) and a marker that can acquire coordinates in the world coordinate system, and goes to a measurement point candidate. And an operator measures the coordinate of the world coordinate system of a measurement point candidate using a sensor. Further, the operator places a marker on the measurement point candidate. The imaging device 301 acquires captured image data in this state. Based on such an operation, the operator obtains information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the position corresponding to the measurement point candidate, and sends the information to the coordinate input device 307. You may enter.

The coordinate input device 307 transmits information (candidate information) related to coordinate conversion of the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the position corresponding to the acquired measurement point candidate to the coordinate conversion table creation unit 308.

In addition to the operation of the coordinate conversion table creation unit 207 of the second embodiment, the coordinate conversion table creation unit 308 uses the acquired candidate information to calculate the coordinates of the camera coordinate system of the small area group and the coordinates of the world coordinate system. A coordinate conversion table is created (step S306). That is, the coordinate conversion table creation unit 308 creates a coordinate conversion table using information (candidate information) related to coordinate conversion of measurement point candidates.

The coordinate conversion table creation unit 308 causes the storage device 309 to store the created coordinate conversion table.

[Description of effects]
The effect of the third embodiment will be described. In addition to the effects of the first and second embodiments, the third embodiment has the effect of further improving accuracy. The reason is as follows.

The measurement point extraction unit 305 of this embodiment extracts measurement point candidates in each small area group. Then, the coordinate conversion table creation unit 308 acquires information (candidate information) related to coordinate conversion in the above measurement point candidates. Here, the candidate information is information relating to coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the measurement point candidate. The candidate information includes, for example, coordinate information measured by an operator or the like. That is, the candidate information (information regarding coordinate conversion) includes at least information regarding accurate coordinate conversion in the measurement point candidate. As described above, the coordinate conversion table creation device 3 creates the coordinate conversion table using the candidate information, so that the accuracy of the coordinate conversion table can be improved.

For example, there are cases where the resolution (resolution) of available 3D terrain data is low, or there are many measurement errors included in the 3D terrain data. However, even in such a case, the coordinate conversion table creation device 3 can improve the accuracy of the coordinate conversion table using the candidate information.

Also, the acquired candidate information includes information related to accurate coordinate conversion in the small area group. In general, the processing amount for ensuring a predetermined accuracy in coordinate transformation is generally inversely proportional to the accuracy of the initial value. When the accuracy of the initial value is high, the processing amount decreases. Therefore, the coordinate conversion table creation device 3 can reduce the calculation cost when creating the coordinate conversion table using the acquired candidate information.

<Fourth Embodiment>
FIG. 13 is a block diagram illustrating an example of a configuration of a coordinate association system 400 according to the fourth embodiment. A coordinate association system 400 according to the fourth embodiment includes an imaging device 401, a three-dimensional terrain database storage unit 402, a coordinate conversion table creation device 4, a display device 407, a coordinate input device 408, and a storage device 410. including. Furthermore, the coordinate matching system 400 of the fourth embodiment includes a measurement ease database storage unit 405. The coordinate conversion table creation device 4 includes a position / orientation acquisition unit 403, a small area grouping unit 404, a measurement point extraction unit 406, and a coordinate conversion table creation unit 409.

The imaging device 401, the three-dimensional terrain database storage unit 402, and the storage device 410 are the same as the imaging device 301, the three-dimensional terrain database storage unit 302, and the storage device 309 in the third embodiment. The display device 407 and the coordinate input device 408 are the same as the display device 306 and the coordinate input device 307 in the third embodiment. The position / orientation acquisition unit 403, the small region grouping unit 404, and the coordinate conversion table creation unit 409 are the same as the position / orientation acquisition unit 303, the small region grouping unit 304, and the coordinate conversion table creation unit 308 in the third embodiment. It is.

The measurement ease database storage unit 405 is a storage device that stores in advance data obtained by quantifying the ease of measurement of coordinates in the world coordinate system at each point included in the three-dimensional terrain data. Note that the storage format of data obtained by digitizing the measurement ease is not particularly limited. For example, the measurement ease database storage unit 405 may store measurement ease data as a database format. Hereinafter, data obtained by quantifying measurement ease may be simply referred to as “measurement ease data”.

計測 Further explanation of the measurement ease data. For example, a region such as a pond or a swamp is a region where it is difficult for an operator to go to that point and measure the coordinates of the world coordinate system. Thus, the measurement ease data of the points included in the difficult region is a small value. A region such as a flat land or a road is a region where an operator can easily go to that point and measure the coordinates of the world coordinate system. The measurement ease data of the points included in such a region is a large value. As described above, in the following description, it is assumed that a point where the coordinates of the world coordinate system are easy to measure has a larger value of measurement ease data.

The measurement point extraction unit 406 extracts measurement point candidates in the small region group based on the small region group grouped by the small region grouping unit 404 and the measurement ease data. That is, the measurement point extraction unit 406 extracts measurement point candidates that can be easily measured.

The position / orientation acquisition unit 403, the small area grouping unit 404, the measurement point extraction unit 406, and the coordinate conversion table creation unit 409 are realized using a CPU of a computer that operates based on a coordinate conversion program, for example. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The computer will be described in detail later.

Next, the operation of the fourth embodiment will be described. Note that detailed descriptions of operations similar to those in the first to third embodiments are omitted as appropriate.

FIG. 14 is a flowchart showing an example of the operation of the coordinate conversion table creation device 4 according to the fourth embodiment. Note that the imaging apparatus 401 captures the monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 409.

The position / orientation acquisition unit 403 acquires the position information and the direction information in the world coordinate system of the imaging apparatus 401, as with the position / orientation acquisition unit 303 of the third embodiment. In addition, the small area grouping unit 404 acquires the 3D terrain data corresponding to the location of the captured image data from the 3D terrain database storage unit 402, similarly to the small area grouping unit 304 of the third embodiment (step S401). ).

Similar to the small region grouping unit 304 of the third embodiment, the small region grouping unit 404 groups small regions of the acquired three-dimensional terrain data based on the normal direction and elevation information of each small region (step) S402).

Next, as in the third embodiment, the small region grouping unit 404 calculates the distance from the image capturing device 401 to the small region based on the position information and the direction information of the image capturing device 401 acquired by the position / orientation acquiring unit 403. Ask. The small area grouping unit 404 divides the small area group according to the distance from the photographing apparatus 401 to the small area (step S403).

The measurement point extraction unit 406 extracts a plurality of measurement point candidates for each small region group, similarly to the measurement point extraction unit 305 of the third embodiment (step S404).

Next, the measurement point extraction unit 406 ranks the measurement point candidates extracted in step S404 based on the measurement ease data of the measurement point candidates. The measurement point extraction unit 406 performs this process for each small area group. That is, the measurement point extraction unit 406 ranks the measurement point candidates in the order in which the coordinates of the world coordinate system are easily measured in each small region group.

It should be noted that the value of the measurement ease data is larger at a point where the coordinates of the world coordinate system are easier to measure. Therefore, the measurement point extraction unit 406 may rank the measurement point candidates in descending order of the measurement ease data for each small region group. Then, the measurement point extraction unit 406 extracts up to a predetermined number of measurement point candidates for each small region group based on the rank (step S405).

For example, it is assumed that the measurement point extraction unit 406 is set to extract measurement point candidates from the highest level to a predetermined number. In this case, the measurement point extraction unit 406 extracts (selects) the measurement point candidates from the highest level to the predetermined number for each small region group. However, the aspect of extracting the measurement point candidates based on the rank is not limited to the above. For example, the measurement point extraction unit 406 may extract measurement point candidates whose measurement ease data value is greater than a predetermined threshold.

FIG. 15 is a schematic diagram showing an example of extraction of measurement point candidates in the fourth embodiment. FIG. 15 shows a measurement point candidate that is desired to be extracted (selected) with a low rank as a triangular marker, and shows the extracted measurement point candidate using an X-type mark. The upper part of FIG. 15 shows positions corresponding to measurement point candidates in the captured image data. Further, the lower diagram in FIG. 15 shows the positions of the measurement point candidates in the three-dimensional terrain data. However, the lower figure of FIG. 15 is not shown in FIG. 4 but a group regrouped based on the distance shown in FIG. Using groups, candidate measurement points were shown.

In the above description, the measurement point extraction unit 406 has extracted a measurement point candidate in step S404, and a measurement point candidate is further extracted from the measurement point candidates in step S405. However, the operation of the measurement point extraction unit 406 is not limited to such an operation. For example, the measurement point extracting unit 406 may extract the measurement point candidates from the top to the predetermined order in descending order of the value of the measurement ease data for each small region group without executing Step S404. The operation of the measurement point extraction unit 406 is the same in each embodiment described later.

The display device 407 displays the measurement point candidates extracted (selected) in step S405, similarly to the display device 306 of the third embodiment.

The coordinate input device 408 operates in the same manner as the coordinate input device 307 of the third embodiment. That is, the coordinate input device 408 obtains information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system in the measurement point candidates displayed on the display device 407 (step S406).

The coordinate conversion table creation unit 409 creates a coordinate conversion table between the coordinates of the camera coordinate system and the coordinates of the world coordinate system, using the acquired candidate information, similarly to the coordinate conversion table creation unit 308 of the third embodiment. (Step S407).

The coordinate conversion table creation unit 409 causes the storage device 410 to store the created coordinate conversion table.

[Description of effects]
The effect of the fourth embodiment will be described. In addition to the effects of the first to third embodiments, the fourth embodiment has an effect of facilitating acquisition of candidate information for further improving accuracy. The reason is as follows.

The measurement point extraction unit 406 of the fourth embodiment extracts measurement point candidates that can be easily measured based on the measurement ease data stored in the measurement ease database storage unit 405. Therefore, the candidate information acquired by the coordinate input device 408 is information that can be easily obtained. Therefore, 4th Embodiment also has an effect of reducing a worker's load.

Also, in general, measurement at a point that is easy to measure is easier to measure with higher accuracy than measurement at a point where measurement is difficult. Therefore, the coordinate conversion table creation unit 409 can acquire highly accurate candidate information.

In the following description of each embodiment, as in the case of the fourth embodiment, a case will be described in which measurement point candidates are ranked, and measurement point candidates are extracted based on the ranks. However, each embodiment described below may be configured not to perform this operation.

<Fifth Embodiment>
Next, a coordinate association system 400 according to the fifth embodiment will be described. The configuration of the coordinate association system 400 according to the fifth embodiment is the same as that of the coordinate association system 400 according to the fourth embodiment. Therefore, the fifth embodiment will be described below with reference to FIG.

Further, in the coordinate association system 400 according to the fifth embodiment, elements other than the coordinate conversion table creation unit 409 are the same as those in the fourth embodiment. Therefore, the description of the same configuration as that of the fourth embodiment is omitted.

The coordinate conversion table creation unit 409 creates a coordinate conversion table for at least one small area group, and then performs coordinate conversion in the remaining small area groups based on the positional relationship between the small area group and the remaining small area groups. presume. Then, the coordinate conversion table creation unit 409 creates a coordinate conversion table for the remaining small region groups based on the estimated coordinate conversion.

The operation of the fifth embodiment is different from the operations of steps S406 and S407 in the flowchart of FIG. 14 showing the operation of the fourth embodiment.

Therefore, description of the operations in steps S401 to S405 in the fifth embodiment is omitted.

The display device 407 according to the fifth embodiment displays the measurement point candidates extracted in step S405 in the same manner as the display device 407 according to the fourth embodiment.

The coordinate input device 408 of the fifth embodiment acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system with respect to the measurement point candidates of at least one small region group (steps). S406).

The coordinate conversion table creation unit 409 obtains coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system with respect to the small area group (hereinafter referred to as “small area group _A ”) corresponding to the acquired candidate information. . More specifically, for the small area group _A , the coordinate conversion table creation unit 409 obtains a coordinate conversion parameter for coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. Then, the coordinate conversion table generation unit 409, by using the coordinate transformation parameters for all the pixels surface of the region corresponding to the small region group _A is displayed, to create the coordinate conversion table.

Next, the coordinate conversion table creation unit 409 performs coordinate conversion (coordinate conversion parameters) in another small region group (hereinafter referred to as “small region group _B ”) based on the relative positional relationship with the small region group _A. ).

For example, the coordinate conversion table creation unit 409 calculates a rotation matrix and a translation vector representing a relative positional relationship between the small area groups based on the three-dimensional terrain data. The coordinate conversion table creation unit 409 may calculate a rotation matrix and a translation vector in advance.

Then, the coordinate conversion table creation unit 409 calculates the coordinate conversion parameter in the small region group _B from the coordinate conversion parameter in the small region group _A based on the positional relationship between the small region group _B and the small region group _A. Specifically, the coordinate conversion table creation unit 409 calculates the coordinate conversion parameter in the small region group _B from the coordinate conversion parameter in the small region group _A using the rotation matrix and the translation vector representing the positional relationship of the small region group. To do.

Then, the coordinate conversion table creation unit 409 creates a coordinate conversion table for all the pixels on which the ground surface in the region corresponding to the small region group _B is displayed, using the calculated coordinate conversion parameter. As described above, the coordinate conversion table creation unit 409 creates a coordinate conversion table for the small area group _B based on the positional relationship between the small area groups and the coordinate conversion parameters for the small area group _A for which candidate information has been acquired ( Step S407).

[Description of effects]
The effect of the fifth embodiment will be described. In addition to the effects of the fourth embodiment, the fifth embodiment has the effect of facilitating acquisition of candidate information. The reason is as follows.

The coordinate conversion table creation unit 409 of the fifth embodiment estimates the coordinate conversion parameters of other small region groups based on at least some of the small region group candidate information, and the coordinate conversion table based on the estimated parameters. Create This is because the coordinate conversion table creation device 4 only needs to acquire candidate information for some of the measurement point candidates. Therefore, the fifth embodiment also has an effect of reducing the load on the worker.

In each of the embodiments described below, as in the fifth embodiment, a correspondence relationship regarding one or a part of small region groups may be used as the correspondence relationship of the small region groups.

<Sixth Embodiment>
FIG. 16 is a block diagram illustrating an example of a configuration of a coordinate association system 500 according to the sixth embodiment. A coordinate association system 500 according to the sixth embodiment includes an imaging device 501, a three-dimensional landform database storage unit 502, a coordinate conversion table creation device 5, a display device 508, a coordinate input device 509, and a storage device 511. including. Furthermore, the coordinate association system 500 of the sixth embodiment includes a measurement ease database storage unit 505. The coordinate conversion table creation device 5 includes a position / orientation acquisition unit 503, a small area grouping unit 504, a measurement point extraction unit 506, a route calculation unit 507, and a coordinate conversion table creation unit 510.

The photographing device 501, the three-dimensional terrain database storage unit 502, and the storage device 511 are the same as the photographing device 401, the three-dimensional terrain database storage unit 402, and the storage device 410 in the fourth embodiment. The measurement ease database storage unit 505, the display device 508, and the coordinate input device 509 are the same as the measurement ease database storage unit 405, the display device 407, and the coordinate input device 408 in the fourth embodiment. However, in the present embodiment, the display device 508 displays the shortest path as described later. The position / orientation acquisition unit 503, the small region grouping unit 504, and the coordinate conversion table creation unit 510 are the same as the position / orientation acquisition unit 403, the small region grouping unit 404, and the coordinate conversion table creation unit 409 in the fourth embodiment. It is. The measurement point extraction unit 506 is the same as the measurement point extraction unit 406 in the fourth embodiment.

The route calculation unit 507 calculates the shortest route that passes through all the measurement point candidates extracted by the measurement point extraction unit 506. Then, the route calculation unit 507 outputs (transmits) the calculated shortest route and measurement point candidate to the display device 508. The display device 508 displays the measurement point candidates and the shortest route.

The position / orientation acquisition unit 503, the small area grouping unit 504, the measurement point extraction unit 506, the route calculation unit 507, and the coordinate conversion table creation unit 510 are realized as follows, for example. These elements are realized using, for example, a CPU of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The computer will be described in detail later.

Next, the operation of the sixth embodiment will be described. Note that detailed description of operations similar to those in the first to fifth embodiments is omitted as appropriate.

FIG. 17 is a flowchart showing an example of the operation of the coordinate conversion table creation device 5 according to the sixth embodiment. Note that the image capturing apparatus 501 captures the monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 510.

The operations of steps S501 to S505 shown in FIG. 17 are the same as the operations of steps S401 to S405 in the fourth embodiment shown in FIG. Therefore, the detailed description is abbreviate | omitted.

In step S505, after acquiring the measurement point candidates extracted by the measurement point extraction unit 506, the route calculation unit 507 calculates the shortest route that passes through all the measurement point candidates based on the distance between the measurement point candidates (step S506). ). The method for calculating the shortest route in the measurement point extraction unit 506 is not particularly limited. For example, the route calculation unit 507 may calculate the shortest route as follows.

The route calculation unit 507 creates a graph in which measurement point candidates are nodes and nodes (measurement point candidates) are connected by edges. At this time, the route calculation unit 507 associates the distance information of the measurement point candidate with the edge. The route calculation unit 507 determines the shortest route that passes through all nodes (all measurement point candidates) for this graph using the Dijkstra method. Note that the route calculation unit 507 may consider not only distance information associated with an edge but also three-dimensional terrain data between nodes (measurement point candidates) in calculating the shortest route. For example, the route calculation unit 507 numerically represents the ease of travel between the nodes (measurement point candidates) (for example, the ease of travel of the vehicle whose coordinates are measured) based on the three-dimensional landform data between the nodes (measurement point candidates). Parameterized). Then, the route calculation unit 507 may perform weighting on the edge using the parameter. However, the route calculation unit 507 may use a route determination method other than the Dijkstra method.

The display device 508 displays the measurement point candidates extracted in step S505 and the shortest route calculated in step S506.

FIG. 18 is a diagram illustrating an example of a measurement point candidate and a calculated shortest route in the sixth embodiment.

When measuring the coordinates of a measurement point candidate, the operator moves a moving object having means for measuring the coordinates of the world coordinate system along the displayed shortest path, and uses the moving object to measure the measurement point candidate. Measure the coordinates of the world coordinate system corresponding to. For example, an operator who inputs candidate information to the coordinate input device 509 performs the measurement of the coordinates of the world coordinate system and the arrangement of the markers at the measurement candidate points described in the third embodiment along the displayed shortest path. Execute.

The coordinate input device 408 operates in the same manner as the coordinate input device 408 of the fourth embodiment. That is, the coordinate input device 408 acquires information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system in the measurement point candidates displayed on the display device 508 (step S508).

The coordinate conversion table creation unit 510 creates a coordinate conversion table between the coordinates of the camera coordinate system and the coordinates of the world coordinate system, using the acquired candidate information, similarly to the coordinate conversion table creation unit 409 of the fourth embodiment. (Step S509).

The coordinate conversion table creation unit 510 stores the created coordinate conversion table in the storage device 511.

[Description of effects]
The effect of the sixth embodiment will be described. In addition to the effects of the first to fourth embodiments, the sixth embodiment has the effect of further reducing the load for acquiring candidate information. The reason is as follows.

The route calculation unit 507 of the present embodiment outputs the measurement point candidate and the shortest route to the display device 508. Therefore, the display device 508 can display the shortest path passing through the measurement point candidate in addition to the measurement point candidate. Therefore, an operator who measures candidate information can move a moving body (for example, a measurement vehicle) used for measurement along the displayed shortest route, reduce the time required for measurement, and measure coordinates. it can.

Note that each embodiment described later may use the shortest path calculation as in the sixth embodiment.

<Seventh Embodiment>
FIG. 19 is a block diagram illustrating an example of a configuration of a coordinate association system 600 according to the seventh embodiment. A coordinate association system 600 according to the seventh embodiment includes a three-dimensional landform database storage unit 602, a coordinate conversion table creation device 6, a measurement ease database storage unit 605, and a storage device 611. Furthermore, the coordinate association system 600 of the seventh embodiment includes a display device 608 and a coordinate input device 609. A coordinate association system 600 according to the seventh embodiment includes a plurality of imaging devices 601. The coordinate conversion table creation device 6 includes a position / orientation acquisition unit 603, a small region grouping unit 604, a measurement point extraction unit 606, a measurement point grouping unit 607, and a coordinate conversion table creation unit 610.

The three-dimensional landform database storage unit 602, the measurement ease database storage unit 605, and the storage device 611 are the three-dimensional landform database storage unit 402, the measurement ease database storage unit 405, and the storage device 410 in the fourth embodiment. It is the same. The display device 608 and the coordinate input device 609 are the same as the display device 407 and the coordinate input device 408 in the fourth embodiment. However, the display device 608 displays the measurement point candidates for each photographing device 601. The small region grouping unit 604 and the measurement point extraction unit 606 are the same as the small region grouping unit 404 and the measurement point extraction unit 406 in the fourth embodiment.

The individual photographing devices 601 are the same as the photographing device 401 in the fourth embodiment.

The position / orientation acquisition unit 603 acquires position information and direction information in the world coordinate system of the image capturing device 601 for each image capturing device 601. The other points are the same as the position / orientation acquisition unit 403 in the fourth embodiment.

The coordinate conversion table creation unit 610 receives photographed image data for each photographing apparatus 601 and creates a coordinate conversion table. Regarding the other points, the coordinate conversion table creation unit 610 is the same as the coordinate conversion table creation unit 409 in the fourth embodiment.

The measurement point grouping unit 607 corresponds to each photographing apparatus 601 from the measurement point candidates extracted by the measurement point extraction unit 606 based on the reliability of coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. Create a group to be used. The reliability is a value indicating the degree of matching (for example, the magnitude of error) between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. The measurement point grouping unit 607 is not limited to a value used as the reliability. However, in the description of the present embodiment, as an example, a case will be described in which the distance between each imaging device 601 and a measurement point candidate is used as the reliability.

As the measurement point candidate is farther from the image capturing device 601, the error in the coordinate system of the world coordinate system (error in the camera depth direction) that is assumed to be a deviation per pixel of the image capturing device 601 (camera) increases. Therefore, the farther the measurement point candidate is from the imaging device 601, the greater the error in the coordinate relationship between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. In other words, the coordinate conversion table creation unit 610 can improve accuracy when operated using candidate information in measurement point candidates close to each imaging device 601. Therefore, the measurement point grouping unit 607 can use the distance between the imaging device 601 and the measurement point candidate as the reliability.

Therefore, the measurement point grouping unit 607 according to the present embodiment divides the measurement point candidates into groups corresponding to the respective image capturing apparatuses 601 based on the distance between the measurement point candidates and the image capturing apparatus 601. Specifically, the measurement point grouping unit 607 creates a group of measurement point candidates whose distance from the imaging device 601 is less than a predetermined threshold as a group for each imaging device 601. That is, the measurement point grouping unit 607 creates a group of measurement point candidates for each photographing apparatus 601 based on the reliability of the coordinates of the world coordinate system in the measurement point candidates and the coordinates of the camera coordinates in the image data.

The position / orientation acquisition unit 603, the small area grouping unit 604, the measurement point extraction unit 606, the measurement point grouping unit 607, and the coordinate conversion table creation unit 610 are realized as follows. These elements are realized using, for example, a CPU of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The computer will be described in detail later.

Next, the operation of the seventh embodiment will be described. Note that detailed description of operations similar to those in the first to sixth embodiments is omitted as appropriate.

FIG. 20 is a flowchart showing an example of the operation of the coordinate conversion table creation device 6 according to the seventh embodiment. Note that each imaging device 601 captures a monitoring target area, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 610.

The operations in steps S601 to S605 shown in FIG. 20 are the same as the operations in steps S401 to S405 in the fourth embodiment shown in FIG. 14, respectively, and detailed description thereof will be omitted.

However, in the present embodiment, there are a plurality of imaging devices 601.

Therefore, in step S603, the small area grouping unit 604 performs the same operation as that in step S403 in FIG. That is, the small area grouping unit 604 obtains the distance from the image capturing apparatus 601 to the small area for each image capturing apparatus 601. Then, the small region grouping unit 604 further divides the small region group grouped in step S602 into groups based on the distance from the photographing device 601 to the small region for each photographing device 601.

In step S604, the measurement point extraction unit 606 performs the same operation as that in step S404 in FIG. 14 for each photographing apparatus 601. That is, the measurement point extraction unit 606 performs a process of extracting measurement point candidates in the small area group for each photographing apparatus 601. Furthermore, in step S605, the measurement point extraction unit 606 executes the same operation as in step S405 of FIG. That is, the measurement point extraction unit 606 extracts measurement point candidates for each photographing apparatus 601.

Next, the measurement point grouping unit 607 divides the measurement point candidates into groups based on the distance to each image capturing device 601 for each image capturing device 601 (step S606). Specifically, the measurement point grouping unit 607 creates a group of measurement point candidates whose distance from the imaging device 601 is less than a threshold for each imaging device 601. That is, in step S606, the measurement point grouping unit 607 determines a measurement point candidate for each photographing apparatus 601.

FIG. 21 is a schematic diagram for explaining the operation in step S606 in the seventh embodiment. However, FIG. 21 uses the group shown in FIG. 4 instead of the group regrouped based on the distance shown in FIG. 9 in order to make the operation easy to understand. The measurement point candidate was shown.

FIG. 21 illustrates a case where there are two imaging devices 601. In the following description, the two photographing devices 601 are denoted by alphabetical signs and are distinguished as the photographing device 601 _a and the photographing device 601 _b .

Measurement point candidate shown with a square marker in FIG. 21 is a measurement point candidate distance from the photographing apparatus 601 _a and the imaging device 601 _b is less than the threshold. Measurement point candidates are shown using markers triangle, the distance from the imaging device 601 _a is less than the threshold value, a measurement point candidate distance from the photographing device 601 _b is greater than or equal to the threshold. X-type marker measurement point candidates are shown with the distance from the imaging device 601 _a and a is equal to or larger than the threshold, a measurement point candidate distance is less than the threshold value from the imaging device 601 _b.

Measurement point grouping unit 607, based on the distance from the imaging device 601 _a, as a group on imaging device 601 _a, selects the measurement point candidates are shown using markers square and triangle. That is, the measurement point grouping unit 607, a group for capturing apparatus 601 _a, excludes measurement point candidate indicated by X-type marker.

Similarly, measurement point grouping unit 607, based on the distance from the imaging device 601 _b, as a group on imaging device 601 _b, selects the measurement point candidates are shown with triangles, and X-type marker. That is, the measurement point grouping unit 607, as a group on imaging device 601 _b, excludes measurement point candidate indicated by the marker triangle.

The measurement point grouping unit 607 transmits to the display device 608 the measurement point candidates and information regarding the group for each photographing device 601. The display device 608 displays measurement point candidates included in the corresponding group for each photographing device 601.

The coordinate input device 609 operates in the same manner as the coordinate input device 408 of the fourth embodiment for each imaging device 601. That is, the coordinate input device 609 obtains information (candidate information) related to coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system in the measurement point candidates displayed on the display device 608 for each photographing device 601. (Step S608).

As with the coordinate conversion table creation unit 409 of the fourth embodiment, the coordinate conversion table creation unit 610 uses the acquired candidate information to determine the coordinates of the camera coordinate system and the coordinates of the world coordinate system for each imaging device 601. Coordinate conversion table is created (step S609).

The coordinate conversion table creation unit 610 stores the coordinate conversion table created for each photographing device 601 in the storage device 611.

[Description of effects]
The effect of the seventh embodiment will be described. In addition to the effects of the first to sixth embodiments, the seventh embodiment has the effect of further improving accuracy. The reason is as follows.

The measurement point grouping unit 607 creates a group of measurement point candidates for each photographing apparatus 601 based on the distance from each photographing apparatus 601. In other words, the measurement point grouping unit 607 creates a group that does not include measurement point candidates for which the reliability of coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system is low. Then, the measurement point grouping unit 607 transmits the measurement point candidates and the created group to the display device 608.

Display device 608 displays the measurement point candidates and groups. That is, the display device 608 displays a measurement point candidate with high reliability for each photographing device 601.

The coordinate input device 609 acquires information (candidate information) related to coordinate conversion for a measurement point candidate with high reliability. Since the coordinate conversion table creation unit 610 creates the coordinate conversion table based on the candidate information acquired as described above, the accuracy of the coordinate conversion table can be improved.

In the following eighth embodiment, grouping of measurement point candidates in the seventh embodiment may be used.

<Eighth Embodiment>
A coordinate association system 700 according to the eighth embodiment includes a plurality of imaging devices 701 as in the seventh embodiment. Then, the coordinate association system 700 of the eighth embodiment creates a coordinate conversion table for the captured image data of at least one imaging device 701. Then, the coordinate association system 700 of the eighth embodiment creates a coordinate conversion table related to the captured image data of the other imaging device 701 based on the coordinate conversion table and the relative positional relationship in the imaging device 701. To do.

FIG. 22 is a block diagram illustrating an example of a configuration of the coordinate association system 700 according to the eighth embodiment. A coordinate association system 700 according to the eighth embodiment includes a three-dimensional landform database storage unit 702, a coordinate conversion table creation device 7, a measurement ease database storage unit 705, and a storage device 711. Furthermore, the coordinate association system 700 includes a display device 707 and a coordinate input device 708. The coordinate association system 700 includes a plurality of imaging devices 701. The coordinate conversion table creation device 7 includes a position / orientation acquisition unit 703, a small area grouping unit 704, a measurement point extraction unit 706, an imaging device relative position calculation unit 709, and a coordinate conversion table creation unit 710.

The imaging device 701 is the same as the imaging device 601 in the seventh embodiment. The three-dimensional landform database storage unit 702, the measurement ease database storage unit 705, and the storage device 711 are the same as the three-dimensional landform database storage unit 602, the measurement ease database storage unit 605, and the storage device 611 in the seventh embodiment. It is the same. The display device 707 and the coordinate input device 708 are the same as the display device 608 and the coordinate input device 609 in the seventh embodiment. However, the display device 707 does not use a group for each photographing device 701 when displaying the measurement point candidate for each photographing device 701. The position / orientation acquisition unit 703, the small region grouping unit 704, and the measurement point extraction unit 706 are the same as the position / orientation acquisition unit 603, the small region grouping unit 604, and the measurement point extraction unit 606 in the seventh embodiment. .

The imaging device relative position calculation unit 709 acquires information (candidate information) on coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the measurement point candidate from the coordinate input device 708. The imaging device relative position calculation unit 709 calculates the relative positional relationship of the imaging device 701 based on the information. That is, the imaging device relative position calculation unit 709 calculates information related to the relative position of the imaging device 701 based on the candidate information of the measurement point candidates.

The coordinate conversion table creation unit 710 creates a coordinate conversion table for at least one imaging device 701 using candidate information. Then, the coordinate conversion table creation unit 710 creates a coordinate conversion table related to the other imaging device 701 based on the coordinate conversion table and the relative positional relationship between the imaging devices 701. That is, the coordinate conversion table creation unit 710 creates a coordinate conversion table based on information (candidate information) related to coordinate conversion of at least one photographing apparatus 701, and based on the created coordinate conversion table and information related to the relative position, A coordinate conversion table for another imaging apparatus 701 is created.

The position / orientation acquisition unit 703, the small area grouping unit 704, the measurement point extraction unit 706, the imaging device relative position calculation unit 709, and the coordinate conversion table creation unit 710 are realized as follows. These elements are realized using, for example, a CPU of a computer that operates based on a coordinate conversion program. In this case, the CPU may read a coordinate conversion program from a program recording medium such as a program storage device (not shown) and operate as each of the above elements based on the read coordinate conversion program. The computer will be described in detail later.

Next, the operation of the eighth embodiment will be described. Note that detailed description of operations similar to those of the first to seventh embodiments is omitted as appropriate.

FIG. 23 is a flowchart showing an example of the operation of the coordinate conversion table creation device 7 according to the eighth embodiment. In addition, each imaging device 701 captures a monitoring target region, generates captured image data, and sends the captured image data to the coordinate conversion table creation unit 710.

23. The operations in steps S701 to S705 shown in FIG. 23 are the same as the operations in steps S601 to S605 in the seventh embodiment shown in FIG. In step S705, when a measurement point candidate is extracted for each photographing device 701, the display device 707 displays the measurement point candidate for each photographing device 701 as in the sixth embodiment. That is, as described above, the display device 707 does not use a group for each photographing device 701.

The coordinate input device 708 operates in the same manner as the coordinate input device 609 of the sixth embodiment for each imaging device 701. That is, the coordinate input device 708 acquires, for each photographing device 701, information (candidate information) regarding coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system at the measurement point candidate displayed on the display device 707. (Step S707).

The imaging device relative position calculation unit 709 calculates the relative positional relationship of the imaging device 701 for each imaging device 701 based on the candidate information of the measurement point candidates for each imaging device 701 (step S708). Specifically, the imaging device relative position calculation unit 709 calculates a rotation matrix and a translation vector that represent the relative positional relationship of the imaging device 701.

FIG. 24 is a schematic diagram for explaining the operation in step S708 in FIG. However, FIG. 24 uses the group shown in FIG. 4 instead of the group regrouped based on the distance shown in FIG. 9 in order to make the operation easy to understand. The measurement point candidate was shown.

FIG. 24 illustrates a case where there are two imaging devices 701. In the following description, the two photographing devices 701 are denoted by alphabetical signs and are distinguished as photographing

devices

701 _a and 701 _b .

Imaging device relative position calculating unit 709 obtains a candidate information at measurement point candidate corresponding to the imaging apparatus 701 _a. Similarly, imaging device relative position calculating unit 709 obtains a candidate information at measurement point candidate corresponding to the photographing apparatus 701 _b. The imaging device relative position calculation unit 709 is a rotation matrix that represents the relative positional relationship between the

imaging devices

701 _a and 701 _b based on the candidate information of the imaging device 701 _{a and} the candidate information of the imaging device 701 _b. And a translation vector is calculated.

FIG. 24 shows two imaging devices 701. However, this is exemplary. Three or more imaging devices 701 may be included. When three or more imaging devices 701 are included, the imaging device relative position calculation unit 709 determines, for example, a reference imaging device 701 (hereinafter referred to as “imaging device 701 _a ”). The photographing device relative position calculation unit 709 may calculate a relative positional relationship between the photographing device 701 _a and the other photographing device 701 (hereinafter referred to as “photographing device 701 _b ”). In this case, in step S709 described later, the coordinate transformation table creating unit 710 first creates a coordinate conversion table of the imaging apparatus 701 _a.

Note that the imaging device relative position calculation unit 709 may use a plurality of imaging devices 701 as the imaging device 701 used as a reference. Alternatively, the imaging device relative position calculation unit 709 may calculate the positional relationship of the imaging device 701 in a column.

The coordinate conversion table creation unit 710 creates a coordinate conversion table for at least one imaging device 701 based on information (candidate information) related to coordinate conversion. Hereinafter, to simplify the description, it is assumed that the coordinate conversion table creation unit 710 first obtains a coordinate conversion table for one imaging device 701 (imaging device 701 _a ).

Coordinate conversion table generating unit 710, with respect to imaging device 701 _a, using the methods already described, to create a coordinate conversion table. That is, the coordinate transformation table creating unit 710, for each small region groups associated with imaging device 701 _a, (more specifically coordinate transformation parameters) coordinate transformation between the camera coordinate system of coordinates and the world coordinate system of the coordinates determining the . Then, the coordinate conversion table creation unit 710 uses the coordinate conversion parameters for each small region group to determine the coordinates of the camera coordinate system and the coordinates of the world coordinate system for all pixels on which the ground surface in the captured image data is displayed. Find the coordinate transformation of. Then, the coordinate conversion table creation unit 710 creates a coordinate conversion table based on the coordinate conversion.

Next, the coordinate conversion table generation unit 710, based on the coordinate conversion table created, and information indicating a relative positional relationship between the imaging device 701 _a and another imaging device 701 _b, other imaging devices 701 _b A coordinate conversion table is created (step S709). That is, the coordinate transformation table generation unit 710, based on the information representing the relative positional relationship between the imaging apparatus 701 _a and the imaging apparatus 701 _b (the rotation matrix and translation vector), the coordinate transformation was developed for capturing apparatus 701 _a Convert the table. The converted coordinate transformation table, the coordinate transformation table for imaging apparatus 701 _b. Coordinate conversion table generation unit 710 similarly to all of the imaging device 701 _b, to create a coordinate conversion table.

The coordinate conversion table creation unit 710 stores the created coordinate conversion table in the storage device 711.

[Description of effects]
The effect of the eighth embodiment will be described. In addition to the effects of the first to seventh embodiments, the eighth embodiment has the effect of reducing the load for creating the coordinate conversion table. The reason is as follows.

The imaging device relative position calculation unit 709 calculates information (rotation matrix and translation vector) regarding the relative position of the imaging device 701. Then, the coordinate conversion table creation unit 710 creates a coordinate conversion table for one or more other image capturing apparatuses 701 based on the coordinate conversion table for one or more image capturing apparatuses 701 and information related to the relative position. The process of creating the coordinate conversion table based on the information on the relative position is matrix multiplication and translation vector addition. This process is less burdensome than the process of creating a coordinate conversion table using information (candidate information) related to coordinate conversion acquired from the coordinate input device 708. Therefore, the coordinate conversion table creation device 7 can reduce the processing load for creating the coordinate conversion table.

<Hardware configuration of each embodiment>
FIG. 25 is a block diagram illustrating an example of a hardware configuration of the coordinate conversion table creation device according to each embodiment of the present invention. The coordinate conversion table creating apparatus shown in FIG. 25 is configured using a computer 1000. The computer 1000 includes a CPU 1001, a main storage device 1002, an auxiliary storage device 1003, an interface 1004, a display device 1005, and an input device 1006.

The CPU 1001 realizes the functions of each embodiment based on a program (coordinate conversion program) for realizing the operation of the coordinate conversion table creation device of each embodiment. The CPU 1001 reads a coordinate conversion program from the auxiliary storage device 1003, develops it in the main storage device 1002, and executes the above-described operation based on the program. Alternatively, the CPU 1001 may acquire the program via a communication line (not shown) and store the program in the auxiliary storage device 1003 or expand the program in the main storage device 1002 to execute the above operation. The program acquired via this communication line may be a program for realizing a part of the processing of the CPU 1001 described above. Furthermore, the program acquired via this communication line may be a program (difference program) used in combination with a program stored in the auxiliary storage device 1003.

The main storage device 1002 temporarily stores programs executed by the CPU 1001 and data. The main storage device 1002 is, for example, a D-RAM (Dynamic-RAM).

The auxiliary storage device 1003 stores the program (coordinate conversion program) and fixed data of each embodiment. The auxiliary storage device 1003 is a non-volatile tangible medium that can be read by a computer. The auxiliary storage device 1003 may be a storage device connected via the interface 1004. The auxiliary storage device 1003 may operate as a storage device, a three-dimensional terrain database storage unit, and / or a measurement ease database storage unit. The auxiliary storage device 1003 is, for example, a magnetic disk, a magneto-optical disk, a CD (Compact Disc) -ROM (Read Only Memory), a DVD (Digital Versatile Disc) -ROM, or a semiconductor memory.

The interface 1004 relays transmission and reception of information between the CPU 1001 and an external device. The interface 1004 is, for example, a USB (Universal Serial Bus) card or a LAN (Local Area Network) card.

The display device 1005 is a device that displays information to the operator of the computer 1000. The display device 1005 is, for example, a liquid crystal display.

The input device 1006 is a device that receives an input operation from an operator of the computer 1000. The input device 1006 is, for example, a keyboard, a mouse, or a touch panel.

In addition, the coordinate matching system of each embodiment may be realized using the computer 1000 shown in FIG. At that time, the display device 1005 may operate as a display device. The input device 1006 may operate as a coordinate input device.

The coordinate conversion table creation device according to each embodiment described above is configured as follows. For example, each component of the coordinate conversion table creation device may be configured with a hardware circuit. Further, in the coordinate conversion table creation device, each component may be configured using a plurality of devices connected via a network (not shown). In the coordinate conversion table creation device, the plurality of components may be configured with a single piece of hardware.

<Outline of the invention>
Next, the outline of each embodiment in the present invention will be described.

FIG. 26 is a block diagram showing an example of the configuration of the information processing apparatus 10 corresponding to the outline of the coordinate conversion table creation apparatuses 1 to 7 according to the embodiment of the present invention. That is, the information processing apparatus 10 is an apparatus corresponding to the minimum configuration of the coordinate conversion table creation apparatus.

The information processing apparatus 10 includes a small area grouping unit 71 corresponding to a small area grouping unit and a coordinate conversion table creating unit 72 corresponding to a coordinate conversion table creating unit.

The small area grouping means 71 converts a small area formed based on a point having three-dimensional information on the terrain corresponding to the area included in the photographed image data photographed by the photographing device into a normal direction and an altitude of the small area. Based on this, it is divided into small area groups, which are small area groups.

The coordinate conversion table creating means 72 uses the group of small regions and the information related to coordinate conversion, the coordinates of the camera coordinate system, which are the coordinates in the captured image data, and the coordinates of the world coordinate system, which are the coordinates of the 3D terrain data. A coordinate conversion table showing the coordinate conversion of is created.

Note that the above coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system corresponds to the correspondence between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. That is, it is repeated, but the coordinate conversion table may be referred to as a correspondence table.

Based on the configuration as described above, the information processing apparatus 10 realizes coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system regardless of the target region, as in the first embodiment. The effect of. More specifically, the information processing apparatus 10 can determine the coordinates of the camera coordinate system and the world coordinate system even in an area where no landmark exists, an area of undulating complex terrain, or an area where a survey vehicle cannot travel. Coordinates can be converted between coordinates.

FIG. 27 is a block diagram showing an example of the configuration of the information processing system 800 corresponding to the outline of the coordinate association systems 100 to 700 according to the embodiment of the present invention. The information processing system 800 is a system corresponding to the minimum configuration of the coordinate association systems 100 to 700.

The information processing system 800 includes a display unit 81 corresponding to a display device, an input unit 82 corresponding to a coordinate input device, and an association unit 83 corresponding to the information processing device 10 (coordinate conversion table creation device).

That is, the association unit 83 corresponds to the information processing apparatus 10.

The display unit 81 displays the three-dimensional information received from the association unit 83 and / or the measurement point candidate.

The input means 82 acquires candidate information on 3D terrain data and / or measurement point candidates.

Based on such a configuration, the information processing system 800 has the same effects as those of the first embodiment.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2015-0665610 filed on Mar. 27, 2015, the entire disclosure of which is incorporated herein.

The above embodiment can be described as the following supplementary notes, but is not limited to the following.

(Appendix 1)
A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. A small area grouping means for dividing the small area group into groups,
Coordinates between the coordinates of the camera coordinate system, which are the coordinates in the captured image data, and the coordinates of the world coordinate system, which is the coordinates of the small area corresponding to the captured image data, using the small area group and candidate information that is information relating to coordinate transformation An information processing apparatus comprising: a coordinate conversion table creating means for creating a coordinate conversion table that is conversion information.

(Appendix 2)
It further includes position and orientation acquisition means for acquiring the distance from the imaging device to the small area and the orientation of the imaging device,
Small area grouping means
The information processing apparatus according to claim 1, further dividing the small area group based on the distance and the direction.

(Appendix 3)
Calculate the center of gravity of the small area group and obtain candidate information that is information related to coordinate conversion between the coordinates of the camera coordinate system and the coordinates of the world coordinate system. Extract a point included in a predetermined number of small region groups that are separated by a predetermined distance or more and the angle between all line segments connecting each point included in the small region group and the center of gravity is equal to or greater than the predetermined angle. Further including a measuring point extracting means for
The coordinate conversion table creation means
The information processing apparatus according to Supplementary Note 1 or Supplementary Note 2, wherein candidate information for a measurement point candidate is acquired and a coordinate conversion table is created based on the candidate information.

(Appendix 4)
And further comprising a measurement ease data storage means for storing measurement ease data, which is data obtained by quantifying the ease of measurement of coordinates in the coordinates of the world coordinate system,
Measuring point extraction means
Extract measurement point candidates based on the measurement ease data.
The information processing apparatus according to attachment 3.

(Appendix 5)
The coordinate conversion table creation means
Coordinate conversion for creating a coordinate conversion table in a small area group for which a coordinate conversion table has not been created, based on parameters used for coordinate conversion for creating a coordinate conversion table in a small area group for which a coordinate conversion table has been created The information processing apparatus according to appendix 3 or appendix 4, wherein the parameter is estimated and a coordinate conversion table of a small area group for which no coordinate conversion table has been generated is generated based on the estimated parameter.

(Appendix 6)
The information processing apparatus according to any one of appendix 3 to appendix 5, further including route calculation means for calculating a shortest route that passes through all the measurement point candidates.

(Appendix 7)
Including multiple photographic devices,
Based on the reliability of the coordinates of the world coordinate system and the coordinates of the camera coordinate system in the measurement point candidates, further includes a measurement point grouping means for creating a group of measurement point candidates for each photographing device,
The coordinate conversion table creation means
The information processing apparatus according to any one of supplementary notes 3 to 6, wherein candidate information is acquired for each photographing apparatus and a coordinate conversion table is created.

(Appendix 8)
Including multiple photographic devices,
Based on the candidate information of the measurement point candidate, further includes an imaging device relative position calculation means for calculating information on the relative position of the imaging device,
The coordinate conversion table creation means
A coordinate conversion table is created based on the candidate information of at least one imaging device, and the coordinate conversion table of the imaging device for which no coordinate conversion table is created based on the created coordinate conversion table and information relating to the imaging device and relative position The information processing apparatus according to any one of appendix 3 to appendix 6.

(Appendix 9)
The information processing apparatus according to any one of appendices 1 to 8,
Display means for displaying three-dimensional information received from the information processing apparatus and / or measurement point candidates;
An information processing system including input means for acquiring candidate information on three-dimensional information and / or measurement point candidates.

(Appendix 10)
A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. Divide into small area groups that are groups,
Coordinates between the coordinates of the camera coordinate system, which are the coordinates in the captured image data, and the coordinates of the world coordinate system, which is the coordinates of the small area corresponding to the captured image data, using the small area group and candidate information that is information relating to coordinate transformation An information processing method for creating a coordinate conversion table, which is conversion information.

(Appendix 11)
A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. Processing to divide into small area groups that are groups,
Coordinates between the coordinates of the camera coordinate system, which are the coordinates in the captured image data, and the coordinates of the world coordinate system, which is the coordinates of the small area corresponding to the captured image data, using the small area group and candidate information that is information relating to coordinate transformation A recording medium on which a computer-readable program for causing a computer to execute a process of creating a coordinate conversion table as conversion information is recorded.

The present invention is suitable for associating the coordinates of the camera coordinate system in the image photographed by the camera with the coordinates of the world coordinate system in the three-dimensional terrain data.

DESCRIPTION OF SYMBOLS 1 Coordinate conversion table creation apparatus 2 Coordinate conversion table creation apparatus 3 Coordinate conversion table creation apparatus 4 Coordinate conversion table creation apparatus 5 Coordinate conversion table creation apparatus 6 Coordinate conversion table creation apparatus 7 Coordinate conversion table creation apparatus 10 Information processing apparatus 71 Small area grouping Means 72 Coordinate conversion table creation means 81 Display means 82 Input means 83 Association means 100 Coordinate association system 101 Imaging device 102 Three-dimensional terrain database storage section 103 Small area grouping section 104 Display apparatus 105 Coordinate input apparatus 106 Coordinate conversion table creation section DESCRIPTION OF SYMBOLS 107 Memory | storage device 200 Coordinate matching system 201 Image pick-up device 202 Three-dimensional landform database memory | storage part 203 Position direction acquisition part 204 Small area grouping part 205 Display apparatus 206 Coordinate Force device 207 Coordinate conversion table creation unit 208 Storage device 300 Coordinate association system 301 Imaging device 302 Three-dimensional terrain database storage unit 303 Position and orientation acquisition unit 304 Small region grouping unit 305 Measurement point extraction unit 306 Display device 307 Coordinate input device 308 Coordinates Conversion table creation unit 309 Storage device 400 Coordinate association system 401 Imaging device 402 Three-dimensional terrain database storage unit 403 Position orientation acquisition unit 404 Small region grouping unit 405 Measurement ease database storage unit 406 Measurement point extraction unit 407 Display device 408 Coordinate input Device 409 Coordinate conversion table creation unit 410 Storage device 500 Coordinate association system 501 Imaging device 502 3D terrain database storage unit 503 Position / orientation acquisition unit 504 Small area grouping unit 505 Measurement ease database storage unit 506 Measurement point extraction unit 507 Route calculation unit 508 Display device 509 Coordinate input device 510 Coordinate conversion table creation unit 511 Storage device 600 Coordinate association system 601 Imaging device 602 3D terrain database storage Unit 603 position / orientation acquisition unit 604 small area grouping unit 605 measurement ease database storage unit 606 measurement point extraction unit 607 measurement point grouping unit 608 display device 609 coordinate input device 610 coordinate conversion table creation unit 611 storage device 700 coordinate association system 701 Imaging device 702 3D terrain database storage unit 703 Position / orientation acquisition unit 704 Small area grouping unit 705 Measurement ease database storage unit 706 Measurement point extraction unit 707 Display device 708 Coordinate input device 709 Imaging device relative position calculation unit 710 Coordinate conversion table creation unit 711 Storage device 800 Information processing system 1000 Computer 1001 CPU
1002 Main storage device 1003 Auxiliary storage device 1004 Interface 1005 Display device 1006 Input device 2001 Object position measurement device 2002 Object detection device 2003 Object position coordinate conversion device

Claims

A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. Small area grouping means for dividing into small area groups, which are groups of areas;
Using the candidate information that is information related to the small area group and coordinate transformation, the coordinates of the camera coordinate system that are the coordinates in the captured image data and the coordinates in the small area that correspond to the captured image data An information processing apparatus comprising: coordinate conversion table creating means for creating a coordinate conversion table that is information of coordinate conversion with coordinates.
It further includes a position and orientation acquisition means for acquiring a distance from the imaging device to the small area and an orientation of the imaging device,
The small area grouping means includes
The information processing apparatus according to claim 1, wherein the small area group is further divided based on the distance and the orientation.
The center of gravity of the small area group is calculated and the candidate information is obtained as a measurement point candidate from among the points included in the small area group at a predetermined distance or more away from the center of gravity and included in the small area group Further comprising a measurement point extracting means for extracting the points included in a predetermined number of the small region groups in which an angle between all line segments connecting the points and the center of gravity is equal to or greater than a predetermined angle;
The coordinate conversion table creation means
The information processing apparatus according to claim 1, wherein the candidate information for the measurement point candidate is acquired, and the coordinate conversion table is created based on the candidate information.
And further comprising a measurement ease data storage means for storing measurement ease data, which is data obtained by quantifying the ease of measurement of coordinates in the coordinates of the world coordinate system,
The measurement point extracting means is
Based on the measurement ease data, the measurement point candidates are extracted.
The information processing apparatus according to claim 3.
The coordinate conversion table creating means includes
Based on the parameters used for the coordinate transformation for creating the coordinate transformation table in the small region group that created the coordinate transformation table, the coordinate transformation table in the small region group that has not created the coordinate transformation table. 5. The parameter conversion table of the small region group that does not create the coordinate conversion table based on the estimated parameter is estimated based on the estimated parameter conversion. The information processing apparatus described.
The information processing apparatus according to claim 3, further comprising a route calculation unit that calculates a shortest route that passes through all the measurement point candidates.
Including a plurality of the photographing devices,
Based on the reliability of the coordinates of the world coordinate system and the coordinates of the camera coordinate system in the measurement point candidates, further includes a measurement point grouping means for creating a group of measurement point candidates for each of the imaging devices,
The coordinate conversion table creation means
The information processing apparatus according to any one of claims 3 to 6, wherein the candidate information is acquired for each of the photographing apparatuses and the coordinate conversion table is created.
Including a plurality of the photographing devices,
Based on the candidate information of the measurement point candidate, further includes imaging device relative position calculation means for calculating information on the relative position of the imaging device,
The coordinate conversion table creating means includes
The coordinate conversion table is created based on the candidate information of at least one of the imaging devices, and the coordinate conversion table is created based on the created coordinate conversion table, the imaging device, and information on the relative position. The information processing apparatus according to any one of claims 3 to 6, wherein the coordinate conversion table of the imaging apparatus that is not present is created.
The information processing apparatus according to any one of claims 1 to 8,
Display means for displaying the three-dimensional information and / or measurement point candidates received from the information processing apparatus;
An information processing system comprising: input means for acquiring the candidate information in the three-dimensional information and / or the measurement point candidate.
A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. Divide into small area groups, which are groups of areas,
Using the candidate information that is information related to the small area group and coordinate transformation, the coordinates of the camera coordinate system that are the coordinates in the captured image data and the coordinates in the small area that correspond to the captured image data An information processing method for creating a coordinate conversion table which is information of coordinate conversion with coordinates.
A small area formed on the basis of a point having three-dimensional information regarding the position on the terrain corresponding to the area included in the photographed image data photographed by the photographing device is determined based on the normal direction and the altitude of the small area. Processing to divide into small area groups that are area groups;
Using the candidate information that is information related to the small area group and coordinate transformation, the coordinates of the camera coordinate system that are the coordinates in the captured image data and the coordinates in the small area that correspond to the captured image data The recording medium which recorded the program which makes a computer perform the process which produces the coordinate conversion table which is the information of coordinate conversion with a coordinate so that computer reading is possible.