JP2024031584A - Information processing system, information processing method and program - Google Patents

Abstract

An object of the present invention is to easily detect the shape and dimensions of an object from an image. An information processing system includes an image acquisition unit that acquires a plurality of images of an object taken from a plurality of positions, and a designation of a position of a vertex of the object in at least one of the plurality of images. The present invention is characterized by comprising: a vertex acquisition unit that receives a vertex, and a model creation unit that creates a three-dimensional model composed of vertices and edges connecting the vertices based on the positions of the vertices and a plurality of images. [Selection diagram] Figure 1

Description

The present invention relates to an information processing system, an information processing method, and a program.

Patent Document 1 discloses a technique for measuring the shape and dimensions of a roof, which is a target object, from an image taken of the target object with a camera mounted on an aircraft, and calculating the area of the roof from the shape and dimensions.

Japanese Patent Application Publication No. 2003-162552

In order to measure the shape of a roof from images, it is necessary to image the object from various angles.

The present invention has been made in view of such a background, and an object of the present invention is to provide a technique that can easily detect the shape and dimensions of an object from an image.

The main invention of the present invention for solving the above problems is an information processing system, which includes an image acquisition unit that acquires a plurality of images of a target object from a plurality of positions, and at least one of the plurality of images. a vertex acquisition unit that receives designation of the position of the apex of the object; and a model that creates a three-dimensional model constituted by the apex and edges connecting the apex based on the position of the apex and the plurality of images. The invention is characterized by comprising a creation section.

Other problems disclosed in the present application and methods for solving the problems will be made clear by the section of the embodiments of the invention and the drawings.

According to the present invention, the shape and dimensions of an object can be easily detected from an image.

FIG. 2 is a diagram illustrating an overview of the present embodiment. 1 is a diagram schematically showing an information processing system 1 according to the present embodiment. 1 is a diagram showing an example of a hardware configuration of an information processing terminal 10. FIG. 1 is a diagram showing an example of a software configuration of an information processing terminal 10. FIG. 3 is a diagram illustrating a display example of image information 121. FIG. FIG. 3 is a diagram illustrating an example of generation of vertex information and line segment information. It is a figure explaining an example of evaluation of a twist degree. 3 is a flowchart showing an example of the flow of processing of the information processing system 1. FIG. FIG. 2 is a diagram schematically showing an information processing system 1' according to another embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

<Summary>

FIG. 1 is a diagram illustrating an overview of this embodiment. The information processing system 1 according to the present embodiment attempts to analyze the structure of the roof 101 from an image taken of the roof 101 of a building 100, which is a target object. Find the length of the roof, ridge, etc.) and create a roof plan.

In the information processing system 1 of this embodiment, the camera 31 attached to the flying object 30 flying above the roof 101 images the roof 101 as a target object.

A GCP (ground control point) 110 is placed near the building 100. The GCP 110 is placed on the ground at a position that is visible from the sky. The size of the GCP 110 is given in advance, and the roof 101 can be measured by detecting the GCP 110 from the captured image.

In the information processing system 1 of the present embodiment, the flying object 30 is moved from a plurality of different altitudes (the example in FIG. 1 shows two altitudes H1 and H2, but three or more altitudes may be used). The roof 101 is imaged. The three-dimensional structure of the roof 101 is calculated by capturing images of the roof 101 from multiple altitudes.

<System configuration>
FIG. 2 is a diagram schematically showing the information processing system 1 according to the present embodiment. As illustrated, the information processing system 1 includes an information processing terminal 10.

The information processing terminal 10 according to this embodiment is implemented by a small computer in the form of a so-called tablet. In other embodiments, the information processing terminal 10 may be realized by a portable information processing terminal such as a smartphone or a game machine, or may be realized by a stationary information processing terminal such as a personal computer. Further, the information processing terminal 10 may be realized by a plurality of pieces of hardware, and may have a configuration in which functions are distributed among them.

FIG. 3 is a diagram showing an example of the hardware configuration of the information processing terminal 10. As shown in FIG. As illustrated, the information processing terminal 10 includes a control section 11 and a touch panel section 12 that is a display section.

The control unit 11 mainly includes a processor 11a, a memory 11b, a storage 11c, a transmitting/receiving unit 11d, and an input/output unit 11e, which are electrically connected to each other via a bus 11f.

The processor 11a is an arithmetic device that controls the operation of the control unit 11, controls data transmission and reception between each element, and performs processing necessary for program execution.

In this embodiment, the processor 11a is, for example, a CPU (Central Processing Unit), and executes a program stored in a storage 11c and developed in a memory 11b to perform various processes.

The memory 11b includes a main storage device made up of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary storage device made up of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). .

The memory 11b is used as a work area for the processor 11a, and also stores a BIOS (Basic Input/Output System) executed when the control unit 11 is started, various setting information, and the like.

The storage 11c stores programs and information used in various processes. For example, when a user operates a flying object for capturing image information of the roof 101 via the information processing terminal 10, a program for controlling the flight of the flying object may be stored in the storage 11c. .

The transmitting/receiving unit 11d connects the control unit 11 to a network such as the Internet, and may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

In this embodiment, for example, a control signal for controlling the flight of the flying object 10 may be transmitted to the flying object 10 via this transmitting/receiving section 11d.

The input/output unit 11e is an interface to which input/output devices are connected, and in this embodiment, the touch panel unit 12 is connected.

The bus 11f transmits, for example, address signals, data signals, and various control signals among the connected processor 11a, memory 11b, storage 11c, transmitting/receiving section 11d, and input/output section 11e.

The touch panel section 12 is an example of a display section, and includes a display surface 12a on which acquired videos and images are displayed. In this embodiment, the display surface 12a receives information input by touching the display surface 12a, and is implemented using various techniques such as a resistive film method and a capacitance method.

For example, the user can input line segment information and node information to the image displayed on the display surface 12a via the touch panel section 12. Furthermore, display information output by the control unit 11 is displayed on the display surface 12a.

FIG. 4 is a diagram showing an example of the software configuration of the information processing terminal 10. As shown in FIG. The processor 11a includes an image acquisition section 111, a model creation section 112, a calculation section 113, a vertex acquisition section 114, a twist evaluation section 115, a determination section 116, a correction section 117, and a drawing creation section 118. Be prepared. The storage 11c also includes an input information database 120.

The input information database 120 stores image information 121, vertex information 123, and line segment information 125.

Image information 121 is an image of the object. In this embodiment, the image information 121 is an image of the roof 101 taken by the camera 31 of the flying object 30. The roof 101 and the GCP 110 are photographed in the photographed image of the image information 121. Image information 121 related to a plurality of images of a target object taken from a plurality of different altitudes is registered. The plurality of images may be images taken at different altitudes at the same location. Note that the angle of view of the camera 31 may be the same.

The vertex information 123 and the line segment information 125 are, for example, information regarding the vertices 102 and line segments 103 for dividing the polygon (region) corresponding to the roof 101. Line segments 103 are part of the outer and inner boundaries of a polygon that partitions the object, and correspond to constituent parts of the object. For example, when the target object is the roof 101, the component corresponding to the line segment 103 is an eave, a ridge, or the like. The vertex 102 means a vertex that connects the plurality of line segments 103. Note that when the object is the roof 101, the component corresponding to the apex 102 is an end (top) of an eave, a ridge, or the like.

The image acquisition unit 111 acquires an image taken by the flying object 30. The image acquisition unit 111 acquires a plurality of images taken from different altitudes. The image acquisition unit 111 can acquire a first image of a target object (roof 101) taken by the flying object 30 at a first altitude (for example, H1 in FIG. 1). The image acquisition unit 111 can also acquire a second image of the object (roof 101) taken by the flying object 30 at a second altitude (for example, H2 in FIG. 1).

The image acquisition unit 111 may communicate with the flying object 30 and acquire images from the camera 31, or may read image information registered in the input information database 120 in advance.

Further, the image acquisition unit 111 can also acquire various information from the input information database 120. For example, the image acquisition unit 111 can acquire image information 121, vertex information 123, and line segment information 125 included in the input information database 120.

The vertex acquisition unit 114 receives designation of the position of the vertex of a polygon that defines the object on the captured image of the object. The vertex acquisition unit 114 can acquire three or more vertices included in the surface portion (roof 101) included in the building 100. For example, the vertex acquisition unit 114 may accept a designation of the position of a vertex on the image from the user, or may detect a polygon from the image through image recognition processing and acquire the vertex of the polygon. good. For this image recognition process, for example, a known machine learning method such as deep learning may be used. Image recognition processing can be performed in the processor 11a of the information processing terminal 10. The vertex acquisition unit 114 can acquire vertices for both the first image (altitude H1) and the second image (altitude H2), for example. The vertex acquisition unit 114 may acquire the vertices of one image, modify the position of the vertices according to the altitude and the angle of view of the camera 31, and estimate the vertices of the other image, or may estimate the vertices of the other image. The vertices of one image may be mapped onto the other image by feature point mapping.

FIG. 5 is a diagram showing a display example of the image information 121. An image including the roof 101 of the building 100 is displayed on the display surface 12a based on the acquired image information 121. A user operating the information processing terminal 10 can visually recognize the structure of the roof 101 from the image displayed on the display surface 12a.

The vertex information 123 and the line segment information 125 may be generated, for example, by a user operating the information processing terminal 10 operating the display surface 12a. Specifically, when an image showing the roof 101 is displayed on the display surface 12a, the user performs an operation on the touch panel section 12 to set the line segment 103 and the vertex 102 corresponding to the constituent parts of the roof 101. You may do so.

FIG. 6 is a diagram illustrating an example of generation of vertex information and line segment information. For example, the vertices 102a to 102l and the line segments 103a to 103o are set by the user's operation on the touch panel unit 12. At this time, the vertices 102a to 102l and the line segments 103a to 103o may be set by the user inputting so that the vertices 102 and line segments 103 are superimposed on the displayed image of the roof 101. . At this time, the polygon that partitions the object refers to a polygon formed by each line segment 103a to 103o.

The set vertices 102a to 102l and line segments 103a to 103o are generated as vertex information and line segment information, respectively, and stored in the input information database 117 of the storage 11c.

Note that in the example shown in FIG. 6, the vertex information 123 and the line segment information 125 are generated by the user's operation on the touch panel unit 12, but the present invention is not limited to this example. For example, the vertex information and line segment information may be generated by extracting portions corresponding to the vertices 102 and line segments 103 through image recognition processing on an image in which the roof 101 is reflected. For this image recognition process, for example, a known machine learning method such as deep learning may be used. In this case, for example, image recognition processing can be realized by using a learning model regarding an image showing the roof 101 and the constituent parts of the roof 101. Image recognition processing can be performed in the processor 11a of the information processing terminal 10.

The model creation unit 112 creates a three-dimensional model of the object based on a plurality of images captured at different altitudes. The model creation unit 112 can create a three-dimensional model based on the positions of vertices on the first and second images acquired by the vertex acquisition unit 114 and the first and second images. For example, the model creation unit 112 calculates the position of each vertex in real space from the position of the vertex on the first and second images based on the position and size of the GCP 110 on the first and second images. Relative positions and distances can be calculated. The model creation unit 112 calculates the relative three-dimensionality of the vertices based on, for example, the angle of view of the camera 31, the relative distance from the GCP 110, and the ratio of the size in pixels of the GCP 110 to the actual size of the GCP 110. Coordinate values (x, y, z) can be calculated. The model creation unit 112 can calculate the three-dimensional coordinate value of each vertex in the three-dimensional space using, for example, an existing algorithm such as a five-point algorithm or SfM (Structure from Motion).

Further, the model creation unit 112 maps feature points on the photographed image using, for example, SLAM (Simultaneous Localization and Mapping) technology (particularly Visual SLAM), and uses the feature points and the vertices 102 to A dimensional model may also be created. For example, the model creation unit 112 can create a three-dimensional model by providing each point including the vertex 102 and feature points to an algorithm such as SfM. Further, the model creation unit 112 may correct the three-dimensional model created based on the vertices 102 as described above using the feature points. Furthermore, the model creation unit 112 creates a three-dimensional model based on feature points obtained by Visual SLAM, and selects the one closest to the vertex 102 among the nodes (vertices) constituting the created three-dimensional model. The position of the identified node may be corrected to match the position of the vertex 102.

Further, the model creation unit 112 may calculate the position of the vertex as a position on the image, or as a relative position with respect to a predetermined position (for example, the position of a certain vertex). For example, based on the position and size of the GCP 110 on the first and second images, the position of the vertex in the three-dimensional space is calculated from the position of the vertex on the first and second images. You can do it like this.

The calculation unit 113 calculates at least one of the area of the object (roof 101 in this embodiment) and the length of the part of the object based on the three-dimensional model. The calculation unit 113 can calculate the length of the region between the vertices using the position of each vertex. The calculation unit 113 can calculate the area of a surface portion of the object, for example, based on the three-dimensional coordinate values of extension points of a polygon that constitutes a surface portion such as a roof surface.

The twist evaluation unit 115 evaluates the degree of twist of the polygon defined by the vertices 102. FIG. 7 is a diagram illustrating an example of evaluation of twist degree. The torsion evaluation unit 115, for example, identifies a plane P where the sum of distances D from each of the vertices 102 in the three-dimensional model is the minimum, and evaluates the degree of torsion of the plane portion according to the sum of these distances D. Can be done. The plane P can be specified by, for example, the least squares method. For example, the twist evaluation unit 115 can evaluate the twist degree such that the longer the total distance D is, the higher the twist degree becomes.

The determining unit 116 can determine whether the creation of the three-dimensional model has failed depending on whether the degree of twist is equal to or greater than a threshold value.

The correction unit 117 corrects the position (three-dimensional coordinate values) of the vertex 102. The correction unit 117 sets the nearest position from the vertex 102 on the plane P described above (the intersection of the straight line extending from the vertex 102 at right angles to the plane P and the plane P) as the position (three-dimensional coordinate value) of the vertex 102. be able to.

In addition, when there are multiple surface portions, the correction unit 117 adjusts the degree of torsion for each surface so that the total sum (for example, the total value of the shortest distance D from the vertex 102 to the plane P) is the minimum. Additionally, the position of the vertex 102 may be corrected. The correction unit 117 performs position correction of the three-dimensional coordinate values using, for example, the steepest descent method.

Further, the correction unit 117 can correct the positions of the vertices 102 so that the slope between the vertices 102 is a predetermined unit. The predetermined unit can be, for example, a slope of 1 inch or 0.5 inch. In cases where the slope is determined by a standard, such as a roof, the slope between the vertices of the polygon that defines the roof is often a unit of sharpness according to the specified standard, so By correcting the coordinate values of the vertex 102, it is expected that the error of the vertex 102 will be corrected.

The drawing creation unit 118 creates a roof plan based on the three-dimensional model.

<Operation>
Next, the flow of processing in the information processing system 1 according to this embodiment will be explained. FIG. 8 is a flowchart illustrating an example of the processing flow of the information processing system 1.

The information processing terminal 10 acquires a plurality of image information obtained by photographing a target object from a plurality of different altitudes (S101), and acquires each vertex 102 of a polygon defining the target object in each image information (S102).

The information processing terminal 10 detects the GCP 110 from the image information (S103), and uses the five-point algorithm or SfM according to the position and size of the detected GCP 110 on the image and the position of the vertex 102 on each image on the image. The three-dimensional coordinate values of the apex 102 are calculated by the following methods, and a three-dimensional model of the object is created (S104).

The information processing terminal 10 corrects the three-dimensional coordinate value of the vertex 102 (S105). As described above, the information processing terminal 10 identifies the plane P that is the minimum distance from each vertex 102 of the polygon indicating the surface portion, and selects the vertices so that the total distance D from the plane P to the vertex 102 is the minimum. 102 three-dimensional coordinate values can be corrected. Further, the information processing terminal 10 can also correct the three-dimensional coordinate values so that the inclination of the parts between the vertices 102 is in a predetermined unit (for example, 1 inch, 0.5 inch, etc.).

The information processing terminal 10 calculates the length of the part between the vertices 102 and the area of the surface portion indicated by the polygon including the vertices 102 (S106), and creates a hidden map based on the three-dimensional model (S107).

As described above, according to the embodiments of the present invention, the shape and dimensions of an object appearing in images taken from a plurality of altitudes can be easily and accurately determined. The flying object 30 may photograph the object by changing only the altitude without moving the horizontal position. Therefore, even if there are obstacles around the object, the shape of the object can be easily determined.

Furthermore, in this embodiment, mapping can be performed after plotting the vertices 102 of the object on a photographed image. In normal SfM, many feature points that can be extracted from an image are mapped, so there may be errors in the accuracy of restoring the position of each part of the object, but in this embodiment, only the plotted vertices 102 are mapped. Since it is only necessary to restore the dimensional position, errors can be reduced.

Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the technical scope of the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

In the above embodiment, the information processing system 1 is realized only by the information processing terminal 10, but the present invention is not limited to such an example. FIG. 9 is a diagram schematically showing an information processing system 1' according to another embodiment. In another embodiment, the information processing system 1' includes an information processing terminal 10 and a server 20. In this case, for example, part or all of the functions possessed by the processor 11a and the storage 11c of the information processing terminal 10 according to the above embodiment may be realized by the processor and storage included in the server 20.

Further, in the above embodiment, the case where the target object is the roof 101 of the building 100 has been described, but it may also be a tree or any ground surface, and furthermore, it may be a temporarily stopped car or an animal. It may be an object such as. The constituent parts of the object are appropriately set depending on the type of the object and the specific structure of the object.

Furthermore, in the above embodiment, the three-dimensional model of the object is created using two images (the first and second images), but of course the three-dimensional model is created based on three or more images. It is also possible to create one.

Further, in the above embodiment, the structure of the roof 101 is analyzed based on images taken by the flying object 10, but it is also possible to analyze the structure of the roof 101 based on images taken by the flying object 10. Analysis can also be performed using images taken from different altitudes.

Furthermore, images may be taken from different altitudes, or may be images moved horizontally or shot in different directions.

Furthermore, in the above embodiment, the three-dimensional structure of the object is estimated based on multiple images taken of the same object from different altitudes, but the same object can be estimated from different positions due to horizontal movement or altitude changes. The three-dimensional structure of the object may be estimated based on a plurality of captured images. Also in this case, by photographing the GCP 110, a three-dimensional model of the object (for example, the roof 101) can be created by the same processing as in the above embodiment.

Further, in the above embodiment, the case where the target object is the roof 101 of the building 100 has been explained, but it may also be a tree or any ground surface, and furthermore, it may be a temporarily stopped car or an animal. It may be an object. The constituent parts of the object are appropriately set depending on the type of the object and the specific structure of the object.

Further, in the above embodiment, images taken at different photographing altitudes are acquired, but the present invention is not limited to this, and any images taken at different photographing positions may be used.

Furthermore, each step in the above embodiments does not necessarily need to be processed chronologically in the order described in the flowchart. For example, each step in the processing of the above embodiment may be processed in a different order from the order described as a flowchart, or may be processed in parallel.

Further, the effects described in this specification are merely explanatory or illustrative, and are not limiting. In other words, the technology according to the present invention can have other effects that are obvious to those skilled in the art from the description of this specification, in addition to or in place of the above effects.

<Disclosure>
Note that the present disclosure also includes the following configurations.
[Item 1]
an image acquisition unit that acquires multiple images of a target object from multiple positions;
a vertex acquisition unit that receives designation of a position of a vertex of the object in at least one of the plurality of images;
a model creation unit that creates a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
An information processing system comprising:
[Item 2]
The information processing system described in item 1,
The image acquisition unit acquires a plurality of images of the object taken from different altitudes;
An information processing system characterized by:
[Item 3]
The information processing system described in item 1,
comprising a calculation unit that calculates at least one of the area of the object and the length of a part of the object based on the three-dimensional model;
An information processing system characterized by:
[Item 4]
The information processing system described in item 1,
the object is a roof;
comprising a drawing creation unit that creates a roof plan based on the three-dimensional model;
An information processing system characterized by:
[Item 5]
The information processing system described in item 1,
The vertex acquisition unit acquires three or more vertices included in a surface portion included in the object,
comprising a torsion evaluation unit that identifies a plane having a minimum distance from each of the vertices in the three-dimensional model and evaluates the degree of torsion of the surface portion according to the distance;
An information processing system characterized by:
[Item 6]
The information processing system described in item 5,
comprising a determination unit that determines whether creation of the three-dimensional model has failed depending on whether the degree of twist is equal to or greater than a threshold;
An information processing system characterized by:
[Item 7]
The information processing system described in item 5,
comprising a correction unit that corrects the position of the vertex to a position on the plane closest to the vertex;
An information processing system characterized by:
[Item 8]
The information processing system described in item 5,
comprising a correction unit that corrects the positions of the vertices so that the distance between the vertices is a predetermined unit;
An information processing system characterized by:
[Item 9]
acquiring a plurality of images of the object from a plurality of positions;
receiving designation of the position of the apex of the object in at least one of the plurality of images;
creating a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
An information processing method, characterized in that an information processing device executes the following.
[Item 10]
acquiring a plurality of images of the object from a plurality of positions;
receiving a designation of the position of a vertex of the object in at least one of the plurality of images;
creating a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
A program that causes an information processing device to execute.

1 Information processing system 10 Information processing terminal 11 Control unit 102 Vertex

Claims (10)

an image acquisition unit that acquires multiple images of a target object from multiple positions;
a vertex acquisition unit that receives designation of a position of a vertex of the object in at least one of the plurality of images;
a model creation unit that creates a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
An information processing system comprising: The information processing system according to claim 1,
The image acquisition unit acquires a plurality of images of the object taken from different altitudes;
An information processing system characterized by: The information processing system according to claim 1,
comprising a calculation unit that calculates at least one of the area of the object and the length of a part of the object based on the three-dimensional model;
An information processing system characterized by: The information processing system according to claim 1,
the object is a roof;
comprising a drawing creation unit that creates a roof plan based on the three-dimensional model;
An information processing system characterized by: The information processing system according to claim 1,
The vertex acquisition unit acquires three or more vertices included in a surface portion included in the object,
comprising a torsion evaluation unit that identifies a plane having a minimum distance from each of the vertices in the three-dimensional model and evaluates the degree of torsion of the surface portion according to the distance;
An information processing system characterized by: The information processing system according to claim 5,
comprising a determination unit that determines whether creation of the three-dimensional model has failed depending on whether the degree of twist is equal to or greater than a threshold;
An information processing system characterized by: The information processing system according to claim 5,
comprising a correction unit that corrects the position of the vertex to a position on the plane closest to the vertex;
An information processing system characterized by: The information processing system according to claim 5,
comprising a correction unit that corrects the positions of the vertices so that the distance between the vertices is a predetermined unit;
An information processing system characterized by: acquiring a plurality of images of the object from a plurality of positions;
accepting designation of the position of the apex of the object in at least one of the plurality of images;
creating a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
An information processing method, characterized in that an information processing device executes the following. acquiring a plurality of images of the object from a plurality of positions;
accepting designation of the position of the apex of the object in at least one of the plurality of images;
creating a three-dimensional model composed of the vertices and edges connecting the vertices based on the positions of the vertices and the plurality of images;
A program that causes an information processing device to execute.

Images