JP7378691B1 - Information processing device, detection method, and detection program - Google Patents

Abstract

The information processing device (100) includes an acquisition unit (120) that acquires a plurality of images obtained by imaging a plurality of objects from a plurality of different angles, and a plurality of layers configured based on the plurality of images. a creation unit (130) that creates three-dimensional colored point cloud information to create three-dimensional colored point cloud information; An extraction unit (140) that extracts the point cloud density of the target object point cloud information, a calculation unit (150) that calculates the point cloud density of the target object point cloud information, and a It has a detection unit (170) that detects layers other than the upper layer.

Description

The present disclosure relates to an information processing device, a detection method, and a detection program.

At the construction site, reinforcement inspection is being carried out. In order to save labor, systems that support reinforcement inspection have been developed. Furthermore, a technique for detecting reinforcement regions has been proposed (see Patent Document 1).

Japanese Patent Application Publication No. 2018-172847

By the way, the objects of reinforcement inspection are a plurality of reinforcing bars. Multiple layers are composed of multiple reinforcing bars. In this way, when a plurality of layers are configured, a technique for detecting the top layer is known. Furthermore, techniques for detecting layers other than the top layer are also known. However, there was a problem in that the detection accuracy of the technology for detecting layers other than the top layer was poor.

An objective of the present disclosure is to improve the accuracy of detecting layers other than the top layer.

An information processing device according to one aspect of the present disclosure is provided. The information processing device includes an acquisition unit that acquires a plurality of images obtained by imaging a plurality of objects from a plurality of different angles, and a three-dimensional colored point forming a plurality of layers based on the plurality of images. a creation unit that creates group information; and a creation unit that estimates a shape of the three-dimensional colored point group information based on the three-dimensional colored point group information, estimates a three-dimensional direction based on the shape, and Based on the direction of and an extraction unit that extracts object point cloud information that is point cloud information of the plurality of objects from the three-dimensional colored point cloud information based on the average value or the mode value; a calculation unit that calculates a point cloud density of object point cloud information; a detection unit that detects a layer other than the top layer among the plurality of layers based on the object point cloud information and the point cloud density; , has.

According to the present disclosure, it is possible to improve the accuracy of detecting layers other than the top layer.

It is a diagram showing an example of a construction site. FIG. 2 is a diagram showing hardware included in an information processing device. FIG. 2 is a block diagram showing the functions of the information processing device. (A) and (B) are diagrams illustrating a specific example of processing executed by an extraction unit. FIG. 3 is a diagram illustrating an example of a method for extracting target object point group information. FIG. 3 is a diagram illustrating an example of intersection detection processing. FIG. 3 is a diagram showing an example of detection processing. (A) and (B) are diagrams illustrating an example of inspection processing. 3 is a flowchart illustrating an example of processing executed by the information processing device.

Embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present disclosure.

Embodiment.
FIG. 1 is a diagram showing an example of a construction site. A plurality of reinforcing bars are placed at a construction site. Multiple layers are composed of multiple reinforcing bars. Figure 1 shows two layers. For example, FIG. 1 shows the first layer, which is the top layer. Further, for example, FIG. 1 shows a second layer that is a layer below the first layer. FIG. 1 thus shows two layers. However, the number of layers constituted by a plurality of reinforcing bars may be three or more.

Further, FIG. 1 shows an information processing apparatus 100. The information processing device 100 is a device that executes a detection method. For example, the information processing device 100 is a tablet terminal, a smartphone, or the like. Further, the information processing device 100 may be a PC (Personal Computer) or the like. The user may use the information processing device 100 to capture images of a plurality of reinforcing bars. When capturing images of a plurality of reinforcing bars, the information processing device 100 includes an image sensor such as a stereo camera, an RGB (Red Green Blue)-D (Depth) camera, an infrared camera, or an X-ray camera. When capturing images of multiple reinforcing bars, the user images the multiple reinforcing bars from multiple different angles. Thereby, the information processing apparatus 100 can acquire a plurality of images. Here, the reinforcing bars are also referred to as objects.

The information processing apparatus 100 executes processing using a plurality of images. The processing executed by the information processing apparatus 100 will be described in detail below.

Next, hardware included in the information processing device 100 will be explained.
FIG. 2 is a diagram showing hardware included in the information processing device. The information processing device 100 includes a processor 101, a volatile storage device 102, and a nonvolatile storage device 103.

Processor 101 controls the entire information processing device 100 . For example, the processor 101 is a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or the like. Processor 101 may be a multiprocessor. Further, the information processing device 100 may include a processing circuit.

The volatile storage device 102 is the main storage device of the information processing device 100. For example, the volatile storage device 102 is a RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the information processing device 100. For example, the nonvolatile storage device 103 is an SSD (Solid State Drive).

Next, the functions of the information processing device 100 will be explained.
FIG. 3 is a block diagram showing the functions of the information processing device. The information processing device 100 includes a storage section 110, an acquisition section 120, a creation section 130, an extraction section 140, a calculation section 150, a direction control section 160, a detection section 170, an inspection section 180, and an output section 190.

The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the nonvolatile storage device 103.

Part or all of the acquisition section 120, creation section 130, extraction section 140, calculation section 150, direction control section 160, detection section 170, inspection section 180, and output section 190 may be realized by a processing circuit. In addition, some or all of the acquisition unit 120, creation unit 130, extraction unit 140, calculation unit 150, direction control unit 160, detection unit 170, inspection unit 180, and output unit 190 are modules of a program executed by the processor 101. It may also be realized as For example, the program executed by processor 101 is also referred to as a detection program. For example, the detection program is recorded on a recording medium.

Storage unit 110 stores various information.
The acquisition unit 120 acquires a plurality of images obtained by capturing images of a plurality of objects (for example, reinforcing bars) from a plurality of different angles. For example, when the information processing device 100 includes a camera (for example, a stereo camera), the acquisition unit 120 acquires the plurality of images generated by the camera. Further, the acquisition unit 120 may acquire the plurality of images from an external device. Note that illustration of external devices is omitted.

The creation unit 130 creates three-dimensional colored point cloud information based on the plurality of images. The three-dimensional colored point cloud information constitutes multiple layers. Note that a point group is a collection of feature points.

The extraction unit 140 extracts object point cloud information from the three-dimensional colored point cloud information using a predetermined method. The target object point cloud information is point cloud information of a plurality of target objects. The processing executed by the extraction unit 140 will be explained in detail.

The extraction unit 140 clusters the colors indicated by the three-dimensional colored point cloud information when the number of colors indicated by the three-dimensional colored point cloud information is equal to or greater than a predetermined threshold. For example, the extraction unit 140 performs clustering using the k-means method. For example, a deep learning method such as a convolutional neural network may be used for clustering. Specifically, the extraction unit 140 extracts RGB values, ratios of RGB values, values based on the HSV color space (for example, hue, saturation, and brightness), or values based on the Lab color space (for example, hue, saturation, and brightness). , and brightness) to cluster the colors indicated by the three-dimensional colored point cloud information. Further, the extraction unit 140 may cluster the colors indicated by the three-dimensional colored point cloud information using RGB values, ratios of RGB values, values based on the HSV color space, and values based on the Lab color space. In clustering, similar colors are grouped into one color. For example, by combining colors, silver is changed to gray. This reduces the number of colors.

Furthermore, when the plurality of images are generated by a monochrome camera such as an infrared camera or an X-ray camera, the extraction unit 140 clusters the colors indicated by the three-dimensional colored point cloud information based on the brightness. By using an infrared camera, it is possible to perform reinforcement inspection at night or in a dark area, for example. Furthermore, by using an X-ray camera, it is possible to inspect the reinforcement arrangement even in a state where the reinforcing bars cannot be directly photographed, such as after concrete is placed.

The extraction unit 140 extracts object point cloud information from the three-dimensional colored point cloud information. The extraction process will be explained in detail. First, the acquisition unit 120 acquires color information corresponding to the target object from the storage unit 110 or an external device. For example, the color is gray. Specifically, the color information is an RGB value. The extraction unit 140 extracts target object point cloud information from the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information and color information corresponding to the target object. Further, the three-dimensional colored point cloud information is unclustered three-dimensional colored point cloud information or clustered three-dimensional colored point cloud information. The extraction unit 140 removes points other than the object from the three-dimensional colored point cloud information.

Here, the processing executed by the extraction unit 140 will be explained using a specific example.
FIGS. 4A and 4B are diagrams showing specific examples of processing executed by the extraction unit. First, it is assumed that the number of colors indicated by the three-dimensional colored point group information is greater than or equal to a threshold value. The extraction unit 140 clusters the colors indicated by the three-dimensional colored point cloud information. FIG. 4(A) shows clustered three-dimensional colored point cloud information. Note that, for example, the straight line shown in FIG. 4(A) is expressed by a plurality of points. FIG. 4A shows a state in which three-dimensional colored point cloud information is clustered into two colors. The two colors are gray and red. The gray areas are reinforcing bars. The red area is the ground.

The extraction unit 140 extracts gray areas from the three-dimensional colored point cloud information as object point cloud information based on the three-dimensional colored point cloud information and information on the color (i.e., gray) corresponding to the object. Extract. As a result, point cloud information indicating reinforcing bars is extracted. The extraction unit 140 removes point clouds other than the object point cloud information from the three-dimensional colored point cloud information. FIG. 4(B) shows the state after the removal has been performed. Therefore, FIG. 4(B) shows target object point group information.

In the above, a case has been described in which object point cloud information is extracted using color information corresponding to the object. The extraction unit 140 may extract target object point group information by the following method. A method for extracting object point group information will be explained using figures.

FIG. 5 is a diagram illustrating an example of a method for extracting object point group information. The extraction unit 140 removes points that are not clustered or points that do not have a certain density from the three-dimensional colored point cloud information. Note that the point or the point group is noise. If the point or the point group does not exist in the three-dimensional colored point cloud information, the extraction unit 140 does not perform the removal process. Further, the three-dimensional colored point cloud information is unclustered three-dimensional colored point cloud information or clustered three-dimensional colored point cloud information. The extraction unit 140 estimates the shape of the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information. The extraction unit 140 estimates a three-dimensional direction based on the estimated shape. Note that the direction may be a general direction. The extraction unit 140 extracts the surface of the top layer based on the three-dimensional direction. The extraction unit 140 calculates the average value of the color values (ie, RGB values) of the surface of the top layer. The extraction unit 140 may calculate the mode of the color values on the surface of the top layer. Further, the extraction unit 140 may calculate the median color value of the surface of the top layer. The extraction unit 140 can use various methods to determine the color of the object from the color value of the surface of the top layer using statistics. The extraction unit 140 uses the average value or the mode as the color of the object. The extraction unit 140 extracts object point cloud information from the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information and the average value or the mode. In addition, when the average value is used, the extraction unit 140 may extract target object point cloud information from the three-dimensional colored point cloud information by using a value close to the average value as the color of the target object. Here, in the method using color information corresponding to the target object, the color information corresponding to the target object may be stored in the storage unit 110. Therefore, in this method, the storage capacity of the storage unit 110 may be increased. However, the method of using the average value or the mode can be implemented without storing information in the storage unit 110. Therefore, the information processing device 100 can suppress an increase in storage capacity.

After the target object point cloud information is extracted, the calculation unit 150 calculates the point cloud density of the target object point cloud information.

Here, as shown in FIG. 4(B), the target object point group information is arranged in a three-dimensional virtual space. The object point group information may be arranged according to the actual direction (for example, the direction of gravity). The process of adjusting the direction will be described below.

If an IMU (Inertial Measurement Unit) sensor is installed in the information processing device 100, the direction control unit 160 acquires information on the direction of gravity from the IMU sensor. The direction control unit 160 aligns the direction perpendicular to the surface of the top layer of the object point cloud information with the direction of gravity. As a result, the direction of the Z axis in FIG. 4(B) coincides with the direction in which layers other than the top layer exist. Note that the surface of the uppermost layer of the object point cloud information may be detected using, for example, a region extraction algorithm such as a region expansion method.

The gravity direction may be detected as follows. The direction control unit 160 detects a plurality of intersection points based on an image including the surface of the top layer. The intersection detection process will be explained using figures.

FIG. 6 is a diagram illustrating an example of intersection detection processing. FIG. 6 shows an image 200. Image 200 is an image that includes the surface of the top layer. Direction control unit 160 detects intersections 201 to 203 based on image 200. Note that Hough transform may be used to detect the intersection. The direction control unit 160 calculates the direction of gravity based on the plurality of intersection points. For example, the direction control unit 160 detects a triangle whose vertices are intersection points 201 to 203. The direction control unit 160 detects the direction perpendicular to the triangle as the direction of gravity.

The detection unit 170 detects layers other than the top layer among the plurality of layers based on the object point cloud information and the calculated point cloud density. The detection process will be specifically explained using figures.

FIG. 7 is a diagram showing an example of detection processing. The upper part of FIG. 7 shows object point cloud information. For example, the detection unit 170 detects a layer other than the top layer (for example, the second layer) by matching a location where the point cloud density is high with a location in the layer of target object point cloud information. In addition, point cloud information tends to have a lower point cloud density in lower layers due to occlusion. Therefore, the detection unit 170 may detect layers other than the top layer based on the tendency.

The inspection unit 180 inspects objects present in layers other than the top layer. Specifically, the acquisition unit 120 acquires test content information indicating the test content. For example, the inspection content is the content of a reinforcement inspection. The inspection unit 180 inspects objects present in layers other than the top layer based on the inspection content information.

In addition, the inspection unit 180 detects objects present in layers other than the uppermost layer using the point cloud density, and inspects objects present in layers other than the top layer based on the inspection content information. You can. The inspection process will be explained using figures.

FIGS. 8A and 8B are diagrams illustrating an example of inspection processing. FIGS. 8A and 8B show the second layer of reinforcing bars (ie, the object). In reinforcement inspection, the spacing between reinforcing bars is inspected. However, there are places where the reinforcing bars intersect, as shown by the circle 300. In order to improve the accuracy of the inspection, the inspection unit 180 needs to accurately detect the position of the reinforcing bars. Therefore, the inspection unit 180 uses point cloud density. Thereby, the inspection unit 180 can accurately detect the position of the reinforcing bars. In this way, the information processing apparatus 100 can perform highly accurate inspection by using the point cloud density.

Moreover, FIG. 8(A) shows the relationship between the position of the reinforcing bars and the point group density in the X direction. FIG. 8(B) shows the relationship between the position of the reinforcing bars and the point group density in the Y direction.

The output unit 190 outputs the test results. For example, if the information processing device 100 has a display, the output unit 190 outputs the test results to the display. Further, for example, the output unit 190 outputs the test results to an external device.

Note that, in the above, the direction control unit 160 calculates the gravity direction based on a plurality of intersection points, but the direction control unit 160 is not limited to this. For example, the direction control unit 160 may calculate the gravitational direction using the point group density. For example, the degree of severity of change in the point group density distribution (that is, the sharpness of the peak of the graph of the point group density distribution (for example, kurtosis)) may be used as the point group density. When the reinforcing bars are photographed from the top layer, the degree of overlap of the reinforcing bars in each layer differs depending on the photographing angle. In the point group density distribution, the sharper the peak of the graph becomes as the degree of overlap of the reinforcing bars in each layer becomes stronger. In other words, when the kurtosis of the graph of the point group density distribution is the strongest, the reinforcing bars of each layer overlap most. In other words, the photographing direction of the camera and the surface of the top layer are at a position opposite to each other. In other words, the camera is located on a perpendicular line to the surface of the top layer. Under this condition, the direction in which the peak value of the graph of the point cloud density distribution gradually decreases can be detected as the direction of gravity in the lower layer. Therefore, the direction control unit 160 can detect the direction of gravity without specifying the top layer by using the point group density. Furthermore, since the actual direction is not limited to the direction of gravity, it is possible to inspect reinforcing bars arranged in a direction different from the direction of gravity, such as walls and columns, for example.

Next, the processing executed by the information processing apparatus 100 will be explained using a flowchart.
FIG. 9 is a flowchart illustrating an example of processing executed by the information processing device.
(Step S11) The acquisition unit 120 acquires a plurality of images obtained by imaging a plurality of objects from a plurality of different angles.
(Step S12) The creation unit 130 creates three-dimensional colored point cloud information based on the plurality of images.

(Step S13) The extraction unit 140 clusters the colors indicated by the three-dimensional colored point cloud information.
(Step S14) The extraction unit 140 extracts target object point cloud information from the clustered three-dimensional colored point cloud information based on the clustered three-dimensional colored point cloud information and color information corresponding to the target object. Extract.

(Step S15) The calculation unit 150 calculates the point cloud density of the target object point cloud information.
(Step S16) The detection unit 170 detects layers other than the top layer based on the object point cloud information and the point cloud density.
(Step S17) The inspection unit 180 inspects objects present in layers other than the top layer.
(Step S18) The output unit 190 outputs the test results.

According to the embodiment, the information processing apparatus 100 can improve the accuracy of detecting layers other than the top layer by using point cloud density.

In the above, the case where the target object is a reinforcing bar has been explained. However, the object may be other than reinforcing steel. Examples of objects other than reinforcing bars include steel frames, scaffolding, wooden structures, piping, and cables. Therefore, the embodiments can be applied even when a plurality of layers are made of materials other than reinforcing bars.

Moreover, although the case where layers other than the top layer are detected has been described above, layers including the top layer may be detected, and the same effect as in the case of layers other than the top layer is achieved.

100 information processing device, 101 processor, 102 volatile storage device, 103 nonvolatile storage device, 110 storage section, 120 acquisition section, 130 creation section, 140 extraction section, 150 calculation section, 160 direction control section, 170 detection section, 180 Inspection section, 190 Output section, 200 Image, 201-203 Intersection, 300 yen.

Claims (6)

an acquisition unit that acquires a plurality of images obtained by imaging a plurality of objects from a plurality of different angles;
a creation unit that creates three-dimensional colored point cloud information constituting a plurality of layers based on the plurality of images;
Based on the three-dimensional colored point cloud information, estimate the shape of the three-dimensional colored point cloud information, estimate the three-dimensional direction based on the shape, and estimate the three-dimensional direction based on the three-dimensional direction. The surface is extracted, the average value of the color values of the surface of the top layer or the mode of the color values of the surface of the top layer is calculated, and the average value or the mode of the color values of the surface of the top layer is calculated. an extraction unit that extracts object point cloud information that is point cloud information of the plurality of objects from the three-dimensional colored point cloud information based on the value ;
a calculation unit that calculates the point cloud density of the target object point cloud information;
a detection unit that detects a layer other than the top layer among the plurality of layers based on the object point cloud information and the point cloud density;
An information processing device having: The extraction section is
If the number of colors indicated by the three-dimensional colored point cloud information is equal to or greater than a predetermined threshold, clustering the colors indicated by the three-dimensional colored point cloud information;
Estimating the shape of the clustered three-dimensional colored point group information based on the clustered three-dimensional colored point group information,
Estimating a three-dimensional direction based on the shape,
extracting the surface of the top layer based on the three-dimensional direction;
Calculating the average value of the color values of the surface of the top layer or the mode of the color values of the surface of the top layer,
extracting the object point cloud information from the clustered three-dimensional colored point cloud information based on the clustered three-dimensional colored point cloud information and the average value or the mode;
The information processing device according to claim 1 . Inspection department and
an output section;
It further has
The acquisition unit acquires test content information indicating test content,
The inspection unit inspects the object present in a layer other than the top layer based on the inspection content information,
The output unit outputs the test results.
The information processing device according to claim 1 or 2 . The inspection department includes:
Detecting the object present in a layer other than the top layer using the point cloud density,
performing an inspection on the object existing in a layer other than the top layer based on the inspection content information;
The information processing device according to claim 3 . The information processing device
Obtain multiple images obtained by capturing multiple objects from multiple different angles,
creating three-dimensional colored point cloud information constituting a plurality of layers based on the plurality of images,
Estimating the shape of the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information,
Estimating a three-dimensional direction based on the shape,
extracting the surface of the top layer based on the three-dimensional direction;
Calculating the average value of the color values of the surface of the top layer or the mode of the color values of the surface of the top layer,
Extracting object point cloud information, which is point cloud information of the plurality of objects, from the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information and the average value or the mode value. death,
Calculating the point cloud density of the target object point cloud information,
Detecting layers other than the top layer among the plurality of layers based on the object point cloud information and the point cloud density;
Detection method. In the information processing device,
Obtain multiple images obtained by capturing multiple objects from multiple different angles,
creating three-dimensional colored point cloud information constituting a plurality of layers based on the plurality of images,
Estimating the shape of the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information,
Estimating a three-dimensional direction based on the shape,
extracting the surface of the top layer based on the three-dimensional direction;
Calculating the average value of the color values of the surface of the top layer or the mode of the color values of the surface of the top layer,
Extracting object point cloud information, which is point cloud information of the plurality of objects, from the three-dimensional colored point cloud information based on the three-dimensional colored point cloud information and the average value or the mode value. death,
Calculating the point cloud density of the target object point cloud information,
Detecting layers other than the top layer among the plurality of layers based on the object point cloud information and the point cloud density;
A detection program that performs processing.

Images