JP7374690B2 - Measurement data processing device, measurement data processing method, and measurement data processing program - Google Patents

Description

The present invention relates to a technique for processing measurement data obtained by a laser scanner with a camera.

There is a laser scanner with a camera that combines a laser scanner and a camera (for example, see Patent Document 1).

JP 2019-128196 Publication

One method of using laser scan data is a technique for obtaining a three-dimensional model of a measurement target based on point cloud data obtained by laser scanning (laser scan point cloud). Point cloud data has a problem in that it has a large amount of data, requires a large data storage capacity , and places a heavy burden on hardware (for example, a PC) when processing it.

Against this background, an object of the present invention is to obtain a technique for suppressing the amount of measurement data obtained by a laser scanner equipped with a camera.

The present invention is a measurement data processing device that processes measurement data obtained by a camera-equipped laser scanning device that combines a laser scanner and a camera with a known relationship between external orientation elements, the laser scan obtained by the laser scanner. a data acquisition unit that acquires image data of a point cloud and an image taken by the camera; a correspondence specifying unit that specifies a correspondence between the laser scan point cloud and the image; and a correspondence specifying unit that accepts specification of a part of the image. a designation reception unit; a laser scan point group extraction unit that extracts a laser scan point group corresponding to the object for which the designation has been accepted from the laser scan point group; and a 3D model based on the extracted laser scan point group. and a 3D model creation unit that creates a 3D model, in which data regarding the standards of the object is acquired in advance, and the standards of the object are read from the print or mark display of the object read from the image. The measurement data processing device obtains the 3D model by acquiring the shape of an object and applying the acquired shape of the object to the laser scan point group. This invention can also be regarded as a method invention and a program invention.

In the present invention, it is preferable that the image forming apparatus further includes a target object extracting section that extracts a specific target object from the image, and in specifying a part of the image, the extracted specific target object is specified.

In the present invention, in identifying the correspondence relationship, a composite image is created by projecting points constituting the laser scan point group onto an image taken by the camera, and the correspondence relationship is identified based on the composite image. The preferred embodiment is as follows.

According to the present invention, a technique for suppressing the amount of measurement data obtained by a laser scanner with a camera can be obtained.

FIG. 1 is an external view of a camera-equipped laser scanner in an embodiment. FIG. 1 is a block diagram of a laser scan data processing device in an embodiment. This is a photograph substituted for a drawing showing an image taken by a camera-equipped laser scanner. This is a photograph substituted for a drawing showing an image taken by a camera-equipped laser scanner. This is a photograph substituted for a drawing showing an example of a composite image in which points constituting a laser scan point group are projected onto an image taken by a camera. 3 is a flowchart illustrating an example of a processing procedure.

1. First embodiment (hardware configuration)
FIG. 1 shows a laser scanner 100 with a camera. The camera-equipped laser scanner 100 includes a horizontal rotation section 101, a support section 102, a base section 103 at the top of the tripod, and a tripod 107 that supports the base section 103 from below.

The horizontal rotation section 101 rotates horizontally with respect to the support section 102 . A vertical rotation section 104 is arranged in the horizontal rotation section 101 . The vertical rotation section 104 rotates vertically with respect to the horizontal rotation section 101. An optical system 105 is disposed in the vertical rotation unit 104 for outputting and detecting distance measuring pulsed light for laser scanning.

With the horizontal rotation unit 101 rotated horizontally and the vertical rotation unit 104 rotated vertically, the optical system 105 repeatedly irradiates distance measuring light outward at a repetition frequency of several kHz to several tens of kHz. Laser scanning is performed by detecting the reflected light by the optical system 105.

By performing the above laser scan, laser scan data whose origin is the optical origin of the optical system 105, which is the viewpoint of the laser scan, is obtained. The laser scan data is obtained by acquiring distance and direction data from the optical origin at each scan point (reflection point of the scan light). Then, data describing the position of each scan point on an appropriate coordinate system becomes a laser scan point group (3D point group data obtained by laser scanning).

Here, if the external orientation elements (the position of the optical origin and the attitude of the optical system 105 in the initial state) in the absolute coordinate system of the camera-equipped laser scanner are known, a group of laser scan points in the absolute coordinate system can be obtained. An absolute coordinate system is a coordinate system used when describing GNSS and map information.

The relationship between the external orientation elements of the camera 106 and the external orientation elements of the optical system 105 constituting the laser scanner is known. Therefore, it is possible to determine the correspondence between the captured image of the camera 106 and the laser scan range.

In this example, the camera-equipped laser scanner 100 outputs laser scan data and image data of an image captured by the camera 106. This laser scan data and image data are sent to a laser scan data processing device, which will be described later.

(Laser scan data processing device)
FIG. 2 shows a block diagram of a laser scan data processing device 200. As shown in FIG. The measurement data processing device 200 processes laser scan data output from the camera-equipped laser scanner 100 of FIG. 1 and image data of a photographed image. The laser scan data processing device 200 is configured using a commercially available PC (personal computer).

Laser scan data processing device 200 functions as a computer. The functions of each functional unit included in the laser scan data processing device 200 are determined by installing application software for realizing each functional unit shown in FIG. It is realized through execution. Part or all of each functional unit may be configured with various processors or electronic circuits. Further, at least a part of the above functional section may be implemented using an external PC (personal computer) or a calculation section of a server.

The laser scan data processing device 200 includes a data acquisition section 201, an object extraction section 202, a correspondence specification section 203, a specification reception section 204, a laser scan point group extraction section 205, and a 3D model creation section 206. In addition, the laser scan data processing device 200 includes a CPU, a storage device (semiconductor memory, hard disk device, etc.), a communication device, an image display device, and a user interface device, which are included in a normal PC.

The data acquisition unit 201 acquires image data of the laser scan point group acquired by the camera-equipped laser scanner 100 and the image photographed by the camera 106. The relationship (relationship between position and orientation) between the optical system 105 that performs the laser scanning function of the laser scanner 100 and the external orientation elements of the camera 106 is known. Therefore, after acquiring the laser scan data and the imaging data, it is possible to specify the correspondence between the laser scan range and the imaging range by the camera 106. Note that the laser scan and the photographing by the camera 106 may be performed synchronously or at different timings.

The object extraction unit 202 extracts a specific object from the image taken by the camera 106. The specific object is predetermined. For example, objects whose appearance is known in advance, such as pipes, fluorescent lights, wiring, generators, and pumps, are registered as a library, and images matching the appearance are extracted. This is realized using image recognition technology that has been put to practical use in image authentication technology and the like. 3 and 4 are photographs substituted for drawings showing images and objects extracted from the images.

The correspondence specifying unit 203 specifies the correspondence between the laser scan point group obtained by the laser scan function of the camera-equipped laser scanner 100 and the image photographed by the camera 106. Here, the above-mentioned correspondence relationship is specified using a composite image obtained by combining the laser scan point group and the photographed image (a composite image of the point group and the image).

The above composite image will be explained below. The external orientation elements of the optical system 105 and camera 106 responsible for the laser scanning function are known. Therefore, it is possible to know the relationship between the range of laser scanning and the angle of view (shooting range) of the corresponding photographed image.

By using the above relationship, it is possible to project the points of the laser scan point group obtained by the optical system 105 onto the image taken by the camera 106 and obtain a displayed composite image. FIG. 5 is an example of this composite image. As illustrated in FIG. 5, in the above composite image, the scan points forming the laser scan point group are displayed as points in the image.

In this composite image, the scan points and images overlap and correspond. Therefore, when the user specifies a specific region in the image, the scan point corresponding to that region can be specified. Therefore, it is possible to extract the laser scan point group obtained from the site in the image specified by the user.

For example, as illustrated in FIGS. 3 and 4, it is assumed that a target object (for example, a fluorescent lamp, a pipe, etc.) is extracted by the target object extraction unit 202 from a photographed image. In this case, when the user specifies a desired object, the specified object is identified on the image. This specification is performed, for example, using the GUI of the PC that constitutes the laser scan data processing device 200.

The designation reception unit 204 receives designation of a part of the image captured by the camera 106. For example, the image in FIG. 3 is displayed on the display screen of a PC that constitutes the laser scan data processing apparatus 200. In this state, the user uses the GUI function of the PC to specify the fluorescent light portion on the image of FIG. Then, an instruction related to this designation is accepted by the designation reception unit 204.

The laser scan point group extraction unit 205 extracts a laser scan point group corresponding to the object for which the designation has been received from the laser scan point group acquired by the camera-equipped laser scanner 100. For example, assume that the fluorescent lamp in FIG. 3 is recognized by image recognition, and that image is designated on the screen. In this case, a laser scan point group corresponding to the designated phosphor is obtained from a composite image (a composite image of a point cloud and an image) created by processing in the correspondence specifying unit 203. In this composite image, the laser scan point cloud and the image are superimposed, so by extracting the laser scan point cloud distributed in the specified image area, the laser scan point cloud acquired by the camera-equipped laser scanner 100 can be extracted. From there, a laser scan point group corresponding to the object for which the designation has been accepted can be extracted.

The 3D model creation unit 206 creates a 3D model based on the laser scan point group extracted by the laser scan point group extraction unit 205. The laser scan point group extracted by the laser scan point group extraction unit 205 is a point group related to the target object extracted from the image by the target object extraction unit 202. Here, the type of object to be extracted is known in advance, such as a fluorescent lamp or a pipe.

Therefore, for example, when the object is a pipe, it is assumed that the laser scan point group extracted by the laser scan point group extraction unit 205 is a point group related to the pipe. When modeling, conditions for fitting to the shape of the pipe can be adopted. As a result, the amount of calculation required for processing can be reduced, and generation of unnecessary noise and erroneous calculation can be suppressed.

It is also possible to read standards such as JIS standards or industry standards from an image, apply a 3D model based on the read standards, and obtain a 3D model from a laser scan point group. For example, suppose that a pipe is extracted from an image, and the standard of the pipe can be read from the print or mark displayed on the pipe. Here, data regarding the pipe standard is acquired in advance, and the corresponding pipe standard (shape and dimensions) is acquired from the standard read from the image. A 3D model corresponding to this known standard is then fitted to the laser scan point cloud. In this way, a more accurate 3D model can be obtained from laser scan data.

(Example of processing)
FIG. 6 is a flowchart showing an example of processing performed by the laser scan data processing device 200. A program that executes the process shown in FIG. 6 is stored in a storage unit included in the laser scan data processing device 200 or a suitable storage medium, read from there, and executed by the CPU of the PC that constitutes the laser scan data processing device 200. Ru. It is also possible to store this program on a server and download it from there via the Internet.

Prior to the process shown in FIG. 6, a laser scanner 100 with a camera is used to scan a laser and the range of the laser scan is photographed using a camera 106. When the process is started, first, laser scan data measured by the camera-equipped laser scanner 100 and image data of an image photographed by the camera-equipped laser scanner 100 are acquired (step S201). Here, in the acquired laser scan data and image data, the relationship among the scanning direction, scanning timing, and photographing direction is associated with each other. The process in step S201 is performed by the data acquisition unit 201.

Next, an image of the object registered in advance is extracted from the image data acquired in step S201 (step S202). This process is performed by the target object extraction unit 202. In this process, objects such as the fluorescent lamp in FIG. 3 and the pipe in FIG. 4 are extracted using image recognition technology, for example.

Next, the correspondence between the laser scan point group acquired in step S201 and the image is specified (step S203). This process is performed by the correspondence specifying unit 203.

Next, the user's designation of the object extracted in step S202 is accepted (step S204). This process is performed by the designation reception unit 204. In this process, an image in which the object extracted in step S202 can be grasped (visually recognized) is displayed on the screen of the PC used as the laser scan data processing device 200, and the user can use the GUI to Accepts specifications of objects. For example, a user specifies an object by left-clicking with a mouse.

Next, points corresponding to the target object received in step S204 are extracted from the laser scan point group received in step S201 (step S205). This processing is performed by the laser scan point group extraction unit 205. In this process, for example, when the pipe in FIG. 4 is designated as the object to be treated, a group of laser scan points corresponding to a portion of this pipe is extracted.

Next, a 3D model is created based on the laser scan point group related to the specific object extracted in step S205 (step S206). In this process, a three-dimensional model is created based on the shape characteristics of the object that are known at this point (for example, a cylindrical shape in the case of a pipe). This process is performed by the 3D model creation unit 206.

It is also possible to output the point cloud data related to the specific object obtained in step S205 to the outside or to store it in an appropriate storage area without performing the process in step S206.

(Superiority)
Since the use of unnecessary laser scan point clouds is restricted, the amount of measurement data obtained by the camera-equipped laser scanner can be reduced. Moreover, by performing three-dimensional modeling of the laser scan point group using the shape characteristics of the object obtained through image recognition, the calculations required for three-dimensional modeling can be reduced.

100...Laser scanner with camera 101...Horizontal rotation part 102...Support part 103...Base part 104...Vertical rotation part 105...Optical system 106...Camera

Claims (5)

A measurement data processing device that processes measurement data obtained by a camera-equipped laser scanning device that combines a laser scanner and a camera with a known relationship between external orientation elements,
a data acquisition unit that acquires image data of a laser scan point group acquired by the laser scanner and an image photographed by the camera;
a correspondence specifying unit that specifies a correspondence between the laser scan point group and the image;
a designation reception unit that accepts designation of a part of the image;
a laser scan point group extraction unit that extracts a laser scan point group corresponding to the object for which the designation has been accepted from the laser scan point group;
a 3D model creation unit that creates a 3D model based on the extracted laser scan point group;
Data regarding the standards of the object is obtained in advance,
The shape of the object is acquired by reading the standard of the object from the print or mark display of the object read from the image, and the acquired shape of the object is applied to the laser scan point group. A measurement data processing device that obtains the 3D model. comprising an object extraction unit that extracts a specific object from the image,
The measurement data processing device according to claim 1, wherein the specification of the part of the image specifies the extracted specific object. In identifying the correspondence relationship,
Creating a composite image by projecting points constituting the laser scan point group into an image taken by the camera;
The measurement data processing device according to claim 1 or 2, wherein the correspondence relationship is specified based on the composite image. A measurement data processing method for processing measurement data obtained by a camera-equipped laser scanning device that combines a laser scanner and a camera with a known relationship between external orientation elements, the method comprising:
a data acquisition step of acquiring image data of a laser scan point group acquired by the laser scanner and an image taken by the camera;
a correspondence identifying step of identifying a correspondence between the laser scan point group and the image;
a designation reception step of accepting designation of a part of the image;
a laser scan point group extraction step of extracting a laser scan point group corresponding to the object for which the designation has been accepted from the laser scan point group;
a 3D model creation step of creating a 3D model based on the extracted laser scan point group;
Data regarding the standards of the object is obtained in advance,
The shape of the object is acquired by reading the standard of the object from the printing or mark display of the object read from the image, and the acquired shape of the object is applied to the laser scan point group. A measurement data processing method for obtaining the 3D model. A program that causes a computer that processes measurement data obtained by a camera-equipped laser scanning device that combines a laser scanner and a camera with a known relationship between external orientation elements to read and execute the program,
a data acquisition step of acquiring image data of the laser scan point cloud acquired by the laser scanner and the image taken by the camera;
a correspondence identifying step of identifying a correspondence between the laser scan point group and the image;
a designation reception step of accepting designation of a part of the image;
a laser scan point group extraction step of extracting a laser scan point group corresponding to the object for which the designation has been accepted from the laser scan point group;
performing a 3D model creation step of creating a 3D model based on the extracted laser scan point group;
Data regarding the standards of the object is obtained in advance,
The shape of the object is acquired by reading the standard of the object from the print or mark display of the object read from the image, and the acquired shape of the object is applied to the laser scan point group. A measurement data processing program from which the 3D model is obtained.

Images