JP7304785B2 - Control device for laser point cloud acquisition device with camera, control method for laser point cloud acquisition device with camera, and program for controlling laser point cloud acquisition device with camera - Google Patents

Description

The present invention relates to control of laser scanners with cameras.

There is a camera-equipped laser scanner in which a laser scanner and a camera are combined (see, for example, Patent Document 1).

JP 2019-128196 A

As one method of using laser scan data, there is a technique of obtaining a three-dimensional model of an object to be measured based on point cloud data (laser scan point cloud) obtained by laser scanning. The point cloud data has a large amount of data, requires a large data storage capacity, and has the problem of placing a heavy load on hardware (for example, a PC, etc.) when performing processing.

In view of this background, an object of the present invention is to obtain a technique for suppressing the amount of measurement data obtained by a camera-equipped laser point group acquisition device.

The present invention is a device for controlling a laser point cloud acquisition device with a camera, which is a combination of a laser point cloud acquisition device with a known relationship of exterior orientation elements and a camera, wherein the camera of the laser point cloud acquisition device with a camera an image data acquisition unit that acquires image data of an image captured by a target object extraction unit that extracts a specific target object from the image; a designation reception unit that receives a designation of a part of the image; and the laser point cloud a laser point cloud acquisition range setting unit configured to limit the acquisition range of the laser point cloud by the acquisition device to the range for which the designation is accepted , wherein the specification of a part of the image includes the extracted specific target It is a device that controls a camera-equipped laser point cloud acquisition device that designates objects .

In the present invention, it is preferable that an object extracting unit extracting a specific object from the image is provided, and the extracted specific object is specified in the part of the image. In the present invention, it is preferable that the laser scanning density corresponding to the extracted specific object is selected.

The present invention can also be understood as the invention of a method and the generation of a program.

ADVANTAGE OF THE INVENTION According to this invention, the technique which suppresses the data amount of the measurement data which the laser point-group acquisition apparatus with a camera acquired is obtained.

1 is an external view of a camera-equipped laser scanner according to an embodiment; FIG. 1 is a block diagram of a laser scanner control device and a camera-equipped laser scanner in an embodiment; FIG. 10 is a drawing-substituting photograph showing an image taken by a camera of a laser scanner with a camera. 10 is a drawing-substituting photograph showing an image taken by a camera of a laser scanner with a camera. 4 is a flow chart showing an example of a procedure of processing;

1. First Embodiment (Structure of Laser Scanner with Camera)
FIG. 1 shows a camera-equipped laser scanner 100 as an example of a camera-equipped laser point group acquisition device. A camera-equipped laser scanner 100 has a horizontal rotating portion 101, a support portion 102, a base portion 103 on the top of the tripod, and a tripod 107 supporting the base portion 103 from below.

The horizontal rotation section 101 horizontally rotates with respect to the support section 102 . A vertical rotation section 104 is arranged in the horizontal rotation section 101 . The vertical rotation section 104 rotates vertically with respect to the horizontal rotation section 101 . An optical system 105 for outputting and detecting a pulsed light for distance measurement for laser scanning is arranged in the vertical rotating section 104 .

With the horizontal rotation unit 101 horizontally rotated and the vertical rotation unit 104 vertically rotated, the optical system 105 repeatedly irradiates the distance measuring light outward at a repetition frequency of several kHz to several tens of kHz. Laser scanning is performed by detecting the reflected light with the optical system 105 .

By performing the above-described laser scanning, laser scanning data whose origin is the optical origin (optical center) of the optical system 105, which is the viewpoint of the laser scanning, is obtained. The laser scan data is obtained by acquiring data on the distance and direction from the optical origin described above at each scan point (reflection point of scan light). Then, data describing the position of each scanning point on an appropriate coordinate system becomes a laser scanning point cloud (3D point cloud data obtained by laser scanning).

Here, if the exterior orientation elements (the position of the optical origin and the attitude of the optical system 105) in the absolute coordinate system of the camera-equipped laser scanner are known, the laser scanning point cloud in the absolute coordinate system can be obtained. The absolute coordinate system is a coordinate system used when describing GNSS and map information.

The relationship between the exterior orientation elements of the camera 106 and the exterior orientation elements of the optical system 105 constituting the laser scanner is known. Therefore, it is possible to determine the correspondence relationship between the image captured by the camera 106 and the laser scan range.

In this example, the camera-equipped laser scanner 100 outputs laser scan data and image data of an image captured by the camera 106 . The imaging by the laser scanner and the camera 106 can be performed simultaneously or separately.

(Block diagram of laser scanner with camera)
A block diagram of the camera-equipped laser scanner 100 is shown in FIG. 2 includes a camera 106, a distance measuring unit 111, a horizontal rotation angle detection unit 112, a vertical rotation angle detection unit 113, a horizontal rotation control unit 114, a vertical rotation control unit 115, and a laser scan data output unit .

Data of an image (image data) captured by the camera 106 is sent to a laser scanner control device 200, which will be described later. At this time, information about the direction of the image (the direction of the image center, the direction of the optical axis of the camera 106 ) and the angle of view (shooting range) is also sent to the laser scanner control device 200 .

A distance measuring unit 111 calculates a distance to an object based on distance measuring light (scanning light) emitted from the optical system 105 . The method of calculating the distance is the same as that of a normal laser rangefinder. As methods for calculating the distance, there are (1) a method of calculating from the propagation time of the distance measuring light, and (2) the detection timing of the distance measuring light and the reference light propagated through the reference optical path provided inside the laser scanner 100 with camera. can be calculated from the difference in detection timing (phase difference).

A horizontal rotation angle detection unit 112 detects the horizontal rotation angle of the horizontal rotation unit 101 . The vertical rotation angle detection unit 113 detects the vertical rotation angle of the vertical rotation unit 104 . Angle detection is performed by a rotary encoder.

The horizontal rotation control unit 114 controls horizontal rotation of the horizontal rotation unit 101 . The actual structure includes a driving mechanism for horizontally rotating the horizontal rotation unit 101, a motor for applying driving force to this driving mechanism, and a control circuit for this motor. A control signal to this control circuit is output from the horizontal rotation control section 114 .

The vertical rotation control unit 115 controls vertical rotation of the vertical rotation unit 104 . The actual structure includes a driving mechanism for vertically rotating the vertical rotating portion 104, a motor for applying driving force to this driving mechanism, and a control circuit for this motor. A control signal to this control circuit is output from the vertical rotation control section 115 .

The laser scan data creation unit 117 creates laser scan data. The laser scan data is data on the distance from the optical origin (optical center) of the optical system 105 and the direction from the optical origin for each scan point. The distance to each scan point is calculated by the distance measuring unit 111 . The direction of each scan point is calculated based on the horizontal rotation angle of the horizontal rotation unit 101 detected by the horizontal rotation angle detection unit 112 and the vertical rotation angle of the vertical rotation unit 104 detected by the vertical rotation angle detection unit 118 .

The laser scan data directly obtained by laser scanning are the distance and direction of each scan point from the optical origin (laser scan viewpoint), and each scan point is described on a polar coordinate system. The laser scan data creation unit 117 may create a coordinate transformation of this polar coordinate system into a three-dimensional rectilinear orthogonal coordinate system.

(laser scanner controller)
A block diagram of the laser scanner controller 200 is shown in FIG. The laser scanner control device 200 controls the laser scanning range of the camera-equipped laser scanner 100 based on the image captured by the camera 106 of the camera-equipped laser scanner 100 in FIG. The laser scanner control device 200 is configured using a commercially available PC (personal computer).

The laser scanner control device 200 functions as a computer. The function of each functional unit provided in the laser scanner control device 200 is achieved by installing application software for realizing each functional unit shown in FIG. It is realized by being A part or all of each functional unit may be configured by various processors and electronic circuits. Moreover, at least part of the above functional units may be implemented using an external PC (personal computer) or a computing unit of a server.

The laser scanner control device 200 includes an image data acquisition section 201 , an object extraction section 202 , a designation acceptance section 203 , a laser scan range setting section 204 and a control signal output section 205 . In addition, the laser scanner control device 200 has a CPU, a storage device (semiconductor memory, hard disk device, etc.), a communication device, an image display device, and a user interface device, which are provided in a normal PC.

The image data acquisition unit 201 acquires image data of an image captured by the camera 106 of the laser scanner 100 with camera. This image data is associated with data regarding the orientation of the camera 106 at the time of photographing, and the data regarding the orientation of the camera is also obtained by the image data obtaining unit 102 .

A target object extraction unit 202 extracts a specific target object from the image captured by the camera 106 . A specific object is predetermined. For example, objects whose appearances are known in advance, such as pipes, fluorescent lamps, wiring, generators, and pumps, are registered as a library, and images that match their appearances are extracted. This is realized using image authentication technology and image recognition technology that have been put into practical use in the security field and the like. 3 and 4 are photographs substituting for drawings showing images and objects extracted from them.

A designation reception unit 203 receives designation of a part of the image captured by the camera 106 . For example, the image shown in FIG. 3 is displayed on the display screen of the PC that constitutes the laser scanner control device 200 . In this state, the user designates a specific portion (for example, the fluorescent lamp portion) on the image in FIG. 3 using the GUI function of the PC. Then, the designation reception unit 203 receives an instruction related to this designation.

The laser scan range setting unit 204 performs processing for setting a limited area including the object accepted by the designation acceptance unit 203 as a laser scan range.

Here, the following processing is performed. First, the direction of a designated object starting from the optical origin of the camera 106 and the range of this object are obtained. The direction of the object with respect to the optical origin (projection origin) of the camera 106 can be obtained by analyzing the image of the object. Here, the optical axis of the camera 106 at the time of photographing this image, that is, the direction of the center of the image can be known from the orientation of the camera 106 at the time of photographing. Also, from the two-dimensional position in the image of the object, the relative relationship between the direction of the center of the image of the object and the direction of the object can be known. From this, the direction of the object originating from the optical origin of the camera 106 of the object is known.

On the other hand, since the relationship between the camera 106 and the exterior orientation elements of the optical system 105 is known, the direction and range of the object viewed from the optical origin of the optical system 105 can be determined from the above analysis. Using this result, the range in which the object exists is set as the laser scanning range. This processing is performed by the laser scan range setting unit 204 .

For example, assume that the pipe portion in FIG. 4 is specified as the object. In this case, the direction and range of the pipe viewed from the camera-equipped laser scanner (the optical origin of the optical system 105) are set as the laser scanning range. This processing is performed by the laser scan range setting unit 204 .

The control signal output unit 205 generates a control signal related to the laser scanning range set by the laser scanning range setting unit 204 and sends it to the camera-equipped laser scanner 100 . For example, a control signal that defines the laser scanning range by horizontal and vertical angles is generated and sent to the camera-equipped laser scanner 100 .

Based on this control signal, the laser scanner 100 with a camera performs a laser scan for a specified range.

(Example of processing)
FIG. 5 is a flowchart showing an example of processing performed by laser scanner control device 200 . A program for executing the processing in FIG. 5 is stored in a storage unit or an appropriate storage medium provided in the laser scanner control device 200, read from there, and executed by the CPU of the PC that constitutes the laser scanner control device 200. It is also possible to store this program in a server and download it from there via the Internet.

Prior to the process of FIG. 5, the camera 106 of the camera-equipped laser scanner 100 is used to photograph the surroundings of the camera-equipped laser scanner 100 . Here, the whole circumference (hemispherical surface) including the upper part is photographed. Imaging is performed all around. At this time, the entire circumference is divided into a plurality of areas, and photographing is performed for each divided range. This is the same as the normal panorama image creation method. A panorama image is created using the plurality of images.

In addition, data on the direction of the optical axis (shooting direction) of the camera 106 relating to each image is also acquired in association with the corresponding image data.

When the process is started, image data (panoramic image data) captured by the camera 106 is first acquired by the image data acquiring unit 201 (step S201). Here, the acquired image data is associated with the direction of the target area of the image (the direction viewed from the camera-equipped laser scanner 200), and the data is also acquired at the same time.

Next, an image of an object registered in advance is extracted from the image data acquired in step S201 (step S202). This processing is performed by the object extraction unit 202 . In this process, for example, objects such as the fluorescent lamp in FIG. 3 and the pipe in FIG. 4 are extracted using image recognition technology.

Next, the user's specification of the object extracted in step S202 is accepted (step S203). This processing is performed by the designation reception unit 203 . In this process, an image is displayed on the screen of the PC used as the laser scanner control device 200 so that the object extracted in step S202 can be grasped (visually recognized), and the object is displayed by the user using the GUI. Accept designation of things. For example, a user designates an object by left-clicking with a mouse.

Next, a laser scan range including the object accepted in step S203 is set (step S204). This processing is performed by the laser scan range setting unit 204 . In this process, an area slightly wider than the designated object area is set as the laser scanning area. Care must be taken here so that the object does not fall outside the laser scanning range. On the other hand, if too much leeway is taken, the portion of unnecessary laser scanning increases. Therefore, for example, an area that is 5 to 20% wider than the specified target area is set as the laser scanning area.

After setting the laser scanning range in step S204, a control signal reflecting the contents of this setting is output to the camera-equipped laser scanner 100 (step S205). This process is performed by control signal output section 205 .

(Superiority)
Using the image, an object requiring laser scanning is selected, and a laser scanning range limited to this object is set. Therefore, the problem of unnecessary laser scanning is suppressed, and the amount of laser scan data is suppressed.

In addition, since the range of laser scanning can be limited, high-density point cloud data can be acquired by concentrating on a specific target.

2. Second Embodiment In the first embodiment, it is also possible to specify or select the scan density (the density of the point cloud to be acquired) corresponding to the object (pipe, pump, etc.) extracted from the image. is.

For example, when an object is specified in an image, a point cloud density (resolution) menu is displayed by right-clicking the mouse. This menu includes items such as 1 point/cm ² , 2 points/cm ² , 4 points/cm ² .

Moreover, it is also possible to set the scan density (point cloud density) to be acquired in advance in correspondence with the target object. It is also possible to present this setting to the user as a recommended acquisition density. For example, if the object is a pipe, n points/cm ² , and if the object is lighting equipment, m points/cm ² . be. This scan density is set by the laser scan range setting unit 204 .

Depending on the object, there are cases where an appropriate point cloud density is known, for example, N points/cm ² for pipes, and M points/cm ² for lighting equipment. In such a case, the above aspect is effective.

3. Third Embodiment In the first embodiment, it is also possible to provide different scan densities according to the shape of the object. For example, even in the pump to be measured, the point density is made coarser in the plane portion, and the portion with changes such as the curved portion is scanned densely. By doing so, it is possible to obtain point cloud data capable of faithfully reproducing the shape of the object while pursuing efficiency in the work related to acquiring the point cloud. It is also possible to recognize the flat portion and curved portion of the object to be measured by image recognition technology and change the scan density accordingly.

4. Fourth Embodiment As a laser point cloud acquisition device with a camera, a combination of a camera and a laser scanner was exemplified. ) using a laser beam can be used. For example, as a camera-equipped laser point cloud acquisition device, a combination of a camera and a flash lidar can be used.

LiDAR is described, for example, in Toshiba Review Vol. 73 No. 6 (November 2018) and Autonomous Driving Lab (October 12, 2018) (https://jidounten-lab.com/y_6506).

For example, a case of using a combination of a camera and a flash lidar as a laser point group acquisition device with a camera will be described. In the flash lidar, light-emitting elements and light-receiving elements are arranged in an array (matrix), and instead of scanning laser range-finding light, array-like (for example, m×n) laser range-finding light is emitted simultaneously, An array of point cloud data is obtained by receiving the reflected light.

In this case, by adjusting the pointing direction of the flash lidar, partial laser point cloud data acquisition for the object is performed. Alternatively, a portion of the array may be emitted and/or received to provide limited laser ranging to the object and partial laser point cloud acquisition.

DESCRIPTION OF SYMBOLS 100... Laser scanner with a camera 101... Horizontal rotation part 102... Support part 103... Base part 104... Vertical rotation part 105... Optical system 106... Camera

Claims (4)

A device for controlling a camera-equipped laser point cloud acquisition device that combines a laser point cloud acquisition device with a known relationship of exterior orientation elements and a camera,
an image data acquisition unit that acquires image data of an image captured by the camera of the laser point cloud acquisition device with a camera;
an object extraction unit that extracts a specific object from the image;
a designation receiving unit that receives designation of a part of the image;
a laser point cloud acquisition range setting unit configured to limit the acquisition range of the laser point cloud by the laser point cloud acquisition device to the range for which the designation is accepted;
with
A device for controlling a camera-equipped laser point group acquisition device in which the specified object extracted is specified in the specification of the part of the image . 2. The apparatus for controlling the camera-equipped laser point cloud acquisition device according to claim 1 , wherein a laser scanning density corresponding to the extracted specific object is selected. A method for controlling a camera-equipped laser point cloud acquisition device that combines a laser point cloud acquisition device with a known relationship between exterior orientation elements and a camera,
an image data acquisition step of acquiring image data of an image captured by the camera of the laser point cloud acquisition device with camera;
an object extraction step of extracting a specific object from the image;
a designation receiving step of receiving a designation of a portion of the image;
a laser point cloud acquisition range setting step for setting the acquisition range of the laser point cloud by the laser point cloud acquisition device to the range for which the designation is accepted;
A control method for a camera-equipped laser point group acquisition device , wherein the designation of the part of the image designates the extracted specific object . A program for controlling a camera-equipped laser point cloud acquisition device in which a camera and a laser point cloud acquisition device with known relationships of exterior orientation elements are combined,
an image data acquisition step of acquiring image data of an image captured by the camera of the laser point cloud acquisition device with camera;
an object extraction step of extracting a specific object from the image;
a designation receiving step of receiving a designation of a portion of the image;
a laser point cloud acquisition range setting step for setting the acquisition range of the laser point cloud by the laser point cloud acquisition device to the range for which the designation is accepted;
A program for controlling a camera-equipped laser point group acquisition device , wherein the specification of the part of the image specifies the extracted specific object .

Images