JP2023105835A - Environment map generation device, environment map generation method and program - Google Patents

Abstract

To provide an environment map generation device, an environment map generation method and a program which can prevent unnecessary reduction of three-dimensional point group data to be used for generation of an environment map.SOLUTION: An environment map generation device (1) comprises: a data acquisition unit (11) which acquires three-dimensional point group data of an object space measured by a LiDAR (2) and video data of the object space imaged by a camera (3); a feature point extraction unit (13) which extracts feature point data of an object in the object space from the three-dimensional point group data; a position posture estimation unit (14) which estimates a position and a posture of the LiDAR (2); an environment map generation unit (15) which generates an environment map of the object space on the basis of the three-dimensional point group data and estimation results of the position and the posture of the LiDAR (2); an object detection unit (16) which determines whether or not a movable object reflected on the video data moves; and a data removal unit (17) which does not remove the three-dimensional point group data of the movable object determined not to move, and removes the three-dimensional point group data of the movable object determined to move from among the three-dimensional point group data to be used for generation of the environment map data.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an environment map generation device, an environment map generation method, and a program.

SLAM (Simultaneous Localization And Mapping) is known as a conventional technique for generating a three-dimensional environment map corresponding to the real space. SLAM is a technique for estimating self-location and generating a three-dimensional environmental map at the same time. Among SLAMs, Visual SLAM uses images to estimate its own position and generate a three-dimensional environmental map. This Visual SLAM may be simply called SLAM.

In Visual SLAM, a camera continuously captures an image of the surrounding space, a three-dimensional measurement sensor continuously measures the surrounding space, and the image of the surrounding space and the coordinates of a three-dimensional point in the surrounding space are obtained. Dimensional point cloud data is acquired. A pixel in each frame image forming an image is associated with a corresponding three-dimensional point, and color information of the corresponding pixel is added to the three-dimensional point. Then, using the 3D point cloud data to which the color information is added, estimation of the self position (position of a 3D measurement sensor or the like) and generation of 3D environment map data are performed.

In addition, for example, a three-dimensional LiDAR (Light Detection And Ranging) is used for the three-dimensional measurement sensor. A three-dimensional LiDAR is a laser sensor that measures the three-dimensional coordinates of a target point using laser light.

In SLAM, self-position (position of LiDAR) is estimated using landmarks (landmarks) in the surrounding space as marks. In three-dimensional measurement of the surrounding space by LiDAR, an object moving in the surrounding space may be erroneously recognized as a landmark. In this case, due to the movement of an object that is erroneously recognized as a landmark, the landmark that is originally at a fixed position is recognized as if it has moved, and there is a possibility that the self-position cannot be estimated and lost. Moreover, even if the self-position can be estimated, an error occurs in the correspondence between the pixels between the frame images and the 3D points, and the accuracy of the 3D environment map is reduced.

In response to the above problems, for example, in Patent Document 1, measurement data of dynamic traffic participants such as pedestrians and vehicles are removed from the point cloud, which is a three-dimensional range image measured by LiDAR, and the remaining point cloud A method for executing SLAM is described using the data of . Since dynamic traffic participants may be erroneously recognized as landmarks, removing their measurement data from the point cloud can reduce the occurrence of problems caused by dynamic traffic participants.

JP 2019-207220 A

The conventional technique described in US Pat. No. 5,900,002 removes the predefined dynamic 3D point cloud data of traffic participants. For this reason, moving objects, such as dynamic traffic participants, but not moving during the measurement and not affecting the accuracy of the environment map when used as landmarks, can be used for 3D point clouds. Data is removed. In this case, since the three-dimensional point cloud data used for generating the environment map data is excessively reduced, there is a problem that the accuracy of the environment map is lowered.

The present disclosure solves the above problems, and provides an environment map generation device, an environment map generation method, and a program that can prevent excessive reduction of 3D point cloud data used to generate environment map data. for the purpose.

An environment map generation device according to the present disclosure includes a data acquisition unit that acquires 3D point cloud data, which is a set of 3D points obtained by measuring a target area by a sensor, and video data obtained by capturing an image of the target area by an imaging device; A feature point extraction unit that extracts feature point data of an object existing in a target region from point cloud data, a position and orientation estimation unit that estimates the position and orientation of a sensor using 3D point cloud data, and feature point data and the position and posture estimation results of the sensor. an object detection unit that determines whether or not the detected movable object is moving; and 3D points of the movable object that are determined not to be moving from the 3D point cloud data used to generate the environment map data. a data removal unit that does not remove the group data but removes the three-dimensional point cloud data of the movable object determined to be in motion.

According to the present disclosure, a movable object, which is an object that can move, is detected from objects appearing in video data, and it is determined whether or not the detected movable object is moving to generate environment map data. The three-dimensional point cloud data of the movable object determined not to be moving is not removed from the three-dimensional point cloud data used, but the three-dimensional point cloud data of the movable object determined to be moving is removed.
As a result, the environment map generation device according to the present disclosure can prevent excessive reduction of the three-dimensional point cloud data used to generate the environment map.

1 is a block diagram showing the configuration of an environment map generating device according to Embodiment 1; FIG. 4 is a block diagram showing the configuration of an object detection unit; FIG. FIG. 4 is a schematic diagram showing an outline of processing for detecting a movable object from a frame image; FIG. 10 is a schematic diagram showing an outline of determination processing as to whether or not a movable object is moving; It is a screen figure which shows the environmental map of embankment. It is a screen figure which shows the environmental map of the embankment which carried out some soil. 4 is a flowchart showing an environment map generation method according to Embodiment 1; 8A and 8B are block diagrams showing the hardware configuration for realizing the functions of the environment map generating device according to Embodiment 1. FIG.

Embodiment 1.
FIG. 1 is a block diagram showing the configuration of an environment map generating device 1 according to Embodiment 1. As shown in FIG. In FIG. 1, the environment map generation device 1 acquires 3D point cloud data obtained by measuring the target area by the LiDAR 2 and image data of the target area captured by the camera 3, and acquires the 3D point cloud data and the image data. is used to generate three-dimensional environmental map data for the region of interest. For example, the environment map generation device 1 is portable by the measurement operator and mounted on a terminal equipped with the LiDAR 2 and the camera 3 . The environment map generation device 1 may be provided separately from the LiDAR 2 and the camera 3 and may be a device capable of communicating with the LiDAR 2 and the camera 3 by wire or wirelessly.

LiDAR2 measures the distance by receiving the reflected light of the laser pulse irradiated to the target area at a predetermined measurement cycle (for example, every 100 milliseconds), and is a set of three-dimensional points specified by the measured distance. It is a sensor that detects three-dimensional point cloud data. LiDAR2 is mounted on a moving body. Examples of mobile objects include vehicles that can move around the target area, flying objects that can move over the target area, and terminals that can be carried and moved by workers. The LiDAR2 measures the target area while the moving object moves. The LiDAR 2 measures the distance to an object existing within the target area as the depth of a 3D point, and outputs 3D point cloud data, which is a set of 3D points.

The camera 3 is an imaging device that captures an image of a target area and obtains video data. The camera 3 may be a monocular camera or a stereo camera. The video data of the target area is composed of, for example, a plurality of frame images captured at a constant frame rate (eg, 20 fps). A frame image is RGB image data, and each pixel has color information.

Moreover, it is desirable that the imaging field of view of the camera 3 is equivalent to the measurement range of the LiDAR 2 . For example, the camera 3 is mounted on the moving object together with the LiDAR 2, and if the LiDAR 2 is a sensor that measures 360 degrees around, the camera 3 is also capable of imaging 360 degrees around. In addition, the number of cameras 3 is not limited to one, and a plurality of cameras 3 may be used to realize an imaging field of view equivalent to the measurement range of the LiDAR 2 .

Further, the positional relationship between the LiDAR 2 and the camera 3 has been calibrated in advance and correlated. For example, when the LiDAR 2 and the camera 3 are mounted on a moving object, and the moving object moves around the target area while performing measurement by the LiDAR 2 and imaging by the camera 3, the three-dimensional point cloud data measured by the LiDAR 2 is Synchronization is achieved for each frame of the video imaged by the camera 3 .

The data acquisition unit 11 acquires three-dimensional point cloud data, which is a set of three-dimensional points obtained by measuring the target area by the LiDAR 2, and video data obtained by imaging the target area by the camera 3. For example, the data acquisition unit 11 is connected to the LiDAR 2 and the camera 3 through wired or wireless communication. The data acquisition unit 11 acquires three-dimensional point cloud data obtained by measuring the target area with the LiDAR 2 at a predetermined measurement cycle, and video data of the target area captured by the camera 3 along with the measurement of the LiDAR 2. , and outputs the acquired data to the color information addition unit 12 .

The color information addition unit 12 specifies the correspondence between the 3D points that make up the 3D point cloud data and the pixels in the frame images that make up the video data, and adds the color information of the pixels corresponding to the 3D points. 3D point cloud data is generated. For example, since the positional relationship between the LiDAR 2 and the camera 3 is calibrated, the 3D point cloud data is synchronized for each frame of video data. Based on this positional relationship, the color information adding unit 12 identifies the positions of the three-dimensional points in the three-dimensional point cloud data and the positions of the corresponding pixels, and adds the color information of the pixels corresponding to the three-dimensional points. do.

Note that if the environment map data is generated using only the depth information of the 3D points in the 3D point cloud data, it is not necessary to add color information to the 3D points. In this case, the environment map generation device 1 does not have to include the color information adding section 12 . That is, the three-dimensional point cloud data acquired by the data acquisition unit 11 is directly output to the feature point extraction unit 13, and feature points are extracted from the three-dimensional point cloud data.

The feature point extraction unit 13 extracts feature point data of an object existing within the target area from the three-dimensional point cloud data. The feature point data is three-dimensional point data corresponding to points (feature points) that can be visually distinguished from other parts such as the surface, unevenness, or edges of an object existing in the target area and that can be regarded as having features. For example, the feature point extracting unit 13 extracts feature points of an object appearing in a frame image that constitutes video data, and extracts three-dimensional point data corresponding to pixels of the extracted feature points from a plurality of points included in the three-dimensional point cloud data. are extracted as feature point data from the three-dimensional point data of .

The position/orientation estimation unit 14 uses the feature point data extracted by the feature point extraction unit 13 to estimate the position and orientation of the LiDAR 2 that is mounted on or carried by a moving object. For example, the position/posture estimation unit 14 uses the feature point data to recognize landmarks in the target area. A landmark is a geographical feature whose position is fixed in the target area and serves as a geographical landmark.

The position/orientation estimation unit 14 expresses the positions of the landmarks by, for example, the center coordinates of the landmarks, and identifies distance data corresponding to the positions of the plurality of landmarks from the feature point data. The position/orientation estimation unit 14 uses SLAM technology to estimate the position and orientation of the LiDAR 2 at time t based on the distance data to the landmark obtained at time t−1 and the traveling direction of the moving object. The posture of the LiDAR 2 is, for example, the azimuth angle and elevation angle with respect to the traveling direction of the moving body. Odometry information indicating the estimation results of the position and orientation of the LiDAR 2 is output to the environment map generator 15 .

The environment map generation unit 15 generates environment map data of the target area based on the 3D point cloud data and the odometry information indicating the estimation result of the position and orientation of the LiDAR 2 . The environment map data is data representing a three-dimensional map constructed from three-dimensional point cloud data. Color information is added to each 3D point of the environment map. For example, the environment map generation unit 15 generates environment map data according to the position and orientation of the LiDAR 2 at time t−1, and generates environment map data according to the position and orientation of the LiDAR 2 at time t.

An object detection unit 16 detects a movable object among the objects appearing in the image data, and determines whether or not the detected movable object is moving. Movable objects are objects that can move, such as pedestrians, vehicles, etc., that are present in the region of interest. Vehicles also include construction machinery. For example, the object detection unit 16 detects a movable object from a frame image by performing image analysis such as pattern recognition for each frame image forming video data.
Note that the movable object may be determined for each application field of the environment map. For example, when an environmental map is used for management of civil engineering work, examples of movable objects include workers who are pedestrians, movable working machines, vehicles for transportation, and the like.

The object detection unit 16 determines that the movable object is moving when the position of the movable object shown in the frame image changes over time. For example, the object detection unit 16 determines whether the position of the movable object continuously changes between a plurality of consecutive frame images during a predetermined determination period, or when the position of the movable object changes between the frame images. If it occurs with a frequency exceeding the set threshold, it is determined that the movable object is moving. In addition, the object detection unit 16 detects intermittent changes in the position of the movable object between a plurality of consecutive frame images during a predetermined determination period, or when changes in the position of the movable object between the frame images occur intermittently. occurs at a frequency equal to or lower than the threshold, it is determined that the movable object is not moving.

In the measurement of the target area by the LiDAR2, the moving object equipped with the LiDAR2 is moving around the target area. When the movable object is moving, the position and orientation of the LiDAR2 change every moment, and the position of the movable object relative to the LiDAR2 also changes every moment. Therefore, it is difficult for the object detection unit 16 to accurately discriminate three-dimensional point cloud data corresponding only to moving movable objects.

The object detection unit 16 identifies, for example, an image region containing a movable object from the frame images that make up the video data, converts 3D point cloud data corresponding to the identified image region into a 3D point cloud corresponding to the movable object. Determined as data. That is, the three-dimensional point cloud data included in the rough range corresponding to the image area including the movable object is determined to be the three-dimensional point cloud data corresponding to the movable object determined to be moving. The object detection unit 16 outputs the determined three-dimensional point cloud data and the determination result indicating whether or not the corresponding movable object is moving to the data removal unit 17 as object detection information.

The data removal unit 17 does not remove the 3D point cloud data of the movable object determined not to be moving from the 3D point cloud data used by the environment map generation unit 15 to generate the environment map data, and determines that the object is moving. 3D point cloud data of the determined movable object is removed. The environment map generation unit 15 removes the 3D point cloud data corresponding to the movable object determined to be moving by the object detection unit 16, the remaining 3D point cloud data, and the estimation result of the position and orientation of the LiDAR 2 is used to generate environmental map data of the target area. In other words, even if a movable object is detected, if it does not move during measurement, and if the movable object is used as a landmark but does not affect the accuracy of the environment map, the 3D point cloud data corresponding to the movable object is not removed from the 3D point cloud data used to generate the environmental map data. As a result, the environment map generation device 1 can prevent excessive reduction of the three-dimensional point cloud data used to generate the environment map data.

Further, the data removal unit 17 removes movable objects determined by the object detection unit 16 not to move from the generated environment map data. The 3D point cloud data corresponding to movable objects that are determined not to be moving are not removed from the 3D point cloud data used to generate the environment map data. For this reason, the environment map data includes the movable object determined not to move, and the data removal unit 17 selects the movable object that is not moving from the three-dimensional point group data constituting the environment map data. The 3D point cloud data corresponding to the determined movable object can be accurately removed.

Further, the object detection unit 16 detects objects to be removed from among the objects appearing in the video data. The object to be removed is an obstacle that is fixedly present in the target area and blocks the three-dimensional measurement of the target area and the imaging of the target area. For example, in the generation of environment map data for embankments, trees (such as roadside trees) that are fixedly present between the embankment and the moving object, or temporarily installed signs, etc., are noise in the environment map.
Note that objects to be removed may be determined for each application field of the environment map. For example, when an environmental map is used to manage civil engineering work, objects to be removed include trees that may block the work area to be managed.

The data removal unit 17 removes the 3D point cloud data of the object to be removed from the 3D point cloud data used by the environment map generation unit 15 to generate the environment map data. As a result, removal target objects such as trees are excluded from the environment map of the target area, so the accuracy of the environment map can be improved. Since the removal target object is fixedly present in the target area, the object detection unit 16 can determine the three-dimensional point cloud data that accurately corresponds to the removal target object.

FIG. 2 is a block diagram showing the configuration of the object detection section 16. As shown in FIG. As shown in FIG. 2, the object detection unit 16 includes, for example, an object inference unit 161 and an object region identification unit 162. As shown in FIG. When video data is input, the object inference unit 161 detects a movable object using a trained model that infers whether or not a movable object is shown in a frame image forming the video data. A trained model is generated by a learning device (not shown in FIGS. 1 and 2), and parameters defining the trained model are stored in a storage device (not shown in FIGS. 1 and 2).

For example, the learning device learns a learned model for inferring whether or not a movable object is shown in the video data, using video data obtained by imaging the target area with the camera 3 as data for machine learning. A trained model is a neural network (NN) trained by deep learning. The learning device generates an NN (learned model) in which the NN weights are optimized by repeatedly updating the weights of the NN so as to improve the detection accuracy of the movable object.

When the learning device generates a learned model for inferring whether or not a movable object is shown in video data, the learning device stores parameters such as weights defining the learned model in a storage device. The object inference unit 161 infers a movable object using a learned model defined by parameters stored in a storage device. As a result, the environment map generation device 1 can improve the detection accuracy of movable objects.

As deep learning, for example, YOLO, EfficientDet, etc. are used. Further, as the learning of the trained model, the learning device may perform supervised learning using video data captured in the past as teacher data, or may perform unsupervised learning without using teacher data. good. The environment map generation device 1 can improve the detection accuracy of the movable object by the object detection unit 16 detecting the movable object using the learned model.

The object area identifying unit 162 identifies the image area of the movable object from the frame images forming the video data, and determines whether or not the movable object is moving based on the identified image area of the movable object. For example, the object region specifying unit 162 specifies an image region in which the movable object inferred by the object inferring unit 161 is shown, and based on the position change of the movable object included in the image region between frame images, the movable object determines whether it is running. The object detection unit 16 outputs to the data removal unit 17 object detection information including information indicating the image area in which the movable object is shown and the determination result as to whether or not the movable object is moving.

In addition, when video data is input, the object inference unit 161 detects an object to be removed using a trained model for inferring whether or not an object to be removed is shown in a frame image forming the video data. good too. For example, the learning device generates an NN (learned model) in which the NN weights are optimized by repeatedly updating the weights of the NN so as to improve the detection accuracy of the object to be removed.

Although the case where the learning device is provided separately from the environment map generation device 1 is shown, the environment map generation device 1 may be provided with the learning device. Moreover, the environment map generation device 1 may include a storage device that stores parameters such as weights that define the learned model.

FIG. 3 is a schematic diagram showing an overview of the process by which the object detection unit 16 detects a movable object from the frame image 3A. In FIG. 3, a frame image 3A is an example of a frame image included in the video data of the target area captured by the camera 3. As shown in FIG. In the frame image 3A, a building 21 fixedly present in the target area is shown, and a pedestrian 22 is shown. The object detection unit 16 analyzes the image data captured by the camera 3 for each frame image, or inputs the frame images of the image data into a trained model to detect walking, which is a movable object existing around the target area. person 22 is detected.

When the pedestrian 22 is detected, the object detection unit 16 specifies an image area 31 including the detected pedestrian 22, converts the 3D point cloud data corresponding to the specified image area into 3D points corresponding to the pedestrian 22, Discriminate as group data. The object detection unit 16 outputs the determined three-dimensional point cloud data and the corresponding determination result indicating whether or not the pedestrian 22 is moving to the data removal unit 17 as object detection information.

FIG. 4 is a schematic diagram showing an overview of the process of determining whether or not a movable object is moving, and shows a case where the pedestrian 22 is detected as the movable object. In FIG. 4, a frame image 3A1 is an image at time t-1 (seconds), and a frame image 3A2 is an image at time t (seconds) when the unit time has advanced by "1". In the frame image 3A2, the pedestrian 22 indicated by a solid line is the position of the pedestrian 22 at time t, and the pedestrian 22 indicated by a broken line is the position of the pedestrian 22 in the frame image 3A1, that is, the pedestrian at time t 22 positions are shown.

As indicated by arrows in FIG. 4, the position of pedestrian 22 changes between frame image 3A1 and frame image 3A2. When the object detection unit 16 detects a change in the position of the pedestrian 22 between the frame image 3A1 and the frame image 3A2, it determines that the pedestrian 22 is moving. Further, the object detection unit 16 may determine that the pedestrian 22 is moving when continuously detecting a change in the position of the pedestrian 22 between frame images within a predetermined determination period. Furthermore, the object detection unit 16 may determine that the movable object is moving when the pedestrian 22 intermittently changes position between frame images at a frequency exceeding a predetermined threshold.

The object detection unit 16 discriminates the three-dimensional point cloud data corresponding to the pedestrian 22 determined to be moving from the three-dimensional point cloud data used to generate the environment map data, and generates the determined three-dimensional point cloud data. And the determination result indicating whether or not the corresponding pedestrian 22 is moving is output to the data removal unit 17 as object detection information.

An environmental map used for managing the work of transporting embankment at a civil engineering work site will be described. FIG. 5 is a screen diagram showing an environmental map of the embankment 41. As shown in FIG. An environment map 15A shown on the left side of FIG. The environment map 15B shown on the right side of FIG. 5 is a three-dimensional map obtained by excluding the three-dimensional point group data corresponding to the excavator 42 from the environment map 15A. In addition, the excavator 42 is a loading machine that loads the soil of the embankment 41 onto a transporting machine.

In the environmental map 15A, the excavator 42 exists on the embankment 41 and shields the state of the embankment 41 . Therefore, in order to manage the state of the embankment 41 using the environmental map 15A, it is necessary to exclude the excavator 42 from the environmental map 15A. In this case, when the environment map generation device 1 detects the excavator 42 as a movable object while measuring the embankment 41, it generates the environment map 15B excluding the excavator 42. FIG.

For example, if the excavator 42 is moving while the embankment 41 is being measured, the environment map generating device 1 can extract three-dimensional points corresponding to the excavator 42 from the three-dimensional point cloud data used to generate the environmental map 15B. The group data are automatically excluded and the environment map 15B is generated using the remaining three-dimensional point cloud data. Also, if the excavator 42 is not moving, the environment map generation device 1 automatically excludes the three-dimensional point cloud data corresponding to the excavator 42 from the generated environment map 15B. Accordingly, the worker can accurately recognize the state of the embankment 41 by referring to the environmental map 15B.

FIG. 6 is a screen view showing an environmental map of the embankment 41 from which a part of the soil has been carried out. An environment map 15C shown on the left side of FIG. The excavator 42 is present above the embankment 41 and carries out soil from the embankment 41 . Therefore, even if the operator refers to the environmental map 15C, the excavator 42 is shielded and the worker cannot recognize how much soil 43 has been carried out from the embankment 41 . In this case, when the environment map generation device 1 detects the excavator 42 as a movable object, it generates an environment map 15D excluding the excavator 42 .

For example, if the excavator 42 is moving for the work of carrying out soil, the environment map generation device 1 generates three-dimensional point cloud data corresponding to the excavator 42 from the three-dimensional point cloud data used to generate the environment map 15D. are automatically excluded, and the environment map 15D is generated using the remaining three-dimensional point cloud data. Moreover, even if the excavator 42 is stopped, the environment map generation device 1 automatically excludes the excavator 42 from the generated environment map 15D. Accordingly, the worker can accurately recognize the amount of soil 43 carried out from the embankment 41 by referring to the environmental map 15D.

Next, an environment map generation method according to Embodiment 1 will be described.
FIG. 7 is a flow chart showing an environment map generation method according to the first embodiment.
The data acquisition unit 11 acquires three-dimensional point cloud data obtained by measuring the target area by the LiDAR 2 and video data obtained by imaging the target area by the camera 3 (step ST1). For example, the worker moves around the target area, and at this time, the LiDAR 2 mounted on the terminal carried by the worker measures the target area at a predetermined measurement cycle, and the camera 3 has a predetermined frame rate. to capture an image of the target area. The data acquisition unit 11 sequentially acquires three-dimensional point cloud data measured by the LiDAR 2 and video data captured by the camera 3 .

The color information addition unit 12 specifies the correspondence relationship between the three-dimensional points forming the three-dimensional point cloud data and the pixels in the frame images forming the video data, and adds color information to the three-dimensional point cloud data. is generated (step ST2). Note that step ST2 is not performed if the environment map generation device 1 does not include the color information addition unit 12 .

The feature point extraction unit 13 extracts feature point data of an object existing within the target area from the three-dimensional point cloud data (step ST3). The position and orientation estimation unit 14 uses the feature point data extracted by the feature point extraction unit 13 to estimate the position and orientation of the LiDAR 2 (step ST4).

The object detection unit 16 detects a movable object among the objects appearing in the image data (step ST5). Here, if no movable object is detected (step ST5; NO), the environment map generation unit 15 generates environment map data of the target area based on the three-dimensional point cloud data and the estimation result of the position and orientation of the LiDAR2. (step ST8).

When a movable object is detected (step ST5; YES), the object detection unit 16 determines whether or not the movable object is moving (step ST6). Here, if the object detection unit 16 determines that the movable object is moving (step ST6; YES), the data removal unit 17 selects the moving object from the three-dimensional point cloud data used to generate the environment map data. The three-dimensional point group data of the movable object determined to be on the screen are removed (step ST7). Further, when the object detection unit 16 detects an object to be removed from the image data, the data removal unit 17 extracts three-dimensional point cloud data corresponding to the object to be removed from the three-dimensional point cloud data used to generate the environment map data. Remove dimensional point cloud data. As a result, the environment map generating unit 15 uses the remaining 3D point cloud data from which the 3D point cloud data corresponding to the movable object or the object to be removed is removed to generate the environment map data of the target area ( step ST8).

On the other hand, when the object detection unit 16 determines that the movable object is not moving (step ST6; NO), the environment map generation unit 15 generates three-dimensional point cloud data including data corresponding to the movable object, the position of the LiDAR 2 and Environment map data of the target area is generated based on the posture estimation result (step ST9). The data removal unit 17 removes the three-dimensional point cloud data corresponding to the movable object from the environment map data generated by the environment map generation unit 15 (step ST10).

Next, a hardware configuration that implements the functions of the environment map generation device 1 will be described.
The functions of the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, the object detection unit 16, and the data removal unit 17 included in the environment map generation device 1 are as follows. It is implemented by a processing circuit. That is, the environment map generation device 1 includes a processing circuit for executing each process from step ST1 to step ST10 shown in FIG. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in memory.

FIG. 8A is a block diagram showing a hardware configuration that implements the functions of the environment map generation device 1. As shown in FIG. FIG. 8B is a block diagram showing a hardware configuration for executing software realizing the functions of the environment map generating device 1. As shown in FIG. 8A and 8B, an input interface 100 is an interface that relays data output from the LiDAR 2 and the camera 3 to the environment map generation device 1. FIG. The output interface 101 is an interface that relays the environment map data output from the environment map generation device 1 to a subsequent device.

If the processing circuit is the dedicated hardware processing circuit 102 shown in FIG. 8A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an Application Specific Integrated Integrated Circuit (ASIC), Circuit), FPGA (Field-Programmable Gate Array), or a combination thereof.
The functions of the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, the object detection unit 16, and the data removal unit 17 included in the environment map generation device 1 are Separate processing circuits may be implemented, or these functions may be combined into one processing circuit.

When the processing circuit is the processor 103 shown in FIG. 8B, the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, which are provided in the environment map generation device 1, The functions of the object detection unit 16 and the data removal unit 17 are realized by software, firmware, or a combination of software and firmware. Note that software or firmware is written as a program and stored in the memory 104 .

The processor 103 reads and executes a program stored in the memory 104 to obtain the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position and orientation estimation unit 14, and the The functions of the environment map generation unit 15, the object detection unit 16, and the data removal unit 17 are realized. For example, the environment map generation device 1 includes a memory 104 for storing a program that, when executed by the processor 103, results in the processing of steps ST1 to ST10 shown in FIG. These programs are procedures or methods of processing performed by the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, the object detection unit 16, and the data removal unit 17. to run on the computer. The memory 104 causes the computer to function as the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, the object detection unit 16, and the data removal unit 17. It may be a computer-readable storage medium storing a program.

The memory 104 includes, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically-EPROM), magnetic Discs, flexible discs, optical discs, compact discs, mini discs, DVDs, and the like are applicable.

One of the functions of the data acquisition unit 11, the color information addition unit 12, the feature point extraction unit 13, the position/orientation estimation unit 14, the environment map generation unit 15, the object detection unit 16, and the data removal unit 17 included in the environment map generation device 1. A portion may be implemented in dedicated hardware, and the remaining portion may be implemented in software or firmware. For example, the data acquisition unit 11 has its functions realized by the processing circuit 102, which is dedicated hardware, and includes a color information addition unit 12, a feature point extraction unit 13, a position/orientation estimation unit 14, an environment map generation unit 15, an object detection unit, and a The functions of the unit 16 and the data removal unit 17 are realized by the processor 103 reading out and executing a program stored in the memory 104 . As such, the processing circuitry may implement the above functions in hardware, software, firmware, or a combination thereof.

FIG. 2 shows a case in which a learning device provided separately from the environment map generation device 1 learns and obtains a learned model for inferring the existence of a movable object, but the environment map generation device 1 It is not limited to this. For example, the environment map generation device 1 may comprise a learning device that learns a trained model that infers the existence of movable objects.

As described above, the environment map generation apparatus 1 according to the first embodiment includes the data acquisition unit 11 that acquires the 3D point cloud data and the video data, and the 3D point cloud data to identify the objects existing in the target area. Based on a feature point extraction unit 13 that extracts feature point data, a position and orientation estimation unit 14 that estimates the position and orientation of the LiDAR 2, three-dimensional point cloud data, and the results of estimating the position and orientation of the LiDAR 2, the target region and an object detection unit that detects a movable object that can move from among the objects shown in the image data and determines whether the detected movable object is moving. 16 and the 3D points of the movable object determined to be moving without removing the 3D point cloud data of the movable object determined not to be moving from the 3D point cloud data used to generate the environment map. A data removing unit 17 for removing group data is provided.
The environment map generation device 1 detects movable objects that can move from among the objects shown in the image data, determines whether the detected movable objects are moving, and uses them to generate an environment map 3 . Three-dimensional point cloud data of a movable object determined not to move is not removed from dimensional point cloud data, and three-dimensional point cloud data of a movable object determined to move is removed. As a result, it is possible to prevent excessive reduction of the three-dimensional point cloud data used to generate the environment map.

In the environment map generation device 1 according to Embodiment 1, the data removal unit 17 removes movable objects determined not to move from the generated environment map. As a result, the environment map generation device 1 can accurately remove the three-dimensional point cloud data corresponding to the movable object determined not to move from the three-dimensional point cloud data constituting the environment map data. .

In the environment map generation device 1 according to Embodiment 1, the correspondence relationship between the 3D points that make up the 3D point cloud data and the pixels in the frame images that make up the video data is specified, and color information is added to the 3D points. A color information addition unit 12 is provided for generating the three-dimensional point cloud data. As a result, the environment map generation device 1 can generate environment map data in which color information is added to three-dimensional points.

In the environment map generation device 1 according to Embodiment 1, when video data is input, the object detection unit 16 creates a learned model for inferring whether or not a movable object is shown in the frame images that make up the video data. and an object inference unit 161 that detects a movable object using an object inference unit 161 that identifies an image area of a movable object appearing in a frame image that constitutes video data, and determines whether or not the movable object is moving based on the identified image area of the movable object. An object region identification unit 162 is provided for determining As a result, the environment map generation device 1 can improve the detection accuracy of movable objects.

In the environment map generation device 1 according to Embodiment 1, the object detection unit 16 detects objects to be removed from objects appearing in video data. The data removal unit 17 removes the 3D point cloud data of the object to be removed from the 3D point cloud data used to generate the environment map data. Since the object to be removed is excluded from the environment map of the target area, the environment map generation device 1 can improve the accuracy of the environment map.

The environment map generation method according to the first embodiment includes steps in which the data acquisition unit 11 acquires three-dimensional point cloud data and image data, and a feature point extraction unit 13 extracts feature point data from the three-dimensional point cloud data. a step of estimating the position and orientation of the LiDAR 2 by the position and orientation estimation unit 14; and a step of the environment map generation unit 15 extracting the target a step of generating environment map data of the area; and an object detection unit 16 detecting a movable object that can move from among the objects appearing in the image data, and determining whether the detected movable object is moving. and the data removal unit 17 does not remove the 3D point cloud data of the movable object determined not to be moving from the 3D point cloud data used to generate the environment map data, and is moving. removing the 3D point cloud data of the movable object determined as As a result, it is possible to prevent excessive reduction of the three-dimensional point cloud data used to generate the environment map.

A program according to Embodiment 1 causes a computer to function as the environment map generation device 1 . As a result, it is possible to provide the environmental map generating device 1 that can prevent excessive reduction of the three-dimensional point cloud data used for generating the environmental map.

It should be noted that any component of the embodiment can be modified or any component of the embodiment can be omitted.

1 environment map generation device 2 LiDAR 3 camera 3A, 3A1, 3A 2 frame image 11 data acquisition unit 12 color information addition unit 13 feature point extraction unit 14 position and orientation estimation unit 15 environment map generation unit 15A 15D environment map, 16 object detection unit, 17 data removal unit, 21 building, 22 pedestrian, 31 image area, 41 embankment, 42 excavator, 43 soil volume, 100 input interface, 101 output interface, 102 processing circuit, 103 Processor, 104 memory, 161 object inference unit, 162 object region identification unit.

An environment map generation device according to the present disclosure is an environment map generation device that generates environment map data indicating a target area of a work site, and is three-dimensional point cloud data that is a set of three-dimensional points measured by a sensor in the target area. and a data acquisition unit for acquiring video data obtained by imaging a target region by an imaging device, a feature point extraction unit for extracting feature point data of an object existing in the target region from the 3D point cloud data, and a 3D point cloud a position/orientation estimation unit that estimates the position and orientation of the sensor using data; an environment map generation unit that generates environment map data of the target area based on the feature point data and the result of estimating the position and orientation of the sensor; Objects to be removed that block the three-dimensional measurement of the work area by the sensor and the imaging of the work area by the imaging device are detected from the objects shown in the video data, and the objects related to the work at the work site shown in the video data An object detection unit that detects a movable object that is an object that can move and determines whether or not the detected movable object is moving; A data removal unit that does not remove the 3D point cloud data of the movable object determined not to be moving, and removes the 3D point cloud data of the movable object determined to be moving and the 3D point cloud data of the object to be removed and the environment map generation unit uses the 3D point cloud data of the objects present in the target area including the movable object determined not to move, excluding the object to be removed, to map the target area of the work site. Generate environmental map data to show

Claims (7)

a data acquisition unit that acquires three-dimensional point cloud data, which is a set of three-dimensional points obtained by measuring a target area with a sensor, and video data obtained by imaging the target area with an imaging device;
a feature point extraction unit that extracts feature point data of an object existing in the target area from the three-dimensional point cloud data;
a position and orientation estimation unit that estimates the position and orientation of the sensor using the feature point data;
an environment map generation unit that generates environment map data of the target area based on the three-dimensional point cloud data and estimation results of the position and orientation of the sensor;
an object detection unit that detects a movable object, which is an object that can move, from objects appearing in the image data and determines whether the detected movable object is moving;
The movable object determined to be moving is not removed from the three-dimensional point cloud data used to generate the environment map data, and the three-dimensional point cloud data of the movable object determined not to be moving is not removed. and a data removing unit that removes the three-dimensional point cloud data of the object. 2. The environment map generating apparatus according to claim 1, wherein the data removing unit removes the movable object determined not to move from the generated environment map data. A color for generating the 3D point cloud data by specifying a correspondence relationship between 3D points constituting the 3D point cloud data and pixels in a frame image constituting the video data, and adding color information to the 3D points. 3. The environment map generation device according to claim 1, further comprising an information addition unit. When the image data is input, the object detection unit detects the movable object using a trained model for inferring whether or not the movable object is shown in the frame images forming the image data. an inference unit; specifying an image area of the movable object appearing in a frame image forming the video data, and determining whether or not the movable object is moving based on the specified image area of the movable object. 4. The environment map generation device according to any one of claims 1 to 3, further comprising a unit. The object detection unit further detects an object to be removed from objects appearing in the image data,
The data removal unit removes the three-dimensional point cloud data of the removal target object from the three-dimensional point cloud data used to generate the environment map data. 5. The environment map generation device according to any one of 4. An environment map generation method for an environment map generation device including a data acquisition unit, a feature point extraction unit, a position and orientation estimation unit, an object detection unit, an environment map generation unit, and a data removal unit,
a step in which the data acquisition unit acquires three-dimensional point cloud data, which is a set of three-dimensional points obtained by measuring a target area by a sensor, and video data obtained by capturing an image of the target area by an imaging device;
a step in which the feature point extraction unit extracts feature point data of an object existing in the target region from the three-dimensional point cloud data;
the position and orientation estimation unit estimating the position and orientation of the sensor using the feature point data;
a step in which the environment map generation unit generates environment map data of the target area based on the three-dimensional point cloud data and estimation results of the position and orientation of the sensor;
a step in which the object detection unit detects a movable object, which is an object that can move, from objects appearing in the video data, and determines whether or not the detected movable object is moving;
The data removal unit does not remove the three-dimensional point cloud data of the movable object determined not to be moving from the three-dimensional point cloud data used to generate the environment map data, and is moving. and removing the three-dimensional point cloud data of the movable object determined to be the environment map generation method. A program for causing a computer to function as the environmental map generation device according to any one of claims 1 to 5.

Images