JP2023138605A - Information processing device - Google Patents

Abstract

To provide an information processing device capable of improving the reliability of point group data after integration.SOLUTION: An information acquisition part 2 acquires travel condition information about a travel condition of a movable body 10 when being imaged. The degree of contribution of respective individual point group data can be determined on the basis of the travel condition information or reliability information and the reliability of total point group data can be improved in the case of collecting and integrating the individual point group data by adding the travel condition information or the reliability information based on the travel condition information to the point group data.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device.

BACKGROUND ART Conventionally, a method (SLAM; Simultaneous Localization and Mapping) is known in which a mobile object simultaneously performs self-position estimation and environmental mapping. As a type of SLAM, a method (VSLAM; Visual SLAM) has been proposed that uses a video camera instead of a laser device (for example, see Patent Document 1). VSLAM as described in Patent Document 1 aims to reduce costs by using a video camera.

JP2014-222550A

In VSLAM as described in Patent Document 1, feature points are extracted from each image (frame) captured by a video camera, and self-position estimation and environment map creation are performed based on changes in feature points between each image. conduct. Point cloud data included in such an environmental map may be collected and integrated by a device such as an external server, and the integrated point cloud data may be used for self-position estimation or the like.

However, the generated point cloud data is not always highly accurate. Therefore, if a plurality of point cloud data to be integrated include point cloud data with low reliability, there is a possibility that the reliability of the integrated point cloud data will decrease. If the reliability of the point cloud data after integration is low, it will be difficult to utilize it for self-position estimation, etc.

Therefore, an example of an object of the present invention is to provide an information processing device that can improve the reliability of point cloud data after integration.

In order to solve the above-mentioned problems and achieve the object, an information processing device of the present invention according to claim 1 includes an imaging information acquisition section that acquires imaging information from an imaging section mounted on a moving object, and a a driving condition acquisition unit that acquires driving condition information regarding the driving condition of the moving body; a generating unit that generates point cloud data based on the imaging information; and an information adding unit that adds the driving condition information to the point cloud data. It is characterized by having the following.

The information processing device of the present invention according to claim 2 includes an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving object, and an imaging information acquisition unit that acquires driving condition information regarding the driving condition of the moving object at the time of imaging. A driving condition acquisition unit, a generation unit that generates point cloud data based on the imaging information, and an information adding unit that adds reliability information based on the driving condition information to the point cloud data. It is a feature.

The information processing method of the present invention according to claim 5 includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, and acquiring running condition information regarding running conditions of the moving body at the time of imaging. The method is characterized by including a driving condition acquisition step, a generation step of generating point cloud data based on the imaging information, and an information adding step of adding the driving condition information to the point cloud data.

The information processing method of the present invention according to claim 6 includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving body, and acquiring running condition information regarding running conditions of the moving body at the time of imaging. A driving condition acquisition step, a generation step of generating point cloud data based on the imaging information, and an information adding step of adding reliability information based on the driving condition information to the point cloud data. It is a feature.

1 is a block diagram schematically showing an information processing apparatus according to an embodiment of the present invention. FIG. 3 is a plan view showing an example of a frame of imaging information acquired by the information processing device. FIG. 7 is a plan view showing another example of a frame of imaging information acquired by the information processing device.

Embodiments of the present invention will be described below. An information processing device according to an embodiment of the present invention includes an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving object, and a driving condition acquisition unit that acquires driving condition information regarding the driving condition of the moving object at the time of imaging. A generation unit that generates point cloud data based on imaging information; and an information provision unit that adds driving condition information to the point cloud data.

According to the information processing device of this embodiment, by adding travel condition information to point cloud data, when collecting and integrating point cloud data, each point cloud data is contribution can be determined. If the driving conditions are such that highly reliable point cloud data can be obtained, increase the contribution of this point cloud data, and if the driving conditions are such that only point cloud data with low reliability can be obtained, increase the contribution of this point cloud data. The contribution of group data can be lowered. Therefore, when integrating a plurality of point cloud data, the reliability of the integrated point cloud data can be improved by determining the degree of contribution of each point cloud data based on the driving condition information.

Note that the imaging information acquisition section, the driving condition acquisition section, the generation section, and the information provision section do not need to be installed on the same object, and may be installed on mutually different devices or devices. That is, each configuration of the information processing device may span multiple physically separated devices.

An information processing device according to another embodiment of the present invention includes an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on a moving object, and a traveling condition information acquisition unit that acquires driving condition information regarding the driving condition of the moving object at the time of imaging. The vehicle includes a condition acquisition unit, a generation unit that generates point cloud data based on imaging information, and an information provision unit that adds reliability information based on driving condition information to the point cloud data.

According to the information processing device of this embodiment, by adding reliability information based on driving condition information to point cloud data, when collecting and integrating point cloud data, the reliability information is added based on the reliability information. The degree of contribution of each point cloud data can be determined, and the reliability of the integrated point cloud data can be improved.

It is preferable that the driving condition acquisition unit acquires, as the driving condition information, at least one of self-information regarding the state or behavior of the mobile object and external information regarding the environment around the mobile object. The self-information includes, for example, driving stability. The lower the speed of the moving body, the lower the acceleration (acceleration/deceleration), and the less wobbling, the higher the running stability. The higher the running stability, the easier it is to generate highly reliable point cloud data.

The external information includes, for example, the imaging difficulty level and the running obstruction level. When the surroundings of a moving object is dark or in rainy weather, it becomes difficult to take an image, or clear image information cannot be obtained, which increases the difficulty of taking an image. During traffic jams, when the distance to the moving object in front is short, or when the moving object in front is large, features around the moving object may be hidden behind other moving objects, making imaging difficult. . Further, when the surroundings of the moving body are dark or on rainy days, the stability of running may decrease, and the degree of hindrance to running increases. When the imaging difficulty level is high or when the degree of travel obstruction is high, the reliability of the generated point cloud data tends to decrease.

Note that the driving condition acquisition unit may directly acquire the driving condition information, or may calculate or estimate by acquiring other parameters. For example, the brightness around the moving body may be detected by a sensor, or may be estimated based on time information and date information. Additionally, information regarding whether or not there are street lights on the road the vehicle is traveling on may also be considered. Further, information regarding the weather may be obtained by detecting rain using a sensor, or may be obtained from outside the mobile object via a network or the like.

It is preferable that the information processing apparatus of any of the embodiments further includes an output unit that outputs point cloud data. This allows the point cloud data to be utilized at the output destination.

The information processing method according to the embodiment of the present invention includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a moving object, and a driving condition acquisition step of acquiring driving condition information regarding the driving condition of the moving object at the time of imaging. a generation step of generating point cloud data based on imaging information; and an information adding step of adding driving condition information to the point cloud data. According to such an information processing method, the reliability of the integrated point cloud data can be improved by adding driving condition information to the point cloud data.

An information processing method according to another embodiment of the present invention includes an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on a mobile object, and a traveling condition information acquisition step of acquiring driving condition information regarding the driving condition of the mobile object at the time of imaging. The method includes a condition acquisition step, a generation step of generating point cloud data based on imaging information, and an information adding step of adding reliability information based on driving condition information to the point cloud data. According to such an information processing method, by adding reliability information based on driving condition information to the point cloud data, it is possible to improve the reliability of the integrated point cloud data.

Further, the information processing method described above may be implemented as an information processing program that causes a computer to execute the information processing method. By doing so, the reliability of the integrated point cloud data can be improved using a computer.

Further, the information processing program described above may be stored in a computer-readable recording medium. By doing so, the program can be distributed as a standalone program in addition to being incorporated into a device, and version upgrades can be easily performed.

Examples of the present invention will be specifically described below. As shown in FIG. 1, the information processing unit 1 of this embodiment includes a first information processing device 1A, which is a VSLAM device mounted on a mobile object 10, and a second information processing device mounted on an external server 100. 1B. The first information processing device 1A includes an information acquisition section 2, a generation section 3, and an output section 4. The moving body 10 is, for example, a vehicle, and is equipped with an imaging section 20, a map information storage section 30, a self-position estimation section 40, a travel information sensor 50, and a date and time information management section 60.

The second information processing device 1B includes a collection section 11 and a generation section 12. The external server 100 is equipped with a communication section 101 and a storage section main body 102. Note that the external server 100 is configured to communicate with a plurality of first information processing devices 1A.

The imaging unit 20 is, for example, a video camera, and is arranged to take an image of the front side of the moving body 10 in the direction of movement. The moving image captured by the imaging unit 20 is composed of a plurality of imaging information (frames), and is transmitted to the information acquisition unit 2. Note that the imaging unit may be capable of continuously capturing still images at predetermined time intervals.

The map information storage unit 30 is composed of, for example, a hard disk or a nonvolatile memory, and stores existing map information. The map information stored in the map information storage unit 30 only needs to include at least information about terrestrial features. Information about features includes not only numerical values about the location and size of features, but also information about the attributes and types of features (distinction between residential and commercial facilities, industry type of commercial facilities, etc.). shall be included.

The self-position estimating unit 40 estimates the current position (absolute position) of the mobile object 10, and may be any GPS receiving unit that receives radio waves transmitted from a plurality of GPS (Global Positioning System) satellites, for example. . The current position estimated by the self-position estimation unit 40 is transmitted to the information acquisition unit 2 as position information indicating the imaging position of the imaging information by the imaging unit 20.

The travel information sensor 50 is a sensor for measuring the amount of displacement of the moving body 10, and includes, for example, a vehicle speed pulse acquisition section for acquiring a vehicle speed pulse of the moving body 10, and a sensor for measuring the amount of azimuth displacement of the moving body 10. A gyro sensor and an acceleration sensor for acquiring the acceleration of the moving body 10. The traveling direction of the mobile object 10 is estimated based on the amount of displacement acquired by the traveling information sensor 50. Imaging orientation information regarding the orientation of the imaging section 20 during imaging is determined by the traveling direction of the moving object 10 and the mounting direction of the imaging section 20 on the moving object 10. When the imaging unit 20 is directed toward the front of the moving body, the traveling direction and the imaging direction substantially match. Therefore, the imaging direction information is transmitted from the driving information sensor 50 to the information acquisition section 2. Note that the traveling information sensor 50 may be any sensor that can at least estimate the traveling direction of the moving body 10.

The date and time information management section 60 has a calendar function and a clock function. Note that the clock function is preferably a radio-controlled clock function that receives standard radio waves and automatically corrects errors. Date and time information is transmitted from the date and time information management section 60 to the information acquisition section 2. Note that based on the date information and time information, solar altitude information and solar azimuth information can be specified. Therefore, the information acquisition unit 2 that acquires date information and time information can be considered to acquire solar altitude information and solar azimuth information.

The information acquisition unit 2 functions as an imaging information acquisition unit by acquiring imaging information from the imaging unit 20 as described above, and functions as a position information acquisition unit by acquiring position information from the self-position estimation unit 40 as described above. It functions as a map information acquisition unit by acquiring map information from the map information storage unit 30, and functions as an imaging azimuth information acquisition unit by acquiring imaging azimuth information from the driving information sensor 50 as described above. However, by acquiring solar altitude information and solar azimuth information from the date and time information management section 60 as described above, it functions as a solar information acquisition section. The information acquisition unit 2 may acquire other information from a data center or the like, for example, via a network such as the Internet or a public line.

The generation unit 3 extracts feature points from the imaging information acquired by the information acquisition unit 2 to generate a two-dimensional feature point group, and further generates point cloud data based on the plurality of feature point groups. Note that the point cloud data generated by the generation unit 3 may be, for example, three-dimensional point cloud data. Note that, for example, an ORB (Oriented fast and Rotated Brief) algorithm may be used to extract feature points, or other algorithms may be used.

The output unit 4 is composed of a circuit, an antenna, etc. for communicating with a network such as the Internet or a public line, and outputs information (point cloud data) generated by the generation unit 3 by communicating with the communication unit 101 of the external server 100. Send. As a result, information is stored in the storage main body 102 of the external server 100.

The collection unit 11 of the second information processing device 1B acquires point cloud data (individual point cloud data) generated by the first information processing device 1A via the communication unit 101. The collection unit 11 collects individual point cloud data by acquiring individual point cloud data from a plurality of moving objects or by acquiring individual point cloud data from one moving object multiple times.

The generation unit 12 generates comprehensive point cloud data based on the plurality of individual point cloud data collected by the collection unit 11. The comprehensive point cloud data may be obtained by superimposing a plurality of individual point cloud data, or may indicate a positional average of constituent points of a plurality of individual point cloud data.

The comprehensive point cloud data generated in this way is used, for example, to estimate the self-position of a moving object. That is, when a terrestrial feature is detected by an optical sensor (so-called LIDAR; Laser Imaging Detection and Ranging) mounted on a moving object, the self-position of this moving object can be estimated by comparing it with comprehensive point cloud data.

[Obtain driving condition information]
The information acquisition unit 2 acquires driving condition information regarding the driving conditions of the moving object 10 at the time of imaging, and functions as a driving condition acquisition unit. Note that the timing at which the information acquisition unit 2 acquires the driving condition information is arbitrary; for example, it may acquire the imaging information and the driving condition information at the same time, or after acquiring the imaging information and generating the point cloud data. Driving condition information may also be acquired.

The running condition information includes self-information regarding the state or behavior of the mobile body 10 and external information regarding the environment around the mobile body 10. Note that the driving condition information may include only one of the self information and external information.

The self-information includes, for example, driving stability. The lower the speed of the moving body 10, the lower the acceleration (less acceleration/deceleration), and the less wobbling, the higher the running stability. The higher the running stability, the easier it is to generate highly reliable point cloud data. The running stability may be calculated based on the speed, acceleration, and wobbling at the time of imaging, or may be calculated based on the past driving history. That is, if the stability was high during past driving, it can be estimated that the driving stability was also high during image capture. Further, the driving stability may be calculated based on the past driving history of the mobile object, or the driving stability may be calculated based on the past driving history of the driver.

Further, the self-information may include information regarding the performance and maintenance status of the mobile body 10. For example, when the acceleration performance and braking performance of the moving body 10 are high, large acceleration and deceleration may occur. Also, depending on the power steering settings, the likelihood of wobbling will change. That is, depending on the performance and maintenance state of the moving object 10, the ease with which the moving object responds to the driver's operations changes, which may affect the driving stability.

The external information includes, for example, the imaging difficulty level and the running obstruction level. When the surroundings of the moving body 10 are dark or in rainy weather, it becomes difficult for the imaging unit 20 to take an image, or clear imaging information cannot be obtained, which increases the difficulty of imaging. During traffic jams, when the distance to the moving object in front is short, or when the moving object in front is large, features around the moving object are hidden by other moving objects, making imaging difficult.

For example, when a large vehicle C such as a truck is traveling in front as shown in FIG. It will be hidden in the shadows, making it impossible to extract the feature points of building B.

Further, when the surroundings of the moving body 10 are dark or on rainy days, the stability of running may decrease, and the degree of inhibition of running becomes high. When the imaging difficulty level is high or when the degree of travel obstruction is high, the reliability of the generated point cloud data tends to decrease.

Note that the driving condition information may be obtained directly, or may be calculated or estimated by obtaining other parameters. For example, the brightness around the mobile object 10 may be detected by a sensor, or may be estimated by acquiring time information and date information from the date and time information management section 60. Furthermore, by acquiring map information from the map information storage unit 30, the brightness may be estimated by taking into account whether or not there are street lights on the road the vehicle is traveling on. Further, information regarding the weather may be obtained by detecting rain using a sensor, or may be obtained by the information obtaining unit 2 from outside the mobile body 10 via a network or the like.

[Assigning information to point cloud data]
The generation unit 3 adds the driving condition information acquired as described above to the generated point cloud data, and functions as an information adding unit. That is, the output unit 4 outputs point cloud data to which driving condition information is added.

Note that the generation unit 3 may function as an information adding unit by adding reliability information to the point cloud data instead of driving condition information. That is, the reliability may be determined based on the driving condition information, and information regarding this reliability may be added to the point cloud data.

[Integration of individual point cloud data with information attached]
As described above, in the second information processing device 1B, the collection unit 11 collects individual point cloud data, and the generation unit 12 generates comprehensive point cloud data based on a plurality of individual point cloud data. At this time, if driving condition information is attached to the individual point cloud data, the generation unit 12 determines the degree of contribution of the individual point cloud data based on the driving condition information, and generates comprehensive point cloud data.

At this time, since the reliability of the individual point cloud data changes depending on the driving conditions as described above, it is only necessary to increase the degree of contribution of the individual point cloud data with high reliability based on the driving condition information. Note that the degree of contribution includes 0, and if the degree of contribution is set to 0, that individual point group data can be removed. Moreover, when reliability information is provided to individual point cloud data instead of driving condition information, the generation unit 12 may determine the degree of contribution of the point cloud data based on the reliability information.

Further, the generation unit 12 may generate comprehensive point cloud data based on individual point cloud data for each driving condition. For example, the individual point cloud data may be classified according to the weather to generate comprehensive point cloud data for clear weather, comprehensive point cloud data for cloudy weather, and comprehensive point cloud data for rainy weather, respectively. Further, comprehensive point cloud data may be generated for each driver or vehicle type, or may be generated for each traveling speed.

As described above, if comprehensive point cloud data is generated for each driving condition, the reliability of the comprehensive point cloud data can be improved. In other words, when utilizing comprehensive point cloud data, by selecting comprehensive point cloud data according to the conditions at that time, it is possible to reduce the discrepancy between the conditions at the time of point cloud data generation and the conditions at the time of utilization. . In the case of the above example, if it is raining when the comprehensive point cloud data is used, the comprehensive point cloud data generated based on the individual point cloud data for rainy weather may be used.

[Acquisition of driving condition information in second information processing device]
In the above example, the first information processing device 1A acquires the driving condition information and adds the information to the individual point cloud data, but the collection unit 11 of the second information processing device 1B Individual point cloud data and driving condition information that have not been updated may be acquired. That is, the collection unit 11 may function as a driving condition acquisition unit. At this time, the driving condition information acquired by the collection unit 11 may include, for example, at least one of self information and external information, as in the above example.

The generation unit 12 determines the degree of contribution of the individual point cloud data based on the driving condition information, and generates comprehensive point cloud data. At this time, the degree of contribution of individual point cloud data with high reliability may be increased based on the driving condition information. Further, the generation unit 12 may generate comprehensive point cloud data based on individual point cloud data for each driving condition, as described above.

With the above configuration, by adding driving condition information or reliability information to point cloud data, when collecting and integrating individual point cloud data, each individual point cloud is The degree of contribution of data can be determined, and the reliability of comprehensive point cloud data can be improved.

Further, by outputting the point cloud data to the external server 100 by the output unit 4, the point cloud data can be utilized in the external server 100 as the output destination.

Furthermore, even when the collection unit 11 of the second information processing device 1B acquires individual point cloud data to which no information is attached and driving condition information, the individual point cloud data are collected and integrated. At this time, the degree of contribution of each individual point cloud data can be determined based on the driving condition information, and the reliability of the comprehensive point cloud data can be improved.

Note that the present invention is not limited to the above-mentioned embodiments, but includes other configurations that can achieve the object of the present invention, and the present invention also includes the following modifications.

For example, in the above embodiment, the information acquisition section 2 and the generation section 3 are mounted on the moving object 10, but the imaging information acquisition section and the driving condition acquisition section that constitute the first information processing device are The generation unit and the information provision unit do not need to be mounted on a moving body, and may be mounted on mutually different devices or devices. That is, each configuration of the information processing device may span multiple physically separated devices.

Furthermore, in the above embodiment, the collection unit 11 and the generation unit 12 are installed in the external server 100, but the collection unit, the generation unit, and the driving condition acquisition unit that constitute the second information processing device are different from each other. , may not be installed in the external server, and may be installed in mutually different devices or devices. That is, each configuration of the information processing device may span multiple physically separated devices.

In addition, the best configuration, method, etc. for carrying out the present invention have been disclosed in the above description, but the present invention is not limited thereto. That is, although the present invention has been specifically illustrated and described primarily with respect to specific embodiments, modifications may be made to the embodiments described above without departing from the spirit and scope of the invention. , materials, quantities, and other detailed configurations, those skilled in the art can make various modifications. Therefore, the descriptions that limit the shapes, materials, etc. disclosed above are provided as examples to facilitate understanding of the present invention, and do not limit the present invention. Descriptions of names of members that exclude some or all of the limitations such as these are included in the present invention.

1A First information processing device 2 Information acquisition unit (imaging information acquisition unit, driving condition acquisition unit)
3 Generation section (information provision section)
4 Output unit 10 Mobile object 20 Imaging unit 1B Second information processing device 11 Collection unit (driving condition acquisition unit)
12 Generation section

Claims (8)

An information processing device that generates point cloud data that is integrated into comprehensive point cloud data along with other point cloud data,
an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on the mobile object;
a driving condition acquisition unit that acquires driving condition information regarding the driving condition of the mobile object at the time of imaging;
a generation unit that generates the point cloud data based on the imaging information;
An information processing device comprising: an information adding section that adds the driving condition information to the point cloud data. An information processing device that generates point cloud data that is integrated into comprehensive point cloud data along with other point cloud data,
an imaging information acquisition unit that acquires imaging information from an imaging unit mounted on the mobile object;
a driving condition acquisition unit that acquires driving condition information regarding the driving condition of the mobile object at the time of imaging;
a generation unit that generates the point cloud data based on the imaging information;
An information processing device comprising: an information adding unit that adds reliability information based on the driving condition information to the point cloud data. The driving condition acquisition unit acquires, as the driving condition information, at least one of self-information regarding the state or behavior of the mobile object and external information regarding the surrounding environment of the mobile object. The information processing device according to item 1 or 2. The information processing apparatus according to any one of claims 1 to 3, further comprising an output unit that outputs the point cloud data. An information processing method executed by an information processing device for generating point cloud data to be integrated into comprehensive point cloud data along with other point cloud data, the method comprising:
an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on the mobile object;
a driving condition acquisition step of acquiring driving condition information regarding the driving condition of the moving object at the time of imaging;
a generation step of generating the point cloud data based on the imaging information;
An information processing method comprising: an information adding step of adding the driving condition information to the point cloud data. An information processing method executed by an information processing device for generating point cloud data to be integrated into comprehensive point cloud data along with other point cloud data, the method comprising:
an imaging information acquisition step of acquiring imaging information from an imaging unit mounted on the mobile object;
a driving condition acquisition step of acquiring driving condition information regarding the driving condition of the moving object at the time of imaging;
a generation step of generating the point cloud data based on the imaging information;
An information processing method comprising: an information adding step of adding reliability information based on the driving condition information to the point cloud data. An information processing program that causes a computer to execute the information processing method according to claim 5 or 6. A computer-readable recording medium storing the information processing program according to claim 7.

Images