JP7160257B2 - Information processing device, information processing method, and program - Google Patents

Description

Application of Patent Act Article 30, Paragraph 2 May 9, 2017 http://www. nippon-control-system. co. jp/products/green/index_lidar. html, http://www. nippon-control-system. co. jp/catalog/GREEENxLIDAR-brochure. pdf, http://www. nippon-control-system. co. jp/en/products/green/index_lidar. html, http://www. nippon-control-system. co. jp/en/catalog/GREEENxLIDAR-brouchure-en. Announced via pdf, May 10-12, 2017 Exhibited at the 20th Embedded System Development Technology Expo (ESEC) sponsored by Reed Exhibitions Japan, May 23-26, 2017 Released at BB Construmat 2017 hosted by Fira de Barcelona, October 1, 2017 Published in "Visual Information Industrial October 2017 Issue Vol.

Application of Article 30, Paragraph 2 of the Patent Act May 26, 2017, May 30, 2017, June 1, 2017, June 2, 2017, June 5, 2017, June 7, 2017, 2017 June 9, June 12, 2017, June 13, 2017, June 14, 2017, June 15, 2017, June 19, 2017, June 2017 21st, June 23, 2017, June 27, 2017, July 3, 2017, July 4, 2017, July 5, 2017, July 7, 2017 , July 11, 2017, July 12, 2017, July 19, 2017, July 21, 2017, July 31, 2017, August 1, 2017, Heisei August 3, 2017, August 4, 2017, August 8, 2017, August 9, 2017, August 10, 2017, August 15, 2017, 2017 August 23, August 25, 2017, August 28, 2017, August 29, 2017, August 30, 2017, September 1, 2017, September 2017 4th, September 6, 2017, September 7, 2017, September 8, 2017, September 12, 2017, September 13, 2017, September 14, 2017 , September 15, 2017, September 20, 2017, September 21, 2017, September 25, 2017, September 26, 2017, September 27, 2017, Heisei September 28, 2017, September 29, 2017, October 3, 2017, October 4, 2017, October 5, 2017, October 10, 2017, 2017 October 13, October 17, 2017, October 18, 2017, October 19, 2017 Released through individual sales and email

The present invention relates to an information processing apparatus or the like that recognizes an object in the vicinity of a moving body and recognizes the position of the moving body, composes and outputs a diagram in which the positional relationship between the surrounding object and the moving body can be grasped, and outputs the diagram. It is.

Conventionally, there is a technology called LIDAR (see, for example, Non-Patent Document 1). LIDAR is one of the remote sensing technologies using light, which measures the scattered light from pulsed laser irradiation and analyzes the distance to a distant target and the properties of the target.

Also, conventionally, there is a technique called a distance image camera (see, for example, Non-Patent Document 2). A depth image camera is a camera that can obtain three-dimensional depth information in real time.

Wikipedia, "LIDAR", [online], [searched on August 26, 2017], Internet [URL: https://ja.wikipedia.org/wiki/LIDAR] "Capability of TOF range imaging camera", [online], [searched on August 26, 2017], Internet [URL: http://www.brainvision.co.jp/wp/support/tech/capability_of_tofcamera.pdf# search=%27%E8%B7%9D%E9%9B%A2%E7%94%BB%E5%83%8F%E3%82%AB%E3%83%A1%E3%83%A9%27]

However, when using conventional technology, sensor fusion is required for applications that require complex and advanced recognition, such as autonomous driving. Sensor fusion is software that intelligently combines data from multiple sensors to improve the performance of applications and systems. Sensor fusion is a technology that compensates for shortcomings and enables more robust recognition by combining sensors with different properties, but it is difficult to build a system.

In addition, for example, in order to combine recognition results using sensors with different characteristics, such as a 3D sensor and a conventional two-dimensional camera, and use them as useful data, it is possible to interpret those recognition results in a single coordinate system. must be installed, requiring precise installation and calibration. In addition, when different types of recognition (for example, recognition of surrounding objects and self-position recognition) are realized using observation data from different sensors, each sensor has different weaknesses (vulnerability to dark places, etc.). In such a situation, the recognition becomes impossible, and as a result, the function of the system becomes imperfect. In addition, the need to perform independent recognition processes for different types of sensor observations individually requires a lot of computational power, adversely affecting the real-time required for advanced applications.

In other words, in the conventional technology, both the position recognition of the moving body and the recognition of the objects around the moving body are performed from one observation data (original information described later), and the moving body and the objects around the moving body are recognized. Could not output the positional relationship of

In view of this background, the present invention aims to realize recognition of two different pieces of information, ie, "object recognition" around a moving object and "self-location recognition" of the moving object, based on the same data obtained from sensors. aim.

An information processing apparatus according to a first aspect of the present invention comprises an original information acquisition unit that acquires one or more pieces of original information that are used as a basis for specifying a position in a three-dimensional space of an object that exists around a moving mobile object; A moving object position information acquiring unit that acquires moving object position information that specifies a position of a moving object in a three-dimensional space using one or more pieces of original information acquired by the original information acquiring unit; an object recognition unit that identifies one or more objects using the above original information and associates each of the one or more objects with two or more pieces of object position information of each object; an output diagram construction unit that constructs an output diagram that is a three-dimensional diagram in which the relative positional relationship between a moving object and one or more objects can be recognized using two or more pieces of object position information; and outputting the output diagram. and a diagram output unit for processing.

With this configuration, based on the same data obtained from the sensor, the position of the moving body and the objects around the moving body are recognized, and the positional relationship between the moving body and the objects around the moving body is output. can.

Further, in the information processing apparatus of the second invention, in contrast to the first invention, the original information acquisition unit acquires one or more pieces of original information covering 360 degrees around the moving body, and the object recognition unit , using one or more pieces of original information to identify one or more objects existing within a 360-degree range, and associate each of the one or more objects with two or more pieces of object position information of each object.

With this configuration, based on the same data acquired from the sensor, the position of the moving body and the objects in all directions of the moving body are recognized, and the positional relationship between the moving body and objects around the moving body is determined. can be output.

Further, in the information processing apparatus of the third invention, in contrast to the first or second invention, the object recognition unit identifies a determining unit for acquiring motion information about the motion of each of the one or more objects, and determining whether or not the motion information matches a predetermined condition; The information processing apparatus further includes a determination information output unit that outputs determination information.

With this configuration, based on the same data acquired from the sensor, the position of the moving body and the objects around the moving body are recognized, and when the object's action meets the conditions (for example, when it is dangerous) ) can be notified to that effect.

Further, in the information processing apparatus of the fourth invention, in contrast to any one of the first to third inventions, the laser identifier for identifying the laser transmission means, the transmission means position information for specifying the position of the laser transmission means, and the laser Further comprising a storage unit for storing two or more laser transmission means information having transmission means direction information specifying the direction of radar transmission of the transmission means, and the original information acquisition part transmits by each of the two or more laser transmission means. receive the reflected laser, acquire the return time from the laser transmission to the reception of the reflection, and obtain the laser information, which is a set of the return time and the laser identifier that identifies the laser transmission means, for each laser transmission means The object recognition unit uses the transmitting means position information and transmitting means direction information paired with the laser identifiers possessed by each of the two or more pieces of laser information acquired by the original information acquiring unit, and the return time possessed by the laser information. , acquiring object position information of an object for each laser information, specifying one or more objects using the acquired two or more object position information, and determining each of the one or more objects and the two or more object position information of each object. is an information processing device that associates

With this configuration, using laser technology, based on the same data obtained from the sensor, the position of the moving body and the objects around the moving body are recognized, and the moving body and the surroundings of the moving body are recognized. You can output the positional relationship with the object.

Further, in the information processing apparatus of the fifth invention, in any one of the first to fourth inventions, the original information acquiring unit acquires one or more pieces of original information at each of two or more points of time, and The position information acquisition unit acquires position information of the mobile object at each of two or more points in time, and the object recognition unit identifies one or more objects at each of the two or more points in time, and identifies each of the one or more objects and two of each object. The above object position information is associated, and the output diagram construction unit is a diagram that includes one or more objects recognized by the object recognition unit at each of two or more points in time. This is an information processing apparatus that configures an output diagram that is a diagram that includes trajectory information indicating the movement trajectory of the mobile body that is configured from the mobile body position information at each of the above points.

With this configuration, based on the same data acquired from the sensor, the position of the moving body and the objects around the moving body are recognized, and the movement history of the moving body and the objects around the moving body are obtained. You can output the positional relationship with

According to the information processing apparatus of the present invention, the position of the moving body and the objects around the moving body are recognized based on the same data acquired from the sensor, and the moving body and the objects around the moving body are recognized. You can output the positional relationship with

Block diagram of information processing device 1 according to Embodiment 1 Flowchart for explaining an operation example of the information processing device 1 Flowchart for explaining same source information acquisition processing Flowchart explaining the same object recognition processing Flowchart for explaining the same output diagram construction processing Flowchart explaining the judgment process Conceptual diagram of the same information processing device 1 Diagram showing the same output example Diagram showing the same output example Diagram showing the same output example Diagram showing the same output example Diagram showing the same output example Overview of the computer system Block diagram of the same computer system

Hereinafter, embodiments of an information processing apparatus and the like will be described with reference to the drawings. It should be noted that, since components denoted by the same reference numerals in the embodiments perform similar operations, repetitive description may be omitted.

(Embodiment 1)

In the present embodiment, an information processing apparatus that performs both peripheral object recognition and position recognition of a moving body, constructs and outputs a diagram that enables the positional relationship between the peripheral object and the moving body to be grasped, will be described. In the present embodiment, an information processing apparatus for recognizing objects in 360-degree directions with respect to a moving object will be described. Note that the moving body is an automobile, a robot, or the like. However, the type of mobile object does not matter.

Further, in the present embodiment, an information processing apparatus that acquires motion information related to the motion of an object and outputs a match when the motion information matches a predetermined condition will be described.

Furthermore, in the present embodiment, an information processing apparatus capable of outputting a movement history of a moving object will be described.

FIG. 1 is a block diagram of an information processing device 1 according to this embodiment. The information processing device 1 includes a storage unit 11 , a reception unit 12 , a processing unit 13 and an output unit 14 . The information processing device 1 may be installed in a moving body, or may be a device that acquires information from the moving body and performs predetermined processing.

The processing unit 13 includes an original information acquisition unit 131 , a moving body position information acquisition unit 132 , an object recognition unit 133 , an output diagram construction unit 134 and a judgment unit 135 .

The output unit 14 includes a diagram output unit 141 and a judgment information output unit 142 .

Various types of information are stored in the storage unit 11 . Various types of information are, for example, two or more laser transmitting means information. The laser transmission means information is information having a laser identifier for identifying the laser transmission means, transmission means position information for specifying the position of the laser transmission means, and transmission means direction information for specifying the radar transmission direction of the laser transmission means. . Note that the laser transmission means is means for transmitting a laser and receiving its reflected signal. For example, LIDAR comprises a number of laser emitting means.

Further, the various types of information are, for example, original information to be described later. The original information is information obtained by the original information obtaining unit 131 .

Various types of information are, for example, object information. Object information is information about objects (for example, people, cars, obstacles, etc.) to be recognized. The object information includes, for example, an object identifier that identifies the object, and information on the contour of the object (for example, coordinate information of a plurality of points forming the contour). An object identifier may be an object name (eg, person, car, etc.). Object information is, for example, an object identifier and an image of one or more objects. Object information is, for example, an object identifier and two or more feature amounts extracted from one or more object images. Also, the object information may be, for example, learning information obtained by learning an object identifier and two or more feature amounts extracted from one or more object images. Note that learning is, for example, learning using deep learning, support vector machines, decision trees, or the like, and the type of machine learning does not matter. An object is a thing such as a person, a moving body (such as a car), or a device that exists independently in a scene, and is not limited to a "moving object" (moving body). What is defined as an object depends on the application. For example, there may be a system that recognizes a post, a sign, or a building as an object. A scene is a scene observed by an observation device.

Further, the various information is, for example, predetermined conditions. The predetermined condition is a condition used by the determination unit 135, for example, a condition using motion information. The predetermined condition is, for example, that the motion information is "information indicating that the distance between the moving body and the object is within a threshold and that the distance between the moving body and the object is decreasing". Further, the predetermined condition is, for example, that "the distance between the position of the moving body indicated by the positional information of the moving body and the position of the object indicated by the positional information of the moving body" of the motion information is within a threshold value.

Moreover, various information is judgment information, for example. Further, the various types of information are, for example, determination information associated with "predetermined conditions". The determination information is, for example, information for calling attention, such as a buzzer sound, a character string for calling attention, or a pattern.

The reception unit 12 receives various instructions, information, and the like. Various instructions, information, and the like are, for example, an instruction to start operation and an instruction to end operation of the information processing apparatus 1 . Further, the operation start instruction is an instruction to start the operation of the information processing device 1, but may be an instruction to start moving the moving object. The operation end instruction is an instruction to end the operation of the information processing device 1 . Here, reception means reception of information input from input devices such as keyboards, mice, and touch panels, reception of information transmitted via wired or wireless communication lines, and recording on optical discs, magnetic discs, semiconductor memories, etc. This is a concept that includes acceptance of information read from a medium.

Any means such as a touch panel, keyboard, mouse, or menu screen may be used as input means for various instructions and information. The reception unit 12 can be realized by a device driver for input means such as a touch panel or a keyboard, control software for a menu screen, or the like.

The processing unit 13 performs various types of processing. The various types of processing are, for example, processing performed by the original information acquisition unit 131, the moving body position information acquisition unit 132, the object recognition unit 133, the output diagram construction unit 134, and the determination unit 135. FIG.

The original information acquisition unit 131 acquires one or more pieces of original information. The original information acquisition unit 131 preferably acquires one or more pieces of original information covering 360 degrees around the moving object. It is preferable that the original information acquisition unit 131 acquires one or more pieces of original information at two or more points in time. It is preferable that the original information acquisition unit 131 acquires one or more pieces of original information covering 360 degrees around the moving object at one point in time. In addition, when the original information acquisition unit 131 acquires two or more pieces of original information covering 360 degrees, a time delay may occur depending on the region within the 360 degree range.

The original information is information that serves as a basis for specifying the position in the three-dimensional space of an object that exists around the moving object. The original information has a set of three-dimensional data of an observation device (not shown). The original information is a set of points representing the 3D coordinates of a specific part such as a person or thing existing in the vicinity observed from the observation device, and is, for example, range image data or point cloud data. Note that the observation device is a device capable of continuously acquiring 3D data as observation data. The observation device is, for example, the above-described LIDAR, three-dimensional camera, ToF sensor, stereo camera, or the like. The observation device may be a system configured by combining two or more devices as long as it outputs 3D data in the same coordinate system as a result. Also, the original information acquisition unit 131 may be considered to include an observation device, or may be considered to acquire original information from the observation device. Also, the three-dimensional data is, for example, three-dimensional coordinate values with an observation device or a moving object as an origin.

The original information is laser information, for example. The laser information is a set of return time and laser identifier. A laser identifier is information for identifying a laser transmission means. The laser identifier may be information specifying the position of the irradiated laser. For example, the original information acquisition unit 131 receives reflections of lasers transmitted by two or more laser transmission means, acquires the return time from the transmission of the laser to the reception of the reflection, and compares the return time with the laser transmission means. Laser information, which is a pair with a laser identifier for identification, is acquired for each laser transmission means. The original information acquisition unit 131 may be considered to include a LIDAR that emits a laser beam, or may be implemented by software, an MPU, or the like that acquires original information from the LIDAR. Note that LIDAR may also be referred to as LiDAR.

The original information may be, for example, a distance image. The original information acquisition unit 131 acquires, for example, one or more distance images captured by a distance image camera. Note that the distance image includes coordinate values of each pixel in the image in a three-dimensional space. Also, the original information acquisition unit 131 may be considered to include a camera (which may be called a distance image sensor) that captures a distance image, and is implemented by software that acquires the distance image captured by the camera, an MPU, and the like. can be

The mobile body position information acquisition unit 132 acquires the mobile body position information of the moving mobile body using one or more pieces of original information acquired by the original information acquisition unit 131 . Mobile body position information is usually information that specifies a position in a three-dimensional space (hereinafter sometimes simply referred to as "space"). Mobile position information usually has a structure of (x, y, z). Also, the mobile body position information is usually information indicating a relative position. When the mobile body position information is information indicating a relative position, for example, the starting point of the mobile body is the origin (0, 0, 0). If the absolute position (usually (latitude, longitude, altitude)) of the starting point of the mobile body is known, the mobile body position information can be converted into information indicating the absolute position. In other words, position information such as mobile body position information and object position information handled in this specification may be information indicating a relative position or information indicating an absolute position.

The mobile body position information acquisition unit 132 normally acquires mobile body position information at two or more points in time using one or more pieces of original information.

More specifically, the moving body position information acquiring unit 132, for example, extracts feature points from two pieces of original information at different points in time, and extracts two feature points corresponding to the two pieces of original information (for example, (x ₁₁ , y ₁₁ , z ₁₁ ) and (x ₁₂ , y ₁₂ , z ₁₂ )) are detected, and the amount of movement (for example, (x ₁₂ −x ₁₁ , y ₁₂ −y ₁₁ , z ₁₂ −z ₁₁ ).Then, the mobile body position information acquisition unit 132 calculates, for example, mobile body position information (eg, (x ₀ , y ₀ , z ₀ )) indicating the immediately preceding position of the mobile body and the amount of movement (eg, , (x ₁₂ −x ₁₁ , y ₁₂ −y ₁₁ , z ₁₂ −z ₁₁ ) and the current position information of the mobile unit (for example, (x ₀ +x ₁₂ −x ₁₁ , y ₀ +y ₁₂ −y ₁₁ , z ₀ +z ₁₂ −z ₁₁ )), where the feature points are corner points of the object, specific points on the outline of the object, etc. From the original information, which is a set of three-dimensional data, the feature points Techniques for obtaining points are well known.

The object recognition unit 133 uses the one or more pieces of original information acquired by the original information acquisition unit 131 to identify one or more objects (“identification” may also be called “recognition”), and identifies the one or more objects. Each object is associated with two or more pieces of object position information of each object. Note that the object recognition unit 133 does not need to recognize all objects around the moving body, and may recognize only specific objects (for example, people, cars, etc.). Also, the object position information is information indicating the positions of points that constitute an object, and is usually three-dimensional coordinate information. Object position information is usually coordinate information indicating a position relative to mobile object position information.

The object recognition unit 133 normally calculates two or more pieces of object position information using each three-dimensional coordinate information and moving body position information included in the original information. That is, the object recognition unit 133 uses the three-dimensional coordinate information (e.g., (x _a , y a , _za )) of the original information as an offset to the mobile body position information (e.g., (x _b , y _b , _{z b} )) to calculate object position information (eg, ( _xa + _xb , ya+ _yb , _za + _{zb )} ), which is relative three-dimensional coordinate information.

Also, it is preferable that the object recognition unit 133 acquires all coordinate information constituting all objects around the moving object using one or more pieces of original information acquired by the original information acquisition unit 131 . This coordinate information is object position information.

Also, the object recognition unit 133 normally uses one or more pieces of original information acquired by the original information acquisition unit 131 to acquire all object position information of objects around the moving body. Next, the object recognition unit 133 identifies one or more objects from all the object position information, and associates each of the one or more objects with two or more pieces of object position information of each object.

The object recognition unit 133 uses one or more pieces of original information to identify one or more objects existing within a 360-degree range, and associates each of the one or more objects with two or more pieces of object position information of each object. preferred. If a time delay occurs when obtaining one or more pieces of original information covering 360 degrees, the object recognition unit 133 normally obtains object position information in consideration of the time delay. Considering the time delay means, for example, calculating the amount of movement corresponding to the time delay and correcting (correcting) the acquired coordinate values by the amount of movement.

Also, the object recognition unit 133 preferably acquires the attribute value of the specified object. The attribute value is, for example, the type of object, the shape feature amount, and the like. Note that the shape feature amount is, for example, size, flatness, and the like.

The object recognition unit 133 performs, for example, one of the following processes (1) to (3).
(1) First processing example of the object recognition unit 133

A first example of processing is object recognition processing when object recognition and self-position recognition are performed by independent recognition processing.

The object recognition unit 133 performs the following processing using, for example, a set of three-dimensional coordinate values (referred to as “3D points”) possessed by one or more pieces of original information. First, the object recognition unit 133 performs object recognition processing using a set of one or more 3D points. Object recognition processing is processing for identifying a set of 3D points that constitute one object. Next, the object recognition unit 133 acquires one or more shape feature amounts, which are feature amounts related to the shape of each object, for example, from a set of 3D points of each object. The shape feature amount is, for example, size (for example, width, height, depth), flatness, and the like. Next, the object recognition unit 133 determines the type of object (an example of the attribute value of the object) using one or more shape feature values, for example. Machine learning techniques such as decision trees, deep learning, support vector machines, and decision trees can be used for the process of determining the type of object. Next, the object recognition unit 133 acquires a set of 3D points (which may also be called object position information) in association with the determined object type information.

In addition, the object recognition process has the following method specifically, for example. First, the first object recognition processing will be explained. The object recognizing unit 133 determines that a set of points that are so close that the distance between each of the plurality of 3D points satisfies a predetermined condition constitutes the same object, and collects the 3D points for each object.

Further, the object recognition unit 133 may perform object recognition processing by, for example, a second method, which is a grid map method described below. In other words, the object recognition unit 133 partitions the three-dimensional space into a plurality of BOXes. Then, the object recognition unit 133 assumes that all 3D points existing in each BOX are 3D points forming the same object. Next, the object recognition unit 133 acquires feature points (for example, the point closest to the center, the median value of a plurality of points, etc.) from the 3D points of each BOX. Next, BOXes whose feature points are close enough to satisfy a predetermined condition are determined to be BOXes forming the same object. As described above, a set of 3D points forming one object has been detected (object recognition has been completed). A BOX is generally a three-dimensional cube or rectangular parallelepiped.

Then, for example, the object recognition unit 133 stores two or more pieces of object position information of each object in a buffer (not shown) in association with the moving body position information acquired from the same original information. It is preferable that the object recognition unit 133 also store the attribute values of each object together in a buffer (not shown).
(2) Second processing example of the object recognition unit 133

A second processing example is a process of storing the history of objects recognized in the past and acquiring past objects corresponding to the current object based on changes in the self-position. In the second processing example, the amount of position change (velocity) of the object can be used for object recognition. In addition, in the second processing example, previously recognized attributes can also be used.

In the second processing example, the position (in the reference coordinate system) where each past object can currently exist is estimated from the history of the past position (in the reference coordinate system) of each object and the movement vector. , based on which each object in the past is associated with the current object. A Kalman filter or the like can be used for such position estimation processing. In order to implement this process, it is necessary to store the position of the moving body and the position of each object at least at one or more times in the past.

More specifically, first, the moving object position information acquiring unit 132 performs the above-described processing to determine the amount of movement of the feature point (for example, (x _t −x _t−1 , y _t −y _t−1 , z _t − z _t−1 )) Calculate. Then, the mobile body position information acquiring unit 132 acquires the mobile body position information (eg, (x _t-1 , y _t-1 , z _{t- 1} ))) and the amount of movement, the position information of the mobile object at the current time point (t) (for example, (x _t , y _t , z _t )) is acquired.

Next, the object recognition unit 133 recognizes an object by the above-described object recognition processing using a set of 3D points possessed by one or more pieces of original information at the current time. Next, the object recognition unit 133 acquires one or more shape feature amounts, which are feature amounts related to the shape of each object, for example, from a set of 3D points of each object. Next, the object recognition unit 133 acquires the current attribute value of the object (for example, the type of object) from one or more shape feature values, for example.

Next, the object recognition unit 133 corrects the object position information of the object at the present time using the mobile body position information in order to associate the object at one or more past points in time with the object at the present time. Next, the current position of each object in the past is estimated from the history of the amount of movement in the past. It should be noted that such processing includes processing using a Kalman filter or the like. Next, the attribute value of the current object and the attribute value of the past object are compared to establish a correspondence between the current object and the past object. The current object and the past object to be associated have the same or similar attribute values, and the difference between the coordinate value corresponding to the current position and the coordinate value corresponding to the predicted position is within a predetermined range. It is an object that stays in the difference between Also, when the object at one or more points in the past and the object at the present point in time can be associated with each other, one or more pieces of object position information of the object at the present point in time are acquired.

Then, for example, the object recognition unit 133 stores one or more pieces of object position information of each object in a buffer (not shown) in association with the moving body position information acquired from the same original information. It is preferable that the object recognition unit 133 also store the attribute values of each object together in a buffer (not shown).
(3) Third processing example of the object recognition unit 133

A third processing example is a method of storing a set of past 3D points, comparing it with a set of 3D points reflecting changes in the self-position, and detecting a set of 3D points divided into moving and non-moving objects. be. In the third processing example, the same effect as the differential processing during fixed-point observation is obtained, so even in the case of moving observation, it is possible to acquire the object point group more robustly. It should be noted that the "moving object" here means "an object whose position (in the reference coordinate system) changes partially or wholly". An object whose position changes partly is, for example, a human being whose only hands are moving. More specifically, first, the moving object position information acquiring unit 132 performs the above-described processing to determine the amount of movement of the feature point (for example, (x _t −x _t−1 , y _t −y _t−1 , z _t − z _t−1 )) Calculate. Then, the mobile body position information acquiring unit 132 acquires the mobile body position information (eg, (x _t-1 , y _t-1 , z _{t- 1} ))) and the amount of movement, the position information of the mobile object at the current time point (t) (for example, (x _t , y _t , z _t )) is acquired.

Next, in order to synchronize the set of 3D points at one or more past points in time (for example, the immediately preceding point) with the set of 3D points at the present time, the object recognition unit 133 converts the set of 3D points at the present point into Correction is performed using the moving object position information.

Next, the object recognition unit 133 performs object recognition processing on the current set of 3D points.

Specifically, the object recognition unit 133 determines whether each current 3D point belongs to a moving object, considering whether it exists at the same position as a past 3D point, object recognition processing. Here, taking into consideration means that, for example, when specifying a set of 3D points forming an object based on the distance between each of the plurality of 3D points, both of the plurality of 3D points exist at the same positions as the past 3D points. It refers to weighting the distance depending on whether or not Next, the object recognition unit 133 recognizes one or more object position information and/or attribute values of each object obtained from a set of 3D points at a past time (for example, immediately before), and a current 3D point. Each object obtained from a set of 3D points in the past and each object for a set of 3D points at the present time by comparing one or more object position information and/or attribute values of each object with respect to the set of points determine how to deal with

Note that an object that was moving or non-moving in the past may now be determined to be non-moving or moving, unlike past determinations. For example, a person who was standing still starts walking, or a moving car is parked. It is also assumed that flag information indicating whether an object is moving or non-moving is a kind of attribute value of the object. Here, the objects may be associated in consideration of the amount of movement of the objects in the past as described in (2).

Then, for example, the object recognition unit 133 stores one or more pieces of object position information of each object in a buffer (not shown) in association with the moving body position information acquired from the same original information. It is preferable that the object recognition unit 133 also store the attribute values of each object together in a buffer (not shown).

The object recognition unit 133 uses, for example, the transmitting means position information and the transmitting means direction information paired with the laser identifiers possessed by the two or more pieces of laser information acquired by the original information acquisition unit 131, and the return time possessed by the laser information. object position information of each object is obtained for each laser information, one or more objects are identified using the obtained two or more object position information, and each of the one or more objects and two or more object position information of each object are identified. and

It is preferable that the object recognition unit 133 identifies one or more objects at each of two or more points in time, and associates each of the one or more objects with two or more pieces of object position information of each object.

When the original information is a set of a laser identifier and a return time, the object recognition unit 133 acquires object position information indicating a relative position within the three-dimensional space with respect to the moving body position information from the original information. For example, the outline of the object is extracted from two or more pieces of object position information, the object information in the storage unit 11 corresponding to the outline is selected, and the object identifier of the object information is acquired. Then, two or more pieces of object position information forming the outline of the object identified by the object identifier and the object identifier are paired and held. In such a case, the storage unit 11 stores one or more pieces of object information. It goes without saying that the object recognition unit 133 may hold object position information inside the object in addition to the two or more pieces of object position information forming the outline of the object, paired with the object identifier.

Also, when the original information is a range image, an object in space is recognized from the range image and an object identifier is acquired. A technique for recognizing an object from an image is a known technique. The object recognition unit 133 can use a machine learning technique such as deep learning, a technique of pattern matching with object information, or the like. Then, the object recognition unit 133 holds the object identifier and two or more pieces of object position information forming the recognized object as a pair.

The location where the object recognition unit 133 holds the pair of object identifiers and two or more pieces of object position information may be the storage unit 11 or a buffer (not shown).

The output diagram constructing unit 134 constructs an output diagram using the moving body position information and two or more object position information of one or more objects. The output drawing is a three-dimensional drawing that allows recognition of the relative positional relationship between the moving object and one or more objects. A specific example of the output diagram will be described later. It should be noted that the moving body position information and the object position information used by the output diagram construction unit 134 are information acquired from the same one or more pieces of original information. Also, it can be said that two or more pieces of object position information of each of one or more objects are recognition results of each of one or more objects. The output diagram construction unit 134 is a diagram including one or more objects recognized by the object recognition unit 133 at each of two or more points in time. An output diagram, which is a diagram including trajectory information indicating the movement trajectory of the moving object configured by the position information, is constructed.

The determination unit 135 acquires motion information about the motion of each of the one or more objects identified by the object recognition unit 133 using the two or more pieces of original information acquired by the original information acquisition unit 131 at two or more points in time, and obtains the motion information. matches a predetermined condition. The motion information is, for example, a history of changes in the distance between the moving object and the object (for example, "15m, 12m, 6m, etc."). The motion information is, for example, information indicating that the distance between the moving body and the object is decreasing. The motion information is, for example, a set of two or more time points of the object position information group of the object. An object position information group is a set of two or more pieces of object position information specifying one object.

The output unit 14 outputs various information. Here, output usually means display on a display, but projection using a projector, printing with a printer, transmission to an external device (usually a display device), storage in a recording medium, and other processing This is a concept including delivery of processing results to devices and other programs.

The diagram output unit 141 outputs the output diagram configured by the output diagram configuration unit 134 . The form of the output diagram does not matter. An example of the output diagram will be described later. In the output diagram, it is preferable that the two or more pieces of object position information specifying the object recognized by the object recognition unit 133 are distinguishable from other object position information. In the output drawing, it is preferable that the object recognized by the object recognition unit 133 is in a recognizable form. In addition, it is preferable that the output diagram be in a mode in which a moving object and a non-moving object can be distinguished.

The determination information output unit 142 outputs determination information regarding the determination when the determination unit 135 determines that the conditions are met. If the determination unit 135 determines that the condition is met, the determination information output unit 142 acquires determination information corresponding to the condition from the storage unit 11, and uses the information to output the determination information.

Here, output means display on a display, projection using a projector, printing on a printer, sound output, transmission to a device in the department (usually a display device), storage in a recording medium, storage in a recording medium, other processing devices, or This is a concept that includes the delivery of processing results to other programs.

The storage unit 11 is preferably a non-volatile recording medium, but can also be realized with a volatile recording medium. The process by which the information is stored in the storage unit 11 does not matter. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or Information input via an input device may be stored in the storage unit 11 .

The processing unit 13, the original information acquisition unit 131, the moving body position information acquisition unit 132, the object recognition unit 133, the output diagram construction unit 134, and the judgment unit 135 can be usually implemented by an MPU, memory, or the like. The processing procedure of the processing unit 13 and the like is normally realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

The output unit 14, the diagram output unit 141, and the judgment information output unit 142 may or may not include output devices such as displays and speakers. The output unit 14 and the like can be realized by output device driver software, or by output device driver software and an output device.

Next, an operation example of the information processing apparatus 1 will be described using the flowchart of FIG.

(Step S201) The reception unit 12 determines whether or not an operation start instruction has been received. If the operation start instruction is received, the process goes to step S202, and if the operation start instruction is not received, the process returns to step S201.

(Step S202) The original information acquisition unit 131 acquires original information. Such original information acquisition processing will be described with reference to the flowchart of FIG.

(Step S203) The mobile position information acquisition unit 132 acquires mobile position information. When the moving body position information acquiring unit 132 acquires moving body position information indicating a relative position, the origin (0, 0, 0) is acquired first, for example. An example of mobile body position information acquisition processing will be described with reference to the flowchart of FIG.

(Step S204) The object recognition unit 133 performs object recognition processing. An example of such object recognition processing will be described with reference to the flowchart of FIG.

(Step S205) The output diagram configuration unit 134 configures the output diagram. Such output diagram construction processing will be described with reference to the flowchart of FIG.

(Step S206) The diagram output unit 141 outputs the output diagram configured in step S205.

(Step S207) The determination unit 135 performs determination processing on the moving object. Such determination processing will be described with reference to the flowchart of FIG.

(Step S208) The determination information output unit 142 determines whether or not determination information has been obtained as a result of the determination processing in step S207. If the judgment information is obtained, the process goes to step S209, and if the judgment information is not obtained, the process goes to step S210.

(Step S209) The determination information output unit 142 outputs one or more pieces of determination information obtained in step S207.

(Step S210) The reception unit 12 determines whether or not an operation end instruction has been received. If the operation end instruction is accepted, the process is terminated, and if the operation end instruction is not accepted, the process returns to step S201.

Next, an example of the original information acquisition process in step S203 will be described using the flowchart of FIG.

(Step S301) The original information acquisition unit 131 substitutes 1 for the counter i.

(Step S302) The original information acquisition unit 131 determines whether or not the i-th laser transmitting means exists. If the i-th laser transmission means exists, go to step S303, and if the i-th laser transmission means does not exist, go to step S307.

(Step S303) The original information acquisition unit 131 acquires the laser identifier of the i-th laser transmission means. Note that the laser identifier is normally stored in the storage unit 11 .

(Step S304) The original information acquisition unit 131 acquires the return time of the laser transmitted by the i-th laser transmission means. It should be noted that such processing is a known technology of LIDAR.

(Step S305) The original information acquisition unit 131 configures original information having the laser identifier acquired in step S303 and the return time acquired in step S304. Note that the original information acquisition unit 131 may calculate three-dimensional coordinate values from the laser identifier and the return time, and configure the original information having the three-dimensional coordinate values. In such a case, the original information acquisition unit 131 acquires the distance to the origin from the return time, acquires the angle to the origin corresponding to the laser identifier, and calculates the three-dimensional coordinate value to the origin from the distance and the angle. Such treatments are known in the art. Also, the angle with respect to the origin is stored in the storage unit 11 in association with the laser identifier.

(Step S306) The original information acquisition unit 131 increments the counter i by 1. Return to step S302.

(Step S307) The original information acquisition unit 131 constructs the original information group at this time and temporarily stores it in a buffer (not shown). Return to upper process. Note that the original information group is a set of two or more pieces of original information configured in step S305.

In the flowchart of FIG. 3, it is preferable that the original information group at this point is, for example, a set of original information for 360 degrees (one round) around the mobile object at one point.

In addition, in the flowchart of FIG. 3, the original information acquisition unit 131 may acquire one or more distance images around the moving object.

Next, an example of the mobile body position information acquisition processing in step S203 will be described using the flowchart of FIG.

(Step S401) The moving body position information acquisition unit 132 acquires from a set of coordinate values in a three-dimensional space possessed by one or more pieces of original information acquired by the original information acquisition unit 131, or from one or more pieces of each piece of original information. One or more feature points are acquired from a set of coordinate values in the three-dimensional space and temporarily stored in a buffer (not shown). Note that the feature points are 3D coordinate values.

(Step S402) The mobile body position information acquisition unit 132 determines whether or not one or more feature points at the immediately preceding point are stored in a buffer (not shown). If one or more feature points are stored, go to step S403, otherwise go to step S408.

(Step S403) The moving body position information acquisition unit 132 reads one or more feature points at the immediately preceding point from a buffer (not shown).

(Step S404) The moving body position information acquiring unit 132 acquires feature points corresponding to one or more feature points among the one or more feature points acquired in step S401. determine the point.

(Step S405) The mobile body position information acquisition unit 132 calculates the movement amount, which is the difference between the two corresponding feature points. Note that the corresponding two feature points are the current feature point acquired in step S401 and the previous feature point. Corresponding means, for example, that the shapes formed by a set of peripheral points are similar and the distance remains within a certain range. Also, the feature points used for calculating the amount of movement are the feature points of the non-moving object. Further, the moving body position information acquiring unit 132 may calculate differences between sets of two or more feature points, and use the largest difference as the amount of movement. Note that a set of feature points is two corresponding feature points.

(Step S406) The moving body position information acquisition unit 132 reads the immediately preceding moving body position information from a buffer (not shown).

(Step S407) The mobile body position information acquisition unit 132 calculates current mobile body position information from the mobile body position information read out in step S406 and the amount of movement acquired in step S405. Return to upper process.

(Step S408) The moving body position information acquisition unit 132 associates the origin with one or more feature points acquired in step S401. Note that the origin is the mobile body position information at the start. Return to upper process. Next, an example of object recognition processing in step S204 will be described using the flowchart of FIG.

(Step S501) The object recognition unit 133 substitutes 1 for the counter i.

(Step S502) The object recognition unit 133 determines whether or not the i-th original information exists in the original information group temporarily accumulated in the buffer (not shown). If the i-th original information exists, go to step S503, and if the i-th original information does not exist, go to step S506.

(Step S503) The object recognition unit 133 acquires the i-th object position information from the i-th original information. For example, the object recognition unit 133 acquires from the storage unit 11 the transmitting means position information and transmitting means direction information paired with the laser identifier possessed by the i-th original information, and stores the transmitting means position information and transmitting means direction information. Using the return time included in the i-th original information, the relative three-dimensional coordinate values with respect to the moving body position information corresponding to the i-th original information are obtained. Then, the object recognition unit 133 acquires the i-th object position information, for example, using the mobile body position information and the three-dimensional coordinate values.

(Step S504) The object recognition unit 133 temporarily accumulates the i-th object position information acquired in step S503 in a buffer (not shown).

(Step S505) The object recognition unit 133 increments the counter i by 1. Return to step S502.

(Step S506) The object recognition unit 133 substitutes 1 for the counter j.

(Step S507) The object recognition unit 133 determines whether or not j-th object position information exists in a buffer (not shown). If the j-th object position information exists, go to step S508, and if the j-th object position information does not exist, return to the upper process.

(Step S508) The object recognition unit 133 acquires j-th object position information from a buffer (not shown).

(Step S509) The object recognition unit 133 recognizes an object position information group including the j-th object position information acquired in step S508, and an object position information group constituting one object formed by the j-th object position information. get. Note that this object position information group is a set of object position information forming the one object, and includes the j-th object position information. A set of object position information forming one object is, for example, a set of object position information of positions close enough to satisfy a predetermined relationship.

(Step S510) The object recognition unit 133 determines whether or not an object position information group corresponding to the object position information group acquired in step S509 exists in the immediately preceding object position information group. If it exists, go to step S511; if not, go to step S514.

The object position information group corresponding to the object position information group acquired in step S509 is the object position information group of the same object as the object position information group acquired in step S509. The object specified by this object position information group may be a stationary object or a moving object. For stationary objects, two sets of object position information specifying the same object at different times are the same information or information contained one in the other. Also, in the case of a moving object, two groups of object position information specifying the same object at different points in time are information in which a plurality of pieces of object position information are shifted in the same width by a certain ratio or more. Note that when recognizing objects around 360 degrees of a moving object at one time, if the current time is the N-th time of recognizing the surrounding objects, the immediately preceding time is the (N−1)th time of This is the point at which object recognition is performed.

(Step S511) The object recognition unit 133, from the object position information group acquired in step S509 and the object position information group corresponding to the object position information group acquired in step S509 (the object position information group at the previous point in time), It is determined whether or not the object corresponding to the object position information group is moving. If it has moved, go to step S512; if not, go to step S513. The object recognition unit 133 uses the amount of movement of the moving object to determine whether the object is moving.

(Step S512) The object recognition unit 133 adds a moving object flag indicating that the object is a moving object in association with the object position information group acquired in step S509.

(Step S513) The object recognition unit 133 adds a non-moving object flag indicating that the object is not a moving object in association with the object position information group acquired in step S509.

(Step S514) The counter i is incremented by N. Return to step S507. Note that N is the number of pieces of object position information forming one object.

In addition, in the flowchart of FIG. 5, you may use the algorithm of the other object recognition processing mentioned above. Next, an example of the output diagram construction processing in step S205 will be described with reference to the flowchart of FIG.

(Step S601) The output diagram configuration unit 134 determines whether or not the previous output diagram exists. If the previous output map exists, the process goes to step S602, and if the previous output map does not exist, the process goes to step S603. It should be noted that the immediately preceding output diagram is the output diagram output immediately before. The output map output immediately before is an output map configured using a set of original information acquired at the point immediately before the current time.

(Step S602) The output diagram construction unit 134 acquires the object position information that constructed the previous output diagram from a buffer (not shown).

(Step S603) The output diagram construction unit 134 substitutes 1 for the counter i.

(Step S604) The output diagram construction unit 134 determines whether or not the i-th object position information group corresponding to the moving object flag exists. If it exists, go to step S605; if not, go to step S607. The object position information group corresponding to the moving object flag is the object position information group of objects that are moving objects.

(Step S605) The output diagram construction unit 134 adds a deletion flag to the i-th object position information group corresponding to the moving object flag. Note that the object position information group to which the deletion flag is added is not used for constructing the output drawing. Adding a deletion flag may be deletion.

(Step S606) The output diagram construction unit 134 increments the counter i by 1. Return to step S604.

(Step S607) The output diagram construction unit 134 adds the currently acquired set of object position information to the set of object position information constituting the output diagram. Here, in the set of object position information acquired at the present time, if the same object position information constituting the same object exists in the set of object position information constituting the output map, the output map configuration Preferably, unit 134 does not add the object location information to the collection of object location information that makes up the output view.

(Step S608) The output diagram construction unit 134 acquires trajectory information indicating the movement trajectory of the moving body using the one or more pieces of moving body position information acquired in step S202.

(Step S609) The output diagram constructing unit 134 constructs an output diagram using the set of object position information obtained in step S607 and the trajectory information obtained in step S608. Return to upper process.

In the flowchart of FIG. 6, the output diagram showing the current position of the moving object may be constructed based on the moving object position information showing the current position of the moving object instead of the trajectory information.

Next, an example of determination processing in step S207 will be described using the flowchart of FIG.

(Step S701) The determination unit 135 substitutes 1 for the counter i.

(Step S702) The determination unit 135 determines whether or not the i-th moving object exists using the object position information group in the buffer. If the i-th moving object exists, go to step S703, and if the i-th moving object does not exist, return to the higher-level processing.

(Step S703) The determination unit 135 acquires the object position information group of the i-th moving object from a buffer (not shown).

(Step S704) The determination unit 135 substitutes 1 for the counter j.

(Step S<b>705 ) The determination unit 135 determines whether or not the j-th predetermined condition exists in the storage unit 11 . If the condition exists, go to step S706; otherwise, go to step S709.

(Step S706) The determination unit 135 determines whether the object of the i-th moving object corresponding to the object position information group of the i-th moving object acquired in step S703 matches the j-th predetermined condition. to judge. If they match, go to step S707; if they do not match, go to step S708. Here, the determination unit 135 applies the object position information group (information indicating the trajectory of movement) of the i-th moving object at two or more different points in time to the j-th predetermined condition, and applies the condition. It is preferable to determine whether it satisfies.

(Step S707) The determination unit 135 acquires determination information corresponding to the j-th predetermined condition. It is preferable that this judgment information also corresponds to the object position information group of the i-th moving object. In this case, the determination information includes information identifying the i-th moving object. The information specifying the object of the i-th moving object may be an object position information group of the i-th moving object or an object identifier of the i-th moving object. Note that the determination unit 135 reads, for example, determination information corresponding to the j-th predetermined condition from the storage unit 11 .

(Step S708) The determination unit 135 increments the counter j by 1. Return to step S705.

(Step S709) The determination unit 135 increments the counter i by 1. Return to step S702.

A specific operation of the information processing apparatus 1 according to the present embodiment will be described below. A conceptual diagram of the information processing apparatus 1 is shown in FIG. In FIG. 8, 801 is the information processing apparatus 1 . 802 is a LIDAR and has a number of laser emitting means. Further, the information processing apparatus 1 is, for example, a robot, a mobile object such as an automobile, an AGV (automated guided vehicle), or the like. In such a case, the following four specific examples will be described.
(Specific example 1)

The original information acquisition unit 131 of the information processing device 1 acquires a large amount of original information from the LIDAR 802 through the above-described processing. It should be noted that the LIDAR 802 emits a laser to 360 degrees around the moving object, receives the reflected laser, and acquires the original information at once.

Next, using the original information acquired by the original information acquisition unit 131, the mobile location information acquisition unit 132 acquires the mobile location information through the above-described processing.

Next, the object recognizing unit 133 uses the same original information as the original information used by the moving body position information acquisition unit 132 and recognizes objects around the information processing device 1, which is a moving body, through the above-described process.

Then, the object recognition unit 133 associates the moving body position information acquired using the same original information with the object recognition result and accumulates them in a buffer. Note that the object recognition result is, for example, two or more pieces of object position information and one or more attribute values of each object.

Then, the output diagram construction unit 134 constructs an output diagram including, for example, a plurality of objects through the above-described processing, and the diagram output unit 141 outputs the output diagram. An example of an output diagram is shown in FIG. In FIG. 9, objects 901, 902, and 903 recognized by the object recognition unit 133 are surrounded by a rectangular parallelepiped BOX.

In FIG. 9, it is possible to detect, identify, analyze, and track people and things around the moving object. Further, for example, according to the information processing device 1, by detecting and tracking a person or vehicle existing within a specific range, it becomes possible to determine the situation in an autonomous mobile system and analyze the flow of people and flow lines. .
(Specific example 2)

The original information acquisition unit 131 of the information processing device 1 acquires a large amount of original information from the LIDAR 802 through the above-described processing.

Next, the object recognition unit 133 recognizes objects around the information processing device 1, which is a moving object, by the above-described processing.

Further, the moving body position information acquisition unit 132 acquires the moving body position information through the above-described processing.

Note that both the moving body position information acquisition unit 132 and the object recognition unit 133 perform processing using the same original information. Note that using the same original information includes, for example, the object recognition unit 133 using information obtained by the moving body position information obtaining unit 132 using the original information.

Then, the output diagram constructing unit 134 constructs an output diagram including, for example, trajectory information indicating trajectories of a plurality of objects and a moving body, through the above-described processing.

Next, the diagram output unit 141 outputs such an output diagram. An example of the output diagram is shown in FIG. In FIG. 10, 1001 is a diagram showing the current position of the moving object.

The mobile body position information acquisition unit 132 estimates the amount of movement of the observation point (information processing device 1) from continuous 3D data (acquires mobile body position information). Then, in FIG. 10, the information processing device 1 can recognize the position of the moving object in the real world in real time. This indicates that it is possible to recognize the self-position in automatic driving and grasp the cargo arrangement by tracing the transport equipment.
(Specific example 3)

The original information acquisition unit 131 of the information processing device 1 acquires a large amount of original information from the LIDAR 802 through the above-described processing.

Next, the moving body position information acquisition unit 132 acquires the moving body position information through the processing described above.

Next, the object recognition unit 133 recognizes objects around the information processing device 1, which is a moving body, by the above-described processing.

In addition, the determination unit 135 can recognize, for example, the movement of a hand or the posture of an approaching person through the above-described processing. Then, the determination unit 135 enables realization of various functions such as detection of a dangerous scene and interaction with the service robot (see FIG. 11).

Note that the judgment information output unit 142 outputs, for example, "information indicating danger" or the like, or outputs a dangerous person in a conspicuous manner in the output diagram.
(Specific example 4)

The original information acquisition unit 131 of the information processing device 1 acquires a large amount of original information from the LIDAR 802 through the above-described processing.

Next, the object recognition unit 133 recognizes objects around the information processing device 1, which is a moving body, by the above-described processing.

Further, the moving body position information acquisition unit 132 acquires the moving body position information through the above-described processing. Then, the output diagram constructing unit 134 constructs an output diagram including, for example, trajectory information indicating trajectories of a plurality of objects and a moving body, through the above-described processing. This output map is a three-dimensional map obtained by integrating data obtained by movement observation by a moving object based on self-position recognition.

Then, the diagram output unit 141 outputs an output diagram. An example of such an output is shown in FIG. According to FIG. 12, for example, it is possible to grasp a specific range three-dimensionally, and it can be used for confirmation of construction results and verification of differences from plans.

As described above, according to the present embodiment, the position of a moving body and the objects around the moving body can be recognized at the same time, and the positional relationship between the moving body and the objects around the moving body can be output in real time.

Further, according to this embodiment, the position of a moving body and the objects in all directions of the moving body can be recognized at the same time, and the positional relationship between the moving body and surrounding objects can be output in real time.

Further, according to the present embodiment, the position of a moving body and the objects around the moving body are recognized at the same time, and when the motion of the object matches the conditions (for example, when it is dangerous), the can be notified.

Furthermore, according to this embodiment, the position of a moving body and the objects around the moving body are recognized at the same time, and the movement history of the moving body and the positional relationship between the moving body and the objects around the moving body are output in real time. can.

According to the information processing apparatus 1 according to the present embodiment, for example, in an autonomous mobile system such as an automobile, a construction machine, a carrier, a service robot, while recognizing its own position in the real world, It detects and identifies people and things, and makes it possible to formulate action plans. Further, according to the information processing device 1, it is possible to recognize the self-position even in a place where it is difficult to use GPS, such as inside a facility or underground. In addition, the information processing device 1 can comprehensively support unmanned construction by detecting surrounding vehicles and workers and by real-time measurement of construction conditions such as dimensions. Further, in the information processing device 1, it is possible to trace the moving route of the conveying equipment and record the location of the conveyed cargo. In addition, it is possible to easily check and grasp the placement of materials that were difficult to manage due to problems such as size, when needed. Furthermore, LIDAR, which has a wide field of view, can measure the flow of people over a wide range. In addition, by efficiently arranging multiple sensors, it is possible to analyze stagnation points even in large-scale facilities such as airports.

Note that the processing in this embodiment may be realized by software. Then, this software may be distributed by software download or the like. Also, this software may be recorded on a recording medium such as a CD-ROM and distributed. Note that this also applies to other embodiments in this specification. The software that implements the information processing apparatus 1 according to the present embodiment is the following program. In other words, the program comprises, for example, a computer, an original information acquisition unit that acquires one or more pieces of original information that are used to specify the position in a three-dimensional space of an object that exists around a moving mobile object; A mobile body position information acquisition unit that acquires mobile body position information specifying a position of a moving mobile body in a three-dimensional space using one or more pieces of original information acquired by the original information acquisition unit, and the original information acquisition unit an object recognition unit that identifies one or more objects using one or more pieces of acquired original information and associates each of the one or more objects with two or more pieces of object position information of each object; An output diagram that constitutes an output diagram that is a three-dimensional diagram in which relative positional relationships between the moving object and the one or more objects can be recognized using two or more pieces of object position information of the one or more objects. A configuration unit and a program for functioning as a diagram output unit for outputting the output diagram.

FIG. 13 is an external view of a computer system 300 that implements the information processing apparatus 1 by executing the program according to this embodiment.

Each device of this embodiment can be realized by computer hardware and a computer program executed thereon. In FIG. 13, computer system 300 comprises computer 301 including disk drive 305 , keyboard 302 , mouse 303 and display 304 . The entire system including the keyboard 302, mouse 303, and display 304 may be called a computer.

FIG. 14 is a diagram showing an example of the internal configuration of the computer system 300. As shown in FIG. 14, a computer 301 includes an MPU 3013, a ROM 3015 for storing programs such as a boot-up program, and a RAM 3016 connected to the MPU 3013 and the like for temporarily storing instructions of application programs and providing a temporary storage space. , a hard disk 3017 for storing application programs, system programs, and data, and a bus 3014 for interconnecting the MPU 3013, ROM 3015, and the like.

A program that causes the computer system 300 to execute the functions of the information processing apparatus 1 may be stored in a disk 3101 such as a CD-ROM, inserted into the disk drive 3012 , and transferred to the hard disk 3017 . Alternatively, the program may be transmitted to computer 301 via a network and stored on hard disk 3017 . Programs are loaded into RAM 3016 during execution. Note that the program may be loaded directly from the hard disk 3016 or from the network. Also, the program may be read into the computer system 300 via another removable recording medium (eg, DVD, memory card, etc.) instead of the hard disk 3016 .

The program does not necessarily include an operating system (OS) or a third-party program that causes the information processing apparatus 1 to execute the functions of the computer details 901 . A program may contain only those portions of instructions that call the appropriate functions or modules in a controlled manner to produce the desired result. How the computer system 300 operates is well known and will not be described in detail. Note that the above is an example, and the hardware configuration of the computer that implements the information processing apparatus 1 does not matter.

Also, the number of computers that execute the above programs may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

Further, in each of the above embodiments, each process may be implemented by centralized processing by a single device, or may be implemented by distributed processing by a plurality of devices.

It goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible and are also included within the scope of the present invention.

As described above, the information processing apparatus according to the present invention can simultaneously recognize the position of a moving body and objects around the moving body, and can output the positional relationship between the moving body and the surrounding objects in real time. It has an effect and is useful as an information processing device or the like.

1 information processing device 11 storage unit 12 reception unit 13 processing unit 14 output unit 131 original information acquisition unit 132 moving body position information acquisition unit 133 object recognition unit 134 output map configuration unit 135 determination unit 141 diagram output unit 142 determination information output unit

Claims (7)

1 or 2 or more pieces of original information that serve as the basis for specifying the position in the 3D space of an object that exists around a moving object, and that is a set of 3D data acquired by an observation device. an original information acquisition unit to acquire;
a moving body position information acquiring unit that acquires moving body position information specifying a position of a moving body in a three-dimensional space using one or more pieces of original information acquired by the original information acquiring unit;
Object recognition that recognizes one or more specific objects using one or more pieces of original information acquired by the original information acquisition unit, and associates each of the one or more pieces of specific objects with two or more pieces of object position information of each object Department and
3D is a three-dimensional diagram in which relative positional relationships between the moving body and the one or more objects can be recognized using the moving body position information and two or more pieces of object position information of the one or more objects. an output diagram configuration unit that configures an output diagram;
a diagram output unit that outputs the output diagram;
Only the one or more specific objects recognized by the object recognition unit using the two or more pieces of original information acquired by the original information acquisition unit at two or more points in time , the specific one or more objects action information that is a history of changes in the distance between the object and the moving object, or action information that is a set of two or more time points of an object position information group that is a set of object position information that specifies the specific one or more objects and a determination unit that determines whether or not the motion information matches a predetermined condition;
and a determination information output unit configured to output determination information related to the determination when the determination unit determines that the condition is met. The object recognition unit
Obtaining two or more shape feature amounts, which are feature amounts relating to the shape of each object, from the set of three-dimensional coordinate values of each of the two or more objects, and acquiring the two or more shape feature amounts and an object identifier indicating the type of the object. Machine learning prediction processing is performed using learning information obtained by learning two or more object information having two or more feature amounts of the object by machine learning learning processing, and the types of each of the two or more objects 2. The information processing apparatus according to claim 1, wherein the type of each of the two or more objects is associated with the position information of each of the two or more objects and stored. Two or more pieces of laser transmission means information having a laser identifier for identifying a laser transmission means, transmission means position information for specifying the position of the laser transmission means, and transmission means direction information for specifying the direction of radar transmission of the laser transmission means. further comprising a storage unit to be stored,
The original information acquisition unit
Receiving the reflection of the laser transmitted by each of the two or more laser transmission means, acquiring the return time from the transmission of the laser to the reception of the reflection, and combining the return time with a laser identifier that identifies the laser transmission means is acquired for each laser transmission means,
The object recognition unit
Object position information of an object using transmitting means position information and transmitting means direction information paired with a laser identifier possessed by each of the two or more pieces of laser information acquired by the original information acquisition unit, and using the return time possessed by the laser information. is acquired for each laser information, the specific one or more objects are recognized using the acquired two or more object position information, and the one or more objects and the two or more object position information of each object are recognized. 3. The information processing apparatus according to claim 1 or 2, to be associated. The object recognition unit
Also determining whether each of the identified two or more objects is a moving body or a non-moving body using the two or more pieces of original information acquired by the original information acquisition unit;
The output diagram configuration unit
4. The information processing apparatus according to any one of claims 1 to 3, wherein each of the two or more objects is a moving object or a non-moving object constitutes an output diagram in a manner distinguishable. The information processing apparatus according to any one of claims 1 to 4, wherein each of the one or more specific objects is a person or a vehicle. An information processing method realized by an original information acquisition unit, a mobile position information acquisition unit, an object recognition unit, an output diagram configuration unit, a diagram output unit, a determination unit, and a determination information output unit,
The original information acquisition unit is a set of three-dimensional data acquired by an observation device, which is one or more pieces of original information that are the basis for specifying the position in the three-dimensional space of an object existing around a moving mobile object. an original information obtaining step of obtaining one or more pieces of original information;
Acquisition of mobile body position information, wherein the mobile body position information acquisition unit acquires mobile body position information specifying a position of the mobile body in a three-dimensional space, using the one or more pieces of original information acquired in the original information acquisition step. a step;
The object recognition unit recognizes one or more specific objects using the one or more original information obtained in the original information obtaining step, and recognizes the one or more specific objects and two or more objects of each object. an object recognition step that associates position information;
The output diagram construction unit can recognize a relative positional relationship between the moving body and the one or more objects using the moving body position information and two or more object position information of the one or more objects. an output diagram construction step for constructing an output diagram that is a three-dimensional diagram;
a diagram output step in which the diagram output unit outputs the output diagram;
The determination unit uses two or more pieces of original information acquired by the original information acquisition unit at two or more points in time to obtain only the one or more specific objects recognized by the object recognition unit. Two or more time points of an object position information group that is a set of object position information specifying one or more specific objects, or motion information that is a history of changes in the distance between one or more objects and the moving body a determination step of acquiring motion information as a set and determining whether or not the motion information matches a predetermined condition;
and a determination information output step, wherein the determination information output unit outputs determination information related to the determination when it is determined in the determination step that the condition is met. the computer,
1 or 2 or more pieces of original information that serve as the basis for specifying the position in the 3D space of an object that exists around a moving object, and that is a set of 3D data acquired by an observation device. an original information acquisition unit to acquire;
a moving body position information acquiring unit that acquires moving body position information specifying a position of a moving body in a three-dimensional space using one or more pieces of original information acquired by the original information acquiring unit;
Object recognition that recognizes one or more specific objects using one or more pieces of original information acquired by the original information acquisition unit, and associates each of the one or more pieces of specific objects with two or more pieces of object position information of each object Department and
3D is a three-dimensional diagram in which relative positional relationships between the moving body and the one or more objects can be recognized using the moving body position information and two or more pieces of object position information of the one or more objects. an output diagram configuration unit that configures an output diagram;
a diagram output unit that outputs the output diagram;
Only the one or more specific objects recognized by the object recognition unit using the two or more pieces of original information acquired by the original information acquisition unit at two or more points in time , the specific one or more objects action information that is a history of changes in the distance between the object and the moving object, or action information that is a set of two or more time points of an object position information group that is a set of object position information that specifies the specific one or more objects and a determination unit that determines whether or not the motion information matches a predetermined condition;
A program for functioning as a determination information output unit that outputs determination information regarding the determination when the determination unit determines that the condition is met.

Images