JPWO2020008684A1 - Object recognition device, object recognition system, and program - Google Patents

Abstract

The object recognition device (200) includes an input unit (21) for inputting three-dimensional information including a three-dimensional position along the outer shape of at least a part of the object, and an arithmetic processing unit (21) for recognizing the object based on the three-dimensional information. 22), and the arithmetic processing unit (22) generates a plurality of two-dimensional information when the three-dimensional position represented by the three-dimensional position is viewed from a plurality of directions based on the three-dimensional information, and the plurality of two-dimensional information is generated. Recognize an object based on information.

Description

The present disclosure relates to an object recognition device, an object recognition system, and a program for recognizing an object.

Non-Patent Document 1 discloses VoxNet. VoxNet is a method of recognizing an object by image processing using a three-dimensional convolutional neural network (3D CNN). Specifically, VoxNet maps three-dimensional point cloud data obtained from LiDAR, RGBD sensors, etc. into a three-dimensional space of a predetermined size to generate three-dimensional information, and the three-dimensional information is converted into a three-dimensional convolutional neural network. It is a method of recognizing an object by inputting to.

Daniel Maturana, Sebastian Scherer, "VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition", Internet <URL: https://www.ri.cmu.edu/pub_files/2015/9/voxnet_maturana_scherer_iros15.pdf>

Image processing by a three-dimensional convolutional neural network as in Non-Patent Document 1 requires a large amount of data and a large network. Therefore, the processing load of image processing is large, and the processing speed of object recognition is slow.

An object of the present disclosure is to provide an object recognition device, an object recognition system, and a program that reduce the processing load of image processing and improve the processing speed of object recognition.

The object recognition device according to the present disclosure includes an input unit for inputting three-dimensional information including a three-dimensional position along the outer shape of at least a part of the object, and an arithmetic processing unit for recognizing the object based on the three-dimensional information. The arithmetic processing unit generates a plurality of two-dimensional information showing a two-dimensional drawing of a solid represented by a three-dimensional position viewed from a plurality of directions based on the three-dimensional information, and is based on the plurality of two-dimensional information. Recognize an object.

The object recognition system according to the present disclosure includes a sensor that measures a distance to an object and generates three-dimensional information, and the object recognition device.

The program according to the present disclosure includes a step of inputting three-dimensional information including a three-dimensional position along the outer shape of at least a part of an object, and a plurality of directions of a solid represented by the three-dimensional position based on the three-dimensional information. The computer is made to perform a step of generating a plurality of two-dimensional information showing the two-dimensional drawing viewed from the above and a step of recognizing an object based on the plurality of two-dimensional information.

According to the object recognition device, the object recognition system, and the program according to the present disclosure, a plurality of two showing a two-dimensional drawing of a solid represented by a three-dimensional position along the outer shape of at least a part of an object as viewed from a plurality of directions. Since the object is recognized based on the dimensional information, the processing load of image processing is reduced. Therefore, the processing speed of object recognition is improved.

The figure for demonstrating the application example of the object recognition system which concerns on this disclosure. Block diagram illustrating the configuration of the object recognition system according to the first and second embodiments. The figure for demonstrating the distance measurement by the distance sensor which concerns on Embodiments 1 and 2. A flowchart showing an example of the object recognition process by the object recognition device according to the first embodiment. The figure which shows an example of the distance image which concerns on Embodiment 1 and the recognition target area which is detected. The figure for demonstrating the three-dimensional space which concerns on Embodiments 1 and 2. The figure for demonstrating the occupancy grid which concerns on Embodiment 1. The figure which shows an example of the plan view of the voxel in the occupied area of FIG. The figure which shows an example of the front view of the voxel in the occupied area of FIG. The figure which shows an example of the side view of the voxel in the occupied area of FIG. The figure for demonstrating the generation of the composite drawing based on the three views of FIGS. 8A-8C. The figure for demonstrating the image processing by the convolutional neural network which concerns on Embodiments 1 and 2. A flowchart showing an example of the learning process of the convolutional neural network according to the first embodiment. A diagram for explaining a comparison between the three views and the number of elements of VoxNet. A flowchart showing an example of the object recognition process by the object recognition device according to the second embodiment. The figure for demonstrating the occupancy grid which concerns on Embodiment 2. The figure which shows the hexagonal view of the voxel in the occupied area of FIG.

Hereinafter, embodiments of the object recognition system according to the present disclosure will be described with reference to the accompanying drawings. In each of the following embodiments, the same reference numerals are given to the same components.

(Application example)
An example to which the object recognition system according to the present disclosure can be applied will be described with reference to FIG. FIG. 1 is a diagram for explaining an application example of the object recognition system 1 according to the present disclosure.

The object recognition system 1 according to the present disclosure can be applied to, for example, in-vehicle use. In the example shown in FIG. 1, the object recognition system 1 is mounted on the vehicle 3. The vehicle 3 is, for example, an autonomous driving vehicle, and includes a vehicle driving device 2 for performing automatic driving. The object recognition system 1 recognizes, for example, an object 4 in the traveling direction of the vehicle 3. The object 4 is, for example, a car, a bus, a motorcycle, a bicycle, a pedestrian, a utility pole, a curb, or a guardrail.

The object recognition system 1 projects light in the traveling direction of the vehicle 3 and receives the reflected light reflected by the object 4. The object recognition system 1 measures the distance from the object recognition system 1 to the object 4 based on the time difference from the light emission to the light reception. The object recognition system 1 generates sensing data including a three-dimensional position along the outer shape of the object 4 based on the measured distance.

The object recognition system 1 generates a plurality of two-dimensional information showing a two-dimensional drawing of a solid represented by a three-dimensional position viewed from a plurality of directions based on sensing data. The object recognition system 1 recognizes an object based on a two-dimensional drawing viewed from a plurality of directions. The object recognition system 1 outputs, for example, object information indicating the distance to the object 4, the direction, the recognition result, and the like to the vehicle driving device 2.

The vehicle driving device 2 includes, for example, a steering mechanism that drives the vehicle 3 by avoiding the object 4 on the road and setting the traveling direction based on the object information output from the object recognition system 1. By recognizing the object 4 by the object recognition system 1, the vehicle driving device 2 can perform automatic driving while avoiding the object 4.

(Configuration example)
Hereinafter, embodiments of the object recognition system 1 as a configuration example will be described.

(Embodiment 1)
The configuration and operation of the object recognition system 1 according to the first embodiment will be described below.

1. 1. Configuration The configuration of the object recognition system 1 according to the present embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram illustrating the configuration of the object recognition system 1. FIG. 3 is a diagram for explaining distance measurement by the distance sensor 100.

The object recognition system 1 includes a distance sensor 100 and an object recognition device 200.

1.1 Configuration of distance sensor The distance sensor 100 includes a light emitting unit 11, a light receiving unit 12, a scanning unit 13, a sensor control unit 14, and an input / output interface unit 15. The distance sensor 100 is, for example, a LIDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging) device.

The light projecting unit 11 projects light to the outside. Specifically, the light projecting unit 11 emits a light flux to the outside according to the control of the sensor control unit 14. The light projecting unit 11 includes, for example, a light source composed of one or more light source elements and a light source driving circuit for pulse-driving the light source. The light source element is, for example, a semiconductor laser (LD) that emits laser light. The light source element may be an LED or the like. For example, the light source elements are arranged in a row in an array in the vertical direction Y shown in FIG. 3, and the light projecting unit 11 projects light toward the light projecting region R11.

The light receiving unit 12 receives light from the outside. The light receiving unit 12 includes a plurality of light receiving elements. When the light receiving element receives light, it generates a light receiving signal according to the amount of light received. The plurality of light receiving elements are arranged in a row in an array along the vertical direction Y, for example. Each light receiving element corresponds to, for example, one pixel of a distance image, and separately receives light incident from a range corresponding to the vertical angle of view of one pixel. The light receiving element is composed of, for example, a SPAD (single photon avalanche photodiode). The light receiving element may be composed of a PD (photodiode) or an APD (avalanche photodiode).

The scanning unit 13 includes, for example, a mirror, a rotation mechanism that rotates the mirror around a rotation axis along the vertical direction Y, and a scanning drive circuit that drives the rotation mechanism. The scanning drive circuit rotationally drives the mirror under the control of the sensor control unit 14. As a result, the scanning unit 13 gradually changes the direction of light projection at regular time intervals, and gradually moves the optical path through which the light travels. For example, as shown in FIG. 3, the scanning unit 13 shifts the light projecting region R11 in the horizontal direction X.

The sensor control unit 14 can be realized by a semiconductor element or the like. The sensor control unit 14 can be composed of, for example, a microcomputer, a CPU, an MPU, a GPU, a DSP, an FPGA, and an ASIC. The function of the sensor control unit 14 may be configured only by hardware, or may be realized by combining hardware and software. The sensor control unit 14 realizes a predetermined function by reading data or a program stored in a storage unit in the distance sensor 100 and performing various arithmetic processes.

The sensor control unit 14 controls the timing of light projection by the light projection unit 11. The sensor control unit 14 generates data of a light receiving waveform indicating the amount of light received according to the elapsed time after the light is projected, for each pixel, based on the timing of the light projection and the light receiving signal obtained from the light receiving unit 12. The sensor control unit 14 calculates the distance for each pixel based on the received light waveform. For example, the sensor control unit 14 measures the flight time of the light from the light projected from the light projecting unit 11 to being reflected by the light receiving unit 12 based on the received light waveform. The sensor control unit 14 calculates the distance from the reference point 5 to the outer shape of the object that reflects light based on the measured flight time. The reference point 5 is, for example, a light outlet of the light projecting unit 11. The sensor control unit 14 generates a distance image based on the distance measured for each pixel.

The sensor control unit 14 measures the distance while scanning the projection surface R10 corresponding to the angle of view of the distance image in the horizontal direction X, and generates a distance image. The resolution of the distance image, that is, the angle of view for each pixel is, for example, 1.0 to 1.6 degrees in the horizontal direction X and 0.3 to 1.2 degrees in the vertical direction Y. By repeating the scanning of the projection surface R10, distance images can be sequentially generated at a desired frame rate. For example, the sensor control unit 14 outputs the generated distance image as sensing data to the object recognition device 200 via the input / output interface unit 15.

The input / output interface unit 15 includes a circuit that communicates with an external device in accordance with a predetermined communication standard. Predetermined communication standards include, for example, LAN, Wi-Fi®, Bluetooth®, USB, and HDMI®.

1.2 Configuration of Object Recognition Device The object recognition device 200 is an information processing device such as a PC or various information terminals. The object recognition device 200 includes an input / output interface unit 21, an arithmetic processing unit 22, and a storage unit 23.

The input / output interface unit 21 includes, for example, a device interface and a network interface. The device interface is a circuit (module) for connecting an external device such as a distance sensor 100 to the object recognition device 200. The device interface is an example of an acquisition unit that performs communication according to a predetermined communication standard. Predetermined standards include USB, HDMI®, IEEE1395, Wi-Fi®, Bluetooth® and the like. The network interface is a circuit (module) for connecting the object recognition device 200 to the communication network via a wireless or wired communication line. The network interface is an example of an acquisition unit that performs communication conforming to a predetermined communication standard. Predetermined communication standards include communication standards such as IEEE802.3 and IEEE802.11a / 11b / 11g / 11ac. The input / output interface unit 21 is an example of an input unit that inputs three-dimensional information including a three-dimensional position along the outer shape of at least a part of an object.

The arithmetic processing unit 22 includes a CPU and a GPU that cooperate with software to realize a predetermined function, and controls the overall operation of the object recognition device 200. The arithmetic processing unit 22 reads data and programs stored in the storage unit 23 and performs various arithmetic processing to realize various functions. The function of the arithmetic processing unit 22 may be configured only by hardware, or may be realized by combining hardware and software. The arithmetic processing unit 22 executes a program for constructing a convolutional neural network, which will be described later. The program is stored in the storage unit 23. The program for constructing the convolutional neural network may be provided from various communication networks, or may be stored in a portable recording medium.

The arithmetic processing unit 22 may be a hardware circuit such as a dedicated electronic circuit or a reconfigurable electronic circuit designed to realize a predetermined function. In addition to the CPU and GPU, the arithmetic processing unit 22 may be composed of various semiconductor integrated circuits such as MPU, GPGPU, TPU, microcomputer, DSP, FPGA, and ASIC.

The arithmetic processing unit 22 includes an area detection unit 22a, an occupied grid generation unit 22b, a 2D drawing generation unit 22c, and an object recognition unit 22d as functional configurations. The area detection unit 22a detects an area in which one object exists in the distance image as an object recognition target area. The occupancy grid generation unit 22b generates occupancy grid data indicating the occupancy grid in which the detected object in the recognition target area is represented by an aggregate of voxels. The 2D drawing generation unit 22c generates a two-dimensional drawing of the voxel aggregate based on the occupied grid data. In the present embodiment, the 2D drawing generation unit 22c generates a three-view drawing. The three-view view is an example of a plurality of two-dimensional information. The object recognition unit 22d executes image processing by a convolutional neural network based on the three views to recognize the type of the object.

The storage unit 23 stores parameters, data, a control program, and the like necessary for realizing a predetermined function. For example, the storage unit 23 stores a program for a convolutional neural network, learning and learned parameters, and the like. The storage unit 23 can be realized by, for example, a hard disk (HDD), SSD, RAM, DRAM, SRAM, ferroelectric memory, flash memory, magnetic disk, or a combination thereof. The storage unit 23 may temporarily store various types of information. The storage unit 23 may be configured to function as, for example, a work area of the arithmetic processing unit 22.

The object recognition device 200 may include an operation unit that is a user interface for the user to operate. The operation unit is composed of, for example, a keyboard, a touch pad, a touch panel, buttons, switches, and a combination thereof. The object recognition device 200 may include a display unit composed of a liquid crystal display or an organic EL display. The object recognition device 200 may include a speaker that outputs voice.

2. Operation 2.1 Object recognition processing The operation related to the object recognition processing of the object recognition system 1 configured as described above will be described with reference to FIGS. 4 to 10.

FIG. 4 is a flowchart illustrating the operation of the arithmetic processing unit 22 of the object recognition device 200.

The area detection unit 22a acquires the sensing data generated by the distance sensor 100 via the input / output interface unit 21 (S101). In the present embodiment, the sensing data is a distance image showing the distance from the reference point 5 to the outer shape of the object. FIG. 5 shows an example of the distance image 30. The distance image 30 shows the distance in the depth direction Z for each pixel arranged in the horizontal direction X and the vertical direction Y. That is, in the distance image 30, each pixel has a pixel value indicating a distance in the depth direction Z. In the distance image 30, for example, the distance in the depth direction Z is identified by the color of each pixel. In one example, the closer the distance, the redder (eg, the pixel value approaches RGB = (255,0,0)), and the farther the distance, the bluer (eg, the pixel value RGB = (0,0,255)). ) Approach). The distance image 30 is an example of three-dimensional information including a three-dimensional position along the outer shape of at least a part of an object.

The region detection unit 22a detects a region in which one object exists as a recognition target region based on the sensing data (S102). The recognition target region can be detected by using a known technique. For example, in the distance image 30, the area detection unit 22a determines that the object is the same when the difference between the pixel value of each pixel and the pixel value of the surrounding eight pixels is equal to or less than a predetermined threshold value, and determines that the recognition target area 35 is the same object. To detect.

The occupied grid generation unit 22b defines a three-dimensional space of a predetermined size (S103). FIG. 6 schematically shows the three-dimensional space 40. The occupied grid generation unit 22b defines, for example, a cubic three-dimensional space 40 composed of 32 × 32 × 32 voxels 41 in a virtual coordinate system (x, y, z).

The occupied grid generation unit 22b generates occupied grid data indicating the occupied grid based on the detected recognition target area 35 and the defined three-dimensional space 40 (S104). FIG. 7 schematically shows an example of the occupied grid 45. The occupied grid 45 is an aggregate of voxels 41 in which the occupied area 46 and the unoccupied area 47 are distinguished in the three-dimensional space 40. The occupied area 46 indicates an area where an object exists, and the unoccupied area 47 indicates an area where an object does not exist. The occupied grid generation unit 22b, for example, transforms the coordinate system in the distance image 30 to associate each pixel in the recognition target area 35 with the voxel 41 in the three-dimensional space 40. The occupied grid generation unit 22b determines which voxel 41 in the three-dimensional space 40 corresponds to each pixel indicating the distance value in the recognition target area 35, and assigns a flag to each voxel 41 to occupy the voxel 41. Occupied grid data indicating grid 45 is generated. The flag is, for example, a binary indicating whether or not it is an object. A solid composed of voxels 41 in the occupied area 46 corresponds to at least a part of an object.

The 2D drawing generation unit 22c generates a three-dimensional view of a solid composed of voxels 41 in the occupied area 46 (S105). For example, the 2D drawing generation unit 22c generates a two-dimensional drawing in which the voxels 41 in the occupied area 46 are viewed from the directions A, B, and C in FIG. 7. FIG. 8A is a plan view 50A viewed from the direction A. FIG. 8B is a front view 50B seen from the direction B. FIG. 8C is a side view 50C seen from the direction C. The plan view 50A, the front view 50B, and the side view 50C are collectively referred to as three views 50A, 50B, and 50C. The 2D drawing generation unit 22c is a pixel corresponding to the distance from the planes 44a, 44b, 44c forming the cube of the three-dimensional space 40 to each voxel in the occupied area 46 in the directions A, B, and C shown in FIG. Generate three views 50A, 50B, 50C with values. In the examples of FIGS. 8A to 8C, the shorter the distance, the lighter the color (for example, the pixel value approaches 255), and the farther the distance, the darker the color (for example, the pixel value approaches 0). The closer the distance, the darker the color, and the farther the distance, the lighter the color. In FIGS. 8A to 8C, the pixels in which no object exists are shown in white, but may be shown in black. Further, the three views 50A, 50B, and 50C have pixel values represented by, for example, RGB. In one example, the plan view 50A is red, the front view 50B is green, and the side view 50C is blue.

The 2D drawing generation unit 22c synthesizes the three views 50A, 50B, and 50C to generate one composite drawing (S106). FIG. 9 shows an example of the composite diagram 50D generated from the three views 50A, 50B, and 50C. The 2D drawing generation unit 22c determines the pixel value of each pixel in the composite drawing 50 based on the pixel value of each pixel in the three views 50A, 50B, and 50C. Therefore, for example, in the composite drawing 50, the shorter the distance, the lighter the color, and the farther the distance, the darker the color. Further, the color of each pixel in the composite view 50 is a color based on the red, green, and blue in the plan view 50A, the front view 50B, and the side view 50C.

The object recognition unit 22d inputs the composite diagram 50D into the trained convolutional neural network and recognizes the type of the object (S107). FIG. 10 is a diagram for explaining image processing by the convolutional neural network 60. The convolutional neural network 60 has been learned in advance to recognize an object by using a composite diagram showing an object such as a car, a bus, a pedestrian, a utility pole, or a curb. The learning method of the convolutional neural network 60 will be described later. The program and parameters for constructing the learned convolutional neural network 60 are stored in, for example, the storage unit 23. The object recognition unit 22d calculates the probability of being a car, a bus, a pedestrian, a utility pole, a curb, a guardrail, etc. from the composite diagram 50D by executing image processing using the convolutional neural network 60. The convolutional neural network 60 includes, for example, convolutional layers L1 and L2, fully connected layers L3 and L4, and an output layer L5 in order from the input side to the output side. In the example of FIG. 10, the number of convolution layers L1 and L2 and the number of fully connected layers L3 and L4 are two, respectively, but the number of layers is not limited to two. Further, there may be a pooling layer after the convolution layers L1 and L2.

The composite diagram 50D is input to the convolution layer L1 of the first layer. In each convolution layer L1 and L2, a convolution operation using each filter is performed. The filters of the convolution layers L1 and L2 are defined by a two-dimensional array of weighting coefficients. The object recognition result is output from the output layer L5. For example, a vector indicating the probability that an object is a car, a bus, a pedestrian, a utility pole, a curb, a guardrail, or the like is output.

The object recognition unit 22d outputs the recognition result (S108). For example, the object recognition unit 22d identifies the object having the highest probability among the probabilities output from the output layer L5 as the object shown in the composite diagram 50D, and identifies the object via the input / output interface unit 21. The object information indicating the type of the object is output to the vehicle driving device 2. When the object recognition device 200 includes a display unit, the type of the object as the recognition result may be displayed on the screen of the display unit. When the object recognition device 200 includes a speaker, the type of the object as the recognition result may be output by voice from the speaker.

2.2 Learning process FIG. 11 shows the learning process of the convolutional neural network 60. For example, the arithmetic processing unit 22 performs the learning process shown in FIG. 11 to learn the convolutional neural network 60 before executing the object recognition process.

The arithmetic processing unit 22 acquires the learning three-view drawing and the data indicating the correct answer label corresponding to the three-view drawing (S201). For example, the arithmetic processing unit 22 acquires learning data indicating the three views and the correct answer labels corresponding to the three views in advance via the input / output interface unit 21 and stores them in the storage unit 23. In step S201, the arithmetic processing unit 22 reads the learning data from the storage unit 23. Correct labels are, for example, cars, buses, pedestrians, utility poles, curbs, and guardrails.

The arithmetic processing unit 22 synthesizes the three views to generate a composite drawing (S202). The arithmetic processing unit 22 inputs the composite diagram into the convolutional neural network 60 and recognizes the type of the object (S203).

The arithmetic processing unit 22 adjusts the parameters of the convolutional neural network 60 based on the recognition result and the correct answer label (S204). For example, the arithmetic processing unit 22 adjusts the weighting coefficient of the filter of the convolution layers L1 and L2 and the weighting coefficient between the neurons of the fully connected layers L3 and L4 according to the backpropagation method.

The arithmetic processing unit 22 determines whether or not the predetermined number of learnings has been completed (S205). Steps S201 to S204 are repeated until a predetermined number of times of learning is completed. When the learning of a predetermined number of times is completed, the learning process shown in FIG. 11 is completed. The arithmetic processing unit 22 stores the programs and parameters corresponding to the learned convolutional neural network 60 in the storage unit 23.

In the example of FIG. 11, the three views were acquired in step S201 and the composite drawing was generated in step S202, but the composite drawing may be acquired in step S201. In this case, step S202 is omitted.

2.3 Comparison of the number of elements in the three-view drawing and the VoxNet FIG. 12 is a diagram comparing the number of elements used for object recognition, and is a diagram comparing the number of elements (the number of voxels) in the conventional VoxNet and the three-view drawing of the present disclosure. Indicates the number of elements (number of pixels) in. When there are 32 elements on one side, the number of elements of VoxNet is 32 × 32 × 32 = 32768, and 32 × 32 × 3 = 3072 in the three-view view. Therefore, the number of elements in the three views is about 1/10 of the number of elements in VoxNet. When there are 64 elements on one side, the number of elements of VoxNet is 64 × 64 × 64 = 262144, and in the three-view drawing, it is 64 × 64 × 3 = 12288. Therefore, the number of elements in the three views is about 1/20 of the number of elements in VoxNet. When there are 128 voxels on one side, the number of VoxNet elements is 2097152, which is 49152 in the three-view drawing. Therefore, the number of elements in the three-view drawing is about 1/40 of the number of elements in VoxNet. As described above, since the number of elements is reduced by forming the three-view drawing, the processing load of the image processing by the two-dimensional CNN using the image of the three-view drawing is reduced as compared with the image processing by the three-dimensional CNN. Therefore, the processing speed of object recognition is improved.

In the measurement of the distance in the direction from the reference point 5 to the projection surface R10 by the distance sensor 100, only the front side of the object (the side where the reference point 5 is located on the projection surface R10) can be measured, and the back side of the object (the reference point on the projection surface R10) can be measured. The distance on the opposite side of 5) cannot be measured. That is, the occupied area 46 does not correspond to the entire object. Therefore, even if the occupied grid 45 is converted into a three-view drawing, the amount of information on the distance value of the object is not significantly reduced, and the object can be recognized with high accuracy.

3. 3. Summary The object recognition system 1 according to the present embodiment includes a distance sensor 100 and an object recognition device 200. The distance sensor 100 measures the distance to the object and generates sensing data including a three-dimensional position along the outer shape of at least a part of the object. The object recognition device 200 according to the present embodiment includes an input / output interface unit 21 and an arithmetic processing unit 22. The input / output interface unit 21 inputs sensing data. The arithmetic processing unit 22 recognizes the object based on the sensing data. Specifically, the arithmetic processing unit 22 generates a plurality of two-dimensional drawings of a solid represented by a three-dimensional position viewed from a plurality of directions based on the sensing data, and an object based on the plurality of two-dimensional drawings. Recognize.

Since the two-dimensional drawing is used for object recognition, the amount of data used for object recognition is reduced. Therefore, the processing load of image processing is reduced, and the processing speed of object recognition is improved.

The arithmetic processing unit 22 executes image processing by the convolutional neural network 60 based on the generated two-dimensional drawings to recognize the object. A two-dimensional convolutional neural network can reduce the scale of the network as compared with a three-dimensional convolutional neural network. For example, the number of layers, the number of neurons, and the like can be reduced.

In the present embodiment, the sensing data is a distance image showing the distance from the reference point 5 to the outer shape of the object. The arithmetic processing unit 22 generates an aggregate of voxels representing a solid based on the distance image. Specifically, the arithmetic processing unit 22 defines a three-dimensional space 40 that can be divided into a plurality of voxels, associates a three-dimensional position along the outer shape of the object with the three-dimensional space, and objects among the plurality of voxels. Represents a solid by the voxels occupied by. This solid corresponds to a voxel in the occupied area 46.

In the present embodiment, the sensing data includes a three-dimensional position along the outer shape of a part of the object, and the plurality of two-dimensional drawings are three-view views of the solid viewed from three orthogonal axes. The three-view drawing has pixel values according to the distance from the planes 44a, 44b, 44c, which are the reference positions, to the solid. The arithmetic processing unit 22 synthesizes the three views to generate a composite drawing, and recognizes the object based on the composite drawing. Since the three-view drawing has about the same amount of information about the distance as the three-dimensional information, the object recognition using the three-view drawing can obtain the same degree of accuracy as the object recognition using the three-dimensional voxel.

(Embodiment 2)
In the first embodiment, the object recognition device 200 recognizes an object by using a three-view drawing generated based on the distance measurement in the projection surface R10 by one distance sensor 100. In this embodiment, object recognition is performed using a hexagonal view. The hexagonal view is an example of a plurality of two-dimensional information.

FIG. 13 is a flowchart showing an example of the object recognition process by the object recognition device 200 according to the second embodiment. Steps S302 to S304, S307, and S308 of FIG. 13 are the same as steps S102 to S104, S107, and S108 of FIG. 4 of the first embodiment.

In the present embodiment, the area detection unit 22a acquires a plurality of sensing data via the input / output interface unit 21 (S301). Each sensing data includes a three-dimensional position along the outer shape of a part of the object, and a plurality of sensing data includes a three-dimensional position along the entire outer shape of the object. The plurality of sensing data are, for example, distance images generated based on distance measurement from a plurality of reference points 5. The plurality of reference points 5 are provided at positions facing each other, for example. In one example, the area detection unit 22a acquires sensing data from a plurality of distance sensors arranged at different positions. In another example, the area detection unit 22a may acquire a plurality of sensing data generated by one distance sensor measuring the distance at different positions.

The region detection unit 22a detects a region in which an object exists as a recognition target region in each distance image (S302). The occupied grid generation unit 22b defines a three-dimensional space 40 (S303). The occupied grid generation unit 22b converts the local coordinates of each recognition target area into world coordinates to generate occupied grid data indicating one occupied grid 45 (S304). FIG. 14 is a diagram for explaining the occupied grid 45 in the second embodiment. The occupied grid 45 is an aggregate of voxels 41 in which the occupied area 46 and the unoccupied area 47 are distinguished in the three-dimensional space 40, as in the first embodiment.

The 2D drawing generation unit 22c generates a six-view drawing of a solid composed of voxels 41 in the occupied area 46 (S305). FIG. 15 shows a six-view view of the voxels 41 in the occupied area 46 from the directions of arrows a to f in FIG. Specifically, FIG. 15A is a front view seen from the direction of arrow a in FIG. FIG. 15B is a rear view seen from the direction of arrow b in FIG. FIG. 15 (c) is a left side view seen from the direction of arrow c in FIG. FIG. 15D is a right side view seen from the direction of arrow d in FIG. FIG. 15 (e) is a plan view (top view) seen from the direction of arrow e in FIG. FIG. 15 (f) is a bottom view seen from the direction of arrow f in FIG. The six views shown in FIGS. 15A to 15F have pixel values corresponding to the distances from each plane of the cube showing the three-dimensional space 40 to each voxel in the occupied area 46. In the examples of FIGS. 15A to 15F, the shorter the distance, the lighter the color (for example, the pixel value approaches 255), and the farther the distance, the darker the color (for example, the pixel value approaches 0). I am trying to do it. However, the closer the distance, the darker the color, and the farther the distance, the lighter the color. In FIGS. 15A to 15F, the pixels in which no object exists are shown in white, but may be shown in black.

The 2D drawing generation unit 22c synthesizes the six views to generate one composite drawing (S306). The object recognition unit 22d inputs the composite diagram into the trained convolutional neural network 60 and recognizes the type of the object (S307). The object recognition unit 22d outputs the recognition result (S308).

As described above, in the present embodiment, the plurality of sensing data include the three-dimensional position along the entire outer shape of the object. The object recognition device 200 generates a hexagonal view of a solid composed of voxels in the occupied area 46 as viewed from the positive and negative directions of three orthogonal axes. The object recognition device 200 recognizes an object using a hexagonal view. As a result, object recognition can be performed based on the entire outer shape of the object. Therefore, the object can be recognized with high accuracy.

(Other embodiments)
In the above embodiment, an example in which the sensing data is a distance image and the object recognition device 200 generates the occupied grid 45 based on the distance image has been described. However, the sensing data is not limited to the distance image. The sensing data may include a three-dimensional position along the outer shape of at least a part of the object. For example, the sensing data may be three-dimensional point cloud information indicating the distance to the outer shape of the object measured by the distance sensor 100. Such three-dimensional point cloud information includes, for example, x-coordinate, y-coordinate, and z-coordinate.

In the above embodiment, the 2D drawing generation unit 22c generates a three-dimensional view or a six-view view composed of the voxels 41 in the occupied area 46, but if the number of two-dimensional drawings to be generated is two or more. Good.

In the above embodiment, an example in which the arithmetic processing unit 22 of the object recognition device 200 performs the learning process shown in FIG. 11 has been described, but the learning process of the convolutional neural network 60 is performed by a device different from the object recognition device 200. You may. For example, the convolutional neural network 60 may be constructed by computer cluster or cloud computing.

In the above embodiment, the example in which the distance sensor 100 and the object recognition device 200 are mounted on the vehicle 3 has been described, but the distance sensor 100 and the object recognition device 200 may be mounted on a self-propelled robot, an AGV (Automated Guided Vehicle), or the like, not limited to the vehicle 3. Further, the object recognition device 200 does not have to be mounted on the vehicle 3 or the like. The object recognition device 200 according to the present disclosure may be various information processing devices. For example, the object recognition device 200 may be one or more server devices such as an ASP server. For example, the object recognition device 200 may acquire sensing data from the distance sensor 100 via a communication network and execute image processing by the convolutional neural network 60. Further, the object recognition device 200 may transmit information indicating the recognition result of the object to the vehicle driving device 2 via the communication network.

In the above embodiment, the distance sensor 100 and the object recognition device 200 are separate devices, but the distance sensor 100 and the object recognition device 200 may be one device. For example, the object recognition device 200 may be provided inside the distance sensor 100, and the distance sensor 100 may have the same function as the object recognition device 200.

In the above embodiment, an example in which the object recognition unit 22d recognizes the type of the object has been described, but the recognition operation is not limited to identifying the type of the object. Recognition includes extracting features of an object. For example, when the object is a car, the recognition performed by the object recognition unit 22d includes extracting "rectangular parallelepiped" and "wheel" as feature quantities of the car.

(Additional note)
As described above, the various embodiments of the present disclosure have been described, but the present disclosure is not limited to the above contents, and various changes can be made within the scope of substantially the same technical idea. .. Hereinafter, various aspects relating to the present disclosure will be added.

The object recognition device of the first aspect according to the present disclosure includes an input unit (21) for inputting three-dimensional information including a three-dimensional position along the outer shape of at least a part of the object, and the object recognition device based on the three-dimensional information. The arithmetic processing unit (22) includes an arithmetic processing unit (22) for recognizing an object, and the arithmetic processing unit (22) has a two-dimensional view of a solid represented by the three-dimensional position from a plurality of directions based on the three-dimensional information. A plurality of two-dimensional information showing a drawing is generated, and the object is recognized based on the plurality of two-dimensional information.

In the second aspect, in the object recognition device of the first aspect, the arithmetic processing unit executes image processing by the convolutional neural network based on the generated two-dimensional information to recognize the object.

In the third aspect, in the object recognition device of the first aspect or the second aspect, the three-dimensional information is a distance image showing a distance from a reference point to the outer shape of at least a part of the object.

In the fourth aspect, in the object recognition device according to any one of the first to third aspects, the arithmetic processing unit generates an aggregate of voxels representing the solid based on the three-dimensional information.

In the fifth aspect, in the object recognition device of the fourth aspect, the arithmetic processing unit defines a three-dimensional space that can be divided into a plurality of voxels, and associates the three-dimensional position with the three-dimensional space. The solid is represented by the voxels occupied by the object among the plurality of voxels.

In the sixth aspect, in the object recognition device according to any one of the first to fifth aspects, the plurality of two-dimensional information has pixel values corresponding to the distance from each reference position to the solid.

In the seventh aspect, in the object recognition device according to any one of the first to sixth aspects, the three-dimensional information includes a three-dimensional position along the outer shape of a part of the object, and the plurality of two. The dimensional information is a three-dimensional view of the solid viewed from each of the three orthogonal axial directions.

In the eighth aspect, in the object recognition device according to any one of the first to sixth aspects, the three-dimensional information includes a three-dimensional position along the entire outer shape of the object, and the plurality of two-dimensional information. Is a hexagonal view of the solid viewed from each of the positive and negative directions of the three orthogonal axes.

In the ninth aspect, in the object recognition device according to any one of the first to eighth aspects, the arithmetic processing unit synthesizes the plurality of two-dimensional information to generate a composite diagram, and the composite diagram is formed. Recognize the object based on.

The object recognition system according to the present disclosure includes a sensor that measures a distance to an object and generates the three-dimensional information, and the object recognition device according to any one of the first to ninth aspects. including.

The program according to the present disclosure includes a step of inputting three-dimensional information including a three-dimensional position along the outer shape of at least a part of an object, and a plurality of solids represented by the three-dimensional position based on the three-dimensional information. A computer is made to execute a step of generating a plurality of two-dimensional information showing a two-dimensional drawing viewed from the above-mentioned direction and a step of recognizing the object based on the plurality of two-dimensional information.

The object recognition device and the object recognition system of the present disclosure can be applied to, for example, an automatic guided vehicle, a self-propelled robot, an AGV, and the like.

1 Object recognition system 2 Vehicle drive device 3 Vehicle 11 Floodlight unit 12 Light receiving unit 13 Scanning unit 14 Sensor control unit 15, 21 Input / output interface unit 22 Arithmetic processing unit 22a Area detection unit 22b Occupied grid generation unit 22c 2D drawing generation unit 22d Object recognition unit 23 Storage unit 100 Distance sensor 200 Object recognition device

Claims (11)

An input unit for inputting 3D information including a 3D position along the outer shape of at least a part of an object.
An arithmetic processing unit that recognizes the object based on the three-dimensional information,
With
The arithmetic processing unit
Based on the three-dimensional information, a plurality of two-dimensional information showing a two-dimensional drawing of the solid represented by the three-dimensional position viewed from a plurality of directions is generated.
Recognizing the object based on the plurality of two-dimensional information.
Object recognition device. The arithmetic processing unit executes image processing by a convolutional neural network based on the generated two-dimensional information, and recognizes the object.
The object recognition device according to claim 1. The three-dimensional information is a distance image showing a distance from a reference point to the outer shape of at least a part of the object.
The object recognition device according to claim 1 or 2. The arithmetic processing unit generates an aggregate of voxels representing the solid based on the three-dimensional information.
The object recognition device according to any one of claims 1 to 3. The arithmetic processing unit defines a three-dimensional space that can be divided into a plurality of voxels, associates the three-dimensional position with the three-dimensional space, and uses the voxels occupied by the object among the plurality of voxels. Represents a solid
The object recognition device according to claim 4. The plurality of two-dimensional information has pixel values according to the distance from each reference position to the solid.
The object recognition device according to any one of claims 1 to 5. The three-dimensional information includes a three-dimensional position along the outer shape of a part of the object.
The plurality of two-dimensional information is a three-view view of the solid viewed from each of the three orthogonal axes.
The object recognition device according to any one of claims 1 to 6. The three-dimensional information includes a three-dimensional position along the entire outer shape of the object.
The plurality of two-dimensional information is a six-view view of the solid viewed from each of the positive and negative directions of the three orthogonal axes.
The object recognition device according to any one of claims 1 to 6. The arithmetic processing unit synthesizes the plurality of two-dimensional information to generate a composite diagram, and recognizes the object based on the composite diagram.
The object recognition device according to any one of claims 1 to 8. A sensor that measures the distance to an object and generates the three-dimensional information,
The object recognition device according to any one of claims 1 to 9.
Object recognition system, including. The step of inputting 3D information including the 3D position along the outer shape of at least a part of the object,
Based on the three-dimensional information, a step of generating a plurality of two-dimensional information when the solid represented by the three-dimensional position is viewed from a plurality of directions, and
A step of recognizing the object based on the plurality of two-dimensional information,
A program that causes a computer to run.

Images