JP2021047120A - Recognition device, control system, recognition method and program - Google Patents

Abstract

To provide a recognition device, a control system, a recognition method and a program, capable of generating an image that is easily used for recognition processing of a moving body based on a detection result of a detection unit.SOLUTION: A recognition device is mounted on a moving body, and comprises: a detection unit that emits electromagnetic waves at different angles with the recognition device as an emission center in vertical and horizontal directions and detects the reflected waves, to detect a distance; and a generation unit that generates a two-dimensional image in which pixels are assigned to an emission direction of each electromagnetic wave with the horizontal direction from the moving body as a vertical reference, based on the detection result of the detection unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a recognition device, a control system, a recognition method, and a program.

Conventionally, a technique for generating a bird's-eye view of the recognition range of a detection unit from information based on a detection result in which a detection unit such as a distance sensor detects the surroundings has been disclosed (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-194729

However, with the conventional technology, it has been difficult to generate an image other than a bird's-eye view.

The present invention has been made in consideration of such circumstances, and is a recognition device, a control system, and a recognition capable of generating an image that can be easily used for recognition processing of a moving object based on the detection result of the detection unit. One of the purposes is to provide methods and programs.

The recognition device, control system, recognition method, and program according to the present invention have the following configurations.
(1) The recognition device according to one aspect of the present invention is a recognition device mounted on a moving body, and detects reflected waves by irradiating electromagnetic waves at different angles with the recognition device as an emission center in the vertical direction and the horizontal direction. A detection unit that detects the distance by doing so, and a generation unit that generates a two-dimensional image in which pixels are assigned to each electromagnetic wave irradiation direction with the horizontal direction from the moving body as a vertical reference based on the detection result of the detection unit. And.

The aspect (2) is that in the recognition device according to the aspect (1) above, the detection unit is a rider.

The aspect (3) is the recognition device according to the aspect (2), wherein the generation unit uses the distance obtained from the detection result of the rider and the intensity of the reflected wave as data elements of the pixel. It produces an image.

In the aspect (4), the recognition device according to the above aspect (3) further includes a recognition unit that performs image recognition processing on the two-dimensional image and recognizes a target in the two-dimensional image. is there.

In the recognition device according to the aspect (4), the recognition unit performs the image recognition process using the trained image recognition model, and the trained image recognition model is a moving body. Machine learning is performed using the peripheral image of the above, and the target in the two-dimensional image and the distance to the target are output.

(6) The control system of another aspect of the present invention includes a recognition device according to any one of the above (1) to (5) and a control device that controls the moving body based on the two-dimensional image. , Is provided.

(7) In the recognition method of another aspect of the present invention, the computer of the recognition device mounted on the moving body irradiates electromagnetic waves at different angles with the recognition device as the emission center in the vertical direction and the horizontal direction to emit reflected waves. By detecting the distance, the distance is detected, and based on the detection result, a two-dimensional image in which pixels are assigned to each irradiation direction of the electromagnetic wave is generated with the horizontal direction from the moving body as a vertical reference.

(8) In the program of another aspect of the present invention, the computer of the recognition device mounted on the moving body is irradiated with electromagnetic waves at different angles with the recognition device as the emission center in the vertical direction and the horizontal direction, and the reflected wave is detected. By doing so, the distance is detected, and based on the detection result, a two-dimensional image in which pixels are assigned to each irradiation direction of the electromagnetic wave is generated with the horizontal direction from the moving body as a vertical reference.

According to (1) to (8), it is possible to generate an image that is easy to be used for the recognition process of the moving object based on the recognition result of the recognition unit.

According to (6), the vehicle can be controlled based on the image.

It is a figure which shows the structure of the vehicle system 1 which uses the recognition device. It is a figure used for the explanation of a rider 314. It is a figure which shows an example of the structure of the object recognition device 316. It is a figure which shows an example of a camera image IMC. It is a figure which shows typically the contents of the data set of a rider 314. It is a figure used for the explanation of the process of generating the 2D image of the image generation part 316B. It is a figure which shows an example of a rider image IML. It is a figure used for the explanation of the recognition process of the recognition unit 430. It is a flowchart which shows an example of the processing flow of the object recognition apparatus 316. It is a flowchart which shows an example of the processing flow of the 1st control unit 420, and the 2nd control unit 460. It is a figure which shows an example of the hardware composition of the automatic operation control device 400 of embodiment.

Hereinafter, embodiments of the recognition device, control system, recognition method, and program of the present invention will be described with reference to the drawings.

<Embodiment>
FIG. 1 is a diagram showing a configuration and the like of a vehicle system 1 that uses a recognition device. The configuration of the vehicle system 1 shown in this figure is mounted on the vehicle M. The vehicle system 1 includes, for example, a camera 310, a radar device 312, a rider (Light Detection and Ranging) 314, an object recognition device 316, a communication device 320, an HMI (Human Machine Interface) 330, and a vehicle sensor 340. , A navigation device 350, an MPU (Map Positioning Unit) 360, a driving operator 380, an automatic driving control device 400, a traveling driving force output device 500, a braking device 510, and a steering device 520. These devices and devices are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like.

The configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted or another configuration may be added. Among the configurations shown in FIG. 1, the object recognition device 316 is an example of the “recognition device”.

The camera 310 is installed at an arbitrary position capable of capturing an image of the periphery of the vehicle M (particularly forward or rear). For example, the camera 310 is installed above the front windshield. The camera 310 is a digital camera including an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and repeatedly images the periphery of the vehicle M at a predetermined cycle.

The radar device 312 radiates radio waves such as millimeter waves around the vehicle M and detects radio waves (reflected waves) reflected by the object to at least detect the position (distance and orientation) of the object. The radar device 312 is attached to an arbitrary position on the vehicle M.

The rider 314 is mounted at a position overlooking any direction of the vehicle M. FIG. 2 is a diagram used to explain the rider 314. In FIG. 2, the rider 314 is attached at a position where the height from the road surface at the front end of the vehicle M is h. The height h is, for example, about several tens [cm].

The rider 314 irradiates an electromagnetic wave to detect a reflected wave, and measures the time from the irradiation to the detection to detect the distance to the contour of the target. The rider 314 can change the irradiation direction of the electromagnetic wave for both the elevation angle or the depression angle (the irradiation direction in the vertical direction) and the azimuth angle (the irradiation direction in the horizontal direction). For example, the rider 314 scans while fixing the irradiation direction in the vertical direction and changing the irradiation direction in the horizontal direction, then changes the irradiation direction in the vertical direction, and fixes the irradiation direction in the vertical direction at the changed angle. The operation of scanning while changing the irradiation direction in the horizontal direction is repeated. Hereinafter, the vertical irradiation direction is referred to as a "layer", and one scan performed while fixing the layer and changing the horizontal irradiation direction is referred to as a "cycle", and is the horizontal irradiation direction. Is called "azimuth". The layers are set to, for example, L1 to Li with a finite number (i is a natural number). The layer change is performed discontinuously with respect to the angle, for example, L0 → L4 → L2 → L5 → L1 ... So that the electromagnetic wave irradiated in the previous cycle does not interfere with the detection in this cycle. Not limited to this, the layer may be changed continuously with respect to the angle.

The rider 314 is provided with a plurality of irradiation units for irradiating electromagnetic waves, and when a plurality of irradiation units are arranged in the vertical direction, each irradiation unit is of each layer without changing the irradiation direction in the vertical direction. It may be the one that irradiates. Further, in the rider 314, when a plurality of irradiation units are arranged in the horizontal direction, each irradiation unit may irradiate each azimuth without changing the irradiation direction in the horizontal direction.

The rider 314 outputs a data set including, for example, {layer, azimuth, distance, intensity of detected reflected wave} to the object recognition device 316. The object recognition device 316 is installed at an arbitrary position in the vehicle M. Hereinafter, as shown in FIG. 2, it is assumed that the object recognition device 316 is installed on the upper part of the vehicle M.

Returning to FIG. 1, the object recognition device 316 performs sensor fusion processing on the detection results of a part or all of the camera 310, the radar device 312, and the rider 314 to determine the position, type, speed, and the like of the target. recognize. FIG. 3 is a diagram showing an example of the configuration of the object recognition device 316. The object recognition device 316 includes, for example, an acquisition unit 316A, an image generation unit 316B, and an output unit 316C. Each of these functional units is realized by executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Some or all of these components are hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), etc. It may be realized by (including circuits), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage device such as a DVD or a CD-ROM. It is stored in a medium, and the storage medium (non-transient storage medium) may be attached to the drive device and installed in the HDD or flash memory of the object recognition device 316.

The acquisition unit 316A acquires the image generated by the camera 310, acquires the detection result of the radar device 312, and acquires the data set from the rider 314. FIG. 4 is a diagram showing an example of an image generated by the camera 310 (hereinafter, camera image IMC). In the example of FIG. 4, the camera 310 is attached to the upper part of the front windshield. In the camera image IMC of FIG. 4, the range AR1 and the range AR2 show other vehicles existing around the vehicle M, respectively, and the range AR3 shows a pedestrian crossing.

FIG. 5 is a diagram schematically showing the contents of the data set of the rider 314. FIG. 5 shows a three-dimensional reflection point distribution recognized from the output of the rider 314 when the vehicle M is present at the same position as in FIG. In FIG. 5, the difference in the color of the reflection point indicates the difference in reflectance (that is, the difference in the intensity of the reflected wave). Here, in terms of the intensity of the reflected wave of the road marking drawn or buried on the road and the intensity of the reflected wave on the road surface, the road marking may have a higher intensity of the reflected wave. This is because the road markings are drawn on the road surface in the form of solid lines or broken lines of white lines and yellow lines. Road markings are, for example, road markings buried in roads in the manner of Bot's Dots or Cat's Eye, legal speeds drawn on the road surface, pedestrian crossings, pedestrian crossing notices, and the like. In FIG. 5, the data set in the real space range corresponding to the range AR1 to AR2 of the camera image IMC shown in FIG. 4 includes data indicating another vehicle and data of a point cloud having a high reflected wave intensity. , The data set of the real space range corresponding to the range AR2 includes the data indicating the white line of the crosswalk and the data of the point cloud having a high intensity of the reflected wave.

Returning to FIG. 2, the image generation unit 316B generates a two-dimensional image in which pixels are assigned for each electromagnetic wave irradiation direction based on the data set acquired by the acquisition unit 316A. Details of the process by which the image generation unit 316B generates a two-dimensional image will be described later. The output unit 316C outputs the image generated by the camera 310, the detection result of the radar device 312, the data set acquired from the rider 314, and the two-dimensional image generated by the image generation unit 316B to the automatic operation control device 400.

The image generation unit 316B generates a two-dimensional image in which pixels are assigned for each electromagnetic wave irradiation direction based on the data set acquired by the acquisition unit 316A. FIG. 6 is a diagram used for explaining a process of generating a two-dimensional image of the image generation unit 316B. Hereinafter, the two-dimensional image generated by the image generation unit 316B will be referred to as a rider image IML. First, the image generation unit 316B is a data set of a part or the whole range (a predetermined range θ1 shown in the drawing) of the data set acquired by the acquisition unit 316A, and is a part in the horizontal direction. , Or the data set of the entire range (predetermined range θ2 shown) is extracted. The predetermined range θ1 and the predetermined range θ2 are, for example, ranges that match the angle of view of the camera image IMC.

Next, the image generation unit 316B associates the extracted data sets with the data elements of the pixels constituting the rider image IML. Hereinafter, it is assumed that the data element of the rider image IML is the pixel value of the pixels constituting the rider image IML. Further, hereinafter, when a specific data set is shown among the data sets acquired by the acquisition unit 316A, it is described as the data set Dxy. Further, when a specific pixel is shown among the pixels constituting the rider image IML, it is described as pixel Pxy. The subscripts x and y are natural numbers, x indicates a layer, and y indicates an azimuth. In FIG. 6, the data set extracted by the image generation unit 316B is a data set related to m [pieces] of layers [1] to [m], and n [n [1] to [n] of azimus [1] to [n]. This is a data set related to hydrangea. The image generation unit 316B associates the extracted data sets with the data sets D11, D12, ..., Dmn as the pixels P11, P12, ..., Pmn of the camera image IMC, for example.

Here, the image generation unit 316B converts the data set in association with the rider image IML so that the layer change direction and the azimuth change direction correspond to the arrangement order of the pixels P. For example, in FIG. 6, since the layer change direction is from top to bottom and the azimuth change direction is from right to left with respect to the rider 314, the image generation unit 316B has pixels P11, P21, ..., Pm1. The camera image IMC is converted so that the arrangement is from top to bottom and the arrangement of pixels P11, P12, ..., P1n is from right to left.

The conventional so-called Lider image is formed as a spatial image as seen from a camera. As a result, in the conventional so-called lidar image, the portion where the lidar does not irradiate the electromagnetic wave occupies a part of the image. On the other hand, since the image generation unit 316B uses only the reflection points corresponding to the layers as image elements (pixels), the processing load is higher than the conventional processing related to the generation of the so-called Layer image when generating the rider image IML. Can be reduced.

In the image generation unit 316B, when the detection range of the rider 314 and the imaging range of the camera 310 are different from each other and the angle of view of the camera image IMC and the angle of view of the rider image IML do not match, A part or all of the rider image IML may be image-converted (for example, enlarged, reduced, etc.) and adjusted so that the angles of view match. Further, in the example shown in FIG. 6, it has been described that the pixels P of the rider image IML are all the same, but the present invention is not limited to this. The image generation unit 316B may change the size of the pixel P of the rider image IML to a different size according to, for example, the detection position of the data set and the distortion caused by the association of the pixels of the rider image IML.

FIG. 7 is a diagram showing an example of a rider image IML. For example, the image generation unit 316B shows the distance to the target reflecting the electromagnetic wave as the blue pixel value (gradation) of the pixel P, and the intensity of the reflected wave as the green pixel value (gradation) of the pixel P. To convert the rider image IML. In FIG. 7, the blue region is indicated by the dot region and the green region is indicated by the shaded region. First, the image generation unit 316B sets the distance from the upper limit value to the lower limit value from 0 to a predetermined gradation (for example, 0 to 255) based on the distance to the target reflecting the electromagnetic wave included in the data set. Convert. In this case, the object recognition device 316 sets a distance shorter than the lower limit value as the maximum value of gradation (“255” in this case) and a distance longer than the upper limit value as the minimum value of gradation (“0” in this case). .. As a result, the image generation unit 316B converts the rider image IML so that the distance to the target that reflects the electromagnetic wave is indicated by the shade of blue, and the target closer to the vehicle M becomes clearer blue. Further, the image generation unit 316B converts the intensity of the reflected wave from the upper limit value to the lower limit value from a predetermined gradation to 0 based on the intensity of the reflected wave included in the data set. In this case, the image generation unit 316B sets the intensity lower than the lower limit value as the minimum value of the gradation (“0” in this case) and the intensity higher than the upper limit value as the maximum value of the gradation (“255” in this case). As a result, the image generation unit 316B converts the rider image IML so that the reflectance of the target that reflects the electromagnetic wave is indicated by the shade of green, and the higher the reflection intensity of the target, the clearer the green.

The color mode associated with the distance to the target reflecting the electromagnetic wave and the intensity of the reflected wave reflected by the target is not limited to this as an example, and other colors are associated with each other. May be good. Further, the rider image IML may be an image showing either the distance to the target that reflected the electromagnetic wave or the intensity of the reflected wave reflected by the target.

Returning to FIG. 1, the communication device 320 communicates with other vehicles existing in the vicinity of the vehicle M, or communicates with various server devices via a radio base station, for example. The HMI 330 presents various information to the occupant of the vehicle M and accepts input operations by the occupant. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys and the like. The vehicle sensor 340 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, an orientation sensor that detects the direction of the vehicle M, and the like.

The navigation device 350 includes, for example, a GNSS (Global Navigation Satellite System) receiver 351, a navigation HMI 352, and a routing unit 353. The navigation device 350 holds the first map information 354 in a storage device such as an HDD or a flash memory. The GNSS receiver 351 identifies the position of the vehicle M based on the signal received from the GNSS satellite. The position of the vehicle M may be specified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 340. The navigation HMI352 includes a display device, a speaker, a touch panel, keys, and the like. The route determination unit 353 uses, for example, a route from the position of the vehicle M specified by the GNSS receiver 351 (or an arbitrary position input) to the destination input by the occupant using the navigation HMI352 (hereinafter, a map). The upper route) is determined with reference to the first map information 354. The route on the map is output to MPU360. The navigation device 350 may provide route guidance using the navigation HMI352 based on the route on the map. The navigation device 350 may be realized by, for example, the function of a terminal device such as a smartphone or a tablet terminal owned by an occupant. The navigation device 350 may transmit the current position and the destination to the navigation server via the communication device 320, and may acquire a route equivalent to the route on the map from the navigation server.

The MPU 360 includes, for example, a recommended lane determination unit 361, and holds the second map information 362 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 361 divides the route on the map provided by the navigation device 350 into a plurality of blocks (for example, divides the route into 100 [m] units with respect to the vehicle M traveling direction), and refers to the second map information 362. Determine the recommended lane for each block. When a branch point exists on the route on the map, the recommended lane determination unit 361 determines the recommended lane so that the vehicle M can travel on a reasonable route to proceed to the branch destination. The second map information 362 is map information with higher accuracy than the first map information 354.

The driver control 380 includes, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a deformed steering wheel, a joystick and other controls. A sensor for detecting the amount of operation or the presence or absence of operation is attached to the operation operator 380, and the detection result is the automatic operation control device 400, or the traveling driving force output device 500, the brake device 510, and the steering device. It is output to a part or all of 520.

The automatic operation control device 400 includes, for example, a first control unit 420, a second control unit 460, and a storage unit 470. The first control unit 420 and the second control unit 460 are realized by, for example, a hardware processor such as a CPU executing a program (software), respectively. Some or all of these components may be realized by hardware (including circuit parts) such as LSI, ASIC, FPGA, GPU, or may be realized by collaboration between software and hardware. .. The program may be stored in advance in a storage device such as an HDD such as a storage unit 470 or a flash memory (a storage device including a non-transient storage medium), or a removable storage such as a DVD or a CD-ROM. It is stored in a medium, and may be installed in the HDD or flash memory of the automatic operation control device 400 by mounting the storage medium (non-transient storage medium) in the drive device. In addition to the program, the storage unit 470 stores the first trained model 472 and the second trained model 474. Details of the first trained model 472 and the second trained model 474 will be described later.

The first control unit 420 includes, for example, a recognition unit 430 and an action plan generation unit 440. The first control unit 420, for example, realizes a function by AI (Artificial Intelligence) and a function by a model given in advance in parallel. For example, the function of "recognizing an intersection" is executed in parallel with the recognition of an intersection by deep learning or the like and the recognition based on predetermined conditions (there are signals that can be pattern matched, road markings, etc.), and both are executed. It may be realized by scoring and comprehensively evaluating. This ensures the reliability of autonomous driving.

The recognition unit 430 recognizes the position, speed, acceleration, and other states of objects around the vehicle M based on the information input from the camera 310, the radar device 312, and the rider 314 via the object recognition device 316. To do. The position of the object is recognized as, for example, a position on absolute coordinates with the representative point of the vehicle M (center of gravity, center of drive axis, etc.) as the origin, and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by a represented area. The "state" of an object may include acceleration or jerk of the object, or "behavioral state" (eg, whether or not the vehicle is changing lanes or is about to change lanes).

Further, the recognition unit 430 recognizes, for example, the lane (traveling lane) in which the vehicle M is traveling. For example, the recognition unit 430 has a road division line pattern (for example, an arrangement of a solid line and a broken line) obtained from the second map information 362 and a road division around the vehicle M recognized from the camera image IMC generated by the camera 310. By comparing with the line pattern, the traveling lane is recognized. The recognition unit 430 may recognize the traveling lane by recognizing not only the road marking line but also the running road boundary (road boundary) including the road marking line, the shoulder, the curb, the median strip, the guardrail, and the like. .. In this recognition, the position of the vehicle M acquired from the navigation device 350 and the processing result by the INS may be added. The recognition unit 430 also recognizes pause lines, obstacles, red lights, tollhouses, and other road events.

When recognizing the traveling lane, the recognition unit 430 recognizes the position and posture of the vehicle M with respect to the traveling lane. For example, the recognition unit 430 sets the deviation of the reference point of the vehicle M from the center of the lane and the angle formed by the center of the lane in the traveling direction of the vehicle M as the relative position and attitude of the vehicle M with respect to the traveling lane. You may recognize it. Instead, the recognition unit 430 recognizes the position of the reference point of the vehicle M with respect to any side end portion (road division line or road boundary) of the traveling lane as the relative position of the vehicle M with respect to the traveling lane. May be good.

The recognition unit 430 uses the camera image IMC, the rider image IML, and the trained model to determine the position, speed, acceleration, and other states of objects around the vehicle M, and the vehicle M is traveling. The lane (driving lane) may be recognized. FIG. 8 is a diagram used for explaining the recognition process of the recognition unit 430. The recognition unit 430 inputs the camera image IMC acquired from the object recognition device 316 into the first trained model 472, and sets the position, speed, acceleration, and the like of the object around the vehicle M, and the vehicle M travels. Acquire the first result indicating the lane in which the vehicle is driving (driving lane). Further, the recognition unit 430 inputs the rider image IML acquired from the object recognition device 316 into the second trained model 474, and sets the position, speed, acceleration, and the like of the object around the vehicle M, and the vehicle M. Acquires the second result indicating the lane in which the vehicle is traveling (traveling lane).

The first trained model 472 uses, for example, teacher data in which other vehicles and road markings shown in the camera image IMC are tagged (annotated) with respect to the camera image IMC acquired so far. It is a trained model that has been machine-learned. Further, the second trained model 474 is, for example, a model of the learning stage of the first trained model 472 or the first trained model 472 with respect to the rider image IML acquired so far. This is a trained model further machine-learned using teacher data tagged (annotated) with other vehicles and road markings shown in. The second trained model 474 is, for example, an example of a “trained image recognition model”, and the camera image IMC is an example of a “peripheral image of a moving body”.

The second trained model 474 is machine-learned using the teacher data tagged (annotated) with the target and the distance to the target shown in the rider image IML with respect to the rider image IML. It may be a trained model. In this case, the recognition unit 430 recognizes the target by inputting the rider image IML into the second trained model 474 and acquiring the second result.

Hereinafter, the recognition unit 430 performs sensor fusion processing on the first result and the second result, and determines the position of an object around the vehicle M, the state of speed, acceleration, etc., and the lane in which the vehicle M is traveling ( The driving lane) shall be recognized. The recognition unit 430 inputting the rider image IML into the second trained model 474 and acquiring the second result is an example of "performing an image recognition process on a two-dimensional image".

The recognition unit 430 may perform the recognition process using either the first result or the second result depending on the state of the detection result of the camera 310 and the detection result of the rider 314. For example, when the camera 310 cannot properly image the surroundings of the vehicle M due to the weather at the position where the vehicle M exists, the recognition unit 430 uses only the second result or obtains the second result rather than the first result. The recognition process may be performed with emphasis (weighting). Further, when the recognition unit 430 has a target having a high reflectance of electromagnetic waves around the vehicle M and the rider 314 cannot appropriately detect the reflected wave, the recognition unit 430 uses only the first result or the second result. The recognition process may be performed with more emphasis (weighting) on the first result than.

Returning to FIG. 1, the action plan generation unit 440 can, in principle, drive in the recommended lane determined by the recommended lane determination unit 361, and can respond to the surrounding conditions of the vehicle M based on the recognition result of the recognition unit 430. As such, the vehicle M automatically generates a target track for future travel (independent of the driver's operation). The target trajectory includes, for example, a velocity element. For example, the target track is expressed as a sequence of points (track points) to be reached by the vehicle M. The track point is a point to be reached by the vehicle M for each predetermined mileage (for example, about several [m]) along the road, and separately, a predetermined sampling time (for example, about 0 comma [sec]). A target velocity and a target acceleration for each are generated as part of the target trajectory. Further, the track point may be a position to be reached by the vehicle M at the sampling time for each predetermined sampling time. In this case, the information of the target velocity and the target acceleration is expressed by the interval of the orbital points.

The action plan generation unit 440 may set an event for automatic driving when generating a target trajectory. Autonomous driving events include constant speed driving events, low speed following driving events, lane change events, branching events, merging events, takeover events, and the like. The action plan generation unit 440 generates a target trajectory according to the activated event.

The second control unit 460 controls the traveling driving force output device 500, the brake device 510, and the steering device 520 so that the vehicle M passes the target trajectory generated by the action plan generation unit 440 at the scheduled time. To do.

[Operation flow]
FIG. 9 is a flowchart showing an example of the processing flow of the object recognition device 316. First, the acquisition unit 316A acquires the camera image IMC from the camera 310, acquires the detection result from the radar device 312, and acquires the data set from the rider 314 (step S100). The image generation unit 316B converts the data set into a rider image IML based on the data set acquired by the acquisition unit 316A (step S102). The output unit 316C outputs various data acquired by the acquisition unit 316A and the rider image IML to the automatic operation control device 400 (step S104).

FIG. 10 is a flowchart showing an example of the processing flow of the first control unit 420 and the second control unit 460. First, the recognition unit 430 inputs the camera image IMC acquired from the object recognition device 316 into the first trained model 472 and acquires the first result (step S200). Next, the recognition unit 430 inputs the rider image IML acquired from the object recognition device 316 into the second trained model 474, and acquires the second result (step S202). Next, the recognition unit 430 performs sensor fusion processing on the first result and the second result, and determines the position, speed, acceleration, and other states of objects around the vehicle M, and the lane in which the vehicle M is traveling. Recognize (traveling lane) (step S204). Next, the action plan generation unit 440, in principle, travels in the recommended lane determined by the recommended lane determination unit 361, and further, can respond to the surrounding situation of the vehicle M based on the recognition result of the recognition unit 430. , The vehicle M automatically generates a target track to be driven in the future (independent of the driver's operation) (step S206). Next, the second control unit 460 uses the drive system device (that is, the traveling driving force output device 500, so that the vehicle M passes the target trajectory generated by the action plan generation unit 440 at the scheduled time. The brake device 510 and the steering device 520) are controlled (step S208).

[Summary of Embodiment]
As described above, in the vehicle system 1 of the present embodiment, the object recognition device 316 performs the recognition process of the moving body (in this case, the vehicle M) based on the detection result of the detection unit (in this case, the rider 314). The rider image IML that is easy to use can be generated, and the target around the vehicle M can be recognized more accurately than when only the camera image IMC is used.

Here, the process of recognizing the target around the vehicle M based on the values such as {layer, azimuth, distance, detected reflected wave intensity} shown in the data set is performed by using the camera image IMC to recognize the vehicle M. It may be a complicated process compared to the process of recognizing the target around. Further, the process of recognizing the target around the vehicle M using the camera image IMC is easily performed by the trained model (in this case, the first trained model 472) learned by machine learning using the camera image IMC. It may be possible to do it. In order to generate the rider image IML so as to match the angle of view of the camera image IMC, the image generation unit 316B uses the first trained model 472 or the learning stage model of the first trained model 472 as the rider image IML. It can be used as a trained model (in this case, the second trained model 474) used for the process of recognizing the target around the vehicle M, or can be diverted to the training of the second trained model 474. Thereby, the vehicle system 1 of the present embodiment can easily perform the recognition process based on the data set (rider image IML) by using the second trained model 474. Further, the vehicle system 1 of the present embodiment can easily perform the processing related to the learning stage of the second trained model 474 by diverting the first trained model 472.

Further, the action plan generation unit 440 of the present embodiment generates an action plan based on the recognition result of the recognition unit 430, and the second control unit 460 uses the vehicle based on the action plan generated by the action plan generation unit 440. The speed or steering of M can be controlled. As a result, the vehicle system 1 of the present embodiment can control the vehicle M while recognizing the periphery of the vehicle M with higher accuracy than the recognition process based only on the camera image IMC.

[About the configuration in which the recognition unit 430 has the functions of the object recognition device 316]
The object recognition device 316 may output the detection results of the camera 310, the radar device 312, and the rider 314 to the automatic driving control device 400 as they are. In this case, the processing executed by the object recognition device 316 is performed by the recognition unit 430. Further, the object recognition device 316 may be omitted. In this case, the camera 310, the radar device 312, and the rider 314 may output the detection result as it is to the automatic driving control device 400.

[Hardware configuration]
FIG. 11 is a diagram showing an example of the hardware configuration of the automatic operation control device 400 of the embodiment. As shown in the figure, the automatic operation control device 400 includes a communication controller 100-1, a CPU 100-2, a RAM (Random Access Memory) 100-3 used as a working memory, a ROM (Read Only Memory) for storing a boot program, and the like. The configuration is such that 100-4, a storage device 100-5 such as a flash memory or an HDD (Hard Disk Drive), a drive device 100-6, and the like are connected to each other by an internal bus or a dedicated communication line. The communication controller 100-1 communicates with a component other than the automatic operation control device 400. The storage device 100-5 stores a program 100-5a executed by the CPU 100-2. This program is expanded into RAM 100-3 by a DMA (Direct Memory Access) controller (not shown) or the like, and is executed by CPU 100-2. As a result, a part or all of the recognition unit 430, the action plan generation unit 440, and the second control unit 460 are realized.

The embodiment described above can be expressed as follows.
A storage device that stores programs and
With a hardware processor,
When the hardware processor executes a program stored in the storage device,
Electromagnetic waves are irradiated at different angles with the recognition device as the emission center in the vertical and horizontal directions.
Detects reflected waves and
Based on the detection result, a two-dimensional image in which pixels are assigned to each electromagnetic wave irradiation direction is generated with the horizontal direction from the moving body as a vertical reference.
A recognition device that is configured to.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Vehicle system, 310 ... Camera, 312 ... Radar device, 314 ... Rider, 316 ... Object recognition device, 316A ... Acquisition unit, 316B ... Image generation unit, 316C ... Output unit, 320 ... Communication device, 340 ... Vehicle sensor, 350 ... Navigation device, 351 ... GNSS receiver, 352 ... Navi HMI, 353 ... Route determination unit, 354 ... First map information, 361 ... Recommended lane determination unit, 362 ... Second map information, 380 ... Driving operator, 400 ... Automatic operation control device, 420 ... First control unit, 430 ... Recognition unit, 440 ... Action plan generation unit, 460 ... Second control unit, 470 ... Storage unit, 472 ... First trained model, 474 ... Second learning Completed model, 500 ... Driving driving force output device, 510 ... Brake device, 520 ... Steering device, IMC ... Camera image, IML ... Rider image, M ... Vehicle, θ1 ... Predetermined range, θ2 ... Predetermined range

Claims (8)

It is a recognition device mounted on a moving body.
A detection unit that detects the distance by irradiating electromagnetic waves at different angles with the recognition device as the emission center in the vertical and horizontal directions and detecting the reflected wave.
Based on the detection result of the detection unit, a generation unit that generates a two-dimensional image in which pixels are assigned to each electromagnetic wave irradiation direction with the horizontal direction from the moving body as a vertical reference.
A recognition device equipped with. The detector is a rider.
The recognition device according to claim 1. The generation unit generates the two-dimensional image in which the distance obtained from the detection result of the rider and the intensity of the reflected wave are used as data elements of the pixel.
The recognition device according to claim 2. An image recognition process is performed on the two-dimensional image, and a recognition unit that recognizes a target in the two-dimensional image is further provided.
The recognition device according to claim 3. The recognition unit performs the image recognition process using the trained image recognition model.
The trained image recognition model is machine-learned using a peripheral image of a moving body, and outputs a target in the two-dimensional image and a distance to the target.
The recognition device according to claim 4. The recognition device according to any one of claims 1 to 5.
A control device that controls the moving body based on the two-dimensional image, and
Control system with. The computer of the recognition device mounted on the mobile body
Electromagnetic waves are irradiated at different angles with the recognition device as the emission center in the vertical and horizontal directions.
Detect the distance by detecting the reflected wave,
Based on the detection result, a two-dimensional image in which pixels are assigned to each electromagnetic wave irradiation direction is generated with the horizontal direction from the moving body as a vertical reference.
Recognition method. To the computer of the recognition device mounted on the mobile body
Electromagnetic waves are irradiated at different angles with the recognition device as the emission center in the vertical and horizontal directions.
The distance is detected by detecting the reflected wave,
Based on the detection result, a two-dimensional image in which pixels are assigned to each electromagnetic wave irradiation direction is generated with the horizontal direction from the moving body as a vertical reference.
program.

Images