JP2022110706A - Vehicle Environment Recognition Method and Vehicle Environment Recognition Device - Google Patents

Abstract

To provide a vehicle environment recognition method and a vehicle environment recognition device capable of appropriately setting a region to which super-resolution processing is applied.SOLUTION: A vehicle environment recognition device 1 comprises: a camera 10 which images a periphery of a self vehicle a plurality of times in different timing; and a controller 20 for processing a plurality of images captured by the camera 10 in the different timing. The controller 20 estimates a travel direction of the self vehicle and sets a predetermined region on the image for each of the plurality of images on the basis of the estimated travel direction of the self vehicle. The controller 20 aligns the plurality of images on the basis of a moving amount on the image in the set region. The controller 20 performs super-resolution processing using the plurality of aligned images and generates a super-resolution image exceeding resolutions of the images captured by the camera 10. The controller 20 then recognizes a road structure on the basis of the super-resolution image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a vehicle environment recognition method and a vehicle environment recognition device.

Conventionally, there has been known an invention in which a license plate of a vehicle is imaged by a camera and recognized (Patent Document 1). The invention described in Patent Document 1 performs super-resolution processing and recognizes a license plate when it is determined that the area of the identification medium is located within a preset area.

U.S. Patent Application Publication No. 2018/0189590

However, there is a possibility that an appropriate super-resolution area cannot be set with the method of setting a super-resolution area in a specific area, such as the invention described in Patent Document 1.

The present invention has been made in view of the above problems, and its object is to provide a vehicular environment recognition method and a vehicular environment recognition device capable of appropriately setting an area to be subjected to super-resolution processing. It is to be.

A vehicle environment recognition method according to an aspect of the present invention estimates a traveling direction of a vehicle, and sets a predetermined region on each of a plurality of images based on the estimated traveling direction of the vehicle. Then, a plurality of images are aligned based on the amount of movement on the image in the set area, super-resolution processing is performed using the aligned images, and an image captured by the camera A super-resolution image exceeding the resolution of is generated, and road structures are recognized based on the super-resolution image.

According to the present invention, it is possible to appropriately set an area for super-resolution processing.

FIG. 1 is a configuration diagram of a vehicle environment recognition device 1 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a method of setting an area for super-resolution processing. FIG. 3 is a diagram illustrating another example of a method of setting an area for super-resolution processing. FIG. 4 is a diagram illustrating still another example of a method of setting an area for super-resolution processing. FIG. 5 is a diagram illustrating still another example of a method of setting an area for super-resolution processing. FIG. 6 is a diagram illustrating still another example of a method of setting an area for super-resolution processing. FIG. 7 is a diagram for explaining still another example of a method of setting an area for super-resolution processing. FIG. 8 is a flowchart for explaining an operation example of the vehicle environment recognition device 1 .

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals, and the description thereof is omitted.

A configuration example of a vehicle environment recognition device 1 will be described with reference to FIG. As shown in FIG. 1, the vehicle environment recognition device 1 includes a camera 10, a yaw rate sensor 11, a GPS receiver 12, a navigation device 13, a controller 20, a steering actuator 14, an accelerator pedal actuator 15, A brake actuator 16 is provided.

The vehicular environment recognition device 1 may be installed in a vehicle having an automatic driving function, or may be installed in a vehicle without an automatic driving function. Further, the vehicle environment recognition device 1 may be installed in a vehicle capable of switching between automatic driving and manual driving. In addition, even if the automatic driving function is a driving support function that automatically controls only a part of the vehicle control functions such as steering control, braking force control, and driving force control to assist the driver's driving. good. In this embodiment, the vehicular environment recognition device 1 is assumed to be installed in a vehicle having an automatic driving function.

The camera 10 has an imaging device such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor). Although the installation location of the camera 10 is not particularly limited, as an example, the camera 10 is installed in front of, on the side of, and behind the own vehicle. The camera 10 continuously images the surroundings of the own vehicle at a predetermined cycle (for example, a short cycle of about 10 msec). That is, the camera 10 acquires a plurality of images captured at different timings. The camera 10 detects objects (pedestrians, bicycles, motorcycles, other vehicles, etc.) existing around the own vehicle and information (laning lines, traffic lights, signs, pedestrian crossings, intersections, etc.) ahead of the own vehicle. Images captured by the camera 10 are output to the controller 20 .

The yaw rate sensor 11 detects the yaw rate of the host vehicle (speed of change in rotation angle in the turning direction). The yaw rate sensor 11 outputs the detected yaw rate to the controller 20 .

The GPS receiver 12 detects positional information of the own vehicle on the ground by receiving radio waves from artificial satellites. The position information of the host vehicle detected by the GPS receiver 12 includes latitude information and longitude information. It should be noted that the method of detecting the position information of the own vehicle is not limited to the GPS receiver 12 . For example, the position may be estimated using a method called odometry. Odometry is a method of estimating the position of the own vehicle by obtaining the movement amount and the movement direction of the own vehicle according to the rotation angle and rotation angular velocity of the own vehicle. The GPS receiver 12 outputs the detected positional information to the controller 20 .

The controller 20 is a general-purpose microcomputer having a CPU (Central Processing Unit), memory, and an input/output unit. A computer program is installed in the microcomputer to function as the vehicle environment recognition device 1 . By executing the computer program, the microcomputer functions as a plurality of information processing circuits included in the vehicle environment recognition device 1 . Here, an example of realizing a plurality of information processing circuits provided in the environment recognition device 1 for a vehicle by software is shown. It is also possible to configure Also, a plurality of information processing circuits may be configured by individual hardware. The controller 20 includes an area setting section 21, an image shift section 22, a super-resolution processing section 23, a road structure recognition section 24, and a vehicle control section 25 as an example of a plurality of information processing circuits.

The region setting unit 21 estimates the traveling direction of the vehicle using the yaw rate obtained from the yaw rate sensor 11, and sets a region for super-resolution processing based on the estimated traveling direction of the vehicle. In this embodiment, the area on which super-resolution processing is performed is part of the image captured by the camera 10 . As will be described later, the method of estimating the traveling direction of the host vehicle is not limited to the method using the yaw rate.

The image shift unit 22 calculates the amount of movement between images, and positions the images by moving the images by the calculated amount of movement. The amount of image movement is the amount of movement between two images. SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), ZNCC (Zero-mean Normalized Cross-Correlation), and the like are used for such movement amount calculation. Specifically, the image shifter 22 calculates the degree of similarity by shifting one pixel each in the vertical and horizontal directions within the search range, and calculates the amount of movement to the location with the highest degree of similarity as the amount of movement. A reference image is required to compare two images. As an example of a method for extracting a reference image, if the number of images required for super-resolution processing is N, the image that was captured the earliest among the N images is extracted as the reference image. be done.

The super-resolution processing unit 23 performs super-resolution processing using the image aligned by the image shift unit 22 to generate an image with high resolution. Super-resolution processing is a technique for generating a high-resolution image by referring to a plurality of images. An image exceeding the resolution of the image captured by the camera 10 is generated by super-resolution processing. Note that super-resolution processing is a well-known technique that has already been publicly known, and therefore will not be described in detail here. The image generated by the super-resolution processing section 23 is output to the road structure recognition section 24 .

The road structure recognition unit 24 uses the image input from the super-resolution processing unit 23 to recognize road structures. A road structure in this embodiment is defined as a stationary object. Specifically, road structures include structures such as lanes, stop lines, pedestrian crossings, road markings such as arrows, roadside strips, curbs, signs, and traffic lights. As an example of a road structure recognition method, sementic segmentation is used to recognize what kind of object each pixel is. The road structure recognition section 24 outputs the recognition result to the vehicle control section 25 .

The vehicle control unit 25 controls the steering actuator 14, the accelerator pedal actuator 15, and the brake actuator 16 using the road structure recognized by the road structure recognition unit 24, and drives the vehicle to the destination input to the navigation device 13. run automatically.

In this embodiment, the GPS receiver 12 and the navigation device 13 may not be mounted on the own vehicle. If the mobile terminal (for example, smart phone) owned by the user can be linked to the function of the own vehicle, the mobile terminal will function as a substitute for the GPS receiver 12 and the navigation device 13 .

Next, an example of a method for setting an area to be subjected to super-resolution processing will be described with reference to FIG.

The scene shown in FIG. 2 is a scene in which the vehicle is traveling straight in the left lane on a three-lane road. An image 30 shown in FIG. 2 is a scene in front of the vehicle captured by the camera 10 . In the scene shown in FIG. 2, the region setting unit 21 estimates the traveling direction of the own vehicle. A yaw rate is used as an example of a method of estimating the traveling direction of the host vehicle. As shown in FIG. 2, when the vehicle is traveling straight ahead, the yaw rate is zero or a value so small that it can be regarded as zero. Therefore, when it is determined that the yaw rate is zero or a value so small as to be regarded as zero, the region setting unit 21 can estimate that the vehicle is traveling straight ahead.

The method of estimating the traveling direction of the own vehicle is not limited to the method using the yaw rate. For example, lane markings may be used. As shown in FIG. 2, when the vehicle is traveling straight, the lane markings are also straight. Therefore, when the traffic lane marking is determined to be a straight line, the area setting unit 21 may estimate that the vehicle is traveling straight ahead. The lane markings are detected by well-known image processing.

Alternatively, the rotation angle of the steering wheel may be used. As shown in FIG. 2, when the host vehicle is running straight, the rotation angle of the steering wheel is zero or a small value that can be regarded as zero. Therefore, when it is determined that the rotation angle of the steering wheel is zero or a value so small as to be regarded as zero, the region setting unit 21 may estimate that the vehicle is traveling straight ahead.

In addition, if the vehicle is an automated driving vehicle that sets a travel trajectory on which the vehicle will travel in the future and is controlled to travel along the set travel trajectory, the vehicle will proceed from the set travel trajectory. Direction can be estimated. Alternatively, when a travel route from the current position of the vehicle to the destination is searched for by a navigation device or the like, the travel direction of the own vehicle can be estimated based on the searched travel route.

After estimating the traveling direction of the own vehicle, the area setting unit 21 sets an area on the image for super-resolution processing based on the estimated traveling direction of the own vehicle. Reference numeral 41 shown in FIG. 2 is an area where super-resolution processing is performed. As shown in FIG. 2 , when the traveling direction of the own vehicle is estimated to be straight ahead, the area setting unit 21 sets a part of the traveling direction of the own vehicle as an area 41 in the image 30 . A region 41 is a region on the image corresponding to a distance from the vehicle, and is also a region on the image corresponding to the lane on which the vehicle is traveling. However, as will be described later, the area set by the area setting unit 21 is not limited to only the area on the image corresponding to the lane on which the host vehicle travels. Also, the area 41 may be set in the upper half range of the image 30 . In the following, unless otherwise specified, the term "region" means a region on the image, and for example, the term "region on the lane" means a region on the image corresponding to the lane.

After the area 41 is set, the image shifter 22 performs alignment with respect to the area 41 using a plurality of images captured at different timings. The super-resolution processing unit 23 performs super-resolution processing using the image aligned by the image shift unit 22 to generate an image with high resolution. The area 41 becomes clear by the super-resolution processing. As described above, according to the present embodiment, the region 41 for performing super-resolution processing is set based on the traveling direction of the vehicle. becomes possible.

Next, another example of a method for setting an area to be subjected to super-resolution processing will be described with reference to FIG.

The scene shown in FIG. 3 is a continuation of the scene shown in FIG. 2, and is a scene in which the host vehicle changes lanes from the left lane to the center lane. In this case, the region setting unit 21 uses the yaw rate to estimate that the traveling direction of the vehicle is from the left lane to the center lane. Note that the yaw rate for changing lanes is obtained in advance through experiments and simulations, and is stored in a storage device (not shown). The region setting unit 21 refers to the storage device and detects a yaw rate that matches or substantially matches the yaw rate detected during running. Then, the region setting unit 21 estimates the traveling direction of the own vehicle based on the detection result.

The storage device stores not only the yaw rate when the host vehicle changes lanes to the left, but also the yaw rate when the host vehicle changes lanes to the right. The storage device also stores the yaw rate when the vehicle turns left and the yaw rate when the vehicle turns right.

In the scene shown in FIG. 3, when the traveling direction of the own vehicle is estimated to be from the left lane to the center lane, the area setting unit 21 sets an area 41 on the center lane.

Next, still another example of a method of setting an area for super-resolution processing will be described with reference to FIG.

The scene shown in FIG. 4 is a scene in which the own vehicle turns left. An image 31 shown in FIG. 4 is a scene in the left direction as seen from the own vehicle captured by the camera 10 . In this case, the region setting unit 21 uses the yaw rate to estimate that the traveling direction of the host vehicle is the left direction. As described above, the yaw rate at which the host vehicle makes a left turn is known in advance, so the area setting unit 21 can refer to this. Alternatively, the region setting unit 21 may use the lane markings to estimate that the traveling direction of the own vehicle is the left direction. A part of the traffic lane marking shown in FIG. 4 is curved with a predetermined curvature. By using such traffic lane markings, the area setting unit 21 can estimate that the traveling direction of the own vehicle is the left direction. When the traveling direction of the own vehicle is estimated to be the left direction, the area setting unit 21 sets an area 42 on the lane along which the own vehicle will travel in the future.

As shown in FIG. 4, the area setting unit 21 may set areas 43 to 44 when setting the area 42 . Areas 43 and 44 are areas set on the opposite lane of the lane on which the vehicle is traveling. Regions 43 to 44 are also regions where super-resolution processing is performed, like region 42 . By extending the area where the super-resolution processing is performed to the oncoming lane in this way, the accuracy of recognizing an obstacle (for example, an oncoming vehicle) that may affect the running of the own vehicle is improved. Note that only the area 42 may be set.

Next, still another example of the method of setting the area for super-resolution processing will be described with reference to FIG.

The scene shown in FIG. 5 is a scene in which the own vehicle runs through an intersection. An image 32 shown in FIG. 5 is a scene in front of the vehicle captured by the camera 10 . In the scene shown in FIG. 5, the traveling direction of the own vehicle is straight ahead. The region setting unit 21 estimates the traveling direction of the own vehicle by the same method as the method described with reference to FIG. When the traveling direction of the own vehicle is estimated to be straight ahead, the area setting unit 21 sets an area 45 on the lane in which the own vehicle travels. The area 45 includes not only the lane in which the vehicle is traveling beyond the intersection (the right lane indicated by the area 45), but also the lane adjacent to the lane in which the own vehicle is traveling (the left lane indicated by the area 45). . However, the area 45 is not limited to this. The area 45 may be a range that includes only the lane in which the vehicle is traveling beyond the intersection.

In the present embodiment, it has been described that the region for super-resolution processing is set based on the traveling direction of the host vehicle. In addition to this, when a pedestrian crossing is included in the image 32 as shown in FIG. 5, the region setting unit 21 may set the pedestrian crossing on the image as regions 46 and 47 for super-resolution processing. Further, when an intersection is included in the image 32 as shown in FIG. 6, the area 45 may be expanded in a direction corresponding to the horizontal direction as seen from the host vehicle. The range to be expanded is not particularly limited, but may be expanded to include the driving lane, oncoming lanes, and portions other than these lanes (for example, sidewalks). Further, as shown in FIG. 7, the area expanded in the direction corresponding to the horizontal direction may be divided into a plurality of areas. In the example shown in FIG. 7, it is divided into four areas (areas 48 to 51). Region 48 contains the sidewalk, region 49 contains the lane in which the vehicle is traveling, and regions 50-51 contain the oncoming lanes. When super-resolution processing is performed, it is preferable that the movements of objects in the regions of interest are the same or similar. Therefore, highly accurate super-resolution processing can be realized by setting areas according to the classification of the driving lane, the oncoming lane, and the sidewalk. The controller 20 determines whether or not the image 32 includes an intersection, in other words, whether or not the vehicle is traveling through an intersection. It is possible to make a determination by referring to the map database that is available. Alternatively, image processing may be used to determine whether or not the vehicle is traveling through an intersection.

Next, an operation example of the vehicle environment recognition device 1 will be described with reference to the flowchart of FIG.

In step S101, the surroundings of the own vehicle are imaged at different timings by the camera 10 mounted on the own vehicle. The process proceeds to step S<b>103 and the controller 20 acquires the yaw rate of the host vehicle from the yaw rate sensor 11 . Also, the controller 20 acquires position information of the own vehicle from the GPS receiver 12 .

The process proceeds to step S<b>105 , and the area setting unit 21 estimates the traveling direction of the vehicle mainly based on the yaw rate of the vehicle from the yaw rate sensor 11 . However, as described above, the method of estimating the traveling direction of the own vehicle is not limited to the method using the yaw rate. The region setting unit 21 sets a region for performing super-resolution processing on each of the plurality of images based on the estimated traveling direction of the own vehicle.

The process advances to step S107, and the image shifter 22 aligns the regions using a plurality of images captured at different timings. The process proceeds to step S109, and the super-resolution processing unit 23 performs super-resolution processing using the image aligned by the image shift unit 22 to generate a high-resolution image. The process proceeds to step S<b>111 , and the image generated by the super-resolution processing unit 23 is output to the road structure recognition unit 24 .

The process proceeds to step S<b>113 , and the road structure recognition unit 24 uses the image input from the super-resolution processing unit 23 to recognize road structures. The process proceeds to step S115, and the vehicle control unit 25 controls the steering actuator 14, the accelerator pedal actuator 15, and the brake actuator 16 using the road structure recognized by the road structure recognition unit 24. FIG.

(Effect)
As described above, according to the vehicle environment recognition device 1 according to the present embodiment, the following effects can be obtained.

The vehicular environment recognition device 1 includes a camera 10 that captures multiple images of the surroundings of the vehicle at different timings, and a controller 20 that processes the multiple images captured by the camera 10 at different timings. The controller 20 estimates the traveling direction of the own vehicle, and sets a predetermined area on each of the plurality of images based on the estimated traveling direction of the own vehicle. The controller 20 aligns a plurality of images based on the amount of movement on the images in the set area. The controller 20 performs super-resolution processing using the aligned images to generate a super-resolution image that exceeds the resolution of the image captured by the camera 10 . The controller 20 then recognizes road structures based on the super-resolution image. According to the present embodiment, the area for performing super-resolution processing is set based on the traveling direction of the own vehicle, so it is possible to set the area for performing super-resolution processing at a position appropriate for the running of the own vehicle. Become.

Further, when it is determined that the vehicle is traveling through an intersection, the controller 20 expands the area for super-resolution processing in a direction corresponding to the left-right direction when viewed from the vehicle. As a result, it becomes possible to set a region for which super-resolution processing is performed within a range that includes an obstacle (for example, an oncoming vehicle) that may affect the traveling of the own vehicle.

Furthermore, the controller 20 may set a region for super-resolution processing at the position of the pedestrian crossing on the image. As a result, it becomes possible to set a region for which super-resolution processing is performed within a range that includes obstacles (for example, pedestrians) that may affect the running of the own vehicle.

Also, the controller 20 may divide the expanded area into a plurality of areas. When super-resolution processing is performed, it is preferable that the movements of objects in the regions of interest are the same or similar. Therefore, by dividing the area so that objects with the same or similar movements are included, highly accurate super-resolution processing can be realized.

The controller 20 estimates the traveling direction of the vehicle based on the yaw rate obtained from the yaw rate sensor 11 that detects the yaw rate of the vehicle, or the information on the traffic lane marking obtained from the image. Since the area for super-resolution processing is set based on the traveling direction of the own vehicle estimated in this way, it is possible to set the area for super-resolution processing at a position appropriate for the running of the own vehicle. Become.

Each function described in the above embodiments may be implemented by one or more processing circuits. Processing circuitry includes programmed processing devices, such as processing devices that include electrical circuitry. Processing circuitry also includes devices such as application specific integrated circuits (ASICs) and circuit components arranged to perform the described functions.

While embodiments of the present invention have been described above, the discussion and drawings forming part of this disclosure should not be construed as limiting the invention. Various alternative embodiments, implementations and operational techniques will become apparent to those skilled in the art from this disclosure.

1 vehicle environment recognition device 10 camera 11 yaw rate sensor 12 GPS receiver 13 navigation device 14 steering actuator 15 accelerator pedal actuator 16 brake actuator 20 controller 21 region setting unit 22 image shift unit 23 super-resolution processing unit 24 road structure recognition unit 25 vehicle control unit

Claims (6)

A vehicular environment recognition method comprising: a camera that captures multiple images of the surroundings of the vehicle at different timings; and a controller that processes the multiple images captured by the camera at different timings,
The controller is
estimating the traveling direction of the own vehicle;
setting a predetermined area on each of the plurality of images based on the estimated traveling direction of the own vehicle;
aligning the plurality of images based on the amount of movement on the image in the set area;
Performing super-resolution processing using the aligned images to generate a super-resolution image exceeding the resolution of the image captured by the camera,
A vehicle environment recognition method, wherein a road structure is recognized based on the super-resolution image. 2. The environment recognition method for a vehicle according to claim 1, wherein, when it is determined that said vehicle will run through an intersection, said controller expands said area in a direction corresponding to a horizontal direction as viewed from said vehicle. 3. The vehicle environment recognition method according to claim 2, wherein the controller sets the region at a position of a pedestrian crossing on the image. 4. The vehicle environment recognition method according to claim 2, wherein the controller divides the expanded area into a plurality of areas. The controller estimates the direction of travel of the vehicle based on the yaw rate obtained from a sensor that detects the yaw rate of the vehicle, or information on traffic lane markings obtained from the image. Item 5. The vehicle environment recognition method according to any one of Items 1 to 4. a camera that captures multiple images of the surroundings of the vehicle at different times;
A controller that processes a plurality of images captured at different timings by the camera,
The controller is
estimating the traveling direction of the own vehicle;
setting a predetermined area on each of the plurality of images based on the estimated traveling direction of the own vehicle;
aligning the plurality of images based on the amount of movement on the image in the set area;
Performing super-resolution processing using the aligned images to generate a super-resolution image exceeding the resolution of the image captured by the camera,
An environment recognition device for a vehicle, wherein a road structure is recognized based on the super-resolution image.

Images