WO2021159397A1

WO2021159397A1 - Vehicle travelable region detection method and detection device

Info

Publication number: WO2021159397A1
Application number: PCT/CN2020/075104
Authority: WO
Inventors: 朱麒文; 崔学理; 郑佳; 吴祖光
Original assignee: 华为技术有限公司
Priority date: 2020-02-13
Filing date: 2020-02-13
Publication date: 2021-08-19
Also published as: CN114981138A

Abstract

A vehicle travelable region detection method and detection device. The detection method comprises: obtaining a binocular image in a vehicle traveling direction, the binocular image comprising a left-eye image and a right-eye image; obtaining parallax information of a travelable region boundary in the binocular image according to a travelable region boundary in the left-eye image and a travelable region boundary in the right-eye image; and obtaining a vehicle travelable region in the binocular image according to the parallax information. The above technical solution can greatly reduce the amount of computation while ensuring the detection accuracy, and improve the road condition detection efficiency of an autonomous vehicle.

Description

Detection method and detection device of vehicle travelable area

Technical field

This application relates to the automotive field, and more specifically, to a detection method and a detection device for a vehicle's drivable area.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theories.

Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on the collaboration of computer vision, radar, monitoring devices, and global positioning systems to enable motor vehicles to achieve autonomous driving without the need for human active operations. Self-driving vehicles use various computing systems to help transport passengers from one location to another; some self-driving vehicles may require some initial or continuous input from an operator (such as a pilot, driver, or passenger) ; Automatic driving vehicles allow the operator to switch from manual mode to automatic driving mode or a mode in between. Since automatic driving technology does not require humans to drive motor vehicles, it can theoretically effectively avoid human driving errors and reduce The occurrence of traffic accidents can improve the efficiency of highway transportation. Therefore, more and more attention has been paid to autonomous driving technology.

At present, it is possible to detect the drivable area of an autonomous vehicle by using a binocular vision method. The binocular vision method refers to the extraction and positioning of the drivable area through the global disparity map of the image output by the binocular camera; The amount of calculation required for the eye camera to obtain the global disparity map is large, which makes the automatic driving vehicle unable to perform real-time processing, resulting in safety risks during the driving process of the automatic driving vehicle; therefore, how to improve the driving ability of the vehicle while ensuring the detection accuracy The detection efficiency of the area detection method has become an urgent problem to be solved.

Summary of the invention

The present application provides a detection method and a detection device for a vehicle travelable area, which can improve the real-time performance of the detection system of an automatic driving vehicle under a certain detection accuracy, and improve the detection efficiency of the vehicle travelable area detection method.

In a first aspect, a method for detecting a vehicle's drivable area is provided, including: acquiring a binocular image of a vehicle traveling direction, wherein the binocular image includes a left-eye image and a right-eye image; according to the drivable area in the left-eye image The area boundary and the travelable area boundary in the right-eye image are used to obtain parallax information of the travelable area boundary in the binocular image; based on the parallax information, the vehicle travelable area in the binocular image is obtained.

The above-mentioned binocular image may include a left-eye image and a right-eye image; for example, it may refer to the left and right two-dimensional images respectively collected by two parallel and equal-height cameras in an autonomous vehicle.

In a possible implementation, the binocular image may be an image of the road surface or the surrounding environment obtained by the autonomous vehicle in the driving direction; for example, it includes the image of the road surface and the image of obstacles and pedestrians attached to the vehicle.

It should be understood that the parallax may refer to the difference in direction caused by observing the same target from two points with a certain distance. For example, the difference in the horizontal direction between the position of the driveable area on the same road collected by the left-eye camera and the right-eye camera in an autonomous vehicle may be parallax information.

In a possible implementation, the obtained left-eye image and right-eye image can be separately input to a deep learning network pre-trained to identify the drivable area in the image; the left-eye image can be identified through the pre-trained deep learning network The area where the vehicle can travel in the image on the right.

In the embodiment of the present application, a binocular image of the driving direction of the vehicle can be obtained, and based on the binocular image, the boundary of the vehicle's travelable area in the left-eye image and the right-eye image can be acquired; according to the travelable area boundary in the left-eye image and the right-eye image The disparity information of the travelable area boundary in the binocular image can be obtained; based on the disparity information, the position of the vehicle travelable area in the binocular image can be obtained; the vehicle travelable area in the embodiment of the present application The detection method can avoid the pixel-by-pixel disparity calculation of the binocular image to obtain the global disparity image. The disparity information of the drivable area boundary in the binocular image can be calculated to realize that the boundary point of the drivable area is in the coordinate system of the autonomous vehicle The positioning can greatly reduce the amount of calculation while ensuring detection accuracy, and improve the detection efficiency of autonomous vehicles on road conditions.

With reference to the first aspect, in some implementations of the first aspect, the method further includes: performing segmentation processing on the drivable area boundary in the first image; Obtaining the disparity information of the drivable area boundary in the right eye image to obtain the disparity information of the drivable area boundary in the binocular image includes:

Based on the N boundary points and line segments obtained by the segmentation process, the second image is matched with the travelable area boundary to obtain the disparity information, wherein the first image is any one of the binocular images; The second image is another image different from the first image in the binocular image; N is an integer greater than or equal to 2.

Wherein, the above-mentioned first image may refer to the left-eye image in the binocular image, and the second image may refer to the right-eye image in the binocular image; or, the first image may refer to the right-eye image in the binocular image, and the second image The image refers to the left eye image in the binocular image.

In a possible implementation, the boundary of the drivable area in the left-eye image can be segmented based on the inflection point of the drivable area boundary in the left-eye image in the binocular image to obtain N segments of the drivable area boundary in the left-eye image ; Matching the boundaries of the drivable area in the right-eye image through the N segments of the drivable area boundary in the left-eye image, so as to obtain the parallax information.

In another possible implementation manner, the boundary of the drivable area in the right-eye image can be segmented based on the inflection point of the boundary of the drivable area in the right-eye image in the binocular image to obtain N segments of drivable areas in the right-eye image Boundary; match the boundaries of the drivable area in the left-eye image through the N segments of the drivable area boundary in the right-eye image to obtain disparity information.

In the embodiment of the present application, when the boundary of the drivable area in the left-eye image is matched with the drivable boundary in the right-eye image, that is, when the disparity information of the boundary of the drivable area in the binocular image is calculated, the The boundary of the drivable area in any one of the images is segmented, and the segmented drivable area boundary can be segmented to match, so as to improve the accuracy of the drivable area boundary matching, which is conducive to more accurate acquisition of vehicles on the road. Information about the drivable area.

With reference to the first aspect, in some implementation manners of the first aspect, the second image is matched with the drivable area boundary based on the N boundary point line segments obtained by the segmentation process to obtain the disparity information, include:

The disparity information is obtained by performing boundary matching of the drivable area on the second image according to the N boundary point line segments and the matching strategy, wherein the matching strategy is determined according to the slopes of the N boundary point line segments.

In the embodiment of the present application, a matching strategy can also be used when segmentally matching the drivable area in the left-eye image and the right-eye image, that is, the boundary point line segments with different slopes can be matched based on different matching strategies, so that Improve the matching accuracy of the boundary points of the drivable area.

With reference to the first aspect, in some implementation manners of the first aspect, the matching of the drivable area boundary of the second image according to the N boundary point line segments and a matching strategy to obtain the disparity information includes:

For the boundary point line segment of the first type among the N boundary point line segments, the first matching strategy in the matching strategy is used to perform the drivable area boundary matching on the second image, wherein the boundary point of the first type The line segment includes the boundary of the drivable area on the edge of the road and the boundary of the drivable area on the side of other vehicles;

For the second type of boundary point line segments among the N boundary point line segments, the second matching strategy in the matching strategy is used to perform the driving area boundary matching on the second image, wherein the second type boundary The boundary point in the dotted line segment is the same distance from the vehicle.

Among them, the first type of boundary point line segment may refer to a boundary point line segment with a relatively large slope; for example, a road edge area or other vehicle side area; the second type of boundary point line segment may refer to a boundary point line segment with a relatively small slope; for example, , The rear area of other vehicles, etc. There is basically no overlap in the drivable area boundary extracted from the left eye image and the right eye image for the boundary point line segment with a larger slope; for the drivable area boundary extracted from the left eye image and the right eye image for the boundary point line segment with a smaller slope There is some overlap of boundary points.

In the embodiment of the present application, a segmented matching strategy based on the magnitude of the slope of the drivable area boundary is proposed, and the detected drivable area is segmented according to the slope distribution. For larger slopes and smaller slopes Different matching strategies are used to match the boundaries of the drivable area, which can improve the matching accuracy of the boundary points of the drivable area.

With reference to the first aspect, in some implementations of the first aspect, the matching strategy includes a first matching strategy and a second matching strategy, wherein the first matching strategy refers to matching through a search area, and the search A region refers to a region generated with any point in the first boundary point line segment as the center, and the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy is Refers to matching through a preset search step, which is determined based on the boundary point disparity of one of the N boundary point line segments.

In a possible implementation manner, for the above-mentioned boundary point line segment with a larger slope, one boundary point of the travelable area boundary point segment of the first image (for example, the left eye image) in the binocular image can be used as a template point, With the boundary point of the second image (for example, the right eye image) corresponding to the same row in the first image as the center, the search area is generated to match the template point in the first image.

In another possible implementation, for the boundary point line segment with a small slope, the boundary point extracted through the first image (for example, the left eye image) will be the same as the boundary point extracted through the second image (for example, the right eye image). There is a phenomenon of overlap of some boundary points of the points, and the method of inflection point parallax correction can be used to match this part of the boundary points.

With reference to the first aspect, in some implementations of the first aspect, the N boundary point line segments are determined according to inflection points of the travelable area boundary in the first image.

In a second aspect, a device for detecting a vehicle's drivable area is provided, including: an acquisition unit for acquiring binocular images of the vehicle traveling direction, wherein the binocular images include a left-eye image and a right-eye image; and the processing unit uses According to the boundary of the drivable area in the left-eye image and the boundary of the drivable area in the right-eye image, disparity information of the boundary of the drivable area in the binocular image is obtained; based on the disparity information, the binocular is obtained The area where the vehicle can travel in the image.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is further configured to: perform segmentation processing on the boundaries of the drivable area in the first image; and based on the N boundaries obtained by the segmentation processing The dotted line segment matches the travelable area boundary on the second image to obtain the disparity information, where N is an integer greater than or equal to 2.

With reference to the second aspect, in some implementations of the second aspect, the processing unit is specifically configured to: perform boundary matching of the drivable area on the second image according to the N boundary point line segments and a matching strategy to obtain the In the disparity information, the matching strategy is determined according to the slopes of the N boundary point line segments.

With reference to the second aspect, in some implementation manners of the second aspect, the processing unit is specifically configured to:

With reference to the second aspect, in some implementations of the second aspect, the matching strategy includes a first matching strategy and a second matching strategy, wherein the first matching strategy refers to matching through a search area, and the search A region refers to a region generated with any point in the first boundary point line segment as the center, and the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy is Refers to matching through a preset search step, which is determined based on the boundary point disparity of one of the N boundary point line segments.

With reference to the second aspect, in some implementations of the second aspect, the N boundary point line segments are determined according to inflection points of the travelable area boundary in the first image.

In a third aspect, a device for detecting a vehicle's travelable area is provided, including: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the The processor is used to perform the following process: acquiring a binocular image of the driving direction of the vehicle, wherein the binocular image includes a left-eye image and a right-eye image; The travel area boundary is obtained, and the parallax information of the travelable area boundary in the binocular image is obtained; based on the parallax information, the vehicle travelable area in the binocular image is obtained.

In a possible implementation manner, the processor included in the foregoing detection device is further configured to execute the method for detecting the travelable area of the vehicle in the first aspect and any one of the implementation manners of the first aspect.

It should be understood that the expansion, limitation, explanation and description of the related content in the above-mentioned first aspect are also applicable to the same content in the third aspect.

In a fourth aspect, a computer-readable storage medium is provided. The computer-readable medium storage medium is used to store program code. When the program code is executed by a computer, the computer is used to execute the first aspect and the first aspect described above. On the one hand, the detection method in any one of the implementations.

In a fifth aspect, a chip is provided, the chip includes a processor, and the processor is configured to execute the detection method in any one of the foregoing first aspect and the first aspect.

In a possible implementation manner, the chip of the fifth aspect described above may be located in an in-vehicle terminal of an autonomous vehicle.

In a sixth aspect, a computer program product is provided, the computer program product comprising: computer program code, when the computer program code runs on a computer, the computer executes any one of the first aspect and the first aspect. The detection method in a kind of realization.

It should be noted that the above-mentioned computer program code may be stored in whole or in part on a first storage medium, where the first storage medium may be packaged with the processor, or may be packaged separately with the processor. There is no specific limitation.

Description of the drawings

Fig. 1 is a schematic structural diagram of a vehicle provided by an embodiment of the present application;

Figure 2 is a schematic structural diagram of a computer system provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of the application of a cloud-side command automatic driving vehicle provided by an embodiment of the present application;

Fig. 4 is a schematic diagram of a detection system for an autonomous vehicle provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of coordinate conversion provided by an embodiment of the present application;

FIG. 6 is a schematic flowchart of a method for detecting a vehicle drivable area provided by an embodiment of the present application;

FIG. 7 is a schematic flowchart of a method for detecting a vehicle drivable area provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of a boundary segment of a drivable area provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of acquiring a road surface image provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of matching a boundary point line segment with a larger slope provided by the implementation of the present application;

FIG. 11 is a schematic diagram of a boundary point line segment with a small slope provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of sub-pixelation processing provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of obtaining positioning of boundary points of a drivable area provided by an embodiment of the present application;

FIG. 14 is a schematic diagram of obtaining positioning of boundary points of a drivable area provided by an embodiment of the present application;

FIG. 15 is a schematic diagram of the method for detecting a vehicle travelable area according to an embodiment of the present application applied to a specific product form;

FIG. 16 is a schematic structural diagram of a detection device for a vehicle travelable area provided by an embodiment of the present application;

FIG. 17 is a schematic structural diagram of a detection device for a vehicle travelable area provided by another embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative work shall fall within the protection scope of this application.

Fig. 1 is a functional block diagram of a vehicle 100 provided by an embodiment of the present application.

Among them, the vehicle 100 may be a manually driven vehicle, or the vehicle 100 may be configured in a fully or partially automatic driving mode.

In one example, the vehicle 100 can control its own vehicle while in the automatic driving mode, and can determine the current state of the vehicle and its surrounding environment through human operations, determine the possible behavior of at least one other vehicle in the surrounding environment, and The confidence level corresponding to the possibility of other vehicles performing possible behaviors is determined, and the vehicle 100 is controlled based on the determined information. When the vehicle 100 is in the automatic driving mode, the vehicle 100 can be placed to operate without human interaction.

The vehicle 100 may include various subsystems, such as a traveling system 110, a sensing system 120, a control system 130, one or more peripheral devices 140 and a power supply 160, a computer system 150, and a user interface 170.

Alternatively, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. In addition, each of the subsystems and elements of the vehicle 100 may be wired or wirelessly interconnected.

Illustratively, the travel system 110 may include components for providing power movement to the vehicle 100. In one embodiment, the travel system 110 may include an engine 111, a transmission 112, an energy source 113, and wheels 114/tires. The engine 111 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations; for example, a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine. The engine 111 can convert the energy source 113 into mechanical energy.

Exemplarily, the energy source 113 may include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other power sources. The energy source 113 may also provide energy for other systems of the vehicle 100.

Exemplarily, the transmission device 112 may include a gearbox, a differential, and a drive shaft; wherein, the transmission device 112 may transmit mechanical power from the engine 111 to the wheels 114.

In an embodiment, the transmission device 112 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 114.

Exemplarily, the sensing system 120 may include several sensors that sense information about the environment around the vehicle 100.

For example, the sensing system 120 may include a positioning system 121 (for example, a GPS system, a Beidou system or other positioning systems), an inertial measurement unit 122 (IMU), a radar 123, a laser rangefinder 124, and a camera 125. The sensing system 120 may also include sensors of the internal system of the monitored vehicle 100 (for example, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and identification are key functions for the safe operation of the autonomous vehicle 100.

Among them, the positioning system 121 can be used to estimate the geographic location of the vehicle 100. The IMU 122 may be used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In an embodiment, the IMU 122 may be a combination of an accelerometer and a gyroscope.

Exemplarily, the radar 123 may use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing the object, the radar 123 may also be used to sense the speed and/or direction of the object.

Exemplarily, the laser rangefinder 124 may use laser light to sense objects in the environment where the vehicle 100 is located. In some embodiments, the laser rangefinder 124 may include one or more laser sources, laser scanners, and one or more detectors, as well as other system components.

Exemplarily, the camera 125 may be used to capture multiple images of the surrounding environment of the vehicle 100. For example, the camera 125 may be a still camera or a video camera.

As shown in FIG. 1, the control system 130 controls the operation of the vehicle 100 and its components. The control system 130 may include various elements, such as a steering system 131, a throttle 132, a braking unit 133, a computer vision system 134, a route control system 135, and an obstacle avoidance system 136.

Illustratively, the steering system 131 may be operated to adjust the forward direction of the vehicle 100. For example, it may be a steering wheel system in one embodiment. The throttle 132 may be used to control the operating speed of the engine 111 and thereby control the speed of the vehicle 100.

Illustratively, the braking unit 133 may be used to control the deceleration of the vehicle 100; the braking unit 133 may use friction to slow down the wheels 114. In other embodiments, the braking unit 133 may convert the kinetic energy of the wheels 114 into electric current. The braking unit 133 may also take other forms to slow down the rotation speed of the wheels 114 to control the speed of the vehicle 100.

As shown in FIG. 1, the computer vision system 134 may be operable to process and analyze the images captured by the camera 125 in order to identify objects and/or features in the surrounding environment of the vehicle 100. The aforementioned objects and/or features may include traffic signals, road boundaries and obstacles. The computer vision system 134 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision technologies. In some embodiments, the computer vision system 134 may be used to map the environment, track objects, estimate the speed of objects, and so on.

Illustratively, the route control system 135 may be used to determine the travel route of the vehicle 100. In some embodiments, the route control system 135 may combine data from sensors, GPS, and one or more predetermined maps to determine a travel route for the vehicle 100.

As shown in FIG. 1, the obstacle avoidance system 136 may be used to identify, evaluate, and avoid or otherwise surpass potential obstacles in the environment of the vehicle 100.

In one example, the control system 130 may additionally or alternatively include components other than those shown and described. Alternatively, a part of the components shown above may be reduced.

As shown in FIG. 1, the vehicle 100 can interact with external sensors, other vehicles, other computer systems, or users through a peripheral device 140; wherein, the peripheral device 140 can include a wireless communication system 141, an onboard computer 142, a microphone 143 and/ Or speaker 144.

In some embodiments, the peripheral device 140 may provide a means for the vehicle 100 to interact with the user interface 170. For example, the onboard computer 142 may provide information to the user of the vehicle 100. The user interface 116 can also operate the onboard computer 142 to receive user input; the onboard computer 142 can be operated through a touch screen. In other cases, the peripheral device 140 may provide a means for the vehicle 100 to communicate with other devices located in the vehicle. For example, the microphone 143 may receive audio (eg, voice commands or other audio input) from the user of the vehicle 100. Similarly, the speaker 144 may output audio to the user of the vehicle 100.

As shown in FIG. 1, the wireless communication system 141 may wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 141 can use 3G cellular communication; for example, code division multiple access (CDMA), EVD0, global system for mobile communications (GSM)/general packet radio service (general packet radio service) packet radio service, GPRS), or 4G cellular communication, such as long term evolution (LTE); or, 5G cellular communication. The wireless communication system 141 can communicate with a wireless local area network (WLAN) using wireless Internet access (WiFi).

In some embodiments, the wireless communication system 141 may directly communicate with the device using an infrared link, Bluetooth, or ZigBee; other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 141 may include one or Multiple dedicated short range communications (DSRC) devices, these devices may include public and/or private data communications between vehicles and/or roadside stations.

As shown in FIG. 1, the power source 160 may provide power to various components of the vehicle 100. In one embodiment, the power source 160 may be a rechargeable lithium ion or lead-acid battery. One or more battery packs of such batteries may be configured as a power source to provide power to various components of the vehicle 100. In some embodiments, the power source 160 and the energy source 113 may be implemented together, such as in some all-electric vehicles.

Exemplarily, part or all of the functions of the vehicle 100 may be controlled by the computer system 150, where the computer system 150 may include at least one processor 151, and the processor 151 is executed in a non-transitory computer readable medium stored in the memory 152, for example. The instruction 153. The computer system 150 may also be multiple computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner.

For example, the processor 151 may be any conventional processor, such as a commercially available CPU.

Alternatively, the processor may be a dedicated device such as an ASIC or other hardware-based processor. Although FIG. 1 functionally illustrates the processor, memory, and other elements of the computer in the same block, those of ordinary skill in the art should understand that the processor, computer, or memory may actually include or may not store Multiple processors, computers or memories in the same physical enclosure. For example, the memory may be a hard disk drive or other storage medium located in a housing other than the computer. Therefore, a reference to a processor or computer will be understood to include a reference to a collection of processors or computers or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described here, some components such as steering components and deceleration components may each have its own processor that only performs calculations related to component-specific functions .

In the various aspects described herein, the processor may be located away from the vehicle and wirelessly communicate with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed in the vehicle and others are executed by a remote processor, including taking the necessary steps to perform a single manipulation.

In some embodiments, the memory 152 may contain instructions 153 (eg, program logic), which may be executed by the processor 151 to perform various functions of the vehicle 100, including those functions described above. The memory 152 may also contain additional instructions, for example, including sending data to, receiving data from, interacting with, and/or performing data to one or more of the traveling system 110, the sensing system 120, the control system 130, and the peripheral device 140. Control instructions.

Exemplarily, in addition to the instructions 153, the memory 152 may also store data, such as road maps, route information, the position, direction, and speed of the vehicle, and other such vehicle data, as well as other information. Such information may be used by the vehicle 100 and the computer system 150 during the operation of the vehicle 100 in autonomous, semi-autonomous, and/or manual modes.

As shown in FIG. 1, the user interface 170 may be used to provide information to or receive information from a user of the vehicle 100. Optionally, the user interface 170 may include one or more input/output devices in the set of peripheral devices 140, for example, a wireless communication system 141, a car computer 142, a microphone 143, and a speaker 144.

In an embodiment of the present application, the computer system 150 may control the functions of the vehicle 100 based on inputs received from various subsystems (for example, the traveling system 110, the sensing system 120, and the control system 130) and from the user interface 170. For example, the computer system 150 may use input from the control system 130 in order to control the braking unit 133 to avoid obstacles detected by the sensing system 120 and the obstacle avoidance system 136. In some embodiments, the computer system 150 is operable to provide control of many aspects of the vehicle 100 and its subsystems.

Optionally, one or more of these components described above may be installed or associated with the vehicle 100 separately. For example, the memory 152 may be partially or completely separated from the vehicle 100. The above-mentioned components may be communicatively coupled together in a wired and/or wireless manner.

Optionally, the above-mentioned components are only an example. In practical applications, the components in the above-mentioned modules may be added or deleted according to actual needs. FIG. 1 should not be construed as a limitation to the embodiments of the present application.

Optionally, the vehicle 100 may be an autonomous vehicle traveling on a road, and may recognize objects in its surrounding environment to determine the adjustment to the current speed. The object may be other vehicles, traffic control equipment, or other types of objects. In some examples, each recognized object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine the speed to be adjusted by the self-driving car.

Optionally, the vehicle 100 or a computing device associated with the vehicle 100 (such as the computer system 150, the computer vision system 134, and the memory 152 in FIG. 1) may be based on the characteristics of the identified object and the state of the surrounding environment (for example, traffic, Rain, ice on the road, etc.) to predict the behavior of the identified object.

Optionally, each recognized object depends on each other's behavior. Therefore, all recognized objects can also be considered together to predict the behavior of a single recognized object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified object. In other words, the self-driving car can determine based on the predicted behavior of the object that the vehicle will need to be adjusted (e.g., accelerate, decelerate, or stop) to a stable state. In this process, other factors may also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 on the road on which it is traveling, the curvature of the road, the proximity of static and dynamic objects, and so on.

In addition to providing instructions to adjust the speed of the self-driving car, the computing device can also provide instructions to modify the steering angle of the vehicle 100 so that the self-driving car follows a given trajectory and/or maintains an object near the self-driving car (for example, , The safe horizontal and vertical distances of cars in adjacent lanes on the road.

The above-mentioned vehicle 100 may be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawn mower, recreational vehicle, playground vehicle, construction equipment, tram, golf cart, train, and trolley, etc. The application examples are not particularly limited.

In a possible implementation manner, the vehicle 100 shown in FIG. 1 may be an automatic driving vehicle, and the automatic driving system will be described in detail below.

Fig. 2 is a schematic diagram of an automatic driving system provided by an embodiment of the present application.

The automatic driving system shown in FIG. 2 includes a computer system 201, where the computer system 201 includes a processor 203, which is coupled to a system bus 205. The processor 203 may be one or more processors, where each processor may include one or more processor cores. The display adapter 207 (video adapter) can drive the display 209, and the display 209 is coupled to the system bus 205. The system bus 205 may be coupled to an input/output (I/O) bus 213 through a bus bridge 211, and an I/O interface 215 is coupled to an I/O bus. The I/O interface 215 communicates with a variety of I/O devices, such as input devices 217 (such as keyboard, mouse, touch screen, etc.), media tray 221, (such as CD-ROM, multimedia interface, etc.) . The transceiver 223 can send and/or receive radio communication signals, and the camera 255 can capture landscape and dynamic digital video images. Wherein, the interface connected to the I/O interface 215 may be the USB port 225.

The processor 203 may be any traditional processor, such as a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, or a combination of the foregoing.

Optionally, the processor 203 may be a dedicated device such as an application specific integrated circuit (ASIC); the processor 203 may be a neural network processor or a combination of a neural network processor and the foregoing traditional processors.

Optionally, in various embodiments described herein, the computer system 201 may be located far away from the autonomous driving vehicle, and may wirelessly communicate with the autonomous driving vehicle. In other respects, some of the processes described herein are executed on a processor provided in an autonomous vehicle, and others are executed by a remote processor, including taking actions required to perform a single manipulation.

The computer system 201 can communicate with the software deployment server 249 through the network interface 229. The network interface 229 may be a hardware network interface, such as a network card. The network 227 may be an external network, such as the Internet, or an internal network, such as an Ethernet or a virtual private network (VPN). Optionally, the network 227 may also be a wireless network, such as a wifi network, a cellular network, and so on.

As shown in FIG. 2, the hard disk drive interface is coupled with the system bus 205, the hardware drive interface 231 can be connected with the hard disk drive 233, and the system memory 235 is coupled with the system bus 205. The data running in the system memory 235 may include an operating system 237 and application programs 243. The operating system 237 may include a parser 239 (shell) and a kernel 241 (kernel). The shell 239 is an interface between the user and the kernel of the operating system. The shell can be the outermost layer of the operating system; the shell can manage the interaction between the user and the operating system, for example, waiting for the user's input, interpreting the user's input to the operating system, and processing various operating systems The output result. The kernel 241 may be composed of those parts of the operating system that are used to manage memory, files, peripherals, and system resources. Directly interact with the hardware. The operating system kernel usually runs processes and provides inter-process communication, providing CPU time slice management, interrupts, memory management, IO management, and so on. Application programs 243 include programs that control auto-driving cars, such as programs that manage the interaction between autonomous vehicles and obstacles on the road, programs that control the route or speed of autonomous vehicles, and programs that control interaction between autonomous vehicles and other autonomous vehicles on the road. . The application program 243 also exists on the system of the software deployment server 249. In one embodiment, the computer system 201 may download the application program from the software deployment server 249 when the automatic driving-related program 247 needs to be executed.

For example, the application program 243 may also be a program for controlling an automatic driving vehicle to perform automatic parking.

Exemplarily, the sensor 253 may be associated with the computer system 201, and the sensor 253 may be used to detect the environment around the computer 201.

For example, the sensor 253 can detect animals, cars, obstacles, and crosswalks. Further, the sensor can also detect the surrounding environment of the above-mentioned animals, cars, obstacles, and crosswalks, such as: the environment around the animals, for example, when the animals appear around them. Other animals, weather conditions, the brightness of the surrounding environment, etc.

Optionally, if the computer 201 is located on a self-driving car, the sensor may be a camera, an infrared sensor, a chemical detector, a microphone, etc.

For example, in an automatic parking scenario, the sensor 253 can be used to detect the size or position of the storage space and surrounding obstacles around the vehicle, so that the vehicle can perceive the distance between the storage space and the surrounding obstacles, and when parking Carry out collision detection to prevent collisions between vehicles and obstacles.

In an example, the computer system 150 shown in FIG. 1 may also receive information from other computer systems or transfer information to other computer systems. Alternatively, the sensor data collected from the sensor system 120 of the vehicle 100 may be transferred to another computer to process the data.

For example, as shown in FIG. 3, data from the computer system 312 may be transmitted to the server 320 on the cloud side via the network for further processing. The network and intermediate nodes can include various configurations and protocols, including the Internet, World Wide Web, Intranet, virtual private network, wide area network, local area network, private network using one or more company’s proprietary communication protocols, Ethernet, WiFi and HTTP, And various combinations of the foregoing; this communication can be by any device capable of transferring data to and from other computers, such as modems and wireless interfaces.

In one example, the server 320 may include a server with multiple computers, such as a load balancing server group, which exchanges information with different nodes of the network for the purpose of receiving, processing, and transmitting data from the computer system 312. The server may be configured similarly to the computer system 312, with a processor 330, a memory 340, instructions 350, and data 360.

Exemplarily, the data 360 of the server 320 may include information related to road conditions around the vehicle. For example, the server 320 may receive, detect, store, update, and transmit information related to the road conditions of the vehicle.

For example, the information related to the road conditions around the vehicle includes information about other vehicles around the vehicle and obstacle information.

At present, the driving area detection of autonomous vehicles can usually adopt monocular vision method or binocular vision method; among them, monocular vision method refers to obtaining images of the road environment through a monocular camera, based on pre-trained deep neural network equipment detection The drivable area in the image; according to the detected drivable area, the principle of plane assumption is used. For example, the autonomous vehicle is in a flat area and there is no slope, etc.; and the drivable area in the image is changed from the two-dimensional pixel coordinate system Down-conversion to the three-dimensional coordinate system where the autonomous vehicle is located to complete the spatial positioning of the drivable area. The binocular vision method refers to obtaining a global disparity map of the road environment by acquiring the images output by the binocular camera separately, detecting the drivable area according to the disparity map, and then dividing the drivable area from the second according to the distance detection feature of the binocular camera. The dimensional pixel coordinate system is converted to the three-dimensional coordinate system where the autonomous vehicle is located to complete the spatial positioning of the drivable area.

However, for the above-mentioned monocular vision method, because the monocular camera cannot perceive the distance, in the process of transforming the drivable area from the pixel coordinate system to the coordinate system of the autonomous vehicle, it is necessary to follow the principle of plane assumption, that is, assume that the vehicle is located. The road surface is completely flat and there is no ramp, etc., which results in low positioning accuracy of the drivable area; for the above-mentioned binocular vision method, the positioning of the drivable area depends on the disparity map, and the calculation of obtaining the global disparity map through the binocular camera is relatively large. As a result, the automatic driving vehicle cannot achieve real-time processing, leading to safety risks in the driving process of the automatic driving vehicle; therefore, how to improve the real-time performance of the detection method of the vehicle's travelable area has become an urgent problem to be solved.

In view of this, the embodiment of the present application provides a method and device for detecting a vehicle drivable area. In the method for detecting a vehicle drivable area in the embodiment of the present application, binocular images of the vehicle traveling direction can be obtained, based on the The binocular image can obtain the boundary of the vehicle's drivable area in the left-eye image and the right-eye image; according to the boundary of the drivable area in the left-eye image and the boundary of the drivable area in the right-eye image, the parallax of the boundary of the drivable area in the binocular image can be obtained Information, based on the disparity information, the position of the vehicle travelable area in the binocular image can be obtained; the method for detecting the vehicle travelable area in the embodiment of the present application can avoid the pixel-by-pixel parallax calculation of the binocular image to obtain the global parallax image, In the embodiment of the present application, there is no need to calculate the global disparity image, and only need to calculate the disparity information of the travelable area boundary in the binocular image to realize the positioning of the self-vehicle coordinate system of the travelable area boundary point. In this case, the amount of calculation is greatly reduced, and the real-time performance of the detection system of autonomous vehicles is improved.

Hereinafter, the method for detecting the travelable area of the vehicle in the embodiment of the present application will be described in detail with reference to FIGS. 4 to 14.

Fig. 4 is a schematic diagram of a detection system for an autonomous vehicle provided by an embodiment of the present application. The detection system 400 may be used to implement a method for detecting a vehicle's drivable area. The detection system 400 may include a perception module 410, a drivable area detection module 420, a drivable area registration module 430, and a coordinate system conversion module 440.

Among them, the perception module 410 can be used to perceive information about the road surface and surrounding environment when the autonomous vehicle is driving; the perception module can include binocular cameras, where the binocular cameras can include left-eye cameras and right-eye cameras, and the binocular cameras can be used for Perceive the environmental information around the vehicle, as the follow-up deep learning network detects the drivable area; the left-eye camera and the right-eye camera can meet the image frame synchronization, which can be achieved through hardware or software. .

For example, in order to ensure that the vehicle-mounted binocular camera can be normally used in a smart driving scene, the baseline distance of the binocular camera can be greater than 30 cm to ensure that it can support the detection of objects of about 100 meters.

The drivable area detection module 420 can be used to detect the drivable area in the pixel coordinate system; the module can be composed of a deep learning network to detect the drivable area in the left-eye image and the right-eye image; the drivable area The input data of the detection module 420 may be the image data collected by the left-eye camera and the right-eye camera included in the above-mentioned perception module 410, and the output data may be the boundary point of the drivable area in the left-eye image and the right-eye image in the pixel coordinate system. The coordinates below.

The drivable area registration module 430 can be used to perform the following five steps at least:

The first step: the left-eye drivable area boundary segmentation processing; that is, in the pixel coordinate system, the boundary points of the left-eye image drivable area extracted above are processed according to the slope transformation, and the continuous drivable area The boundary point is divided into N segments.

The second step: segment matching processing between the boundary points of the left-eye drivable area and the boundary points of the right-eye drivable area; that is, in the pixel coordinate system, different segments are used according to the boundary points of the segmented drivable area The matching strategy of the matching strategy is to match the boundary points of the left-eye drivable area with the boundary points of the right-eye drivable area.

The third step: registration filtering processing of the drivable area; that is, in the pixel coordinate system, the boundary points of the matched drivable area are filtered by a filtering algorithm to ensure the accuracy of the boundary point registration;

The fourth step: the disparity calculation of the boundary points of the drivable area; that is, in the pixel coordinate system, the disparity calculation corresponding to the boundary points of the drivable area is performed according to the registered boundary points of the drivable area;

The fifth step: Parallax sub-pixelation processing; that is, in the pixel coordinate system, the parallax obtained above is sub-pixelated to ensure that the coordinate positioning of the boundary points of the travelable area at a longer distance is still ensured. High positioning accuracy.

Among them, sub-pixel may refer to subdividing two adjacent pixels, which means that each pixel will be divided into smaller units.

The coordinate system conversion module 440 can be used to detect the positioning of the boundary point of the drivable area in the X and Y coordinates. The calculation of the X distance between the boundary point of the drivable area and the self-vehicle can be based on the parallax obtained above, through the triangulation The measurement method or the least square method is used to convert the boundary point from the pixel coordinate system to the X-direction coordinate of the self-car coordinate system; the Y distance calculation from the boundary point of the travelable area to the self-car can be carried out by triangulation based on the calculated X distance The conversion of the boundary point of the drivable area from the pixel coordinate system to the Y coordinate of the vehicle coordinate system.

Illustratively, Figure 5 is a schematic diagram of coordinate conversion; the point P(x, y) on the boundary of the drivable area can be converted from the pixel coordinates of the two-dimensional image to the three-dimensional coordinate system where the autonomous vehicle is located, where , The X direction can represent the forward direction along the location of the autonomous vehicle; the Y direction can represent the left direction along the location of the autonomous vehicle.

It should be understood that the foregoing FIG. 5 is an example of the coordinate system, and does not limit the direction in the coordinate system in any way.

Fig. 6 is a schematic flowchart of a method for detecting a vehicle drivable area provided by an embodiment of the present application. The detection method shown in FIG. 6 may be executed by the vehicle shown in FIG. 1, or the automatic driving system shown in FIG. 2, or the detection system shown in FIG. 4; the detection method shown in FIG. 6 includes steps 510 to 530 , These steps are described in detail below.

Step 510: Obtain a binocular image of the driving direction of the vehicle.

Among them, the above-mentioned binocular images may include left-eye images and right-eye images; for example, it may refer to the left and right two-dimensional images collected by two parallel and equal-height cameras in an autonomous vehicle; The image acquired by the binocular camera shown.

For example, the binocular image may be an image of the road surface or the surrounding environment obtained by the autonomous vehicle in the driving direction; for example, the image of the road surface and the obstacles and pedestrians attached to the vehicle are included.

Step 520: Obtain the disparity information of the drivable area boundary in the binocular image according to the drivable area boundary in the left-eye image and the drivable area boundary in the right-eye image.

It should be noted that the disparity information refers to the information of the direction difference caused by observing the same target from two points with a certain distance. For example, the difference in the horizontal direction between the position of the driveable area on the same road collected by the left-eye camera and the right-eye camera in an autonomous vehicle may be parallax information.

Exemplarily, the obtained left-eye image and right-eye image can be respectively input to a deep learning network pre-trained to recognize the drivable area in the image; the pre-trained deep learning network can identify the left-eye image and the right-eye image. The area where the vehicle can travel.

Step 530: Based on the disparity information, obtain the travelable area of the vehicle in the binocular image.

For example, based on the parallax information of the boundary of the drivable area in the binocular image, the position of the vehicle drivable area in the binocular image can be obtained by triangulation.

Further, in the embodiment of the present application, when the boundary of the drivable area in the left-eye image and the drivable boundary in the right-eye image are matched to calculate the parallax information of the boundary of the drivable area in the binocular image, the binocular image The drivable area boundary in any one of the images in the image is segmented, and the segmented drivable area boundary can be used for segment matching, which can improve the accuracy of the drivable area boundary matching, which is conducive to more accurate road acquisition. Information about the area where the vehicle can travel.

In a possible implementation, the method for detecting the drivable area of the vehicle further includes segmenting the drivable area boundary in the first image in the binocular image; the foregoing step 520 is based on the drivable area in the left eye image The boundary and the boundary of the drivable area in the right eye image to obtain the parallax information of the drivable area boundary in the binocular image may include the second image of the Binocular image based on the N boundary points and line segments obtained by the segmentation process to perform the drivable area boundary Match to obtain disparity information, where N is an integer greater than or equal to 2.

Exemplarily, the foregoing segmentation processing is performed on the boundary of the drivable area in the first image in the binocular image, and the boundary matching of the drivable area in the second image in the binocular image may mean that the boundary of the drivable area in the binocular image The boundary of the drivable area in the left-eye image is segmented, and based on the N boundary point line segments in the left-eye image obtained by the segmentation processing, the drivable area boundary is matched in the right-eye image to obtain disparity information; or, it can also refer to The boundary of the drivable area in the right-eye image in the binocular image is segmented, and the travelable area boundary is matched in the left-eye image based on the N boundary point line segments in the right-eye image obtained by the segmentation process to obtain parallax information.

In the embodiment of the present application, the boundary point line segment can be divided into a boundary point line segment with a smaller slope and a boundary point line segment with a larger slope according to the slopes of the N boundary point line segments after the segmentation process, so as to control the binocular based on the matching strategy. The boundary of the vehicle in the image can be matched, and the disparity information can be obtained according to the matching result.

In a possible implementation manner, the second image is matched with the travelable area boundary based on the N boundary point line segments obtained by the segmentation process to obtain the disparity information, including: matching the second image according to the N boundary point line segments and the matching strategy Perform boundary matching of the drivable area to obtain disparity information, where the matching strategy is determined according to the slopes of the line segments of the N boundary points.

For example, for the first type of boundary point line segment among the N boundary point line segments, the first matching strategy in the matching strategy can be used to match the travelable area boundary of the second image, where the first type of boundary point line segment may include the road edge The boundary of the drivable area and the boundary of the drivable area on the side of other vehicles;

For the second type of boundary point line segment in the N boundary point line segments, the second matching strategy in the matching strategy can be used to match the travelable area boundary of the second image, where the boundary point of the second type of boundary point line segment matches the vehicle The distance is the same.

Optionally, the above-mentioned matching strategy may include a first matching strategy and a second matching strategy. The first matching strategy may refer to matching through a search area, and the search area may refer to any point in a line segment with a first boundary point. As the center generated area, the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy can refer to matching through a preset search step, which is based on N The boundary point parallax of a boundary point line segment in the boundary point line segment is determined.

Exemplarily, for the above-mentioned boundary point line segment with a larger slope, the first image (for example, the left eye image) in the binocular image can be used as a template point by using a boundary point of the travelable area boundary point segment, and the first image The boundary point of the second image (for example, the right eye image) corresponding to the same row of is the center, and the search area is generated to match the template point in the first image. The specific process is shown in Figure 7 below.

Exemplarily, for the boundary point line segment with a small slope, because the boundary point extracted through the first image (for example, the left eye image) will partially overlap with the boundary point extracted through the second image (for example, the right eye image) Phenomenon, the inflection point parallax correction method can be used to match the boundary points of this part. The specific process is shown in Figure 7 below.

Fig. 7 is a schematic flowchart of a method for detecting a vehicle drivable area provided by an embodiment of the present application. The detection method shown in FIG. 7 can be executed by the vehicle shown in FIG. 1, or the automatic driving system shown in FIG. 2, or the detection system shown in FIG. 4; the detection method shown in FIG. 7 includes steps 601 to 614 , These steps are described in detail below.

Step 601: Start; that is, start to execute the detection method of the vehicle's drivable area.

Step 602: Acquire a left eye image.

The left-eye image may refer to an image of the road surface or the surrounding environment acquired by one of the binocular cameras (for example, the left-eye camera).

Step 603: Acquire a right eye image.

The right-eye image may refer to the image of the road surface or the surrounding environment acquired by another camera (for example, the left-eye camera) among the binocular cameras.

It should be noted that the foregoing step 602 and step 603 may be performed at the same time; alternatively, step 603 may be performed first and then step 602 may be performed, which is not limited in this application.

Step 604: Perform a vehicle-driving area detection on the acquired left-eye image.

For example, a deep learning network for identifying a drivable area in an image can be pre-trained through training data; the pre-trained deep learning network can identify a drivable area of a vehicle in the left-eye image.

Exemplarily, a pre-trained deep learning network can be used to detect the drivable area in the pixel coordinate system; the input data can be the collected left-eye image, and the output data can be the detected drivable area on the left. The coordinate vector in the pixel coordinate system in the eye image; based on the output coordinate vector, the coordinates of the drivable area in the pixel coordinate system can be located in the left eye image.

Step 605: Perform a vehicle-driving area detection on the acquired right-eye image.

In the same way, the pre-trained deep learning network in step 604 can be used to detect the drivable area in the right-eye image.

Exemplarily, the obtained left-eye image and right-eye image can be simultaneously input to a pre-trained deep learning network for detection; or, the obtained left-eye image and right-eye image can also be input one after the other to the pre-trained deep learning network. The trained deep learning network can detect the coordinates of the drivable area in the left-eye image and the right-eye image.

Step 606: Perform segmentation processing on the drivable area in the left-eye image.

Exemplarily, the boundary of the drivable area may be segmented according to different slopes of the boundary points of the drivable area detected in the left-eye image.

Illustratively, assuming that the coordinate vector P{p ₁ (x ₁ ,y ₁ ),p ₂ (x ₂ ,y ₂ )..p _n (x _n ,y _n )} of the boundary of the vehicle's drivable area can be passed The following formula calculates the slope of each point:

Among them, K _i can represent the slope of the i-th point; _xi can represent the abscissa of the i-th point in the pixel coordinate system _{; xi-1 can represent the abscissa of the i-1th point in the pixel coordinate system} ; Y _i can represent the ordinate of the i-th point in the pixel coordinate system; y _i-1 can represent the ordinate of the i-1th point in the pixel coordinate system.

Further, through the above formula, the slope vector of each point on the boundary of the vehicle's drivable area can be obtained as K{k ₁ , k ₂ ... k _n }; the inflection point of the boundary of the drivable area is detected according to the jump of the slope; For example, as shown in Figure 8, the inflection points on the boundary of the vehicle's drivable area can be obtained, including point B, point C, point D, point E, point F, and point G; according to the inflection point, the boundary of the vehicle drivable area can be segmented, as shown in the figure As shown in 8, it can be divided into AB section, BC section, CD section, DE section, EF section, FG section and GH section.

Exemplarily, the boundary of the above-mentioned drivable area can be segmented to obtain N segments; the boundary points of the N segments can be classified into two categories according to the slope distribution of the segment, that is, the boundary point line segment with a smaller slope and the boundary with a larger slope Point line segment.

For example, (a) in Figure 9 shows a road surface image (for example, the left-eye image) obtained by one of the binocular cameras. From the image shown in Figure 9 (a), it can be seen that the vehicle can travel The boundary of the area includes point A, point B, point C, and point D. If the detected drivable area boundary is located on the edge of the road or on the side of other vehicles, the slope of the corresponding drivable area boundary in the pixel coordinate system will be larger; the line segment AB included on the drivable area boundary as shown in Figure 8 , Line segment CD, line segment EF, line segment GH; if the detected drivable area boundary is located at the rear of other vehicles, the slope of the corresponding drivable area boundary is smaller, as shown in Figure 8 for the line segment included on the drivable area boundary BC, line segment DE, line segment FG.

Step 607: Segment matching of the boundary of the drivable area.

Among them, the above step 606 illustrates the segmentation of the boundary of the vehicle travelable area detected in the left-eye image; in addition, the boundary point line segment can be divided into the boundary point with a smaller slope according to the slope of the boundary point line segment after the segmentation process Line segments and boundary point line segments with larger slopes; step 607 will perform matching in the right-eye image with the boundary point line segments in the left-eye image.

Exemplarily, in the right-eye image, based on the segmentation result of the drivable area boundary of the right-eye image, the boundary points of the drivable area in the acquired left-eye image may be matched according to different matching strategies.

For example, for the boundary point line segment with a large slope, it may correspond to the road edge in the real scene, or the side of the vehicle, such as the AB line segment and the CD line segment shown in Figure 9 (a); Figure 9 (b) As shown, for the boundary point line segment with a larger slope (for example, the edge of the road), there is basically no overlap in the boundary of the drivable area extracted from the left image and the right eye image.

The first matching strategy: for the above-mentioned boundary point line segment with a larger slope, one boundary point of the left-eye image's travelable area boundary point segment can be used as a template point, and the right-eye image boundary point corresponding to the same row in the left-eye image As the center, the search area is generated to match the template points in the left-eye image.

For example, (a) in FIG. 10 shows a left-eye image; (b) in FIG. 10 shows a schematic diagram of generating a search area in the right-eye image to match template points in the left-eye image.

Exemplarily, the above-mentioned search area may be a search area obtained by an eight-dimensional descriptor; wherein, 360 degrees may be divided into eight equal parts first, that is, 0 degrees to 45 degrees, 45 degrees to 90 degrees, and 90 degrees to 135 degrees. , 135 degrees to 180 degrees, 180 degrees to 225 degrees, 225 degrees to 270 degrees, 270 degrees to 315 degrees, 315 degrees to 360 degrees, the eight regions can represent eight angular distributions; corresponding to the eight-dimensional subsequent generation Descriptor, count the angles of each pixel in the above 5*5 neighborhood, and if the angle falls in the corresponding area, the corresponding area assignment is accumulated in a 1*range manner, and the assignment of each area is calculated to generate 8-dimensional Descriptor of; can be calculated using the following formula.

Among them, S can be represented as a descriptor, Angle can represent the corresponding 8 angle regions, and range can represent the gradient size or amplitude corresponding to each point; through the above formula, the corresponding boundary point of each travelable area can be obtained. Eight-dimensional descriptor, the subsequent matching process is based on the realization of the descriptor.

For example, you can generate a search area with a size of 5*5, match the descriptors of the template points with the descriptors of each point in the search area, and determine the matching points that match the template points, so as to achieve a boundary with a larger slope. Point segment matching.

Among them, each point in the boundary point of the drivable area that can be extracted from the left-eye image is the center, in the same area of the right-eye image corresponding to the left-eye image, for example, in the left-eye image corresponding to a boundary point The descriptor is generated in the 5*5 neighborhood around the right-eye image, and the gradient and angle value of each point in the neighborhood in the right-eye image are calculated using the following formula.

Among them, dx represents the x-direction gradient of the pixel, dy represents the y-direction gradient of the pixel, Angle represents the angle value of the pixel, and range represents the gradient magnitude of the pixel.

It should be noted that the above description is based on the eight-dimensional descriptor generating a search area with a size of 5*5 as an example. The size of the search area can also be other scale sizes, which is not limited in this application.

The second matching strategy: For the boundary point line segment with a small slope, because the boundary point extracted by the right eye image will overlap with the boundary point extracted by the left eye image, the inflection point parallax correction method can be used for this Matching of partial boundary points.

Specifically, the BC line segment shown in FIG. 11, because the slope of the line segment of the boundary point of the drivable area is relatively small, that is, the boundary of the drivable area detected by the left-eye image and the right-eye image may overlap in the line segment of the boundary point. Area; for the adjacent area of the BC line segment, for example, the line segment on the left side of point B and the line segment on the right side of point C are boundary point line segments with a larger slope. Through the above matching strategy one, the boundary points with a larger slope can be matched Matching; in the matching process of the adjacent line segments of the BC line segment, the parallax between the left eye image and the right eye image at point B and C can be obtained, and the parallax between the two points B and C can be used as the basis for the BC line segment to The average value of the above-mentioned two-point disparity is the search length, and the boundary point line segment with the smaller slope in the left-eye image and the right-eye image is matched by the search length.

It should be understood that the above-mentioned boundary point matching through the descriptor is also applicable to the boundary point line segment with a small slope, for example, matching the BC line segment of the travelable area boundary point in the left-eye image and the right-eye image.

It should also be understood that the above is an example of the process of matching the drivable area boundary in the left-eye image with the drivable area boundary in the right-eye image after segmentation processing; in the same way, the acquired right eye The boundary of the drivable area in the image is segmented to match the boundary of the drivable area in the left-eye image, which is not limited in this application.

In the embodiment of the present application, the extracted drivable area boundary of the left-eye image is segmented, and different matching strategies are adopted for the drivable area boundary with a larger slope and a smaller slope. Matching the boundary of the driving area can improve the matching accuracy of the boundary points of the drivable area.

Step 608: Filter the matching result.

Wherein, the matching result filtering may refer to a process of filtering the matching result of step 607, so as to eliminate erroneous matching points.

Exemplarily, for the matched boundary points, there may be a mismatch situation; therefore, the matched boundary point pairs may be filtered in a filtering manner, so as to ensure the matching accuracy of the boundary points of the drivable area.

Further, considering the influence of external factors such as light in a real scene, the robustness of filtering with a fixed threshold is low; therefore, filtering can be performed with a dynamic threshold.

For example, according to the matching degree of the matched boundary points, the matching degree of the boundary points can be sorted, and abnormal matching points can be eliminated according to the method of box plot, and the matching points with higher matching degree can be retained as the final left-eye image and right-eye image. The boundary point where the drivable area is successfully matched in the eye image.

Step 609: Calculate the disparity of the boundary points of the drivable area.

For example, the disparity calculation is performed based on the above-mentioned matched boundary points. For example, the coordinate difference of the matched boundary points in the left-eye image and the right-eye image in the pixel coordinate system corresponding to the boundary point in the Y direction can be calculated.

Step 610: Parallax filtering of boundary points of the drivable area.

In the embodiment of the present application, the pixel points obtained by the disparity calculation in step 609 may be discrete points, and further, the disparity filtering of the boundary points of the drivable area in step 610 may be used to perform continuous operations on the discrete points.

Exemplarily, by using interpolation and compensation methods to perform parallax filtering on the boundary points, the process of parallax filtering can be mainly divided into filtering and interpolation (for example, continuity processing); first, considering that it starts from the bottom of the image, It is the drivable area closest to the self-driving vehicle; starting from the bottom of the image and gradually upward, the parallax corresponding to the boundary points of the matched drivable area should gradually decrease, that is, there may not be an increase in the parallax of the boundary points. Therefore, you can According to this idea, the error parallax is filtered out, that is, if it starts from the bottom of the image and gradually goes up, the deeper the depth of field, the smaller the parallax between the left-eye image and the right-eye image of the travelable area; if it is at the boundary point corresponding to the deeper area If the parallax increases, the boundary point may be a matching deviation point, and the deviation point can be eliminated. Second, considering that in the pixel coordinate system, the distance between the border points corresponding to the same row in the image and the autonomous vehicle is the same (that is, the depth of field can be the same, or it can mean that the coordinates in the Y direction are the same), that is, the left eye image and the right The disparity of the boundary points corresponding to the same row in the eye image should be the same. Therefore, the second filtering can be performed based on this situation, and the boundary points after the filtering process can be continuously corrected by interpolation using the disparity of the adjacent boundary points.

Step 611: Parallax sub-pixelation processing.

Further, in the embodiment of the present application, in order to ensure the accuracy in the calculation of the distance between the binocular images, the extracted boundary point disparity may be subjected to sub-pixelation processing.

Exemplarily, as shown in Figure 12, suppose point b is the coordinate of the matched boundary point, and the gradient values of points a, b, and c are respectively denoted as: g _a , g _b , g _c ; a, b, and c are three The coordinates of the points are respectively recorded as: p _a (x _a , y _a ), p _b (x _b , y _b ), p _c (x _c , y _c ); the following formula can be used to calculate the coordinates of the boundary point after sub-pixel processing :

Among them, M represents the descriptor of sub-pixel processing; N represents the descriptor of sub-pixel processing; y represents the y offset of the image coordinate system.

Step 612: The boundary point of the drivable area is positioned on the X coordinate in the coordinate system of the autonomous vehicle.

For example, based on the obtained parallax of the boundary point of the drivable area, the triangulation method can be used to obtain the distance of the boundary point in the X direction in the vehicle coordinate system.

Step 613: The boundary point of the drivable area is positioned on the Y coordinate in the coordinate system of the autonomous vehicle.

For example, based on the obtained parallax of the boundary point of the drivable area, the triangulation method can be used to obtain the distance of the boundary point in the Y direction in the vehicle coordinate system.

Exemplarily, as shown in FIG. 13, triangulation is used to measure the distances of boundary points in the X direction and Y direction in the coordinate system where the autonomous vehicle is located, where f represents the focal length; B represents the reference line; y represents the image coordinate system The offset in.

According to the triangle similarity relationship, that is, the similarity between △PLR and △PL1R1 can be obtained:

Get the distance in X direction after sorting:

d=uL-uR is the parallax.

As shown in Figure 14, Y is the horizontal distance from the object to the camera; y is the pixel coordinates on the picture; and the distance in the Y direction:

In the embodiment of the present application, through the above steps, without relying on the assumption of the plane, the detection of the drivable area can be realized in time and the drivable area from the pixel coordinate system to the self-car coordinate can be realized through a small amount of calculation. System mapping, thereby improving the real-time detection of autonomous vehicles on the vehicle's drivable area.

FIG. 15 is a schematic diagram of the method for detecting a vehicle travelable area according to an embodiment of the present application applied to a specific product form.

The product form shown in Figure 15 can refer to a vehicle-mounted visual perception device, and a method of detecting the drivable area and positioning the space coordinate can be realized through the software algorithm deployed on the computing node of the related device. Among them, it can mainly include three parts: the first part is to obtain the image, that is, the left-eye image and the right-eye image can be obtained through the left-eye camera and the right-eye camera, and the left-eye camera and the right-eye camera can meet the frame synchronization; the second part is Obtain the drivable area in each image, for example, the deep learning algorithm can output the drivable area (Freespace) in the left-eye image and the right-eye image, or the boundary of the drivable area; the above-mentioned deep learning algorithm can be deployed with AI chips , Can be based on the parallel acceleration processing of multiple AI chips to output Freespace; the third part is to obtain the parallax of the boundary points of the drivable area, for example, based on the serial processing to output the parallax; finally, the disparity of the drivable area can be obtained based on the parallax of the drivable area boundary. The positioning of the coordinate system where the autonomous vehicle is located is to obtain the distance of the drivable area in the X direction and the Y direction based on the boundary parallax.

It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples given, and such modifications or changes also fall within the scope of the embodiments of the present application.

The detection method of the vehicle drivable area of the embodiment of the present application is described in detail above with reference to Figs. 1 to 15, and the device embodiment of the present application will be described in detail below with reference to Figs. 16-17. It should be understood that the detection device for the vehicle travelable area in the embodiment of the present application can execute the various vehicle travelable area detection methods of the foregoing embodiments of the present application, that is, for the specific working process of the following various products, you can refer to the foregoing method embodiment Corresponding process in.

Fig. 16 is a schematic block diagram of a device for detecting a vehicle travelable area provided by an embodiment of the present application. It should be understood that the detection device 700 may execute the method for detecting the drivable area shown in FIGS. 6 to 15. The detection device 700 includes: an acquisition unit 710 and a processing unit 720.

Wherein, the acquiring unit 710 is configured to acquire binocular images of the driving direction of the vehicle, where the binocular images include left-eye images and right-eye images; From the boundary of the drivable area in the image, the disparity information of the boundary of the drivable area in the binocular image is obtained; and based on the disparity information, the drivable area of the vehicle in the binocular image is obtained.

Optionally, as an embodiment, the processing unit 720 is further configured to:

Perform segmentation processing on the travelable area boundary in the first image; perform segmentation processing on the travelable area boundary of the second image based on the N boundary point line segments obtained by the segmentation processing to obtain the disparity information, where N is greater than Or an integer equal to 2.

Optionally, as an embodiment, the processing unit 720 is specifically configured to:

Optionally, as an embodiment, the matching strategy includes a first matching strategy and a second matching strategy, wherein the first matching strategy refers to matching through a search area, and the search area refers to a first boundary. Any one of the dotted line segments is a region generated at the center, the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy refers to a preset search step For matching, the preset search step size is determined based on the boundary point disparity of one boundary point line segment among the N boundary point line segments.

Optionally, as an embodiment, the N boundary point line segments are determined according to inflection points of the boundary of the drivable area in the first image.

It should be noted that the detection device 700 described above is embodied in the form of a functional unit. The term "unit" herein can be implemented in the form of software and/or hardware, which is not specifically limited.

For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.

Therefore, the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

FIG. 17 is a schematic diagram of the hardware structure of a detection device for a vehicle travelable area provided by an embodiment of the present application.

As shown in FIG. 17, the detection apparatus 800 (the detection apparatus 800 may specifically be a computer device) includes a memory 801, a processor 802, a communication interface 803, and a bus 804. Among them, the memory 801, the processor 802, and the communication interface 803 realize the communication connection between each other through the bus 804.

The memory 801 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 801 may store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 is configured to execute each step of the method for detecting a vehicle drivable area in the embodiment of the present application, for example, execute FIGS. 6 to 15 The various steps shown.

It should be understood that the vehicle travelable area detection device shown in the embodiment of the present application may be a server, for example, it may be a cloud server, or may also be a chip configured in a cloud server.

The processor 802 may adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC) or one or more integrated circuits for executing related programs to realize this The method for detecting a vehicle's drivable area in the embodiment of the application method.

The processor 802 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the method for detecting a vehicle travelable area of the present application can be completed by an integrated logic circuit of hardware in the processor 802 or instructions in the form of software.

The above-mentioned processor 802 may also be a general-purpose processor, a digital signal processing (digital signal processing, DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, Discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and, in combination with its hardware, completes the functions required by the units included in the detection device for the vehicle travelable area shown in FIG. 16 in the implementation of this application, or, The method for detecting the vehicle's drivable area shown in FIGS. 6 to 15 of the method embodiment of the present application is executed.

The communication interface 803 uses a transceiver device such as but not limited to a transceiver to implement communication between the detection device 800 and other devices or communication networks.

The bus 804 may include a path for transmitting information between various components of the detection device 800 (for example, the memory 801, the processor 802, and the communication interface 803).

It should be noted that although the foregoing detection device 800 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the detection device 800 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the detection device 800 may also include hardware devices that implement other additional functions.

In addition, those skilled in the art should understand that the detection device 800 described above may also only include the components necessary to implement the embodiments of the present application, and not necessarily include all the components shown in FIG. 17.

It should be understood that the term "and/or" in this text is only an association relationship describing the associated objects, indicating that there can be three types of relationships, for example, A and/or B, which can mean: A alone exists, and both A and B exist. , There are three cases of B alone. In addition, the character "/" in this text generally indicates that the associated objects before and after are in an "or" relationship.

It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A method for detecting a vehicle's drivable area, which is characterized in that it comprises:

Acquiring a binocular image of the driving direction of the vehicle, where the binocular image includes a left-eye image and a right-eye image;

Obtaining the disparity information of the drivable area boundary in the binocular image according to the drivable area boundary in the left-eye image and the drivable area boundary in the right-eye image;

Based on the disparity information, the vehicle-drivable area in the binocular image is obtained.
The detection method of claim 1, further comprising:

Perform segmentation processing on the boundary of the drivable area in the first image;

The obtaining the disparity information of the drivable area boundary in the binocular image according to the drivable area boundary in the left-eye image and the drivable area boundary in the right-eye image includes:

Performing boundary matching of the drivable area on the second image based on the N boundary point line segments obtained by the segmentation process to obtain the disparity information,

Wherein, the first image is any one of the binocular images, the second image is another image in the binocular images that is different from the first image, and N is greater than or equal to 2 Integer.
The detection method according to claim 2, wherein the matching of the drivable area boundary on the second image based on the N boundary point line segments obtained by the segmentation process to obtain the disparity information comprises:

The disparity information is obtained by performing boundary matching of the drivable area on the second image according to the N boundary point line segments and the matching strategy, wherein the matching strategy is determined according to the slopes of the N boundary point line segments.
The detection method according to claim 3, wherein the matching of the travelable area boundary of the second image according to the N boundary point line segments and a matching strategy to obtain the disparity information comprises:

For the boundary point line segment of the first type among the N boundary point line segments, the first matching strategy in the matching strategy is used to perform the drivable area boundary matching on the second image, wherein the boundary point of the first type The line segment includes the boundary of the drivable area on the edge of the road and the boundary of the drivable area on the side of other vehicles;

For the second type of boundary point line segments among the N boundary point line segments, the second matching strategy in the matching strategy is used to perform the driving area boundary matching on the second image, wherein the second type boundary The boundary point in the dotted line segment is the same distance from the vehicle.
The detection method according to claim 3 or 4, wherein the matching strategy includes a first matching strategy and a second matching strategy, wherein the first matching strategy refers to matching through a search area, and the search A region refers to a region generated with any point in the first boundary point line segment as the center, and the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy is Refers to matching through a preset search step, which is determined based on the boundary point disparity of one of the N boundary point line segments.
The detection method according to any one of claims 2 to 5, wherein the N boundary point line segments are determined according to the inflection point of the boundary of the drivable area in the first image.
A device for detecting a vehicle's travelable area is characterized in that it comprises:

An acquiring unit for acquiring a binocular image of a vehicle traveling direction, wherein the binocular image includes a left-eye image and a right-eye image;

The processing unit is configured to obtain the disparity information of the drivable area boundary in the binocular image according to the drivable area boundary in the left-eye image and the drivable area boundary in the right-eye image; and obtain the disparity information based on the disparity information The travelable area of the vehicle in the binocular image.
8. The detection device according to claim 7, wherein the processing unit is further configured to:

Perform segmentation processing on the boundary of the drivable area in the first image;

Performing boundary matching of the drivable area on the second image based on the N boundary point line segments obtained by the segmentation process to obtain the disparity information,

Wherein, the first image is any one of the binocular images, the second image is another image in the binocular images that is different from the first image, and N is greater than or equal to 2 Integer.
The detection device according to claim 8, wherein the processing unit is specifically configured to:

The disparity information is obtained by performing boundary matching of the drivable area on the second image according to the N boundary point line segments and the matching strategy, wherein the matching strategy is determined according to the slopes of the N boundary point line segments.
The detection device according to claim 9, wherein the processing unit is specifically configured to:

For the boundary point line segment of the first type among the N boundary point line segments, the first matching strategy in the matching strategy is used to perform the drivable area boundary matching on the second image, wherein the boundary point of the first type The line segment includes the boundary of the drivable area on the edge of the road and the boundary of the drivable area on the side of other vehicles;

For the second type of boundary point line segments among the N boundary point line segments, the second matching strategy in the matching strategy is used to perform the driving area boundary matching on the second image, wherein the second type boundary The boundary point in the dotted line segment is the same distance from the vehicle.
The detection device according to claim 9 or 10, wherein the matching strategy includes a first matching strategy and a second matching strategy, wherein the first matching strategy refers to matching through a search area, and the search A region refers to a region generated with any point in the first boundary point line segment as the center, and the first boundary point line segment is any one of the N boundary point line segments; the second matching strategy is Refers to matching through a preset search step, which is determined based on the boundary point disparity of one of the N boundary point line segments.
The detection device according to any one of claims 8 to 11, wherein the N boundary point line segments are determined according to the inflection point of the boundary of the drivable area in the first image.
A device for detecting a travelable area of a vehicle, characterized in that it comprises at least one processor and a memory, the at least one processor is coupled with the memory, and is configured to read and execute instructions in the memory to execute such as The detection method according to any one of claims 1 to 6.
A computer-readable medium, wherein the computer-readable medium stores a program code, and when the computer program code runs on a computer, the computer executes the method described in any one of claims 1 to 6 Detection method.