WO2023184051A1

WO2023184051A1 - Intrusion detection method and apparatus, and vehicle

Info

Publication number: WO2023184051A1
Application number: PCT/CN2022/083224
Authority: WO
Inventors: 李华宇; 张峻豪; 李腾; 杨京寰
Original assignee: 华为技术有限公司
Priority date: 2022-03-26
Filing date: 2022-03-26
Publication date: 2023-10-05
Also published as: CN117157681A

Abstract

An intrusion detection method and apparatus, and a vehicle. The method comprises: acquiring image information, which is from a first photographic apparatus (S401); according to the image information, acquiring an area where a target object which is near a vehicle is located and a negative distance intrusion area of the vehicle (S402), wherein the negative distance intrusion area is related to a projection area of the vehicle on the ground; according to the relative position of the area where the target object is located and the negative distance intrusion area, determining whether the target object enters the negative distance intrusion area (S403); and giving an early warning when the target object enters the negative distance intrusion area (S404). In this way, whether a target object enters an area which is related to a projection area of a vehicle on the ground can be detected, thereby improving the accuracy of intrusion detection for the vehicle.

Description

Intrusion detection methods, devices and vehicles

Technical field

The embodiments of the present application relate to the field of smart vehicles, and more specifically, to an infringement detection method, device and vehicle.

Background technique

Infringement detection around vehicles is a safety warning scenario that is usually considered in the automotive field. It is often used in applications such as scratch warning, door opening warning, automatic parking, and sentry mode. Through vehicle perimeter infringement detection, objects around the vehicle can be detected and early warnings can be issued in a timely manner, which can protect the driver's personal safety and property safety. At present, vehicle circumferential violation detection has gradually begun to be applied to various vehicle models.

Currently, infringement warning scenarios usually use visual analysis methods, using sensors such as radar to obtain image data around the vehicle, and use data analysis to determine whether there is an object and issue an early warning. For example, ultrasonic radar is mainly used to identify objects at relatively long distances, short-range ultrasonic radar is generally used to identify objects at a distance of 15 to 250 centimeters (cm), and long-range ultrasonic radar is generally used to identify objects at a distance of 30 to 500 cm. . For objects that are relatively close to the vehicle, since the radar has a low field of view (FOV), especially the vertical FOV is relatively small, there is a certain blind spot in the radar's perception of close distances around the vehicle.

Contents of the invention

Embodiments of the present application provide a vehicle infringement detection method, which can detect whether a target object infringes on a vehicle, thereby improving the accuracy of vehicle infringement detection.

In the first aspect, a method for vehicle infringement detection is provided, including: acquiring image information from a first camera device; obtaining, based on the image information, an area where a target object is located near the vehicle and a negative distance infringement area of the vehicle. The distance violation area is related to the projection area of the vehicle on the ground; determine whether the target object enters the negative distance violation area based on the relative position of the area where the target object is located and the negative distance violation area; and provide an early warning when the target object enters the negative distance violation area.

The vehicle infringement detection solution provided by the embodiment of the present application uses the projection area of the vehicle and the ground to determine the negative distance infringement area, thereby being able to detect whether the target object enters a range relatively close to the vehicle, improving the efficiency of vehicle infringement detection. accuracy.

Alternatively, the negative distance violation area may be an area enclosed by the sides of the vehicle.

In this way, when the target object enters the negative distance violation area, it can be understood that the target object may come into contact with the vehicle and may actually invade the vehicle.

In connection with the first aspect, in some implementations of the first aspect, before acquiring the image information from the first camera device, the method further includes: acquiring the orientation information of the target object and the distance between the target object and the vehicle. Information; turning on at least one camera device according to the orientation information and the distance information, and the at least one camera device includes the first camera device.

The vehicle infringement detection solution provided by the embodiment of the present application can wake up one or more camera devices based on the orientation information of the target object and the distance information between the target object and the vehicle. In other words, the solution of this application can wake up the camera device corresponding to the target object's orientation information and distance information, instead of all camera devices, which can save energy consumption.

Combined with the first aspect, in some implementations of the first aspect, the orientation information and the distance information are determined based on the information collected by the detection device, and the orientation corresponding to the detection range of the detection device corresponds to the detection range of the first camera device The orientations partially or completely coincide with each other; after turning on at least one camera device according to the orientation information and the distance information, the method further includes: closing the detection device.

The vehicle infringement detection solution provided by the embodiment of the present application can turn off the detection device after waking up the camera device based on the orientation information and distance information of the target object determined by the information collected by the detection device. In this way, energy consumption can be saved and unnecessary waste of energy can be avoided.

With reference to the first aspect, in some implementations of the first aspect, the orientation information and the distance information are determined based on the information collected by the second camera device, and the distance between the second camera device and the first camera device is less than or equal to a predetermined value. Set a distance threshold; after turning on at least one camera device according to the orientation information and distance information, the method also includes: when the time when there is no object within the detection range of the second camera device is greater than or equal to the first time threshold, close the second camera device .

The vehicle infringement detection solution provided by the embodiment of the present application can wake up the first camera device based on the orientation information and distance information determined by the information collected by other camera devices (second camera devices) other than the first camera device. In addition, if no object appears within the detection range of the second camera device within a certain period of time, the second camera device can be turned off. In this way, energy consumption can be saved.

On the other hand, it can also be decided whether to turn off the second camera device based on the time when there is no object within the detection range of the second camera device, so as to ensure comprehensive detection of objects near the vehicle, and at the same time, to minimize the frequent opening or closing of the camera device. Life loss caused by shutdown.

Combined with the first aspect, in some implementations of the first aspect, when the target object enters the negative distance violation area, an early warning is provided, including: when the target object enters the negative distance violation area, determining the first characteristic information based on the image information, The first characteristic information includes the duration of movement of the target object in the negative distance violation area and/or the movement trajectory of the target object in the negative distance violation area; an early warning is performed based on the first characteristic information.

The vehicle infringement detection solution provided by the embodiment of the present application can provide an early warning based on the target object's movement trajectory and movement duration in the negative distance infringement area when the target object enters the negative distance infringement area, thereby improving the user experience.

In connection with the first aspect, in some implementations of the first aspect, the negative distance violation area is the projection area of the part of the vehicle on the ground within the detection range of the first camera device.

In connection with the first aspect, in some implementations of the first aspect, the outer edge line of the negative distance violation area is determined based on the outline of the projection area of the vehicle on the ground.

In this way, whether the target object enters the negative distance invasion area can be determined based on the relative position of the outer edge line of the negative distance invasion area and the area where the target object is located. In this way, when determining whether the target object enters the negative distance violation area, it is not necessary to determine all the coordinate sets of the negative distance violation area, which can save data overhead during violation detection.

In conjunction with the first aspect, in some implementations of the first aspect, the inner edge line of the negative distance violation area is determined based on the outline of the projection area of the top curve of the vehicle on the ground.

In a second aspect, a device for vehicle infringement detection is provided, including: an acquisition unit, used to acquire image information from the first camera device; and a processing unit, used to acquire the area where the target object is located near the vehicle based on the image information. and the negative distance violation area of the vehicle. The negative distance violation area is related to the projection area of the vehicle on the ground; the processing unit is also used to determine whether the target object has entered the negative distance based on the relative position of the area where the target object is located and the negative distance violation area. Violation area; the processing unit is also used to provide early warning when the target object enters the negative distance violation area.

At this time, when the target object enters the negative distance violation area, it can be understood that the target object may come into contact with the vehicle and may actually invade the vehicle.

Combined with the second aspect, in some implementations of the second aspect, the acquisition unit is also used to obtain the orientation information of the target object and the distance information between the target object and the vehicle; the processing unit is also used to obtain the orientation information according to the orientation information. and the distance information activate at least one camera device, the at least one camera device including the first camera device.

Combined with the second aspect, in some implementations of the second aspect, the orientation information and the distance information are determined based on the information collected by the detection device, and the orientation corresponding to the detection range of the detection device corresponds to the detection range of the first camera device The orientations partially or completely coincide; the processing unit is also used to turn off the detection device.

In connection with the second aspect, in some implementations of the second aspect, the orientation information and the distance information are determined based on the information collected by the second camera device, and the distance between the second camera device and the first camera device is less than or equal to a predetermined value. Set a distance threshold; the processing unit is also configured to: turn off the second camera device when the time when there is no object within the detection range of the second camera device is greater than or equal to the first time threshold.

In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: when the target object enters the negative distance violation area, determine first feature information based on the image information, where the first feature information includes the target object The duration of movement in the negative distance violation area and/or the movement trajectory of the target object in the negative distance violation area; an early warning is performed based on the first feature information.

Combined with the second aspect, in some implementations of the second aspect, the negative distance violation area is the projection area of the part of the vehicle on the ground within the detection range of the first camera device.

Combined with the second aspect, in some implementations of the second aspect, the outer edge line of the negative distance violation area is determined based on the outline of the projection area of the vehicle on the ground.

Combined with the second aspect, in some implementations of the second aspect, the inner edge line of the negative distance violation area is determined based on the contour line of the projection area of the top curve of the vehicle on the ground.

In a third aspect, a computer-readable medium is provided. The computer-readable medium stores program code. When the computer program code is run on a computer, it causes the computer to perform the method described in any one of the first aspects. .

A fourth aspect provides a chip system, including: a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method described in any one of the above first aspects.

In a fifth aspect, a vehicle infringement detection device is provided, including: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory to execute the above The method according to any one of the first aspects.

A sixth aspect provides a computer program product. The computer product includes: a computer program, which when the computer program is run, causes the computer to execute the method described in any one of the above first aspects.

A seventh aspect provides a vehicle, which includes the device according to any one of the above-mentioned first aspects.

The infringement detection method provided by the embodiment of the present application can determine whether the target object enters an area close to the vehicle (negative distance infringement area) based on the image information from the first camera device and provide an early warning, thereby improving the accuracy of infringement detection. . One or more camera devices are awakened according to the orientation information of the target object and the distance information between the target object and the vehicle. That is, the camera device corresponding to the target object's orientation information and distance information is awakened, instead of all camera devices, to save energy consumption. The first camera device is awakened based on the orientation information and distance information of the target object determined by the information collected by the detection device, and the detection device is turned off after the first camera device is turned on. In this way, energy consumption can be saved and unnecessary waste of energy can be avoided. The first camera is turned on based on the orientation information and distance information determined by the information collected by the second camera, and when there is no object within the detection range of the second camera within a certain period of time, the second camera is turned off. In this way, the target objects near the vehicle can be fully detected as much as possible, while power consumption can be saved. In addition, the life loss caused by frequent opening or closing of the camera device can be reduced as much as possible. Early warning is provided based on the movement trajectory and movement duration of the target object in the negative distance violation area to remind the car owner or driver, which can ensure the personal safety and property safety of the car owner or driver. Whether the target object enters the negative distance invasion area is determined based on the relative position of the outer edge line of the negative distance invasion area and the area where the target object is located. In this way, when determining whether the target object enters the negative distance violation area, it is not necessary to determine all the coordinate sets of the negative distance violation area, which can save data overhead during violation detection.

Description of drawings

FIG. 1 is a functional block diagram of a vehicle 100 to which the embodiment of the present application is applicable.

Figure 2 is a schematic architectural diagram of an intrusion detection system provided by an embodiment of the present application.

FIG. 3 is a schematic structural diagram of a camera device for a vehicle provided by an embodiment of the present application.

Figure 4 is a schematic flowchart of an infringement detection method provided by an embodiment of the present application.

Figure 5 is a schematic diagram of the negative distance invasion area provided by the embodiment of the present application.

Figure 6 is a schematic flowchart of an infringement detection method provided by an embodiment of the present application.

Figure 7 is a schematic flowchart of negative distance infringement warning provided by an embodiment of the present application.

Figure 8 is a schematic diagram of the relative positions of the area where the object is located and the infringement area provided by the embodiment of the present application.

Figure 9 is a schematic diagram of the relative positions of the area where the object is located and the infringement area provided by the embodiment of the present application.

Figure 10 is a schematic diagram of the human-computer interaction interface for infringement warning provided by this application.

Figure 11 is a schematic block diagram of an infringement detection device provided by an embodiment of the present application.

Figure 12 is a schematic block diagram of an infringement detection device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

Various subsystems may be included in vehicle 100, such as travel system 110, perception system 120, display device 130, and computing platform 140.

Alternatively, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. Additionally, each subsystem and element of vehicle 100 may be interconnected via wires or wirelessly.

Illustratively, travel system 110 may include components for providing powered motion to vehicle 100 . In one embodiment, travel system 110 may include engine 111, transmission 112, energy source 113, and wheels/tires 114. The engine 111 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines; for example, a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine. Engine 111 may convert energy source 113 into mechanical energy.

By way of example, energy source 113 may include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 113 may also provide energy for other systems of vehicle 100 .

By way of example, transmission 112 may include a gearbox, a differential, and a drive shaft; wherein transmission 112 may transmit mechanical power from engine 111 to wheels 114 .

In one embodiment, the transmission device 112 may also include other components, such as a clutch. Among other things, the drive shaft may include one or more axles that may be coupled to one or more wheels 114 .

Illustratively, the sensing system 120 may include several sensors that sense information about the environment surrounding the vehicle 100 .

For example, the sensing system 120 may include a positioning system 121 (such as a global positioning system (GPS), Beidou system or other positioning systems), an inertial measurement unit (IMU) 122, a lidar 123, a millimeter wave Radar 124, ultrasonic radar 125 and camera device 126. Sensing system 120 may also include sensors that monitor internal systems of vehicle 100 (eg, interior air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding properties (position, shape, orientation, speed, etc.). This detection and identification is a critical function for the safe operation of the autonomous vehicle 100 .

Positioning system 121 may be used to estimate the geographic location of vehicle 100 .

The inertial measurement unit 122 is used to sense position and orientation changes of the vehicle 100 based on inertial acceleration. In some embodiments, inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

LiDAR 123 may utilize laser light to sense objects in the environment in which vehicle 100 is located. In some embodiments, lidar 123 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.

Millimeter wave radar 124 may utilize radio signals to sense objects within the surrounding environment of vehicle 100 . In some embodiments, in addition to sensing objects, radar 126 may be used to sense the speed and/or heading of the object.

The ultrasonic radar 125 can sense objects around the vehicle 100 using ultrasonic signals.

The camera device 126 may be used to capture image information of the surrounding environment of the vehicle 100 . The camera device 126 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information acquired by the camera device 126 may include still images or video stream information.

As shown in FIG. 1 , the vehicle 100 can interact with the user through the display device 130 .

For example, information may be provided to a user of vehicle 100 via display device 130 .

Some or all functions of vehicle 100 may be controlled by computing platform 140 . The computing platform 140 may include processors 141 to 14n (n is a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor may be a circuit with instruction reading and execution capabilities. For example, central processing unit (CPU), microprocessor, graphics processing unit (GPU) (can be understood as a microprocessor), or digital signal processor (DSP), etc. ; In another implementation, the processor can realize certain functions through the logical relationship of the hardware circuit. The logical relationship of the hardware circuit is fixed or can be reconstructed. For example, the processor is an application-specific integrated circuit (application-specific integrated circuit). Hardware circuits implemented by ASIC) or programmable logic device (PLD), such as off-the-shelf programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading the configuration file and realizing the hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing Unit (deep learning processing unit, DPU), etc. In addition, the computing platform 140 may also include a memory, which is used to store instructions. Some or all of the processors 141 to 14n may call instructions in the memory to execute the quality to implement corresponding functions.

Computing platform 140 may control functionality of vehicle 100 based on input received from various subsystems (eg, perception system 120 ). In some embodiments, computing platform 140 is operable to provide control of many aspects of vehicle 100 and its subsystems.

Optionally, the above components are just examples. In actual applications, the components in each of the above modules may be added or deleted according to actual needs. Figure 1 should not be understood as limiting the embodiments of the present application.

An autonomous vehicle traveling on the road, such as vehicle 100 above, can identify objects within its surrounding environment to determine adjustments to its current speed. The objects may be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently and based on the object's respective characteristics, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine the speed to which the autonomous vehicle will adjust.

Optionally, the vehicle 100 or a sensing and computing device associated with the vehicle 100 (eg, computing platform 140 ) may determine the behavior based on the characteristics of the identified object and the state of the surrounding environment (eg, traffic, rain, ice on the road, etc. etc.) to predict the behavior of the identified object. Alternatively, each recognized object depends on the behavior of each other, so it is also possible to predict the behavior of a single recognized object by considering all recognized objects together. The vehicle 100 is able to adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle is able to determine what stable state the vehicle will need to adjust to (eg, accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors may also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road on which it is traveling, the curvature of the road, the proximity of static and dynamic objects, and so on.

In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device may also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a given trajectory and/or maintains contact with objects in the vicinity of the autonomous vehicle (e.g., , the safe lateral and longitudinal distance between cars in adjacent lanes on the road).

The above-mentioned vehicle 100 may be a car, a truck, a motorcycle, a public vehicle, a boat, an airplane, a helicopter, a lawnmower, an entertainment vehicle, a playground vehicle, construction equipment, a tram, a golf cart, a train, etc., in the embodiment of the present application No special restrictions are made.

Infringement detection around vehicles is a safety warning scenario commonly considered in the automotive field, such as scratch warning, door opening warning, sentry mode, etc. These can protect the driver's personal and property safety while the vehicle is driving or parked.

Currently, the most common vehicle perimeter infringement warning is based on sensors such as radar. For example, ultrasonic radar is mainly used to identify objects at a longer distance, short-range ultrasonic radar is generally used to identify objects at a distance of 15 to 250cm, and long-range ultrasonic radar is generally used to identify objects at a distance of 30 to 500cm. For objects that are relatively close to the vehicle, since the radar has a low FOV, especially the vertical FOV is relatively small, there is a certain blind spot in the radar's perception at close range around the vehicle, and it is unable to detect whether the object has violated the vehicle body.

Especially when most areas of the object are far away from the vehicle body (for example, beyond 20cm), and only partially touch the vehicle, or when part of the object is relatively close to the vehicle body (for example, within 20cm), it is impossible to accurately detect whether part of the object has violated the vehicle. For example, when a person stands 20cm away from the vehicle and only extends his arm and is about to touch or has touched the vehicle, the vehicle cannot effectively identify the violation and issue an early warning, and cannot ensure vehicle safety.

In response to the above problems, this application proposes a vehicle infringement detection method, which can detect whether an object has violated the vehicle and provide an early warning, thereby improving the accuracy of vehicle infringement detection.

Figure 2 shows a schematic architectural diagram of an infringement detection system provided by an embodiment of the present application. The vehicle detection system of FIG. 2 can be applied to the vehicle 100 of FIG. 1 .

The infringement detection system may include an object segmentation module 210, an infringement detection module 220, a camera adaptive scheduling module 230, and a hierarchical early warning module 240. Among them, the above-mentioned object segmentation module 210, infringement detection module 220, camera adaptive scheduling module 230 and hierarchical early warning module 240 can be implemented by one or more modules respectively, and the above-mentioned modules can also be implemented by the same module.

The object segmentation module 210 may include a video stream acquisition module 211 and an object segmentation module 212. The object segmentation module 212 may determine the precise object segmentation area based on the data collection by the video stream collection module 211 . When segmenting objects, you can add ground segmentation to remove shadows, ghosts, ghosts and other problems, and you can also add a multi-source data fusion step to compensate for long-range errors and insufficient information.

The violation detection module 220 may include a violation area mapping module 221, a close range violation detection module 222, a negative distance violation detection module 223, and a vehicle body detection module 224. Among them, the violation area may include a close range violation area and a negative distance violation area.

The violation area mapping module 221 can map the violation area of the world coordinate system into the image, and the vehicle body detection module 224 can obtain the vehicle body segmentation area based on the image and determine the outer edge line of the negative distance violation area. The close range infringement detection module 222 and the negative distance infringement detection module 223 can determine whether the object enters the negative distance infringement area based on the relative position of the area where the object is located in the image coordinate system and the infringement area.

The camera adaptive scheduling module 230 may include a multi-source data fusion module 231, a path following module 232, and a camera adaptive scheduling module 233. The multi-source data fusion module 231 can fuse one or more data such as radar signal data, video image data, infrared data, vibration sensor data, vehicle movement speed data, etc. The path following module 232 can determine the movement path of the object and predict the object's future action direction, and send this information to the camera adaptive scheduling module 233 . The camera adaptive scheduling module 233 can turn on or off the corresponding camera based on this information.

For example, if no object appears within the detection range of the camera within a certain period of time, the camera adaptive scheduling module 233 can control to turn off the camera. After the camera is turned off, the sensor corresponding to the camera will be turned on. When an object appears, a corresponding location detection device (such as a radar sensor) can detect the location of the object and report the information to the path following module 233 . When the distance between the object and the vehicle is less than or equal to a certain distance, the camera adaptive scheduling module 233 will turn on the camera corresponding to the corresponding orientation of the sensor detection range.

The hierarchical warning module 240 can present warning information in different forms on the vehicle side, user terminal side, etc. Early warning information can include different levels, for example, it can be divided into low-level warning, medium-level warning and high-level warning. The hierarchical early warning module can trigger different levels of alarms on the vehicle side and/or the user terminal side, and transmit different levels of early warning information through the human-computer interaction interface, such as the movement trajectory of the violation target, distance from the vehicle body, relative position to the vehicle body, etc. Presented to users in different ways. The hierarchical warning module can also store warning information according to different warning levels.

The module of the infringement detection system can be implemented in the form of a processor calling software. For example, the infringement detection system includes a processor. The processor is connected to a memory. Instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above. Method or implement the functions of each unit of the device, where the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the module of the infringement detection system can be implemented in the form of a hardware circuit, and some or all of the unit functions can be implemented through the design of the hardware circuit, which can be understood as one or more processors; for example, in one implementation , the hardware circuit is an ASIC, which realizes the functions of some or all of the above units through the design of the logical relationship of the components in the circuit; as another example, in another implementation, the hardware circuit can be realized by PLD, taking FPGA as an example, It can include a large number of logic gate circuits, and the connection relationships between the logic gate circuits are configured through configuration files, thereby realizing the functions of some or all of the above units. All modules of the above infringement detection system can be fully implemented in the form of software called by the processor, or fully implemented in the form of hardware circuits, or partially implemented in the form of software called by the processor, and the remaining part is implemented in the form of hardware circuits.

FIG. 3 is a schematic structural diagram of a camera device for a vehicle provided by an embodiment of the present application. Violation detection of vehicles can be realized through the camera device in Figure 3 . The camera device of FIG. 3 can be applied to the vehicle 100 of FIG. 1 .

Figure 3 depicts the number and installation positions of the camera devices. The number and position of the camera devices in the embodiment of the present application are only examples and are not limiting.

By way of example, a vehicle may include multiple camera devices. For example, it may include 1 front camera, 1 left camera, 1 right camera, 1 rear camera, 1 front left and 1 front right camera.

In this embodiment of the present application, multiple camera devices in the vehicle may include a first camera device and a second camera device, and the distance between the second camera device and the first camera device is less than or equal to a preset distance threshold. The vehicle can determine the orientation information of the target object and the distance information between the target object and the vehicle based on the second camera device, and then turn on the first camera device based on the orientation information and distance information. The vehicle can determine whether there is a target object near the vehicle and the positional relationship between the area where the target object is near the vehicle and the negative distance violation area, and determine whether the target object enters the negative distance violation area based on the image information from the first camera device.

Alternatively, the camera device may be a linear camera. Alternatively, it can also be a non-linear camera, such as a fisheye camera.

Alternatively, the camera device may be a color camera. Alternatively, it could be a depth camera.

Optionally, the FOV of the camera device may be greater than or equal to 90°. The FOV of the camera device can also be greater than or equal to other thresholds, such as 120°, 110°, 100°, 80°, etc.

The embodiments of the present application do not limit the type of the camera device. By adjusting the installation position, installation angle and quantity of the camera device, the camera device can detect the negative distance violation area around the vehicle as comprehensively as possible.

Figure 4 is a schematic flowchart of a vehicle infringement detection method provided by an embodiment of the present application. The vehicle infringement detection method in Figure 4 may be executed by the vehicle, or may be executed by a computing platform, or may be executed by a system composed of a computing platform and a camera device. In one embodiment, when the method is performed by a computing platform, the computing platform may be located in the vehicle, such as computing platform 140 shown in FIG. 1 . Alternatively, the computing platform can be located in a cloud server. The vehicle collects image information through the camera and sends the image information to the cloud server. The cloud server can determine whether the target object enters the negative distance violation area based on the image information sent by the vehicle. When the cloud server determines that the target object has entered the negative distance violation area, it can send instruction information to the vehicle. The instruction information is used to instruct the target object to enter the negative distance violation area. The method includes:

S401. Obtain image information from the first camera device.

The plurality of camera devices in FIG. 3 may include a first camera device.

S402: According to the image information, obtain the area where the target object is located near the vehicle and the negative distance violation area of the vehicle.

It should be understood that the area involved in the embodiments of the present application may be a three-dimensional space. For the convenience of description, the embodiments of the present application do not distinguish between areas and spaces, but use areas (for example, negative distance violation areas, areas where target objects are located) for description. For example, a space can be represented as a projected area.

In some embodiments, the area where the target object is located may be a set of coordinates in the image coordinate system of the projection area of the target object and the ground.

The target object can be moving or stationary. The vehicle can be in motion or stationary. This application does not limit this.

In some embodiments, image segmentation technology can be used to obtain the area where the target object is located, such as frame search algorithm, neural network model, etc.

In some embodiments, a multi-source data fusion module can be used to analyze data from different perspectives, for example, one or more data such as radar signal data, video image data, infrared data, vibration sensor data, vehicle speed data, etc. Perform fusion analysis to obtain the area where the target object is located.

In some embodiments, the negative range violation area is related to the projected area of the vehicle on the ground. FIG. 5 is a schematic diagram of the negative distance encroachment area provided by the embodiment of the present application. The negative distance encroachment area will be briefly introduced below with reference to FIG. 5 .

In some embodiments, the negative distance violation area may be the projected area between the vehicle and the ground. As shown in (a) of Figure 5 , the projected area of the vehicle and the ground can be understood as the area surrounded by the contour lines of the projected area of the vehicle and the ground.

In some embodiments, the negative distance violation area may be within the projected area of the vehicle and the ground.

For example, as shown in (b) of Figure 5 , the annular area enclosed by the outline of the vehicle's projection area on the ground and the outline of the roof line's projection on the ground can be used as a negative distance violation area.

For example, the negative distance violation area may also be within an annular area surrounded by the outline of the vehicle's projected area on the ground and the outline of the roof line's projection on the ground.

For example, the negative distance violation area may be a projection area on the ground of a body part of the vehicle within the detection range of the first camera device. Figure 5(c) is an image captured by the camera device on the left rearview mirror of the vehicle. The camera device detects the negative distance violation area within the detection range, that is, the negative distance violation area in Figure 5(d) (top view) .

When the negative distance violation area can be within the projection area of the vehicle and the ground, the negative distance violation area can be determined based on the inner edge line and the outer edge line of the negative distance violation area. For example, the outer edge line of the negative distance violation area is determined based on the outline of the projected area between the vehicle and the ground. The inner edge line of the negative distance violation area can be determined based on the outline of the roof line and the ground projection area. Within the detection range of the first camera device, the contour line projected on the ground outside the vehicle body part within the detection range of the first camera device can be used as the outer edge line of the negative distance violation area; the first camera device can be The roof line of this part of the car body within the detection range, the outline of the projected area on the ground as the negative distance violates the inner edge line of the area.

In some embodiments, the negative distance violation area may be associated with an area bounded by bodywork around the vehicle. Alternatively, it can be understood that the negative distance violation area is related to the area enclosed by the vehicle body on the sides of the vehicle (front, rear, left, and right).

In some embodiments, the negative distance violation area may be an area surrounded by the body of the vehicle. Alternatively, it can be understood that the negative distance violation area is the area surrounded by the vehicle body on the sides of the vehicle (front, rear, left, and right). At this time, when the target object enters the negative distance violation area, it can be considered that the target object may come into contact with the vehicle and may actually invade the vehicle.

The negative distance violation area may be a set of coordinates in a world coordinate system preset based on the vehicle model. The vehicle body detection module can obtain the vehicle body segmentation area according to the image information from the first camera device, and then map the coordinate set of the negative distance violation area in the world coordinate system to the image coordinate system according to the vehicle body segmentation area, thereby obtaining the image coordinate system. The negative distance violates the area.

In the image coordinate system, the shape of the negative distance invasion area can be determined based on different parameters such as the installation position, installation angle, and vehicle type of the camera device. This application is only used as an example and is not limited.

S403: Determine whether the target object enters the negative distance violation area based on the relative position of the area where the target object is located and the negative distance violation area.

In some embodiments, whether the target object enters the negative distance violation area can be determined based on the intersection ratio between the area where the target object is located and the negative distance violation area. For example, it can be determined according to formula (1) whether the target object enters the negative distance violation area. When the intersection ratio is greater than or equal to the preset threshold, the target object enters the negative distance violation area; when the intersection ratio is less than the preset threshold, the target object does not enter. Negative distance violates the area.

J _L1 =F((∑L1∩L2)/∑L2,I1) Formula (1)

J _L1 is the output result of whether the target object enters the negative distance violation area, F is the comparison function, L1 is the coordinate set of the area where the target object is located in the image coordinate system, L2 is the coordinate set of the negative distance violation area in the image coordinate system, I1 is For the first preset threshold, / is the division symbol, Σ is the summation symbol, and ∩ is the intersection symbol. Among them, L1∩L2 is the intersection of the coordinate set of the area where the target object is located and the coordinate set of the negative distance violation area, ∑L1∩L2 is the number of coordinate points of L1∩L2, and ∑L2 is the number of coordinate points of L2.

In some embodiments, whether the target object enters the negative distance violation area may be determined based on the relative position of the area where the target object is located and the outer edge line of the negative distance violation area. In this way, by determining the coordinate set of the outer edge line of the negative distance violation area and the coordinate set of the area where the target object is located through the vehicle body segmentation area, it can be determined whether the target object has entered the negative distance violation area. For example, whether the target object enters the negative distance violation area can be determined based on the intersection ratio between the area where the target object is located and the outer edge line of the negative distance violation area, as shown in formula (2). When the intersection ratio is greater than or equal to the preset threshold, the target object enters the negative distance violation area; when the intersection ratio is less than the preset threshold, the target object does not enter the negative distance violation area.

J _S =F((∑L1∩S)/∑S,I2) Formula (2)

J _S is the output result of whether the target object enters the negative distance violation area, F is the comparison function, L1 is the coordinate set of the area where the target object is located in the image coordinate system, and S is the coordinate of the outer edge line of the negative distance violation area in the image coordinates Set, I2 is the second preset threshold, / is the division symbol, ∑ is the summation symbol, and ∩ is the intersection symbol. Among them, L1∩S is the intersection of the coordinate set of the area where the target object is located and the coordinate set of the outer edge of the negative distance violation area, ∑L1∩S is the number of coordinate points of L1∩S, and ∑S is the number of coordinate points of S.

S404: Provide an early warning when the target object enters the negative distance violation area.

In some embodiments, when the target object enters the negative range violation area, a negative distance violation warning is performed.

In some embodiments, when the target object enters the negative distance violation area, the first characteristic information may be determined based on the image information, and the first characteristic information may include the movement trajectory and/or movement duration information of the target object in the negative distance violation area. An early warning can then be issued based on the movement trajectory and/or movement duration of the target object in the negative distance violation area. For example, early warning can be divided into low-level early warning, medium-level early warning, high-level early warning, etc. The longer the movement trajectory and/or the longer the movement duration of the target object in the negative distance violation area, the higher the early warning level.

Figure 6 is a schematic flowchart of a vehicle infringement detection method provided by an embodiment of the present application.

The vehicle infringement detection method in Figure 6 may be executed by the vehicle, or may be executed by a computing platform, or may be executed by a system composed of a computing platform and a camera device. In one embodiment, when the method is performed by a computing platform, the computing platform may be located in the vehicle, such as computing platform 140 shown in FIG. 1 . Alternatively, the computing platform can be located in a cloud server. The vehicle collects information through the camera (second camera device) or detection device and sends it to the cloud server. The cloud server can determine the orientation information and distance information of the target object based on the information sent by the vehicle, and open the orientation of the target object based on the orientation information and distance information. Corresponding camera (including the first camera device). The vehicle collects image information through a camera (first camera device) and sends the image information to the cloud server. The cloud server can determine whether the target object enters the negative distance violation area based on the image information sent by the vehicle. When the cloud server determines that the target object has entered the negative distance violation area, it can send instruction information to the vehicle. The instruction information is used to instruct the target object to enter the negative distance violation area. The method includes:

S601: Obtain the orientation information of the target object and the distance information between the target object and the vehicle.

In some embodiments, the orientation information of the target object and the distance information between the target object and the vehicle can be obtained based on information collected by a detection device (such as a radar). For example, when a target object appears at a certain position far away from the vehicle (for example, 60 cm away), the detection device corresponding to the position can first detect the target object. Based on the information collected by the detection device, the orientation information and distance information of the target object can be obtained.

In some implementations, the orientation information of the target object and the distance information between the target object and the vehicle may be obtained according to the second camera device. When the target object is at a certain position close to the vehicle (for example, within 60 cm), the position information and distance information of the target object can be obtained through the information collected by the second camera device corresponding to the position.

S602: Turn on at least one camera device based on the orientation information and distance information.

In some embodiments, when the target object appears at a certain position far away from the vehicle, the detection device corresponding to the position may first detect the target object. When the distance between the target object and the vehicle is less than or equal to the target distance, at least one camera device can be turned on, wherein the at least one camera device includes a first camera device, and the orientation corresponding to the detection range of the detection device is consistent with the detection range of the first camera device. The corresponding orientations partially or completely overlap. For example, the orientation information of the target object may only correspond to the detection range of one camera device. When the target object appears in this orientation, a camera device may be turned on, and the camera device may be the first camera device. The orientation information of the target object may correspond to the detection ranges of multiple camera devices. When the target object appears in the orientation, multiple camera devices may be turned on. The multiple camera devices may include the first camera device.

After at least one camera device is turned on, the detection devices corresponding to the directions corresponding to the detection ranges of these camera devices can be turned off to save energy consumption.

In some embodiments, when the target object is relatively close to the vehicle, and in the process of moving from the orientation corresponding to the detection range of the second camera device to the orientation corresponding to the detection range of the first camera device, the image collected by the second camera device can be The information determines that the target object is about to enter or has entered the position corresponding to the first camera device, and then turns on the first camera device. After turning on the first camera device, when no object (including the target object and other objects) appears within the detection range of the second camera device within a certain period of time, the second camera device can be turned off.

Within a preset time after the second camera device is turned off, the detection device corresponding to the orientation corresponding to the detection range of the second camera device can be turned on.

In some implementations, the distance between the second camera device and the first camera device is less than or equal to the preset distance threshold. For example, the camera devices around the vehicle are camera device 1, camera device 2, camera device 3, camera device 4, camera device 5, and camera device 6 in order. For example, the first camera device may be camera device 1, and the second camera device may be camera device 2 or camera 6. The distance between the first camera device and the second camera device is less than or equal to the preset distance threshold.

When the first camera device is in an open state, the first camera device may not be turned on repeatedly.

S603. Obtain image information from the first camera device.

S604: Obtain the area where the target object is near the vehicle and the violation area of the vehicle based on the image information.

In some embodiments, the violation area may include a close range violation area and a negative range violation area.

In some embodiments, the close range violation area may be determined based on an area outside the projected area of the vehicle and the ground. For example, the inner edge line of the close range infringement area can be the outer edge line of the negative distance infringement area, and the distance between the outer edge line of the close range infringement area and the inner edge line of the close range infringement area can be a certain distance (such as 25cm) .

In some embodiments, both the short-distance violation area and the negative-distance violation area can be a set of coordinates in a preset world coordinate system, and the coordinates of the short-distance violation area and the negative-distance violation area in the world coordinate system are calculated according to the image information. The collection is mapped into the image coordinate system. The coordinate set of the violation area in the world coordinate system can be converted into the coordinate set of the violation area in the image coordinate system based on one or more parameters such as the correction coefficient rotation matrix, translation matrix, and distortion coefficient of the camera device.

For example, formula (3) can be used to obtain the coordinate set L2 of the negative distance violation area in the image coordinate system.

in,

is the coordinate set of the negative distance violation area in the world coordinate system, R is the camera correction coefficient rotation matrix, and T is the translation matrix.

For example, formula (4) can be used to obtain the coordinate set L3 of the close-range violation area in the image coordinate system.

in,

is the coordinate set of the close-range violation area in the world coordinate system, R is the camera correction coefficient rotation matrix, and T is the translation matrix.

In some embodiments, the vehicle body segmentation area L4 (X4, Y4) can be obtained from the image information through the vehicle body detection module, and the coordinate set S (Xs, Ys) of the outer edge line of the negative distance violation area is determined based on the vehicle body segmentation area. For example, in the left scene, formula (5) can be used to obtain the coordinate set S(Xs, Ys) of the outer edge line of the negative distance violation area.

X _S =max{X,(X,Ys)∈L4} Formula (5)

Among them, L4 is the coordinate set of the body segmentation area, (X, Ys) is the coordinate set of the body segmentation area when Y is equal to Ys, and Ys is the ordinate of any point on L4. max is the maximum value symbol, and ∈ is the belonging symbol.

S605: Determine whether the target object enters the violation area based on the relative position of the area where the target object is located and the violation area.

In some embodiments, whether the target object enters the negative distance violation area can also be determined based on the relative position of the area where the target object is located and the negative distance violation area. For the determination method, please refer to the relevant description in S403, which will not be described in detail here in this application.

In some embodiments, whether the target object enters the near-infringement area can be determined based on the relative position of the area where the target object is located and the close-infringement area.

For example, the intersection ratio can be used to determine whether the target object enters the close range violation area. When the intersection ratio is greater than or equal to the preset threshold, the target object enters the close range invasion area; when the intersection ratio is less than the preset threshold, the target object does not enter the close range invasion area. Whether the target object enters the close range violation area can be determined according to formula (6).

J _L3 =F((∑L1∩L3)/∑L3,I3) Formula (6)

Among them, J _L3 is the output result of whether the target object enters the close range violation area, F is the comparison function, I3 is the third preset threshold, L1 is the coordinate set of the area where the target object is located in the image coordinate system, and L3 is the image coordinate The set of coordinates of the close range violation area in the system. / is the division symbol, ∑ is the summation symbol, and ∩ is the intersection symbol. Among them, L1∩L3 is the intersection of the coordinate set of the area where the target object is located and the coordinate set of the close-range violation area, ∑L1∩L3 is the number of coordinate points of L1∩L3, and L3 is the number of coordinate points of L3.

S606: Provide an early warning when the target object enters the violation area.

In some embodiments, when the target object enters the close range violation area, a close range violation warning is performed.

In some embodiments, when the target object enters the near-infringement area, information such as the movement trajectory and/or movement duration of the target object in the near-intrusion area can also be obtained based on the image information, and then the target object's movement in the near-intrusion area can be obtained based on the image information. Use information such as the movement trajectory and/or movement duration to provide early warning of close-range infringement. For example, early warnings may include low-level warnings, medium-level warnings, and high-level warnings. The longer the movement trajectory of the target object in the near violation area, and/or the longer the movement duration of the target object in the near violation area, the higher the level of the near violation warning.

In some embodiments, when the target object enters the negative distance violation area, information such as the movement trajectory and/or movement duration of the target object in the negative distance violation area can also be obtained based on the image information, and then based on the target object's movement in the negative distance violation area Negative distance violation warning based on information such as movement trajectory and/or movement duration. For example, early warnings may include low-level warnings, medium-level warnings, and high-level warnings. The longer the movement trajectory of the target object in the negative distance violation area, and/or the longer the movement duration of the target object in the negative distance violation area, the higher the level of the negative distance violation warning.

Figure 7 shows a schematic flow chart of the negative distance infringement warning provided by the embodiment of the present application. The negative distance infringement early warning involved in the embodiment of the present application will be introduced below with reference to Figure 7 . The negative distance violation early warning methods include S701 to S704, which are introduced respectively below.

S701: Obtain characteristic information of the target object.

In some embodiments, the first feature information can be obtained based on the image information. The first feature information includes the movement trajectory of the target object in the negative distance violation area and/or the movement duration of the target object in the negative distance violation area, etc.

In some embodiments, the movement trajectory of the target object in the negative distance violation area can be represented by length. For example, the target object is within the negative distance violation area and moves clockwise from the first position on the left side of the vehicle to the second position on the rear side of the vehicle. The movement trajectory is D1; the target object is within the negative distance violation area and moves clockwise from the left side of the vehicle. The first position moves clockwise to the third position on the right side of the vehicle, and the motion trajectory can be D2. Then, the length of the motion trajectory D2 is greater than the length of the motion trajectory D1. For another example, the target object is within the negative distance violation area and moves clockwise from the first position on the left side of the vehicle to the second position on the rear side of the vehicle, and back and forth once. The movement trajectory is D3; the target object is within the negative distance violation area. From the first position on the left side of the vehicle to the second position on the rear side of the vehicle, and back and forth twice, the movement trajectory is D4. Then, the length of the motion trajectory D4 is greater than the length of the motion trajectory D3.

In some embodiments, the maintenance time of the target object in the negative distance violation area may include a static maintenance time and a movement maintenance time. The movement maintenance time represents the movement duration of the target object in the negative distance violation area, and the static maintenance time represents the target object. The duration of inactivity in the negative range violation zone.

In some implementations, the first characteristic information may also include one or more of the following: the length of time the target object remains stationary in the negative distance violation area, the number of times the target object violates the negative distance violation area, and the distance between the target object and the vehicle. , the running speed of the target object in the negative distance violation area, the number of target objects in the negative distance violation area and the type of the target object, whether the target object is a registered person of the vehicle, etc.

In some implementations, second characteristic information can also be obtained. The second characteristic information can be one or more of the following: the running speed of the vehicle, the complexity of the environment in which the vehicle is located, etc.

The categories and specific descriptions of the first characteristic information and the second characteristic information are as shown in Table 1.

Table 1

S702: Determine the early warning coefficient according to the characteristic information of the target object.

In some implementations, one or more of the first feature information and/or the second feature information can be fused to obtain an early warning coefficient.

For example, a2, a3, a6, and b2 are continuous feature information. One or more of the continuous feature information can be normalized and weighted to obtain the continuous violation feature value _PC , as shown in formula (7) Show.

P _C =∑[r _C,n f _C,n (C _n )] Formula (7)

Among them, f _C,n (*) is the vector mapping function, C _n is the value of continuous feature information, r _C,n is the weight of continuous feature information, and ∑ is the summation symbol.

For example, a1, a4, a5, a7, a8, a9, b1 are discrete feature information. One or more of the discrete feature information can be weighted to obtain the discontinuous violation feature value P _D , as shown in formula (8) shown.

P _D =∑[r _D,n g _D,n (D _n )] Formula (8)

Among them, g _D,n (*) is the normalization function, D _n is the value of discontinuous feature information, r _D,n is the weight of continuous feature information, and ∑ is the summation symbol.

For example, the continuous intrusion characteristic value P _C and the discontinuous infringement characteristic value P _D can be added together according to a certain weight to obtain the early warning coefficient P.

S703: Determine the early warning level based on the relationship between the early warning coefficient and the early warning threshold.

In some implementations, the warning levels may include low-level warning, medium-level warning, and high-level warning.

For example, two early warning thresholds Y ₁ and Y ₂ can be set (Y ₁ < Y ₂ ). When P ≤ Y ₁ , a low-level early warning is carried out; when Y ₁ < P < Y ₂ , a medium-level early warning is carried out; when P > Y ₂ , a high-level early warning is carried out. The greater the length of the target object's movement trajectory in the negative distance violation area, the greater the early warning coefficient and the higher the early warning level. The longer the movement duration of the target object in the negative distance violation area, the greater the early warning coefficient and the higher the early warning level.

S704, perform early warning according to the early warning level.

Different levels of early warning information can be delivered to users in different forms, allowing users to know the early warning level and the infringement information of the target object.

For example, a low-level warning can be used on the vehicle side and/or the user terminal side to remind the driver of the presence of a target intrusion in the form of a normal notification ringtone (such as a short ringtone), and on the human-computer interaction interface to remind the driver in the form of pictures and text. It can also store the data, pictures and/or videos of the target object invading the vehicle according to the level.

For example, mid-level early warning can remind the driver of target intrusion in the form of important notification ringtones (such as long-lasting ringtones) and vibrations on the vehicle side and/or the user terminal side, and in the form of pictures and text on the human-computer interaction interface. Notify the driver of the warning level, the target object's movement trajectory, violation area, relative position to the vehicle body and other information. At the same time, the data, pictures and/or videos of the target object violating the vehicle can also be stored according to the level.

For example, advanced warning can warn the driver of target intrusion in the form of alarm sounds and vibrations on the vehicle side and/or user terminal side, and on the human-computer interaction interface in the form of pictures and text in conspicuous colors such as red. Information such as early warning level, movement trajectory of the target object, violation area, relative position to the vehicle body, etc. It can also store data, pictures and/or video levels of the target object violating the vehicle. In addition, warnings can be given through lights inside and outside the car.

FIG. 8 shows a schematic diagram of the relative positions of the area where the target object is located and the vehicle periphery infringement area provided by the embodiment of the present application.

Taking the image information obtained by the camera device on the left rearview mirror of the vehicle as an example, based on the relative position of the area where the target object is located and the violation area around the vehicle, it is judged whether the target object has entered the close range violation area ABCD and the negative distance violation area ABGF of the vehicle. .

For example, the preset threshold I1 can be set to 0, and then it can be determined whether the target object enters the negative distance violation area based on whether there is an intersection between the area where the target object is located and the negative distance violation area. When the area where the target object is located intersects with the negative distance violation area, the target object enters the negative distance violation area; when the area where the target object is located does not intersect with the negative distance violation area, the target object does not enter the negative distance violation area.

For example, the preset threshold I3 can be set to 0, and then it can be determined whether the target object enters the near-infringement area based on whether there is an intersection between the area where the target object is located and the close-infringement area. When the area where the target object is located intersects with the near-infringement area, the target object enters the close-infringement area; when there is no intersection between the area where the target object is and the near-infringement area, the target object does not enter the close-infringement area.

As shown in (a) of Figure 8, at time T1, when the target object is far away from the vehicle (for example, greater than 60 cm), the detection device such as radar detects the presence of the target object. When the distance between the target object and the vehicle is less than or equal to the target distance, the left camera device can be turned on. At time T1, the area where the target object is located has no intersection with the close-range invasion area ABCD, and the target object has not entered the close-range invasion area. At time T1, the area where the target object is located has no intersection with the negative distance violation area ABGF, and it can be considered that the target object has not entered the negative distance violation area.

As shown in (b) in Figure 8, at time T2, the target object gradually approaches the vehicle. At T2, the area where the target object is located intersects with the close-range violation area ABCD, and it can be considered that the target object has entered the close-range violation area. At time T2, the target object area has no intersection with the negative distance violation area ABGF, and it can be considered that the target object has not entered the negative distance violation area.

As shown in (c) in Figure 8, at time T3, the target object continues to approach the vehicle, and part of the target object enters the negative distance violation area. At T3, the area where the target object is located intersects with the close-range invasion area ABCD, and it can be considered that the target object has entered the close-range invasion area. At time T3, the target object area intersects with the negative distance violation area ABGF, and it can be considered that the target object enters the negative distance violation area.

FIG. 9 shows a schematic diagram of the relative positions of the area where the target object is located and the area violated by the vehicle provided by the embodiment of the present application.

Taking the image information of the camera device on the rear side of the vehicle as an example, based on the relative position of the area where the target object is located and the violation area, it is determined whether the target object has entered the close range violation area MNJK and the negative distance violation area MNOQ of the vehicle.

The target moves away from the left side of the vehicle and gradually approaches the rear side of the vehicle. When the target object approaches the rear side of the vehicle, and the distance between the target object and the vehicle is always smaller than the target distance, the left camera device can detect whether the target object is about to enter or has entered the detection range of the rear camera device. When the target object is about to enter or has entered the detection range of the rear camera device, the rear camera device is turned on.

For example, the first preset threshold I1 can be set to 0, and then it can be determined whether the target object enters the negative distance violation area based on whether there is an intersection between the area where the target object is located and the negative distance violation area. When the area where the target object is located intersects with the negative distance violation area, the target object enters the negative distance violation area; when the area where the target object is located does not intersect with the negative distance violation area, the target object does not enter the negative distance violation area.

For example, the third preset threshold I3 can be set to 0, and then it can be determined whether the target object enters the near-infringement area based on whether there is an intersection between the area where the target object is located and the close-infringement area. When the area where the target object is located intersects with the near-infringement area, the target object enters the close-infringement area; when there is no intersection between the area where the target object is and the near-infringement area, the target object does not enter the close-infringement area.

As shown in (a) in Figure 9, at time T4, the vehicle appears behind the vehicle. At time T4, the area where the target object is located has no intersection with the close range invasion area MNJK, and the target object has not entered the close range invasion area. At time T4, the area where the target object is located has no intersection with the negative distance violation area MNOQ, and the target object does not enter the negative distance violation area.

As shown in (b) in Figure 9, at time T5, the target object continues to approach the vehicle. At time T5, the area where the target object is located intersects with the close range invasion area MNJK, and the target object enters the close range invasion area. At time T5, the area where the target object is located has no intersection with the negative distance violation area MNOQ, and the target object does not enter the negative distance violation area.

As shown in (c) in Figure 9, at time T6, the target object continues to approach the vehicle. At time T6, the area where the target object is located intersects with the close range invasion area MNJK, and the target object enters the close range invasion area. At time T6, the area where the target object is located intersects with the negative distance violation area MNOQ, and the target object enters the close range violation area.

The human-computer interaction interface can display the current status of warning information, and you can also view historical warning information.

For example, when a target object enters the violation area, the human-computer interaction interface can display key frame photos of the target object when the violation occurred, and can also display the area and location of the violation by the target object, and inform the target object of the level of violation. For example, during the advanced warning of negative distance infringement, the human-computer interaction interface can display key frame photos of the target object's violation, and can also display the warning level, the target object's movement trajectory, the target object's area and location, etc. in conspicuous colors such as red. information.

For example, historical infringement transactions can also be queried through the human-computer interaction interface. For example, you can query historical transactions by warning level, or you can query historical violation transactions by violation type, or you can query historical violation transactions by time.

Embodiments of the present application also provide a device for implementing any of the above methods. For example, a device is provided that includes units (or means) for implementing each step performed by a vehicle in any of the above methods.

Figure 11 is a schematic block diagram of a vehicle intrusion detection device according to an embodiment of the present application. The device 4000 shown in FIG. 11 includes an acquisition unit 4010 and a processing unit 4020.

The acquisition unit 4010 and the processing unit 4020 may be used to perform the infringement detection method in the embodiment of the present application.

The acquisition unit 4010 is used to acquire image information from the first camera device.

The processing unit 4020 is configured to obtain the area where the target object is located near the vehicle and the negative distance violation area of the vehicle based on the image information. The negative distance violation area is related to the projection area of the vehicle on the ground.

The processing unit 4020 is configured to determine whether the target object enters the negative distance violation area based on the relative position of the area where the target object is located and the negative distance violation area.

The processing unit 4020 is also used to issue an early warning when the target object enters the negative distance violation area.

Optionally, as an embodiment, the acquisition unit 4010 is also configured to acquire the orientation information of the target object and the distance information between the target object and the vehicle; the processing unit 4020 is configured to turn on at least one camera device according to the orientation information and distance information. , at least one camera device includes a first camera device.

Optionally, as an embodiment, the orientation information and distance information are determined based on the information collected by the detection device, and the orientation corresponding to the detection range of the detection device partially or completely coincides with the orientation corresponding to the detection range of the first camera device. ; After turning on at least one camera device according to the orientation information and distance information, the processing unit 4020 is also used to turn off the detection device.

Optionally, as an embodiment, the orientation information and distance information are determined based on the information collected by the second camera device, and the distance between the second camera device and the first camera device is less than or equal to the preset distance threshold; based on the orientation information and distance information. After turning on at least one camera device, the processing unit is also configured to: turn off the second camera device when there is no object within the detection range of the second camera device for a time greater than or equal to the first time threshold.

Optionally, as an embodiment, the processing unit 4020 is also configured to: when the target object enters the negative distance violation area, determine first feature information based on the image information, where the first feature information includes the movement trajectory of the target object in the negative distance violation area. And/or the duration of movement of the target object in the negative distance violation area; an early warning is performed based on the first characteristic information.

Optionally, as an embodiment, the outer edge line of the negative distance violation area is determined based on the outline of the projection area of the vehicle on the ground.

Optionally, as an embodiment, it is characterized in that the inner edge line of the negative distance violation area is determined according to the contour line of the projection area of the top curve of the vehicle on the ground.

Optionally, as an embodiment, the negative distance violation area is the projection area of part of the vehicle on the ground within the detection range of the first camera device.

It should be understood that the division of each unit in the above-mentioned device 4000 is only a division of logical functions. In actual implementation, all or part of the units may be integrated into one physical entity, or may be physically separated. In addition, the unit in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, instructions are stored in the memory, and the processor calls the instructions stored in the memory to implement any of the above methods. Or realize the functions of each unit of the device, where the processor is, for example, a general-purpose processor, such as a CPU or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and some or all of the functions of the units can be implemented through the design of the hardware circuits, which can be understood as one or more processors; for example, in one implementation, The hardware circuit is an ASIC, which realizes the functions of some or all of the above units through the design of the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented through PLD, taking FPGA as an example. It can include a large number of logic gate circuits, and the connection relationships between the logic gate circuits can be configured through configuration files to realize the functions of some or all of the above units. All units of the above device may be fully realized by the processor calling software, or may be fully realized by hardware circuits, or part of the units may be realized by the processor calling software, and the remaining part may be realized by hardware circuits.

In the embodiment of the present application, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, and a GPU (which can Understood as a microprocessor), or DSP, etc.; in another implementation, the processor can achieve certain functions through the logical relationship of the hardware circuit. The logical relationship of the hardware circuit is fixed or can be reconstructed, such as processing The processor is a hardware circuit implemented by an application-specific integrated circuit ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading the configuration file and realizing the hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as NPU, TPU, DPU, etc.

It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA , or a combination of at least two of these processor forms.

In addition, each unit in the above device may be integrated together in whole or in part, or may be implemented independently. In one implementation, these units are integrated together and implemented as a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The at least one processor may be of different types, such as a CPU and an FPGA, or a CPU and an artificial intelligence processor. CPU and GPU etc.

Figure 12 is a schematic diagram of the hardware structure of a vehicle intrusion detection device provided by an embodiment of the present application. The vehicle infringement detection device 5000 shown in Figure 12 includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. Among them, the memory 5001, the processor 5002, and the communication interface 5003 implement communication connections between each other through the bus 5004.

The memory 5001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 is used to execute various steps of the vehicle infringement detection method according to the embodiment of the present application.

The processor 5002 can use a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits to execute relevant programs to implement the vehicle infringement detection method of the method embodiment of the present application.

The processor 5002 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the vehicle infringement detection method of the present application can be completed by instructions in the form of hardware integrated logic circuits or software in the processor 5002 .

The above-mentioned processor 5002 can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. Each method, step and logical block diagram disclosed in the embodiment of this application can be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 5001. The processor 5002 reads the information in the memory 5001, and combines its hardware to complete the functions required to be performed by the units included in the device shown in Figure 12, or to perform vehicle infringement according to the method embodiment of the present application. Detection method.

The communication interface 5003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 5000 and other devices or communication networks. For example, image information can be obtained through the communication interface 5003.

Bus 5004 may include a path that carries information between various components of device 5000 (eg, memory 5001, processor 5002, communication interface 5003).

Embodiments of the present application also provide a computer-readable medium that stores program code for device execution. The program code includes a method for performing vehicle infringement detection in the embodiment of the present application.

An embodiment of the present application also provides a computer program product containing instructions. When the computer program product is run on a computer, it causes the computer to execute the vehicle intrusion detection method in the embodiment of the present application.

An embodiment of the present application also provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the vehicle infringement detection method in the embodiment of the present application.

Optionally, as an implementation manner, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the present application. The vehicle infringement detection method in the embodiment.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

It should be understood that the term "and/or" in this article is only an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, and A and B exist simultaneously. , there are three cases of B alone, where A and B can be singular or plural. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship, but it may also indicate an "and/or" relationship. For details, please refer to the previous and later contexts for understanding.

In this application, "at least one" refers to one or more, and "plurality" refers to two or more. "At least one of the following" or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items). For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

It should be understood that in the various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

A vehicle infringement detection method is characterized by including:

Obtain image information from the first camera device;

According to the image information, the area where the target object is located near the vehicle and the negative distance violation area of the vehicle are obtained, and the negative distance violation area is related to the projection area of the vehicle on the ground;

Determine whether the target object enters the negative distance violation area based on the relative position of the area where the target object is located and the negative distance violation area;

When the target object enters the negative distance violation area, an early warning is provided.
The method of claim 1, wherein before acquiring the image information from the first camera device, the method further includes:

Obtain the orientation information of the target object and the distance information between the target object and the vehicle;

At least one camera device is activated based on the orientation information and the distance information, and the at least one camera device includes the first camera device.
The method of claim 2, wherein the azimuth information and the distance information are determined based on information collected by a detection device, and the azimuth corresponding to the detection range of the detection device is consistent with the first camera device. The orientations corresponding to the detection ranges partially or completely overlap;

After turning on at least one camera device according to the orientation information and the distance information, the method further includes:

Turn off the detection device.
The method of claim 2, wherein the orientation information and the distance information are determined based on information collected by a second camera device, and the distance between the second camera device and the first camera device is The distance is less than or equal to the preset distance threshold;

After turning on at least one camera device according to the orientation information and the distance information, the method further includes:

When the time when there is no object within the detection range of the second camera is greater than or equal to the first time threshold, the second camera is turned off.
The method according to any one of claims 1 to 4, characterized in that, when the target object enters the negative distance violation area, an early warning is provided, including:

When the target object enters the negative distance violation area, first characteristic information is determined according to the image information, and the first characteristic information includes the movement duration and/or the movement duration of the target object in the negative distance violation area. The movement trajectory of the target object in the negative distance violation area;

An early warning is performed based on the first characteristic information.
The method according to any one of claims 1 to 5, characterized in that the negative distance violation area is the projection area of the part of the vehicle on the ground within the detection range of the first camera device.
The method according to any one of claims 1 to 6, characterized in that the outer edge line of the negative distance violation area is determined based on the outline of the projection area of the vehicle on the ground.
The method according to any one of claims 1 to 7, characterized in that the inner edge line of the negative distance violation area is determined based on the outline of the projection area of the top curve of the vehicle on the ground.
A vehicle infringement detection device, characterized by including:

An acquisition unit acquires image information from the first camera device;

A processing unit configured to obtain, based on the image information, the area where the target object is located near the vehicle and the negative distance violation area of the vehicle, where the negative distance violation area is related to the projection area of the vehicle on the ground. ;

The processing unit is configured to determine whether the target object enters the negative distance violation area based on the relative position of the area where the target object is located and the negative distance violation area;

The processing unit is also configured to issue an early warning when the target object enters the negative distance violation area.
The device according to claim 9, characterized in that it includes:

The acquisition unit is also used to acquire the orientation information of the target object and the distance information between the target object and the vehicle;

The processing unit is further configured to turn on at least one camera device according to the orientation information and the distance information, and the at least one camera device includes the first camera device.
The device of claim 10, wherein the azimuth information and the distance information are determined based on information collected by a detection device, and the azimuth corresponding to the detection range of the detection device is consistent with the first camera device. The orientations corresponding to the detection ranges partially or completely overlap;

The processing unit is also configured to turn off the detection device.
The device of claim 10, wherein the orientation information and the distance information are determined based on information collected by a second camera device, and the distance between the second camera device and the first camera device is The distance is less than or equal to the preset distance threshold;

The processing unit is also configured to: turn off the second camera device when the time when there is no object within the detection range of the second camera device is greater than or equal to the first time threshold.
The device according to any one of claims 9 to 12, characterized in that the processing unit is also used for:

When the target object enters the negative distance violation area, first characteristic information is determined according to the image information, and the first characteristic information includes the movement duration and/or the movement duration of the target object in the negative distance violation area. The movement trajectory of the target object in the negative distance violation area;

An early warning is performed based on the first characteristic information.
The device according to any one of claims 9 to 13, wherein the negative distance violation area is the projection area of the part of the vehicle on the ground within the detection range of the first camera device.
The device according to any one of claims 9 to 14, wherein the outer edge line of the negative distance violation area is determined based on the contour line of the projection area of the vehicle on the ground.
The device according to any one of claims 9 to 15, characterized in that the inner edge line of the negative distance violation area is determined based on the contour line of the projection area of the top curve of the vehicle on the ground.
A computer-readable medium, characterized in that the computer-readable medium stores program code. When the computer program code is run on a computer, it causes the computer to execute the method as claimed in any one of claims 1 to 8. method described.
A chip system, characterized in that it includes: at least one processor and a memory, the at least one processor is coupled to the memory, and is used to read and execute instructions in the memory to execute the instructions of claims 1 to 1 The method described in any one of 8.
A vehicle infringement detection device, characterized by comprising: at least one processor and a memory, the at least one processor being coupled to the memory and configured to read and execute instructions in the memory to execute the rights The method according to any one of claims 1 to 8.
A computer program product, characterized in that the computer product includes: a computer program, which when the computer program is run, causes the computer to perform the method according to any one of claims 1 to 8.
A vehicle, characterized by comprising the device according to any one of claims 9 to 16 and 19.