CN118015559A

CN118015559A - Object identification method and device, electronic equipment and storage medium

Info

Publication number: CN118015559A
Application number: CN202410096205.8A
Authority: CN
Inventors: 吴函; 李冬冬; 李乾坤; 侯壮; 高存璋; 王凯
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2024-01-23
Filing date: 2024-01-23
Publication date: 2024-05-10

Abstract

The application relates to the technical field of data processing, in particular to an object identification method, an object identification device, electronic equipment and a storage medium, which are used for improving the accuracy of object identification. The method comprises the following steps: respectively acquiring spatial attribute information of each radar object and each visual object detected in the current detection period by radar and video acquisition equipment; matching each radar object with each visual object based on the detected position information, and determining at least one fusion object and spatial attribute information of the at least one fusion object based on a matching result; and updating the current object set based on the at least one piece of fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period. According to the application, the radar and the object detected by the video acquisition equipment are fused, so that the obtained position information of the fused object is more accurate and stable, and the accuracy of object identification is improved.

Description

Object identification method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to an object identification method, an object identification device, an electronic device, and a storage medium.

Background

In the field of security protection, monitoring devices based on radar and video acquisition devices are becoming more and more important. Radar and video acquisition devices, while both can be used for target detection, are quite different in their applicable scenarios.

The radar can accurately acquire the space position information and the movement speed information of the target, can detect the target at a distance, is not influenced by environments such as illumination, but cannot acquire the target identification information with high accuracy, and cannot accurately detect the target when the target is in a static state.

The video acquisition equipment can obtain target identification information with high accuracy, can accurately detect all targets in a middle and close range, but cannot easily and accurately obtain the space position information and the movement speed information of the targets, has poor detection effect on the targets at a distance, and is easily influenced by environments such as poor illumination, heavy fog weather and the like.

In view of the advantages and limitations of target detection of the radar and the video acquisition equipment, how to effectively fuse the detection data of the radar and the video acquisition equipment, so as to improve the accuracy of object identification in a detection area, and the method is a problem to be solved in the present day.

Disclosure of Invention

The embodiment of the application provides an object identification method, an object identification device, electronic equipment and a storage medium, which are used for improving the accuracy of object identification.

The object identification method provided by the embodiment of the application comprises the following steps:

respectively acquiring spatial attribute information of each radar object and each visual object detected in a current detection period by radar and video acquisition equipment, wherein the spatial attribute information at least comprises position information;

Matching each radar object with each visual object based on each detected position information, and determining at least one fusion object and spatial attribute information of the at least one fusion object based on a matching result, wherein the spatial attribute information of the fusion object at least comprises fusion position information;

updating the current object set based on the obtained at least one fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period, wherein each candidate object is obtained based on the fusion object in at least one previous detection period.

Optionally, the matching result includes: at least one of the presence of a first radar object that matches the visual object, the presence of a second radar object that does not match the visual object, and the presence of an initial visual object that does not match the radar object;

The determining at least one fusion object based on the matching result includes:

if the matching result comprises the presence of the first radar object, determining a first fusion object based on the first radar object and a visual object to which the first radar object is matched;

If the matching result comprises the existence of the second radar object, determining a second fusion object based on the second radar object;

If the matching result includes the presence of the initial visual object, a third fusion object is determined based on the initial visual object.

Optionally, the fused position information of the first fused object is determined based on the position information of the first radar object; the fusion position information of the second fusion object is determined based on the position information of the second radar object; the fused position information of the third fused object is determined based on the position information of the initial visual object.

Optionally, the matching the radar objects with the visual objects based on the detected position information, and determining at least one fusion object based on the matching result, includes:

Obtaining respective association degrees of object groups based on the position information, wherein each object group comprises a radar object and/or a visual object, and the association degrees are obtained based on the position information of the radar object and/or the position information of the visual object contained in the corresponding object group;

Combining the object groups to obtain a plurality of object group sets, wherein each object group set comprises the radar objects and the visual objects, and no coincident radar objects or visual objects exist in one object group set;

Determining the at least one fusion object based on a target object group set of the plurality of object group sets, the target object group set being: and the object group sets with minimum correlation degree sum of the object groups contained in the plurality of object group sets.

Optionally, the determining the at least one fusion object based on the target object group set in the plurality of object group sets includes at least one of:

determining a first fusion object based on an object group comprising a radar object and a visual object in object groups comprising the target object group set;

determining a second fusion object based on an object group only containing radar objects in the object groups contained in the target object group set;

And obtaining a third fusion object based on the object group only containing the visual object in the object groups contained in the target object group set.

Alternatively, each degree of association is obtained by:

for the plurality of object groups, the following operations are performed respectively:

When one object group contains a radar object and a visual object, obtaining a degree of association of the one object group based on a distance between position information of the radar object contained in the one object group and position information of the visual object;

when the one object group includes only the radar object or the visual object, the first association degree is regarded as the association degree of the one object group.

Optionally, the method further comprises:

And when the distance is larger than a preset distance threshold value, taking the second association degree as the association degree of the object group.

Optionally, the method further comprises:

And when the detection period does not exist before the current detection period, respectively adding the at least one fusion object to the current object set as a candidate object.

An object recognition device provided by an embodiment of the present application includes:

the acquisition unit is used for respectively acquiring the radar and the video acquisition equipment, and each spatial attribute information of each radar object and each visual object detected in the current detection period at least comprises position information;

The matching unit is used for matching each radar object with each visual object based on the detected position information, and determining at least one fusion object and the spatial attribute information of the at least one fusion object based on a matching result, wherein the spatial attribute information of the fusion object at least comprises fusion position information;

And the updating unit is used for updating the current object set based on the obtained at least one fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period, wherein each candidate object is obtained based on the fusion object in at least one previous detection period.

The matching unit is specifically configured to:

Optionally, the spatial attribute information of the radar object further includes speed information, and the spatial attribute information of the visual object further includes speed information and boundary information in the target video frame; the target video frame is a video frame acquired by the video acquisition object in the current detection period; the spatial attribute information of the fusion object also comprises fusion speed information and fusion boundary information;

The matching unit is further configured to:

Acquiring fusion speed information and fusion boundary information of the first fusion object based on the speed information of the first radar object and the boundary information of the visual object matched with the first radar object;

acquiring fusion speed information and fusion boundary information of the second fusion object based on the speed information and the position information of the second radar object;

and obtaining the fusion speed information and the fusion boundary information of the third fusion object based on the speed information and the boundary information of the initial visual object.

Optionally, the matching unit is specifically configured to:

Optionally, the matching unit is specifically configured to obtain each association degree by:

Optionally, the matching unit is specifically configured to determine the fusion object by at least one of the following ways:

The optional matching unit is further configured to:

Optionally, the updating unit is further configured to obtain each predicted position information by:

For each candidate object, the following operations are respectively executed:

based on the position information and the speed information of one candidate object in the last detection period and the time interval between the last detection period and the current detection period, the predicted position information of the one candidate object is obtained.

Optionally, the updating unit is specifically configured to:

matching the at least one fusion object with each candidate object based on the at least one fusion position information and each predicted position information;

updating the spatial attribute information of the corresponding candidate object based on the spatial attribute information of the fusion object matched to the candidate object, wherein the spatial attribute information of the candidate object is obtained based on the spatial attribute information of the fusion object matched in the previous detection period;

And adding the fusion object which is not matched with the candidate object to the current object set as the candidate object.

Optionally, the updating unit is specifically configured to:

determining a decoupling object in the current object set, wherein the decoupling object is a candidate object which is not matched with a fusion object in the candidate objects;

and removing the disconnection object from the current object set when the number of times that the disconnection object cannot be matched with the fusion object is greater than a preset number of times threshold.

Optionally, the apparatus further comprises an adding unit for:

An electronic device provided in an embodiment of the present application includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, causes the processor to execute any one of the steps of the object recognition method described above.

An embodiment of the present application provides a computer-readable storage medium including a computer program for causing an electronic device to execute the steps of any one of the above-described object recognition methods when the computer program is run on the electronic device.

Embodiments of the present application provide a computer program product comprising a computer program stored in a computer readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program so that the electronic device performs the steps of any one of the object recognition methods described above.

The application has the following beneficial effects:

the embodiment of the application provides an object identification method, an object identification device, electronic equipment and a storage medium, wherein in a current detection period, all radar objects detected by a radar are matched with all visual objects detected by video acquisition equipment, at least one fusion object and corresponding fusion position information are determined, and the current object set is updated based on the obtained at least one fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period. The radar object and the visual object are fused to obtain the fusion object, the detection results of the radar and the video acquisition equipment are fully utilized, the spatial attribute information of the fusion object can be obtained more stably, the stability and the accuracy of object identification are improved, the condition of object loss in the object identification process is reduced, the dependence on the high precision of the radar or the video acquisition equipment is reduced, the equipment cost is reduced, and the object set is updated through fusion position information, so that the spatial attribute information of the object is updated in real time.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a schematic diagram of the target detection of the front and rear frames in radar video monitoring after calibration;

FIG. 2 is an alternative schematic diagram of an application scenario in an embodiment of the present application;

FIG. 3 is a flowchart of an object recognition method according to an embodiment of the present application;

FIG. 4 is a schematic diagram of the detection principle of a radar object and a visual object in an embodiment of the present application;

FIG. 5 is a flow chart of matching fusion objects with candidate objects in an embodiment of the application;

FIG. 6 is a schematic overall flow chart of an object recognition method according to an embodiment of the application;

FIG. 7 is a schematic diagram of an object recognition device according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a hardware configuration of an electronic device to which embodiments of the present application are applied;

Fig. 9 is a schematic diagram of a hardware composition structure of another electronic device to which the embodiment of the present application is applied.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the technical solutions of the present application, but not all embodiments. All other embodiments, based on the embodiments described in the present document, which can be obtained by a person skilled in the art without any creative effort, are within the scope of protection of the technical solutions of the present application.

In the description of the present application, "a plurality of" means "at least two". "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. A is connected with B, and can be represented as follows: both cases of direct connection of A and B and connection of A and B through C. In addition, in the description of the present application, the words "first," "second," and the like are used merely for distinguishing between the descriptions and not be construed as indicating or implying a relative importance or order.

In addition, in the technical scheme of the application, the data is collected, transmitted, used and the like, and all meet the requirements of national relevant laws and regulations.

In the field of security protection, monitoring equipment based on radar and video all-in-one machines is increasingly valued. Radar and vision sensors, while both can detect objects, are quite different in the applicable scenario of both sensors. The radar has the advantages that the radar can accurately acquire the space information and the movement speed information of the target, can detect the target at a distance, is not influenced by environments such as illumination and the like, and has the defects that the target identification information with high accuracy cannot be obtained and all the targets cannot be accurately detected when the vehicle is stationary. The visual sensor has the advantages that the target identification information with high accuracy can be obtained, and all targets can be accurately detected in the video in a middle-near distance. The method has the defects that accurate movement speed information and spatial position information of the target are not easy to obtain, the detection effect on the target at a distance is poor, and the target is easily influenced by environments such as poor illumination, heavy fog weather and the like.

The traditional security terminal equipment mainly comprises video acquisition equipment, such as a visible light camera, wherein the visible light camera has the advantages of obtaining category information of a target, but not easily obtaining operation information of the target. For example, the type of object (man-machine-not-machine) is easily identified, but the speed and spatial position of the object are not easily estimated. The accuracy of identifying the target in the long-distance, night and rainy and foggy days is lower than that of the target in the short-distance, namely the time-space accuracy of the target in the monitoring area is inconsistent (even has a significant difference). The millimeter wave Radar actively emits electromagnetic waves and receives signals with the same frequency, has very high detection probability for moving objects or objects with larger Radar Cross-Section (RCS), and has lower detection probability (the detection probability is not zero) for stationary objects. Millimeter wave radars can operate for 24 hours throughout the day and are less affected by weather. Therefore, the two are combined with each other, so that the effects of 1+1 being more than 2 can be obtained by taking the advantages of the two as the complement: the target detection recognition probability is further improved, and the target distance, azimuth and speed information can be obtained. At present, the demand for monitoring products based on radar video integrated machines is vigorous in the market.

The millimeter wave radar can obtain measurement information of a target, including: distance, angle, radial velocity (RADIAL SPEED), RCS. It should be noted that the normal millimeter wave radar detection target is a clustered region, but for decision-level target fusion, the radar target (radar object) is just a clustered point, and the visual target is a rectangular region in the image. When the target is positioned, the visual target (visual object) can not directly obtain the distance information of the target, but after the radar coordinate system and the visual coordinate system are calibrated, the mapping position (namely the distance) of the visual target under the radar coordinate system can be obtained, under the normal calibration condition, the radar measurement position can only accurately describe the whole position of one target, and when no shielding exists nearby, the visual mapping position can describe the distance (such as the head, the tail and the license plate) of one specific part of the target in detail.

In the field of surveillance, such as parks, roads, bridges, parks, squares, etc., the type of target is obtained by video sensors, the spatial position and speed of the target is obtained by radar, and the radar target and the visual target are correctly matched/fused. Based on the correct fusion, the data service can be provided for road flow indexes (such as traffic flow, space occupancy, time occupancy and queuing length).

When the radar target and the visual target are fused, an accurate fusion result is the basis of subsequent event processing. As shown in fig. 1, the target detection situation of the front and rear frames in the radar video monitoring after calibration is schematically shown. The left image is a monitoring scene of the previous frame, the right image is a monitoring scene of the next frame (namely, after a period of time, the object moves from the first image to the second image), ABCD in the scene respectively represents 4 visual objects, ①②③④ respectively represents radar objects mapped into the image after calibration, if a solid line frame exists in the visual objects, the current object can be detected by the visual sensor, otherwise, the object is still present but cannot be detected by the visual sensor due to too far distance.

It should be noted that, in the embodiment of the present application, the object/target to be identified and tracked is a vehicle, and all the objects/targets mentioned below refer to vehicles.

In the left view, since all vehicles can be detected by the radar during running of the vehicle, the video sensor can detect only BC two vehicles at the near end. After a period of time, all vehicle states are shown in the right graph, and the radar sensor cannot detect the C vehicle due to the reasons of vehicle standstill, vehicle congestion, too close a vehicle distance and the like, and the vision sensor cannot detect the A vehicle due to too far a distance. In the process of turning from the left image to the right image, the expected pursuit result should be 4 fusion Identity Identifiers (IDs) corresponding to the ABCD four vehicles respectively. However, in the decision-level fusion method, the single-sensor tracking result is generally directly used, and the switching between different sensors of the same vehicle is completed by matching with a certain logic process. The case of pursuit errors is easy to occur. Such as: when video is the main source, the D vehicle can not be searched to generate a new ID, and when radar is the main source, the C vehicle can fail to search to generate a new ID. Even if a logical processing step such as switching the sensor by distance is added, the sensor cannot be adapted to different scenes.

The radar and the video sensor are used for detecting the target, although the advantages can be complemented, the detection scene of the target is enlarged, high-precision target identification accuracy can not be obtained at any time, for example, the calibration error is overlarge, the fusion method is not selected properly, and the accuracy of target identification and target tracking can be reduced by using the two sensors. By effectively fusing the radar and video data through a proper method, the equipment can work in more detection scenes, and meanwhile, more detailed and accurate identification information of the target can be obtained.

In view of this, the embodiments of the present application provide an object recognition method, apparatus, electronic device, and storage medium, in a current detection period, each radar object detected by a radar is matched with each visual object detected by a video acquisition device, at least one fusion object and corresponding fusion position information are determined, and a current object set is updated based on the obtained at least one fusion position information and predicted position information of each candidate object included in the current object set in the current detection period. The radar object and the visual object are fused to obtain the fusion object, the detection results of the radar and the video acquisition equipment are fully utilized, the spatial attribute information of the fusion object can be obtained more stably, the stability and the accuracy of object identification are improved, the condition of object loss in the object identification process is reduced, the dependence on the high precision of the radar or the video acquisition equipment is reduced, the equipment cost is reduced, and the object set is updated through fusion position information, so that the spatial attribute information of the object is updated in real time.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification, it being understood that the preferred embodiments described herein are for illustration and explanation only, and not for limitation of the present application, and embodiments of the present application and features of the embodiments may be combined with each other without conflict.

Fig. 2 is a schematic diagram of an application scenario according to an embodiment of the present application. The application scene graph comprises a server 201, a terminal device 202, a video acquisition device 203 and a radar 204. Information interaction can be performed between the server 201 and the terminal device 202 through a communication network, where a communication manner adopted by the communication network may include: both wireless communication and wired communication, video capture device 203 and radar 204 may transmit captured ground data or information to terminal device 202.

The server 201 may illustratively communicate with the terminal device 202 via a cellular mobile communication technology, such as, for example, fifth generation mobile communication (5th Generation Mobile Networks,5G) technology, access to the network.

Alternatively, the server 201 may access the network to communicate with the terminal device 202 via short-range wireless communication means, including, for example, wireless fidelity (WIRELESS FIDELITY, wi-Fi) technology.

The server 201 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), basic cloud computing services such as big data and an artificial intelligent platform.

It should be noted that, in the embodiment of the present application, the server 201 is configured to obtain the spatial attribute information of each radar object and each visual object detected in the current detection period, match each radar object with each visual object based on each detected position information, determine the spatial attribute information of at least one fusion object and at least one fusion object, and update the current object set based on the obtained at least one fusion position information and the predicted position information of each candidate object included in the current object set in the current detection period.

Terminal device 202 is a device that may provide voice and/or data connectivity to a user, comprising: a handheld terminal device with a wireless connection function, a vehicle-mounted terminal device, and the like.

Exemplary terminal devices 202 include, but are not limited to: a Mobile phone, a tablet computer, a notebook computer, a palm computer, a Mobile internet device (Mobile INTERNET DEVICE, MID), a wearable device, a Virtual Reality (VR) device, an augmented Reality (Augmented Reality, AR) device, a wireless terminal device in industrial control, a wireless terminal device in unmanned driving, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, a wireless terminal device in smart home, or the like.

In addition, the terminal device 202 may have installed thereon an associated client, which may be software (e.g., an application, browser, short video software, etc.), a web page, an applet, etc. In an embodiment of the present application, the acquired radar object and visual object may be transmitted to the server 201 by the terminal device 202.

The video acquisition device 203 converts analog video into digital video and stores the digital video in a format of a digital video file, and the video acquisition is a process of converting video signals output by an analog video camera, a video recorder, a laser video disc (English: LASERDISC, abbreviated: LD) video disc player and a television into binary digital information through special analog and digital conversion devices.

In the working process of video acquisition, the video acquisition card is main equipment, and is divided into two levels of profession and household, the professional level video acquisition card not only can carry out video acquisition, but also can realize hardware level video compression and video editing, the household level video acquisition card can only carry out video acquisition and preliminary hardware level compression, and the television card with a lower end can carry out video acquisition, but the video acquisition function of hardware level is generally omitted.

Radar 204 refers to radio detection and ranging, i.e., finding a target and determining its spatial position by radio, and is therefore also referred to as "radio positioning" as an electronic device that detects a target using electromagnetic waves. The radar emits electromagnetic waves to irradiate the target and receives echoes thereof, thereby obtaining information such as the distance from the target to the electromagnetic wave emission point, the distance change rate (radial velocity), the azimuth, the altitude and the like.

Illustratively, the radar includes a millimeter wave radar that obtains metrology information for the target including: the distance, angle, radial speed and radar reflection area are that the target detected by the normal millimeter wave radar is a cluster aggregation area, but the fusion of the radar object and the visual object in the embodiment of the application belongs to a decision-level fusion method, and for decision-level target fusion, the radar object is only a clustered point, and the visual object is a rectangular area in an image.

It should be noted that, the number of terminal devices and servers shown in fig. 2 is merely illustrative, and the number of terminal devices and servers is not limited in practice, and is not particularly limited in the embodiment of the present application.

In the embodiment of the application, when the number of the servers is multiple, the multiple servers can be formed into a blockchain, and the servers are nodes on the blockchain; according to the object identification method disclosed by the embodiment of the application, the spatial attribute information of each object can be stored on the blockchain.

In addition, the embodiment of the application can be applied to various scenes, including not only object recognition scenes, but also but not limited to cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and other scenes.

The object recognition method provided by the exemplary embodiments of the present application will be described below with reference to the accompanying drawings in conjunction with the above-described application scenario, and it should be noted that the above-described application scenario is merely illustrated for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not limited in any way in this respect.

Referring to fig. 3, a flowchart of an implementation of an object recognition method according to an embodiment of the present application is shown, taking an execution subject as a server as an example, where the implementation flow of the method includes steps S31 to S33 as follows:

s31: the method comprises the steps that a server respectively acquires spatial attribute information of each radar object and each visual object detected in a current detection period;

Specifically, the radar and the video capturing device are each configured to periodically detect an object in the target area, for example, each detection period of the radar is spaced by 0.5 seconds, each detection period of the video capturing device is spaced by 1 second, and the intervals between the detection periods of the radar and the video capturing device may be different, but the current detection period acquired by the server needs to correspond to the same time, for example, the current detection period is 2023, 12, 1, 2, 15 minutes, and 5 seconds.

The spatial attribute information includes at least position information, for example, (x, y) coordinate information of the object in a world coordinate system, the spatial attribute information of the radar object may further include velocity information and point cloud data, the spatial attribute information of the visual object may further include velocity information, and boundary information of the visual object in a video frame, that is, an object detection frame.

In order to more clearly describe the object recognition method in the present application, in the embodiment of the present application, an object (also referred to as a target) is taken as an example of a vehicle. The following describes a process of acquiring spatial attribute information of a radar object and a visual object:

As shown in fig. 4, in the embodiment of the application, the radar object and the visual object detection principle are schematically shown, and the radar-based radar tracking algorithm and the video tracking algorithm based on the video acquisition device can accurately detect the number and the position of the vehicle (except for the condition that the target is out of the monitoring range and is blocked), and the video tracking algorithm can give the position of the target frame in real time. A target sequence and a corresponding target box sequence may be established:

Wherein, Representing a visual target (visual object) ID, box _i (boundary information) represents the position of a visual target frame.

TABLE 1

Wherein,And respectively representing pixel coordinates of four points of the upper left, the upper right, the lower left and the lower right of the visual target frame.

Similarly, the radar can detect and track down the number of targets and point cloud data, and can establish the following target sequences and corresponding point cloud set sequences:

Rtrg₁,CartPtSet₁

Rtrg₂,CartPtSet₂

Rtrg₃,CartPtSet₃

…

Wherein, Representing a radar target (radar object) ID, cartPtSet _i represents a radar target point cloud location set. And obtaining the corresponding number and positions of the vehicles after inputting the radar point cloud data. The clustering method which can be used is a Density-based clustering method (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) with noise, and the DBSCAN is a spatial clustering algorithm based on Density and is a clustering method which is widely used in a radar algorithm.

All measurement data are processed through clustering and calibration algorithms, and the following mapping relation can be constructed:

Rloc_i＝h_cluster(CartPtSet_i)

Vloc_i＝h_calib(BoxPt_i)

Wherein Rloc _i represents the position of the radar target in the world coordinate system, and h _cluster (x) represents a clustering algorithm for processing Lei Dadian cloud data. Vloc _i represents the position of the visual target in the world coordinate system. h _calib (x) represents a calibration function that converts the pixel points of the target frame into positions in the world coordinate system.

S32: the server matches each radar object with each visual object based on the detected position information, and determines at least one fusion object and spatial attribute information of the at least one fusion object based on a matching result;

Specifically, the radar object and the visual object each have corresponding position information, and based on the position information, the distance between the radar object and the visual object can be calculated, and the radar object and the visual object which are closer in distance are generally considered to match each other. The radar object may be matched with the visual object with the smallest distance, or a distance threshold may be set, and if the distance between the radar object and the visual object is smaller than the distance threshold, the radar object and the visual object are considered to be matched, or whether the radar object and the visual object are matched may be determined based on other manners, which is not limited specifically herein.

In the matching process, since the number of radar objects and visual objects do not necessarily correspond, each radar object does not necessarily have a corresponding visual object, and the matching result includes: at least one of a first radar object that matches the visual object, a second radar object that does not match the visual object, and an initial visual object that does not match the radar object.

For example, radar objects 1,2,3,4, match visual objects 1,2,3,4, radar object 1 matches visual object 2, radar object 2 matches visual object 4, radar object 3 matches visual object 1, radar object 4 does not match visual object, visual object 3 does not match radar object, then the first radar object comprises: radar object 1, radar object 2 and radar object 3, the second radar object comprising radar object 4 and the initial visual object comprising visual object 3.

After the matching result is obtained, a fusion object is obtained by:

If the matching result comprises that the first radar object exists, determining a first fusion object based on the first radar object and a visual object matched with the first radar object;

If the matching result comprises the existence of a second radar object, determining a second fusion object based on the second radar object;

if the matching result includes the presence of an initial visual object, a third fusion object is determined based on the initial visual object.

Specifically, for different matching results, the fusion objects can also be divided into three categories: the first fusion object, the second fusion object and the third fusion object, wherein in practical application, the fusion objects comprise part or all of the first fusion object, the second fusion object and the third fusion object.

Still taking radar objects 1-4 as an example, a first fused object 1 is obtained based on radar object 1 and visual object 2, a first fused object 2 is obtained based on radar object 2 and visual object 4, a first fused object 3 is obtained based on radar object 3 and visual object 1, a second fused object 1 is obtained based on radar object 4, and a third fused object 2 is obtained based on visual object 3.

Correspondingly, the spatial attribute information of the fusion object at least comprises fusion position information, wherein the fusion position information is determined based on the position information of the corresponding radar object and/or video object, and the fusion position information of the first fusion object is determined based on the position information of the first radar object; the fused position information of the second fused object is determined based on the position information of the second radar object; the fused position information of the third fused object is determined based on the position information of the initial visual object.

Specifically, the position information of the first radar object is used as the fusion position information of the first fusion object, the position information of the second radar object is used as the fusion position information of the second fusion object, and the position information of the initial visual object is used as the position information of the third fusion object. The position information of the visual object is obtained by mapping the center point of a target frame in a visual object video frame to a world coordinate system through a calibration function

The spatial attribute information of the radar object further comprises speed information, the spatial attribute information of the visual object further comprises speed information and boundary information in a target video frame, the spatial attribute information of the fusion object further comprises fusion speed information and fusion boundary information, and the target video frame is a video frame acquired by the video acquisition object in the current detection period; fusion speed information and fusion boundary information may also be obtained by:

Acquiring fusion speed information and fusion boundary information of a first fusion object based on speed information of the first radar object and boundary information of a visual object matched with the first radar object; acquiring fusion speed information and fusion boundary information of a second fusion object based on the speed information and the position information of the second radar object; and obtaining the fusion speed information and the fusion boundary information of the third fusion object based on the speed information and the boundary information of the initial visual object.

Specifically, the speed information of the radar object can be directly detected by the radar, and the boundary information of the visual object can be directly obtained by detecting the target video frame by the visual sensor, so that the speed information of the first radar object can be directly used as the fusion speed information of the first fusion object, and the boundary information of the visual object matched with the first radar object can be directly used as the fusion boundary information of the first fusion object.

For the second fusion object, the speed information of the second radar object can be directly used as the fusion speed information of the second fusion object, and in order to ensure the integrity of the attribute of the fusion object, the fusion boundary information (target frame) of the second fusion object uses the virtual boundary information (virtual frame) of the position fitting of the second radar object.

For the third fusion object, the boundary information of the initial visual object can be directly used as the fusion boundary information of the third fusion object, and in order to ensure the integrity of the attribute of the fusion object, the fusion speed information (target frame) of the third fusion object is estimated based on the initial visual object, the position change amount is calculated based on the difference between the position information of the initial visual object in the target visual frame and the position information in the last video frame, and the speed information of the third fusion object is obtained based on the position change amount and the time difference between the target video frame and the last video frame.

S33: the server updates the current object set based on the obtained at least one piece of fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period;

The method comprises the steps that each candidate object is obtained based on a fusion object in at least one previous detection period, fusion position information is the position of the fusion object in the current detection period, prediction position information is the position of the predicted candidate object in the current detection period, whether the fusion object and the candidate object are obtained for the same object or not can be judged by comparing the fusion position information and the prediction position information, for example, the fusion object 1 is a vehicle 1 detected in 1 minute and 14 seconds, the candidate object 1 is a vehicle 1 detected in 1 minute and 13 seconds, the position information of the candidate object 1 can be updated through the fusion position information of the fusion object 1, and when the candidate object corresponding to the fusion object does not exist, the fusion object can be used as the candidate object to be newly added into an object set.

In addition, when the detection period does not exist before the current detection period, no candidate object exists in the current object set, and at least one fusion object is respectively added to the current object set as a candidate object.

Optionally, in step S33, for one candidate object, the corresponding predicted position information is obtained by:

the predicted position information of one candidate object is obtained based on the position information, the speed information, and the time interval between the last detection period and the current detection period of the one candidate object.

Specifically, the last detection period is a detection period in which the candidate object is detected last time, and one or more detection periods may exist between the last detection period and the current detection period.

The predicted position information of the current detection period candidate may be predicted by the following formula:

Wherein, Predicted position information representing a candidate object of a current detection cycle, Δt representing a time interval between a previous detection cycle and a current detection cycle,/>Representing speed information.

In the embodiment of the application, in the current detection period, each radar object detected by the radar is matched with each visual object detected by the video acquisition equipment, at least one fusion object and corresponding fusion position information are determined, and the current object set is updated based on the obtained at least one fusion position information and the predicted position information of each candidate object contained in the current object set in the current detection period. The radar object and the visual object are fused to obtain the fusion object, the detection results of the radar and the video acquisition equipment are fully utilized, the spatial attribute information of the fusion object can be obtained more stably, the stability and the accuracy of object identification are improved, the condition of object loss in the object identification process is reduced, the dependence on the high precision of the radar or the video acquisition equipment is reduced, the equipment cost is reduced, and the object set is updated through fusion position information, so that the spatial attribute information of the object is updated in real time.

Alternatively, step S32 may be implemented as steps S321-S323:

S321: based on the position information, obtaining the association degree of each object group, wherein each object group comprises a radar object and/or a visual object;

Specifically, the radar object and the visual object are grouped, and all the groups including one radar object and/or one visual object are enumerated, taking radar object 1, radar object 2, visual object 1 and visual object 2 as examples, 8 object groups are obtained, wherein object group 1 includes radar object 1, object group 2 includes radar object 2, object group 3 includes visual object 1, object group 4 includes visual object 2, object group 5 includes radar object 1 and visual object 1, object group 6 includes radar object 1 and visual object 2, object group 7 includes radar object 2 and visual object 1, and object group 8 includes radar object 2 and visual object 2.

The association degree of each object group is obtained based on the position information of the radar object and/or the position information of the visual object, and for the class of the object contained in each object group, the corresponding association degree can be obtained by the following two ways:

Mode 1: when one object group contains a radar object and a visual object, obtaining the association degree of the one object group based on the distance between the position information of the radar object contained in the one object group and the position information of the visual object;

Specifically, the calculation mode of the association degree can adopt the distance between two objects for judgment. The position information of the radar object and the visual object is a point (X, Y) in the world coordinate system, and thus the distance Wij between the radar object i and the visual object j can be calculated by the following formula:

Where (x _i,y_i) represents the position information of the radar object and (x _j,y_j) represents the position information of the visual object.

Optionally, when the distance between the position information of the radar object and the position information of the visual object is greater than a preset distance threshold, the second association degree is used as the association degree of one object group.

Specifically, to prevent the misassociation of two objects far away from each other, a maximum distance (preset distance threshold) should be set, and when the maximum distance is exceeded, wij is set to a maximum value (second association degree), namely:

if Wij>d_maxthen Wij＝W_max

The following association matrix table 2 may be established to store the association degrees, where the rows represent the association degrees of radar objects and the columns represent the association degrees of visual objects:

TABLE 2

Mode 2: when one object group contains only a radar object or a visual object, the first degree of association is regarded as the degree of association of one object group.

If the object group only includes one object, the distance cannot be calculated, and the first association degree is set as the association degree of the object group, where the first association degree may be set according to the actual requirement, for example, set to 0.

S322: combining the object groups to obtain a plurality of object group sets, wherein each object group set comprises radar objects and visual objects, and no coincident radar objects or visual objects exist in one object group set;

Specifically, all of the radar objects and the visual objects need to be contained in each object group set, and there is no repetition, and 7 object group sets are obtained by combining among the above object groups 1 to 8, taking the above object groups 1 to 8 as an example, wherein the object group set 1 contains { (radar object 1), (visual object 1), (radar object 2, visual object 2) }, the object group set 2 contains { (radar object 1, visual object 1), (radar object 2, visual object 2) }, the object group set 3 contains { (radar object 1, visual object 1), (radar object 2), (visual object 2) }, the object group set 4 contains { (radar object 1), (visual object 1), (radar object 2), (visual object 2) }, the object group set 5 contains { (radar object 1), (visual object 2), (radar object 2, visual object 1) }, the object group set 6 contains { (radar object 1, visual object 2), (radar object 1) }, the object group set 7 contains { (radar object 1, visual object 2) }.

S323: determining at least one fusion object based on a target object group set of the plurality of object group sets, the target object group set being: and an object group set having a minimum sum of association degrees of the object groups included in the plurality of object group sets.

Specifically, after the object groups are combined into the object group set, there is a sum of association degrees for any one of the combination results, for example,And/>Is a group,/>And/>Is a group,/>And/>Is a group,/>And/>When the object group is a group, the sum of the association degrees of the object group sets is:

W_num＝W₁₁+W₂₂+W₃₃+W₄₄

When W _num is minimum, the current object combination set is considered to be the optimal combination mode, namely the target object group set, and the minimum weight matching of the incidence matrix can be solved by using a computer algorithm (KM algorithm).

A fusion object may be determined based on each object group in the set of target object groups, taking the set of target object groups as { (radar object 1), (vision object 1), (radar object 2, vision object 2) } for example, fusion object 1 is determined based on radar object 1, fusion object 2 is determined based on vision object 1, and fusion object 3 is determined based on radar object 2 and vision object 2.

Optionally, after the target object group set is determined in step S323, the fusion object is determined by at least one of the following means:

mode 1: determining a first fusion object based on object groups containing radar objects and visual objects in object groups contained in the target object group set;

Mode 2: determining a second fusion object based on the object group only containing radar objects in the object groups contained in the target object group set;

Mode 3: and obtaining a third fusion object based on the object group only comprising the visual object in the object groups contained in the target object group set.

By the above method, three kinds of fusion objects can be determined, and the method for acquiring the spatial attribute information of each fusion object is referred to the above embodiment, which is not described herein.

Alternatively, step S33 may be implemented as:

Matching the at least one fusion object with each candidate object based on the at least one fusion position information and each predicted position information; updating the spatial attribute information of the corresponding candidate object based on the spatial attribute information of the fusion object matched to the candidate object; and adding the fusion object which is not matched with the candidate object to the current object set as the candidate object.

The method comprises the steps of determining candidate objects matched with fusion objects based on fusion position information and predicted position information, and obtaining respective association degrees of object groups based on at least one fusion position information and each predicted position information, wherein each object group comprises one fusion object and/or one candidate object, and the association degrees are obtained based on the position information of the radar objects and/or the position information of the visual objects contained in the corresponding object group, similar to the process of matching the visual objects with the radar objects; combining the object groups to obtain a plurality of object group sets, wherein each object group set comprises each fusion object and each candidate object, and no coincident fusion object or candidate object exists in one object group set; determining candidate objects matched by fusion objects based on target object group sets in a plurality of object group sets, wherein the target object group sets are as follows: and an object group set having a minimum sum of association degrees of the object groups included in the plurality of object group sets.

The spatial attribute information of the candidate object is obtained based on the spatial attribute information of the fusion object matched in the previous detection period, the spatial attribute information of the matched candidate object can be updated through the spatial attribute information of the fusion object, and the fusion object is newly added into the current object set as the candidate object for the fusion object which is not matched with the candidate object.

Further, candidate objects which are not matched with the fusion object may exist in each candidate object included in the current object set, namely, the disconnection object in the current object set is determined, and when the number of times that the disconnection object cannot be matched with the fusion object is greater than a preset number of times threshold, the disconnection object is removed from the current object set.

For example, if the prediction number is 5, the missing object is removed when the number of times that the missing object cannot be matched with the fusion object is greater than 5.

As shown in fig. 5, a flowchart of matching fusion objects and candidate objects in an embodiment of the present application is shown, and the matching process is specifically described below with reference to fig. 5:

s501: inputting all fusion targets of the current frame;

S502: judging whether the current frame is the first frame, if yes, executing step S508, and if not, executing step S503;

s503, according to the position and speed of each candidate object in the list, obtaining the predicted position of the candidate object in the current frame;

s504: constructing an association matrix according to the predicted position of each candidate target and the detection position of the current frame target;

The association matrix is used for describing whether each candidate object and the fusion object of the current frame are the same object. An association matrix table 3 may be established to store the association degrees in which rows represent the association degrees of the fusion targets and columns represent the association degrees of candidate targets (note that the fusion targets have no ID for the current frame, here replaced with a sequence number).

/>

TABLE 3 Table 3

Wij in the association matrix represents the association degree of the ith candidate target and the jth fusion target of the current frame. The association degree calculation mode can adopt the distance between two targets for judgment. The distance calculation formula can be written as:

Where (x _i,y_i) represents the predicted position of the candidate object in the current frame and (x _j,y_j) represents the detected position of the current frame fusion object.

In order to prevent the error association of two objects with a longer distance, a maximum association distance should be set, and when the maximum distance is exceeded, wij is set as a maximum value, namely:

if Wij>d_maxthen Wij＝W_max

s505: obtaining a target matching result according to the constructed incidence matrix;

The correlation matrix can be solved through a KM algorithm, when the sum of the weights is minimum, the best correlation result is considered at the moment, and three matching results can be obtained at the moment:

First kind: successfully matched fusion targets;

Second kind: fusion targets that did not match successfully;

Third kind: candidate targets that did not successfully match;

S506: updating the spatial attribute information of the candidate targets in the list by using the spatial attribute information of the successfully matched fusion targets;

S507: assigning candidate object IDs to the matched fusion objects for identifying the same object;

s508: adding the fusion target which is not successfully matched to the list;

S509: judging whether the number of unassociated times of the candidate targets which are not successfully matched exceeds a threshold value, if so, executing the step S510, and if not, executing the step S511;

S510: deleting from the list;

S511: incrementing a target loss of association count value;

S512: and after the end of the search, outputting the current frame search result externally.

Fig. 6 is a schematic overall flow chart of an object recognition method according to an embodiment of the application, which includes the following steps:

S601: the camera inner and outer parameters/four points are marked;

The purpose of S601 is to configure parameters on which the video radar calibration method depends, where the video radar calibration accuracy affects the subsequent fusion accuracy.

S602: detecting a visual object based on the video acquisition device;

detecting the number and the positions of targets by adopting artificial intelligence methods such as deep learning and the like, and endowing each target with unique ID;

S603: visual object searching;

The purpose of this step is that the same target has a stable ID.

S604: based on the position of a target frame of a visual object in a video frame, extracting a target frame center point, and mapping the target frame center point into an actual position under a world coordinate system through a calibration algorithm to obtain the position of the visual object;

s605: acquiring radar point cloud data based on radar;

Radar can obtain measurements in the environment, including point cloud data from real targets and point cloud data from other false targets.

S606: clustering Lei Dadian cloud data to obtain the number of radar objects and point cloud data corresponding to each radar object;

S607: determining the rectangular outline of the point cloud data of each radar object, and calculating the position of the radar object under a world coordinate system;

s608: acquiring a radar object and a visual object corresponding to the same time stamp;

s609: fusing the visual object and the radar object to obtain a fused object;

S610: and based on the candidate objects, finishing ID assignment on all the fusion objects of the current frame.

Based on the mode, after the single sensor finishes the tracking, the fusion target after the association is tracked once again, so that the accuracy of multi-sensor tracking can be effectively improved. Therefore, the requirement on the single sensor on the pursuit accuracy is reduced, and the normal operation of the system is not affected when the single sensor cannot pursue. In the fusion process of the radar object and the visual object, the radar and the visual object are associated by adopting a global nearest neighbor method, so that the positioning accuracy requirement of the sensor is reduced, the calibration accuracy requirement is reduced, and the scene adaptability is stronger.

Based on the same inventive concept, the embodiment of the application also provides an object recognition device. As shown in fig. 7, which is a schematic structural diagram of the object recognition device 700, may include:

an acquiring unit 701, configured to acquire, respectively, spatial attribute information of each radar object and each visual object detected in a current detection period, where the spatial attribute information includes at least position information;

A matching unit 702, configured to match each radar object with each visual object based on each detected position information, and determine at least one fusion object and spatial attribute information of at least one fusion object based on a matching result, where the spatial attribute information of the fusion object at least includes fusion position information;

An updating unit 703, configured to update the current object set based on the obtained at least one fused position information and the predicted position information of each candidate object included in the current object set in the current detection period, where each candidate object is obtained based on the fused object in the previous at least one detection period.

the matching unit 702 is specifically configured to:

Optionally, the fused position information of the first fused object is determined based on the position information of the first radar object; the fused position information of the second fused object is determined based on the position information of the second radar object; the fused position information of the third fused object is determined based on the position information of the initial visual object.

Optionally, the spatial attribute information of the radar object further includes speed information, and the spatial attribute information of the visual object further includes speed information and boundary information in the target video frame; the target video frame is a video frame acquired by a video acquisition object in the current detection period; the spatial attribute information of the fusion object also comprises fusion speed information and fusion boundary information;

The matching unit 702 is further configured to:

acquiring fusion speed information and fusion boundary information of a first fusion object based on speed information of the first radar object and boundary information of a visual object matched with the first radar object;

acquiring fusion speed information and fusion boundary information of a second fusion object based on the speed information and the position information of the second radar object;

Optionally, the matching unit 702 is specifically configured to:

Combining the object groups to obtain a plurality of object group sets, wherein each object group set comprises radar objects and visual objects, and no coincident radar objects or visual objects exist in one object group set;

Determining at least one fusion object based on a target object group set of the plurality of object group sets, the target object group set being: and an object group set having a minimum sum of association degrees of the object groups included in the plurality of object group sets.

Optionally, the matching unit 702 is specifically configured to obtain each association degree by:

For a plurality of object groups, the following operations are performed:

when one object group contains a radar object and a visual object, obtaining the association degree of the one object group based on the distance between the position information of the radar object contained in the one object group and the position information of the visual object;

When one object group contains only a radar object or a visual object, the first degree of association is regarded as the degree of association of one object group.

Optionally, the matching unit 702 is specifically configured to determine the fusion object by at least one of the following:

Determining a first fusion object based on object groups containing radar objects and visual objects in object groups contained in the target object group set;

determining a second fusion object based on the object group only containing radar objects in the object groups contained in the target object group set;

And obtaining a third fusion object based on the object group only comprising the visual object in the object groups contained in the target object group set.

Optionally, the matching unit 702 is further configured to:

And when the distance is larger than a preset distance threshold value, taking the second association degree as the association degree of one object group.

For each candidate object, the following operations are respectively executed:

Optionally, the updating unit 703 is specifically configured to:

determining a decoupling object in the current object set, wherein the decoupling object is a candidate object which is not matched with the fusion object in each candidate object;

and when the number of times that the missing link object cannot be matched with the fusion object is greater than a preset number of times threshold, removing the missing link object from the current object set.

Optionally, the apparatus further comprises an adding unit 704, configured to:

And when the detection period does not exist before the current detection period, at least one fusion object is respectively used as a candidate object to be added to the current object set.

For convenience of description, the above parts are described as being functionally divided into modules (or units) respectively. Of course, the functions of each module (or unit) may be implemented in the same piece or pieces of software or hardware when implementing the present application.

In the present embodiment, the term "module" or "unit" refers to a computer program or a part of a computer program having a predetermined function and working together with other relevant parts to achieve a predetermined object, and may be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Also, a processor (or multiple processors or memories) may be used to implement one or more modules or units. Furthermore, each module or unit may be part of an overall module or unit that incorporates the functionality of the module or unit.

Those skilled in the art will appreciate that the various aspects of the application may be implemented as a system, method, or program product. Accordingly, aspects of the application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

The embodiment of the application also provides electronic equipment based on the same conception as the embodiment of the method. In one embodiment, the electronic device may be a server, such as the server shown in FIG. 2. In this embodiment, the electronic device may be configured as shown in fig. 8, including a memory 801, a communication module 803, and one or more processors 802.

A memory 801 for storing a computer program for execution by the processor 802. The memory 801 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a program required for running an instant communication function, and the like; the storage data area can store various instant messaging information, operation instruction sets and the like.

The memory 801 may be a volatile memory (RAM) such as a random-access memory (RAM); the memory 801 may also be a nonvolatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk (HARD DISK DRIVE, HDD) or a solid state disk (solid-state disk) (STATE DRIVE, SSD); or memory 801, is any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 801 may be a combination of the above memories.

The processor 802 may include one or more central processing units (central processing unit, CPUs) or digital processing units, or the like. A processor 802 for implementing the above-described object recognition method when calling the computer program stored in the memory 801.

The communication module 803 is used for communicating with a terminal device and other servers.

The specific connection medium between the memory 801, the communication module 803, and the processor 802 is not limited in the embodiment of the present application. The embodiment of the present application is illustrated in fig. 8 by a bus 804 between the memory 801 and the processor 802, where the bus 804 is illustrated in fig. 8 by a bold line, and the connection between other components is merely illustrative, and not limiting. The bus 804 may be classified as an address bus, a data bus, a control bus, or the like. For ease of description, only one thick line is depicted in fig. 8, but only one bus or one type of bus is not depicted.

The memory 801 has stored therein a computer storage medium having stored therein computer executable instructions for implementing the object recognition method of the embodiment of the present application. The processor 802 is configured to perform the object recognition method described above, as shown in fig. 3.

In another embodiment, the electronic device may also be other electronic devices, such as the terminal device shown in fig. 2. In this embodiment, the structure of the electronic device may include, as shown in fig. 9: communication component 910, memory 920, display unit 930, camera 940, sensor 950, audio circuit 960, bluetooth module 970, processor 980, and so forth.

The communication component 910 is configured to communicate with a server. In some embodiments, a circuit wireless fidelity (WIRELESS FIDELITY, WIFI) module may be included, the WiFi module belongs to a short-range wireless transmission technology, and the electronic device may help the user to send and receive information through the WiFi module.

Memory 920 may be used to store software programs and data. The processor 980 performs various functions of the terminal device as well as data processing by executing software programs or data stored in the memory 920. Memory 920 may include high-speed random access memory, but may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. The memory 920 stores an operating system that enables the terminal device to operate. The memory 920 may store an operating system and various application programs, and may also store a computer program for executing the object recognition method according to the embodiment of the present application.

The display unit 930 may also be used to display information input by a user or information provided to the user and a graphical user interface (GRAPHICAL USER INTERFACE, GUI) of various menus of the terminal device. Specifically, the display unit 930 may include a display 932 provided on the front surface of the terminal device. The display 932 may be configured in the form of a liquid crystal display, light emitting diodes, or the like. The display unit 930 may be used to display an object recognition user interface or the like in the embodiment of the present application.

The display unit 930 may also be used to receive input numeric or character information, generate signal inputs related to user settings and function control of the terminal device, and in particular, the display unit 930 may include a touch screen 931 provided on the front surface of the terminal device, and may collect touch operations on or near the user, such as clicking buttons, dragging scroll boxes, and the like.

The touch screen 931 may be covered on the display screen 932, or the touch screen 931 may be integrated with the display screen 932 to implement input and output functions of the terminal device, and after integration, the touch screen may be simply referred to as a touch screen. The display unit 930 may display the application program and the corresponding operation steps in the present application.

Camera 940 may be used to capture still images and a user may comment on the image captured by camera 940 through an application. The number of cameras 940 may be one or more. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal to an electrical signal, which is then passed to processor 980 for conversion to a digital image signal.

The terminal device may further comprise at least one sensor 950, such as an acceleration sensor 951, a distance sensor 952, a fingerprint sensor 953, a temperature sensor 954. The terminal device may also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, light sensors, motion sensors, and the like.

Audio circuitry 960, speaker 961, microphone 962 may provide an audio interface between a user and a terminal device. Audio circuit 960 may transmit the received electrical signal converted from audio data to speaker 961, where it is converted to a sound signal output by speaker 961. The terminal device may also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, microphone 962 converts the collected sound signals into electrical signals, which are received by audio circuit 960 and converted into audio data, which are output to communication component 910 for transmission to, for example, another terminal device, or to memory 920 for further processing.

The bluetooth module 970 is used for exchanging information with other bluetooth devices having a bluetooth module through a bluetooth protocol. For example, the terminal device may establish a bluetooth connection with a wearable electronic device (e.g., a smart watch) that also has a bluetooth module through the bluetooth module 970, thereby performing data interaction.

The processor 980 is a control center of the terminal device, and connects various parts of the entire terminal using various interfaces and lines, performs various functions of the terminal device and processes data by running or executing software programs stored in the memory 920, and calling data stored in the memory 920. In some embodiments, processor 980 may include one or more processing units; processor 980 may also integrate an application processor primarily handling operating systems, user interfaces, applications programs, etc., with a baseband processor primarily handling wireless communications. It will be appreciated that the baseband processor described above may not be integrated into the processor 980. The processor 980 may run an operating system, applications, user interface displays, and touch responses, as well as the object recognition methods of embodiments of the present application. In addition, the processor 980 is coupled to the display unit 930.

In some possible embodiments, aspects of the object recognition method provided by the present application may also be implemented in the form of a program product comprising a computer program for causing an electronic device to perform the steps of the object recognition method according to the various exemplary embodiments of the application described herein above when the program product is run on the electronic device, e.g. the electronic device may perform the steps as shown in fig. 3.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product of embodiments of the present application may take the form of a portable compact disc read only memory (CD-ROM) and comprise a computer program and may be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a command execution system, apparatus, or device.

The readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave in which a readable computer program is embodied. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device.

A computer program embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic device may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (e.g., connected through the internet using an internet service provider).

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the elements described above may be embodied in one element in accordance with embodiments of the present application. Conversely, the features and functions of one unit described above may be further divided into a plurality of units to be embodied.

Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having a computer-usable computer program embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program commands may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the commands executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program commands may also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the commands stored in the computer readable memory produce an article of manufacture including command means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Claims

1. An object recognition method, the method comprising:

2. The method of claim 1, wherein the matching result comprises: at least one of the presence of a first radar object that matches the visual object, the presence of a second radar object that does not match the visual object, and the presence of an initial visual object that does not match the radar object;

3. The method of claim 2, wherein the spatial attribute information of the radar object further comprises speed information, and the spatial attribute information of the visual object further comprises speed information and boundary information in the target video frame; the target video frame is a video frame acquired by the video acquisition object in the current detection period; the spatial attribute information of the fusion object also comprises fusion speed information and fusion boundary information;

the method further comprises the steps of:

4. The method of claim 1, wherein the matching of the radar objects with the visual objects is based on the detected positional information, and wherein the determining of at least one fusion object is based on the matching:

5. The method of claim 4, wherein each degree of association is obtained by:

6. A method according to any one of claims 1-5, characterized in that each predicted position information is obtained by:

For each candidate object, the following operations are respectively executed:

7. The method according to any one of claims 1 to 5, wherein updating the current object set based on the obtained at least one piece of fused position information and the predicted position information of each candidate object included in the current object set in the current detection period, comprises:

8. The method of claim 7, wherein the method further comprises:

9. An object recognition apparatus, comprising:

10. An electronic device comprising a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, causes the processor to perform the steps of the method of any of claims 1 to 8.