WO2021227790A1

WO2021227790A1 - Video image target association method, apparatus, and terminal device

Info

Publication number: WO2021227790A1
Application number: PCT/CN2021/088459
Authority: WO
Inventors: 王健; 宁哲
Original assignee: 深圳云天励飞技术股份有限公司
Priority date: 2020-05-15
Filing date: 2021-04-20
Publication date: 2021-11-18
Also published as: CN111695429B; CN111695429A

Abstract

A video image target association method, an apparatus, and a terminal device, applicable in the technical field of data processing, the method comprising: obtaining first frame sequence number information and first position information for a reference target, and obtaining second frame sequence number information and second position information for a target to be associated; determining target video frames according to the first frame sequence number information and the second frame sequence number information; determining a temporal coincidence degree for the reference target and the target to be associated according to the number of frames among the target video frames where the reference target and the target to be associated are simultaneously present; determining a spatial association degree for the reference target and the target to be associated according to the first position information and the second position information among the target video frames; calculating an association confidence score for the target to be associated according to the temporal coincidence degree and the spatial association degree; and if the association confidence score for the target to be associated satisfies a preset association condition, determining the target to be associated as an associated target which possesses an association relationship with the reference target. The present method can improve target association accuracy among video images.

Description

Video image target association method, device and terminal equipment

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on May 15, 2020, the application number is 202010412909.3, and the invention title is "Video Image Target Association Method, Apparatus, and Terminal Equipment", the entire content of which is incorporated by reference In this application.

Technical field

This application belongs to the field of data processing technology, and in particular relates to a method, device and terminal device for associating video image objects.

Background technique

At present, in the processing and application of video data, there are methods to realize the monitoring and analysis of specific people or specific objects by searching for the correlation between different targets in the video image. However, the accuracy of the existing target association methods for video images is low.

Summary of the invention

In view of this, the embodiments of the present application provide a video image target association method, device, and terminal equipment to solve the problem of how to improve the accuracy of target association in a video image in the prior art.

The first aspect of the embodiments of the present application provides a method for associating video image objects, including:

Acquire first data corresponding to the reference target, and acquire second data corresponding to the target to be associated, where the first data includes the first frame number information of the reference target in the preset video image sequence and the corresponding First location information, where the second data includes second frame number information and corresponding second location information of the target to be associated in the preset video image sequence;

Determine, according to the first frame sequence number information and the second frame sequence number information, the appearance and disappearance moments of the reference target and the target to be associated in the preset video image sequence to obtain a target video frame;

Determine the temporal coincidence degree of the reference target and the target to be associated according to the number of frames in the target video frame where the reference target and the target to be associated simultaneously exist;

Determine the degree of spatial association between the reference target and the target to be associated according to the first position information and the second position information in the target video frame;

Calculating the correlation confidence of the target to be correlated according to the temporal coincidence degree and the spatial correlation degree;

If the association confidence of the target to be associated satisfies a preset association condition, the target to be associated is determined as an association target that has an association relationship with the reference target.

A second aspect of the embodiments of the present application provides a device for associating a video image target, including:

The acquiring unit is configured to acquire first data corresponding to the reference target and acquire second data corresponding to the target to be associated, wherein the first data includes the first frame of the reference target in the preset video image sequence Serial number information and corresponding first position information, where the second data includes second frame serial number information and corresponding second position information of the target to be associated in the preset video image sequence;

A target video frame determining unit, configured to determine the appearance and disappearance of the reference target and the target to be associated in the preset video image sequence according to the first frame sequence number information and the second frame sequence number information At the moment, get the target video frame;

A time coincidence degree determining unit, configured to determine the time coincidence degree of the reference target and the target to be associated according to the number of frames in the target video frame where the reference target and the target to be associated exist at the same time;

A spatial association degree determining unit, configured to determine the spatial association degree between the reference target and the target to be associated according to the first position information and the second position information in the target video frame;

An association confidence degree calculation unit, configured to calculate the association confidence degree of the target to be associated according to the temporal coincidence degree and the spatial correlation degree;

The associated target determination unit is configured to determine the to-be-associated target as an associated target that has an association relationship with the reference target if the association confidence of the to-be-associated target meets a preset association condition.

The third aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program When the time, the terminal device is made to realize the steps of associating the video image target.

The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the terminal device realizes the video image Steps of target association.

The fifth aspect of the embodiments of the present application provides a computer program product, which when the computer program product runs on a terminal device, causes the terminal device to execute the video image target association method as described in the first aspect.

Compared with the prior art, the embodiment of the present application has the following beneficial effects: In the embodiment of the present application, since the time coincidence degree and the spatial relevance degree of the reference target and the target to be associated can be combined, it can be determined from both the time dimension and the space dimension. Accurately calculate the association confidence of the target to be associated, and accurately determine the association target that has an association relationship with the reference target according to the association confidence, so that the accuracy of the target association of the video image can be improved.

Description of the drawings

FIG. 1 is a schematic diagram of the implementation process of a method for associating video image objects provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a video image target association device provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of a terminal device provided by an embodiment of the present application.

Detailed ways

Example one:

FIG. 1 shows a schematic flowchart of a first method for associating video image objects provided by an embodiment of the present application, and the details are as follows:

In S101, first data corresponding to a reference target is acquired, and second data corresponding to a target to be associated is acquired, where the first data includes the first frame number of the reference target in a preset video image sequence Information and corresponding first position information, and the second data includes second frame number information and corresponding second position information of the target to be associated in the preset video image sequence.

The preset video image sequence in the embodiment of this application is a sequence composed of several frames of video images, and this sequence is the video data captured by the camera equipment in advance; for simplicity of description, the preset video image sequence is The video image is called a video frame. The target in the embodiments of this application refers to an image corresponding to a specific physical object (such as a person, an object, a human body part, or an object part) in a preset video image sequence, and the reference target is the selected main subject that needs to be associated with the video image target. Target, the target to be associated is the selected target that needs to be detected whether there is an association relationship with the reference target.

In the embodiment of the present application, the first data corresponding to the reference target and the second data corresponding to the target to be associated can be obtained from the target data set. The target data set is a collection of data obtained after multi-target tracking or multi-target detection is performed on a preset video image sequence in advance, and the target data set includes frame number information of the video frame corresponding to the target detected in the preset video image sequence And the position information of the target in the video frame, the position information may specifically be coordinate data of the target frame corresponding to the target (may include center coordinate information of the target frame, edge point coordinate information, etc.). Specifically, the data corresponding to the reference target is called first data. The first data includes first frame number information and first position information. The first frame number information is the first video corresponding to the reference target in the preset video image sequence. The sequence number of the frame, and the first position information is the position information of the reference target in the first video frame; wherein, the first video frame is a video frame containing the reference target in the preset video image sequence. Specifically, the second data corresponding to the target to be associated includes second frame number information and second position information. The second frame number information is the sequence number of the second video frame corresponding to the target to be associated in the preset video image sequence. The location information is the location information of the target to be associated in the second video frame; the second video frame is a video frame that contains the target to be associated in a preset video image sequence.

Optionally, the reference target and the target to be associated in the embodiment of the present application are targets corresponding to two different entity object types;

Correspondingly, the acquiring the first data corresponding to the reference target and acquiring the second data corresponding to the target to be associated specifically includes:

A1: Determine the first target type and the second target type that need to be associated in the preset video image sequence, where the first target type and the second target type are respectively two different entity object types;

A2: Obtain a first target data set corresponding to the first target type from the target data set, and obtain a second target data set corresponding to the second target type, wherein the target data set is a preset video image sequence performed in advance The collection of data obtained after multi-target tracking or multi-target detection;

A3: Determine the reference target belonging to the first target type, and obtain the first data corresponding to the reference target from the first target data set;

A4: Select one of the targets belonging to the second target type as the target to be associated, and obtain the second data corresponding to the target to be associated from the second target data set.

Exemplarily, the entity object type includes, but is not limited to, a human face, a human body, a certain organ of a person, a designated object, and the like. By selecting two targets belonging to different entity object types as the reference target and the target to be associated, the association relationship between the different types of targets in the video image sequence can be established for specific monitoring and analysis. For example, the reference target may be the face image of the target person, and the target to be associated may be the vehicle image; through the method of the embodiment of this application, the association relationship between the face image and the vehicle image in the video image sequence can be accurately established, so as to be able to analyze Determine the information of the vehicle driven by the target person.

Optionally, the reference target and the target to be associated in the embodiment of the present application are targets corresponding to the same entity object type. For example, the reference target is the face image of the person A, and the face image of the target person B is to be associated; through the method of the embodiment of the present application, the association relationship between the person A and the person B can be accurately determined, so that the person A and the person B can be analyzed and determined. Whether character B is a companion and other information.

In S102, according to the first frame sequence number information and the second frame sequence number information, determine the appearance and disappearance moments of the reference target and the target to be associated in the preset video image sequence to obtain the target Video frame.

In the embodiment of this application, the first starting frame sequence number in the first frame sequence number information is the frame sequence number corresponding to the time when the reference target appears in the preset video image sequence, and the first ending frame sequence number is the reference target in the preset video sequence. The frame number corresponding to the disappearing moment in the image sequence. The second starting frame number in the second frame number information is the frame number corresponding to the moment when the target to be associated appears in the preset video image sequence, and the second end frame number is the target to be associated in the preset video image sequence The frame number corresponding to the moment of disappearance. Specifically, according to the first starting frame sequence number and the second starting frame sequence number, it is determined that the larger frame sequence number is the starting frame sequence number corresponding to the target video frame, that is, the reference target and the target to be associated correspond to the later time. The video frame is used as the starting frame in the target video frame. Specifically, according to the first end frame sequence number and the second end frame sequence number, the smaller frame sequence number is the end frame sequence number corresponding to the target video frame, that is, the video frame corresponding to the earlier disappearance of the reference target and the target to be associated Is the end frame in the target video frame. According to the determined start frame number and end frame number in the determined target video frame, the target video frame is obtained from the preset video image sequence.

In S103, according to the number of frames of the target video frame, the temporal coincidence of the reference target and the target to be associated;

After the target video frame is obtained, the target video frame is detected one by one (for example, the frame number of each frame of the target video frame is compared with the first frame number information and the second frame number information), and the reference target is determined from it The number of video frames that exist at the same time as the target to be associated. According to the number of frames, the time coincidence of the reference target and the target to be associated is determined. The number of frames can be directly used as the time coincidence degree, or the number of frames can also be divided by the total number of frames of the preset video image sequence as the time coincidence degree.

In S104, determine the degree of spatial association between the reference target and the target to be associated according to the first position information and the second position information in the target video frame;

According to the first position information corresponding to the reference target in the target video frame and the second position information corresponding to the target to be associated, the position relationship between the reference target and the target to be associated is analyzed in the target video frame, and the reference target and the target to be associated are determined according to the position relationship The spatial relevance of the target.

Optionally, the spatial correlation degree includes any one or more of the area coincidence degree, the target distance value, the target direction angle, and the target displacement deviation value.

Among them, the area overlap refers to the overlap ratio of the area where the reference target is located and the area where the target to be associated is located in the same video frame; the target distance value refers to the distance between the reference target and the target to be associated in the same video frame; the target direction angle refers to It is the angle between the line connecting the reference target and the target to be associated with the first specified direction; the target displacement deviation value refers to the displacement deviation of the reference target and the target to be associated after a preset time has passed.

Optionally, the location information corresponding to the target in the embodiment of the present application (that is, the first location information corresponding to the reference target or the second location information corresponding to the target to be associated) specifically includes the target frame corresponding to the target (that is, the area where the target is located). The coordinate information of the smallest enclosing rectangular box), which can specifically include the center coordinates of the target box

The coordinate information of the two opposite corners of the target box, such as the coordinates of the upper left corner of the target box

And the coordinates of the lower right point

In the embodiments of the present application, N is used to represent the identification of a certain target. For example, N ¹ can be used to represent the identification of the reference target, N ² is the identification of the target to be associated, and f is the corresponding frame number in the preset video image sequence. E.g,

Indicates the center coordinates of the target frame of the reference target in _{the video frame with the frame number f 0.}

Optionally, the spatial correlation degree includes an area coincidence degree, and the reference target and the target to be associated are determined according to the first position information and the second position information in the target video frame. Spatial relevance, including:

According to the first position information and the second position information in the target video frame, the target frame overlap area between the target frames corresponding to the reference target and the target to be associated in the same video frame is calculated ；

The degree of overlap of the area of the reference target and the target to be associated is determined according to the overlapping area of the target frame.

The spatial correlation degree in the embodiment of this application at least includes the area coincidence degree

The first position information in the embodiment of the present application includes at least the coordinate information of two diagonal points of the reference target frame corresponding to the reference target, and the second position information in the embodiment of the present application includes at least the target frame to be associated corresponding to the target to be associated. The coordinate information of the two diagonal points. _{Determine the area S 1 of the} reference target frame according to the coordinate information of the two diagonal points of the reference target frame _{; calculate the area S 2 of the} target frame to be associated according to the coordinate information of the two diagonal points of the target frame to be associated; According to the positional relationship between the reference target frame and the target frame to be associated in the same video frame, the target frame overlap area (S ₁ ∩S ₂ ) between the two target frames is determined.

Then, according to the overlapping area of the target frame, calculate the area overlap degree of the reference target and the target to be associated

Specifically, the overlapping area of the target frame is divided by the area of the target frame with the smaller area in the two target frames corresponding to the reference target and the target to be associated, respectively, to obtain the area overlap degree

The calculation formula is as follows:

Optionally, in the embodiment of the present application, the area coincidence degree of the reference target and the target to be associated is specifically calculated in M target video frames, so the average value of the area coincidence degree in the M target video frames can be calculated

Sum variance

As an indicator of the final area overlap. Among them, M is the frame number of the target video frame determined in step B1, M>0; set the frame number f=1 of the first target video frame, and the frame number f=M of the last frame in the target video frame, then:

Mean value of area coincidence

The calculation formula is as follows:

Variance of area coincidence

The calculation formula is as follows:

Optionally, the degree of spatial association includes a target distance value, and the relationship between the reference target and the target to be associated is determined according to the first position information and the second position information in the target video frame. Spatial relevance, including:

Calculate the distance between the center coordinates of the target frame corresponding to the reference target and the target to be associated in the same video frame according to the first position information and the second position information in the target video frame, Obtain the target distance value.

In the embodiment of the present application, the spatial correlation degree at least includes the target distance value, and the first position information includes at least the center coordinates of the target frame corresponding to the reference target

The second location information includes at least the center coordinates of the target frame corresponding to the target to be associated

Calculate the distance between the center coordinates of the two target frames in the same video frame, and obtain the target distance value L ^f between the reference target and the target to be associated in the same video frame. The calculation formula is as follows:

Optionally, in the embodiment of the present application, the correlation between the reference target and the target to be associated is specifically analyzed in M target video frames. Therefore, the average value of the target distance value in the M target video frame can be calculated.

The sum variance D(L ^f ) is used as the index of the final target distance value. Among them, M is the frame number of the target video frame determined in step B1, M>0; set the frame number f=1 of the first target video frame, and the frame number f=M of the last frame in the target video frame, then:

Mean value of target distance

The calculation formula is as follows:

The calculation formula of the variance D(L ^f ) of the target distance value is as follows:

Optionally, the spatial association degree includes a target direction angle, and the reference target and the target to be associated are determined according to the first position information and the second position information in the target video frame. Spatial relevance, including:

According to the first position information and the second position information in the target video frame, determine the connection between the center coordinates of the target frame corresponding to the reference target and the target to be associated in the same video frame Line segment

The target direction angle is determined according to the first designated direction and the connecting line segment.

In the embodiment of the present application, the spatial correlation degree includes at least the target direction angle, and the first position information includes at least the center coordinates of the target frame corresponding to the reference target

Determine the line segment between the reference target and the target to be associated in the same video frame according to the two coordinate information.

The first designated direction in the embodiment of the present application is a reference direction set in advance, and the angle formed by the aforementioned determined connecting line segment and the first designated direction is taken as the target direction angle. Exemplarily, suppose the first designated direction is the vertical direction, and use

Indicates the target direction angle, the calculation formula of the target direction angle is as follows:

Optionally, in the embodiment of the present application, the correlation between the reference target and the target to be associated is specifically analyzed in M target video frames. Therefore, the average value of the target direction angles in the M target video frames can be calculated.

The sum variance D(α ^f ) is used as the index of the final target distance value. Among them, M is the frame number of the target video frame determined in step B1, M>0; set the frame number f=1 of the first target video frame, and the frame number f=M of the last frame in the target video frame, then:

Mean value of target direction angle

The calculation formula is as follows:

The calculation formula of the variance D(α ^f ) of the target direction angle is as follows:

Optionally, the spatial correlation degree includes a target displacement deviation value, the target displacement deviation value includes a displacement distance deviation value and/or a displacement direction angle deviation value, and the first position in the target video frame is Information and the second location information, determining the degree of spatial association between the reference target and the target to be associated, includes:

C1: According to the first position information, calculate the distance between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target to obtain the corresponding The first displacement distance value;

C2: According to the second location information, calculate the distance between the center coordinates of the third target frame corresponding to the first appearance of the target to be associated and the center coordinates of the fourth target frame corresponding to the last appearance of the target to be associated to obtain the target to be associated The corresponding second displacement distance value;

C3: Determine the displacement distance deviation value according to the first displacement distance value and the second displacement distance value;

and / or:

D1: According to the first position information, determine the first displacement line segment between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target;

D2: Determine the first displacement direction angle corresponding to the reference target according to the second designated direction and the first displacement line segment;

D3: According to the second position information, determine the second displacement line segment between the center coordinates of the third target frame corresponding to the first appearance of the target to be associated and the center coordinates of the fourth target frame corresponding to the last appearance;

D4: Determine the second displacement direction angle corresponding to the target to be associated according to the second designated direction and the second displacement line segment;

D5: Determine the angular deviation value of the displacement direction according to the first displacement direction angle and the second displacement direction angle.

In the embodiment of the present application, the degree of spatial correlation includes at least a target displacement deviation value, and the target displacement deviation value may include a displacement distance deviation value and/or a displacement direction angular deviation value. The first position information in the embodiment of the present application includes at least the center coordinates of the target frame corresponding to the reference target

According to the first position information and the second position information, the deviation value of the displacement distance between the reference target and the target to be associated can be determined through the process of step C1 to step C3; the reference target can be determined through the process of step D1 to step D5 The angular deviation of the displacement direction from the target to be associated.

Specifically, in this embodiment of the present application, the association relationship between the reference target and the target to be associated is analyzed in M target video frames, and the M target video frame is the reference target and the target to be associated determined in step B1. Video frame, where M>0. Assuming that the frame number of the first target video frame is f = f ₀ , and the frame number of the last frame in the target video frame is f = f _n , then in the M target video frame, the reference target and the target to be associated correspond to when they first appear The frame numbers of the video frames are both f ₀ , and the frame numbers of the corresponding video frames when the reference target and the target to be associated last appear are both f _n . Correspondingly:

In step C1, the target frame center coordinates of the reference target included in the first position information corresponding to the _{frame number f 0 are determined as the first target frame center coordinates}

Determine the target frame center coordinates of the reference target included in the first position information corresponding to the frame number f _{n as the second target frame center coordinates}

Determine the first displacement distance value corresponding to the reference target according to these two coordinates

The calculation formula is as follows:

In step C2, the center coordinates of the target frame of the target to be associated included in the second position information corresponding to the _{frame number f 0 are determined as the center coordinates of the third target frame}

Determine the center coordinates of the target frame of the target to be associated included in the second position information corresponding to the frame number f _{n as the center coordinates of the fourth target frame}

Determine the second displacement distance value corresponding to the reference target according to these two coordinates

The calculation formula is as follows:

In step C3, calculate the first displacement distance value

And the second displacement distance value

To obtain the displacement distance deviation value Δx, the calculation formula is as follows:

In step D1, according to the center coordinates of the first target frame corresponding to the first appearance of the reference target

And the center coordinates of the second target frame corresponding to the last appearance

Determine the line between these two coordinates as the first displacement line segment corresponding to the reference target.

In step D2, the second designated direction is another reference direction set in advance, and the second designated direction may be the same as or different from the first designated direction. The angle formed by the first displacement line segment determined in step D1 and the second specified direction is determined as the first displacement direction angle corresponding to the reference target. Optionally, the second designated direction is the horizontal direction, using

Represents the first displacement direction angle, the calculation formula of the first displacement direction angle is as follows:

In step D3, according to the center coordinates of the third target frame corresponding to the first appearance of the target to be associated

And the center coordinates of the fourth target frame corresponding to the last appearance

It is determined that the line between these two coordinates is the second displacement line segment corresponding to the target to be associated.

In step D4, compare the second displacement line segment determined in step D3 with

The angle formed by the second designated direction is determined as the second displacement direction angle corresponding to the target to be associated. To express the second displacement direction angle, the calculation formula for the second displacement direction angle is as follows:

In step D5, calculate the first displacement direction angle

And the second displacement direction angle

The deviation value of the displacement direction angle Δα is obtained, and the calculation formula is as follows:

In S105, the correlation confidence of the target to be correlated is calculated according to the temporal coincidence degree and the spatial correlation degree.

Specifically, the time coincidence degree and the spatial correlation degree of the determined reference target and the target to be associated are respectively multiplied by the corresponding weight parameter ω and summed to obtain the correlation confidence degree corresponding to the target to be correlated, and the correlation confidence degree is used to represent the The possibility that the target to be associated has an association relationship with the reference target. Among them, the value of the weight parameter ω can be set according to the actual situation, and ω∈[0,1).

Optionally, the spatial correlation degree includes an area coincidence degree, a target distance value, a target direction angle, and a target displacement deviation value. In the embodiments of the present application, the values of the time correlation degree and the area coincidence degree and the correlation confidence degree are positive correlations. The higher the time correlation degree and the area coincidence degree, the more likely the reference target and the target to be correlated are related; The target distance value, target direction angle, and target displacement deviation value are negatively correlated with the associated confidence. The greater the target distance value, target direction angle, and target displacement deviation value, the more likely the reference target and the target to be associated are related. small. Exemplarily, in the M target video frame, the calculation formula of the correlation confidence is:

Among them, Score represents the value of the correlation confidence, O _t represents the time correlation,

Represents the mean value of area coincidence degree,

Represents the variance of the area coincidence degree,

Represents the mean value of the target distance value, D(L ^f ) represents the variance of the target distance value,

Represents the average value of the target direction angle, D(α ^f ) represents the variance of the target direction angle, Δx is the displacement distance deviation value in the target displacement deviation value, Δα is the displacement direction angle deviation value in the target displacement deviation value, ω ₁ ～ω ₉ is the weight parameter.

In the embodiments of the present application, the value of the correlation confidence is calculated by combining the time coincidence degree and the spatial coincidence degree of multiple indicators including the area coincidence degree, the target distance value, the target direction angle, and the target displacement deviation value, so that the correlation confidence degree calculation can be improved. Moreover, since the weight parameter corresponding to each indicator can be set according to the actual situation, the accuracy and flexibility of the calculation of the correlation confidence can be further improved.

In S106, if the association confidence of the target to be associated satisfies a preset association condition, the target to be associated is determined as an association target that has an association relationship with the reference target.

When the association confidence of the target to be associated satisfies the preset association condition, it means that the target to be associated is more likely to have an association relationship with the reference target, so the target to be associated is determined as an association target that has an association relationship with the reference target. . Optionally, the preset association condition may be that the association confidence is greater than or equal to a preset association confidence threshold.

Optionally, suppose that the reference target is a determined target, and the target to be associated is a pending target among a plurality of preset targets. Correspondingly, before the step S101, the method further includes:

S100: Select one target from the multiple preset targets as the target to be associated;

Correspondingly, the step S106 includes:

If it is detected that there is still a target with an undetermined association confidence level among the plurality of preset targets, return to perform the step S100;

If it is detected that each of the plurality of preset targets has determined the corresponding association confidence, then the target to be associated with the highest corresponding association confidence among the plurality of preset targets is regarded as the to-be-associated target that meets the preset association conditions Target, and determine the to-be-associated target as an associated target that has an associated relationship with the reference target.

Through the method of the embodiment of the present application, it is possible to accurately determine an associated target that has an associated relationship with the reference target from a plurality of preset targets.

In the embodiments of the present application, since the temporal coincidence degree and the spatial correlation degree of the reference target and the target to be associated can be combined, the correlation confidence of the target to be correlated can be accurately calculated from the two aspects of the time dimension and the space dimension, and the correlation confidence is determined according to the correlation confidence. The association target that has an association relationship with the reference target is accurately determined, so that the accuracy of the target association of the video image can be improved.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

Embodiment two:

FIG. 2 shows a schematic structural diagram of a video image target association device provided by an embodiment of the present application. For ease of description, only the parts related to the embodiment of the present application are shown:

The video image target association device includes: an acquisition unit 21, a target video frame determination unit 22, a temporal coincidence degree determination unit 23, a spatial correlation degree determination unit 24, an association confidence degree calculation unit 25, and an association target determination unit 26. in:

The acquiring unit 21 is configured to acquire first data corresponding to a reference target and acquire second data corresponding to a target to be associated, wherein the first data includes the first data of the reference target in a preset video image sequence. Frame number information and corresponding first position information, and the second data includes second frame number information and corresponding second position information of the target to be associated in the preset video image sequence.

The target video frame determining unit 22 is configured to determine, according to the first frame sequence number information and the second frame sequence number information, the time of appearance of the reference target and the target to be associated in the preset video image sequence and At the moment of disappearance, the target video frame is obtained.

The time coincidence degree determining unit 23 is configured to determine the time coincidence degree between the reference target and the target to be associated according to the number of frames in which the reference target and the target to be associated exist simultaneously in the target video frame.

The spatial association degree determining unit 24 is configured to determine the spatial association degree between the reference target and the target to be associated according to the first position information and the second position information in the target video frame.

The correlation confidence calculation unit 25 is configured to calculate the correlation confidence of the target to be associated according to the temporal coincidence degree and the spatial correlation degree.

The associated target determination unit 26 is configured to determine the to-be-associated target as an associated target that has an associated relationship with the reference target if the association confidence of the to-be-associated target satisfies a preset association condition.

Optionally, the spatial correlation degree includes an area coincidence degree, and the spatial correlation degree determining unit 24 includes:

The area coincidence degree determining unit is configured to calculate the target corresponding to the reference target and the target to be associated in the same video frame according to the first position information and the second position information in the target video frame The overlap area of the target frame between the frames; the overlap degree of the area of the reference target and the target to be associated is determined according to the overlap area of the target frame.

Optionally, the spatial correlation degree includes a target distance value, and the spatial correlation degree determining unit 24 includes:

The target distance value determining unit is configured to calculate the target corresponding to the reference target and the target to be associated in the same video frame according to the first position information and the second position information in the target video frame The distance between the center coordinates of the frame to obtain the target distance value.

Optionally, the spatial correlation degree includes a target direction angle, and the spatial correlation degree determining unit 24 includes:

A target direction angle determination unit, configured to determine the target corresponding to the reference target and the target to be associated in the same video frame according to the first position information and the second position information in the target video frame The connecting line segment between the frame center coordinates; the target direction angle is determined according to the first designated direction and the connecting line segment.

Optionally, the spatial correlation degree includes a target displacement offset value, the target displacement offset value includes a displacement distance deviation value and/or a displacement direction angle deviation value, and the spatial correlation degree determining unit 24 includes a displacement distance deviation value Determining the unit and/or the displacement direction angular deviation value determining unit:

The displacement distance deviation value determining unit is configured to calculate the distance between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target according to the first position information, Obtain the first displacement distance value corresponding to the reference target; according to the second position information, calculate the center coordinates of the third target frame corresponding to the first appearance of the target to be associated and the center coordinates of the fourth target frame corresponding to the last appearance of the target to be associated To obtain the second displacement distance value corresponding to the target to be associated; determine the displacement distance deviation value according to the first displacement distance value and the second displacement distance value;

The displacement direction angle determining unit is configured to determine, according to the first position information, the first displacement between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target Line segment; according to the second designated direction and the first displacement line segment, determine the first displacement direction angle corresponding to the reference target; according to the second position information, determine the third target corresponding to the first appearance of the target to be associated A second displacement line segment between the frame center coordinates and the corresponding fourth target frame center coordinates when it appeared last; determining the second displacement direction angle corresponding to the target to be associated according to the second designated direction and the second displacement line segment; The deviation value of the displacement direction angle is determined according to the first displacement direction angle and the second displacement direction angle.

It should be noted that the information interaction and execution process between the above-mentioned devices/units are based on the same concept as the method embodiment of this application, and its specific functions and technical effects can be found in the method embodiment section. I won't repeat it here.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The above-mentioned integrated units can be hardware-based Formal realization can also be realized in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working process of the units and modules in the foregoing system, reference may be made to the corresponding process in the foregoing method embodiment, which will not be repeated here.

Embodiment three:

Fig. 3 is a schematic diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 3, the terminal device 3 of this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and running on the processor 30, such as a video image object association program . When the processor 30 executes the computer program 32, the steps in the above embodiments of the video image object association method are implemented, for example, steps S101 to S106 shown in Fig. 1. Alternatively, when the processor 30 executes the computer program 32, the functions of the modules/units in the foregoing device embodiments, for example, the functions of the units 21 to 26 shown in FIG. 2 are realized.

Exemplarily, the computer program 32 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 31 and executed by the processor 30 to complete This application. The one or more modules/units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the terminal device 3. For example, the computer program 32 may be divided into an acquisition unit, an association degree determination unit, an association confidence degree calculation unit, and an association target determination unit. The specific functions of each unit are as follows:

The acquiring unit is configured to acquire first data corresponding to the reference target and acquire second data corresponding to the target to be associated, wherein the first data includes the first frame of the reference target in the preset video image sequence Sequence number information and corresponding first location information, and the second data includes second frame sequence number information and corresponding second location information of the target to be associated in the preset video image sequence.

A target video frame determining unit, configured to determine the appearance and disappearance of the reference target and the target to be associated in the preset video image sequence according to the first frame sequence number information and the second frame sequence number information At the moment, the target video frame is obtained.

The time coincidence degree determining unit is configured to determine the time coincidence degree of the reference target and the target to be associated according to the number of frames in the target video frame where the reference target and the target to be associated exist simultaneously.

The spatial association degree determining unit is configured to determine the spatial association degree between the reference target and the target to be associated according to the first position information and the second position information in the target video frame.

The correlation confidence calculation unit is configured to calculate the correlation confidence of the target to be associated according to the temporal coincidence degree and the spatial correlation degree.

The terminal device 3 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art can understand that FIG. 3 is only an example of the terminal device 3, and does not constitute a limitation on the terminal device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, and so on.

The so-called processor 30 may be a central processing unit (Central Processing Unit, CPU), other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.

The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or a memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk equipped on the terminal device 3, a smart memory card (Smart Media Card, SMC), and a Secure Digital (SD) Card, Flash Card, etc. Further, the memory 31 may also include both an internal storage unit of the terminal device 3 and an external storage device. The memory 31 is used to store the computer program and other programs and data required by the terminal device. The memory 31 can also be used to temporarily store data that has been output or will be output.

In the above-mentioned embodiments, the description of each embodiment has its own focus. For parts that are not described in detail or recorded in an embodiment, reference may be made to related descriptions of other embodiments.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Claims

A method for associating video image targets, characterized in that it comprises:

Acquire first data corresponding to the reference target, and acquire second data corresponding to the target to be associated, where the first data includes the first frame number information of the reference target in the preset video image sequence and the corresponding First location information, where the second data includes second frame number information and corresponding second location information of the target to be associated in the preset video image sequence;

Determine, according to the first frame sequence number information and the second frame sequence number information, the appearance and disappearance moments of the reference target and the target to be associated in the preset video image sequence to obtain a target video frame;

Determine the temporal coincidence degree of the reference target and the target to be associated according to the number of frames in the target video frame where the reference target and the target to be associated simultaneously exist;

Determine the degree of spatial association between the reference target and the target to be associated according to the first position information and the second position information in the target video frame;

Calculating the correlation confidence of the target to be correlated according to the temporal coincidence degree and the spatial correlation degree;

If the association confidence of the target to be associated satisfies a preset association condition, the target to be associated is determined as an association target that has an association relationship with the reference target.
The video image target association method according to claim 1, wherein the spatial association degree includes any one or more of area coincidence, target distance value, target direction angle, and target displacement deviation value.
The method for associating video image objects according to claim 1, wherein the spatial association degree includes an area coincidence degree, and the first position information and the second position information in the target video frame are , Determining the degree of spatial association between the reference target and the target to be associated includes:

According to the first position information and the second position information in the target video frame, the target frame overlap area between the target frames corresponding to the reference target and the target to be associated in the same video frame is calculated ；

The degree of overlap of the area of the reference target and the target to be associated is determined according to the overlapping area of the target frame.
The method for associating video image objects according to claim 1, wherein the spatial association degree includes a target distance value, and the first position information and the second position information in the target video frame are , Determining the degree of spatial association between the reference target and the target to be associated includes:

Calculate the distance between the center coordinates of the target frame corresponding to the reference target and the target to be associated in the same video frame according to the first position information and the second position information in the target video frame, Obtain the target distance value.
The method for associating video image objects according to claim 1, wherein the spatial association degree includes a target direction angle, and the first position information and the second position information in the target video frame are , Determining the degree of spatial association between the reference target and the target to be associated includes:

According to the first position information and the second position information in the target video frame, determine the connection between the center coordinates of the target frame corresponding to the reference target and the target to be associated in the same video frame Line segment

The target direction angle is determined according to the first designated direction and the connecting line segment.
The method for associating a video image target according to claim 1, wherein the spatial correlation degree includes a target displacement deviation value, and the target displacement deviation value includes a displacement distance deviation value and/or a displacement direction angular deviation value, and the According to the first position information and the second position information in the target video frame, determining the degree of spatial association between the reference target and the target to be associated includes:

According to the first position information, the distance between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target is calculated, and the first corresponding to the reference target is obtained. Displacement distance value;

According to the second position information, the distance between the center coordinates of the third target frame corresponding to the first appearance of the target to be associated and the center coordinates of the fourth target frame corresponding to the last appearance of the target to be associated is calculated to obtain the corresponding target of the target to be associated The second displacement distance value;

Determining the displacement distance deviation value according to the first displacement distance value and the second displacement distance value;

and / or:

Determine, according to the first position information, a first displacement line segment between the center coordinates of the first target frame corresponding to the first appearance of the reference target and the center coordinates of the second target frame corresponding to the last appearance of the reference target;

Determine the first displacement direction angle corresponding to the reference target according to the second designated direction and the first displacement line segment;

Determine, according to the second position information, a second displacement line segment between the center coordinates of the third target frame corresponding to the first appearance of the target to be associated and the center coordinates of the fourth target frame corresponding to the last appearance;

Determine the second displacement direction angle corresponding to the target to be associated according to the second designated direction and the second displacement line segment;

The deviation value of the displacement direction angle is determined according to the first displacement direction angle and the second displacement direction angle.
A device for associating video and image objects, which is characterized in that it comprises:

The acquiring unit is configured to acquire first data corresponding to the reference target and acquire second data corresponding to the target to be associated, wherein the first data includes the first frame of the reference target in the preset video image sequence Serial number information and corresponding first position information, where the second data includes second frame serial number information and corresponding second position information of the target to be associated in the preset video image sequence;

A target video frame determining unit, configured to determine the appearance and disappearance of the reference target and the target to be associated in the preset video image sequence according to the first frame sequence number information and the second frame sequence number information At the moment, get the target video frame;

A time coincidence degree determining unit, configured to determine the time coincidence degree of the reference target and the target to be associated according to the number of frames in the target video frame where the reference target and the target to be associated exist at the same time;

A spatial association degree determining unit, configured to determine the spatial association degree between the reference target and the target to be associated according to the first position information and the second position information in the target video frame;

An association confidence degree calculation unit, configured to calculate the association confidence degree of the target to be associated according to the temporal coincidence degree and the spatial correlation degree;

The associated target determination unit is configured to determine the to-be-associated target as an associated target that has an association relationship with the reference target if the association confidence of the to-be-associated target meets a preset association condition.
8. The video image target association device according to claim 7, wherein the spatial association degree comprises any one or more of area coincidence degree, target distance value, target direction angle, and target displacement deviation value.
A terminal device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, when the processor executes the computer program, the terminal device realizes The steps of the method according to any one of claims 1 to 6.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, a terminal device is enabled to implement any one of claims 1 to 6 Method steps.