KR20210129189A - Target detection method, apparatus, electronic device and computer readable storage medium - Google Patents

Abstract

An embodiment of the present invention provides a target detection method, an apparatus, an electronic device, and a computer-readable storage medium, wherein the target detection method includes: acquiring an image to be detected (S201); Based on the image to be detected, each vertex determines vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex (S202) - the vertex determines the position of the target object in the image to be detected indicate; and determining ( S203 ) a target object in the to-be-detected image based on the vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex.

Description

Target detection method, apparatus, electronic device and computer readable storage medium

[Cross-Reference to Related Applications]

This application has an application number 202010073142.6, filed on January 22, 2020 based on a Chinese patent application, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference do.

The present invention relates to the field of image recognition technology, and more particularly, to a target detection method, an apparatus, an electronic device, and a computer-readable storage medium.

Target detection is an important fundamental problem in computer vision, and many computer vision applications all rely on target detection, such as autonomous driving, video monitoring and mobile entertainment.

In case of target detection, the main task is to use the detection frame to mark the position of the object in the image, the process can determine the position of the object in the image based on the target detection algorithm of the object key point, After determining all the object key points of , the detection frame of the object is obtained by matching the object key points belonging to the same object.

However, when the image contains several similarly shaped objects, the degree of agreement between the key points of the objects corresponding to the similar objects is relatively high, and it is easy to cause erroneous detection results, for example, in the detection frame with the same detection results can include several objects, and therefore, the current target detection method has a relatively low detection accuracy.

An embodiment of the present invention provides at least one target detection method.

According to a first aspect of an embodiment of the present invention, there is provided a method for detecting a target, the method comprising:

acquiring an image to be detected;

Determining a centripetal shift tensor corresponding to each vertex and vertex position information in the image where each vertex is to be detected, based on the to-be-detected image; Indicates location - ; and

and determining a target object in the image to be detected based on vertex position information in the image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex.

In the method provided in the embodiment of the present invention, after obtaining an image to be detected, vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex must first be determined, which means that the vertices are located in the image. Since it is a main feature point, the position information of the vertices in the image to be detected may indicate the position of each target object in the image to be detected, for example, the vertices may include an upper left vertex and a lower right vertex, where the left The upper vertex indicates the intersection of the line corresponding to the upper contour of the target object and the line corresponding to the left contour of the target object, and the lower right vertex indicates the intersection of the line corresponding to the lower contour of the target object and the line corresponding to the right contour of the target object. , when the upper left vertex and the lower right vertex belong to the detection frame of the same target, the positions indicated by the centripetal shift tensors corresponding to the upper left vertex and the lower right vertex may be relatively close. The proposed target detection method determines vertices belonging to the same target object based on vertex position information in the image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex, thereby determining the same target object based on the determined vertices. can be detected.

In one possible embodiment, based on the image to be detected, each vertex determining vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex comprises:

performing feature extraction on the to-be-detected image to obtain an initial feature map corresponding to the to-be-detected image;

performing vertex pooling on the initial feature map to obtain a vertex pooled feature map; and

and obtaining vertex position information in the image where each vertex is to be detected, and a centripetal shift tensor corresponding to each vertex, based on the vertex pooled feature map.

In the method provided in the embodiment of the present invention, an initial feature map is obtained through feature extraction on the image to be detected, and vertex pooling is performed on the initial feature map, thereby vertex and centripetal corresponding to the vertex. A feature map that is easy to extract the shift tensor is obtained, that is, a vertex pooled feature map is obtained.

In one possible embodiment, based on the vertex pooled feature map, the step of obtaining vertex position information in the image where each vertex is to be detected comprises:

generating a vertex heat map corresponding to the image to be detected based on the vertex pooled feature map;

Based on the vertex heat map, determining a probability value using each feature point in the vertex heat map as a vertex, and selecting the vertex from the feature point of the vertex heat map based on the probability value using each feature point as a vertex ;

Obtaining the location information of each selected vertex in the vertex heat map and local shift information corresponding to each vertex - The local shift information includes the actual physical point indicated by the corresponding vertex position shift information in the vertex heat map. Used to indicate - ; and

Based on the position information of each obtained vertex in the vertex heatmap, local shift information corresponding to each vertex, and the size ratio between the vertex heatmap and the image to be detected, each vertex is a vertex in the image to be detected. determining location information.

The method of determining the vertex position information in the image where each vertex is to be detected provided in the embodiment of the present invention is through quoting the vertex heat map in the above process, and through a probability value using each feature point as a vertex as a vertex Determine the vertex position information in the image where the vertex is to be detected by determining the feature points that can be used, selecting the vertices, and then modifying the vertex position information in the vertex heatmap, the method is relatively accurate Since the vertex position information of the high vertex can be obtained, it is easy to detect the position in the image where the target object is to be detected based on the vertex.

In one possible embodiment, the step of determining a centripetal shift tensor corresponding to each vertex based on the vertex pooled feature map comprises:

determining a guide shift tensor corresponding to each feature point in the vertex pooled feature map, based on the vertex pooled feature map; Represents the shift tensor of the target target center point - ;

Determining shift domain information of the feature point based on the guide shift tensor corresponding to each feature point, wherein the shift domain information is a shift tensor in which a plurality of initial feature points associated with the feature point respectively indicate shifted feature points respectively. including - ;

Based on the shift domain information of the feature points in the vertex pooled feature map and the vertex pooled feature map, adjusting the feature data of feature points in the vertex pooled feature map to obtain an adjusted feature map step - ; and

and determining a centripetal shift tensor corresponding to each vertex based on the adjusted feature map.

The process of determining the centripetal shift tensor provided in the embodiment of the present invention is a vertex-pooled feature map by considering the target target information, for example, by citing the guide shift tensor corresponding to the vertex and the shift domain information of the feature point. By adjusting the feature data of the feature points in , the feature data of the feature points in the obtained adjusted feature map includes richer target target information, so that a more accurate centripetal shift tensor corresponding to each vertex can be determined, Through an accurate centripetal shift tensor, accurate center point position information pointed to by a vertex can be obtained, so that a position of a target object in an image to be detected can be accurately detected.

In one possible embodiment, the vertex heatmap corresponding to the image to be detected comprises a vertex heatmap respectively corresponding to a plurality of channels, each channel in the plurality of channels corresponding to one preset target category, ; After determining a probability value using each feature point as a vertex in the vertex heat map based on the vertex heat map, the detection method includes:

For each channel in the plurality of channels, based on a probability value using each feature point in the vertex heatmap corresponding to the channel as a vertex, it is determined whether the vertex exists in the vertex heatmap corresponding to the channel. to do; and

The method further includes, when the vertex exists in the vertex heatmap corresponding to the channel, including a target object of a preset target category corresponding to the channel in the image to be detected.

The method provided in the embodiment of the present invention obtains a vertex heatmap including the number of prior channels by inputting the vertex pooled feature map to the vertex heatmap prediction network, and in the vertex heatmap corresponding to each channel By determining whether a vertex exists, it may be determined whether a target object of a category corresponding to the channel exists in the image to be detected.

In one possible embodiment, the step of determining a target object in the image to be detected comprises: based on the vertex position information in the image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex,

and determining a detection frame of a target object in the image to be detected based on a centripetal shift tensor corresponding to each vertex of the vertex position information in the image where each vertex is to be detected.

The method provided in the embodiment of the present invention determines the detection frame of each target object based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex, and each target object is Position information in the image to be detected may be acquired.

In one possible embodiment, the step of determining the detection frame of the target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex comprises: ,

selecting a pair of candidate vertices capable of constituting a candidate detection frame based on vertex position information in the image where each vertex is to be detected;

determining center point position information indicated by the vertex based on the vertex position information in the image where each vertex is to be detected in each candidate vertex pair and a centripetal shift tensor corresponding to each vertex;

determining center region information corresponding to the candidate vertex pair based on the vertex position information in the image in which each vertex is to be detected in each candidate vertex pair; and

and determining a detection frame of the target object in the candidate detection frame based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair.

In the method of determining the detection frame of the target object proposed in the embodiment of the present invention, the candidate vertex pair constituting the candidate detection frame is first determined through the vertex position information of the vertex, and then corresponds to each vertex in the candidate vertex pair. The detection frames of all target objects in the image to be detected can be detected relatively accurately by determining whether the target object surrounding the candidate detection frame is the same target object based on the obtained centripetal shift tensor.

In one possible embodiment, the determining of center region information corresponding to the candidate vertex pair comprises:

determining vertex position information indicating a central region frame corresponding to the candidate vertex pair based on vertex position information of each vertex in the candidate vertex pair; and

and determining a coordinate range of the central region frame corresponding to the candidate vertex pair based on the vertex position information of the central region frame.

In one possible embodiment, the detection frame of the target object in the candidate detection frame is determined based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair step is,

determining a valid candidate vertex pair based on center location information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair;

Corresponding to each valid candidate vertex pair based on center point location information indicated by each vertex in the valid candidate vertex pair, center region information corresponding to the valid candidate vertex pair, and a probability value corresponding to each vertex in the valid candidate vertex pair determining a score of a candidate detection frame, wherein a probability value corresponding to each vertex is used to indicate a probability value in which the vertex uses a feature point corresponding to a vertex heat map as a vertex; and

and determining a detection frame of the target object in the candidate detection frame based on a score of a candidate detection frame corresponding to each valid candidate vertex pair and a size of an overlapping region between adjacent candidate detection frames.

The method provided in the embodiment of the present invention determines to select a candidate detection frame representing only one target target candidate through valid selection of candidate vertex pairs constituting the candidate detection frame, and then A detection frame indicating an accurate target object is obtained by performing soft band non-band suppression selection on a candidate detection frame indicating only the target object candidate.

In one possible embodiment, after determining the detection frame of the target object in the image to be detected, the target detection method comprises:

The method further includes determining instance information of the target object in the image to be detected based on the detection frame of the target object and an initial feature map obtained by performing feature extraction on the image to be detected.

The method provided in the embodiment of the present invention may determine instance information of a target object, wherein the instance refers to providing a pixel of each target object in a pixel layer after performing instance segmentation on the target object in the image, Instance segmentation is accurate up to the edge of the object, so that more accurate location information can be obtained from the image in which the target object is to be detected.

In one possible embodiment, based on an initial feature map obtained by performing feature extraction on the detection frame of the target object and the image to be detected, determining instance information of the target object in the image to be detected step is,

extracting, by the initial feature map, feature data of feature points in the detection frame based on the detection frame of the target object and the initial feature map; and

and determining, by the initial feature map, instance information of the target object in the image to be detected, based on feature data of feature points in the detection frame.

In one possible embodiment, the target detection method is implemented through a neural network, and the neural network is acquired by training using a sample image including a labeling target sample object.

In one possible embodiment, the neural network comprises:

acquiring a sample image;

Based on the sample image, each sample vertex determines vertex position information in the sample image and a centripetal shift tensor corresponding to each sample vertex, wherein the sample vertex represents the position of a target sample object in the sample image. ;

predicting a target object in the sample image based on vertex position information of each sample vertex in the sample image and a centripetal shift tensor corresponding to each sample vertex; and

Based on the predicted target sample object in the sample image and the labeling target sample object in the sample image, it is obtained by training using the step of adjusting the value of the network parameter of the neural network.

The training method of a neural network provided in an embodiment of the present invention includes obtaining a sample image, based on the sample image, in which each sample vertex corresponds to vertex position information in the sample image, and each sample vertex. By determining the centripetal shift tensor, a target sample object in the sample image is detected where each sample vertex is based on the vertex position information in the sample image and the centripetal shift tensor corresponding to each sample vertex, where the sample vertex is located in the image. This is because it indicates a main feature point, for example, the sample vertex may include an upper left sample vertex and a lower right sample vertex, where the upper left sample vertex is a line corresponding to the upper contour of the target sample object and the left side of the target sample object It points to the intersection of the lines corresponding to the contour, and the lower right sample vertex points to the intersection of the line corresponding to the lower contour of the target sample object and the line corresponding to the right contour of the target sample object, and the upper left sample vertex and the lower right sample vertex are the same target When the sample object belongs to the detection frame, the positions indicated by the centripetal shift tensors corresponding to the upper left sample vertex and the lower right sample vertex may be relatively close. Therefore, the training method of the neural network proposed in the embodiment of the present invention is , by determining a sample vertex belonging to the same target sample object based on the vertex position information of the vertex indicating the position in the sample image of the target sample object and the centripetal shift tensor corresponding to each sample vertex, based on the determined sample vertex. can detect the same target sample object, and then continuously based on the labeling object in the sample image, through continuously adjusting the neural network parameter value, thereby obtaining a neural network with a relatively high accuracy, wherein the accuracy is It is possible to accurately detect a target object based on a relatively high neural network. According to a second aspect of an embodiment of the present invention, there is provided a target detection apparatus, the apparatus comprising:

an acquiring unit configured to acquire an image to be detected;

a determiner configured to determine, based on the image to be detected, a centripetal shift tensor corresponding to each vertex and vertex position information in the image for each vertex to be detected, the vertex indicating the position of the target object in the image to be detected - ;

and a detection unit configured to determine a target object in the image where each vertex is to be detected based on vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the determining unit comprises:

extracting a feature from the image to be detected to obtain an initial feature map corresponding to the image to be detected;

vertex pooling is performed on the initial feature map to obtain a vertex pooled feature map;

and determine vertex position information for each vertex in the to-be-detected image, and a centripetal shift tensor corresponding to each vertex, based on the vertex pooled feature map.

In one possible embodiment, when the determining unit is configured to determine vertex position information in the image where each vertex is to be detected, based on the vertex pooled feature map, the determining unit comprises:

generating a vertex heat map corresponding to the image to be detected based on the vertex pooled feature map;

determining a probability value using each feature point in the vertex heat map as a vertex based on the vertex heat map, and selecting the vertex from the feature point based on a probability value using each feature point as a vertex;

Each of the selected vertices obtains position information in the vertex heat map and local shift information corresponding to each vertex, and the local shift information indicates that the actual physical point indicated by the corresponding vertex indicates position shift information in the vertex heat map. used to - ;

Based on the position information of each obtained vertex in the vertex heatmap, local shift information corresponding to each vertex, and the size ratio between the vertex heatmap and the image to be detected, each vertex is a vertex in the image to be detected. and determine location information.

In one possible embodiment, when the determination unit determines a centripetal shift tensor corresponding to each vertex based on the vertex pooled feature map, the determination unit comprises:

Based on the vertex-pooled feature map, a guide shift tensor corresponding to each feature in the vertex-pooled feature map is determined; represents a shift tensor pointing to - ;

Determining shift domain information of the feature point based on the guide shift tensor corresponding to each feature point, wherein the shift domain information includes a shift tensor in which a plurality of initial feature points associated with the feature point respectively indicate shifted feature points respectively Ham - ;

Based on the shift domain information of the feature points in the vertex pooled feature map and the vertex pooled feature map, adjusting the feature data of the feature points in the vertex pooled feature map to obtain an adjusted feature map, ;

Based on the adjusted feature map, it is configured to determine a centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the vertex heatmap corresponding to the image to be detected comprises a vertex heatmap respectively corresponding to a plurality of channels, each channel in the plurality of channels corresponding to one preset target category, ; After the determining unit is configured to determine a probability value using each feature point as a vertex in the vertex heat map based on the vertex heat map, the determining unit further comprises:

For each channel in the plurality of channels, based on a probability value using each feature point in the vertex heatmap corresponding to the channel as a vertex, determine whether the vertex exists in the vertex heatmap corresponding to the channel, and ,

and when the vertex exists in the vertex heatmap corresponding to the channel, determine that the target object of a preset object category corresponding to the channel is included in the image to be detected.

In one possible embodiment, the detection unit comprises:

and determine a detection frame of a target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the detection unit determines the detection frame of the target object in the image to be detected, based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex. When configured to, the detection unit,

Based on the vertex position information in the image where each vertex is to be detected, a pair of candidate vertices that can constitute a candidate detection frame is selected,

determining center point location information indicated by the vertex based on the vertex location information in the image where each vertex is to be detected in each candidate vertex pair and the centripetal shift tensor corresponding to each vertex;

determining center region information corresponding to the candidate vertex pair based on the vertex position information in the image where each vertex is to be detected in each candidate vertex pair;

and determine the detection frame of the target object in the candidate detection frame based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair.

In one possible embodiment, when the detection unit is configured to determine center region information corresponding to the candidate vertex pair based on the vertex position information in the image where each vertex is to be detected in each candidate vertex pair, the the detection unit,

determining vertex position information indicating a central region frame corresponding to the candidate vertex pair based on vertex position information of each vertex in the candidate vertex pair;

and determine a coordinate range of the central region frame corresponding to the candidate vertex pair based on the vertex position information of the central region frame.

In one possible embodiment, the detection unit is based on the center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair, the detection frame of the target object in the candidate detection frame When configured to determine, the detection unit,

determining a valid candidate vertex pair based on center position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair;

Corresponding to each valid candidate vertex pair based on center point location information indicated by each vertex in the valid candidate vertex pair, center region information corresponding to the valid candidate vertex pair, and a probability value corresponding to each vertex in the valid candidate vertex pair determines the score of the candidate detection frame to be used - a probability value corresponding to each vertex is used to indicate a probability value using a feature point corresponding to the vertex heat map as a vertex;

and determine the detection frame of the target object in the candidate detection frame, based on a score of a candidate detection frame corresponding to each valid candidate vertex pair and a size of an overlapping region between adjacent candidate detection frames.

In one possible embodiment, the detection unit further comprises:

After determining the detection frame of the target object in the image to be detected, based on the detection frame of the target object and the initial feature map obtained by performing feature extraction on the image to be detected, the target in the image to be detected and determine instance information of the target.

In one possible embodiment, based on the detection unit detection frame of the target object and an initial feature map obtained by performing feature extraction on the image to be detected, instance information of the target object in the image to be detected When configured to determine, the detection unit,

the initial feature map extracts feature data of feature points in the detection frame based on the detection frame of the target object and the initial feature map;

The initial feature map is configured to determine instance information of the target object in the to-be-detected image based on feature data of feature points in the detection frame.

In one possible embodiment, the target detection device further comprises a neural network training unit, the neural network training unit,

configured to train a neural network to proceed with target detection, wherein the neural network is obtained by training using a sample image including a labeling target sample object;

In one possible embodiment, the neural network training unit comprises:

acquiring a sample image;

based on the sample image, each sample vertex determines vertex position information in the sample image and a centripetal shift tensor corresponding to each sample vertex, wherein the sample vertex indicates a position of a target sample object in the sample image;

predicting a target object in the sample image based on vertex position information of each sample vertex in the sample image and a centripetal shift tensor corresponding to each sample vertex;

and proceed to adjust the value of the network parameter of the neural network based on the predicted target sample object in the sample image and the labeling target sample object in the sample image.

According to a third aspect of an embodiment of the present invention, there is provided an electronic device, the electronic device comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, the electronic device is activated, communicates between the processor and the memory via a bus, and executes the target detection method step of the first aspect when the machine readable instructions are executed by the processor.

According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is operated by a processor, the target of the first aspect Execute the detection method steps.

According to a fifth aspect of an embodiment of the present invention, there is provided a computer program including computer readable code, wherein when the computer readable code is run in an electronic device, the processor in the electronic device detects the target of the first aspect to implement the method steps.

In order to make the above features and advantages of the present invention clearer and easier to understand, a comparatively preferred embodiment is given below as an example, and in combination with the accompanying drawings, it will be described in detail.

In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings to be used in the embodiments are briefly described below, and the drawings are a part of the specification and constitute the entire specification, and these drawings represent the present invention. It shows an embodiment conforming to the above, and is for explaining the technical solution of the present invention together with the specification. It is to be understood that the drawings below only show some embodiments of the present invention, and therefore should not be considered as limiting in scope, and those skilled in the art may use other Associated drawings can be obtained.
1 is an exemplary diagram of results obtained when detection is performed on an image to be detected.
2 is a flowchart of an exemplary target detection method provided in an embodiment of the present invention.
3 is a flowchart for determining vertex position information and a centripetal shift tensor corresponding to the vertex provided in an embodiment of the present invention.
4 is another flowchart of determining the vertex position information and the centripetal shift tensor corresponding to the vertex provided in the embodiment of the present invention.
5 is a flowchart of determining a centripetal shift tensor corresponding to a vertex provided in an embodiment of the present invention.
6 is a flowchart illustrating an exemplary feature adjustment network provided in an embodiment of the present invention for adjusting a vertex-pooled feature map.
7 is a flowchart for determining a category of a target object provided in an embodiment of the present invention.
8 is a flowchart of determining a detection frame of a target object provided in an embodiment of the present invention.
9 is a flowchart of determining a detection frame of a target object based on each candidate vertex pair provided in an embodiment of the present invention.
10 is a flowchart corresponding to an exemplary target detection method provided in an embodiment of the present invention.
11 is a flowchart of a neural network training method provided in an embodiment of the present invention.
12 is an exemplary structural diagram of a target detection apparatus provided in an embodiment of the present invention.
13 is an exemplary structural diagram of an electronic device provided in an embodiment of the present invention.

In order to make the technical solutions and advantages of the present invention more clear, below, in combination with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described, It is apparent that the described embodiments are only some of the embodiments rather than all of the embodiments of the present invention. In general, the components of the embodiments of the invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Accordingly, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the protection scope of the present invention, but to represent selected embodiments of the present invention. Based on the embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative labor should fall within the protection scope of the embodiments of the present invention.

When target detection is performed on an image, when a plurality of similar target objects are included in the image, as shown in FIG. 1 , when a plurality of similar planes exist in the image, a key point based on the plane is used in the image When detection is carried out on the plane of When detection is performed on a target object in , the precision of the detection result is relatively low. In contrast, the target detection method provided in the embodiment of the present invention can increase the accuracy of the detection result.

In the target detection method provided in the embodiment of the present invention based on the above study, after acquiring an image to be detected, first, each vertex determines vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex. However, since the vertex is a major feature point in the image, the position information of the vertex in the image to be detected may indicate the position of each target object in the image to be detected, for example, the vertex is the upper left vertex and the lower right and a vertex, wherein the upper left point indicates an intersection of a line corresponding to the upper contour of the target object and a line corresponding to the left contour of the target object, and the lower right vertex is a line corresponding to the lower contour of the target object and the target object. It points to the intersection of the lines corresponding to the right contour, and if the upper left vertex and the lower right vertex are the same target target detection frame, the position indicated by the centripetal shift tensor corresponding to the upper left vertex and the lower right vertex may be relatively close. The target detection method provided by the embodiment of the present invention determines the vertices belonging to the same target object based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex, thereby determining the determined vertices. based on the same target object can be detected.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is clear that the described embodiments are only some of the embodiments of the present invention, and not all embodiments. do. In general, the components of the embodiments of the invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Accordingly, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the protection scope of the embodiments of the present invention, but to represent selected embodiments of the present invention. Based on the embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative labor should fall within the protection scope of the embodiments of the present invention.

It should be noted that, since similar symbols and characters indicate similar items in the drawings below, if any one item is defined in one drawing, there is no need to proceed with definition and interpretation in subsequent drawings.

In order to facilitate understanding of the embodiments of the present invention, first, the target detection method disclosed in the embodiments of the present invention will be described in detail, and the execution subject of the target detection method provided in the embodiments of the present invention is generally a computer having a certain computational capability. device, and the computer device includes a terminal device or a server or other processing device. In some possible implementation manners, the target detection method may be implemented through a method in which a processor calls a computer readable instruction stored in a memory.

Referring to FIG. 2 , it is a flowchart of a target detection method provided in an embodiment of the present invention, and the method includes steps S201 to 203 .

In step S201, an image to be detected is acquired.

Here, the image to be detected may be an image to be detected for a specific environment. For example, when detecting a vehicle at a specified traffic entrance, a camera may be installed at the traffic entrance, and the camera may be used for a certain time of the traffic entrance. after collecting the video stream in the video stream, framing processing is performed on the video stream to obtain an image to be detected; Alternatively, the animals of a specified zoo may be detected, a camera may be installed in the zoo, and a video stream within a certain time of the zoo may be collected with the camera, and then framing processing is performed on the video stream to be detected. Acquire an image.

Here, the image to be detected may include a target object, wherein the target object points to an object to be detected under a specific environment, for example, a vehicle at a traffic entrance specified as described above, and an animal in a specified zoo, The target object may not be included, and when the target object is not included, the detection result is empty, and the embodiment of the present invention proceeds to explain the image to be detected including the target object

In step S202, based on the image to be detected, vertex position information for each vertex in the image to be detected and a centripetal shift tensor corresponding to each vertex are determined, wherein the vertex determines the position of the target object in the image to be detected. indicates.

The position in the image where the target object is to be detected may be indicated through the detection frame, and the embodiment of the present invention indicates the position in the image where the target object is to be detected through a vertex, where the vertex may be a vertex of the detection frame, , for example, the upper-left vertex and the lower-right vertex indicate the position in the image where the target object is to be detected, the upper-left vertex is the upper-left vertex of the detection frame, and the lower-right vertex is the lower-right vertex of the detection frame, where , the upper left vertex indicates the intersection of a line corresponding to the upper contour of the corresponding target object and a line corresponding to the left contour of the corresponding target object, and the lower right vertex is the intersection of the line corresponding to the corresponding target object lower contour and the line corresponding to the right contour of the corresponding target object. points to

Of course, it is not limited to indicating the position of the target object through the upper left vertex and the lower right vertex, but it is also possible to indicate the position of the target object through the upper right vertex and the lower left vertex, and the embodiment of the present invention is the upper left vertex and The lower-right corner was described as an example.

Here, the centripetal shift tensor points to a shift tensor whose vertices point to the target object central position, which is because the image to be detected is a two-dimensional image, where the centripetal shift tensor contains shift values in both directions, and the two directions are in the X-axis direction, respectively. and in the Y-axis direction, the centripetal shift tensor includes a shift value in the X-axis direction and a shift value in the Y-axis direction. Through the vertex and the centripetal shift tensor corresponding to the vertex, the central position pointed to by the vertex can be determined, and when the upper left vertex and the lower right vertex are located in the same detection frame, the pointed center positions must be the same or similar , therefore, by determining a vertex belonging to the same target object based on the centripetal shift tensor corresponding to each vertex, a detection frame of the target object may be determined based on the determined vertex.

An embodiment of the present invention determines a vertex and a centripetal shift tensor corresponding to the vertex through a neural network, and the following embodiments are combined and described repeatedly.

In step S203, a target object in the image to be detected is determined based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex.

Here, the vertex position information in the image in which each vertex is to be detected refers to vertex position information in the image in which each vertex is to be detected from a plurality of vertices, and the centripetal shift tensor corresponding to each vertex is at each vertex in the plurality of vertices. Points to the corresponding centripetal shift tensors, respectively.

Detecting the target object in the image to be detected may include detecting a position of the target object, for example determining a detection frame of the target object in the image to be detected, and an instance of the target object in the image to be detected The information can be determined, or the detection frame and instance information of the target object in the image to be detected are simultaneously determined, and the method of determining the target object in the image to be detected will be interpreted in detail below.

In the target detection method provided by the steps S201 to S203, after obtaining the image to be detected, vertex position information for each vertex in the to-be-detected image and the centripetal shift tensor corresponding to each vertex must be determined first, which Since a vertex is a major feature point in an image, the position information of the vertex in the image to be detected may indicate the position of each target object in the image to be detected, for example, the vertex may include an upper left vertex and a lower right vertex. wherein the upper left vertex indicates the intersection of the line corresponding to the upper contour of the target object and the line corresponding to the left contour of the target object, and the lower right vertex corresponds to the line corresponding to the lower contour of the target object and the right contour of the target object When the upper left vertex and the lower right vertex belong to the detection frame of the same target, the positions indicated by the centripetal shift tensors corresponding to the upper left vertex and the lower right vertex may be relatively close. The target detection method provided by the embodiment of the present invention is based on the determined vertices by determining the vertices belonging to the same target object based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex. Thus, the same target object can be detected.

Next, the above steps S201 to S203 will be described continuously.

In one embodiment for step S202, when a vertex determines vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex, based on the image to be detected, as shown in FIG. Likewise, the following steps S301 to 303 may be included.

In step S301, an initial feature map corresponding to the image to be detected is obtained by performing feature extraction on the image to be detected.

In step S301, vertex pooling is performed on the initial feature map to obtain a vertex pooled feature map.

In step S303, based on the vertex pooled feature map, vertex position information in an image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex are determined.

Here, the size of the image to be detected is constant, for example, when the size is H*W, where H and W respectively indicate pixel values in the length and width directions in the image to be detected, and then pre-select the image to be detected. It is possible to obtain an initial feature map corresponding to the image to be detected by inputting it into the trained hourglass convolutional neural network and extracting features, for example, extracting texture features, extracting color features, and extracting edge features.

Of course, since the input end of the hourglass convolutional neural network has a request for image size, when the size of the image to be detected and the size of the image to be detected of the reception set size does not match the set size, the size of the image to be detected first After adjustment, the size-adjusted image to be detected is input again into the hourglass convolutional neural network, and feature extraction and size compression are performed to obtain an initial feature map of size h*w*c. where c is the number of channels of the initial feature map, and h and w indicate the size of the initial feature map of each channel.

The initial feature map includes a plurality of feature points, each feature point has feature data, and this feature data displays global information of the image to be detected. For the initial feature map (Corner Pooling), vertex pooling is proposed to obtain a vertex pooled feature map, and the vertex pooled feature map is compared to the initial feature map. Therefore, based on the vertex pooled feature map, vertex position information of each vertex in the image to be detected and the centripetal shift tensor corresponding to each vertex can be more accurately determined.

Here, by performing feature extraction on the image to be detected, obtaining an initial feature map, performing vertex pooling on the initial feature map, and extracting the feature map of the vertex and the centripetal shift tensor corresponding to the vertex An easy feature map is obtained, that is, a vertex pooled feature map is obtained.

After obtaining the vertex pooled feature map, based on the vertex pooled feature map and the pre-trained neural network, it is possible to determine whether vertices exist, and if there are vertices, each vertex is a vertex in the image to be detected. Determining position information, an embodiment of the present invention determines vertex position information in an image in which a target object is to be detected through an upper left vertex and a lower right vertex, and determines vertex position information in an image in which each vertex is to be detected. The process may be a process of determining vertex position information in an image in which an upper left vertex is to be detected, and vertex position information in an image in which a lower right vertex is to be detected, where the upper left vertex is determined through a left upper vertex detection network. It is possible to detect the vertex position information in the image to be detected, and the vertex position information in the image where the lower right vertex is to be detected through a lower right vertex detection network, which is the upper left vertex in the image to be detected. This is because the vertex position information and the lower right vertex are similar to the vertex position information in the image to be detected, and in the embodiment of the present invention, the determination of the vertex position information in the image where the upper left vertex is to be detected will be described in detail.

In one embodiment, the upper-left vertex detection network may include an upper-left vertex heat map prediction network and an upper-left vertex local shift prediction network, and based on the vertex pooled feature map, each vertex in the image to be detected When the vertex location information is determined, as shown in FIG. 4 , the following steps S401 to S404 may be included.

In step S401, a vertex heat map corresponding to the image to be detected is generated based on the vertex pooled feature map.

When the vertex position information in the image where the upper left vertex is to be detected is predicted, the vertex heat map can be obtained through the upper left vertex heat map prediction network in the upper left vertex detection network, and the vertex pooled feature map is shown in the upper left. By inputting the vertex heat map prediction network, an upper left vertex heat map corresponding to an image to be detected may be obtained, wherein the upper left vertex heat map includes a plurality of feature points, each feature data corresponding to the feature point , and based on the feature data of the feature point in the vertex heat map, it is possible to determine a probability value using the feature point as the upper left vertex.

The size of the obtained upper left vertex heatmap is h*w*m, where h*w represents the size of the vertex heatmap in each channel, m represents the pre-channel quantity, and each pre-channel has one Corresponding to a preset object category, for example, m=40, indicates that there are 40 preset object categories, and the upper left vertex heat map is used to determine the upper left vertex in the image to be detected. In addition, the upper left vertex is also used to determine the category of the target object indicated in the image to be detected, and the process of determining the category of the target object will be interpreted in detail below.

In step S402, based on the vertex heat map, a probability value of using each feature point as a vertex in the vertex heat map is determined, and based on the probability value using each feature point as a vertex, vertices are selected from the feature points in the vertex heat map. .

Based on the probability value of using each feature point as the upper left vertex, the probability that the feature point is the upper left vertex can be determined, so that the feature point whose probability value is greater than the threshold setting is used as the upper left vertex.

In step S403, location information of each selected vertex in the vertex heat map and local shift information corresponding to each vertex are acquired.

The local shift information is used to display position shift information in the vertex heat map of the actual physical point indicated by the corresponding vertex, where the local shift information corresponding to each upper left vertex is the actual physical point indicated by the upper left vertex is the upper left It is used to display position shift information in the vertex heatmap.

Here, the local shift information may be displayed through a local shift tensor, and the local shift tensor may also be displayed as a shift value in both directions in the upper left vertex heat map, for example, the coordinate system in the upper left vertex heat map is It includes both directions, in an x-axis direction and a y-axis direction, respectively, and the local shift tensor includes a shift value in the x-axis direction and a shift value in the y-axis direction.

Based on the coordinate system corresponding to the upper left vertex heat map, each feature point in the upper left vertex heat map may acquire location information in the upper left vertex heat map, and the obtained location information of the upper left vertex and the left Considering that there may be an error between the position information of the actual physical point indicated by the upper vertex, for example, by performing position detection on the upper left vertex heat map, the specified position information of the upper left vertex is obtained, and the There is a certain deviation between the position information of the actual physical point indicated by the upper left vertex and the detected position information of the upper left vertex, and the local shift information is used to indicate the deviation.

In order to increase the accuracy of target object detection, here we cite the pre-trained upper-left vertex local shift prediction network, and input the next vertex pooled feature map into the upper-left vertex local shift prediction network in the upper-left vertex prediction network. , determine local shift information corresponding to each feature point in the upper left vertex heat map, then modify the location information of the feature point in the vertex heat map based on the local shift information, and then based on the modified location information Thus, the vertex position information in the image where the upper left vertex is to be detected is determined.

In step S404, each vertex is located in the to-be-detected image based on location information of each obtained vertex in the vertex heatmap, local shift information corresponding to each vertex, and a size ratio between the vertex heatmap and the image to be detected. Determine the vertex position information.

Here, the obtained position information of each of the upper left vertices in the upper left vertex heat map may include a coordinate value x in the x-axis direction in the upper left vertex heat map, and a coordinate value y in the y-axis direction, The vertex position information of each upper left vertex in the image to be detected may include a coordinate value X in the X-axis direction and a coordinate value Y in the Y-axis direction.

Here, vertex position information in the image in which the i-th upper left vertex is to be detected may be determined according to formulas (1) and (2) below.

는 i번째 좌측 상단 꼭짓점이 검출될 이미지에서의 X축 방향에서의 좌표값을 표시하고,

는 i번째 좌측 상단 꼭짓점이 검출될 이미지에서의 Y축 방향에서의 좌표값을 표시하고; n은 좌측 상단 꼭짓점 히트맵 및 검출될 이미지 간의 크기 비율을 표시하고;

는 i번째 좌측 상단 꼭짓점이 좌측 상단 꼭짓점 히트맵에서의 x축 방향에서의 좌표값을 표시하고,

는 i번째 좌측 상단 꼭짓점이 꼭짓점 히트맵에서의 y축 방향에서의 좌표값을 표시하고;

는 i번째 좌측 상단 꼭짓점이 나타내는 실제 물리점이 꼭짓점 히트맵에서의 x축 방향에서의 시프트 값을 표시하고,

는 i번째 좌측 상단 꼭짓점이 나타내는 실제 물리점이 꼭짓점 히트맵에서의 y축 방향에서의 시프트 값을 표시한다.here,

represents the coordinate value in the X-axis direction in the image where the i-th upper left vertex is to be detected,

denotes the coordinate value in the Y-axis direction in the image where the i-th upper left vertex is to be detected; n denotes the size ratio between the top left vertex heatmap and the image to be detected;

indicates the coordinate value of the i-th upper-left vertex in the x-axis direction in the upper-left vertex heat map,

denotes the coordinate value of the i-th upper left vertex in the y-axis direction in the vertex heat map;

indicates the shift value in the x-axis direction of the actual physical point indicated by the i-th upper left vertex in the vertex heat map,

indicates the shift value in the y-axis direction of the actual physical point indicated by the i-th upper left vertex in the vertex heat map.

The above-described process is a process of determining vertex position information in the image in which the upper left vertex is to be detected, and the process of determining the vertex position information in the image in which the lower right vertex is to be detected is also the process of determining the vertex pooled feature map to the lower right vertex. Lower-right vertex heatmap in the prediction network By inputting into the prediction network and obtaining the lower-right vertex heatmap, the probability value of using each feature point in the lower-right vertex heatmap as the lower-right vertex is determined, where the lower-right vertex is and at the same time combine the local shift information corresponding to the lower right vertex determined through the lower right vertex local shift network in the lower right vertex prediction network to determine the vertex position information in the image where each lower right vertex is to be detected, No further explanation will be repeated here.

Similarly, vertex position information in the image in which the j-th lower-right vertex is to be detected may be determined according to formulas (3) and (4) below.

는 j번째 우측 하단 꼭짓점이 검출될 이미지에서의 X축 방향에서의 좌표값을 표시하고,

는 j번째 우측 하단 꼭짓점이 검출될 이미지에서의 Y축 방향에서의 좌표값을 표시하고; n은 우측 하단 꼭짓점 히트맵 및 검출될 이미지 간의 크기 비율을 표시하고;

는 j번째 우측 하단 꼭짓점이 우측 하단 꼭짓점 히트맵에서의 x축 방향에서의 좌표값을 표시하고,

는 j번째 우측 하단 꼭짓점이 꼭짓점 히트맵에서의 y축 방향에서의 좌표값을 표시하고;

는 j번째 우측 하단 꼭짓점이 나타내는 실제 물리점이 꼭짓점 히트맵에서의 x축 방향에서의 시프트 값을 표시하고,

는 j번째 우측 하단 꼭짓점이 나타내는 실제 물리점이 꼭짓점 히트맵에서의 y축 방향에서의 시프트 값을 표시한다.here,

represents the coordinate value in the X-axis direction in the image where the j-th lower right vertex is to be detected,

denotes a coordinate value in the Y-axis direction in the image where the j-th lower right vertex is to be detected; n denotes the size ratio between the bottom right vertex heatmap and the image to be detected;

indicates the coordinate value of the j-th lower-right vertex in the x-axis direction in the lower-right vertex heat map,

denotes the coordinate value of the j-th lower right vertex in the y-axis direction in the vertex heat map;

represents the shift value in the x-axis direction of the actual physical point indicated by the j-th lower right vertex in the vertex heat map,

indicates the shift value in the y-axis direction of the actual physical point indicated by the j-th lower right vertex in the vertex heat map.

The above-described steps S401 to S404 are a method in which each vertex provided in an embodiment of the present invention determines vertex position information in the to-be-detected image. Determines feature points that can become vertices through the probability value used as In this way, it is possible to obtain vertex position information of a vertex with relatively high accuracy, so that it is easy to detect a position in an image where a target object is to be detected based on the vertex.

The following introduces the process of determining the centripetal shift tensor corresponding to each vertex, and when a vertex is divided into an upper left vertex and a lower right vertex, the centripetal shift tensor corresponding to the upper left vertex, and the centripetal corresponding to the lower right vertex Each of the shift tensors should be checked, and the embodiment of the present invention will be described as an example of determining the centripetal shift tensor corresponding to the upper left vertex, and the centripetal shift tensor corresponding to the lower right vertex and the centripetal shift corresponding to the upper left vertex The method of determining the tensor is similar, and the embodiment of the present invention will not be repeated any longer.

In one embodiment, in order to obtain a more accurate centripetal shift tensor, before determining the centripetal shift tensor, after adjusting the vertex pooled feature map by citing the feature data process, the centripetal shift tensor is determined, Here, when determining a centripetal shift tensor corresponding to each vertex based on the vertex pooled feature map, as shown in FIG. 5 , the following steps S501 to S504 may be included.

In step S501, based on the vertex-pooled feature map, a guide shift tensor corresponding to each feature in the vertex-pooled feature map is determined.

Here, the guide shift tensor corresponding to each feature point represents the shift tensor of the target center point in the image to be detected indicated by the feature point.

Considering the location of the target object in the image to be detected and related to the target object information, it is hoped that the feature data of the vertices of the vertex pooled feature map will include richer target objects, so that each feature point can be considered here, and in particular the vertex The feature vector is displayed to include more abundant target target information, and accordingly, based on the guide shift tensor corresponding to each feature point, feature adjustment can be performed on the vertex pooled feature map, and the adjusted feature map can be Each feature point, especially a vertex, can contain richer target target information.

Here, by performing a convolution operation on the vertex pooled feature map, a guide shift tensor corresponding to each feature point in the vertex pooled feature map is obtained, and the guide shift tensor is a shift value along the x-axis direction and Contains shift values along the y-axis direction.

To determine the centripetal shift tensor corresponding to the upper left vertex, for example, a convolution operation is performed on the vertex pooled feature map, and the main acquired feature point is used as a guide shift tensor corresponding to the upper left vertex.

In step S502, shift domain information of the feature point is determined based on the guide shift tensor corresponding to each feature point.

Here, the shift domain information includes a shift tensor in which a plurality of initial feature points associated with the feature point respectively indicate shifted feature points corresponding to each.

After obtaining the guide shift tensor corresponding to each feature point, a convolution operation is performed based on the guide shift tensor corresponding to each feature point to obtain shift domain information of the feature point.

Determining the centripetal shift tensor corresponding to the upper left vertex corresponds to, for example, obtaining a guide shift tensor corresponding to the case where each feature point is used as the upper left vertex, and then using each feature point as the upper left vertex By performing a convolution operation on the guide shift tensor, shift domain information is obtained when the feature point is used as the upper left vertex.

In step S503, based on the vertex pooled feature map and the shift domain information of the feature points in the vertex pooled feature map, the feature data of the feature points in the vertex pooled feature map are adjusted, and the adjusted feature map to acquire

After obtaining shift domain information when using the feature point of the vertex pooled feature map as the upper left vertex, the vertex pooled feature map and each feature point in the vertex pooled feature map are used as the upper left vertex. It is an adjusted feature map corresponding to the upper left vertex obtained by simultaneously performing a variable convolution operation on domain information.

Here, the processes of steps S501 to S503 may be determined through a feature coordination network as shown in FIG. 6 ;

A convolution operation is performed on the vertex pooled feature map to obtain a guide shift tensor corresponding to the case where each feature point in the vertex pooled feature map is used as the upper left vertex, and then the guide shift tensor is convolved. A solution operation is performed to obtain shift domain information, where the interpretation of the shift domain information is as follows;

When a convolution operation is performed on the feature data of the feature point in the vertex pooled feature map, for example, when the convolution operation is performed on the feature data of feature point A in the vertex pooled feature map, shift domain information is present. Otherwise, when convolution operation is performed on the feature data of feature point A according to 3*3 convolution, the feature data of the initial feature point displayed through 9 solid line frames including feature point A in the vertex pooled feature map is used. It can be obtained by performing a convolution operation, and after considering the shift domain information, it is desired to perform feature adjustment on the feature point A by including feature data of the feature point of the more abundant target target information, for example, Based on the guide shift vector corresponding to each feature point, it is for shifting the feature point that performs feature adjustment on feature point A. For example, the shifted feature point is 9 in the vertex pooled feature map in FIG. By performing a convolution operation on the feature data of feature points that are displayed through a dotted line and shifted by nine, feature adjustment can be performed on the feature data of feature point A, where the shift domain information is shifted in FIG. Each shift tensor in the shift tensor, that is, the shift tensor of the feature point after the shift corresponding to the initial feature point pointed to by each initial feature point, is expressed through the progression of the tensor, and the initial feature point is shifted in the x-axis direction and the y-axis direction. After that, the shifted feature point corresponding to the obtained initial feature point is displayed.

In consideration of the guide shift tensor corresponding to the case of using each feature point as the upper left vertex, the target target information included in the feature data at the feature-adjusted feature point is enriched for each left side based on the adjusted feature map in the future. When determining the centripetal shift tensor corresponding to the upper vertex, it is easy to obtain a more accurate centripetal shift tensor.

Similarly, when each feature point is used as the lower right vertex, the guide shift tensor of the center point of the relative target object is taken into consideration, and the target target information including the feature-adjusted feature point is further enriched, based on the adjusted feature map in the future. When determining the centripetal shift tensor corresponding to each lower right vertex, it is easy to obtain a more accurate centripetal shift tensor.

In step S504, a centripetal shift tensor corresponding to each vertex is determined based on the adjusted feature map.

Here, a centripetal shift tensor corresponding to each vertex is obtained based on performing a convolution operation on the feature data corresponding to the vertices in the adjusted feature map.

Here, the adjusted feature map may include an adjusted feature map corresponding to the upper left vertex and an adjusted feature map corresponding to the lower right vertex, and based on the adjusted feature map corresponding to the upper left vertex, each left When the centripetal shift tensor corresponding to the upper vertex is determined, it can be determined through the centripetal shift prediction network corresponding to the upper left vertex, and the centripetal shift corresponding to each lower right vertex based on the adjusted feature map corresponding to the lower right vertex. When determining the tensor, it can be determined through the centripetal shift prediction network corresponding to the lower right vertex.

The above-described steps S501 to S504 are a process of determining the centripetal shift tensor provided in the embodiment of the present invention, and through consideration of target target information, for example, a guide shift tensor corresponding to a vertex and a shift domain of a feature point By citing information and adjusting the feature data of the feature points in the vertex pooled feature map, the feature data of the feature points in the obtained adjusted feature map includes richer target target information, so that By allowing the centripetal shift tensor to be determined more accurately, and by obtaining accurate center position information pointed to by a vertex through the accurate centripetal shift tensor, the position of the target object in the image to be detected can be accurately detected.

As described above, the category of the target object included in the image to be detected can be determined through the vertex heatmap. Here, a method of determining the category of the target object according to the vertex heatmap is introduced, and as is known from the above, the the vertex heatmap corresponding to the image includes a vertex heatmap respectively corresponding to a plurality of channels, each channel corresponding to one preset target category; As described above, after determining a probability value using each feature point in the vertex heat map as a vertex based on the vertex heat map, as shown in FIG. 7 , the detection method provided in the embodiment of the present invention includes the following steps S701 to step S702 are further included.

In step S701, for each channel in the plurality of channels, based on a probability value using each feature point in the vertex heat map corresponding to the channel as a vertex, whether a vertex of the vertex heat map corresponding to the channel exists decide whether

In step S702, if a vertex exists in the vertex heat map corresponding to the channel, it is determined that a target object of a preset target category corresponding to the channel is included in the image to be detected.

In the case of a vertex heatmap corresponding to an image to be detected, based on a probability value of using each feature point in the vertex heatmap corresponding to each channel as a vertex, it can be determined whether a vertex exists in the vertex heatmap of the channel. and, for example, if the vertex feature map of the specified channel includes a feature point having a plurality of corresponding probability values greater than the threshold value, it is explained that the vertex feature map of the channel has a high probability of including the vertex, and the vertex is the target target. It is used to indicate a position in this to-be-detected image, whereby it can be explained that the to-be-detected image includes a target object of a preset object category corresponding to the channel.

For example, when animal detection is performed for a specified zoo, the number of channels may be set to 100, the obtained vertex heatmap is h*w*100, and each channel corresponds to one preset target category, For the detected image to be detected, in 100 channels of the acquired vertex heatmap corresponding to the to-be-detected image, only the vertex heatmaps in the first channel and the second channel include vertices, and When the preset target category is 01 and the preset target category corresponding to the second channel is 02, it may be explained that the target objects having categories 01 and 02 are included in the image to be detected.

In an embodiment of the present invention, by inputting the vertex pooled feature map to the vertex heatmap prediction network, a vertex heatmap including the number of preset channels can be obtained, and vertices exist in the vertex heatmap corresponding to each channel. It has been proposed that it is possible to determine whether or not there is a target object corresponding to the channel in the image to be detected.

In addition, here, after detecting the vertices included in the vertex heatmap of each channel, the centripetal shift tensor corresponding to the vertex can be determined, thereby determining the position in the image where the target object corresponding to each channel is to be detected. In combination with the category of the target object corresponding to , the category of each target object in the image to be detected is determined.

In step S203, when determining a target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex,

and determining a detection frame of the target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex.

Here, when generating a detection frame of a target object from an image to be detected, vertex position information of at least one upper left vertex and lower right vertex must be determined, or vertex position information of at least one upper right vertex and lower left vertex should be determined, and the embodiment of the present invention has been described as an example of determining the detection frame through the upper left vertex and the lower right vertex.

Here, when determining the detection frame of the target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex, as shown in FIG. 8 ,

In step S801, the method may include selecting a pair of candidate vertices constituting a candidate detection frame based on vertex position information in an image in which each vertex is to be detected.

Taking the candidate vertices including the upper left vertex and the lower right vertex as an example, when selecting the upper left vertex and the lower right vertex that can form a candidate detection frame, in order to improve the selection speed, first the upper left vertex and the lower right vertex It can be determined whether or not they belong to the same target target category, and when it is determined that any upper left vertex and lower right vertex belong to the same target target category, the arbitrary upper left vertex and lower right vertex in the image to be detected It is continuously determined whether or not the vertex position information of .

For example, the upper left vertex should be located at the upper left of the lower right vertex in the image to be detected, and according to the vertex position information of the upper left vertex and the lower right vertex, for example, the upper left vertex in the image to be detected According to the position coordinates and the position coordinates in the image where the lower right vertex is to be detected, if the upper left vertex cannot be located in the upper left of the lower right vertex, the upper left vertex and the lower right vertex can form a pair of candidate vertices. can't

Here, a coordinate system may be established in the image to be detected, the coordinate system includes an X-axis and a Y-axis, and the vertex position information of each vertex in the coordinate system is an abscissa value in the X-axis direction, and a ordinate in the Y-axis direction. value, and then, in the coordinate system, the upper left vertex and the lower right vertex from which each vertex can form a candidate detection frame are selected according to a coordinate value corresponding to the coordinate system.

In step S802, center point position information indicated by the vertex is determined based on the vertex position information in the image in which each vertex is to be detected in each candidate vertex pair and the centripetal shift tensor corresponding to each vertex.

Here, according to the formula (5) below, the center point position information pointed to by the upper left vertex in each candidate vertex pair can be determined, and the center point position information pointed to by the lower right vertex in each candidate vertex pair can be determined according to the formula (6) below. have.

는 i 번째 좌측 상단 꼭짓점이 가리키는 중심점 위치 정보에서의 X축 방향에 대응되는 횡좌표값을 표시하고,

는 i 번째 좌측 상단 꼭짓점이 가리키는 중심점 위치 정보에서의 Y축 방향에 대응되는 종좌표값을 표시하며;

는 i 번째 좌측 상단 꼭짓점이 검출될 이미지에서의 꼭짓점 위치 정보에서의 X축 방향에 대응되는 횡좌표값을 표시하고;

는 i 번째 좌측 상단 꼭짓점이 검출될 이미지에서의 꼭짓점 위치 정보에서의 Y축방향에 대응되는 종좌표값을 표시하며 ;

는i 번째 좌측 상단 꼭짓점의 구심 시프트 텐서에서의 X축 방향의 시프트 값을 표시하고,

는 i 번째 좌측 상단 꼭짓점의 구심 시프트 텐서에서의 Y축 방향의 시프트 값을 표시한다.here,

represents the abscissa value corresponding to the X-axis direction in the center point position information indicated by the i-th upper left vertex,

denotes a ordinate value corresponding to the Y-axis direction in the center point position information indicated by the i-th upper left vertex;

denotes an abscissa value corresponding to the X-axis direction in the vertex position information in the image where the i-th upper left vertex is to be detected;

denotes the ordinate value corresponding to the Y-axis direction in the vertex position information in the image where the i-th upper left vertex is to be detected;

denotes the shift value in the X-axis direction in the centripetal shift tensor of the i-th upper left vertex,

denotes the shift value in the Y-axis direction in the centripetal shift tensor of the i-th upper left vertex.

는 j 번째 우측 하단 꼭짓점이 가리키는 중심점 위치 정보에서의 X축 방향에 대응되는 횡좌표값을 표시하고,

는 j 번째 우측 하단 꼭짓점이 가리키는 중심점 위치 정보에서의 Y축 방향에 대응되는 종좌표값을 표시하며;

는 j 번째 우측 하단 꼭짓점이 검출될 이미지에서의 꼭짓점 위치 정보에서의 X축 방향에 대응되는 횡좌표값을 표시하고;

는 j 번째 우측 하단 꼭짓점이 검출될 이미지에서의 꼭짓점 위치 정보에서의 Y축방향에 대응되는 종좌표값을 표시하며 ;

는 j 번째 우측 하단 꼭짓점의 구심 시프트 텐서에서의 X축 방향의 시프트 값을 표시하고,

는 j 번째 우측 하단 꼭짓점의 구심 시프트 텐서에서의 Y축 방향의 시프트 값을 표시한다.here,

represents the abscissa value corresponding to the X-axis direction in the center point position information indicated by the j-th lower right vertex,

denotes a ordinate value corresponding to the Y-axis direction in the center point position information indicated by the j-th lower right vertex;

denotes an abscissa value corresponding to the X-axis direction in the vertex position information in the image where the j-th lower right vertex is to be detected;

denotes a ordinate value corresponding to the Y-axis direction in the vertex position information in the image where the j-th lower right vertex is to be detected;

denotes the shift value in the X-axis direction in the centripetal shift tensor of the j-th lower right vertex,

denotes the shift value in the Y-axis direction in the centripetal shift tensor of the j-th lower right vertex.

In step S803, center region information corresponding to the candidate vertex pair is determined based on the vertex position information in the image in which each vertex is to be detected in each candidate vertex pair.

Here, the central region information may be preset, and is defined as a coordinate range of the central region frame overlapping the center of the detection frame of the target object, and whether the only target object is included in the candidate detection frame through the coordinate range of the central region frame can be detected.

For example, in the candidate vertex pair, when the center point location information pointed to by the upper left vertex and the center point location information pointed to by the lower right vertex are located within the coordinate range of the center area frame, and the coordinate range of the center area frame is relatively small, the upper left By recognizing that the center point position information pointed to by the vertex and the center point position information pointed to by the lower right vertex are relatively close, it is determined that the candidate detection frame generated by the candidate vertex contains the unique target object.

Here, when determining the center region information corresponding to the candidate vertex pair based on the vertex position information in the image where each vertex is to be detected in each candidate vertex pair,

(1) determining vertex position information indicating a central region frame corresponding to the candidate vertex pair based on vertex position information of each vertex in the candidate vertex pair; and

and (2) determining the coordinate range of the central region frame corresponding to the candidate vertex pair based on the vertex position information of the central region frame.

When the m-th candidate vertex is composed of the i-th upper left vertex and the j-th lower right vertex, the vertex position information of the central region frame corresponding to the m-th candidate vertex pair is obtained according to formulas (7) to (10) below. can decide;

는 m 번째 후보 꼭짓점 페어에 대응되는 중심 영역 프레임의 좌측 상단 꼭짓점이 검출될 이미지에서의 X축 방향에서의 횡좌표값을 표시하고;

는 m 번째 후보 꼭짓점 페어에 대응되는 중심 영역 프레임의 좌측 상단 꼭짓점이 검출될 이미지에서의 Y축 방향에서의 횡좌표값을 표시하며;

는 m 번째 후보 꼭짓점 페어에 대응되는 중심 영역 프레임의 우측 하단 꼭짓점이 검출될 이미지에서의 X축 방향에서의 횡좌표값을 표시하고;

는 m 번째 후보 꼭짓점 페어에 대응되는 중심 영역 프레임의 우측 하단 꼭짓점이 검출될 이미지에서의 Y축 방향에서의 횡좌표값을 표시하며;

는 중심 영역 프레임의 길이 너비와 후보 검출 프레임의 길이 너비의 비율을 표시하고, 상기 비율은 기설정된 것이고,

이다.here,

denotes the abscissa value in the X-axis direction in the image in which the upper left vertex of the center region frame corresponding to the m-th candidate vertex pair is to be detected;

denotes the abscissa value in the Y-axis direction in the image in which the upper left vertex of the center region frame corresponding to the m-th candidate vertex pair is to be detected;

denotes the abscissa value in the X-axis direction in the image in which the lower right vertex of the center region frame corresponding to the m-th candidate vertex pair is to be detected;

denotes the abscissa value in the Y-axis direction in the image in which the lower right vertex of the center region frame corresponding to the m-th candidate vertex pair is to be detected;

denotes the ratio of the length width of the central region frame to the length width of the candidate detection frame, the ratio being a predetermined one,

am.

After determining the vertex position information of the central region frame corresponding to the m-th candidate vertex pair, the coordinate range of the central region frame may be determined according to the following formula (11);

는 m 번째 후보 꼭짓점에 대응되는중심 영역 프레임의 좌표 범위를 표시하고, 상기 중심 영역 프레임의 좌표 범위는 X축 방향에서의

값, 및 Y축 방향에서의

값을 통해 표시되고, 여기서

의 범위는

를 만족하고,

의 범위는

를 만족한다.here,

denotes the coordinate range of the central region frame corresponding to the m-th candidate vertex, and the coordinate range of the central region frame is in the X-axis direction.

value, and in the Y-axis direction

It is expressed through a value, where

the range of

satisfied with

the range of

is satisfied with

In step S804, the detection frame of the target object in the candidate detection frame is determined based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair.

The center region information corresponding to each candidate vertex is used to restrain the proximity between the center point position information indicated by each vertex in the candidate vertex pair, and the center point position information indicated by each vertex in the specified candidate vertex is the candidate vertex. When located within the frame of the corresponding central region, it can be explained that the center points indicated by each vertex in the pair of candidate vertices are relatively close, and the target object included in the candidate detection frame created by the candidate vertex is the only target object.

The method of determining the detection frame of the target object proposed in the embodiment of the present invention is to first determine a candidate vertex pair constituting a candidate detection frame based on vertex position information of the vertex, and then target each vertex in the candidate vertex pair. The detection frames of all target objects in the image to be detected can be detected relatively accurately by determining whether the target object surrounding the candidate detection frame is the same target object based on the obtained centripetal shift tensor.

Here, when determining the detection frame of the target in the candidate detection frame based on the center point position information indicated by each vertex in each candidate vertex pair and the center region information corresponding to the vertex pair, as shown in FIG. Likewise, it may include the following steps S901 to S902;

In step S901, a valid candidate vertex pair is determined based on center position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair.

Here, when the center position information indicated by each vertex in the specified candidate vertex pair is located in the center region frame corresponding to the vertex pair, the valid candidate vertex pair is used as a valid candidate vertex pair.

Here, it is determined whether the candidate vertex pair formed by the i-th upper left vertex and the j-th lower right vertex is a valid candidate vertex pair through the following formula (12), that is, the i-th upper left vertex and the j-th lower right vertex are It is determined whether the coordinate range of the m-th center region frame corresponding to the created candidate detection frame and the center point location information indicated by the i-th upper left vertex and the j-th lower right vertex respectively satisfy Equation (12) below.

The coordinate range of the m-th center region frame corresponding to the candidate detection frame created by the i-th upper-left vertex and the j-th lower-right vertex and the center point location information pointed to by the i-th upper-left vertex and the j-th lower-right vertex are determined by the anomaly formula (12 ), it is explained that the candidate vertex pair formed by the i-th upper left vertex and the j-th lower right vertex is a valid candidate vertex pair, and then step S902 is continuously performed for the valid candidate vertex. Otherwise, i If the candidate vertex pair created by the th upper-left vertex and the j-th lower-right vertex is an invalid candidate vertex pair, the i-th upper-left vertex and other lower-right vertices are valid candidate vertices until a valid candidate vertex pair is obtained. A decision is made as to whether to establish a pair, and the subsequent steps are executed.

In step S902, each valid candidate vertex pair is based on the center point position information pointed to by each vertex in the valid candidate vertex pair, the center region information corresponding to the valid candidate vertex pair, and the probability value corresponding to each vertex in the valid candidate vertex pair. Determine the score of the candidate detection frame corresponding to .

Here, the probability value corresponding to each vertex is for displaying a probability value in which the vertex uses a feature point corresponding to the vertex heat map as a vertex.

When detection is performed on an image to be detected, since there may exist a case where a plurality of candidate detection frames may appear for the same target object, the accuracy of the location of the target object indicated by some candidate detection frames in the image to be detected may be relatively low. , where, by citing the score of the candidate detection frame corresponding to each valid candidate vertex, for example, in the valid candidate vertex pair, the area created by the central point pointed to by each vertex and the central region frame corresponding to the valid candidate vertex are area relations, and The score of the candidate detection frame corresponding to each valid candidate vertex is displayed through the probability value corresponding to each vertex in the valid candidate vertex pair, and the probability of using the high-scoring candidate detection frame as the target target detection frame is relatively high, Through this, selection is performed on candidate detection frames.

Here, with respect to the valid candidate vertex pair formed by the i-th upper left vertex and the j-th lower right vertex, the score of the candidate detection frame corresponding to the valid candidate vertex can be determined according to the following formula (13):

는 i 번째 좌측 상단 꼭짓점 및 j 번째 우측 하단 꼭짓점이 조성한 유효 후보 꼭짓점 페어에 대응되는 후보 검출 프레임의 점수를 표시하고;

는 i 번째 좌측 상단 꼭짓점이 좌측 상단 꼭짓점 히트맵에 대응되는 특징점을 좌측 상단 꼭짓점으로 사용하는 확률값을 표시하고;

는 j 번째 우측 하단 꼭짓점이 우측 하단 꼭짓점 히트맵에 대응되는 특징점을 우측 하단 꼭짓점으로 사용하는 확률값을 표시한다.here,

denotes the score of the candidate detection frame corresponding to the valid candidate vertex pair formed by the i-th upper-left vertex and the j-th lower-right vertex;

denotes a probability value in which the i-th upper-left vertex uses a feature point corresponding to the upper-left vertex heat map as the upper-left vertex;

denotes a probability value in which the j-th lower right vertex uses a feature point corresponding to the lower right vertex heat map as the lower right vertex.

In step S903, a detection frame of the target object in the candidate detection frame is determined based on the score of the candidate detection frame corresponding to each valid candidate vertex and the size of the overlapping region between adjacent candidate detection frames.

Here, the overlapping area can be determined based on the size of the image in which the overlapping area is to be detected, and the following is the target target based on the score of the candidate detection frame corresponding to each valid candidate vertex, and the overlapping area between adjacent candidate detection frames. A method of selecting a detection frame is introduced.

Here, the detection frame of the target object from the plurality of candidate detection frames may be selected through soft band suppression, and when there are a plurality of candidate detection frames exceeding the overlapping region threshold, the candidate with the highest correspondence score The detection frame may be used as the detection frame of the target object, and other candidate detection frames in the plurality of candidate detection frames may be deleted, thereby obtaining the detection frame of the target object in the image to be detected.

In steps S901 to S903 described above, it is determined that a candidate detection frame representing only one target target candidate is selected by performing effective selection on the candidate vertex pairs constituting the candidate detection frame, and then, this single target target The detection frame indicating the target object is accurately obtained by performing the maximal suppression selection on the candidate detection frame indicating only the candidate.

After obtaining the detection frame of the target object in the image to be detected, instance information of the target object in the detection frame may be determined, wherein the detection frame of the target object and the initial obtained by performing feature extraction on the image to be detected Instance information of the target object in the image to be detected may be determined based on the feature map.

Here, the instance information can be displayed through a mask, where the mask refers to providing the pixels of each target object in terms of the pixel layer, after performing instance segmentation on the target object in the image, so that the mask extends to the edge of the object. By being accurate, it is possible to obtain a more accurate position in the image where the target object is to be detected, in addition, the mask can also indicate the shape of the target object, so that the determination of the category of the target object based on the shape is Whether or not it is correct may be authenticated, and subsequent motion analysis may be performed on the target object based on the appearance of the target object indicated by the mask, which will not be repeated in the embodiment of the present invention.

Here, when determining the instance information of the target object from the image to be detected based on the initial feature map obtained by performing feature extraction on the detection frame of the target object and the image to be detected,

(1) based on the detection frame of the target object and the initial feature map, the initial feature map extracts feature data of feature points in the detection frame;

(2) the initial feature map determines, based on feature data of feature points in the detection frame, instance information of a target object in an image to be detected.

Here, the detection frame of the target object and the initial feature map corresponding to the image to be detected are input to the region of interest extraction network, and the region of interest extraction network may first extract the region of interest matching the size of the initial feature map, and then Through alignment pooling, the initial feature map acquires feature data of feature points in the detection frame (region of interest), and then the initial feature map inputs feature data of feature points in the detection frame to the mask prediction network. Next, instance information of the target object may be generated, the instance information may be displayed in the form of a mask, and the mask of the target object is then scaled up to the size of the target object in the image to be detected, and the target of the image to be detected It is possible to obtain instance information of the target.

A general description will be given of the target detection method proposed in the embodiment of the present invention in conjunction with FIG. 10 below;

An image to be detected is input to the hourglass convolutional neural network to obtain an initial feature map f corresponding to the image to be detected, and then, when detection is performed on a target object in the image to be detected, the initial feature map f By performing vertex pooling and obtaining a vertex-pooled feature map p, upper-left vertex detection and feature adjustment are performed on the vertex-pooled feature map p, corresponding to the upper-left vertex and the upper-left vertex. A centripetal shift tensor can be obtained, and the process of obtaining the upper-left vertex is determined through the upper-left vertex detection network, and the upper-left vertex detection network is the upper-left vertex heatmap prediction network and the upper-left vertex local shift prediction network. Including (not shown in Fig. 10), before obtaining the centripetal shift tensor corresponding to the upper left vertex, first, feature adjustment is performed on the vertex-pooled feature map p for the feature adjustment network, and the process is in the upper left The convolution operation is performed by including the guide shift tensor and shift domain information corresponding to the vertices, and performing adjustment on the vertex pooled feature map p based on the following variable convolution operation to obtain an adjusted feature map g. Through this, a centripetal shift tensor corresponding to the upper left vertex is determined.

The lower right vertex is determined through the lower right vertex detection network, the centripetal shift tensor corresponding to the lower right vertex is determined through feature adjustment and convolution operation, and the centripetal shift tensor corresponding to the upper left vertex and the upper left vertex is determined. The process is similar, and based on the centripetal shift tensor corresponding to the next upper left vertex and the upper left vertex, and the centripetal shift tensor corresponding to the lower right vertex and the lower right vertex, the detection frame of the target object is jointly determined.

After obtaining the detection frame of the target object, by extracting the region of interest based on the detection frame of the target object and the initial feature map f, the region of interest alignment vertex pooling is performed on the region of interest, and the region of interest feature (initial feature The map acquires feature data of feature points in the detection frame), obtains a mask of a target object through convolution operation in a mask prediction network, and scales up the mask to be detected. Obtain a mask of the same size as the image (instance information of the target object).

Through the target detection method proposed in the embodiment of the present invention, it is possible to output the detection frame of the target object, the mask of the target object, and the category of the target object, and according to a predetermined request, a desired result can be obtained, for example, Outputting the detection frame of the target object, or outputting the mask image of the target object, or outputting the detection frame of the target object, outputting the mask image of the target object, and simultaneously outputting the category of the target object, according to the present invention Examples of is not limited.

The target detection method of the embodiment of the present invention can be implemented through a neural network, and the neural network is acquired by training using a sample image including a labeling target sample object.

Here, as shown in Fig. 11, the neural network of the target detection method proposed by the embodiment of the present invention includes steps S1101 to S1104;

In step S1101, a sample image is acquired.

Here, the sample image may include a positive sample of the labeling target sample object, and does not include a negative sample of the target sample object, and the target object included in the positive sample may include a plurality of categories.

Here, the positive sample of the labeling target sample object is divided into a target sample object that has passed detection frame labeling, and a target sample object that has passed mask labeling.

In step S1102, based on the sample image, each sample vertex determines vertex position information in the sample image and a centripetal shift tensor corresponding to each sample vertex, and the sample vertex indicates the position of the target sample object in the sample image. .

Here, based on the sample image, the sample vertices determine the vertex position information in the sample image, and the process of the centripetal shift tensor respectively corresponding to each sample vertex, and as described above, vertex position information in the image where the vertices are to be detected. Since it is the same as the determination method of the centripetal shift tensor corresponding to each vertex, , and , the description is not repeated here.

In step S1103, a target sample object in the sample image is predicted based on the vertex position information of each sample vertex in the sample image and the centripetal shift tensor corresponding to each sample vertex.

Here, since the process of predicting the target sample object in the sample image is the same as the method of determining the target object in the image to be detected as described above, it will not be repeatedly described herein.

In step S1104, based on the predicted target sample object in the sample image and the labeling target sample object in the sample image, a network parameter of the neural network is adjusted.

Here, it is possible to determine the corresponding loss value when predicting the target sample object by citing the loss function, and after training several times, adjust the network parameter value of the neural network through the loss value, for example, When the loss value is set to be smaller than the set threshold, the network parameter value of the neural network can be obtained by stopping the training.

In addition, the process of determining the detection frame of the target sample object, the mask of the target sample object, and the category of the target sample object, as described above, includes determining the detection frame of the target object, the mask of the target object, and the category of the target object as described above. Since it is similar to the process, it will not be repeated here any longer.

The training method of the neural network provided in the embodiment of the present invention is based on the sample image through sample image acquisition, vertex position information of each sample vertex in the sample image, and centripetal shift corresponding to each sample vertex. By determining the tensor, each sample vertex detects a target sample object in the sample image based on the vertex position information in the sample image and the centripetal shift tensor corresponding to each sample vertex, where the sample vertex is a key feature point in the image. For example, the sample vertex may include an upper-left sample vertex and a lower-right sample vertex, where the upper-left sample vertex is a line corresponding to the upper contour of the target sample object and the left contour of the target sample object. indicates the intersection of the lines corresponding to , and the lower right sample vertex points to the intersection of the line corresponding to the lower contour of the target sample object and the line corresponding to the right contour of the target sample object, and the upper left sample vertex and the lower right sample vertex are the same target sample In the case of the target detection frame, the positions indicated by the centripetal shift tensors corresponding to the upper left sample vertex and the lower right sample vertex may be relatively close. Based on the vertex position information of the vertices indicating the position of the sample object in the sample image and the centripetal shift tensor corresponding to each sample vertex, the sample vertices belonging to the same target sample object are determined, and the same target sample is determined based on the determined sample vertices. A target can be detected, and a neural network with relatively high accuracy can be obtained by continuously adjusting the neural network parameter value based on the labeling target in the sample image, and a target based on the neural network with relatively high accuracy can be obtained. The target can be accurately detected.

A person skilled in the art can understand that in the above method, the recording order of each step does not imply a strict execution order and does not constitute any limitation in the implementation process, and the execution order of each step is determined according to a function and a possible internal logic.

Based on the same technical idea, the embodiment of the present invention further provides a target detection apparatus corresponding to the target detection method, and the technical principle of the apparatus in the embodiment of the present invention is similar to that of the other detection method according to the embodiment of the present invention. Therefore, the implementation of the apparatus may refer to the implementation of the method, and overlapping parts are not described repeatedly.

Referring to FIG. 12 , it is an exemplary view of a target detection apparatus 1200 provided in an embodiment of the present invention, and the apparatus includes an acquisition unit 1201 , a determination unit 1202 , and a detection unit 1203 .

Here, the acquiring unit 1201 acquires an image to be detected;

The determining unit 1202 determines, based on the image to be detected, vertex position information in the image where each vertex is to be detected, and a centripetal shift tensor corresponding to each vertex - the vertex indicates the position of the target object in the image to be detected - ;

The detection unit 1203 is configured to determine a target object in the image to be detected based on vertex position information in the image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the decision unit 1202 comprises:

performing feature extraction on the image to be detected to obtain an initial feature map corresponding to the image to be detected;

vertex pooling is performed on the initial feature map to obtain a vertex pooled feature map;

and to determine vertex position information in an image where each vertex is to be detected, and a centripetal shift tensor corresponding to each vertex, based on the vertex pooled feature map.

In one possible embodiment, when the determiner 1202 is configured to determine vertex position information in the image where each vertex is to be detected, based on the vertex pooled feature map, the determiner comprises:

generating a vertex heatmap corresponding to the image to be detected based on the vertex pooled feature map;

based on the vertex heat map, determining a probability value using each feature point as a vertex in the vertex heat map, and selecting vertices from the feature points of the vertex heat map based on the probability value using each feature point as a vertex;

Each selected vertex acquires position information in the vertex heat map and local shift information corresponding to each vertex - The local shift information is used for displaying position shift information in the vertex heat map at the actual physical point indicated by the corresponding vertex. Used - ;

Each vertex determines the vertex position information in the to-be-detected image based on the obtained position information in the vertex heatmap, local shift information corresponding to each vertex, and the size ratio between the vertex heatmap and the image to be detected. configured to do

In one possible embodiment, when the determining unit 1202 is configured to determine a centripetal shift tensor corresponding to each vertex based on the vertex pooled feature map, the determining unit comprises:

Based on the vertex-pooled feature map, a guide shift tensor corresponding to each feature in the vertex-pooled feature map is determined; represents a tensor - ;

Determining shift domain information of the feature point based on the guide shift tensor corresponding to each feature point - The shift domain information includes indicating the shift tensor of the feature point after shift corresponding to a plurality of initial feature points associated with the feature point - ;

based on the vertex pooled feature map and the shift domain information of the feature points in the vertex pooled feature map, adjusting the feature data of the feature points in the vertex pooled feature map to obtain an adjusted feature map;

Based on the adjusted feature map, it is configured to obtain a centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the vertex heatmap corresponding to the image to be detected comprises a vertex heatmap respectively corresponding to a plurality of channels, each channel in the plurality of channels corresponding to one preset target category; When the determining unit 1202 is configured to determine a probability value of using each feature point as a vertex in the vertex heat map based on the vertex heat map, the determining unit further includes:

for each channel in the plurality of channels, based on a probability value using each feature point in the vertex heatmap corresponding to the channel as a vertex, determining whether a vertex exists in the vertex heatmap corresponding to the channel;

and determine that, when a vertex exists in the vertex heatmap corresponding to the channel, a target object of a preset object category corresponding to the channel is included in the image to be detected.

In one possible embodiment, the detection unit further comprises:

and determine a detection frame of a target object in the image to be detected, based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex.

In one possible embodiment, the detection unit 1203 determines the detection frame of the target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex. When configured to determine, the detection unit,

selecting a pair of candidate vertices constituting a candidate detection frame based on vertex position information in an image in which each vertex is to be detected;

determining center point location information indicated by the vertex based on the vertex location information in the image where each vertex is to be detected in each candidate vertex pair and the centripetal shift tensor corresponding to each vertex;

determining center region information corresponding to the candidate vertex pair based on vertex position information in an image in which each vertex is to be detected in each candidate vertex pair;

and determine the detection frame of the target object in the candidate detection frame based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair.

In one possible embodiment, when the detection unit 1203 is configured to determine center region information corresponding to the candidate vertex pair based on the vertex position information in the image where each vertex is to be detected in each candidate vertex pair , the detection unit,

determining vertex position information indicating a central region frame corresponding to the candidate vertex pair based on vertex position information of each vertex in the candidate vertex pair;

and determine a coordinate range of the central region frame corresponding to the candidate vertex pair based on the vertex position information of the central region frame.

In one possible embodiment, the detection unit 1203 determines the target target in the candidate detection frame based on center point position information pointed to by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair. When configured to determine a detection frame, the detection unit,

determining a valid candidate vertex pair based on center position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair;

Candidates corresponding to each valid candidate vertex pair based on center point position information pointed to by each vertex in the valid candidate vertex pair, center region information corresponding to the valid candidate vertex pair, and a probability value corresponding to each vertex in the valid candidate vertex pair determining a score of the detection frame, wherein a probability value corresponding to each vertex is used to indicate a probability value in which the vertex uses a feature point corresponding to a vertex heat map as a vertex;

and determine the detection frame of the target object in the candidate detection frame based on the score of the candidate detection frame corresponding to each valid candidate vertex pair and the size of the overlapping region between the adjacent candidate detection frames.

In one possible embodiment, the detection unit 1203 is also

After determining the detection frame of the target object in the image to be detected, the instance information of the target object in the image to be detected is determined based on the detection frame of the target object and the initial feature map obtained by performing feature extraction on the image to be detected configured to do

In one possible embodiment, based on the detection unit 1203 performing feature extraction on the detection frame of the target object and the image to be detected to obtain an initial feature map, When configured to determine instance information, the detection unit,

based on the detection frame of the target object and the initial feature map, the initial feature map extracts feature data of feature points in the detection frame;

The initial feature map is configured to determine instance information of the target object in the image to be detected, based on the feature data of the feature points in the detection frame.

In one possible embodiment, the target detection device 1200 further comprises a neural network training unit 1204, the neural network training unit 1204 comprising:

It is configured to train a neural network for target detection, and the neural network is acquired by training using a sample image including a labeling target sample object.

In one possible embodiment, the neural network training unit 1204 comprises:

acquiring a sample image;

based on the sample image, each sample vertex determines vertex position information in the sample image and a centripetal shift tensor corresponding to each sample vertex, wherein the sample vertex indicates a position of a target sample object in the sample image;

predict a target sample object in the sample image based on the vertex position information of each sample vertex in the sample image and the centripetal shift tensor corresponding to each sample vertex;

and adjust a network parameter value of the neural network based on the predicted sample object in the sample image and the labeling target sample object in the sample image.

In the embodiments of the present invention and other embodiments, a “part” may be a partial circuit, a partial processor, a partial program or software, etc., of course, may be a unit, and may be modular or non-modular.

Corresponding to the target detection method in FIG. 13 , the embodiment of the present invention further provides an electronic device 1300 , and as shown in FIG. 13 , it is an exemplary diagram of the structure of the electronic device 1300 provided in the embodiment of the present invention. , the electronic device 1300,

includes a processor 1301 , a memory 1302 , and a bus 1303 ; the memory 1302 is configured to store executable instructions, and includes an internal storage 13021 and an external memory 13022; Here, the internal storage 13021 is also referred to as an internal memory, and temporarily stores data exchanged with the external memory 13022 such as operation data in the processor 1301 and hardware, and the processor 1301 is the internal storage 13021 . is configured to exchange data with the external memory 13022 through When the instruction is executed by the processor 1301, the process

acquiring an image to be detected;

determining, based on the image to be detected, vertex position information in which each vertex is in the image to be detected and a centripetal shift tensor corresponding to each vertex, wherein the vertex represents the position of a target object in the image to be detected;

Based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex, a target object in the image to be detected is determined.

An embodiment of the present invention further provides a computer readable storage medium, the computer program is stored in the computer readable storage medium, and when the computer program is operated by a processor, the target detection method according to the embodiment of the method run the steps Here, the computer-readable storage medium may be a volatile computer-readable or non-volatile computer-readable storage medium.

An embodiment of the present invention further provides a computer program including a computer readable code, wherein when the computer readable code is run in an electronic device, the processor in the electronic device implements the target detection method step of the first aspect it is to do

The computer program product of the target detection method provided in the embodiment of the present invention includes a computer-readable storage medium in which a program code is recorded, and the instructions included in the program code are the steps of the target detection method according to the embodiment of the method. may be used to perform the above method, and reference may be made to the above method embodiments, which will not be repeatedly described herein.

An embodiment of the present invention further provides a computer program, and when the computer program is executed by a processor, any method of the above embodiment is implemented. The computer program product may be specifically implemented through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically implemented as a computer storage medium, and in some other embodiments, the computer program product is specifically implemented as a software product such as a Software Development Kit (SDK) or the like.

A person of ordinary skill in the art may understand that, for convenience and conciseness of the description, the operation process of the above-described system and apparatus may refer to a corresponding process among the above-described method embodiments, which will be described repeatedly I never do that. In the several embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. The above-described device embodiment is merely exemplary, for example, the division for the unit is merely logical function division, and when actually implemented, there may be other division methods, and for example, a plurality of units or components may be It may be combined or integrated into other systems, or some features may be ignored or not implemented. Further, any coupling or direct coupling or communication connection between each other, shown or discussed, is implemented through some communication interface, and the indirect coupling or communication connection through a device or unit may be electrical, mechanical, or other form.

The unit described as the separation member may or may not be physically separated, and the member shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed in a plurality of network units. According to actual needs, some or all of the units may be selected to implement the purpose of the present embodiment method.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may be an independent physical entity, and two or two or more units may be integrated into one unit. .

When the above functions are implemented in the form of a software function unit and sold or used as a standalone product, one processor may be stored in an executable nonvolatile computer readable storage medium. Based on this understanding, the technical solution of the present invention, that is, a part contributing to the prior art or a part of the technical solution may be implemented in the form of a software product, the computer software product being stored in one storage medium, one It includes a plurality of instructions used to cause a computer device (which may be a personal computer, a server or a network device, etc.) of The above-described storage medium includes a variety of media capable of storing a program code, such as a U disk, a mobile disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. include

The last thing to be described is that the above embodiment is only a specific embodiment of the present invention, and is not intended to limit the technical solution of the present invention, but is intended to illustrate, and the claims of the present invention are not limited thereto, although Although the present invention has been described in detail with reference to the above-described embodiments, any person skilled in the art will find that within the technical scope disclosed in the present invention, the technical solutions described in the above-described embodiments are still It should be understood that changes may be readily devised, or some technical features thereof may be substituted equally; It should be understood that such modifications, changes or replacements should not depart from the spirit and scope of the technical solutions of the present invention, and should all fall within the protection scope of the present invention. Accordingly, the protection scope of the present invention should be based on the protection scope of the above claims.

An embodiment of the present invention provides a target detection method, an apparatus, an electronic device and a computer-readable storage medium, wherein the target detection method includes: acquiring an image to be detected; determining, based on the to-be-detected image, vertex position information for each vertex in the to-be-detected image and a centripetal shift tensor corresponding to each vertex, the vertex indicating the position of the target object in the to-be-detected image; and determining a target object in the to-be-detected image based on vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex. The target detection method proposed in the embodiment of the present invention determines vertices belonging to the same target object based on vertex position information in an image in which each vertex is to be detected and a centripetal shift tensor corresponding to each vertex, so that the determined vertex is based on the same target object can be detected.

Claims (17)

A target detection method comprising:
acquiring an image to be detected;
determining, based on the image to be detected, vertex position information for each vertex in the image to be detected and a centripetal shift tensor corresponding to each vertex, the vertex indicating the position of the target object in the image to be detected; ;and
and determining a target object in the image where each vertex is to be detected based on vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex. The method of claim 1,
Determining the vertex position information and the centripetal shift tensor corresponding to each vertex in the image where each vertex is to be detected, based on the image to be detected,
obtaining an initial feature map corresponding to the image to be detected by extracting features from the image to be detected;
performing vertex pooling on the initial feature map to obtain a vertex pooled feature map; and
and obtaining vertex position information in the image where each vertex is to be detected, and a centripetal shift tensor corresponding to each vertex, based on the vertex pooled feature map. 3. The method of claim 2,
The step of determining vertex position information in the image where each vertex is to be detected based on the vertex pooled feature map comprises:
generating a vertex heat map corresponding to the image to be detected based on the vertex pooled feature map;
Based on the vertex heat map, a probability value of using each feature point in the vertex heat map as a vertex is determined, and based on a probability value of using each feature point as a vertex, the vertex is selected from the feature points of the vertex heat map. step;
Obtaining the location information of each selected vertex in the vertex heat map and local shift information corresponding to each vertex - The local shift information includes the actual physical point indicated by the corresponding vertex position shift information in the vertex heat map. Used to indicate - ; and
Each vertex is a vertex position in the to-be-detected image based on position information of each obtained vertex in the vertex heatmap, local shift information corresponding to each vertex, and a size ratio between the vertex heatmap and the image to be detected A method for detecting a target comprising determining information. 4. The method of claim 2 or 3,
Determining a centripetal shift tensor corresponding to each vertex based on the vertex pooled feature map comprises:
determining a guide shift tensor corresponding to each feature point in the vertex pooled feature map based on the vertex-pooled feature map; Represents a shift tensor pointing to the centroid - ;
Determining shift domain information of the feature point based on the guide shift tensor corresponding to each feature point, wherein the shift domain information includes a shift tensor indicating a shifted feature point corresponding to a plurality of initial feature points associated with the feature point, respectively. Included - ;
Based on the shift domain information of the feature points in the vertex pooled feature map and the vertex pooled feature map, adjusting the feature data of the feature points in the vertex pooled feature map to obtain an adjusted feature map step; and
and determining a centripetal shift tensor corresponding to each vertex based on the adjusted feature map. 4. The method of claim 3,
the vertex heatmap corresponding to the image to be detected includes a vertex heatmap respectively corresponding to a plurality of channels, wherein each channel in the plurality of channels corresponds to one preset target category; After determining a probability value using each feature point as a vertex in the vertex heatmap based on the vertex heatmap, the detection method includes:
For each channel in the plurality of channels, based on a probability value using each feature point in the vertex heatmap corresponding to the channel as a vertex, it is determined whether the vertex exists in the vertex heatmap corresponding to the channel. to do; and
and determining that a target object of a preset target category corresponding to the channel is included in the image to be detected when the vertex exists in the vertex heatmap corresponding to the channel. The method of claim 1,
Determining a target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex includes:
and determining a detection frame of a target object in the image to be detected based on vertex position information in the image where each vertex is to be detected and a centripetal shift tensor corresponding to each vertex. 7. The method of claim 6,
The step of determining the detection frame of the target object in the image to be detected based on the vertex position information in the image where each vertex is to be detected and the centripetal shift tensor corresponding to each vertex,
selecting a pair of candidate vertices constituting a candidate detection frame based on vertex position information in the image where each vertex is to be detected;
determining center point location information indicated by the vertex based on the vertex location information in the image where each vertex is to be detected in each candidate vertex pair and a centripetal shift tensor corresponding to the vertex;
determining center region information corresponding to the candidate vertex pair based on the vertex position information in the image in which each vertex is to be detected in each candidate vertex pair; and
Target detection comprising the step of determining the detection frame of the target object from the candidate detection frame based on center point position information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair Way. 8. The method of claim 7,
Determining center region information corresponding to the candidate vertex pair based on the vertex position information in the image in which each vertex is to be detected in each candidate vertex pair comprises:
determining vertex position information indicating a central region frame corresponding to the candidate vertex pair based on vertex position information of each vertex in the candidate vertex pair; and
and determining a coordinate range of a central region frame corresponding to the candidate vertex pair based on vertex position information of the central region frame. 9. The method according to claim 7 or 8,
The step of determining the detection frame of the target object in the candidate detection frame based on the center point position information indicated by each vertex in each candidate vertex pair and the center region information corresponding to the vertex pair comprises:
determining a valid candidate vertex based on center location information indicated by each vertex in each candidate vertex pair and center region information corresponding to the vertex pair;
Corresponding to each valid candidate vertex pair based on center point location information indicated by each vertex in the valid candidate vertex pair, center region information corresponding to the valid candidate vertex pair, and a probability value corresponding to each vertex in the valid candidate vertex pair determining a score of a candidate detection frame to be used, wherein a probability value corresponding to each vertex is used to indicate a probability value in which the vertex uses a feature point corresponding to a vertex heat map as a vertex; and
Target detection comprising the step of determining the detection frame of the target object in the candidate detection frame based on the score of the candidate detection frame corresponding to each valid candidate vertex pair and the size of the overlapping region between adjacent candidate detection frames Way. 7. The method of claim 6,
After determining the detection frame of the target object in the image to be detected, the target detection method includes:
The method further comprising the step of determining instance information of the target object in the image to be detected based on the detection frame of the target object and an initial feature map obtained by performing feature extraction on the image to be detected target detection method. 11. The method of claim 10,
Determining instance information of the target object in the image to be detected based on an initial feature map obtained by performing feature extraction on the detection frame of the target object and the image to be detected,
extracting, by the initial feature map, feature data of feature points in the detection frame based on the detection frame of the target object and the initial feature map; and
and determining the instance information of the target object in the image to be detected based on the feature data of the feature points in the detection frame by the initial feature map. 12. The method according to any one of claims 1 to 11,
The target detection method is implemented through a neural network, and the neural network is obtained by training using a sample image including a labeling target sample object. 13. The method of claim 12,
The neural network is
acquiring a sample image;
Based on the sample image, each sample vertex determines vertex position information in the sample image and a centripetal shift tensor corresponding to each sample vertex, wherein the sample vertex represents the position of a target sample object in the sample image. ;
predicting a target sample object in the sample image based on vertex position information of each sample vertex in the sample image and a centripetal shift tensor corresponding to each sample vertex; and
Target obtained by training using the step of adjusting the value of the network parameter of the neural network based on the predicted target sample object in the sample image and the labeling target sample object in the sample image detection method. A target detection device comprising:
an acquiring unit configured to acquire an image to be detected;
a determining unit configured to determine, based on the image to be detected, a centripetal shift tensor corresponding to each vertex and vertex position information for each vertex in the to-be-detected image, wherein the vertex determines the position of the target object in the image to be detected indicates - ;
and a detection unit configured to determine a target object in the image where each vertex is to be detected based on vertex position information in the to-be-detected image and a centripetal shift tensor corresponding to each vertex. As an electronic device,
a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when an electronic device is operated, the processor and the memory communicate through a bus, and the machine-readable instructions 14. An electronic device characterized in that when is executed by the processor, the step of the target detection method according to any one of claims 1 to 13 is executed. A computer-readable storage medium comprising:
A computer program is stored in the computer readable storage medium, and when the computer program is executed by a processor, the computer readable method according to any one of claims 1 to 13 implements the steps of the target detection method. possible storage medium. A computer program comprising:
14. A method comprising computer readable code, wherein when the computer readable code is operated in an electronic device, a processor in the electronic device implements the method for detecting a target according to any one of claims 1 to 13. computer program to do.

Images