TWI739601B - Image processing method, electronic equipment and storage medium - Google Patents

Abstract

This application discloses an image processing method, electronic equipment and computer readable storage medium. The method includes: obtaining a first position of a first character point in an image to be processed and a second position of a first character frame in the image to be processed; the first position and the second position are both used to characterize the position of the first person in the image to be processed; according to the first position and the second position, the third position of the first person in the image to be processed is obtained.

Description

Image processing method, electronic equipment and storage medium

The present invention relates to the field of computer vision technology, but is not limited to an image processing method, electronic equipment, and computer-readable storage medium.

In public places (such as squares, supermarkets, subway stations, docks, etc.), sometimes there will be too many people, which will lead to too crowded. At this time, some public accidents are prone to occur, such as stampede incidents. Therefore, how to determine the position of a person in an image is of great significance.

At present, based on computer vision technology, the human head detection processing can be performed on the image to obtain the position of the person in the image, but the accuracy of the position is low.

The embodiment of the present invention provides an image processing method, an electronic device, and a computer-readable storage medium.

The embodiment of the present invention provides an image processing method, and the method includes: Acquire the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed; both the first position and the second position are used to characterize the first The position of the character in the image to be processed; According to the first position and the second position, the third position of the first person in the image to be processed is obtained.

In some embodiments of the present invention, the method further includes: Acquire the first confidence level of the first location and the second confidence level of the second location; the first confidence level is negatively correlated with the scale of the first location; the second confidence level is related to the The scale of the second position is positively correlated; The obtaining the third position of the first person in the image to be processed according to the first position and the second position includes: Taking the position with the highest confidence level among the first position and the second position as the fourth position; According to the fourth position, the third position is obtained.

In some embodiments of the present invention, the acquiring the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed includes: Performing character positioning processing on the image to be processed to obtain the first position and the position of the at least one character frame; Determine the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance; Use the person frame corresponding to the second distance as the first person frame, and determine the second position of the first person frame in the image to be processed; the second distance is the at least one first distance The distance in which the distance threshold is not exceeded.

In some embodiments of the present invention, the performing character positioning processing on the to-be-processed image to obtain the first position includes: Performing character positioning processing on the image to be processed to obtain the position of at least one character point; The position with the highest confidence among the positions of the at least one character point is taken as the first position.

In some embodiments of the present invention, the at least one character frame includes a second character frame; the at least one first distance includes a third distance between the first character point and the second character frame; The determining the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance includes: Obtaining the fourth distance between the first character point and the second character frame according to the first position and the position of the second character frame; Determine the difference between the first scale and the second scale to obtain the first difference; the first scale is the scale of the first person point in the image to be processed; the second scale is the first 2. The scale of the character frame in the image to be processed; According to the fourth distance and the first difference, the third distance is obtained; the third distance is positively correlated with the first difference.

In some embodiments of the present invention, the method further includes: Determine the second character point according to the position of the second character frame; Determining a midpoint between the first character point and the second character point to obtain a third character point; Obtain a first scale index; the first scale index represents the mapping between the first size and the second size; the first size is the size of the first reference object located at the first scale position; the second size is The size of the first reference object in the real world; the first scale position is the position of the third person point in the image to be processed; The determining the difference between the first scale and the second scale to obtain the first difference includes: According to the first scale index, the first difference is obtained.

In some embodiments of the present invention, the obtaining the first scale index includes: Performing object detection processing on the image to be processed to obtain a first object frame and a second object frame; The first length is obtained according to the length of the first object frame in the y-axis direction, and the second length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the length of the image to be processed The vertical axis of the pixel coordinate system; Obtaining a second scale index according to the first length and the third length of the first object in the real world, and obtaining a third scale index according to the second length and the fourth length of the second object in the real world; The first object is the detection object contained in the first object frame; the second object is the detection object contained in the second object frame; the second scale index represents the difference between the third size and the fourth size The third size is the size of the second reference object located at the second scale position; the fourth size is the size of the second reference object in the real world; the second scale position is based on The position of the first object frame is determined in the image to be processed; the third scale index represents the mapping between the fifth size and the sixth size; the fifth size is the position at the third scale The size of the third reference object; the sixth size is the size of the third reference object in the real world; the third scale position is based on the position of the second object frame in the image to be processed Determined location Perform curve fitting processing on the second scale index and the third scale index to obtain the scale index diagram of the image to be processed; the first pixel value in the scale index diagram represents the seventh size and the eighth size Mapping between sizes; the seventh size is the size of the fourth reference object at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; the first pixel value Is the pixel value of the first pixel; the fourth scale position is the position of the second pixel in the image to be processed; the position of the first pixel in the scale index map is the same as that of the first pixel The positions of the two pixels in the image to be processed are the same; According to the scale index map and the position of the third person point, the first scale index is obtained.

In some embodiments of the present invention, the second scale position is the position of the first object point in the image to be processed; the third scale position is the position of the second object point in the image to be processed Location; The first object point is one of the following: the geometric center of the first object frame, the vertex of the first object frame; the second object point is one of the following: The geometric center and the vertex of the second object frame.

In some embodiments of the present invention, the second character point is one of the following: a geometric center of the second character frame, and a vertex of the second character frame.

In some embodiments of the present invention, the pixel area covered by the first character dot and the pixel area included in the first character frame are both human head areas.

In some embodiments of the present invention, the shape of the first character frame is a rectangle, a diamond, a circle, an ellipse, or a polygon.

An embodiment of the present invention also provides an image processing device, which includes: An acquiring unit configured to acquire the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed; both the first position and the second position Used to characterize the position of the first person in the image to be processed; The first processing unit is configured to obtain a third position of the first person in the image to be processed according to the first position and the second position.

In some embodiments of the present invention, the acquiring unit is further configured to acquire a first confidence level of the first position and a second confidence level of the second position; the first confidence level is compared with the first confidence level The scale of the position is negatively correlated; the second confidence level is positively correlated with the scale of the second position; The first processing unit is configured as: Taking the position with the highest confidence level among the first position and the second position as the fourth position; According to the fourth position, the third position is obtained.

In some embodiments of the present invention, the acquiring unit is further configured to perform character positioning processing on the image to be processed to obtain the first position and the position of the at least one character frame; Determine the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance; A person frame corresponding to the second distance is used as the first person frame; the second distance is a distance that does not exceed a distance threshold in the at least one first distance.

In some embodiments of the present invention, the acquiring unit is configured to: Performing character positioning processing on the image to be processed to obtain the position of at least one character point; The position with the highest confidence among the positions of the at least one character point is taken as the first position.

In some embodiments of the present invention, the at least one character frame includes a second character frame; the at least one first distance includes a third distance between the first character point and the second character frame; The acquiring unit is configured to: Obtaining the fourth distance between the first character point and the second character frame according to the first position and the position of the second character frame; Determine the difference between the first scale and the second scale to obtain the first difference; the first scale is the scale of the first person point in the image to be processed; the second scale is the first 2. The scale of the character frame in the image to be processed; According to the fourth distance and the first difference, the third distance is obtained; the third distance is positively correlated with the first difference.

In some embodiments of the present invention, the device further includes a second processing unit, and the second processing unit is configured to: Determine the second character point according to the position of the second character frame; Determining a midpoint between the first character point and the second character point to obtain a third character point; The acquiring unit is further configured to acquire a first scale index; the first scale index represents a mapping between a first size and a second size; the first size is the value of the first reference object located at the first scale position Size; the second size is the size of the first reference object in the real world; the first scale position is the position of the third person point in the image to be processed; According to the first scale index, the first difference is obtained.

In some embodiments of the present invention, the acquiring unit is configured to: Performing object detection processing on the image to be processed to obtain a first object frame and a second object frame; The first length is obtained according to the length of the first object frame in the y-axis direction, and the second length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the length of the image to be processed The vertical axis of the pixel coordinate system; Obtaining a second scale index according to the first length and the third length of the first object in the real world, and obtaining a third scale index according to the second length and the fourth length of the second object in the real world; The first object is the detection object contained in the first object frame; the second object is the detection object contained in the second object frame; the second scale index represents the difference between the third size and the fourth size The third size is the size of the second reference object located at the second scale position; the fourth size is the size of the second reference object in the real world; the second scale position is based on The position of the first object frame is determined in the image to be processed; the third scale index represents the mapping between the fifth size and the sixth size; the fifth size is the position at the third scale The size of the third reference object; the sixth size is the size of the third reference object in the real world; the third scale position is based on the position of the second object frame in the image to be processed Determined location Perform curve fitting processing on the second scale index and the third scale index to obtain the scale index diagram of the image to be processed; the first pixel value in the scale index diagram represents the seventh size and the eighth size Mapping between sizes; the seventh size is the size of the fourth reference object at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; the first pixel value Is the pixel value of the first pixel; the fourth scale position is the position of the second pixel in the image to be processed; the position of the first pixel in the scale index map is the same as that of the first pixel The positions of the two pixels in the image to be processed are the same; According to the scale index map and the position of the third person point, the first scale index is obtained.

In some embodiments of the present invention, the second scale position is the position of the first object point in the image to be processed; the third scale position is the position of the second object point in the image to be processed Location; The first object point is one of the following: the geometric center of the first object frame, the vertex of the first object frame; the second object point is one of the following: The geometric center and the vertex of the second object frame.

In some embodiments of the present invention, the second character point is one of the following: a geometric center of the second character frame, and a vertex of the second character frame.

In some embodiments of the present invention, the pixel area covered by the first character dot and the pixel area included in the first character frame are both human head areas.

In some embodiments of the present invention, the shape of the first character frame is a rectangle, a diamond, a circle, an ellipse, or a polygon.

The embodiment of the present invention also provides a processor configured to execute any one of the above-mentioned image processing methods.

An embodiment of the present invention also provides an electronic device, including: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, the processor executes the computer instructions In this case, the electronic device executes any one of the above-mentioned image processing methods.

The embodiment of the present invention also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, The processor executes any one of the above-mentioned image processing methods.

An embodiment of the present invention also provides a computer program, the computer program includes a computer program or instruction, and when the computer program or instruction runs on a computer, the computer is caused to execute any one of the above-mentioned image processing methods.

In the embodiment of the present invention, when the person is far away in the image to be processed, the position of the person in the image to be processed is determined according to the position of the person point; when the person is close to the image to be processed In this case, the position of the character in the image to be processed is determined according to the position of the character frame. In this way, the accuracy of the position of the character in the image to be processed is improved.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present invention.

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only These are a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

The terms "first" and "second" in the specification and patent application scope of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

Reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

First, define some concepts that will appear below. In the embodiment of the present invention, [a, b] represents a value interval greater than or equal to a and less than or equal to b.

In the embodiment of the present invention, the image scale corresponding to a person near the image is large, and the image scale corresponding to a person far away in the image is small. In the embodiment of the present invention, "far" refers to the distance between the real person corresponding to the person in the image and the imaging device that captures the image, and "near" refers to the real person corresponding to the person in the image and the image capture device. The distance between the imaging devices of the image is close.

In the image, the area of the pixel area covered by a person in the vicinity is larger than the area of the pixel area covered by a person in the distance. For example, the person A in FIG. 1 is a close person compared to the person B, and the area of the pixel area covered by the person A is larger than the area of the pixel area covered by the person B. However, the scale of the pixel area covered by the people in the vicinity is large, and the scale of the pixel area covered by the people in the distance is small. Therefore, the area of the pixel area covered by the person is positively correlated with the scale of the pixel area covered by the person.

In the embodiment of the present invention, the positions in the image all refer to the positions under the pixel coordinates of the image. In the embodiment of the present invention, the abscissa of the pixel coordinate system is used to indicate the number of columns where the pixel is located, and the ordinate under the pixel coordinate system is used to indicate the number of rows where the pixel is located. For example, in the image shown in Figure 2, the upper left corner of the image is the coordinate origin O , the direction parallel to the image row is the direction of the X axis, and the direction parallel to the image column is the direction of the Y axis. Direction, the construction pixel coordinate system is XOY . The units of the abscissa and ordinate are pixels. For example, _{the coordinates of pixel A 11} in Figure 2 are (1, 1), the coordinates of pixel A ₂₃ are (3, 2), _{the coordinates of pixel A 42} are (2, 4), and the coordinates of pixel A ₃₄ are The coordinates are (4, 3).

In public places (such as squares, supermarkets, subway stations, docks, etc.), sometimes there will be too many people, which will lead to too crowded. At this time, some public accidents are prone to occur, such as stampede incidents. Therefore, how to determine the number of people in the image, the density of the crowd in the image, and the distribution of the crowd in the image are of very important significance.

In some embodiments of the present invention, the position of each person in the image can be determined to determine the number of people in the image, the density of the crowd in the image, and the distribution of the crowd in the image. With the development of computer vision technology, computer vision-based methods can determine the location of each person in the image (for ease of presentation, determining the location of each person in the image is referred to as crowd positioning below).

In related technologies, a human head frame (that is, a frame containing a human head) in the image can be obtained by performing head detection processing on the image based on computer vision technology. The position of the character in the image can be determined according to the position of the head frame. Since the heads in the near area are larger than the heads in the distance in the image, and the density of the heads in the distance is higher when the crowd density is high, the accuracy of the position of the head frame in the distance is low. In turn, the accuracy of crowd positioning is low. Based on this, the embodiment of the present invention provides a method for improving the accuracy of crowd positioning.

The image processing method of the embodiment of the present invention may be implemented by an image processing device. In some embodiments, the image processing device may be one of the following: a mobile phone, a computer, a server, and a tablet computer. The image processing method of the embodiment of the present invention can also be implemented by a processor executing computer code. The embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention.

Please refer to FIG. 3, which is a schematic flowchart of an image processing method according to an embodiment of the present invention.

Step 301: Obtain the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed, wherein both the first position and the second position are used for characterization The position of the first person in the image to be processed.

In the embodiment of the present invention, both the first person point and the first person frame can be obtained by performing person detection processing on the image to be processed. In some embodiments, the person detection processing of the image to be processed can be implemented through the person detection network in the image to be processed. The person detection network is obtained by training a convolutional neural network with a training image, where the label information of the training image includes at least one of the following: the position of the character point and the position of the character frame. In the case that the annotation information of the training image includes the position of the person point, the person detection network is used to process the image to be processed, and the position of at least one person point including the above-mentioned first position can be obtained; In the case where the annotation information includes the position of the person frame, the person detection network is used to process the image to be processed, and the position of at least one person frame including the above-mentioned second position can be obtained; the annotation information of the training image includes the person point In the case of the position of the person frame and the position of the person frame, using the person detection network to process the image to be processed, the position of at least one person point including the above-mentioned first position and at least the position of at least one person including the above-mentioned second position can be obtained The position of a character frame.

In the image to be processed, the pixel area covered by the character point can be regarded as the character area, and the character area is the pixel point area covered by the human body. For example, the area covered by the first character point belongs to the pixel point area covered by the human head. For another example, the area covered by the first character point belongs to the pixel point area covered by the arm. For another example, the area covered by the first character point belongs to the pixel point area covered by the torso. In the image to be processed, the pixel area contained in the character frame can be regarded as the character area. For example, the area contained in the first character frame is the pixel area covered by the human head. For another example, the area contained in the first person frame is the pixel area covered by the face. For another example, the area included in the first character frame is the pixel area covered by the torso.

In the embodiment of the present invention, the first character point may have any shape, and the present invention does not limit the shape of the first character point. In some embodiments, the shape of the first character point includes at least one of the following: a circle, a diamond, a rectangle, an ellipse, and a polygon.

In the embodiment of the present invention, the first position is the position of the first character point in the pixel coordinate system of the image to be processed. For example, in the case where the shape of the first character point is a circle, the first position may be the position of the center of the first character point under the pixel coordinates. For another example, when the shape of the first character point is a rectangle, the first position may be the position of the geometric center of the first character point under the pixel coordinates.

In some embodiments of the present invention, the first character point is a pixel point in the image to be processed, and the first position is the position of the pixel point in the pixel coordinate system. For example, in the to-be-processed image shown in FIG. 4, the first character point is the pixel point A ₁₃ , and the first position is the position of the pixel point A ₁₃ in the pixel coordinate system.

In some embodiments of the present invention, the first position also carries size information of the first character point. For example, in the case where the shape of the first character point is a circle, the first position also carries the radius of the first character point. For another example, when the first character point is a rectangle, the first position also carries the length and width of the first character point. According to the first position, the pixel area covered by the first character point in the image to be processed can be determined. It is understandable that if the first character point is a pixel in the image to be processed, even if the first position does not carry the size information of the first character point, it can be determined that the first character point is in the image to be processed according to the first position. The area of pixels covered by.

In the embodiment of the present invention, the first character frame may have any shape, and the present invention does not limit the shape of the first character frame. In some embodiments of the present invention, the shape of the first character frame includes at least one of the following: a rectangle, a diamond, a circle, an ellipse, and a polygon.

In the embodiment of the present invention, the second position is the position of the first character frame in the image to be processed. For example, when the shape of the first character frame is a rectangle, the second position may include coordinates of any pair of diagonal corners in the rectangle, wherein the pair of diagonal corners refers to two vertices on the diagonal of the rectangle. For another example, when the shape of the first character frame is a rectangle, the second position may include: the position of the geometric center of the rectangle, the length of the rectangle, and the width of the rectangle. For another example, when the shape of the first character frame is circular, the second position may include: the center of the first character frame and the radius of the first character frame. According to the second position, the pixel area included in the image to be processed by the first character frame can be determined.

In the embodiment of the present invention, the first position may be used to indicate the position of the person in the image to be processed. In a possible implementation manner, the first position indicates that there is a person at the first position in the image to be processed. For example, if the first position is (7, 8), the position of the person in the image to be processed is (7, 8).

In some embodiments of the present invention, the first position is represented in the image to be processed, and the pixel neighborhood constructed based on the first position is a character area. For example, suppose the first position is (7, 8), the first position is the center of the circle, and the 2 pixels are the radius to construct the pixel neighborhood n1, and then n1 is the character area. At this time, the position of any pixel in n1 can be used as the position of the person in the image to be processed.

In the embodiment of the present invention, the second position may also be used to indicate the position of the person in the image to be processed. In some embodiments of the present invention, the position of any pixel in the first character frame has a character, and the position of any pixel in the first character frame can be determined according to the second position. Therefore, according to the first character frame The second position can determine the position of the person in the image to be processed. For example, suppose that the shape of the first character frame is rectangular, and the second position includes the coordinates of a pair of diagonal corners of the first character frame: (7, 8), (10, 10). At this time, the coordinates (x, y) can be used as the position of the person in the image to be processed, where the value range of x is [7, 10], and the value range of y is [8, 10].

In some embodiments of the present invention, it is determined that there is a character at the geometric center of the first character frame according to the second position. For example, suppose that the shape of the first character frame is a rectangle, and the second position includes the coordinates of a pair of diagonal corners of the first character frame: (8, 8), (12, 10). At this time, the coordinates of the geometric center of the first character frame are (10, 9), that is, the position of the character in the image to be processed is (7, 8).

In the embodiment of the present invention, the first position and the second position are both used to indicate the position of the same person (that is, the above-mentioned first person) in the image to be processed. In an implementation manner of acquiring the first position, the image processing device receives the first position input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc. In another implementation manner of acquiring the first position, the image processing apparatus receives the first position sent by the first terminal. In some embodiments of the present invention, the first terminal may be any of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. In yet another implementation manner for obtaining the first position, the image processing device performs person detection processing on the image to be processed to obtain the first position.

In an implementation manner of acquiring the second position, the image processing device receives the second position input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc. In another implementation manner of acquiring the second location, the image processing apparatus receives the second location sent by the second terminal. In some embodiments of the present invention, the second terminal may be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. The second terminal and the first terminal may be the same or different, which is not limited in the present invention. In yet another implementation manner for acquiring the second position, the image processing device performs person detection processing on the image to be processed to obtain the second position.

Step 302: Obtain the third position of the first person in the image to be processed according to the first position and the second position.

Because in the image to be processed, people in the vicinity are larger than those in the distance, and when the crowd density is high, the crowd density in the distance is high (such as people next to each other, or even in the image to be processed). , There is an overlapping area between two different character areas), the accuracy of the position of the first character frame in the distance is low, which in turn leads to the low accuracy of the position of the character determined according to the first character frame.

Since the area of the nearby person in the image to be processed is large, the number of person points corresponding to the nearby person area may exceed one. At this time, the position of the person corresponding to the person area exceeds 1. , Which in turn leads to low accuracy of the position of the character determined according to the first character frame. That is, according to the position of the character frame, the accuracy of determining the position of a nearby person is high; according to the position of the character frame, the accuracy of determining the position of a distant person is low. According to the position of the character point, the accuracy of determining the position of the nearby character is low; according to the position of the character point, the accuracy of determining the position of the distant character is high.

In the embodiment of the present invention, for the same character in the image to be processed, the position of the character is determined according to the position of the character point of the character and the position of the character frame of the character. When the image is close to the image, the position of the character is determined according to the character frame of the character; and when the character is far away in the image to be processed, the position of the character is determined according to the character point of the character. In this way, the accuracy of the position of the person in the image to be processed can be improved.

In some embodiments of the present invention, the ordinate of the center of the first character point (hereinafter referred to as the first ordinate) is obtained according to the first position, and the ordinate of the center of the first character frame (hereinafter referred to as the first ordinate) is obtained according to the second position. Will be called the second ordinate). Since the size of the ordinate in the image to be processed can be used to represent the distance and nearness, it can be determined according to the size of the ordinate to whether the first person is located close to the image to be processed or far away from the image to be processed. In the case where the first person is located close to the image to be processed, the position of the first person in the image to be processed is determined according to the first position; in the case where the first person is located far away from the image to be processed, the position of the first person in the image to be processed is determined according to the The second position determines the position of the first person in the image to be processed.

For example, when the first ordinate is at [first value, second value], the position of the first person in the image to be processed is obtained according to the first position, that is, the third position; when the first ordinate is at ( In the case of the second value, the third value], the position of the first person in the image to be processed is obtained according to the second position, that is, the third position. The first value is the largest ordinate in the image to be processed, and the third The value is the smallest ordinate in the image to be processed, and the second value is the average of the first value and the third value.

In some embodiments of the present invention, the farther the distance between the real person and the imaging device that collects the above-mentioned image, the smaller the size of the person in the image. Therefore, the image processing device can determine that the first person is far away based on the ratio of the length of a certain part of the body of the first person in the image to be processed to the length of the part in the real world (hereinafter referred to as the reference ratio). The place is still close.

For example, suppose that the first character frame is a human body frame, that is, the first frame contains the entire body of the first character. At this time, the image processing device may use the ratio of the height of the first person in the image to be processed to the height of the first person in the real world as a reference ratio. The image processing device can then determine whether the first person is far away or near based on the reference ratio. In some embodiments of the present invention, the fourth value (such as the average height of a human) may be used as the height of the first person in the real world. For another example, suppose that the first person frame is a face frame, that is, the first frame contains the face of the first person. At this time, the image processing device can use the ratio of the length of the first person’s face in the image to be processed to the length of the first person’s face in the real world as a reference ratio, and the image processing device can then rely on the image The processing device determines whether the first person is far away or close. In some embodiments of the present invention, the fifth value (average length of the face) may be used as the length of the face of the first person in the real world. For another example, suppose that the first character frame is a human head frame, that is, the first frame contains the first person's head. At this time, the image processing device may use the ratio of the length of the head of the first person in the image to be processed to the length of the head of the first person in the real world as a reference ratio. The image processing device can then determine whether the first person is far away or near based on the reference ratio. In some embodiments of the present invention, the sixth value (the average length of the human head) may be used as the length of the first character's head in the real world.

In the case of determining whether the first person is close or far away according to the reference ratio, the image processing device may further determine whether to determine the position of the first person according to the first position or to determine the position of the first person according to the second position.

For example, if the reference ratio does not exceed the seventh value, it indicates that the first person is far away. At this time, the position of the first person in the image to be processed is obtained according to the first position, that is, the third position; the reference ratio exceeds the seventh value, indicating that The first person is close, and at this time, the position of the first person in the image to be processed is obtained according to the second position, that is, the third position. In some embodiments of the present invention, the seventh value is 1.

In the embodiment of the present invention, when the person is far away in the image to be processed, the position of the person in the image to be processed is determined according to the position of the person point; in the case where the person is close to the image to be processed Next, determine the position of the character in the image to be processed according to the position of the character frame. In this way, the accuracy of the position of the character in the image to be processed is improved.

In some embodiments of the present invention, before performing step 302, the image processing apparatus further performs the following steps.

Step 1: Obtain the first confidence level of the above-mentioned first position and the second confidence level of the above-mentioned second position.

In the embodiment of the present invention, the first confidence level may be obtained in the process of performing person detection processing on the image to be processed to obtain the first position. The second degree of confidence can be obtained in the process of performing person detection processing on the image to be processed to obtain the second position.

The first degree of confidence is negatively related to the scale of the first position, that is, the smaller the distance between the first position and the x-axis of the pixel coordinate system, the higher the first degree of confidence; the difference between the first position and the x-axis of the pixel coordinate system The greater the distance between, the lower the first confidence. The second degree of confidence is positively correlated with the scale of the second position, that is, the smaller the distance between the second position and the x-axis of the pixel coordinate system, the lower the second degree of confidence; between the second position and the x-axis of the pixel coordinate system The greater the distance, the higher the first confidence.

In an implementation manner for obtaining the first degree of confidence, the image processing device receives the first degree of confidence input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc. In another implementation manner for obtaining the first confidence level, the image processing apparatus receives the first confidence level sent by the third terminal. Optionally, the third terminal may be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. The third terminal and the first terminal may be the same or different, which is not limited in the present invention. In yet another implementation manner for obtaining the first confidence level, the first location carries the confidence information of the first location, and the image processing device obtains the first confidence level by obtaining the first location.

In an implementation manner for obtaining the second degree of confidence, the image processing device receives the second degree of confidence input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc. In another implementation manner for obtaining the second confidence level, the image processing apparatus receives the second confidence level sent by the fourth terminal. Optionally, the third terminal can be any two of the following: mobile phones, computers, tablet computers, servers, and wearable devices. The fourth terminal and the third terminal may be the same or different, which is not limited in the present invention. In yet another implementation manner for obtaining the second confidence level, the second location carries the confidence level information of the second location, and the image processing apparatus obtains the second confidence level by obtaining the second location.

After performing step 1, the image processing apparatus specifically performs the following steps in the process of performing step 302.

Step 2: Use the position with the highest confidence level among the first position and the second position as the fourth position.

For example, the first confidence is 0.8, the second confidence is 0.9, and the position with the highest confidence among the first position and the second position is the second position. At this time, the image processing device regards the second position as the fourth position.

Step 3: According to the fourth position, the third position is obtained.

，第四位置為

。在一種可能實現的方式中，

、

滿足下式：

（1） 其中， k為正數。例如， k=1。 Suppose the third position is

, The fourth position is

. In one possible way,

,

Meet the following formula:

(1) Among them, k is a positive number. For example, k =1.

、

滿足下式：

（2） 其中， k為正數、 c為實數。例如， k=1， c=0。 In another possible way,

,

Meet the following formula:

(2) Among them, k is a positive number and c is a real number. For example, k =1 and c =0.

、

滿足下式：

（3） 其中， k為正數、 c為實數。例如， k=1， c=0。 In another possible way,

,

Meet the following formula:

(3) Among them, k is a positive number and c is a real number. For example, k =1 and c =0.

In the embodiment of the present invention, for the same character, the position of the character is determined according to the position of the character point and the position of the character frame with the highest confidence, which can improve the accuracy of the position of the character.

As an optional implementation manner, before step 301 is executed, the image processing apparatus further executes the following steps.

Step 4: Obtain the above-mentioned image to be processed.

In the embodiment of the present invention, the image to be processed may be any image. For example, the image to be processed may contain people. The image to be processed may only include the human head, without the torso and limbs (the torso and limbs are referred to as the human body below). The image to be processed can also include only the human body and not the human head. The image to be processed can also include only lower limbs or upper limbs. The present invention does not limit the area of the human body specifically included in the image to be processed. For another example, the image to be processed may contain animals. For another example, the image to be processed may include plants. The present invention does not limit the content contained in the image to be processed.

In an implementation manner of acquiring the image to be processed, the image processing device receives the image to be processed input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc. In another implementation manner of acquiring the image to be processed, the image processing apparatus receives the image to be processed sent by the fifth terminal. In some embodiments, the fifth terminal may be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. The fifth terminal and the first terminal may be the same or different, which is not limited in the present invention. In yet another implementation manner for acquiring the image to be processed, the image processing device may acquire the image to be processed through the imaging element. In some embodiments, the aforementioned imaging element may be a camera.

After performing step 4, the image processing apparatus specifically performs the following steps during the process of performing step 301.

Step 5: Perform character positioning processing on the image to be processed to obtain the first position and the position of the at least one character frame.

In the embodiment of the present invention, by performing character positioning processing on the image to be processed, it can be determined whether the image to be processed contains a person. In the case that the image to be processed contains a person, the position of the person in the image to be processed can also be obtained. The above position includes the position of the character point and the position of the character frame.

In some embodiments of the present invention, the character positioning processing on the image to be processed may be implemented by a convolutional neural network. By using the image with annotated information as training data, the convolutional neural network is trained, so that the trained convolutional neural network can complete the character positioning processing of the image. The label information of the image in the training data is the position of the character point and the position of the character frame. In the process of using training data to train the convolutional neural network, the convolutional neural network extracts the characteristic data of the image from the image, and determines whether there are people in the image based on the characteristic data. In the case of a person, the position of the person point and the position of the person frame are obtained according to the feature data of the image. Use the labeled information as the supervision information to supervise the results obtained in the training process of the convolutional neural network, and update the parameters of the convolutional neural network to complete the training of the convolutional neural network. In this way, the trained convolutional neural network can be used to process the image to be processed to obtain the position of the character in the image to be processed (including the position of the character point and the position of the character frame).

In some embodiments of the present invention, the person location processing can be implemented by a person detection algorithm, where the person detection algorithm can be one of the following: you only look once (YOLO), target detection algorithm Method (deformable part model, DMP), single shot multiBox detector (SSD), Faster Region-based Convolutional Neural Networks (Faster-RCNN) algorithm, etc. Etc., the present invention does not limit the person detection algorithm that realizes the person positioning process.

Since the number of characters in the image to be processed may be more than one, by performing character positioning processing on the image to be processed, the number of character points and the number of character frames obtained are both at least one. Therefore, the number of character point positions is at least one, and the number of character frame positions is also at least one.

In some embodiments of the present invention, when the image processing device performs person detection processing on the image to be processed to obtain the position of at least one character point and the position of the at least one character frame, it will also obtain the position of each character point. The confidence level and the confidence level of the position of each character frame. When the number of character points exceeds 1, the position of the character point with the highest confidence is taken as the first position. For example, perform person positioning processing on the image to be processed to obtain position 1, position 2, and position 3, where the confidence of position 1 is 0.7, the confidence of position 2 is 0.9, and the confidence of position 3 is 0.8. Since position 2 has the highest confidence, position 2 is taken as the first position.

Step 6: Determine the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance.

。 According to the first position and the position of a character frame, the distance between a character frame and the first character point can be determined to obtain a first distance. For example, the first position is (7, 8), at least one character frame includes character frame 1, the shape of character frame 1 is rectangular, and the position of character frame 1 includes the coordinates of a pair of diagonal corners of character frame 1: (6, 8 ), (4, 12). The distance between the first character point and the character frame 1 may be the distance between the first character point and the center of the character frame 1, and the distance is:

.

。 For another example, the first position is (7, 8), at least one character frame includes character frame 2, the shape of character frame 2 is rectangular, and the position of character frame 2 includes a pair of diagonal coordinates of character frame 1: (6, 8), (4, 12). The distance between the first character point and the character frame 1 may be the distance between the first character point and the top character point in the character frame 2 that is closest to the original character point of the pixel coordinate system, and the distance is:

.

The distance between the first character point and each character frame is determined respectively, and at least one first distance can be obtained.

Step 7: Use the person frame corresponding to the second distance as the first person frame, and determine the second position of the first person frame in the image to be processed, wherein the second distance is the at least one first distance. The distance in which the distance threshold is not exceeded.

In the embodiment of the present invention, the distance between the character point and the character frame does not exceed the distance threshold, indicating that the character point and the character frame belong to the same character, that is, the position of the character point and the position of the character frame are used to indicate that the same character is in the process to be processed. The position in the image.

Therefore, it is determined that at least one of the first distances does not exceed the distance threshold, and the second distance is obtained. The character frame corresponding to the second distance is taken as the first character frame. For example, at least one character frame includes character frame 1 and character frame 2. The distance between the first point and the character frame 1 is 20, and the distance between the first point and the character frame 2 is 30. Assuming that the distance threshold is 25, the second distance is 20. At this time, the character frame corresponding to the second distance is the character frame 1.

In some embodiments of the present invention, when the number of character frames corresponding to the second distance exceeds 1, the position of the character frame with the highest confidence is taken as the second position, and the character frame corresponding to the second position is taken as the second position. As the first character frame. In the embodiment of the present invention, the distance between the first point and the character frame is used to determine the character frame belonging to the same character as the first point, and then the first character frame is determined.

In some embodiments of the present invention, after obtaining the position of at least one character point and the position of at least one character frame by performing step 5, the image processing device may use the position with the highest confidence among the positions of the at least one character frame as the first Two positions, and the character frame corresponding to the second position is used as the first character frame. The image processing device determines the distance between the at least one character point and the first character frame according to the second position and the position of the at least one character point to obtain at least one first intermediate distance. The image processing device uses the person point corresponding to the second intermediate distance as the first person point, where the second intermediate distance is at least one of the first intermediate distances that does not exceed the distance threshold.

Please refer to FIG. 5, which is a schematic flowchart of a possible implementation method of step 6 according to an embodiment of the present invention.

Step 501: Obtain a fourth distance between the first character point and the second character frame according to the first position and the position of the second character frame.

In the implementation of the present invention, at least one character frame includes a second character frame. According to the position of the first position and the second character frame, the distance between the first point and the second character frame, that is, the fourth distance, is determined. For the implementation manner of obtaining the fourth distance according to the position of the first position and the second character frame, refer to the implementation manner of obtaining a first distance according to the first position and the position of a character frame in step 5.

It should be understood that in this step, the distance obtained according to the position of the first position and the position of the second character frame is not the first distance but the fourth distance.

Step 502: Determine the difference between the first scale and the second scale to obtain the first difference, where the first scale is the scale of the first person point in the image to be processed, and the second scale is the first The scale of the second person frame in the image to be processed.

In some embodiments of the present invention, the first difference can be obtained by calculating the difference between the ordinate of the first character point and the ordinate of the center of the second character frame. In some embodiments of the present invention, the scale neural network is used to process the image to be processed, and the first scale and the second scale can be obtained. The scale neural network is trained with the scale of the person in the image as the supervisory information.

Step 503: Obtain the third distance according to the fourth distance and the first difference, where the third distance is positively correlated with the first difference.

In the embodiment of the present invention, the third distance is the distance between the first character point and the second character frame, that is, at least one first distance includes the third distance.

Since in the image to be processed, the unit length of the place with a small scale corresponds to the length of the real world than the unit length of the place with a large scale corresponds to the length of the real world, the distance between the character point and the character frame should be positively correlated with the scale, That is, the third distance is positively correlated with the first difference. For example, suppose the unit length is 10 pixels. In the image to be processed, 10 pixels in a large-scale area indicate that the length in the real world is 0.5 m, and 10 pixels in a small-scale area indicate that the length in the real world is 1 m.

，第三距離為

，第四距離為

。 Suppose the first difference is

, The third distance is

, The fourth distance is

.

、

、

滿足下式：

（4） 其中， t為正數。例如， t=1。 In some embodiments of the invention,

,

,

Meet the following formula:

(4) Among them, t is a positive number. For example, t =1.

、

滿足下式：

（5） 其中， t為正數、 b為實數。例如， t=1， b=0。 In some embodiments of the invention,

,

Meet the following formula:

(5) Among them, t is a positive number and b is a real number. For example, t =1, b =0.

、

滿足下式：

（6） 其中，， t為正數、 b為實數。例如， t=1， b=0。 In some embodiments of the invention,

,

Meet the following formula:

(6) Among them, t is a positive number and b is a real number. For example, t =1, b =0.

In the embodiment of the present invention, the third distance is determined according to the first difference and the fourth distance, which can improve the accuracy of the third distance.

In some embodiments of the present invention, before performing step 502, the image processing apparatus further performs the following steps.

Step 8: Determine the second character point according to the position of the second character frame.

In the embodiment of the present invention, a character point can be determined according to the position of a character frame. For example, the shape of the character frame 1 is a rectangle. The image processing device can determine the position of any vertex of the character frame 1 according to the position of the character frame 1, and then can use any vertex of the character frame 1 as a character point. For another example, the shape of the character frame 1 is rectangular abcd. The center of the rectangle abcd is point e. The image processing device can determine the coordinates of the point e according to the position of the character frame 1, and then use the point e as the character point. For another example, the shape of the character frame 1 is a circle. The image processing device can determine the position of any point on the circle according to the position of the character frame 1, and then can use any point on the circle as a character point. In the embodiment of the present invention, the character point obtained according to the position of the second character frame is the second character point. In some embodiments, the second character point is one of the following: the geometric center of the second character frame, and the vertex of the second character frame.

Step 9: Determine the midpoint between the first character point and the second character point to obtain a third character point.

Step 10: Obtain the first scale index.

In the embodiment of the present invention, in the image, the scale index (including the above-mentioned first scale index, and the second scale index and the third scale index which will appear below) in an image represents the size of the object located there and the object The mapping relationship between dimensions in the real world.

In some embodiments of the present invention, the scale index in a certain place represents the number of pixels required to represent 1 meter in the real world at that place. For example, suppose that in the image shown in FIG. 4, _{the scale index of the position of the pixel point A 31} is 50, and the scale index of the position of the pixel point A ₁₃ is 20. Then the _{position of pixel A 31} indicates that the number of pixels required for 1 meter in the real world is 50.

In some embodiments of the present invention, the scale index of a certain place represents the ratio between the size of the object located there and the size of the object in the real world. For example, suppose that in the image shown in FIG. 4, the object 1 is located at _{the position where the pixel point A 13} is located, and the object 2 is located at the position where the _{pixel point A 31 is located.} The scale index of the location of the pixel point A ₃₁ is 50, and the scale index of the location of the pixel point A ₁₃ is 20. Then the ratio between the size of object 1 in the image and the size of object 1 in the real world is 20, and the ratio between the size of object 2 in the image and the size of object 2 in the real world is 50.

In some embodiments of the present invention, the scale index of a certain place represents the reciprocal of the ratio between the size of the object located there and the size of the object in the real world. For example, suppose that in the image shown in FIG. 4, the object 1 is located at _{the position where the pixel point A 13} is located, and the object 2 is located at the position where the _{pixel point A 31 is located.} The scale index of the location of the pixel point A ₃₁ is 50, and the scale index of the location of the pixel point A ₁₃ is 20. Then the ratio between the size of object 1 in the real world and the size of object 1 in the image is 20, and the ratio between the size of object 2 in the real world and the size of object 2 in the image is 50.

In some embodiments of the present invention, positions with the same scale have the same scale index. For example, in the image shown in Figure 4, _{the scale of pixel point A 11} , the scale of pixel point A ₁₂ , and the scale of pixel point A ₁₃ are all the same, the scale of pixel point A ₂₁ , the scale of pixel point A ₂₂ , and the pixel The scale of the point A ₂₃ is the same, the scale of the pixel point A ₃₁ , the scale of the pixel point A ₃₂ , and the scale of the pixel point A ₃₃ are all the same. The scale index of pixel point A ₁₁ , the scale index of pixel point A ₁₂ , and the scale index of pixel point A ₁₃ are all the same. The scale index of pixel point A ₂₁ , the scale index of pixel point A ₂₂ , and the scale index of pixel point A ₂₃ are all the same. The same, _{the scale index of pixel point A 31} , the scale index of pixel point A ₃₂ , and the scale index of pixel point A ₃₃ are all the same

In the embodiment of the present invention, the first scale index is the scale index of the first scale position, where the first scale position is the position of the third person point in the image to be processed. Assuming that the first reference object is located at the first scale position, the first scale indicator represents the mapping between the first size and the second size, where the first size is the size of the first reference object in the image to be labeled, and the first The second size is the size of the first reference object in the real world. In an implementation manner for obtaining the first scale index, the image processing device receives the first scale index input by the user through the input element. The above-mentioned input elements include: keyboard, mouse, touch screen, touch pad, audio input device, etc.

In another implementation manner for acquiring the first metric index, the image processing apparatus receives the first metric index sent by a terminal different from the image processing apparatus. Optionally, the terminal different from the image processing device may be any one of the following: a mobile phone, a computer, a tablet computer, a server, and a wearable device. The terminal different from the image processing apparatus and the first terminal may be the same or different.

After acquiring the first scale index, the image processing apparatus specifically executes the following steps in the process of executing step 502.

Step 11: Obtain the above-mentioned first difference according to the above-mentioned first scale index.

，第一尺度指標為

。 In the embodiment of the present invention, the first scale index is positively correlated with the first difference. Suppose the first difference is

, The first scale index is

.

、

滿足下式：

（7） 其中， r為正數。例如， r=1/2。 In some embodiments of the invention,

,

Meet the following formula:

(7) Among them, r is a positive number. For example, r = 1/2.

、

滿足下式：

（8） 其中， r為正數、 a為實數。可選的， r=1/2， a=0。 In some embodiments of the invention,

,

Meet the following formula:

(8) Among them, r is a positive number and a is a real number. Optional, r = 1/2, a =0.

、

滿足下式：

（9） 其中， r為正數、 a為實數。可選的， r=1/2， a=0。 In some embodiments of the invention,

,

Meet the following formula:

(9) Among them, r is a positive number and a is a real number. Optional, r = 1/2, a =0.

Since the first scale index can more accurately reflect the scale of the position of the third person's point, the first difference can be determined according to the first scale index, which can improve the accuracy of the first difference.

Please refer to FIG. 6, which is a schematic flowchart of a possible implementation method of step 10 according to an embodiment of the present invention.

Step 601: Perform object detection processing on the image to be processed to obtain a first object frame and a second object frame.

In the embodiment of the present invention, the length of the detection object of the object detection process in the real world is near a certain value. For example, the average length of a human face is 20 cm, and the detection object of the object detection process may be a human face. For another example, the average height of a person is 1.65 meters, and the detection object of the object detection process may be a human body. For another example, in the waiting room, the height of the sign as shown in FIG. 7A is determined (for example, 2.5 meters), and the detection object of the object detection process may be the sign. Optionally, the object detection processing is face detection processing.

In some embodiments of the present invention, object detection processing on the image to be processed may be implemented by a convolutional neural network. By using the image with annotated information as training data, the convolutional neural network is trained, so that the trained convolutional neural network can complete the object detection processing on the image. The label information of the image in the training data is the position information of the object frame, and the object frame contains the detection object of the object detection process.

In some embodiments of the present invention, the object detection processing can be implemented by a person detection algorithm, where the person detection algorithm can be one of the following: YOLO, DMP, SSD, Faster-RCNN algorithm, etc. The present invention is The person detection algorithm that realizes the object detection processing is not limited.

In the embodiment of the present invention, the detection object contained in the first object frame is different from the detection object contained in the second object frame. For example, the detection object contained in the first object frame is Zhang San's face, and the detection object contained in the second object frame is Li Si's face. For another example, the detection object contained in the first object frame is Zhang San's face, and the detection object contained in the second object frame is a sign.

Step 602: Obtain a first length according to the length of the first object frame in the y-axis direction, and obtain a second length according to the length of the second object frame in the y-axis direction.

In the embodiment of the present invention, the y-axis is the vertical axis of the pixel coordinate system of the image to be processed. The image processing device can obtain the position of the object frame by executing step 601. According to the position of the object frame, the length of the object frame in the y-axis direction can be obtained.

For example, the rectangular frame abcd is the object frame 1, where the coordinates of a are (4, 8), the coordinates of b are (6, 8), the coordinates of c are (6, 12), and the coordinates of d are (4, 12). ). At this time, the length of the object frame 1 in the y-axis direction is 12-8=4.

The image processing device can obtain the length of the first object frame in the y-axis direction according to the position of the first object frame, that is, the first length. The image processing can obtain the length of the second object frame in the y-axis direction according to the position of the second object frame, that is, the second length.

Step 603: Obtain a second scale index according to the first length and the third length of the first object in the real world, and obtain a third scale index according to the second length and the fourth length of the second object in the real world.

In the embodiment of the present invention, the second scale index is a scale index of a second scale position, where the second scale position is a position determined in the image to be annotated according to the position of the first object frame. Assuming that the second reference object is located at the second scale position, the second scale index represents the mapping between the third size and the fourth size, where the third size is the size of the second reference object in the image to be labeled, and The fourth size is the size of the second reference object in the real world. The third scale index is a scale index of a third scale position, where the third scale position is a position determined in the image to be marked according to the position of the second object frame. Assuming that the third reference object is located at the third scale position, the third scale index represents the mapping between the fifth size and the sixth size, where the fifth size is the size of the third reference object in the image to be labeled, and the The sixth size is the size of the third reference object in the real world.

In the embodiment of the present invention, the image processing device can determine an object point according to an object frame. For the specific implementation of this step, please refer to the implementation of determining a character point according to a character frame in step 8, which will not be repeated here.

The image processing device determines the first object point according to the position of the first object frame. The image processing device determines the second object point according to the position of the second object frame.

In some embodiments of the present invention, the first object point is one of the following: the geometric center of the first object frame, and the vertex of the first object frame. The second object point is one of the following: the geometric center of the second object frame, and the vertex of the second object frame.

After determining the position of the first object point and the position of the second object point, the image processing device may use the position of the first object point as the second-scale position and the position of the second object point as the third-scale position.

In the embodiment of the present invention, both the first object and the second object are detection objects of object detection processing. The first object is a detection object included in the first object frame, and the second object is a detection object included in the second object frame. The length of the first object in the real world is the third length, and the length of the second object in the real world is the fourth length. For example, the first object and the second object are both human faces, and both the third length and the fourth length may be 20 cm. For another example, the first object is a human face, the second object is a human body, the third length may be 20 cm, and the fourth length may be 170 cm.

，第二長度為

，第三長度為

，第四長度為

，第二尺度指標為

，第三尺度指標為

。 Suppose the first length is

, The second length is

, The third length is

, The fourth length is

, The second scale index is

, The third scale index is

.

、

、

、

、

、

滿足下式：

（10） 其中， q為正數。例如， q=1。 In some embodiments of the invention,

,

,

,

,

,

Meet the following formula:

(10) Among them, q is a positive number. For example, q =1.

、

、

、

、

、

滿足下式：

（11） 其中， q為正數、 m為實數。例如， q=1， m=0。 In some embodiments of the invention,

,

,

,

,

,

Meet the following formula:

(11) Among them, q is a positive number and m is a real number. For example, q =1, m =0.

、

、

、

、

、

滿足下式：

（12） 其中， q為正數、 m為實數。例如， q=1， m=0。 In some embodiments of the invention,

,

,

,

,

,

Meet the following formula:

(12) Among them, q is a positive number and m is a real number. For example, q =1, m =0.

Step 604: Perform curve fitting processing on the second scale index and the third scale index to obtain the scale index map of the image to be processed.

Since in the image to be processed, the relationship between the scale and the ordinate can be regarded as a linear correlation, and the scale index is used to characterize the scale, the image processing device performs curve fitting processing on the second scale index and the third scale index, The scale index map of the image to be processed can be obtained. The scale index map includes the scale index of the position of any pixel in the image to be processed.

Take the first pixel in the scale index map as an example. Assuming that the pixel value of the first pixel (ie, the first pixel value) is 40, the position of the first pixel in the scale index map is the same as the position of the second pixel in the image to be processed. Then the scale index of the position of the second pixel in the image to be processed (that is, the fourth scale position) is the first pixel value. Assuming that the fourth reference object is located at the fourth scale position, the first pixel value represents the mapping between the seventh size and the eighth size, where the seventh size is the size of the fourth reference object located at the fourth scale position, and the eighth The size is the size of the fourth reference object in the real world.

Step 605: Obtain the first scale index according to the scale index map and the position of the third person point.

As described in step 604, the scale index map includes the scale index of the location of any pixel in the image to be processed. Therefore, according to the scale index map and the position of the third person point in the image to be processed, the scale index of the third person point, that is, the first scale index, can be determined.

In the embodiment of the present invention, the second scale index is obtained according to the first length and the third length, and the third scale index is obtained according to the second length and the fourth length. By performing curve fitting processing on the second scale index and the third scale index, a scale index map is obtained, and then the scale index of the location of any pixel in the image to be processed can be determined according to the scale index map.

In some embodiments of the present invention, the character points (including: the first character point, the second character point, and the third character point) in the embodiment of the present invention may be the head point, and the character frame (including: the first character frame) , The second character frame) can be a human head frame. The pixel area covered by the human head dot and the pixel area included in the human head frame are both the human head area.

Based on the technical solutions provided by the present invention, the embodiments of the present invention also provide some possible application scenarios.

As mentioned above, in public places, crowds are often too crowded due to excessive flow of people, and then some public accidents occur. How to locate the crowd in public places is of very important significance.

In related technologies, in order to enhance safety in work, life, or social environments, surveillance camera equipment will be installed in various public places to perform security protection based on video stream information. By using the technical solution provided by the embodiment of the present invention to process the video stream collected by the surveillance camera equipment, the location of each person in the image can be determined, thereby effectively preventing the occurrence of public accidents.

For example, a server of a video stream processing center of a surveillance camera device can execute the technical solution provided in the embodiment of the present invention, and the server can be connected to at least one surveillance camera. After the server obtains the video stream sent by the surveillance camera, it can perform person detection processing on each frame of image in the video stream, and obtain the position of the person point in each frame of image and the position of each frame of image. The position of the character frame. FIG. 7B is a schematic diagram of an application scenario provided by an embodiment of the present invention. As shown in FIG. 7B, the image to be processed 71 is each of the above-mentioned video frames, and the image to be processed is input to the image processing device 1 Later, the technical solution provided by the embodiment of the present invention can be used to process the position of the character point in each frame of image and the position of the character frame in each frame of image to determine the position of the character in each frame of image. Location. The user can then view the position of each character in the image through the server, so as to further determine the position of the character in the image in the real world.

In some embodiments, after obtaining the position of the person in each frame of the image in the video stream, the server can determine the number of persons in each frame of the image according to the position of the person in each frame of the image. In the case that the number of people in the image is greater than or equal to the number threshold, the server can send instructions to related devices to prompt or alarm.

For example, the server may send an instruction to the camera that collects the image, and the instruction is used to instruct the camera that collects the image to give an alarm. For another example, the server may send an instruction to the terminal of the management personnel in the area where the camera collecting the image is located, and the instruction is used to prompt the terminal to output prompt information that the number of people exceeds the threshold of the number of people.

Those skilled in the art can understand that in the above-mentioned methods of the specific implementation, the writing order of the steps does not mean a strict execution order but constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possibility. The inner logic is determined.

The above describes the method of the embodiment of the present invention in detail, and the device of the embodiment of the present invention is provided below.

Please refer to FIG. 8. FIG. 8 is a schematic structural diagram of an image processing device according to an embodiment of the present invention. The image processing device 1 includes: an acquisition unit 11, a first processing unit 12, and a second processing unit 13. in: The acquiring unit 11 is configured to acquire the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed; the first position and the second position Both are used to characterize the position of the first person in the image to be processed; The first processing unit 12 is configured to obtain a third position of the first person in the image to be processed according to the first position and the second position.

In some embodiments of the present invention, the acquiring unit 11 is further configured to acquire a first confidence level of the first position and a second confidence level of the second position; the first confidence level and the first confidence level are The scale of a position is negatively correlated; the second confidence level is positively correlated with the scale of the second position; The first processing unit 12 is configured to: Taking the position with the highest confidence level among the first position and the second position as the fourth position; According to the fourth position, the third position is obtained.

In some embodiments of the present invention, the acquiring unit 11 is further configured to perform character positioning processing on the image to be processed to obtain the first position and the position of the at least one character frame; Determine the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance; A person frame corresponding to the second distance is used as the first person frame; the second distance is a distance that does not exceed a distance threshold in the at least one first distance.

In some embodiments of the present invention, the acquiring unit 11 is configured to: Performing character positioning processing on the image to be processed to obtain the position of at least one character point; The position with the highest confidence among the positions of the at least one character point is taken as the first position.

In some embodiments of the present invention, the at least one character frame includes a second character frame; the at least one first distance includes a third distance between the first character point and the second character frame; The obtaining unit 11 is configured to: Obtaining the fourth distance between the first character point and the second character frame according to the first position and the position of the second character frame; Determine the difference between the first scale and the second scale to obtain the first difference; the first scale is the scale of the first person point in the image to be processed; the second scale is the first 2. The scale of the character frame in the image to be processed; According to the fourth distance and the first difference, the third distance is obtained; the third distance is positively correlated with the first difference.

In some embodiments of the present invention, the device further includes a second processing unit 13, and the second processing unit 13 is configured to: Determine the second character point according to the position of the second character frame; Determining a midpoint between the first character point and the second character point to obtain a third character point; The acquiring unit 11 is further configured to acquire a first scale index; the first scale index represents a mapping between a first size and a second size; the first size is a first reference object located at a first scale position The size of; the second size is the size of the first reference object in the real world; the first scale position is the position of the third person point in the image to be processed; According to the first scale index, the first difference is obtained.

In some embodiments of the present invention, the acquiring unit 11 is configured to: Performing object detection processing on the image to be processed to obtain a first object frame and a second object frame; The first length is obtained according to the length of the first object frame in the y-axis direction, and the second length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the length of the image to be processed The vertical axis of the pixel coordinate system; Obtaining a second scale index according to the first length and the third length of the first object in the real world, and obtaining a third scale index according to the second length and the fourth length of the second object in the real world; The first object is the detection object contained in the first object frame; the second object is the detection object contained in the second object frame; the second scale index represents the difference between the third size and the fourth size The third size is the size of the second reference object located at the second scale position; the fourth size is the size of the second reference object in the real world; the second scale position is based on The position of the first object frame is determined in the image to be processed; the third scale index represents the mapping between the fifth size and the sixth size; the fifth size is the position at the third scale The size of the third reference object; the sixth size is the size of the third reference object in the real world; the third scale position is based on the position of the second object frame in the image to be processed Determined location Perform curve fitting processing on the second scale index and the third scale index to obtain the scale index diagram of the image to be processed; the first pixel value in the scale index diagram represents the seventh size and the eighth size Mapping between sizes; the seventh size is the size of the fourth reference object at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; the first pixel value Is the pixel value of the first pixel; the fourth scale position is the position of the second pixel in the image to be processed; the position of the first pixel in the scale index map is the same as that of the first pixel The positions of the two pixels in the image to be processed are the same; According to the scale index map and the position of the third person point, the first scale index is obtained.

In some embodiments of the present invention, the second scale position is the position of the first object point in the image to be processed; the third scale position is the position of the second object point in the image to be processed Location; The first object point is one of the following: the geometric center of the first object frame, the vertex of the first object frame; the second object point is one of the following: The geometric center and the vertex of the second object frame.

In some embodiments of the present invention, the second character point is one of the following: a geometric center of the second character frame, and a vertex of the second character frame.

In some embodiments of the present invention, the pixel area covered by the first character dot and the pixel area included in the first character frame are both human head areas.

In some embodiments of the present invention, the shape of the first character frame is a rectangle, a diamond, a circle, an ellipse, or a polygon.

In the embodiment of the present invention, when the person is far away in the image to be processed, the position of the person in the image to be processed is determined according to the position of the person point; in the case where the person is close to the image to be processed Next, determine the position of the character in the image to be processed according to the position of the character frame. In this way, the accuracy of the position of the character in the image to be processed is improved.

In some embodiments, the functions or modules contained in the device provided by the embodiments of the present invention can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For brevity, I won't repeat it here.

An embodiment of the present invention also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any one of the above-mentioned image processing methods is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

The embodiment of the present invention also provides a computer program, including computer-readable code, when the computer-readable code is run in an electronic device, the processor in the electronic device is executed to implement any of the above-mentioned image processing methods .

An embodiment of the present invention also provides an electronic device. FIG. 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present invention. The electronic device 2 may include a processor 21 and a memory 22. The memory 22 is used to store computer program codes. The computer program codes include computer instructions. When the processor 21 executes the computer instructions, The electronic device 2 executes any one of the above-mentioned image processing methods.

The processor 21 may be one or more graphics processing units (graphics processing unit, GPU). In the case where the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Optionally, the processor 21 may be a processor group composed of multiple GPUs, and the multiple processors are coupled to each other through one or more buses. In some embodiments, the processor may also be other types of processors, etc., which is not limited in the embodiment of the present invention. The memory 22 can be used to store computer program instructions and various computer program codes including program codes used to execute the solution of the present invention. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (read-only memory, ROM), erasable programmable read-only memory (erasable programmable read-only memory). only memory, EPROM), or portable read-only memory (compact disc read-only memory, CD-ROM), which is used for related instructions and data.

In some optional implementation manners, the aforementioned electronic device 2 may further include an input device 23 and/or an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled through a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiment of the present invention. It should be understood that in the various embodiments of the present invention, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, such as connection through various interfaces, transmission lines, bus bars, and the like. The input device 23 is used for inputting data and/or signals, and the output device 24 is used for outputting data and/or signals. The input device 23 and the output device 24 may be independent devices or a whole device.

It can be understood that, in the embodiment of the present invention, the memory 22 can be used not only to store related commands, but also to store related data. For example, the memory 22 can be used to store the first position and the second position obtained through the input device 23, or The memory 22 can also be used to store the third position obtained by the processor 21, etc. The embodiment of the present invention does not limit the specific data stored in the memory.

It can be understood that FIG. 9 only shows a simplified design of an electronic device. In practical applications, electronic equipment may also contain other necessary components, including but not limited to any number of input/output devices, processors, memory, etc., and all electronic equipment that can implement the embodiments of the present invention are included in the present invention. Within the scope of protection.

A person of ordinary skill in the art may be aware that the units and algorithm steps described in the examples in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered as going beyond the scope of the present invention.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. Those skilled in the art can also clearly understand that the description of each embodiment of the present invention has its own focus. For the convenience and brevity of the description, the same or similar parts may not be repeated in different embodiments. Therefore, in a certain embodiment For parts that are not described or described in detail, reference may be made to the records of other embodiments.

In the several embodiments provided by the present invention, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or elements may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. . Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be sent from a website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) Another website, computer, server or data center for transmission. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a digital versatile disc (DVD)), or a semiconductor medium (for example, a solid state drive).

A person of ordinary skill in the art can understand that all or part of the process in the above-mentioned embodiment method can be realized. The process can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium, and the program is executing At this time, it may include the process of each method embodiment described above. The aforementioned storage media include: ROM or RAM, magnetic disks or optical disks and other media that can store program codes.

301, 302: steps 501~503: steps 601~605: steps 1: Image processing device 11: Get unit 12: The first processing unit 13: The second processing unit 2: electronic equipment 21: processor 22: Memory 23: Input device 24: output device 71: Image to be processed

In order to more clearly describe the technical solutions in the embodiments of the present invention or the background art, the following will describe the drawings that need to be used in the embodiments of the present invention or the background art. The drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments in accordance with the present invention and are used together with the specification to illustrate the technical solution of the present invention. FIG. 1 is a schematic diagram of a crowd image provided by an embodiment of the present invention; 2 is a schematic diagram of a pixel coordinate system provided by an embodiment of the present invention; FIG. 3 is a schematic flowchart of an image processing method provided by an embodiment of the present invention; FIG. 4 is a schematic diagram of an image provided by an embodiment of the present invention; FIG. 5 is a schematic flowchart of another image processing method provided by an embodiment of the present invention; 6 is a schematic flowchart of another image processing method provided by an embodiment of the present invention; FIG. 7A is a schematic diagram of a sign provided by an embodiment of the present invention; FIG. 7B is a schematic diagram of an application scenario provided by an embodiment of the present invention; FIG. 8 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present invention; FIG. 9 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention.

301, 302: steps

Claims (13)

An image processing method, the method comprising: obtaining a first position of a first character point in an image to be processed and a second position of a first character frame in the image to be processed; the first position and The second position is used to characterize the position of the first person in the image to be processed; according to the first position and the second position, the first person’s position in the image to be processed is obtained. Three positions; including: when the first person is in the vicinity of the image to be processed, a third position is obtained according to the second position; and when the first person is in the image to be processed In the case of a remote location, the third position is determined according to the first position. The method according to claim 1, further comprising: obtaining a first confidence degree of the first position and a second confidence degree of the second position; the first confidence degree is proportional to the scale of the first position Negative correlation; the second confidence level is positively correlated with the scale of the second position; the first position and the second position are used to obtain the position of the first person in the image to be processed The third position includes: taking the position with the highest confidence level among the first position and the second position as the fourth position; and obtaining the third position according to the fourth position. The method according to claim 1 or 2, wherein the acquiring the first position of the first character point in the image to be processed and the second position of the first character frame in the image to be processed includes: Perform character positioning processing on the image to be processed to obtain the first position Position and the position of the at least one character frame; according to the first position and the position of the at least one character frame, determine the distance between the at least one character frame and the first character point to obtain at least one A distance; the person frame corresponding to the second distance is used as the first person frame, and the second position of the first person frame in the image to be processed is determined; the second distance is the at least one The first distance does not exceed the distance threshold. The method according to claim 3, wherein the performing character positioning processing on the to-be-processed image to obtain the first position includes: performing character positioning processing on the to-be-processed image to obtain at least one person The position of the point; the position with the highest confidence among the positions of the at least one character point is taken as the first position. The method according to claim 3, wherein the at least one character frame includes a second character frame; the at least one first distance includes a third distance between the first character point and the second character frame The determining the distance between the at least one character frame and the first character point according to the first position and the position of the at least one character frame to obtain at least one first distance includes: according to the The first position and the position of the second character frame are obtained to obtain the fourth distance between the first character point and the second character frame; the difference between the first scale and the second scale is determined to obtain the first difference; The first scale is the scale of the first person point in the image to be processed; the second scale is the scale of the second person frame in the image to be processed; according to the first Four distances and the first difference to obtain the third distance; the third distance is positively correlated with the first difference. The method according to claim 5, further comprising: determining a second character point according to the position of the second character frame; determining a midpoint between the first character point and the second character point to obtain a third character Point; obtain a first scale index; the first scale index represents the mapping between the first size and the second size; the first size is the size of the first reference object located at the first scale position; the second The size is the size of the first reference object in the real world; the first scale position is the position of the third person point in the image to be processed; the determining one of the first scale and the second scale Obtaining the first difference includes: obtaining the first difference according to the first scale index. The method according to claim 6, wherein the obtaining the first scale index includes: performing object detection processing on the image to be processed to obtain a first object frame and a second object frame; The length of the frame in the y-axis direction obtains the first length, and the second length is obtained according to the length of the second object frame in the y-axis direction; the y-axis is the vertical axis of the pixel coordinate system of the image to be processed ； Obtaining a second scale index according to the first length and the third length of the first object in the real world, and obtaining a third scale index according to the second length and the fourth length of the second object in the real world; The first object is the detection object contained in the first object frame; the second object is the detection object contained in the second object frame; the second scale index represents the difference between the third size and the fourth size The third size is the size of the second reference object located at the second scale position; the fourth size is the size of the second reference object in the real world; the second scale position is based on The position of the first object frame is determined in the image to be processed; the third scale index represents the mapping between the fifth size and the sixth size; the fifth size is the position at the third scale The size of the third reference object; the sixth size is the size of the third reference object in the real world; the third scale position is based on the position of the second object frame in the image to be processed The determined position; curve fitting processing is performed on the second scale index and the third scale index to obtain the scale index map of the image to be processed; the first pixel value in the scale index map represents the seventh The mapping between the size and the eighth size; the seventh size is the size of the fourth reference object located at the fourth scale position; the eighth size is the size of the fourth reference object in the real world; The first pixel value is the pixel value of the first pixel; the fourth scale position is the position of the second pixel in the image to be processed; the position of the first pixel in the scale index map The position of the second pixel point in the image to be processed is the same; according to the scale index map and the position of the third person point, all the points are obtained The first scale index. The method according to claim 7, wherein the second scale position is the position of the first object point in the image to be processed; the third scale position is the position of the second object point in the image to be processed The position in the image; the first object point is one of the following: the geometric center of the first object frame, the vertex of the first object frame; the second object point is one of the following: the The geometric center of the second object frame and the vertices of the second object frame. The method according to claim 6, wherein the second character point is one of the following: a geometric center of the second character frame, and a vertex of the second character frame. The method according to claim 1 or 2, wherein the pixel area covered by the first character dot and the pixel area included in the first character frame are both human head areas. The method according to claim 1 or 2, wherein the shape of the first character frame is a rectangle, a diamond, a circle, an ellipse, or a polygon. An electronic device, comprising: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, when the processor executes the computer instructions, the electronic device Perform the method described in any one of Claims 1 to 11. A computer-readable storage medium in which a computer program is stored, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes a request item The method described in any one of 1 to 11.

Images