TW202301276A - Depth detection method and device, electronic equipment and storage medium - Google Patents

Abstract

The invention relates to a depth detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of to-be-detected frames which comprise image frames obtained by carrying out the image collection of a target object from at least two collection visual angles; carrying out key point detection on a target area in the target object according to the to-be-detected frames, determining multiple key point detection results corresponding to the multiple to-be-detected frames, wherein the target area comprises a head area and/or a shoulder area; and determining depth information of the target object according to the plurality of key point detection results.

Description

In-depth detection method and device, electronic equipment, storage medium and program product

The present invention claims the priority of the Chinese patent application filed with the China Patent Office on June 28, 2021, with the application number 202110721270.1, and the title of the invention is "deep detection method and device, electronic equipment and storage medium", the entire contents of which are incorporated by reference In the present invention.

The invention relates to the field of computer technology, in particular to a depth detection method and device, electronic equipment, storage media and program products.

The depth information can reflect the distance of the human body in the image relative to the image acquisition device, and based on the depth information, the human body object in the image can be spatially positioned. The binocular camera is a relatively common and widely used image acquisition device. Based on at least two images collected by the binocular camera, the depth information of the human body in the image can be determined through the matching between the images. However, the difference between the images The matching calculation is complex and the accuracy is easily affected. How to conveniently and accurately determine the depth information of the human body in the image has become an urgent problem to be solved.

The invention proposes a technical scheme of depth detection.

According to an aspect of the present invention, a depth detection method is provided, comprising:

Acquiring multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained by collecting images of the target object from at least two acquisition angles of view; Key point detection of the region, determining a plurality of key point detection results corresponding to the multiple frames to be detected, wherein the target area includes a head region and/or a shoulder region; according to the plurality of key point detection results , to determine the depth information of the target object.

In a possible implementation manner, the determining the depth information of the target object according to the multiple key point detection results includes: acquiring at least two preset device parameters respectively corresponding to at least two acquisition devices, the At least two acquisition devices are used to acquire images of the target object from at least two acquisition angles of view; according to the at least two preset device parameters and the plurality of key point detection results, determine the Depth information of the target object.

In a possible implementation manner, the depth information includes a depth distance, and the depth distance includes a distance between the target object and the optical center of the acquisition device; the at least two preset device parameters and the The plurality of key point detection results, determining the depth information of the target object in the frame to be detected includes: according to the preset external parameters in the at least two preset device parameters and the plurality of key point detection results Coordinates in at least two forms obtain the depth distance; wherein, the preset external parameters include relative parameters formed between the at least two acquisition devices.

In a possible implementation manner, the depth information includes an offset angle, and the offset angle includes a spatial angle of the target object relative to the optical axis of the acquisition device; the at least two preset Determining the depth information of the target object in the frame to be detected based on the device parameters and the multiple key point detection results includes: according to the preset internal parameters in the at least two preset device parameters and the multiple The coordinates of the key point detection results in at least two forms obtain the offset angle; wherein, the preset internal parameters include device parameters corresponding to the at least two devices respectively.

In a possible implementation manner, the performing the key point detection of the target area in the target object according to the frame to be detected includes: according to the position information of the target object in the reference frame, The key point detection is performed on the target area of the target object, and the key point detection result corresponding to the frame to be detected is obtained, wherein the reference frame is the target film to which the frame to be detected belongs, and is located in the frame to be detected Detects the movie frame before the frame.

In a possible implementation manner, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the key point detection corresponding to the frame to be detected is obtained. The corresponding key point detection result includes: clipping the frame to be detected according to the first position of the target object in the reference frame to obtain a clipping result; Key point detection, obtaining a key point detection result corresponding to the frame to be detected.

In a possible implementation manner, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the key point detection corresponding to the frame to be detected is obtained. The corresponding key point detection result includes: obtaining a second position of the target area of the target object in the reference frame; clipping the frame to be detected according to the second position to obtain a clipping result; The key point detection is performed on the target area of the target object in the clipping result, and the key point detection result corresponding to the frame to be detected is obtained.

In a possible implementation manner, the obtaining the second position of the target area of the target object in the reference frame includes: using a first neural network to identify the target area in the reference frame to obtain The second position output by the first neural network; and/or, according to the key point detection result corresponding to the reference frame, the second position of the target area in the reference frame is obtained.

In a possible implementation manner, the method further includes: determining a position of the target object in a three-dimensional space according to depth information of the target object.

According to an aspect of the present invention, a depth detection device is provided, comprising:

The acquisition module is used to acquire multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained by collecting images of the target object from at least two acquisition angles of view; the key point detection module is used to Perform key point detection of the target area in the target object according to the frame to be detected, and determine a plurality of key point detection results corresponding to the multiple frames to be detected, wherein the target area includes a head area and/or or the shoulder area; a depth detection module, configured to determine the depth information of the target object according to the detection results of the plurality of key points.

In a possible implementation manner, the depth detection module is configured to: acquire at least two preset device parameters respectively corresponding to at least two acquisition devices, and the at least two acquisition devices are used to measure performing image acquisition on the target object; determining depth information of the target object in the frame to be detected according to the at least two preset device parameters and the plurality of key point detection results.

In a possible implementation manner, the depth information includes a depth distance, and the depth distance includes a distance between the target object and the optical center of the acquisition device; the depth detection module is further configured to: according to at least The preset external parameters among the two preset device parameters and the coordinates of the plurality of key point detection results in at least two forms obtain the depth distance; wherein, the preset external parameters include the at least two Collect relative parameters formed between devices.

In a possible implementation manner, the depth information includes an offset angle, and the offset angle includes a spatial angle of the target object relative to the optical axis of the acquisition device; the depth detection module is further configured to: According to the preset internal parameters in the at least two preset device parameters and the coordinates of the plurality of key point detection results in at least two forms, the offset angle is obtained; wherein the preset internal parameters include Device parameters respectively corresponding to the at least two devices.

In a possible implementation manner, the key point detection module is configured to: perform key point detection on the target area of the target object in the frame to be detected according to the position information of the target object in the reference frame , to obtain a key point detection result corresponding to the frame to be detected, wherein the reference frame is a movie frame before the frame to be detected in the target movie to which the frame to be detected belongs.

In a possible implementation manner, the key point detection module is further configured to: clip the frame to be detected according to the first position of the target object in the reference frame to obtain a clipping result; The key point detection is performed on the target area of the target object in the clipping result, and the key point detection result corresponding to the frame to be detected is obtained.

In a possible implementation manner, the key point detection module is further configured to: acquire a second position of the target area of the target object in the reference frame; Clipping the frame to obtain a clipping result; performing key point detection on the target area of the target object in the clipping result to obtain a key point detection result corresponding to the frame to be detected.

In a possible implementation manner, the key point detection module is further configured to: use the first neural network to identify the target area in the reference frame, and obtain the second position output by the first neural network and/or, according to the key point detection result corresponding to the reference frame, obtain the second position of the target area in the reference frame.

In a possible implementation manner, the device is further configured to: determine the position of the target object in a three-dimensional space according to the depth information of the target object.

According to an aspect of the present invention, there is provided an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

According to one aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor.

According to one aspect of the present invention, a computer program product is provided, including computer readable codes, and when the computer readable codes are run in an electronic device, a processor in the electronic device executes the above method.

In the embodiment of the present invention, by acquiring multiple frames to be detected from at least two acquisition angles of view, and performing key point detection of the target area according to the frames to be detected, multiple key point detections corresponding to multiple frames to be detected are determined As a result, and based on the detection results of multiple key points, the depth information of the target object is determined. Through the embodiment of the present invention, the parallax formed by the multiple frames to be detected collected under at least two acquisition angles can be used to utilize the multiple frames to be detected. The detection results of multiple key points corresponding to the target area in the frame realize the calculation based on parallax to obtain depth information, effectively reduce the amount of data processed in the process of calculation based on parallax, and improve the efficiency and accuracy of depth detection.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. Other features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, may mean including the composition of A, B, and C Any one or more elements selected in the collection.

In addition, in order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. It will be understood by those skilled in the art that the present invention may be practiced without certain of the specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail in order to highlight the gist of the present invention.

Fig. 1 shows a flowchart of a depth detection method according to an embodiment of the present invention. The method can be performed by a depth detection device, and the depth detection device can be an electronic device such as a terminal device or a server, and the terminal device can be user equipment (User Equipment, UE), mobile device, user terminal, terminal, mobile phone, indoor wireless phone , Personal Digital Assistant (PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. In some possible implementation manners, the method may be implemented by the processor calling computer-readable instructions stored in the memory. Alternatively, the method may be performed by a server. As shown in Figure 1, the method may include:

Step S11 , acquiring multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained by collecting images of the target object from at least two collection angles of view.

Wherein, the frame to be detected may be any image frame that requires depth detection, for example, it may be an image frame extracted from a filmed film, or an image frame obtained by shooting an image. The number of multiple frames to be detected is not limited in this embodiment of the present invention, and may include two or more frames.

The acquisition angle of view can be the angle of image acquisition of the target object. Different frames to be detected can be collected by image acquisition devices set at different acquisition angles of view, or can be collected by the same image device under different acquisition angles of view. collection.

The frame to be detected includes the target object to be subjected to depth detection. The type of the target object is not limited in the embodiment of the present invention, and may include various human objects, animal objects or some mechanical objects, such as robots. Subsequent disclosed embodiments are described by taking the target object as a person object as an example. Implementations in which the target object is other types can be flexibly expanded by referring to the subsequent disclosed embodiments, and will not be elaborated one by one.

The number of target objects contained in the frame to be detected is also not limited in this embodiment of the present invention, and may contain one or more target objects, which can be flexibly determined according to actual conditions.

The method of obtaining multiple frames to be detected is not limited in the embodiment of the present invention. In a possible implementation, frame extraction may be performed from one or more films to obtain multiple frames to be detected, wherein the frame extraction It may include one or more methods such as frame-by-frame extraction, frame sampling at a certain interval, or random frame sampling. In a possible implementation, it is also possible to collect images from multiple angles of the target object to obtain multiple frames to be detected; Multiple frames to be detected, etc.

Step S12 , performing key point detection of the target area in the target object according to the frame to be detected, and determining multiple key point detection results corresponding to multiple frames to be detected.

Wherein, the key point detection result may include the position of the detected key point in the frame to be detected. Among them, the number and types of detected key points can be flexibly determined according to the actual situation. In some possible implementations, the number of detected key points can include 2 to 150, etc. In one example, the detected key points can be Contains 14 body key points of the human body (such as head key points, shoulder key points, neck key points, elbow key points, wrist key points, crotch key points, leg key points, foot key points, etc.) , or include 59 contour key points on the contour of the human body (such as some key points on the periphery of the head or the periphery of the shoulders), etc. In a possible implementation manner, in order to reduce the amount of calculation, the detected key points may also only include three key points including the key point of the head, the key point of the left shoulder and the key point of the right shoulder.

Multiple key point detection results can correspond to multiple frames to be detected respectively. For example, if key point detection is performed on multiple frames to be detected, each frame to be detected can correspond to a key point detection result, so that it can be obtained Multiple keypoint detection results.

The target area may include a head area and/or a shoulder area, and the head area of the target object may be the area where the head of the target object is located, such as the area formed between the key points of the head and the key points of the neck; the shoulder area Then it may be the area where the shoulder and neck of the target object are located, such as the area formed between the key points of the neck and the key points of the shoulder.

Fig. 2 shows a schematic diagram of the target area according to an embodiment of the present invention. As shown in Fig. 2 , in a possible implementation, when the target area includes the head area and the shoulder area, the key points of the head can be , the left shoulder key point and the right shoulder key point are connected as the head and shoulders frame as the target area. In one example, the head-shoulders frame can be a rectangle as shown in Figure 2. It can be seen from Figure 2 that the head-shoulders frame can be connected to the head key point at the head vertex of the target object and the left shoulder key point at the left shoulder joint. and the right shoulder key point at the right shoulder joint. In an example, the head-and-shoulders frame may also be in other shapes, such as polygons, circles, or other irregular shapes.

The way of key point detection can be flexibly determined according to the actual situation. In one possible implementation, the frame to be detected can be input into any neural network with key point detection function to realize key point detection; in some possible implementations, It is also possible to perform key point identification on the frame to be detected through a relevant key point identification algorithm to obtain a key point detection result; Position, perform key point detection on a part of the image area in the frame to be detected to obtain key point detection results, etc. For some possible specific implementations of step S12, reference may be made to the following disclosed embodiments in detail, which will not be expanded here.

Step S13 , according to the multiple key point detection results, determine the depth information of the target object in the frame to be detected.

The content of the information included in the depth information can be flexibly determined according to the actual situation, and any information that can reflect the depth of the target object in the three-dimensional space can be used as the implementation of the depth information. In a possible implementation manner, the depth information may include a depth distance and/or an offset angle.

The depth distance can be the distance between the target object and the collection device, and the collection device can be any device that collects images of the target object. In some possible implementations, the collection device can be a static image collection device, such as A camera, etc.; in some possible implementation manners, the acquisition device may also be a device for acquiring dynamic images, such as a video camera or a photographic lens.

As described in the above disclosed embodiments, different frames to be detected can be collected by image acquisition devices set at different acquisition angles of view, or can be acquired by the same image device at different acquisition angles of view. Therefore, the acquisition device The number of can be one or more. In a possible implementation manner, the depth detection method proposed in the embodiment of the present invention can be implemented based on at least two acquisition devices. In this case, the at least two acquisition devices can detect the target object from at least two acquisition perspectives. Image acquisition to obtain multiple frames to be detected.

In the case where the collection device includes at least two collection devices, the types of different collection devices may be the same or different, which can be flexibly selected according to the actual situation, and there is no limitation in this embodiment of the present invention.

The depth distance can be the distance between the target object and the collection device, the distance can be the distance between the target object and the collection device as a whole, or the distance between the target object and a certain equipment part of the collection device, in some possible In an implementation manner of , the distance between the target object and the optical center of the acquisition device may be used as the depth distance.

The offset angle may be an offset angle of the target object relative to the collection device, and in a possible implementation manner, the offset angle may be a spatial angle of the target object relative to the optical axis of the collection device.

Since multiple key point detection results can correspond to multiple frames to be detected, and multiple frames to be detected can be obtained by collecting images of the target object from at least two acquisition angles of view, therefore, based on multiple key point detection results , the disparity formed between multiple frames to be detected can be determined, and then the depth information calculation based on the disparity can be realized to obtain the depth information of the target object. Among them, the parallax-based calculation method based on the key point detection results can be flexibly determined according to the actual situation. Any method for realizing depth ranging based on parallax can be used in the implementation process of step S13. For details, see the following disclosed embodiments. , do not expand here.

In the embodiment of the present invention, by acquiring multiple frames to be detected from at least two acquisition angles of view, and performing key point detection of the target area according to the frames to be detected, multiple key point detections corresponding to multiple frames to be detected are determined As a result, and based on the detection results of multiple key points, the depth information of the target object is determined. Through the embodiment of the present invention, the parallax formed by the multiple frames to be detected collected under at least two acquisition angles can be used to utilize the multiple frames to be detected. The detection results of multiple key points corresponding to the target area in the frame realize the calculation based on parallax to obtain depth information, effectively reduce the amount of data processed in the process of calculation based on parallax, and improve the efficiency and accuracy of depth detection.

In a possible implementation manner, step S12 may include:

According to the position information of the target object in the reference frame, key point detection is performed on the target area of the target object in the frame to be detected, and a key point detection result corresponding to the frame to be detected is obtained.

Wherein, the reference frame may be a film frame located before the frame to be detected in the target film, and the target film may be a film containing the frame to be detected. In some possible implementation manners, different frames to be detected may respectively belong to different target films. In this case, different frames to be detected may correspond to different reference frames.

In some possible implementations, the reference frame can be the previous frame of the frame to be detected in the target movie, and in some possible implementations, the reference frame can also be the frame in the target movie that is located before the frame to be detected and is the same as the frame to be detected The distance between the movie frames does not exceed the preset distance, the number of preset distances can be flexibly determined according to the actual situation, and can be one or more frames apart, which is not limited in the embodiment of the present invention.

Since the reference frame is located before the frame to be detected, and the distance from the frame to be detected does not exceed the preset distance, the position of the target object in the reference frame may be relatively close to the position of the target object in the frame to be detected. In this case, According to the position information of the target object in the reference frame, the position information of the target object in the frame to be detected can be roughly determined. In this case, the target area of the target object in the frame to be detected can be more targeted. Detection, and the amount of data detected will be smaller, so that more accurate key point detection results can be obtained, and the efficiency of key point detection can also be improved.

In some possible implementations, according to the position information of the target object in the reference frame, the key point detection method of the target area of the target object in the frame to be detected can be flexibly determined according to the actual situation, for example, according to the position information of the target object in the reference frame The position information in the to-be-detected frame is cropped and then the key point detection is performed, or according to the position information of the target object in the reference frame, the key point detection is directly performed on the image area corresponding to the position in the to-be-detected frame, etc., various possible implementations The methods can be referred to the following disclosed embodiments in detail, and will not be expanded here.

Through the embodiment of the present invention, according to the position information of the target object in the reference frame, more targeted key point detection can be realized for the target area in the frame to be detected, and the efficiency and accuracy of key point detection can be improved, thereby improving the depth detection method. efficiency and precision.

In a possible implementation manner, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the key point detection result corresponding to the frame to be detected is obtained, including:

clipping the frame to be detected according to the first position of the target object in the reference frame to obtain a clipping result;

The key point detection is performed on the target area of the target object in the clipping result, and the key point detection result corresponding to the frame to be detected is obtained.

Wherein, the first position may be the overall position coordinates of the target object in the reference frame. For example, if the target object is a person object, the first position may be the position coordinates of the target object's body frame in the reference frame.

The manner of clipping the frame to be detected according to the first position is also not limited in this embodiment of the present invention, and is not limited to the following disclosed embodiments. In a possible implementation, the first coordinates of the human body frame in the reference frame can be determined according to the first position, and combined with the position coordinate correspondence between the reference frame and the frame to be detected, it can be determined that the human body frame of the target object is in the frame to be detected The second coordinate in the frame, based on the second coordinate, the frame to be detected is clipped to obtain a clipping result.

In some possible implementations, the first coordinates of the body frame in the reference frame and the border length of the body frame can also be determined according to the first position, and combined with the position coordinate correspondence between the reference frame and the frame to be detected, determine The second coordinates of the human body frame of the target object in the frame to be detected, and the frame to be detected is cropped based on the second coordinates and the frame length to obtain a clipping result, wherein, the clipping based on the second coordinates and the frame length can be based on the first The two coordinates determine the position of the clipping endpoint, and the frame length determines the length of the clipping result. In one example, the length of the clipping result can be consistent with the frame length. In one example, the length of the clipping result can also be proportional to the frame length. For example, it is N times the length of the frame, and N can be any value not less than 1.

The manner of performing key point detection on the target object in the clipping result can be flexibly determined according to the actual situation. For details, please refer to the following disclosed embodiments, which will not be expanded here.

Through the embodiment of the present invention, the target object in the frame to be detected can be preliminarily positioned according to the first position of the target object in the reference frame, and the clipping result is obtained, and the key point detection of the target area is performed based on the clipping result, on the one hand, it can reduce The amount of detected data improves the detection efficiency. On the other hand, since the target object accounts for a large proportion in the cropped result after cropping, the accuracy of key point detection can be improved.

In a possible implementation manner, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the key point detection result corresponding to the frame to be detected is obtained, including:

Acquiring a second position of the target area of the target object in the reference frame;

Cutting the frame to be detected according to the second position to obtain a cutting result;

Perform key point detection on the target object in the clipping result to obtain the key point detection result.

Wherein, the second position may be the position coordinates of the target area of the target object in the reference frame. As described in the above disclosed embodiments, the target area may include the head area and/or the shoulder area, so in a possible implementation manner In , the second position may be the position coordinates of the head and shoulders frame of the target object in the reference frame.

How to determine the second position of the target area in the reference frame, the implementation form can be flexibly determined according to the actual situation, for example, it can be realized by performing head and shoulder frame and/or key point recognition on the reference frame, see the following publications for details Embodiment, do not expand here.

For the manner of clipping the frame to be detected according to the second position, reference may be made to the manner of clipping the frame to be detected according to the first position, which will not be repeated here.

The key point detection method for the target object in the clipping result can be the same as the key point detection method based on the clipping result obtained at the first position, or it can be different. Do unfold.

Through the embodiment of the present invention, the key point detection result can be obtained according to the second position of the target area of the target object in the reference frame. In this way, the target area can be more targeted, thereby further reducing the amount of data processing. Therefore, the accuracy and efficiency of depth detection are further improved.

In a possible implementation manner, obtaining the second position of the target area of the target object in the reference frame may include:

Identifying the target area in the reference frame through the first neural network to obtain the second position output by the first neural network; and/or,

According to the key point detection result corresponding to the reference frame, the second position of the target area in the reference frame is obtained.

Wherein, the first neural network may be any network used to determine the second position, and its implementation form is not limited in this embodiment of the present invention. In some possible implementations, the first neural network may be an object area detection network for identifying the second location of the object area directly from a reference frame, and in one example, the object area detection network may be more Faster region-based convolutional neural network (Faster Regions with Convolutional Neural Networks, Faster RCNN); in some possible implementations, the first neural network can also be a key point detection network for Identify one or more key points, and then determine the second position of the target area in the reference frame according to the positions of the identified key points.

In some possible implementation manners, the reference frame may also be used as the frame to be detected for depth detection. In this case, the reference frame may have undergone key point detection and a corresponding key point detection result has been obtained. Therefore, in some possible implementation manners, the second position of the target area in the reference frame may be obtained according to the key point detection result corresponding to the reference frame.

In some possible implementation manners, the key point detection may also be directly performed on the reference frame to obtain the key point detection result. For the key point detection method, reference may be made to other disclosed embodiments, which will not be repeated here.

Through the embodiment of the present invention, the second position of the target area in the reference frame can be flexibly determined in multiple ways according to the actual situation of the reference frame, which improves the flexibility and versatility of depth detection; and in some possible implementations , when the reference frame before the frame to be detected has participated in the depth detection, the second position can be determined directly based on the intermediate result of the reference frame in the depth detection, thereby reducing the repeated calculation of data and improving the efficiency of depth detection and precision.

In a possible implementation manner, the key point detection is performed on the target object in the clipping result to obtain the key point detection result, which may include:

The second neural network is used to perform key point detection on the target object in the clipping result to obtain a key point detection result.

Wherein, the second neural network can be any neural network used to realize key point detection, and its implementation mode is not limited in the embodiment of the present invention, wherein, the first neural network can be a key point detection network In some cases, the second neural network may be implemented in the same or different manner as the first neural network.

In some possible implementation manners, key point detection may also be performed on the target object in the clipping result through a related key point recognition algorithm, and the key point recognition algorithm to be used is also not limited in the embodiment of the present invention.

FIG. 3 shows a flow chart of a depth detection method according to an embodiment of the present invention. As shown in FIG. 3, in a possible implementation, step S13 may include:

Step S131, acquiring at least two preset device parameters respectively corresponding to at least two capture devices, the at least two capture devices are used to capture images of the target object from at least two capture angles of view.

Step S132: Determine the depth information of the target object in the frame to be detected according to at least two preset device parameters and a plurality of key point detection results.

For the implementation manner of the acquisition device, reference may be made to the above-mentioned disclosed embodiments, which will not be repeated here.

In some possible implementation manners, the at least two preset device parameters may include preset internal parameters respectively corresponding to at least two acquisition devices. The preset internal parameters may be some calibration parameters of the collection device itself, and the types and types of parameters contained therein may be flexibly determined according to the actual situation of the collection device. In some possible implementation manners, the preset internal parameters may include an internal reference matrix of the acquisition device, and the internal reference matrix may include one or more focal length parameters of the camera, principal point positions of one or more cameras, and the like.

In some possible implementations, since the collection device may include at least two collection devices, at least two preset device parameters may also include preset external parameters, wherein the preset external parameters may be between different collection devices The formed relative parameters are used to describe the relative positions of different acquisition devices in the world coordinate system. In some possible implementation manners, the preset external parameters may include an external parameter matrix formed between different acquisition devices. In an example, the external parameter matrix may include a rotation matrix and/or a translation vector matrix, and the like.

The method of obtaining preset device parameters is not limited in the embodiment of the present invention. In some possible implementations, the preset device parameters can be directly obtained according to the actual situation of the acquisition device. In some possible implementations, you can also The preset device parameters are obtained by calibrating the acquisition device.

According to the positional relationship among the multiple key point detection results, combined with at least two preset device parameters, the parallax formed between different frames to be detected in the three-dimensional world coordinate system can be determined. As mentioned in the above-mentioned disclosed embodiments, the content of the information contained in the depth information can be flexibly determined according to the actual situation. Therefore, as the content of the depth information is different, the process of determining the depth information according to the preset device parameters and the detection results of multiple key points can also be determined at any time. For the changes, see the following disclosed embodiments for details, and will not be expanded here.

Through the embodiment of the present invention, at least two preset device parameters and multiple key point detection results can be used to determine the parallax formed between different frames to be detected, and to determine the depth information simply and conveniently. This method has a small amount of calculation and is The result is more accurate, which can improve the accuracy and efficiency of depth detection.

In a possible implementation manner, step S132 may include:

The depth distance is obtained according to the preset external parameters among the at least two preset device parameters and the coordinates of the multiple key point detection results in at least two forms.

Wherein, the implementation form of preset external parameters may refer to the above-mentioned disclosed embodiments, which will not be repeated here. The coordinates of the key point detection results in at least two forms may be the corresponding coordinates of the key point detection results in different coordinate systems, for example, may include the pixel coordinates formed by the key point detection results in the image coordinate system, and /or, homogeneous coordinates formed separately in different acquisition devices, etc. Which form of coordinates to choose can be flexibly selected according to actual conditions, and is not limited to the following disclosed embodiments.

In the process of obtaining the depth distance, the coordinates of the key points in the key point detection results are not limited in the embodiment of the present invention. In some possible implementations, the head key point, left shoulder key point and right shoulder key point can be selected. One or more of the shoulder keys and, in one example, the head key. In some possible implementations, the center of the head and shoulders can also be chosen.

Wherein, the center point of the head and shoulders may be the center point of the head and shoulders frame mentioned in the above-mentioned disclosed embodiments. In some possible implementation manners, according to the position coordinates of the head key point, left shoulder key point and right shoulder key point, Determine the overall position coordinates of the head and shoulders frame, and determine the position coordinates of the center point of the head and shoulders based on the overall position coordinates of the head and shoulders frame; in some possible implementations, the center point of the head and shoulders can also be directly used as the key to be detected point, so that the position coordinates of the center point of the head and shoulders can be directly obtained in the key point detection results.

As the number of acquisition devices is different, the calculation method for obtaining the depth distance can be flexibly changed, and is not limited to the following disclosed embodiments. In one example, it may include two acquisition devices, a left camera and a right camera. In this case, according to the external parameters in at least two preset device parameters and the coordinates of a plurality of key point detection results in at least two forms , the process of obtaining the depth distance can be expressed by the following formulas (1) and (2):

（1）

(1)

（2）

(2)

為深度距離，

為左相機採集的待檢測幀中關鍵點在齊次形式下的原始坐標，

為對原始坐標進行線性變換後所得到的變換坐標，

為右相機採集的待檢測幀中關鍵點在齊次形式下的坐標，

為預設外部參數中右相機相對於左相機的旋轉矩陣

，

為預設外部參數中右相機相對於左相機的平移向量矩陣

。 in,

is the depth distance,

is the original coordinates of the key points in the homogeneous form in the frame to be detected collected by the left camera,

is the transformed coordinate obtained by linearly transforming the original coordinate,

is the coordinates of the key points in the homogeneous form in the frame to be detected collected by the right camera,

is the rotation matrix of the right camera relative to the left camera in the preset extrinsic parameters

,

is the translation vector matrix of the right camera relative to the left camera in the preset extrinsic parameters

.

Through the embodiment of the present invention, the homogeneous form coordinates of key points in different camera coordinate systems, and the coordinates of key points in the form of linear transformation can be combined with the relative preset external parameters between different cameras, with a small amount of calculation Accurately determine the depth distance, thereby improving the accuracy and efficiency of depth detection.

In a possible implementation manner, step S132 may also include:

The offset angle is obtained according to the preset internal parameters in the at least two preset device parameters and the coordinates of the multiple key point detection results in at least two forms.

Wherein, the implementation forms of preset internal parameters and coordinates of key point detection results in at least two forms can also refer to the above disclosed embodiments, and will not be repeated here.

According to the preset internal parameters and the coordinates of the key point detection results in at least two forms, the way of obtaining the offset angle can also be flexibly selected, and is not limited to the following disclosed embodiments. In the process of determining the offset angle, the type of selected key points can also be flexibly selected according to the actual situation. You can refer to the type of key points selected in the above-mentioned determination of the depth distance, which will not be repeated here.

Similar to the determination process of the depth distance, with the different number of acquisition devices, the calculation method for obtaining the offset angle can also be flexibly changed, and is not limited to the following disclosed embodiments. In an example, taking the acquisition device including a certain target camera as an example, the process of obtaining the offset angle relative to the target camera can be expressed by the following formulas (3) to (5):

（3）

(3)

（4）

(4)

（5）

(5)

為目標對象在x軸方向上的偏移角度，

為目標對象在y軸方向上的偏移角度，

為目標相機採集的待檢測幀中關鍵點在齊次形式下的坐標，

為目標相機採集的待檢測幀中關鍵點的像素坐標，

和

為目標相機的內參矩陣

中的焦距參數，

和

為目標相機的內參矩陣

中的主點位置。 in,

is the offset angle of the target object in the x-axis direction,

is the offset angle of the target object in the y-axis direction,

is the coordinates of the key points in the homogeneous form in the frame to be detected collected by the target camera,

is the pixel coordinate of the key point in the frame to be detected collected by the target camera,

and

is the intrinsic parameter matrix of the target camera

The focal length parameter in ,

and

is the intrinsic parameter matrix of the target camera

The location of the principal point in .

Through the embodiment of the present invention, the offset angle can be determined simply and conveniently by using the preset internal parameters and the coordinates of the key point detection results obtained in the depth detection process in different forms. This determination method does not need to obtain additional data and is convenient. Computing can improve the efficiency and convenience of in-depth detection.

In a possible implementation manner, the method proposed in the embodiment of the present invention may also include:

According to the depth information of the target object, the position of the target object in the three-dimensional space is determined.

Wherein, the position of the target object in the three-dimensional space may be the three-dimensional coordinates of the target object in the three-dimensional space. The way to determine the position in the three-dimensional space based on the depth information can be flexibly selected according to the actual situation. In a possible implementation mode, the two-dimensional coordinates of the target object in the frame to be detected can be determined according to the key point detection results of the target object, and The 2D coordinates are combined with the depth distance and/or offset angle in the depth information to determine the 3D coordinates of the target object in the 3D space.

After determining the position of the target object in the three-dimensional space, based on the three-dimensional position information, face recognition, living body recognition, route tracking or application to virtual reality (Virtual Reality, VR) or augmented reality (Augmented Reality) can be performed on the target object. Reality, AR) and other scenarios. Through the embodiments of the present invention, the depth information can be used to perform three-dimensional positioning of the target object, so as to realize various operations such as interaction with the target object. For example, in some possible implementations, the distance and angle between the target object and the smart air conditioner can be determined according to the position of the target object in three-dimensional space, so as to dynamically adjust the wind direction and/or wind speed of the smart air conditioner; in some possible In the implementation mode, the target object can also be positioned in the game scene based on the position of the target object in the three-dimensional space in the AR game platform, so that the human-computer interaction in the AR scene can be realized more realistically and naturally.

It can be understood that the above-mentioned method embodiments mentioned in the present invention can all be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, the present invention will not repeat them. Those skilled in the art can understand that, in the above method in the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

In addition, the present invention also provides depth detection devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any depth detection method provided by the present invention, corresponding technical solutions and descriptions, and refer to the corresponding records in the method section, No longer.

Fig. 4 shows a block diagram of a depth detection device according to an embodiment of the present invention. As shown in Figure 4, device 20 includes:

The obtaining module 21 is configured to obtain multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained by collecting images of the target object from at least two collection angles of view.

The key point detection module 22 is used to perform key point detection of the target area in the target object according to the frame to be detected, and determine a plurality of key point detection results corresponding to multiple frames to be detected, wherein the target area includes the head area and/or or the shoulder area.

The depth detection module 23 is configured to determine the depth information of the target object according to the detection results of multiple key points.

In a possible implementation manner, the depth detection module is used to: obtain at least two preset device parameters respectively corresponding to at least two collection devices, and the at least two collection devices are used to map the target object from at least two collection perspectives. Image acquisition; according to at least two preset device parameters and multiple key point detection results, determine the depth information of the target object in the frame to be detected.

In a possible implementation manner, the depth information includes a depth distance, and the depth distance includes a distance between the target object and the optical center of the acquisition device; the depth detection module is further used for: according to presets in at least two preset device parameters The external parameters and the coordinates of the multiple key point detection results in at least two forms can be used to obtain the depth distance; wherein, the preset external parameters include relative parameters formed between at least two acquisition devices.

In a possible implementation, the depth information includes an offset angle, and the offset angle includes a spatial angle of the target object relative to the optical axis of the acquisition device; the depth detection module is further used to: according to at least two preset device parameters The preset internal parameters and the coordinates of the multiple key point detection results in at least two forms are used to obtain the offset angle; wherein the preset internal parameters include device parameters corresponding to at least two devices respectively.

In a possible implementation, the key point detection module is used to: perform key point detection on the target area of the target object in the frame to be detected according to the position information of the target object in the reference frame, and obtain the key point corresponding to the frame to be detected The point detection result, wherein, the reference frame is the movie frame before the frame to be detected in the target movie to which the frame to be detected belongs.

In a possible implementation, the key point detection module is further used to: crop the frame to be detected according to the first position of the target object in the reference frame to obtain the cropping result; key the target area of the target object in the cropping result Point detection to obtain key point detection results corresponding to the frame to be detected.

In a possible implementation, the key point detection module is further used to: obtain the second position of the target area of the target object in the reference frame; according to the second position, the frame to be detected is cropped to obtain the cropping result; the cropping result The key point detection is performed on the target area of the target object in , and the key point detection result corresponding to the frame to be detected is obtained.

In a possible implementation, the key point detection module is further used to: use the first neural network to identify the target area in the reference frame to obtain the second position output by the first neural network; and/or, according to The key point detection result corresponding to the reference frame is used to obtain the second position of the target area in the reference frame.

In a possible implementation manner, the device is further configured to: determine the position of the target object in the three-dimensional space according to the depth information of the target object.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present invention can be used to execute the methods described in the above method embodiments, and its specific implementation can refer to the description of the above method embodiments. For brevity, I won't go into details here.

Application Scenario Example

Fig. 5 shows a schematic diagram of an application example according to the present invention. As shown in Fig. 5, the application example of the present invention proposes a depth detection method, which may include the following process:

Step S31, use the Faster RCNN neural network to detect the head and shoulders frame of the human body from the two frames to be detected taken by the binocular camera (including the left camera and the right camera), and obtain the head and shoulders frame in the first frame of the left camera. position, and the position of the head-and-shoulders box in the first frame of the right camera.

Step S32, obtain the target films corresponding to the left camera and the right camera respectively, start from the second frame of the target film, use this film frame as the frame to be detected, and use the previous frame of the frame to be detected as the reference frame, according to the header in the reference frame For the second position of the shoulder frame, the key point detection is performed on the frame to be detected through the key point detection network, and the position coordinates of the three key points of the head key point, left shoulder key point and right shoulder key point are obtained, and the three key points The circumscribed rectangle of the point is used as the head and shoulders frame in the frame to be detected.

Step S33, according to the coordinates of the key points in the frame to be detected in at least two forms, and the internal reference matrix of the camera, calculate the offset angle of the target object relative to the camera:

和

。 Among them, according to the pixel coordinates (u, v, 1) of the key points of the head in the frame to be detected and the internal reference matrix K of the camera, the head The homogeneous coordinates (x/z, y/z, 1) corresponding to the internal key points, and the offset angle relative to the camera optical axis

and

.

Step S34, according to the homogeneous coordinates of the key points in the frame to be detected in the left camera and the right camera, and the extrinsic matrix of the right camera relative to the left camera, calculate the depth distance of the target object:

Among them, according to the homogeneous coordinates of the same key point in the left and right cameras respectively, and the extrinsic matrix R and T of the right camera relative to the left camera, through the formula (1) mentioned in the above disclosed embodiments and (2), calculate the depth distance d of the target object.

In one example, after the depth information of the target object in the frame to be detected is determined through steps S33 and S34, the next frame of the frame to be detected in the target video corresponding to the left camera and the right camera can also be used as the frame to be detected , and return to step S32 to perform depth detection again.

Through the application example of the present invention, the head and shoulders frame of the human body and the key points in the head and shoulders frame can be used to calculate the parallax formed by the frames to be detected collected under different viewing angles. Compared with the parallax estimation method based on image matching, the calculation amount Smaller, wider application scenarios.

It can be understood that the above-mentioned method embodiments mentioned in the present invention can all be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, the present invention will not repeat them.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

The embodiment of the present invention also proposes a computer-readable storage medium on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor. A computer readable storage medium may be a non-volatile computer readable storage medium or a volatile computer readable storage medium.

An embodiment of the present invention also proposes an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

An embodiment of the present invention also provides a computer program product, including computer-readable codes. When the computer-readable codes run on the device, the processor in the device executes instructions for implementing the depth detection method provided in any of the above embodiments. .

The embodiment of the present invention also provides another computer program product, which is used for storing computer-readable instructions. When the instructions are executed, the computer executes the operation of the depth detection method provided by any one of the above-mentioned embodiments.

Electronic devices may be provided as terminals, servers, or other types of devices.

FIG. 6 shows a block diagram of an electronic device 800 according to an embodiment of the present invention. As shown in FIG. 6 , the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message sending and receiving device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant and other terminals.

Referring to FIG. 6, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor The implementer component 814, and the communication component 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

The memory 804 is configured to store various types of data to support operations in the electronic device 800 . Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. Memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electronically erasable rewritable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk.

The power supply component 806 provides power to various components of the electronic device 800 . Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 800 .

The multimedia component 808 includes a screen providing an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a movie mode, the front camera lens and/or the rear camera lens can receive external multimedia data. Each front camera lens and rear camera lens can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in operation modes, such as call mode, recording mode and voice recognition mode. The received audio signal can be further stored in the memory 804 or sent via the communication component 816 . In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing unit 802 and peripheral interface modules, such as a keyboard, a click wheel, and buttons. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor assembly 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of user contact with the electronic device 800 , the orientation or acceleration/deceleration of the electronic device 800 and the temperature of the electronic device 800 change. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 814 may also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field Programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the method described above.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to implement the above method.

FIG. 7 shows a block diagram of an electronic device 1900 according to an embodiment of the present invention. As shown in FIG. 7, the electronic device 1900 may be provided as a server. Referring to FIG. 7 , electronic device 1900 includes processing component 1922 , which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by processing component 1922 , such as application programs. The application programs stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input/output (I/O) Interface 1958. The electronic device 1900 can operate based on the operating system stored in the memory 1932, such as the Microsoft server operating system (Windows Server ^TM ), the operating system based on the graphical user interface (Mac OS X ^TM ) introduced by Apple Inc., multi-user and multi-process computer Operating system (Unix ^™ ), free and open source Unix-like operating system (Linux ^™ ), open source Unix-like operating system (FreeBSD ^™ ) or similar.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to implement the above method.

The present invention can be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for causing a processor to implement various aspects of the invention.

A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only Memory (EPROM or Flash), Static Random Access Memory (SRAM), Compact Disk Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD), Memory Cards, Diskettes, Mechanical Encoding devices, such as punched cards or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing/processing device, or to an external computer over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage in each computing/processing device in storage media.

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or any Combining source or object code written in programming languages including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server to execute. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or it can be connected to an external computer (for example, using the Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), are customized by utilizing state information of computer readable program instructions, The electronic circuit can execute computer-readable program instructions to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart and/or block diagrams, and combinations of blocks in the flowchart and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus, the Means that implement the functions/actions specified in one or more blocks in the flowchart and/or block diagrams are produced. These computer-readable program instructions can also be stored in computer-readable storage media, and these instructions cause computers, programmable data processing devices and/or other devices to operate in a specific way, so that computer-readable media storing instructions include An article of manufacture comprising instructions for implementing aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flow charts and/or block diagrams.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of an instruction that contains one or more Executable instructions for logical functions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block in the block diagrams and/or flow diagrams, and combinations of blocks in the block diagrams and/or flow diagrams, can be implemented with a dedicated hardware-based computer that performs the specified function or action. system, or may be implemented using a combination of dedicated hardware and computer instructions.

The computer program product can be realized by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. wait.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or improvement of technology in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

S11~S13: Steps S31~S34: steps S131~S132: Steps 20: Depth detection device 21: Get the mod 22: Key point detection module 23: Depth detection module 800: Electronic equipment 802: processing components 804: memory 806: Power components 808:Multimedia component 810:Audio components 812: input/output interface 814: Sensor component 816:Communication component 820: Processor 1900: Electronic equipment 1922: Processing components 1926: Power components 1932: Memory 1950: Web interface 1958: Input and output interface

The accompanying drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments consistent with the present invention, and are used together with the description to explain the technical solution of the present invention. Fig. 1 shows the flowchart of the depth detection method according to an embodiment of the present invention; Fig. 2 shows a schematic diagram of a target area according to an embodiment of the present invention; Fig. 3 shows the flowchart of the depth detection method according to an embodiment of the present invention; 4 shows a block diagram of a depth detection device according to an embodiment of the present invention; Fig. 5 shows a schematic diagram of an application example according to the present invention; Figure 6 shows a block diagram of an electronic device according to an embodiment of the present invention; and Fig. 7 shows a block diagram of an electronic device according to an embodiment of the present invention.

S11~S13: Steps

Claims (13)

A deep detection method, comprising: Acquiring multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained by collecting images of the target object from at least two acquisition angles of view; Perform key point detection of a target area in the target object according to the frame to be detected, and determine a plurality of key point detection results corresponding to the multiple frames to be detected, wherein the target area includes a head area and/or shoulder area; Depth information of the target object is determined according to the plurality of key point detection results. According to the method described in claim 1, wherein the determining the depth information of the target object according to the multiple key point detection results includes: Acquire at least two preset device parameters respectively corresponding to at least two capture devices, the at least two capture devices are used to capture images of the target object from at least two capture angles of view; Depth information of the target object in the frame to be detected is determined according to the at least two preset device parameters and the plurality of key point detection results. The method according to claim 2, wherein the depth information includes a depth distance, and the depth distance includes a distance between the target object and the optical center of the acquisition device; The determining the depth information of the target object in the frame to be detected according to the at least two preset device parameters and the multiple key point detection results includes: According to the preset external parameters in the at least two preset device parameters and the coordinates of the plurality of key point detection results in at least two forms, the depth distance is obtained; wherein the preset external parameters include the The relative parameters formed between the at least two acquisition devices. The method according to claim 2, wherein the depth information comprises an offset angle comprising a spatial angle of the target object relative to an optical axis of the acquisition device; The determining the depth information of the target object in the frame to be detected according to the at least two preset device parameters and the multiple key point detection results includes: According to the preset internal parameters in the at least two preset device parameters and the coordinates of the plurality of key point detection results in at least two forms, the offset angle is obtained; wherein the preset internal parameters include Device parameters respectively corresponding to the at least two devices. According to the method described in claim 1, wherein the key point detection of the target area in the target object according to the frame to be detected includes: According to the position information of the target object in the reference frame, perform key point detection on the target area of the target object in the frame to be detected, and obtain a key point detection result corresponding to the frame to be detected, wherein the The reference frame is a film frame before the frame to be detected in the target film to which the frame to be detected belongs. According to the method described in claim 5, wherein, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the obtained Key point detection results corresponding to the frame to be detected, including: clipping the frame to be detected according to the first position of the target object in the reference frame to obtain a clipping result; Key point detection is performed on the target area of the target object in the clipping result to obtain a key point detection result corresponding to the frame to be detected. According to the method described in claim 5, wherein, according to the position information of the target object in the reference frame, the key point detection is performed on the target area of the target object in the frame to be detected, and the obtained Key point detection results corresponding to the frame to be detected, including: Acquiring a second position of the target area of the target object in the reference frame; clipping the frame to be detected according to the second position to obtain a clipping result; Key point detection is performed on the target area of the target object in the clipping result to obtain a key point detection result corresponding to the frame to be detected. The method according to claim 7, wherein said obtaining the second position of the target area of the target object in the reference frame includes: Using the first neural network to identify the target area in the reference frame to obtain the second position output by the first neural network; and/or, Obtain a second position of the target area in the reference frame according to the key point detection result corresponding to the reference frame. The method according to any one of claims 1 to 8, wherein the method further comprises: According to the depth information of the target object, the position of the target object in the three-dimensional space is determined. A depth detection device, comprising: An acquisition module, configured to acquire multiple frames to be detected, wherein the multiple frames to be detected include image frames obtained from image acquisition of the target object from at least two acquisition angles of view; The key point detection module is used to perform key point detection of the target area in the target object according to the frame to be detected, and determine a plurality of key point detection results corresponding to the multiple frames to be detected, wherein the target Areas include head area and/or shoulder area; The depth detection module is used to determine the depth information of the target object according to the detection results of the plurality of key points. An electronic device comprising: processor; memory for storing processor-executable instructions; Wherein, the processor is configured to invoke the instructions stored in the memory to execute the method described in any one of claims 1-9. A computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in any one of claims 1 to 9 is realized. A computer program product, including computer readable code, when the computer readable code is run in the electronic device, the processor in the electronic device executes the method for implementing any one of claims 1 to 9 .

Images