EP3827371A1 - Image processing method - Google Patents

Image processing method

Info

Publication number
EP3827371A1
EP3827371A1 EP19745257.6A EP19745257A EP3827371A1 EP 3827371 A1 EP3827371 A1 EP 3827371A1 EP 19745257 A EP19745257 A EP 19745257A EP 3827371 A1 EP3827371 A1 EP 3827371A1
Authority
EP
European Patent Office
Prior art keywords
image
continuous probability
probability distribution
location
point
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP19745257.6A
Other languages
German (de)
French (fr)
Inventor
Andrea Vedaldi
Lukas Neumann
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Oxford University Innovation Ltd
Original Assignee
Oxford University Innovation Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oxford University Innovation Ltd filed Critical Oxford University Innovation Ltd
Publication of EP3827371A1 publication Critical patent/EP3827371A1/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present invention relates to a method of determining the location of a point in an image using a deep neural network.
  • Deep learning techniques have improved the performance of generic object detection and segmentation methods when applied to images. These techniques can provide reasonable performance when applied to interpreting images containing people.
  • One objective of interpreting images containing people is pose recognition, or the process of identifying the location of particular parts of the human body (such as shoulders, wrists or hips) and using these locations to define the shape and therefore pose of the person within the image (e.g. standing, lying, which way they are facing, etc.).
  • a deep neural network is adapted to predict a heat map for each location of a part of a human body in an image, where each location is found as position of a maximal value in the corresponding heat map.
  • In“Stacked hourglass net-works for human pose estimation” (A. Newell, K. Yang, and J. Deng. In European Conference on Computer Vision, pages 483-499. Springer, 2016) an improved network architecture for the deep neural network referred to as a“Stacked Hourglass” is applied which allows features of different scales in the image to be identified in the deep neural network.
  • Part Affinity Fields a set of confidence maps of a location of a part of the human body together with 2D vector fiel ds that encode the position and orientation of parts of the human body are output by the deep neural network.
  • the size of a person in the image may be much smaller.
  • a method of determining the locations of points in images of low resolution would be advantageous.
  • a method of determining the location of at least one point in an image comprising: generating a first continuous probability distribution function from the image using the output of a deep neural network; and using the first continuous probability function to determine the location of the point in the image.
  • an image analysis device for determining the location of at least one point in an image, compri sing: a calculator unit configured to receive an output from a neural network and generate a first continuous probability distribution function from the image; and a locator unit configured to receive the first continuous probability distribution function from the calculator and to use the first continuous probability function to determine the location of the point in the image.
  • the position of the maximum value of the first continuous probability function may indicate the most likely position in the image of the location of the point in the image.
  • the method may further comprise any one of dividing the image into a plurality of sub images; generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
  • the image analysis device may further comprise: an image processing unit configured to divide the image into a plurality of sub images; wherein the calculator unit is configured to generate a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and the calculator unit is configured to calculate the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
  • Each of the plurali ty of sub images may be associated with at least one of the plurality of second continuous probability distribution functions.
  • Each one of the plurality of sub images may be associated with a different one of the plurality of second continuous probability distribution functions.
  • Each of the plurality of second continuous probability distributions may be defined by a first distance and a second distance.
  • the first distance may indicate a distance of a peak of the second continuous probability distribution from a point in the sub image associated with the second continuous probability distribution in a first direction.
  • the second distance may indicate a distance of a peak of the second continuous probability distribution from the point in the sub image associated with the second continuous probability distribution in a second direction.
  • the first direction may be parallel to the x-axis of the image and the second direction may be parallel to the y-axis of the image.
  • Each of the plurality of second continuous probability distributions may be defined by a first standard deviation and a second standard deviation.
  • the first standard deviation may indicate an uncertainty associated with the first distance and the second standard deviation may indicate an uncertainty associated with the second distance.
  • At least one of the plurality of second continuous probability distributions may be a Gaussian distribution.
  • the method may further comprise generating a weighting associated with each of the plurality of second continuous probability distribution functions: and the weighting associated with each of the plurality of second continuous probability distribution functions may be applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
  • the method may further comprise at least one of generating at least one further first continuous probability distribution function associated with the image and using the at least one further first probability function to determine the location of at least one further point in the image.
  • the location of each of the at least one points in the image may correspond to the location of a part of the human body in the image.
  • the image may be a low resolution image.
  • a feature associated with the at least one point in the image may have a size of less than or equal to 64 pixels.
  • the image may be divided into greater than or equal to 2 sub images and less than or equal to 64 sub images.
  • the image may be divided into 16 sub images.
  • an image analysis device comprising an analyser configured to perform the method according to the present invention to determine the location of at least one point in an image.
  • a method of training a deep neural network comprising: providing the deep neural network with an image;
  • the prediction of the location of the point in the image to the location of the point in the image may be compared by maximizing the logarithmic probability of the first probability distribution at the location of the point in the image.
  • the method of training a deep neural network may further comprise any one of dividing the image into a plurality of sub images; generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability functions is associated with at least one of the plurality of sub images; and calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
  • the method of training a deep neural network may further comprise generating a weighting associated with each of the plurality of second continuous probability distribution functions using the output of the deep neural network; and the weighting associated with each of the plurality of second continuous probability distribution functions may be applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
  • the comparing of the prediction of the location of the point in the image to the location of the point in the image may further comprise increasing the weighting associated with second continuous probability distributions associated with sub images that are close to the position of the point in the image.
  • Figure 1 illustrates a section of an image containing a person.
  • Figure 2 illustrates a method of determining the location of at least one point in an image according to an embodiment of the present invention.
  • Figure 3 illustrates an image divided into a plurality of sub-images.
  • Figure 4 illustrates a schematic diagram of a process performed by a deep neural network according to an embodiment of the present invention.
  • Figure 5 illustrates a flow chart of a method of generating a probability distribution function according to an embodiment of the present invention.
  • Figure 6 illustrates a method of training a deep neural network according to an
  • Figure 7 illustrates data comparing the performance of the method of determining the location of at least one point in an image of an embodiment of the present invention to other methods.
  • Figure 1 illustrates an image 1 which includes a person 100.
  • the person 100 is visible in the image 1, various different body parts of the person 100 may be visible in the image 1 depending on the pose of the person 100 within the image 1.
  • the person 100 is facing the image 1 such that both legs and arms of the person are visible in the image.
  • Points 5 in the image 1 may be identified as points that correspond to parts of the human body of the person 100 visible in the image 1. These points 5 may be referred to as landmark points.
  • the location of points 5 in the image associated with the left hand 5a, left elbow 5b and chest 5c of the person 100 have been identified.
  • the points 5 identified in the image 1 may then be connected to provide a representation or an estimation of the pose of the person 100 in the image 1.
  • the points 5a, 5b and 5c may be connected to produce a representation or an estimation of the location of the left arm of the person 100 in the image 1.
  • the location of three points 5 associated with different parts of the human body have been identified. However, the location of any number of points 5 may be identified in the image 1. In the case where the points 5 correspond to parts of the human body, the location of further points 5 such as the head, right arm, right elbow, hips, left and right knees and left and right feet may be identified.
  • the points 5 that may be identified in the image are not limited to points on the human body, and the location of other categories of points 5 may be identified.
  • the locations of points 5 may be identified that are associated with particular features or buildings in an image 1 of an aerial photograph of an area.
  • the locations of points 5 may be identified that are associated with features on a human face in an image 1.
  • the location of human features such as the nose, left and right eyes or mouth on a face may be identified.
  • the size of a feature within an image 1, such as a person 100 may be measured in the number of pixels that the feature occupies within the image 1.
  • the image 1 may be a low resolution image.
  • the height of the feature in the image 1, such as a person 100 may be 128, 96, 64, 48, 32 or 24 pixels high.
  • the width of the feature in the image 1 may be a corresponding or different number of pixels.
  • the size of the feature may be determined by the height and/or width of the feature.
  • Figure 2 illustrates a method of determining the location of at least one point 5 in an image 1 according to the present invention.
  • the image 1 is obtained.
  • the image 1 may have been obtained shortly before the method is performed.
  • the location of the at least one point 5 in the image may be determined in real time.
  • the image 1 may be obtained by part of an autonomous driving system.
  • the image 1 may have been obtained at an earlier point in time.
  • a first continuous probability distribution function is generated from the image 1 using the output of a deep neural network.
  • the deep neural network is a deep learning system for analysing data representations such as an image 1.
  • a deep neural network is a network of nodes which may be referred to as neurons.
  • the deep neural network may consist of at least three layers of nodes where each node is linked to at least one node in the layer above and at least one node in the layer below.
  • Each node receives an input from each linked node in the layer below, applies a weight to the input, sums the weighted input, applies a non-linear function to the su and to obtain an output. The output is then provided to each of the linked nodes in the layer above.
  • the first continuous probability distribution function is a two dimensional continuous function associated with the image 1.
  • the first CPDF may be continuous across the entire area of the image 1
  • the first CPDF indicates a probability that a point 5 in the image I is located a location in the image 1.
  • the first CPDF has an infinite resolution. The resolution of the first CPDF is therefore unrelated to the resolution of the image 1.
  • step 10c the location of the point 5 in the image is determined using the first CPDF.
  • the location of the point 5 in the image 1 may be determined as the location corresponding to the maximum probability of the first CPDF. Alternatively, the location of the point 5 in the image 1 may be determined using other features of the first CPDF.
  • Steps 10b and 10c may be repeated any number of times to determine the location of a plurality of points 5 within the image.
  • at least one further first CPDF may be generated from the output of the deep neural network.
  • a first CPDF may be generated for each points 5 to be located within the image 1. For example, when determining a pose of a person within the image 1, the location on of a plurality of points 5 corresponding to different parts of the human body of a person 100 may be determined. The location of the plurality of points 100 in the image 1 may be used to produce an estimate of the pose of the person 100 within the image 1.
  • the image 11 may be divided into a plurality of sub-images 11, as illustrated in Figure 3.
  • the image 1 is divided into sixteen sub-images 11.
  • the image 1 may be divided into any number of sub-images 11.
  • the image may be divided into greater than or equal to 2 sub images and less than or equal to 64 sub images. If the image 1 is divided into a smaller number of sub-images, the deep neural network may be able to produce an output more quickly. If the image 1 is divided into a larger number of sub-images 11, the deep learning network may be able to produce a more accurate output.
  • the image 1 may be upscaled or downscaled when the image 1 is divided into the plurality of sub images 11.
  • a second CPDF associated with each sub-image 1 1 of the image 1 may be generated from the output of the deep neural network.
  • One second CPDF may be associated with each of the plurality of sub-images 11
  • a plurality of second CPDFs may be associated one sub image 11.
  • a second CPDF may not be generated for at least one of the plurality of sub images 11 in the image 1.
  • the output 3 of the deep learning network may include a displacement associated with each second CPDF.
  • the displacement may indicate the distance of the peak 12 or maximum value of the second CPDF from a point in the sub-image 14.
  • the displacement may comprise a first distance parallel to the x-axis of the image 1 and a second distance parallel to the y- axis of the image 1.
  • the location of the point in the sub-image 14 may correspond to the centre of the sub-image 14.
  • the location of the point in the sub-image may 14 also correspond to one of the corners of the sub image 14
  • the output 3 of the deep neural network may include a first uncertainty associated with the first distance associated with each second CPDF and a second uncertainty associated with the second distance associated with each second CPDF.
  • the uncertainty in the first distance may correspond to a standard deviation of the first distance and the uncertain in the second distance may correspond to a standard deviation of the second distance.
  • Each of the plurality of second CPDFs may be a Gaussian distribution.
  • the first and second distance and the first and second uncertainty are the parameters which may define each of the second CPDFs
  • Each of the plurality of CPDFs may be a different CPDF which may he defined by different parameters that are associated with that CDPF.
  • the output 3 of the deep neural net rk may include a weighting associated with each second CPDF.
  • the weighting may be used when calculating the first CPDF from the plurality of second CPDFs as described below.
  • An example of the process of generating the output 3 from the deep neural network is illustrated in Figure 4.
  • An image 1 is provided that contains a person 100.
  • the image 1 is provided to the deep neural network.
  • the deep neural network analyses the image 1 and provides an output 3.
  • the output 3 may include the plurality of first distances m ch 3a associated with each of the plurality of second CDPFs.
  • the output 3 may include the plurality of second distances p yn 3b associated with each of the plurality of second CDPFs.
  • the output 3 may include the plurality of first uncertainties s ch 3c associated with each of the plurality of second CDPFs.
  • the output 3 may include the plurality of second uncertainties o yn 3d associated with each of the plurality of second CDPFs.
  • the output 3 may include the plurality of weightings W 3e associated with each of the plurality of second CDPFs. In the example shown in figure 4, only tw ? o of the each of the first and second distances, uncertainties and the
  • the plurality of second CPDFs may be combined to form the first continuous probability distribution. If each of the second CPDFs is a Gaussian distribution, the first CPDF may ⁇ be defined by a Gaussian mixture model as follows:
  • F is a function mapping the image x to the point in the image it, v is the point in the sub image, A k (v; F ) is a displacement, ⁇ k( v; F ) is a covariance matrix and p fc (v
  • F may be the operation performed by the deep neural network which may be a fully convolutional neural network.
  • each of the parameters associated with each sub image 11 are obtained from the deep neural network in step 20a and the parameters are combined to generate the first CPDF in step 20b.
  • An advantage determining the location of at least on point in an image of using a CPDF may be that while the first CPDF may be obtained from the discrete data provided by the pixels of the image 1, the first continuous probability distribution is continuous and therefore has infinite resolution. The method may therefore allow sub-pixel interpolation of the point 5 in the image 1.
  • the deep learning network may be provided on an analyser.
  • the analyser may form part of an image analysis device which receives the input of an image 1 and determines the position of at least one point 5 within the image 1.
  • Such an image analysis device may be provided in a self-driving system.
  • the self driving system may acquire images containing pedestrians which may be processed by the image analysis device to output the positions of the body part of the pedestrians within the image 1. The pose of the pedestrians may therefore be determined to allow potential movements of the pedestrians to be predicted.
  • Figure 6 illustrates a method of training a deep neural network according to the present invention.
  • an image 1 containing at least one point 5 of interest is provided to the deep neural network in step 30a.
  • the image may be one of a plurality of images 1 containing at least one point 5 of interest which have been selected for training the deep neural network.
  • the plurality of images may consist of images of people where the deep learning network is being training to determine points corresponding to locations on the human body within the plurality of images.
  • the image 1 is provided to the deep neural network and a first CPDF is generated from the image 1 using the output of deep neural network. This process is substantially the same as the process of generating the first CPDF that has already been described in the method of determining the location of a point 5 in an image 1 according to the present invention..
  • the first CPDF may be used to predict the location of a point in the image.
  • This process is substantially the same as the process of determining the location of a point in the image using the first CPDF described in the method of determining the location of a point 5 in an image 1 according to the present invention..
  • step 30d the predicted location of the point 5 in the image 1 is compared to the actual location of the point 5 in the image 1. For a plurali ty of images 1 each containing a plurality of points 5 of interest this may be compared by optimizing the loss:
  • X-i is one of the plurality of images 1 and u k is the plurality of points 5 in each image 1.
  • step 30e the deep network is adjusted based on the result of the comparison performed in step 30d.
  • the weight applied to the inputs received by each node in the deep neural network may be adjusted by a positive or a negative increment.
  • the adjustment may be performed to gradually minimise the loss £ 3.
  • the minimization process may be performed by a Stochastic Gradient Descent algorithm.
  • the method of training the deep neural network may be repeated a large number of times to optimize the accuracy of the output of the deep neural network when used to generate a first CPDF that can be used to determine the location of a point 5 in the image 1.
  • the comparison of the predicted location of the point 5 in the image 1 to the actual location of the point 5 in the image 1 may further include increasing the weighting associated with second continuous probability distributions associated with sub images that are close to the position of the point in the image.
  • the optimization may include a further term:
  • 0 is a term representing the number of sub images and the Gaussian mixture weight associated with each sub image.
  • Q is a term that may be set to 3.
  • Figure 7 shows examples of the results of the accuracy of the determination of a location of a plurality of points 5 in an image 1 using the method according to the present invention compared to other methods.
  • results of the accuracy of the determination of a location of a plurality of points 5 in an image 1 according to the present invention are compared to the methods for human pose estimation Stacked Hourglass and Part Affinity Field discussed above, using models provided by their authors.
  • the input images for the Stacked Hourglass and Part Affinity Field methods is up-sampled to 256 c 256 pixels using bilinear interpolation while maintaining centring of the image.
  • Input images to the method according to the present invention rvere down or up-sampled to 64 x 64 pixels using the same mechanism.
  • the deep neural network was built on the Spatial ConvNet architecture described in“Flowing convnets for human pose estimation in videos” (T. Pfister J. Charles, and A. Zisserman., Proceedings of the IEEE International Conference on Computer Vision, pages 1913-1921, 2015 ).
  • Graph 50 in figure 7 shows the standard PCKh@0.5metric to measure point detection accuracy, which requires a detection to be within certain distance from the actual point position, where the thresholding distance is given by 50% of the head segment length.
  • the method according to the present invention shown as the solid line 50a, outperforms both the Stacked Hourglass method, shown as the wide dashed line 50b and the Part Affinity Field method, shown as the narrow dashed line 50c, when the size of the person in the image is below 64 pixels.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Image Analysis (AREA)

Abstract

A method of determining the location of at least one point in an image, the method comprising generating a first continuous probability distribution function from the image using the output of a deep neural network; and using the first continuous probability function to determine the location of the point in the image.

Description

IMAGE PROCESSING METHOD
The present invention relates to a method of determining the location of a point in an image using a deep neural network.
Deep learning techniques have improved the performance of generic object detection and segmentation methods when applied to images. These techniques can provide reasonable performance when applied to interpreting images containing people. One objective of interpreting images containing people is pose recognition, or the process of identifying the location of particular parts of the human body (such as shoulders, wrists or hips) and using these locations to define the shape and therefore pose of the person within the image (e.g. standing, lying, which way they are facing, etc.).
In“Joint training of a convolutional network and a graphical model for human pose estimation” (J. J. Tompson, A. Jain,Y. LeCun, and C.Bregler. Advances in neural information processing systems, pages 1799-1807, 2014) a deep neural network is adapted to predict a heat map for each location of a part of a human body in an image, where each location is found as position of a maximal value in the corresponding heat map. In“Stacked hourglass net-works for human pose estimation” (A. Newell, K. Yang, and J. Deng. In European Conference on Computer Vision, pages 483-499. Springer, 2016) an improved network architecture for the deep neural network referred to as a“Stacked Hourglass” is applied which allows features of different scales in the image to be identified in the deep neural network.
In“Realtime multi-person 2d pose estimation using part affinity fields” (Z. Cao, T. Simon, S.-E.Wei, and Y. Sheikh, CVPR, 2017) a n on-parametric representation referred to as “Part Affinity Fields” where a set of confidence maps of a location of a part of the human body together with 2D vector fiel ds that encode the position and orientation of parts of the human body are output by the deep neural network. The above methods were performed on images of a relatively high resolution. In fact, most approaches to pose recognition assume that input images have a relatively large resolution, such that the apparent height of a person is in the order of a few hundred pixels which gives good resolution of different parts of the body. However, in certain
applications the size of a person in the image may be much smaller.
A method of determining the locations of points in images of low resolution would be advantageous.
According to the present invention there is provided a method of determining the location of at least one point in an image, the method comprising: generating a first continuous probability distribution function from the image using the output of a deep neural network; and using the first continuous probability function to determine the location of the point in the image.
According to the present invention there is also provided an image analysis device for determining the location of at least one point in an image, compri sing: a calculator unit configured to receive an output from a neural network and generate a first continuous probability distribution function from the image; and a locator unit configured to receive the first continuous probability distribution function from the calculator and to use the first continuous probability function to determine the location of the point in the image.
The position of the maximum value of the first continuous probability function may indicate the most likely position in the image of the location of the point in the image.
The method may further comprise any one of dividing the image into a plurality of sub images; generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions. The image analysis device may further comprise: an image processing unit configured to divide the image into a plurality of sub images; wherein the calculator unit is configured to generate a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and the calculator unit is configured to calculate the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
Each of the plurali ty of sub images may be associated with at least one of the plurality of second continuous probability distribution functions. Each one of the plurality of sub images may be associated with a different one of the plurality of second continuous probability distribution functions.
Each of the plurality of second continuous probability distributions may be defined by a first distance and a second distance. The first distance may indicate a distance of a peak of the second continuous probability distribution from a point in the sub image associated with the second continuous probability distribution in a first direction. The second distance may indicate a distance of a peak of the second continuous probability distribution from the point in the sub image associated with the second continuous probability distribution in a second direction.
The first direction may be parallel to the x-axis of the image and the second direction may be parallel to the y-axis of the image.
Each of the plurality of second continuous probability distributions may be defined by a first standard deviation and a second standard deviation. The first standard deviation may indicate an uncertainty associated with the first distance and the second standard deviation may indicate an uncertainty associated with the second distance.
At least one of the plurality of second continuous probability distributions may be a Gaussian distribution. The method may further comprise generating a weighting associated with each of the plurality of second continuous probability distribution functions: and the weighting associated with each of the plurality of second continuous probability distribution functions may be applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
The method may further comprise at least one of generating at least one further first continuous probability distribution function associated with the image and using the at least one further first probability function to determine the location of at least one further point in the image.
The location of each of the at least one points in the image may correspond to the location of a part of the human body in the image.
The image may be a low resolution image. A feature associated with the at least one point in the image may have a size of less than or equal to 64 pixels.
The image may be divided into greater than or equal to 2 sub images and less than or equal to 64 sub images. The image may be divided into 16 sub images.
According to the present invention there is provided an image analysis device, comprising an analyser configured to perform the method according to the present invention to determine the location of at least one point in an image.
According to the present invention there is provided a method of training a deep neural network, the method comprising: providing the deep neural network with an image;
generating a first continuous probability distribution function from the image using the output of the deep neural network, using the first continuous probability function to predict the location of a point in the image; comparing the prediction of the location of the point in the image to the location of the point in the image: and adjusting the parameters of the deep neural network based on the result of the comparison.
The prediction of the location of the point in the image to the location of the point in the image may be compared by maximizing the logarithmic probability of the first probability distribution at the location of the point in the image.
The method of training a deep neural network may further comprise any one of dividing the image into a plurality of sub images; generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability functions is associated with at least one of the plurality of sub images; and calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
The method of training a deep neural network may further comprise generating a weighting associated with each of the plurality of second continuous probability distribution functions using the output of the deep neural network; and the weighting associated with each of the plurality of second continuous probability distribution functions may be applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
The comparing of the prediction of the location of the point in the image to the location of the point in the image may further comprise increasing the weighting associated with second continuous probability distributions associated with sub images that are close to the position of the point in the image.
The invention will further be described by way of non-limitative example with reference to the accompanying drawings in which:
Figure 1 illustrates a section of an image containing a person. Figure 2 illustrates a method of determining the location of at least one point in an image according to an embodiment of the present invention.
Figure 3 illustrates an image divided into a plurality of sub-images.
Figure 4 illustrates a schematic diagram of a process performed by a deep neural network according to an embodiment of the present invention.
Figure 5 illustrates a flow chart of a method of generating a probability distribution function according to an embodiment of the present invention.
Figure 6 illustrates a method of training a deep neural network according to an
embodiment of the present invention.
Figure 7 illustrates data comparing the performance of the method of determining the location of at least one point in an image of an embodiment of the present invention to other methods.
Figure 1 illustrates an image 1 which includes a person 100. When the person 100 is visible in the image 1, various different body parts of the person 100 may be visible in the image 1 depending on the pose of the person 100 within the image 1. In the example shown in Figure 1, the person 100 is facing the image 1 such that both legs and arms of the person are visible in the image.
Various points 5 in the image 1 may be identified as points that correspond to parts of the human body of the person 100 visible in the image 1. These points 5 may be referred to as landmark points. In the examples shown in Figure 1, the location of points 5 in the image associated with the left hand 5a, left elbow 5b and chest 5c of the person 100 have been identified. The points 5 identified in the image 1 may then be connected to provide a representation or an estimation of the pose of the person 100 in the image 1. For example, the points 5a, 5b and 5c may be connected to produce a representation or an estimation of the location of the left arm of the person 100 in the image 1.
In the example shown in Figure 1, the location of three points 5 associated with different parts of the human body have been identified. However, the location of any number of points 5 may be identified in the image 1. In the case where the points 5 correspond to parts of the human body, the location of further points 5 such as the head, right arm, right elbow, hips, left and right knees and left and right feet may be identified.
The points 5 that may be identified in the image are not limited to points on the human body, and the location of other categories of points 5 may be identified. For example, the locations of points 5 may be identified that are associated with particular features or buildings in an image 1 of an aerial photograph of an area. In a further example, the locations of points 5 may be identified that are associated with features on a human face in an image 1. For example, the location of human features such as the nose, left and right eyes or mouth on a face may be identified.
The size of a feature within an image 1, such as a person 100, may be measured in the number of pixels that the feature occupies within the image 1. The image 1 may be a low resolution image. For example, the height of the feature in the image 1, such as a person 100, may be 128, 96, 64, 48, 32 or 24 pixels high. The width of the feature in the image 1 may be a corresponding or different number of pixels. The size of the feature may be determined by the height and/or width of the feature.
Figure 2 illustrates a method of determining the location of at least one point 5 in an image 1 according to the present invention. In step 10a, the image 1 is obtained. The image 1 may have been obtained shortly before the method is performed. The location of the at least one point 5 in the image may be determined in real time. The image 1 may be obtained by part of an autonomous driving system. The image 1 may have been obtained at an earlier point in time. In step 10b, a first continuous probability distribution function is generated from the image 1 using the output of a deep neural network. The deep neural network is a deep learning system for analysing data representations such as an image 1. A deep neural network is a network of nodes which may be referred to as neurons. The deep neural network may consist of at least three layers of nodes where each node is linked to at least one node in the layer above and at least one node in the layer below. Each node receives an input from each linked node in the layer below, applies a weight to the input, sums the weighted input, applies a non-linear function to the su and to obtain an output. The output is then provided to each of the linked nodes in the layer above.
The first continuous probability distribution function (CPDF) is a two dimensional continuous function associated with the image 1. The first CPDF may be continuous across the entire area of the image 1 The first CPDF indicates a probability that a point 5 in the image I is located a location in the image 1. The first CPDF has an infinite resolution. The resolution of the first CPDF is therefore unrelated to the resolution of the image 1.
In step 10c, the location of the point 5 in the image is determined using the first CPDF.
The location of the point 5 in the image 1 may be determined as the location corresponding to the maximum probability of the first CPDF. Alternatively, the location of the point 5 in the image 1 may be determined using other features of the first CPDF.
Steps 10b and 10c may be repeated any number of times to determine the location of a plurality of points 5 within the image. When the location of a plurality of points 5 is determined, at least one further first CPDF may be generated from the output of the deep neural network. A first CPDF may be generated for each points 5 to be located within the image 1. For example, when determining a pose of a person within the image 1, the locati on of a plurality of points 5 corresponding to different parts of the human body of a person 100 may be determined. The location of the plurality of points 100 in the image 1 may be used to produce an estimate of the pose of the person 100 within the image 1. The image 11 may be divided into a plurality of sub-images 11, as illustrated in Figure 3.
In the example shown in Figure 3, the image 1 is divided into sixteen sub-images 11. However, the image 1 may be divided into any number of sub-images 11. For example, the image may be divided into greater than or equal to 2 sub images and less than or equal to 64 sub images. If the image 1 is divided into a smaller number of sub-images, the deep neural network may be able to produce an output more quickly. If the image 1 is divided into a larger number of sub-images 11, the deep learning network may be able to produce a more accurate output. The image 1 may be upscaled or downscaled when the image 1 is divided into the plurality of sub images 11.
A second CPDF associated with each sub-image 1 1 of the image 1 may be generated from the output of the deep neural network. One second CPDF may be associated with each of the plurality of sub-images 11 A plurality of second CPDFs may be associated one sub image 11. A second CPDF may not be generated for at least one of the plurality of sub images 11 in the image 1.
The output of the deep neural network will now be discussed in more detail. The output 3 of the deep learning network may include a displacement associated with each second CPDF. The displacement may indicate the distance of the peak 12 or maximum value of the second CPDF from a point in the sub-image 14. The displacement may comprise a first distance parallel to the x-axis of the image 1 and a second distance parallel to the y- axis of the image 1. The location of the point in the sub-image 14 may correspond to the centre of the sub-image 14. The location of the point in the sub-image may 14 also correspond to one of the corners of the sub image 14
The output 3 of the deep neural network may include a first uncertainty associated with the first distance associated with each second CPDF and a second uncertainty associated with the second distance associated with each second CPDF. The uncertainty in the first distance may correspond to a standard deviation of the first distance and the uncertain in the second distance may correspond to a standard deviation of the second distance. Each of the plurality of second CPDFs may be a Gaussian distribution. The first and second distance and the first and second uncertainty are the parameters which may define each of the second CPDFs Each of the plurality of CPDFs may be a different CPDF which may he defined by different parameters that are associated with that CDPF.
The output 3 of the deep neural net rk may include a weighting associated with each second CPDF. The weighting may be used when calculating the first CPDF from the plurality of second CPDFs as described below.
An example of the process of generating the output 3 from the deep neural network is illustrated in Figure 4. An image 1 is provided that contains a person 100. The image 1 is provided to the deep neural network. The deep neural network analyses the image 1 and provides an output 3. The output 3 may include the plurality of first distances mch 3a associated with each of the plurality of second CDPFs. The output 3 may include the plurality of second distances pyn 3b associated with each of the plurality of second CDPFs. The output 3 may include the plurality of first uncertainties sch 3c associated with each of the plurality of second CDPFs. The output 3 may include the plurality of second uncertainties oyn 3d associated with each of the plurality of second CDPFs. The output 3 may include the plurality of weightings W 3e associated with each of the plurality of second CDPFs. In the example shown in figure 4, only tw?o of the each of the first and second distances, uncertainties and the weightings are shown.
The plurality of second CPDFs may be combined to form the first continuous probability distribution. If each of the second CPDFs is a Gaussian distribution, the first CPDF may¬ be defined by a Gaussian mixture model as follows:
Where is x the image 1, F is a function mapping the image x to the point in the image it, v is the point in the sub image, Ak(v; F ) is a displacement, åk(v; F ) is a covariance matrix and pfc(v |F ) is a Gaussian mixture weight at each location v in the feature space iL made up of the plurality of sub images 11. F may be the operation performed by the deep neural network which may be a fully convolutional neural network.
In the example shown in Figure 5, each of the parameters associated with each sub image 11 are obtained from the deep neural network in step 20a and the parameters are combined to generate the first CPDF in step 20b.
An advantage determining the location of at least on point in an image of using a CPDF may be that while the first CPDF may be obtained from the discrete data provided by the pixels of the image 1, the first continuous probability distribution is continuous and therefore has infinite resolution. The method may therefore allow sub-pixel interpolation of the point 5 in the image 1.
The deep learning network may be provided on an analyser. The analyser may form part of an image analysis device which receives the input of an image 1 and determines the position of at least one point 5 within the image 1. Such an image analysis device may be provided in a self-driving system.. The self driving system may acquire images containing pedestrians which may be processed by the image analysis device to output the positions of the body part of the pedestrians within the image 1. The pose of the pedestrians may therefore be determined to allow potential movements of the pedestrians to be predicted.
Figure 6 illustrates a method of training a deep neural network according to the present invention. In the method of training the deep neural network, an image 1 containing at least one point 5 of interest is provided to the deep neural network in step 30a. The image may be one of a plurality of images 1 containing at least one point 5 of interest which have been selected for training the deep neural network. For example, the plurality of images may consist of images of people where the deep learning network is being training to determine points corresponding to locations on the human body within the plurality of images. In step 30b, the image 1 is provided to the deep neural network and a first CPDF is generated from the image 1 using the output of deep neural network. This process is substantially the same as the process of generating the first CPDF that has already been described in the method of determining the location of a point 5 in an image 1 according to the present invention..
In step 30c, the first CPDF may be used to predict the location of a point in the image.
This process is substantially the same as the process of determining the location of a point in the image using the first CPDF described in the method of determining the location of a point 5 in an image 1 according to the present invention..
In step 30d, the predicted location of the point 5 in the image 1 is compared to the actual location of the point 5 in the image 1. For a plurali ty of images 1 each containing a plurality of points 5 of interest this may be compared by optimizing the loss:
Where X-i is one of the plurality of images 1 and uk is the plurality of points 5 in each image 1.
In step 30e, the deep network is adjusted based on the result of the comparison performed in step 30d. For example, the weight applied to the inputs received by each node in the deep neural network may be adjusted by a positive or a negative increment. The adjustment may be performed to gradually minimise the loss £3. The minimization process may be performed by a Stochastic Gradient Descent algorithm.
The method of training the deep neural network may be repeated a large number of times to optimize the accuracy of the output of the deep neural network when used to generate a first CPDF that can be used to determine the location of a point 5 in the image 1. The comparison of the predicted location of the point 5 in the image 1 to the actual location of the point 5 in the image 1 may further include increasing the weighting associated with second continuous probability distributions associated with sub images that are close to the position of the point in the image. For example, the optimization may include a further term:
Where s is a softmax operator, 0’ is a term representing the number of sub images and the Gaussian mixture weight associated with each sub image. Q is a term that may be set to 3.
Figure 7 shows examples of the results of the accuracy of the determination of a location of a plurality of points 5 in an image 1 using the method according to the present invention compared to other methods.
For the evaluation, images from a popular standard human pose benchmark provided in “2d human pose estimation: New benchmark and state of the art analysis” (M. Andriluka,
L. Pishchulin, P. Gehler, and S. Bernt, IEEE Conference on Computer Vision and Pat tern Recognition, CVPR, June 2014). Images are down sampled to create people of predefined sizes from 128 pixels to 24 pixels in height, in order to evaluate the impact of decreasing person size to pose estimation accuracy.
In Figure 7, results of the accuracy of the determination of a location of a plurality of points 5 in an image 1 according to the present invention are compared to the methods for human pose estimation Stacked Hourglass and Part Affinity Field discussed above, using models provided by their authors. In order to make a comparison possible, the input images for the Stacked Hourglass and Part Affinity Field methods is up-sampled to 256 c 256 pixels using bilinear interpolation while maintaining centring of the image. Input images to the method according to the present invention rvere down or up-sampled to 64 x 64 pixels using the same mechanism. The deep neural network was built on the Spatial ConvNet architecture described in“Flowing convnets for human pose estimation in videos” (T. Pfister J. Charles, and A. Zisserman., Proceedings of the IEEE International Conference on Computer Vision, pages 1913-1921, 2015 ).
Graph 50 in figure 7 shows the standard PCKh@0.5metric to measure point detection accuracy, which requires a detection to be within certain distance from the actual point position, where the thresholding distance is given by 50% of the head segment length. The method according to the present invention, shown as the solid line 50a, outperforms both the Stacked Hourglass method, shown as the wide dashed line 50b and the Part Affinity Field method, shown as the narrow dashed line 50c, when the size of the person in the image is below 64 pixels.

Claims

Claims
1. A method of determining the location of at least one point in an image, the method comprising:
generating a first continuous probability distribution function from the image using the output of a deep neural network, and
using the first continuous probability function to determine the location of the point in the image.
2. The method of claim 1, wherein the position of the maximum value of the first continuous probability function indicates the most likely position in the image of the location of the point in the image.
3. The method of claims 1 or 2, further comprising:
dividing the image into a plurality of sub images;
generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and
calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
4. The method of claim 3, wherein each of the plurality of sub images is associated with at least one of the plurality of second continuous probability distribution functions; and
wherein each one of the plurality of sub images is associated with a different one of the plurality of second continuous probability distribution functions.
5. The method of claim 3 or 4, wherein each of the plurality of second continuous probability distributions is defined by a first distance and a second distance; wherein the first distance indicates a distance of a peak of the second continuous probability distribution from a point in the sub image associated with the second continuous probability distribution in a first direction; and
wherein the second distance indicates a distance of a peak of the second continuous probability distribution from the point in the sub image associated with the second continuous probability distribution in a second direction.
6. The method of claim 5, wherein the first direction is parallel to the x-axis of the image and the second direction is parallel to the y-axis of the image.
7. The method of any one of claims 3 to 6, wherein each of the plurality of second continuous probability distributions is defined by a first standard deviation and a second standard deviation;
wherein the first standard deviation indicates an uncertainty associated with the first distance; and
the second standard deviation indicates an uncertainty associated with the second distance.
8. The method of any one of claims 3 to 7, wherein at least one of the plurality of second continuous probability distributions is a Gaussian distribution.
9. The method of claims 3 to 8, wherein the method further comprises generating a weighting associated with each of the plurality of second continuous probability distribution functions; and
the weighting associated with each of the plurality of second continuous probability distribution functions is applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions
10. The method of any one of claim s 1 to 9, further comprising generating at least one further first continuous probability distribution function associated with the image; and using the at least one further first probability function to determine the location of at least one further point in the image
11. The method of any one of claims 1 to 10, wherein the location of each of the at least one points in the image correspond s to the location of a part of the human body in the image.
12. The method of any one of claims 1 to 11, wherein the image is a low resolution image, preferably wherein a feature associated with the at least one point in the image has a size of less than or equal to 64 pixels.
13. The method of any one of claims 1 to 12, wherein the image is divided into greater than or equal to 2 sub images and less than or equal to 64 sub images, preferably wherein the image is divided into 16 sub images.
14. A method of traini ng a deep neural network, the method comprising:
providing the deep neural network with an image;
generating a first continuous probability distribution function from the image using the output of the deep neural network;
using the first continuous probability function to predict the location of a point in the image,
comparing the prediction of the location of the point in the image to the location of the point in the image; and
adjusting the parameters of the deep neural network based on the result of the comparison.
15. The method of claim 14, wherein the prediction of the location of the point in the image to the location of the point in the image is compared by maximizing the logarithmic probability of the first probability distribution at the locati on of the point in the image.
16. The method of claims 14 or 15, further comprising:
dividing the image into a plurality of sub images;
generating a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability functions is associated with at least one of the plurality of sub images; and calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
17. The method of claim 16, wherein the method further comprises generating a weighting associated with each of the plurality of second continuous probabili ty distribution functions using the output of the deep neural network, and
the weighting associated with each of the plurality of second continuous probability distribution functions is applied to the second continuous probability distribution function when calculating the first continuous probability distribution function using the plurality of second continuous probability distribution functions
18 . The method of claim 17, wherein comparing the prediction of the location of the point in the image to the location of the point in the image further comprises increasing the weighting associated with second continuous probability distributions associated with sub images that are close to the position of the point in the image.
19 An image analysis device, comprising:
an analyser configured to perform the method of any one of claims 1 to 13 to determine the location of at least one point in an image.
20. An image analysis device for determining the location of at least one point in an image, comprising:
a calculator unit configured to receive an output from a neural network and generate a first continuous probability distribution function from the image; and
a locator unit configured to receive the first continuous probability distribution function from the calculator and to use the first continuous probability function to determine the location of the point in the image.
21. The image analysis device of claim 20 , further comprising:
an image processing unit configured to divide the image into a plurality of sub images; wherein
the calculator unit is configured to generate a plurality of second continuous probability distribution functions using the output of the deep neural network, wherein each of the plurality of second continuous probability distribution functions is associated with at least one of the plurality of sub images; and
the calculator unit is configured to calculate the first continuous probability distribution function using the plurality of second continuous probability distribution functions.
22. The image analysis device of claims 20 or 21, wherein the location of each of the at least one points in the image determined by the calculator unit corresponds to the location of a part of the human body in the image.
EP19745257.6A 2018-07-26 2019-07-23 Image processing method Withdrawn EP3827371A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GBGB1812206.9A GB201812206D0 (en) 2018-07-26 2018-07-26 Image processing method
PCT/GB2019/052059 WO2020021250A1 (en) 2018-07-26 2019-07-23 Image processing method

Publications (1)

Publication Number Publication Date
EP3827371A1 true EP3827371A1 (en) 2021-06-02

Family

ID=63518178

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19745257.6A Withdrawn EP3827371A1 (en) 2018-07-26 2019-07-23 Image processing method

Country Status (3)

Country Link
EP (1) EP3827371A1 (en)
GB (1) GB201812206D0 (en)
WO (1) WO2020021250A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114139701A (en) * 2021-11-15 2022-03-04 土巴兔集团股份有限公司 Neural network model training method for boundary line extraction and related equipment

Also Published As

Publication number Publication date
GB201812206D0 (en) 2018-09-12
WO2020021250A1 (en) 2020-01-30

Similar Documents

Publication Publication Date Title
US11468697B2 (en) Pedestrian re-identification method based on spatio-temporal joint model of residual attention mechanism and device thereof
CN116188893B (en) Image detection model training and target detection method and device based on BEV
CN110033473B (en) Moving target tracking method based on template matching and depth classification network
CN110472542A (en) A kind of infrared image pedestrian detection method and detection system based on deep learning
CN113609999B (en) Human body model building method based on gesture recognition
Dudek et al. Vision-based robot localization without explicit object models
CN113686314A (en) Monocular water surface target segmentation and monocular distance measurement method of shipborne camera
CN109886356A (en) A target tracking method based on three-branch neural network
CN109063549A (en) High-resolution based on deep neural network is taken photo by plane video moving object detection method
CN117576149A (en) A single target tracking method based on attention mechanism
CN110097599A (en) A kind of workpiece position and orientation estimation method based on partial model expression
CN111914751A (en) Image crowd density identification and detection method and system
Zhang et al. Deep learning framework with Local Sparse Transformer for construction worker detection in 3D with LiDAR
CN110569706A (en) A Deeply Integrated Target Tracking Algorithm Based on Time and Space Network
CN114494594B (en) Deep learning-based astronaut operation equipment state identification method
Faujdar et al. Human pose estimation using artificial intelligence with virtual gym tracker
Anilkumar et al. An adaptive DeepLabv3+ for semantic segmentation of aerial images using improved golden eagle optimization algorithm
CN109461177A (en) A kind of monocular image depth prediction approach neural network based
CN110298402A (en) A kind of small target deteection performance optimization method
CN109345559B (en) A moving target tracking method based on sample augmentation and deep classification network
Oh et al. Local selective vision transformer for depth estimation using a compound eye camera
CN109242882A (en) Visual tracking method, device, medium and equipment
CN112989952A (en) Crowd density estimation method and device based on mask guidance
EP3827371A1 (en) Image processing method
Chen et al. Bridging the gap between explicit and implicit representations: Cross-data association for VSLAM

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201221

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
18W Application withdrawn

Effective date: 20211014