WO2020011014A1 - 实时视频的目标检测与识别方法、系统、存储介质及设备 - Google Patents

实时视频的目标检测与识别方法、系统、存储介质及设备 Download PDF

Info

Publication number
WO2020011014A1
WO2020011014A1 PCT/CN2019/092791 CN2019092791W WO2020011014A1 WO 2020011014 A1 WO2020011014 A1 WO 2020011014A1 CN 2019092791 W CN2019092791 W CN 2019092791W WO 2020011014 A1 WO2020011014 A1 WO 2020011014A1
Authority
WO
WIPO (PCT)
Prior art keywords
target object
frame image
current frame
neural network
position range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/092791
Other languages
English (en)
French (fr)
Inventor
程君
尚海豹
李峰
李昊沅
左小祥
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Publication of WO2020011014A1 publication Critical patent/WO2020011014A1/zh
Priority to US17/012,761 priority Critical patent/US11625921B2/en
Anticipated expiration legal-status Critical
Priority to US18/114,513 priority patent/US12347223B2/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G06V40/113—Recognition of static hand signs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G06V40/176—Dynamic expression
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection

Definitions

  • the present application relates to the field of artificial intelligence and image recognition technologies, and in particular, to a method, system, storage medium, and device for real-time video target detection and recognition.
  • CNN Convolutional Neural Networks
  • AlexNet Alex Network
  • VGG Visual Geometry Group
  • GoogLeNet Google's LeCun Network
  • ResNet Residual Network (Residual Network) model
  • the various network structures described above can basically achieve good results only when the target area of the image accounts for more than 50% of the image. Classification effect. Taking gesture recognition as an example, for the case where the gesture area is relatively small, it is difficult to focus on the gesture area to learn gesture-related features and fail to recognize the corresponding gesture.
  • a processing method is to use a cascaded CNN network to sequentially perform object localization and classification recognition in a two-level CNN network.
  • the two-level neural network operations will occupy a large amount of CPU resources.
  • applications such as synchronous target classification and recognition during real-time video playback Scenarios will place a huge burden on the CPU and place higher requirements on the computing power of the CPU.
  • the huge heat generated by the high-load operation of the CPU is also a problem that the existing hardware of the mobile terminal equipment cannot solve and bear.
  • This application is intended to solve at least one of the technical problems in the related technologies described above. Provided are a method, a system, a storage medium, and a device for real-time video target detection and recognition.
  • a method for detecting and identifying a target of a real-time video is proposed.
  • the method is executed by a computing device and includes:
  • the target object position detection is performed on the current frame image through the first-level neural network to obtain the position range C X of the target object in the current frame image; when two recognition result R X-1 and R X-2 are the same, without passing through the first stage neural network, determining the position of the range C X current image of the target object based on a position of the front range image of the target object C X-1;
  • target object recognition is performed on the current frame image according to the position range C X of the target object in the current frame image, and the target object recognition result Rx of the current frame image is obtained.
  • performing target object position detection on the current frame image through a first-level neural network includes:
  • the position range C X of the target object in the current frame image is obtained through the first-level neural network
  • the interval for determining whether the frame image contains the target object is set to be executed every predetermined number of frames, and the determination is continued through the first-level neural network until the frame image includes the target object.
  • the second-level neural network is used to perform target object recognition on the current frame image according to the position range C X of the target object in the current frame image to obtain the target object recognition result Rx of the current frame image, including:
  • Target object recognition is performed within the extended position range through a second-level neural network to obtain a target object recognition result Rx of the current frame image.
  • C X-1 to determine the current position of the range C X frame image of the target object based on a position of the front range image of the target object comprising:
  • the position range C x of the target object in the current frame image is predicted according to the position range of the target object in the previous frame image and the position range of the target object in the previously predefined frame number of images.
  • the target object is a human hand
  • the step of detecting the position of the target object in the current frame image by using the first-level neural network to obtain the position range C X of the target object in the current frame image includes: identifying the position range of the human hand in the current frame image through the first-level neural network;
  • the second-stage neural network performs target object recognition on the current frame image according to the position range C X of the target object in the current frame image, and obtains the target object recognition result Rx of the current frame image.
  • the method includes: , According to the position range C X of the target object in the current frame image, perform gesture recognition on the human hand in the current frame image, and the recognition result is the gesture gesture.
  • the target object position detection result of the previous frame image can be used to reduce the running frequency of the first-level neural network used for position detection, and accelerate the speed of the entire target object detection and recognition.
  • the CPU and memory resources are reduced, and the performance of the recognition algorithm is improved as a whole.
  • using the method of the present application can save more than 50% of CPU resources compared with the scheme of running the first-level neural network for each frame of image recognition.
  • An embodiment according to the second aspect of the present application provides a target detection and recognition system for real-time video, which includes:
  • the first level of neural network is used to detect the position of the target object on the video frame image to obtain the position range of the target object;
  • a second-level neural network for identifying a target object in a video frame image based on the position range of the target object to obtain a recognition result
  • Neural network operation control module for:
  • the position range of the target object in the current frame image is determined based on the position range C X-1 of the target object in the previous frame without going through the first-level neural network.
  • the second-level neural network is controlled to perform target object recognition on the current frame image according to the position range C X of the target object in the current frame image to obtain the target object recognition result Rx of the current frame image.
  • the neural network when controlling the first-level neural network to perform target object position detection on the current frame image, runs a control module, configured to:
  • the current frame image includes a target object, controlling the first neural network to obtain a position range C X of the target object in the current frame image;
  • the interval for determining whether the frame image includes the target object is set to be executed every predetermined number of frames, and the first-level neural network is controlled to continue to perform the determination until the frame image includes the target object.
  • the neural network when the second-level neural network is controlled to perform target object recognition on the current frame image based on the position range C X of the target object in the current frame image, and the target object in the current frame image obtains the recognition result Rx, the neural network operates Control module for,
  • the second-level neural network is controlled to perform target object recognition within the extended position range to obtain a target object recognition result Rx of the current frame image.
  • the position range C x of the target object in the current frame image is predicted according to the position range of the target object in the previous frame image and the position range of the target object in the previously predefined frame number of images.
  • the target object is a human hand
  • the neural network runs a control module, Configured to control the first-level neural network to recognize a position range of a human hand in a current frame image
  • the neural network operation control module When controlling the second-level neural network, performing target object recognition on the current frame image according to the position range C X of the target object in the current frame image, and obtaining the target object recognition result Rx of the current frame image, the neural network operation control module, It is used to control the second-level neural network to perform gesture recognition on the human hand in the current frame image according to the position range C x of the target object in the current frame image, and the recognition result is the gesture gesture.
  • the target object position detection result of the previous frame image can be used to reduce the running frequency of the first-level neural network used for position detection, and accelerate the speed of the entire target object detection and recognition.
  • the CPU and memory resources are reduced, and the performance of the recognition algorithm is improved as a whole.
  • using the method of the present application can save more than 50% of CPU resources compared with the scheme of running the first-level neural network for each frame of image recognition.
  • an embodiment of the third aspect of the present application provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, real-time according to the first aspect of the present application Object detection and recognition method of video.
  • an embodiment of the fourth aspect of the present application provides a computer program product, and when an instruction in the computer program product is executed by a processor, the target detection of the real-time video according to the first aspect of the present application is implemented. And identification methods.
  • an embodiment of the fifth aspect of the present application provides a computing device including a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor executes the program, A method for detecting and identifying a target of a real-time video according to the first aspect of the present application is implemented.
  • the computer program product and the computing device have similar advantageous effects as the target detection and recognition method and system of real-time video according to the first and second aspects of the present application. , Will not repeat them here.
  • FIG. 1 is a schematic diagram of the working principle of a cascaded neural network for target detection and recognition according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of a method for detecting and identifying a target of a real-time video according to an embodiment of the present application
  • FIG. 3 is a schematic flowchart of a target object position detection process performed by the first God network according to an embodiment of the present application
  • FIG. 4 is a schematic diagram of extending a position range of a target object according to the present application.
  • FIG. 5 is a schematic flowchart of a gesture detection program according to an embodiment of the present application.
  • FIG. 6 is a structural block diagram of a target detection and recognition system for real-time video according to an embodiment of the present application
  • FIG. 7 is a structural block diagram of a neural network operation control module of a real-time video target detection and recognition system according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a computing device according to an embodiment of the present application.
  • the video When performing target detection and recognition of real-time video, the video needs to be split into frames of images, and target detection and recognition is performed on the basis of single-frame images. From the perspective of the limit resolution frequency of the naked eye, the human eye can no longer distinguish the video when the flow rate of the video reaches 30 frames / second or more. Therefore, the frame rate of the current video is generally set to 30 frames / second or less.
  • the first-level neural network uses U-net (U-shaped network, which is a full convolutional neural network FCN (A suitable for image semantic segmentation).
  • the second-level neural network uses the VGG structure as an example.
  • the hardware device uses iphoneX (which belongs to the better performance level in existing mobile terminals) when running real-time video target detection and recognition. , CPU usage can reach about 45%. If a more complex network structure is used, the CPU usage will further increase.
  • the video information processing itself uses a very high CPU. Coupled with the target object recognition, the CPU will always be in a very high load state. For mobile terminals, the continuous high-load operation of the CPU will make the fuselage hot and hot, which is unacceptable to the user or the hardware of the machine most of the time.
  • the main thread may need to be blocked during recognition.
  • the processing of target recognition takes a long time, which will cause the video rendering frame rate to drop, video stuttering and other phenomena, making the target recognition neural network on low-end machines Not available on models.
  • This paper proposes a high-performance method for target detection and recognition. Utilizing the feature that the probability of a large-scale displacement of the target object in adjacent video frames is small, and when the target object does not move between the video frames or the displacement does not affect the recognition, the historical target position range recognition results are used to determine the current frame The target object position range reduces the operation of the previous-stage neural network for position detection, thereby reducing the amount of calculation.
  • FIG. 1 is a schematic diagram of the working principle of a cascaded neural network for target detection and recognition according to an embodiment of the present application.
  • the neural network used for target detection and recognition may include a first-stage neural network 110 (also referred to as a pre-stage network) and a second-stage neural network 120 (also referred to as a post-stage network).
  • the first-level neural network 110 is configured to detect a position of a target object on a video frame image, and obtain a position range of the target object, that is, a region of the target object in the figure.
  • the position detection can be implemented by, for example, semantic segmentation of the image.
  • the output can be the position range of the position detection and recognition result containing the contour information of the target object, and the image area containing the target object can be obtained after cropping and resizing. For example, it can be used to identify a human hand in an image, and determine the position range (the area in the image) of the human hand. Or it is used to recognize a human face and obtain the position range (the area in the image) of the human face.
  • the target object and the human hand or face are used as an example for description.
  • the target object may be any other type of object, for example, the target object may be other torso such as a human foot, or
  • the target object is the entire human body, or the target object can also be a specific animal, plant, house, or other type of object.
  • the type of the target object is not limited in this application.
  • the second-level neural network 120 is configured to identify the details or features of the target object on the video frame image according to the position range (the area in the image) of the target object to obtain the recognition result. For example, gesture classification is performed to obtain gesture types. Or it can be used to recognize human faces and get facial expressions.
  • the image 210 to be identified is an image containing a human hand, and the image 210 may be a frame in a video of a human hand motion.
  • the target object is a human hand, the purpose is to recognize the gesture of the human hand.
  • the human hand (corresponding to 220 in FIG. 1) is obtained.
  • the position range of the target object can be determined according to the contour of the human hand. According to the position range, a sub-image corresponding to the position range of the target object in the image 210 is acquired, and can be used for further recognition of the target object in the next step.
  • the image 210 may be cropped and resized (corresponding to 230 in FIG. 1) to obtain a sub-image 240 suitable for gesture recognition that contains a human hand and the human hand occupies most of the screen area.
  • the sub-image 240 is input to the second neural network 120, and gesture recognition is performed to obtain the processed output image 250 and the output feature vector (represented by a histogram 260) corresponding to the gesture category.
  • the histogram 260 indicates that the meaning of the gesture is "good” or "praise”.
  • the position detection task of the target object may involve distinguishing the target object from the background, and the edge detection of the target object. , May require more convolutional layers, and the amount of calculation required is relatively large. Compared to target detection, the features of classification tasks are easier to extract, so the network structure required by the second neural network is relatively less complicated.
  • gesture recognition experiments have also proven that in the entire gesture recognition process, the time-consuming and CPU consumption of the first-level neural network is about five times that of the second-level neural network. In other words, the main consumption in the process of target detection and recognition is on the first-level neural network.
  • the gestures and expressions may constantly change, but most of the time the position of the hand or face changes little. , Basically not moving or moving slowly.
  • FIG. 2 is a schematic flowchart of a method for detecting and identifying a target of a real-time video according to an embodiment of the present application.
  • the detection and recognition of the target object in each video frame may include steps S100 and S200, where step S100 further includes sub-steps S110 to S130.
  • Both the first-level neural network and the second-level neural network may use a convolutional neural network.
  • training is performed in advance to obtain corresponding model parameters, and calculation is performed according to the trained model parameters in the process of target detection and recognition.
  • the first-level neural network may use various network models suitable for detecting the position of the target object, for example, a full convolutional neural network (FCN) model, which is not limited in this application.
  • FCN full convolutional neural network
  • the first-level neural network may use a U-net model suitable for implementing image semantic segmentation.
  • the second-level neural network can use various network models suitable for target object recognition, for example, AlexNet, VGG, GoogLeNet, and ResNet, etc. There is no limitation in this application.
  • the corresponding network model can be selected according to the identified accuracy requirements and hardware computing capabilities.
  • the second-level neural network can use the VGG model.
  • VGG for target object feature recognition in the related art
  • Various methods using VGG for target object feature recognition in the related art can be used for the second neural network of this application, and there are no special requirements for this, and details are not described again.
  • expression recognition with the rapid development of deep learning technology, the accuracy of expression recognition based on non-display features using deep neural networks has become higher and higher.
  • the use of cascaded two-level neural networks for facial expression recognition has become more and more accurate. Shows very good application prospects. AlexNet, VGG, GoogLeNet, and ResNet can also be used for expression recognition and gesture recognition.
  • the parameters of the first-level neural network can be obtained by model training according to the purpose of target position detection, while the parameters of the second-level neural network can be trained according to the purpose of target recognition, for example, according to adaptive classification, or to model based on labeled data classified by a given category. Get it.
  • step S100 the position range C x of the target object in the current frame image is obtained.
  • the position range C X of the target object may be a range of a regular shape determined according to the outline of the target object, for example, a rectangular range shown by an image 240 in FIG. 1, so as to facilitate further recognition of the target object in subsequent images.
  • the position range C X can be represented by a coordinate range capable of characterizing it. For example, the four vertices of a rectangle, the center and radius of a circle, etc. are mostly used in image processing in a rectangular range.
  • a corresponding sub-image suitable for further recognition by the second neural network can be obtained from the current frame image.
  • Step S100 may be specifically implemented according to the following steps S110 to S130.
  • step S110 it is determined whether the target object recognition result R X-1 of the image of the previous frame of the current frame and the target object recognition result R X-2 of the image of the previous two frames are the same.
  • This step is mainly to estimate the feasibility of using the position detection result of the previous frame image for this frame.
  • the recognition results R X-1 and R X-2 of the previous frame and the previous frame are the same, it usually means that the target object is in a relatively stable state, and there may be no movement or posture change.
  • R X-1 and R X-2 are not the same, the probability of a change in the position of the target object is greater.
  • the recognition result when the gesture changes, the recognition result is naturally different.
  • the recognition result when the position of the target human hand changes, so that the input image of the second-level network does not include the human hand, the recognition result may still be obtained due to the presence of the background object, that is, the position range C X
  • the case where "hand" is not included in -1 may also be reflected in the change of the gesture recognition result R X-1 .
  • step S130 is performed to perform target object position detection on the current frame image through the first-level neural network to obtain the position range of the target object in the current frame image.
  • FIG. 3 is a schematic flowchart of a target object position detection process performed by the first God network according to an embodiment of the present application.
  • the detection of the position of the target object in the current frame image through the first-level neural network may include: Step S131: firstly determine whether the current frame image includes the target object through the first-level neural network; if the current frame image includes the target object, further execute In step S133, the position range C X of the target object in the current frame image is obtained through the first-level neural network; if the current frame image does not include the target object, obviously no further recognition is required. At this time, step S132 is performed. For a new video frame, the step of determining whether there is a hand may be repeatedly performed for each frame or a predetermined number of frames.
  • the human body's action speed is generally not too high.
  • the target object for example, a human hand or a face
  • the interval for determining whether the target object is included in the frame image can be set to be executed every predetermined number of frames, and the determination is continuously performed by the first-level neural network until the target object is included in the frame image. For example, for gesture recognition, you can determine whether there is a hand in the frame image every 10-20 frames.
  • any one of the related art position detection algorithms can be used, which is not limited in this application.
  • step S120 is performed without determining the target object position range C X-1 in the previous frame image without going through the first-level neural network.
  • Target object position range C X Specifically, the position range C X-1 of the target object in the previous frame image may be directly used as the position range C X of the target object in the current frame image. This method is more suitable for situations where the target object is stationary or the motion amplitude is small.
  • the target object may be moving, for example, during gesture recognition, the user may be tired after holding the same position for a long time, and the fatigue can be relieved by changing the position of the hand.
  • the position of the hand in the frame image may exist. Slow change.
  • the head may rotate and move, etc., causing the position of the face to change. Therefore, in some embodiments, the position range C x of the target object in the current frame image may be predicted according to the target object position range in the previous frame image and the target object position range in the previously predefined frame number of images. Based on the continuity of hand movements, a simple linear or low-level polynomial form of the motion path fitting algorithm can obtain better prediction results, thereby obtaining a more accurate target object position range C X.
  • step S140 according to the position of the range C X current image of the target object, the current frame image of the target object identified by the neural network of the second stage, to obtain a current frame image of the target object recognition result Rx.
  • the target object in each frame of the image can be The range of positions is expanded to take a larger range of images as input to the second-level neural network.
  • a box 410 in the inner circle shows a position range C X of the target object in the current frame image, and a range in the outer box represents a new position expanded for the second-level neural network. range.
  • the method of the present application is used for human gesture recognition.
  • the target object is a human hand
  • detecting the position of the target object in the current frame image by using the first-level neural network to obtain the position range C X of the target object in the current frame image includes: The position range in the frame image; the second-level neural network performs target object recognition on the current frame image based on the position range C X of the target object in the current frame image to obtain the target object recognition result Rx of the current frame image, including :
  • Through the second-level neural network according to the position range C X of the target object in the current frame image, perform gesture recognition on the human hand in the current frame image, and the recognition result is the gesture gesture.
  • FIG. 5 is a schematic flowchart of a gesture detection program according to an embodiment of the present application.
  • the variable doFcn is set to determine whether the identification of the first-level neural network needs to be run.
  • the variable doFcn can be set as a binary variable, such as the values "Yes” and "No". Target detection with and without first-level neural networks, respectively.
  • step S400 execution is started.
  • step S410 the value of the doFcn variable is determined before the detection of each frame is started. If the doFcn value is Yes, step S420 is performed, and if the doFcn value is No, step S430 is performed.
  • step S420 doFcn is Yes.
  • a previous network is performed on the frame, that is, the first neural network calculation is used to detect the position of the target object in the frame image.
  • step S422 it is determined whether there is a hand in the current frame image. If there is a hand, step S430 is performed, and if there is no hand, step S424 is performed.
  • step S424 since the first-level neural network determines that there is no hand in the current frame, the doFcn value is set to Yes, and the determination interval is set to be detected every 15 frames, that is, the subsequent detection of the current frame is skipped directly, and this skip Fourteen frames after this frame have passed the pre-stage detection and post-stage pre-stage, and directly feedback to the user that there is no hand in the video. At the fifteenth frame, detection resumes.
  • every 15 frames are taken as an example, and other frame intervals are also possible.
  • step S430 the first-level neural network determines that there is a hand in the current frame, and is calculated by the second-level neural network to perform target recognition within the detected target position range. Using the position range of the target object in the previous frame, the current frame image is cropped to obtain the sub-image corresponding to the position range of the target object. The second-level neural network calculates based on the area of the sub-image and obtains the gesture type recognition result as R X.
  • step S432 doFcn is set to No, and the determination interval is set to be detected every frame, that is, the execution of step S422 is set to make a determination every frame.
  • step S430 executed directly with a doFcn value of No is calculated using a second-level neural network to perform target recognition within the detected target position range.
  • S440 may be further performed to determine whether the doFcn value needs to be modified.
  • step S440 the recognition result of the current frame is compared with the recognition result of the previous frame, so as to predict whether the target position range of the current frame can be used for the next frame and set the corresponding doFcn value.
  • step S444 is performed, and step S442 is not performed at the same time.
  • step S442 the doFcn value is set to Yes, and the execution of step S422 is set to make a judgment every frame.
  • step S44 the doFcn value is set to No, and the execution of step S422 is set to make a judgment every frame.
  • step S450 the processing of the current frame is completed, and it is determined whether a new video frame arrives. If there is, it returns to step S410 to restart the determination of the doFcn value. If there is no new video frame, step S460 is performed to end the recognition process.
  • gestures can be quickly and accurately recognized.
  • the process of facial expression recognition is similar, except that human hands and gestures are used to replace adult faces and expressions.
  • the structure and parameters of the neural network model are adjusted accordingly, but the overall idea is the same, so it will not be described again.
  • the target object position detection result of the previous frame image can be used to reduce the running frequency of the first-level neural network used for position detection, and accelerate the speed of the entire target object detection and recognition.
  • the CPU and memory resources are reduced, and the performance of the recognition algorithm is improved as a whole.
  • using the method of the present application can save more than 50% of CPU resources compared with the scheme of running the first-level neural network for each frame of image recognition.
  • the embodiment of the second aspect of the present application proposes a target detection and recognition system for real-time video.
  • the implementation of the real-time video object detection and recognition system may include one or more computing devices, the computing device including a processor and a memory, and the memory storing computer program instructions including instructions that can be run on the processor.
  • the application program can be divided into a plurality of program modules for corresponding functions of each component of the system. The division of program modules is logical rather than physical. Each program module can run on one or more computing devices, and one computing device can also run one or more program modules. In the following, the system of the present application is described in detail according to the functional logic division of the program module.
  • FIG. 6 is a structural block diagram of a target detection and recognition system for real-time video according to an embodiment of the present application.
  • the system 100 may include a first-level neural network 110, a second-level neural network 120, and a neural network operation control module 130.
  • the real-time video object detection and recognition system 100 may be implemented by using a computing device including a processor and a memory.
  • the memory stores program modules that can be executed by the processor. When each program module is executed, the program module is controlled.
  • the computing device implements the corresponding function.
  • the first-level neural network 110, the second-level neural network 120, and the neural network operation control module 130 may be implemented as such a program module.
  • the first-level neural network 110 is configured to detect a position of a target object on a video frame image, and obtain a position range of the target object.
  • the second-level neural network 120 is configured to recognize a target object in a video frame image according to a position range of the target object to obtain a recognition result.
  • the neural network operation control module 130 is configured to control the operation of the first neural network and the second neural network according to a target object recognition result of a past video frame image.
  • the neural network operation control module 130 is configured to determine whether the target object recognition result R X-1 of the previous frame image of the current frame image is the same as the target object recognition result R X-2 of the image two frames ago; When the recognition results R X-1 and R X-2 are different, control the first-level neural network to detect the position of the target object in the current frame image to obtain the position range C X of the target object in the current frame image; when the two recognition results R X-1 R X-2 and the same, without passing through the first stage neural network 110, C X-1 C X determine the position of the range of the target object in the current frame image based on the position in the previous frame image of the target object range; and a control
  • the second-level neural network 120 performs target object recognition on the current frame image according to the position range C X of the target object in the current frame image to obtain
  • FIG. 7 is a structural block diagram of a neural network operation control module of a real-time video target detection and recognition system according to an embodiment of the present application.
  • the neural network operation control module 130 may further include a recognition result comparison unit 131, a first-stage neural network operation control unit 132, a target position object range determination unit 133, and a second-stage neural network operation control unit 134.
  • the recognition result comparison unit 131 is configured to determine whether the target object recognition result R X-1 of the previous frame image of the current frame image and the target object recognition result R X-2 of the image two frames ago are the same.
  • the first-level neural network operation control unit 132 is configured to control the first-level neural network to detect the target object position of the current frame image when the two recognition results R X-1 and R X-2 are different.
  • controlling the first-level neural network 110 to perform target object position detection on the image of the current frame may include: controlling the first-level neural network 110 to determine whether the current frame image includes the target object; if the current frame image includes the target Object, the first neural network is controlled to further obtain the position range C X of the target object in the current frame image; if the current frame image does not include the target object, the interval for determining whether the target image is included in the frame image is set to every predetermined number of frames Execute it once and control the first-level neural network to continue to make judgments until the frame image contains the target object.
  • the target position object range determination unit 133 is configured to obtain the target object position range C X in the current frame image according to the target object position detection result of the first-level neural network of the current frame or the target object range determined by history.
  • the target position of the object range determination unit 133 when the C X-1 C X determine the position of the range of the target object in the current frame image based on the position in the previous frame image of the target object range, the target position of the object range determination unit 133, for the previous frame
  • the position range C X-1 of the target object in the image is taken as the position range C X of the target object in the current frame image.
  • the position range C x of the target object in the current frame image may be predicted according to the position range of the target object in the previous frame image and the position range of the target object in the previously predefined frame number image.
  • the second-level neural network operation control unit 134 is configured to control the second-level neural network 120 to perform target object recognition on the current frame image according to the position range C X of the target object in the current frame image to obtain target object recognition of the current frame image.
  • the result is Rx.
  • the second-level neural network operation control unit 134 is configured to expand the position range C X of the target object in the current frame image before performing target object recognition on the current frame image, to obtain an area that is increased by a predefined multiple. Extend the position range; and control the second-level neural network to perform target object recognition within the extended position range to obtain the target object recognition result Rx of the current frame image.
  • the target object may be a human hand; when the first-level neural network 110 is controlled to perform target object position detection on the current frame image to obtain the target object position range C X in the current frame image, the first The first-level neural network operation control unit 132 is used to control the first-level neural network to recognize the position range of the human hand in the current frame image; in controlling the second-level neural network 120, according to the position range C x of the target object in the current frame image, When the target object recognition is performed on the current frame image, and the target object recognition result Rx of the current frame image is obtained, the second-level neural network operation control unit 134 is used to control the second-level neural network, and according to the position range of the target object in the current frame image C X : Perform gesture recognition on the human hand in the current frame image, and the recognition result is the gesture gesture.
  • the target object position detection result of the previous frame image can be used to reduce the running frequency of the first-level neural network used for position detection, and accelerate the speed of the entire target object detection and recognition.
  • the CPU and memory resources are reduced, and the performance of the recognition algorithm is improved as a whole.
  • the method of the present application can save more than 50% of CPU resources compared with the scheme of running the first-level neural network for each frame of image recognition.
  • the method or system related to the above embodiments of the present application may be applied to Artificial Intelligence (AI) to realize an AI application scenario based on target detection and recognition, for example, a scenario in which an AI robot interacts with gestures or expressions Wait.
  • AI Artificial Intelligence
  • An embodiment of the third aspect of the present application proposes a non-transitory computer-readable storage medium having executable instructions stored thereon, and when the executable instructions are run on a processor, the implementation as in the embodiment of the first aspect of the present application
  • the method for detecting and identifying a target of a real-time video may be provided on the device as part of the device; or when the device can be remotely controlled by the server, the storage medium may be provided on a remote server that controls the device.
  • the computer instructions used to implement the methods of the present application may be carried in any combination of one or more computer-readable media.
  • the so-called non-transitory computer-readable medium may include any computer-readable medium, except for the signal itself which is temporarily propagated.
  • the computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus, or device.
  • An embodiment of the fourth aspect of the present application provides a computer program product.
  • instructions in the computer program product are executed by a processor, the target detection and recognition of the real-time video according to the embodiment of the first aspect of the present application is implemented. method.
  • the computer program code for performing the operations of this application may be written in one or more programming languages, or a combination thereof, including programming languages such as Java, Smalltalk, C ++, and also conventional Procedural programming language—such as "C" or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, as an independent software package, partly on the user's computer, partly on a remote computer, or entirely on a remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider) Internet connection).
  • LAN local area network
  • WAN wide area network
  • Internet service provider Internet service provider
  • An embodiment of the fifth aspect of the present application provides a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor.
  • the processor executes the program, the program according to the first aspect of the present application is implemented.
  • the method for object detection and recognition of real-time video in one aspect.
  • FIG. 8 illustrates a block diagram of an exemplary computing device suitable for use in implementing embodiments of the present application.
  • the computing device 12 shown in FIG. 8 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.
  • the computing device 12 may be implemented in the form of a general-purpose computing device.
  • the components of the computing device 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
  • the bus 18 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local area bus using any of a variety of bus structures.
  • these architectures include, but are not limited to, Industry Standard Architecture (hereinafter referred to as ISA) bus, Micro Channel Architecture (hereinafter referred to as MAC) bus, enhanced ISA bus, video electronics Standards Association (Video Electronics Standards Association; hereinafter referred to as: VESA) local bus and Peripheral Component Interconnection (hereinafter referred to as: PCI) bus.
  • ISA Industry Standard Architecture
  • MAC Micro Channel Architecture
  • VESA Video Electronics Standards Association
  • PCI Peripheral Component Interconnection
  • Computing device 12 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the computing device 12, including volatile and non-volatile media, removable and non-removable media.
  • the memory 28 may include a computer system-readable medium in the form of volatile memory, such as Random Access Memory (hereinafter referred to as RAM) 30 and / or cache memory 32.
  • RAM Random Access Memory
  • the computing device 12 may further include other removable / non-removable, volatile / nonvolatile computer-readable storage media.
  • the storage system 34 may be used to read and write non-removable, non-volatile magnetic media (not shown in the figure, commonly referred to as a "hard drive").
  • a disk drive for reading and writing to a removable non-volatile disk (for example, a “floppy disk”) and a removable non-volatile optical disk (for example, a compact disk read-only memory (Compact Disc ReadRead Only Memory (hereinafter referred to as: CD-ROM), Digital Video Disc (Digital Video Read Read Only Memory; hereinafter referred to as: DVD-ROM) or other optical media) read and write optical disc drive.
  • each drive may be connected to the bus 18 through one or more data medium interfaces.
  • the memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments of the present application.
  • a program / utility tool 40 having a set (at least one) of program modules 42 may be stored in, for example, the memory 28.
  • Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data Each of these examples, or some combination, may include an implementation of a network environment.
  • the program module 42 generally performs functions and / or methods in the embodiments described in this application.
  • the computing device 12 may also communicate with one or more external devices 14 (such as a keyboard, pointing device, display 24, etc.), and may also communicate with one or more devices that enable users to interact with the computer system / server 12, and / Or communicate with any device (such as a network card, modem, etc.) that enables the computer system / server 12 to communicate with one or more other computing devices. This communication can be performed through an input / output (I / O) interface 22.
  • the computing device 12 may also communicate with one or more networks (such as a local area network (hereinafter referred to as a LAN), a wide area network (hereinafter referred to as a WAN), and / or a public network such as the Internet through the network adapter 20 ) Communication.
  • networks such as a local area network (hereinafter referred to as a LAN), a wide area network (hereinafter referred to as a WAN), and / or a public network such as the Internet through the network adapter 20 ) Communication.
  • the network adapter 20 communicates with other modules of the computing device 12 via the bus 18. It should be noted that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, Tape drives and data backup storage systems.
  • the processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing the methods mentioned in the foregoing embodiments.
  • the non-transitory computer-readable storage medium, computer program product, and computing device according to the third to fifth aspects of the present application can be implemented with reference to the content specifically described in the embodiment of the first aspect of the present application, and have the same characteristics as those according to the first aspect of the present application.
  • the beneficial effects of the real-time video target detection and recognition method are similar, and are not repeated here.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as “first” and “second” may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of “multiple” is two or more, such as two, three, etc., unless it is specifically and specifically defined otherwise.
  • any process or method description in a flowchart or otherwise described herein may be understood to mean an instruction that includes one or more executable instructions for implementing a particular logical function or process step.
  • Modules, snippets or parts of code, and the scope of the preferred embodiments of this application includes additional implementations, which may not be in the order shown or discussed, including in a substantially simultaneous manner or in the reverse order according to the functions involved To perform functions, which should be understood by those skilled in the art to which the embodiments of the present application belong.
  • a sequenced list of executable instructions that can be considered to implement a logical function can be embodied in any computer-readable medium,
  • the instruction execution system, device, or device such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from the instruction execution system, device, or device), or combine these instruction execution systems, devices, or devices Or equipment.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
  • each part of the application may be implemented by hardware, software, firmware, or a combination thereof.
  • multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits, application specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Human Computer Interaction (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

本申请提出了一种用于实时视频的目标检测与识别的方法和系统。方法包括：确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；不相同时，通过第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X；相同时，根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X；以及通过第二级神经网络，根据位置范围C X，对当前帧图像进行目标物体识别。从而，能够降低用于位置检测的第一级神经网络的运行频率，加快识别速度并降低CPU和内存资源的占用。

Description

实时视频的目标检测与识别方法、系统、存储介质及设备
本申请要求于2018年07月13日提交的申请号为201810771226.X、发明名称为“一种实时视频的目标检测与识别方法和系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能和图像识别技术领域，尤其涉及一种实时视频的目标检测与识别方法、系统、存储介质及设备。
背景技术
随着识别精度地不断提高，用CNN(Convolutional Neural Networks，卷积神经网络)来进行图像识别和分类的方法已逐渐被业界接受。在评价模型或者算法的性能时，则多采用ImageNet(图像网络)数据集作为测试数据集。近年来，各种新的CNN网络结构被不断提出，从2012年的AlexNet(Alex网络)模型，到2014年的VGG(VisualGeometryGroup，视觉几何组)和随后的GoogLeNet(谷歌的LeCun网络)和ResNet(Residual Network，残差网络)模型，CNN在最初的LeNet(LeCun网络)基础上取得了长足进展，ImageNet数据集的最佳定位和分类成绩也在不断刷新。ImageNet大规模视觉识别挑战赛(ILSVRC)的历年成绩中，top-5的错误率从2012年AlexNet的16.4％逐年下降到2015年ResNet的3.57％，在2016年更进一步下降到3％以下。与模型精度的提高向对应的，是网络结构的深度也在不断增加，ILSVRC 2012时，AlexNet只有8层，到ILSVRC 2015，ResNet网络深度已经达到152层，当前ResNet的层数更是已经可以达到10^3的数量级。虽然比起AlexNet和VGG，GoogLeNet和ResNet的全连接层数和全连接层的节点数量都有所减少，但由于层数的大量增加，总的模型运算量还是远增。
在目标定位和分类联合应用的场景(例如，手势识别这样的典型场景)，上述介绍的各种网络结构目前基本都只能在图像目标区域占图像比例大于50％的情况下才能取得较好的分类效果。以手势识别为例，对于手势区域占比较小的情况，就很难聚焦到手势区域学习到手势相关特征，无法识别出相应的手势。
为了解决这一问题，一种处理方式是使用级联的CNN网络，在两级CNN网络中依次进行物体定位和分类识别。为了取得更好的识别效果，随着神经网络的深度增加，运算量也逐步加大，两级神经网络的运算将占用大量CPU资源，对于实时视频播放过程中的同步目标分类和识别这样的应用场景，将会给CPU带来巨大负担，对CPU的运算能力提出了更高的要求。此外，在移动终端的应用中，CPU高负荷运行产生的巨大热量也几乎是现有移动终端设备的硬件无法解决和承受的问题。
发明内容
本申请旨在至少在一定程度上解决上述相关技术中的技术问题之一。提供一种实时视频的目标检测与识别方法、系统、存储介质及设备。
为了达到上述目的，根据本申请第一方面的实施例提出了一种实时视频的目标检测与识别方法，该方法由计算设备执行，其包括：
确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；
当两个识别结果R X-1和R X-2不相同时，通过第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X；当两个识别结果R X-1和R X-2相同时，不经过第一级神经网络，根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X；
通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
在一些实施例中，通过第一级神经网络对当前帧图像进行目标物体位置检测，包括：
通过第一级神经网络判断当前帧图像中是否包含目标物体；
若当前帧图像包含目标物体，则通过第一级神经网络获取当前帧图像中目标物体的位置范围C X；
若当前帧图像不含目标物体，则将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并通过第一级神经网络持续进行判断直至帧图像中包含目标物体。
在一些实施例中，通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别 结果Rx，包括：
将当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；
通过第二级神经网络在所述扩展位置范围内进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
在一些实施例中，根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X，包括：
将前一帧图像中目标物体的位置范围C X-1作为当前帧图像中目标物体的位置范围C X；或者
根据前一帧图像中目标物体的位置范围，以及之前预定义帧数的图像中目标物体的位置范围，预测当前帧图像中目标物体的位置范围C X。
在一些实施例中，所述目标物体为人手；
所述通过第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X包括：通过第一级神经网络识别人手在当前帧图像中的位置范围；
所述通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx，包括：通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
使用本申请的实时视频的目标检测与识别方法，能够利用前帧图像的目标物体位置检测结果，降低用于位置检测的第一级神经网络的运行频率，加快整个目标物体检测和识别的速度并同时降低CPU和内存资源的占用，从整体上改善识别算法的性能。在两级神经网络分别采用相同结构的前提下，使用本申请的方法与每帧图像的识别均运行第一级神经网络的方案相比，可节约CPU资源50％以上。
根据本申请第二方面的实施例提供了一种实时视频的目标检测与识别系统，其包括：
第一级神经网络，用于对视频帧图像进行目标物体的位置检测，获取目标物体位置范围；
第二级神经网络，用于根据目标物体位置范围，在视频帧图像中进行目标物体的识别，得到识别结果；以及
神经网络运行控制模块，用于，
确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；
当两个识别结果R X-1和R X-2不相同时，控制第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X；
当两个识别结果R X-1和R X-2相同时，不经过第一级神经网络，根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X；
控制第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
在一些实施例中，在控制第一级神经网络对当前帧图像进行目标物体位置检测时，神经网络运行控制模块，用于，
控制所述第一级神经网络判断当前帧图像中是否包含目标物体；
若当前帧图像包含目标物体，则控制第一神经网络获取当前帧图像中目标物体的位置范围C X；
若当前帧图像不含目标物体，则将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并控制第一级神经网络持续进行判断直至帧图像中包含目标物体。
在一些实施例中，在控制第二级神经网络根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，当前帧图像的目标物体得到识别结果Rx时，神经网络运行控制模块，用于，
将当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；
控制第二级神经网络在所述扩展位置范围内进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
在一些实施例中，在根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X时，神经网络运行控制模块，用于，
将前一帧图像中目标物体的位置范围C X-1作为当前帧图像中目标物体的位置范围C X；或者
根据前一帧图像中目标物体的位置范围，以及之前预定义帧数的图像中目标物体的位置范围，预测当前帧图像中目标物体的位置范围C X。
在一些实施例中，所述目标物体为人手；在控制第一级神经网络对当前帧 图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X时，神经网络运行控制模块，用于控制所述第一级神经网络识别人手在当前帧图像中的位置范围；
在控制第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx时，所述神经网络运行控制模块，用于控制所述第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
使用本申请的实时视频的目标检测与识别系统，能够利用前帧图像的目标物体位置检测结果，降低用于位置检测的第一级神经网络的运行频率，加快整个目标物体检测和识别的速度并同时降低CPU和内存资源的占用，从整体上改善识别算法的性能。在两级神经网络分别采用相同结构的前提下，使用本申请的方法与每帧图像的识别均运行第一级神经网络的方案相比，可节约CPU资源50％以上。
为了实现上述目的，本申请第三方面实施例提供了一种非临时性计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现根据本申请第一方面所述的实时视频的目标检测与识别方法。
为了实现上述目的，本申请第四方面实施例提供了一种计算机程序产品，当所述计算机程序产品中的指令由处理器执行时，实现根据本申请第一方面所述的实时视频的目标检测与识别方法。
为了实现上述目的，本申请第五方面实施例提供了一种计算设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现根据本申请第一方面所述的实时视频的目标检测与识别方法。
根据本申请第三到五方面的非临时性计算机可读存储介质，计算机程序产品和计算设备具有与根据本申请第一和第二方面的实时视频的目标检测与识别方法和系统类似的有益效果，在此不再赘述。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1是根据本申请实施例的目标检测与识别的级联神经网络工作原理示意图；
图2是根据本申请实施例的实时视频的目标检测与识别方法的流程示意图；
图3是根据本申请实施例的第一神网络进行目标物体位置检测的流程示意图；
图4是根据本申请的对目标物体位置范围进行扩展的示意图；
图5是根据本申请实施例的手势检测程序实现流程示意图；
图6是根据本申请实施例的实时视频的目标检测与识别系统的结构框图；
图7是根据本申请实施例的实时视频的目标检测与识别系统的神经网络运行控制模块结构框图；
图8是根据本申请实施例的计算设备的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
在进行实时视频的目标检测与识别之时，需要将视频拆分成一帧帧的图像，在单帧图像的基础上进行目标检测与识别。从肉眼的极限分辨频率角度来说，视频的流速达到30帧/秒以上时人眼就已经无法进一步分辨，因此，目前视频的帧率一般都会设置在30帧/秒或以下。
以30帧/秒为例，经测试，按照每帧图像通过两级CNN神经网络进行目标检测和识别的方法，第一级神经网络使用U-net(U形网络，是全卷积神经网络FCN的一种，适于图像语义分割)，第二级神经网络使用VGG结构为例，硬件设备选用iphoneX(属于现有移动终端中性能较佳的那一个级别)，运行实时视频目标检测和识别时，CPU的占用率可达45％左右。如果采用更复杂的网络结构，CPU的占用率还会进一步增加。
对于边播放视频，边通过摄像头拍摄手势识别用户评价这样的应用场景，或者是视频直播，同时对手势、表情或者其他目标进行识别的场景，由于视频信息处理本身对CPU的使用就非常高，再加上目标物体识别所占用的，CPU会一直处于一种负载非常高的状态。对于移动终端来说，CPU持续高负荷运行会 使机身发热发烫，无论对于用户还是机器的硬件本身大部分时候都是无法接受的。
并且，为了使目标识别结果和视频帧同步，进行识别时也可能会需要阻塞主线程。对于计算能力比较低的中低端设备，由于CPU的算力较差，处理目标识别时间较长，会导致视频渲染帧率下降，视频卡顿等现象，使得目标识别的神经网络在低端机型上无法使用。
为了解决相关技术中的目标识别方案CPU占用高，识别时间长的问题。本文提出一种高性能的目标检测与识别的方法。利用相邻视频帧中，目标物体的大范围位移概率较小的特点，在视频帧之间目标物体不发生位移或者位移不影响识别时，利用历史上的目标物体位置范围识别结果来确定当前帧的目标物体位置范围，减少用于位置检测的前级神经网络的运行，从而降低计算量。
下面参考附图对本申请实施例的方法和系统进行详细的说明。
为了更好地理解本申请，首先介绍用级联的两级神经网络进行目标检测与识别的工作原理。图1是根据本申请实施例的目标检测与识别的级联神经网络工作原理示意图。其中，用于目标检测与识别的神经网络可包括第一级神经网络110(也称为前级网络)和第二级神经网络120(也称为后级网络)。
第一级神经网络110，用于对视频帧图像进行目标物体的位置检测，获取目标物体位置范围，即目标物体在图中的区域。位置检测可以通过例如对图像进行语义分割等方式实现。其输出可以是包含目标物体的轮廓信息的位置检测识别结果位置范围，经过裁切和调整大小后得到包含目标物体的图像区域。例如，可以用来识别图像中的人手，并确定人手的位置范围(在图像中的所在区域)。或者是用来识别人脸，得到人脸的位置范围(在图像中的所在区域)。
在本申请实施例中，仅以目标物体和人手或人脸为例进行说明，除此之外，目标物体也可以是其它任意类型的物体，比如，目标物体可以是人脚等其它躯干，或者，目标物体是整个的人体，或者，目标物体也可以是特定的动物、植物、房屋等其它类型物体。对于目标物体的类型，本申请不做限定。
第二级神经网络120，用于根据目标物体位置范围(在图像中的所在区域)，对视频帧图像进行目标物体的细节或者特征的识别，得到识别结果。例如，进行手势分类，得到手势类型。或者是用来识别人脸，得到人脸的表情等。
参见图1，其中，待识别的图像210为一幅包含人手的图像，图像210可以是一段人手动作的视频中的一帧。目标物体为人手，目的是识别人手的手势。 首先经过第一级神经网络110，通过语义分割进行目标位置检测，得到图像中目标物体——人手的轮廓(对应图1中220)。根据人手的轮廓可确定目标物体的位置范围。根据位置范围，获取图像210中对应目标物体位置范围的子图像，即可用于下一步的目标物体的进一步识别。例如，可对图像210进行裁切和调整大小(对应图1中230)，得到含有人手，且人手占据绝大部分画面面积的适于手势识别的子图像240。之后，将子图像240输入到第二神经网络120，进行手势识别，得到处理后的输出图像250和输出的对应手势类别的特征向量(以柱状图260表示)，例如，在本实施例中，该柱状图260表示的是手势的含义是“好”或者“称赞”。
通过对第一级神经网络和第二级神经网络运行的分析，就目前常用的几种CNN结构来说，目标物体的位置检测任务由于可能涉及区分目标物体和背景，以及目标物体的边缘检测等，可能需要较多的卷积层，所需的计算量也相对较大。相对于目标检测，分类任务的特征更加容易提取，因此第二神经网络所需的网络结构相对没有那么复杂。以手势识别为例，实验也证明，在整个手势识别过程中，第一级神经网络的耗时和CPU占用约是第二级神经网络的五倍。也就是说，目标检测和识别过程中的主要消耗是在第一级神经网络上。
在实际场景中，例如利用移动终端进行直播时，用户在对着手机比手势或者做表情的时候，手势和表情可能会不断变换，但是手的位置或脸的位置大部分时间是变化很小的，基本不动或者缓慢移动，对于前级网络，可以参用上一次前级的结果对当前帧的手或脸的范围进行估计，并用后级网络计算出的结果对估计的范围进行有效性验证。
图2是根据本申请实施例的实时视频的目标检测与识别方法的流程示意图。其中，对各个视频帧中目标物体的检测与识别，可以包括步骤S100和S200，其中步骤S100进一步包括子步骤S110到S130。
第一级神经网络和第二级神经网络可以均使用卷积神经网络。并且预先进行训练，得到相应的模型参数后，在进行目标检测与识别的过程中直接根据训练好的模型参数进行计算。
其中，第一级神经网络可以使用各种适用于进行目标物体位置检测的网络模型，例如，全卷积神经网络(FCN)模型，本申请对此没有限制。在一些实施例中，第一级神经网络可以使用适于实现图像语义分割的U-net模型。
关于使用FCN以及U-net实现视频帧图像语义分割的具体方式，与相关技 术中的图像语义分割实现类似，在此不再赘述。
第二级神经网络可以使用各种适用于进行目标物体识别的网络模型，例如，AlexNet、VGG、GoogLeNet和ResNet等，本申请对此没有限制。可以根据识别的精度需求和硬件计算能力选择相应的网络模型。
例如，对于手势识别，综合考虑模型复杂度和识别精度的需求，第二级神经网络可以使用VGG模型。相关技术中使用VGG进行目标物体特征识别的各种方法均可以用于本申请的第二神经网络，对此没有特殊的要求，也不再赘述细节。对于表情识别，随着深度学习技术的飞速发展，使用深度神经网络进行基于非显示特征的表情识别的精确度已经越来越高，直接使用级联的两级神经网络进行人脸的表情识别已展现出非常好的应用前景。AlexNet、VGG、GoogLeNet和ResNet等网络结构也均可用于表情识别和手势识别。
第一级神经网络参数可按照目标位置检测目的进行模型训练得到，而第二级神经网络的参数可按照目标识别的目的训练，例如按照自适应分类，或者根据给定类别分类的标注数据进行模型训练得到。
在进行实时视频的目标检测与识别的过程中，对于拆分的每幅视频帧图像，首先，在步骤S100，获取当前帧图像中目标物体的位置范围C X。
所述目标物体的位置范围C X可以是根据目标物体的轮廓确定的一个规则形状的范围，例如，图1中图像240所示的矩形范围，以便于后续的图像中目标物体的进一步识别。位置范围C X可以用能够表征其的坐标范围来表示。例如，矩形的4个顶点，圆形的圆心和半径等，在图像处理中，以矩形范围的使用居多。根据目标物体的位置范围C X即可以从当前帧图像中获得相应的适用于第二神经网络进一步识别的子图像。
由于相邻视频帧中，目标物体的位置较接近或者不变的概率很大，因此，可根据当前帧之前的历史帧图像的识别结果来确定是否需要运行第一神经网络进行位置检测，从而节约算力。步骤S100可以具体按照下述步骤S110到S130实现。
在步骤S110，确定当前帧的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同。
这一步骤主要为了估计将上一帧图像的位置检测结果用于本帧的可行性。前一帧和之前的一帧的识别结果R X-1和R X-2相同时，通常意味着目标物体处于较稳定的状态，可能没有发生移动和姿势变化。而R X-1和R X-2不相同时，则目 标物体位置发生变化的概率较大。
以手势识别为例，当手势发生变化时，识别结果自然是不同的。除此之外，当目标人手位置发生了变化，而使得第二级网络的输入图像中不含人手时，可能由于背景物体的存在，仍旧可能得出识别结果，也就是说，位置范围C X-1内不含“手”的情况，也可能反映在手势识别结果R X-1的变化上。
因此，当两个识别结果R X-1和R X-2不相同时，执行步骤S130，通过第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X。
参见图3，图3是根据本申请实施例的第一神网络进行目标物体位置检测的流程示意图。通过第一级神经网络对当前帧图像进行目标物体位置检测，可包括：步骤S131，首先通过第一级神经网络判断当前帧图像中是否包含目标物体；若当前帧图像包含目标物体，则进一步执行步骤S133，通过第一级神经网络获取当前帧图像中目标物体的位置范围C X；若当前帧图像不含目标物体，则显然无需进一步的识别。此时，执行步骤S132，对于新的视频帧，可对于每帧或预定帧数重复执行判断是否有手的步骤。
考虑到对于手势识别或者表情识别等应用场景，人体的动作速度一般不会过高，按照30帧/秒的帧率，如果当前帧中不含目标物体(例如，人手或人脸)，可能随后的若干帧中，仍然是不含目标物体的。因此，可将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并通过第一级神经网络持续进行判断直至帧图像中包含目标物体。例如，对于手势识别，可每隔10-20帧判断一次帧图像中是否有手。
至于的具体的位置检测过程，则可以使用任意一种相关技术中的位置检测算法，本申请对此没有限定。
当两个识别结果R X-1和R X-2相同时，执行步骤S120，不经过第一级神经网络，根据前一帧图像中的目标物体位置范围C X-1确定当前帧图像中的目标物体位置范围C X。具体而言，可以直接将前一帧图像中目标物体的位置范围C X-1作为当前帧图像中目标物体的位置范围C X。此种方式比较适合目标物体静止或者运动幅度微小的情况。
考虑到有时目标物体可能是移动的，例如手势识别时，用户可能长时间手部保持同一位置会疲惫，可以通过改变手部位置缓解疲劳，此时手在帧图像中的位置则可能是存在较缓慢的变动。或者是，表情识别时，头部可能会转动移 动等，导致人脸位置的改变。因此，在一些实施例中，可根据上一帧图像中的目标物体位置范围，以及之前预定义帧数的图像中的目标物体位置范围，预测当前帧图像中目标物体的位置范围C X。基于手部动作的连续性，简单的线性或者低级多项式形式的移动路径拟合算法，即可得到较好的预测效果，从而获取较准确的目标物体位置范围C X。
在步骤S140，通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
考虑到目标物体在两帧之间可能发生的小范围移动，为了使利用前帧的位置检测检测结果来预测本帧中目标物体的位置范围的准确度更高，可以将每帧图像中目标物体位置范围进行扩展，以一个更大范围的图像来作为第二级神经网络的输入。
即，在通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别之前，将当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；并通过第二级神经网络，在所述扩展位置范围内进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
参见图4，其中内圈的方框410中示出的是当前帧图像中目标物体的位置范围C X，外圈的方框内的范围表示扩展后用于第二级神经网络的新的位置范围。
在一些实施例中，本申请的方法被用于人的手势识别。其中，上述目标物体为人手；所述通过第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X包括：通过第一级神经网络识别人手在当前帧图像中的位置范围；所述通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx，包括：通过第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
下面结合图5，对使用本申请的方法进行手势识别的实施例进行详细说明。图5是根据本申请实施例的手势检测程序实现流程示意图。其中，设置变量doFcn用于判断是否需要运行第一级神经网络的标识，doFcn变量可设为二值变量，例如取值“Yes”(是)和“No”(否)。分别对应执行和不执行第一级神经网络的目标检测。
在步骤S400，开始执行。
步骤S410，在每帧检测开始前，先对doFcn变量的值进行判断。如果doFcn值为Yes则执行步骤S420，如果doFcn值为No则执行步骤S430。
一般来说，doFcn的值可根据上一帧后级网络和上上一帧的第二级神经网络的识别结果而定。假设上一帧后级网络的识别结果为R X-1，上上一帧的后级网络的识别R X-2，如果R X-1不等于R X-2，则doFcn＝Yes；相等则doFcn＝No，第一帧时doFcn＝Yes。
步骤S420，doFcn为Yes，先对此帧执行前级网络，即使用第一神经网络计算，进行帧图像中目标物体的位置检测。
首先，在步骤S422，判断当前帧图像中是否有手。如果有手，则执行步骤S430，无手则执行步骤S424。
步骤S424，由于第一级神经网络判断当前帧中没手，则将doFcn值为Yes，并将判定间隔设置为每15帧检测一次，即直接跳过当前帧的后级检测，并此跳过此帧过后的十四帧的前级检测和后级前级，并直接反馈给用户图视频中没有手。第十五帧开始重新恢复检测。当然，本实施例以每隔15帧为例，其它的帧间隔也是可以的。
在步骤S430，第一级神经网络判断当前帧中有手，则由第二级神经网络计算，在检测到的目标位置范围内，进行目标识别。利用上一帧中目标物体的位置范围，对当前帧图像进行裁剪，得到目标物体位置范围对应的子图像，第二级神经网络根据子图像的区域进行计算，并得出手势类型识别结果为R X。
之后，由于“手”已经进入了视频帧的范围内，不必再以大的帧间隔进行检测。因此，在接下来的步骤S432，将doFcn置为No，并将判定间隔设置为每帧都检测，也就是将步骤S422的执行设置为每帧进行一次判断。
如果是在步骤S410中，doFcn值为No而直接执行的步骤S430，使用第二级神经网络计算，在检测到的目标位置范围内，进行目标识别。则执行之后，可以进一步执行S440，判断doFcn值是否需要修改。
在步骤S440，将当前帧的识别结果与上一帧的识别结果进行比较，从而用于预测下一帧是否可以使用当前帧的目标位置范围和设置相应的doFcn值。在当前帧的识别结果与上一帧的识别结果相同时，执行步骤S444，不同时执行步骤S442。
在步骤S442，将doFcn值设置为Yes，并将步骤S422的执行设置为每帧进 行一次判断。
在步骤S44，将doFcn值设置为No，并将步骤S422的执行设置为每帧进行一次判断。
在步骤S450，完成对当前帧的处理，判断是否有新的视频帧到达，如果有，则返回步骤S410，重新开始判断doFcn值。如果没有新的视频帧，则执行步骤S460，结束识别过程。
通过上述步骤，可以对手势进行快速准确的识别。进行人脸的表情识别的过程与此类似，只是在将人手和手势替换成人脸和表情，神经网络模型的结构和参数进行相应调整，但是整体的思路是一致的，因此不再赘述。
使用本申请的实时视频的目标检测与识别方法，能够利用前帧图像的目标物体位置检测结果，降低用于位置检测的第一级神经网络的运行频率，加快整个目标物体检测和识别的速度并同时降低CPU和内存资源的占用，从整体上改善识别算法的性能。在两级神经网络分别采用相同结构的前提下，使用本申请的方法与每帧图像的识别均运行第一级神经网络的方案相比，可节约CPU资源50％以上。
以手势识别为例，按照每秒30帧的帧率，设对于每帧图像，第一级神经网络的执行耗时为T1，第二级神经网络的执行耗时为T2，T1＝5T2。则，在出现视频帧内无手的情况时，使用本申请方法与不使用的情况的耗时比为：2*T1/30*(T1+T2)＝2*10*T2/30*(5T2+T2)＝11.11％，相应地，可视为其计算量和CPU占用上约为使用本方法之前的百分之11.11。
对于视频帧内有手的情况，经统计，在手势变换和正常速度移动情况下，手势位置的预测在每3帧中大约可以命中2帧，使用本申请方法与不使用的情况的耗时比为：(10*T1+30*T2)/30*(T1+T2)＝60*T2/30*(5T2+T2)＝44.44％，相应地，可视为其计算量和CPU占用上约为使用本方法之前的百分之44.44。
为了实现上述第一方面实施例中的方法，本申请第二方面的实施例提出了一种实时视频的目标检测与识别系统。
所述实时视频的目标检测与识别系统的实现可包括一个或多个计算设备，所述计算设备包括处理器和存储器，所述存储器上存储有包括可在所述处理器上运行的计算机程序指令的应用程序。所述应用程序可以划分为多个程序模块，用于系统各个组成部分的相应功能。其中，程序的模块的划分是逻辑上的而非物理上的，每个程序模块可以运行在一个或多个计算设备上，一个计算设备上 也可以运行一个或一个以上的程序模块。以下对本申请的系统按照程序模块的功能逻辑划分进行详细说明。
图6是根据本申请实施例的实时视频的目标检测与识别系统结构框图。其中，根据本申请的系统100可以包括第一级神经网络110，第二级神经网络120和神经网络运行控制模块130。
所述实时视频的目标检测与识别系统100可以采用包括处理器和存储器的计算设备实现，所述存储器中存储有可被所述处理器执行的程序模块，各个程序模块被执行时，控制所述计算设备实现相应的功能。例如，第一级神经网络110，第二级神经网络120和神经网络运行控制模块130可以实现为这样的程序模块。
第一级神经网络110，用于对视频帧图像进行目标物体的位置检测，获取目标物体位置范围。
第二级神经网络120，用于根据目标物体位置范围，在视频帧图像中进行目标物体的识别，得到识别结果。
神经网络运行控制模块130，用于根据过往视频帧图像的目标物体识别结果控制第一神经网络和第二神经网络工作。其中，神经网络运行控制模块130，用于确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；当两个识别结果R X-1和R X-2不相同时，控制第一级神经网络对当前帧图像进行目标物体位置检测，获取当前帧图像中目标物体的位置范围C X；当两个识别结果R X-1和R X-2相同时，不经过第一级神经网络110，根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X；以及控制第二级神经网络120，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
参见图7，图7是根据本申请实施例的实时视频的目标检测与识别系统的神经网络运行控制模块结构框图。神经网络运行控制模块130可进一步包括：识别结果比较单元131、第一级神经网络运行控制单元132、目标位置物体范围确定单元133和第二级神经网络运行控制单元134。
识别结果比较单元131，用于确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同。
第一级神经网络运行控制单元132，用于当两个识别结果R X-1和R X-2不相同 时，控制第一级神经网络对当前帧图像进行目标物体位置检测。
在一些实施例中，控制第一级神经网络110对当前帧的图像进行目标物体位置检测，可包括：控制第一级神经网络110判断当前帧图像中是否包含目标物体；若当前帧图像包含目标物体，则控制第一神经网络进一步获取当前帧图像中目标物体的位置范围C X；若当前帧图像不含目标物体，则将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并控制第一级神经网络持续进行判断直至帧图像中包含目标物体。
目标位置物体范围确定单元133，用于根据当前帧第一级神经网络的目标物体位置检测结果或者历史确定的目标物体范围，获取当前帧图像中的目标物体位置范围C X。
在一些实施例中，在根据前一帧图像中目标物体的位置范围C X-1确定当前帧图像中目标物体的位置范围C X时，目标位置物体范围确定单元133，用于将前一帧图像中目标物体的位置范围C X-1作为当前帧图像中目标物体的位置范围C X。或者，在另一些实施例中，可根据前一帧图像中目标物体的位置范围，以及之前预定义帧数的图像中目标物体的位置范围，预测当前帧图像中目标物体的位置范围C X。
第二级神经网络运行控制单元134，用于控制第二级神经网络120，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
在一些实施例中，第二级神经网络运行控制单元134，用于对当前帧图像进行目标物体识别之前，将当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；并控制第二级神经网络在所述扩展位置范围内进行目标物体识别，得到当前帧图像的目标物体识别结果Rx。
特别是，在一些实施例中，所述目标物体可为人手；在控制第一级神经网络110对当前帧图像进行目标物体位置检测，获取当前帧图像中的目标物体位置范围C X时，第一级神经网络运行控制单元132，用于控制第一级神经网络识别人手在当前帧图像中的位置范围；在控制第二级神经网络120，根据当前帧图像中目标物体的位置范围C X，对当前帧图像进行目标物体识别，得到当前帧图像的目标物体识别结果Rx时，第二级神经网络运行控制单元134，用于控制第二级神经网络，根据当前帧图像中目标物体的位置范围C X，对当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
本申请实时视频的目标检测与识别系统中的各个模块的功能和作用的实现过程具体详情可参见上述方法中对应步骤的实现过程。对于系统实施例而言，由于其基本对应于方法实施例，前述对本申请的方法实施例的解释说明也适用于本申请的系统实施例。为避免冗余，在系统实施例中将不会对所有细节进行重复，相关未尽之处可参见上述结合图1到图5对本申请实时视频的目标检测与识别方法实施例的相关描述。
使用本申请的实时视频的目标检测与识别方法，能够利用前帧图像的目标物体位置检测结果，降低用于位置检测的第一级神经网络的运行频率，加快整个目标物体检测和识别的速度并同时降低CPU和内存资源的占用，从整体上改善识别算法的性能。在两级神经网络分别采用相同结构的前提下，使用本申请的方法与每帧图像的识别均运行第一级神经网络的方案相比可节约CPU资源50％以上。
本申请上述实施例涉及的方法或系统，可以应用于AI(Artificial Intelligence，人工智能)中，以实现基于目标检测与识别的AI应用场景，比如，实现通过手势或者表情与AI机器人进行交互的场景等。
本申请第三方面的实施例提出了一种非临时性计算机可读存储介质，其上存储有可执行指令，所述可执行指令在处理器上运行时，实现如本申请第一方面实施例所述的实时视频的目标检测与识别方法。该存储介质可以作为设备的一部分设置在其上；或者当该设备可以被服务器远程控制时，该存储介质可以设置在对该设备进行控制的远程服务器上。
用于实现本申请方法的计算机指令的可以采用一个或多个计算机可读的介质的任意组合来承载。所谓非临时性计算机可读介质可以包括任何计算机可读介质，除了临时性地传播中的信号本身。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
本申请第四方面的实施例提供了一种计算机程序产品，当所述计算机程序产品中的指令由处理器执行时，实现根据本申请第一方面实施例所述的实时视频的目标检测与识别方法。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如”C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
本申请第五方面实施例提供了一种计算设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现根据本申请第一方面所述的实时视频的目标检测与识别方法。
上述使用根据本申请的存储介质、计算机程序产品和计算设备，其相关部分的具体实施方式可以从相应的本申请的实时视频的目标检测与识别方法或系统的实施例中获得，并具有与相应的本申请的实时视频的目标检测与识别方法或系统相似的有益效果，在此不再赘述。
图8示出了适于用来实现本申请实施方式的示例性计算设备的框图。图8显示的计算设备12仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图8所示，计算设备12可以通用计算设备的形式实现。计算设备12的组件可以包括但不限于：一个或者多个处理器或者处理单元16，系统存储器28，连接不同系统组件(包括系统存储器28和处理单元16)的总线18。
总线18表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(Industry Standard Architecture；以下简称：ISA)总线，微通道体系结构(Micro Channel Architecture；以下简称：MAC)总线，增强型ISA总线、视频电子标准协会(Video Electronics Standards Association；以下简称：VESA)局域总线以及 外围组件互连(Peripheral Component Interconnection；以下简称：PCI)总线。
计算设备12典型地包括多种计算机系统可读介质。这些介质可以是任何能够被计算设备12访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
存储器28可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(Random Access Memory；以下简称：RAM)30和/或高速缓存存储器32。计算设备12可以进一步包括其它可移动/不可移动的、易失性/非易失性的计算机可读存储介质。仅作为举例，存储系统34可以用于读写不可移动的、非易失性磁介质(图中未显示，通常称为“硬盘驱动器”)。尽管图8中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如：光盘只读存储器(Compact Disc Read Only Memory；以下简称：CD-ROM)、数字多功能只读光盘(Digital Video Disc Read Only Memory；以下简称：DVD-ROM)或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线18相连。存储器28可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。
具有一组(至少一个)程序模块42的程序/实用工具40，可以存储在例如存储器28中，这样的程序模块42包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块42通常执行本申请所描述的实施例中的功能和/或方法。
计算设备12也可以与一个或多个外部设备14(例如键盘、指向设备、显示器24等)通信，还可与一个或者多个使得用户能与该计算机系统/服务器12交互的设备通信，和/或与使得该计算机系统/服务器12能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口22进行。并且，计算设备12还可以通过网络适配器20与一个或者多个网络(例如局域网(Local Area Network；以下简称：LAN)，广域网(Wide Area Network；以下简称：WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器20通过总线18与计算设备12的其它模块通信。要说明的是，尽管图中未示出，可以结合计算设备12使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、 RAID系统、磁带驱动器以及数据备份存储系统等。
处理单元16通过运行存储在系统存储器28中的程序，从而执行各种功能应用以及数据处理，例如实现前述实施例中提及的方法。
根据本申请第三到五方面的非临时性计算机可读存储介质，计算机程序产品和计算设备，可以参照根据本申请第一方面实施例具体描述的内容实现，并具有与根据本申请第一方面的实时视频的目标检测与识别方法类似的有益效果，在此不再赘述。
需要说明的是，在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是两个或两个以上，例如两个，三个等，除非另有明确具体的限定。
本技术领域的普通技术人员可以理解实现上述实施例的方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
在本说明书的描述中，流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机 可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一个实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (13)

  1. 一种实时视频的目标检测与识别方法，其特征在于，所述方法由计算设备执行，所述方法包括：
    确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；
    当两个识别结果R X-1和R X-2不相同时，通过第一级神经网络对所述当前帧图像进行目标物体位置检测，获取所述当前帧图像中目标物体的位置范围C X；当两个识别结果R X-1和R X-2相同时，不经过所述第一级神经网络，根据所述前一帧图像中目标物体的位置范围C X-1确定所述当前帧图像中目标物体的位置范围C X；
    通过第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx。
  2. 根据权利要求1所述的实时视频的目标检测与识别方法，其特征在于，所述通过第一级神经网络对所述当前帧图像进行目标物体位置检测，包括：
    通过所述第一级神经网络判断所述当前帧图像中是否包含目标物体；
    若所述当前帧图像包含目标物体，则通过所述第一级神经网络获取所述当前帧图像中目标物体的位置范围C X；
    若所述当前帧图像不含目标物体，则将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并通过所述第一级神经网络持续进行判断直至帧图像中包含目标物体。
  3. 根据权利要求1所述的实时视频的目标检测与识别方法，其特征在于，所述通过第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx，包括：
    将所述当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；
    通过所述第二级神经网络在所述扩展位置范围内进行目标物体识别，得到 所述当前帧图像的目标物体识别结果Rx。
  4. 根据权利要求1所述的实时视频的目标检测与识别方法，其特征在于，所述根据所述前一帧图像中目标物体的位置范围C X-1确定所述当前帧图像中目标物体的位置范围C X，包括：
    将所述前一帧图像中目标物体的位置范围C X-1作为所述当前帧图像中目标物体的位置范围C X；或者
    根据所述前一帧图像中目标物体的位置范围，以及之前预定义帧数的图像中目标物体的位置范围，预测所述当前帧图像中目标物体的位置范围C X。
  5. 根据权利要求1-4中任一项所述的实时视频的目标检测与识别方法，其特征在于，
    所述目标物体为人手；
    所述通过第一级神经网络对所述当前帧图像进行目标物体位置检测，获取所述当前帧图像中目标物体的位置范围C X，包括：通过所述第一级神经网络识别人手在所述当前帧图像中的位置范围；
    所述通过第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx，包括：通过所述第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
  6. 一种实时视频的目标检测与识别系统，其特征在于，包括：
    第一级神经网络，用于对视频帧图像进行目标物体的位置检测，获取目标物体位置范围；
    第二级神经网络，用于根据目标物体位置范围，在视频帧图像中进行目标物体的识别，得到识别结果；以及
    神经网络运行控制模块，用于，
    确定当前帧图像的前一帧图像的目标物体识别结果R X-1和两帧之前的图像的目标物体识别结果R X-2是否相同；
    当两个识别结果R X-1和R X-2不相同时，控制第一级神经网络对所述当前帧 图像进行目标物体位置检测，获取所述当前帧图像中目标物体的位置范围C X；当两个识别结果R X-1和R X-2相同时，不经过所述第一级神经网络，根据所述前一帧图像中目标物体的位置范围C X-1确定所述当前帧图像中目标物体的位置范围C X；
    控制第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx。
  7. 根据权利要求6所述的实时视频的目标检测与识别系统，其特征在于，在控制第一级神经网络对所述当前帧图像进行目标物体位置检测时，所述神经网络运行控制模块，用于，
    控制所述第一级神经网络判断所述当前帧图像中是否包含目标物体；
    若所述当前帧图像包含目标物体，则控制所述第一神经网络获取所述当前帧图像中目标物体的位置范围C X；
    若所述当前帧图像不含目标物体，则将判断帧图像中是否包含目标物体的间隔设置为每隔预定帧数执行一次，并控制所述第一级神经网络持续进行判断直至帧图像中包含目标物体。
  8. 根据权利要求6所述的实时视频的目标检测与识别系统，其特征在于，在控制第二级神经网络根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx时，所述神经网络运行控制模块，用于，
    将所述当前帧图像中目标物体的位置范围C X扩展，得到面积增大预定义倍数的扩展位置范围；
    控制所述第二级神经网络在所述扩展位置范围内进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx。
  9. 根据权利要求6所述的实时视频的目标检测与识别系统，其特征在于，在根据所述前一帧图像中目标物体的位置范围C X-1确定所述当前帧图像中目标物体的位置范围C X时，所述神经网络运行控制模块，用于，
    将所述前一帧图像中目标物体的位置范围C X-1作为所述当前帧图像中目标物体的位置范围C X；或者
    根据所述前一帧图像中目标物体的位置范围，以及之前预定义帧数的图像中目标物体的位置范围，预测所述当前帧图像中目标物体的位置范围C X。
  10. 根据权利要求6-9中任一项所述的实时视频的目标检测与识别系统，其特征在于，
    所述目标物体为人手；
    在控制第一级神经网络对所述当前帧图像进行目标物体位置检测，获取所述当前帧图像中目标物体的位置范围C X时，所述神经网络运行控制模块，用于控制所述第一级神经网络识别人手在所述当前帧图像中的位置范围；
    在控制第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像进行目标物体识别，得到所述当前帧图像的目标物体识别结果Rx时，所述神经网络运行控制模块，用于控制所述第二级神经网络，根据所述当前帧图像中目标物体的位置范围C X，对所述当前帧图像中的人手进行手势识别，识别结果为手势的姿态。
  11. 一种非临时性计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时，实现根据权利要求1-5中任一项所述的实时视频的目标检测与识别方法。
  12. 一种计算机程序产品，其特征在于，当所述计算机程序产品中的指令由处理器执行时，实现根据权利要求1-5中任一项所述的实时视频的目标检测与识别方法。
  13. 一种计算设备，其特征在于，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现根据权利要求1-5中任一项所述的实时视频的目标检测与识别方法。
PCT/CN2019/092791 2018-07-13 2019-06-25 实时视频的目标检测与识别方法、系统、存储介质及设备 Ceased WO2020011014A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/012,761 US11625921B2 (en) 2018-07-13 2020-09-04 Method and system for detecting and recognizing target in real-time video, storage medium, and device
US18/114,513 US12347223B2 (en) 2018-07-13 2023-02-27 Method and system for detecting and recognizing target in real-time video, storage medium, and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810771226.X 2018-07-13
CN201810771226.XA CN110147702B (zh) 2018-07-13 2018-07-13 一种实时视频的目标检测与识别方法和系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/012,761 Continuation US11625921B2 (en) 2018-07-13 2020-09-04 Method and system for detecting and recognizing target in real-time video, storage medium, and device

Publications (1)

Publication Number Publication Date
WO2020011014A1 true WO2020011014A1 (zh) 2020-01-16

Family

ID=67589262

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/092791 Ceased WO2020011014A1 (zh) 2018-07-13 2019-06-25 实时视频的目标检测与识别方法、系统、存储介质及设备

Country Status (3)

Country Link
US (2) US11625921B2 (zh)
CN (1) CN110147702B (zh)
WO (1) WO2020011014A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612034A (zh) * 2020-04-15 2020-09-01 中国科学院上海微系统与信息技术研究所 一种对象识别模型的确定方法、装置、电子设备及存储介质
EP3905127A1 (en) * 2020-04-28 2021-11-03 Roland Corporation Image processing program, image processing method and image processing apparatus
CN113763425A (zh) * 2021-08-30 2021-12-07 青岛海信网络科技股份有限公司 道路区域标定方法及电子设备

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102222011B1 (ko) * 2019-07-29 2021-03-04 주식회사 코어라인소프트 의료용 인공 신경망의 분석 결과를 평가하는 의료용 인공 신경망 기반 의료 영상 분석 장치 및 방법
CN110503042B (zh) * 2019-08-23 2022-04-19 Oppo广东移动通信有限公司 图像处理方法、装置以及电子设备
CN112446229B (zh) * 2019-08-27 2024-07-16 北京地平线机器人技术研发有限公司 一种标志杆的像素坐标获取方法及装置
CN111242081B (zh) * 2020-01-19 2023-05-12 深圳云天励飞技术有限公司 视频检测方法、目标检测网络训练方法、装置及终端设备
CN111259907B (zh) * 2020-03-12 2024-03-12 Oppo广东移动通信有限公司 内容识别方法、装置以及电子设备
CN113763456B (zh) * 2020-06-01 2024-11-19 聚晶半导体股份有限公司 图像检测装置以及图像检测方法
JP7443965B2 (ja) * 2020-07-13 2024-03-06 オムロン株式会社 情報処理装置、補正方法、プログラム
CN111986690B (zh) * 2020-08-27 2024-08-02 三星电子（中国）研发中心 一种视频的语音降噪方法和装置
CN112101223B (zh) * 2020-09-16 2024-04-12 阿波罗智联(北京)科技有限公司 检测方法、装置、设备和计算机存储介质
CN112580450B (zh) * 2020-12-03 2022-11-18 天津大学 一种基于快进策略的视频中动物状态快速检测方法
CN112836568B (zh) * 2020-12-08 2024-08-23 北京迈格威科技有限公司 位置预测方法、装置、电子设备及可读存储介质
CN113034518A (zh) * 2021-04-16 2021-06-25 佛山市南海区广工大数控装备协同创新研究院 一种基于卷积神经网络的肝脏病灶分割方法
CN113409355A (zh) * 2021-05-13 2021-09-17 杭州电子科技大学 一种基于fpga的运动目标识别系统及方法
CN113408531B (zh) * 2021-07-19 2023-07-14 北博（厦门）智能科技有限公司 一种基于图像识别的目标物形状框选方法及终端
CN115862126A (zh) * 2021-09-24 2023-03-28 深圳千木知微科技有限公司 远距离手势识别系统及其方法
KR20230056407A (ko) * 2021-10-20 2023-04-27 삼성전자주식회사 영상 정보를 생성하는 방법 및 장치
CN114268807B (zh) * 2021-12-24 2023-08-01 杭州当虹科技股份有限公司 一种实时智能遮台标的自动化测试方法
CN114004963B (zh) * 2021-12-31 2022-03-29 深圳比特微电子科技有限公司 目标类别识别方法、装置及可读存储介质
CN114842377B (zh) * 2022-04-22 2025-04-29 上海西井科技股份有限公司 目标检测模型训练方法、系统、设备及存储介质
CN114926623B (zh) * 2022-05-12 2024-07-12 重庆大学 一种基于目标抛雪区自动识别的抛雪管转向控制方法
CN115063724B (zh) * 2022-06-22 2025-05-16 南京理工大学 一种果树田垄的识别方法及电子设备
CN115393851B (zh) * 2022-08-30 2026-04-14 网易有道(杭州)智能科技有限公司 图像处理方法和终端、装置、电子设备、存储介质
CN116110031A (zh) * 2023-01-12 2023-05-12 杭州海康机器人股份有限公司 一种目标检测方法、装置、电子设备及存储介质
US12450851B2 (en) * 2023-02-06 2025-10-21 Snap Inc. Fingernail segmentation and tracking
CN116721384A (zh) * 2023-05-29 2023-09-08 深圳奇迹智慧网络有限公司 基于神经网络的暴露垃圾监控方法及系统
CN119126053A (zh) * 2023-06-12 2024-12-13 北京图森智途科技有限公司 目标检测方法、车辆、计算设备及计算机可读存储介质
US20250126348A1 (en) * 2023-10-17 2025-04-17 Zoom Video Communications, Inc. Generating An Image In A Video Conference
CN117437678A (zh) * 2023-11-01 2024-01-23 烟台持久钟表有限公司 正面人脸持续时间统计方法、系统、装置、存储介质
CN119364110B (zh) * 2024-12-26 2025-04-22 浙江宇视科技有限公司 视频处理方法、装置、电子设备以及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787867A (zh) * 2016-04-21 2016-07-20 华为技术有限公司 基于神经网络算法的处理视频图像的方法和装置
CN107492115A (zh) * 2017-08-30 2017-12-19 北京小米移动软件有限公司 目标对象的检测方法及装置
CN107944376A (zh) * 2017-11-20 2018-04-20 北京奇虎科技有限公司 视频数据实时姿态识别方法及装置、计算设备
CN108229277A (zh) * 2017-03-31 2018-06-29 北京市商汤科技开发有限公司 手势识别、控制及神经网络训练方法、装置及电子设备

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6964023B2 (en) * 2001-02-05 2005-11-08 International Business Machines Corporation System and method for multi-modal focus detection, referential ambiguity resolution and mood classification using multi-modal input
CN103150546B (zh) * 2012-12-26 2016-03-16 冉阳 视频人脸识别方法和装置
US20170161607A1 (en) * 2015-12-04 2017-06-08 Pilot Ai Labs, Inc. System and method for improved gesture recognition using neural networks
US11144761B2 (en) * 2016-04-04 2021-10-12 Xerox Corporation Deep data association for online multi-class multi-object tracking
US10163042B2 (en) * 2016-08-02 2018-12-25 International Business Machines Corporation Finding missing persons by learning features for person attribute classification based on deep learning
CN109416728A (zh) * 2016-09-30 2019-03-01 富士通株式会社 目标检测方法、装置以及计算机系统
CN108256562B (zh) * 2018-01-09 2022-04-15 深圳大学 基于弱监督时空级联神经网络的显著目标检测方法及系统
US10489639B2 (en) * 2018-02-12 2019-11-26 Avodah Labs, Inc. Automated sign language translation and communication using multiple input and output modalities
US10304208B1 (en) * 2018-02-12 2019-05-28 Avodah Labs, Inc. Automated gesture identification using neural networks
WO2019157344A1 (en) * 2018-02-12 2019-08-15 Avodah Labs, Inc. Real-time gesture recognition method and apparatus

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787867A (zh) * 2016-04-21 2016-07-20 华为技术有限公司 基于神经网络算法的处理视频图像的方法和装置
CN108229277A (zh) * 2017-03-31 2018-06-29 北京市商汤科技开发有限公司 手势识别、控制及神经网络训练方法、装置及电子设备
CN107492115A (zh) * 2017-08-30 2017-12-19 北京小米移动软件有限公司 目标对象的检测方法及装置
CN107944376A (zh) * 2017-11-20 2018-04-20 北京奇虎科技有限公司 视频数据实时姿态识别方法及装置、计算设备

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111612034A (zh) * 2020-04-15 2020-09-01 中国科学院上海微系统与信息技术研究所 一种对象识别模型的确定方法、装置、电子设备及存储介质
CN111612034B (zh) * 2020-04-15 2024-04-12 中国科学院上海微系统与信息技术研究所 一种对象识别模型的确定方法、装置、电子设备及存储介质
EP3905127A1 (en) * 2020-04-28 2021-11-03 Roland Corporation Image processing program, image processing method and image processing apparatus
US11776309B2 (en) 2020-04-28 2023-10-03 Roland Corporation Image processing method, image processing apparatus and non-transitory computer readable medium
CN113763425A (zh) * 2021-08-30 2021-12-07 青岛海信网络科技股份有限公司 道路区域标定方法及电子设备

Also Published As

Publication number Publication date
US11625921B2 (en) 2023-04-11
US12347223B2 (en) 2025-07-01
CN110147702B (zh) 2023-05-23
US20200401812A1 (en) 2020-12-24
CN110147702A (zh) 2019-08-20
US20230222838A1 (en) 2023-07-13

Similar Documents

Publication Publication Date Title
WO2020011014A1 (zh) 实时视频的目标检测与识别方法、系统、存储介质及设备
CN110458095B (zh) 一种有效手势的识别方法、控制方法、装置和电子设备
CN110610510B (zh) 目标跟踪方法、装置、电子设备及存储介质
US10983596B2 (en) Gesture recognition method, device, electronic device, and storage medium
CN111311634B (zh) 一种人脸图像检测方法、装置及设备
Xiao et al. Robust facial landmark detection via recurrent attentive-refinement networks
US10726313B2 (en) Active learning method for temporal action localization in untrimmed videos
US10488939B2 (en) Gesture recognition
CN108960163B (zh) 手势识别方法、装置、设备和存储介质
US11948340B2 (en) Detecting objects in video frames using similarity detectors
CN105825524A (zh) 目标跟踪方法和装置
US20130329946A1 (en) Fast pose detector
WO2020133442A1 (zh) 一种识别文本的方法及终端设备
CN111797733A (zh) 一种基于图像的行为识别方法、装置、设备和存储介质
CN112199994B (zh) 一种实时检测rgb视频中的3d手与未知物体交互的方法和装置
US20230306609A1 (en) Method, device, and computer program for detecting boundary of object in image
CN111444807B (zh) 目标检测方法、装置、电子设备和计算机可读介质
CN111291713A (zh) 一种基于骨架的手势识别方法及系统
WO2020103462A1 (zh) 视觉搜索方法、装置、计算机设备及存储介质
US20250384565A1 (en) Image processing apparatus, image processing method, and non-transitory computer-readable storage medium to perform a tracking process, using a tracking model
CN114139726A (zh) 数据处理方法及装置、电子设备、存储介质
CN115937970A (zh) 手部关键点识别方法、装置、设备及存储介质
US12293559B2 (en) Image data augmentation device and method
CN110147724B (zh) 用于检测视频中的文本区域的方法、装置、设备以及介质
WO2023146470A2 (en) Dual-level model for segmentation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19834401

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19834401

Country of ref document: EP

Kind code of ref document: A1