WO2018107760A1 - 一种用于行人检测的协同式深度网络模型方法 - Google Patents

一种用于行人检测的协同式深度网络模型方法 Download PDF

Info

Publication number
WO2018107760A1
WO2018107760A1 PCT/CN2017/094016 CN2017094016W WO2018107760A1 WO 2018107760 A1 WO2018107760 A1 WO 2018107760A1 CN 2017094016 W CN2017094016 W CN 2017094016W WO 2018107760 A1 WO2018107760 A1 WO 2018107760A1
Authority
WO
WIPO (PCT)
Prior art keywords
training
network model
sample
classification
model
Prior art date
Application number
PCT/CN2017/094016
Other languages
English (en)
French (fr)
Inventor
王文敏
松鸿蒙
王荣刚
李革
董胜富
王振宇
李英
赵辉
高文
Original Assignee
北京大学深圳研究生院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京大学深圳研究生院 filed Critical 北京大学深圳研究生院
Priority to US16/466,386 priority Critical patent/US10867167B2/en
Publication of WO2018107760A1 publication Critical patent/WO2018107760A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition

Definitions

  • the invention belongs to the field of information technology and relates to digital file content protection technology, in particular to a collaborative deep network model method for pedestrian detection.
  • pedestrian detection refers to judging whether a pedestrian or a video frame contains pedestrians. If it includes and marks the position of the pedestrian, it can be divided into two parts: classification and positioning of the sample. Among them, fast and accurate sample classification is the premise and key of pedestrian detection technology.
  • the existing pedestrian detection technology is relatively mature, and its basic framework is derived from the HOG+SVM model.
  • the classification of pedestrian samples is roughly divided into five main steps: sample collection (candidate frame extraction), preprocessing, feature extraction, classifier training and testing.
  • feature extraction and classifier training are the key factors affecting detection performance.
  • the pedestrian detection algorithm can be divided into a single classifier model algorithm and a multi-classifier model algorithm.
  • artificial feature-based detection algorithms and deep learning-based detection algorithms dominate the mainstream.
  • the former first extracts features from the image by manual definition, including HOG features, LBP features, ACF, etc., and then sends them into the common classifier for training.
  • the trained model is used to distinguish between pedestrian and non-pedestrian samples.
  • deep learning algorithms greatly improved the accuracy of feature extraction and the detection performance of classifiers, but it was still limited by the limitations of single classifiers in feature learning ability, and the results still did not reach the ideal level.
  • the second type of algorithm successfully breaks through the bottleneck of the single classifier model. They use multiple classifiers to learn sample features and integrate the classification results to achieve more accurate classification results, such as the common component model based classification. Algorithms, cascading models, and integration models. Component-based models often start from the various parts of the sample, extract local features, train local classifiers, and finally integrate analysis.
  • the cascading model uses the idea of Boosting to arrange multiple classifiers sequentially, and then trains these classifiers one by one with different samples, so that they have different classification capabilities, and the latter classifier is designed according to the previous classifier.
  • Boosting to arrange multiple classifiers sequentially, and then trains these classifiers one by one with different samples, so that they have different classification capabilities, and the latter classifier is designed according to the previous classifier.
  • all the weak classifiers in the final cascade form a strong classifier, which improves the performance of the detection algorithm from the perspective of complementarity of the classifier; the integrated model is more common in the multi-classifier model.
  • the integrated model that has been successfully verified and applied in the field of pedestrian detection is only integrated with CNN (Convolutional Neural Network) model, which trains multiple CNN models in parallel, and then inputs each sample into each CNN model. Take the maximum, minimum, or average of the scores of the output as the final classification result for the sample.
  • CNN Convolutional Neural Network
  • the specific process of the classification algorithm based on the integrated CNN model includes:
  • test data set is sequentially sent to the trained model for classification of pedestrian samples. Each sample obtains several detection scores, and the maximum, minimum or average value of the score is calculated as the final discrimination score.
  • the original classification algorithm based on the integrated CNN model only integrates the CNN model.
  • the model is more scientific than the component-based model and the cascade model, and can solve the pedestrian deformation and occlusion problems.
  • it has drawbacks in the integration method, does not have general and theoretical basis, and the sample classification is not accurate enough.
  • the present invention provides a Collaborative Deep Network (CDN) method for pedestrian detection, which mainly aims at a classification process in pedestrian detection technology, and designs a cooperative depth.
  • CDN Collaborative Deep Network
  • the network model uses different types of training data sets obtained by the K-means clustering algorithm to train multiple deep networks in parallel, and then integrates and synthesizes the classification results of the original data sets on these deep networks through artificial neural networks. Analysis to achieve a more accurate sample classification, can be used for pedestrian sample classification in pedestrian detection.
  • the principle of the invention is: the invention improves the classification algorithm based on the integrated CNN model, and the idea of the classification algorithm based on the integrated CNN model is to construct a plurality of different CNN networks by using different nodes of the dropout in the full connection layer.
  • the sample trains each CNN network, and finally the maximum, minimum, and average values of each network's output are used to complete the classification.
  • the invention draws on the working mode of human social division of labor, first training a plurality of different deep network models as members in parallel, and then training an artificial neural network similar to the decision maker to learn each sample in the data set in each sub-classifier
  • the classification results information on the classification, so that it can learn to comprehensively analyze this information and get more accurate conclusions.
  • the present invention also proposes a re-sampling method based on the K-means clustering algorithm, which first Candidate samples extracted from the original data set This box is clustered according to certain characteristics, and different types of pedestrian samples and non-pedestrian samples are obtained, and then sent to different detection models for learning, so that the classifier can learn more concentrated sample characteristics.
  • the present invention actually provides a new integrated model framework, using a variety of deep network models, wherein the deep network model itself is better, and the effect of integration is more significant.
  • the cooperative deep network model method adopted by the present invention can integrate multiple different types of deep network models, and use artificial neural networks to replace the existing original algorithms to calculate the maximum, minimum or average methods to integrate different deep networks.
  • a re-sampling technique based on K-means clustering algorithm is designed to achieve more accurate pedestrian classification.
  • a collaborative deep network model method (CDN for pedestrian detection), by establishing a cooperative deep network model, using different types of training data sets obtained by the clustering algorithm for training multiple deep network models in parallel, and then Through the artificial neural network, the classification results of the original data set on each deep network model are integrated and comprehensively analyzed to achieve more accurate sample classification; including the following steps:
  • the original training sample data set is divided into different sub-sample sets according to different features
  • the re-sampling method based on the K-means clustering algorithm in step 1), the K-means clustering algorithm is used to automatically segment the original training sample data set, and the re-sampling is obtained.
  • a training subsample set with different characteristics makes the difference between the training subsample sets larger and the training subsample sets have smaller differences; the following steps are included:
  • Extracting the feature vector for each sample in the original training sample data set D n is the total number of samples
  • Equation 1 x i is a feature vector of each sample in the original training sample set D; c j is each cluster center;
  • the feature vector is divided into classes in which the center vector closest to each other is located;
  • C j represents all sample labels contained in each category
  • the feature vector is a three-channel feature, and each sample corresponds to three feature vectors, and each of the three channels is respectively clustered with the original training samples; Step 12)
  • the model initialization value of the classification number k is set to 2, indicating that the corresponding sample contains attributes of both pedestrian and non-pedestrian.
  • the deep network model as the sub-classifier includes a basic depth network model, a strong deep learning detector, and a strong deep learning detector trained with weak training samples.
  • the basic deep network model includes a convolutional neuron network model and a perceptron model; the strong deep learning detector includes a joint deep learning model.
  • x i represents the value of the i-th node of the input layer
  • w ij represents the connection weight of the i-th node of the input layer to the j-th node of the output layer
  • n is the number of nodes of the input layer
  • b j represents the output The offset of the jth node of the layer
  • the artificial neural network can be trained using a back propagation algorithm.
  • step 2) selects the joint
  • the deep learning model is a sub-classifier.
  • the original training sample data set uses the original image in the Caltech pedestrian database and the ETH pedestrian database.
  • the training process of the sub-classifier is divided into the following steps:
  • the original image is first transformed into the YUV color space, and the three-channel feature is extracted.
  • the obtained three-channel feature input is combined with the deep learning model, and the input three-channel feature is subjected to two convolution transformations and one pooling operation to obtain a plurality of component detection maps;
  • the component detection map is deformed by Equation 4, and the score s p of each component is calculated:
  • M p represents a pth component detection map
  • D np and c np respectively represent an nth deformation map corresponding to the pth component and a weight thereof; Corresponding to the element at the (x, y) position in B p ;
  • the component scoring vector is used to train the visual reasoning and classification network, that is, the training of the above sub-classifier is completed, and the trained sub-classifier is obtained.
  • the first three-channel feature wherein the first channel feature corresponds to the Y channel feature of the original image;
  • the second channel feature is divided into four parts: an upper left corner, an upper right corner, a lower left corner, and a lower right corner.
  • the upper left corner, the upper right corner, and the lower left corner respectively correspond to the feature maps of the Y, U, and V channels of the original image whose size is reduced to half of the original image, and the lower right corner is filled with 0;
  • the third channel feature is also divided into four parts, reflecting the original
  • the edge information of the picture, the upper left corner, the upper right corner, and the lower left corner are the edge maps obtained by transforming the feature maps of the original channels Y, U, and V through the Sobel operator and scaling them.
  • Each of the three edge maps is composed of pixel values having the largest amplitude value.
  • the second step uses twenty convolution kernels to calculate twenty component detection maps, which are feature maps of the components.
  • the deformation map used in the third step is a two-dimensional matrix of the same size as the component inspection map, and the values in each matrix are between 0 and 255.
  • the invention is directed to the field of pedestrian detection in computer vision, and the limitation of the classification algorithm used in the existing pedestrian detection technology, and a new collaborative deep network model algorithm is proposed.
  • K-means clustering algorithm Different types of training sub-data sets are segmented, and then these data sets are used to train multiple deep network models in parallel.
  • the artificial neuron network is used to integrate the classification results of all the samples on these trained deep network models.
  • the present invention has a better effect on a plurality of experimental data sets than other algorithms.
  • a new collaborative multi-model learning framework is constructed to complete the classification process in pedestrian detection.
  • Several different deep network models are trained in parallel within the framework.
  • the classification results of each model are integrated to make decisions together. The limitations of single classifiers in feature extraction and feature learning are avoided.
  • Constructing a new collaborative multi-model learning framework is not considered in the existing pedestrian detection technology.
  • the classification model of multiple deep network models provided by the present invention can effectively compensate the feature extraction and feature learning of a single depth model.
  • the limitations, especially for pedestrians with more severe deformation and occlusion, make full use of the characteristics of each classifier and eliminate detection errors, thus achieving more accurate pedestrian detection.
  • the strategy of using artificial neural networks to integrate each deep network in the collaborative model is closer to the human brain decision-making method than the existing manual rules such as using average or maximum value, which is more scientific and can achieve better detection results.
  • a resampling technique based on K-means clustering algorithm is proposed. Firstly, the candidate sample boxes extracted from the original data set are clustered according to certain characteristics to obtain different types of pedestrian samples and non-pedestrian samples. By training different detection models and using K-means clustering algorithm to resample the samples, each sub-classifier can learn different and more concentrated pedestrian characteristics, so that each classifier can learn more specific samples. Features, enhance the classification ability of a certain type of sample, enhance the classification effect of each classifier in the collaborative model, and then improve the overall classification effect.
  • FIG. 1 is a flow chart of a collaborative deep network model method provided by the present invention.
  • FIG. 2 is a flow chart of a training process in which a joint depth model is used as a sub-classifier in an embodiment of the present invention.
  • the invention provides a cooperative deep network model algorithm (CDN for pedestrian detection), which is a sample classification method, which does not include candidate frame extraction and pedestrian positioning process in pedestrian detection process; mainly for pedestrian detection
  • CDN for pedestrian detection is a sample classification method, which does not include candidate frame extraction and pedestrian positioning process in pedestrian detection process; mainly for pedestrian detection
  • the classification process in technology, designing a collaborative deep network model, different types obtained by K-means clustering algorithm
  • the training data set is used to train multiple deep networks in parallel, and then integrate and comprehensively analyze the classification results of the original data sets on these deep networks through artificial neural networks to achieve more accurate sample classification, which can be used for pedestrian detection.
  • FIG. 1 is a flow chart of the collaborative deep network model method provided by the present invention, which mainly includes the following steps:
  • Each of the deep network models is trained using the training method of the original model itself;
  • step 1) the re-sampling technique based on the K-means clustering algorithm is used to prepare the training sample data set, that is, the K-means clustering algorithm is used to automatically segment the original sample data set, and re-sampling to obtain a plurality of training members having different characteristics.
  • the sample set makes the difference between different sample sets larger and the difference within the sample set smaller, which enables multiple deep networks to learn different but more concentrated types of features and enhance the ability to distinguish specific samples.
  • Extracting the feature vector for each sample in the original training sample set D n is the total number of samples
  • Equation 1 x i is a feature vector of each sample in the original training sample set D; c j is each cluster center;
  • the feature vector is divided into classes in which the center vector closest to each other is located;
  • C j represents all sample labels contained in each category
  • each sample should correspond to three feature vectors, and in the resampling process, the original training samples are separately clustered according to each of the three channels.
  • the model initializes k equal to 2 to correspond to the attributes of the sample containing both pedestrian and non-pedestrian.
  • step 2) the training subsample set and the original data set obtained by the above clustering are respectively used to train a plurality of different deep network models, and the training process of each model is performed by multi-thread parallel computing.
  • the deep network models that can be used here include the following three categories:
  • Basic depth network models such as CNN and perceptron models, which have good feature learning and classification capabilities, but are not sufficient to complete pedestrian detection tasks in complex scenes where there are a large number of pedestrian deformations and occlusions. Cooperate with each other to better avoid the detection error of its own existence;
  • Strong deep learning detectors such as the joint deep learning model described above, which can detect pedestrians in the image more accurately and quickly than the basic deep network model, and have better countermeasures for complex scenes in CDN.
  • the addition of a strong depth learning detector can effectively ensure the detection effect of the overall model and further improve the detection performance of the strong depth learning detector;
  • step 3 after obtaining a plurality of trained deep network models, each sample in the original training data set is simultaneously sent to the depth models for classification and identification, and several detection scores are obtained (each depth network model is classified) The score of the test), these output scores are composed of the vector as the observation information of each sample to train an artificial neural network to obtain a collaborative classification model, that is, a collaborative deep network model, in which a plurality of different models are embedded.
  • the deep network sub-classifier can effectively use this information to complete more accurate pedestrian detection tasks after comprehensively learning the preliminary classification information of each sample.
  • the feedforward model of the artificial neural network is Equation 3:
  • x i represents the value of the i-th node of the input layer
  • w ij represents the connection weight of the i-th node of the input layer to the j-th node of the output layer
  • n is the number of nodes of the input layer
  • b j represents the output The offset of the jth node of the layer.
  • the BP (Back Propagation) algorithm can be used to train the artificial neural network.
  • the collaborative classification model is obtained after comprehensively learning the preliminary classification information of each sample, and can effectively use the preliminary classification information of each sample to complete more accurate pedestrian detection tasks.
  • the test data set only needs to be input into the trained collaborative classification model for classification, that is, the pedestrian sample classification of the test data set is obtained.
  • the following embodiment adopts a joint deep learning model (hereinafter referred to as UDN) as a sub-classifier for the collaborative deep network model selection of the present invention, on the Caltech pedestrian database and the ETH pedestrian database (the original database is used as the original training sample data).
  • UDN joint deep learning model
  • the original database is used as the original training sample data.
  • the image in the original database is preprocessed to obtain a rectangular candidate frame image and used for training).
  • the UDN model successfully integrates feature extraction, deformation processing, occlusion processing and classification into a CNN model, effectively solving the problems of pedestrian deformation and occlusion.
  • the flow of the training process of UDN as a sub-classifier is shown in Figure 2.
  • the input picture is defined as three-channel image data of size 84 ⁇ 28, which is convoluted through 64 9 ⁇ 9 ⁇ 3 convolution kernels, and then subjected to 4 ⁇ 4 pooling operation to obtain 64 19 ⁇ 5 feature maps, these feature maps are then input into the second convolutional layer, and 20 component detection maps are obtained through 20 convolution kernels designed to process deformation, and then the component is detected by the deformation layer calculation component, and finally sent to visual reasoning and The classification model performs an estimation of the category.
  • the specific training process is divided into the following steps:
  • the image is first transformed into the YUV color space, and then the three-channel feature is extracted.
  • the first channel feature corresponds to the Y channel feature of the original image; the second channel feature is divided into four blocks, the upper left corner, the upper right corner, and the lower left corner.
  • the feature maps of the three channels corresponding to the original picture, Y, U, and V, are reduced to the original half, and the lower right corner is filled with 0.
  • the third channel feature is also divided into four parts, reflecting It is the edge information of the original picture, and the upper left corner, the upper right corner, and the lower left corner are the edge maps obtained by the feature map of the three channels of the original picture, Y, U, and V, which are transformed by the Sobel operator and scaled.
  • the lower corner is composed of the pixel values having the largest amplitude value at each of the above three edge maps.
  • this embodiment uses 20 different convolution kernels to calculate 20 component detections, which are characteristic maps of human body components.
  • the third step is performed by treatment of formula 4 pairs of strain detecting member of FIG, each component calculated score s p:
  • M p represents a pth component detection map
  • D np and c np respectively represent an nth deformation map corresponding to the pth component and a weight thereof; Corresponding to the element at the (x, y) position in B p ;
  • the deformation map designed in this embodiment is a two-dimensional matrix having the same size as the component detection pattern, and the values in each matrix are between 0 and 255, including four matrices, and the first and third are all divided into matrices.
  • the values of each area are the same, and the value of the left area is always larger than the right area, the difference is that the jump between the values of the first type is small, the third area The jump between the values is larger; the second and fourth divide the matrix into six long strips arranged from top to bottom, the values in each region are the same, the difference is that the second satisfies the upper region The value of the value is greater than the value of the lower area, and the fourth type satisfies the change of the value from the top to the bottom of the region.
  • the preset values of the specific parameter values in these deformation maps may not be fixed, and the training process is used to optimize these values. parameter.
  • the scores of all components constitute the component score vector s, as in Equation 5; the component score vector is used to train the visual reasoning and classification network, and the training method adopts the standard BP algorithm; that is, the training of the above sub-classifier is completed, and the training is completed. Sub-classifier.
  • the visual reasoning and classification network is an artificial neural network.
  • the input information received by each hidden layer in the network comes not only from the upper layer but also from the upper and upper layers.
  • CDN has better performance and detection effect than other more advanced algorithms in the field of pedestrian detection, which can effectively improve the classification ability of a single model.
  • other advanced algorithms in the field of pedestrian detection include HOG (Histogram of Oriented Gradient, That is, gradient histogram), HOGLBP (Histogram of Oriented Gradient and Local Binary Pattern), DPM (Deformable Part Models), DDM (Discriminative Deep Model) Model), ICF (Integral Channel Features), CNN (Convolutional Neural Network), ACF (Aggregated Channel Features), UDN (United Deep Model) ).
  • HOG Histogram of Oriented Gradient, That is, gradient histogram
  • HOGLBP Histogram of Oriented Gradient and Local Binary Pattern
  • DPM Deformable Part Models
  • DDM Discriminative Deep Model
  • ICF Intelligent Deep Model
  • CNN Convolutional Neural Network
  • ACF Aggregated Channel Features
  • UDN United Deep Model
  • Table 2 shows the detection effect of different designed CDNs on the Caltech dataset. It can be found that CDN clusters based on three-channel features and the number of clusters is 2, and the sub-trainets obtained by clustering are integrated. The joint deep training model is trained in the original training set, and the artificial neural network is used to integrate the results, which will obtain the best detection. can.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

本发明公布了一种用于行人检测的协同式深度网络模型方法，包括：构建一种新的协同式多模型学习框架来完成行人检测中的分类过程；利用人工神经元网络来整合协同式模型中各子分类器的判决结果，并借用机器学习的方法训练该网络，能够更加有效地综合各分类器反馈的信息；提出一种基于K-means聚类算法的重采样方法，增强协同模型中各分类器的分类效果，进而提升整体分类效果。本发明通过建立协同式深度网络模型，将由聚类算法得到的不同类型的训练数据集用于并行地训练多个深度网络模型，再通过人工神经元网络将原始数据集在各个深度网络模型上的分类结果进行整合和综合分析，以实现更加精准的样本分类。

Description

一种用于行人检测的协同式深度网络模型方法 技术领域
本发明属于信息技术领域，涉及数字文件内容保护技术，尤其涉及一种用于行人检测的协同式深度网络模型方法。
背景技术
无论是自然科学研究还是社会科学研究，人类自身一直是其中最重要的关注对象，因此，利用计算机分析人类活动一直是计算机视觉领域中最热门的研究课题。而人类动作识别和事件检测等智能化行为的分析都需要以快速准确的行人检测为前提，因而行人检测技术的研究日益受到学术界和工业界的广泛关注，对现代视频监控、智能机器人和无人驾驶技术的发展均具有深远意义。一般来说，行人检测是指判断给定图像或视频帧中是否包含行人，如果包含，并标出行人位置的过程，因而可以分为样本的分类和定位两大部分。其中，快速准确的样本分类是行人检测技术的前提和关键。
现有的行人检测技术已较为成熟，其基本框架来源于HOG+SVM模型。其中，行人样本的分类大致分为样本收集(候选框提取)、预处理、特征提取、分类器训练和测试五个主要步骤。其中，特征提取和分类器训练是影响检测性能的关键。按照分类过程所使用分类器数目的多少，行人检测算法可以分为单分类器模型算法和多分类器模型算法。在第一类中，基于人工特征的检测算法和基于深度学习的检测算法占据主流。前者先用人工定义的方法从图像中提取特征，包括HOG特征、LBP特征、ACF等，再送入常用分类器中进行训练，最后利用训练好的模型去区分行人和非行人样本。后来，深度学习算法的出现大大提高了特征提取的准确性和分类器的检测性能，但仍然受限于单个分类器在特征学习能力上的局限性，其结果仍未达到理想水平。第二类算法成功突破了单分类器模型的瓶颈，它们使用多个分类器去学习样本特征，并将各分类结果整合在一起，实现了更加精准的分类效果，如常用的基于部件模型的分类算法、级联模型和集成模型等。基于部件的模型往往从样本的各个部分出发，提取局部特征，训练局部分类器，最后再整合分析，对于遮挡较为严重的样本能够较好地学习到有用的轮廓特征，从而大大提升了检测性能；级联模型则使用Boosting的思想，将多个分类器顺序排列，然后用不同的样本逐一训练这些分类器，使得它们具备不同的分类能力，其中后一分类器的设计需依据前一分类器的分类结果，最终级联所有的弱分类器形成一个强分类器，从分类器互补的角度改进了检测算法的性能；集成模型则是多分类器模型中较为罕 见的一类，在行人检测领域尚未被人关注和使用，它采用并行的方式来整合多个分类器，让所有子分类器共同完成最终的决策，这与人类社会协同合作的决策方式极其相似。目前在行人检测领域成功得到验证和应用的集成模型仅有集成CNN(Convolutional Neural Network，卷积神经元网络)模型，它并行地训练多个CNN模型，然后将每个样本输入各CNN模型，最后取输出的得分中最大、最小或平均值作为该样本的最终分类结果。基于集成CNN模型的分类算法具体过程包括：
1)准备训练数据集，用滑动窗口法从原始数据集中提取多尺度的行人和非行人候选框；
2)将每个样本依次送入CNN进行卷积和pooling计算，得到特征图；
3)采用不同的dropout比例设置全连接层，然后将从原始样本中提取的特征图送入训练不同的CNN模型；
4)将测试数据集依次送入训练好的模型中进行行人样本的分类，每个样本均得到若干检测得分，计算得分的最大、最小或平均值作为最终的判别得分。
可见，上述原始基于集成CNN模型的分类算法仅仅集成CNN模型。该模型比基于部件的模型和级联模型更具科学性，可以解决行人形变和遮挡问题，但是，它在整合方式上存在弊端，不具备一般性和理论依据，样本分类的精准性不足。
发明内容
为了克服上述现有技术的不足，本发明提供一种用于行人检测的协同式深度网络模型方法(Collaborative Deep Network，简称CDN)，主要针对行人检测技术中的分类过程，设计一种协同式深度网络模型，将由K-means聚类算法得到的不同类型的训练数据集用于并行地训练多个深度网络，再通过人工神经元网络将原始数据集在这些深度网络上的分类结果进行整合和综合分析，以实现更加精准的样本分类，可用于行人检测中行人样本分类。
本发明的原理是：本发明对基于集成CNN模型的分类算法进行改进，基于集成CNN模型的分类算法的思想是利用全连接层中dropout的结点的不同来构建多个不同的CNN网络，用样本训练每个CNN网络，最终对每个网络的输出取最大、最小和平均值，进而完成分类。本发明借鉴了人类社会分工协作的工作模式，先并行地训练多个不同的深度网络模型作为成员，然后训练一个类似决策者的人工神经元网络去学习数据集中的每个样本在各个子分类器上的分类结果信息，使其学会综合分析这些信息并得到更加准确的结论。同时，为了增强团队每个成员的能力，即各个子分类器的分类效果，进而提升整个分类模型的分类能力，本发明还提出了一种基于K-means聚类算法的重采样方法，先将原始数据集中提取出来的候选样 本框按照某种特征进行聚类，得到不同类别的行人样本和非行人样本，再送入不同的检测模型进行学习，以使得分类器能够学到更加集中的样本特性。需要注意的是，本发明实际上提供了一种新的集成模型框架，使用多种深度网络模型，其中使用的深度网络模型本身效果越好，其集成后的效果也越显著。总之，本发明采用的协同式深度网络模型方法能够集成多个不同类型的深度网络模型，并且使用人工神经元网络取代现有的原始算法中计算最大、最小或平均值的方法来整合不同深度网络的分类结果，为了提升每个子深度网络的分类能力，还设计了基于K-means聚类算法的重采样技术，从而达到实现更加准确的行人分类的效果。
本发明提供的技术方案是：
一种用于行人检测的协同式深度网络模型方法(简称CDN)，通过建立协同式深度网络模型，将由聚类算法得到的不同类型的训练数据集用于并行地训练多个深度网络模型，再通过人工神经元网络将原始数据集在各个深度网络模型上的分类结果进行整合和综合分析，以实现更加精准的样本分类；包括如下步骤：
1)采用基于K-means聚类算法的重采样方法，将原始训练样本数据集按照不同特征划分为不同的子样本集；
2)选取多个深度网络模型分别作为子分类器，利用上述子样本集并行地训练所述多个深度网络模型，得到多个训练好的子分类器；
3)将原始训练样本数据集同时送入所述多个训练好的子分类器中，得到检测得分，将所述检测得分连接成检测得分向量，利用所述检测得分向量训练一个人工神经元网络，得到训练好的协同式深度网络模型；
4)将测试数据集输入到所述训练好的协同式深度网络模型中对行人样本进行分类，得到行人样本分类。
针对上述协同式深度网络模型方法，进一步地，步骤1)所述基于K-means聚类算法的重采样方法，具体采用K-means聚类算法自动分割原始训练样本数据集，通过重采样得到多个具备不同特性的训练子样本集，使得训练子样本集之间差异较大而训练子样本集内差异较小；包括如下步骤：
11)对原始训练样本数据集D中的每一个样本，提取得到特征向量
Figure PCTCN2017094016-appb-000001
n为样本总数；
12)随机选择k个样本的特征向量作为聚类中心，记为
Figure PCTCN2017094016-appb-000002
13)通过式1计算得到每个特征向量与每个聚类中心的距离dij：
dij＝||xi-cj||2        (式1)
式1中，xi为原始训练样本集D中的每个样本的特征向量；cj为每个聚类中心；
14)对于每个特征向量xi，依据其与各个聚类中心的距离大小，将该特征向量划分至彼此间距离最近的中心向量所在的类；
15)通过式2更新每类的中心向量：
Figure PCTCN2017094016-appb-000003
其中，Cj表示每个类别中所包含的所有样本标号；
16)当Cj不再变化时，停止聚类过程，得到不同的子样本集；否则返回步骤13)。
针对上述协同式深度网络模型方法，进一步地，所述特征向量为三通道特征，每个样本对应三个特征向量，将三个通道中的每一个特征向量分别对原始训练样本进行聚类；将步骤12)所述分类个数k的模型初始化值设为2，代表对应样本包含行人和非行人两类的属性。
针对上述协同式深度网络模型方法，进一步地，步骤2)作为子分类器的深度网络模型包括基础深度网络模型、强深度学习检测器和用弱训练样本训练的强深度学习检测器。其中，基础深度网络模型包括卷积神经元网络模型和感知器模型；所述强深度学习检测器包括联合深度学习模型。
针对上述协同式深度网络模型方法，进一步地，步骤3)所述人工神经元网络的前馈模型为式3：
Figure PCTCN2017094016-appb-000004
其中，xi代表输入层第i个结点的值，wij代表输入层第i个结点到输出层第j个结点的连接权重，n是输入层结点个数，bj代表输出层第j个结点的偏置；
可采用反向传播算法训练所述人工神经元网络。
针对上述协同式深度网络模型方法，进一步地，在本发明实施例中，步骤2)选取联合 深度学习模型为子分类器，原始训练样本数据集采用Caltech行人数据库和ETH行人数据库中的原始图像，对该子分类器的训练过程分为以下几步：
第一步，先将原始图像变换到YUV颜色空间，提取得到三通道特征；
第二步，将得到的三通道特征输入联合深度学习模型，对输入的三通道特征进行两次卷积变换和一次pooling操作，得到多个部件检测图；
第三步，通过式4对部件检测图进行形变处理，计算得到各部件的得分sp：
Figure PCTCN2017094016-appb-000005
其中，Mp代表第p个部件检测图；Dnp和cnp分别代表第p个部件对应的第n个形变图及其权重；
Figure PCTCN2017094016-appb-000006
对应Bp中(x,y)位置上的元素；
第四步，通过式5将所有部件的得分构成部件得分向量s：
Figure PCTCN2017094016-appb-000007
再用所述部件得分向量训练视觉推理和分类网络，即完成上述子分类器的训练，得到训练好的子分类器。
上述训练过程中，第一步所述三通道特征，其中，第一通道特征对应原始图像的Y通道特征；将第二通道特征分为四部分：左上角、右上角、左下角、右下角，左上角、右上角、左下角分别对应尺度缩小为原来一半的原始图片的Y、U、V三个通道的特征图，右下角用0填充；将第三通道特征也分为四部分，反映原始图片的边缘信息，左上角、右上角、左下角分别是由原始图片的Y、U、V三个通道的特征图经过Sobel算子变换并进行尺寸放缩后得到的边缘图，右下角由上述三个边缘图中每个位置取幅值最大的像素值构成。在本发明实施例中，第二步使用二十个卷积核，计算得到二十个部件检测图，为部件的特征图。第三步中采用的形变图为与部件检测图尺寸相同的二维矩阵，每个矩阵中的值均在0到255之间。
与现有技术相比，本发明的有益效果是：
本发明针对计算机视觉中的行人检测领域，现有行人检测技术中使用的分类算法所存在的局限性，提出了一种新的协同式深度网络模型算法。先由K-means聚类算法从原始数据集 中分割出不同类型的训练子数据集，再用这些数据集并行地训练多个深度网络模型，最后通过人工神经元网络将全部样本在这些训练好的深度网络模型上的分类结果整合在一起，综合分析以实现更加精准的样本分类。本发明在多个实验数据集上相比其他算法都有更好的效果。
与现有技术相比，本发明的核心优点体现在以下方面：
(一)构建了一种新的协同式多模型学习框架来完成行人检测中的分类过程，框架内并行地训练若干不同的深度网络模型，最后整合各模型的分类结果来共同做出决策，有效避免了单一分类器在特征提取和特征学习上的局限性。
构建一种新的协同式多模型学习框架是现有的行人检测技术中基本没有考虑过的，本发明提供的多个深度网络模型协作的分类模式可以有效弥补单一深度模型在特征提取和特征学习上的局限性，尤其是对于形变和遮挡较为严重的行人样本，充分发挥每个分类器的特长，消除检测误差，从而实现更加准确的行人检测。
(二)提出利用人工神经元网络来整合协同式模型中各子分类器的判决结果，并借用机器学习的方法训练该网络，能够更加有效地综合各分类器反馈的信息。
利用人工神经元网络来整合协同式模型中的各个深度网络的策略比现有的利用平均或取最大值等人工规则更贴近人脑决策方式，更具科学性，能够实现更好的检测效果。
(三)提出了一种基于K-means聚类算法的重采样技术，先将原始数据集中提取出来的候选样本框按照某种特征进行聚类，得到不同类型的行人样本和非行人样本，用以训练不同的检测模型，利用K-means聚类算法进行样本的重采样可以使每个子分类器能学到不同且更加集中的行人特征，以使得各分类器均能够学到更加专一的样本特征，增强某一类样本的分类能力，增强协同模型中各分类器的分类效果，进而提升整体分类效果。
附图说明
图1是本发明提供的协同式深度网络模型方法的流程框图。
图2本发明实施例中以联合深度模型为子分类器的训练过程的流程框图。
具体实施方式
下面结合附图，通过实施例进一步描述本发明，但不以任何方式限制本发明的范围。
本发明提出了一种用于行人检测的协同式深度网络模型算法(简称CDN)，该算法是一种样本分类方法，不包含行人检测过程中的候选框提取和行人定位过程；主要针对行人检测技术中的分类过程，设计一种协同式深度网络模型，将由K-means聚类算法得到的不同类型 的训练数据集用于并行地训练多个深度网络，再通过人工神经元网络将原始数据集在这些深度网络上的分类结果进行整合和综合分析，以实现更加精准的样本分类，可用于行人检测中行人样本分类；图1是本发明提供的协同式深度网络模型方法的流程框图，主要包括如下步骤：
1)采用基于K-means聚类算法的重采样技术准备训练样本数据集，将原始样本数据集按照某种特征的不同划分为不同的子样本集；
2)选取多个深度网络模型作为子分类器，并利用原始训练样本集和重采样的上述子样本集分别并行地训练它们；
其中每个深度网络模型均采用原模型本身的训练方法进行训练；
3)将原始样本数据集同时送入多个训练好的子分类器中，得到检测得分，然后将这些得分连接成向量去训练一个人工神经元网络，最终得到训练好的协同式深度网络模型；
4)将测试数据集输入训练好的模型中进行行人样本的分类。
步骤1)中，使用基于K-means聚类算法的重采样技术准备训练样本数据集，即采用K-means聚类算法自动分割原始样本数据集，以重采样得到多个具备不同特性的训练子样本集，使得不同样本集间差异较大而样本集内差异较小，进而使得多个深度网络能够学习到不同但更加集中的某类特征，并增强区分特定样本的能力。具体步骤如下：
11)对原始训练样本集D中的每一个样本，提取得到特征向量
Figure PCTCN2017094016-appb-000008
n为样本总数；
12)随机选择k个样本的特征向量作为聚类中心，记为
Figure PCTCN2017094016-appb-000009
13)通过式1计算得到每个特征向量与每个聚类中心的距离dij：
dij＝||xi-cj||2       (式1)
式1中，xi为原始训练样本集D中的每个样本的特征向量；cj为每个聚类中心；
14)对于每个特征向量xi，依据其与各个聚类中心的距离dij的大小，将该特征向量划分至彼此间距离最近的中心向量所在的类；
15)通过式2更新每类的中心向量：
Figure PCTCN2017094016-appb-000010
其中，Cj表示每个类别中所包含的所有样本标号；
16)如果Cj不再变化，则停止聚类过程，得到不同的子样本集；否则返回步骤13)。
由于这里提取的特征向量为三通道特征，故每个样本应对应三个特征向量，且在重采样过程中，需依据这三个通道中的每一个分别对原始训练样本进行聚类。另外，考虑到K-means算法需要事先规定分类个数k，本模型初始化k等于2，以对应样本所具有的包含行人和非行人两类的属性。
步骤2)中，使用上述聚类得到的训练子样本集和原始数据集分别去训练多个不同的深度网络模型，每个模型的训练过程采用多线程并行计算。这里可以使用的深度网络模型包括以下三类：
2A)基础深度网络模型，如CNN、感知器模型等，这些模型已经具有较好的特征学习和分类能力，但不足以完成存在大量行人形变和遮挡的复杂场景下的行人检测任务，故需要互相配合才能更好地避免自身存在的检测误差；
2B)强深度学习检测器，如上述联合深度学习模型等，这些模型比基础深度网络模型更能准确快速地检测出图像中的行人，且对于复杂场景已有较好的应对措施，在CDN中加入强深度学习检测器可以有效保证整体模型的检测效果，并进一步提升强深度学习检测器的检测性能；
2C)用弱训练样本训练的强深度学习检测器，这一类模型较为特殊，它们具有较强的分类能力，但被用某一类的训练样本训练，进而对某类样本有极致的识别能力，通过将具备检测不同类型样本能力的分类器进行整合，CDN模型会比普通的多模型融合具有更全面的检测能力。
步骤3)中，在得到多个训练好的深度网络模型后，将原始训练数据集中的各个样本同时送入这些深度模型进行分类识别，并得到若干检测得分(每个深度网络模型均得出分类检测的得分)，将这些输出得分组成向量作为每个样本的观测信息去训练一个人工神经元网络，以得到一个协同式分类模型，即协同式深度网络模型，该模型中内嵌多个不同的深度网络子分类器，可在综合学习各样本的初步分类信息以后能够有效利用这些信息去完成更加准确的行人检测任务。人工神经元网络的前馈模型为式3：
Figure PCTCN2017094016-appb-000011
其中，xi代表输入层第i个结点的值，wij代表输入层第i个结点到输出层第j个结点的连 接权重，n是输入层结点个数，bj代表输出层第j个结点的偏置。
训练人工神经元网络可采用BP(Back Propagation，即反向传播)算法。
协同式分类模型是在综合学习各样本的初步分类信息以后得到的，能够有效利用各样本的初步分类信息去完成更加准确的行人检测任务。在线测试时，只需将测试数据集输入训练好的协同式分类模型中进行分类，即获得测试数据集的行人样本分类。
为了便于实验验证，以下实施例采用联合深度学习模型(以下简称UDN)作为本发明的协同式深度网络模型选取的子分类器，在Caltech行人数据库和ETH行人数据库上(原始数据库作为原始训练样本数据集，先将原始数据库中的图像经过预处理得到矩形候选框图像后用于训练)分别做了测试。UDN模型成功将特征提取、形变处理、遮挡处理和分类四个环节融入到一个CNN模型当中，有效地解决了行人形变和遮挡等问题。
UDN作为子分类器的训练过程的流程如图2所示。其中，输入图片定义为三通道的、大小为84×28的图像数据，经由64个9×9×3的卷积核做卷积操作，再经过4×4的pooling操作，得到64个19×5的特征图，这些特征图再输入第二卷积层，经过20个设计好的处理形变的卷积核得到20个部件检测图，随后利用形变层计算部件检测得分，最后送入视觉推理和分类模型进行类别的估计。具体训练过程分为以下几步：
第一步，先将图像变换到YUV颜色空间，然后提取三通道特征，其中，第一通道特征对应原始图片的Y通道特征；第二通道特征分为4块，左上角、右上角、左下角三块分别对应原始图片的Y、U、V三个通道的特征图其尺度缩小为原来一半的结果，右下角那一块用0填充；同样，第三通道特征也分为四个部分，反映的是原始图片的边缘信息，其左上角、右上角、左下角分别是由原始图片的Y、U、V三个通道的特征图经过Sobel算子变换并进行尺寸放缩后得到的边缘图，右下角则由上述三个边缘图中每个位置取幅值最大的像素值构成。
第二步，对输入的三通道特征进行两次卷积变换和一次pooling操作，得到20个部件检测图；
在训练过程中，本实施例使用20个不同的卷积核，计算得到20个部件检测，为人体部件的特征图。
第三步，通过式4对部件检测图进行形变处理，计算得到各部件得分sp：
Figure PCTCN2017094016-appb-000012
其中，Mp代表第p个部件检测图；Dnp和cnp分别代表第p个部件对应的第n个形变图及其权重；
Figure PCTCN2017094016-appb-000013
对应Bp中(x,y)位置上的元素；
本实施例中设计的形变图为与部件检测图尺寸相同的二维矩阵，每个矩阵中的值均在0到255之间，包括四种矩阵，第一种和第三种均将矩阵划分为从左至右排列的六个长条形区域，每个区域的值相同，且左边区域的值始终大于右边区域，区别在于第一种的区域间的值的跳跃较小，第三种区域间的值的跳跃较大；第二种和第四种则将矩阵划分为从上至下排列的六个长条形区域，每个区域中的值均相同，区别在于第二种满足上边区域的值要大于下边区域的值，而第四种则满足从上至下区域的值先递增后递减的变化规律；这些形变图中具体参数值的预设可以不固定，通过训练过程来优化这些参数。
第四步，所有部件的得分构成部件得分向量s，如式5；再用部件得分向量训练视觉推理和分类网络，训练方法采用标准BP算法；即完成上述子分类器的训练，得到训练好的子分类器。
Figure PCTCN2017094016-appb-000014
其中，在联合深度模型中，视觉推理和分类网络是一种人工神经元网络，网络中每个隐藏层接收的输入信息不仅来自于上层还来自于上上层。
实验结果表明，CDN比行人检测领域其他较为先进的算法具有更好的性能和检测效果，可以有效提升单一模型的分类能力；其中，行人检测领域其他较为先进的算法包括HOG(Histogram of Oriented Gradient，即梯度直方图)、HOGLBP(Histogram of Oriented Gradient and Local Binary Pattern，即梯度直方图和局部二值化模式)、DPM(Deformable Part Models，即形变部件模型)、DDM(Discriminative Deep Model，即辨别深度模型)、ICF(Integral Channel Features，即积分通道特征)、CNN(Convolutional Neural Network，即卷积神经网络)、ACF(Aggregated Channel Features，即聚合通道特征)、UDN(United Deep Model，即联合深度模型)。对比结果如表1所示：
表1不同行人检测模型实验结果统计表
Figure PCTCN2017094016-appb-000015
Figure PCTCN2017094016-appb-000016
表2不同设计的CDN模型在Caltech数据集上实验结果统计表
Figure PCTCN2017094016-appb-000017
表2则给出了不同设计的CDN在Caltech数据集上的检测效果，可以发现，CDN若基于三通道特征进行聚类且聚类数目均为2，同时集成用聚类得到的子训练集和原始训练集分别训练的联合深度学习模型，并且采用人工神经元网络进行结果的整合，会获得最好的检测性 能。
需要注意的是，公布实施例的目的在于帮助进一步理解本发明，但是本领域的技术人员可以理解：在不脱离本发明及所附权利要求的精神和范围内，各种替换和修改都是可能的。因此，本发明不应局限于实施例所公开的内容，本发明要求保护的范围以权利要求书界定的范围为准。

Claims (10)

  1. 一种用于行人检测的协同式深度网络模型方法，通过建立协同式深度网络模型，将由聚类算法得到的不同类型的训练数据集用于并行地训练多个深度网络模型，再通过人工神经元网络将原始数据集在各个深度网络模型上的分类结果进行整合和综合分析，以实现更加精准的样本分类；包括如下步骤：
    1)采用基于K-means聚类算法的重采样方法，将原始训练样本数据集按照不同特征划分为不同的子样本集；
    2)选取多个深度网络模型分别作为子分类器，利用上述子样本集并行地训练所述多个深度网络模型，得到多个训练好的子分类器；
    3)将原始训练样本数据集同时送入所述多个训练好的子分类器中，得到检测得分，将所述检测得分连接成检测得分向量，利用所述检测得分向量训练一个人工神经元网络，得到训练好的协同式深度网络模型；
    4)将测试数据集输入到所述训练好的协同式深度网络模型中对行人样本进行分类，得到行人样本分类。
  2. 如权利要求1所述协同式深度网络模型方法，其特征是，步骤1)所述基于K-means聚类算法的重采样方法，具体采用K-means聚类算法自动分割原始训练样本数据集，通过重采样得到多个具备不同特性的训练子样本集，使得训练子样本集之间差异较大而训练子样本集内差异较小；包括如下步骤：
    11)对原始训练样本数据集D中的每一个样本，提取得到特征向量
    Figure PCTCN2017094016-appb-100001
    n为样本总数；
    12)随机选择k个样本的特征向量作为聚类中心，记为
    Figure PCTCN2017094016-appb-100002
    13)通过式1计算得到每个特征向量与每个聚类中心的距离dij：
    dij＝||xi-cj||2        (式1)
    式1中，xi为原始训练样本集D中的每个样本的特征向量；cj为每个聚类中心；
    14)对于每个特征向量xi，依据其与各个聚类中心的距离大小，将该特征向量划分至彼此间距离最近的中心向量所在的类；
    15)通过式2更新每类的中心向量：
    Figure PCTCN2017094016-appb-100003
    其中，Cj表示每个类别中所包含的所有样本标号；
    16)当Cj不再变化时，停止聚类过程，得到不同的子样本集；否则返回步骤13)。
  3. 如权利要求2所述协同式深度网络模型方法，其特征是，所述特征向量为三通道特征，每个样本对应三个特征向量，将三个通道中的每一个特征向量分别对原始训练样本进行聚类；将步骤12)所述分类个数k的模型初始化值设为2，代表对应样本包含行人和非行人两类的属性。
  4. 如权利要求1所述协同式深度网络模型方法，其特征是，步骤2)作为子分类器的深度网络模型包括基础深度网络模型、强深度学习检测器和用弱训练样本训练的强深度学习检测器。
  5. 如权利要求4所述协同式深度网络模型方法，其特征是，所述基础深度网络模型包括卷积神经元网络模型和感知器模型；所述强深度学习检测器包括联合深度学习模型。
  6. 如权利要求1所述协同式深度网络模型方法，其特征是，步骤3)所述人工神经元网络的前馈模型为式3：
    Figure PCTCN2017094016-appb-100004
    其中，xi代表输入层第i个结点的值，wij代表输入层第i个结点到输出层第j个结点的连接权重，n是输入层结点个数，bj代表输出层第j个结点的偏置；
    可采用反向传播算法训练所述人工神经元网络。
  7. 如权利要求1所述协同式深度网络模型方法，其特征是，步骤2)选取联合深度学习模型为子分类器，原始训练样本数据集采用Caltech行人数据库和ETH行人数据库中的原始图像，对该子分类器的训练过程分为以下几步：
    第一步，先将原始图像变换到YUV颜色空间，提取得到三通道特征；
    第二步，将得到的三通道特征输入联合深度学习模型，对输入的三通道特征进行两次卷积变换和一次pooling操作，得到多个部件检测图；
    第三步，通过式4对部件检测图进行形变处理，计算得到各部件的得分sp：
    Figure PCTCN2017094016-appb-100005
    其中，Mp代表第p个部件检测图；Dnp和cnp分别代表第p个部件对应的第n个形变图及其权重；
    Figure PCTCN2017094016-appb-100006
    对应Bp中(x,y)位置上的元素；
    第四步，通过式5将所有部件的得分构成部件得分向量s：
    Figure PCTCN2017094016-appb-100007
    再用所述部件得分向量训练视觉推理和分类网络，即完成上述子分类器的训练，得到训练好的子分类器。
  8. 如权利要求7所述协同式深度网络模型方法，其特征是，第一步所述三通道特征中，第一通道特征对应原始图像的Y通道特征；将第二通道特征分为四部分：左上角、右上角、左下角、右下角，左上角、右上角、左下角分别对应尺度缩小为原来一半的原始图片的Y、U、V三个通道的特征图，右下角用0填充；将第三通道特征也分为四部分，反映原始图片的边缘信息，左上角、右上角、左下角分别是由原始图片的Y、U、V三个通道的特征图经过Sobel算子变换并进行尺寸放缩后得到的边缘图，右下角由上述三个边缘图中每个位置取幅值最大的像素值构成。
  9. 如权利要求7所述协同式深度网络模型方法，其特征是，第二步使用二十个卷积核，计算得到二十个部件检测图，为部件的特征图。
  10. 如权利要求7所述协同式深度网络模型方法，其特征是，第三步中采用的形变图为与部件检测图尺寸相同的二维矩阵，每个矩阵中的值均在0到255之间。
PCT/CN2017/094016 2016-12-16 2017-07-24 一种用于行人检测的协同式深度网络模型方法 WO2018107760A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/466,386 US10867167B2 (en) 2016-12-16 2017-07-24 Collaborative deep network model method for pedestrian detection

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611166903.2 2016-12-16
CN201611166903.2A CN106650806B (zh) 2016-12-16 2016-12-16 一种用于行人检测的协同式深度网络模型方法

Publications (1)

Publication Number Publication Date
WO2018107760A1 true WO2018107760A1 (zh) 2018-06-21

Family

ID=58823680

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/094016 WO2018107760A1 (zh) 2016-12-16 2017-07-24 一种用于行人检测的协同式深度网络模型方法

Country Status (3)

Country Link
US (1) US10867167B2 (zh)
CN (1) CN106650806B (zh)
WO (1) WO2018107760A1 (zh)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110472586A (zh) * 2019-08-17 2019-11-19 四创科技有限公司 一种基于迁移学习的安全帽检测的方法
CN111126240A (zh) * 2019-12-19 2020-05-08 西安工程大学 一种三通道特征融合人脸识别方法
CN111160226A (zh) * 2019-12-26 2020-05-15 华侨大学 一种基于视角自适应特征学习的行人性别识别方法
CN111325223A (zh) * 2018-12-13 2020-06-23 中国电信股份有限公司 深度学习模型的训练方法、装置和计算机可读存储介质
CN111428675A (zh) * 2020-04-02 2020-07-17 南开大学 一种融入行人体态姿态特征的行人重识别方法
CN111709336A (zh) * 2020-06-08 2020-09-25 杭州像素元科技有限公司 一种高速公路行人检测方法、设备及可读存储介质
CN111985546A (zh) * 2020-08-10 2020-11-24 西北工业大学 基于单分类极限学习机算法的飞机发动机多工况检测方法
CN112307955A (zh) * 2020-10-29 2021-02-02 广西科技大学 一种基于ssd红外图像行人检测的优化方法
CN112465124A (zh) * 2020-12-15 2021-03-09 武汉智能装备工业技术研究院有限公司 孪生深度时空神经网络模型获取/故障诊断方法、装置
CN112861615A (zh) * 2020-12-30 2021-05-28 江苏大学 一种基于深度目标检测的垃圾乱投放行为识别方法
CN113139597A (zh) * 2021-04-19 2021-07-20 中国人民解放军91054部队 一种基于统计思想的图像分布外检测方法
CN113536946A (zh) * 2021-06-21 2021-10-22 清华大学 一种基于摄像头关系的自监督行人重识别方法
CN114120453A (zh) * 2021-11-29 2022-03-01 北京百度网讯科技有限公司 一种活体检测方法、装置、电子设备及存储介质
CN114548366A (zh) * 2022-01-12 2022-05-27 湖南大学 基于难样本挖掘的深度网络模型构建方法及系统

Families Citing this family (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106650806B (zh) * 2016-12-16 2019-07-26 北京大学深圳研究生院 一种用于行人检测的协同式深度网络模型方法
CN107169527B (zh) * 2017-06-06 2020-04-03 西北工业大学 基于协同深度学习的医学图像分类方法
CN107273929A (zh) * 2017-06-14 2017-10-20 电子科技大学 一种基于深度协同神经网络的无人机自主降落方法
CN107316031B (zh) * 2017-07-04 2020-07-10 北京大学深圳研究生院 用于行人重识别的图像特征提取方法
CN107273872B (zh) * 2017-07-13 2020-05-05 北京大学深圳研究生院 用于图像或视频中行人重识别的深度判别网络模型方法
US10417527B2 (en) * 2017-09-06 2019-09-17 Irdeto B.V. Identifying an object within content
US20200279156A1 (en) * 2017-10-09 2020-09-03 Intel Corporation Feature fusion for multi-modal machine learning analysis
CN107729948A (zh) * 2017-10-31 2018-02-23 京东方科技集团股份有限公司 图像处理方法及装置、计算机产品和存储介质
CN108241745B (zh) * 2018-01-08 2020-04-28 阿里巴巴集团控股有限公司 样本集的处理方法及装置、样本的查询方法及装置
JP7063052B2 (ja) * 2018-03-28 2022-05-09 富士通株式会社 適合度算出プログラム、適合度算出方法、適合度算出装置、識別プログラム、識別方法及び識別装置
CN108564030A (zh) * 2018-04-12 2018-09-21 广州飒特红外股份有限公司 面向车载热成像行人检测的分类器训练方法和装置
CN110335269A (zh) * 2018-05-16 2019-10-15 腾讯医疗健康(深圳)有限公司 眼底图像的类别识别方法和装置
CN108898076B (zh) * 2018-06-13 2022-07-01 北京大学深圳研究生院 一种视频行为时间轴定位及候选框提取的方法
US11615208B2 (en) * 2018-07-06 2023-03-28 Capital One Services, Llc Systems and methods for synthetic data generation
US10832003B2 (en) * 2018-08-26 2020-11-10 CloudMinds Technology, Inc. Method and system for intent classification
CN109448005B (zh) * 2018-10-31 2019-12-27 数坤（北京）网络科技有限公司 一种用于冠状动脉的网络模型分割方法及设备
CN109547546B (zh) * 2018-11-12 2020-06-05 网宿科技股份有限公司 一种请求任务的调度方法及调度中心服务器
CN109522855B (zh) * 2018-11-23 2020-07-14 广州广电银通金融电子科技有限公司 结合ResNet和SENet的低分辨率行人检测方法、系统及存储介质
CN109635708B (zh) * 2018-12-06 2023-01-03 中山大学 一种基于三数据集交叉迁移学习的无监督行人再识别方法
CN111723617B (zh) * 2019-03-20 2023-10-27 顺丰科技有限公司 动作识别的方法、装置、设备及存储介质
CN110009038B (zh) * 2019-04-04 2022-02-22 北京百度网讯科技有限公司 筛查模型的训练方法、装置及存储介质
CN110674875A (zh) * 2019-09-25 2020-01-10 电子科技大学 一种基于深度混合模型的行人运动模式识别方法
US11630988B2 (en) * 2019-11-18 2023-04-18 International Business Machines Corporation Multi-sample dropout for faster deep neural network training
CN111062493B (zh) * 2019-12-20 2021-06-15 深圳前海微众银行股份有限公司 基于公共数据的纵向联邦方法、装置、设备及介质
CN111582363B (zh) * 2020-05-06 2023-04-07 西安电子科技大学 一种基于多视点深度特征融合SENet网络的分类方法
TWI769466B (zh) * 2020-06-17 2022-07-01 台達電子工業股份有限公司 類神經網路系統及其操作方法
CN111982910B (zh) * 2020-07-06 2022-10-25 华南理工大学 一种基于人工缺陷仿真的弱监督机器视觉检测方法及系统
CN111797778B (zh) * 2020-07-08 2023-06-02 龙岩学院 一种用于breaking街舞主播连麦斗舞的自动评分方法
CN111797795A (zh) * 2020-07-13 2020-10-20 燕山大学 一种基于YOLOv3与SSR的行人检测算法
CN111860638B (zh) * 2020-07-17 2022-06-28 湖南大学 基于不平衡数据深度信念网络的并行入侵检测方法和系统
CN111860679B (zh) * 2020-07-29 2022-04-26 浙江理工大学 一种基于YOLO v3改进算法的车辆检测方法
CN111814755A (zh) * 2020-08-18 2020-10-23 深延科技(北京)有限公司 面向夜间运动场景的多帧图像行人检测方法和装置
CN112132005A (zh) * 2020-09-21 2020-12-25 福州大学 一种基于聚类分析与模型压缩的人脸检测方法
CN112382382B (zh) * 2020-10-23 2024-04-12 北京科技大学 一种代价敏感的集成学习分类方法及系统
CN112949688A (zh) * 2021-02-01 2021-06-11 哈尔滨市科佳通用机电股份有限公司 一种动车组底板胶皮破损故障检测方法、系统及装置
CN113343760A (zh) * 2021-04-29 2021-09-03 暖屋信息科技(苏州)有限公司 一种基于多尺度特征神经网络的人体行为识别方法
CN113177514B (zh) * 2021-05-20 2023-06-16 浙江波誓盾科技有限公司 无人机信号检测方法、装置及计算机可读存储介质
CN113343949B (zh) * 2021-08-03 2021-10-29 中国航空油料集团有限公司 一种通用的嵌入式平台的行人检测模型训练方法
CN114140731B (zh) * 2021-12-08 2023-04-25 西南交通大学 一种牵引变电所异常检测方法
CN115098854A (zh) * 2021-12-20 2022-09-23 中国科学院沈阳自动化研究所 一种融合工艺行为与网络行为的入侵检测方法
CN115098771B (zh) * 2022-06-09 2024-10-25 阿里巴巴（中国）有限公司 推荐模型更新方法、推荐模型训练方法及计算设备
CN116232967B (zh) * 2023-05-09 2023-07-04 中国科学技术大学先进技术研究院 网络安全检测方法、装置、设备及存储介质
CN117422963B (zh) * 2023-09-11 2024-08-06 南通大学 基于高维度特征映射和特征聚合的跨模态地点识别方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1908960A (zh) * 2005-08-02 2007-02-07 中国科学院计算技术研究所 一种基于特征分组的多分类器组合人脸识别方法
CN103489004A (zh) * 2013-09-30 2014-01-01 华南理工大学 一种深度学习网络实现大类别图像识别的方法
US8644624B2 (en) * 2009-07-28 2014-02-04 Samsung Electronics Co., Ltd. System and method for indoor-outdoor scene classification
CN106650806A (zh) * 2016-12-16 2017-05-10 北京大学深圳研究生院 一种用于行人检测的协同式深度网络模型方法

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160026915A1 (en) * 2001-01-05 2016-01-28 In-Depth Test Llc Methods and Apparatus for Data Analysis
US6778705B2 (en) * 2001-02-27 2004-08-17 Koninklijke Philips Electronics N.V. Classification of objects through model ensembles
US7308133B2 (en) * 2001-09-28 2007-12-11 Koninklijke Philips Elecyronics N.V. System and method of face recognition using proportions of learned model
US7480414B2 (en) * 2004-10-14 2009-01-20 International Business Machines Corporation Method and apparatus for object normalization using object classification
JP5742873B2 (ja) * 2013-05-08 2015-07-01 株式会社デンソー 物体識別装置
US10339685B2 (en) * 2014-02-23 2019-07-02 Northeastern University System for beauty, cosmetic, and fashion analysis
US20150310365A1 (en) * 2014-04-25 2015-10-29 Xerox Corporation System and method for video-based detection of goods received event in a vehicular drive-thru
CN105095835A (zh) * 2014-05-12 2015-11-25 比亚迪股份有限公司 行人检测方法及系统
JP2017525064A (ja) * 2014-05-30 2017-08-31 プレイスメーター インコーポレイテッドＰｌａｃｅｍｅｔｅｒ Ｉｎｃ． ビデオデータを用いた活動モニタリングのためのシステム及び方法
CN104035996B (zh) * 2014-06-11 2017-06-16 华东师范大学 基于Deep Learning的领域概念抽取方法
CN105046197B (zh) * 2015-06-11 2018-04-17 西安电子科技大学 基于聚类的多模板行人检测方法
US10325204B2 (en) * 2015-07-06 2019-06-18 Texas Instruments Incorporated Efficient decision tree traversal in an adaptive boosting (AdaBoost) classifier
US9818032B2 (en) * 2015-10-28 2017-11-14 Intel Corporation Automatic video summarization
US11514244B2 (en) * 2015-11-11 2022-11-29 Adobe Inc. Structured knowledge modeling and extraction from images
CN105975929A (zh) * 2016-05-04 2016-09-28 北京大学深圳研究生院 一种基于聚合通道特征的快速行人检测方法
CN106022231A (zh) * 2016-05-11 2016-10-12 浙江理工大学 一种基于多特征融合的行人快速检测的技术方法
US10387765B2 (en) * 2016-06-23 2019-08-20 Siemens Healthcare Gmbh Image correction using a deep generative machine-learning model
EP4283563A3 (en) * 2016-06-27 2024-05-29 Mobileye Vision Technologies Ltd. Controlling host vehicle based on detected parked vehicle characteristics
CN106204449B (zh) * 2016-07-06 2019-09-10 安徽工业大学 一种基于对称深度网络的单幅图像超分辨率重建方法
US10262239B2 (en) * 2016-07-26 2019-04-16 Viisights Solutions Ltd. Video content contextual classification
US10083347B2 (en) * 2016-07-29 2018-09-25 NTech lab LLC Face identification using artificial neural network
WO2018022821A1 (en) * 2016-07-29 2018-02-01 Arizona Board Of Regents On Behalf Of Arizona State University Memory compression in a deep neural network
US10152637B2 (en) * 2016-09-14 2018-12-11 Canon Kabushiki Kaisha Temporal segmentation of actions using context features
US10074038B2 (en) * 2016-11-23 2018-09-11 General Electric Company Deep learning medical systems and methods for image reconstruction and quality evaluation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1908960A (zh) * 2005-08-02 2007-02-07 中国科学院计算技术研究所 一种基于特征分组的多分类器组合人脸识别方法
US8644624B2 (en) * 2009-07-28 2014-02-04 Samsung Electronics Co., Ltd. System and method for indoor-outdoor scene classification
CN103489004A (zh) * 2013-09-30 2014-01-01 华南理工大学 一种深度学习网络实现大类别图像识别的方法
CN106650806A (zh) * 2016-12-16 2017-05-10 北京大学深圳研究生院 一种用于行人检测的协同式深度网络模型方法

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111325223B (zh) * 2018-12-13 2023-10-24 中国电信股份有限公司 深度学习模型的训练方法、装置和计算机可读存储介质
CN111325223A (zh) * 2018-12-13 2020-06-23 中国电信股份有限公司 深度学习模型的训练方法、装置和计算机可读存储介质
CN110472586B (zh) * 2019-08-17 2022-10-14 四创科技有限公司 一种基于迁移学习的安全帽检测的方法
CN110472586A (zh) * 2019-08-17 2019-11-19 四创科技有限公司 一种基于迁移学习的安全帽检测的方法
CN111126240B (zh) * 2019-12-19 2023-04-07 西安工程大学 一种三通道特征融合人脸识别方法
CN111126240A (zh) * 2019-12-19 2020-05-08 西安工程大学 一种三通道特征融合人脸识别方法
CN111160226B (zh) * 2019-12-26 2023-03-31 华侨大学 一种基于视角自适应特征学习的行人性别识别方法
CN111160226A (zh) * 2019-12-26 2020-05-15 华侨大学 一种基于视角自适应特征学习的行人性别识别方法
CN111428675A (zh) * 2020-04-02 2020-07-17 南开大学 一种融入行人体态姿态特征的行人重识别方法
CN111709336A (zh) * 2020-06-08 2020-09-25 杭州像素元科技有限公司 一种高速公路行人检测方法、设备及可读存储介质
CN111709336B (zh) * 2020-06-08 2024-04-26 杭州像素元科技有限公司 一种高速公路行人检测方法、设备及可读存储介质
CN111985546A (zh) * 2020-08-10 2020-11-24 西北工业大学 基于单分类极限学习机算法的飞机发动机多工况检测方法
CN112307955A (zh) * 2020-10-29 2021-02-02 广西科技大学 一种基于ssd红外图像行人检测的优化方法
CN112465124A (zh) * 2020-12-15 2021-03-09 武汉智能装备工业技术研究院有限公司 孪生深度时空神经网络模型获取/故障诊断方法、装置
CN112861615B (zh) * 2020-12-30 2024-04-09 江苏大学 一种基于深度目标检测的垃圾乱投放行为识别方法
CN112861615A (zh) * 2020-12-30 2021-05-28 江苏大学 一种基于深度目标检测的垃圾乱投放行为识别方法
CN113139597A (zh) * 2021-04-19 2021-07-20 中国人民解放军91054部队 一种基于统计思想的图像分布外检测方法
CN113139597B (zh) * 2021-04-19 2022-11-04 中国人民解放军91054部队 一种基于统计思想的图像分布外检测方法
CN113536946B (zh) * 2021-06-21 2024-04-19 清华大学 一种基于摄像头关系的自监督行人重识别方法
CN113536946A (zh) * 2021-06-21 2021-10-22 清华大学 一种基于摄像头关系的自监督行人重识别方法
CN114120453A (zh) * 2021-11-29 2022-03-01 北京百度网讯科技有限公司 一种活体检测方法、装置、电子设备及存储介质
CN114548366A (zh) * 2022-01-12 2022-05-27 湖南大学 基于难样本挖掘的深度网络模型构建方法及系统

Also Published As

Publication number Publication date
CN106650806A (zh) 2017-05-10
US20200082165A1 (en) 2020-03-12
US10867167B2 (en) 2020-12-15
CN106650806B (zh) 2019-07-26

Similar Documents

Publication Publication Date Title
WO2018107760A1 (zh) 一种用于行人检测的协同式深度网络模型方法
CN107832672B (zh) 一种利用姿态信息设计多损失函数的行人重识别方法
Hsiao et al. Occlusion reasoning for object detectionunder arbitrary viewpoint
CN107273845B (zh) 一种基于置信区域和多特征加权融合的人脸表情识别方法
CN108520226B (zh) 一种基于躯体分解和显著性检测的行人重识别方法
CN111079674B (zh) 一种基于全局和局部信息融合的目标检测方法
CN106529442B (zh) 一种行人识别方法和装置
Li et al. Robust visual tracking based on convolutional features with illumination and occlusion handing
WO2020114118A1 (zh) 面部属性识别方法、装置、存储介质及处理器
US9317785B1 (en) Method and system for determining ethnicity category of facial images based on multi-level primary and auxiliary classifiers
CN110033007B (zh) 基于深度姿态预估和多特征融合的行人衣着属性识别方法
CN110728209A (zh) 一种姿态识别方法、装置、电子设备及存储介质
CN109934195A (zh) 一种基于信息融合的反欺骗三维人脸识别方法
CN108171133B (zh) 一种基于特征协方差矩阵的动态手势识别方法
CN109583482A (zh) 一种基于多特征融合与多核迁移学习的红外人体目标图像识别方法
CN103854016B (zh) 基于方向性共同发生特征的人体行为分类识别方法及系统
Presti et al. Boosting Hankel matrices for face emotion recognition and pain detection
CN102930300A (zh) 一种飞机目标识别方法及系统
CN111062928A (zh) 一种医学ct图像中病变的识别方法
WO2013075295A1 (zh) 低分辨率视频的服装识别方法及系统
Xia et al. Face occlusion detection using deep convolutional neural networks
CN111723600A (zh) 一种基于多任务学习的行人重识别特征描述子
Chen et al. A multi-scale fusion convolutional neural network for face detection
CN114818963A (zh) 一种基于跨图像特征融合的小样本检测算法
CN114782979A (zh) 一种行人重识别模型的训练方法、装置、存储介质及终端

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17880572

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17880572

Country of ref document: EP

Kind code of ref document: A1