CN110866425A - Pedestrian identification method based on light field camera and depth migration learning - Google Patents

Abstract

A pedestrian recognition method based on a light field camera and depth migration learning comprises the steps of ① obtaining a plurality of pedestrian images through the light field camera, ② obtaining color pedestrian images and depth pedestrian images through Lytro desktop software, ③ preprocessing the color images and the depth images obtained in the step ②, classifying the color images and the depth images into uniform sizes, dividing the images into positive and negative samples to obtain light field image data sets, ④ model initialization, ⑤ freezing previous convolution blocks through an existing VGG16 image classification model trained on an ImageNet data set, reserving parameters of the last convolution block to obtain initial values of a neural network, ② 0 processing the color pedestrian images and the depth pedestrian images in the step ① through the neural network in ④ to obtain mixed convolution characteristics, and ⑦ conducting repeated training of the neural network according to the convolution characteristics obtained in the step ⑥ and conducting fine tuning on the model to obtain a new classification model.

Description

Pedestrian recognition method based on light field camera and deep transfer learning

technical field

The invention belongs to the field of computer vision, and particularly relates to a pedestrian recognition method based on a light field camera and deep migration learning.

Background technique

Pedestrian recognition is an important part of the field of computer vision research, and has played a very important role in intelligent transportation, video surveillance, artificial intelligence, and autonomous driving. In recent years, based on the rapid development of computer hardware equipment and new photographing technologies, the industry has put forward more stringent requirements for the performance and accuracy of pedestrian recognition.

Due to the rapid development of autonomous driving technology in recent years, the accuracy of pedestrian recognition is particularly important. Because pedestrians have the characteristics of rigid objects and non-rigid objects, such as the variability of pedestrian shooting angles, the existence of factors such as illumination and occlusion, and the appearance of a large number of human bodies in traffic signs and street view advertising signs, the false detection of pedestrians has always been a problem. Key issues affecting pedestrian detection performance. Therefore, in recent years, researchers have carried out a lot of work on the acquisition of pedestrian features and the optimization of detection methods, and comprehensively use multi-sensor methods to extract pedestrian features, thereby reducing the false detection rate and improving the pedestrian detection rate.

Wu Yiren created a "light field camera" with other researchers in the laboratory of Professor Hanraen at Stanford University. It is understood that the body of the "light field camera" is similar to that of a general digital camera, but the internal structure is very different. Generally, the camera captures the light with the main lens, and then focuses it on the film or photoreceptor behind the lens. The sum of all the light forms a small dot on the photo to display the image. This special camera is placed between the main lens and the photoreceptor, and there is a microscope array covered with 90,000 microscopic lenses. Focus the light and convert the light data to record it digitally. The camera's built-in software operates an "expanded light field" that tracks where each ray of light falls on the image at different distances, and digitally refocuses it to take the perfect photo.

Moreover, the "light field camera" is unconventional, reducing the lens aperture size and depth of field, using a small mirror array to control the extra light, revealing the depth of field of each image, and then projecting a tiny secondary image onto the photoreceptor, all around the focused image. The hazy aperture becomes "sharp," maintaining the increased luminosity, reduced photo-taking time, and graining of the older cameras' large aperture, without sacrificing depth of field and image clarity.

Compared with digital cameras, light field cameras have several distinguishing features.

1. Take pictures first, then focus: digital cameras only capture a glossy focus image, with a clear center and blurry out-of-focus; light field cameras record the data of beams in all directions, and select the focus point in the computer later as needed. The final imaging effect of the photo should be processed on the computer.

2. Small size and fast speed: Due to the use of different imaging technologies from digital cameras, light field cameras do not have the complex focusing system of digital cameras, the overall size is small, and the operation is relatively simple; at the same time, because there is no need to select focusing, the shooting speed is also faster. quick.

At present, the commonly used target recognition method is to take pictures based on a color camera to obtain samples, and to perform target recognition through machine learning or deep learning methods. These methods show poor accuracy and robustness when faced with the pedestrian recognition problem in a two-dimensional plane.

The machine learning-based method learns the laws of the human body from the training samples to obtain the model, and then tests it on the test set. If the data and features can be selected reasonably, coupled with a reasonable algorithm for training, the problem of false detection of pedestrians in two-dimensional planes can be well overcome.

Machine learning-based methods generally include three parts: feature extraction, classifier training and detection. The most commonly used feature in the field of pedestrian recognition is the histogram of oriented gradient (HOG). The application of HOG feature combined with Support Vector Machine (SVM) in pedestrian recognition has indeed achieved good results. However, HOG is a typical handcrafted feature, which is not satisfactory for image classification and recognition and detection of pedestrians, animals, buildings and other objects with arbitrary poses. And this kind of HOG-like manual feature design requires the designer to have excellent visual research ability and rich research experience. Looking back at the research progress of object recognition in ten years, it can be found that the proposed models and algorithms are all artificially designed based on features, and the progress is slow.

In recent years, with the development of science and technology, deep learning has become one of the most popular research directions in the field of computer vision. For image recognition tasks and other tasks such as detection, segmentation, etc., various research teams have also achieved very good results in subsequent research. Therefore, according to the current situation, the application of deep learning to pedestrian recognition has become a trend, and has very broad research significance and application prospects.

SUMMARY OF THE INVENTION

The invention improves the shortcomings of the existing pedestrian recognition methods, and provides a pedestrian recognition method based on a light field camera and deep migration learning, and the accuracy and robustness of the pedestrian detection method can be effectively improved by using the method.

As conceived above, the technical solution of the present invention is: a pedestrian recognition method based on a light field camera and deep transfer learning, characterized in that it includes the following steps:

①Using a light field camera to acquire multiple pedestrian images;

②Using the Lytro desktop software to process the original pedestrian image obtained in step ① to obtain a color pedestrian image and a deep pedestrian image;

(3) Preprocess the color image and depth image obtained in step (2) and reduce it to a uniform size, and divide the image into positive and negative samples to obtain a light field image dataset;

④Model initialization: fine-tuning is implemented by fine-tuning a "step-by-step" strategy based on transfer learning;

⑤Using the existing VGG16 image classification model trained on the ImageNet dataset, freeze the previous convolution block, retain the parameters of the last convolution block, and obtain the initial value of the neural network;

⑥ The color pedestrian image and the depth pedestrian image in step ① are respectively processed by the neural network in ④ to obtain the mixed convolution feature;

⑦ According to the convolution features obtained in ⑥, the neural network is repeatedly trained and the model is fine-tuned to obtain a new classification model.

The transfer learning adopts the Keras framework.

The present invention has the following advantages and positive effects:

1. The present invention has indeed played a good role in the misidentification of pedestrians on a two-dimensional plane. As a supplement to the existing pedestrian recognition method, the method has a small amount of computation, a small amount of data required, and low requirements on machine hardware, and can be applied to actual industrial environments.

2. The present invention can effectively improve the accuracy and robustness of the pedestrian detection method.

Description of drawings

FIG. 1 is a flowchart of a pedestrian recognition method based on a light field camera and deep transfer learning of the present invention.

FIG. 2 is a schematic structural diagram of a deep convolutional network used in the present invention.

Detailed ways

Below in conjunction with specific embodiments, the present invention will be further illustrated, and it should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. The modifications all fall within the scope defined by the appended claims of this application.

The light field camera of the present invention adopts the light field camera produced by Lytro Company. The experimental platform used: PC.

As shown in Figure 1: a pedestrian recognition method based on light field camera and deep transfer learning, including the following steps:

①Using a light field camera to acquire multiple pedestrian images;

②Using the Lytro desktop software to process the original pedestrian image obtained in step ① to obtain a color pedestrian image and a deep pedestrian image;

(3) Preprocess the color image and depth image obtained in step (2) and reduce it to a uniform size, and divide the image into positive and negative samples to obtain a light field image dataset;

④Model initialization: Based on migration learning, fine-tuning is implemented by adopting a "step-by-step migration" strategy; the so-called "fine-tuning" is to use the trained model to initialize the parameters of the target network, and continue training on this basis. The purpose is to Get a nice initial value for the neural network.

⑤Using the existing VGG16 image classification model trained on the ImageNet dataset, freeze the previous convolution block, retain the parameters of the last convolution block, and obtain the initial value of the neural network;

⑥ The color pedestrian image and the depth pedestrian image in step ① are respectively processed by the neural network in ④ to obtain the mixed convolution feature;

⑦ According to the convolution features obtained in ⑥, the neural network is repeatedly trained and the model is fine-tuned to obtain a new classification model.

So far, the pedestrian recognition method based on the light field camera and deep learning provided by the present invention is completed.

As shown in Figure 1, 2: the step that the present invention specifically adopts is:

1. First, use the Lytro light field camera to take multiple pictures containing pedestrian scenes, and then use the Lytro official software to process these pictures to obtain color images and depth images of pedestrians.

2. Preprocess the color image and depth image obtained in step 1 and reduce it to a uniform size, and divide the image into 500 positive and negative samples to obtain a light field image dataset.

3. Download the existing VGG16 image classification model trained on the ImageNet dataset, and adapt to the format of the Keras framework used in the present invention.

4. Using the model parameters processed in step 3, freeze the parameters of the first 4 convolution blocks, unfreeze the parameters in the fifth convolution block, participate in the neural network training and update.

5. Before training the neural network, the data set needs to be augmented. Mainly include color channel offset, image rotation, mirroring, random cropping, etc. Using the Keras deep learning framework, the above image processing operations can be easily performed through its built-in functions.

6. Using the parameters in step 4, pass the color image through all convolutional layers of VGG16 to obtain the feature vector X1.

7. Input the depth image into a convolutional layer with a convolution kernel size of 3*3, and obtain the feature vector X2 after the convolution operation.

8. The convolution features in step 6 and step 7 are weighted and added together to form a new convolution feature, that is, X=W1*X1+W2*X2, where X is a new convolution feature.

9. Input the convolution feature X obtained in step 8 into the Fully Connected Layer to obtain the final classification result.

10. According to the classification result, the misrecognition rate of pedestrian recognition can be reduced, especially when the pedestrian dataset contains pedestrians in a two-dimensional plane.

In the present invention, the number of iterations of the neural network should be set to 50, and the optimizer should be set to Momentum.

As above, a detection method for pedestrian misidentification in a two-dimensional plane proposed by the present invention has been described clearly and in detail.

Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, The scope of the invention is defined by the claims and their equivalents.

Claims (2)

1. a pedestrian recognition method based on light field camera and deep transfer learning, is characterized in that: comprise the steps: ①Using a light field camera to acquire multiple pedestrian images; ②Using the Lytro desktop software to process the original pedestrian image obtained in step ① to obtain a color pedestrian image and a deep pedestrian image; (3) Preprocess the color image and depth image obtained in step (2) and reduce it to a uniform size, and divide the image into positive and negative samples to obtain a light field image dataset; ④Model initialization: fine-tuning is implemented by fine-tuning a "step-by-step" strategy based on transfer learning; ⑤Using the existing VGG16 image classification model trained on the ImageNet dataset, freeze the previous convolution block, retain the parameters of the last convolution block, and obtain the initial value of the neural network; ⑥ The color pedestrian image and the depth pedestrian image in step ① are respectively processed by the neural network in ④ to obtain the mixed convolution feature; ⑦ According to the convolution features obtained in ⑥, the neural network is repeatedly trained and the model is fine-tuned to obtain a new classification model. 2 . The pedestrian recognition method based on a light field camera and deep transfer learning according to claim 1 , wherein the transfer learning adopts a Keras framework. 3 .

Images