TW201520978A - Method for estimating two-dimensional image depth values and system thereof - Google Patents

Abstract

The present invention relates to a method for estimating two-dimensional image depth values and a system thereof, and characterized in that a neural network is used, and during a training process of the neural network, single-resolution and multiple-resolution texture images of a 2D sample image are cut into a block pixel for use as an input vector, of which a known depth value is used as a corresponding output, so as to obtain a set of neural weights; then the neural network is used again and loaded with trained neural weights, and the 2D image is used as a redefined input vector, and the input vector is fed into the neural network for obtaining the corresponding depth value, thereby obtaining a depth map of the image in order to facilitate the subsequent transformation of a 2D image into a 3D image.

Description

Method and system for estimating depth value of two-dimensional image

The present invention relates to a 2D to 3D image processing technology, and more particularly to an estimation technique for performing 2D image depth values using a neural network-like algorithm.

Press, Depth Map has always played an important role in 2D to 3D conversion. The 2D image plus the depth map estimated by the 2D image through a certain algorithm, and then the left eye and right eye images are synthesized by the technique of Depth-image-based rendering (DIBR), and then matched with 3D. Hardware devices can present stereoscopic effects, as in the literature [12].

The earliest 2D image depth value estimation technique uses the left and right eye image depth estimation method, which is based on the principle that the two eyes have a Disparity distance of about 6 cm. Since the eyes of the person can continuously judge the distance of the object, that is, the depth (Depth). Place the two cameras on the same horizontal line with the camera spaced six centimeters apart and simultaneously shoot the same object. The actual depth of the camera to the object can be calculated by the calculation of the formula, as in [4], the depth map can be generated.

In the development of the second-dimensional image depth value estimation technology in the future, the present invention classifies it into the following two categories: the first category, which is an estimation of image classification such as: k-means, watershed or edge information. (Edge information). Estimates on depth cues such as Mobile parallax, Linear perspective, Atmospheric perspective, Texture gradient, and more.

However, only Stanford University uses the (Markov Random Field, MRF) training method to perform range sensor depth map estimation. For example, the literature [13] adopts the method of supervised learning (Markov Random Field, MRF) and divides the image into indoors. And do training outdoors. Since the local image features are insufficient, the image global content is added, and the hierarchical structure is used to build the model. The resulting depth map is similar to the actual depth.

However, the above-mentioned two-dimensional image depth value estimation technique is cumbersome, time-consuming and the estimated depth value is still not accurate enough, so the current 2D image depth value estimation technology has room for improvement.

Citations: [4]: A. Klaus, M. Sormann, and K. Karner, "Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure," Pattern Recognition, 2006. ICPR 2006. 18th International Conference On, 2006, pp. 15-18. Literature [12]: Cheng. Chao-Chung, Li. Chung-Te, and Chen. Liang-Gee, "A novel 2Dd-to-3D conversion system using edge information," Consumer Electronics, IEEE Transactions on, vol. 56, Pp. 1739-1745, 2010. [13]: Ashutosh Saxena, H. Chung-Sung, and Y. Ng-Andrew, "3-D Depth Reconstruction from a Single Still Image," International Journal of Computer Vision (IJCV) , Aug 2007. Literature [21]: The application of neural network-MATLAB (third edition), edited by Luo Huaqiang.

Therefore, the main object of the present invention is to provide an estimation method and system for estimating a depth value of a two-dimensional image with a relatively accurate depth value of a two-dimensional image.

In order to achieve the above object, the technical means for estimating the depth value of the two-dimensional image of the present invention comprises: (a) a first defining step, taking the same image and defining (x, y) _T f input vector and the target vector of the corresponding depth value d(x, y) ; (b), a network creation step, establish a type of neural network and parameterize it; (c), a training network step Entering the target vector value of the (x, y) _T f input vector and its depth value, thereby training the neural network to obtain a NN weight value; (d), a second definition step, and a system definition (x, y) _t f input vector of the sample image; and (e), an output step, inputting the (x, y) _t f input vector to the trained neural network, and loading the The NN weight value gives the estimated depth value de(x, y) of the sample image.

The above estimation method further includes a step (f), wherein the step (f) is a comparison step, according to the depth value de(x, y) , to evaluate the training effect of the neural network. If it meets the requirements, it stops training and can be used for this type of neural network; if it does not meet the requirements, the (x, y) _T f input vector is set to multiple input vectors, or / and increase The number of neurons in a neural network is re-trained and re-trained to meet the needs.

The above estimation method uses the Mean absolute error (MAE) to evaluate the training effect of this type of neural network.

The (x, y) _T f multiple input vector in the above step (f) is set as a first re-input vector, a second re-input vector, a third re-input vector, and a fourth multi-input vector.

The input method of the input vector (x, y) _T f in the above step (f) is a head-to-head connection method, a zig-zag method or a head-to-tail connection method.

The number of neurons in the above step (f) is set to be at least 700.

The neural network in the above step (b) is set as a combination of a reverse transmission network and a creation of a feedforward network.

A hidden layer set in the above-mentioned neural network uses a log double bending transfer function (logsig) or a tangent double bending transfer function (tansig), and an output layer set inside the neural network uses linear transfer. Function (purelin).

The NN weight value described above includes a weight (W) and an offset value (b).

A system for estimating a depth value of a two-dimensional image is a system constructed using the above estimation method.

The above technical means have the following effects: 1. The present invention is the first to propose a new method for estimating the depth value of a two-dimensional image, and using the neural network to estimate the depth map of the two-dimensional image can effectively Precisely predicting the depth value of the depth map of the 2D image; and 2. The present invention proposes a multiple resolution prediction method under double prediction (MAE can reach 8.47), and under triple prediction (MAE can reach 7.23), at ( MAE can reach 5.47). In order to improve its prediction accuracy, the number of network neurons is increased during the training of such nerves. When the number of neurons increases to 1000 (MAE can reach 1.0). Therefore, the present invention can effectively improve the prediction accuracy of the depth value of the 2D image when the number of the number of neurons and the number of neurons are increased.

Referring to FIG. 1 and FIG. 2 , the present invention relates to a method for estimating a depth value of a two-dimensional image and a system thereof, comprising: (a) a first defining step, taking the same image and Defining its (x,y) _T f input vector and the target vector of the corresponding depth value d(x,y) ; (b), creating a network, establishing a neural network and parameterizing it; c) a training network step of inputting the target vector value of the (x, y) _T f input vector and its depth value, thereby training the neural network to obtain a NN weight value; (d), a second defining step of defining an (x, y) _t f input vector of the sample image; (e) an output step of inputting the (x, y) _t f input vector to the trained neural field The network, and loading the NN weight value, obtains the estimated depth value de(x, y) of the sample image; and (f), a comparison step according to the depth value de(x, y) , According to the evaluation of the training effect of the neural network, if the requirements are met, the training of the neural network is stopped and used; if the demand is not met, the input vector (x, y _T f) is set to multiple input vectors, or / and increase the number of neurons in this type of neural network, and retrain this type of neural network to meet the needs. Further illustrate that the steps of the method of the invention are as follows:

The present invention uses a neural network-like (hereinafter referred to as NN) technique and a gray-scale texture image to perform depth map estimation, and divides the depth map estimation into two processes: (a) a training process, and (b) a prediction process. As shown in Fig. 2, the system architecture of the NN depth map training and prediction of the present invention is shown. Table 1 below is the NN depth map training and prediction symbol definition. 【Table 1】

Regarding the NN training process of the present invention, in the training process, first input gray scale image (x, y) _T f and gray depth map d (x, y) , and cut square pixels from single or multiple gray scale image The vector is input, and its known depth value is used as the corresponding output to train the neural network. The training result is the weight value (W) of a group of neurons.

Regarding the prediction process of the present invention: in the prediction process, two samples are divided into tests. First, the test sample 1. Grayscale texture image (x, y) _t1 f. Second, the test sample 2. Grayscale texture image (x, y) _t2 f. The input vector is extracted according to the input two-dimensional gray-scale texture image, and then the trained neuron weight value is loaded, and finally the estimated depth value de(x, y) is output, that is, the entire depth map training and prediction process is completed.

Regarding the first definition step (a) of the present invention, before defining the input vector and the target value, the image of the test sample used in the present invention and the relationship between its multi-resolution are first introduced. The following table 2 is the correspondence between grayscale image and depth map. 【Table 2】

Each Input Vector input of the Neural Network of the present invention is a fixed 5*5 block that can be taken from a different Texture Layer. The correspondence between the gray-scale texture image and the depth map in Table 2 is as follows: N(i)=(I(i)-n+1)*(J(i)-n+1)(block:n* n, i: 1~8); P(i)*Q(i)=2 ^(i-1) *2 ^(i-1) .

The present invention has found through preliminary experimental results that the computer device cannot load its operation due to too many input vectors. In order to reduce the input vector, the relationship between the grayscale texture image size and the depth map and its relation are redefined, as shown in Table 3. 【table 3】

The relationship of the above table: N(i)=(I(i)-n+1)*(J(i)-n+1); (block:n*n,i:4~8); 4 is the meaning of the above abbreviation. 【Table 4】

Single resolution defines the input vector and the target value: the input grayscale texture image (x, y) _T f and the grayscale depth value d (x, y) used in the present invention are both 68*90 pixels, and the vector is captured. The size is 5*5 pixels, and the overlay method is used for vector capture. The capture method is from top to bottom, from left to right (the interval is one pixel).

First, input the grayscale image (x, y) ₄ f and its corresponding grayscale depth value d (x, y) , and define the input vector and target value, such as the 3a, by the vector size and mode proposed in this paper. The figure shown.

Wherein (x, y) ₄ F is a 5*5 vector centered on (x, y) ₄ f . Take the vector formula = (x-2: x + 2, y-2: y + 2), as shown in Figure 3b. Moreover, the (x, y) ₄ f image starting range: x ∈ [1: M], y ∈ [1: N]; and (x, y) ₄ F input vector center point effective coordinate range: x ∈ [3 :M-2], y∈[3:N-2]. Wherein the d(x,y) target value starting range: x∈[1:M], y∈[1:N], and the d(x,y) target value is valid corresponding to the coordinate range: x∈[3: M-2], y∈[3:N-2].

Regarding the step (b) of creating a network network according to the present invention: the present invention uses the Matlab version 7.14 Neural Network Toolbox 7.0.3 to establish an architecture of the reverse transfer network. Each Input is weighted with an appropriate weight, and the weighted input and the sum of the offsets form the input to the transfer function f . The neuron can use any divisible transfer function f to produce the output of the neuron, as shown in Figure 4. The inverted transfer network has a weight value, an offset value, and its hidden layer (Hidden Layer) has a double-bend transfer function (logsig or tansig function), as shown in Figures 5a and 5b, and its output layer ( The Output Layer) has a linear transfer function (purelin function), as shown in Figure 5c, which allows the network to approximate any function of a finite number of discontinuities. Table 5 below is a description of each abbreviation of the neuron model. 【table 5】

The single- or multi-grain grayscale image is used as an input vector, and its known depth value is used as a corresponding output to train the neural network of the present invention. Because the MATLAB Neural Network Toolbox network only allows one-dimensional input vectors, the 5*5 input vector is converted to 25*1 (the way the head is connected), as shown in Figure 6. Therefore, the inverse transfer network architecture (single resolution depth prediction) of the present invention can be as shown in FIG.

Parameter setting for the neural network of the present invention: the depth is estimated by known actual, as shown in Table 6. Preliminary experiments, training and predictions were performed using 68*90 image sizes, and local predictions were taken for observation. When the hidden layer uses the logsig transfer function, the prediction results are not good, as shown in Table 7. The prediction effect is similar to the actual estimated depth when using the tansig transfer function, as shown in Table 8. Therefore, it is better to use tansig for the hidden layer transfer function of the present invention, and purein for the output layer. [Table 6] [Table 7] [Table 8]

The present invention further creates a Feedforward Network MATLAB instruction: newff, whose instruction syntax is as follows: net=newff(Iv,[N ₁ N ₂ ...N _i ],...{TF ₁ TF ₂ ...TF _i }, BTF , BLF, PF). Table 9 shows the description of the feedforward network symbol. [Table 9]

Usually, the feedforward network often has one or more hidden layers, and the single-resolution deep prediction network architecture of the present invention is referred to [21], and the total weight value and offset value used in the network are 18,901. The algorithm is as shown in Equation 1: [Formula 1]

First create a two-layer feedforward network with input elements, where the input layer inputs 25 representative feature vectors, the hidden layer (the first layer) has 700 neurons, and the output layer (the second layer) has 1 nerve. The transfer function of the first layer is the tangent double bending transfer function tansig, and the transfer function of the second layer is the linear transfer function purelin, and the output of the network can take any value. The training function is the Levenberg-Marquardt algorithm trainlm. And the maximum number of iterations for training is 200, and the final performance goal is 1e-5. Each input is weighted with an appropriate weight value (W). The input of this weighted sum and the offset value (b) form the input of the transfer function f . The neuron can use any divisible transfer function f to produce the output of the neuron, as shown in Figure 8a.

Regarding the step (c) of the training network of the present invention: after the network is created, the action of the training network can be started. The invention uses the Mean absolute error (MAE) to evaluate the network learning effect, and stops the network during the training network process until the depth value of the network training and the known actual value estimation depth are not large. Road training, and store this network for the next deep prediction. The training image is 68*90 in size.

The so-called training process (Training Process): training image (x, y) ₄ f is 68 * 90, take 25 * 1 pixel as the input vector and its corresponding depth value d (x, y) to train the network model, Call it the "training process." Until the network can approach a depth value that meets the required depth, that is, stop network training, Figure 8b is the BPN training process.

A second defining step (d) The aspect of the invention relates to: define a new input vector, and the training input grayscale image lines (x, y) _T f and test grayscale image lines (x, y) _t f, which is input to the capture The amount and size are equivalent to step (a) of the present invention. The newly defined input vector will be used in the prediction network, and its scope is as follows: (x, y) _T f input vector range: x [1:M],y [1:N], input vector center point effective coordinate range x [3:M-2],y [3: N-2]. And (x,y) _t f input vector range: x [1:2*M],y [1:2*N], input vector center point effective coordinate range: x [3:2*M-2],y [3:2*N-2]. Table 10 below defines a new list of input vectors. [Table 10]

It also predicts the network's prediction process, which is based on the trained network model to predict the answer, which is called the "predictive process." Given an input vector that the network has never seen before, the network tends to give a reasonable answer and output the required depth value, as shown in Figure 9a for the BPN prediction process.

The present invention divides into two test samples for depth prediction, test sample 1. Prediction: f _T = f ₄ , f _t = f ₄ ; test sample 2. Prediction: f _T = f ₄ , f _t = f ₃ . And for d(x, y), the starting range of d(x, y): x [1:M],y [1:N], the valid range of d(x,y): x [3:M-2],y [3: N-2]. Regarding de(x, y), the starting range of de(x, y): x [3:M-2],y [3:N-2], the de(x,y) effective range: x [3:M-2],y [3: N-2], as shown in Figure 9b.

The present invention takes two images Image01 and Image02 as tests, and Figures 10a and 10b show the predicted results and parameter settings. In order to compare the difference between the known actual depth d(x, y) and the predicted depth de(x, y) , the present invention uses the Mean absolute error (MAE) to evaluate the network learning effect, and the calculation formula is as follows. 2. Where n is the number of output depth values, d is the known actual depth value, and de is the predicted depth value. [Formula 2]

The Neural Network training parameters are (#neuron=700, hidden layer=1, epochs=200, goal=1e–5).

The principle and structure of the preliminary depth prediction proposed by the present invention are obtained through experimental results, and the results predicted by the test sample 1. and the test sample 2. are displayed under the single-resolution prediction. When the process at the predicted depth resolution singlet, although predictable (x, y) of the input vector ₄ f, but for the (x, y) ₃ f prediction can not give the desired results. First, the reason for the prediction failure is that the input gray-scale image vector is not enough during the training process, resulting in failure in prediction. Second, it is impossible to grasp the parameter configuration of the inverted transit neural network, such as the number of neurons, the number of hidden layers and the adjustment of some detailed parameters. Currently, there is no configuration of the number of neurons and the number of hidden layers. Certainly standard, can only try to find the best answer.

In order to improve the lack of single-resolution prediction, the present invention uses the concept of Hierarchical Structure proposed by Stanford University, as in the literature [13]. Using this concept, the present invention improves the single-resolution prediction and develops multiple-resolution depth prediction. The present invention first introduces the multi-resolution depth prediction result, and then the subjective and objective evaluation of the predicted depth map, and finally uses the Depth-image-based rendering (DIBR) technique to synthesize the left and right eye images. . Stereoscopic Image can be presented through the 3D display.

The present invention adds multiple purposes: in the training process, the grayscale texture image vector is added for training, so that the accuracy can be improved when performing depth prediction. Next, the present invention will begin to introduce multiple resolution predictions.

First, define each heavy input vector, the difference is that the input vector (x, y) _T f defined in the training process is different from the corresponding target value d (x, y) , and the remaining steps are the same as the single-resolution depth prediction. As shown in Figure 11 and Figure 12a. A first weighted input vector F ₄ (x, y): In (x, y) ₄ f 5 * 5 vector centers, (x, y) ₄ f start range: x [1:M],y [1:N]. The second heavy input vector F ₅ (x, y): at ₅ f (⌈ Hey, hey. ⌉) is the center of the 5*5 vector, (x, y) ₅ f starting range: x [1:M/2],y [1:N/2]. The third heavy input vector F ₆ (x, y): with ₆ f (⌈ Hey, hey. ⌉) is the center of the 5*5 vector, (x, y) ₆ f starting range: x [1:M/4],y [1:N/4]. The fourth heavy input vector F ₇ (x, y): at ₇ f (⌈ Hey, hey. ⌉) is the center of the 5*5 vector, (x, y) ₇ f starting range: x [1:M/8],y [1:N/8].

Regarding the double-resolution prediction aspect, the difference is that the input vector defined in the training process is different from the target value, and the remaining steps are the same as the single-resolution prediction, as shown in Fig. 12b as the multi-resolution training process. Its inverted network architecture (multiple resolution prediction) is shown in Figure 13.

For double resolution, define the input vector and the target value: the first heavy input vector (x, y) ₄ F, the second heavy input vector (x, y) ₅ F, whose vector draws the center point (x, y) ₄ f And (x, y) ₅ f and its corresponding depth value d (x, y) center point, as shown in Figure 14, the algorithm and process are as follows:

The first heavy input vector F ₄ (x, y): (x, y) ₄ f starting range: x [1:M],y [1:N], and (x,y) ₄ F input vector center point effective coordinate range: x [5:M-4],y [5: N-4]. The second heavy input vector F ₅ (x, y): firstly (x, y) ₄ f is first averaged into (x, y) ₅ f, and its formula is as follows: (x, ₅ f = f ₄ (x , y), its x, y range: x [1:M],y [1:N], and , the (x,y) ₅ f starting range: x [1:M/2],y [1:N/2], and the (x,y) ₅ F input vector center point effective coordinate range: x [3:M/2-2],y [3:N/2-2].

Another d(x,y) , the starting range of the d(x,y) : x [1:M],y [1:N], and the d(x,y) target value center point effective coordinate range: x [5:M-4],y [5: N-4].

Regarding triple resolution prediction, triple resolution defines the input vector and the target value: the first heavy input vector (x, y) ₄ F, the second heavy input vector (x, y) ₅ F, and the third heavy input vector (x, y) ₆ F, whose vector is extracted from the center point (x, y) ₄ f, (x, y) ₅ f and (x, y) ₆ f and its corresponding depth value d (x, y) center point, such as As shown in Figure 15, the algorithm and process are as follows:

The first heavy input vector F ₄ (x, y): 1.f ₄ (x, y) starting range: x [1:M],y [1:N];2.F ₄ (x,y) input vector center point effective coordinate range: x [10:M-9],y [10: N-9] The second heavy input vector F ₅ (x, y): 1. First, do (x, y) ₄ f average sub-sampling into (x, y) ₅ f, the formula is as follows: x ₅ f=f ₄ (x,y), its x,y range:x [1:M],y [1:N]. Where (x,y) ₅ f starting range: x [1:M/2],y [1:N/2], the (x,y) ₅ F input vector center point effective coordinate range: x [5:M/2-4],y [5: N/2-4]. The third heavy input vector F ₆ (x, y): 1. First, do (x, y) ₅ f average sub-sample into (x, y) ₆ f, the formula is as follows: 2.f ₆ (x, y ) = f ₅ (x, y), its x, y range: x [1:M/2],y [1:N/2], ;3.f ₆ (x,y) starting range: x [1:M/4],y [1:N/4];4.F ₆ (x,y) input vector center point effective coordinate range: x [3:M/4-2],y [3: N/4-2]. Another d(x,y) :1. d(x,y) starting range: x [1:M],y [1:N];2. d(x,y) target value center point effective coordinate range: x [10:M-9],y [10:N-9].

Regarding quadratic resolution prediction: quadruple resolution defines the input vector and the target value, the first heavy input vector (x, y) ₄ F, the second heavy input vector (x, y) ₅ F, and the third heavy input vector ( x, y) ₆ F, the fourth heavy input vector (x, y) ₇ F, whose vector draws the center point (x, y) ₄ f, (x, y) ₅ f, (x, y) ₆ f and (x, y) ₇ f corresponds to the depth point d (x, y) center point, as shown in Figure 11, the algorithm and process are as follows:

The first heavy input vector F ₄ (x, y): 1.f ₄ (x, y) starting range: x [1:M],y [1:N];2.F ₄ (x,y) input vector center point effective coordinate range: x [20:M-19],y [20:N-19]. The second heavy input vector F ₅ (x, y): 1. First, do (x, y) ₄ f average sub-sampling into (x, y) ₅ f, the formula is as follows; 2.f ₅ (x, y =f ₄ (x,y), its x,y range:x [1:M],y [1:N], ;3.f ₅ (x,y) starting range: x [1:M/2],y [1:N/2];4.F ₅ (x,y) input vector center point effective coordinate range: x [10:M/2-9],y [10: N/2-9]. The third input vector F ₆ (x, y): 1. First, do (x, y) ₅ f average sub-sample into (x, y) ₆ f, the formula is as follows; 2.f ₆ (x, y )=f ₅ (x,y), its x,y range:x [1:M/2],y [1:N/2], ;3.f ₆ (x,y) starting range: x [1:M/4],y [1:N/4];4.F ₆ (x,y) input vector center point effective coordinate range: x [5:M/4-4],y [5: N/4-4]. The fourth heavy input vector F ₇ (x, y): 1. First, do (x, y) ₆ f average sub-sample into (x, y) ₇ f, the formula is as follows; 2.f ₇ (x, y )=f ₆ (x,y), its x,y range:x [1:M/4],y [1:N/4], ;3.f ₇ (x,y) starting range: x [1:M/8],y [1:N/8];4.F ₇ (x,y) input vector center point effective coordinate range: x [3:M/4-2],y [3:N/4-2]; another d(x,y) :1. d(x,y) starting range: x [1:M],y [1:N];2. d(x,y) target value center point effective coordinate range: x [20:M-19],y [20:N-19]; 3.Add extra global information, and use the MATLAB command: imresize, and use imresize to reduce the M*N image to 5*5.

The present invention will take the following two test samples for the depth value estimation experiment, test sample one: 1. double prediction (f _T = f ₄ + f ₅ , f _t = f ₄ ); 2. triple prediction (f _T = f ₄ +f ₅ +f ₆ , f _t =f ₄ ); 3. Four-fold prediction (f _T = f ₄ + f ₅ + f ₆ + f ₇ , f _t = f ₄ ). Test sample two: 1. Double prediction (f _T = f ₄ + f ₅ , f _t = f ₃ ); 2. Triple prediction (f _T = f ₄ + f ₅ + f ₆ , f _t = f ₃ ); Quadruple prediction (f _T = f ₄ + f ₅ + f ₆ + f ₇ , f _t = f ₃ ).

Then, the above test samples were subjected to two experiments, the first one was a multi-resolution prediction, and the second was an increase in the number of neurons. First, the present invention describes a multiple resolution prediction experiment.

Regarding the dual, triple and quadruple training: the Neural Network training parameters are (#neuron=700, hiddenlayer=1, epochs=200, goal=1e–5). As shown in Fig. 16, it is the multiple training time of sample one; as shown in Fig. 17, it is the depth map corresponding to sample one; as shown in Fig. 18, it is the multiple prediction of sample one. result. As shown in Fig. 19, it is the depth map corresponding to sample 2; as shown in Fig. 20, it is the multiple prediction result of sample 2.

The conclusion of Experiment 1 above is that the test sample (x, y) _t f = f ₃ (x, y) is predicted via the network trained by the training sample _T f = f ₄ (x, y) , although multiple The resolution can be effectively predicted. Double prediction (MAE can reach 8.47), triple prediction (MAE can reach 7.23), and quadruple prediction (MAE can reach 5.47). However, its prediction accuracy is not optimal. In the second experiment below, the present invention will increase the number of neurons to improve its prediction accuracy.

Experiment 2, increase the number of neurons: Neural Network training parameters are (#neuron=700, hiddenlayer=1, epochs=200, goal=1e–5). Using the best results of Experiment I, the quadruple resolution predicted to increase the number of neurons. In addition to training with 700 neurons, the present invention additionally added 750, 800 and 1000 neurons for testing. For the experimental results, please refer to the figures shown in Figures 20 to 22.

Statistics can also be used to derive the relationship between a neuron and training time and MAE. Please refer to Figures 23a and 23b.

However, the present invention can be concluded by the above experiment 2: in addition to the original fixation of 700 neurons, the experiment of increasing the number of neurons 750, 800 and 1000, the experimental results show that when the number of neurons and the number of neurons are increased, there are Helps improve the accuracy of depth prediction.

In addition, the present invention performs an experiment three, which resets the input vector and adjusts each pixel position. There are two ways: 1.zig-zag scan method; 2. head and tail connection method. After the zig-zag scan method completes the 5*5 vector, use zig-zag to reorder the position of each pixel, as shown in Figure 24a, and retrain. After the head-to-tail method is used to retrieve the 5*5 vector, the position of each pixel is reordered using the head-to-tail connection, as shown in Figure 24b, and training is resumed. The Neural Network training parameters of the above two methods are (#neuron=700, hiddenlayer=1, epochs=200, goal=1e–5). The result is shown in Fig. 25. Therefore, using zig-zag and head-to-tail to reorder each pixel position and train, the prediction result is not as good as the initial method. Based on the best method described above, it is used to make multiple image depth map estimates.

In the third experiment, a plurality of images are further used for depth map prediction, and fourteen images and depth maps (both sizes are 34*45) are selected, as shown in Fig. 26. The Inside image and the Outside image are fixed during the test. Its purpose: to observe the law of the quality of inside and outside when the number of training sheets is increased. 1.Lab01: Train two images, test two inside and (fixed) outside four. 2.Lab02: Train five images, test four inside and four (fixed) outside. 3.Lab03: Train ten images, test four inside and four (fixed) outside.

The present invention performs Lab01: training two images, testing two inside and two (fixed) Outside. The Neuralnetwork training parameters are (#neuron=200, hiddenlayer=1, epochs=200, goal=1e–5). The results are as shown in Figure 27.

The present invention is performing Lab02: training five images, testing four inside and four (fixed) Outside. The Neuralnetwork training parameters are (#neuron=500, hiddenlayer=1, epochs=200, goal=1e–5). The results are as shown in Figure 28.

The present invention performs Lab03: training ten images, testing four inside and four (fixed) Outside. The Neuralnetwork training parameters are (#neuron=1000, hiddenlayer=1, epochs=200, goal=1e–5). The results are as shown in Figure 29.

The conclusions obtained through the above experiment 3 are as follows: the Inside image (2) and the Outside image (12) selected in the present invention are discussed. First, when the present invention is trained with two images, its Inside prediction (MAE can reach 2.47), but the Outside prediction effect is poor (MAE is). Secondly, when training with five images, Inside predicts (MAE can reach 2.65) and Outside predicts (MAE is 27.25). Finally, when training with ten images, Inside predicts (MAE can reach 2.70) and Outside predicts (MAE is 10.92). Combining the above results in a result, when the number of training sheets increases, its Inside prediction will slightly degrade, but its Outside prediction will have significant progress. In addition, the present invention can use a median filter to filter out small black spots in the image to make the image look smooth. The results of the Inside prediction are post-processed. As shown in Figure 30a, the MAE is preferably adjusted to 2.03 after being corrected by the post-processing median filter.

Through the above experiments, when the number of neurons and neurons are increased, the MAE value is significantly decreased, and it is estimated that the depth map is more and more close to the actual depth map, and then the actual depth map is synthesized by DIBR technology. The interlaced stereo image of the left and right eye images is outputted on the display, as shown in Fig. 30b, which is the image selected for the present invention, the estimated depth map, and the interlaced stereo image.

In summary, the present invention relates to a "two-dimensional image depth value estimation system", which is the first to propose a new two-dimensional image depth value estimation method, using a neural network to the depth image of a two-dimensional image. It is estimated that the depth value of the depth map can be predicted effectively and accurately, and then the operating system (ie, the program software) is produced in this way, and the method and the system formed by the method are not seen in the publications or publicly used. Integrity meets the requirements of the invention patent application, and asks the Bureau to give a clear indication of the patent.

(a), (b), (c), (d), (e), (f) ‧ ‧ steps

Figure 1 is a flow chart showing the steps of the method of the present invention. Figure 2 is a system architecture diagram of the depth map training and prediction of the present invention. Figure 3a is a schematic diagram of the input vector (x, y) ₄ f and its corresponding depth value d (x, y) when the network is trained in the present invention. Of FIG. 3b: a schematic diagram of FIG. ₄ f 3a of the vector (x, y) captured by the system of the present invention. Figure 4 is a schematic representation of a neuronal model of the invention. Figure 5a is a schematic diagram of the log double bending transfer function used for the hidden layer of the neural network of the invention. Figure 5b is a schematic diagram of the tangent double bending transfer function used by the hidden layer of the neural network of the invention. Figure 5c is a schematic diagram of the linear transfer function used by the output layer of the neural network of the present invention. Fig. 6 is a schematic diagram showing the conversion of a two-dimensional vector of Fig. 3b of the present invention into one dimension. Figure 7 is an architectural diagram of the inverse transfer network created by the single-resolution depth prediction of the present invention. Figure 8a is an architectural diagram of a feedforward network created by the single-resolution depth prediction of the present invention. Figure 8b is a schematic diagram of a training neural network of the present invention. Figure 9a is a schematic diagram of predicted depth values for a neural network of the invention. Fig. 9b is a schematic diagram showing the effective range of d(x, y) and de (x, y) depth values of the present invention. Fig. 10a is a schematic diagram showing the results of single prediction (f _T = f ₄ , f _t = f ₄ ) of test sample 1 of the present invention. Figure 10b is a graphical representation of the results of a single prediction (f _T = f ₄ , f _t = f ₃ ) for the test sample 2 of the present invention. Figure 11 is a schematic diagram of multiple input vectors and their corresponding target values of the present invention. Figure 12a is a schematic diagram of multiple input vectors of the present invention. Figure 12b is a schematic diagram of the multiple resolution training process of the present invention. Figure 13 is a diagram of the inverse transfer network architecture of the multiple resolution prediction of the present invention. Figure 14 is a schematic diagram of (x,y) ₄ f&f ₅ (x,y) and its corresponding d(x,y) of the present invention. Figure 15 is a schematic diagram of (x, y) ₄ f & f ₅ (x, y) & f ₆ (x, y) and their corresponding d (x, y) . Figure 16: This is the result of the multiple training time of the present invention. Figure 17 is a depth map corresponding to the texture image of the test sample 1 of the present invention. Figure 18 is a multi-predictive result graph of test sample 1 of the present invention. Figure 19 is a depth map corresponding to the texture image of the test sample 2 of the present invention. Figure 20 is a multi-predictive result graph of Test Sample 2 of the present invention. Fig. 21 is a graph showing the results of increasing the training time of neuron quadruple prediction for the present invention. Figure 22: This is the result of increasing the depth of neuron quadruple prediction for the present invention. Figure 23a is a diagram showing the relationship between the number of neurons and the training time of the present invention. Figure 23b is a diagram showing the relationship between the number of neurons and the MAE value of the present invention. Of FIG. 24a: zig-zag scan method to set the input vector (x, y) _T f system of the present invention. Of FIG. 24b: end to end process to set the input vector (x, y) _T f system of the present invention. Fig. 25 is a schematic diagram showing the results of the depth map predicted by Initial (head-to-head), zig-zag and head-to-tail connection method. Figure 26: This is a 14-gray test image map selected for the present invention. Figure 27: This is the Lab01 of the invention: training two images, testing the results of the inside two and (fixed) Outside four. Figure 28: This is the Lab02 of the invention: training five images, testing the results of the four inside and four (fixed) Outside. Figure 29: This is the Lab03 of the invention: training ten images, testing the results of four inside and four (fixed) Outside. Figure 30a is a result of the present invention using a median filter to filter out small black spots in the image to make the image look smooth. Figure 30b is an interlaced stereo image created using the neural network of the present invention.

(a), (b), (c), (d), (e), (f) ‧ ‧ steps

Claims (10)

A method for estimating a depth value of a two-dimensional image includes: (a) a first defining step of taking the same image and defining its (x, y) _T f input vector and a corresponding depth value d (x) , y) the target vector; (b), a network creation step, establish a type of neural network and parameterize it; (c), a training network step, enter the (x, y) _T f Inputting the target vector value of the vector and its depth value, thereby training the neural network to obtain a NN weight value; (d), a second defining step, defining (x, y) _t f of the sample image Input vector; and (e), an output step, inputting the (x, y) _t f input vector to the trained neural network, and loading the NN weight value to obtain an estimate of the sample image Measure the depth value de(x,y) . According to the method for estimating the depth value of the two-dimensional image described in claim 1, the method further comprises a step (f), wherein the step (f) is a comparison step according to the depth value de(x, y ) , according to the evaluation of the training effect of this type of neural network, if it meets the demand, stop training the neural network and use it; if it does not meet the demand, the input is (x, y) _T The f-quantity is set to multiple input vectors, or / and increase the number of neurons in this type of neural network, and retrain the neural network to meet the demand. According to the estimation method of the two-dimensional image depth value described in Item 2 of the patent application, the Mean absolute error (MAE) is used to evaluate the training effect of the neural network. According to the method for estimating the depth value of the two-dimensional image described in claim 2, wherein the (x, y) _T f multiple input vector in the step (f) is set as a first heavy input vector, a first A double input vector, a third heavy input vector, and a fourth heavy input vector. According to the method for estimating the depth value of the two-dimensional image described in claim 4, wherein the input method of the input vector (x, y) _T f in the step (f) is a head-to-head method and a zig-zag method. Or head-to-tail connection. According to the estimation method of the two-dimensional image depth value described in claim 2, wherein the number of neurons in the step (f) is at least 700. The estimation method of the two-dimensional image depth value according to claim 1 or 2, wherein the neural network in the step (b) is set as a reverse transmission network and a feedforward network is created. combination. According to the method for estimating the depth value of the two-dimensional image described in claim 7, wherein a hidden layer inside the neural network uses a log double bending transfer function (logsig) or a tangent double bending transfer function ( Tansig), an output layer set up inside such a neural network uses a linear transfer function (purelin). The method for estimating a two-dimensional image depth value according to claim 1 or 2, wherein the NN weight value comprises a weight (W) and an offset value (b). A system for estimating a depth value of a two-dimensional image is a system constructed by using the estimation method according to any one of the above-mentioned claims.