KR101938491B1 - Deep learning-based streetscape safety score prediction method - Google Patents

Abstract

The present invention relates to a deep learning-based street safety score prediction method comprising: (a) a step of constructing a city image data set consisting of a plurality of city image data; (b) a step of applying safety base data assigned to the city image data to a preregistered ranking determination algorithm to calculate a safety score of the city image data; (c) a step of applying the city image data to a saliency estimation algorithm to extract an environmental context map; and (d) a step of applying the environmental context map and the safety score for the city image data to a convolutional neural network to train the convolutional neural network. Accordingly, environmental context which is an abstract high-level feature is extracted, and city safety can be accurately predicted by using the environmental context.

Description

[0001] DEEP LEARNING BASED STREETSCAPE SAFETY SCORE PREDICTION METHOD [0002]

The present invention relates to a method for predicting a degree of distance safety based on deep learning,

Over the past several decades, various studies have been conducted in order to revitalize the local economy of the city. These studies have created attractive cities that attract tourists, such as attempts to make the city a space of pleasure and consumption through the creation of the night-time economy (NTE) concept, and attempts to obtain economic effects by hosting local festivals It is focused on.

It is known that creating a safe space positively affects the number of visitors and consumer spending, so that mitigation of fear of crime plays an important role in revitalizing the local economy. Therefore, in order to reduce fear of crime, researches on crime occurrence or safety prediction are under way.

In the past, this analysis required a lot of information and costs. However, as computer vision and big data developed recently, it became possible to predict the urban environment from the images. In particular, in relation to the prediction of crime, Bachner et al., "Predictive policing: "The Santa Cruz Experiment: Can a City's Crime Be Predicted (Popular Science, 2011, 2011-10)" by Thompson et al. I am receiving great attention.

There is a method based on Wilson and Kelling's "broken window theory" in predicting city safety. The theory of broken windows is that crime spreads around the point if you leave a broken window on the street. If you leave a minor disorder, it will spread to the whole city. In other words, a place that is visually disordered in the city can be judged to have a low degree of safety. Based on these theories, we tried to evaluate the safety of urban distance images using visual perception and to learn and predict the values through machine learning algorithms.

On the other hand, various studies on the prediction of crime and the safety prediction of the city are being conducted. In the above-mentioned paper by Bachner et al., We use business technologies developed by retailers such as Netflix and WalMart to predict consumer behavior and predict crime based on this.

In addition, Thompson et al. Described in the above-mentioned study has been applied to Santa Cruz, California, to enable effective and efficient crime response even with a small number of people, and predicts the time and place where crime is likely to occur based on the Big Data .

Most of these existing studies are conducted in three stages. First, data to be used for learning are collected. Records of past crimes, questionnaires, SNS, and images are mainly used as learning data, and then features are extracted from the data. The characteristics of crime occurrence time and location are searched. When images are used as data, global features such as color and gradient are extracted. Finally, a prediction model is generated based on the extracted features.

Most existing studies have obtained low level features such as color, gradient, and texture through global feature extraction in urban images. Therefore, there are limitations in predicting abstract information such as safety assessment of urban environment.

Accordingly, the present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a system and method for extracting an environmental context, which is an abstract high-level feature, Based distance safety score prediction method.

According to another aspect of the present invention, there is provided a method of predicting distance learning based on depth learning, the method comprising the steps of: (a) constructing a city image data set composed of a plurality of city image data; (b) calculating a safety degree score of each of the plurality of pieces of the urban image data by applying a safety degree basic data to each of the plurality of the urban image data to a pre-registered ranking algorithm; (c) extracting an environmental context map by applying each of the image data to a saliency estimation algorithm; (d) extracting an environmental context map for each of the image data; Wherein the degree of safety score is applied to a convolutional neural network to learn the convolutional neural network.

Here, in the step (b), the safety-based data is given through a result of a pairwise comparison test on two pieces of the urban image data; The ranking algorithm may include a Trueskill algorithm.

In the step (c), the upper 30% of the urban image data is extracted as the safety image data and the lower 30% of the urban image data is extracted as the safety data based on the safety degree score, and the saliency estimation algorithm Lt; / RTI >

In the step (d), a loss between the estimated safety degree estimated by the convolutional neural network and the safety degree score may be calculated and reflected in the learning.

In addition, the loss can be calculated through an Euclidean loss.

According to the present invention, according to the present invention, it is possible to extract an environmental context which is an abstract high-level feature and to use it to predict a degree of safety of a city, A score prediction method is provided.

FIG. 1 is a view for explaining a method of predicting a depth-based distance safety score according to the present invention,
2 is a diagram for explaining a process of determining the safety score of the urban image data in the depth learning based distance safety score prediction method according to the present invention,
3 is a view for explaining a process of extracting an environmental context map in the depth learning based distance safety score prediction method according to the present invention,
4 is a diagram for explaining a CS reconstruction model according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram for explaining a depth learning based distance safety score prediction method according to the present invention.

First, a plurality of city image data is collected to form a city image data set (S10). In the present invention, the Place Pulse 1.0 data and the urban image data of P. Salesses et al., "The collaborative image of the city: mapping the inequality of urban perception (PloS one 8.7 (2013): e68400) A city image data set including urban image data is constructed.

When the urban image data set is constructed, safety comparison basic data for each of the urban image data is obtained by performing a pairwise comparison on a plurality of participants (S11). More specifically, as shown in FIG. 2A, the participant is allowed to show two pieces of the urban image data and select which one looks safe (or the same). In the present invention, a total of 32 participants using the above-described 2,286 urban image data are subjected to a 23361 pair comparison experiment to acquire safety-based data according to visual recognition.

Then, the safety degree data obtained through the pair comparison test is applied to the Trueskill algorithm to calculate the safety score of each of the urban image data (S12). The Trueskill algorithm uses a Bayesian graphical model to determine the ranking of a player in an online game. In the present invention, the selected urban image data is regarded as a winner in a one-to-one competition according to a query given in the pair comparison experiment.

으로 모델링되며 실험이 진행될 때마다 업데이트 된다. 두 명의 플레이어, 즉 두 개의 도시 영상 데이터 x와 y에 대해서 플레이어 x가 y를 이겼을 경우 업데이트는 [수학식 1] 및 [수학식 2]를 통해 수행된다.The skill of each city image data is

And is updated as the experiment progresses. If the player x has won y with respect to two players, that is, two city image data x and y, the updating is performed through [Equation 1] and [Equation 2].

[Equation 1]

&Quot; (2) "

,

이다.

는 사전 정의된 상수로 게임 별 편차를 의미하며,

은 경험적으로 추정된 x, y 두 도시 영상 데이터가 비길 확률이다. 함수

와

는 정규 확률 밀도 함수

와 정규 누적 밀도 함수

가 적용되었다.Here, trueskill of x and y

,

to be.

Means a game-specific deviation with a predefined constant,

Is the likelihood ratio of the two urban image data, which are empirically estimated x and y. function

Wow

Normal probability density function

And normal cumulative density function

Respectively.

을 초기값으로 적용하였고,

를 적용하였다. [수학식 1]은 도시 영상 데이터의 평균값을, [수학식 2]는 도시 영상 데이터의 분산 값을 업데이트 한다. [수학식 1] 및 [수학식 2]를 통해 업데이트 되는 도시 영상 데이터들의 trueskill에 대해서 평균값을 의미하는

가 최종 순위를 결정하는데 사용된다.In the present invention,

As initial values,

Respectively. Equation (1) updates the average value of the urban image data, and (2) updates the variance value of the urban image data. Means an average value for the trueskill of the urban image data updated through [Equation 1] and [Equation 2]

Is used to determine the final ranking.

In the present invention, it is assumed that the final ranking of the urban image data is normalized to a value between 0 and 10, and the safety score of each urban image data is calculated. Fig. 2 (b) shows an example of the degree of safety calculated for each piece of the urban image data.

As described above, when the safety degree score is calculated for the city image data, the process of extracting the environmental context map from the city image data (S14) is performed. In the present invention, the upper 30% of the safety score is extracted as the safety image data, and the lower 30% of the safety score is extracted as the dangerous image data to extract the environmental context map (S13).

3 is a diagram for explaining a process of extracting an environmental context map in the depth learning based distance safety score prediction method according to the present invention. Referring to FIG. 3, in the present invention, as disclosed in Xia et al., "Bottom-up visualization estimation with deep autoencoder-based sparse reconstruction (IEEE Transactions on Neural Networks and Learning Systems 27.6 (2016): 1227-1240) An example is to extract a saliency map using the Saliency Estimation Algorithm.

The silence prediction algorithm extracts 8,000 training images having a size of 16 × 16 × 3 arbitrarily from the input urban image data and generates 8000 target images having a size of 8 × 8 × 3 at the center of each sample image . Then, the C-S reconstruction model is learned through the training image and the target image.

4 is a diagram for explaining a C-S reconfiguration model according to the present invention. Referring to FIG. 3 and FIG. 4, the C-S reconstruction model is composed of four encoder layers, four decoder layers, and two inference layers.

The encoder layer repeats convolution and Max-pooling twice as shown in FIG. 4 (a), and the decoder layer repeats Max-unpooling and decon- Repeat the deconvolution twice. In the inference layer, the target image is inferred based on the urban image data obtained through the autoencoder, and a completely connected layer is used. As shown in FIG. 4 (b) 768, and 192, respectively. The neuron thus obtained has an output size of 8 × 8 × 3, which corresponds to the size of the target image. Here, the results of each layer are applied to ReLU and dropout techniques, and 3 × 3 and 5 × 5 kernels are used for convolution, respectively.

Then, using the CS reconstruction model that has been learned, the residual map between the predicted image and the original image is obtained as a silence map for the entire input image, and the original image is multiplied by the silence map to eliminate unnecessary information, And extracts an environmental context map as shown in FIG. 4 (c).

When the extraction of the environmental context map is completed for the safety image data and the desired image data as described above (S14), the extracted environmental context map is applied to the Convolutional Neural Network (S15) Convolutional Neural Network).

The convolutional neural network according to the present invention is exemplified by two convolution layers and two Max-pooling layers, and the result of each side is applied to ReLU and dropout techniques as an example do. In the fully connected layer of the convolution neural network, the loss can be calculated using the difference between the safety score of the observers participating in the experiment and the predicted score of the model.

More specifically, the first convolution layer of the convolution neural network obtains 96 feature maps using a 11 × 11 × 3 kernel on Stride 4, and the second convolution layer is the Max- Using a 27 × 27 × 96 size image as input, 256 feature maps are obtained through a 3 × 3 × 96 kernel on Stride 2. And the Max-Pulling is again performed on the result of the second convolution layer.

The total connectivity layer, which is connected to the convolution layer, is composed of 4096 and 1 neuron, respectively. To estimate the human perceived safety score, the number of neurons in the last complete connection layer is set to 1, An example is the calculation through loss (Euclidean loss).

Through the extraction of the environmental context map and the learning in the convolutional neural network using the environment context map, the method of predicting the distance safety score based on the deep learning is constructed and the safety of the newly input urban image can be more accurately predicted .

Although several embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various modifications may be made without departing from the principles and spirit of the invention . The scope of the invention will be determined by the appended claims and their equivalents.

Claims (5)

A depth learning based distance safety score prediction method implemented by a computer program,
(a) registering a city image data set composed of a plurality of city image data in the computer program;
(b) a step of calculating a safety degree score of each of the city image data by applying a safety degree basic data given to each of the city image data to a ranking algorithm previously registered in the computer program,
(c) applying the respective urban image data to a saliency estimation algorithm previously registered in the computer program to extract an environmental context map;
(d) the environment context map and the safety degree score for each of the city image data are applied to a convolutional neural network previously registered in the computer program, so that the convolutional neural network is learned. Based safety score prediction method. The method according to claim 1,
In the step (b), the safety-based data is given through a result of a pairwise comparison experiment on two pieces of the urban image data;
Wherein the ranking algorithm includes a Trueskill algorithm. ≪ RTI ID = 0.0 > 11. < / RTI > The method according to claim 1,
In step (c), the upper 30% of the urban image data is extracted as the safety image data and the lower 30% of the urban image data is extracted as the safety data based on the degree of safety score, and the data is applied to the saliency estimation algorism Wherein the step of estimating a distance safety score based on the depth learning is performed. The method according to claim 1,
In the step (d), the loss between the estimated safety degree estimated by the convolutional neural network and the safety degree score is calculated and reflected in the learning. 5. The method of claim 4,
Wherein the loss is calculated through an Euclidean loss. ≪ Desc / Clms Page number 20 >

Images