JP2024002745A - Information processing device, area division method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, an area division method and a program capable of dividing an area of traffic congestion on a road from an image of an aerial photograph.

SOLUTION: An information processing device includes: an aerial photograph collection part 10 for acquiring an image photographed from the sky; an algorithm calculation part 120 for simultaneously dividing a road area and a congestion area from each other from the image by using a model of a neural network; an output processing part 130 for outputting a division result obtained by the algorithm calculation part 120; and a learning part 140 for inputting the image of the aerial photograph to the model and adjusting the parameter of the model so as to minimize the error between the output from the model and a correct answer.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to technology for extracting road information from aerial photographic images.

By detecting traffic congestion and estimating traffic density from aerial photographic images, real-time traffic status information can be provided to city monitoring systems and drivers. With such traffic condition information, for example, an appropriate driving route can be determined.

The technology disclosed in Non-Patent Document 2, which is a conventional technology for detecting traffic congestion, discloses a technology that treats traffic congestion detection as a classification problem by estimating traffic density from images taken with cameras installed at intersections. .

In addition, the technology disclosed in Non-Patent Document 3 uses an open source application programming interface (API) provided by the Land Transport Authority (LTA) to collect data and estimate traffic density. proposed a CNN (convolutional neural network).

However, since the images used in Non-Patent Documents 2 and 3 were taken by traffic cameras (cameras installed at intersections, etc.), they can only provide road condition information for a very small area.

Also, many methods have been proposed for extracting roads from aerial photography images based on semantic segmentation. For example, Non-Patent Document 1 discloses a technique in which road extraction is divided into three interrelated subtasks: road surface segmentation, road edge detection, and road centerline extraction. However, although it is possible to extract the road network, it is not possible to segment (divide) areas of traffic congestion from aerial photographs.

Y. Liu, J. Yao, X. Lu, M. Xia, X. Wang, and Y. Liu. RoadNet: Learning to Comprehensively Analyze Road Networks in Complex Urban Scenes from High-Resolution Remotely Sensed Images. IEEE Transactions on Geoscience and Remote Sensing, 57(4):2043-2056, 2019. J. Nubert, N. G. Truong, A. Lim, H. I. Tanujaya, L. Lim, M. A. Vu, "Traffic Density Estimation using a Conbolutional Neural Network," arXiv preprint arXiv : 1809.01564, 2018. M. Hasan, S. Das and M. N. T. Akhand, "Estimating Traffic Density on Roads using Convolutional Neural Network with Batch Normalization," 2021 5th International Conference on Electrical Engineering and Information Communication Technology (ICEEICT), 2021, pp. 1-6, doi : 10.1109/ICEEICT53905.2021.9667860.

The present invention has been made in view of the above points, and an object of the present invention is to provide a technique that makes it possible to classify areas of traffic congestion on a road from an aerial photographic image.

According to the disclosed technology, an acquisition unit that acquires an image taken from above;
a calculation unit that simultaneously divides the image into a road area and a traffic congestion area using a neural network model;
An information processing device is provided, comprising: an output processing section that outputs the classification results obtained by the calculation section.

According to the disclosed technology, a technology is provided that makes it possible to classify congested areas on a road from an aerial photographic image.

FIG. 1 is a configuration diagram of an information processing device in an embodiment of the present invention. It is a flowchart for explaining the flow of processing. FIG. 2 is a diagram showing the overall configuration of a context-enhanced traffic segmentation model. FIG. 3 is a diagram showing the configuration of an original transportation module. FIG. 3 is a diagram showing the configuration of an attention calculation block. It is a diagram showing an example of the hardware configuration of the device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention (this embodiment) will be described below with reference to the drawings. The embodiments described below are merely examples, and embodiments to which the present invention is applied are not limited to the following embodiments.

Note that in this specification and claims, "division," "segmentation," "extraction," "classification," "classification," and "segmentation" may be used interchangeably. In other words, "division," "segmentation," "extraction," "classification," "classification," and "segmentation" described in the specification or claims each refer to any other of these. May be replaced.

Furthermore, "aerial photograph" may be replaced with "aerial photograph." An aerial photo image may also be referred to as an "aerial image" or "aerial image." An aerial photograph/aerial photograph is a photograph taken of the ground from a flying object in the sky, and the flying object is not limited to a specific one. For example, the flying object may be an airplane, a satellite, or a drone.

Furthermore, "enhance" has the meaning of, for example, increasing the accuracy of classifying congested areas, or clarifying the boundaries between congested areas and non-congested areas.

(Equipment configuration example, operation example)
FIG. 1 shows a configuration example of an information processing apparatus 100 in this embodiment. As shown in FIG. 1, the information processing device 100 includes an aerial photo collection section 110, an algorithm calculation section 120, an output processing section 130, and a learning section 140. Note that the aerial photo collection unit 110 may also be referred to as an acquisition unit. Further, the algorithm calculation unit 120 may be called a calculation unit.

The flow of processing at the time of inference (during testing) by the information processing apparatus 100 will be described with reference to the flowchart of FIG. 2. In S101, the aerial photo collection unit 110 acquires photos or videos taken by a drone, satellite, aircraft, or the like. These photographs and videos will be collectively referred to as "aerial photographs." The aerial photo image acquired by the aerial photo collection unit 110 is input to the algorithm calculation unit 120.

The algorithm calculation unit 120 has a neural network model (end-to-end model), which will be described later. It is assumed here that the model has already been trained. In S102, the algorithm calculation unit 120 inputs the aerial photograph image to the model, and divides the road area and the congestion area (the road area into the congestion area and the non-congestion area) as an output from the model. Obtain a segmented image.

In S103, the output processing unit 130 may output the image obtained by the algorithm calculation unit 120 (the image in which the road area and the traffic congestion area are divided) as is, or may perform processing on the image and output the image after the processing. You may also output an image of For example, the output processing unit 130 can extract and output only the congested area on the road from the image obtained by the algorithm calculation unit 120.

When training the model, a large amount of aerial photographic images and their label data (for example, images labeled with roads, traffic jams, and other than traffic jams) are used. The learning unit 140 inputs an aerial photograph image to the model, and adjusts the parameters (weights) of the model so that the error between the output from the model and the correct answer is minimized.

Note that the device that performs learning (the device that includes the learning section 140) and the device that performs inference may be separate devices. In this case, the device that performs learning may be called a learning device. Hereinafter, the configuration and operation of the algorithm calculation section 120 will be explained in detail. Furthermore, the device that performs inference does not need to include the learning unit 140.

The algorithm calculation unit 120 has a neural network model. Using this model, it is possible to simultaneously segment the road surface and the congested area on the road surface on an aerial photograph image.

In aerial photographic images, vehicle segmentation is generally very difficult because the size (scale) of a vehicle seen from above is smaller than the size of a road surface when seen from above. This is called the scale variation problem. Therefore, in the conventional technology, it is very difficult to accurately segment the boundary between a congested area and a non-congested area.

The model constituting the algorithm calculation unit 120 according to the present embodiment solves the above problems and is capable of accurately segmenting a congested area from an aerial photograph image.

The configuration and operation of the context-enhanced traffic segmentation model in this embodiment will be described in detail below. Hereinafter, for convenience of description, the context-enhanced traffic segmentation model may be referred to as a "model".

(Overall configuration of model)
Figure 3 shows an example of the overall configuration of the context-enhanced traffic segmentation model. As shown in FIG. 3, this model includes a feature pyramid network (FPN) 210, an original traffic module 220, and a context attention module 230.

The context attention module 230 includes a Global Context Generator 240 and an Attention Computation Block 250.

Since multi-level prediction is effective for scale variation problems, an aerial photographic image is first input into a feature pyramid network 210 that performs multi-level prediction. The feature pyramid network 210 generates a feature pyramid consisting of five feature maps (P2 to P6) of different scales from the input image. The P6 feature map contains the highest (top) level semantic information.

The five feature maps (P2 to P6) are input to the original traffic module 220, which generates segmentation results of Road Surface and Original Traffic Jam.

Further, the feature map of P6 is input to the global context generator 240, and the global context generator 240 generates a global context feature from the feature map of P6.

The global context features and the original congestion segmentation results are input to the attention calculation block 250, which outputs an enhanced (enhanced) congestion segmentation (congested region).

By combining the enhanced traffic congestion segmentation and the road surface segmentation obtained by the original traffic module 220, the final output can be obtained.

(Original transportation module 220)
Next, the original transportation module 220 will be explained. In order to divide (segment) a congested area on a road, it is necessary to segment a group of vehicles on the road. However, when viewed from above, the scale of the vehicle relative to the road is small, causing a scale variation problem. The original traffic module 200 has a multi-scale feature fusion network configuration to address this problem.

FIG. 4 shows an example of the configuration of the original transportation module 220. As shown in FIG. 4, the original traffic module 220 includes a convolution layer 221 and a fusion layer 222. Convolution layers 221 include three consecutive 3x3 convolution layers for each feature map.

As shown in FIG. 4, each feature map PεR ^256×H×W (size: 256×H×W) is input to the convolution layer 221. The same convolution process is performed on each feature map. The convolution process generates a new feature map ^~ P∈R ^1×H×W . Note that in the text of this specification, for convenience of description, symbols written at the beginning of letters are written before the letters. “ ^～ P” is an example.

Next, each feature map is resized to the original input R ^h×w scale by bilinear interpolation. After that, the five feature maps are fused by concatenation, and the fused feature map is input to one 3×3 convolutional layer (fusion layer 222), thereby creating a traffic congestion map A∈R ^1×h×w and a road surface map. Obtain the original segmentation result with B∈R ^1×h×w .

(Context attention module 230)
Next, the context attention module 230 will be described.

Generally, in an aerial photographic image, the boundary between a congested area on a road and other areas is ambiguous, so it is difficult to explicitly distinguish between a congested area and a passable road area. Contextual attention module 230 solves this problem and makes the boundaries between congested areas and other areas clear. The context attention module 230 refines the original congestion map obtained by the original traffic module 220 by using the global context feature map to clarify the boundaries.

Global context module 240 includes a pyramid pooling module (PPM). As mentioned above, feature map P6 has the most powerful semantic information, and global context module 240 receives feature map P6 as input.

That is, first, the global context module 240 applies a pyramid pooling module (PPM) to the feature map of P6 to further exploit the region representation and context dependencies. The global context module 240 obtains a global context feature map C∈R ^1×h×w .

The pyramid pooling module uses multiple grids with pyramid-shaped size hierarchies to perform pooling on the input, thereby generating a global model that shows how many features of each class are included in which grid. (Global) rough context information can be obtained.

The global context feature map and the original congestion map are input to the attention calculation block 250.

FIG. 5 shows the processing configuration of the attention calculation block 250. A neural network is configured to enable this processing.

As shown in FIG. 5, first, the original traffic congestion map A and the global context feature map C are each downsampled to obtain { ^−A , ^−C }∈R ^1×h/4×w/4 . Next, these are reshaped (transformed) to obtain two new feature maps { ^~ A, ^~ C}∈R ^1×n . Here, n is n=h/4×w/4 and indicates the number of pixels in the feature map.

As shown in FIG. 5 and Equation (1) below, matrix multiplication is performed between ^~ A and the transposed ^~ C, and the context attention map S∈R ^n×n is calculated by the softmax layer.

S=Softmax( ^~ C ^T × ^~ A) (1)
In order to enhance (clarify) the congestion area using context information, as shown in ^{FIG .} Reshape to ^4×w/4 . Then ^- add A to obtain the context-enhanced congestion map ^~ A ^S ∈R ^1×h/4×w/4 . Here, α is a learnable weight parameter initialized as 0.

^~ A ^S = α (Reshape ( ^~ A × S)) + ^- A (2)
Then resize ^~ A ^S to obtain the final enhanced congestion segmentation result A ^S ∈R ^1×h×w .

The above is the processing of the context attention module 230. Finally, the enhanced congestion segmentation and the original road surface segmentation are combined and input into a 3×3 convolution layer to generate the final congestion segmentation result. In the final traffic congestion segmentation result, for example, a road area is divided and shown on an aerial photograph image, and a congestion area and a non-congestion area in the road area are also shown divided.

(Summary of context-enhanced traffic segmentation model)
As described above, in this embodiment, the context-enhanced traffic segmentation model divides (classifies) traffic congestion and road surfaces from an aerial photographic image using an end-to-end method. The "context-enhanced traffic segmentation model" is a traffic segmentation model whose performance is enhanced by context.

The model in this embodiment consists of two modules (original traffic module 220 and context attention module 230) that make it possible to explicitly segment traffic congestion and road surface.

The original traffic module 220 is a module for solving the scale variation problem in aerial photographic images. That is, this module utilizes a multi-scale feature map based on feature pyramids to extract further features by the convolutional layer 221. Then, the fusion layer 222 fuses the features at different scales to obtain the original (initial) segmentation of the traffic congestion and road surface.

Contextual attention module 230 enforces traffic jam boundaries. Context attention module 230 consists of an attention calculation block 250 and a corresponding global context generator 240. The feature map at the top level of the feature pyramid contains the strongest semantic information. Therefore, it is input to the global context generator 240 and a global context map is obtained through a pyramid pooling operation. Then, in attention calculation block 250, an attention map between the global context map and the original segmentation of the traffic jam is calculated. Finally, the attention map is used to strengthen the boundaries of traffic congestion (clarify the boundaries) and obtain the final traffic congestion segmentation result.
This allows the road surface and the congested area to be simultaneously and accurately classified. It also solves the problem of scale in aerial photographic images, where the scale of the vehicle is smaller than the scale of the road surface seen from the air, making segmentation of the vehicle very difficult.

(Hardware configuration example)
The information processing device 100 can be implemented, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. . The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. It is also possible to provide the above program through a network such as the Internet or e-mail.

FIG. 6 is a diagram showing an example of the hardware configuration of the computer. The computer in FIG. 6 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are interconnected by a bus BS.

A program for realizing processing by the computer is provided, for example, by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily need to be installed from the recording medium 1001, and may be downloaded from another computer via a network. The auxiliary storage device 1002 stores installed programs as well as necessary files, data, and the like.

The memory device 1003 reads the program from the auxiliary storage device 1002 and stores it when there is an instruction to start the program. The CPU 1004 implements functions related to the information processing apparatus 100 according to programs stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, various measuring devices, exercise intervention devices, and the like. A display device 1006 displays a GUI (Graphical User Interface) or the like based on a program. The input device 1007 is composed of a keyboard, a mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. An output device 1008 outputs the calculation result.

(Effects of embodiment)
With the technology according to the present embodiment, road areas and congestion areas can be automatically output, instead of extracting road and congestion information from images with the human eye as in the past. In addition, it is possible to divide the image into regions with high precision, taking into account the problem of scale variation on conventional images (small cars, large roads, etc.). Furthermore, the boundary line that divides the regions can also be displayed smoothly.

Further, the segmentation map obtained by the technology according to the present embodiment allows visually understanding the location of traffic jams and the ratio of the area occupied by the traffic jam to the road area. Furthermore, even if there is traffic jam, it is possible to know which places are passable, so it can be determined, for example, whether an emergency vehicle can pass. These points are superior to conventional techniques for detecting traffic jams and estimating density.

(Additional note)
Regarding the above embodiments, the following additional notes are further disclosed.
(Additional note 1)
memory and
comprising a processor;
The processor includes:
Obtain images taken from above,
simultaneously classifying road areas and congestion areas from the image using a neural network model;
An information processing device that outputs the obtained classification results.
(Additional note 2)
The model is
a first module that classifies a road area and a traffic congestion area in the image using a plurality of feature maps obtained from the image;
The information processing device according to supplementary note 1, further comprising: a second module that enhances the congested area obtained by the first module using a specific feature map among the plurality of feature maps.
(Additional note 3)
The information processing according to appendix 2, wherein the plurality of feature maps are generated by a feature pyramid network included in the model, and the specific feature map is the highest level feature map among the plurality of feature maps. Device.
(Additional note 4)
The second module is
a context generator that generates a global context from the specific feature map;
the information according to supplementary note 2 or 3, comprising: an attention calculation block that uses the global context and the congestion area obtained by the first module to generate a congestion area with higher accuracy than the congestion area; Processing equipment.
(Additional note 5)
An area segmentation method executed by an information processing device, the method comprising:
an acquisition step of acquiring an image taken from above;
a calculation step of simultaneously classifying a road area and a traffic congestion area from the image using a neural network model;
An area segmentation method comprising: an output step of outputting the segmentation result obtained by the calculation step.
(Additional note 6)
A non-temporary storage medium storing a program for causing a computer to function as each part of the information processing apparatus according to any one of Supplementary Notes 1 to 4.

Although the present embodiment has been described above, the present invention is not limited to such specific embodiment, and various modifications and changes can be made within the scope of the gist of the present invention as described in the claims. It is possible.

100 Information processing device 110 Aerial photo collection unit 120 Algorithm calculation unit 130 Output processing unit 140 Learning unit 210 Feature pyramid network 220 Original traffic module 221 Convolution layer 222 Fusion layer 230 Context attention module 240 Global context generator 250 Attention calculation block 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims (6)

an acquisition unit that acquires an image taken from the sky;
a calculation unit that simultaneously divides the image into a road area and a traffic congestion area using a neural network model;
An information processing device comprising: an output processing section that outputs the classification results obtained by the calculation section. The model is
a first module that classifies a road area and a traffic congestion area in the image using a plurality of feature maps obtained from the image;
The information processing apparatus according to claim 1, further comprising: a second module that uses a specific feature map among the plurality of feature maps to strengthen the congested area obtained by the first module. The information processing according to claim 2, wherein the plurality of feature maps are generated by a feature pyramid network included in the model, and the specific feature map is a top-level feature map among the plurality of feature maps. Device. The second module includes:
a context generator that generates a global context from the specific feature map;
The information processing device according to claim 2, further comprising: an attention calculation block that uses the global context and the congestion area obtained by the first module to generate a congestion area with higher accuracy than the congestion area. . An area segmentation method executed by an information processing device, the method comprising:
an acquisition step of acquiring an image taken from above;
a calculation step of simultaneously classifying a road area and a traffic congestion area from the image using a neural network model;
An area segmentation method comprising: an output step of outputting the segmentation result obtained by the calculation step. A program for causing a computer to function as each part of an information processing apparatus according to any one of claims 1 to 4.

Images