KR102383566B1 - Method and system for learning lane changeable time based on camera and method and system for predicting time lane changeable time - Google Patents

Abstract

Disclosed are a method and system for learning a camera-based lane change timing, and a method and system for predicting a lane change timing. The method of learning a lane change time according to an embodiment of the present invention comprises the steps of receiving an image input through a camera in relation to a side-rear area of a vehicle and assigned a first label or a second label to individual frames (the first step) The image to which the first label is assigned includes an image indicating a state in which lane change is blocked (BLOCKED), and the image to which the second label is assigned includes an image indicating a state in which lane change is free (FREE), the Lane change occurs after n seconds (where n is a natural number) in at least one frame before the transition time according to the number of frames per second (FPS) based on the transition time from the blocked state to the free state. The method may include assigning a label defining the possibility and learning the lane change timing based on the frames of the image and labels included in the frames.

Description

A method and system for learning when a camera-based lane change is possible, and a method and system for predicting when a lane change is possible

The following description provides a method and system for learning a camera-based lane change time, and a method and system for predicting a lane change time.

As technologies for lane change assistance, blind spot detection (BSD), blind spot information system (BLIS), and the like exist. However, in these conventional technologies, in the case of a device (or system) using a radar sensor, the sensor itself is expensive, and the device is buried in the exterior (eg, bumper) of the vehicle, so that it is damaged even in a minor accident or the calibration information is damaged. There is a problem in that replacement or reinstallation is required. Therefore, it has a disadvantage that the system installation and maintenance cost is high. On the other hand, the camera sensor has a competitive advantage in terms of price, and the proposed system has robust characteristics with respect to the installed location. More specifically, in the case of a radar sensor, there is a disadvantage in that a recognition error for a stationary object and a very close object increases due to the physical characteristics of the corresponding sensor. In addition, the existing BLIS has a problem of notifying the driver only when it is impossible to change a lane depending on the occupancy state of the corresponding lane.

It receives an image from a camera installed to obtain a viewpoint similar to the side-rear area observed with the side mirror of the vehicle and collects information on the time of lane change (for example, 3 seconds ago, 2 seconds ago, 1 second ago, possible, impossible). It provides a camera-based lane change timing prediction/learning method and system that can predict and notify the driver or autonomous driving AI.

Step of receiving an image to which the first label or the second label is assigned to each frame input through the camera in relation to the side-rear area of the vehicle - The image to which the first label is assigned is blocked from changing lanes an image indicating the state, and the image to which the second label is assigned includes an image indicating a state in which lane change is free; Lane changes after n seconds (where n is a natural number) in at least one frame before the transition time according to the number of frames per second (FPS) based on the transition time from the blocked state to the free state assigning a label defining that this is possible; and learning the lane change timing based on the frames of the image and labels included in the frames.

determining a current state related to a lane change by analyzing an image input through a camera in relation to a side-rear area of the vehicle; and providing different types of notifications for a state in which a lane change is possible and a state in which a lane change is free after n seconds (where n is a natural number) as the current state. do.

Provided is a computer program stored in a computer-readable recording medium in combination with a computer to execute the method on the computer.

There is provided a computer-readable recording medium in which a program for executing the method in a computer is recorded.

A computer device comprising: at least one processor embodied to execute computer-readable instructions, by the at least one processor being inputted through a camera in relation to a side-rear area of a vehicle to receive a first message in a separate frame An image to which a label or a second label is assigned is received - the image to which the first label is assigned is an image indicating a state in which lane change is blocked (BLOCKED), and the image to which the second label is assigned is free of lane change Each includes an image indicating a free state - at least one before the transition time according to the number of frames per second (Frame Per Second, FPS) based on the transition time from the blocked state to the free state A computer device characterized in that a label defining that lane change is possible after n seconds (where n is a natural number) is given to the frame of provides

A computer device comprising: at least one processor implemented to execute computer-readable instructions; A current state related to change is determined, and as the current state, different methods are provided for a state in which a lane change is possible and a state in which a lane change is free (FREE) after n seconds (where n is a natural number) as the current state. A computer device is provided.

It receives an image from a camera installed to obtain a viewpoint similar to the side-rear area observed with the side mirror of the vehicle and collects information on the time of lane change (for example, 3 seconds ago, 2 seconds ago, 1 second ago, possible, impossible). It can predict and alert the driver or autonomous driving AI.

1 is a diagram illustrating an example of annotated images according to an embodiment of the present invention.
2 shows an example of comparing the attention result of the CAM with the attention result of the method according to an embodiment of the present invention.
3 is a diagram illustrating an example of an integration method according to an embodiment of the present invention.
4 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention.
5 is a flowchart illustrating an example of a method for learning a lane change time point according to an embodiment of the present invention.
6 is a flowchart illustrating an example of a method for predicting a lane change time point according to an embodiment of the present invention.
7 is a diagram illustrating an example of assigning a label according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

The lane change time learning method according to embodiments of the present invention may be performed by at least one computer device to be described later. In this case, the computer program according to an embodiment of the present invention may be installed and driven in the computer device, and the computer device performs the lane change time learning method according to the embodiments of the present invention under the control of the driven computer program. can The above-described computer program may be stored in a computer-readable recording medium in combination with a computer device to execute the method according to the embodiments of the present invention in the computer device.

In embodiments of the present invention, weak supervision is used to overcome severe data imbalance issues by extracting vehicle-aware image features and few-shot classification. We propose a system for lane-changing assistance. For self-driving agents and human drivers, the proposed system can monitor the side-to-rear space and predict when a lane change is possible. By using weakly supervised learning and attention maps, the system can extract new image features of objects moving behind the ego-vehicle. In addition to converting binary labels into quantized time slots, a side-back view image dataset to which binary labels (FREE or BLOCKED) are manually assigned can be reused. By using the few-shot classification method, it is possible to indirectly predict the target lane state and quickly adapt to various road scenarios. temporal properties can be recognized.

1. Introduction

For safety and convenience, lane-change decision aid is one of the key functions in ADAS (Advanced Driving Assistance System). Accordingly, automakers are trying to equip drivers with blind spot detection systems that alert and warn drivers of potential collisions in the side-to-rear space. It is also required for autonomous vehicles to perform cooperative lane-changing even on roads where human drivers coexist. Embedded computer vision systems are becoming more popular for implementing vehicle safety functions because they are cost-effective.

The end-to-end learning framework may classify spatial attributes of images to help determine a lane-change. In this end-to-end learning framework, side-rear view images can be collected and annotated with binary classes (or labels) of FREE if the self-vehicle can move to the target lane, otherwise BLOCKED. Although the side-to-rear viewpoint image dataset is a valuable asset for image-based lane-changing studies, this binary classification is insufficient to interpret various road scenarios, especially driving environments with interactions with other vehicles. Before a lane-changing action, a safe distance from the vehicle in front or behind the vehicle on the target lane must be secured, and the definition of the safe distance is defined by the relative speed between the two vehicles and the type of road (e.g., city road vs. highway). may change according to Therefore, there is a need for a new system that can inform the remaining time to change lanes regardless of road conditions.

The recent success of supervised learning-based computer vision algorithms requires a large dataset. However, it is inefficient to collect all datasets and annotate each of them in all possible scenarios, especially in the field of intelligent vehicles. In order to reduce the burden on this location-level annotation task, weakly supervised learning methods for object localization have been studied. Interestingly, by designing image features through weak maps, the internal mechanisms of deep convolutional neural networks can be understood. Based on such weak supervised learning, a vehicle-aware image feature extraction method focusing on a primary target on the side-to-rear space may be proposed. Pixelwise regularization and multiscale aggregation are described in order to focus on the overall shape of the vehicle rather than some distinctive parts of the vehicle, such as wheels and grills.

In order for the system to be able to interpret temporal attributes as well as spatial attributes on the scene, we change the binary-class labels of the dataset to n (eg 5) class labels and , these indicate the time remaining to perform a safe lane-change. This change task is done systematically and can alleviate the need for tedious manual annotation work. However, it is inappropriate to apply simple image classification to the dataset because the modified dataset will have severe data imbalance with huge intra-class variation. Instead, a few-shot classification scheme in which the entire dataset is separated according to various road types and driving scenarios so that the subset contains representative examples for various road environments and driving styles. ) can be used. As a result, the classification method by fractional learning can enable the proposed system to quickly adapt to various road scenarios that it sees for the first time.

In the following, we apply a new vehicle computer vision application that predicts the time of lane change, a task-specific image feature focusing on the main target based on weak supervised learning, and the proposed lane-changing system in various road environments. In order to explain the formulation of the prime learning problem.

2. Related Studies

Deep Learning for Auto-Driving. Large road datasets have led to the incorporation of intelligent vehicle and computer vision research. It has contributed to advances in machine perception algorithms such as detection, tracking, stereo matching, optical flow and semantic segmentation. Recently, some studies have used deep learning techniques to make computer vision applications leap from perception to control. For example, an end-to-end driving model has been proposed that directly creates vehicle control factors such as steering angle from image input, and generates structured calculation results for the controller of an auto-driving agent or various driving patterns. There are studies that use large-scale video datasets collected in the wild to implement a comprehensive model that reflects or simplify the lane-change decision problem to a binary classification that measures the occupancy status of the target lane. . However, a technique for estimating the time to change lanes has not been developed so far.

Weakly Supervised Object Localization. Existing weakly supervised object localization methods can be classified into sequential approaches and integrated approaches. Sequential methods may first suggest a region in which an object may appear, and then perform classification. Multiple Instance Learning (MIL) method generates positive and negative labeled bags of the instance region and selects some instances within non-convex optimization positively labeled bags. do. Weakly Supervised Deep Detection Networks (WSDDN) use spatial pyramid pooling for region proposals and combine recognition and detection figures through elementwise multiplication.

On the other hand, the integrated methods simultaneously perform object classification and location recognition through class-level saliency maps, which are incidental results in the learning process. The Class Activation Mapping (CAM) algorithm uses fully connected layers for the classification task to learn adaptive weights of feature maps with respect to class-specific activations. ) with a global average pooling layer. To obtain a fine-grained attention region, the derived backpropagation scores can be multiplied by the weighted feature map. However, since these algorithms create an importance map based on cross entropy for image classification, the activation result deals with some distinctive parts rather than the whole object. To solve this problem, the Hide-and-Seek algorithm was implemented to intentionally hide a part of the image in order to learn an error resilience model. An adversarial erasing technique has been proposed so that the network understands the complete body of an object by iteratively erasing the most identifiable parts. In embodiments of the present invention, per-pixel attention regularization and multiscale inference are proposed to derive an attention map covering the entire object region without fragmenting parts. .

Meta-learning for classification by fractional learning. Meta-learning aims to mimic human intelligence, which can quickly adapt to new tasks with a small number of examples. In computer vision, few-shot classification is a difficult task because the scarce data is insufficient to include pristine diversified visual features. In order to deal with unbalanced training data, a model regression network has been proposed to learn the classifier transformation from a small-sample model to a large-sample model. In model-agnostic meta-learning, an objective function is defined that accepts parameters across all ancillary tasks. Through gradient-based optimization, the parameters cause rapid adaptation of the updateability model sensitively to the loss of new tasks. Mechanisms exist for training models with unlabeled data. On the other hand, a non-parametric approach is to train an embedding function that produces a representative value for each class in units of distance. Embodiments of the present invention follow a non-parametric approach as most drivers naturally react to new road environments based on their previous driving experiences.

3. Dataset

Previously, it was explained that a binary label indicating whether the target lane is free (FREE) or blocked (BLOCKED) may be assigned to each side-rear view image according to the spatial properties of the scene. The system according to an embodiment may additionally change the dataset so as to predict when the target lane becomes free for lane-changing within a very short time by using temporal information of the scene. For example, images to which the label BLOCKED has been previously assigned may be remapped to one of 1s, 2s, 3s, and BLOCKED labels. Here, ns may indicate that the target lane is currently blocked but will be freed within about n seconds. Meanwhile, all images to which the previous label FREE is assigned may remain as they are. In order to avoid expensive and time-consuming manual annotation work, a systematic method utilizing a set of raw data sets can be utilized. More specifically, given a bundle of side-back view scenes, the minimum number of frames from the BLOCKED state to the FREE state may be counted, and the frame count may be converted into seconds according to the number of frames per second of the bundle. When allocating a new label, ±1 frame can be tolerated to mitigate the quantization error.

1 is a diagram illustrating an example of annotated images according to an embodiment of the present invention. The images in the first row in FIG. 1 show the order in which the self-vehicle overtakes another vehicle in the target lane. The images in the second row show the reverse order. Also, it is shown that the images of the dataset of Fig. 1 are obtained from various places including residential areas (images in the first row) and highways (images in the second row).

Table 1 below shows an example of overall statistics of a data set converted according to an embodiment.

existing label converted label proportions FREE FREE 55.41% BLOCKED FREE in 1s 1.17% FREE in 2s 1.04% FREE in 3s 0.47% BLOCKED 41.91%

Table 1 shows an example of data distribution in the case of allocating a FREE label and a BLOCKED label to side-rear view scenes, and a FREE label, FREE in 1s label, FREE in -rear view scenes according to an embodiment of the present invention An example of data distribution in the case of assigning a 2s label, a FREE in 3s label, and a BLOCKED label is shown. For example, the images of FIG. 1 show an example in which such a converted label is assigned. In relation to a lane change, a pre-established dataset including a total of 109,416 images exists. However, this dataset has two problems to become suitable training data for deep learning. First, images within the same class show large variation. For example, the fourth column of FIG. 1 is two images in the same 1s class (label), where the vehicle position on the target lane is very different. This dataset is because images are obtained from different types of roads (urban roads vs. highways), and covers different road structures and lane-changing scenarios, such as self-vehicle overtaking other vehicles and vice versa. do. Second, the relabeling result shows a severe imbalance across the five classes, as shown in Table 1. In Table 1, classes 1s, 2s, and 3s occupy only 3% of the dataset, whereas the FREE and BLOCKED classes overwhelmingly outnumber the entire dataset.

4. Methodology

Hereinafter, a new image feature extraction method focusing on a primary target in the lateral-posterior space will be described. Then, how classification by fractional learning enables lane-change decision in a system according to embodiments of the present invention is described.

4.1 Learning Features via Weak Supervision

Pre-trained features have been widely used in various image recognition tasks. In embodiments of the present invention, new image features for handling movable objects on the road may be learned for more reliable results on a task. Using an attention mechanism can improve perception of image features associated with fast-moving objects while suppressing the effects of features that are not task-relevant. To obtain these task-specific characteristics, weak supervised learning can be used with an attention mechanism that uses a saliency map of a specific class. In addition, there is a need to understand the inside of the Convolutional Neural Network (CNN) so that the proposed system achieves safety and reliability as a vehicle application.

4.1.1 Class Activation Mapping

The CNN architecture for image classification is mostly composed of convolutional layers for feature extraction and linear layers followed by classification tasks. In the process of reshaping the output of convolutional layers, it may be required to read spatial information of image features. The main idea of the Class Activation Mapping (CAM) algorithm is that each activation unit encodes an individual image feature, and thus the activation units bring different importance to predict the class. In particular, the CAM may perform global average pooling on the last convolutional layer and compute adaptive weights of feature maps across activation units.

를 내재적으로(implicitly) 학습하여 클래스가 지정된 중요도 맵

을 얻을 수 있다.Formulately, during learning, the CAM algorithm calculates a weight for each class as shown in Equation 1 below.

A classed importance map by implicitly learning

can get

는 예를 들어, 소프트맥스(softmax)와 같은 활성화 함수(activation function)를 나타낼 수 있고,

는 마지막 합성곱 계층에서 k 번째 활성화 유닛을 나타낼 수 있다. 달리 말하면, 클래스가 지정된 가중치

와 특징 맵

의 선형 조합은 분류를 위해 지역적으로 의미 있는 영역을 강조하는 픽셀 단위의 중요도 맵

가 될 수 있다. 그러므로,

는 클래스

가 될 가능성이므로 CAM은 클래스의 예상을 위해 이 수치를 이용할 수 있다.here

may represent, for example, an activation function such as softmax,

may represent the k -th activation unit in the last convolutional layer. In other words, class-assigned weights

and feature map

A linear combination of the pixel-by-pixel importance maps highlighting regions that are regionally significant for classification

can be therefore,

is the class

, so the CAM can use this figure for class prediction.

4.1.2 Pixelwise Regularization

In one embodiment of the present invention, the importance map for vehicle-related classes may be used as an attention mask to focus on a movable object on the road. Because CAM uses cross-entropy loss to make predictions that maximize the overall attention associated with a class, it tends to emphasize only the most distinctive parts of an object. In order to tackle this problem, in this embodiment, pixel-wise normalization is applied to the CAM and the effect of a local characteristic common to all classes can be minimized. As a result, for each class, the normalization method according to the present embodiment can make regions corresponding to the class emphasized while suppressing irrelevant regions.

의 집합으로 중요도 맵

의 집합에 대한 평균적인 이진 크로스 엔트로피 손실을 계산할 수 있다. 여기서

는 클래스의 총 개수를 나타낸다. 다른 정보(예를 들어, 이진 분할 맵(binary segmentation map))가 이용 가능하다면, 클래스들과 관련 있는 모양 정보(shape information)를 만들기 위하여 픽셀 단위의 실측 자료 맵(정답 맵)이 정의될 수 있다. 그러나, 본 실시예에서는 확장된 실측 자료를 이진 행렬로 간단히 설계할 수 있다. 예를 들어, 모든

에 대하여

이고, 여기서

와

는 각각 영상의 폭(width)과 높이(height)를 나타낼 수 있다. 이때, 픽셀 단위의 회귀 손실(pixelwise regression loss)은 다음 수학식 2와 같이 정의될 수 있다.In this embodiment, the expanded actual data (correct answer)

Importance map as a set of

We can compute the average binary cross-entropy loss for a set of . here

represents the total number of classes. If other information (eg, binary segmentation map) is available, a pixel-level ground truth map (correct answer map) can be defined to produce shape information related to classes. . However, in this embodiment, the extended actual data can be simply designed as a binary matrix. For example, all

about

and where

Wow

may represent the width and height of the image, respectively. In this case, the pixelwise regression loss may be defined as in Equation 2 below.

Finally, the attention model can learn a linear combination of losses as shown in Equation 3 below:

는 분류를 위한 크로스 엔트로피 손실일 수 있으며,

는 초매개변수(hyperparameter)일 수 있다. 이진 분류기로서 주의 엔진(attention engine)을 학습시키면서, 차량-유사 객체(양성 샘플) 및 그 외의 것(음성 샘플)과 관련된 서브세트가 활용될 수 있다. 데이터 샘플링에 대해서는 이후 더욱 자세히 설명한다.here,

may be the cross-entropy loss for classification,

may be a hyperparameter. Subsets related to vehicle-like objects (positive samples) and others (negative samples) can be utilized while training the attention engine as a binary classifier. Data sampling will be described in more detail later.

4.1.3 Multiscale Aggregation

Since the dataset to be utilized is mostly composed of object-oriented images, it is difficult to learn size-invariant features from it. However, moving objects often appear in various sizes, especially on side-rear view road scenes. Therefore, a multi-scale integration method capable of obtaining an importance map regardless of object size can be used. The proposed integration method can reuse the model learned by Equation 3 and can operate in the reasoning step. Therefore, no additional learning is required.

을 생성할 수 있다. 이 이후부터, 명료함을 위하여

의 표기법을

으로 대체한다.

을 영상의 사이즈를 앵커 영상의 사이즈로 바꾸게 하는 변환(예를 들어, 이중 선형 보간법(bilinear interpoliation))이라고 하자. 이때 다중 스케일의 주의 맵

을 얻기 위하여 다음 수학식 4와 같이 모든 스케일에 대해 중요도 맵들이 적응적으로 통합될 수 있다.The system may generate image pyramids having various sizes for one input image. In this case, the initial input image may be later referred to as an "anchor image" to distinguish it from a resized image. The system can iteratively input a set of scaled images into the model to obtain multi-scale importance maps. In order to accommodate different sizes for the input, a minimal transformation to the model can be applied by scaling the global average pooling operation on top of the last convolutional layer. For the mth pyramidal input image, the attention engine generates an importance map for the car class.

can create From this point on, for clarity

notation of

be replaced with

Let be a transformation (eg, bilinear interpolation) that changes the size of the image to the size of the anchor image. In this case, the multi-scale attention map

In order to obtain , the importance maps can be adaptively integrated for all scales as shown in Equation 4 below.

은 m 번째 크기 변경된(scaled) 입력에 대한 예상 수치를 나타내고,

은 다양한 크기 변경된 입력들에 대한 예상 수치들 중에서 소프트맥스(softmax)가 계산한 각 크기의 상대적인 중요도를 결정할 수 있다.here,

represents the expected value for the m -th scaled input,

may determine the relative importance of each size calculated by softmax among expected values for various size-changed inputs.

2 shows an example of comparing the attention result of the CAM with the attention result of the method according to an embodiment of the present invention. Pixelwise regularization (PR) and multiscale aggregation (MA) have been described above in FIG. 2 , and it is shown that the proposed method can improve recognition of the entire vehicle shape in an image. As such, the pixel-by-pixel normalization and multi-scale integration approach according to the present embodiment can extend the limited cognitive area of the CNN without an additional learning procedure.

4.2 Few-Shot Classification for Lane-Change

To overcome the serious data imbalance problem, prime learning can be applied to the dataset. In the sense that humans can adapt their behavior patterns to various driving scenarios based on their experiences, the system can adopt a non-parametric algorithm, a prototype network.

4.2.1 Prototype Network

에 대해

개의 레이블링된(labeled) 예시들을 포함하는 지원 세트(support set)인

를 갖는다고 가정하자. 여기서,

은 데이터 지점(data point)이고,

는 대응하는 정답 레이블(ground truth label)을 나타낼 수 있다. 프로토타입 네트워크는 각 클래스에 대해 프로토타입(prototype)인

를 표현하기 위하여 임베딩 함수(embedding function)

를 학습할 수 있다. 여기서

는 학습 변수(learnable parameters)를 의미할 수 있다. 임베딩 공간(embedding space) 내 거리 단위

로, 프로토타입 네트워크는 소프트맥스(softmax)를 사용하여 클래스들 상의 분포를 도출할 수 있으며, 쿼리

에 대한 레이블을 다음 수학식 5와 같이 예상할 수 있다.For completeness sake, the prototype network is briefly reviewed here. set class

About

A support set comprising of labeled examples

Suppose we have here,

is a data point,

may represent a corresponding ground truth label. The prototype network is a prototype for each class.

embedding function to express

can learn here

may mean learning parameters. Unit of distance in embedding space

Thus, the prototype network can derive the distribution over the classes using softmax, and query

A label for can be predicted as in Equation 5 below.

4.2.2 Feature Embedding for Lane-Change

In this embodiment, it is assumed that a place where pre-computed image features are used is a fixed feature space. In order to consider the temporal properties on the scene, image features calculated at 1-second intervals may be concatenated, and the annotation of the connected features may be copied from the annotation of the last time slot. In this case, image features may be integrated with attention. This can be either fuse after concatenation (FAC) or concatenate after fusion (CAF). Using FAC, attention results are applied globally to the extended feature map, whereas CAF can link the individual attention regions of each feature map. 3 is a diagram illustrating an example of an integration method according to an embodiment of the present invention. In detail, FIG. 3 shows a comparison of different spatiotemporal integration methods of image features and attention maps. The left side of FIG. 3 shows an example using the FAC method, and the right side of FIG. 3 shows an example using the CAF method. Image features and attention masks can be combined by multiplication or addition. Table 2 shows examples of performance comparison for various integration scenarios.

Fusion Op. 1-shot (5-way) 5-shot (5-way) FAC Mul 56.00±0.78% 75.95±0.70% CAF 56.70±0.81% 81.04±0.67% FAC Add 68.76±0.80% 84.88±0.64% CAF 74.09±0.83% 86.48±0.65%

Table 2 shows the performance of VGG-16 embedding in relation to the integration method. A Euclidean distance metric may be applied to the embedding space. Feature embedding networks consist of three convolutional building blocks, each block being a convolutional layer (with 3Х3 kernel size), batch normalization, Rectified Linear Unit (ReLU) and max pooling (with 2Х2 kernel size). ) can be formed as a stack of If the dimension of the input features depends on the backbone model, the dimension of the hidden unit in the embedding network can be fixed at 512 and 256, respectively.

4 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. The systems according to the embodiments of the present invention described above may be implemented by the computer device 400 shown in FIG. 4 , and the method according to the embodiments of the present invention may be performed by the computer device 400 . can

In this case, as shown in FIG. 4 , the computer device 400 may include a memory 410 , a processor 420 , a communication interface 430 , and an input/output interface 440 . The memory 410 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer device 400 as a separate permanent storage device distinct from the memory 410 . Also, the memory 410 may store an operating system and at least one program code. These software components may be loaded into the memory 410 from a computer-readable recording medium separate from the memory 410 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 410 through the communication interface 430 instead of a computer-readable recording medium. For example, the software components may be loaded into the memory 410 of the computer device 400 based on a computer program installed by files received over the network 460 .

The processor 420 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 420 by memory 410 or communication interface 430 . For example, the processor 420 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 410 .

The communication interface 430 may provide a function for the computer device 400 to communicate with other devices via the network 460 . For example, a request, command, data, file, etc. generated by the processor 420 of the computer device 400 according to a program code stored in a recording device such as the memory 410 is transmitted to the network ( 460) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 400 through the communication interface 430 of the computer device 400 via the network 460 . A signal, command, or data received through the communication interface 430 may be transferred to the processor 420 or the memory 410 , and the file may be a storage medium (described above) that the computer device 400 may further include. persistent storage).

The input/output interface 440 may be a means for an interface with the input/output device 450 . For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 440 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 450 may be configured as one device with the computer device 400 .

Also, in other embodiments, the computer device 400 may include fewer or more components than those of FIG. 4 . However, there is no need to clearly show most of the prior art components. For example, the computer device 400 may be implemented to include at least a portion of the above-described input/output device 450 or may further include other components such as a transceiver and a database.

The communication method is not limited, and not only a communication method using a communication network (eg, a mobile communication network, wired Internet, wireless Internet, broadcasting network) that the network 460 may include, but also short-distance wired/wireless communication between devices may be included. there is. For example, the network 460 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 460 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

5 is a flowchart illustrating an example of a method for learning a lane change time point according to an embodiment of the present invention. The lane change time learning method according to the present embodiment may be performed by the computer device 400 for implementing the lane change time learning system. For example, the processor 420 of the computer device 400 may be implemented to execute a control instruction according to a code of an operating system included in the memory 410 or a code of at least one program. Here, the processor 420 causes the computer device 400 to perform the steps 510 to 550 included in the method of FIG. 5 according to a control command provided by the code stored in the computer device 400 . can control

In operation 510 , the computer device 400 may receive an image in which the first label or the second label is assigned to individual frames by being input through the camera in relation to the side-rear region of the vehicle. Here, the image to which the first label is assigned may include an image indicating a state in which a lane change is blocked, and an image to which the second label is assigned may include an image indicating a state in which a lane change is free. For example, the first label may mean the previously described label 'BLOCKED' and the second label may mean the previously described label 'FREE'.

In step 520, the computer device 400 transmits n (the above) to at least one frame before the transition time according to the number of frames per second (Frame Per Second, FPS) based on the transition time from the blocked state to the free state. where n is a natural number) may give a label defining that lane changes are possible after seconds. For example, in step 520 , the computer device 400 may assign a label indicating that lane change is possible after n seconds to the m-th frame before the time of change. In this case, m may be determined by a multiplication operation having n and the number of frames per second as parameters. As a more specific example, it is assumed that the index of the frame corresponding to the transition time from the blocked state to the free state is 110 in a moving picture in which the FPS is fixed to 10. In this case, the label defining that lane change is possible after 1 second may be assigned to the frame of index 100, which is the 10th (1(sec) Х 10(FPS))-th frame before the transition time. Similarly, a label defining that lane change is possible after 2 seconds may be assigned to a frame of index 90, which is a 20 (2 (second) Х 10 (FPS))-th frame before the transition time.

In operation 530, the computer device 400 may learn the lane change timing based on the frames of the image and labels included in the frames. According to an embodiment, the computer device 400 performs weakly supervised learning for each of the frames of the image based on whether a vehicle or a moving object exists, and the vehicle or moving object occupies in each frame. The region can be output in the form of a heat map. In this case, the computer device 400 may learn the lane change timing by further using the heat map and the feature vector of the image in step 430 . In this case, the computer device 400 may increase the weight of the area occupied by the vehicle or the moving object in the image by adding the values of the heat map to all dimensions of the feature vector. In addition, as shown in FIG. 1 , since images assigned with the same label may have various situations, the image features corresponding to a preset time interval (eg, 1 second) are connected into one vector to determine the temporal Features can be encoded. For example, different situations shown in FIG. 1 may be distinguished according to a case in which an area occupied by a vehicle or a moving object in an image increases or decreases with time.

Also, the computer device 400 may utilize multi-scale integration to obtain an importance map for objects that appear in different sizes in the image regardless of the size of the object. For example, the computer device 400 generates a set of scaled images for the image, and repeatedly inputs the set of scaled images to an attention model to obtain multi-scale saliency maps. After creating maps), by integrating the importance maps for all scales, it is possible to obtain an importance map regardless of the size of the object for objects that appear in different sizes in the image.

6 is a flowchart illustrating an example of a method for predicting a lane change time point according to an embodiment of the present invention. The lane change time prediction method according to the present embodiment may be performed by the computer device 400 for implementing the lane change time prediction system. For example, the processor 420 of the computer device 400 may be implemented to execute a control instruction according to a code of an operating system included in the memory 410 or a code of at least one program. Here, the processor 420 causes the computer device 400 to perform the steps 610 to 650 included in the method of FIG. 6 according to a control command provided by the code stored in the computer device 400 . can control

In operation 610 , the computer device 400 may analyze an image input through the camera in relation to the side-rear area of the vehicle to determine the current state related to the lane change. Such a determination may be made through the artificial intelligence model previously learned through FIG. 5 . For example, the AI model may determine the current state as a state in which lane change is possible and a state in which lane change is free (FREE) after n seconds (n is a natural number).

Step 610 is a step of acquiring an image input through a camera in relation to a side-rear area of the vehicle and a current state related to a lane change from such an image, a state in which a lane change is impossible, a state in which a lane change is possible, and n ( Where n is a natural number), one of the states in which a lane change is possible after seconds may be divided into a step of predicting, and the step of predicting may be using a model learned through FIG. 5 .

In operation 620 , the computer device 400 may provide different types of notifications for a state in which a lane change is possible and a state in which a lane change is free after n (n is a natural number) seconds as the current state. For example, the computer device 400 provides information on the remaining time until the lane change every second from n seconds to 1 second for a state in which lane change is possible after n seconds, and lane change in the state where the lane change is free You can provide information on the possibility of this. More specifically, as an example, the computer device 400 connected to the display device installed in the vehicle, when the display device is n is 3, information on the state in which a lane change is possible after n seconds, such as 3 seconds, 2 seconds, and 1 second You can provide information about the time remaining until the change. Also, when a lane change is possible, the computer device may provide a notification so that the display device outputs information about the lane change possibility. In addition to providing a visual notification through the display device, provision of an audible notification using a speaker may be considered. Meanwhile, in addition to providing the notification to the driver of the vehicle, it may also be considered that the notification is provided to the artificial intelligence that controls the autonomous driving of the vehicle.

According to an embodiment, the computer device 400 may activate the lane change preparation function as a notification of a state in which a lane change is possible is provided after n seconds. In this case, the lane change preparation function may include a preparation function such as calculating detailed parameters for lane change or calculating a route.

The computer device 400 may provide a notification according to a lane change request, but may continuously output different types of notifications in relation to a left lane change and a right lane change of the vehicle.

In addition, the computer device 400 may store information on the driving tendency of the driver of the vehicle in step 620 and set the intensity of the notification based on the driving tendency.

The computer device 400 may determine the strength of the notification based on the number of frames per label for frames to which a label defining that a lane change is possible after n seconds is assigned. For example, based on 10 FPS, if the number of frames assigned a 3-second label is 10, while the number of frames assigned a 2-second label is 1 or 0, it indicates that there is a vehicle or object accelerating rapidly in the lane to be changed. can mean In this case, the computer device 400 may provide a relatively stronger notification or a signal for driving a safety device for preventing a lane change.

7 is a diagram illustrating an example of assigning a label according to an embodiment of the present invention. 7 shows a total of five frames 710 to 750, one for every 10 frames among the frames included in the image. In this case, the first frame 710 of FIG. 7 represents a frame at the time of transition from the blocked state to the free state. The first frame 710 may be in a state in which a label 'FREE', which is a label indicating a free state, is attached. Frames before the first frame 710 may be frames to which a label 'BLOCKED', which is a label indicating a blocked state, is assigned. In this case, the second frame 720 , the third frame 730 , the fourth frame 740 , and the fifth frame 750 illustrated in FIG. 7 may also be included in the frames to which the label 'BLOCKED' is assigned.

Assuming that the number of frames per second is 10, as already described, the computer device 400 may assign a label defining that lane change is possible after n seconds to the m-th frame preceding the first frame 710 . Here, m may be determined through a multiplication operation between the number of frames per second and n.

For example, in FIG. 7 , the second frame 720 represents a tenth frame preceding the first frame 710 . In this case, a label defining that lane change is possible after 1 second may be provided to the second frame 720 , which is the 10th frame (the number of frames per second 10 Х 1 second) before the first frame 710 .

Similarly, the third frame 730 represents the 20th frame preceding the first frame 710 . In this case, a label defining that lane change is possible after 2 seconds may be provided to the third frame 730 that is the 20th frame (the number of frames per second 10 Х 2 seconds) before the first frame 710 .

The fourth frame 740 may represent the 30th frame preceding the first frame 710 , and it is easy to see that this fourth frame 740 may be given a label defining that a lane change is possible after 3 seconds. you will understand

Although Table 1 and FIG. 7 have described an example in which n is a natural number less than or equal to 3, it will be easily understood that n may be set more variously. If n is a natural number less than or equal to 4, a label defining that lane change is possible after 4 seconds will be attached to the fifth frame 750 of FIG. 7 .

A label defining that lane change is possible after n seconds may be added to the existing label 'BLOCKED', or may be applied in place of the existing label 'BLOCKED'.

As such, according to the embodiments of the present invention, information on a lane change possible time by receiving an image from a camera installed so as to obtain a viewpoint similar to the side-rear region observed by the side mirror of the vehicle (eg, 3 seconds before, 2 It can predict the first time before, one second before, possible, impossible) to give a notification to the driver or autonomous driving AI.

The system or apparatus described above may be implemented as a hardware component or a combination of a hardware component and a software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The medium may continuously store a computer executable program, or may be a temporary storage for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (8)

Step of receiving an image to which the first label or the second label is assigned to each frame input through the camera in relation to the side-rear area of the vehicle - The image to which the first label is assigned is blocked from changing lanes an image indicating the state, and the image to which the second label is assigned includes an image indicating a state in which lane change is free;
Based on the transition time from the blocked state to the free state, at least one frame before the transition time is selected according to the number of frames per preset time interval, and in the selected at least one frame, n ( applying a label defining that lane change is possible after n is a natural number) preset time intervals; and
Learning a lane change time based on the frames of the image and labels included in the frames
A method of learning when to change lanes, including According to claim 1,
The step of receiving the image is,
acquiring an image for learner training by mounting the camera at a position for observation of the side-rear region of the vehicle; and
applying the first label or the second label sequentially according to time to the individual frames of the acquired image
Lane change time learning method, characterized in that it comprises a. determining a current state related to a lane change by analyzing an image input through a camera in relation to a side-rear area of the vehicle; and
Providing different types of notifications for a state in which a lane change is possible and a state in which a lane change is free after n (wherein n is a natural number) preset time intervals as the current state
Lane change timing prediction method including 4. The method of claim 3,
The step of providing the different types of notifications includes:
A method for predicting a lane change time, characterized in that the different types of notifications are provided by artificial intelligence that controls the autonomous driving of the vehicle. A computer program stored in a computer-readable recording medium in combination with a computer to cause the computer to execute the method of any one of claims 1 to 4. A computer-readable recording medium, characterized in that a program for executing the method of any one of claims 1 to 4 in a computer is recorded. A computer device comprising:
at least one processor implemented to execute computer-readable instructions
including,
by the at least one processor;
In relation to the side-rear area of the vehicle, it is input through the camera to receive an image to which the first label or the second label is assigned to individual frames - The image to which the first label is assigned is a state in which lane change is blocked (BLOCKED) and the image to which the second label is assigned includes an image indicating a lane change free state, respectively;
Based on the transition time from the blocked state to the free state, at least one frame before the transition time is selected according to the number of frames per preset time interval, and in the selected at least one frame, n ( Wherein n is a natural number) a label defining that a lane change is possible after a preset time interval,
Learning a lane change time based on the frames of the image and labels included in the frames
A computer device comprising a. A computer device comprising:
at least one processor implemented to execute computer-readable instructions
including,
by the at least one processor;
Analyzing the image input through the camera in relation to the side-rear area of the vehicle to determine the current state related to lane change,
Providing different types of notifications for a state in which lane change is possible and a state in which lane change is free after n (where n is a natural number) preset time intervals as the current state
A computer device comprising a.

Images