KR102360181B1 - Electronic device and method for controlling operation of vehicle - Google Patents

Abstract

obtaining a video sequence including a plurality of frames from a camera installed in the vehicle; detecting an object included in the plurality of frames from the plurality of frames; obtaining location information of an object for each of a plurality of frames; determining whether an event related to driving of a vehicle has occurred by analyzing time-series changes of positions of objects in a plurality of frames; based on the determination, generating a notification message informing of the event; and outputting the generated notification message, wherein detecting the object, obtaining location information, determining whether an event has occurred, and generating the notification message are performed using a plurality of learning models A method for controlling the operation of a vehicle is disclosed.

Description

ELECTRONIC DEVICE AND METHOD FOR CONTROLLING OPERATION OF VEHICLE

The present disclosure relates to an electronic device and method for controlling an operation of a vehicle, and more particularly, to an electronic device that provides a notification message for notifying occurrence of an event related to driving of a vehicle according to a location of an object in a plurality of frames and methods.

As multimedia technology and network technology develop, users can receive various services using electronic devices. As technologies applied to vehicles develop, various methods for recognizing whether an event related to driving of a vehicle has occurred have been developed.

On the other hand, in the field of autonomous driving technology that requires a fast and accurate cognitive function, the demand for a technology that provides a notification message to a user by determining whether an event related to vehicle driving has occurred more accurately with a limited amount of data is increasing.

SUMMARY One embodiment is to provide an electronic device and method for providing a notification message for notifying occurrence of an event related to driving of a vehicle according to the location of an object in a plurality of frames using a plurality of learning models.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure is a memory in which at least one program is stored; a communication unit for receiving a call request from a counterpart electronic device; and at least one processor configured to provide a notification message by executing at least one program, the at least one program comprising: acquiring a video sequence including a plurality of frames from a camera installed in the vehicle; detecting an object included in the plurality of frames from the plurality of frames; obtaining location information of an object for each of a plurality of frames; determining whether an event related to driving of the vehicle has occurred by analyzing a time-series change of the position of the object in the plurality of frames; based on the determination, generating a notification message informing of the event; and outputting the generated notification message: including instructions for executing: detecting an object, obtaining location information, determining whether an event has occurred, and generating a notification message includes a plurality of learning It is possible to provide an electronic device, which is performed using the models.

In addition, a second aspect of the present disclosure includes: obtaining a video sequence including a plurality of frames from a camera installed in a vehicle; detecting an object included in the plurality of frames from the plurality of frames; obtaining location information of an object for each of a plurality of frames; determining whether an event related to driving of the vehicle has occurred by analyzing a time-series change of the position of the object in the plurality of frames; based on the determination, generating a notification message informing of the event; and outputting the generated notification message, wherein the detecting an object, obtaining location information, determining whether an event has occurred, and generating the notification message are performed using a plurality of learning models. may provide a method of controlling the operation of a vehicle, which is performed.

In addition, a third aspect of the present disclosure may provide a computer-readable recording medium in which a program for executing the method of the second aspect is recorded on a computer.

1 is a diagram illustrating an example in which an electronic device controls an operation of a vehicle according to an exemplary embodiment.
2 is a flowchart illustrating a method for an electronic device to provide a notification message to a user, according to an embodiment.
3 is a flowchart illustrating a method in which an electronic device provides a notification message to a user by using a learning model, according to an exemplary embodiment.
4 is a diagram illustrating an example of generating a notification message using a learning model according to an embodiment.
5 is a diagram for explaining how content of a notification message is determined, according to an embodiment.
6 is a diagram illustrating an example of outputting a notification message, according to an embodiment.
7 is a diagram illustrating an example of outputting a notification message, according to an embodiment.
8 and 9 are block diagrams illustrating an electronic device according to some exemplary embodiments.
10 is a block diagram illustrating a control unit according to an exemplary embodiment.
11 is a block diagram illustrating a data learning unit according to an exemplary embodiment.
12 is a block diagram illustrating a data recognition unit according to an exemplary embodiment.
13 is a diagram illustrating an example of learning and recognizing data by interworking with an electronic device and a server according to an embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an example in which an electronic device 1000 controls an operation of a vehicle according to an exemplary embodiment.

Referring to FIG. 1 , the electronic device 1000 may be a device installed in a vehicle, and the electronic device 1000 may receive a video sequence from a camera installed in the vehicle and provide a user with a notification message notifying the occurrence of various events. can

Previously, it has been described that the electronic device 1000 receives a video sequence from a camera installed in a vehicle, but the present invention is not limited thereto. The electronic device 1000 may also receive a video sequence from a camera capable of photographing the surroundings of the vehicle. The perimeter of the vehicle may include, for example, the front, sides, and rear of the vehicle.

According to an embodiment, the electronic device 1000 may provide a notification message including different contents according to the type of event and the degree of risk of driving. For example, if it is determined that it is better to provide action instructions in response to the event rather than simply notifying the event based on the type of event and the risk of driving, a notification message containing the event and action instructions in response to the event can be provided to users. For example, the action instruction corresponding to the event may include a method for reducing the risk of driving.

In an embodiment, if it is determined that it is desirable to control the operation of the module installed in the vehicle based on the type of event and the degree of risk of driving, the electronic device 1000 may transmit a command for controlling the operation of the module installed in the vehicle installed in the vehicle. can be sent to the module. For example, the electronic device 1000 may output a notification message and simultaneously control the operation of a module installed in the vehicle, based on the type of event and the driving risk. Alternatively, if the electronic device 1000 does not receive a user input for controlling the operation of the module installed in the vehicle within a preset time after outputting the notification message, the electronic device 1000 controls the operation of the module installed in the vehicle. Commands can be sent to modules installed in the vehicle. Alternatively, the electronic device 1000 may control the operation of a module installed in the vehicle without outputting a notification message, based on the type of event and the driving risk.

The electronic device 1000 is a head unit in a vehicle, an embedded board, a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a vehicle, a media player, a micro server, a global positioning system (GPS) ) devices, e-book terminals, digital broadcast terminals, navigation devices, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but is not limited thereto. Also, the electronic device 1000 may be a wearable electronic device such as a watch, glasses, a hair band, and a ring having a communication function and a data processing function. However, the present invention is not limited thereto, and the electronic device 1000 may obtain an image (eg, a video and a still image) from a camera and provide a notification message to the user based on the obtained image. It can include all possible types of devices.

In an embodiment, the electronic device 1000 is a module installed in a vehicle, and may control an operation of the vehicle, and may communicate with other modules installed in the vehicle through a predetermined network.

In an embodiment, the electronic device 1000 may be a device separate from the vehicle, such as a smartphone, and in this case, the electronic device 1000 may obtain a video sequence by using it from the camera of the electronic device 1000, A video sequence may be received from a camera capable of photographing the surroundings of the vehicle through a predetermined network. It can communicate with a module included in the vehicle. For example, the electronic device 1000 may obtain a video sequence by using the camera of the electronic device 100 , and may receive the video sequence from a camera capable of photographing the surroundings of the vehicle. Also, when the electronic device 1000 is a device separate from the vehicle, it may communicate with a module installed in the vehicle to control the operation of the vehicle.

A vehicle may be a means of transportation, such as a car, bus, truck, train, bicycle, motorcycle, etc., having a communication function, a data processing function, and a transportation function.

In addition, the electronic device 1000 receives a video sequence, transmits a notification message, and transmits a command for controlling an operation of another electronic device (not shown), through a predetermined network to the server 2000 and other devices. It can communicate with an electronic device (not shown). In this case, the network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and their mutual It is a data communication network in a comprehensive sense that includes a combination and enables each network constituent entity to communicate smoothly with each other, and may include a wired Internet, a wireless Internet, and a mobile wireless communication network. Wireless communication is, for example, wireless LAN (Wi-Fi), Bluetooth, Bluetooth low energy, Zigbee, WFD (Wi-Fi Direct), UWB (ultra wideband), infrared communication (IrDA, infrared Data Association) ), NFC (Near Field Communication), etc. may be there, but is not limited thereto.

2 is a flowchart illustrating a method for the electronic device 1000 to provide a notification message to a user, according to an embodiment.

In operation S210, the electronic device 1000 may obtain a video sequence including a plurality of frames from a camera installed in the vehicle. In an embodiment, the electronic device 1000 may receive a video sequence by communicating with a camera installed in a vehicle through a predetermined network. For example, the video sequence may be a blackbox image of a vehicle, or may be an image received from a stereo camera of the vehicle. In an embodiment, the electronic device 1000 may include a camera to obtain a video sequence from a camera included in the electronic device 1000 .

A video sequence may consist of a series of still images. Each of the still images may mean a picture or a frame.

In operation S220, the electronic device 1000 may detect an object included in the plurality of frames from the plurality of frames included in the video sequence. In an embodiment, the electronic device 1000 may detect one or more objects from one frame included in a video sequence. One or more objects detected from one frame may also be detected from other frames included in the same video sequence. Alternatively, one or more objects detected from one frame may not be detected from other frames included in the same video sequence. For example, in the first frame, roadway, sidewalk, first car, second car, third car, traffic sign were detected, and in a second frame within the same video sequence as the first frame, driveway, sidewalk, first car, Only the third car may be detected, and the second car and the traffic sign may not be detected. Also, in the second frame, a motorcycle that was not detected in the first frame may be detected.

In an embodiment, the electronic device 1000 may determine the type of the object.

The types of objects are road, sidewalk, building, wall, fence, pole, traffic light, traffic sign, plant ( vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle, and bicycle (bicycle) and the like.

For example, the electronic device 1000 may detect a plurality of objects from one frame and determine the type of each of the plurality of objects. Also, even if some of the plurality of objects have the same type, the electronic device 1000 may distinguish the same type of objects. For example, if three cars are detected in one frame, the electronic device 1000 may distinguish the three cars into a first car, a second car, and a third car, respectively.

In an embodiment, the electronic device 1000 may use the first learning model to detect an object included in the frame. When frames obtained from the video sequence are input to the first learning model, information about an object detected from the frames may be output from the first learning model. An operation of detecting an object using the first learning model will be described later with reference to FIG. 3 .

In operation S230, the electronic device 1000 may obtain location information of an object for each of a plurality of frames included in the video sequence.

In an embodiment, the electronic device 1000 may determine the positions of the object in the plurality of frames for each of the plurality of frames in order to obtain location information of the object. For example, with respect to one frame, the electronic device 1000 may determine at which position the object is located within the frame. For example, the electronic device 1000 may determine at which position the object is located within the frame with respect to another frame. Also, for example, for one frame, the electronic device 1000 may determine at which positions a plurality of objects are located within each frame. For example, the electronic device 1000 may determine at which positions a plurality of objects are located in each frame with respect to another frame. Accordingly, for each of the plurality of frames, the electronic device 1000 may determine at which positions the plurality of objects are located within each frame.

In an embodiment, the electronic device 1000 may determine the positions of the object in units of pixels. For example, the electronic device 1000 may determine pixels representing an object from among pixels constituting the frame. For example, when one frame includes a plurality of objects, the electronic device 1000 may determine pixels representing each of the plurality of objects. For example, the electronic device 1000 may determine which of the detected objects a random pixel constituting the frame represents.

A method in which the electronic device 1000 precisely acquires object location information in units of pixels instead of a bounding box may be applied to a technical field requiring an accurate recognition function. For example, the electronic device 1000 may obtain location information of an object in units of pixels, analyze a time-series location change of the object, and use it for autonomous driving technology requiring a fast and accurate recognition function.

In an embodiment, the electronic device 1000 may use the first learning model to acquire location information of the object. When a plurality of frames are input to the first learning model, pixel information may be output. The pixel information may be information regarding which objects each pixel constituting a frame represents. Previously, although the operation S220 and the operation S230 were described as separate operations, the present invention is not limited thereto. For example, when a plurality of frames are input to the first learning model, information about an object detected from the frames and pixel information may be output together. Or, for example, only pixel information may be output. An operation of acquiring location information of an object using the first learning model will be described later with reference to FIG. 3 .

In operation S240 , the electronic device 1000 may determine whether an event related to driving of a vehicle has occurred by analyzing a time-series change of positions of objects in a plurality of frames.

According to an embodiment, the electronic device 1000 may analyze a change in the position of an object from a previous frame to a next frame according to a display order of a video sequence. For example, the electronic device 1000 analyzes a change in the position of an object by comparing the position information of the object included in a first frame having a playback order earlier with position information of the same object included in a second frame having a later playback order. can do. For example, the electronic device 1000 may determine whether an event has occurred by analyzing a change in the position of each of the plurality of objects over time. Accordingly, the electronic device 1000 does not track a bounding box related to a region of interest (ROI), but rather recognizes a change in positions of a plurality of objects determined in units of pixels to be more accurately You can decide whether an event has occurred or not. For example, if the first car and the second car are stopped in front, and the third car, the fourth car, and the fifth car continuously change lanes to the right, the electronic device 1000 may cause the accidental vehicles in front It can be determined that an event has occurred. In addition, since there are accident vehicles ahead, an action guideline corresponding to the event that it is advisable to change the lane to the right can be determined.

In an embodiment, the electronic device 1000 may determine an event type by analyzing a time-series change of positions of objects in a plurality of frames.

The type of event related to the driving of the vehicle may include, but is not limited to, a traffic signal change, an accident expected, a road condition change, a terrain change, and the like. As an example in which the traffic signal is changed, a traffic light may be changed from green to red or from red to green. Examples of the expected accident may include not securing a safe distance from the vehicle in front and/or the vehicle behind, and the appearance of an unexpected person. As an example in which the road condition is changed, there may be an accident vehicle ahead and the road being blocked. As an example in which the terrain is changed, there may be a winding road in front, a hill in front, and the like.

In an embodiment, the electronic device 1000 may determine the risk of driving by analyzing a time-series change of positions of objects in a plurality of frames.

For example, the risk of driving may be expressed as a numerical value, and a higher numerical value may mean a higher risk. For example, the risk level may be expressed as an integer of 1 or more and 100 or less, and when the risk level is greater than or equal to a threshold value, the electronic device 1000 may be set to include the driving risk in the notification message. Also, for example, the degree of risk of driving may be expressed as high, medium, and low.

In an embodiment, the electronic device 1000 may use the second learning model to determine whether an event has occurred. When pixel information output from the first learning model is input to the second learning model, whether an event has occurred may be output. An operation of determining whether an event has occurred using the second learning model will be described later with reference to FIG. 3 .

In operation S260, the electronic device 1000 may generate a notification message informing of the event based on the determination of the occurrence of the event.

In an embodiment, different notification messages may be generated according to the type of object, changes in time-series positions of the object, and whether an event occurs.

If it is determined in operation S250 that an event has occurred, in operation S260 , the electronic device 1000 may generate a notification message notifying the event. Also, if it is determined that the event has not occurred in operation S250 , in operation S260 , the electronic device 1000 may not generate a notification message for notifying the event. In an embodiment, when the electronic device 1000 determines not to generate a notification message for notifying an event, the electronic device 1000 may not generate the notification message at all, or a preset content that does not include event notification. You can also create notification messages. For example, the electronic device 1000 may generate a notification message including at least one of a current temperature, an rpm value of the vehicle, a forward direction of the vehicle, a traffic situation, and a driving risk. In an embodiment, the notification message that does not include the event notification content may be set as a default value in the electronic device 1000 .

In an embodiment, the electronic device 1000 may generate a notification message based on the type of event and the driving risk. How the electronic device determines the content of the notification message based on the type of event and the driving risk will be described later with reference to FIG. 5 .

The notification message according to an embodiment may be generated as a text message or a voice message, but is not limited thereto. Also, for example, a notification message in a voice format may be obtained by converting a text-to-speech (TSS) message generated in a text format.

When pixel information output from the first learning model is input to the second learning model, a notification message may be output. Although operation S250 and operation S260 have been described as separate operations, the present invention is not limited thereto. For example, when pixel information is input to the first learning model, whether an event has occurred and a notification message may be output together. Or, for example, only a notification message may be output. An operation of generating a notification message using the second learning model will be described later with reference to FIG. 3 .

In an embodiment, the operation of detecting the object, the operation of acquiring location information of the object, the operation of determining whether an event has occurred, and the operation of generating a notification message may be performed using a plurality of learning models.

In operation S270, the electronic device 1000 may output the generated notification message.

In an embodiment, the notification message may be output in the form of sound, text, image, and/or vibration.

In an embodiment, the electronic device 1000 may display a notification message on a head up display (HUD) or dashboard of the vehicle.

In an embodiment, when the notification message is in the form of a voice, the electronic device 1000 may output it through a speaker of the vehicle. For example, the electronic device 1000 may transmit a command for controlling the speaker of the vehicle to output a notification message in the form of a voice to the speaker of the vehicle.

In an embodiment, a command for controlling the operation of a module installed in the vehicle may be transmitted to the corresponding module according to the type of event and the degree of risk of driving. If it is determined that it is desirable to control the operation of the module installed in the vehicle based on the type of event and the driving risk, the electronic device 1000 may transmit a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle. . For example, the electronic device 1000 may output a notification message and simultaneously control the operation of a module installed in the vehicle, based on the type of event and the driving risk. Alternatively, if the electronic device 1000 does not receive a user input for controlling the operation of the module installed in the vehicle within a preset time after outputting the notification message, the electronic device 1000 controls the operation of the module installed in the vehicle. Commands can be sent to modules installed in the vehicle. Alternatively, the electronic device 1000 may transmit a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle, without outputting a notification message, based on the type of event and the driving risk.

The user input according to an embodiment may include at least one of a step on input, a steering input, a voice input, a key input, a touch input, a bending input, and a multimodal input, The present invention is not limited thereto. A step on input may mean an input in which a user applies a brake to control a brake of the vehicle. A steering input may mean an input in which a user rotates a steering wheel to control a steering wheel of a vehicle.

3 is a flowchart illustrating a method in which the electronic device 1000 provides a notification message to a user using a learning model, according to an embodiment.

In operation S310 , the electronic device 1000 may apply a filter for flattening the brightness of the plurality of frames to the plurality of frames to input to the first learning model.

The first learning model is to be created by learning a criterion for judging the type of an object using a Fully Convolutional Network (FCN) and a criterion for judging positions within a plurality of frames of the object for each of a plurality of frames. can

In an embodiment, the electronic device 1000 may convert a frame made of an RGB channel into a Luminance-Chromatic (Lab) channel. The L value of the converted Lab channel is the luminance value of the image and represents the brightness of the image excluding color information. Before input to the first learning model, the electronic device 1000 may perform preprocessing of applying a median filter for flattening the L values of the plurality of frames to the plurality of frames included in the video sequence. have. By performing the pre-processing, an object can be more easily detected even when it is dark or raining, and a plurality of objects can be distinguished from each other.

In operation S320, the electronic device 1000 may determine the type of object included in the plurality of frames by using the first learning model.

For example, the electronic device 1000 may detect a plurality of objects from one frame by using the first learning model and determine the type of each of the plurality of objects. For example, when one frame is input to the first learning model, different values may be output according to the type of object included in one frame. For example, in the first learning model, the sky may be set to 12, the plant to 10, the roadway to 4, the sidewalk to 3, the car to 6, the person to 8, and the like. For example, if plants, roads, cars, and people are included in the input frame, 4, 6, 8, and 10 may be included in the output of the first learning model. Accordingly, it is possible to detect which object is included in the frame by using the first learning model. Also, for example, when a frame is input to the first learning model, a value corresponding to the type of object may not be output, but pixel information regarding which objects the pixels constituting the frame represent may be output. The pixel information may be a matrix in which a value corresponding to a type of an object is matched with a position within a frame of the object.

In operation S330 , the electronic device 1000 may determine the positions of the object in the plurality of frames for each of the plurality of frames by using the first learning model.

In an embodiment, the electronic device 1000 may determine not only the type of the object but also the positions of the object in the frame by using the first learning model. For example, the electronic device 1000 may determine the positions of the objects in units of pixels by using the first learning model. For example, since the electronic device 1000 may determine which object pixels constituting the frame represent, when a frame is input to the first learning model, a value corresponding to the type of the object is matched with a position within the frame of the object. A matrix can be output. Since the positions of the objects are determined in units of pixels, for example, if the size of the frame is 512 x 256, the size of the matrix may also be 512 x 256. That is, as an output of the first learning model with respect to the input of the frame, a matrix including type information of the object and location information of the object may be obtained.

In operation S340 , the electronic device 1000 may reduce the dimension of the output of the first learning model to be input to the second learning model.

The second learning model uses a Recurrent Neural Network (RNN) to analyze the time-series change of the position of the object in a plurality of frames, thereby determining whether an event related to vehicle driving has occurred and the content of the notification message. It may be generated by learning the determining criteria.

In one embodiment, the output of the first learning model may be used as an input of the second learning model. In another embodiment, the electronic device 1000 generates a matrix in which the dimension of the matrix output from the first learning model is reduced in order to reduce the amount of computation of the second learning model for determining whether an event has occurred and generating a notification message. 2 can be used as input to the learning model. For example, to reduce the dimension of a matrix, dilated convolution can be used.

Also, in an embodiment, in order to reduce the amount of computation of the first learning model, the electronic device 1000 performs 1x1 convolutional filtering on the output of the layer included in the first learning model, and includes Dimensions may be matched between layers.

In operation S350, the electronic device 1000 may determine whether an event related to driving of a vehicle has occurred using the second learning model.

In an embodiment, when the first car and the second car are stopped in front, and the third car, the fourth car, and the fifth car successively change lanes to the right, the electronic device 1000 may display the accident vehicle in front. It can be determined that an event has occurred. Also, it is possible to determine an action guideline corresponding to the event that it is better to change the lane to the right because there are accident vehicles ahead.

As described above, since the electronic device 1000 can acquire object type information and object location information on the entire screen with a small amount of computation by using the first learning model, in a method of tracking an object, Likewise, it is possible to quickly and accurately analyze time-series changes in the positions of objects without setting an ROI (Region of Interest).

Accordingly, the electronic device 1000 may determine the occurrence of an event that can be detected by analyzing a time-series location change of an object in addition to an event related to driving of the vehicle by using the second learning model. For example, the electronic device 1000 may use the second learning model to generate subtitles in real time while a movie is being played.

In operation S360, the electronic device 1000 may generate a notification message by using the second learning model.

In an embodiment, when the output of the first learning model is processed to reduce the amount of computation and input to the second learning model, a notification message may be output. Alternatively, a control operation corresponding to the event may be output, or a control operation corresponding to the event may be output together with a notification message.

In an embodiment, the notification message generated using the second learning model may be different according to the type of event and the degree of risk of driving. For example, content included in the notification message may be different depending on the type of event and the degree of risk of driving. For example, the content included in the notification message may include content for notifying an event, an action guide corresponding to the event, and an alarm sound. For example, when there are accident vehicles in front, the electronic device 1000 includes a notification of the event and action instructions in response to the event, and a notification of “Please change the lane to the right because there is an accident vehicle ahead.” You can create messages.

The content included in the notification message will be described later with reference to FIG. 5 .

4 is a diagram illustrating an example of generating a notification message using a learning model according to an embodiment.

Convolutional Neural Network (CNN) includes a fully connected layer in the rear layer to classify the class of image data. There is a problem that the location information of the disappears. In order to improve this problem, FCN (Fully Convolutional Network) considers the fully connected layer of CNN as a 1x1 convolution, allowing location information to be maintained.

In an embodiment, the electronic device 1000 may input the video sequence 410 that has been preprocessed for flattening the brightness into the first learning model. Since the first learning model uses FCN, when the electronic device 1000 inputs the video sequence 410 to the first learning model, a series of matrices including object type information and object location information may be output. have. According to an embodiment, the video sequence 410 on which preprocessing has been performed to be input to the first learning model may be input to the first learning model in the order of reproduction. An output order of a matrix output from the first learning model may be the same as an input order of a video sequence input to the first learning model.

When a series of matrices output from the first learning model are imaged, a video sequence 420 expressed in a different color for each type of object included in the video sequence 410 may be obtained. If extended convolution is performed on the video sequence 430 in which the video sequence 420 is divided into pixel units, a matrix 440 in which the dimension of the matrix output from the first learning model is reduced can be obtained. The extended convolution is a method of performing convolution using only some pixels among pixels included in the video sequence 430 . For example, by performing convolution and skipping in units of one or more pixels, the size of a receptive field (RF) may be expanded to reduce the dimension and amount of computation of a matrix.

In an embodiment, when the matrix 440 is input to the second learning model, a notification message 460 may be output. The second learning model, which uses RNN, is called a Recurrent Neural Network (RNN), which is a neural network with recurrent connections between nodes in different time intervals. The RNN according to an embodiment may recognize sequential data. The sequential data is data having temporality or order, such as audio data, image data, biometric data, handwriting data, and the like. For example, the recognition model of the RNN may recognize what pattern the input image data changes according to.

The RNN may be trained through supervised learning in which the training data and the corresponding output data are input together into the neural network, and the connection weights of the connecting lines are updated so that the output data corresponding to the training data is output. For example, the RNN may update connection weights between neurons through a delta rule and error backpropagation learning.

The RNN may have a structure including a long short-term memory (LSTM) network 450 . The LSTM network 450 is a type of RNN capable of long-term dependency learning. The RNN that does not include the LSTM network 450 can connect previous information to the current task, but has a disadvantage in that it is difficult to connect the temporally distant information of the previous task to the current task. The LSTM network 450 may be a structure designed to avoid this long-term dependency problem. The LSTM network 450 may extract a relative change amount that changes with time from the input data as a feature value, and may determine whether an event has occurred by analyzing the time-series change in the position of the object.

Since the second learning model uses the RNN including the LSTM network 450, the structure for all of the previous time step, the current time step, and the next time step is used for learning, and the information of the current step is passed to the next step. This may affect the output value.

According to an embodiment, the matrix 440 in which the dimension of the output of the first learning model is reduced may be input to the second learning model in the order outputted from the first learning model. The second learning model may generate a notification message in consideration of whether an event has occurred, the type of the event, and the risk of driving.

For convenience of explanation, the first learning model and the second learning model have been described separately, but depending on the functions and roles of the first learning model and the second learning model, a plurality of learning models may exist, and one learning model may be integrated into

5 is a diagram for explaining how content of a notification message is determined, according to an embodiment.

Referring to FIG. 5 , the table 500 shows how the content of the notification message is determined according to the type of event and the driving risk. The table 500 according to an embodiment is only an example, and the plurality of learning models may be continuously updated. Accordingly, an output value according to an input value to the plurality of learning models may be continuously updated. The electronic device 1000 may output different notification messages according to the type of event and the driving risk by using the second learning model. As shown in FIG. 5 , for example, if the type of event is that an accident is expected due to not securing a safe distance from the vehicle in front, and the driving risk is higher, the electronic device 1000 displays the event together with an alarm sound in the notification message. You can create a notification message that includes "Stop the brake immediately", which is an action guide to respond to. Also, according to an embodiment, the electronic device 1000 may determine a time during which a user input for performing an action guide may be received, based on the type of event and the driving risk. For example, the time during which the user input may be received may be determined based on the degree of risk of driving. Also, for example, which data to use to determine a time for which a user input can be received may be set and changed based on learning according to a preset criterion. For example, when it is determined that the driving risk is higher, if a user input for controlling an operation of a module installed in the vehicle is not received within a preset time, a command for the electronic device 1000 to control an operation of a module installed in the vehicle can be transmitted to a module installed in the vehicle.

For example, if the type of event is changed to a terrain with a winding road ahead and the driving risk is high, the electronic device 1000 includes the event notification and action instructions corresponding to the event in the notification message If possible, you can create a notification message that reads, “Be careful, there is a winding road ahead.”

For example, if the type of event is that the road conditions have changed because there is an accident vehicle ahead, and the driving risk is high, the notification message includes information to notify the event and action instructions to respond to the event, “ Please change lanes to the right.”

For example, if the type of event indicates that there is a change in the traffic signal because the traffic light is changed from green to red, and the driving risk is different, the electronic device 1000 displays an event notification and an action guide corresponding to the event as a notification message To be included in the , you can create a notification message that reads, “The traffic signal has changed, please stop.”

For example, if the type of event indicates that there is a change in the traffic signal because the traffic light is changed from red to green, and the driving risk is high, the electronic device 1000 displays the event notification and action instructions corresponding to the event as a notification message To be included in the , you can create a notification message that reads, “Traffic signal has changed, please leave.”

6 is a diagram illustrating an example of outputting a notification message, according to an embodiment.

In an embodiment, the electronic device 1000 may display a notification message on a head up display (HUD) of the vehicle.

For example, when the electronic device 1000 determines that an accident is expected due to not securing a safe distance from the vehicle 610 in front and the driving risk 630 is 35, the electronic device 1000 sets the driving risk level A notification message including a 630 and a virtual image 620 for securing a safe distance may be controlled to be displayed on the HUD of the vehicle. In addition, it is possible to output a notification message in the form of a voice along with an alarm sound and the contents of "Stop the brake immediately," which is an action guideline for responding to an event. Also, for example, if a step on input for stepping on the brake is not received from the user within a preset time after outputting the notification message, the electronic device 1000 sends a command for controlling the operation of the brake to the brake. can be transmitted For example, the preset time may be set by learning, and may be different according to the degree of risk 630 . For example, the higher the risk 630, the smaller the preset time interval may be set. For example, if the distance between the front vehicle 610 and the user's vehicle is too short and an accident is expected if the brake is not applied immediately, a notification message is output and a command for controlling the operation of the brake is transmitted to the brake. can

7 is a diagram illustrating an example of outputting a notification message, according to an embodiment.

In an embodiment, the electronic device 1000 may transmit a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle.

For example, if the electronic device 1000 determines that an event related to a change in road conditions that a vehicle cannot be driven in a corresponding lane due to accidents in front has occurred, the electronic device 1000 does not include information notifying the event. It is possible to control to display a preset notification message on the HUD of the vehicle. For example, the electronic device 1000 may display a notification message including at least one of a current temperature, an rpm value of the vehicle, a forward direction of the vehicle, a traffic situation, and a driving risk on the HUD. Also, “Change lanes to the right as there is an accident vehicle ahead.” Notification messages can be output in voice form. Also, for example, if a steering input for rotating the steering wheel 710 is not received from the user within a preset time after outputting a notification message in the form of a voice, a command for rotating the steering wheel 710 is sent to the steering wheel can be transmitted. Accordingly, the electronic device 1000 may autonomously adjust the driving route by inducing the user to adjust the steering wheel 710 or transmitting a command to adjust the steering wheel 710 to the steering wheel 710 .

8 and 9 are block diagrams illustrating an electronic device 1000 according to some exemplary embodiments.

As shown in FIG. 8 , the electronic device 1000 according to an embodiment may include a control unit 1300 , a communication unit 1500 , and a memory 1700 . However, not all of the components illustrated in FIG. 8 are essential components of the electronic device 1000 . The electronic device 1000 may be implemented by more components than those illustrated in FIG. 8 , or the electronic device 1000 may be implemented by fewer components than those illustrated in FIG. 8 .

For example, as shown in FIG. 9 , the electronic device 1000 according to an embodiment includes a user input unit 1100 and an output unit 1200 in addition to the control unit 1300 , the communication unit 1500 , and the memory 1700 . ), a sensing unit 1400 , and an A/V input unit 1600 may be further included.

For example, the electronic device 1000 according to an embodiment may include the control unit 1300 , the communication unit 1500 , and the memory 1700 as a vehicle board, and the electronic device 1000 according to an embodiment includes the control unit ( 1300), the communication unit 1500, and the memory 1700 in addition to the user input unit 1100, the output unit 1200, the sensing unit 1400, and the A/V input unit 1600 can be a vehicle further including at least one have.

The user input unit 1100 means a means for a user to input data for controlling the electronic device 1000 . For example, the user input unit 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, integral type). There may be a tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

The user input unit 1100 may receive a user input for controlling an operation of a module installed in the vehicle.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . have. In an embodiment, the output unit 1200 may output a notification message in the form of audio, video, and/or vibration.

The display unit 1210 displays and outputs information processed by the electronic device 1000 . For example, the display unit 1210 may display a notification message on a head up display (HUD) of the vehicle.

The sound output unit 1220 outputs audio data received from the communication unit 1500 or stored in the memory 1700 . Also, the sound output unit 1220 outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the electronic device 1000 . For example, the sound output unit 1220 may output an alarm sound to notify that an event has occurred.

The controller 1300 generally controls the overall operation of the electronic device 1000 . For example, the control unit 1300 executes programs stored in the memory 1700 , and thus the user input unit 1100 , the output unit 1200 , the sensing unit 1400 , the communication unit 1500 , and the A/V input unit 1600 . ) can be controlled in general. Also, the controller 1300 may execute the programs stored in the memory 1700 to perform the functions of the electronic device 1000 described in FIGS. 1 to 13 . The controller 1300 may include at least one processor. The control unit 1300 may include a plurality of processors or a single processor in an integrated form according to functions and roles thereof. In an embodiment, the controller 1300 may include at least one processor that provides a notification message by executing at least one program stored in the memory 1700 .

In an embodiment, the controller 1300 may obtain a video sequence including a plurality of frames from a camera installed in a vehicle through the communication unit 1500 . In an embodiment, the controller 1300 may transmit a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle, based on the type of event and the degree of risk of driving through the communication unit 1500 .

In an embodiment, the controller 1300 may detect an object included in the plurality of frames from the plurality of frames. In an embodiment, the controller 1300 may obtain location information of an object for each of a plurality of frames. The controller 1300 may determine the positions of the objects in units of pixels. In an embodiment, the controller 1300 may determine whether an event related to driving of a vehicle has occurred by analyzing a time-series change of positions of objects in a plurality of frames. In an embodiment, the controller 1300 may determine the type of event and the risk of driving by analyzing time-series changes of positions of objects in a plurality of frames. In an embodiment, the controller 1300 may generate a notification message informing of an event based on a determination of whether an event has occurred. In an embodiment, the controller 1300 may generate a notification message informing of an event based on the type of event and the degree of risk of driving. In an embodiment, the controller 1300 may control to output the generated notification message through the output unit 1200 . In an embodiment, the controller 1300 may control to display the generated notification message through the display unit 1210 . In an embodiment, the controller 1300 may perform detection of an object, acquisition of location information of an object, determination of whether an event occurs, and generation of a notification message using different learning models.

In an embodiment, the first learning model uses a Fully Convolutional Network (FCN) to determine a reference for determining the type of an object and a reference for determining positions within a plurality of frames of the object for each of a plurality of frames. It may be generated by learning, and the controller 1300 may determine the type of the object using the first learning model, and may determine positions within a plurality of frames of the object.

In an embodiment, the second learning model uses a Recurrent Neural Network (RNN) to analyze a time-series change of a position of an object in a plurality of frames, thereby determining whether an event related to driving of a vehicle has occurred. The message may be generated by learning the criteria for determining the content of the message, and the controller 1300 may determine whether an event related to driving of the vehicle has occurred using the second learning model, and may determine the content of the notification message.

In one embodiment, the controller 1300 may apply a filter for flattening the brightness of the plurality of frames to the plurality of frames to input to the first learning model, and perform the second learning For input to the model, the dimensionality of the output of the first learning model may be reduced.

The sensing unit 1400 may detect a state of the electronic device 1000 , a user state, or a state around the electronic device 1000 , and transmit the sensed information to the controller 1300 .

The sensing unit 1400 includes a magnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and at least one of an RGB sensor 1490 , but is not limited thereto. Since a function of each sensor can be intuitively inferred from the name of a person skilled in the art, a detailed description thereof will be omitted.

The communication unit 1500 may include one or more components that allow the electronic device 1000 to communicate with another electronic device (not shown) and the server 2000 . Another electronic device (not shown) may be a computing device or a sensing device, but is not limited thereto. Also, for example, the other electronic device may be a module included in a vehicle, such as the electronic device 1000 . For example, the communication unit 1500 may include a short-distance communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit 1510, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.

The mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an implementation example, the electronic device 1000 may not include the broadcast receiver 1530 .

In an embodiment, the communication unit 1500 may receive a video sequence including a plurality of frames from a camera installed in a vehicle. In an embodiment, the communication unit 1500 may transmit a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle.

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like. The camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the controller 1300 or a separate image processing unit (not shown). For example, an image captured by the camera 1610 may be used as information for determining whether an event has occurred.

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 may receive an acoustic signal from an external electronic device or a user. The microphone 1620 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal.

The memory 1700 may store a program for processing and control of the controller 1300 , and may also store data input to or output from the electronic device 1000 .

The memory 1700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , and the like. .

The UI module 1710 may provide a specialized UI, GUI, etc. interworking with the electronic device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the controller 1300 . The touch screen module 1720 according to an embodiment may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

The notification module 1730 may generate a signal for notifying the occurrence of an event. The notification module 1730 may output a notification signal in the form of a video signal through the display unit 1210 , may output a notification signal in the form of an audio signal through the sound output unit 1220 , and the vibration motor 1230 . It is also possible to output a notification signal in the form of a vibration signal through

10 is a block diagram illustrating a processor 1300 according to an embodiment.

Referring to FIG. 10 , a processor 1300 according to an embodiment may include a data learning unit 1310 and a data recognition unit 1320 .

The data learning unit 1310 may acquire pixel information and learn a criterion for generating a notification message. The data learning unit 1310 obtains pixel information and learns criteria for which data to use to generate a notification message, how to obtain pixel information using the data, and how to generate a notification message. . The data learner 1310 acquires data to be used for learning, and applies the acquired data to a data recognition model to be described later, thereby acquiring pixel information and learning a criterion for generating a notification message.

1 to 9, detecting an object, obtaining location information of the object, determining the type of object, determining the location of the object, determining whether an event has occurred, determining the type of event, determining the risk of driving, notification message Although it has been described that an operation for generating a , and an operation for determining the content of the notification message is performed as each operation, the present invention is not limited thereto. Detect object, obtain object location information, determine the type of object, determine the location of the object, determine whether an event has occurred, determine the type of event, determine the risk of driving, generate a notification message, content of the notification message At least two or more of performing the operation for determining , may be performed based on learning according to a preset criterion.

The data recognition unit 1320 may obtain pixel information based on data and generate a notification message. The data recognition unit 1320 may recognize pixel information and a notification message from predetermined data by using the learned data recognition model. The data recognition unit 1320 acquires predetermined data according to a preset criterion by learning, and uses the data recognition model using the acquired data as an input value, thereby acquiring and notifying how pixel information is obtained based on the predetermined data. You can decide how to create a message. In addition, a result value output by the data recognition model using the obtained data as an input value may be used to update the data recognition model.

At least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor (eg, CPU) Alternatively, it may be manufactured as a part of an application processor) or a graphics-only processor (eg, GPU) and mounted on the various electronic devices described above.

In this case, the data learning unit 1310 and the data recognition unit 1320 may be mounted on one electronic device or may be mounted on separate electronic devices, respectively. For example, one of the data learning unit 1310 and the data recognition unit 1320 may be included in the electronic device, and the other one may be included in the server. In addition, the data learning unit 1310 and the data recognition unit 1320 may provide the model information constructed by the data learning unit 1310 to the data recognition unit 1320 through wired or wireless communication, and the data recognition unit ( Data input to 1320 may be provided to the data learning unit 1310 as additional learning data.

Meanwhile, at least one of the data learning unit 1310 and the data recognition unit 1320 may be implemented as a software module. When at least one of the data learning unit 1310 and the data recognition unit 1320 is implemented as a software module (or a program module including instructions), the software module is a computer-readable non-transitory readable It may be stored in a recording medium (non-transitory computer readable media). Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

11 is a block diagram illustrating a data learning unit 1310 according to an exemplary embodiment.

Referring to FIG. 11 , the data learning unit 1310 according to an exemplary embodiment includes a data acquiring unit 1310-1, a preprocessing unit 1310-2, a training data selection unit 1310-3, and a model learning unit 1310. -4) and a model evaluation unit 1310-5.

The data acquisition unit 1310-1 may acquire data necessary for determining how to acquire pixel information and how to generate a notification message. The data acquisition unit 1310-1 may acquire data necessary for learning for determining how to acquire pixel information and how to generate a notification message.

For example, the data acquisition unit 1310-1 may acquire voice data, image data, text data, or biosignal data. For example, the data acquisition unit 1310-1 may receive data through an input device (eg, a microphone, a camera, or a sensor) of the electronic device 1000 . Alternatively, the data acquisition unit 1310-1 may acquire data through another electronic device that communicates with the electronic device 1000 . Alternatively, the data acquisition unit 1310-1 may acquire data through a server that communicates with the electronic device 1000 .

For example, the data acquisition unit 1310-1 may receive a video sequence from a camera installed in a vehicle. Also, for example, the data acquisition unit 1310-1 may receive a video sequence from a camera capable of photographing the surroundings of the vehicle. Also, for example, the data acquisition unit 1310-1 may acquire a video sequence from a camera included in the electronic device 1000 .

The preprocessor 1310 - 2 may preprocess the acquired data so that the acquired data can be used for learning for determining how to acquire pixel information and how to generate a notification message. The preprocessor 1310-2 obtains the data obtained so that the model learning unit 1310-4, which will be described later, uses the acquired data for learning to determine how to acquire pixel information and how to generate a notification message. data can be processed into a preset format. For example, the preprocessor 1310 - 2 may perform preprocessing by applying a filter for flattening the brightness of the plurality of frames included in the video sequence to the plurality of frames.

The learning data selection unit 1310 - 3 may select data necessary for learning from among the pre-processed data. The selected data may be provided to the model learning unit 1310 - 4 . The learning data selector 1310 - 3 may select data required for learning from among preprocessed data according to preset criteria for determining how to obtain pixel information and how to generate a notification message. In addition, the training data selection unit 1310 - 3 may select data according to a preset criterion by learning by the model learning unit 1310 - 4 to be described later.

The model learning unit 1310 - 4 may learn a criterion for how to acquire pixel information and how to generate a notification message based on the training data. Also, the model learning unit 1310 - 4 may learn a criterion for which training data to use to determine how to acquire pixel information and how to generate a notification message.

Also, the model learning unit 1310 - 4 may train a data recognition model used for determining how to acquire pixel information and how to generate a notification message based on the training data, using the training data. In this case, the data recognition model may be a pre-built model. For example, the data recognition model may be a model built in advance by receiving basic learning data (eg, an image of a black box of a vehicle, etc.).

The data recognition model may be constructed in consideration of the field of application of the recognition model, the purpose of learning, or the computer performance of the device. The data recognition model may be, for example, a model based on a neural network. For example, a model such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Fully Convolutional Network (FCN), or a Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the data recognition model, but is not limited thereto.

According to various embodiments, when a plurality of pre-built data recognition models exist, the model learning unit 1310 - 4 uses a data recognition model that has a high correlation between the input training data and the basic learning data as a data recognition model to learn. can decide In this case, the basic learning data may be pre-classified for each type of data, and the data recognition model may be previously built for each type of data. For example, the basic learning data may be pre-classified according to various criteria such as a region where the learning data is generated, a time when the training data is generated, a size of the training data, a genre of the training data, a generator of the training data, and the like.

Also, the model learning unit 1310 - 4 may train the data recognition model using, for example, a learning algorithm including error back-propagation or gradient descent. .

Also, the model learning unit 1310 - 4 may train the data recognition model through, for example, supervised learning using learning data as an input value. In addition, the model learning unit 1310-4 learns, for example, the type of data necessary for determining how to acquire pixel information and how to generate a notification message based on the training data without any guidance. By doing so, it is possible to train the data recognition model through unsupervised learning to find a criterion for determining how to obtain pixel information and how to generate a notification message based on the training data. In addition, the model learning unit 1310-4, for example, based on the training data according to the learning, how to obtain the pixel information and how to generate the notification message feedback on whether the result of the determination is correct A data recognition model can be trained through reinforcement learning that is used.

Also, when the data recognition model is learned, the model learning unit 1310 - 4 may store the learned data recognition model. In this case, the model learning unit 1310 - 4 may store the learned data recognition model in the memory of the electronic device 1000 including the data recognition unit 1320 . Alternatively, the model learning unit 1310 - 4 may store the learned data recognition model in the memory of the electronic device 1000 including the data recognition unit 1320 to be described later. Alternatively, the model learning unit 1310 - 4 may store the learned data recognition model in a memory of a server connected to the electronic device 1000 through a wired or wireless network.

In this case, the memory in which the learned data recognition model is stored may also store, for example, commands or data related to at least one other component of the electronic device 1000 . The memory may also store software and/or programs. A program may include, for example, a kernel, middleware, an application programming interface (API) and/or an application program (or "application"), and the like.

The model evaluation unit 1310-5 may input evaluation data into the data recognition model, and when the recognition result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit 1310-4 may be trained again. have. In this case, the evaluation data may be preset data for evaluating the data recognition model.

For example, the model evaluator 1310-5 sets a predetermined criterion when the number or ratio of evaluation data for which the recognition result is not accurate among the recognition results of the learned data recognition model for the evaluation data exceeds a preset threshold. It can be evaluated as unsatisfactory. For example, when the predetermined criterion is defined as a ratio of 2%, when the learned data recognition model outputs an erroneous recognition result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit 1310-5 can be evaluated that the trained data recognition model is not suitable.

On the other hand, when there are a plurality of learned data recognition models, the model evaluation unit 1310 - 5 evaluates whether each learned data recognition model satisfies a predetermined criterion, and recognizes a model that satisfies the predetermined criterion as final data It can be determined as a model. In this case, when there are a plurality of models that satisfy the predetermined criteria, the model evaluator 1310 - 5 may determine any one or a predetermined number of models preset in the order of the highest evaluation score as the final data recognition model.

On the other hand, in the data learning unit 1310, the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310 At least one of -5) may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 One may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or it may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU) as described above. It may be mounted on various electronic devices.

In addition, the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 are one electronic unit. It may be mounted on the device, or may be respectively mounted on separate electronic devices. For example, some of the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 may be included in the electronic device, and the remaining part may be included in the server.

In addition, at least one of the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 is It may be implemented as a software module. At least one of the data acquisition unit 1310-1, the preprocessor 1310-2, the training data selection unit 1310-3, the model learning unit 1310-4, and the model evaluation unit 1310-5 is a software module When implemented as (or, a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

12 is a block diagram of a data recognition unit 1320 according to an exemplary embodiment.

Referring to FIG. 12 , the data recognition unit 1320 according to an embodiment includes a data acquisition unit 1320-1, a preprocessor 1320-2, a recognition data selection unit 1320-3, and a recognition result providing unit ( 1320-4) and a model update unit 1320-5.

The data acquisition unit 1320-1 may acquire data necessary for determining how to acquire pixel information and how to generate a notification message based on the training data, and the pre-processing unit 1320-2 determines how the pixel information is to be generated. The acquired data may be pre-processed so that the acquired data can be used for determining whether to acquire information and how to generate a notification message. The preprocessor 1320-2 is configured to use the acquired data to determine how the recognition result providing unit 1320-4, which will be described later, acquires pixel information and how to generate a notification message. can be processed in a preset format.

The recognition data selection unit 1320 - 3 may select data necessary for determining how to acquire pixel information and how to generate a notification message from among the preprocessed data. The selected data may be provided to the recognition result providing unit 1320 - 4 . The recognition data selection unit 1320 - 3 may select some or all of the preprocessed data according to preset criteria for how to obtain pixel information and how to generate a notification message. Also, the recognition data selection unit 1320 - 3 may select data according to a preset criterion by learning by the model learning unit 1310 - 4 to be described later.

The recognition result providing unit 1320 - 4 may determine how to obtain pixel information and how to generate a notification message by applying the selected data to the data recognition model. The recognition result providing unit 1320 - 4 may provide a recognition result according to the purpose of data recognition. The recognition result providing unit 1320-4 may apply the selected data to the data recognition model by using the data selected by the recognition data selection unit 1320-3 as an input value. Also, the recognition result may be determined by a data recognition model.

The model updater 1320 - 5 may update the data recognition model based on the evaluation of the recognition result provided by the recognition result providing unit 1320 - 4 . For example, the model updating unit 1320-5 provides the recognition result provided by the recognition result providing unit 1320-4 to the model learning unit 1310-4, so that the model learning unit 1310-4 is The data recognition model can be updated.

On the other hand, in the data recognition unit 1320, the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit ( 1320-5) may be manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, among the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result provision unit 1320-4, and the model update unit 1320-5 At least one may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). It may be mounted on one various electronic devices.

In addition, the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit 1320-5 are one It may be mounted on an electronic device, or may be respectively mounted on separate electronic devices. For example, among the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result provision unit 1320-4, and the model update unit 1320-5 Some may be included in the electronic device, and some may be included in the server.

In addition, at least one of the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit 1320-5 may be implemented as a software module. At least one of the data acquisition unit 1320-1, the preprocessor 1320-2, the recognition data selection unit 1320-3, the recognition result providing unit 1320-4, and the model update unit 1320-5 is software. When implemented as a module (or a program module including instructions), the software module may be stored in a computer-readable non-transitory computer readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the at least one software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

13 is a diagram illustrating an example of learning and recognizing data by interworking with the electronic device 1000 and the server 2000 according to an embodiment.

Referring to FIG. 13 , the server 2000 may learn criteria for how to obtain pixel information and how to generate a notification message. The electronic device 1000 may determine how to acquire pixel information and how to generate a notification message based on the learning result by the server 2000 .

In this case, the model learning unit 2340 of the server 2000 may perform the function of the data learning unit 1310 illustrated in FIG. 11 . The model learning unit 2340 of the server 2000 determines which data to use to determine how to obtain pixel information and how to generate a notification message, and how to obtain pixel information using predetermined data. , and a criterion for how to generate a notification message may be learned. The model learning unit 2340 acquires data to be used for learning, and applies the acquired data to a data recognition model to be described later, thereby determining how to acquire pixel information and how to generate a notification message. can learn

In addition, the recognition result providing unit 1320 - 4 of the electronic device 1000 applies the data selected by the recognition data selection unit 1320 - 3 to the data recognition model generated by the server 2000 to obtain pixel information. It is possible to determine whether to obtain it, and how to generate a notification message. For example, the recognition result providing unit 1320-4 transmits the data selected by the recognition data selection unit 1320-3 to the server 2000, and the server 2000 performs the recognition data selection unit 1320-3 It may be requested to determine how to obtain pixel information and how to generate a notification message by applying the data selected by the to the recognition model. Also, the recognition result providing unit 1320 - 4 may receive from the server 2000 information on how to obtain pixel information and how to generate a notification message determined by the server 2000 .

Alternatively, the recognition result providing unit 1320 - 4 of the electronic device 1000 receives the recognition model generated by the server 2000 from the server 2000, and uses the received recognition model to obtain pixel information. and how to generate a notification message. In this case, the recognition result providing unit 1320 - 4 of the electronic device 1000 applies the data selected by the recognition data selection unit 1320 - 3 to the data recognition model received from the server 2000 to obtain pixel information. It is possible to determine whether to obtain it, and how to generate a notification message.

Some embodiments may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules to be executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Also, in this specification, “unit” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (20)

An electronic device for controlling the operation of a vehicle, comprising:
a memory in which at least one program is stored;
At least one processor configured to provide a notification message by executing the at least one program,
the at least one program,
obtaining a video sequence including a plurality of frames from a camera installed in the vehicle;
detecting an object included in the plurality of frames from the plurality of frames;
obtaining location information of the object for each of the plurality of frames;
determining whether an event related to driving of the vehicle has occurred by analyzing a time-series change of positions of the object in the plurality of frames;
generating a notification message informing of the event based on the determination; and
outputting the generated notification message;
Includes commands that execute
Detecting the object and acquiring the location information are performed using a first learning model, and determining whether the event has occurred and generating the notification message are performed using a second learning model being an electronic device.
The method of claim 1,
The electronic device, wherein the location information of the object output from the first learning model is input to the second learning model.
The method of claim 1,
The step of determining whether the event has occurred,
determining the type of the event; and
determining the degree of risk of the driving;
including,
The step of generating the notification message is,
and generating the notification message based on the type of the event and the driving risk.
4. The method of claim 3,
According to the type of the event and the degree of risk of driving, a command for controlling an operation of a module installed in the vehicle is transmitted to the module.
The method of claim 1,
The step of outputting the notification message is,
and displaying the notification message on a head up display (HUD) of the vehicle.
The method of claim 1,
Obtaining the location information of the object comprises:
and determining the positions of the object in units of pixels from the plurality of frames.
The method of claim 1,
The first learning model is a criterion for determining the type of the object using a Fully Convolutional Network (FCN), and a criterion for determining the positions of the object in the plurality of frames for each of the plurality of frames is created by learning
The step of detecting the object is to determine the type of the object using the first learning model,
The obtaining of the location information of the object may include determining positions of the object within the plurality of frames for each of the plurality of frames by using the first learning model.
The method of claim 1,
The input of the second learning model, the electronic device of which the dimension (dimension) of the output of the first learning model is reduced.
The method of claim 1,
The second learning model is generated by learning a criterion for determining whether an event related to driving of the vehicle has occurred and a criterion for determining the content of the notification message using a Recurrent Neural Network (RNN),
The step of determining whether the event has occurred determines whether an event related to driving of the vehicle has occurred using the second learning model,
The generating of the notification message may include determining content of the notification message by using the second learning model.
The method of claim 1,
the at least one program,
and applying to the plurality of frames a filter for flattening a lightening degree of the plurality of frames to be input to the first learning model.
A method for controlling the operation of a vehicle, the method comprising:
obtaining a video sequence including a plurality of frames from a camera installed in the vehicle;
detecting an object included in the plurality of frames from the plurality of frames;
obtaining location information of the object for each of the plurality of frames;
determining whether an event related to driving of the vehicle has occurred by analyzing a time-series change of positions of the object in the plurality of frames;
generating a notification message informing of the event based on the determination; and
outputting the generated notification message;
includes,
Detecting the object and acquiring the location information are performed using a first learning model, and determining whether the event has occurred and generating the notification message are performed using a second learning model How to become.
12. The method of claim 11,
The method, wherein the position information of the object output from the first learning model is input to the second learning model.
12. The method of claim 11,
The step of determining whether the event has occurred,
determining the type of the event; and
determining the degree of risk of the driving;
including,
The step of generating the notification message is,
and generating the notification message based on the type of the event and the degree of risk of driving.
14. The method of claim 13,
According to the type of the event and the degree of risk of driving, a command for controlling an operation of a module installed in the vehicle is transmitted to the module.
12. The method of claim 11,
Obtaining the location information of the object comprises:
and determining the positions of the object in units of pixels from the plurality of frames.
12. The method of claim 11,
The first learning model is a criterion for determining the type of the object using a Fully Convolutional Network (FCN), and a criterion for determining the positions of the object in the plurality of frames for each of the plurality of frames is created by learning
The step of detecting the object is to determine the type of the object using the first learning model,
The obtaining of the location information of the object may include determining positions within the plurality of frames of the object for each of the plurality of frames by using the first learning model.
12. The method of claim 11,
The method of claim 1, wherein the input of the second learning model is reduced in dimension of the output of the first learning model.
12. The method of claim 11,
The second learning model is generated by learning a criterion for determining whether an event related to driving of the vehicle has occurred and a criterion for determining the content of the notification message using a Recurrent Neural Network (RNN),
The step of determining whether the event has occurred determines whether an event related to driving of the vehicle has occurred using the second learning model,
In the generating of the notification message, the content of the notification message is determined by using the second learning model.
12. The method of claim 11,
and applying to the plurality of frames a filter for flattening a lightening degree of the plurality of frames for input to the first learning model.
A computer-readable recording medium in which a program for executing the method of claim 11 in a computer is recorded.

Images