KR20220059923A - Electronic device and method for processing data received from in vehicle electronic device - Google Patents

Abstract

The present invention relates to an electronic device and method for processing data received from an electronic device in a vehicle. According to various embodiments of the present invention, the electronic device is able to process a video streamed on a first external electronic device such as an image acquisition device for a vehicle, based on a designated status (e.g., traffic accident) identified from the first external electronic device. For instance, when a notification related to a designated status is received from the first external electronic device, the electronic device is able to acquire the information related to the designated status and broadcast the acquired information to one or more terminals (e.g., the terminal in Fig. 5 (130)) through a second external electronic device such as a navigation server. The terminals can respond to the broadcast and provide information related to the designated status to a user of the terminals. The information can be used for changing a navigation path provided by the terminals. The electronic device and method for processing data received from an electronic device in a vehicle are capable of rapidly recognizing an event occurring in the vehicle.

Description

ELECTRONIC DEVICE AND METHOD FOR PROCESSING DATA RECEIVED FROM IN VEHICLE ELECTRONIC DEVICE

Various embodiments disclosed in this document relate to an electronic device and method for processing data obtained and received by an electronic device for a vehicle.

A vehicle image acquisition device called a so-called vehicle dash cam (Car Dashboard Camera) may be disposed in the vehicle. In a state of driving the vehicle, the vehicle image acquisition device may detect a specified state (eg, sudden braking and/or collision) of the vehicle and store a video based thereon.

With the development of communication technology, there is a method for quickly recognizing data being captured by the vehicle data acquisition device and/or a specified state and/or event recognized by the vehicle data acquisition device, and performing a related function based on the recognition result. may be requested

An electronic device according to various embodiments includes a communication circuit, a memory, and at least one processor operably coupled to the communication circuit and the memory, wherein the memory includes a plurality of store instructions, wherein, when the plurality of instructions are executed by the at least one processor, the at least one processor receives, using the communication circuit, a video from a first external electronic device that is distinct from the electronic device and, in a state of receiving the video, receives a first signal indicating entry into a specified state from the first external electronic device, and in response to the reception of the first signal, identifies information related to the received video and, in response to the identification of the information, control to transmit a second signal including at least a portion of the information to a second external electronic device distinct from the first external electronic device.

A terminal according to various embodiments includes a built-in display, a communication circuit, a memory and at least one processor operably coupled to the display, the communication circuit, and the memory, wherein the memory includes a plurality of store instructions, wherein, when the plurality of instructions are executed by the at least one processor, the at least one processor communicates with a first external electronic device based on a designated application, and receives the instructions from the first external electronic device. , receive information related to a second external electronic device that is distinguished from the first external electronic device, and in response to receiving the information, identify an external display that is distinct from the built-in display, and the external display is connected to the terminal In a first state, display a first visual element related to the information on the external display, and in a second state distinct from the first state, display the first visual element on the built-in display, and in a state in which the first visual element is displayed, in response to identification of a user input associated with the first visual element, control to display a second visual element associated with the first visual element and comprising at least a portion of the information; , the second visual element may be displayed on the built-in display in the first state and displayed on the external display in the second state.

The electronic device according to various embodiments of the present disclosure may quickly recognize data being captured by the in-vehicle data acquisition device and/or an event occurring in a vehicle in which the in-vehicle data acquisition device is disposed, and perform a related function based on the recognition result. can

1 is a block diagram related to an embodiment of an electronic device connected through a network.
2 is a signal flow diagram illustrating an operation of an electronic device according to various embodiments of the present disclosure;
3 is a flowchart of an electronic device according to an exemplary embodiment.
4 is a flowchart of a terminal processing information provided from an electronic device according to an exemplary embodiment.
5 is a diagram illustrating an example of a UI output by a terminal in response to information provided from an electronic device according to an embodiment.
6 is a block diagram of an image acquisition apparatus 601 for a vehicle according to various embodiments.
7 is a diagram illustrating a data structure of minimum accident information (MSD) generated by an electronic device according to an exemplary embodiment.
FIG. 8 is a diagram illustrating data types, restrictions, and definitions of information (VehicleIdentificationNumber) referenced in sequence number 4 of FIG. 7 .
9 is a block diagram illustrating an autonomous driving system of a vehicle, according to an exemplary embodiment.
10 is a block diagram of an electronic device according to an embodiment.
11 is a block diagram of a server according to an embodiment.
12 is a signal flow diagram illustrating an operation of an electronic device according to various embodiments of the present disclosure;
13 is a signal flow diagram illustrating an operation of a server according to various embodiments of the present disclosure.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between elements, for example, “between” and “between” or “directly adjacent to”, etc., should be interpreted similarly.

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference numerals in each figure indicate like elements.

1 is a block diagram related to an embodiment of an electronic device 101 connected through a network. Referring to FIG. 1 , the electronic device 101 may be connected to the first external electronic device 110 and the second external electronic device 120 based on a wired network and/or a wireless network. The second external electronic device 120 may be connected to one or more terminals 130 through a wired network and/or a wireless network. In an embodiment, the first external electronic device 110 may include at least one vehicle image acquisition device disposed in a vehicle. In an embodiment, the vehicle image acquisition device includes a device capable of receiving a light signal reflected from a subject by an image sensor, such as a car video recorder, a car dashboard camera, and the like and converting it into an electrical image signal. may include

In an embodiment, the second external electronic device 120 may provide information (eg, navigation information) for guiding the movement of a vehicle to one or more terminals 130 . In an embodiment, the second external electronic device 120 may correspond to a server (eg, a navigation server) of a service provider of a navigation service for providing navigation information. In an embodiment, the second external electronic device 120 may correspond to an autonomous driving server that generates an autonomous driving algorithm for supporting an autonomous driving function of a vehicle. For example, the second external electronic device 120 receives the video streaming obtained by the first external electronic device 110 through the electronic device 101 connected to the network, and according to the machine learning-based autonomous driving algorithm After generating an algorithm or artificial neural network for autonomous driving of a vehicle equipped with the electronic device 101, the generated path information for autonomous driving is transmitted to the electronic device 101 to assist the autonomous driving function of the vehicle. ) can respond to autonomous driving servers. In an embodiment, the processor 103 of the electronic device 101 may generate a control command for controlling autonomous driving of the vehicle. The control command for controlling the autonomous driving of the vehicle may include an acceleration command, a deceleration command, a steering wheel angle control command, a turn indicator light control command, a wiper control command, and the like. In an embodiment, when a separate processor of the autonomous driving control system that generates a control command for autonomous driving of the vehicle exists, the processor 103 of the electronic device 101 controls the autonomous driving control system using the processor of the autonomous driving control system. command can be printed.

Although only one first external electronic device 110 and one second external electronic device 120 are illustrated, the number of first external electronic devices 110 with which the electronic device 101 can communicate simultaneously and the second external electronic device The number of 120 may be one or more. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network is a network such as Long Term Evolution (LTE), 5g New Radio (NR), Wireless Fidelity (WiFi), Zigbee, Near Field Communication (NFC), Bluetooth, Bluetooth Low-Energy (BLE), or a combination thereof. may include Although the electronic device 101 , the first external electronic device 110 , and the second external electronic device 120 are illustrated as being directly connected, the electronic device 101 and the first external electronic device 110 . and the second external electronic device 120 may be indirectly connected through one or more routers and/or access points (APs).

Referring to FIG. 1 , an electronic device 101 according to an embodiment may include at least one of a processor 103 , a memory 105 , and a communication circuit 107 . The processor 103 , the memory 105 , and the communication circuit 107 may be electrically and/or operatively coupled to each other by an electronic component, such as a communication bus. operably coupled with each other). The type and/or number of hardware components included in the electronic device 101 is not limited to that illustrated in FIG. 1 . For example, the electronic device 101 may include only some of the hardware components illustrated in FIG. 1 .

The processor 103 of the electronic device 101 according to an embodiment may include a hardware component for processing data based on one or more instructions. Hardware components for processing data may include, for example, an Arithmetic and Logic Unit (ALU), a Field Programmable Gate Array (FPGA), and/or a Central Processing Unit (CPU). The number of processors 103 may be one or more. For example, the processor 103 may have a structure of a multi-core processor such as a dual core, a quad core, or a hexa core.

The memory 105 of the electronic device 101 according to an embodiment may include a hardware component for storing data and/or instructions input and/or output to the processor 103 . The memory 105 may include, for example, volatile memory such as random-access memory (RAM) and/or non-volatile memory such as read-only memory (ROM). there is. The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disk, and an embedded multi media card (eMMC). can

In the memory 105 , one or more instructions indicating an operation to be performed by the processor 103 on data may be stored. A set of instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and/or an application. For example, the electronic device 101 and/or the processor 103 of the electronic device 101 executes a set of a plurality of instructions distributed in the form of an application, so that FIGS. 2 to 3 . at least one of the operations of

The communication circuit 107 of the electronic device 101 according to an embodiment includes the electronic device 101 and another electronic device distinct from the electronic device 101 (eg, the first external electronic device 110 and/or A hardware component for supporting transmission and/or reception of an electrical signal between the second external electronic device 120 may be included. The communication circuit 107 may include, for example, at least one of a modem (MODEM), an antenna, and an O/E (Optic/Electronic) converter. Communication circuit 107, Ethernet (ethernet), LAN (Local Area Network), WAN (Wide Area Network), WiFi (Wireless Fidelity), Bluetooth, BLE (Bluetooth Low Energy), ZigBee, LTE (Long Term Evolution), Transmission and/or reception of electrical signals may be supported based on various types of protocols such as 5G New Radio (NR).

The electronic device 101 according to various embodiments may receive a video being captured by the first external electronic device 110 . In the state of receiving the video, the electronic device 101 may receive a notification related to a specified state transmitted from the first external electronic device 110 . The specified state may include, for example, a state in which an impact amount exceeding a specified threshold is applied to the first external electronic device 110 and/or the vehicle in which the first external electronic device 110 is disposed. The amount of impact may be identified by, for example, an acceleration sensor and/or a gyro sensor included in the first external electronic device 110 . The amount of impact may be related to, for example, a collision and/or an accident occurring in a vehicle in which the first external electronic device 110 is disposed.

In response to receiving the notification related to the specified state, the electronic device 101 may store the video received from the first external electronic device 110 . The electronic device 101 may obtain information related to the stored video. The information may include, for example, information related to the location of the first external electronic device 110 and/or the amount of impact identified in the first external electronic device 110 . The electronic device 101 may transmit the obtained information to the second external electronic device 120 . The second external electronic device 120 may transmit information received from the electronic device 101 to the at least one terminal 130 related to the location of the first external electronic device 110 .

Although it has been described that the first external electronic device 110 acquires and stores the video in the above-described exemplary embodiment, the present invention is not limited thereto. For example, if the first external electronic device 110 includes a Light Detection and Ranging (LiDAR) sensor and/or a RAdio Detection and Ranging (RADAR) sensor, sensor information obtained from the LiDAR sensor and/or RADAR sensor or , information combined with information reconstructed using the camera may be transmitted to the second external electronic device 120 .

Referring to FIG. 1 , the terminal 130 may include at least one of a processor 131 , a memory 132 , a communication circuit 133 , a display 134 , and a sensor 135 . The processor 131 , the memory 132 , the communication circuit 133 , the display 134 , and/or the sensor 135 may be electrically connected by a communication bus. Each of the processor 131 , the memory 132 , and the communication circuit 133 may operate at least partially similarly to the processor 103 , the memory 105 , and the communication circuit 107 .

The terminal 130 may display a user interface (UI) based on information received from the second external electronic device 120 on the display 134 . An operation in which the terminal 130 displays the UI based on information received from the second external electronic device 120 will be described in detail with reference to FIGS. 4 to 5 .

2 is a signal flow diagram for explaining an operation of the electronic device 101 according to various embodiments of the present disclosure. The electronic device 101 , the first external electronic device 110 , the second external electronic device 120 , and the terminal 130 of FIG. 2 are the electronic device 101 and the first external electronic device 110 of FIG. 1 , respectively. , the second external electronic device 120 and the terminal 130 may be included. For example, at least one of the operations of the electronic device 101 illustrated in FIG. 2 may be performed by the electronic device 101 and/or the processor 103 of FIG. 1 .

Referring to FIG. 2 , in operation 205 , the electronic device 101 according to an embodiment may receive a video being captured from the first external electronic device 110 . The video may be streamed based on a real time video transferring protocol.

In an embodiment, the streaming of the video may be performed on a condition that satisfies a specified criterion. For example, the first external electronic device 110 uses an electronic map stored in advance in an electronic device (eg, the terminal 130 ) related to the first external electronic device 110 and the first external electronic device 110 . When the images (or videos) obtained through the process are different from each other, streaming of the video may be performed. As another example, the first external electronic device 110 may stream the video based on identifying that the mode of the vehicle including the first external electronic device 110 is changed from the autonomous driving mode to the manual driving mode. can run However, it is not limited thereto. As another example, the first external electronic device 110 includes information (position, shape) of objects located on a road that is stored in advance in an electronic device (eg, the terminal 130 ) related to the first external electronic device 110 . ) and information (position, shape) of objects existing in the video acquired through the first external electronic device 110 are different from each other, streaming of the video may be performed. Here, the objects may include a traffic light, a crosswalk, a boundary stone, a tree, a speed bump, a street lamp, a median, a lane, a toll gate, a road sign, a road mark, and the like. In an embodiment, the streaming of the video may be performed for a specified time from the timing of detecting the specified reference. However, it is not limited thereto.

In a state in which streaming based on operation 205 is being performed in the electronic device 101 and the first external electronic device 110 , the first external electronic device 110 may An amount of impact exceeding a threshold applied to a vehicle (eg, a vehicle in which the first external electronic device 110 is disposed) may be identified. In response to the identification of the impact amount, the first external electronic device 110 may transmit a signal 210 informing the electronic device 101 of entering a specified state related to the impact amount. The signal 210 includes the GPS coordinates of the first external electronic device 110 , the amount of impact identified in the first external electronic device 110 , and a video frame captured by the first external electronic device 110 at the moment when the amount of impact is identified. , may include at least one of audio frames.

In response to the reception of the signal 210 , in operation 215 , the electronic device 101 according to an embodiment may store the video being streamed. Storing the video by the electronic device 101 may include storing at least a portion of information included in the signal 210 in metadata of the video. In one embodiment, storing the video means storing the video in a different storage device different from a buffer used for streaming the video so that the video streamed through operation 205 can be further processed. can do. In one embodiment, storing the video means distinguishing the video from the electronic device 101 and executing the recognition in a storage device of another electronic device so as to perform recognition of the video. This may include storing However, it is not limited thereto. In an embodiment, the length of the time interval for maintaining the storage may be changed according to the type of the signal in operation 210 . However, it is not limited thereto.

In an embodiment, the video may be discarded after a specified time elapses when the signal is not received. For example, the designated time may be determined based on a size of a buffer configured to temporarily store the video. However, it is not limited thereto.

Referring to FIG. 2 , in operation 220 , the electronic device 101 according to an embodiment may acquire information related to a stored video. The information may include at least a part of information related to a specified state included in the signal 210 and/or a result of recognizing the video being streamed. For example, the electronic device 101 may recognize a video using a neural network, and identify information related to a specified state (eg, an external vehicle and/or a pedestrian related to the identified impact amount) from the video.

In response to the acquisition of the information, the electronic device 101 transmits a signal 225 including at least a portion of the acquired information to the second external electronic device 120 distinct from the first external electronic device 110 . can The signal 225 may include at least a portion of information obtained in operation 220 and/or information included in signal 210 .

In response to the reception of the signal 225 , in operation 230 , the second external electronic device 120 may identify one or more terminals related to information included in the signal. For example, based on the location of the first external electronic device 110 included in the signal 210 , the second external electronic device 120 selects one or more terminals related to the location of the first external electronic device 110 . can be identified. For example, the second external electronic device 120 may identify one or more terminals existing within a specified distance based on the location of the first external electronic device 110 . For example, the second external electronic device 120 may identify one or more moving terminals based on a path that at least partially overlaps with the location of the first external electronic device 110 . As shown in FIG. 2 , it is assumed that the second external electronic device 120 identifies one or more terminals including the terminal 130 .

The second external electronic device 120 is one or more terminals related to the location of the first external electronic device 110 including the terminal 130 , and includes information related to a specified state of the first external electronic device 110 . Signal 235 may be transmitted. The terminal 130 may receive the signal including the information. In an embodiment, the terminal 130 may change at least a part of the navigation service provided from the second external electronic device 120 based on the reception of the signal. For example, the terminal 130 may change the route being guided in the vehicle for the movement of the vehicle including the terminal 130 . For example, when the information indicates that an external object such as a port hole or a speed bump not included in the electronic map exists, the terminal 130 bypasses the location of the external object, the navigation service For this purpose, the path displayed on the display of the terminal 130 may be changed. However, it is not limited thereto. In an embodiment, the terminal 130 may change at least a part of an autonomous driving algorithm provided from the second external electronic device 120 or an autonomous driving function according to an artificial neural network based on the reception of the signal. For example, the terminal 130 may change the autonomous driving path of the vehicle to move the vehicle including the terminal 130 . For example, when the information indicates that there is an external object that is not included in the pre-stored electronic map for autonomous driving, the terminal 130 is configured to avoid a collision with the external object or to avoid a collision with the external object. A control command for changing the autonomous driving path may be generated to bypass the . However, it is not limited thereto.

Also, although it has been described that the first external electronic device 110 performs video streaming to the electronic device 101 in the above-described exemplary embodiment, the present invention is not limited thereto. For example, if the first external electronic device 110 further includes non-visual sensors such as Lidar and RADAR in addition to visual sensors such as an image sensor, Lidar and RADAR in addition to video streaming It may stream to the electronic device 101 including the sensor information obtained by . Additionally, when the first external electronic device 110 streams the video and the sensor information to the electronic device 101 , location information obtained by acquiring the video and the sensor information may be included. In an embodiment, the terminal 130 may identify that security of the traffic infrastructure is necessary based on receiving the signal, and execute at least one operation based on the identification. For example, when the terminal 130 is a device included in a vehicle for repairing traffic infrastructure, the terminal 130 sends a notification indicating where the traffic infrastructure requiring repair is located to the terminal 130 ) can be displayed through the display. However, it is not limited thereto.

In an embodiment, the signal 235 may include information for changing the UI displayed on the terminal 130 . An example of the UI displayed by the terminal 130 in response to the reception of the signal 235 will be described in detail with reference to FIG. 5 .

3 is a flowchart of an electronic device according to an exemplary embodiment. The electronic device of FIG. 3 may include the electronic device 101 of FIGS. 1 to 2 . At least one of the operations of FIG. 3 may be performed by the electronic device 101 and/or the processor 103 of FIG. 1 .

Referring to FIG. 3 , in operation 310 , the electronic device according to an embodiment may receive a video from the first external electronic device (eg, the first external electronic device 110 of FIGS. 1 to 2 ). . Operation 310 may be performed similarly to operation 205 of FIG. 2 .

Referring to FIG. 3 , in operation 320 , the electronic device according to an embodiment may determine whether or not a signal notifying entry into a specified state is received from the first external electronic device. The signal may include, for example, the signal 210 of FIG. 2 . For example, the signal may include information related to an accident identified in the first external electronic device.

If a signal for notifying entry into the specified state is not received from the first external electronic device (320 - NO), the electronic device may maintain a state of receiving a video based on operation 310 . When a signal for notifying the entry to the specified state is received from the first external electronic device ( 320 - Yes), in operation 330 , the electronic device may store the received video. Operation 330 may be performed similarly to operation 215 of FIG. 2 .

Referring to FIG. 3 , in operation 340 , the electronic device according to an embodiment may identify information related to a stored video. Operation 340 may be performed similarly to operation 220 of FIG. 2 .

In response to the identification of the video-related information, in operation 350 , the electronic device according to an embodiment is identified as a second external electronic device (eg, the second external electronic device 120 of FIGS. 1 to 2 ). It is possible to transmit at least a portion of the received information. Information transmitted from the electronic device to the second external electronic device 120 may correspond to, for example, information included in the signal 225 of FIG. 2 . Upon receiving the information in operation 350 , the second external electronic device may broadcast information related to the entry into the specified state of the first external electronic device to one or more terminals.

4 is a flowchart of a terminal processing information provided from an electronic device according to an exemplary embodiment. The information may be provided to the terminal 130 through the electronic device 101 and the second external electronic device 120 of FIGS. 1 to 2 . The terminal of FIG. 4 may include the terminal 130 of FIGS. 1 to 2 . The operation of FIG. 4 may be performed by the terminal 130 and/or the processor 131 of FIGS. 1 to 2 .

Referring to FIG. 4 , in operation 410 , the terminal according to an embodiment may control at least one display based on a specified application. The designated application may include, for example, a navigation application. The at least one display may include a built-in display (eg, a touch screen) included in the terminal. The at least one display may include an external display connected to the terminal based on an interface such as HDMI, USB-C, D-SUB and/or WIDI. Operation 410 may be performed, for example, in a state of guiding a route based on a navigation application.

In a state of controlling at least one display based on operation 410 , in operation 420 , the terminal according to an embodiment performs the second external electronic device (eg, the second external electronic device (eg, the second external electronic device of FIGS. 1 to 2 ) 120)), it may be determined whether information related to the first external electronic device (eg, the first external electronic device 110 of FIGS. 1 to 2 ) has been received. The information is, for example, a signal included in the signal 235 of FIG. 2 and may include information related to an accident identified in the first external electronic device.

In a state in which information related to the first external electronic device is not received ( 420 - No), the terminal may maintain control of at least one display based on operation 410 . In response to receiving the information related to the first external electronic device (420 - Yes), in operation 430 , the terminal according to an embodiment may identify whether an external display is connected to the terminal.

In a state in which the external display is not connected to the terminal ( 430 - NO), in operation 440 , the terminal may display a first visual element related to the first external electronic device on the built-in display. The first visual element may include at least one of a text, an icon, an image, and an animation for notifying an accident identified by the first external electronic device. In a state in which the external display is connected to the terminal (430 - Yes), in operation 450, the terminal may display the first visual element related to the first external electronic device on the external display.

In a state in which the first visual element is displayed based on operation 440 , in operation 470 , the terminal according to an embodiment may identify a user input related to the first visual element from the user. The user input may include a user's gesture of touching and/or clicking the first visual element. In response to identifying the user input associated with the first visual element (470-Yes), in operation 460 , the terminal may display the second visual element associated with the first visual element on the at least one display. The second visual element may include information related to an accident identified in the first external electronic device.

Referring to FIG. 4 , displaying the second visual element based on operation 460 may be performed differently depending on whether an external display is connected. For example, when an external display is connected to the terminal, the terminal displays the first visual element on the external display based on the operation 450 , and based on the operation 460 , an external display and/or a built-in display At least one of the second visual elements may be displayed. For example, when the external display is not connected to the terminal, the terminal may display the second visual element in the built-in display while the user input is performed based on the operations 440 and 470 .

5 is a diagram illustrating an example of a UI output by the terminal 130 in response to information provided from an electronic device according to an embodiment. The electronic device may include the electronic device 101 of FIGS. 1 to 2 . The terminal may include the terminal 130 of FIGS. 1 to 2 . The UI of FIG. 5 may be displayed based on at least one of the operations of FIG. 4 .

Referring to FIG. 5 , an external display 510 connectable to the terminal 130 is illustrated. The external display 510 may include, for example, a display disposed in a vehicle. The external display 510 may correspond to the external display of FIG. 4 .

Referring to FIG. 5 , the terminal 130 may display a screen 520 based on a designated application, such as a navigation application, on the built-in display and/or the external display 510 of the terminal 130 . For example, in a state in which the terminal 130 and the external display 510 are connected through a wired network and/or a wireless network, the terminal 130 may display the screen 520 on the external display 510 . For another example, in a state in which the terminal 130 and the external display 510 are not connected, the terminal 130 may display the screen 520 in the built-in display.

In response to an accident occurring in a vehicle that is distinct from the terminal 130 and/or the external display 510 , as described with reference to FIGS. 1 to 4 , information related to the accident may be provided to the terminal 130 . For example, a first external electronic device, such as an image acquisition device for a vehicle disposed in a vehicle in which an accident has occurred, notifies the occurrence of the accident to the electronic device according to an embodiment, and the electronic device navigates information related to the accident. The data may be transmitted to the second external electronic device corresponding to the server. The second external electronic device may transmit information related to the accident to the terminal 130 moving along a path adjacent to and/or overlapping the location where the accident occurred.

In response to the reception of the information, the terminal 130 may display information related to the accident in the screen 520 . For example, the terminal 130 may display the first visual element 522 notifying the accident. The first visual element 522 may correspond to the first visual element of operations 440 and 450 of FIG. 4 . Referring to FIG. 5 , the first visual element 522 may have coordinates corresponding to the location where the accident occurred in the map image on the screen 520 , and information related to the accident (eg, the accident occurred) time) may be included. The terminal 130 may further display, on the screen 520 , a message 524 notifying the change of the route according to the occurrence of an accident.

When the user touches and/or clicks the first visual element 522 , the terminal 130 may further display a screen 530 including detailed information related to the accident. Referring to the screen 530 , the terminal 130 may further display the type of accident, the lane closed by the accident, a CCTV screen related to the accident, and/or a visual element related to a video reported by another user. Although an embodiment in which the terminal 130 displays the screen 530 on the built-in display is illustrated, the terminal 130 may display the screen 530 on at least a portion of the external display 510 .

The electronic device according to various embodiments may process a video being streamed from a first external electronic device, such as an image acquisition device for a vehicle, based on a specified state (eg, a traffic accident) identified in the first external electronic device. . For example, when receiving a notification related to a specified state from the first external electronic device, the electronic device obtains information related to the specified state, and then transmits the obtained information to a second external electronic device such as a navigation server or an autonomous driving server. It can be broadcast to one or more terminals (eg, the terminal 130 of FIG. 5 ) through . The terminal may provide information related to the designated state to the user of the terminal in response to the broadcasting. The information may be used to change the navigation route being provided by the terminal or change the autonomous driving route.

As described above, an electronic device according to various embodiments includes a communication circuit, a memory, and at least one processor operably coupled to the communication circuit and the memory, wherein the The memory stores a plurality of instructions, and when the plurality of instructions are executed by the at least one processor, the at least one processor uses the communication circuit to cause a first external electronic device to be distinguished from the electronic device. Receive a video from a device, and in a state of receiving the video, receive a first signal for notifying entry to a specified state from the first external electronic device, and in response to receiving the first signal, the received video and control to transmit a second signal including at least a portion of the information to a second external electronic device distinguished from the first external electronic device in response to the identification of the information.

In an embodiment, the video is associated with the second external electronic device, and an electronic map displayed on a third external electronic device located in a vehicle including the first external electronic device is displayed on the first external electronic device. It may be transmitted from the first external electronic device to the electronic device based on the third external electronic device identifying that it does not correspond to the video acquired by the .

In an embodiment, in the video, an object recognized by a third external electronic device connected to the second external electronic device and located in a vehicle including the first external electronic device is displayed by the first external electronic device. The data may be transmitted from the first external electronic device to the electronic device based on the third external electronic device identifying that it does not correspond to the object detected from the acquired video.

In an embodiment, the video may be transmitted from the first external electronic device to the electronic device under a condition that a driving mode of a vehicle including the electronic device is changed from an autonomous driving mode to a manual driving mode.

In an embodiment, the plurality of instructions, when executed by the at least one processor, cause the at least one processor to store the video based on reception of the first signal, and not receive the first signal. Based on , it is possible to control the video to be discarded after a specified time has elapsed. In an embodiment, the length of the time interval for maintaining storing the video may be changed based on the type of the first signal.

In an embodiment, the plurality of instructions, when executed by the at least one processor, the at least one processor recognizes the video using a neural network based on the first signal, thereby It can be controlled to identify the relevant information.

In an embodiment, the second signal may cause the second external electronic device to identify one or more terminals related to at least a part of the information.

As described above, the terminal according to various embodiments includes a built-in display, a communication circuit, a memory and at least one processor operably coupled to the display, the communication circuit and the memory, and the The memory stores a plurality of instructions, and when the plurality of instructions are executed by the at least one processor, the at least one processor communicates with a first external electronic device based on a specified application, and 1 Receive, from an external electronic device, information related to a second external electronic device that is distinguished from the first external electronic device, and in response to receiving the information, identify an external display that is distinct from the built-in display; displays a first visual element related to the information on the external display in a first state connected to the terminal, and displays the first visual element on the built-in display in a second state distinct from the first state a second visual element associated with the first visual element and including at least a portion of the information, in response to identification of a user input associated with the first visual element, in the displayed state of the first visual element control to be displayed, and the second visual element may be displayed on the built-in display in the first state and displayed on the external display in the second state.

In an embodiment, the first visual element may be displayed to provide information related to an accident, and may be displayed as an overlap on an electronic map provided through the designated application. In an embodiment, the first visual element may be displayed together with a message for notifying the accident.

In an embodiment, the second visual element may include detailed information about the accident indicated by the first visual element. In an embodiment, the plurality of instructions, when executed by the at least one processor, cause the at least one processor to display the first visual element superimposed on the electronic map in response to the identification of the user input. Stop displaying and display the second visual element, it is possible to control.

In one embodiment. The visual element may be displayed in association with information on the time of occurrence of the accident.

6 is a block diagram of an image acquisition apparatus 601 for a vehicle according to various embodiments of the present disclosure. The vehicle image acquisition device 601 may include at least one of a processor 602 , a memory 603 , a communication circuit 604 , a sensor 605 , and a battery 606 . The vehicle image acquisition device 601 may be connected to at least one of the vehicle 610 and the terminal 620 through a wired network and/or a wireless network. According to embodiments, the vehicle image acquisition device 601 and the terminal 620 may be integrally implemented. According to embodiments, the vehicle image acquisition device 601 and the vehicle 610 may be integrally implemented. According to embodiments, the vehicle image acquisition device 601 , the vehicle 610 , and the terminal 620 may be integrally implemented.

Hereinafter, functions performed by the vehicle image acquisition apparatus 601 according to various embodiments will be described.

1. Battery discharge warning in vehicle 610

In the vehicle image acquisition device 601 operating while the vehicle 610 is parked, since power is provided to the vehicle image acquisition device 601 in a state where the engine of the vehicle 610 is stopped, the battery in the vehicle 610 is can reduce the average lifespan of For example, when the vehicle 610 is parked, since there are few electronic devices in the vehicle 610 that operate as a load, the battery management system in the vehicle 610 is When the vehicle 610 is parked, it may be difficult to obtain information on a meaningful discharge characteristic of the battery in the vehicle 610 (eg, a change in at least one of a voltage or a current over time). In other words, with respect to the parking of the vehicle 610 , it may be difficult for the battery management system in the vehicle 610 to identify when the battery of the vehicle 610 needs to be replaced. Since the vehicle image acquisition device 601 is a device that consumes a relatively large amount of power in a state in which the generator in the vehicle 610 is off (eg, while the vehicle 610 is parked), the vehicle image acquisition device 601 is, A discharge characteristic of the battery of the vehicle 610 may be monitored. The vehicle image acquisition device 601 records the result of the monitoring, so that direct discharge characteristics (eg, changes in at least one of voltage or current over time) or indirect discharge characteristics (eg, the vehicle 610 up to a specified voltage) ) of the time when the voltage of the battery drops or the time during which recording is continued while the vehicle 610 is parked) may be acquired. The vehicle image acquisition apparatus 601 may determine when replacement of the battery of the vehicle 610 is required based on the acquired data. For example, the vehicle image acquisition device 601 identifies whether the recording duration while the vehicle 610 is parked is less than or equal to a reference time, and the duration of the recording duration while the vehicle 610 is parked is the Based on the identification that is less than a reference time, it is possible to determine when replacement of the battery is required. As another example, the vehicle image acquisition device 601 may identify a trend of a recording duration while the vehicle 610 is parked. For example, in the vehicle image acquisition device 601 , the recording continues during the first parking time of the vehicle 610 and the recording continues during the second parking of the vehicle 610 after the first parking of the vehicle 610 . It is possible to determine when a replacement of the battery is required based on identifying whether a difference between the times when the battery needs to be replaced is greater than a threshold, and based on identifying whether the difference is greater than the threshold.

In one embodiment, in order to identify the time for which recording is continued while the vehicle 610 is parked, the timing of starting to acquire an image through the vehicle image acquisition device 601 while parking is that the voltage provided from the battery It may be the timing of reaching a specified voltage. In an embodiment, the specified voltage may be changed according to a state of charge of the battery. For example, the specified voltage may be 12 (V) under the condition that the battery is fully charged and the specified voltage may be 11.5 (V) under the condition that the battery is not fully charged. However, it is not limited thereto.

The vehicle image acquisition device 601 according to an embodiment monitors a moving average of the recording time while the vehicle 610 is parked, and when the slope at which the recording time decreases exceeds a certain threshold, the life of the battery of the vehicle 610 is exceeded. You can warn that your State of Health (SoH) has expired. For example, deterioration of the performance of the battery of the vehicle 610 (eg, expiration of the life of the battery of the vehicle 610 ) causes a decrease in the capacity of the battery of the vehicle 610 , It is possible to reduce the recording time of the vehicle image acquisition device 601 during parking. In an embodiment, the predetermined threshold may be changed according to the specified voltage. For example, the predetermined threshold when the specified voltage is 12 (V) may be different from the predetermined threshold when the specified voltage is 11.5 (V). However, it is not limited thereto.

In an embodiment, when monitoring the moving average of the recording time using 601, the moving average of the recording performed through the vehicle image acquisition device 601 during the most recent parking of the vehicle 610 is the vehicle ( 610) may be used as data for estimating how long the recording to be performed through the vehicle image acquisition device 601 will last during the next parking. The vehicle image acquisition device 601 may be configured to display the vehicle 610 through at least one of a display in the vehicle 610 , a display of the terminal 620 , or a display of the vehicle image acquisition device 601 based on the data. An estimate of the duration of recording to be performed through the vehicle image acquisition device 601 may be displayed at the next parking time. When the estimated duration of the recording is displayed through the display in the vehicle 610 or the display of the terminal 620 , the vehicle image acquisition device 601 receives the estimated duration of the recording through the communication circuit 604 . Information for display can be transmitted. For example, the information may be transmitted from the vehicle image acquisition device 601 to the vehicle 610 or the terminal 620 in response to detecting that the engine of the vehicle 610 is driven. However, it is not limited thereto.

Meanwhile, the vehicle image acquisition device 601 may identify whether the estimated duration of the recording is less than a reference value. Based on identifying that the estimated duration of the recording is less than the reference value, the vehicle image acquisition device 601 executes an operation for providing a notification indicating that the battery replacement time of the vehicle 610 is coming. can For example, the vehicle image acquisition device 601 may output the notification through a speaker of the vehicle image acquisition device 601 or display the notification through the display of the vehicle image acquisition device 601 . As another example, the vehicle image acquisition device 601 may transmit information for displaying the notification or information for outputting the notification as an audio signal to at least one of the vehicle 610 and the terminal 620 . . At least one of the vehicle 610 and the terminal 620 may display or output the notification based on the information received from the vehicle image acquisition device 601 .

In one embodiment, the vehicle image acquisition device 601 determines the duration of the recording until the recording of the vehicle image acquisition device 601 is stopped while the vehicle 610 is parked due to the low voltage of the battery in the vehicle 610 . Alternatively, a change trend in the voltage of the battery in the vehicle 610 may be recorded from when the recording is started to when the recording is finished. For example, in order to improve the accuracy of the recording related to the performance of the battery in the vehicle 610 , the vehicle image acquisition device 601 may determine the previous charging time (the time the vehicle's engine is turned on) or the parking time of the vehicle 610 . It is possible to store the voltage at the time the recording started. The vehicle image acquisition device 601 processes the stored record, and a time period related to the recording (eg, a recording time while the vehicle 610 is parked or the voltage of the battery of the vehicle 610 while the vehicle 610 is parked) It is possible to extract the state of the battery in the vehicle 610 for the time elapsed until the voltage falls below the specified voltage). The vehicle image acquisition device 601 includes an extraction value (eg, a short-term extraction value) obtained during parking of the most recent vehicle 610 and a recording time during parking of the most recent vehicle 610 for a specified period of time. By comparing the ratios (eg, long-term extraction values/short-term extraction values) of the extracted values (eg, long-term extraction values), if the ratio exceeds a reference value, it is determined that the performance of the battery of the vehicle 610 is deteriorated, or can recognize When the ratio falls below the reference value and the extracted value increases significantly after the ratio falls below the reference value, the vehicle image acquisition apparatus 601 may determine that the battery of the vehicle 610 has been replaced.

In an embodiment, the vehicle image acquisition device 601 uses the communication circuit 604 to provide a long-term extracted value or a short-term extracted value to the server directly or to the server through the vehicle 610 or the terminal 620 . , or a value indicating whether replacement of the battery in the vehicle 610 is required may be transmitted.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

In an embodiment, the vehicle image acquisition apparatus 601 may identify whether the battery is to be replaced based on the change trend. For example, when the duration of acquiring an image by using the vehicle image acquisition device 601 while parking is increased, the vehicle image acquisition device 601 may identify that the battery is to be replaced. However, it is not limited thereto.

2. Continuous recording prohibited mode (Incognito Mode)

In the continuous recording prohibition mode described below, a designated time section around the timing at which an impact recognized as being related to an accident while driving the vehicle 610 is detected, and a designated time around the timing at which the sudden acceleration event of the vehicle 610 is detected Only partial images for at least one section or a designated time section around the timing at which the rapid deceleration event of the vehicle 610 is detected are stored, and the remaining partial images except for the partial images among the images recorded while the vehicle 610 is driving It may mean a mode for discarding . When the location in which the vehicle 610 is located is changed (eg, when movement between countries occurs), the vehicle image acquisition device 601 may inquire whether or not to activate the always-on recording prohibition mode without a user input.

When the vehicle 610 is driven, the vehicle image acquisition apparatus 601 according to an embodiment may ask the user whether to store or delete the (always) recorded image while the vehicle 610 is operating. For example, the vehicle image acquisition device 601 may display a message for inquiring whether to store or delete a recorded image while the vehicle 610 is driving, or output the message as a voice signal. Alternatively, the vehicle image acquisition device 601 may store only the event image and not the normally recorded image. In order to protect personal privacy, improve storage memory lifespan, and avoid specific national privacy regulations, the vehicle image acquisition device 601 may discard the always-recorded image without storing it or inquire whether to store the always-recorded image.

The vehicle image acquisition apparatus 601 may store the data frames output from the image and audio codecs in the memory 603 for a predetermined period of time. In the memory 603 , an image acquired through the vehicle image acquisition device 601 may be stored based on a FIFO (first input first output) structure that extracts old data frames and stores the latest ones. The vehicle image acquisition device 601 may obtain data frames output from the video and audio codecs from the buffer, mux them if necessary, and store the data frames in a file. Different files for each event type may be stored in different parts of the memory 603 . The in-vehicle image acquisition device 601 controls to bypass the recording unit in which the file for constant recording is generated while the regular recording prohibition mode is activated, or the file for the constant recording by a user input in the deletion mode after recording. If deletion of is selected, the file can be found on the SDCard and the file can be deleted. The vehicle image acquisition device 601 may output whether the constant recording prohibition mode is used. The in-vehicle image acquisition device 601 may further include an input device that receives an input from the user whether to use the continuous recording prohibition mode. The input for inquiring whether to use the regular recording prohibition mode may be input through the vehicle image acquisition device 601 , the vehicle 610 , or the terminal 620 . .

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

3. Implementation of low-power parking recording by turning off unnecessary cameras

It is possible to reduce current consumption by turning off a camera adjacent to a wall in parking recording among one or more cameras included in the vehicle image acquisition device 601 . A camera adjacent to the wall may be identified using a sensor (radar, ultrasound) of the vehicle 610 (or built into the camera) or through camera image recognition. For example, from the image before entering the parking mode, the vehicle image acquisition device 601 may distinguish a wall. In the 2ch vehicle image acquisition device 601 with a front and rear camera, 30-40% of current consumption can be saved by turning off one camera, which can take a long time for parking recording.

The vehicle image acquisition device 601 recognizes an image (photo) taken in the vehicle traveling direction just before parking or using data acquired from an ultrasonic sensor or a radar sensor in the vehicle 610, the wall is located within a specified distance from the exterior of the vehicle. It can be determined whether The vehicle image acquisition device 601 may recognize whether there is a flat obstacle close to a predetermined level or more in the vehicle traveling direction immediately before parking using an ultrasonic sensor or a radar sensor (of the vehicle). The vehicle image acquisition device 601 may control to cut off the power of the camera on the side determined that there is a wall in the recognition unit in the parking mode and not store the image of the corresponding camera.

In an embodiment, the ultrasonic sensor in the vehicle 610 may be connected to the vehicle image acquisition device 601 through intra-vehicle communication (eg, CAN communication, LIN communication, etc.). The vehicle image acquisition device 601 receives a signal for instructing that an external object approaches the vehicle 610 from the ultrasonic sensor activated while the vehicle 610 is parked, and based on the reception of the signal, the vehicle ( The camera of the image acquisition device 601 for a vehicle in which the power is cut off during parking of the 610 may be driven. Through such driving of the camera, the vehicle image acquisition device 601 may acquire an image of an accident caused while the power provided to the camera is cut off. Meanwhile, the ultrasonic sensor may be operatively coupled to an auxiliary processor of the vehicle 610 . When the generator of the vehicle 610 is turned off, the auxiliary processor obtains low power from the battery of the vehicle 610 unlike the main processor in the vehicle 610 in which power provided from the battery of the vehicle 610 is cut off. can Due to the use of the auxiliary processor, the amount of power consumed by the battery of the vehicle 610 may be reduced while the generator of the vehicle 610 is turned off. According to embodiments, when there are a plurality of ultrasonic sensors connected to the auxiliary processor, the auxiliary processor obtains information about a camera whose power is cut off from the vehicle image acquisition device 601, and based on the information, At least one ultrasonic sensor related to a field of view (FOV) of the camera whose power is cut off may be identified among the plurality of ultrasonic sensors, and only the identified at least one ultrasonic sensor may be activated.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

4. The language of the vehicle image acquisition device 601, the user's country, local time, and summer time synchronization with the smartphone

The vehicle image acquisition device 601 may accurately adjust the accident time in the vehicle image acquisition device through tethering or information received from the terminal 620 connected through BT. For example, the vehicle image acquisition device 601 may identify the presence or absence of summer time in a corresponding country or region, different from the time received through GPS.

For example, the vehicle image acquisition device 601, from the terminal 620, the country in which the terminal 620 is currently located, the language of the country, the local time of the country (summer time is applied), the terminal 620 position of the terminal 620 (or the vehicle image acquisition device 601, the vehicle 610) moving speed, or the moving direction of the terminal 620 (or the vehicle image acquisition device 601, the vehicle 610) ( or heading) may be received. The vehicle image acquisition device 601 may configure header information of an image acquired through the vehicle image acquisition device 601 or change information used in the vehicle image acquisition device 601 based on the data. . Meanwhile, one or more application programs may be stored in order for the terminal 620 to transmit the data to the vehicle image acquisition device 601 .

This is, despite the fact that the vehicle image acquisition device 601 is used as evidence in an accident, the current vehicle image acquisition device only acquires the longitude and latitude and the GMT period according to the GPS measurement. This may be because the data for calculating the local time of the designated place is not obtained. On the other hand, when data on the moving speed of the terminal 620 and/or the moving direction (or heading) of the terminal 620 is acquired, the vehicle image acquisition device 601 has a built-in GPS or the terminal 620 or the vehicle ( 610) may be connected.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

5. Important parking shock notification (IMU related)

The vehicle image acquisition device 601 detects the impact of getting on and off (occurring in the longitudinal direction) before and after entering the parking mode and before and after the time when the engine is started to change to the regular mode after riding. Only parking shocks that are likely to damage the actual vehicle can be notified to the user.

For example, when the engine of the parked vehicle 610 is driven, a general vehicle image acquisition apparatus may provide one to three images acquired according to the impact of getting on and off. Such an image may lengthen a time for searching for at least one image including a scene causing damage to a vehicle among a plurality of images recorded while parking. The vehicle image acquisition device 601 deletes an image acquired according to the user's impact when getting on and off, through image analysis, or a designated time period before and after the stopped timing of the vehicle 610 (or before and after the ignition is turned off) At least one of the images acquired in , or images acquired in a specified time period before and after the driving start timing of the vehicle 610 (or before and after the ignition is turned on) may be deleted. The vehicle image acquisition apparatus 601 may prevent an unnecessary image from becoming an unnecessary image through such deletion and may reduce a search time for a meaningful image.

In one embodiment, the vehicle image acquisition device 601 indicates the direction of gravity vector obtained through the acceleration sensor of the vehicle 610, the terminal 620, or the vehicle image acquisition device 601 just before entering the parking mode. Through this data, data for a Norm vector perpendicular to the ground on which the vehicle 610 is parked may be acquired.

On the other hand, in the vehicle image acquisition device 601 , at least some of the sound signals related to the vehicle 610 acquired through the vehicle 610 , the terminal 620 , or the microphone of the vehicle image acquisition device 601 are included in the vehicle 610 . ) can be identified whether it is related to the impact sound of the door.

For example, the vehicle image acquisition device 601 may include data for the Norm vector, identification data indicating that at least some of the sound signals are related to the impact sound of a door of the vehicle 610, and data for the Norm vector Based on at least one of the acquired time, the state of the vehicle 610 , the state of the power supply of the vehicle 610 , or the state of the power supply of the vehicle image acquisition device 601 , It can be identified whether the impact is an impact generated when the parked vehicle 610 gets on and off. In addition, the vehicle image acquisition device 601 is used for the identification in order to improve the performance of identifying whether the impact caused in the parked vehicle 610 is an impact generated when the parked vehicle 610 gets on and off. Data (data about the Norm vector, the identification data, etc.) and information about a result of the identification may be provided to the neural network. The neural network may be trained using data used for the identification and information about a result of the identification. Through this learning, the vehicle image acquisition device 601 may improve the identification performance.

Meanwhile, the vehicle image acquisition device 601 may monitor whether at least some of the images acquired while the vehicle 610 is parked are selected by a user input for private playback. For example, for the monitoring, the vehicle image acquisition device 601 determines whether at least some of the images are reproduced, whether at least some of the images are downloaded, and whether at least some of the images are copied to the library. At least one of whether or not or obtaining a response result value for the validity of at least some of the images may be performed. The vehicle image acquisition device 601 uses the data obtained based on the execution of the monitoring to identify whether the impact caused in the parked vehicle 610 is an impact generated when the parked vehicle 610 gets on and off. In the resulting data, it can be added as a label. The label may be used to improve the performance of identifying whether an impact caused in the parked vehicle 610 is an impact generated when the parked vehicle 610 gets on and off. For example, the label may be referenced by the neural network.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

6. Parking impact direction notification (IMU related)

The vehicle image acquisition device 601 may display the largest impact direction among the parking impacts with an LED in the front (windshield) direction of the vehicle image acquisition device 601 . Alternatively, the vehicle image acquisition device 601 may output the parking impact direction and intensity by using the terminal 620 . For example, the vehicle image acquisition device 601 may operate the sensor 605 (eg, an IMU sensor) at 1000 Hz or more. For example, the vehicle image acquisition device 601 may switch the frequency at which the sensor 605 operates between a first state operating in a range of 10 to 50 Hz and a second state operating at 1000 Hz or higher. When the vehicle image acquisition device 601 informs the user of the impact direction before boarding the vehicle, the user can quickly check the damaged part of the vehicle before boarding.

In one embodiment, the vehicle image acquisition device 601 may acquire the data for the Norm vector in the same or similar manner as described in the table of contents “5. Important Parking Impact Notification (IMU Related)”. The vehicle image acquisition device 601 provides data for the Norm vector and an acceleration sensor (vehicle image) based on the assumption that the front camera of the vehicle image acquisition device 601 is looking in a direction approximate to the front of the vehicle 610 . Using the coordinate system of the acquisition device 601, the vehicle 610, or the terminal 620), the unit vector for the front direction of the vehicle 610 that is parallel to the ground on which the vehicle 610 is parked and has a unit length data can be calculated.

The vehicle image acquisition device 601 determines a coordinate system based on the vehicle 610 when the vehicle 610 is parked based on the data on the Norm vector and the data on the unit vector for the front direction of the vehicle 610, , by processing the at least one value obtained through the acceleration sensor during parking of the vehicle 610 and the time at which the at least one value is obtained based on the determined coordinate system, the initial direction of the impact of the vehicle 610 during parking is determined You can decide which direction to take. The vehicle image acquisition device 601 records the result data of the determination during the parking of the vehicle 610 , and based on the result data, the largest impact among the plurality of impacts caused during the parking of the vehicle 610 , based on the result data. You can decide which direction to take.

The vehicle image acquisition device 601 may provide information on the direction of the greatest impact. For example, the vehicle image acquisition device 601 may transmit the information to a display related to the vehicle image acquisition device 601 (a display of the vehicle 610 , a display of the vehicle image acquisition device 601 , or a terminal 620 ). can be displayed on the display). As another example, the vehicle image acquisition device 601 may output the information as an audio signal through a speaker associated with the vehicle image acquisition device 601 . However, it is not limited thereto.

Some of the operations of the vehicle image obtaining apparatus 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server instead of the vehicle image obtaining apparatus 601 .

7.Towing notification (IMU related)

Unlike a simple shock, the vehicle image acquisition device 601 can detect and notify the towing because the angle of the parked vehicle changes from when the first towing is started. For example, the vehicle image acquisition device 601 may be implemented in the same or similar manner to the method described in the description of the table of contents "6. Parking impact direction notification (IMU-related)". Through this execution, the vehicle image acquisition device 601 may distinguish an impact event while the vehicle 610 is parked and a traction event of the vehicle 610 unlike a general vehicle image acquisition device.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

8. Remaining passengers notification:

When the vehicle image acquisition device 601 includes an indoor camera, when there is a movement indoors while parking, the vehicle image acquisition device 601 may detect and notify the vehicle owner. For example, it is possible to detect movement in the room by children or companion animals. The method of determining whether there is a movement in the interior may include, for example, a method of confirming whether there is a motion vector in the vehicle, and/or a method of confirming through recognition. In this case, it may be more cost effective than using ultrasound, radar, or UWB (usually millimeter wave). The vehicle image acquisition device 601 may transmit a picture related to indoor movement using a connected service based on the communication circuit 604 .

In an embodiment, the vehicle image acquisition device 601 includes an indoor camera (a vehicle image acquisition device 601 camera, a vehicle 610 camera, and/or a vehicle 610 related to the vehicle image acquisition device 601 ). Alternatively, an image inside the vehicle 610 may be acquired using the camera of the terminal 620). The vehicle image acquisition device 601 may analyze the image. The vehicle image acquisition device 601 may detect a motion in the acquired image. The vehicle image acquisition device 601 may determine whether the detected motion is caused by a shadow. The vehicle image acquisition apparatus 601 may exclude a partial image corresponding to a window area of the vehicle 610 from the acquired image.

The vehicle image acquisition device 601 may acquire an acoustic signal within the vehicle 610 through a microphone associated with the vehicle image acquisition device 601 while acquiring the image. The vehicle image acquisition apparatus 601 may analyze the generation time of the sound signal, the intensity of the sound signal, and the type of the sound signal.

The vehicle image acquisition device 601 is based on at least one of the image acquired during parking except for the partial image, data on the detected motion, data as a result of analysis of the image, or data on the sound signal. , it is possible to determine whether there is an object moving indoors while the vehicle 610 is parked. The vehicle image acquisition device 601 may, if necessary, provide information on the result of the determination to an electronic device related to the vehicle image acquisition device 601 (eg, the vehicle 610 , the terminal 620 , or an external server). can send

As described above, the vehicle image acquisition apparatus 601 may determine the indoor movement of the vehicle 610 by using a motion vector and image analysis to distinguish it from a shadow or a light. Also, the vehicle image acquisition apparatus 601 may further use the acquired sound signal to improve the accuracy of the determination.

At least some of the operations of the vehicle image acquisition device 601 described with respect to the present embodiment may be executed by the terminal 620 , the vehicle 610 , or an external server rather than the vehicle image acquisition device 601 .

9. Low voltage auto start charging

When the voltage of the battery of the vehicle 610 falls below a predetermined value, the vehicle image acquisition device 601 may start the vehicle 610 with reference to the fuel gauge or start the vehicle 610 only a specified number of times. there is. The vehicle image acquisition device 601 charges the battery of the vehicle 610 for a specified time (eg, 30 minutes to 1 hour), then lengthens the parking recording period and prevents the battery of the vehicle 610 from being discharged. can

Hereinafter, one or more functions performed by the vehicle image acquisition apparatus 601 according to an embodiment by communicating with the vehicle 610 and/or the terminal 620 will be described.

1. Update traffic information using photo information

When the ratio of the vehicle average speed (gps or vehicle speed) and the current road average speed on the received real-time traffic information exceeds a certain range from 1, the Traffic information Speed I > Threshold) Send one or several (taken at regular time intervals) photos of the current camera to the server to determine if the road condition is an accident, congestion, or leak (construction cone), etc. I can update the map.

2.. Vehicle image acquisition device 601 processes IMU sensor data based on vehicle speed (or gps speed) , it is possible to obtain the direction in which gravity acts.The vehicle image acquisition device 601 is based on the acceleration value in the direction of gravity obtained from the direction of gravity and the secondary rate of change of the vehicle speed, the inclination rate of the road, the pothole, the speed bump, It is possible to determine the amount of horizontal acceleration on the left and right and the evenness of the road surface, which may be information that cannot be known from GPS. It is possible to detect a safe distance The vehicle image acquisition device 601 can accurately search and guide by identifying the road surface, knowing whether a junction or a ramp has entered. The vehicle image acquisition device 601 can also update a map.

3. Pedestrian caution warning in child protection area

The vehicle image acquisition device 601 may output a caution warning to recognize a pedestrian behind an object (a stopped vehicle, etc.) that is obscured through a camera (radiator grill, front bumper, front hood) installed in front of the driver's point of view. The vehicle image acquisition device 601 may effectively recognize a pedestrian and a child based on, for example, face recognition. The vehicle image acquisition device 601 may identify a distance between a vehicle and a pedestrian by calculating a change rate with time of a face size (a horizontal pixel length or a vertical pixel length). For example, when a face having a size greater than or equal to a predetermined pixel is identified, the vehicle image acquisition device 601 may notify the driver of the risk of a traffic accident.

Traffic accident emergency notification system (NG e-Call, Next Generation e-Call) enables two-way VoIP voice calls and two-way data transmission of 200 bytes or more using a telecommunication-based IP network environment such as LTE/5G. It may refer to a driver rescue system through emergency notification in the event of a traffic accident such as a collision.

The traffic accident emergency notification system terminal (NG e-Call IVS, NG e-Call In-Vehicle System) is installed directly inside the vehicle during the manufacturing stage of the finished vehicle and receives various information necessary for accident analysis from the sensors of the vehicle or terminal. It may refer to a device capable of automatically determining an accident by an algorithm or transmitting minimum accident information and extended accident information data by generating a manual trigger, and capable of voice communication with an emergency rescue center.

The traffic accident emergency notification reception center (e-Call Center) receives the minimum accident information, extended information information, and still image information from the terminal of the emergency notification system for traffic accidents, determines whether an accident has occurred, It may refer to a center that plays a role in delivering additional information such as a rescue request and injury severity to secondary related organizations such as the center.

Minimum set of data (MSD) may mean minimum information including basic accident information transmitted by the terminal of the emergency notification system for traffic accidents to the traffic accident emergency notification reception center when an accident occurs.

Extended Set of Data (ESD) is 30 points (20 points before the accident and 10 points after the accident, 10 points after the accident, 20 points before the accident and 10 points after the accident) Example: It refers to the information of 20 seconds before the accident and 10 seconds after the accident) based on the record in seconds, and may include information directly related to the accident and additional information.

7 is a diagram illustrating a data structure of minimum accident information (MSD) generated by an electronic device according to an exemplary embodiment.

Minimum set of data (MSD) may be the first information transmitted from the terminal of the traffic accident emergency notification system to the traffic accident emergency notification collection center by an automatic trigger or manual operation of the driver's SOS button when a vehicle accident or emergency occurs. . The minimum accident information is a message that includes the transmission of the accident occurrence and accident location information, and is divided into essential information and optional information according to the importance of the information.

The minimum accident information is a data message that promptly transmits basic accident information to the traffic accident emergency notification collection center so that the accident can be notified quickly and an emergency rescue response can be made. It means information such as the time of occurrence of an accident and the location of the accident vehicle that should be transmitted. Optional information refers to other information that is selectively transmitted when it is possible to collect from the terminal of the emergency notification system for traffic accidents.

The data arrangement order of the minimum accident information (MSD) may be based on the data structure of the minimum accident information (MSD) defined in the EU 15722:2015 standard to maintain data compatibility with the European Standard. Encoding and encoding of the minimum accident information (MSD) can be based on the ASN.1 UPER method.

Referring to FIG. 7 , recentVehicleLocationN1 and recentVehicleLocationN2 referenced in sequence numbers 9 to 10 may be information recorded as mandatory (M). The information is information for indicating the location where the accident occurred, and may be essentially recorded in order to estimate the location of the accident despite the GPS hearing loss area.

In an embodiment, the minimum accident information may further include data on the timing of measuring the vehicleLocation of the turn 7 and data on the timing of measuring the vehicleDirection of the turn 8. In an embodiment, the data for the timing at which the vehicleLocation of the turn 7 is measured may indicate a difference between the timing indicated by the timeStamp of the turn 6 and the timing at which the vehicleLocation of the turn 7 is measured. In an embodiment, the data for the timing at which the vehicleDirection of the turn 8 is measured may indicate a difference between the timing indicated by the timeStamp of the turn 6 and the timing at which the vehicleDirection of the turn 8 is measured. In an embodiment, the minimum accident information may further include data about the timing of measuring the recentvehicleLocationN1 and recentvehicleLocationN2 of sequence numbers 9 and 10. The data may be data distinguished from the timeStamp of sequence number 6 . The data may be recorded in a separate field in the minimum accident information, or may be recorded in a field for additionalData of sequence number 12.

FIG. 8 is a diagram illustrating data types, restrictions, and definitions of information (VehicleIdentificationNumber) referenced in sequence number 4 of FIG. 7 . The information may include not only the vehicle's unique identifier defined as ISO 3779:2009 Road vehicles, but also the vehicle registration number plate number determined by the laws of the country related to the vehicle. For example, in the case of the Republic of Korea, a vehicle number based on a notice on standards such as a vehicle registration number plate may be included in the information.

As the vehicle identifier based on the statute is further included, the minimum accident information can be related to the product for AM. As the minimum accident information is related to AM products, the number of vehicles to which the minimum accident information is applicable can be increased, and the minimum accident information can be widely used for rescue/rescue. The minimum accident information generated by the electronic device according to an embodiment may be compatible with the ITU NG eCall.

The minimum accident information (hereinafter, MSD) defined through the description of FIG. 7 or FIG. 8 may be utilized as follows. For example, the electronic device included in the vehicle may detect that an accident of the vehicle occurs. For example, the electronic device may be a device embedded in the vehicle. As another example, the electronic device may be a device detachably attached to the vehicle. However, it is not limited thereto.

In an embodiment, the detection of the occurrence of an accident in the vehicle may be performed based on information about values received from at least one sensor in the vehicle. For example, the electronic device may detect the occurrence of the vehicle accident by identifying the occurrence of the vehicle accident based on the information. In another embodiment, the detection of the occurrence of the accident in the vehicle may be performed based on a message for instructing to identify the occurrence of the accident in the vehicle from another electronic device embedded in the vehicle. However, it is not limited thereto.

The electronic device may transmit the MSD to a server related to an automotive emergency service in response to or based on the detection. The server may receive the MSD from the electronic device. The server may, in response to receiving the MSD, identify that the MSD is related to the accident of the vehicle, and start a timer for confirming that the MSD is related to the accident of the vehicle. Also, in response to the reception of the MSD, the server may transmit an acknowledgment (acknowledgement) message for indicating that the MSD is received to the electronic device.

The electronic device may drive a timer for determining that the MSD is related to the accident of the vehicle in response to the reception of the Ack message. While driving the timer, the electronic device may output a guide voice or display a visual guide in order to determine related to the accident of the vehicle. When the electronic device does not receive a response to the guide voice or the visual guide from the user until the driving of the timer expires, the electronic device sends a message for instructing the server to determine that it is related to the accident of the vehicle. can send

On the other hand, the server identifies whether the message is received until the running of the timer of the server expires, and based on identifying that the message is not received until the running of the timer of the server expires Thus, the MSD can be transmitted to another server connected to the server. The MSD transmitted from the server to the other server may be used to provide a rescue service for the occurrence of the accident of the vehicle. For example, the other server receiving the MSD may use the MSD to make a call or send a message to a public safety agent. However, it is not limited thereto.

9 is a block diagram illustrating an autonomous driving system 900 for a vehicle, according to an exemplary embodiment.

The autonomous driving system 900 of the vehicle according to FIG. 9 includes sensors 903 , an image preprocessor 905 , a deep learning network 907 , an artificial intelligence (AI) processor 909 , a vehicle control module 911 , It may be a deep learning network including a network interface 913 and a communication unit 915 . In various embodiments, each element may be connected through various interfaces. For example, sensor data sensed and output by the sensors 903 may be fed to the image preprocessor 905 . The sensor data processed by the image preprocessor 905 may be fed to a deep learning network 907 that runs on the AI processor 909 . The output of the deep learning network 907 run by the AI processor 909 may be fed to the vehicle control module 911 . Intermediate results of the deep learning network 907 running on the AI processor 907 may be fed to the AI processor 909 . In various embodiments, the network interface 913 communicates with the electronic device 101 to transmit autonomous driving path information and/or autonomous driving control commands for autonomous driving of the vehicle to internal block components. In an embodiment, the network interface 913 may be used to transmit sensor data acquired through the sensor(s) 903 to the second external electronic device 120 . In some embodiments, autonomous driving control system 900 may include additional or fewer components as appropriate. For example, in some embodiments, image preprocessor 905 may be an optional component. For another example, a post-processing component (not shown in FIG. 9 ) may be configured to perform post-processing on the output of the deep learning network 907 before the output is provided to the vehicle control module 911 in the autonomous driving control system ( 900) may be included.

In some embodiments, sensors 903 may include one or more sensors. In various embodiments, the sensors 903 may be attached to different locations on the vehicle. The sensors 903 may face one or more different directions. For example, the sensors 903 may be directed toward the front, sides, rear, etc. of the vehicle in forward-facing, rear-facing, side-facing, etc. directions. ), and/or to a loop. In some embodiments, the sensors 903 may be image sensors, such as high dynamic range cameras. In some embodiments, sensors 903 include non-visual sensors. In some embodiments, the sensors 903 include radar (RADAR), Light Detection And Ranging (LiDAR), and/or ultrasonic sensors in addition to an image sensor. In some embodiments, the sensors 903 are not mounted on a vehicle having the vehicle control module 911 . For example, sensors 903 may be included as part of a deep learning system for capturing sensor data and attached to an environment or roadway and/or mounted to surrounding vehicles.

In some embodiments, an image pre-processor 905 may be used to pre-process the sensor data of the sensors 903 . For example, image preprocessor 905 may be used to preprocess sensor data, split sensor data into one or more components, and/or post-process one or more components. In some embodiments, image preprocessor 905 may be a graphics processing unit (GPU), central processing unit (CPU), image signal processor, or specialized image processor. there is. In various embodiments, the image preprocessor 905 may be a tone-mapper processor for processing high dynamic range data. In some embodiments, image preprocessor 1505 may be a component of AI processor 909 .

In some embodiments, the deep learning network 907 may be a deep learning network for implementing control instructions for controlling an autonomous vehicle. For example, the deep learning network 907 may be an artificial neural network such as a convolutional neural network (CNN) trained using sensor data, and the output of the deep learning network 907 is sent to the vehicle control module 911 provided

In some embodiments, the artificial intelligence (AI) processor 909 may be a hardware processor for running the deep learning network 907 . In some embodiments, the AI processor 909 is a specialized AI processor for performing inference through a convolutional neural network (CNN) on sensor data. In some embodiments, AI processor 909 may be optimized for bit depth of sensor data. In some embodiments, AI processor 909 may be optimized for deep learning operations, such as operations of a neural network including convolution, dot product, vector and/or matrix operations. In some embodiments, the AI processor 909 may be implemented through a plurality of graphic processing units (GPUs) that can effectively perform parallel processing.

In various embodiments, the AI processor 909 is configured to perform deep learning analysis on sensor data received from the sensor(s) 903 while the AI processor 909 is running, and to operate the vehicle at least partially autonomously. may be coupled via an input/output interface to a memory configured to provide an AI processor having instructions that cause the used machine learning result to be determined. In some embodiments, the vehicle control module (Vehicle Control Module) 911 processes commands for vehicle control output from the artificial intelligence (AI) processor 1509, and the AI processor 909 to control various modules of the vehicle. ) can be used to translate the output of each vehicle into commands for controlling the module of each vehicle. In some embodiments, the vehicle control module 911 is used to control a vehicle for autonomous driving. In some embodiments, the vehicle control module 911 may adjust the steering and/or speed of the vehicle. For example, the vehicle control module 911 may be used to control driving of the vehicle, such as deceleration, acceleration, steering, lane change, and lane maintenance. In some embodiments, the vehicle control module 911 may generate control signals for controlling vehicle lighting such as brake lights, turns signals, headlights, etc. can In some embodiments, the vehicle control module 911 may include a vehicle's sound system, a vehicle's audio warnings, a vehicle's microphone system, and a vehicle's horn system. system) may be used to control vehicle audio-related systems, such as.

In some embodiments, vehicle control module 911 provides notification systems, including warning systems, to notify passengers and/or driver of driving events, such as a potential collision or approach of an intended destination. ) can be used to control In some embodiments, vehicle control module 911 may be used to calibrate sensors, such as sensors 903 of a vehicle. For example, the vehicle control module 911 may modify the orientation of the sensors 903 , change the output resolution and/or format type of the sensors 903 , and set the capture rate You can increase or decrease it, adjust the dynamic range, and adjust the focus of the camera. Also, the vehicle control module 911 may individually or collectively turn on/off the operations of the sensors.

In some embodiments, the vehicle control module 911 modifies the frequency range of filters, adjusts features and/or edge detection parameters for object detection, or sets bit depth and channels may be used to vary parameters of the image preprocessor 905 in a manner such as adjusting channels and bit depth. In various embodiments, the vehicle control module 911 may be used to control autonomous driving of a vehicle and/or a driver assistance function of the vehicle.

In some embodiments, the network interface 913 may be in charge of an internal interface between the block components of the autonomous driving control system 900 and the communication unit 915 . Specifically, the network interface 913 may be a communication interface for receiving and/or transmitting data including voice data. In various embodiments, the network interface 913 connects voice calls or receives and/or sends text messages via the communication unit 915 , transmits sensor data, updates the vehicle's software with an autonomous driving system, or the vehicle It can be connected to external servers to update the software of the autonomous driving system.

In various embodiments, the communication unit 915 may include various wireless interfaces of a cellular or WiFi method. For example, the network interface 913 includes the sensors 903, the image preprocessor 905, the deep learning network 907, and the AI processor from the second external electronic device 120 connected through the communication unit 915. 909 ), to receive updates to operating parameters and/or commands for the vehicle control module 911 . For example, the machine learning model of the deep learning network 907 may be updated using the communication unit 915 . According to another example, the communication unit 915 may be used to update operating parameters of the image preprocessor 905 such as image processing parameters and/or firmware of the sensors 903 .

In another embodiment, the communication unit 915 may be used to activate communication for emergency services and emergency contact in an accident or near-accident event. For example, in a crash event, the communication unit 915 may be used to call emergency services for assistance, and may be used to inform the outside of emergency services of the vehicle's location and crash details. In various embodiments, the communication unit 915 may update or obtain the expected arrival time and/or destination location.

According to an embodiment, the autonomous driving system 900 illustrated in FIG. 9 may be configured as an electronic device of a vehicle. According to an embodiment, when an autonomous driving cancellation event occurs from a user during autonomous driving of the vehicle, the AI processor 909 of the autonomous driving system 900 converts the autonomous driving cancellation event related information into training set data of the deep learning network. By controlling the input, it is possible to control the vehicle's autonomous driving software to learn.

10 is a block diagram of an electronic device 1000 according to an embodiment.

Referring to FIG. 10 , the electronic device 1000 according to an embodiment includes sensors 1003 , a positioning unit 1005 , a memory 1007 , a processor 1009 , a driver gesture acquiring unit 1011 , and a communication circuit. At least one of a 1013 , a display unit 1015 , and an autonomous driving system 1017 may be included. In various embodiments, each element may be connected through various interfaces. In some embodiments, the electronic device 1000 may include additional or fewer components as appropriate. For example, the sensor(s) 1003 may be a component of an external device separate from the electronic device 1000 . In some embodiments, the sensors 903 include radar (RADAR), Light Detection And Ranging (LiDAR), and/or ultrasonic sensors in addition to an image sensor.

The positioning unit 1005 positions the vehicle's position in real time through communication with a global positioning system (GPS), a global navigation satellite system (GNSS) such as GLONASS, or a base station of a cellular network, The positioned location may be provided to the processor 1009 .

The memory 1007 may store at least one of various control information for driving a vehicle, driving information generated according to driving of the vehicle, operating system software of the vehicle, and electronic map data for driving the vehicle.

The processor 1009 may include hardware components for processing data based on one or more instructions. In an embodiment, the processor 1009 transmits autonomous driving disengagement event associated information to the server through the communication circuit 1013 on a condition that satisfies a specified criterion. can

In an embodiment, the processor 1009 may store sensor data and location information acquired by the sensor(s) 1003 during autonomous driving of the vehicle in the memory 1007 .

In an embodiment, the autonomous driving disengagement event associated information includes sensor data acquired by the sensor(s) 1003 acquired at the time when the autonomous driving disengagement event occurs, and sensor data acquired It includes at least one of location information and driver driving information. In an embodiment, the sensor data may include at least one of data acquired by an image sensor, radar, LiDAR, and ultrasonic sensors. In an embodiment, autonomous driving disengagement event associated information includes driver identification information (User ID, Driver license information, driver name, etc.) capable of identifying the driver in order to protect the driver's privacy and/or Alternatively, it may be processed so as to be independent of vehicle identification information (License plate information, Vehicle Identification Number), and the like.

In an embodiment, the specified criterion may be a time point at which an autonomous driving disengagement event occurs. For example, it may be a time when a driver intervention occurs while the vehicle is driving in the autonomous driving mode, or a driver gesture requesting to change the driving mode from the autonomous driving mode to the performing driving mode occurs. there is. In one embodiment, the driver gesture acquisition unit 1011 identifies that the driver gesture acquisition unit 1011 operates the steering wheel of the vehicle, operates the accelerator pedal/decelerator pedal of the vehicle, or operates the gear of the vehicle. Based on this, it is possible to determine the driver's intervention. In an embodiment, the driver's gesture acquisition unit 1011 may determine the driver's intervention based on identifying the hand gesture or body gesture of the driver indicating the driving mode conversion from the autonomous driving mode to the manual driving mode. According to an embodiment, the autonomous driving cancellation event may occur at a point where the autonomous driving system 1017 of the vehicle fails to smoothly autonomously drive according to a pre-learned autonomous driving algorithm. For example, in a situation in which a vehicle traveling on a driving route determined according to the autonomous driving mode enters a round about without a traffic light, the presence of another vehicle entering the roundabout is detected, and the other vehicle is predicted Based on the processor 1009 identifying that the vehicle does not proceed in the direction and speed, the driver gesture acquisition unit 1011 changes the mode to the autonomous driving release mode by the driver gesture generated at the time point identified by the processor 1009 can determine driver intervention for In another example, the vehicle traveling on the driving route determined according to the autonomous driving mode is notified of unexpected road conditions (during road construction), traffic conditions, road accidents, vehicle failure notifications, etc. Based on the identification by the processor 1009, the driver gesture acquisition unit 1011 may determine the driver intervention for mode change to the autonomous driving release mode by the driver gesture generated at the time point identified by the processor 1009. there is.

In an embodiment, the driver gesture acquisition unit 1011 may determine whether a user gesture recognized through a visible light camera and/or an infrared camera mounted inside the vehicle is a gesture corresponding to canceling a predetermined autonomous driving mode. . Also, according to an embodiment, the driver gesture acquirer 1011 may identify occurrence of an autonomous driving cancellation event according to a user input selected through a user eXperience (UX) displayed on the display 1013 .

In an embodiment, the processor 1009 acquires driver driving information on a condition that satisfies a specified criterion, and communicates the acquired driving information and the location information from which the driving information is obtained to the communication circuit 1013 . may be transmitted to the server 1100 through In this case, the driver driving information may include at least one of a steering wheel manipulation angle, accelerator pedal manipulation information, deceleration pedal manipulation information, and gear information manipulated by the driver at the time when the autonomous driving cancellation event occurs.

In an embodiment, the processor 1009 is configured to provide autonomous driving disengagement event associated information obtained at the time when the autonomous driving disengagement event occurs to reduce congestion of reverse traffic. When transmitting to the server, only some of the data acquired by the sensor(s) 1003 may be transmitted as autonomous driving cancellation event related information.

For example, when a total of 10 sensors are installed in the vehicle and each sensor acquires sensor data at 30 frames per second (30 fps), the processor 1009 cancels the autonomous driving among the sensor data acquired from the 10 sensors. Only some frames (100 frames) (10 seconds X 10 frames) out of a total of 300 frames (10 seconds X 30 frames) generated for a certain period of time (eg 10 seconds) based on the time when the event occurred are transmitted to the server can do.

In another embodiment, when the processor 1009 transmits the autonomous driving disengagement event associated information acquired at the time when the autonomous driving disengagement event occurs to the server, the sensor(s) 1003 Full data of the data acquired by the . For example, when a total of 10 sensors are installed in the vehicle, and each sensor acquires sensor data at 30 frames per second (30 fps), the processor 1009 only partially data ( For example, 10 frames per second) are stored in the memory 1007, and a total of 300 frames (10 seconds X 30 frames) generated for a predetermined time (eg, 10 seconds) based on the time when the autonomous driving cancellation event occurs. The whole can be sent to the server.

Of course, it goes without saying that the processor 1009 must match the time synchronization of the sensor data obtained from each of the sensors 1003 . According to an embodiment, the autonomous driving system 1017 provides an autonomous driving function to a vehicle using a neural network learned using sensor data acquired by the sensor(s), and through the communication circuit 1013, OTA ( You can update or download the autonomous driving software using the Over The Air) method.

11 is a block diagram of a server 1100 according to an embodiment.

Referring to FIG. 11 , the server 1100 according to an embodiment includes at least one of a processor 1103 , a memory 1105 , a training set generation unit 1107 , a deep learning processing unit 1109 , and a communication circuit 1111 . may include In various embodiments, each element may be connected through various interfaces. In some embodiments, server 1100 may include additional or fewer components as appropriate.

In an embodiment, the processor 1103 distributes the software (algorithm) for autonomous driving learned by the deep learning processing unit 1109 to the electronic device 1000 in an OTA method through the communication circuit 1111 . In an embodiment, the processor 1103 transmits the autonomous driving cancellation event related information received from the electronic device 1000 to the training set generation unit 1107 to train the deep learning processing unit 1109 for learning. control to generate data.

In an embodiment, the memory 1105 stores electronic map data required for autonomous driving of a vehicle, sensor data obtained from vehicles that are connected to a network and performing autonomous driving, and location information thereof, with identification information of a user and/or vehicle and Save regardless.

According to an embodiment, the memory 1105 may store only sensor data and location information generated at the time when the autonomous driving cancel event occurs during autonomous driving of the vehicle.

In an embodiment, the deep learning processing unit 1109 performs learning of the deep learning algorithm of autonomous driving using the training data generated by the training set generating unit 1107, and the autonomous driving algorithm using the performed learning result. update

In an embodiment, the processor 1103 may distribute the autonomous driving algorithm updated by the deep learning processing unit 1109 to the electronic device 1000 connected to the network through an OTA method.

In an embodiment, the processor 1103 is the autonomous driving control system of the vehicle B via the location where the autonomous driving release event related information received from the vehicle A through the communication circuit 1111 is generated, and is configured to update the autonomous driving software. request, vehicle B can download the updated autonomous driving software.

12 is a signal flow diagram for explaining an operation of an electronic device according to various embodiments of the present disclosure;

Referring to FIG. 12 , in operation 1200 , the electronic device 1000 according to an embodiment operates in an autonomous driving mode, and in operation 1202 , acquires sensor data by sensor(s). In operation 1204, when an autonomous driving cancellation event occurs, the electronic device 1000 according to an embodiment converts to a manual driving mode and drives according to a control command generated according to a user's manual operation.

In operation 1206 , the electronic device 1000 according to an embodiment generates information related to the autonomous driving cancellation event and transmits an autonomous driving cancellation event generation notification message to the server 1100 in operation 1210 .

In response to obtaining the autonomous driving cancellation event occurrence notification message, in operation 1212 , the server 1100 transmits a request message for information transmission related to the autonomous driving cancellation event to the electronic device 1000 .

In response to the acquisition of the autonomous driving cancellation event related information transmission request message, in operation 1214 , the electronic device 1000 transmits the autonomous driving canceling event related information to the server 1100 .

In response to the acquisition of the autonomous driving cancellation event related information, in operation 1216, the server 1100 generates training set data for deep learning by using the autonomous driving cancellation event related information.

In operation 1218, the server 1100 performs deep learning using the training set data, and in operation 1220, the server 1100 updates the autonomous driving algorithm.

In operation 1222 , if the electronic device 1000 is switched to the autonomous driving mode by the user, in operation 1224 , the electronic device 1000 drives in the autonomous driving mode. In response to receiving the new version of the autonomous driving software from the server 1100 in operation 1226 , the electronic device 1000 performs autonomous driving using the new version of the autonomous driving software.

13 is a signal flow diagram illustrating an operation of a server according to various embodiments of the present disclosure;

When it is confirmed that the autonomous driving release event occurs from vehicle A in operation 1300 , in operation 1302 , the server 1100 acquires autonomous driving cancellation event related information from vehicle A in operation 1302 .

When the autonomous driving cancellation event-related information is obtained, the server 1100 generates the autonomous driving cancellation event-related information as training set data for deep learning in operation 1304, and the training set data generated in operation 1306 to perform deep learning, and in operation 1308 , the autonomous driving software is updated based on the result of the deep learning performed.

In operation 1310 , if it is confirmed in the server 1100 that the vehicle B is going to pass the point where the autonomous driving release event occurs in the vehicle A ( 1310 - Yes), the server 1100 switches to the vehicle B in operation 1312 . By requesting an autonomous driving software update, and by sending the autonomous driving software to vehicle B in operation 1314 , vehicle B may prevent occurrence of an autonomous driving unlock event similar to that occurring in vehicle A. In operation 1310, the server 1100 is connected to autonomously driving vehicles through a network and can check the route and location of each autonomously driving vehicle in real time, so that a specific vehicle autonomously drives. It can be checked whether the following vehicle will enter/pass through the point where the release event occurs. Of course, for the server 1100 to protect the driver's personal information, location information obtained from each vehicle may be obtained regardless of driver and/or vehicle identification information.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

In an electronic device,
communication circuit;
Memory; and
at least one processor operably coupled to the communication circuitry and the memory;
The memory stores a plurality of instructions,
The plurality of instructions, when executed by the at least one processor, cause the at least one processor to:
receive a video from a first external electronic device distinct from the electronic device by using the communication circuit;
receiving a first signal informing of entering a specified state from the first external electronic device in the state of receiving the video;
in response to receiving the first signal, identify information related to the received video;
In response to the identification of the information, the electronic device controls to transmit a second signal including at least a portion of the information to a second external electronic device that is distinct from the first external electronic device.
The method according to claim 1, wherein the video,
that the electronic map displayed on a third external electronic device linked to the second external electronic device and located in a vehicle including the first external electronic device does not correspond to the video acquired by the first external electronic device transmitted from the first external electronic device to the electronic device based on the identification of the third external electronic device;
electronic device.
The method according to claim 1, wherein the video,
transmitted from the first external electronic device to the electronic device under a condition that a driving mode of a vehicle including the electronic device is changed from an autonomous driving mode to a manual driving mode;
electronic device.
The method according to claim 1, The plurality of instructions, when executed by the at least one processor, the at least one processor,
store the video based on the reception of the first signal;
Based on the non-receipt of the first signal, controlling to discard the video after a specified time has elapsed,
electronic device.
5. The method of claim 4, wherein the length of the time interval for maintaining the storage of the video is:
changed based on the type of the first signal,
electronic device.
The method according to claim 1, The plurality of instructions, when executed by the at least one processor, the at least one processor,
controlling to identify the information related to the video by recognizing the video using a neural network based on the first signal;
electronic device.
The method according to claim 1, The second signal,
causing the second external electronic device to identify one or more terminals associated with at least a portion of the information;
electronic device.
In the terminal,
built-in display;
communication circuit;
Memory; and
at least one processor operably coupled to the display, the communication circuitry and the memory;
The memory stores a plurality of instructions,
The plurality of instructions, when executed by the at least one processor, cause the at least one processor to:
based on the specified application, communicate with the first external electronic device;
receive, from the first external electronic device, information related to a second external electronic device distinguished from the first external electronic device;
in response to receiving the information, identify an external display distinct from the embedded display;
displaying a first visual element related to the information on the external display in a first state in which the external display is connected to the terminal;
display the first visual element on the built-in display in a second state distinct from the first state;
In the displayed state of the first visual element, in response to identification of a user input associated with the first visual element,
control to display a second visual element associated with the first visual element and comprising at least a portion of the information;
The second visual element is displayed on the built-in display in the first state and is displayed on the external display in the second state.
The method according to claim 8, wherein the first visual element,
Displayed to provide information related to the accident;
Displayed as superimposed on the electronic map, provided through the specified application,
terminal.
The method according to claim 9, wherein the first visual element,
Displayed along with a message to inform the accident,
terminal.
The method according to claim 9, wherein the second visual element,
including detailed information about the accident indicated by the first visual element;
terminal.
The method of claim 11 , wherein the plurality of instructions, when executed by the at least one processor, cause the at least one processor to:
in response to identifying the user input, ceasing to display the first visual element superimposed on the electronic map;
controlling to display the second visual element;
terminal.
The method according to claim 9, wherein the visual element,
Displayed in connection with information about the time of occurrence of the accident,
terminal.
A method for operating an electronic device, comprising:
receiving a video from a first external electronic device distinguished from the electronic device by using the communication circuit;
receiving a first signal informing of entering a specified state from the first external electronic device in the state of receiving the video;
in response to receiving the first signal, identifying information related to the received video;
and in response to the identification of the information, transmitting a second signal including at least a portion of the information to a second external electronic device that is distinct from the first external electronic device.
The method according to claim 14, wherein the video,
that the electronic map displayed on a third external electronic device linked to the second external electronic device and located in a vehicle including the first external electronic device does not correspond to the video acquired by the first external electronic device transmitted from the first external electronic device to the electronic device based on the identification of the third external electronic device;
Way.
The method according to claim 14, wherein the video,
transmitted from the first external electronic device to the electronic device under a condition that a driving mode of a vehicle including the electronic device is changed from an autonomous driving mode to a manual driving mode;
Way.
15. The method of claim 14,
storing the video based on the reception of the first signal;
Based on the non-receipt of the first signal, the method further comprising: discarding the video after a specified time elapses,
Way.
18. The method of claim 17, wherein the length of the time interval maintaining storing the video is:
changed based on the type of the first signal,
Way.
The method according to claim 14, wherein the identifying the information comprises:
and identifying the information related to the video by recognizing the video using a neural network based on the first signal.
The method according to claim 14, wherein the second signal,
causing the second external electronic device to identify one or more terminals associated with at least a portion of the information;
Way.

Images