KR20210005818A - Electronic apparatus and control method thereof - Google Patents

Abstract

The present invention provides an electronic apparatus capable of performing interface output and various information processing, and a method of operating the same. According to an embodiment of the present invention, an electronic apparatus comprises: a communication unit for receiving image data including a driving image of a vehicle; a driving lane information generation module for identifying a lane area from the image data and generating driving lane information corresponding to a lane on which the vehicle is traveling in the driving image from the image data of the identified lane area; and a control unit which performs necessary processing according to the driving lane information.

Description

Electronic device and its control method TECHNICAL FIELD [ELECTRONIC APPARATUS AND CONTROL METHOD THEREOF}

The present invention relates to an electronic device and a control method thereof, and more particularly, to an electronic device provided in a vehicle and capable of performing lane recognition and notification, and a control method thereof.

The most important thing when driving a vehicle is safe driving and prevention of traffic accidents, and for this purpose, various auxiliary devices that perform the vehicle's attitude control and function control of vehicle components, and safety devices such as seat belts and airbags are installed on the vehicle. have.

In addition, recently, devices such as a black box that are located in a vehicle and store driving images of the vehicle and data transmitted from various sensors are being provided in vehicles to identify the cause of an accident in the vehicle when an accident occurs. Portable terminals such as smartphones and tablets can also be equipped with a black box or a navigation application, and thus are used as such a vehicle device.

However, currently, the utilization of driving images in such vehicle devices is low. More specifically, even if a driving image of the vehicle is acquired through a vision sensor such as a camera currently mounted on the vehicle, the electronic device of the vehicle simply displays and transmits the driving image, or only generates simple surrounding notification information such as whether or not a lane is departed.

In addition, a HUD (Head Up Display) or an augmented reality interface has also been proposed as an emerging electronic device of a vehicle, but the utilization of the driving image of the vehicle is limited to simple display or simple notification information generation.

It was conceived to solve the above problems of the present invention, and an object of the present invention is an electronic device capable of generating driving lane information based on a driving image of a vehicle, and performing interface output and various information processing. It is to provide a method of operation.

A method according to an exemplary embodiment of the present invention for solving the above problems includes: receiving image data including a driving image of a vehicle in a method of controlling an electronic device; Identifying a lane area from the image data; Generating driving lane information corresponding to a lane on which the vehicle is traveling in the driving image from the image data of the identified lane area portion; And performing necessary processing according to the driving lane information.

In addition, a method according to another embodiment of the present invention for solving the above problems, in the control method of a vehicle information notification device based on an augmented reality, receiving image data including a driving image of the vehicle; Identifying a lane area from the image data; Generating driving lane information of the vehicle from the image data of the identified lane area; Generating a lane information interface overlaid on the augmented reality according to the driving lane information; And outputting the lane information interface through the augmented reality.

In addition, an apparatus according to an exemplary embodiment of the present invention for solving the above problems includes: a communication unit for receiving image data including a driving image of a vehicle; A driving lane information generation module for identifying a lane area from the image data and generating driving lane information corresponding to a lane on which the vehicle is traveling in the driving image from the image data of the identified lane area portion; And a controller that performs necessary processing according to the driving lane information.

In addition, an apparatus according to another embodiment of the present invention for solving the above-described problem is an augmented reality-based vehicle information notification device, comprising: a communication unit for receiving image data including a driving image of a vehicle; A driving lane information generation module that identifies a lane area from the image data and generates driving lane information of the vehicle from the image data of the identified lane area; A control unit for generating a lane information interface overlaid on the augmented reality according to the driving lane information; And an output unit for outputting the lane information interface through the augmented reality.

The methods according to an embodiment of the present invention as described above may be implemented as a recording medium in which a program for execution on a computer is recorded.

According to an embodiment of the present invention, driving lane information corresponding to a lane on which a vehicle is traveling in a driving image may be generated from image data of a portion of a lane area, and necessary processing may be performed.

Accordingly, various information processing including outputting a lane interface using driving lane information and generating an augmented reality interface may be performed.

In addition, according to an embodiment of the present invention, it is possible to perform a driver assistance role by determining a current driving lane based on the driving lane information and guiding the driver to the driving lane.

In addition, according to an embodiment of the present invention, user convenience may be promoted by performing an accurate lane change notification using the route information of the navigation and the driving lane information.

In addition, according to an embodiment of the present invention, appropriate lane change notification can be selectively performed according to the types of lanes on both sides of the driving lane using driving lane information, thereby improving the performance of lane change notification and providing rich information. Can do

1 is a block diagram schematically illustrating an electronic device according to an embodiment of the present invention.
2 is a diagram illustrating a system network connected to an electronic device according to an embodiment of the present invention.
3 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.
4 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention in more detail.
5 to 7 are diagrams showing step-by-step an image processing process according to a control method of an electronic device according to an exemplary embodiment of the present invention.
8 is an exemplary diagram illustrating driving lane information generated by an electronic device according to an embodiment of the present invention.
9 to 10 are diagrams for explaining a method of guiding a lane change according to driving lane information of an electronic device according to an exemplary embodiment of the present invention.
11 is a flowchart illustrating a lane departure notification method according to driving lane information of an electronic device according to an embodiment of the present invention.

The following content merely illustrates the principles of the present invention. Therefore, those skilled in the art can implement the principles of the present invention and invent various devices included in the concept and scope of the present invention, although not clearly described or illustrated herein. In addition, all conditional terms and examples listed in this specification are, in principle, intended to be clearly intended only for the purpose of making the concept of the present invention understood, and should be understood as not limiting to the embodiments and states specifically listed as described above. do.

In addition, it is to be understood that all detailed descriptions listing specific embodiments as well as principles, aspects and embodiments of the present invention are intended to include structural and functional equivalents of these matters. It should also be understood that these equivalents include not only currently known equivalents, but also equivalents to be developed in the future, that is, all devices invented to perform the same function regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing a conceptual perspective of exemplary circuits embodying the principles of the invention. Similarly, all flowcharts, state transition diagrams, pseudocodes, etc. are understood to represent various processes performed by a computer or processor, whether or not the computer or processor is clearly depicted and that can be represented substantially in a computer-readable medium. Should be.

The functions of the various elements shown in the figures, including a processor or functional block represented by a similar concept, may be provided by the use of dedicated hardware as well as hardware having the ability to execute software in association with appropriate software. When provided by a processor, the function may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

In addition, the explicit use of terms presented as processor, control, or similar concepts should not be interpreted exclusively by referring to hardware capable of executing software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM, and non-volatile memory. Other commonly used hardware may also be included.

In the claims of this specification, components expressed as means for performing the functions described in the detailed description include all types of software including, for example, a combination of circuit elements or firmware/microcode that perform the above functions. It is intended to include all methods of performing a function to perform the function, and is combined with suitable circuitry for executing the software to perform the function. Since the invention defined by these claims is combined with the functions provided by the various enumerated means and combined with the manner required by the claims, any means capable of providing the above functions are equivalent to those conceived from this specification. It should be understood as.

The above-described objects, features, and advantages will become more apparent through the following detailed description in connection with the accompanying drawings, whereby those of ordinary skill in the technical field to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically illustrating an electronic device according to an embodiment of the present invention.

The electronic device 100 described below may be implemented as various devices provided or mounted on a vehicle. For example, the electronic device 100 includes a navigation system capable of performing route guidance to a specific point, a black box that stores or transmits image information by photographing the surroundings of a vehicle, a smart phone, a tablet computer, a notebook computer, and a personal digital assistant (PDA). Digital assistant), PMP (portable multimedia player), etc. can be implemented in a variety of devices.

The electronic device 100 according to an embodiment of the present invention capable of performing such an operation may be implemented as, for example, a vehicle navigation system, a vehicle black box, or an augmented reality system.

When the electronic device 100 is a vehicle navigation system or an augmented reality system, the electronic device 100 may refer to a system that informs the driver and passengers of the vehicle of various data related to operation and maintenance of the vehicle. In this case, in a narrow sense, the electronic device 100 may be a vehicle navigation itself, and in a broad sense, the electronic device 100 may be regarded as a concept including various electronic devices that operate in connection with the electronic device 100 by wired or wirelessly.

This may mean that the electronic device 100 may configure an integrated system by connecting various electronic devices capable of supplementing and increasing the function of the vehicle navigation system. An electronic device capable of configuring the entire system may be another mobile terminal capable of accessing a mobile communication network, a remote control, or the like.

When the electronic device 100 is implemented as a vehicle black box, it may be implemented by itself or integrally with a navigation system.

When the electronic device 100 is implemented as a vehicle black box, the electronic device 100 may store data necessary for a vehicle accident processing process by exchanging signals with a vehicle navigation system or a mobile terminal. For example, when an accident occurs while the vehicle is running, the image data stored in the electronic device 100 may be analyzed and used to determine the status of the accident and the degree of negligence. In addition, when connected to a vehicle navigation system or other mobile terminal, the electronic device 100 may utilize various data stored in the vehicle navigation system or other mobile terminal. For example, this means that the effectiveness of the electronic device 100 can be increased by matching the image acquired by the electronic device 100 with map data stored in the mobile terminal.

When the electronic device 100 is implemented as a vehicle black box, vehicle data may be obtained while driving and while stopping. In other words, it is possible to capture an image while the vehicle is driving, as well as to capture an image even when the vehicle is stopped. The quality of the image acquired through the electronic device 100 may be constant or may be changed. For example, by increasing the image quality before and after the occurrence of an accident, and lowering the image quality in a normal case, it is possible to store essential images while minimizing the required storage space.

The entire system network including the electronic device 100 according to an embodiment of the present invention includes Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), and Ultra WideBand (UWB). , ZigBee, etc. can communicate.

In addition, the electronic device 100 according to an embodiment of the present invention may analyze the captured driving image to generate driving lane information, determine a driving lane, and provide it to a user through a lane information interface.

Hereinafter, the configuration and operation of the electronic device 100 will be described in more detail.

Referring to FIG. 1, an electronic device 100 according to an embodiment of the present invention includes a communication unit 110, an input unit 120, a sensing unit 130, an output unit 140, a storage unit 150, and a power supply unit ( 160), a control unit 170, a driving lane information generation module 180, a driving lane determination module 185, and an augmented reality interface module 190.

The communication unit 110 may be provided for the electronic device 100 to communicate with other devices. The communication unit 110 may include all or part of a location data module, a wireless Internet module, a broadcast transmission/reception module, a short range communication module, and a wired communication module. For example, the communication unit 110 may receive image data including a driving image of a vehicle from another device or a camera module connected to the electronic device 100.

The location data module is a device that acquires location data through the Global Navigation Satellite System (GNSS). GNSS refers to a navigation system capable of calculating the location of a receiving terminal using radio signals received from satellites. Specific examples of GNSS include GPS (Global Positioning System), Galileo, GLONASS (Global Orbiting Navigational Satellite System), COMPASS, IRNSS (Indian Regional Navigational Satellite System), QZSS (Quasi-Zenith Satellite System), etc. I can. The location data module of the electronic device 100 according to an embodiment of the present invention may obtain location data by receiving a GNSS signal serving in an area where the electronic device 100 is used.

The wireless Internet module is a device that acquires or transmits data by accessing the wireless Internet. The wireless Internet accessible through the wireless Internet module may be a wireless LAN (WLAN), a wireless broadband (Wibro), a world interoperability for microwave access (Wimax), and a high speed downlink packet access (HSDPA).

The short-range communication module is a device for short-range communication. As described above, the short-range communication module may communicate through Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra WideBand (UWB), ZigBee, and the like.

The wired communication module is an interface device capable of connecting the electronic device 100 to other devices by wire. The wired communication module may be a USB module capable of communicating through a USB port or a cable interface for transmitting other data signals.

The electronic device 100 according to an embodiment of the present invention and another embodiment may communicate with other devices through a short-range communication module or a wired communication module as necessary. Furthermore, when communicating with a plurality of devices, one of the devices communicates through a short-range communication module and the other communicates through a wired communication module. Accordingly, in the case of an electronic device that does not include a camera function according to an embodiment of the present invention, a driving image photographed from a black box is received through the communication unit 110, or a separate camera installed toward the front of the vehicle is driven. You can receive video.

The input unit 120 is a device that converts a physical input from the outside of the electronic device 100 into a specific electrical signal. The input unit 120 may include a user input module and all or part of a microphone module.

The user input module is an input device that a user can input through a touch or a push operation. Here, the user input module may be implemented using at least one of various button types, a touch sensor receiving a touch input, and a proximity sensor receiving an approaching motion.

The microphone module is a device that receives a user's voice and sound generated inside and outside the vehicle.

The sensing unit 130 is a device capable of detecting the current state of the electronic device 100. The sensing unit 130 may include a motion sensing module and all or part of a light sensing module.

The motion sensing module may detect motion of the electronic device 100 in a 3D space. The motion sensing module may include a 3-axis geomagnetic sensor and a 3-axis acceleration sensor. By combining the motion data acquired through the motion sensing module with the location data acquired through the location data module, a more accurate trajectory of the vehicle to which the electronic device 100 is attached may be calculated.

The light sensing module is a device that measures ambient illumination of the electronic device 100. The brightness of the display module included in the output unit 140 may be changed to correspond to the ambient brightness using the illumination data acquired through the light sensing module.

The output unit 140 is a device that outputs data of the electronic device 100. The output unit 140 may include all or part of a display module and an audio output module.

The display module is a device that outputs data that can be visually recognized by the electronic device 100. The display module may be implemented as a display unit provided on the front of the housing of the electronic device 100. Here, the display module is integrally formed with the electronic device 100 to output visual recognition data, and is installed separately from the electronic device 100 such as a head up display (HUD) to receive visual recognition data through augmented reality, etc. You can also print it out.

The audio output module is a device that outputs data that can be audibly recognized by the electronic device 100. The audio output module may be implemented as a speaker that expresses sound data to be informed to the user of the electronic device 100.

The storage unit 150 is a device that stores data required for the operation of the electronic device 100 and data generated by the operation. The storage unit 150 may be a memory built into the electronic device 100 or a removable memory. Data necessary for the operation of the electronic device 100 may include an OS, an area information table, driving lane information, a route search application, a map, and the like. In addition, data generated by the operation of the electronic device 100 may be a driving lane information table, a lane information interface, or the like.

The power supply unit 160 is a device that supplies power necessary for the operation of the electronic device 100 or for the operation of another device connected to the electronic device 100. The power supply unit 160 may be a device that receives power from an external power source such as a battery or a vehicle built in the electronic device 100. In addition, the power supply unit 160 may be implemented as a wired communication module or a device that is supplied wirelessly according to a type of receiving power.

The controller 170 is a device that outputs control signals for controlling various operations of the electronic device 100. Furthermore, the controller 170 may output a control signal for controlling another device connected to the electronic device 100.

In particular, according to an embodiment of the present invention, the control unit 170 controls the driving lane information generation module 180 and the driving lane determination module 185 to identify a lane area from the received image data, and Driving lane information corresponding to a lane on which the vehicle is traveling in the driving image may be generated from the image data of the region portion, and necessary processing may be performed according to the driving lane information.

More specifically, the driving lane information generation module 180 identifies the lane area from the image data including the driving image received from the control unit 170, and is located on both sides of the lane on which the vehicle is traveling from the lane area. Driving lane information corresponding to the lane may be generated. The driving lane information may include at least one of lane type information and lane color information corresponding to each lane located on both sides of the lane on which the vehicle is driving.

In addition, the driving lane information generation module 180 may perform binarization on the image data of the lane area portion to generate driving lane information, and obtain lane type information from the binarized partial image data.

In addition, the driving lane information generating module 180 analyzes the binary partial image data using at least one of the time continuity information and the lane speed information of the lane to determine whether the types of both lanes of the lane the vehicle is driving are solid lines. You can identify if it is a dotted line. In addition, the driving lane information generation module 180 may extract color information corresponding to each lane whose type is identified from the image data and generate the driving lane information.

Meanwhile, the driving lane determination module 185 may identify a location of a lane in which the vehicle is driving in the driving image based on driving lane information under the control of the controller 170.

Here, the driving lane information may mean information indicating each of the lines on both sides forming a lane in which the vehicle is driving, and the driving lane may be determined according to the driving lane information.

Further, the driving lane may mean a position of a relative lane based on a center line such as a first lane, a central lane, or an end lane. Further, even if it is determined based on the same driving lane information, the driving lane determined by the driving lane determination module 185 may be different according to national traffic laws and the like. Accordingly, the driving lane determination module 185 may determine the current driving lane based on preset country information or a lane determination table.

In addition, the driving lane determination module 185 may transmit the determined driving lane information to the control unit 170.

The control unit 170 performs necessary processing based on the driving lane information, and outputs it through the output unit 140.

For example, the controller 170 may output lane guidance information generated based on the position of the lane determined by the driving lane determination module 185 through the output unit 140. When the electronic device 100 provides a vehicle navigation function, the controller 170 may output a lane change notification according to a route to a destination and the determined lane position. When the distance to the left or right turn guidance point is less than a certain distance, the controller 170 may determine whether the current lane is capable of turning left or right and output a lane change notification to the user.

In addition, for example, the controller 170 may select an appropriate lane departure warning according to the type of lanes on both sides of the driving lane of the vehicle identified based on the driving lane information and output this to the user. When the electronic device 100 provides a lane departure warning function, the controller 170 may provide different warnings according to the type and color of the lane departure. For example, the control unit 170 selects different warning images or warning voices according to the case where the vehicle invades the center line, the white solid line, the white dotted line, or the blue line. Can be printed.

And, for example, when the electronic device 100 performs vehicle information notification based on augmented reality, the control unit 170 generates a lane information interface in connection with the augmented reality interface module 190, and the output unit ( 140). Augmented reality may be provided through a HUD using a vehicle windshield or an image overlay using a separate image output device, and the augmented reality interface module 190 may generate a reality image or an interface image overlaid on the glass. I can. Through this, an augmented reality navigation or vehicle infotainment system may be implemented.

In addition, the augmented reality interface module 190 may generate a lane information interface overlaid on the augmented reality according to the driving lane information generated by the driving lane generating module 180 or the lane information determined by the driving lane determining module 185. I can.

In addition, the control unit 170 may control the output unit 140 to set a position or region where the lane information interface is displayed through the augmented reality interface module 190 and to be overlaid on the augmented reality and output. The controller 170 may perform a lane change notification or a lane departure warning, as described above, through the lane information interface overlaid on the augmented reality.

2 is a diagram illustrating a system network connected to an electronic device according to an embodiment of the present invention.

As shown in FIG. 2, the electronic device 100 according to an embodiment of the present invention may be implemented with various devices provided in a vehicle such as a navigation system, a black box, or other vehicle augmented reality interface providing device, and various communication networks. And other electronic devices 61 to 64.

In addition, the electronic device 100 may calculate a current location and a current time zone by interworking with the GPS module according to the radio signal received from the satellite 20.

Each satellite 20 may transmit an L-band frequency having a different frequency band. The electronic device 100 may calculate a current location based on a time taken for the L-band frequency transmitted from each satellite 20 to reach the electronic device 100.

Meanwhile, the electronic device 100 may wirelessly access the network 30 through the control station 40 (ACR), the base station 50 (RAS), etc. through the communication unit 110. When the electronic device 100 is connected to the network 30, data can be exchanged by indirectly connecting with other electronic devices 61 and 62 connected to the network 30.

Meanwhile, the electronic device 100 may indirectly access the network 30 through another device 63 having a communication function. For example, when a module capable of accessing the network 30 is not provided in the electronic device 100, communication with another device 63 having a communication function may be performed through a short-range communication module or the like.

3 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention.

Referring to FIG. 3, first, the electronic device 100 receives image data (S101).

The image data may include a driving image of a specific vehicle, and the driving image of the vehicle may include an image captured by a camera module installed in the electronic device 100 or an image captured by another device. The control unit 170 may receive image data through the communication unit 110 or directly input through a separate interface.

For example, the photographed driving image may mean an RGB (Red Green Blue) (color) image. In addition, the driving image may include a front image of the vehicle obtained from a camera mounted at a position facing the windshield of the vehicle.

Then, the electronic device 100 identifies a lane area from the image data (S103).

The driving lane information generation module 180 may identify a lane area from a driving image of the image data. To this end, the driving lane information generation module 180 converts the driving image into a gray image and performs a lane detection algorithm to determine an area where each lane located on both sides of the driving vehicle can be identified as the lane area.

Then, the electronic device 100 generates driving lane information by analyzing partial image data of the identified lane area (S103).

The driving lane information generation module 180 may generate driving lane information corresponding to each lane by analyzing lane pattern information and color information for the detected lane area. The driving lane information may include at least one of line type information and line color information corresponding to each lane located on both sides of the driving vehicle.

Then, the electronic device 100 performs necessary processing according to the driving lane information (S107).

The controller 170 may perform a lane change notification, a lane departure warning, or a lane information notification through an augmented reality interface using the generated driving lane information. To this end, the driving lane determination unit 185 may determine a driving lane identified from the driving lane information and transmit it to the control unit 170.

Hereinafter, in the method of controlling an electronic device, a process of acquiring driving lane information will be described in more detail with reference to FIGS. 4 to 7.

4 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present invention in more detail.

Referring to FIG. 4, the electronic device 100 according to an embodiment of the present invention converts received color image data into a gray image (S201), and detects a lane area from the converted gray image (S203).

The driving lane information generation module 180 may extract a region for detecting a lane from the photographed driving image. In addition, the driving lane information generation module 180 may perform light source correction on the original driving image in order to minimize the shadow influence because it is difficult to detect lanes when a part of the road is affected by a shadow.

In addition, the driving lane information generating module 180 may detect an area in which a lane can exist as a lane area according to a predetermined camera position or a camera installation angle. For example, the driving lane information generation module 180 may determine a lane area based on a position at which a lane can start. In addition, the driving lane information generation module 180 may estimate the position where the lane area starts and the length of the lane area as the width of the lane in the driving image (the maximum width between the left and right lane areas) and the viewing angle of the camera. .

In addition, the driving lane information generation module 180 may convert a gray image corresponding to the lane detection region into an edge image, and detect the lane region based on a straight line position extracted from the converted edge image. More specifically, the driving image may be converted into an edge image through various known algorithms, and the edge image may include edges representing a plurality of straight lines. In this case, the driving lane information generating module 180 may recognize the position of the detected straight line as a lane. In addition, the driving lane information generating module 180 may determine a lane area based on a position of a straight line having a predetermined lane width with respect to the driving direction of the vehicle among a plurality of straight line candidates.

5 shows a process of converting a gray image and detecting a lane area.

Referring to FIG. 5, the first input driving image may be converted into a gray image 200, and straight lane areas 201 and 202 may be detected through a lane detection algorithm such as edge detection. The lane area may be divided into a left lane area 201 and a right lane area 202 based on the driving direction of the vehicle.

4 will be described again.

Thereafter, when a lane area is detected, the electronic device 100 sets a lane type ROI based on the lane area (S205).

When a lane area is detected, the driving lane information generation module 180 may set a lane type region of interest (ROI) based on the detected lane area. The lane type interest area may mean a portion of a driving image including a lane for determining the line type and color of the lane and a predetermined area around the lane.

More specifically, FIG. 6 shows a lane type interest area in a gray image.

As shown in FIG. 6, the lane type ROI 210 and 220 may partially include a previously detected lane area and a peripheral area thereof. Further, the lane type ROI may be divided into a left lane type ROI 210 and a right lane type ROI 220 based on the vehicle traveling direction.

For example, the previously detected lane area may be generally formed as a straight line, and when expressed by an equation such as y = a * x + b, the lane type region of interest is y = a * x + b + m and y = a * x + b-m can be represented as a region including all. Unlike the existing simple lane detection method, this is a method for generating detailed and various driving lane information. The driving lane information generation module 180 can expand the detected line lane area and set the surrounding area as a region of interest. have.

4 will be described again.

Thereafter, the electronic device 100 performs binarization on the lane type ROI (S207), maps the binarized partial gray image to a one-dimensional region (S209), and uses at least one of visual continuity and speed To identify (S211).

The driving lane information generation module 180 may extract a partial gray image of a lane type ROI from the converted gray image and perform binarization on the partial gray image. The reference value for binarization may be determined as an average luminance value of the partial gray image of the ROI. Accordingly, the driving lane information generation module 180 can clearly distinguish only a portion determined as a lane from the partial gray image.

In addition, the driving lane information generation module 180 may map each line (left and right) identified in the binary partial gray image to a one-dimensional area. And, by analyzing the pattern of each line mapped to the one-dimensional area, the line type can be identified.

More specifically, FIG. 7 shows the binarization and one-dimensional mapping of a lane type ROI in a gray image.

As shown in FIG. 7, when binarization is performed on a lane-type ROI, a binarized image 300 may be obtained. In the binarized image 300, a portion indicated by white may be identified as a lane, and other portions may be identified as black.

In addition, each line identified in the binarized image 300 may be mapped to a one-dimensional region. The driving lane information generation module 180 may easily determine a line type by using the image 310 mapped to the 1D area.

For example, the driving lane information generation module 180 may determine whether a dotted line or a solid line is based on a starting point and a length feature of each line mapped in one dimension. In addition, the driving lane information generating module 180 may determine whether a dotted line or a solid line is made using the continuity and speed of each line mapped in one dimension. Further, the driving lane information generation module 180 may first determine a dotted line or a solid line according to the starting point and length characteristics, and then secondly determine a dotted line or a solid line using time continuity and speed.

More specifically, the driving lane information generation module 180 may first determine whether a dotted line or a solid line by comparing and determining a line length according to a starting point position of each line. In this case, the driving lane information generation module 180 may determine whether there is a dotted line or a solid line with only one image frame.

In addition, the driving lane information generation module 180 may more clearly determine whether each line is a dotted line or a solid line depending on whether each line is continuously formed over time. For example, the driving lane information generation module 180 may preset a degree of continuity according to a line movement speed in an image, and may determine a continuity degree as a dotted line when the continuity of each line is lower than a preset continuity degree value.

Accordingly, according to an exemplary embodiment of the present invention, it is possible to determine a final line type by pre-distinguishing whether a dotted line or a solid line is performed through one frame and verifying it through successive frames.

Referring back to FIG. 4, the electronic device 100 detects a color of a line portion whose type is identified from original image data that is color (S213).

The driving lane information generation module 180 may analyze a color image, detect a color of a portion corresponding to a line whose type has been previously identified, and classify it. For example, the driving lane information generation module 180 may classify the detected color into white, yellow, or blue.

Thereafter, the electronic device 100 generates driving lane information based on the identified line type and the classified color (S215), and performs necessary processing according to the generated driving lane information (S217).

8 is an exemplary diagram illustrating driving lane information generated by an electronic device according to an embodiment of the present invention.

As shown in FIG. 8, the driving lane information may include at least one of line position, line type, or line color information, and may be configured in the form of a table. This is for exemplifying driving lane information, and it is obvious that other information may be added or changed.

As shown in the example of FIG. 8, the line position may be a left line or a right line, and the line type may be a dotted line or a solid line according to the pattern. In addition, the line color may be classified as yellow, white, or blue. However, the line type and color may be assigned different values according to the traffic laws of each country.

9 to 11 are diagrams for explaining a processing method according to driving lane information of an electronic device according to an embodiment of the present invention.

9 is a flowchart illustrating a process of guiding a change in a driving lane of an electronic device according to an embodiment of the present invention.

Referring to FIG. 9, first, the electronic device 100 generates driving lane information (S303). The driving lane information may be generated by a method as shown in FIG. 4.

Then, the electronic device 100 identifies the current lane location according to the driving lane information (S305), and performs a driving lane change guide according to the lane location (S307).

As described above, the control unit 170 of the electronic device 100 identifies the current lane location according to the driving lane determined by the driving lane determination module 185, and changes the lane when guidance on changing the driving lane is required according to the lane location. Guidance can be followed.

10 shows a lane information table for determining such a driving lane. The driving lane determination module 185 may determine a location of a relative lane based on the center line by comparing the lane information table with the driving lane information, and may output it to the control unit 170.

Here, the position of the relative lane may be any one of a first lane, a center lane, and an end lane, and may be differently designated according to traffic laws for each country. Accordingly, the lane information table may include information on a driving lane determined according to a country, a left line type and color, and a right line type and color.

Accordingly, the driving lane determination module 185 may determine the position of the lane based on driving lane information generated based on the driving direction of the current vehicle. For example, when the country is designated as Korea, when the left line is a solid yellow line and the right line is a white dotted line, the driving lane determination module 185 may determine as the closest primary lane based on the center line. In addition, when the left line is a white dotted line and the right line is a white solid line, the driving lane determination module 185 may determine the end lane. However, the table shown in FIG. 10 may be an example, and may be set to different values for each country or situation according to the setting.

The control unit 170 may perform a lane change guide based on the determined lane position. For example, when performing a navigation guide, the controller 170 may determine whether a lane change is necessary according to a position of a lane in advance before reaching a turn point on a path, and may notify the user of a lane change.

According to such an embodiment of the present invention, a location of a relative lane can be clearly distinguished by supplementing the existing simple lane recognition. Accordingly, in the lane change notification, the control unit 170 can continuously induce a lane change so as to change to a turnable lane.

11 illustrates a lane departure warning method according to driving lane information of the electronic device 100.

Referring to FIG. 11, first, the electronic device 100 generates driving lane information (S303). The driving lane information may be generated by a method as shown in FIG. 4.

Further, the electronic device 100 identifies a line type and a line color of both lanes of the vehicle based on the driving direction according to the driving lane information (S305).

Thereafter, when the electronic device 100 leaves a lane, the electronic device 100 selects and outputs an appropriate lane departure warning according to the line type and color of the lane (S307).

As described above, the control unit 170 of the electronic device 100 may select an appropriate lane departure warning according to the type of lanes on both sides of the driving lane of the vehicle identified based on driving lane information, and output this to the user. have. When the electronic device 100 provides a lane departure warning function, the controller 170 may provide different warnings according to the type and color of the lane departure. For example, the control unit 170 selects different warning images or warning voices according to the case where the vehicle invades the center line, the white solid line, the white dotted line, or the blue line. Can be printed.

In addition, the lane change notification or lane departure warning described in FIGS. 9 to 11 may be performed based on the augmented reality described above. In this case, the controller 170 may generate a lane information interface in conjunction with the augmented reality interface module 190 and output through the output unit 140.

Meanwhile, according to an embodiment of the present invention, it is illustrated that the driving lane information generating module 180 and the driving lane determining module 185 are included in the electronic device 100, but each module is located in an external device or an external separate server. It can also be operated independently. For example, the controller 170 may transmit a driving image to an external server, and the external server may generate driving lane information or determine a driving lane and transmit a response to the controller 170.

Meanwhile, the method for controlling an electronic device according to various embodiments of the present disclosure described above may be implemented as a program code and provided to each server or devices while being stored in various non-transitory computer readable media. have.

The non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, and ROM.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

Claims (21)

In a computer readable storage medium storing one or more programs,
The one or more programs,
When executed by at least one processor of an electronic device having a camera and a display,
While guiding the path of the vehicle being moved, a plurality of images of the front of the vehicle being moved are acquired through the camera,
Based on at least some of the plurality of images, identify whether the vehicle can perform a turn in a driving lane of the vehicle in order to move along the path,
While identifying that the vehicle is unable to perform a turn in the driving lane, identifying a distance between the vehicle and a position on the path for which the turn of the vehicle is required,
Based on identifying that the identified distance is less than or equal to a specified distance, to display a notification through the display for guiding that a change in the driving lane of the vehicle is required for the turn,
A computer-readable storage medium containing instructions that cause the electronic device.
The method of claim 1, wherein the plurality of images,
A first image in a first frame and a second image in a second frame following the first frame,
When the one or more programs are executed by the at least one processor of the electronic device,
Identify lanes adjacent to the vehicle from the first image,
Determining a type of each of the lanes identified from the first image,
To verify the result of the determination based on a change in position between the lanes in the first image and the lanes in the second image,
A computer-readable storage medium containing instructions that further cause the electronic device.
The method of claim 2, wherein the one or more programs are executed by the at least one processor of the electronic device,
Based on the type of each of the lanes, to identify a relative positional relationship between the driving lane among the plurality of lanes arranged along the route and the remaining lanes of the plurality of lanes,
A computer-readable storage medium containing instructions that further cause the electronic device.
The method of claim 3, wherein the one or more programs are executed by the at least one processor of the electronic device,
Based on identifying that the vehicle being moved deviates from the driving lane, superimposing on the user interface displayed through the display to provide a guide of the route, guiding that the vehicle leaves the driving lane To display a notification for
A computer-readable storage medium containing instructions that further cause the electronic device.
The method of claim 4, wherein the notification superimposed on the user interface,
A computer-readable storage medium displayed through the display that is a heap-up display (HUD).
The method of claim 2, wherein the one or more programs are executed by the at least one processor of the electronic device,
Acquiring a binary image converted from the first image,
To identify the lanes from the binary image,
A computer-readable storage medium containing instructions that further cause the electronic device.
The method of claim 6, wherein the one or more programs are executed by the at least one processor of the electronic device,
Identify regions including the lanes identified from the binary image as regions of interest,
To determine a type of each of the lanes by performing binarization on the regions identified as the regions of interest,
A computer-readable storage medium containing instructions that cause the electronic device.
In the method for operating an electronic device having a camera and a display,
An operation of acquiring through the camera a plurality of images of the front of the vehicle being moved while guiding the path of the vehicle being moved;
An operation of identifying whether a turn can be performed in a driving lane of the vehicle in order to move the vehicle along the path, based on at least a portion of the plurality of images; and
An operation of identifying a distance between the vehicle and a position on the route for which the turn of the vehicle is required while identifying that the vehicle cannot perform a turn in the driving lane; and
And displaying, on the display, a notification for guiding that a change in the driving lane of the vehicle is required for the turn, based on identifying that the identified distance is less than or equal to a specified distance.
The method of claim 8, wherein the plurality of images,
A first image in a first frame and a second image in a second frame following the first frame,
The above method,
Identifying lanes adjacent to the vehicle from the first image;
Determining a type of each of the lanes identified from the first image; and
And verifying a result of the determination based on a change in position between the lanes in the first image and the lanes in the second image.
The method of claim 9,
The method further comprising: identifying a relative positional relationship between the driving lane among the plurality of lanes arranged along the route and the remaining lanes among the plurality of lanes, based on the type of each of the lanes.
The method of claim 10,
Based on identifying that the vehicle being moved deviates from the driving lane, superimposing on the user interface displayed through the display to provide a guide of the route, guiding that the vehicle leaves the driving lane The method further comprising displaying a notification for.
The method of claim 11, wherein the notification superimposed on the user interface,
A method displayed through the display, which is a heap-up display (HUD).
The method of claim 9, wherein the operation of identifying the lanes,
Obtaining a binary image converted from the first image, and
And identifying the lanes from the binary image.
The method of claim 13, wherein determining the type of each of the lanes,
An operation of identifying regions including the lanes identified from the binary image as regions of interest,
And determining a type of each of the lanes by performing binarization on the regions identified as the regions of interest.
In the electronic device,
display;
camera;
A memory configured to store instructions; And
A processor, wherein the processor, when executing the instructions,
While guiding the path of the vehicle being moved, a plurality of images of the front of the vehicle being moved are acquired through the camera,
Based on at least some of the plurality of images, identify whether the vehicle can perform a turn in a driving lane of the vehicle in order to move along the path,
While identifying that the vehicle is unable to perform a turn in the driving lane, identifying a distance between the vehicle and a position on the path for which the turn of the vehicle is required,
Based on identifying that the identified distance is less than or equal to a specified distance, to display a notification through the display for guiding that a change in the driving lane of the vehicle is required for the turn,
Electronic device configured.
The method of claim 15, wherein the plurality of images,
A first image in a first frame and a second image in a second frame following the first frame,
The processor, when executing the instructions,
Identify lanes adjacent to the vehicle from the first image,
Determining a type of each of the lanes identified from the first image,
To verify the result of the determination based on a change in position between the lanes in the first image and the lanes in the second image,
Electronic device further configured.
The method of claim 16, wherein the processor, when executing the instructions,
Based on the type of each of the lanes, to identify a relative positional relationship between the driving lane among the plurality of lanes arranged along the route and the remaining lanes of the plurality of lanes,
Electronic device further configured.
The method of claim 17, wherein the processor, when executing the instructions,
Based on identifying that the vehicle being moved deviates from the driving lane, superimposing on the user interface displayed through the display to provide a guide of the route, guiding that the vehicle leaves the driving lane To display a notification for
Electronic device further configured.
The method of claim 18, wherein the display,
An electronic device further comprising a head-up display (HUD).
The method of claim 16, wherein the processor, when executing the instructions,
Acquiring a binary image converted from the first image,
To identify the lanes from the binary image,
Electronic device further configured.
The method of claim 20, wherein the processor, when executing the instructions,
Identify regions including the lanes identified from the binary image as regions of interest,
To determine a type of each of the lanes by performing binarization on the regions identified as the regions of interest,
Electronic device configured.

Images