KR102383425B1 - Electronic apparatus, control method of electronic apparatus, computer program and computer readable recording medium - Google Patents

Abstract

A method of controlling an electronic device is disclosed. The control method includes detecting a crosswalk from image data captured by a camera while driving a vehicle, generating an object representing the detected crosswalk, and outputting the generated object through augmented reality.

Description

Electronic devices, control methods for electronic devices, computer programs, and computer-readable recording media

The present invention relates to an electronic device, a method for controlling an electronic device, a computer program, and a computer-readable recording medium, and more particularly, to an electronic device for providing driving-related guidance to a user in augmented reality, a method for controlling an electronic device, and a computer program and to a computer-readable recording medium.

The most important thing when driving a vehicle is safe driving and prevention of traffic accidents. there is.

In addition, recently, devices such as a black box are located in a vehicle to store driving images of the vehicle and data transmitted from various sensors, thereby identifying the cause of the vehicle accident when the vehicle is in an accident. Portable terminals such as smartphones and tablets can also be equipped with a black box or a navigation application, and thus are being used as such vehicle devices.

However, the current utilization of driving images in such vehicle devices is low. More specifically, even when a driving image of a vehicle is acquired through a vision sensor such as a camera currently mounted on the vehicle, the electronic device of the vehicle simply displays and transmits it, or generates only simple surrounding notification information such as lane departure.

In addition, although a head-up display (HUD) or an augmented reality interface has been proposed as an electronic device for a vehicle that is newly emerging, the utilization of the driving image of the vehicle is limited to a simple display or simple notification information generation here.

The present invention has been devised in accordance with the above-mentioned necessity of the present invention, and an object of the present invention is to generate an object representing a crosswalk in a situation such as when a vehicle stops at an intersection while driving on a road, and output the object through augmented reality. To provide a control method, a computer program, and a computer-readable recording medium.

A method of controlling an electronic device according to an embodiment of the present invention for achieving the above object includes detecting a crosswalk from image data captured by a camera while driving a vehicle, and generating an object representing the detected crosswalk and outputting the generated object through augmented reality.

And, further comprising the step of determining whether the vehicle is in a stopped state, wherein the generating is a first object for recognizing that the crosswalk is located in front of the vehicle when it is determined that the vehicle is in a stopped state can create

The method may further include determining signal type information by using image data of a signal region portion of a traffic light among the image data, wherein the generating may include a state in which the vehicle is stopped in a state in which the signal type information is a stop signal. maintain, create a first object for recognizing that the crosswalk is located in front of the vehicle, and when the vehicle starts in a state where the signal type information is a stop signal, the crosswalk is located in front of the vehicle You can create a second object for warning that it is located in .

In addition, the first and second objects may be distinguished by different colors.

In addition, the first and second objects may be implemented in a form including an alpha channel related to color transparency, and the first and second objects may include a transparent region according to the alpha channel.

The method may further include determining whether a pedestrian exists in the crosswalk using the captured image data and generating an object indicating whether the pedestrian exists.

In addition, when the vehicle in front of the vehicle starts in a state in which a pedestrian is present in the crosswalk, the method may further include controlling not to perform a departure guidance of the vehicle ahead.

And, when the vehicle starts in a state in which a pedestrian is present in the crosswalk, the method may further include controlling to perform a guide to warn that a pedestrian is present in the crosswalk.

In addition, the display position of the first and second objects may be an area where a crosswalk is located in the augmented reality.

In addition, the outputting step may include calculating a camera parameter by performing calibration on the camera, and generating a virtual 3D (3-dimensional) space for the captured image of the camera based on the camera parameter. and locating the created object in the virtual 3D space.

On the other hand, in an electronic device according to an embodiment of the present invention for achieving the above object, when a crosswalk is detected from image data captured by a display unit for displaying a screen and a camera while driving a vehicle, the detected crosswalk and an object generating unit generating an object representing

Then, the control unit determines whether the vehicle is in a stopped state, and when the vehicle is determined to be in a stopped state, the object generating unit is configured to generate a first object for recognizing that the crosswalk is located in front of the vehicle. can be controlled

In addition, the control unit determines the signal type information by using the image data of the signal area portion of the traffic light among the image data, and when the vehicle maintains a stopped state in a state in which the signal type information is a stop signal, the crosswalk Creates a first object for recognizing that is located in front of the vehicle, and when the vehicle starts in a state where the signal type information is a stop signal, for warning that the crosswalk is located in front of the vehicle The object generator may be controlled to create a second object.

In addition, the first and second objects may be distinguished by different colors.

In addition, the first and second objects may be implemented in a form including an alpha channel related to color transparency, and the first and second objects may include a transparent region according to the alpha channel.

In addition, the controller may control the object generator to determine whether a pedestrian exists in the crosswalk by using the captured image data, and to generate an object indicating the presence of the pedestrian.

In addition, the controller may control not to perform the forward vehicle departure guidance when the vehicle in front of the vehicle starts in a state where a pedestrian is present in the crosswalk.

And, the control unit, when the vehicle starts in a state in which the pedestrian is present in the crosswalk, the step of controlling to perform a guide to warn the presence of the pedestrian in the crosswalk may further include.

In addition, the display position of the first and second objects may be an area where a crosswalk is located in the augmented reality.

In addition, the controller calculates a camera parameter by performing calibration on the camera, and generates a virtual 3D (3-dimensional) space for the captured image of the camera based on the camera parameter, and the virtual 3D It is possible to control to position the created object in space.

On the other hand, the computer program stored in the recording medium according to an embodiment of the present invention for achieving the above object is combined with an electronic device, the step of detecting a crosswalk from image data captured by a camera while the vehicle is driving, the The steps of generating an object representing the detected crosswalk and outputting the generated object through augmented reality may be executed.

Meanwhile, in a computer-readable recording medium storing a computer program for executing a control method of an electronic device according to an embodiment of the present invention for achieving the above object, the control method is captured by a camera while driving a vehicle It may include detecting a crosswalk from the image data, generating an object representing the detected crosswalk, and outputting the generated object through augmented reality.

According to various embodiments of the present invention described above, by dynamically expressing guidance information in an augmented reality technique in a section with a crosswalk, effective guidance and interest are induced to the driver, and at the same time, the driver's safe driving and convenience for the vehicle are improved. can be promoted

In addition, according to various embodiments of the present disclosure described above, it is possible to promote safe driving and convenience of the driver by performing related guidance according to the presence or absence of pedestrians in the crosswalk.

1 is a block diagram illustrating an electronic device according to an embodiment of the present invention.
2 is a block diagram specifically illustrating an augmented reality providing unit according to an embodiment of the present invention.
3 is a diagram for explaining a system network connected to an electronic device according to an embodiment of the present invention.
4 is a flowchart schematically illustrating a method for controlling an electronic device according to an embodiment of the present invention.
5 is a flowchart specifically illustrating a method for controlling an electronic device according to an embodiment of the present invention.
6 is a flowchart specifically illustrating a method of controlling an electronic device according to another embodiment of the present invention.
7 is a flowchart specifically illustrating a method of controlling an electronic device according to another embodiment of the present invention.
8 is a diagram illustrating an augmented reality screen displaying a crosswalk object according to an embodiment of the present invention.
9 is a diagram illustrating a texture image of a crosswalk object according to an embodiment of the present invention.
10 is a diagram illustrating an augmented reality screen displaying a notification object of a pedestrian according to an embodiment of the present invention.
11 is a diagram illustrating an implementation form when a camera and an electronic device are separated according to an embodiment of the present invention.
12 is a diagram illustrating an implementation form when a camera and an electronic device are integrated according to an embodiment of the present invention.
13 is a diagram illustrating an implementation form using a head-up display (HUD) and an electronic device according to an embodiment of the present invention.

The following is merely illustrative of the principles of the invention. Therefore, those skilled in the art will be able to devise various devices that, although not explicitly described or shown herein, embody the principles of the present invention and are included within the spirit and scope of the present invention. In addition, all conditional terms and examples listed herein are, in principle, expressly intended only for the purpose of understanding the concept of the present invention, and it should be understood that they are not limited to the specifically enumerated embodiments and states as such. do.

Moreover, it is to be understood that all detailed description reciting specific embodiments, as well as principles, aspects, and embodiments of the invention, are intended to include structural and functional equivalents of such matters. It should also be understood that such equivalents include not only currently known equivalents, but also equivalents developed in the future, i.e., all devices invented to perform the same function, regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing conceptual views of illustrative circuitry embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc. may be tangibly embodied on computer-readable media and be understood to represent various processes performed by a computer or processor, whether or not a computer or processor is explicitly shown. should be

The functions of the various elements shown in the figures including a processor or functional blocks represented by similar concepts may be provided by the use of dedicated hardware as well as hardware having the ability to execute software in association with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or a plurality of separate processors, some of which may be shared.

In addition, the clear use of terms presented as processor, control, or similar concepts should not be construed as exclusively referring to hardware having the ability to execute software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM (RAM) and non-volatile memory. Other common hardware may also be included.

In the claims of the present specification, a component expressed as a means for performing the function described in the detailed description includes, for example, a combination of circuit elements that perform the function or software in any form including firmware/microcode, etc. It is intended to include all methods of performing the functions of the device, coupled with suitable circuitry for executing the software to perform the functions. Since the present invention defined by these claims is combined with the functions provided by the various enumerated means and in a manner required by the claims, any means capable of providing the functions are equivalent to those contemplated from the present specification. should be understood as

The above objects, features and advantages will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in the description of the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating an electronic device according to an embodiment of the present invention. Referring to FIG. 1 , the electronic device 100 includes a storage unit 110 , an input unit 120 , an output unit 130 , a crosswalk detection unit 140 , a signal type information determination unit 150 , and a driving state determination unit ( 155 ), the augmented reality providing unit 160 , the control unit 170 , the communication unit 180 , the sensing unit 190 , and all or a part of the power supply unit 195 .

Here, the electronic device 100 includes a smart phone, a tablet computer, a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), smart glasses, which can provide driving-related guidance to a driver of a vehicle in a driving state; It may be implemented in various devices such as project glasses, navigation, black-box, and the like, and may be provided in a vehicle.

Here, the driving state of the vehicle may include various states in which the vehicle is being driven by the driver, such as a stopped state of the vehicle, a running state of the vehicle, and a parking state of the vehicle.

The driving-related guidance may include various guidance for assisting the driver in driving, such as route guidance, lane departure guidance, forward vehicle departure guidance, traffic light change guidance, forward collision avoidance guidance, lane change guidance, lane guidance, and the like.

Here, the route guidance includes augmented reality route guidance that performs route guidance by combining various information such as the location and direction of the user with an image taken in front of the vehicle in motion, and 2D (2-Dimensional) or 3D (3-D) 2D (2-Dimensional) or 3D (3-Dimensional) route guidance for performing route guidance by combining various types of information such as a user's location and direction with map data of Dimensional) may be included. Here, the route guidance may be interpreted as a concept including route guidance when the user moves while walking or running as well as when the user rides and drives in a vehicle.

In addition, the lane departure guidance may be guiding whether the driving vehicle has departed from a lane.

In addition, the front vehicle departure guidance may indicate whether a vehicle located in front of a stopped vehicle departs.

Also, the traffic light change guidance may be to guide whether a traffic light positioned in front of a stopped vehicle is changed. For example, when a red light indicating a stop signal is turned on to a blue light indicating a departure signal, this may be guiding.

In addition, the front vehicle collision avoidance guidance may be to guide the vehicle in front of a stopped or driving vehicle to prevent a collision with the vehicle in front when the distance to the vehicle located in front is within a predetermined distance.

In addition, the lane change guidance may indicate a change from a lane in which the vehicle is located to another lane in order to guide a route to a destination.

Also, the lane guidance may indicate a lane in which the vehicle is currently located.

A driving-related image, such as a front image of a vehicle that enables the provision of various guidance, may be captured by a camera mounted on the vehicle. Here, the camera may be a camera integrally formed with the electronic device 100 mounted on the vehicle to photograph the front of the vehicle. In this case, the camera may be integrally formed with a smart phone, a navigation system, or a black box, and the electronic device 100 may receive an image captured by the integrally formed camera.

As another example, the camera may be mounted on a vehicle separately from the electronic device 100 to photograph the front of the vehicle. In this case, the camera may be a separate black box mounted toward the front of the vehicle, and the electronic device 100 receives a photographed image through wired/wireless communication with the separately mounted black box, or receives a photographed image of the black box. When the storage medium to be stored is inserted into the electronic device 100 , the electronic device 100 may receive a captured image.

Hereinafter, the electronic device 100 according to an embodiment of the present invention will be described in more detail based on the above description.

The storage unit 110 functions to store various data and applications required for the operation of the electronic device 100 . In particular, the storage 110 may store data necessary for the operation of the electronic device 100 , for example, an OS, a route search application, map data, and the like. Also, the storage 110 may store data generated by the operation of the electronic device 100 , for example, searched route data, a received image, and the like.

Here, the storage unit 110 includes a random access memory (RAM), a flash memory, a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a register, a hard disk, a removable disk, and a memory. It may be implemented as a storage device in a built-in type such as a card or a Universal Subscriber Identity Module (USIM), as well as a storage device in a removable type such as a USB memory.

The input unit 120 functions to convert a physical input from the outside of the electronic device 100 into a specific electrical signal. Here, the input unit 120 may include all or part of the user input unit 121 and the microphone unit 123 .

The user input unit 121 may receive a user input such as a touch or a push operation. Here, the user input unit 120 may be implemented using at least one of various button shapes, a touch sensor for receiving a touch input, and a proximity sensor for receiving an approaching motion.

The microphone unit 123 may receive a user's voice and a sound generated inside or outside the vehicle.

The output unit 130 is a device that outputs data of the electronic device 100 . Here, the output unit 130 may include all or part of the display unit 131 and the audio output unit 133 .

The display unit 131 is a device that outputs data that can be visually recognized by the electronic device 100 . The display unit 131 may be implemented as a display unit provided on the front of the housing of the electronic device 100 . Also, the display unit 131 may be integrally formed with the electronic device 100 to output visual recognition data, and may be installed separately from the electronic device 100 such as a Head Up Display (HUD) to output visual recognition data. You may.

The audio output unit 133 is a device for outputting data that can be recognized aurally by the electronic device 100 . The audio output unit 133 may be implemented as a speaker expressing sound of data to be notified to the user of the electronic device 100 .

The communication unit 180 may be provided for the electronic device 100 to communicate with other devices. The communication unit 180 includes all or part of the location data unit 181 , the wireless Internet unit 183 , the broadcast transceiver unit 185 , the mobile communication unit 186 , the short-distance communication unit 187 , and the wired communication unit 189 . may include

The location data unit 181 is a device for acquiring location data through a Global Navigation Satellite System (GNSS). GNSS refers to a navigation system capable of calculating the position of a receiving terminal using a radio signal received from an artificial satellite. Specific examples of GNSS include GPS (Global Positioning System), Galileo, GLONASS (Global Orbiting Navigational Satellite System), COMPASS, IRNSS (Indian Regional Navigational Satellite System), QZSS (Quasi-Zenith Satellite System), etc. can The location data unit 181 of the electronic device 100 according to an embodiment of the present invention may obtain location data by receiving a GNSS signal serviced in a region where the electronic device 100 is used.

The wireless Internet unit 183 is a device for acquiring or transmitting data by accessing the wireless Internet. The wireless Internet accessible through the wireless Internet unit 183 may be a wireless LAN (WLAN), a wireless broadband (Wibro), a world interoperability for microwave access (Wimax), a high speed downlink packet access (HSDPA), or the like.

The broadcast transceiver 185 is a device for transmitting and receiving broadcast signals through various broadcast systems. Broadcast systems that can be transmitted and received through the broadcast transceiver 185 include Digital Multimedia Broadcasting Terrestrial (DMBT), Digital Multimedia Broadcasting Satellite (DMBS), Media Forward Link Only (MediaFLO), Digital Video Broadcast Handheld (DVBH), and ISDBT (Digital Multimedia Broadcasting Terrestrial). Integrated Services Digital Broadcast Terrestrial) or the like. The broadcast signal transmitted and received through the broadcast transceiver 185 may include traffic data, life data, and the like.

The mobile communication unit 186 may access and communicate with a mobile communication network according to various mobile communication standards such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE).

The short-distance communication unit 187 is a device for short-distance communication. The short-range communication unit 187, as described above, Bluetooth (Bluetooth), RFID (Radio Frequency Identification), infrared communication (IrDA, Infrared Data Association), UWB (Ultra WideBand), ZigBee, NFC (Near Field Communication), Wi -Communication is possible through Fi (Wireless-Fidelity), etc.

The wired communication unit 189 is an interface device that can connect the electronic device 100 to another device by wire. The wired communication unit 119 may be a USB module capable of communicating through a USB port.

The communication unit 180 includes at least one of a location data unit 181 , a wireless Internet unit 183 , a broadcast transceiver unit 185 , a mobile communication unit 186 , a short-distance communication unit 187 , and a wired communication unit 189 . can be used to communicate with other devices.

For example, when the electronic device 100 does not include a camera function, an image captured by a vehicle camera such as a black box may be received using at least one of the short-range communication unit 187 and the wired communication unit 189 .

As another example, when communicating with a plurality of devices, one may communicate with the short-range communication unit 187 and the other may communicate through the wired communication unit 119 .

The sensing unit 190 is a device capable of detecting the current state of the electronic device 100 . The sensing unit 190 may include all or a part of the motion sensing unit 191 and the light sensing unit 193 .

The motion sensing unit 191 may detect a motion of the electronic device 100 in a three-dimensional space. The motion sensing unit 191 may include a 3-axis geomagnetic sensor and a 3-axis acceleration sensor. A more accurate trajectory of the vehicle to which the electronic device 100 is attached may be calculated by combining the motion data acquired through the motion sensing unit 191 with the position data acquired through the position data unit 181 .

The light sensing unit 193 is a device for measuring peripheral illuminance of the electronic device 100 . By using the illuminance data obtained through the light sensing unit 193, the brightness of the display unit 131 may be changed to correspond to the ambient brightness.

The power supply unit 195 is a device that supplies power necessary for the operation of the electronic device 100 or the operation of another device connected to the electronic device 100 . The power supply unit 195 may be a device that receives power from an external power source such as a battery built in the electronic device 100 or a vehicle. In addition, the power supply unit 195 may be implemented as a wired communication module 119 or a wirelessly supplied device depending on the form of receiving power.

The crosswalk detection unit 140 may detect a crosswalk from image data captured by the camera. Specifically, the crosswalk detection unit 140 may determine the region of interest including the crosswalk in the image data by using a vanishing point of the captured image. Here, the vanishing point may be determined by extracting a lane from photographing data captured by a camera while the vehicle is driving, and calculating a point at which the extracted lane crosses each other by extending it. Also, since the crosswalk is formed on the road at the lower end of the vanishing point, the crosswalk detection unit 140 may determine the lower end of the vanishing point as the region of interest.

Meanwhile, the crosswalk detection unit 140 may perform image processing on the determined image data of the ROI to determine whether a crosswalk is located in the ROI.

However, unlike the above-described example, according to another embodiment of the present invention, the crosswalk detection unit 140 may detect a crosswalk without using camera-captured image data. For example, the crosswalk detection unit 140 determines whether the current vehicle is located at the crosswalk by using the current location information of the vehicle determined through the location data unit 181 and the crosswalk location of the map data pre-stored in the storage unit 110 . can determine whether Alternatively, for more accurate determination, the crosswalk detection unit 140 may determine whether the current vehicle is located at the crosswalk in consideration of camera-captured image data, map data, and location data.

The signal type information determining unit 150 may determine the signal type information by using the image data of the signal area portion of the traffic light among the image data captured by the camera. Specifically, the signal type information determining unit 150 determines a region of interest including a traffic light from image data using a vanishing point of the captured image, and converts the determined image data of the region of interest based on a preset pixel value. to generate ROI image data. Here, the traffic light is located in the upper region of the vanishing point, and the signal type information determining unit 150 may determine the upper region of the determined vanishing point as the region of interest.

Meanwhile, the signal type information determining unit 150 may detect image data of a signal region portion of a traffic light from the determined region of interest image data. In addition, the signal type information determining unit 150 may determine the signal type information based on the image data of the signal area portion.

Here, the signal type information is information for identifying each of a plurality of signals that can be displayed in a traffic light, and may include stop signal information, straight forward signal information, a left turn signal, a right turn signal, and the like.

The driving state determining unit 155 may determine the driving state of the vehicle, for example, whether the vehicle is in a stopped state, in a driving state, or in a parked state. Specifically, the driving state determining unit 155 may determine whether the vehicle is in a stopped state by using image data captured by the camera. More specifically, the driving state determining unit 155 generates gray image data for the image data, and sequentially compares each of a plurality of frames included in the generated gray image data in chronological order to determine whether the vehicle is in a stopped state. can do. However, the present invention is not limited thereto, and the driving state determining unit 155 determines whether the vehicle is in a stopped state based on the signal detected by the sensing unit 190 and the motion data obtained from the location data unit 181 , or The driving state of the vehicle may be determined based on real-time speed information of the vehicle obtained using CAN communication (Controller Area Network).

Meanwhile, the electronic device 100 according to an embodiment of the present invention may include the augmented reality providing unit 160 that provides the augmented reality view mode, which will be described in detail with reference to FIG. 2 .

2 is a block diagram specifically illustrating the augmented reality providing unit 160 according to an embodiment of the present invention. Referring to FIG. 2 , the augmented reality providing unit 160 may include all or a part of a calibrator 161 , a 3D space generator 162 , an object generator 163 , and a mapping unit 164 .

The calibrator 161 may perform calibration for estimating a camera parameter corresponding to a camera from a captured image captured by the camera. Here, the camera parameter may be a parameter constituting the camera matrix, which is information indicating the relationship between the actual space and the photograph.

The 3D space generator 162 may generate a virtual 3D space based on a captured image captured by the camera. Specifically, the 3D space generator 162 obtains depth information from an image captured by the camera based on the camera parameter estimated by the calibrator 161, and based on the obtained depth information and the captured image You can create a virtual 3D space.

The object generator 163 may generate an object for guidance in the augmented reality, for example, a route guidance object, a lane change guidance object, a lane departure guidance object, a crosswalk object, a pedestrian guidance object, and the like. Here, the object may be implemented as a 3D object, a texture image, or an art line.

The mapping unit 164 may map the object generated by the object generating unit 163 to the virtual 3D space generated by the 3D space generating unit 162 .

Meanwhile, the controller 170 controls the overall operation of the electronic device 100 . Specifically, the control unit 170 includes a storage unit 110 , an input unit 120 , an output unit 130 , a crosswalk detection unit 140 , a signal type information determination unit 150 , a driving state determination unit 155 , and augmented reality. All or part of the providing unit 160 , the communication unit 180 , the sensing unit 190 , and the power supply unit 195 may be controlled.

In particular, when a crosswalk is detected from image data captured by the camera while the vehicle is driving, the controller 170 controls the object generator 163 to generate an object representing the detected crosswalk, and converts the created object into augmented reality. It can be controlled to output through .

For example, when the driving state determination unit 155 determines that the vehicle is in a stopped state, and the crosswalk detection unit 140 determines that there is a crosswalk in front of the vehicle, the controller 170 controls the first crosswalk indicating the crosswalk. The object generator 163 may be controlled to create an object. In addition, the controller 170 may control to output the created object through augmented reality. Here, the first object may be an object for recognizing to the driver that a crosswalk exists in front of the vehicle.

As another example, the determination result of the driving state determination unit 155 indicates that the vehicle is in a stopped state, the determination result of the signal type information determination unit 150 indicates a stop signal, and the determination result of the crosswalk detection unit 140 indicates that a crosswalk is in front of the vehicle. If present, the controller 170 may control the object generator 163 to generate a first object representing a crosswalk while the vehicle maintains a stopped state. In addition, the controller 170 may control to output the generated first object through augmented reality. Here, the first object may be an object for recognizing to the driver that a crosswalk exists in front of the vehicle.

However, when the vehicle moves in a stopped state among the stop signals, the controller 170 may control the object generator 163 to generate the second object. In addition, the controller 170 may control to output the generated second object through augmented reality. Here, the second object may be an object for warning the driver that a crosswalk exists in front of the vehicle.

Here, the first object for making the driver recognize that the crosswalk exists in front of the vehicle and the second object for warning the driver that the crosswalk exists in front of the vehicle may be distinguished from each other.

Specifically, the first and second objects may be distinguished from each other by different colors. For example, the first object may be implemented in a white color similar to the color of a crosswalk in the real world and displayed on the augmented reality screen, and the second object may be displayed in a color for recognizing that the driver is in a dangerous state, for example, red. It may be implemented and displayed on the augmented reality screen.

In addition, the first and second objects may be implemented in a form including an alpha channel related to color transparency, for example, RGBA (Red, Green, Blue, Alpha). In this case, the first object according to the alpha channel , the second object may include a transparent area. That is, the first and second objects may be implemented by including a color-expressing area and a transparent area to correspond to a crosswalk in the real world.

In addition, the display position of the first and second objects may be an area where a crosswalk is located in the augmented reality. For example, when the crosswalk detection unit 140 detects a crosswalk from the image data captured by the camera, the controller 170 controls the first and second objects at positions in the augmented reality screen corresponding to the position where the crosswalk is detected. The mapping unit 164 may be controlled to express . According to an embodiment of the present invention, the object representing the crosswalk may be displayed as if it is located on the crosswalk on the augmented reality screen, and accordingly, guidance may be provided to the driver in a more intuitive way.

Meanwhile, according to an embodiment of the present invention, it is possible to determine whether a pedestrian is present in a crosswalk and perform a corresponding notification.

Specifically, the controller 170 may control the object generator 163 to determine whether a pedestrian exists in the crosswalk by using the captured image data, and to generate a third object indicating the presence of the pedestrian. In addition, the controller 170 may control the generated third object to be output through augmented reality. Here, the third object may be an object for recognizing the presence of a pedestrian in the crosswalk to the driver.

In addition, when the vehicle in front of the host vehicle starts while there is a pedestrian in the crosswalk, the controller 170 may control not to guide the departure of the vehicle ahead. That is, if the forward vehicle departure guidance is performed in a state where pedestrians are present in the crosswalk, a collision between the own vehicle and the pedestrian may occur. In this case, the controller 170 may control not to perform the forward vehicle departure guidance.

In addition, when the vehicle starts from a stopped state in a state in which a pedestrian is present in the crosswalk, the control unit 170 may control to perform guidance to warn that a pedestrian is present in the crosswalk. That is, when the own vehicle starts from a stopped state in the presence of pedestrians in the crosswalk, collisions between the own vehicle and the pedestrian may occur. In this case, the controller 170 may control to perform a guidance that the own vehicle should not depart.

Meanwhile, according to the above-described examples, it has been described as an example that the guidance of a crosswalk, a pedestrian, etc. is displayed in the form of an image on the augmented reality screen, but the present invention is not limited thereto. Accordingly, according to another embodiment of the present invention, the control unit 170 may control the audio output unit 133 to output the guidance as voice.

3 is a diagram for explaining a system network connected to an electronic device according to an embodiment of the present invention. Referring to FIG. 3 , the electronic device 100 according to an embodiment of the present invention may be implemented as various devices provided in a vehicle, such as a navigation device, a black box, a smart phone, or other device for providing an augmented reality interface for a vehicle. It can connect to a communication network and other electronic devices 61 to 64 .

In addition, the electronic device 100 may calculate the current location and the current time zone by interworking with the GPS module according to the radio signal received from the artificial satellite 20 .

Each of the satellites 20 may transmit an L-band frequency having a different frequency band. The electronic device 100 may calculate the current location based on the time it takes for the L-band frequency transmitted from each satellite 20 to reach the electronic device 100 .

Meanwhile, the electronic device 100 may wirelessly access the network 30 through the control station 40 , ACR, the base station 50 , RAS, etc. through the communication unit 180 . When the electronic device 100 is connected to the network 30 , it may also indirectly connect with other electronic devices 61 and 62 connected to the network 30 to exchange data.

Meanwhile, the electronic device 100 may indirectly access the network 30 through another device 63 having a communication function. For example, when the electronic device 100 does not include a module that can connect to the network 30 , it can communicate with another device 63 having a communication function through a short-range communication module or the like.

4 is a flowchart schematically illustrating a method for controlling an electronic device according to an embodiment of the present invention. Referring to FIG. 4 , the electronic device 100 may detect a crosswalk from image data captured by a camera while driving a vehicle ( S101 ). Here, the camera may be a camera integrally formed with the electronic device 100 mounted on the vehicle to photograph the front of the vehicle, or the camera may be a separate black box mounted toward the front of the vehicle.

Then, the electronic device 100 may create an object representing the detected crosswalk (S102). Here, the object may be implemented as a 3D object, a texture image, or an art line.

Then, the electronic device 100 may output the generated object through augmented reality (S103). Here, the step of outputting (S103) is a step of performing calibration on the camera to calculate camera parameters, generating a virtual 3D (3-dimensional) space for the captured image of the camera based on the camera parameters, and It may include positioning the created object in the virtual 3D space.

Hereinafter, a method of controlling the electronic device 100 will be described in more detail with reference to FIGS. 5 to 7 .

5 is a flowchart specifically illustrating a method for controlling an electronic device according to an embodiment of the present invention. Referring to FIG. 5 , the electronic device 100 may determine whether the vehicle is in a stopped state ( S201 ). Here, whether the vehicle is in a stopped state may be performed using the above-described driving state determining unit 155 .

When the vehicle is in a stopped state, the electronic device 100 may detect a crosswalk from image data captured by a camera while the vehicle is driving ( S202 ). Here, the detection of the crosswalk may be performed using the above-described crosswalk detection unit 140 .

When a crosswalk is detected, the electronic device 100 may generate a first object for recognizing that the crosswalk is located in front of the vehicle ( S203 ).

Then, the electronic device 100 may output the generated first object through augmented reality (S204). In this case, the controller 170 may control the mapping unit 164 so that the generated first object is displayed in proximity to the own vehicle on the augmented reality screen.

Accordingly, the first object may be displayed in proximity to the front of the own vehicle in the road area of the augmented reality screen, and the driver may easily recognize that a crosswalk exists near the front of the own vehicle.

6 is a flowchart specifically illustrating a method of controlling an electronic device according to another embodiment of the present invention. Referring to FIG. 6 , the electronic device 100 may determine whether the vehicle is in a stopped state ( S301 ). Here, whether the vehicle is in a stopped state may be performed using the above-described driving state determining unit 155 .

When it is determined that the vehicle is in a stopped state, the electronic device 100 may determine signal type information by using image data of a signal region portion of a traffic light among the image data ( S302 ). Here, the signal type information determination may be performed using the above-described signal type information determination unit 150 .

When it is determined that the signal type information is a stop signal, the electronic device 100 may detect a crosswalk from image data captured by the camera while the vehicle is being driven ( S303 ). Here, the detection of the crosswalk may be performed using the above-described crosswalk detection unit 140 .

In addition, when the vehicle maintains a stopped state in a state in which the signal type information is a stop signal, the electronic device 100 may generate a first object for recognizing that a crosswalk is located in front of the vehicle ( S304 ).

Then, when the vehicle starts in a state in which the signal type information is a stop signal, the electronic device 100 may create a second object for warning that a crosswalk is located in front of the vehicle ( S305 ).

Then, the electronic device 100 may output the generated object through augmented reality (S306). Here, the generated first and second objects may be displayed in different forms in order to provide different guidance to the driver.

Accordingly, the driver can not only easily recognize that a crosswalk exists near the front of the own vehicle, but can also easily recognize whether the vehicle can start.

7 is a flowchart specifically illustrating a method of controlling an electronic device according to another embodiment of the present invention. Referring to FIG. 7 , the electronic device 100 may determine whether the vehicle is in a stopped state ( S401 ). Here, whether the vehicle is in a stopped state may be performed using the above-described driving state determining unit 155 .

If it is determined that the vehicle is stopped, the electronic device 100 may detect a crosswalk from image data captured by the camera while the vehicle is driving ( S402 ). Here, the detection of the crosswalk may be performed using the above-described crosswalk detection unit 140 .

When a crosswalk is detected, the electronic device 100 may determine whether a pedestrian exists in the crosswalk using the captured image data (S403).

Then, the electronic device 100 may create an object indicating whether a pedestrian is present (S404).

Then, the electronic device 100 may output the generated object through augmented reality (S405). Accordingly, the driver can easily recognize that a pedestrian is walking near the front of the own vehicle.

Meanwhile, according to the present invention, when the vehicle in front of the own vehicle starts while there is a pedestrian in the crosswalk, the electronic device 100 may control not to perform the forward vehicle departure guidance.

Also, according to the present invention, when the own vehicle starts from a stop state in a state in which a pedestrian is present in the crosswalk, the electronic device 100 may control to perform a guide to warn that a pedestrian is present in the crosswalk.

8 is a diagram illustrating an augmented reality screen displaying a crosswalk object according to an embodiment of the present invention. 8A is a diagram illustrating an augmented reality screen when a vehicle is stopped behind a crosswalk among the stop signals 810 . Referring to FIG. 8A , the electronic device 100 may generate a first object 801 representing a crosswalk located in front of a vehicle, and output the generated first object 801 through augmented reality. there is. Here, the first object 801 may be displayed in proximity to the front of the own vehicle among the road regions of the augmented reality screen, and accordingly, the driver may easily recognize that a crosswalk exists near the front of the own vehicle.

Meanwhile, FIG. 8B is a diagram illustrating an augmented reality screen when the vehicle moves in a stopped state among the stop signals 810 . Referring to FIG. 8B , the electronic device 100 creates a second object 802 for warning the driver that a crosswalk exists in front of the vehicle, and augments the generated second object 802 . It can be output through reality. Here, the second object 802 may be displayed in proximity to the front of the own vehicle in the road area of the augmented reality screen, and may be implemented in different colors to be distinguished from the first object 801 . Accordingly, the driver can easily recognize that the current vehicle should not start.

Meanwhile, the above-described first and second objects may be implemented as texture images and expressed through augmented reality. This will be described in detail with reference to FIG. 9 .

Referring to FIG. 9 , the first object 801 for recognizing to the driver that there is a crosswalk in front of the vehicle may be implemented in white color similar to the color of the crosswalk in the real world, and the crosswalk in front of the vehicle The second object 802 for warning the driver of the existence may be implemented in a color for recognizing that a dangerous state is present, for example, red, and displayed on the augmented reality screen.

Here, the first object 801 and the second object 802 are implemented to include areas 801-1 and 802-1 and transparent areas 801-2 and 802-2 for expressing colors so as to correspond to a crosswalk in the real world. can be In this case, transparency of the transparent regions 801 - 2 and 802 - 2 may be adjusted by, for example, changing the value A, which is an alpha channel value of RGBA (Red, Green, Blue, Alpha). In an embodiment of the present invention, the alpha channel value has a value between 0.0 (completely transparent) and 1.0 (completely opaque). In the embodiment of the present invention, RGBA values are used as values for color display, but HSLA (Hue, Saturation, Lightness, Alpha), which is another color display unit for displaying an alpha channel value indicating transparency, may be used.

10 is a diagram illustrating an augmented reality screen displaying a notification object of a pedestrian according to an embodiment of the present invention. Referring to FIG. 10 , the electronic device 100 creates a third object 1001 for guiding a pedestrian 1002 when there is a pedestrian in a crosswalk, and uses the generated third object 1001 through augmented reality. output can be controlled. Accordingly, the driver can recognize that a pedestrian is walking in front of the current vehicle and can easily recognize that the current vehicle must not start.

11 is a diagram illustrating an implementation form when a navigation device according to an embodiment of the present invention does not include a photographing unit. Referring to FIG. 11 , a vehicle black box 200 provided separately from the vehicle navigation device 100 may configure a system according to an embodiment of the present invention using a wired/wireless communication method.

The vehicle navigation device 100 may include a display unit 131 provided on a front surface of the navigation housing 191 , a navigation operation key 121 , and a navigation microphone 123 .

The vehicle black box 200 may include a black box camera 222 , a black box microphone 224 , and an attachment unit 281 .

12 is a diagram illustrating an implementation form when a navigation device according to an embodiment of the present invention includes a photographing unit. Referring to FIG. 12 , when the navigation device 100 includes the photographing unit 125 , the user may use the photographing unit 125 of the navigation device 100 to photograph the front of the vehicle and display the navigation device 100 . The navigation device 100 may be mounted so that the portion is recognizable by the user. Accordingly, it is possible to implement a system according to an embodiment of the present invention.

13 is a diagram illustrating an implementation form using a head-up display (HUD) according to an embodiment of the present invention. Referring to FIG. 13 , the HUD may display the augmented reality guidance screen on the heads-up display through wired/wireless communication with other devices.

For example, the augmented reality may be provided through a HUD using a vehicle windshield or an image overlay using a separate image output device, etc. can create Through this, an augmented reality navigation system or a vehicle infotainment system may be implemented.

Meanwhile, the above-described control method according to various embodiments of the present invention may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing the server or device in which the program is stored.

In addition, the above-described control method according to various embodiments of the present invention may be implemented as a program and stored in various non-transitory computer readable media to be provided. The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: electronic device 110: storage unit
120: input unit 130: output unit
140: crosswalk detection unit 150: signal type information determination unit
155: driving state determination unit 160: augmented reality providing unit 170: controller 180: communication unit
190: sensing unit 195: power unit

Claims (22)

A method for controlling an electronic device, comprising:
detecting a crosswalk from image data captured by a camera while the vehicle is being driven;
determining signal type information by using image data of a signal region of a traffic light among the image data;
generating a first object for recognizing that the crosswalk is located in front of the vehicle when the vehicle maintains a stopped state in a state in which the signal type information is a stop signal;
generating a second object for warning that the crosswalk is located in front of the vehicle when the vehicle starts from a stopped state in a state in which the signal type information is a stop signal; and
and outputting the first object or the second object through augmented reality. According to claim 1,
Further comprising; determining whether the vehicle is in a stopped state;
The generating step is
When it is determined that the vehicle is in a stopped state, the first object for recognizing that the crosswalk is located in front of the vehicle is generated. delete According to claim 1,
The control method, characterized in that the first and second objects are distinguished by different colors. According to claim 1,
The first and second objects are implemented in a form including an alpha channel related to color transparency,
The control method according to claim 1, wherein the first and second objects include a transparent region according to the alpha channel. According to claim 1,
determining whether there is a pedestrian in the crosswalk by using the photographed image data; and
The control method further comprising; generating an object indicating the presence of the pedestrian. 7. The method of claim 6,
When the vehicle in front of the vehicle starts in a state where a pedestrian is present in the crosswalk, controlling the vehicle in front to not perform departure guidance. 7. The method of claim 6,
When the vehicle starts in a state in which a pedestrian is present in the crosswalk, controlling to perform a guide to warn that a pedestrian is present in the crosswalk; Control method comprising a further comprising. According to claim 1,
The display positions of the first and second objects are,
Control method, characterized in that the area where the crosswalk is located in the augmented reality. According to claim 1,
The output step is
calculating camera parameters by performing calibration on the camera;
generating a virtual 3D (3-dimensional) space for the captured image of the camera based on the camera parameter; and
and locating the created object in the virtual 3D space. In an electronic device,
a display unit for displaying a screen;
When a crosswalk is detected from the image data captured by the camera while the vehicle is driving, an object generator for generating an object representing the detected crosswalk; and
Containing a; control unit for controlling the display unit to output the generated object through augmented reality;
The control unit is
Determining signal type information by using the image data of the signal area part of the traffic light among the image data,
When the vehicle maintains a stopped state in a state in which the signal type information is a stop signal, a first object is created for recognizing that the crosswalk is located in front of the vehicle, and the signal type information is a stop signal When the vehicle starts in a stopped state, the electronic device according to claim 1, wherein the object generating unit is controlled to generate a second object for warning that the crosswalk is located in front of the vehicle. 12. The method of claim 11,
The control unit is
Determining whether the vehicle is in a stopped state,
When it is determined that the vehicle is in a stopped state, the electronic device of claim 1, wherein the object generating unit is controlled to generate a first object for recognizing that the crosswalk is located in front of the vehicle. delete 12. The method of claim 11,
The electronic device of claim 1, wherein the first and second objects are distinguished by different colors. 12. The method of claim 11,
The first and second objects are implemented in a form including an alpha channel related to color transparency,
The electronic device of claim 1, wherein the first and second objects include a transparent region according to the alpha channel. 12. The method of claim 11,
The control unit is
The electronic device of claim 1, wherein the object generator is configured to determine whether a pedestrian exists in the crosswalk by using the photographed image data, and to generate an object indicating whether or not the pedestrian is present. 17. The method of claim 16,
The control unit is
The electronic device of claim 1, wherein when a vehicle in front of the vehicle starts in a state in which a pedestrian is present in the crosswalk, the electronic device is configured to control not to perform a departure guidance of the vehicle ahead. 17. The method of claim 16,
The control unit is
The electronic device of claim 1, further comprising: when the vehicle starts in a state in which a pedestrian is present in the crosswalk, controlling to perform a guide to warn that a pedestrian is present in the crosswalk. 12. The method of claim 11,
The display positions of the first and second objects are,
The electronic device, characterized in that the area where the crosswalk is located in the augmented reality. 12. The method of claim 11,
The control unit is
A camera parameter is calculated by performing calibration on the camera, a virtual 3D (3-dimensional) space is generated for the captured image of the camera based on the camera parameter, and the generated object is located in the virtual 3D space. Electronic device, characterized in that the control to position. combined with electronic devices,
detecting a crosswalk from image data captured by a camera while the vehicle is being driven;
determining signal type information by using image data of a signal region of a traffic light among the image data;
generating a first object for recognizing that the crosswalk is located in front of the vehicle when the vehicle maintains a stopped state in a state in which the signal type information is a stop signal;
generating a second object for warning that the crosswalk is located in front of the vehicle when the vehicle starts from a stopped state in a state in which the signal type information is a stop signal; and
A computer program stored in a recording medium for executing; outputting the first object or the second object through augmented reality. A computer-readable recording medium storing a computer program for executing a control method of an electronic device,
detecting a crosswalk from image data captured by a camera while the vehicle is being driven;
determining signal type information by using image data of a signal region of a traffic light among the image data;
generating a first object for recognizing that the crosswalk is located in front of the vehicle when the vehicle maintains a stopped state in a state in which the signal type information is a stop signal;
generating a second object for warning that the crosswalk is located in front of the vehicle when the vehicle starts from a stopped state in a state in which the signal type information is a stop signal; and
A computer-readable recording medium comprising a; outputting the first object or the second object through augmented reality.

Images