KR102545199B1 - Electronic apparatus and control method thereof - Google Patents

Abstract

An electronic device and a control method thereof are disclosed. According to the present invention, a control method of an electronic device includes the steps of receiving image data, obtaining feature information indicating an object from image data using a plurality of filters, and acquiring through at least two or more filters among the plurality of filters. The method includes detecting an object included in image data using feature information and providing information on the detected object. Accordingly, the electronic device can more accurately detect surrounding vehicles and pedestrians not only in normal road conditions but also in road conditions at night and in bad weather.

Description

Electronic apparatus and control method thereof

The present invention relates to an electronic device and a control method thereof, and more particularly, to an electronic device for detecting an object from image data input to the electronic device and a control method thereof.

With the recent development of autonomous driving-related technologies, electronic devices such as a head-up display mounted in a vehicle to provide driving information or a black box mounted in a vehicle to record the driving situation in real time monitor the state of the vehicle and monitor the result. provides In addition, the electronic device provides driving information including road conditions on which the vehicle is driving. Specifically, the electronic device photographs the road conditions on which the vehicle is driving, detects objects including surrounding vehicles and pedestrians around the vehicle from the captured image data, and detects type information, location information, and movement information of the detected object. Provides driving information including at least one of

Accordingly, the driver can prevent the risk of an accident between vehicles or between a vehicle and a pedestrian by checking the distance between vehicles in front, a blind spot risk factor, and whether there is a pedestrian crossing the road through various driving information provided through the electronic device.

However, such a conventional electronic device is a road condition under conditions where a plurality of nearby vehicles are parked or a plurality of pedestrians cross a vehicle road together, a road condition in bad weather (snow, rain, fog), and a night time zone. In a road situation, there arises a problem of not detecting surrounding vehicles and pedestrians from captured image data.

The present invention was conceived to solve the above-mentioned problems and to respond to the above-mentioned technical development request, and the present invention is to facilitate detection of surrounding vehicles and pedestrians not only in general road conditions but also in night and bad weather road conditions. is aimed at

A control method of an electronic device according to an embodiment of the present invention includes receiving image data, obtaining characteristic information representing an object from the image data using a plurality of filters, and using at least two or more of the plurality of filters. The method may include detecting an object included in the image data using feature information acquired through a filter and providing information on the detected object.

Meanwhile, according to another embodiment of the present invention, the electronic device features an input unit for receiving image data, an output unit for outputting information on an object of the image data, and a plurality of filters to represent an object from the image data. A processor that obtains information, detects an object included in the image data using feature information acquired through at least two of the plurality of filters, and controls the output unit to output information on the detected object. includes

According to various embodiments as described above, the electronic device according to the present invention can more accurately detect surrounding vehicles and pedestrians not only in normal road conditions but also in night and bad weather conditions.

1 is an exemplary view showing an environment in which an object is not recognized in a general electronic device;
2 is a schematic block diagram of an electronic device for detecting an object from image data according to an embodiment of the present invention;
3 is a detailed block diagram of an electronic device for detecting an object from image data according to an embodiment of the present invention;
4 is an exemplary view showing an object detection process and detection result in a conventional electronic device;
5 is an exemplary view showing an object detection process and detection result in an electronic device according to an embodiment of the present invention;
6 is an example of adjusting the size of a reference confidence map used for generating a final confidence map in an electronic device according to the present invention;
7 is an exemplary view showing a detection result of detecting an object from image data in an electronic device according to the present invention and a conventional electronic device;
8 is an exemplary diagram for providing driving information according to an object detection result of an electronic device according to an embodiment of the present invention;
9 is a detailed block diagram of an electronic device according to an embodiment of the present invention;
10 is a flowchart of a control method of an electronic device for detecting an object from captured image data according to an embodiment of the present invention;
11 is a flowchart of a method of obtaining characteristic information representing an object from image data in an electronic device according to an embodiment of the present invention;
12 is a flowchart of a method of detecting an object in an electronic device according to an embodiment of the present invention.

Prior to a detailed description of the present invention, the description method of the present specification and drawings will be described.

First, terms used in the present specification and claims are general terms in consideration of functions in various embodiments of the present invention. However, these terms may vary depending on the intention of a technician working in the field, legal or technical interpretation, and the emergence of new technologies. In addition, some terms are arbitrarily selected by the applicant. These terms may be interpreted as the meanings defined in this specification, and if there is no specific term definition, they may be interpreted based on the overall content of this specification and common technical knowledge in the art.

In addition, the same reference numerals or numerals in each drawing attached to this specification indicate parts or components that perform substantially the same function. For convenience of description and understanding, the same reference numerals or symbols are used in different embodiments. That is, even if all components having the same reference numerals are shown in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present specification and claims, terms including ordinal numbers such as “first” and “second” may be used to distinguish between elements. These ordinal numbers are used to distinguish the same or similar components from each other, and the meaning of the term should not be construed as being limited due to the use of these ordinal numbers. For example, the order of use or arrangement of elements associated with such an ordinal number should not be limited by the number. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Terms such as "module", "unit", and "part" in the embodiments of the present invention are terms used to refer to components that perform at least one function or operation, and these components are hardware or software. It may be implemented or implemented as a combination of hardware and software. In addition, a plurality of "modules", "units", "parts", etc. are integrated into at least one module or chip, except for cases where each of them needs to be implemented with separate specific hardware, so that at least one processor (not shown).

Also, in an embodiment of the present invention, when a part is said to be connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, the meaning that a certain part includes a certain component means that it may further include other components without excluding other components unless otherwise specified.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram illustrating an environment in which an object is not recognized in a general electronic device.

As shown in FIG. 1 , the electronic device 100 included in a transportation device such as a vehicle may be, for example, a black box, a smart phone, or the like that captures road conditions on which the transportation device is driving in real time. The electronic device 100 may capture the front area and the rear area of the road on which the transportation device is driving in real time, and detect objects such as people and vehicles from the captured image data using an object detection algorithm.

Then, the electronic device 100 provides at least one of type information of the detected object, location information of the object, and movement information of the object. Accordingly, the driver driving the transportation device can recognize the environmental conditions of the road in front or behind while driving based on the information provided through the electronic device 100 and can drive safely.

As such, the electronic device 100 that provides information on various objects existing on the road cannot detect the object under the following environmental conditions.

Specifically, as shown in FIG. 1 , in the first driving environment 10 in which a transportation device drives on a road at night, the electronic device 100 cannot detect an object for a pedestrian from captured image data. In addition, in the second driving environment 20 in which the transportation device drives on the road in rainy weather, the electronic device 100 cannot detect objects for pedestrians or surrounding vehicles from captured image data. . Also, in the third driving environment 30 where a part of the pedestrian is covered by an umbrella, the electronic device 100 cannot detect an object for the pedestrian covered by the umbrella from the photographed image data.

Such environmental conditions may be difficult to operate the transportation device. Under these environmental conditions, when the electronic device 100 cannot detect an object present on the road on which it is driving, the probability of an accident occurring for a driver driving the transportation device may increase.

Hereinafter, each component of the electronic device 100 will be described in detail.

2 is a schematic block diagram of an electronic device for detecting an object from image data according to an embodiment of the present invention.

As shown in FIG. 2 , the electronic device 100 includes an input unit 110 , an output unit 120 , a storage unit 130 and a processor 140 .

The input unit 110 receives image data photographed through a camera. The output unit 120 outputs at least one of video and audio from image data input through the input unit 110 . Also, the output unit 120 outputs information about an object detected from image data through at least one of video and audio. Here, the information about the object detected from the image data may include at least one of object type information, location information, and movement information.

The storage unit 130 may store a filter set including a plurality of filters for detecting feature points representing objects included in captured image data.

Accordingly, the processor 140, which will be described later, may detect an object from image data input through the input unit 110 using a filter set stored in the storage unit 130.

Specifically, the processor 140 obtains feature information representing an object from image data input through the input unit 110 using a plurality of filters included in the filter set. Thereafter, the processor 1409 detects an object included in the corresponding image data using feature information obtained through at least two of the plurality of filters, and outputs information on the detected object to the output unit 120. to control

Here, the feature information may include a probability value representing the reliability of the object and location information at which the object is located.

Specifically, the processor 140 generates a confidence map corresponding to each of a plurality of filters by using each feature information obtained from a plurality of filters constituting the filter set. Thereafter, the processor 140 obtains first and second confidence maps among confidence maps corresponding to each of a plurality of filters, and generates a final confidence map using the obtained first and second confidence maps. Thereafter, the processor 140 may analyze the final confidence map to detect an object from image data input through the input unit 110 .

Here, the first confidence map is a map generated using feature information obtained from a filter next to a preset filter among a plurality of filters, and the second confidence map is generated using feature information obtained from the last filter among the plurality of filters. can be a map.

In the present invention, the description was limited to generating a final confidence map using the first and second confidence maps, but the present invention is not limited thereto, and the first and second confidence maps and feature information obtained from another filter A final confidence map can be created using the confidence map generated using .

More specifically, the processor 140 obtains first characteristic information indicating an object included in the image data from image data input through the input unit 110 by using a first filter among a plurality of filters constituting the filter set. . Then, the processor 140 generates a confidence map based on the first feature information as described above. Thereafter, the processor 140 obtains second characteristic information representing the corresponding object from the confidence map generated based on the first characteristic information by using a second filter among a plurality of filters. Thereafter, the processor 140 polls the confidence map generated based on the second feature information to a predetermined size. Thereafter, the processor 140 obtains third feature information representing the corresponding object from the polled confidence map by using a third filter.

If the third filter is the last filter, the processor 140 generates a confidence map based on feature information obtained from at least one of the first and second filters and the third feature information obtained from the last filter. A final confidence map may be generated using the confidence map generated based thereon.

Thereafter, the processor 140 compares a probability value representing the reliability of an object from the previously generated final confidence map with a preset threshold value, and if the probability value is equal to or greater than the preset threshold value, the object is detected from the image data. Thereafter, the processor 140 may obtain type information of a corresponding object based on the detected shape of the object.

Accordingly, the processor 140 may control the output unit 120 to output at least one of type information, location information, and movement information of an object detected from input image data.

In addition, the processor 140 uses at least one of type information, location information, and movement information of the object detected from the input image data to provide distance information between the electronic device 100 and the corresponding object, dangerous situation information, and blind spot notification. generate at least one of the information. Thereafter, the processor 140 transmits the driving information of the transportation device to another electronic device (not shown) provided in the transportation device through a communication unit 150 to be described later. Accordingly, the other electronic device (not shown) may provide at least one of distance information to a pre-detected object, danger situation information, and blind spot notification information based on information received from the electronic device 100 .

Hereinafter, an operation of detecting an object from image data in the electronic device 100 according to the present invention will be described in detail.

3 is a detailed block diagram of an electronic device for detecting an object from image data according to an embodiment of the present invention.

As shown in FIG. 3, the processor 140 includes first to third calculation units 141 to 143, and the storage unit 130 includes a filter storage unit 131, a characteristic result storage unit 132, and a characteristic storage unit 130. An information storage unit 133 may be included.

Specifically, when initial image data is input through the input unit 110, the first operation unit 141 uses parameters set in the first filter among a plurality of filters constituting the filter set previously stored in the filter storage unit 131. The first feature value of the object is acquired from the image data and stored in the feature result storage unit 132 .

The second operation unit 142 obtains first characteristic information including a probability value representing the reliability of an object and location information on which the corresponding object is located based on the first characteristic value stored in the characteristic result storage unit 132, and stores the characteristic information. stored in section 133. Accordingly, the third operation unit 143 generates a first confidence map corresponding to the first filter based on the first characteristic information stored in the characteristic information storage unit 133 .

When the first confidence map is generated, the first operation unit 141 obtains a second feature value for the corresponding object from the first confidence map using the parameter set in the second filter and stores it in the feature result storage unit 132. .

Accordingly, the second operation unit 142 obtains second characteristic information including a probability value representing the reliability of the corresponding object and location information based on the second characteristic value stored in the characteristic result storage unit 132, thereby obtaining the characteristic information storage unit. Save to (133). Accordingly, the third operation unit 143 generates a second confidence map corresponding to the second filter based on the second characteristic information stored in the characteristic information storage unit 133 .

When the third confidence map corresponding to the third filter, which is the last filter among the plurality of filters, is generated through such a series of processes, the third operation unit 143 calculates the final confidence map using the previously generated first to third confidence maps. You can create maps.

However, the present invention is not limited thereto, and the third operation unit 143 uses one of the pre-generated first and second confidence maps and the third confidence map generated in correspondence with the third filter, which is the last filter, to determine the final Confidence maps can be created.

When the final confidence map is generated, the third operation unit 143 compares a probability value indicating the reliability of the corresponding object from the final confidence map with a preset threshold value, and if the probability value is greater than or equal to the preset threshold value, the corresponding object is detected, and the detected Information about the object is transferred to the output unit 120 . Accordingly, the output unit 120 may output information about the object detected from the input image data through at least one of video and audio.

Hereinafter, a process and detection result of detecting an object from image data input in an electronic device according to the present invention and a conventional electronic device will be compared and described.

4 is an exemplary diagram illustrating an object detection process and detection result in a conventional electronic device.

As shown in FIG. 4, when image data 410 is input, the conventional electronic device 100' obtains feature values for an object from the input image data using a plurality of filters included in a filter set. .

In this case, when the image data 410 is input, the conventional electronic device 100' obtains first feature information about the object from the image data 410 using a parameter set in the first filter. Thereafter, the electronic device 100' obtains a first confidence map (conv 1_1) 411 corresponding to the first filter based on the first feature information.

Thereafter, the electronic device 100' obtains second feature information from the first confidence map 411 using the parameter set in the second filter, and based on the obtained second feature information, the electronic device 100' obtains second feature information corresponding to the second filter. 2 A confidence map (conv 1_2) 412 is acquired.

Thereafter, the electronic device 100' polls (pool 1) 413 the second confidence map 412 with a preset size to obtain the polled confidence map. Thereafter, the electronic device 100 ′ obtains third feature information from the polled confidence map using a third filter, and based on the obtained third feature information, a third confidence map corresponding to the third filter (conv 2_1 ) (414) is obtained.

When the n-th confidence map (conv 5_3) 415 corresponding to the last n-th filter is obtained through such a series of processing processes, the conventional electronic device 100' converts the obtained n-th confidence map 415 to the final It is determined by the confidence map 420 .

Thereafter, when the final feature information is obtained from the predetermined final confidence map 420, the conventional electronic device 100 ′ uses the position predictor (RPN) 416 to obtain first image data from the final feature information ( 410) acquires location information of an object included. Here, the aforementioned feature information may be a feature value for an object.

Accordingly, the conventional electronic device 100' may obtain location information where an object is located from feature values included in final feature information specially obtained through a location predictor (RPN).

Thereafter, the conventional electronic device 100' obtains object type information from a feature value included in the specially obtained final feature information. Thereafter, the conventional electronic device 100' may provide object detection result information 430 indicating the type and location of the corresponding object based on the type information and location information of the object that have been obtained in advance.

That is, the conventional electronic device 100 ′ provides only object detection result information 430 in which the first object corresponding to the car 431 is detected through the object detection result information 430 .

Meanwhile, a first object corresponding to the car 431 and a second object corresponding to a person partially overlapped with the first object may be included on the input image data. However, the conventional electronic device 100' does not detect the second object overlapping the first object, and provides only the object detection result information 430 in which the first object corresponding to the aforementioned car 431 is detected. do.

5 is an exemplary diagram illustrating an object detection process and a detection result in an electronic device according to an embodiment of the present invention.

As shown in FIG. 5 , when image data 510 is input, the electronic device 100 uses a parameter set for a first filter among a plurality of filters included in a filter set, and uses the input image data to provide characteristics of an object. get the value Thereafter, the electronic device 100 obtains feature information including a probability value indicating reliability of an object and location information on which the corresponding object is located, based on the specially obtained feature value. Thereafter, the electronic device 100 generates a first confidence map (conv 1_1) 511 corresponding to the first filter based on the specially obtained feature information.

When the first confidence map 511 is generated, the electronic device 100 obtains a feature value for a corresponding object from the first confidence map 511 using parameters set in the second filter. Thereafter, the electronic device 100 obtains feature information including a probability value indicating reliability of an object and location information on which the corresponding object is located, based on the specially obtained feature value. Thereafter, the electronic device 100 generates a second confidence map (conv 1_2) 512 corresponding to the second filter based on the specially obtained feature information.

When the second confidence map 512 is generated, the electronic device 100 polls the pre-generated second confidence map 512 in a preset size to generate the polled confidence map 513 . Thereafter, the electronic device 100 obtains feature values of the object from the polled confidence map 513 using parameters set in the third filter. Thereafter, the electronic device 100 obtains feature information including a probability value indicating reliability of an object and location information on which the corresponding object is located, based on the specially obtained feature value. Thereafter, the electronic device 100 generates a third confidence map (conv 2_1) 514 corresponding to the third filter based on the specially obtained feature information.

When the n-th confidence map (conv 8_2) 515 corresponding to the last n-th filter is obtained through such a series of processing processes, the electronic device 100 determines the n-th confidence map (conv 8_2) 515 and the previous one. Among the confidence maps, a confidence map corresponding to a predefined filter is obtained.

Thereafter, the electronic device 100 determines the n-th confidence map (conv 8_2) 515 and the confidence map corresponding to the predefined filter as reference confidence maps (hereinafter, referred to as first to fourth reference confidence maps). do. Then, the electronic device 100 adjusts the size of the first to fourth reference confidence maps 521 to 525 to the same size. Thereafter, the electronic device 100 generates a final confidence map 530 using the first to reference confidence maps 521 to 525 adjusted to the same size.

However, the present invention is not limited thereto, and when a confidence map is generated corresponding to each of a plurality of filters, the electronic device 100 may adjust the size of each confidence map to the same preset size.

When the final confidence map 530 is generated, the electronic device 100 obtains feature information indicating an object from the previously created final confidence map 530, and based on the acquired feature information, the electronic device 100 is included in the previously input image data. detect an object Then, the electronic device 100 provides an object detection result 540 for the input image data based on the location and type information of the detected object.

As described with reference to FIG. 4 , the input image data 510 may include a first object corresponding to a car and a second object corresponding to a person partially overlapped with the first object.

In this case, the electronic device 100 according to the present invention corresponds to a first object 541 corresponding to a car and a person partially overlapped with the first object from the image data 510 input through the above-described series of execution processes. detects the second object 542 and provides object detection result information 540 in which the first and second objects 541 and 542 are detected.

Hereinafter, an operation of adjusting the size of the first to fifth reference confidence maps 521 to 525 used to generate the aforementioned final confidence map 530 in the electronic device 100 according to the present invention will be described. .

6 is an exemplary view of adjusting the size of a reference confidence map used to generate a final confidence map in an electronic device according to the present invention.

As described in FIG. 5 , the electronic device 100 selects a confidence map corresponding to a predefined filter among confidence maps corresponding to a plurality of filters and a confidence map corresponding to the last n-th filter as first to fifth reference confidence maps. It can be determined by the maps 521 to 525.

Each confidence map determined as the first to fifth reference confidence maps 521 to 525 may be polled with a predetermined size.

For example, the confidence map determined as the first reference confidence map 521 may be a map scaled in pixel units of 80 * 48, and the confidence map determined as the second reference confidence map 522 may be 80 * 48 It can be a map scaled from pixel units of 40 * 24 pixels.

In addition, the confidence map determined as the third reference confidence map 523 may be a map whose size is adjusted from 40 * 24 pixel units to 20 * 12 pixel units, and the confidence map determined as the fourth reference confidence map 524 The map may be a map scaled from 20 * 12 pixels to 10 * 6 pixels. In addition, the confidence map corresponding to the last n-th filter determined as the fifth reference confidence map 525 may be a map whose size is adjusted from 10 * 6 pixel units to 5 * 3 pixel units.

Accordingly, the electronic device 100 adjusts the size of each confidence map determined as the first to fifth reference confidence maps 521 to 525 to the same preset size.

According to an embodiment, the electronic device 100 corresponds to the first reference confidence map 521 represented by the largest size among the sizes of the respective confidence maps determined by the first to fifth reference confidence maps 521 to 525. The size of the confidence maps corresponding to the remaining second to fifth confidence maps 522 to 525 may be adjusted according to the size of the confidence map.

That is, the electronic device 100 adjusts each confidence map corresponding to the second to fifth confidence maps 522 to 525 to be expressed in a size of 80 * 48 pixels. Accordingly, the confidence maps corresponding to each of the first to fifth reference confidence maps 521 to 525 may have a size of 80 * 48 pixels.

Then, as described above, the electronic device 100 may generate the final confidence map 540 as described in FIG. 5 using the first to fifth reference confidence maps 521 to 525 adjusted to the same size. there is.

7 is an exemplary diagram illustrating a detection result of detecting an object from image data in an electronic device according to the present invention and a conventional electronic device.

As shown in (a) of FIG. 7 , when a plurality of objects overlap each other on input image data, the conventional electronic device 10 cannot detect the overlapped objects. For example, when a bus stopped in front of a crosswalk is partially covered by a person crossing a crosswalk, a vehicle and a bus stopped in front of a crosswalk overlap each other, or a group of people cross a crosswalk, a conventional electronic device (10 ) cannot detect objects for buses, vehicles, and a group of people crossing the crosswalk from the image data 710 photographing the road environment.

On the other hand, the electronic device 10 according to the present invention may detect objects for each bus, vehicle, and group of people crossing the crosswalk from the image data 710 of the road environment captured through the above-described execution process.

In addition, as shown in (b) of FIG. 7, the conventional electronic device 10 cannot detect an object for a vehicle in motion from image data 720 photographing a road environment at night. The other electronic device 10 may detect an object for a vehicle in motion from image data 720 capturing a road environment at night.

8 is an exemplary diagram for providing driving information according to an object detection result of an electronic device according to an embodiment of the present invention.

As shown in FIG. 8 , the electronic device 100 uses at least one of object type information, location information, and movement information detected from captured image data to obtain distance information from a previously detected object, dangerous situation information, and At least one of the blind spot notification information is generated and transmitted to the other electronic device 200 provided in the transportation device.

Accordingly, the other electronic device 200 may provide at least one of distance information to an object, danger situation information, and blind spot notification information received from the electronic device 100 .

For example, as shown in FIG. 8 , another electronic device 200 is an electronic device 100 through an E-Mirror 810 that visually provides risk factors for surrounding situations of the transportation device. Blind spot notification information received from may be provided.

Specifically, when the blind spot notification information is received from the electronic device 100, the other electronic device 200 displays one area of the electronic mirror 810 differently from the other areas based on the received blind spot notification information. Controls the mirror 810. Accordingly, one area of the electronic mirror 810 is displayed in red 811, so that the driver of the transportation device can recognize that the vehicle has approached the vicinity of the transportation device based on the notification information displayed on the electronic mirror 810. .

In addition, the other electronic device 200 may provide blind spot notification information received from the electronic device 100 through the head-up display (HUD) 820 that provides driving information.

Specifically, when the blind spot notification information is received from the electronic device 100, the other electronic device 200 controls the head-up display to display that a nearby vehicle is close to the transportation vehicle based on the received blind spot notification information. . Accordingly, the head-up display 820 displays on one area 821 of the screen that a nearby vehicle is close to the transport vehicle, so that the driver of the transport device can view the transport device based on notification information displayed on the head-up display 820. It can be recognized that a vehicle is approaching nearby.

In addition, the other electronic device 200 may provide dangerous situation information about surrounding vehicles received from the electronic device 100 through the head-up display (HUD) 820 that provides driving information.

Specifically, when the dangerous situation information is received from the electronic device 100, the other electronic device 200 transmits the surrounding vehicle information 822 indicating that a nearby vehicle is in front of the transportation vehicle based on the received dangerous situation information. Controls the head-up display 820 to display. Accordingly, the head-up display 820 displays nearby vehicles close to the transport vehicle differently from other surrounding vehicles on one area 821 of the screen, so that the driver of the transport device can view the transport vehicle through the head-up display 820. It can be recognized that it is approaching a nearby vehicle in front.

Meanwhile, the electronic device 100 according to the present invention independently transmits at least one of distance information to a corresponding object, dangerous situation information, and blind spot notification information using at least one of type information, location information, and movement information of the detected object. can be provided with

Hereinafter, the detailed configuration of the electronic device 100 according to the present invention will be described in detail.

9 is a detailed block diagram of an electronic device according to an embodiment of the present invention.

As shown in FIG. 9 , the electronic device 100 includes an input unit 110, an output unit 120, a storage unit 130, and a processor 140, as well as a communication unit 150, a photographing unit 160, and a sensing unit ( 170) may be further included.

First, as described above, the input unit 110 receives image data photographed through a camera. In addition, the input unit 110 may be an input means for receiving various user commands and transmitting them to the processor 140 . In this case, the input unit 110 may be implemented as a touch pad capable of touch input or a key pad equipped with various function keys, numeric keys, character keys, and the like.

In addition, the aforementioned output unit 120 includes a display unit 121 and an audio output unit 122. The display unit 121 displays driving information of the transportation vehicle and video images included in captured image data on a screen. In addition, the display unit 121 may display at least one of distance information, dangerous situation information, and blind spot notification information of a corresponding object generated based on at least one of type information, location information, and movement information of the object detected from the captured image data. can be provided visually.

The audio output unit 122 may output vehicle driving information and an audio signal included in captured image data in the form of audible sound through a speaker. In addition, the audio output unit 122 may include at least one of distance information, dangerous situation information, and blind spot notification information of a corresponding object generated based on at least one of type information, location information, and movement information of the object detected from the captured image data. An audio signal for one can be output.

Meanwhile, the display unit 121 may be implemented as a liquid crystal display (LCD), an organic light emitting display (OLED), or the like. In particular, when the above-described input unit 110 includes a touch input unit (not shown) capable of a user's touch input, the display unit 121 is implemented in the form of a touch screen forming a mutual layer structure together with a touch input unit (not shown). It can be.

The above-described processor 140 controls the overall operation of each element constituting the electronic device 100 . In particular, as described above, the processor 140 may detect an object from image data input through the input unit 110 and provide at least one of type information, location information, and movement information of the detected object.

The processor 140 may include a CPU 141, a GPU 142, a ROM 143, and a RAM 144, and may include the CPU 141, the GPU 142, the ROM 143, and the RAM 144 ) may be connected to each other through the bus 145.

The CPU 141 accesses the storage unit 130 and performs booting using the OS stored in the storage unit 130 . In addition, the CPU 141 performs various operations using various programs, contents, and data stored in the storage unit 130 .

The GPU 142 creates a display screen including various objects such as icons, images, text, and the like. Specifically, the GPU 142 calculates attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen based on the received control command, and selects the object based on the associated attribute values. Create display screens of various layouts including

The ROM 143 stores command sets for system booting and the like. When a turn-on command is input and power is supplied, the CPU 141 copies the OS stored in the storage 130 to the RAM 144 according to the command stored in the ROM 143 and executes the OS to boot the system. When booting is completed, the CPU 141 copies various programs stored in the storage unit 130 to the RAM 144 and executes the programs copied to the RAM 144 to perform various operations.

Such a processor 140 may be implemented as a single-chip system (System-on-a-chip or System on chip, SOC, SoC) by combining the above-described components.

Meanwhile, the above-described operation of the processor 140 may be performed by a program stored in the storage unit 130 . Here, the storage unit 130 may include a ROM 143, a RAM 144, or a memory card (eg, an SD card or a memory stick) removable/mountable in the electronic device 100, a non-volatile memory, a volatile memory, or a hard disk drive. (HDD) or solid state drive (SSD).

The communication unit 150 includes the above-described other electronic devices 200 included in the transportation vehicle, a terminal device such as a smart phone in the transportation vehicle (not shown), an electronic device 100 provided in the surrounding vehicles, and a terminal device provided in the surrounding vehicles. Communication may be performed with other electronic devices 200 and a server (not shown) that collects traffic conditions.

The communication unit 150 may be implemented as a communication module such as a short-distance wireless communication module (not shown) or a wireless communication module (not shown). Here, the short-distance wireless communication module (not shown) is a communication module that performs wireless communication with another electronic device 200 located in a short distance or a terminal device (not shown) in a transportation vehicle. For example, Bluetooth, It can be Zigbee, Near Field Communication (NFC), and the like.

A wireless communication module (not shown) connects to a mobile communication network according to various mobile communication standards such as WiFi, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Ling Term Evolution), etc. It is possible to perform communication with a collecting server (not shown) and the like.

In addition, the communication unit 150 is a connector (not shown) including at least one of wired communication modules such as High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), and Institute of Electrical and Electronic Engineers (IEEE) 1394. may further include. Such a connector (not shown) receives content data transmitted from an external server (not shown) through a wired cable connected to the connector (not shown) according to a control command of the processor 140, or externally records previously stored image data. can be transmitted on medium. In addition, a connector (not shown) may receive power from a power source through a wired cable physically connected to the connector (not shown).

The photographing unit 160 is for capturing a still image or a video according to a user command or an event occurrence, and may be implemented in plural units such as a front camera and a rear camera.

The sensing unit 170 is a sensor that detects ambient brightness, external temperature, and movement of the electronic device 100 . The sensing unit 170 includes an illuminance sensor (not shown), a temperature sensor (not shown), a motion sensor (not shown), a geomagnetic sensor (not shown), a gravity sensor (not shown), a gyro sensor (not shown), and the like. can include

The illuminance sensor (not shown) may detect the brightness of the surrounding environment, and the temperature sensor (not shown) may be a sensor that detects an external temperature.

An accelerometer sensor (not shown) is an acceleration sensor that measures acceleration of the moving electronic device 100 or intensity of impact. And, the magnetic sensor (not shown) is a sensor that can detect the azimuth using the earth's magnetic field, and the gravity sensor (not shown) is a sensor that detects which direction gravity acts, The transportation vehicle equipped with the electronic device 100 detects the direction by automatically rotating according to the moving direction. Lastly, a gyroscope sensor (not shown) is a sensor that helps to recognize a more detailed and precise motion by recognizing the 6-axis direction by inserting rotation into each of the existing motion sensors (not shown).

So far, an operation of detecting an object from image data photographed by the electronic device 100 according to the present invention has been described in detail. Hereinafter, a method of detecting an object from image data photographed by the electronic device 100 according to the present invention will be described in detail.

10 is a flowchart of a control method of an electronic device for detecting an object from photographed image data according to an embodiment of the present invention.

As shown in FIG. 10 , the electronic device 100 receives image data photographed through a camera (S1010). When image data is input, the electronic device 100 obtains feature information indicating an object from the previously input image data by using a plurality of filters constituting a pre-stored filter set (S1020). Thereafter, the electronic device 100 detects an object included in the image data using feature information acquired through at least two or more filters among a plurality of filters (S1030). Then, the electronic device 100 provides information to the detected object (S1040).

Here, the feature information may include a probability value representing the reliability of the object and location information at which the object is located.

Specifically, the electronic device 100 generates a confidence map corresponding to each of the plurality of filters by using each of the characteristic information obtained from the plurality of filters constituting the filter set. Thereafter, the electronic device 100 acquires first and second confidence maps corresponding to each of at least two predefined filters among the confidence maps corresponding to each of a plurality of filters, and obtains the first and second confidence maps. Create a final confidence map using . Thereafter, the electronic device 100 may analyze the final confidence map generated using the first and second confidence maps to detect an object from previously input image data.

Hereinafter, a method of acquiring feature information indicating an object from image data using a plurality of filters in the electronic device 100 and a method of detecting an object included in image data based on the specially obtained feature information will be described in detail. do.

11 is a flowchart of a method of obtaining characteristic information representing an object from image data in an electronic device according to an embodiment of the present invention.

As shown in FIG. 11 , when image data is input, the electronic device 100 obtains first characteristic information indicating an object from the previously input image data by using a first filter among a plurality of filters (S1110).

Thereafter, the electronic device 100 obtains second characteristic information indicating an object from a confidence map generated based on the first characteristic information by using a second filter (S1120). Thereafter, the electronic device 100 polls the confidence map generated based on the second feature information to a preset size (S1130). Thereafter, the electronic device 100 obtains third characteristic information indicating an object from a confidence map polled with a preset size by using a third filter (S1140).

Specifically, when initial image data is input, the electronic device 100 obtains a first characteristic value of an object from the image data by using a parameter set in a first filter among a plurality of filters. After that, the electronic device 100 obtains first feature information including a probability value representing the reliability of the object and location information on which the object is located, based on the first feature value. Thereafter, the electronic device 100 generates a first confidence map corresponding to the first filter based on the acquired first feature information.

Thereafter, the electronic device 100 obtains a second characteristic value of the corresponding object from the pre-generated first confidence map using the parameter set in the first filter. Thereafter, the electronic device 100 obtains second feature information including a probability value representing the reliability of the corresponding object and location information based on the second feature value, and responds to a second filter based on the obtained second feature information. A second confidence map is created.

Then, the electronic device 100 polls the second confidence map with a preset size. For example, if the second confidence map is a map expressed in units of 80 * 48 pixels, the electronic device 100 may poll the second confidence map in units of 40 * 24 pixels. In this way, when the second confidence map is polled with a preset size, the electronic device 100 determines a third characteristic value for the corresponding object from the second confidence map polled with a preset size using a parameter set in the third filter. Acquire Thereafter, the electronic device 100 obtains second feature information including a probability value representing the reliability of the corresponding object and location information based on the third feature value, and responds to a second filter based on the obtained second feature information. A third confidence map is created.

When a confidence map corresponding to each of a plurality of filters is generated through such a series of processing processes, the electronic device 100 generates a final confidence map using at least two confidence maps among the plurality of confidence maps, and the generated final confidence map Objects can be detected from the confidence map.

12 is a flowchart of a method of detecting an object in an electronic device according to an embodiment of the present invention.

As shown in FIG. 12 , the electronic device 100 generates a confidence map corresponding to each of a plurality of filters using each feature information obtained from a plurality of filters (S1210). Thereafter, the electronic device 100 obtains first and second confidence maps from among the plurality of confidence maps corresponding to each filter (S1220).

Here, the first confidence map is a map generated using feature information obtained from a filter next to a preset filter among a plurality of filters, and the second confidence map is generated using feature information obtained from the last filter among the plurality of filters. can be a map.

When the first and second confidence maps are obtained, the electronic device 100 adjusts the first and second confidence maps to the same size (S1220 and S1230). Thereafter, the electronic device 100 generates a final confidence map using the first and second confidence maps adjusted to the same size, analyzes the generated final confidence map, and detects an object included in the image data (S1240). S1250).

Specifically, the electronic device 100 may obtain feature information indicating an object from the final confidence map, and detect an object included in pre-input image data based on the acquired feature information.

More specifically, when characteristic information representing an object is obtained from the final confidence map, the electronic device 100 compares a probability value representing reliability of an object included in the obtained characteristic information with a preset threshold. As a result of the comparison, if the probability value is greater than or equal to a predetermined threshold value, the electronic device 100 detects the corresponding object based on the location information of the object included in the specially obtained feature information. Then, the electronic device 100 obtains type information of the object detected from the final confidence map. Thereafter, the electronic device 100 provides an object detection result for previously input image data based on at least one of location information of the detected object, type information of the object, and movement information of the object.

Meanwhile, the electronic device 100 is a device included in the transport device and can communicate with another electronic device 200 provided in the transport device and providing driving information. In this case, the electronic device 100 uses at least one of type information, location information, and movement information of the object detected from the image data to obtain at least one of distance information to the corresponding object, dangerous situation information, and blind spot notification information. It can be transmitted to the electronic device 200. Accordingly, the other electronic device 200 may provide notification information on a corresponding object through an E-mirror or a head-up display based on the information received from the electronic device 100 .

Meanwhile, the control method of the electronic device 100 as described above may be implemented as at least one execution program, and such an execution program may be stored in a non-transitory computer readable medium.

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the above-described programs include RAM (Random Access Memory), flash memory, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electronically Erasable and Programmable ROM), registers, hard disks, removable disks, memory It may be stored in various types of recording media readable by a terminal, such as a card, USB memory, or CD-ROM.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is common in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: electronic device 110: input unit
120: output unit 121: display unit
122: audio output unit 130: storage unit
140: processor 150: communication unit
160: photographing unit 170: detection unit

Claims (20)

In the control method of an electronic device,
receiving image data;
First feature information indicating an object is obtained from the image data by using a first filter among a plurality of filters, a first confidence map is generated based on the first feature information, and a second filter is obtained from the first confidence map. A polled confidence map obtained by obtaining second feature information representing the object using , generating a second confidence map based on the second feature information, and polling the second confidence map with a preset size. Obtains third feature information representing the object using a third filter from , generates a third confidence map based on the third feature information, and uses an n-th filter from the n-1 th confidence map to obtain the object obtaining n-th feature information representing , and generating an n-th confidence map based on the n-th feature information;
At least one confidence map among the 1st to n-1th confidence maps is acquired as a reference confidence map, a final confidence map is generated using the reference confidence map and the nth confidence map, and based on the final confidence map detecting the object; and
Including; providing type information on the detected object;
The electronic device is a device provided in a transportation device,
The image data includes image data of a rear area of the road on which the transportation device travels,
The reference confidence map,
Includes a map generated using feature information obtained from a filter next to a predetermined filter among the plurality of filters;
The n th confidence map,
A map generated using feature information obtained from a last filter among the plurality of filters. According to claim 1,
The feature information
The control method characterized in that it comprises a probability value representing the reliability of the object and location information where the object is located. delete delete delete According to claim 1,
adjusting the size of the reference confidence map and the n-th confidence map to have the same size;
A control method characterized in that it further comprises. According to claim 1,
Detecting the object is
and detecting the object according to whether a probability value representing the reliability of the object from the final confidence map is greater than or equal to a predetermined threshold value. According to claim 1,
The step of providing,
and providing at least one of location information and movement information of the detected object. According to claim 8,
The electronic device communicates with another electronic device that provides driving information,
The step of providing,
and providing at least one of distance information to the object, dangerous situation information, and blind spot notification information to the other electronic device using at least one of type information, location information, and movement information of the detected object. method. According to claim 1,
The control method of claim 1 , wherein the image data further includes image data of an area in front of a road on which the transportation device travels. In electronic devices,
an input unit for receiving image data;
an output unit outputting information about an object of the image data; and
Obtaining feature information indicating an object from the image data using a plurality of filters;
A processor for controlling the output unit to detect an object included in the image data using feature information acquired through at least two of the plurality of filters and output information on the detected object;
the processor,
First feature information indicating an object is obtained from the image data by using a first filter among a plurality of filters, a first confidence map is generated based on the first feature information, and a second filter is obtained from the first confidence map. A polled confidence map obtained by obtaining second feature information representing the object using , generating a second confidence map based on the second feature information, and polling the second confidence map with a preset size. Obtains third feature information representing the object using a third filter from , generates a third confidence map based on the third feature information, and uses an n-th filter from the n-1 th confidence map to obtain the object Obtaining n-th feature information representing , generating an n-th confidence map based on the n-th feature information,
At least one confidence map among the 1st to n-1th confidence maps is acquired as a reference confidence map, a final confidence map is generated using the reference confidence map and the nth confidence map, and based on the final confidence map detect the object,
The electronic device is a device provided in a transportation device,
The image data includes image data of a rear area of the road on which the transportation device travels,
The reference confidence map,
Includes a map generated using feature information obtained from a filter next to a predetermined filter among the plurality of filters;
The n th confidence map,
An electronic device that is a map generated using feature information obtained from a last filter among the plurality of filters. According to claim 11,
The feature information
The electronic device characterized in that it includes a probability value representing the reliability of the object and location information where the object is located. delete delete delete According to claim 11,
the processor,
The electronic device characterized in that the size of the reference confidence map and the size of the n-th confidence map are adjusted to the same size. According to claim 11,
the processor,
The electronic device characterized in that the object is detected according to whether a probability value representing the reliability of the object from the final confidence map is equal to or greater than a predetermined threshold value. According to claim 11,
the processor,
The electronic device characterized by controlling the output unit to output at least one of location information and movement information of the detected object. According to claim 18,
It further includes; a communication unit that communicates with other electronic devices that provide driving information;
the processor,
Controlling the communication unit to transmit at least one of distance information to the object, danger situation information, and blind spot notification information to the other electronic device using at least one of type information, location information, and movement information of the detected object Electronic device characterized in that. According to claim 11,
The electronic device according to claim 1 , wherein the image data further includes image data of an area in front of a road on which the transportation device travels.

Images