KR102300652B1 - vehicle and control method thereof - Google Patents

Abstract

The present invention receives a plurality of cameras and a plurality of images related to a vehicle periphery from the plurality of cameras, determines whether an object is detected in at least one of the plurality of images, and determines whether the object is detected in at least one of the overlapping regions of the plurality of images. By determining which one is located, if the object is located in the overlap area, the overlap area is processed based on the object detection information, and when the object is not detected or the object is not located in the overlap area, It relates to a vehicle comprising a; a control unit for blending the overlap area according to a preset ratio.

Description

A vehicle and its control method {vehicle and control method thereof}

The present invention relates to a vehicle having an AVM (Around View Monitoring) device for displaying an image around the vehicle.

Recently, according to the luxury and multi-function of the vehicle, an around view monitoring (hereinafter, AVM, Around View Monitoring) system is mounted on the vehicle.

The AVM device acquires images around the vehicle through cameras mounted on all sides of the vehicle, and when the vehicle is parked, the driver can check the surroundings of the vehicle through the indoor display device without having to look around or look around. it is a system In addition, the around-view monitoring system provides an around view as if viewed from the top of a vehicle by synthesizing the surrounding images. By using such an around-view monitoring system, the driver can safely park the vehicle or pass the narrow road by grasping the situation around the vehicle at a glance.

Meanwhile, although the AVM device is used as a parking assistance device, object detection is also possible based on images acquired through a plurality of cameras. It is necessary to study an operation of detecting an object through a plurality of cameras provided in the AVM device.

Prior literature on the AVM system: Publication No. 10-2013-0028230

An object of the present invention is to provide a vehicle that performs object detection on images received from a plurality of cameras in order to solve the above problems.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, a vehicle according to an embodiment of the present invention receives a plurality of cameras and a plurality of images related to the vehicle surroundings from the plurality of cameras, and determines whether an object is detected in at least one of the plurality of images and determining whether the object is located in at least one of the overlapping areas of the plurality of images, and when the object is located in the overlapping area, processing the overlapping area based on the object detection information, and the object is detected or a control unit for blending the overlapping area according to a preset ratio when the object is not located in the overlapping area.

On the other hand, the plurality of cameras, a first camera to obtain a first image, a second camera to obtain a second image, a third camera to obtain a third image, and a fourth camera to obtain a fourth image .

Meanwhile, when the object is detected in the overlap region of the plurality of images, the controller compares the moving speed, the moving direction, or the size of the object in the plurality of images, and the reliability of any image among the plurality of images It is determined whether is higher, and the overlap region is processed based on the reliability.

Meanwhile, the controller processes the overlap region only with the image having the higher reliability among the plurality of images.

Meanwhile, when the reliability is determined based on the moving speed, the controller gives higher reliability to an image in which the pixel movement amount per unit time of the object is greater in the plurality of images.

Meanwhile, when determining the reliability based on the movement direction, the controller gives higher reliability to an image having a greater horizontal movement in the plurality of images.

Meanwhile, when the reliability is determined based on the size, the controller gives higher reliability to an image having a larger number of pixels occupied by the object in the plurality of images.

A method of controlling a vehicle according to an embodiment of the present invention includes: receiving a plurality of images related to the surroundings of the vehicle from a plurality of cameras, respectively; determining whether an object is detected in at least one of the plurality of images; determining whether the object is located in at least one of the overlapping areas of a plurality of images; if the object is located in the overlapping area, processing the overlapping area based on the object detection information; and blending the overlapping area according to a preset ratio when the object is not located in the overlapping area.

On the other hand, the plurality of cameras, a first camera to obtain a first image, a second camera to obtain a second image, a third camera to obtain a third image, and a fourth camera to obtain a fourth image .

Meanwhile, the processing of the overlap region based on the object detection information may include determining the moving speed, moving direction or size of the object in the plurality of images when the object is detected in the overlapping region of the plurality of images. By comparison, it is determined which image has higher reliability among the plurality of images, and the overlap area is processed based on the reliability.

Meanwhile, in the processing of the overlap region based on the object detection information, the overlap region is processed only with the image having the higher reliability among the plurality of images.

Meanwhile, in the processing of the overlap region based on the object detection information, when reliability is determined based on the movement speed, in the plurality of images, an image with a larger pixel movement amount per unit time of the object has a higher value. give credibility

Meanwhile, in the processing of the overlap region based on the object detection information, when reliability is determined based on the movement direction, in the plurality of images, higher reliability is given to an image having a greater horizontal movement. do.

Meanwhile, in the processing of the overlap region based on the object detection information, when reliability is determined based on the size, in the plurality of images, an image with a larger number of pixels occupied by the object is given higher reliability. do.

The details of other embodiments are included in the detailed description and drawings.

According to an embodiment of the present invention, there are one or more of the following effects.

The vehicle according to an embodiment of the present invention has an effect of detecting an object in an acquired image and tracking the detected object, thereby preventing an accident in advance.

The vehicle according to the embodiment of the present invention has an effect of providing a natural AVM screen by processing the overlap area based on the object detection information even when an object is located in the overlap area.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a view showing the exterior of a vehicle having a plurality of cameras according to an embodiment of the present invention.
FIG. 2 is a diagram schematically illustrating positions of a plurality of cameras attached to the vehicle of FIG. 1 .
3A illustrates an around view image based on images captured by the plurality of cameras of FIG. 2 ;
3B is a diagram referenced for explaining an overlap region according to an embodiment of the present invention.
4 is a block diagram of a vehicle according to an embodiment of the present invention.
5 is a block diagram of a display device according to an embodiment of the present invention.
6 is a detailed block diagram of a control unit according to an embodiment of the present invention.
7 is a flowchart illustrating an operation of a vehicle according to an embodiment of the present invention.
8 is a diagram referenced for explaining an operation of granting reliability when an object is detected in an overlap area according to an embodiment of the present invention.
9 to 11 are diagrams referenced for explaining an operation of generating an around-view image by synthesizing a plurality of images, according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as "comprises" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The vehicle described herein may be a concept including all of an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, an electric vehicle having an electric motor as a power source, and the like.

In the following description, the left side of the vehicle refers to the left side of the driving direction of the vehicle, that is, the driver's seat side, and the right side of the vehicle refers to the right side in the driving direction of the vehicle, that is, the passenger side side.

Meanwhile, the AVM device described herein may be a device that includes a plurality of cameras and provides an around view image by combining a plurality of images captured by the plurality of cameras. In particular, it may be a device that provides a top view or a bird eye view based on the vehicle. Hereinafter, an AVM apparatus for a vehicle and a vehicle having the same according to various embodiments of the present invention will be described.

Meanwhile, in this specification, data exchange may be performed through a vehicle communication network. Here, the vehicle communication network is preferably CAN. According to an embodiment, the vehicle communication network may be established using an Ethernet protocol.

1 is a view showing the exterior of a vehicle having a plurality of cameras according to an embodiment of the present invention.

Referring to FIG. 1 , a vehicle 10 includes wheels 20FR, 20FL, 20RL,.. ) may be provided with a plurality of cameras (110a, 110b, 110c, 110d) mounted on. Meanwhile, in the drawings, only the left camera 110a and the front camera 110d are shown for convenience.

The plurality of cameras 110a, 110b, 110c, and 110d may be activated when the speed of the vehicle is less than or equal to a predetermined speed, or when the vehicle is propelled to obtain a photographed image, respectively. The images obtained by the plurality of cameras may be signal-processed in the controller ( 180 in FIG. 4 ) or the processor ( 280 in FIG. 5 ).

FIG. 2 is a diagram schematically illustrating positions of a plurality of cameras attached to the vehicle of FIG. 1 , and FIG. 3A illustrates an around-view image based on images captured by the plurality of cameras of FIG. 2 .

First, referring to FIG. 2 , a plurality of cameras 110a , 110b , 110c , and 110d may be disposed on the left, rear, right, and front of the vehicle, respectively.

In particular, the left camera 110a and the right camera 110c may be disposed in a case surrounding the left side mirror and a case surrounding the right side mirror, respectively.

Meanwhile, the rear camera 110b and the front camera 110d may be disposed in the vicinity of the trunk switch and in the vicinity of the emblem or emblem, respectively.

Each of the plurality of images photographed by the plurality of cameras 110a, 110b, 110c, and 110d is transmitted to the controller (180 of FIG. 4 ) in the vehicle 10 and the like, and the controller (180 of FIG. 4 ), the plurality of images are combined to create an around-view image.

3A illustrates an example of an around-view image 810 . The around view image 810 includes a first image area 110ai from the left camera 110a, a second image area 110bi from the rear camera 110b, and a third image area 110a from the right camera 110c. 110ci), and a fourth image area 110di from the front camera 110d.

On the other hand, when an around-view image is generated from a plurality of cameras, a boundary portion between each image area is generated. This boundary part is processed by image blending so that it can be displayed naturally.

Meanwhile, boundary lines 111a , 111b , 111c , and 111d may be displayed on the boundary of each of the plurality of images.

3B is a diagram referenced for explaining an overlap region according to an embodiment of the present invention.

Referring to FIG. 3B , a plurality of cameras may use a wide-angle lens. As a result, an overlap region may occur in images acquired from a plurality of cameras. For example, a first overlap region 112a may be generated in the first image acquired by the first camera 110a and the second image acquired by the second camera 110b. In addition, a second overlap region 112b may be generated in the second image acquired by the second camera 110b and the third image acquired by the third camera 110c. Also, a third overlap region 112c may be generated in the third image acquired by the third camera 110c and the fourth image acquired by the fourth camera 110d. In addition, a fourth overlap region 112d may be generated in the fourth image acquired by the fourth camera 110d and the first image acquired by the first camera 110a.

Meanwhile, when an object is located in the first to fourth overlap regions 112a, 112b, 112c, and 112d, two objects may appear or disappear when converted to an around-view image. In this case, a problem may occur in object detection, and a problem may occur in that accurate information cannot be transmitted to the passenger.

4 is a block diagram of a vehicle according to an embodiment of the present invention.

4 , the vehicle 10 includes a plurality of cameras 110a, 110b, 110c, and 110d, a first input unit 120, an alarm unit 130, a first communication unit 140, a display device 200, It may include a first memory 160 and a controller 180 .

The plurality of cameras may include first to fourth cameras 110a, 110b, 110c, and 110d. The first camera 110a acquires an image around the left side of the vehicle. The second camera 110b acquires an image around the rear of the vehicle. The third camera 110c acquires an image around the right side of the vehicle. The fourth camera 110d acquires an image around the front of the vehicle.

The plurality of images respectively acquired by the first to fourth cameras 110a, 110b, 110c, and 110d are transmitted to the controller 180 .

Meanwhile, the first to fourth cameras 110a, 110b, 110c, and 110d each include a lens and an image sensor (eg, CCD or CMOS). Here, it is preferable that the lens is a fisheye lens having a wide angle of 180° or more.

The first input unit 120 may receive a user input. The first input unit 120 may include means for receiving an input from the outside, such as a touch pad, a physical button, a dial, a slider switch, and a click wheel. A user input received through the first input unit 120 is transmitted to the control unit 180 .

The alarm unit 130 outputs an alarm according to the information processed by the control unit 180 . The alarm unit 130 may include a sound output unit and a display. The sound output unit may output audio data under the control of the controller 180 . The sound output unit may include a receiver, a speaker, a buzzer, and the like. The display displays alarm information on a screen under the control of the controller 180 .

Meanwhile, the alarm unit 130 may output an alarm based on the position of the detected object. Meanwhile, the display included in the alarm unit 130 may be a concept including a cluster or a head up display (HUD) on the front of the vehicle.

The first communication unit 140 communicates with an external electronic device. The first communication unit 140 may exchange data with an external server, a nearby vehicle, an external base station, and the like. The first communication unit 140 may include a communication module that enables communication with an external electronic device, where the communication module may use a known technology.

Meanwhile, the first communication unit 140 may include a short-range communication module, and may exchange data with the passenger's portable terminal through the short-distance communication module. The first communication unit 140 may transmit the around view screen to the passenger's portable terminal. Also, the first communication unit 140 may transmit a control command received from the portable terminal to the control unit 180 . Meanwhile, the first communication unit 140 may transmit information according to object detection to the portable terminal. In this case, the portable terminal may output an alarm notifying the detection of an object through vibration, sound output, or the like.

The display apparatus 200 decompresses the compressed image and displays the around-view image. The display apparatus 200 may be Audio Video Navigation (AVN). The configuration of the display apparatus 200 will be described in detail with reference to FIG. 5 .

The first memory 160 stores data supporting various functions of the vehicle 10 . The first memory 160 may store a plurality of application programs driven in the vehicle 10 , data for operation of the vehicle 10 , and commands.

The first memory 160 may include a high-speed random access memory. The first memory 160 may also include, but is not limited to, non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices, and may include a readable storage medium. .

For example, the first memory 160 may include an Electronically Erasable and Programmable Read Only Memory (EEP-ROM), but is not limited thereto. In the EEP-ROM, writing and erasing of information may be performed by the control unit 180 while the control unit 180 is operating. The EEP-ROM may be a storage device in which information stored therein is maintained without being erased even when the power supply of the controller is turned off and the power supply is stopped.

Meanwhile, the first memory 160 may store images obtained from the plurality of cameras 110a, 110b, 110c, and 110d. For example, when a collision of the vehicle 10 is detected, images obtained from the plurality of cameras 110a, 110b, 110c, and 110d may be stored.

The controller 180 controls the overall operation of each unit in the vehicle 10 . The controller 180 may execute or execute various software programs and/or sets of instructions stored in the first memory 160 to perform various functions to control the vehicle 10 and process data. The controller 180 may process a signal based on information stored in the first memory 160 .

The controller 180 performs pre-processing on images received from the plurality of cameras 110a, 110b, 110c, and 110d. The controller 180 removes noise from the image by using various filters or histogram equalization. Meanwhile, the pre-processing of the image is not necessarily a necessary procedure, and may be omitted depending on the state of the image or the purpose of image processing.

The controller 180 may detect an object based on the pre-processed image. Here, the object may be a concept including a pedestrian, an obstacle, a surrounding vehicle, and the like. Meanwhile, the around view screen displayed through the display apparatus 200 may correspond to a partial region of an original image acquired through the plurality of cameras 110a, 110b, 110c, and 110d. The controller 180 may perform object detection based on the entire original image, including the image displayed on the display apparatus 200 .

The controller 180 compares the detected object with the object stored in the first memory 160 to classify and confirm the object.

The controller 180 tracks the detected object. For example, the object tracking unit 430 sequentially identifies an object in the acquired images, calculates a motion or a motion vector of the identified object, and based on the calculated motion or motion vector, the corresponding The movement of an object can be tracked.

The controller 180 synthesizes an image by processing the overlap region based on the object detection information.

When an object is detected in the overlap region of the plurality of images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the plurality of images. The controller 180 determines which image among a plurality of images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. The controller 180 processes the overlap area only with images having higher reliability among the plurality of images. For example, when an object is detected in the overlapping region of the first and second images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the first and second images. The controller 180 determines which image among the first and second images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. The controller 180 processes the overlap region only with images having higher reliability among the first and second images.

If the reliability is determined based on the moving speed of the object, the controller 180 may give higher reliability to an image in which the amount of movement of a pixel per unit time of the object is greater in a plurality of images. For example, when an object is detected in an overlapping region of the first and second images, the controller 180 applies higher reliability to an image in which the amount of movement of a pixel per unit time of the object in the first and second images is greater. can be given

If the reliability is determined based on the moving direction of the object, the controller 180 may give higher reliability to an image in which the horizontal movement of the object is greater in the plurality of images. In the case of vertical movement, since an actual object approaches a vehicle, only the size of the object increases, it is disadvantageous compared to the horizontal movement when determining object detection and tracking. For example, when an object is detected in the overlap region of the first and second images, the controller 180 gives higher reliability to an image in which the horizontal movement of the object in the first and second images is greater. can do.

If the reliability is determined based on the size of the object, the controller 180 may assign higher reliability to an image in which the number of pixels occupied by the object is greater in the plurality of images. Alternatively, in the plurality of images, the controller 180 may further assign a weight to an image having a large virtual rectangular area surrounding the object. For example, when an object is detected in the overlap region of the first and second images, the controller 180 may give higher reliability to an image in which the number of pixels occupied by the object in the first and second images is greater. can

Meanwhile, when an object is not detected or the object is not located in the overlap region, the controller 180 blends the overlap region according to a preset ratio to synthesize the image.

The controller 180 generates an around-view image based on the synthesized image.

Here, the around-view image may be an image converted into a top-view image by synthesizing images received from a plurality of cameras 110a, 110b, 110c, and 110d for capturing images around the vehicle.

For example, the controller 180 may synthesize a plurality of images using a look-up table (LUT) and convert the images into an around-view image. The lookup table is a table storing the relationship between which one pixel of the synthesized image corresponds to which pixel of the four original images.

Thereafter, the controller 180 generates a virtual vehicle image in the around-view image. Specifically, the controller 180 overlays a virtual vehicle image on the around-view image.

Next, the controller 180 transmits the compressed data to the display apparatus 200 to display the around-view image.

Meanwhile, the controller 180 may overlay and display an image corresponding to the object detected in step S730 on the around-view image. The controller 180 may overlay an image corresponding to the tracked object on the around-view image and display it.

5 is a block diagram of a display device according to an embodiment of the present invention.

Referring to FIG. 5 , the display apparatus 200 includes a second input unit 220 , a second communication unit 240 , a display unit 250 , a sound output unit 255 , a second memory 260 , and a processor 280 . may include.

The second input unit 220 may receive a user input. The second input unit 220 may include means for receiving an input from the outside, such as a touch pad, a physical button, a dial, a slider switch, and a click wheel. A user input received through the first input unit 220 is transmitted to the control unit 180 .

The second communication unit 240 may be communicatively connected with an external electronic device to exchange data. For example, the second communication unit 240 may be connected to a broadcaster server to receive broadcast content. For example, the second communication unit 240 may be connected to a traffic information providing server to receive Transport Protocol Experts Group (TPEG) information.

The display unit 250 displays information processed by the processor 270 . For example, the display unit 250 may display execution screen information of an application program driven by the processor 270 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

Meanwhile, when the touch pad forms a layer structure with the display unit 250 , it may be referred to as a touch screen. The touch screen may function as the second input unit 220 .

The sound output unit 255 may output audio data. The sound output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The second memory 260 stores data supporting various functions of the display apparatus 200 . The second memory 260 may store a plurality of application programs driven in the display apparatus 200 , data for operation of the display apparatus 200 , and commands.

The second memory 260 may include a high-speed random access memory. The second memory 260 may also include, but is not limited to, non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices, and may include a readable storage medium. .

For example, the second memory 260 may include an Electronically Erasable and Programmable Read Only Memory (EEP-ROM), but is not limited thereto. In the EEP-ROM, writing and erasing of information may be performed by the processor 280 while the processor 280 is operating. The EEP-ROM may be a storage device in which information stored therein is maintained without being erased even when the power supply of the controller is turned off and the power supply is stopped.

The processor 280 controls the overall operation of each unit in the display apparatus 200 . The processor 280 may execute or execute various software programs and/or sets of instructions stored in the second memory 260 to perform various functions to control the display device 20 and process data. The processor 280 may process a signal based on information stored in the second memory 260 .

The processor 280 displays the around-view image.

6 is a detailed block diagram of a control unit according to an embodiment of the present invention.

Referring to FIG. 6 , the controller 180 includes a preprocessor 610 , an object detector 620 , an object checker 630 , an object tracker 640 , an overlap region processor 650 , and an around-view image generator. (660).

The preprocessor 610 performs preprocessing on images received from the plurality of cameras 110a, 110b, 110c, and 110d. The preprocessor 610 removes noise from the image by using various filters or histogram equalization. Meanwhile, the pre-processing of the image is not necessarily a necessary procedure, and may be omitted depending on the state of the image or the purpose of image processing.

The object detector 620 may detect an object based on the pre-processed image. Here, the object may be a concept including a pedestrian, an obstacle, a surrounding vehicle, and the like. Meanwhile, the around view screen displayed through the display apparatus 200 may correspond to a partial region of an original image acquired through the plurality of cameras 110a, 110b, 110c, and 110d. The object detection unit 620 may perform object detection based on the entire original image including the image displayed on the display apparatus 200 .

The object check unit 630 compares the detected object with the object stored in the first memory 160 to classify and confirm the object.

The object tracking unit 640 tracks the detected object. For example, the object tracking unit 430 sequentially identifies an object in the acquired images, calculates a motion or a motion vector of the identified object, and based on the calculated motion or motion vector, the corresponding The movement of an object can be tracked.

The overlap region processing unit 650 synthesizes an image by processing the overlap region based on the object detection information.

When an object is detected in the overlap region of the plurality of images, the overlap region processing unit 650 compares the moving speed, the moving direction, or the size of the object in the plurality of images. The overlap region processing unit 650 determines which image among a plurality of images has higher reliability based on the comparison result. The overlap area processing unit 650 processes the overlap area based on reliability. The overlap area processing unit 650 processes the overlap area only with images having higher reliability among the plurality of images. For example, when an object is detected in the overlap region of the first and second images, the overlap region processing unit 650 compares the moving speed, the moving direction, or the size of the object in the first and second images. The overlap region processing unit 650 determines which of the first and second images has higher reliability based on the comparison result. The overlap area processing unit 650 processes the overlap area based on reliability. The overlap area processing unit 650 processes the overlap area only with images having higher reliability among the first and second images.

If the reliability is determined based on the moving speed of the object, the overlap region processing unit 650 may provide higher reliability to an image in which the pixel movement amount per unit time of the object is greater in the plurality of images. For example, when an object is detected in the overlap region of the first and second images, the overlap region processing unit 650 may set a higher value in the image in which the pixel movement amount per unit time of the object in the first image and the second image is larger. reliability can be given.

If the reliability is determined based on the movement direction of the object, the overlap area processing unit 650 may give higher reliability to an image in which the horizontal movement of the object is greater in the plurality of images. In the case of vertical movement, since an actual object approaches a vehicle, only the size of the object increases, it is disadvantageous compared to the horizontal movement when determining object detection and tracking. For example, when an object is detected in the overlap region of the first and second images, the overlap region processing unit 650 provides a higher reliability to an image in which the horizontal movement of the object in the first and second images is greater. can be given

If the reliability is determined based on the size of the object, the overlap region processing unit 650 may provide higher reliability to an image in which the number of pixels occupied by the object is greater in a plurality of images. Alternatively, the overlap region processing unit 650 may further assign a weight to an image having a large virtual quadrangular area surrounding the object in the plurality of images. For example, when an object is detected in the overlap region of the first and second images, the overlap region processing unit 650 provides higher reliability to an image in which the number of pixels occupied by the object in the first and second images is larger. can be given

Meanwhile, when no object is detected or the object is not located in the overlap area, the overlap area processing unit 650 blends the overlap area according to a preset ratio to synthesize an image.

The around-view image generator 660 generates an around-view image based on the synthesized image.

Here, the around-view image may be an image converted into a top-view image by synthesizing images received from a plurality of cameras 110a, 110b, 110c, and 110d for capturing images around the vehicle.

For example, the around-view image generator 660 may synthesize a plurality of images using a look-up table (LUT) and convert the images into an around-view image. The lookup table is a table storing the relationship between which one pixel of the synthesized image corresponds to which pixel of the four original images.

Thereafter, the around-view image generator 660 generates a virtual vehicle image in the around-view image. Specifically, the around-view image generator 660 overlays a virtual vehicle image on the around-view image.

Next, the around-view image generator 660 transmits the compressed data to the display apparatus 200 to display the around-view image.

Meanwhile, the around-view image generator 660 may overlay an image corresponding to the object detected in step S730 on the around-view image and display it. The around-view image generator 660 may display an image corresponding to the tracked object by overlaying it on the around-view image.

7 is a flowchart illustrating an operation of a vehicle according to an embodiment of the present invention.

Referring to FIG. 7 , the controller 180 receives first to fourth images from a plurality of cameras 110a, 110b, 110c, and 110d (S710).

The controller 180 performs pre-processing on each of the plurality of received images (S720). The controller 180 removes noise from the image by using various filters or histogram equalization. Meanwhile, the pre-processing of the image is not necessarily a necessary procedure, and may be omitted depending on the state of the image or the purpose of image processing.

The controller 180 determines whether an object is detected based on the received first to fourth images or pre-processed images (S730). Here, the object may be a concept including a pedestrian, an obstacle, a surrounding vehicle, and the like.

When the object is detected, the controller 180 determines whether the object is located in the overlap area (S740). Specifically, the controller 180 determines whether the object is located in any one of the first to fourth overlap areas 112a, 112b, 112c, and 112d described with reference to FIG. 3B .

When the object is located in the overlap region 112a, 112b, 112c, 112d, the controller 180 processes the overlap region based on the object detection information to synthesize an image (S750).

When an object is detected in the overlap region of the plurality of images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the plurality of images. The controller 180 determines which image among a plurality of images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. The controller 180 processes the overlap area only with images having higher reliability among the plurality of images. For example, when an object is detected in the overlapping region of the first and second images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the first and second images. The controller 180 determines which image among the first and second images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. The controller 180 processes the overlap region only with the image having higher reliability among the first and second images.

If the reliability is determined based on the moving speed of the object, the controller 180 may give higher reliability to an image in which the amount of movement of a pixel per unit time of the object is greater in a plurality of images. For example, when an object is detected in an overlapping region of the first and second images, the controller 180 applies higher reliability to an image in which the amount of movement of a pixel per unit time of the object in the first and second images is greater. can be given

If the reliability is determined based on the movement direction of the object, the controller 180 may give higher reliability to an image in which the horizontal movement of the object is greater in the plurality of images. In the case of vertical movement, since an actual object approaches a vehicle, only the size of the object increases, it is disadvantageous compared to the horizontal movement when determining object detection and tracking. For example, when an object is detected in the overlap region of the first and second images, the controller 180 gives higher reliability to an image in which the horizontal movement of the object in the first and second images is greater. can do.

If the reliability is determined based on the size of the object, the controller 180 may assign higher reliability to an image in which the number of pixels occupied by the object is greater in the plurality of images. Alternatively, in the plurality of images, the controller 180 may further assign a weight to an image having a large virtual rectangular area surrounding the object. For example, when an object is detected in the overlap region of the first and second images, the controller 180 may give higher reliability to an image in which the number of pixels occupied by the object in the first and second images is greater. can

Next, the controller 180 generates an around-view image based on the synthesized image (S760). Here, the around-view image may be an image converted into a top-view image by synthesizing images received from a plurality of cameras 110a, 110b, 110c, and 110d for capturing images around the vehicle.

For example, the controller 180 may synthesize a plurality of images using a look-up table (LUT) and convert the images into an around-view image. The lookup table is a table storing the relationship between which one pixel of the synthesized image corresponds to which pixel of the four original images.

Thereafter, the controller 180 generates a virtual vehicle image in the around-view image (S770). Specifically, the controller 180 overlays a virtual vehicle image on the around-view image.

Next, the controller 180 transmits the compressed data to the display apparatus 200 to display the around-view image (S780).

Meanwhile, the controller 180 may overlay and display an image corresponding to the object detected in step S730 on the around-view image. The controller 180 may overlay an image corresponding to the tracked object on the around-view image and display it.

On the other hand, if the object is not detected in step S730 or the object is not located in the overlap area in step S740, the controller 180 blends the overlap area according to a preset ratio to synthesize an image (S790).

8 to 11 are diagrams referenced for explaining an operation of generating an around-view image by synthesizing a plurality of images, according to an embodiment of the present invention.

8 illustrates a case in which an object is not detected in a plurality of images according to an embodiment of the present invention.

Referring to FIG. 8 , when a plurality of cameras is configured of four, four overlapping regions 810 , 820 , 830 , and 840 are generated. If an object is not detected in a plurality of images, the controller 180 blends all overlapping regions 810 , 820 , 830 , and 840 to synthesize them. By synthesizing a plurality of images by blending the overlap regions 810 , 820 , 830 , and 840 , a natural image may be provided to the vehicle occupant.

9 illustrates a case in which an object is detected in a non-overlapping area according to an embodiment of the present invention.

Referring to FIG. 9 , when an object is detected in regions 950 , 960 , 970 , 980 other than the overlap regions 910 , 920 , 930 and 940 , the controller 180 controls the overlap regions 910 , 920 , 930 , 940) is synthesized by blending.

10 illustrates a case in which an object is detected in an overlap area according to an embodiment of the present invention.

Referring to FIG. 10 , when an object 1050 is detected in overlap areas 1010 , 1020 , 1030 , and 1040 , the controller 180 processes the overlap area based on object detection information to synthesize an image. Specifically, when an object is detected in the overlap region of the plurality of images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the plurality of images. Thereafter, the controller 180 determines which image among the plurality of images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. For example, the controller 180 processes the overlap area only with images having higher reliability among the plurality of images.

11 is a diagram referenced for explaining an operation of granting reliability when an object is detected in an overlap area according to an embodiment of the present invention.

Referring to FIG. 11 , when an object is detected in the overlap region of the first and second images, the controller 180 compares the moving speed, the moving direction, or the size of the object in the first and second images. The controller 180 determines which image among the first and second images has higher reliability based on the comparison result. The controller 180 processes the overlap area based on the reliability. The controller 180 processes the overlap region only with the image having higher reliability among the first and second images.

When the objects 1110 and 1120 are detected in the overlap region of the first image and the second image, the controller 180 may determine reliability based on the moving speed of the objects 1110 and 1120 . 11A , when the moving speed of the object 1110 in the first image is greater than the moving speed of the object 1120 in the second image, the controller 180 controls the overlap region only in the first image. can be processed. Here, the movement speed may be determined as the movement amount of the pixel per unit time of the object in the image.

When the objects 1130 and 1140 are detected in the overlap region of the first image and the second image, the controller 180 may determine reliability based on the moving directions of the objects 1130 and 1140 . 11B , when the object 1130 in the first image moves in the horizontal direction and the object 1140 in the second image moves in the vertical direction, the controller 180 controls the first The overlap area can be processed only with the image. This is because, in the case of vertical movement, when an actual object approaches a vehicle, only the size of the object increases, which is disadvantageous compared to horizontal movement when determining object detection and tracking.

When the objects 1150 and 1160 are detected in the overlap region of the first image and the second image, the controller 180 may determine reliability based on the sizes of the objects 1150 and 1160 . As shown in (c) of FIG. 11 , when the size of the object 1150 in the first image is larger than the size of the object 1160 in the second image, the controller 180 sets the overlap region only in the first image. can be processed The size of the object may be determined based on the number of pixels the object finds in the image. Alternatively, the size of the object may be determined based on the size of a rectangle surrounding the object.

In the above, preferred embodiments of the present invention have been shown and described, but the present invention is not limited to the specific embodiments described above, and in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

10: vehicle
110: camera
120: first input unit
130: alarm unit
140: first communication unit
160: first memory
180: control unit
200: display device
220: second input unit
240: second communication unit
250: display unit
255: sound output unit
260: second memory
280: processor

Claims (14)

a plurality of cameras; and
Receive a plurality of images related to the vehicle surroundings from the plurality of cameras, determine whether an object is detected in at least one of the plurality of images, and determine whether the object is located in at least one of the overlapping areas of the plurality of images ,
When the object is located in the overlap area, processing the overlap area based on the object detection information,
When the object is not detected or the object is not located in the overlap area, the overlap area is blended according to a preset ratio,
When the object is detected in the overlap region of the plurality of images, by comparing the moving speed, the moving direction, or the size of the object in the plurality of images, it is determined which image among the plurality of images has higher reliability, a control unit that processes the overlap area based on the reliability;
vehicle including. The method of claim 1,
The plurality of cameras includes a first camera that acquires a first image, a second camera that acquires a second image, a third camera that acquires a third image, and a fourth camera that acquires a fourth image. delete The method of claim 1,
The control unit is
A vehicle that processes the overlap area only with the image having the higher reliability among the plurality of images. 5. The method of claim 4,
The control unit is
When the reliability is determined based on the moving speed, in the plurality of images, a vehicle for giving higher reliability to an image in which a pixel movement amount per unit time of the object is larger. 5. The method of claim 4,
The control unit is
When the reliability is determined based on the moving direction, in the plurality of images, a vehicle giving higher reliability to an image having a greater horizontal movement. 5. The method of claim 4,
The control unit is
When the reliability is determined based on the size, a vehicle for giving higher reliability to an image in which the number of pixels occupied by the object in the plurality of images is greater. Receiving each of a plurality of images related to the vehicle surroundings from a plurality of cameras;
determining whether an object is detected in at least one of the plurality of images;
determining whether the object is located in at least one of the overlapping regions of the plurality of images;
When the object is located in the overlapping area, by comparing the moving speed, moving direction, or size of the object in the plurality of images, it is determined which image has higher reliability among the plurality of images, and based on the reliability processing the overlap area; and
and blending the overlap area according to a preset ratio when the object is not detected or the object is not located in the overlap area. 9. The method of claim 8,
The plurality of cameras may include a first camera configured to acquire a first image, a second camera configured to acquire a second image, a third camera configured to acquire a third image, and a fourth camera configured to acquire a fourth image. control method. delete 9. The method of claim 8,
The processing of the overlap area based on the object detection information includes:
A method of controlling a vehicle in which the overlap area is processed only with the image having the higher reliability among the plurality of images. 12. The method of claim 11,
The processing of the overlap area based on the object detection information includes:
When the reliability is determined based on the moving speed, in the plurality of images, a method of controlling a vehicle to give higher reliability to an image in which a pixel movement amount per unit time of the object is greater. 12. The method of claim 11,
The processing of the overlap area based on the object detection information includes:
When the reliability is determined based on the moving direction, in the plurality of images, a vehicle control method for giving higher reliability to an image having a greater horizontal movement. 12. The method of claim 11,
The processing of the overlap area based on the object detection information includes:
When the reliability is determined based on the size, in the plurality of images, the method of controlling a vehicle to give higher reliability to an image having a larger number of pixels occupied by the object.

Images