KR20160089222A - vehicle and control method thereof - Google Patents

Abstract

The present invention relates to a vehicle which comprises: a plurality of cameras; and a control unit receiving a plurality of images relative to the surroundings of a vehicle from the cameras, determining whether at least one object of the images is detected, and determining whether the object is placed in at least one of overlapped areas of the images, to treat the overlapped areas based on object detection information when the object is placed in the overlapped areas, and to blend the overlapped areas in accordance with a predetermined ratio when the object is not detected, or the object in not placed in the overlapped areas.

Description

[0001] VEHICLE AND CONTROL METHOD [0002]

The present invention relates to a vehicle equipped with an AVM (Around View Monitoring) device for displaying an image around a vehicle.

In recent years, an AVM (Around View Monitoring) system is mounted on a vehicle in accordance with the upgrading and multi-functionalization of a vehicle.

The AVM device acquires images of the surroundings of the vehicle through a camera mounted on all sides of the vehicle, and can display the surroundings of the vehicle through a display device installed in the room without the driver looking back or looking around when the vehicle is parked System. In addition, the surround view monitoring system also synthesizes the surrounding images to provide an around view as if viewed from above the vehicle. Using these surround view monitoring systems, the driver can grasp the situation around the vehicle at a glance, park securely, or go a narrow path.

On the other hand, the AVM apparatus is used as a parking assisting apparatus, but it is also possible to detect an object based on an image obtained through a plurality of cameras. It is necessary to study an operation of detecting an object through a plurality of cameras provided in the AVM apparatus.

Preceding literature on AVM systems: Publication No. 10-2013-0028230

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and it is an object of the present invention to provide a vehicle that performs object detection on an image received from a plurality of cameras.

The problems of the present invention are not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, a vehicle according to an embodiment of the present invention receives a plurality of images related to a vehicle periphery from a plurality of cameras and the plurality of cameras, determines whether or not an object is detected in at least one of the plurality of images And determines whether the object is located in at least one of the overlap regions of the plurality of images. If the object is located in the overlap region, the overlap region is processed based on the object detection information, And blending the overlap region according to a predetermined ratio when the object is not located in the overlap region.

The plurality of cameras may include a first camera that acquires a first image, a second camera that acquires a second image, a third camera that acquires a third image, and a fourth camera that acquires a fourth image .

Meanwhile, the control unit may compare the moving speed, moving direction, or size of the object in the plurality of images when the object is detected in the overlap region of the plurality of images, Is determined to be higher, and processes the overlap region based on the reliability.

Meanwhile, the control unit processes the overlap region only with the image having the higher reliability among the plurality of images.

Meanwhile, when the reliability is determined on the basis of the moving speed, the control unit gives higher reliability to the image having a larger pixel shift amount per unit time of the object in the plurality of images.

Meanwhile, when the reliability is determined on the basis of the moving direction, the control unit gives a higher reliability to an image in which motion in the horizontal direction is larger in the plurality of images.

On the other hand, when the reliability is determined based on the size, the control unit gives a higher reliability to an image having a larger number of pixels occupied by the object in the plurality of images.

A method of controlling a vehicle according to an embodiment of the present invention includes the steps of: receiving a plurality of images related to a vehicle periphery from a plurality of cameras; determining whether an object is detected in at least one of the plurality of images; The method comprising the steps of: determining whether the object is located in at least one of overlapping regions of a plurality of images; processing the overlapping region based on the object detection information when the object is located in the overlapping region; And blending the overlap region according to a predetermined ratio when the object is not located in the overlap region.

The plurality of cameras may include a first camera that acquires a first image, a second camera that acquires a second image, a third camera that acquires a third image, and a fourth camera that acquires a fourth image .

The step of processing the overlap area based on the object detection information may include a step of detecting a moving speed, a moving direction or a size of the object in the plurality of images when the object is detected in the overlap area of the plurality of images Determines which of the plurality of images has higher reliability, and processes the overlap region based on the reliability.

On the other hand, the step of processing the overlap region based on the object detection information may process the overlap region only with the image having the higher reliability among the plurality of images.

Meanwhile, the step of processing the overlap area based on the object detection information may include a step of, when the reliability is determined on the basis of the moving speed, determining whether the pixel shift amount per unit time of the object in the plurality of images is higher Thereby giving credibility.

The step of processing the overlap area based on the object detection information may further include the step of, when the reliability is determined based on the moving direction, giving a higher reliability to the image in which the motion in the horizontal direction is larger in the plurality of images do.

The step of processing the overlap region based on the object detection information may further include the step of, when the reliability is determined on the basis of the size, giving a higher reliability to an image having a larger number of pixels occupied by the object in the plurality of images do.

The details of other embodiments are included in the detailed description and drawings.

According to an embodiment of the present invention, there is one or more of the following effects.

The vehicle according to the embodiment of the present invention has an effect of detecting an object in an acquired image and tracking the detected object to prevent an accident in advance.

The vehicle according to the embodiment of the present invention has an effect of providing a natural AVM screen by processing the overlap area based on the object detection information even when the object is located in the overlap area.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a view showing the appearance of a vehicle having a plurality of cameras according to an embodiment of the present invention; FIG.
Fig. 2 is a view schematically showing positions of a plurality of cameras attached to the vehicle of Fig. 1. Fig.
Figure 3A illustrates an example of an ambient view image based on images photographed by the plurality of cameras of Figure 2
FIG. 3B is a diagram referred to explain overlap regions according to an embodiment of the present invention. FIG.
4 is a block diagram of a vehicle according to an embodiment of the present invention.
5 is a block diagram of a display device according to an embodiment of the present invention.
6 is a detailed block diagram of a control unit according to an embodiment of the present invention.
7 is a flowchart illustrating the operation of the vehicle according to the embodiment of the present invention.
Fig. 8 is a diagram referred to explain an operation of giving reliability when an object is detected in an overlap region, according to an embodiment of the present invention. Fig.
Figs. 9 to 11 are diagrams referred to in describing an operation of composing a plurality of images to generate an ambient view image, according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The vehicle described in the present specification may be a concept including both an internal combustion engine vehicle having an engine as a power source, a hybrid vehicle having an engine and an electric motor as a power source, and an electric vehicle having an electric motor as a power source.

In the following description, the left side of the vehicle means the left side in the running direction of the vehicle, that is, the driver's seat side, and the right side of the vehicle means the right side in the running direction of the vehicle, that is, the assistant's seat side.

On the other hand, the AVM apparatus described in the present specification may be an apparatus provided with a plurality of cameras and combining a plurality of images photographed by a plurality of cameras to provide an overview image. In particular, it may be a device that provides a top view or a bird eye view based on a vehicle. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an AVM apparatus and a vehicle including the AVM apparatus according to various embodiments of the present invention will be described.

On the other hand, in the present specification, the exchange of data can be performed through the vehicle communication network. Here, the vehicle communication network is preferably CAN. According to an embodiment, the vehicle communication network may be constructed using an Ethernet protocol.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a view showing the appearance of a vehicle having a plurality of cameras according to an embodiment of the present invention; FIG.

1, the vehicle 10 includes wheels 20FR, 20FL, 20RL, ... rotated by a power source, a steering wheel 30 for adjusting the traveling direction of the vehicle 10, (110a, 110b, 110c, 110d) mounted on the camera body (not shown). On the other hand, only the left camera 110a and the front camera 110d are shown in FIG.

The plurality of cameras 110a, 110b, 110c, and 110d may be activated when the speed of the vehicle is less than or equal to a predetermined speed, or when the vehicle is propelled, to obtain a shot image, respectively. The image obtained by the plurality of cameras may be signal processed within the control unit (180 in FIG. 4) or the processor (280 in FIG. 5).

Fig. 2 is a view schematically showing the positions of a plurality of cameras attached to the vehicle of Fig. 1, and Fig. 3a illustrates an example of an around view image based on images photographed by the plurality of cameras of Fig.

2, a plurality of cameras 110a, 110b, 110c, and 110d may be disposed on the left, rear, right, and front of the vehicle, respectively.

In particular, the left camera 110a and the right camera 110c may be disposed in a case surrounding the left side mirror and a case surrounding the right side mirror, respectively.

On the other hand, the rear camera 110b and the front camera 110d can be disposed in the vicinity of the trunk switch and in the vicinity of the ambulance or the ambulance, respectively.

Each of the plurality of images photographed by the plurality of cameras 110a, 110b, 110c and 110d is transmitted to a control section (180 in Fig. 4) or the like in the vehicle 10 and the control section (180 in Fig. 4) To generate an ambient view image.

FIG. 3A illustrates an example of an around view image 810. FIG. The surround view image 810 includes a first image area 110ai from the left camera 110a, a second image area 110bi from the rear camera 110b, a third image area 110b from the right camera 110c 110ci, and a fourth image area 110di from the front camera 110d.

On the other hand, when a surround view image is generated from a plurality of cameras, a boundary portion between each image area occurs. Such boundary portions can be blended with the image so that they can be displayed naturally.

On the other hand, the boundaries 111a, 111b, 111c, and 111d may be displayed at the boundaries of the plurality of images.

FIG. 3B is a diagram referred to explain overlap regions according to an embodiment of the present invention. FIG.

Referring to FIG. 3B, a plurality of cameras can use a wide-angle lens. Thereby, an overlap region may occur in an image obtained from a plurality of cameras. For example, a first overlap region 112a may be generated in a first image obtained by the first camera 110a and a second image obtained by the second camera 110b. In addition, a second overlap region 112b may be generated in the second image obtained by the second camera 110b and the third image obtained by the third camera 110c. In addition, a third overlap region 112c may be generated in the third image obtained by the third camera 110c and the fourth image obtained by the fourth camera 110d. In addition, a fourth overlap region 112d may be generated in the fourth image obtained by the fourth camera 110d and the first image obtained by the first camera 110a.

On the other hand, when the object is located in the first to fourth overlap regions 112a, 112b, 112c, and 112d, two objects may appear or disappear when converting into the surround view image. In such a case, a problem may arise in object detection and a problem that correct information can not be transmitted to the occupant may arise.

4 is a block diagram of a vehicle according to an embodiment of the present invention.

4, the vehicle 10 includes a plurality of cameras 110a, 110b, 110c and 110d, a first input unit 120, an alarm unit 130, a first communication unit 140, a display device 200, And may include a first memory 160 and a controller 180.

The plurality of cameras may include first to fourth cameras 110a, 110b, 110c, and 110d. The first camera 110a acquires the left surroundings image of the vehicle. And the second camera 110b acquires the vehicle rear periphery image. And the third camera 110c acquires the surroundings of the right side room of the vehicle. And the fourth camera 110d acquires the vehicle front periphery image.

The plurality of images acquired by the first to fourth cameras 110a, 110b, 110c, and 110d are transmitted to the controller 180. [

Meanwhile, the first to fourth cameras 110a, 110b, 110c, and 110d each include a lens and an image sensor (for example, a CCD or a CMOS). Here, the lens is preferably a fisheye lens having a wide angle of 180 DEG or more.

The first input unit 120 may receive a user input. The first input unit 120 may include means for accepting input from the outside such as a touch pad, a physical button, a dial, a slider switch, and a click wheel. The user input received through the first input unit 120 is transmitted to the controller 180.

The alarm unit 130 outputs an alarm in accordance with the information processed by the control unit 180. The alarm unit 130 may include a sound output unit and a display. The audio output unit can output audio data under the control of the control unit 180. [ The sound output unit may include a receiver, a speaker, a buzzer, and the like. The display displays the alarm information on the screen under the control of the control unit 180.

On the other hand, the alarm unit 130 can output an alarm based on the position of the detected object. Meanwhile, the display included in the alarm unit 130 may be a concept including a cluster or a head up display (HUD) on the inside of the vehicle interior.

The first communication unit 140 performs communication with an external electronic device. The first communication unit 140 may exchange data with an external server, a neighboring vehicle, an external base station, or the like. The first communication unit 140 may include a communication module that enables communication with an external electronic device. Here, the communication module may use a known technique.

Meanwhile, the first communication unit 140 may include a short-range communication module, and may exchange data with a portable terminal or the like of the passenger through the short-range communication module. The first communication unit 140 may transmit the overview view screen to the portable terminal of the passenger. Also, the first communication unit 140 may transmit a control command received from the portable terminal to the controller 180. Meanwhile, the first communication unit 140 may transmit information according to object detection to the portable terminal. In this case, the portable terminal can output an alarm notifying the object detection through vibration, sound output, or the like.

The display device 200 releases the compressed image and displays the surround view image. The display device 200 may be an AVN (Audio Video Navigation). The configuration of the display device 200 will be described in detail with reference to FIG.

The first memory 160 stores data supporting various functions of the vehicle 10. [ The first memory 160 may store a plurality of application programs driven by the vehicle 10, data for operation of the vehicle 10, and commands.

The first memory 160 may include a high speed random access memory. The first memory 160 may also include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices, but may include readable storage media .

For example, the first memory 160 may include an EEP-ROM (Electronically Erasable and Programmable Read Only Memory), but is not limited thereto. In the EEP-ROM, information can be written and erased by the control unit 180 during the operation of the control unit 180. [ The EEP-ROM may be a storage device in which information stored in the EEP-ROM is held without being erased even when the power supply of the controller is turned off and the power supply is stopped.

Meanwhile, the first memory 160 may store images obtained from the plurality of cameras 110a, 110b, 110c, and 110d. For example, when a collision of the vehicle 10 is detected, an image obtained from the plurality of cameras 110a, 110b, 110c, and 110d may be stored.

The control unit 180 controls the overall operation of each unit in the vehicle 10. [ The control unit 180 may perform various functions for controlling the vehicle 10 and may execute or execute various software programs and / or sets of instructions stored in the first memory 160 for processing data. The control unit 180 may process the signal based on the information stored in the first memory 160.

The control unit 180 preprocesses images received from the plurality of cameras 110a, 110b, 110c, and 110d. The control unit 180 removes noise of the image using various filters or histogram equalization. On the other hand, the preprocessing of such an image is not necessarily required, and may be omitted depending on the state of the image and the purpose of image processing.

The control unit 180 can detect the object based on the preprocessed image. Here, the object may be a concept including a pedestrian, an obstacle, a nearby vehicle, and the like. Meanwhile, the surrounding view screen displayed through the display device 200 may correspond to a part of the original image obtained through the plurality of cameras 110a, 110b, 110c, and 110d. The control unit 180 may perform object detection based on the entire original image including the image displayed on the display device 200. [

The control unit 180 compares the detected object with the object stored in the first memory 160, classifies and confirms the object.

The control unit 180 tracks the detected object. For example, the object tracking unit 430 sequentially identifies an object in the acquired images, calculates a motion or a motion vector of the identified object, Movement of an object, and the like can be tracked.

The control unit 180 processes the overlap area based on the object detection information to synthesize the images.

When an object is detected in an overlap area of a plurality of images, the control unit 180 compares the movement speed, the movement direction, or the size of the object in the plurality of images. The control unit 180 determines which of the plurality of images has higher reliability based on the comparison result. The control unit 180 processes the overlap area based on the reliability. The control unit 180 processes the overlap area only with the images of higher reliability among the plurality of images. For example, when an object is detected in the overlap area of the first and second images, the controller 180 compares the moving speed, moving direction, or size of the object in the first and second images. The control unit 180 determines which one of the first and second images is more reliable based on the comparison result. The control unit 180 processes the overlap area based on the reliability. The control unit 180 processes the overlap region only with the higher reliability image among the first and second images.

If the reliability is determined on the basis of the moving speed of the object, the control unit 180 can give a higher reliability to an image having a larger moving amount of pixels per unit time of the object in a plurality of images. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 determines that the moving amount of the pixel per unit time of the object in the first and second images is higher .

If the reliability is determined on the basis of the moving direction of the object, the control unit 180 can give a higher reliability to an image in which the motion of the object in the horizontal direction is larger in a plurality of images. In the case of the longitudinal direction movement, the actual object approaches the vehicle, which is disadvantageous compared with the lateral direction motion when judging object detection and tracking because the size of the object is increased. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 gives a higher reliability to the image in which the horizontal movement of the object in the first and second images is larger can do.

If the reliability is determined on the basis of the size of the object, the control unit 180 can give a higher reliability to an image having a larger number of pixels occupied by the object in a plurality of images. Alternatively, the control unit 180 may assign a weight to an image having a large virtual rectangular area surrounding the object, from a plurality of images. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 gives a higher reliability to the image in which the number of pixels occupied by the object in the first and second images is larger .

On the other hand, when the object is not detected or the object is not located in the overlap region, the control section 180 blends the overlap region according to a predetermined ratio to synthesize the image.

The control unit 180 generates the surround view image based on the synthesized image.

Here, the surround view image may be an image obtained by synthesizing the images received from the plurality of cameras 110a, 110b, 110c, and 110d that photograph the surroundings of the vehicle, and converting them into a top view image.

For example, the control unit 180 can synthesize a plurality of images by using a look up table (LUT), and switch to an ambient view image. The look-up table is a table that stores the relationship between one pixel of the composite image and which pixel of the four original images.

Thereafter, the control unit 180 generates a virtual vehicle image on the surround view image. Specifically, the control unit 180 overlays the virtual vehicle image on the surround view image.

Next, the control unit 180 transmits the compressed data to the display device 200 to display the surround view image.

On the other hand, the controller 180 may overlay the image corresponding to the object detected in step S730 on the surround view image. The control unit 180 may overlay the image corresponding to the object to be tracked on the surround view image.

5 is a block diagram of a display device according to an embodiment of the present invention.

5, the display device 200 includes a second input unit 220, a second communication unit 240, a display unit 250, an audio output unit 255, a second memory 260, . ≪ / RTI >

The second input unit 220 can receive a user input. The second input unit 220 may include means for accepting input from the outside such as a touch pad, a physical button, a dial, a slider switch, and a click wheel. The user input received through the first input unit 220 is transmitted to the controller 180.

The second communication unit 240 can communicate with an external electronic device to exchange data. For example, the second communication unit 240 may be connected to a broadcasting company server to receive broadcasting contents. For example, the second communication unit 240 may be connected to a traffic information providing server to receive TPEG (Transport Protocol Experts Group) information.

The display unit 250 displays information processed by the processor 270. [ For example, the display unit 250 may display execution screen information of an application program driven by the processor 270 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

Meanwhile, when the touch pad has a mutual layer structure with the display unit 250, it can be called a touch screen. The touch screen may function as the second input unit 220.

The sound output unit 255 can output audio data. The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The second memory 260 stores data supporting various functions of the display device 200. [ The second memory 260 may store a plurality of application programs driven by the display device 200, data for operation of the display device 200, and commands.

The second memory 260 may include a high-speed random access memory. The second memory 260 may also include non-volatile memory such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state memory devices, but may include readable storage media .

For example, the second memory 260 may include, but is not limited to, Electronically Erasable and Programmable Read Only Memory (EEP-ROM). The EEP-ROM may be written and erased by the processor 280 during the operation of the processor 280. [ The EEP-ROM may be a storage device in which information stored in the EEP-ROM is held without being erased even when the power supply of the controller is turned off and the power supply is stopped.

The processor 280 controls the overall operation of each unit in the display device 200. The processor 280 may perform various functions for controlling the display device 20 and may execute or perform various software programs and / or sets of instructions stored in the second memory 260 to process the data. The processor 280 may process the signal based on information stored in the second memory 260.

The processor 280 displays the surround view image.

6 is a detailed block diagram of a control unit according to an embodiment of the present invention.

6, the control unit 180 includes a preprocessing unit 610, an object detecting unit 620, an object identifying unit 630, an object tracking unit 640, an overlap area processing unit 650, 660 < / RTI >

The preprocessing unit 610 preprocesses images received from the plurality of cameras 110a, 110b, 110c, and 110d. The pre-processing unit 610 removes noise of an image using various filters or histogram equalization. On the other hand, the preprocessing of such an image is not necessarily required, and may be omitted depending on the state of the image and the purpose of image processing.

The object detecting unit 620 can detect an object based on the preprocessed image. Here, the object may be a concept including a pedestrian, an obstacle, a nearby vehicle, and the like. Meanwhile, the surrounding view screen displayed through the display device 200 may correspond to a part of the original image obtained through the plurality of cameras 110a, 110b, 110c, and 110d. The object detecting unit 620 can perform object detection based on the entire original image including the image displayed on the display device 200. [

The object verification unit 630 compares the detected object with the object stored in the first memory 160, classifies the object, and confirms the object.

The object tracking unit 640 tracks the detected object. For example, the object tracking unit 430 sequentially identifies an object in the acquired images, calculates a motion or a motion vector of the identified object, Movement of an object, and the like can be tracked.

The overlap area processing unit 650 processes the overlap area based on the object detection information to synthesize the images.

When an object is detected in the overlap region of a plurality of images, the overlap region processing unit 650 compares the movement speed, the movement direction, or the size of the object in the plurality of images. The overlap area processing unit 650 determines which of the plurality of images has higher reliability based on the comparison result. The overlap area processing unit 650 processes the overlap area based on the reliability. The overlap area processing unit 650 processes the overlap area with only images having higher reliability among a plurality of images. For example, when an object is detected in the overlap area of the first and second images, the overlap area processing unit 650 compares the movement speed, the movement direction, or the size of the object in the first and second images. The overlap area processor 650 determines which one of the first and second images is more reliable based on the comparison result. The overlap area processing unit 650 processes the overlap area based on the reliability. The overlap region processing unit 650 processes the overlap region only with the images having higher reliability among the first and second images.

If the reliability is determined on the basis of the moving speed of the object, the overlap area processing unit 650 can give a higher reliability to the image having the larger moving amount of pixels per unit time of the object in a plurality of images. For example, when an object is detected in the overlap area of the first and second images, the overlap area processing unit 650 may detect that the moving amount of the pixel per unit time of the object in the first and second images is higher Reliability can be given.

If reliability is determined on the basis of the moving direction of the object, the overlap area processing unit 650 can give a higher reliability to an image in which the motion of the object in the horizontal direction is larger in a plurality of images. In the case of the longitudinal direction movement, the actual object approaches the vehicle, which is disadvantageous compared with the lateral direction motion when judging object detection and tracking because the size of the object is increased. For example, when an object is detected in the overlap region of the first and second images, the overlap region processing unit 650 may determine that the motion of the object in the first and second images in the horizontal direction has a higher reliability Can be given.

If the reliability is determined based on the size of the object, the overlap area processing unit 650 can give a higher reliability to an image having a larger number of pixels occupied by the object in a plurality of images. Alternatively, the overlap area processing unit 650 can give a weight to an image having a large virtual rectangular area surrounding the object in a plurality of images. For example, when an object is detected in the overlap area of the first and second images, the overlap area processing unit 650 sets the reliability of the image in which the number of pixels occupied by the object in the first and second images is larger .

On the other hand, when the object is not detected or the object is not located in the overlap region, the overlap region processing section 650 blends the overlap region according to a predetermined ratio to synthesize the image.

The surround view image generation unit 660 generates an ambient view image based on the synthesized image.

Here, the surround view image may be an image obtained by synthesizing the images received from the plurality of cameras 110a, 110b, 110c, and 110d that photograph the surroundings of the vehicle, and converting them into a top view image.

For example, the surround view image generating unit 660 can synthesize a plurality of images using a lookup table (LUT), and switch to an ambient view image. The look-up table is a table that stores the relationship between one pixel of the composite image and which pixel of the four original images.

Subsequently, the surround view image generating unit 660 generates a virtual vehicle image on the surrounding view image. Specifically, the surround view image generator 660 overlaid a virtual vehicle image on the surround view image.

Next, the surround view image generator 660 transmits the compressed data to the display device 200 to display the surround view image.

On the other hand, the surround view image generator 660 can overlay the image corresponding to the object detected in step S730 on the surround view image. The surround view image generator 660 may overlay the image corresponding to the object to be tracked on the surround view image.

7 is a flowchart illustrating the operation of the vehicle according to the embodiment of the present invention.

Referring to FIG. 7, the controller 180 receives the first through fourth images from the plurality of cameras 110a, 110b, 110c, and 110d (S710).

The control unit 180 performs preprocessing on each of the plurality of received images (S720). The control unit 180 removes noise of the image using various filters or histogram equalization. On the other hand, the preprocessing of such an image is not necessarily required, and may be omitted depending on the state of the image and the purpose of image processing.

The control unit 180 determines whether an object is detected based on the received first through fourth images or the preprocessed image (S730). Here, the object may be a concept including a pedestrian, an obstacle, a nearby vehicle, and the like.

When the object is detected, the control unit 180 determines whether the object is located in the overlap region (S740). Specifically, the control unit 180 determines whether the object is located in one of the first to fourth overlap areas 112a, 112b, 112c, and 112d described with reference to FIG. 3B.

When the object is located in the overlap region 112a, 112b, 112c, and 112d, the control unit 180 processes the overlap region based on the object detection information to synthesize the image (S750).

When an object is detected in an overlap area of a plurality of images, the control unit 180 compares the movement speed, the movement direction, or the size of the object in the plurality of images. The control unit 180 determines which of the plurality of images has higher reliability based on the comparison result. The control unit 180 processes the overlap area based on the reliability. The control unit 180 processes the overlap area only with the images of higher reliability among the plurality of images. For example, when an object is detected in the overlap area of the first and second images, the controller 180 compares the moving speed, moving direction, or size of the object in the first and second images. The control unit 180 determines which one of the first and second images is more reliable based on the comparison result. The control unit 180 processes the overlap area based on the reliability. The control unit 180 processes the overlap region only with the higher reliability image among the first and second images.

If the reliability is determined on the basis of the moving speed of the object, the control unit 180 can give a higher reliability to an image having a larger moving amount of pixels per unit time of the object in a plurality of images. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 determines that the moving amount of the pixel per unit time of the object in the first and second images is higher .

If the reliability is determined on the basis of the moving direction of the object, the control unit 180 can give a higher reliability to an image in which the motion of the object in the horizontal direction is larger in a plurality of images. In the case of the longitudinal direction movement, the actual object approaches the vehicle, which is disadvantageous compared with the lateral direction motion when judging object detection and tracking because the size of the object is increased. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 gives a higher reliability to the image in which the horizontal movement of the object in the first and second images is larger can do.

If the reliability is determined on the basis of the size of the object, the control unit 180 can give a higher reliability to an image having a larger number of pixels occupied by the object in a plurality of images. Alternatively, the control unit 180 may assign a weight to an image having a large virtual rectangular area surrounding the object, from a plurality of images. For example, when an object is detected in the overlap region of the first and second images, the control unit 180 gives a higher reliability to the image in which the number of pixels occupied by the object in the first and second images is larger .

Next, the controller 180 generates an overview image based on the synthesized image (S760). Here, the surround view image may be an image obtained by synthesizing the images received from the plurality of cameras 110a, 110b, 110c, and 110d that photograph the surroundings of the vehicle, and converting them into a top view image.

For example, the control unit 180 can synthesize a plurality of images by using a look up table (LUT), and switch to an ambient view image. The look-up table is a table that stores the relationship between one pixel of the composite image and which pixel of the four original images.

Thereafter, the control unit 180 generates a virtual vehicle image in the surround view image (S770). Specifically, the control unit 180 overlays the virtual vehicle image on the surround view image.

Next, the control unit 180 transmits the compressed data to the display device 200 to display the surround view image (S780).

On the other hand, the controller 180 may overlay the image corresponding to the object detected in step S730 on the surround view image. The control unit 180 may overlay the image corresponding to the object to be tracked on the surround view image.

If the object is not detected in step S730, or if the object is not located in the overlap area in step S740, the control unit 180 blends the overlapped area according to a preset ratio to synthesize the image (S790).

FIGS. 8 to 11 are diagrams for explaining an operation of composing a plurality of images to generate an overview image according to an embodiment of the present invention.

Fig. 8 shows a case where an object is not detected in a plurality of images, according to an embodiment of the present invention.

Referring to FIG. 8, when a plurality of cameras are configured with four cameras, four overlap regions 810, 820, 830 and 840 are generated. If an object is not detected in a plurality of images, the controller 180 blends and synthesizes all the overlap regions 810, 820, 830, and 840. By blending the overlap regions 810, 820, 830, and 840 to synthesize a plurality of images, a natural image can be provided to the vehicle occupant.

9 shows a case where an object is detected in an area other than an overlap area according to an embodiment of the present invention.

9, when an object is detected in areas 950, 960, 970 and 980 other than the overlap areas 910, 920, 930 and 940, the control part 180 controls the overlap areas 910, 920, 930, 940) are blended and synthesized.

10 shows a case where an object is detected in an overlap area according to an embodiment of the present invention.

Referring to FIG. 10, when an object 1050 is detected in the overlap regions 1010, 1020, 1030, and 1040, the control unit 180 processes the overlap region based on the object detection information to synthesize the images. Specifically, when an object is detected in the overlap area of a plurality of images, the control unit 180 compares the moving speed, the moving direction, or the size of the object in the plurality of images. Thereafter, the control unit 180 determines which of the plurality of images has higher reliability based on the comparison result. The control unit 180 processes the overlap area based on the reliability. For example, the control unit 180 processes overlapping regions only with images having higher reliability among a plurality of images.

Fig. 11 is a diagram referred to explain an operation of assigning reliability when an object is detected in an overlap region, according to an embodiment of the present invention. Fig.

Referring to FIG. 11, when an object is detected in the overlap region of the first and second images, the controller 180 compares the moving speed, moving direction, or size of the object in the first and second images. The control unit 180 determines which one of the first and second images is more reliable based on the comparison result. The control unit 180 processes the overlap area based on the reliability. The control unit 180 processes the overlap region only with the higher reliability image among the first and second images.

When the objects 1110 and 1120 are detected in the overlap region of the first image and the second image, the control unit 180 can determine the reliability based on the moving speed of the objects 1110 and 1120. 11A, when the moving speed of the object 1110 in the first image is larger than the moving speed of the object 1120 in the second image, the controller 180 controls the overlapping area Lt; / RTI > Here, the moving speed can be determined by the amount of movement of the pixel per unit time of the object in the image.

When the objects 1130 and 1140 are detected in the overlap region of the first image and the second image, the control unit 180 can determine the reliability based on the moving directions of the objects 1130 and 1140. 11 (b), when the object 1130 moves in the horizontal direction in the first image and the object 1140 moves in the vertical direction in the second image, the control unit 180 controls the first The overlap region can be processed only by the video. In the case of the vertical direction movement, the actual object approaches the vehicle, and since the size of the object becomes larger, it is disadvantageous in comparison with the horizontal direction movement when judging object detection and tracking.

When the objects 1150 and 1160 are detected in the overlap region of the first image and the second image, the control unit 180 can determine the reliability based on the sizes of the objects 1150 and 1160. 11 (c), when the size of the object 1150 in the first image is larger than the size of the object 1160 in the second image, the controller 180 sets the overlap area as the first image alone Can be processed. The size of the object can be determined based on the number of pixels that the object looks for in the image. Alternatively, the size of the object may be determined based on the size of the rectangle surrounding the object.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It should be understood that various modifications may be made by those skilled in the art without departing from the spirit and scope of the present invention.

10: Vehicle
110: camera
120: first input unit
130:
140: first communication section
160: first memory
180:
200: display device
220: second input section
240: second communication section
250:
255:
260: second memory
280: Processor

Claims (14)

A plurality of cameras; And
A plurality of images related to the surroundings of the vehicle are received from the plurality of cameras, and it is determined whether or not an object is detected in at least one of the plurality of images, and it is determined whether the object is located in at least one of overlapping regions of the plurality of images ,
Processing the overlap area based on the object detection information when the object is located in the overlap area,
And a controller for blending the overlap region according to a predetermined ratio when the object is not detected or the object is not located in the overlap region. The method according to claim 1,
The plurality of cameras include a first camera for acquiring a first image, a second camera for acquiring a second image, a third camera acquiring a third image, and a fourth camera acquiring a fourth image. 3. The method of claim 2,
Wherein,
Comparing the moving speed, the moving direction, or the size of the object in the plurality of images when the object is detected in the overlap region of the plurality of images, determining which one of the plurality of images is more reliable, And processing the overlap region based on the reliability. The method of claim 3,
Wherein,
And processes the overlap region with only the image having the higher reliability among the plurality of images. 5. The method of claim 4,
Wherein,
And when the reliability is determined on the basis of the moving speed, a higher reliability is given to the image in which the pixel shift amount per unit time of the object is larger in the plurality of images. 5. The method of claim 4,
Wherein,
And when the reliability is determined based on the moving direction, a higher reliability is given to the image in which the motion in the horizontal direction is larger in the plurality of images. 5. The method of claim 4,
Wherein,
And when reliability is judged on the basis of the size, a higher reliability is given to an image in which the number of pixels occupied by the object is larger in the plurality of images. Receiving a plurality of images related to the surroundings of the vehicle from a plurality of cameras, respectively;
Determining whether an object is detected in at least one of the plurality of images;
Determining whether the object is located in at least one of overlapping regions of the plurality of images;
Processing the overlap region based on the object detection information when the object is located in the overlap region; And
And blending the overlap region according to a predetermined ratio when the object is not detected or the object is not located in the overlap region. 9. The method of claim 8,
The plurality of cameras include a first camera for acquiring a first image, a second camera for acquiring a second image, a third camera for acquiring a third image, and a fourth camera for acquiring a fourth image Control method. 10. The method of claim 9,
Wherein the step of processing the overlap region based on the object detection information comprises:
Comparing the moving speed, moving direction, or size of the object in the plurality of images when the object is detected in the overlap region of the plurality of images to determine which one of the plurality of images has higher reliability, And processing the overlap region based on the reliability. 11. The method of claim 10,
Wherein the step of processing the overlap region based on the object detection information comprises:
And processing the overlap region only with the image having the higher reliability among the plurality of images. 12. The method of claim 11,
Wherein the step of processing the overlap region based on the object detection information comprises:
Wherein when reliability is determined based on the moving speed, a higher reliability is given to an image having a larger pixel shift amount per unit time of the object in the plurality of images. 12. The method of claim 11,
Wherein the step of processing the overlap region based on the object detection information comprises:
Wherein when the reliability is determined based on the moving direction, a higher reliability is given to an image in which motion in the horizontal direction is larger in the plurality of images. 12. The method of claim 11,
Wherein the step of processing the overlap region based on the object detection information comprises:
Wherein when the reliability is determined based on the size, a higher reliability is given to an image having a larger number of pixels occupied by the object in the plurality of images.

Images