KR102235951B1 - Imaging Apparatus and method for Automobile - Google Patents

Abstract

The present invention relates to a vehicle image generating apparatus and method capable of displaying images around a vehicle and inside a vehicle on a screen according to a user's selection, a driving state of a vehicle, a driver's driving state, etc., and a vehicle image generation according to the present invention The device includes a multi-camera unit including a plurality of wide-angle cameras attached to the vehicle, a plurality of image generating cores, and an image synthesizing unit, and each of the plurality of image generating cores inputs one of images captured by the wide-angle camera. As an image, a virtual camera having a viewpoint according to a set parameter is modeled, an image viewed from the viewpoint of the virtual camera is generated and output from the input image, and the image synthesizing unit is an image output from the plurality of image generating cores. An image generation unit that generates an output image by synthesizing according to a set screen configuration, each image generation core of the plurality of image generation cores sets a parameter including a viewpoint of the virtual camera required to generate an image, and the image It may include a processing unit for setting the screen configuration for image synthesis in the synthesis unit and a screen unit for outputting the output image generated by the image generation unit.
According to the present invention, by using a plurality of wide-angle cameras to capture the surrounding and/or indoor images of the vehicle, it is possible to generate and provide various types of images that enable the driver to safely drive, and also the driving speed or driving of the vehicle. It will be possible to help improve driving convenience and safety by automatically generating necessary images according to the direction and providing them to the driver.

Description

Vehicle image generation apparatus and method {Imaging Apparatus and method for Automobile}

The present invention relates to an apparatus and method for generating an image for a vehicle, and more particularly, an apparatus for generating an image for a vehicle capable of displaying images around a vehicle and inside the vehicle according to a user's selection, a driving state of a vehicle, a driver's driving state, etc. And a method.

For the safe operation of the vehicle, it is necessary for the driver to recognize and judge the surrounding situation. To this end, the front of the vehicle is designed to secure the driver's view as much as possible, and side mirrors or rearview mirrors are placed in the left/right and interior of the vehicle to secure side and rear views that the driver cannot see immediately. have. However, in the conventional rear/side recognition by a mirror, there is a considerable blind spot that the driver cannot see, and this has an inherent possibility of causing an accident.

Patents (Korean Patent Registration Nos. 10-1486670, 10-1660806) have been proposed to solve these problems, remove the side mirrors, and use a camera to obtain rear or side images and show them to the driver. Its application is limited to a side mirror substitute or supplement.

[1] KR10-1486670,'Digital camera side mirror,' Registration date: January 20, 2015 [2] KR10-1660806,'System for securing side and rear view of vehicle,' registration date September 22, 2016

It is an object of the present invention to display images around and/or inside the vehicle according to the user's selection, the driving state of the vehicle, the driving state of the driver, etc. in order to quickly recognize the situation around and inside the vehicle to the user. It is to provide an apparatus and method for generating an image for a type vehicle.

An image generating apparatus for a vehicle according to the present invention for achieving the above object includes a multi-camera unit including a plurality of wide-angle cameras attached to the vehicle, a plurality of image generating cores, and an image synthesizing unit, and the plurality of image generating cores Each takes one of the images captured by the wide-angle camera as an input image, models a virtual camera having a viewpoint according to a set parameter, generates and outputs an image viewed from the viewpoint of the virtual camera from the input image, The image synthesizing unit generates an output image by synthesizing the images output from the plurality of image generation cores according to a set screen configuration, and the image generation cores of the plurality of image generation cores are required to generate an image. A processing unit configured to set a parameter including a viewpoint of a virtual camera, set the screen configuration for image synthesis in the image synthesizing unit, and a screen unit to output the output image generated by the image generating unit. In addition, the vehicle sensor unit for obtaining driving state information of the vehicle and/or an input unit for receiving setting information and/or a request for generating the output image from a user, wherein the processing unit is input through the input unit. The parameters and/or the screen configuration may be set based on the set setting information, requirements, and/or the driving state information obtained from the vehicle sensor unit.

In addition, the image generating apparatus for a vehicle according to the present invention further comprises a storage unit for storing at least one of an image captured by the plurality of wide-angle cameras and an output image generated by the image generating unit, wherein the storage unit In the case of storing the image captured by the plurality of wide-angle cameras, the images captured by the plurality of wide-angle cameras are synchronized and stored by assigning the same index to the image captured by each of the wide-angle cameras in the same time zone, and the image generator When the output image generated in is stored, a parameter including a viewpoint of the virtual camera of the plurality of image generating cores set to generate the output image and the screen configuration may be stored together.

And the plurality of wide-angle cameras of the multi-camera unit include a front camera for photographing the front of the vehicle, a rear camera for photographing the rear of the vehicle, a side camera for photographing left and right sides of the vehicle, and an indoor photographing room. The processing unit may include a camera, and the processing unit is a front wide-angle view of the front camera, a top view, a bird's eye view, a blind spot view of the side camera, an adjacent lane view, a top view, and a rear wide-angle view of the rear camera. , A top view, a road image view, a driver's seat view, an assistant seat view, a front view, and an omnidirectional indoor view of the indoor camera may be set to be output on the screen unit.

In addition, the screen unit may be positioned on at least one of a left window, a right window, a center fascia, a room mirror, a windshield, and a rear of the front seat of the vehicle.

In more detail, the processing unit includes a pre-processing unit, a post-processing unit, and a main processing unit, and the pre-processing unit receives images captured by a plurality of wide-angle cameras of the multi-camera unit, and Pre-processed images are generated through a pre-processing process that performs at least one of defect correction, noise reduction, and wide dynamic range, and transfers the pre-processed images to the post-processing unit, and the For each of the pre-processed images received from the pre-processing unit, the post-processing unit includes brightness and color signal separation, edge enhancement, gamma correction, color correction, and contrast enhancement. A post-processing process including at least one is performed to generate post-processed images and transfer them to the image generation unit, and the main processing unit includes setting information input through the input unit, requirements, and the driving obtained from the vehicle sensor unit. Based on at least one of the state information, each image generation core of the plurality of image generation cores sets a parameter including a viewpoint of the virtual camera required to generate an image, and the image synthesis unit for image synthesis You can set the screen composition.

In addition, the main processing unit is set to perform the wide-angle correction on an image captured by at least one of the plurality of wide-angle cameras, and the exposure is less for at least one wide-angle camera set to perform wide-angle correction on the captured image. A parameter is set so that a low-exposure image and a high-exposure image with high exposure can be captured and transmitted to the preprocessor, and the preprocessor selects a pixel value corresponding to one of the low exposure image and the high exposure image for each pixel. Thus, the wide area correction can be performed by determining the value.

In addition, the post-processing unit includes at least one of a motion detector for detecting motion in the post-processed image, a lane detector for detecting a lane in the post-processed image, and an object tracker for tracking an object in the post-processed image, and the motion detector Generates at least one of the motion information obtained from, the lane information obtained from the lane detector, and the object tracking information obtained from the object tracker, and the main processing unit generates setting information, requirements, and the vehicle sensor input through the input unit. The parameter and/or the screen configuration may be set based on at least one of the driving state information obtained by the unit, the motion information generated by the post-processing unit, the lane information, and the object tracking information.

In addition, the main processing unit is configured to additionally synthesize a lane, a parking line, and/or a vehicle on the screen based on at least one of the motion information, the lane information, and the object tracking information generated by the post processing unit. A screen configuration can be set, and among the setting information input through the input unit, requirements, the driving state information obtained from the vehicle sensor unit, the motion information generated by the post-processing unit, the lane information, and the object tracking information Based on at least one, the parameter and/or the screen configuration may be set so that an image is output to the screen unit according to a preset scenario.

In addition, the preprocessor includes information related to at least one of auto white balance, auto exposure, and auto focus for each image captured by the plurality of wide-angle cameras from the received image. Is extracted and transmitted to the main processing unit, and the main processing unit sets parameters for photographing of the plurality of wide-angle cameras based on information related to at least one of automatic color temperature correction, automatic exposure, and auto focus received from the preprocessor. For setting, at least one of a shutter speed, color gains, image resolution, and frame rate may be used as a parameter for photographing the plurality of wide-angle cameras.

In addition, the preprocessor includes at least one of auto color temperature correction (Auto White Balance), auto exposure, and auto focus for each image captured by the plurality of wide-angle cameras from the received image. Related information is extracted and transmitted to the main processing unit, and the main processing unit includes information related to at least one of automatic color temperature correction, automatic exposure, and auto focus received from the preprocessor, setting information input through the input unit, and request Information, parameters for photographing the plurality of wide-angle cameras based on at least one of the driving state information acquired by the vehicle sensor unit, the motion information generated by the post-processing unit, the lane information, and the object tracking information Can be set.

In addition, when the storage unit is present, the post-processing unit may compress the post-processed images using a video codec and additionally transmit the compressed images to the storage unit.

In addition, the image generation core is a virtual camera model unit that models one or more virtual cameras having respective viewpoints and types according to parameters set by the processing unit, and an output that is an image viewed by each virtual camera of the one or more virtual cameras. A second curved surface model unit that provides a second curved surface used to add an image effect when mapping between an image and an input image that is an image captured by the wide-angle camera, and each virtual camera while removing distortion included in the input image And a camera projection model unit that provides a camera projection model for performing mapping between the output image and the input image for each, wherein the camera projection model unit By calculating an intersection point between a projection line connecting points and the second-order curved surface, and mapping the intersection point to a point on the input image using a distortion correction mapping function for removing distortion included in the input image, Mapping between the output image for each virtual camera and the input image may be performed.

In order to achieve the above object, the method for generating an image for a vehicle according to the present invention includes: capturing a plurality of images with a plurality of wide-angle cameras, configuring an output screen and setting an output image, configuring the output screen and generating the output image. Setting a parameter including a viewpoint of a virtual camera in order to do so, wherein each of a plurality of image generation cores in a virtual camera-based image generator uses one of the plurality of images as an input image, and includes the set viewpoint of the virtual camera. A virtual camera is modeled according to a parameter to be displayed, an image viewed from the viewpoint of the virtual camera is generated from the input image, and the images generated by the plurality of image generation cores are synthesized according to the set output screen configuration to generate an output image. It may include generating and displaying the output image on a screen.

In addition, at least one of the steps of acquiring driving state information of the vehicle, receiving input of the configuration of the output screen and user setting information for generating the output image from a user, and configuring and outputting the output screen Setting an image includes configuring an output screen and setting an output image based on at least one of the driving state information and the user setting information, and setting the parameter includes the driving state information and the user setting It may include setting a parameter based on at least one of the information, and the step of configuring an output screen and setting an output image based on at least one of the driving state information and the user setting information may include a vehicle among the driving state information. Setting the view point of the virtual camera by setting the zoom and tilt parameters of the virtual camera according to the progress speed, or by setting the zoom and pan parameters of the virtual camera according to the direction of vehicle travel among the driving state information to view the view of the virtual camera And setting a viewpoint of the virtual camera by setting zoom, pan, and tilt parameters of the virtual camera according to the user setting information.

In addition, the method may further include storing at least one of an image photographed by the plurality of wide-angle cameras and the output image, wherein this step is performed when the image photographed by the plurality of wide-angle cameras is stored, the It may include the step of synchronizing and storing images captured by the plurality of wide-angle cameras by assigning the same index to images captured by each wide-angle camera of the plurality of wide-angle cameras in the same time zone, and storing the output image, A parameter including a viewpoint of the virtual camera set to generate the output image may be stored together.

In addition, the step of configuring the output screen and setting the output image includes a front wide-angle view of the front camera, a top view, a bird's eye view, a blind spot view of the lateral camera, an adjacent lane view, a top view, and a rear wide-angle view of the rear camera. , A top view, a road image view, a driver's seat view of an indoor camera, an auxiliary seat view, a front view, and an omnidirectional indoor view.

In addition, the generating of the output image includes receiving a plurality of images photographed by the plurality of wide-angle cameras, and correcting the value of a defective pixel for each of the plurality of received images (Defect correction) and noise reduction (Noise Reduction). , And a pre-processing step of generating pre-processed images by performing at least one of wide dynamic range, separation of brightness and color signals for each of the pre-processed images generated in the pre-processing step, edge enhancement, and gamma A post-processing step of generating post-processed images by performing at least one of Gamma Correction, Color Correction, and Contrast Enhancement, each of a plurality of image generation cores in a virtual camera-based image generation unit Taking one of the post-processed images as an input image, modeling a virtual camera according to a parameter including a set viewpoint of the virtual camera, and generating an image viewed from the viewpoint of the virtual camera from the input image; and And generating an output image by synthesizing the images generated by the plurality of image generating cores according to the set output screen configuration.

In addition, the wide-area correction in the pre-processing step includes generating two wide-area correction images with different exposures by adjusting a shutter speed of a wide-angle camera, setting a wide-area correction threshold based on the two wide-area correction images, and Based on the correction threshold, determine whether to use the corresponding pixel value of the low-exposure image of the low-exposure image or the corresponding pixel value of the high-exposure image with high exposure for each pixel value in the image, and save it as exposure area information. And in order to generate a wide-area-corrected pre-processed image, one pixel of two images having different exposures input from the wide-angle camera selected according to the exposure area information is determined, wherein each pixel value of the wide-angle-corrected pre-processed image is determined. It may include the step of determining the value.

In addition, the step of setting the parameter includes detecting motion in the post-processed image to obtain motion information, detecting a lane in the post-processed image to obtain lane information, and tracking an object in the post-processed image. And at least one of obtaining object tracking information, and setting a parameter based on at least one of the motion information, the lane information, the object tracking information, the driving state information, and the user setting information. can do.

In addition, the pre-processing step includes at least one of auto color temperature correction (Auto White Balance), auto exposure, and auto focus for each image captured by the plurality of wide-angle cameras from the received image. The step of extracting related information, wherein the step of setting the parameter includes information related to at least one of the automatic color temperature correction, automatic exposure, and auto focus extracted in the preprocessing unit step, the motion information, the lane information, And setting parameters for photographing of the plurality of wide-angle cameras based on at least one of the object tracking information, the driving state information, and the user setting information.

In addition, each of a plurality of image generation cores in the virtual camera-based image generation unit uses one of the post-processed images as an input image, and models a virtual camera according to a parameter including a set viewpoint of the virtual camera, and the The step of generating an image viewed from the viewpoint of the virtual camera from the input image includes modeling a virtual camera according to the set parameters, in order to add a video effect when mapping between the image viewed by the virtual camera and the input image Modeling a second-order curved surface to be used, generating a camera projection model to perform mapping between the image viewed from the viewpoint of the virtual camera and the input image while removing distortion included in the input image, and the optical axis of the virtual camera Calculating an intersection point of the second-order curved surface and a projection line connecting the center and a point on the image viewed from the viewpoint of the virtual camera, and distortion correction mapping to remove the distortion included in the input image Mapping to a point on the input image using a function may be included.

Further, it may include a computer program stored in a computer-readable recording medium that executes a method according to any one of the above-described methods when executed on a computer or processor.

According to the present invention, there is an effect of generating and providing various types of images that enable a driver to safely drive by capturing images around and/or inside a vehicle using a plurality of wide-angle cameras.

In addition, according to the present invention, an image that automatically zooms and pans according to the driving direction of the vehicle, an image that automatically zooms and tilts according to the driving speed of the vehicle, and the driving direction of the vehicle An image that automatically zooms and pans accordingly, an image of a blind spot in an area that cannot be seen outside of the driver's field of view, a top view image that looks as if viewed from the top of a vehicle, and controls the vehicle from the outside. There is an effect of generating and providing a bird-eye image of the same third-person view point, or a tracking image that automatically tracks and shows so that the movement and speed of vehicles in adjacent lanes can be checked.

In addition, according to the present invention, it provides a multi-view image, such as that taken by installing several cameras indoors, and in addition to this, a driver's seat view that can check the driver's driving condition, an assistant seat view that can check the condition of the passenger seat, and indoors. There is an effect of creating and providing an all-round interior view that can check the overall situation, a front view that can also check the driving situation in front along with views showing the indoor situation.

In addition, according to the present invention, when generating a front view, a wide dynamic range technology is applied to match a severe difference in illuminance between indoors and outdoors, and an effect of providing a function of automatically setting an arbitrary correction window such as a glass window. There is.

In addition, according to the present invention, by controlling an image sensor appropriately according to the speed of the vehicle, the image resolution and frame rate are varied to provide a function to shoot and store, thereby increasing the frame rate during high-speed driving or vehicle collision. By providing an ultra-high resolution image by increasing the resolution while lowering the frame rate while driving at low speed, it is possible to help the driver to quickly recognize the situation and to facilitate post-processing of an accident with the stored image.

In addition, according to the present invention, when the door is opened after parking, an image of a convenient viewing angle is generated from the side and rear cameras so that the vehicle coming from the rear can be checked, and displayed on the mirror or the indoor display window, and the detection vehicle It has the effect of promoting safety when getting off by including a function that also displays information such as the location of the vehicle, whether lane change is possible, and parking lanes.

)의 일 예를 도시한 도면이다.
도 12는 본 발명의 일 실시 예에 따른 장면 좌표계상의 한 점(

)을 왜곡 없는 이상적인 영상 평면(

)의 대응하는 점으로 매핑하는 과정을 도시한다.
도 13은 본 발명의 일 실시 예에 따른 가상 카메라의 투영선과 2차 곡면 모델과의 교차점(

)이 왜곡이 해소된 이상적인 영상 평면상의 좌표(

)에 재투영되는 것을 도시한 도면이다.
도 14는 영상생성부(700)의 영상 생성 코어에 입력되는 입력 영상 및 이에 포함된 왜곡을 보정하여 영상 생성 코어에서 출력하는 영상의 일 예를 도시한 도면이다.
도 15는 본 발명의 일 실시 예에 따른 차량 진행 속도에 따라 자동으로 가상 카메라가 줌 및 틸트되도록 영상 생성 코어를 설정한 결과를 도시한 도면이다.
도 16은 본 발명의 일 실시 예에 따른 차량 진행 방향에 따라 자동으로 가상 카메라가 줌 및 팬이 되도록 영상 생성 코어를 설정한 결과를 도시한 도면이다.
도 17은 측면 광각 카메라(도 2의 C2 또는 C3)가 촬영한 영상(도 17 (a))으로부터 사각 지대 영상, 위에서 보는 듯한 영상, 인접 차선 영상을 생성한 결과를 도시한 도면이다.
도 18은 후방 광각 카메라(도 2의 C4)가 촬영한 영상(도 18 (a))으로부터 주행 시 필요한 광각의 후방 영상, 후진 주차 시 차량과 주차선을 함께 표시한 도로 영상, 정확한 주차선을 보여주기 위한 위에서 보는 듯한 top 뷰 영상 등을 생성한 결과를 도시한 도면이다.
도 19는 실내 광각 카메라(도 2의 C5)가 촬영한 영상(도 19 (a))으로부터 운전자 상황을 확인할 수 있는 운전석 뷰, 보조석 뷰, 전방위 실내 뷰, 전방 주행상황을 동시에 확인할 수 있는 전방 뷰 등을 생성한 결과를 도시한 도면이다.
도 20은 본 발명의 일 실시 예에 따른 주행감응형 차량용 영상 생성 방법을 도시한 도면이다.
도 21은 본 발명의 일 실시 예에 따른 차량용 영상 생성 장치에서 출력 영상을 생성하는 방법을 도시한 도면이다.
도 22는 본 발명의 일 실시 예에 따른 광역 보정 방법을 도시한 도면이다.
도 23은 본 발명의 일 실시 예에 따른 영상 생성 코어에서 영상을 생성하는 방법을 도시한 도면이다.1 is a diagram illustrating an apparatus for generating an image for a vehicle according to an exemplary embodiment of the present invention.
2 is a diagram illustrating a multi-camera unit 100 configured using a plurality of wide-angle cameras installed in a vehicle according to an embodiment of the present invention.
3 is a diagram illustrating an image output configuration of the screen unit 800 according to an embodiment of the present invention.
4 is a diagram illustrating a configuration of a virtual camera-based image generator 700 at an arbitrary point in time according to an embodiment of the present invention.
5 is a diagram showing a schematic configuration of a virtual camera model unit 710 according to an embodiment of the present invention.
6 shows a virtual camera coordinate system {n} compared with a world coordinate system.
7 is a diagram for explaining the center of an optical axis of a virtual camera.
8 is a diagram showing a schematic configuration of a secondary curved model unit according to an embodiment of the present invention.
9A and 9B are diagrams showing parameters for an ellipsoid model 324, a cylinder model 325, and a parabolic model 326 according to an embodiment of the present invention.
10 is a diagram showing a schematic configuration of a camera projection model unit 730 according to an embodiment of the present invention.
11 shows the intersection of the second-order curved model and the projection line of the virtual camera (

) Is a diagram showing an example of.
12 is a point on the scene coordinate system according to an embodiment of the present invention (

) To the ideal image plane without distortion (

) Shows a process of mapping to the corresponding point.
13 is an intersection point of a projection line of a virtual camera and a second-order curved surface model according to an embodiment of the present invention (

) Is the coordinates on the ideal image plane where the distortion is resolved (

) Is a view showing the reprojection.
14 is a diagram illustrating an example of an input image input to the image generation core of the image generation unit 700 and an image output from the image generation core by correcting distortion included therein.
15 is a diagram illustrating a result of setting an image generating core so that a virtual camera automatically zooms and tilts according to a vehicle traveling speed according to an embodiment of the present invention.
16 is a diagram illustrating a result of setting an image generating core so that a virtual camera automatically zooms and pans according to a vehicle traveling direction according to an embodiment of the present invention.
FIG. 17 is a diagram showing a result of generating a blind spot image, a top-view image, and an adjacent lane image from an image captured by a side wide-angle camera (C2 or C3 in FIG. 2) (FIG. 17(a)).
18 is a wide-angle rear image required for driving, a road image showing a vehicle and a parking line when parking backwards, and an accurate parking line from the image taken by the rear wide-angle camera (C4 in FIG. 2) (FIG. 18(a)). This is a drawing showing the result of creating a top view image that looks like you are viewing from above.
FIG. 19 is a driver's seat view, a driver's seat view, an all-around interior view, and a front view that can simultaneously check the driving situation from the image (FIG. 19 (a)) captured by the indoor wide-angle camera (C5 in FIG. 2). It is a diagram showing the result of generating the etc.
20 is a diagram illustrating a method of generating an image for a driving-sensitive vehicle according to an embodiment of the present invention.
21 is a diagram illustrating a method of generating an output image in an image generating apparatus for a vehicle according to an embodiment of the present invention.
22 is a diagram illustrating a wide area correction method according to an embodiment of the present invention.
23 is a diagram illustrating a method of generating an image in an image generating core according to an embodiment of the present invention.

In order to clearly describe the present invention, parts irrelevant to the description have been omitted, and the same reference numerals are attached to the same or similar components throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

When a part is referred to as being "on" another part, it may be directly on top of another part, or other parts may be involved in between. In contrast, when a part is referred to as being "directly above" another part, no other part is involved in between.

Terms such as first, second and third are used to describe various parts, components, regions, layers and/or sections, but are not limited thereto. These terms are only used to distinguish one part, component, region, layer or section from another part, component, region, layer or section. Accordingly, a first part, component, region, layer or section described below may be referred to as a second part, component, region, layer or section without departing from the scope of the present invention.

The terminology used herein is only for referring to specific embodiments and is not intended to limit the present invention. Singular forms as used herein also include plural forms unless the phrases clearly indicate the opposite. As used in the specification, the meaning of "comprising" specifies a specific characteristic, region, integer, step, action, element and/or component, and the presence of another characteristic, region, integer, step, action, element and/or component, or It does not exclude additions.

Terms indicating a relative space such as "below" and "above" may be used to more easily describe the relationship of one part to another part shown in the drawings. These terms are intended to include other meanings or operations of the device in use together with their intended meaning in the drawings. For example, if the device in the drawings is turned over, certain parts described as being "below" other parts are described as being "above" other parts. Thus, the exemplary term “down” includes both up and down directions. The device can be rotated by 90 degrees or other angles, and terms that refer to relative space are interpreted accordingly.

Although not defined differently, all terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms defined in a commonly used dictionary are additionally interpreted as having a meaning consistent with the related technical literature and the presently disclosed content, and are not interpreted in an ideal or very formal meaning unless defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein.

The apparatus and method for generating an image for a vehicle proposed in the present invention may capture images around and/or an interior of a vehicle using a plurality of wide-angle cameras, and generate and provide various types of images that enable a driver to safely drive.

1 is a diagram illustrating an apparatus for generating an image for a vehicle according to an exemplary embodiment of the present invention.

Referring to FIG. 1, in the apparatus for generating an image for a vehicle according to an embodiment of the present invention, a preprocessor 200 for preprocessing an image coming from the multi-camera unit 100 and the multi-camera unit 100 and transmitting the preprocessor to the post-processor 400. ), a post-processing unit 400 that acquires motion information or object tracking information from the current image coming from the pre-processing unit 200 or the past image read from the storage unit 300 and delivers it to the main processing unit 500, Input unit 600 for receiving requirements, setting information, etc., vehicle sensor unit 900 for acquiring driving state information such as current speed, acceleration, angular speed of the vehicle, and user requirements input through the input unit 600 , The main processing unit for setting parameters for the image generating unit 700 based on the state information of the vehicle input through the vehicle sensor unit 900, and/or motion information or object tracking information from the post-processing unit 400 ( 500), an image generation unit 700 for generating various images by generating a virtual camera model at an arbitrary point of view based on the image input from the post-processing unit 400 and the settings input from the main processing unit 500, and an image generation unit It may include a screen unit 800 that displays the image generated in 700. The image generated by the image generating unit 700 may also be stored in the storage unit 300. The pre-processing unit 200, the post-processing unit 400, and the main processing unit 500 may be implemented as a single process to constitute a processing unit.

Hereinafter, each part will be examined in more detail.

2 is a diagram illustrating a multi-camera unit 100 configured using a plurality of wide-angle cameras installed in a vehicle according to an embodiment of the present invention.

Referring to FIG. 2, according to an exemplary embodiment of the present invention, five wide-angle cameras from C1 to C5 may be provided in a vehicle in front, left, right, rear, and interior, respectively. The position of each wide-angle camera is marked with a cross mark ('+'), and the field of view they can see can be expressed in a sector shape or a circular gray area. The front wide-angle camera C1 may be located at the front end of the vehicle, and may photograph a situation ahead of the driver's field of view. During reverse parking in a parking lot, the driver's view may be obscured by cars located on the left and right, and accordingly, it is difficult for the driver to check the approaching vehicle from the side, thereby increasing the possibility of an accident. In this case, the image of the front wide-angle camera C1 located at the front end may help the driver to easily check the situation ahead. The left and right side wide-angle cameras C2 and C3 may be positioned near the conventional side mirror in order to obtain a wide range of side images with a view similar to that of the driver. The rear wide-angle camera (C4) can be located near the trunk handle or at the top of the trunk so that the rear situation can be checked first, and the indoor wide-angle camera (C5) includes all the interior of the vehicle, such as the driver's seat, assistant seat, rear row seat and front windshield. It can be placed on top of the front windshield or near the room mirror.

In the exemplary embodiment of FIG. 2, it has been described that five wide-angle cameras are provided in the vehicle, but one omni camera or two wide-angle cameras are provided on the roof of the vehicle to photograph the surroundings of the vehicle. The multi-camera unit 100 may be configured with two or three cameras for photographing indoors.

As shown in the exemplary embodiment of FIG. 2, the multi-camera unit 100 includes a plurality of wide-angle cameras, each wide-angle camera is equipped with a wide-angle lens, and a serial digital interface (SDI) or parallel. The image captured using a parallel digital interface (PDI) method can be transferred to the preprocessor 200, and control parameters including resolution and frame rate for controlling each wide-angle camera are given. It can be received from the processing unit 500.

3 is a diagram illustrating an image output configuration of the screen unit 800 according to an embodiment of the present invention.

The image generated by the image generating unit 700 may be displayed on the screen unit 800, and the screen of the screen unit 800 may be located in the left window and the right window of the vehicle. It can be located between the left window and the right window of the field of view. Alternatively, it may be located on a center fascia of a dash board, a room mirror, or a windshield of a vehicle, and may be located behind a front seat of a vehicle to provide an image to the occupant. The screen unit 800 may display images generated by the image generator 700 in various combinations including a picture in picture (PIP) function according to a user input.

Referring to FIG. 3, (a) of FIG. 3 shows a configuration of a screen unit 800 that selects and displays one of the five wide-angle camera images of FIG. 2 on the entire screen unit 800, and (b) Shows the configuration of the screen unit 800 that divides the screen unit 800 into four and displays images captured by the wide-angle cameras of C1 to C4 on each divided screen, and (c) and (d) are the screen units ( 800) is divided into four to an arbitrary size, and an image captured by a wide-angle camera of C1 to C4 is shown on each divided screen, and (e) shows the configuration of the screen unit 800 to an arbitrary size. It shows the configuration of the screen unit 800, which is divided into 5 and displays an image captured by the wide-angle cameras of C1 to C5 on each divided screen. In particular, in the configuration of each screen unit 800, it is preferable that the position of the split screen on which each wide-angle camera is displayed is matched to the position of the vehicle of each wide-angle camera. In addition, the screen unit 800 may be configured in various combinations according to user settings.

In addition, when the entire screen of the screen unit 800 is divided to form several screens, the size of each screen may be changed in real time according to the driving situation of the vehicle. As an example, when a vehicle intends to turn left, a screen outputting images captured by a left side camera and a front camera may be enlarged, and a screen outputting an image captured by a rear camera and a right side camera may become smaller. And if you go straight again, you can return to the original screen size. As another example, if another vehicle is passing your vehicle in the left passing lane while the vehicle is going straight, the screen that outputs the image captured by the rear camera is enlarged, and then the image taken by the left camera. It can be displayed while tracking other vehicles passing by while growing the screen that outputs.

Referring back to FIG. 1, the preprocessor 200 receives an image input from the multi-camera unit 100 and performs automatic color temperature correction (Auto White Balance; AWB), an automatic exposure (AE), Information necessary to control a wide-angle camera such as Auto Focus (AF) is extracted and transmitted to the main processing unit 500, and defective pixel value correction (Defect correction), noise reduction (NR) and wide area correction are performed. A pre-processed image obtained by performing the process of (Wide Dynamic Range; WDR) may be generated, and the pre-processed image may be transmitted to the post-processing unit 400. This pre-processing may be performed separately for each image captured by each camera of the multi-camera unit 100 or may be performed in combination.

Among them, the wide area correction function of the preprocessor 200 may be performed in order to generate a clear image by removing a difference in illuminance inside and outside the vehicle. In the case of a camera installed indoors, such as the C5 wide-angle camera of FIG. 2, both indoors and outdoors of the vehicle can be photographed. However, since the difference in illuminance between indoor and outdoor is large, wide area correction technology is absolutely required in order to capture both indoor and outdoor images using a single camera. However, the conventional wide area correction technique takes a low-exposure image and a high-exposure image by varying the shutter speed for one screen, and combines the corresponding pixel values of the low-exposure image and the high-exposure image for each pixel. The method of determining the pixel value is generally used. Here, the low-exposure image is an image in which bright portions are also expressed dark because the exposure of the camera is reduced, and the high-exposure image is an image in which dark portions are also expressed brightly because the exposure of the camera is increased. However, such a conventional wide area correction technique is not suitable for a vehicle camera image in which a high-illuminance image and a low-illuminance image are divided and included in one image. In the case of such a vehicle camera image, it may be more suitable to select only one of a high-exposure image or a low-exposure image for each pixel. That is, a pixel corresponding to a bright portion with high illuminance may have a corresponding pixel value of a low-exposure image, and a pixel corresponding to a dark portion with low illuminance may have a corresponding pixel value of a high exposure image.

In the wide-angle correction technology presented in the present invention, it is common that a wide-angle camera is fixed and attached to a vehicle, and therefore, a high-illumination image (image of a bright part; mainly outside the vehicle) and a low-light image ( The location of the dark part of the image (usually the interior of the vehicle) will always be constant. Based on this, in the initial setting mode, determine whether to use a high-exposure image or a low-exposure image for each pixel and set it as the exposure area information. By using an image shot with exposure or an image shot with low exposure, it is possible to create a clear image by removing the difference in illuminance inside and outside the vehicle.

Looking at the initial setting mode operation of the wide-angle correction technology proposed in the present invention, first, the shutter speed of the wide-angle camera is adjusted, and two wide-area correction images are applied, that is, a low-exposure image and a high-exposure image (I ₁ And I ₂ ) are inputted. In this case, the frame rate of the wide-angle camera may be 60 frames per second (fps). In general, an image is generated at 30 fps, and a wide-angle camera needs to generate an image at a speed of 60 fps because the wide-angle correction technique requires using two low-exposure images and high-exposure images for one screen. The wide-band image correction function of the preprocessor 200 _{sets a wide-area correction threshold based on the input wide-area correction images I 1} and I _2, and based on the wide-area correction threshold, the corresponding low-exposure image for each pixel in the image. It determines whether to use a pixel value or a corresponding pixel value of a high-exposure image, and generates exposure area information. The exposed area information generated in this way may be stored in the storage unit 300 and may be used for wide area correction during actual operation later.

Even in the general operation mode, the wide-angle camera captures a low-exposure image and a high-exposure image by adjusting the shutter speed, and transmits it to the preprocessor 200. The broadband image correction module of the preprocessor 200 obtains a pixel value of a low-exposure image for each pixel based on the exposure area information determined in the initial setting mode or a pixel value of a high-exposure image to generate a new image, and the post-processing unit You can pass it to 400. Through the above-described method, a difference in illuminance between indoor and outdoor can be eliminated by automatically determining an area corresponding to indoor or outdoor from an image and setting exposure area information.

The broadband image correction function of the preprocessor 200 may be applied to images of all wide-angle cameras provided in the vehicle, but the image of a wide-angle camera (C5 in the example of FIG. 2) capable of simultaneously photographing indoors and outdoors where a difference in illuminance is evident. Applicable only to Accordingly, the main processing unit 500 may set the frame rate to 60 fps for a wide-angle camera that will generate an image to which the wide-area correction technology is applied among images input from a plurality of wide-angle cameras of the multi-camera unit 100, and also, a low-exposure image. You can set the shutter speed for high-exposure images and shutter speed for high-exposure images. On the other hand, for a wide-angle camera to which the wide-angle correction technology is not applied, the frame rate can be set at 30 fps, and the shutter speed can also be set to one value.

The post-processing unit 400 may receive pre-processed images of images captured by the multi-camera unit 100 through the pre-processing unit 200 or may receive past images captured and stored in the past from the storage unit 300. . The post-processing unit 400 performs post-processing processes such as separation of brightness and color signals, edge enhancement, gamma correction, color correction, and contrast enhancement for pre-processed images. The performed post-processed image is generated, and the post-processed image is compressed using a video codec and transmitted to the storage unit 300 to be stored in a memory or the like in the storage unit. In addition, the post-processing unit 400 detects motion from a post-processed image or a past image received from the storage unit 300 using a motion detector, a lane detector, and/or an object tracker. Information, lane detection information, object tracking information, etc. may be acquired and transmitted to the main processing unit 500. In addition, the post-processing unit 400 may transmit a post-processed image or a past image received from the storage unit 300 to the image generating unit 700.

The main processing unit 500 is an image based on information from the post processing unit 400, user input information from the input unit 600, and/or vehicle state information such as speed, acceleration, and angular acceleration from the vehicle sensor unit 900. Setting information necessary for the generation unit 700 may be generated and transmitted to the image generation unit 700, and parameters for photographing a plurality of wide-angle cameras provided in the multi-camera unit 100 may be set.

As an embodiment, the main processing unit 500 may change the image resolution, shutter speed, and frame rate for each of a plurality of wide-angle cameras provided in the multi-camera unit 100 according to the vehicle speed. For example, the main processing unit 500 controls to increase the frame rate of the wide-angle camera when the vehicle is driving at high speed or when a vehicle crashes, so that the wide-angle camera can generate a super-high-speed image, and when driving at a low speed, the frame rate is adjusted. It can be controlled to lower and increase the resolution to generate an ultra-high resolution image of the surrounding situation. Also, as described above, the frame rate may be different depending on whether or not to apply the broadband correction technique.

The settings for the wide-angle cameras of the multi-camera unit 100 by the main processing unit 500 include user input at the input unit 600, motion detection information at the post processing unit 400, lane detection information, object tracking information, and vehicle. Includes vehicle driving information from the sensor unit 900 and/or automatic color temperature correction from the preprocessor 200 (Auto White Balance; AWB), Auto Exposure (AE), and Auto Focus (AF). It may be performed based on the image information and the like, and may be set to a reasonable resolution and frame rate according to each combined condition.

In addition, the main processing unit 500 may generate setting information necessary for the image generation unit 700 and control the image generation unit 700 to generate the required image. Continue after explanation.

The image generator 700 may generate an image displayed at a specific point in time according to a user's setting based on an image photographed in a wide range using a wide-angle camera. That is, the image generation unit 700 includes a plurality of image generation cores and an image synthesis unit, and each image generation core receives one of the images captured by a plurality of wide-angle cameras in the multi-camera unit 100 as an input image. Then, a virtual camera having a viewpoint according to a parameter set by the above-described processing unit is modeled, and an image viewed from the viewpoint of the virtual camera is generated from the input image and output. In addition, the image synthesizing unit may generate a final output image by synthesizing the output images of each image generating core according to a screen configuration set by the processing unit.

4 is a diagram illustrating a configuration of a virtual camera-based image generator 700 at an arbitrary point in time according to an embodiment of the present invention.

As shown in FIG. 4, the image generation unit 700 includes image generation cores 701, 702, and 703 capable of generating an image according to a setting input corresponding to each of a plurality of wide-angle cameras provided in the vehicle. , Images generated by each of the image generation cores 701, 702, and 703 may be synthesized into an output image in accordance with the configuration of the screen unit 800 by the image synthesis unit 740. In addition, the synthesized output image may be output to the screen unit 800 or may be stored in the storage unit 300.

Setting inputs required for each image generation core 701, 702, 703 of the image generation unit 700 are provided by the main processing unit 500, and each image input may be input through the post processing unit 400.

Hereinafter, the function of one image generating core will be described.

4, each image generating core 701, 702, 703 of the image generating unit 700 according to an embodiment of the present invention includes a virtual camera model unit 710, a second curved surface model unit 720, and And a camera projection model unit 730.

The virtual camera model unit 710 may perform modeling as if there is a virtual camera at a set arbitrary position in order to display an image generated by each image generating core as if viewed from an arbitrary viewpoint. In addition, the virtual camera model unit 710 can generate a digital PTZ (Pan, Tilt, Zoom) function and a special lens effect. To this end, a perspective projection camera having a general perspective effect and a perspective effect are removed. Virtual cameras can be modeled with an orthographic projection camera that can imitate a lens, or a panorama camera model that generates a panoramic image using a polar coordinate image model. In this case, one or more virtual cameras may be modeled to output a plurality of images for one input image. The second curved surface model unit 720 is an image displayed by a virtual camera modeled in the virtual camera model unit 710, which is an image generated by the image generation cores 701, 702, 703, and various images during mapping. A second-order curved surface that functions as a mirror that can be used to add effects is provided, and the camera projection model unit 730 removes distortion included in the input image and selects a point of the image displayed by the virtual camera as an input image. It can be mapped to a point of.

Before looking at each part in more detail, some coordinate systems used in the present invention will be described first. In the present invention, a virtual camera coordinate system {n} to indicate a point or pixel of an image viewed by a virtual camera, a second curved coordinate system {q} indicating coordinates with the center of the second curved surface as a zero point, and the actual surroundings are photographed. A camera coordinate system {c} is used to indicate the coordinates in each image taken by a wide-angle camera. In addition, the coordinates displayed by this coordinate system may be converted into coordinates on the scene coordinate system {0}, which is a reference coordinate system.

The virtual camera model unit 710 is a virtual camera similar to a PTZ (pan, tilt, zoom) camera having pan, tilt, and zoom functions using a mechanical driving unit and an electric zoom lens. The position and posture of the virtual camera can be modeled so that the (virtual camera) can create the same image as seen from an arbitrary point of view, and the point (pixel) of the image (J) seen from the virtual camera's point of view can be modeled accordingly. have. In this case, a virtual camera coordinate system may be used.

5 is a diagram showing a schematic configuration of a virtual camera model unit 710 according to an embodiment of the present invention.

5, a virtual camera model unit 710 according to an embodiment of the present invention includes a rectangular coordinate display module 311, a polar coordinate display module 312, a size adjustment module 313, and a position setting module 314. , A conversion module 315, and an optical axis center setting module 316.

에 의해서 구해지는

의 직각 좌표 형식으로 좌표(u)를 표시할 수 있다. 여기서

는 가상 카메라의 시점에서 보여지는 영상(J)의 중심을 직각 좌표 형식으로 나타낸 것이고, 설정 가능한 파라미터이다. 이처럼 영상 좌표를 직각 좌표 형식을 이용하여 표시하는 영상은 원근 투영 카메라 또는 직각 투영 카메라에서 사용될 수 있다.The rectangular coordinate display module 311 may display the coordinate u of a point of the image J viewed from the viewpoint of the virtual camera in a rectangular coordinate format. The rectangular coordinate display module 311 may represent the coordinate u as a relative coordinate with respect to the center of the image J viewed from the viewpoint of the virtual camera. Therefore, the rectangular coordinate display module 311

Obtained by

You can display the coordinates (u ) in a rectangular coordinate format of. here

Denotes the center of the image (J) viewed from the viewpoint of the virtual camera in a rectangular coordinate format, and is a parameter that can be set. As such, an image displaying image coordinates using a rectangular coordinate format may be used in a perspective projection camera or a rectangular projection camera.

)과 최대값(

), 반지름 방향 픽셀 수(

), 각도의 최소값(

) 과 최대값(

), 그리고 가상 카메라상에서의 극좌표 변환을 위한 극좌표 형식에서의 중심 (

)을 포함하며, 설정가능한 파라미터들이다. 그러면 좌표(u)를 나타내는 극좌표 (r, θ)는 다음과 같이 표현될 수 있다.The polar coordinate display module 312 may display the coordinate u of a point of the image J viewed from the viewpoint of the virtual camera in a polar coordinate format. The parameter used at this time is the minimum value of the radius to be used in the polar coordinate format (

) And the maximum value (

), the number of radial pixels (

), the minimum value of the angle (

) And the maximum value (

), and the center in polar coordinate format for polar coordinate conversion on the virtual camera (

) And are configurable parameters. Then, the polar coordinates (r, θ) representing the coordinates u can be expressed as follows.

이다. 그리고 극좌표 형식으로 표시되는 영상 좌표

는 다음과 같이 표현될 수 있다.here,

to be. And image coordinates displayed in polar coordinate format

Can be expressed as

의 극좌표 형식으로 좌표(u)를 표시할 수 있다. 이러한 극 좌표 형식의 표시는 파노라마 영상을 생성하는데 이용될 수 있다.That is, the polar coordinate display module 312 is

You can display the coordinates (u ) in the form of polar coordinates of. This polar coordinate format display can be used to create a panoramic image.

)에 의하여 어떤 표현 방식을 사용할 것인지를 결정할 수 있다. The virtual camera model can express the coordinates (u ) of a point in the image (J ) viewed from the viewpoint of the virtual camera in a rectangular coordinate format or a polar coordinate format, and the user's choice (

), you can decide which expression method to use.

)에 의해 직각 좌표 형식 또는 극좌표 형식으로 표현되는 가상 카메라 영상의 한 점의 좌표 (u')를 크기 보상 행렬과 초점거리를 이용하여 가상 카메라 좌표계 상의 좌표 (

)로 표현할 수 있다. 이때의 좌표(

)는 다음 식을 이용하여 구할 수 있다.The size adjustment module 313 is a process of compensating the size of a virtual camera image in a form convenient for a user to view, and may adjust a size ratio of the image to a horizontal axis and a vertical axis. This size ratio may be changed according to the above-described various screen configuration methods, and may be set differently for each image generating core. And accordingly, it is possible to adjust the coordinates u of one point of the image J viewed from the viewpoint of the virtual camera. That is, when the size adjustment module 313 has a feeling of disparity between the image generated by the virtual camera and the real space visible to the user's eyes, the user can feel dizziness while viewing the image, and the sense of distance disappears, resulting in a spatial sense. Falls. In order to minimize this sense of heterogeneity, a size adjustment module 313 may be used. That is, the resizing module 313 is selected by the user (

) By using the coordinates (u' ) of a point in the virtual camera image expressed in a rectangular coordinate format or a polar coordinate format by using the size compensation matrix and the focal length in the virtual camera coordinate system (

) Can be expressed. The coordinates at this time (

) Can be obtained using the following equation.

은 가로축과 세로축 크기 보상값들로 구성된 대각행렬(diagonal matrix)이고,

은 가상카메라의 초점 거리(focal length)다.here,

Is a diagonal matrix composed of size compensation values on the horizontal and vertical axes,

Is the focal length of the virtual camera.

The position setting module 314 sets the position and posture of the virtual camera.

The virtual camera model is a virtual camera similar to a PTZ (pan, tilt, zoom) camera with pan, tilt, and zoom functions using a mechanical driving unit and an electric zoom lens. It is possible to model the position and posture of the virtual camera so that it can generate the same image as seen from an arbitrary viewpoint. In this case, a virtual camera coordinate system may be used to set the position and posture of the virtual camera.

6 shows a virtual camera coordinate system {n} compared with a world coordinate system.

을 장면 좌표계 {0} 상에서의 가상 카메라의 위치,

를 장면 좌표계 {0} 상에서의 2차 곡면모델의 중심 위치이며 가상 카메라의 팬, 틸트 회전의 중심으로 두면, 장면 좌표계 {0} 상에서 2차 곡면모델로 매핑되는 가상 카메라 좌표계 {n}를 표현한 변환 행렬(

)은 다음과 같이 표현할 수 있다.The virtual camera coordinate system {n} can be rotated vertically and horizontally based on the center 10 of the scene coordinate system for pan and tilt functions, and the position of the viewpoint can be moved _{in the z n direction for the zoom function.} Referring to FIG. 5, φ _n and θ _n may be set as a horizontal rotation angle (azimuth) and a vertical rotation angle (elevation) with respect to the z-axis 11 of the scene coordinate system of the virtual camera, respectively,

The position of the virtual camera in the scene coordinate system {0},

Is the center position of the second-order surface model on the scene coordinate system {0} and is the center of the virtual camera's pan and tilt rotation, the transformation representing the virtual camera coordinate system {n} that is mapped to the second-order surface model on the scene coordinate system {0} procession(

) Can be expressed as follows.

)은

의 단위 벡터 형태로 표현될 수 있다. 여기서,

,

, 및

은 각각 가상 카메라 좌표계 {n} 상에서의 각 단위 벡터({1,0,0}, {0,1,0}, 및 {0,0,1})를 장면 좌표계 {0} 상의 좌표로 변환했을 때의 단위 벡터 표현이고,

은 가상 카메라 좌표계 {n} 상에서의 원점({0,0,0})을 장면 좌표계 {0} 상의 좌표로 변환했을 때의 단위 벡터 표현이다. 상술한 것과 같이 수평 회전각과 수직 회전각을 사용한 가상 좌표계의 표현은 상하가 뒤집히지 않고, 어떤 중심을 기준으로 하여도 회전이 가능하며, 줌 기능을 효과적으로 표현할 수 있으므로 기계적인 구동부를 사용한 PTZ 기능을 거의 흡사하게 동작시킬 수 있는 장점이 있다. 이러한 가상 좌표계의 표현 이외에도 사용자의 설정에 따라 임의 방식으로 가상 좌표계를 표현하는 것도 가능하다.Here, trans(·) denotes a 3-D translational matrix in 3D space, and rot _z (·) and rot _x (·) denote rotation matrices with the z-axis and the x-axis as rotation axes (3- D rotational matrices). And the transformation matrix (

)silver

It can be expressed in the form of a unit vector of. here,

,

, And

Transforms each unit vector ({1,0,0}, {0,1,0}, and {0,0,1}) on the virtual camera coordinate system {n} into coordinates on the scene coordinate system {0}. Is the unit vector representation of time,

Is a unit vector expression when the origin ({0,0,0}) on the virtual camera coordinate system {n} is converted into coordinates on the scene coordinate system {0}. As described above, the expression of the virtual coordinate system using the horizontal and vertical rotation angles is not upside down, can be rotated based on any center, and can effectively express the zoom function. It has the advantage of being able to operate almost similarly. In addition to the expression of the virtual coordinate system, it is also possible to express the virtual coordinate system in an arbitrary manner according to the user's settings.

), 장면 좌표계상에서의 가상 카메라의 위치(

), 및 가상 카메라의 장면 좌표계의 z축(11)에 대한 수평 회전각(φ_n) 및 수직 회전각(θ_n)을 파라미터로 입력받아 장면 좌표계 상에서 가상 좌표계를 표현한 변환 행렬(

)을 계산한다. 이 변환 행렬에 의하여 가상 카메라의 시점에서 보여지는 영상(J)의 한 점의 좌표(u)를 장면 좌표계의 한 좌표(

)로 변환할 수 있다.That is, the position setting module 314 is the center of the second-order curved surface model on the scene coordinate system (

), the position of the virtual camera in the scene coordinate system (

), and a transformation matrix representing the virtual coordinate system in the scene coordinate system by receiving _{the horizontal rotation angle (φ n} ) and the vertical rotation angle (θ _n ) with respect to the z-axis 11 of the scene coordinate system of the virtual camera as parameters (

) Is calculated. By this transformation matrix, the coordinates (u ) of one point of the image (J ) viewed from the viewpoint of the virtual camera are converted into one coordinate (

) Can be converted.

)을 이용하여 가상 카메라의 시점에서 보여지는 영상(J)의 한 점의 좌표(u)에 대응되는 장면 좌표계의 한 좌표(

)를 계산한다.The transformation module 315 is a transformation matrix calculated by the position setting module 314 (

) Using the coordinates of the scene coordinate system corresponding to the coordinates (u ) of a point in the image (J ) viewed from the viewpoint of the virtual camera (

) Is calculated.

The optical axis center setting module 316 is for setting the optical axis center of the virtual camera.

7 is a diagram for explaining the center of an optical axis of a virtual camera.

)은 다음과 같이 표현될 수 있다. 여기서

의 왼쪽 위에 있는 '0'은 좌표계를 나타내는 것으로 {0}은 장면 좌표계상의 좌표로 표시됨을 나타낸다.Referring to FIG. 7, the perspective projection camera and the orthogonal projection camera may vary depending on whether a projection ray converges to an optical center. In the perspective projection camera, as shown in Fig. 7(a), all the projection lines are gathered at one center point, and in the orthogonal projection camera, as shown in Fig. 7(b), all the projection lines are parallel to the optical axis. do. Therefore, the center of the optical axis (

) Can be expressed as here

'0' at the top left of '0' indicates the coordinate system, and {0} indicates that it is displayed as coordinates on the scene coordinate system.

은 초점 거리(focal length)이고,

는 상술한 바와 같이 가상 카메라의 시점에서 보여지는 영상(J)의 한 점의 좌표(u)에 대응되는 장면 좌표계의 좌표이고,

및

은 상술한 것처럼 변환 행렬(

)을 단위 벡터 형태인

으로 표현할 때의 단위 벡터들이다. here,

Is the focal length,

Is the coordinates of the scene coordinate system corresponding to the coordinates (u ) of one point of the image (J ) viewed from the viewpoint of the virtual camera as described above,

And

Is the transformation matrix (

) As the unit vector

These are unit vectors when expressed as.

) 및 가상 카메라의 광축(

) 중심에 대한 정보를 계산하여 출력할 수 있다.As described above, the virtual camera model receives parameters regarding the location of the virtual camera, whether to use a rectangular coordinate format or a polar coordinate format, and is assigned to the coordinates (u ) of a point in the image (J) viewed from the viewpoint of the virtual camera. The coordinates of the corresponding scene coordinate system (

) And the optical axis of the virtual camera (

) Information about the center can be calculated and printed.

Referring back to FIG. 4, the second curved model generated by the second curved model unit 720 is a corresponding point (pixel) of the input image including the distortion of one point of the image J viewed from the viewpoint of the virtual camera. ) Can perform the same function as a mirror mapping. In this case, a second-order surface model for modeling various second-order surfaces may be used according to a desired effect. As an example, the ellipsoid model is expressed in a spherical mirror shape and can be mainly used to obtain the PTZ effect of a virtual camera, and the cylinder model is expressed in a cylinder column shape mirror to create a wide-angle image. It can be used mainly in time, and the Paraboloid model is expressed in the shape of a mirror in the shape of a hemisphere, and can be used when creating an effect that maps only a certain part of the image.

8 is a diagram showing a schematic configuration of a secondary curved model unit according to an embodiment of the present invention.

Referring to FIG. 8, the second curved surface model unit 720 according to an embodiment of the present invention may include a second curved surface model selection module 321, a coordinate setting module 322, and a conversion module 323. .

The second curved model selection module 321 may allow a user to select one of a plurality of second curved models in order to obtain a desired effect. That is, as shown in FIG. 7, one of an ellipsoid model 324, a cylinder model 325, a parabolic model 326, and a user-set secondary curved surface model 327 may be selected using the _{parameter q sel.} .

The above-described second-order curved surface models in 3D space can be expressed in the form of a common matrix as follows.

는

로 모델링되는 2차 곡면상의 한점이고, 2차 곡면 모델(

)은 다음과 같이 4x4 대칭 행렬로 표현될 수 있다.here,

Is

It is a point on the quadratic surface modeled as, and the quadratic surface model (

) Can be expressed as a 4x4 symmetric matrix as follows.

) 행렬 내의 각 값은 2차 곡면별로 다를 수 있는데, 도 9a 및 도 9b는 본 발명의 일 실시 예에 따른 타원체 모델(324), 실린더 모델(325), 및 포물면 모델(326)에 대한 파라미터를 도시하고 있다. 여기서 도 9a 및 도 9b의 x_c, y_c, z_c는 장면 좌표계 {0} 상에서의 각 모델의 중심 위치를 나타낸다.At this time, the second-order surface model (

) Each value in the matrix may be different for each quadratic surface, and FIGS. 9A and 9B illustrate parameters for an ellipsoid model 324, a cylinder model 325, and a parabolic surface model 326 according to an embodiment of the present invention. Is shown. _{Here, x c} , y _c , and z _c of FIGS. 9A and 9B represent the center positions of each model on the scene coordinate system {0}.

)을 생성할 수 있고, 이때의 변환행렬(

)은 다음 식을 이용하여 구할 수 있다. The coordinate setting module 322 is used to define the secondary curved surface on the scene coordinate system {0}, and, like the position setting of the virtual camera, the horizontal rotation angle (φ _q ) and the vertical rotation angle (θ _q ), which are easy to set by the user, are set. It can be expressed using. That is, the coordinate setting module 322 is a transformation matrix for converting points on a secondary curved surface into points on a corresponding scene coordinate system (

) Can be created, and the transformation matrix (

) Can be obtained using the following equation.

)을 장면 좌표계 상에서의 선택된 2차 곡면 모델을 표현하는 행렬(

)로 변환할 수 있다. 장면 좌표계에서의 2차 곡면 모델을 표현하는 행렬 (

)는 다음 식을 이용하여 구할 수 있다.The transformation module 323 is a matrix representing a quadratic surface generated by an ellipsoid model 324, a cylinder model 325, a parabolic surface model 326, or a user-set secondary surface model 327 (

) As a matrix representing the selected quadratic surface model in the scene coordinate system (

) Can be converted. Matrix representing the quadratic surface model in the scene coordinate system (

) Can be obtained using the following equation.

The camera projection model unit 730 reprojects an input image including image distortion after mapping to a secondary curved model in a virtual camera, and may include a camera calibration and a camera projection model for correcting image distortion. .

10 is a diagram showing a schematic configuration of a camera projection model unit 730 according to an embodiment of the present invention.

10, the camera projection model unit 730 according to an embodiment of the present invention includes an intersection module 331, a transformation module 332, a coordinate calculation module 333, an interpolation module 334, and a memory ( 335).

The intersection module 331 may calculate an intersection of the second curved model selected by the second curved model unit 720 and the projection line of the virtual camera modeled by the virtual camera model unit 710.

)의 일 예를 도시한 도면이다.11 shows the intersection of the second-order curved model and the projection line of the virtual camera (

) Is a diagram showing an example of.

)과 가상 카메라의 시점에서 보여지는 영상(J)의 한 점(

)을 잇는 직선인 투영선(910)이 2차 곡면 모델과 만나기 위하여는 다음 식을 만족하여야 한다.11, the center of the optical axis of the virtual camera (

) And a point of the image ( J ) viewed from the viewpoint of the virtual camera (

In order for the projection line 910, which is a straight line connecting ), to meet the second-order curved surface model, the following equation must be satisfied.

은 가상 카메라의 광축 중심이고,

으로 표현되는데

는 가상 카메라의 시점에서 보여지는 영상(J)의 한 점이다. 그리고 점

는 투영선(910) 상에 존재하는 점이면서, 2차 곡면 모델 상에 존재하는 교차점이 된다. 이 교차점의 위치는

에 의해서 결정되는데,

는 다음과 같이 계산될 수 있다.here,

Is the center of the optical axis of the virtual camera,

It is expressed as

Is a point in the image (J ) viewed from the viewpoint of the virtual camera. And point

Is a point that exists on the projection line 910 and an intersection point that exists on the quadratic curved surface model. The location of this intersection is

Is determined by

Can be calculated as follows.

이고,

이다.here,

ego,

to be.

)의 투영선(910)과 교차하는 2차 곡면상의 한 점(

)을 계산하여 출력할 수 있다.That is, the intersection module 331 is based on the above-described equation, based on the point of the image ( J ) viewed from the viewpoint of the virtual camera (

A point on the quadratic surface that intersects the projection line 910 of (

) Can be calculated and printed.

)은 전방향 카메라 모델(336)에 의하여 입력 영상의 한 점으로 투영될 수 있다.And one point (

) May be projected to a point of the input image by the omnidirectional camera model 336.

)을 왜곡 없는 이상적인 영상 평면(

)의 대응하는 점으로 매핑하는 과정을 도시한다.12 is a point on the scene coordinate system according to an embodiment of the present invention (

) To the ideal image plane without distortion (

) Shows a process of mapping to the corresponding point.

에서 광각 카메라의 광축 중심으로의 투영선(1010) 상에 벡터

가 존재하고, 이는 일반적인 원근 투영 기하를 따르고 다음 식의 관계가 성립된다.12, a point on the scene coordinate system

On the projection line (1010) to the center of the optical axis of the wide-angle camera at

Is present, which follows the general perspective projection geometry, and the relationship of the following equation is established.

는 회전 행렬

과 이동 벡터

로 구성된 카메라 투영 행렬(projection matrix)이며,

는 장면 좌표계상의 한 점

의 동차 표현(homogenous representation)이다. here

Is the rotation matrix

And move vector

Is a camera projection matrix consisting of,

Is a point in the scene coordinate system

It is a homogenous representation of.

의

는 장면 좌표계 상의 한 점

를 이상적인 영상 평면(

) 상의 한 점(

)으로 매핑하는 왜곡보정 매핑함수이며, 광각 카메라의 왜곡 현상을 이론적으로 분석한 모델(analytic model) 또는 수치적으로 근사한 모델(approximation model) 등을 사용하여 표현할 수 있다. 이에 의하여 이상적인 영상 평면(

)은 왜곡이 없는 영상 평면을 의미할 수 있다. 일 실시 예로서 Scaramuzza가 제안한 것과 같은 테일러 확장(Taylor expansion series) 기반의 근사 모델을 사용하여 표현할 수 있다.vector

of

Is a point in the scene coordinate system

Is the ideal image plane (

) On one point (

), and can be expressed using a model that theoretically analyzes the distortion phenomenon of a wide-angle camera or a numerical approximation model. Thereby, the ideal image plane (

) May mean an image plane without distortion. As an embodiment, it may be expressed using an approximate model based on a Taylor expansion series as suggested by Scaramuzza.

을 입력 영상(I)에서의 실제 픽셀 좌표라고 가정하면, 입력 영상(I)과 이상적인 영상 평면(

)의 관계는 어파인(affine) 변환 행렬로 다음과 같이 표현될 수 있다.And vector

Assuming that the actual coordinates of the pixel in the input image (I), the input image (I) and the ideal image plane (

The relationship of) can be expressed as an affine transformation matrix as follows.

는 어파인 변환 행렬이고,

는 이동 벡터를 나타낸다. From here

Is the affine transformation matrix,

Denotes a motion vector.

는 입력 영상(I)에서 픽셀 좌표

으로 매핑되므로, 만약 이식들이 역변환 관계가 존재한다면, 입력 영상(I)에서 원래 장면 좌표계 상의 한 점

의 픽셀 값을 구할 수 있다. One point on the scene coordinate system as described above

Is the pixel coordinates in the input image ( I)

Is mapped to, so if there is an inverse transformation relationship between the implants, a point on the original scene coordinate system in the input image (I)

You can get the pixel value of.

를 카메라 좌표계 {c} 상의 한 점

로 왜곡 보정을 하면서 매핑시키기 위하여서는 상술한 바처럼 광각 카메라의 위치와 자세 및 왜곡보정 매핑함수(

)의 계수들을 미리 알아야만 한다. A point on the scene coordinate system {0}

A point on the camera coordinate system {c}

In order to map while performing distortion correction, as described above, the position and posture of the wide-angle camera and the distortion correction mapping function (

You must know the coefficients of) in advance.

과 이동 벡터

왜곡보정 매핑함수(

)의 계수, 및 어파인 변환 행렬(

)를 획득할 수 있다.To this end, the position and posture of the image sensor, and distortion included in the captured image may be recognized using a calibration chart in which the horizontal and vertical grid sizes are known in advance. As an example, the horizontal and vertical size of the correction chart can be set to 20cm x 20cm, and it is included in the photographed image by measuring the slope of each grid and the size of the grid on the outer side of the correction chart image taken by a wide-angle camera. Distortion can be measured. And based on the measured distortion, the above-described rotation matrix

And move vector

Distortion correction mapping function (

) Of the coefficients, and the affine transformation matrix (

) Can be obtained.

)이 왜곡이 해소된 이상적인 영상 평면상의 좌표(

)에 재투영되는 것을 도시한 도면이다.13 is an intersection point of a projection line of a virtual camera and a second-order curved surface model according to an embodiment of the present invention (

) Is the coordinates on the ideal image plane where the distortion is resolved (

) Is a view showing the reprojection.

)을 연결하는 선이 된다.Referring to FIG. 13, the secondary curved surface may serve as a mirror reflecting the projection line 1110 of the virtual camera to the projection line 1120 of the wide-angle camera. Here, the projection line 1120 of the wide-angle camera is the intersection with the center of the optical axis of the wide-angle camera (

).

)가 되고,The point where the projection line (1110) of the virtual camera and the quadratic surface meet is the intersection (

) Becomes,

)은 다음 식을 이용하여 카메라 좌표계의 좌표(

)로 변환될 수 있다. crossing(

) Is the coordinates of the camera coordinate system (

) Can be converted.

는 장면 좌표계상에서 표현된 광각 카메라의 위치와 자세를 나타내는 변환 행렬로써 카메라 투영행렬(

)의

과

로 구성될 수 있다. here

Is a transformation matrix representing the position and posture of the wide-angle camera expressed in the scene coordinate system.

)of

and

It can be composed of.

)은 도 12에 도시된 한 점(

)과 마찬가지로 광각 카메라 광축 중심으로의 투영선(1120) 상에 존재할 수 있고, 이 투영선(1120)은 광각 카메라의 광축을 중심으로 방사형으로 대칭이므로 이상적인 영상 평면(

)과 교차점(

)과의 각도로 표현할 수 있다. 즉,

의 관계식을 얻을 수 있는데 여기서,

는 이상적인 영상 평면(

)과 교차점(

)와의 각도로서,

로 표현된다.The intersection of the projection line (1110) of the virtual camera and the quadratic surface (

) Is a point shown in FIG. 12 (

) Can exist on the projection line 1120 about the optical axis of the wide-angle camera, and this projection line 1120 is radially symmetrical about the optical axis of the wide-angle camera, so the ideal image plane (

) And intersection (

It can be expressed as an angle with ). In other words,

You can get the relational expression of where,

Is the ideal image plane (

) And intersection (

) As an angle,

It is expressed as

)는 상술한 바처럼 모두

의 함수이므로, 이들의 해를 구하면 이상적인 영상 평면상의 좌표(

)를 구할 수 있고, 어파인 변환 행렬을 이용하여 입력 영상(I)에서의 픽셀값(

)을 구할 수 있다. 따라서, 가상 카메라의 시점에서 보여지는 영상(J) 내의 한 점(

)의 픽셀값은 광각 카메라에서 촬영한 왜곡이 포함된 원본 영상(I)으로부터 얻어지고, 가상 카메라의 시점에서 보여지는 영상(J) 내의 각 점에 대하여 상술한 기능을 수행하면 각 영상 생성 코어의 최종 결과 영상인 가상 카메라의 시점에서 보여지는 영상(J)을 획득할 수 있고, 이 영상은 임의 시점에서 생성한 렌즈 왜곡이 제거된 영상이 된다. And the projection line 1120 to the center of the optical axis of the wide-angle camera and the distortion correction mapping function (

) Are all

Since it is a function of, solving for these coordinates on the ideal image plane (

) A can be obtained, the pixel values in using the affine transformation matrix input image (I) (

) Can be obtained. Therefore, a point in the image ( J ) viewed from the viewpoint of the virtual camera (

The pixel value of) is obtained from the original image (I ) including the distortion captured by the wide-angle camera, and if the above-described function is performed for each point in the image (J ) viewed from the viewpoint of the virtual camera, each image generating core An image J viewed from the viewpoint of the virtual camera, which is the final result image, can be obtained, and this image becomes an image from which lens distortion generated at an arbitrary viewpoint has been removed.

)를 이용하여 교차점(

)을 카메라 좌표계의 좌표(

)로 변환한다. Referring back to FIG. 10, in order to perform the above-described operation, the transformation module 332 of the camera projection model unit 730 is a transformation matrix representing the position and posture of the wide-angle camera expressed in the scene coordinate system (

) Using the intersection (

) To the coordinates of the camera coordinate system (

).

), 이동 벡터(

), 장면 좌표계상의 한 점

를 이상적인 영상 평면(

) 상의 한 점(

)으로 매핑하는 역할을 수행하는 테일러 확장 기반의 근사 모델의 계수들을 파라미터로 입력받아 가상 카메라의 투영선과 2차 곡면 모델과의 교차점(

)으로부터 왜곡이 포함된 입력 영상(I)에서의 실제 픽셀 좌표를 나타내는 벡터(

)를 획득한다. 좌표 계산 모듈(333)에서 사용하는 상술한 테일러 확장 기반의 근사 모델은 광각 렌즈의 왜곡 현상을 이론적으로 분석하고, 이를 해소하기 위하여 수치적으로 근사화한 것으로 이에 의하여 원본 영상에 존재하는 왜곡 현상을 제거할 수 있다. 도 14는 입력 영상 (a)로부터 왜곡이 보정된 영상 (b), (c)를 생성한 일 예를 보여준다.And the coordinate calculation module 333 is an affine transformation matrix (

), motion vector (

), a point in the scene coordinate system

Is the ideal image plane (

) On one point (

), the coefficients of the Taylor extension-based approximation model, which play a role of mapping, are input as parameters, and the intersection of the projection line of the virtual camera and the second-order curved surface model (

) From the vector representing the actual pixel coordinates in the input image (I) containing distortion (

). The Taylor extension-based approximation model used in the coordinate calculation module 333 theoretically analyzes the distortion phenomenon of the wide-angle lens, and is numerically approximated to solve the distortion phenomenon, thereby removing the distortion phenomenon existing in the original image. can do. 14 shows an example of generating images (b) and (c) whose distortion is corrected from an input image (a).

를 이용하여 입력 영상(I)에서 해당 좌표의 주변 픽셀값을 읽어 들이고 양선형 보간 (bilinear interpolation) 등의 방법 등으로 해당 좌표의 값을 계산 하여 출력할 수 있다.And the interpolation module 334 is a vector representing the actual pixel coordinates

By using, the value of the surrounding pixel of the corresponding coordinate is read from the input image (I ), and the value of the corresponding coordinate can be calculated and output using a method such as bilinear interpolation.

By using the image generating unit 700 described above, the present invention can generate an image that is viewed at one or more arbitrary viewpoints, and corrects the image so that there is no distortion for each image, thereby creating an image that is comfortable for the user to see. have.

14 is a diagram illustrating an example of an input image input to the image generation core of the image generation unit 700 and an image output from the image generation core by correcting distortion included therein.

Referring to FIG. 14, (a) shows a wide-angle camera image including distortion obtained using a fisheye lens, and (b) shows an image whose distortion is corrected by a general method. In the distortion-corrected image as shown in (b), the difference between the actual size and the sense of distance occurs as it goes toward the end of the image. The reason is that there can be numerous points in space on the projection line that comes as a point of the distorted image of the camera, and for structure reconstruction, two or more projection lines must exist so that the actual position of this point in space can be known. Because it becomes. However, if the method proposed in the present application is used and necessary parameters are appropriately set, an image that is comfortable for the user to see can be generated as shown in (c).

), 가상 카메라의 시점에서 보여지는 영상(J)의 직각 좌표 형식의 중심(

), 극좌표 형식에서 사용할 반지름의 최소값(

)과 최대값(

), 반지름 방향 픽셀 수(

), 각도의 최소값(

) 과 최대값(

), 그리고 가상 카메라상에서의 극좌표 변환을 위한 극좌표 형식에서의 중심 (

), 크기 보상을 위한 파라미터

, 가상카메라의 초점 거리(focal length)

, 가상 카메라의 장면 좌표계의 z축(11)에 대한 수평 회전각(azimuth) φ_n 및 수직 회전각(elevation) θ_n, 장면 좌표계 {0} 상에서의 가상 카메라의 위치

, 장면 좌표계 {0} 상에서의 2차 곡면모델의 중심 위치이며 가상 카메라의 팬, 틸트 회전의 중심

, 광축 중심 선택 파라미터 (V_psel)등을 설정할 필요가 있다. As described above, each image generation core 701, 702, 703 of the image generation unit 700 may generate an image viewed at an arbitrary viewpoint while removing distortion if necessary parameters are appropriately set. At this time, if the parameters required by the image generation cores 701, 702, 703 of the image generation unit 700 are summarized, parameters required for each virtual camera model of the virtual camera model unit 710 are displayed from the viewpoint of the virtual camera. A parameter to set whether to display each point of the losing image ( J) in rectangular coordinate format or polar coordinate format (

), the center of the rectangular coordinate format of the image (J ) displayed from the viewpoint of the virtual camera (

), the minimum value of the radius to be used in polar coordinate format (

) And the maximum value (

), the number of radial pixels (

), the minimum value of the angle (

) And the maximum value (

), and the center in polar coordinate format for polar coordinate conversion on the virtual camera (

), parameters for size compensation

, The focal length of the virtual camera

_{, The horizontal rotation angle (azimuth) φ n} and the vertical rotation angle (elevation) θ _n about the z-axis (11) of the scene coordinate system of the virtual camera, and the position of the virtual camera on the scene coordinate system {0}

, The center position of the second-order surface model on the scene coordinate system {0}, and the center of the pan and tilt rotation of the virtual camera.

, It is necessary to set the optical axis center selection parameter (V _{psel ).}

_{In addition, the parameter q sel} indicating which secondary surface model is to be used as a parameter for the secondary surface model unit, a plurality of parameters for setting the characteristics of each secondary surface model, and the secondary surface are defined in the scene coordinate system {0}. There may be a horizontal rotation angle (φ _q ) and a vertical rotation angle (θ _{q ), which are parameters for doing so.}

, 장면 좌표계상에서 표현된 광각 카메라의 위치와 자세를 나타내는 변환 행렬

등이 있을 수 있다.In addition, as parameters for the camera projection model unit 730, each coefficient of the distortion correction mapping function for correcting distortion included in the input image, and an affine transformation matrix

, Transformation matrix representing the position and posture of the wide-angle camera expressed in the scene coordinate system

There may be a back.

Parameters necessary for each image generation core of the image generation unit 700 may be set by the main processing unit 500. That is, the main processing unit 500 includes vehicle driving information including the vehicle traveling speed and direction provided by the vehicle sensor unit 900, motion information provided by the post processing unit 400, object tracking information, and lane information, and an input unit. Based on the user's setting information input through 600, parameters necessary for each image generation core of the image generation unit 700 may be set. These parameters may be changed every image frame so that an image desired by a user or an image that enables the safety of the vehicle and the driver to be secured may be output on the screen unit 800.

As an embodiment, the main processing unit 500 may set the driving speed of the vehicle by dividing it into fast, normal, and slow according to the driving speed of the vehicle, and accordingly, the image generating core automatically zooms, pans, and tilts the virtual camera. May be set, and the image generation core may be set so that the virtual camera automatically zooms and pans according to the vehicle traveling direction, that is, left turn, right turn, and reverse direction.

15 is a diagram illustrating a result of setting an image generating core so that a virtual camera automatically zooms and tilts according to a vehicle traveling speed according to an embodiment of the present invention.

Referring to FIG. 15, the relative relationship between the images displayed on the screen 800 according to the speed of the vehicle is shown in the upper part of the drawing, the input images 1512, 1522 and 1532 in the middle, and the screen part ( Output images 1514, 1524, and 1534 displayed on 800 are shown. As shown at the top of the drawing, according to the speed of the vehicle, the zoom and tilt parameters of the virtual camera are set to show a wider area at low speed (1511), and at high speed, the zoom and tilt parameters of the virtual camera are set to enlarge and show the far area (1531). You can set the camera's zoom and tilt parameters. Accordingly, in the enlarged output image 1534 during high-speed driving, the driver can identify a vehicle far from the center through the image. The grids 1513, 1523, 1533 displayed in the center of the input images 1512, 1522, 1532 represent the regions mapped to the input image to generate the output images 1514, 1524, 1534, and the mapping shape of these grid regions. According to this, various types of images may be generated.

16 is a diagram illustrating a result of setting an image generating core so that a virtual camera automatically zooms and pans according to a vehicle traveling direction according to an embodiment of the present invention.

Referring to FIG. 16, the relative relationship between the images displayed on the screen 800 according to the traveling direction of the vehicle is shown at the top of the drawing, and the input images 1612, 1622, 1632 are shown in the middle, and the screen unit is shown at the bottom. Output images 1614, 1624, and 1634 displayed on 800 are shown.

At the top of the drawing, according to the vehicle heading direction information acquired from the vehicle or the degree of rotation of the steering wheel, if the right turn degree is low, the zoom and pan parameters of the virtual camera are set to show the center part 1611, and if the right turn degree is severe At, zoom and pan parameters of the virtual camera may be set to show the right part 1631. Accordingly, when the driving direction is changed, portions that have not yet entered the driver's field of view are displayed through the output images 1614, 1624, and 1634, so that the driver can grasp the road situation in advance through the image. The grids 1613, 1623, 1633 displayed in the center of the input images 1612, 1622, 1632 represent the regions mapped to the input image to generate the output images 1614, 1624, 1634, and the mapping shape of these grid regions. According to this, various types of images can be generated.

As described above, in addition to being able to set the image generation core so that the virtual camera automatically zooms, pans, and/or tilts according to the vehicle traveling speed or the vehicle traveling direction, the main processing unit 500 performs another task. For example, an image of a blind spot that is not visible outside of the driver's field of view, a top view image such as viewed from above, a bird-eye image from a third-person view such as controlling the vehicle from the outside, and an object tracker in adjacent lanes. An image generation core may be set so that a tracking image, etc., which is automatically tracked and displayed, is output on the screen unit 800. In addition, a wide-angle rear image that provides a wide field of view when driving, a road image that simultaneously displays the rear vehicle and the parking line when parking in reverse, and a top view image showing the parking line from above to accurately check the parking situation are displayed on the screen 800. You can set the image generation core to be output to. In addition, a multi-view image, such as the one shot by installing multiple cameras indoors, a driver's seat view that can check the driver's driving status, an assistant seat view that can check the situation of the passenger seat, an all-around indoor view that can check the indoor situation as a whole, An image generation core can be set to generate a front view image that can also check the driving situation in the front along with views showing the indoor situation. In particular, when generating the front view, the above-described wide-area correction technique can be applied to match the severe difference in illumination between indoor and outdoor. For this purpose, the control signal of the wide-angle camera of the multi-camera unit 100 can be set together with setting the image generation core. have. Alternatively, the virtual camera may be appropriately controlled so that an image is output on the screen unit 800 according to a user setting input.

17 to 19 are diagrams illustrating input images captured by a wide-angle camera and output images output to the screen unit 800 according to an embodiment of the present invention.

Referring to FIGS. 17 to 19, an output image output to the screen unit 800 may show an image that is viewed simultaneously from multiple viewpoints with respect to an input image captured by a single wide-angle camera. That is, parameters of the image generator 700 may be set to have the same effect as the presence of cameras at various positions and angles. These images are obtained by providing the same input image (for example, an image captured by the C1 wide-angle camera in Fig. 2) to a plurality of image generating cores and setting different viewpoints of the virtual cameras modeled by the virtual camera modeling unit of each image generating core. Can be. Alternatively, it may be obtained by placing a plurality of virtual cameras modeled by the virtual camera modeling unit of one image generating core and setting different viewpoints of each virtual camera.

FIG. 17 is a diagram showing a result of generating a blind spot image, a top-view image, and an adjacent lane image from an image captured by a side wide-angle camera (C2 or C3 in FIG. 2) (FIG. 17(a)). The grids of FIG. 17A represent regions mapped to an input image to generate an output image, and various types of images may be generated according to the mapping shape of the grid regions.

18 is a wide-angle rear image required for driving, a road image showing a vehicle and a parking line when parking backwards, and an accurate parking line from the image taken by the rear wide-angle camera (C4 in FIG. 2) (FIG. 18(a)). This is a drawing showing the result of creating a top view image that looks like you are viewing from above.

FIG. 19 is a driver's seat view, a driver's seat view, an all-around interior view, and a front view that can simultaneously check the driving situation from the image (FIG. 19 (a)) captured by the indoor wide-angle camera (C5 in FIG. 2). It is a diagram showing the result of generating the etc.

The user may select an appropriate camera and view according to the driving situation using the input unit 600 and set a desired image to be displayed on the screen unit 800. Here, the camera may include front, rear, left and right side, and indoor cameras, and the views of the front camera include a front wide-angle view, a top view, and a bird's eye view, and in the case of a side camera, a blind spot view, an adjacent lane view, and There may be a top view, and in the case of a rear camera, there may be a rear wide-angle view, a top view, and a road image view, and in the case of an indoor camera, there may be a driver's seat view, an assistant seat view, a front view, and an all-around indoor view. In particular, in the case of the side camera, the same view as the conventional side mirror can be set to be output as an image. For each view, zoom, pan, tilt, and viewpoint conversion may be automatically adjusted according to a driving situation or by the main processing unit 500 according to a user setting. In the case of outputting the same view as the conventional side mirror described above, the user may change the setting of the output image by using the conventional side mirror adjustment switch as it is.

In addition, information recognized from an image, such as a lane, a parking line, and a vehicle, may be additionally synthesized and displayed in an augmented reality method on the screen unit 800.

Possible positions of the output device of the screen unit 800 include the left and right windows of the vehicle, in particular, the left and right windows between the conventional side mirror and the driver's view, and the center fascia of the dash board. ), a room mirror, the windshield of the vehicle, or the rear of the front seat of the vehicle.

In addition, the main processing unit 500 may set the image generation unit 700 to generate an image according to a preset scenario based on the driving state of the vehicle.

As an example of possible scenarios, when driving, the screen 800 is configured to be configured as shown in Fig. 3(d), and a front wide-angle view of the front camera, a blind spot view of the left and right side cameras, or an adjacent lane view or tracking view. , And a wide-angle view of the rear camera may be output to the screen unit 800 to help the driver secure a view of a wide range possible. In this case, each image generating core of the image generating unit 700 takes a photographed image of the front camera, a photographed image of the left and right side cameras, and a photographed image of the rear camera as input images, and according to the setting of the main processing unit 500, the front wide angle A view, a left and right blind spot view or an adjacent lane view or a tracking view, and a rear wide-angle view are generated, and the image synthesizing unit 740 synthesizes each view generated by the image generation core to come to a set position of the screen unit 800. The final output image may be generated and output to the screen unit 800.

As another example of possible scenarios, when T-shape forward parking is performed, an image captured by a side wide-angle camera is mainly used to select an adjacent lane view, a bird's eye view, a top view, etc., and output them to the screen unit 800. In the case of T-shape reverse parking, the image captured by the rear camera is mainly used, and the rear wide-angle view, the top view, the road image view, etc., as shown in FIG. 17 are selected to be output to the screen unit 800. In the case of parallel parking, images taken by the side and rear wide-angle cameras are properly mixed to provide an image, thereby promoting smooth parking for the user.

As another embodiment of possible scenarios, a camera may be appropriately selected according to a door that opens after stopping and set to provide an image, so that a person getting off may easily check whether a vehicle is coming from the rear. By selecting the left camera for the left door open and the right camera for the right door open, you can create an image with a convenient viewing angle for checking the approaching vehicle.At the same time, the rear camera also creates an image with a convenient viewing angle to check the vehicle behind you. It may be output on the screen unit 800. In this case, information such as the location and distance of the detected vehicle may be displayed together with the image output, and a function of giving an alarm to the occupant using sound may be added.

Images captured by each camera of the driving-sensitive vehicle image generating apparatus proposed in the present invention and images generated by the image generating unit may be directly output to the screen unit 800, but may be simultaneously stored in a storage device. In particular, images captured by each wide-angle camera of the multi-camera unit 100 may be stored in a storage device for re-viewing purposes such as preservation and post-accident processing, and in this case, a plurality of original images are synchronized to one file. Can be saved. In order to synchronize and store the images taken by each wide-angle camera, the same index (for example, 33.3ms if 30fps is the default frame rate) to the images taken by each wide-angle camera based on a timer built into the device. index), and an image with the same index is determined as an image captured at the same time, so that images captured by each wide-angle camera can be synchronized.

At this time, when the original image is stored in a general recording device, the size of the storage space of the storage device may be a problem because the data size is large, so the image data compressed through the video codec can be stored in consideration of the performance and storage space of the storage device. . Various voice information, vehicle status information, acceleration information, and/or GPS information may be stored in the stored file.

According to a setting using the input unit 600 of the user, an image previously generated and stored or an original stored image may be reproduced on the screen unit 800. In addition, the image generated by the image generator 700 may be output to the screen unit 800 with a new setting based on the previously stored original image. A newly created image or a previously stored image may be transmitted to another display device, and an image at a random viewpoint may be displayed in real time by receiving a user input in real time and changing the position and posture of the virtual camera. Examples of file storage formats include AVI, MP4, and the like, Micro SD card, etc. as a storage device, and H.264/MPEG-4 AVC or H.265 can be used as a video codec that can be used. .

As described above, the input unit 600 may receive requirements, setting information, and the like from a user. That is, screen configuration information to be displayed on the screen unit 800, information on installation locations of a plurality of wide-angle cameras of the multi-camera unit 100, scenario information, and information on whether to use a stored image or a currently captured image. The user may input such as through the input unit 600.

20 is a diagram illustrating a method of generating an image for a driving-sensitive vehicle according to an embodiment of the present invention.

Referring to FIG. 20, in a method for generating an image for a driving-sensitive vehicle according to an embodiment of the present invention, a plurality of images including a vehicle surrounding and/or a vehicle interior are captured using a plurality of wide-angle cameras attached to the vehicle (S100). )can do. At this time, since it is photographed by a wide-angle camera such as a fisheye lens, the captured image contains considerable distortion.

In addition, an output screen may be configured and an image to be output may be set manually or automatically according to object tracking information, motion detection information, or vehicle driving information according to a user's input (S200). Alternatively, the device may automatically configure the required output screen and set the image to be output according to the user's input.

In the configuration of the output screen, as shown in the example of FIG. 3, it may be configured to output one image on the entire screen, or by dividing the entire screen into two, four, or five, different images are displayed on each divided screen. Can be configured to output. Alternatively, it may be configured to output an image on the entire screen in the form of a picture in picture (PIP) while outputting another image on a part of the image.

In addition, an image to be displayed on each divided screen or a full screen may be set according to the configuration of the output screen.

As an example, a front wide-angle image, an image as seen from the sky, and a bird's eye view may be set to be output on the screen based on the image captured by the front camera, or a blind spot image or an adjacent lane based on the image captured by the side camera. You can set the screen to display a video, an adjacent vehicle tracking video, and a video like seen from the sky, and the rear wide-angle video, a video like seen from the sky, and a road video can be set to be output based on the video taken from the rear camera. Also, a driver's seat image, an assistant seat image, a front image, and an all-around indoor image may be set to be output on the screen based on the image captured by the indoor camera.

As an example of configuring the output screen and setting the output image, it is possible to set one of the above-described images to be displayed on the entire screen according to the user's selection, and as another embodiment, the entire screen is divided into three according to the user's selection. , The left blind spot image is displayed on the upper left screen, the right blind spot image is displayed on the upper right screen, and the rear wide-angle image is displayed on the lower screen.

As another embodiment, it is possible to set to recognize that the vehicle is driving and to automatically configure the entire screen as shown in Fig. 3(d), and to output a front wide-angle image, a left/right blind spot image, and a rear wide-angle image. have.

In another embodiment, it is possible to recognize that a T-shape forward parking is in progress, automatically divide the entire screen into four, and set to output an adjacent lane image, a bird's eye image, an image as seen from the sky, and a rear wide-angle image.

In addition, in order to generate an output screen configuration and an output image set as described above, a parameter for image generation may be set (S300) in the image generator including the viewpoint of the virtual camera.

Then, the output image set by the virtual camera-based image generator is generated based on the parameter set in step S300 (S400). That is, as described above, the image generation unit 700 of the vehicle image generation apparatus proposed in the present invention includes a plurality of image generation cores, and by appropriately setting parameters for each image generation core, each image generation core is One of the images captured by a plurality of wide-angle cameras in step S100 is used as an input image, a virtual camera is modeled according to a parameter set in step S300, and an image displayed from the viewpoint of the virtual camera from the input image May be generated, and an output image may be generated by synthesizing the images generated by the plurality of image generating cores according to the set output screen configuration.

The image generated in step S400 may be displayed on the screen (S500).

In addition, the vehicle image generating apparatus acquires driving state information of the vehicle or receives user setting information for generating an output screen configuration and an output image from a user, and sets an output screen configuration and an output image based on this, and the virtual camera The parameters of the base image generator can be set.

In addition, it is possible to additionally control a plurality of wide-angle cameras. The frame rate and resolution settings of the wide-angle camera may be automatically changed according to the output image set in step S200. For example, when driving at a high speed or when a vehicle crashes, a frame rate of a wide-angle camera may be automatically increased to provide a super-high-speed image, or when driving at a low speed, a frame rate may be lowered and resolution may be increased to provide a super-resolution image.

In addition, images captured by a plurality of wide-angle cameras and/or images output in step S400 may be stored in a storage device. In particular, images captured by a plurality of wide-angle cameras may be stored in synchronization with each other. In more detail, images captured at the same time can be stored with the same index.

In addition, the above-described generation method can manually or automatically change the setting of the output screen configuration or change the image to be output in real time, so that various images that can help the driver to drive safely can be generated and provided according to the situation.

21 is a diagram illustrating a method of generating an output image in an image generating apparatus for a vehicle according to an embodiment of the present invention.

Referring to FIG. 21, in order to generate an output image, an image generating apparatus for a vehicle according to an embodiment of the present invention first corrects the values of defective pixels and removes noise with respect to a plurality of images captured by a plurality of wide-angle cameras. (Noise Reduction; NR) and wide dynamic range (WDR) are performed to generate a pre-processed image (S410).

In particular, the wide area correction function may be performed to generate a clear image by removing a difference in illuminance inside and outside the vehicle.

22 is a diagram illustrating a wide area correction method according to an embodiment of the present invention.

Referring to FIG. 22, there may be an initial setting mode and an operation mode for wide area correction. The initial setting mode determines which of a low-exposure image and a high-exposure image is to be used, and sets exposure area information including the result. In the operation mode, a preprocessed image corrected for a wide area is generated based on the exposure area information set in the initial setting mode in real time during operation. To this end, in the initial setting mode, two images for wide area correction having different exposures are first generated by varying the shutter speed of the wide area camera (S510). And, of the two images for wide area correction with different exposures, the area with very low pixel values in the low exposure image and the portion with very high pixel values in the high exposure image are compared with each other, so that the overlapping area between the two images is minimized. Is set (S520). In addition, as the value of each pixel of the preprocessed image corrected for wide area, whether to use the pixel value of the low-exposure image with low exposure or the pixel value of the high-exposure image with high exposure is selected based on the wide area correction threshold, corresponding to each pixel Exposure area information indicating whether a pixel corresponds to a low exposure area or a high exposure area is stored in the form of an image (S530). In the actual operation mode, one of the corresponding pixel values of two images (low-exposure image and high-exposure image) with different exposures input from the wide-angle camera for each frame is selected according to the exposure area information of each pixel already stored. It is determined as the pixel value of the pre-processed image corrected for wide area (S540).

In this way, since the wide-area correction is to create one wide-area corrected image by synthesizing a low-exposure image and a high-exposure image, two images with different exposures for each frame must be photographed and input by a wide-angle camera. Therefore, it is necessary to set the frame rate to be twice as large as that of a camera that does not require wide area correction.

Referring to FIG. 21 again, the pre-processed image undergoes a post-processing step. In the post-processing step, separation of brightness and color signals, edge enhancement, and gamma correction for each of the pre-processed images generated in the pre-processing step (S410). Post-processed images may be generated (S420) by performing at least one of Gamma Correction, Color Correction, and Contrast Enhancement.

In addition, each of the plurality of image generation cores in the virtual camera-based image generation unit takes one of the post-processed images as an input image, models a virtual camera according to the set parameters, and generates an image viewed from the viewpoint of the virtual camera from the input image. It can be generated (S430).

The image generation core may generate an image viewed from the viewpoint of the virtual camera from the input image according to the method illustrated in FIG. 21.

23 is a diagram illustrating a method of generating an image in an image generating core according to an embodiment of the present invention.

) 및 광축 중심(

)을 계산할 수 있다. Referring to FIG. 23, each image generating core according to an embodiment of the present invention generates a virtual camera model according to a set parameter (S610). In this case, when one or more images are to be generated from one input image, one or more virtual camera models may be generated. That is, the main processing unit 500 may set the number of virtual cameras and the position and posture of each virtual camera as parameters. The position and posture set here emulate the digital pan, tilt, and zoom functions of a mechanical camera. In addition, the types of virtual cameras to have a special lens effect and parameters for each type are set. When the settings for the type, position, and posture of the virtual camera are completed, based on this, the coordinates of the scene coordinate system for a point in the image (J) displayed from the viewpoint of the virtual camera (

) And the center of the optical axis (

) Can be calculated.

)로 표현될 수 있고, 설정된 자세 및 위치에 따라 장면 좌표계상의 2차 곡면 모델(

)로 변환될 수 있다. In addition, a second curved surface model may be generated (S620). The second-order surface model can be selected from a plurality of models according to the desired effect. The multiple models include an ellipsoid model that can be used to obtain the pan, tilt, and zoom effects of a virtual camera, a cylinder model for creating a wide-angle image, and a parabolic model to obtain the effect of mapping only a certain couple of images. In addition, a user secondary curved surface model that can be arbitrarily set by the user may be included. When selecting a second-order curved model, parameters suitable for each model can be set together. The set quadratic surface model is a 4x4 matrix (

), and according to the set posture and position, a second-order surface model on the scene coordinate system (

) Can be converted.

Next, a camera projection model for removing distortion included in an image captured by a wide-angle camera may be generated (S630). In order to remove the distortion contained in the image, the camera projection model can use a distortion correction mapping function obtained by a model that is theoretically analyzed and a numerically approximate model, and is preferably an approximation based on a Taylor expansion series. You can use the model. The distortion correction mapping function can be used to map a point of an input image capable of removing distortion.

Then, based on the model generated according to the set parameters as described above, the intersection point of the projection line of the virtual camera model and the second-order curved surface model is calculated (S640). Here, the projection line is a straight line connecting the center of the optical axis of the virtual camera and a point of the image viewed from the viewpoint of the virtual camera. In other words, in order to create an image displayed by a virtual camera, a point (pixel) of the input image must be mapped to a point (pixel) of the image viewed by the virtual camera. It is to calculate the intersection point. Then, the intersection point is mapped back to one point of the input image using a camera projection model including a distortion correction mapping function to generate a final image viewed from the virtual camera (S650). One point of the input image at this time is that distortion included in the input image can be removed.

Each of the plurality of image generation cores may generate one or more images using the method described above. Referring to FIG. 19 again, a plurality of images generated by the plurality of image generation cores are synthesized and output according to a set output screen configuration. An image may be generated (S440). The output image generated in this way may be displayed on the screen.

As described above, according to the vehicle image generating apparatus and method presented in the present invention, various images that can help the driver to drive safely can be provided to the driver based on images captured by a plurality of wide-angle cameras attached to the vehicle. There will be. In particular, it is possible to help improve driving convenience and safety by automatically generating necessary images according to the driving speed or driving direction of the vehicle and providing them to the driver.

Those skilled in the art to which the present invention pertains, since the present invention may be implemented in other specific forms without changing the technical spirit or essential features thereof, the embodiments described above are illustrative in all respects and should be understood as non-limiting. Only do it. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

100: multiple camera unit
200: pretreatment unit
300: storage
400: post-processing unit
500: main processing unit
600: input
700: image generation unit
701, 702, 703: image generation core
710: virtual camera model unit
720: quadratic surface model part
730: camera projection model unit
800: screen portion
900: vehicle sensor unit

Claims (29)

As an image generating device for a vehicle,
A multi-camera unit including a plurality of wide-angle cameras attached to the vehicle; A vehicle sensor unit for acquiring driving state information of the vehicle; A scenario corresponding to the driving state information acquired through the vehicle sensor unit is determined, and an output screen configuration including the positions and sizes of a plurality of divided screens based on the scenario, and an output to be output to each of the plurality of divided screens A processing unit for setting images; A final output image is synthesized so as to be displayed on the plurality of divided screens according to the set output screen configuration according to the set output screen configuration and a plurality of image generation cores based on a virtual camera that generates the output images and output images generated by the plurality of image generation cores. An image generating unit including an image synthesizing unit to generate; A screen unit for outputting the final output image generated by the image generating unit; And a storage unit for storing at least one of an image photographed by the plurality of wide-angle cameras and the final output image generated by the image generation unit,
The processing unit sets a parameter including a virtual camera viewpoint of each of the plurality of image generation cores based on an output image to be generated by each of the plurality of image generation cores, and each of the plurality of image generation cores is generated. Based on the output image to be performed, one of the input images captured by the plurality of wide-angle cameras for each of the plurality of image generating cores is set as an input image,
Each of the plurality of image generation cores receives an input image set by the processing unit as an input, models a virtual camera according to a parameter set by the processing unit, and displays a view of the modeled virtual camera in the set input image. Create a losing image and output it as an output image,
The screen unit may output a plurality of output images, which are viewed simultaneously from multiple viewpoints with respect to one input image captured by one wide-angle camera, on a plurality of split screens,
The processing unit may change the frame rate of the plurality of wide-angle cameras according to the speed of the vehicle,
Vehicle image generating device. The method of claim 1,
The scenario includes a scenario in which the vehicle is running, a scenario in which the vehicle is turning left or right, a scenario in which the vehicle is parked in a forward T-shape, a scenario in which the vehicle is parked in a T-shaped reverse direction, a scenario in which the vehicle is parked in parallel, and the vehicle is At least one of the scenarios where the door opens after stopping,
Vehicle image generating device. delete The method of claim 1, wherein the storage unit,
When storing the final output image generated by the image generator,
To store together a parameter including a viewpoint of the virtual camera of the plurality of image generation cores set to generate the final output image and the output screen configuration,
Vehicle image generating device.
delete delete delete The method of claim 1,
The processing unit includes a pre-processing unit, a post-processing unit, and a main processing unit,
The preprocessor receives images captured by a plurality of wide-angle cameras of the multi-camera unit, and corrects the value of defective pixels for each of the received images (Defect correction), noise reduction, and wide dynamic range. ) Through a pre-processing process that performs at least one of), and transfers the pre-processed images to the post-processing unit,
The post-processing unit separates brightness and color signals, edge enhancement, gamma correction, color correction, and contrast enhancement for each of the preprocessed images received from the preprocessor. Performing a post-processing process including at least one of the post-processed images is generated and transmitted to the image generating unit,
The main processing unit determines a scenario corresponding to the driving state information acquired through the vehicle sensor unit, and an output screen configuration including positions and sizes of a plurality of divided screens based on the scenario, and each of the plurality of divided screens To set the output images to be output to,
Vehicle image generating device. The method of claim 8,
The main processing unit,
Two images with different exposures are captured for at least one wide-angle camera set to perform the wide-angle correction on an image captured by at least one of the plurality of wide-angle cameras and perform wide-angle correction on the captured image To set the parameters so that they can be transferred to the preprocessor,
The pretreatment unit,
For each pixel, performing wide area correction by selecting a corresponding pixel value of one of two images having different exposures and determining a value,
Vehicle image generating device. delete The method of claim 8,
The pretreatment unit,
For each image captured by the plurality of wide-angle cameras from the received image, information associated with at least one of Auto White Balance, Auto Exposure, and Auto Focus is extracted, and the To the main processing unit,
The main processing unit,
Setting parameters for photographing of the plurality of wide-angle cameras based on information related to at least one of automatic color temperature correction, automatic exposure, and auto focus received from the preprocessor,
Vehicle image generating device. delete delete delete The method of claim 1, wherein each of the image generation cores,
A virtual camera model unit for modeling a virtual camera having a viewpoint and a type according to a parameter set by the processing unit;
A second curved surface model unit that provides a second curved surface used to add an image effect when mapping between an output image that is an image viewed by the modeled virtual camera and an input image that is an image captured by the wide-angle camera; And
Including; a camera projection model unit for providing a camera projection model for performing mapping between the output image of the modeled virtual camera and the input image while removing distortion included in the input image; including,
Vehicle image generating device. The method of claim 15,
The camera projection model unit,
Calculate the intersection of the projection line and the secondary curved surface, which is a straight line connecting the center of the optical axis of the virtual camera modeled by the virtual camera model and a point on the output image
By mapping the intersection point to a point on the input image using a distortion correction mapping function for removing distortion included in the input image,
Performing mapping between the output image of the modeled virtual camera and the input image,
Vehicle image generating device. In the vehicle image generation method,
Capturing a plurality of images with a plurality of wide-angle cameras; Configuring an output screen including a plurality of divided screens and setting output images to be output to each of the plurality of divided screens; Generating the set output images; Synthesizing the generated output images to be displayed on the plurality of split screens according to the set output screen configuration to generate a final output image; Storing at least one of an image captured by the plurality of wide-angle cameras and the final output image; And displaying the final output image; Including,
The configuring of the output screen and setting the output images may include: acquiring driving state information of the vehicle; Determining a scenario corresponding to the driving state information; And setting an output screen configuration including positions and sizes of the plurality of divided screens and output images to be output to each of the plurality of divided screens based on the scenario; Including,
The generating of the set output images may include generating one output image by each of a plurality of image generation cores based on a virtual camera,
The step of generating one output image by each of the plurality of image generating cores may include: setting a parameter including a viewpoint of a virtual camera of each image generating core based on the output image to be generated; Selecting one of the plurality of images as an input image based on the output image to be generated; Modeling a virtual camera according to a parameter including a set point of view of the virtual camera; And generating an image displayed according to the viewpoint of the modeled virtual camera from the selected input image as an output image; Including,
Displaying the generated output images on the plurality of divided screens based on the setting includes displaying the output images generated by the plurality of image generating cores on the plurality of divided screens according to the set output screen configuration. Including;
It is possible to output a plurality of output images, such as viewed from multiple viewpoints at the same time, on a plurality of split screens with respect to one input image captured by a single wide-angle camera,
It is possible to change the frame rate of the plurality of wide-angle cameras according to the speed of the vehicle,
How to create an image for a vehicle. The method of claim 17,
The scenario includes a scenario in which the vehicle is running, a scenario in which the vehicle is turning left or right, a scenario in which the vehicle is parked in a forward T-shape, a scenario in which the vehicle is parked in a T-shaped reverse direction, a scenario in which the vehicle is parked in parallel, and the vehicle is At least one of the scenarios where the door opens after stopping,
How to create an image for a vehicle. The method of claim 17,
The step of configuring the output screen and setting the output image,
Setting a viewpoint of the virtual camera by setting zoom and tilt parameters of a virtual camera according to a vehicle progress speed among the driving state information;
Or setting a viewpoint of the virtual camera by setting zoom and pan parameters of the virtual camera according to the vehicle traveling direction of the driving state information;
Or setting zoom, pan, and tilt parameters of the virtual camera according to the setting information to set the viewpoint of the virtual camera; including,
How to create an image for a vehicle. delete delete The method of claim 17,
The storing of at least one of an image photographed by the plurality of wide-angle cameras and the final output image may include storing the final output image,
Including, storing parameters including viewpoints of the virtual cameras of the plurality of image generation cores set to generate the final output image and the output screen configuration; including,
How to create an image for a vehicle. delete The method of claim 17,
Generating the set output image,
The plurality of images captured by the plurality of wide-angle cameras are received, and values of defective pixels are corrected for each of the plurality of received images (Defect correction), noise reduction, and wide dynamic range. A pre-processing step of generating pre-processed images by performing at least one of); And
At least one of brightness and color signal separation, edge enhancement, gamma correction, color correction, and contrast enhancement for each of the preprocessed images generated in the preprocessing step A post-processing step of generating post-processed images by performing; further comprising,
Selecting one of the plurality of images as an input image based on the output image to be generated,
In the step of selecting one of the post-processed images as an input image based on the output image to be generated,
How to create an image for a vehicle. The method of claim 24,
The wide area correction of the pre-processing step,
Generating two wide-area correction images having different exposures by adjusting a shutter speed of a wide-angle camera;
Setting a wide area correction threshold based on the two wide area correction images;
Determining an image in which a pixel value is to be used among two images for wide area correction having different exposures for each pixel value in the image based on the wide area correction threshold value, and storing the image as exposure area information; And
In order to generate a wide-area-corrected pre-processed image, a pixel value of each pixel of a wide-area-corrected pre-processed image is determined, and a pixel of one selected image among two images having different exposures input from the wide-angle camera selected according to the exposure area information Including; determining as a value
How to create an image for a vehicle. delete The method of claim 24,
The pretreatment step,
Extracting information associated with at least one of auto color temperature correction, auto exposure, and auto focus for each of the plurality of received images; further comprising,
Setting the parameter is
Including, additionally based on information related to at least one of automatic color temperature correction, automatic exposure, and auto focus received from the pre-processing step, and setting parameters for photographing of the plurality of wide-angle cameras; including,
How to create an image for a vehicle. The method of claim 17,
The step of generating an image displayed according to the viewpoint of the modeled virtual camera from the selected input image as an output image,
Modeling a second-order curved surface used to add an image effect when mapping between the image displayed by the modeled virtual camera and the input image;
Generating a camera projection model to perform mapping between an image viewed from a viewpoint of the virtual camera and an input image while removing distortion included in the input image;
Calculating an intersection point of the second curved surface and a projection line that is a straight line connecting a point on the image viewed from the viewpoint of the virtual camera and the center of the optical axis of the virtual camera; And
Mapping the intersection point to a point on the input image using a distortion correction mapping function for removing distortion included in the input image; including,
How to create an image for a vehicle. A computer program stored on a computer-readable recording medium for executing the method according to claim 17 when executed on a computer or processor.

Images