KR102441052B1 - Apparatus and method for transforming avm image - Google Patents

Abstract

The present invention relates to an AVM image conversion apparatus and method. The apparatus according to the present invention maps pixels of a wide-angle image input from a camera to corresponding coordinates of a narrow-angle image using a predefined lookup table, and performs hole filling for holes formed in the pixel-mapped narrow-angle image. a mapping unit generating a virtual narrow-angle image by performing and a synthesizing unit configured to generate an output narrow-angle image of a current frame by synthesizing effective pixels mapped from the wide-angle image to the narrow-angle image with the effective pixels of the output narrow-angle image of the previous frame.

Description

AVM image conversion apparatus and method {APPARATUS AND METHOD FOR TRANSFORMING AVM IMAGE}

The present invention relates to an AVM image conversion apparatus and method.

The AVM system converts images around the vehicle acquired using multiple wide-angle cameras into narrow-angle images, synthesizes them, and provides a top-view image to the user so that the user can secure a field of view in the blind area.

The AVM system uses a warping technique to convert a wide-angle image into a narrow-angle image. Examples of the warping technique include backward warping and forward warping.

However, when a part of a wide-angle image is converted into a narrow-angle image by using a warping technique and the converted image is enlarged, the number of pixels present in the conversion region of the wide-angle image is small, so pixels transmitted to the narrow-angle image information is limited. Accordingly, the image quality of the converted narrow-angle image is deteriorated and the sharpness is deteriorated.

An object of the embodiments of the present invention is to convert a wide-angle image to a narrow-angle image from a wide-angle image to an effective pixel in an output narrow-angle image of a previous frame by using a forward warping technique of a progressive method when a viewpoint is converted into a narrow-angle image. By accumulating and synthesizing the effective pixels mapped with have.

An AVM image conversion apparatus according to the present invention for achieving the above object maps pixels of a wide-angle image input from a camera to corresponding coordinates of a narrow-angle image using a predefined lookup table, and maps the pixels to the pixel-mapped narrow-angle image. A mapping unit generating a virtual narrow-angle image by performing hole filling on the formed hole, estimating the motion of pixels between the virtual narrow-angle image and the output narrow-angle image of the previous frame, and according to the motion estimation result, A matching unit for moving the pixels of the output narrow-angle image, and a synthesizing unit for generating the output narrow-angle image of the current frame by synthesizing the effective pixels mapped from the wide-angle image to the narrow-angle image with the effective pixels of the output narrow-angle image of the previous frame characterized in that

In addition, in an AVM image conversion method according to an embodiment of the present invention for achieving the above object, mapping pixels of a wide-angle image input from a camera to corresponding coordinates of a narrow-angle image using a predefined lookup table , generating a virtual narrow-angle image by performing hole filling on the hole formed in the pixel-mapped narrow-angle image, estimating the motion of pixels between the virtual narrow-angle image and the output narrow-angle image of the previous frame, and the motion estimation result moving pixels of the output narrow-angle image of the previous frame according to It is characterized in that it comprises the step of

According to embodiments of the present invention, when a view is converted from a wide-angle image to a narrow-angle image, an effective pixel in the output narrow-angle image of the previous frame is converted from a wide-angle image to a narrow-angle image by using a forward warping technique of a progressive method. By accumulating and synthesizing effective pixels mapped to , there is an advantage in that it is possible to minimize the image quality degradation caused by the limitation of the number of effective pixels when converting a wide-angle image to a narrow-angle image and to output a clearer result image.

1 is a diagram illustrating the configuration of an AVM image conversion apparatus according to an embodiment of the present invention.
2 to 5 are diagrams illustrating an embodiment referenced to describe an image conversion operation flow of an AVM image conversion apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating an operation flow of an AVM image conversion method according to an embodiment of the present invention.

It should be noted that the technical terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in the present invention should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise specifically defined in the present invention. It should not be construed as meaning or in an excessively reduced meaning. In addition, when the technical term used in the present invention is an incorrect technical term that does not accurately express the spirit of the present invention, it should be understood by being replaced with a technical term that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted as defined in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, the singular expression used in the present invention includes the plural expression unless the context clearly dictates otherwise. In the present invention, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the invention, and some components or some steps may not be included. It should be construed that it may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used in the present invention may be used to describe the components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of the reference numerals, and the redundant description thereof will be omitted.

In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention and should not be construed as limiting the spirit of the present invention by the accompanying drawings.

1 is a diagram illustrating the configuration of an AVM image conversion apparatus according to an embodiment of the present invention.

The AVM image conversion apparatus 100 according to the present invention may be implemented inside a vehicle. In this case, the AVM image conversion apparatus 100 may be integrally formed with the vehicle's internal control units, or may be implemented as a separate device and connected to the vehicle's control units by a separate connection means. Here, the AVM image conversion apparatus 100 may operate in connection with the AVM system provided in the vehicle, or may be implemented and operated in the AVM system.

Here, the AVM system is an abbreviation of the Around View Monitoring system, which converts wide-angle images captured using one or more wide-angle cameras into a narrow-angle image, such as looking down from the top of the vehicle, and then synthesizes them to provide a top-view image so that the driver can monitor the surroundings of the vehicle. This is to ensure visibility into the blind area.

Accordingly, referring to FIG. 1 , the AVM image conversion apparatus 100 may include a control unit 110 , a camera 120 , an interface unit 130 , a storage unit 140 , and an image conversion unit 150 . Here, the controller 110 processes a signal transmitted between each unit of the AVM image conversion apparatus.

The camera 120 is a device for capturing images around the vehicle, and may be provided, for example, in the front, rear, left and right sides of the vehicle. Here, the camera 120 may be a wide-angle camera. The camera 120 transmits a wide-angle image captured around the vehicle to the controller 110 . In this case, the controller 110 may store the wide-angle image transmitted from the camera 120 in the storage 140 and provide it to the mapping unit 160 .

Of course, the AMV image conversion apparatus may receive a wide-angle image captured around the vehicle from the AVM system. In this case, the AVM image conversion apparatus 100 may not include a separate camera 120 .

The interface unit 130 may include an input unit for receiving a control command from a user and an output unit for outputting an operation state and result of the AVM image conversion apparatus 100 .

Here, the input means may correspond to a key button, and may correspond to a mouse, a joystick, a jog shuttle, a stylus pen, or the like. In addition, the input means may correspond to a soft key implemented on the display.

The output means may include a display, and may include an audio output means such as a speaker. In this case, when a touch sensor such as a touch film, a touch sheet, or a touch pad is provided in the display, the display operates as a touch screen, and the input means and the output means are integrated.

In this case, the display includes a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (Flexible Display). , a field emission display (FED), and a three-dimensional display (3D display) may include at least one.

The storage unit 140 stores data and programs necessary for the AVM image conversion apparatus 100 to operate.

As an example, the storage 140 may store a look-up table generated by modeling a pixel mapping relationship between a wide-angle image and a narrow-angle image in advance. The stored lookup table may be called by the mapping unit 160 and used for pixel mapping based on a wide-angle image.

Also, the storage unit 140 may store an algorithm for pixel mapping, hole filling, and/or motion estimation in the AVM image conversion apparatus 100 .

Here, the storage unit 140 is a flash memory type (Flash Memory Type), a hard disk type (Hard Disk Type), a multimedia card micro type (Multimedia Card Micro Type), a card type memory (eg, SD or XD memory) etc.), magnetic memory, magnetic disk, optical disk, RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), PROM (Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory) may include at least one storage medium.

The image converter 150 converts a wide-angle image into a narrow-angle image by using a warping technique based on a single image. The image converter 150 converts a wide-angle image into a narrow-angle image by using a forward warping technique among warping techniques. The forward warping technique refers to performing image conversion by transferring a pixel value of a pixel existing in a wide-angle image to a position of a corresponding narrow-angle image.

Here, the image conversion unit 150 may include a mapping unit 160 , a matching unit 170 , and a synthesis unit 180 .

The mapping unit 160 calls a lookup table generated by modeling the pixel relationship between the wide-angle image and the narrow-angle image in advance from the storage unit 140 . In this case, the mapping unit 160 may determine to which position in the corresponding narrow-angle image each pixel of the wide-angle image is transferred from the called lookup table. Accordingly, the mapping unit 160 maps the pixels of the wide-angle image captured by the camera 120 to corresponding coordinates in the narrow-angle image by using the lookup table. Here, the mapping unit 160 may store information on the effective pixel mapped to the narrow-angle image in the storage unit 140 .

When the position of the pixel transmitted to the narrow-angle image is a non-integer position, the mapping unit 160 may perform interpolation processing using neighboring pixels.

Meanwhile, in the process of mapping the pixels of the wide-angle image to the narrow-angle image, a hole may be generated at a position where the pixel is not transmitted in the narrow-angle image. In this case, the mapping unit 160 may perform hole filling through interpolation or filtering using pixels surrounding the corresponding hole in order to remove the hole generated in the narrow-angle image. In this case, the mapping unit 160 may generate the hole-filled image as a virtual narrow-angle image.

The controller 110 may store the virtual narrow-angle image generated by the mapping unit 160 in the storage 140 , and may transmit it to the matching unit 170 for registration processing.

The matching unit 170 performs image registration between the virtual narrow-angle image generated by the mapping unit 160 and the output narrow-angle image of a previous frame stored in advance in the storage 140 . Here, the matching unit 170 performs motion estimation between the virtual narrow-angle image and the output narrow-angle image of the previous frame, and obtains a motion vector for the entire output narrow-angle image of the previous frame from the motion estimation result. estimate

In this case, the matching unit 170 may move all pixels of the output narrow-angle image of the previous frame to correspond to the narrow-angle image of the current frame by using the estimated motion vector.

The synthesizing unit 180 synthesizes the effective pixels mapped from the wide-angle image to the narrow-angle image by the mapping unit 160 with the output narrow-angle image of the previous frame pixel-shifted by the matching unit 170 to combine the output narrow-angle image of the current frame. can create The output narrow-angle image of the current frame is stored in the storage unit 140 and may be used to generate an output narrow-angle image of the next frame.

As described above, since the present invention progressively accumulates and synthesizes effective pixels mapped from wide-angle images to narrow-angle images to effective pixels in the output narrow-angle image of the previous frame, the number of effective pixels when converting a wide-angle image to a narrow-angle image It is possible to minimize the image quality degradation caused by the limitation of

2 to 5 are diagrams illustrating an embodiment referenced to describe an image conversion operation flow of an AVM image conversion apparatus according to an embodiment of the present invention.

First, FIG. 2 shows a flow of an operation of converting a wide-angle image into a narrow-angle image in the image converter 150 of FIG. 1 .

Referring to FIG. 2 , when the wide-angle image 210 of the current frame captured by the camera 120 is input, the image converter 150 models the pixel relationship between the wide-angle image and the narrow-angle image from the storage 140 in advance. The generated lookup table 215 is called, and each pixel of the input wide-angle image 210 is mapped to corresponding coordinates in the narrow-angle image as in operation 220 based on the called lookup table 215 .

The image converter 150 may generate a narrow-angle image 225 corresponding to the input wide-angle image of the current frame by pixel mapping at 220, and in this case, effective pixel information of the generated narrow-angle image 225 is performed in operation 260 It can be applied to pixel synthesis to generate an output narrow-angle image in

Meanwhile, in operation 230 , the image converter 150 performs interpolation or filtering using neighboring pixels of the corresponding hole in order to remove a hole generated because a pixel is not transmitted from the generated narrow-angle image 225 . After the execution, a virtual narrow-angle image of the current frame may be generated and used to estimate motion with respect to the output narrow-angle image of the previous frame.

The image conversion apparatus compares the output narrow-angle image of the previous frame with the virtual narrow-angle image of the current frame to estimate motion of all pixels with respect to the output narrow-angle image of the previous frame in operation 250 . In this case, the image conversion apparatus may estimate the motion vector from the motion estimation result.

The image conversion apparatus moves all pixels of the output narrow-angle image of the previous frame according to the narrow-angle image of the current frame by using the estimated motion vector.

The image conversion apparatus may generate an output narrow-angle image of the current frame by synthesizing effective pixels in the narrow-angle image of the pixel-mapped current frame in operation 220 with the output narrow-angle image of the previous frame, which is pixel-shifted in operation 260 .

FIG. 3 illustrates a pixel change of an operation of converting a wide-angle image into a narrow-angle image in the image converter 150 of FIG. 1 .

Referring to FIG. 3 , (a) of FIG. 3 shows the input wide-angle image of the current frame, and when pixels of the input wide-angle image are mapped to corresponding coordinates in the narrow-angle image using a pre-stored lookup table, (b) of FIG. ), a narrow-angle image is generated.

However, in the narrow-angle image shown in FIG. 3B , a hole may occur due to a lack of effective pixels 310 . At this time, the image conversion unit 150 fills the hole 320 around the effective pixel as shown in FIG. 3C by performing hole filling.

The image converter 150 compares the output narrow-angle image of the previous frame with the virtual narrow-angle image filled with holes in FIG. As in d), the effective pixel 330 in the output narrow-angle image of the previous frame is moved according to the estimated motion information.

Accordingly, the image conversion unit 150 accumulates and synthesizes the effective pixels 310 of the narrow-angle image shown in FIG. Outputs the output narrow-angle image of the current frame.

In FIG. 3E , reference numeral 340 denotes a pixel obtained by accumulating and synthesizing effective pixels 310 pixel-mapped from the wide-angle image with the effective pixels 330 of the output narrow-angle image of the previous frame, and the pixel-mapped effective pixels 310 ) can have higher sharpness.

In addition, in the output narrow-angle image of the current frame, holes in the narrow-angle image shown in FIG. have.

4 shows an input wide-angle image captured by a camera, and FIG. 5 shows a narrow-angle image obtained by converting the input wide-angle image using a forward warping technique.

As shown in FIG. 4, when the AVM image conversion apparatus according to the present invention performs forward warping on a partial region 415 of the wide-angle image 410 captured by the camera 120, as shown in FIG. A narrow-angle image is obtained. In this case, in the obtained narrow-angle image, image quality is deteriorated due to a hole occurring in many pixel coordinates due to the limitation of effective pixels.

The AVM image conversion apparatus according to the present invention may perform hole filling on a hole generated in the narrow-angle image of FIG. 5A . At this time, the narrow-angle image filled with holes is in a state as shown in (b) of FIG. 5 .

It can be seen that the image quality deterioration of the narrow-angle image shown in FIG. 5(b) is improved compared to the narrow-angle image shown in (a) due to hole filling, but image quality deterioration still occurs.

Therefore, the AVM image conversion apparatus compares the narrow-angle image shown in FIG. 5(b) with the narrow-angle image generated as a result of the output of the previous frame, and performs image registration for the output narrow-angle image of the previous frame using the estimated motion vector. If the effective pixels of the narrow-angle image shown in FIG. 5A are synthesized with the resulting image, an output narrow-angle image as shown in FIG. 5C can be obtained.

The operation flow of the device according to the present invention configured as described above will be described in more detail as follows.

6 is a flowchart illustrating an operation flow of an AVM image conversion method according to an embodiment of the present invention.

As shown in FIG. 6 , when a wide-angle image photographed around a vehicle is input by the camera 120 ( S110 ), the AVM image conversion apparatus converts pixels of the wide-angle image to the corresponding pixels of the narrow-angle image using a pre-stored lookup table. By mapping the coordinates (S120), the wide-angle image is converted into a narrow-angle image (S130).

Holes may occur in the narrow-angle image converted in the process 'S130' due to the limitation of effective pixels. At this time, the AVM image conversion apparatus performs hole filling on the narrow-angle image (S140) to generate a virtual narrow-angle image (S150) .

On the other hand, the AVM image conversion apparatus calls the output narrow-angle image of the previous frame (S160), and compares the output narrow-angle image of the called previous frame with the virtual narrow-angle image generated in the 'S150' process to estimate motion for all pixels. . In this case, the AVM image conversion apparatus may estimate a motion vector from the motion estimation result (S170).

The AVM image conversion apparatus moves the pixels of the output composite image of the previous frame according to the narrow-angle image of the current frame by using the motion vector estimated in the 'S170' of the output narrow-angle image of the previous frame (S180).

Accordingly, the AVM image conversion apparatus accumulates and synthesizes the effective pixels of the narrow-angle image of the current frame converted in the process 'S130' with the effective pixels of the output synthesized image of the previous frame pixel-shifted in the process 'S180' (S190), and the current frame to output the final narrow-angle image of (S200).

The above processes may be directly implemented in hardware executed by a processor, a software module, or a combination of the two. A software module may reside in a storage medium such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, ie, memory and/or storage. An exemplary storage medium is coupled to the processor, the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral with the processor. The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and all technical ideas with equivalent or equivalent modifications to the claims as well as the claims to be described later are included in the scope of the present invention. should be interpreted as

110: control unit 120: camera
130: interface unit 140: storage unit
150: image conversion unit 160: mapping unit
170: matching unit 180: compounding unit

Claims (12)

A virtual narrow-angle image is created by mapping the pixels of the wide-angle image input from the camera to the corresponding coordinates of the narrow-angle image using a predefined lookup table, and performing hole filling on the hole formed in the pixel-mapped narrow-angle image. a mapping unit to generate;
a matching unit for estimating a motion of a pixel between the virtual narrow-angle image and an output narrow-angle image of a previous frame, and moving the pixel of the output narrow-angle image of the previous frame according to a motion estimation result; and
A synthesizing unit generating an output narrow-angle image of a current frame by synthesizing effective pixels mapped from the wide-angle image to the narrow-angle image with the effective pixels of the output narrow-angle image of the previous frame
including,
The synthesis part,
The AVM image conversion apparatus according to claim 1, wherein effective pixels mapped from the wide-angle image to the narrow-angle image are progressively accumulated and synthesized with effective pixels in the output narrow-angle image of the previous frame. The method according to claim 1,
The lookup table is
AVM image conversion device, characterized in that it is generated by modeling the pixel relationship between the wide-angle image and the narrow-angle image in advance. The method according to claim 1,
The matching part,
and calculating a motion vector by estimating a motion of a pixel between the virtual narrow-angle image and an output narrow-angle image of a previous frame. 4. The method of claim 3,
The matching part,
AVM image conversion apparatus, characterized in that moving the effective pixel in the output narrow-angle image of the previous frame according to the motion vector. delete The method according to claim 1,
The mapping unit,
The AVM image conversion apparatus according to claim 1, wherein the hole filling is performed through interpolation or filtering on the hole using pixels around the hole formed in the narrow-angle image. mapping pixels of a wide-angle image input from a camera to corresponding coordinates of a narrow-angle image using a predefined lookup table;
generating a virtual narrow-angle image by performing hole filling on a hole formed in the pixel-mapped narrow-angle image;
estimating a motion of a pixel between the virtual narrow-angle image and an output narrow-angle image of a previous frame, and moving pixels of the output narrow-angle image of the previous frame according to a result of motion estimation; and
generating an output narrow-angle image of the current frame by synthesizing effective pixels mapped from the wide-angle image to the narrow-angle image with the effective pixels of the output narrow-angle image of the previous frame;
including,
The step of generating the output narrow-angle image of the current frame comprises:
An AVM image conversion method, characterized in that effective pixels mapped from the wide-angle image to the narrow-angle image are progressively accumulated and synthesized with effective pixels in the output narrow-angle image of the previous frame. 8. The method of claim 7,
The lookup table is
AVM image conversion method, characterized in that it is generated by modeling the pixel relationship between the wide-angle image and the narrow-angle image in advance. 8. The method of claim 7,
The step of moving the pixel of the output narrow-angle image of the previous frame comprises:
and calculating a motion vector by estimating a motion of a pixel between the virtual narrow-angle image and an output narrow-angle image of a previous frame. 10. The method of claim 9,
The step of moving the pixel of the output narrow-angle image of the previous frame comprises:
AVM image conversion method, characterized in that moving the effective pixel in the output narrow-angle image of the previous frame according to the motion vector. delete 8. The method of claim 7,
The step of generating the virtual narrow-angle image comprises:
AVM image conversion method, characterized in that the hole filling is performed through interpolation or filtering on the hole using pixels around the hole formed in the narrow-angle image.

Images