WO2017033691A1 - Image generation device, image generation method, and program - Google Patents

Abstract

An image generation unit (20) performs distortion correction for correcting distortion of a captured image captured by an imaging unit (10) installed in a vehicle to generate an object image covering a maximum range of an image to be displayed. An image display unit (20) displays the generated object image in a display area of a display unit. An indefinite region determination unit (20) determines whether at least one indefinite region in which no image is formed in the object image is present or not, on the basis of image height data indicating the distance from an imaging center of the imaging unit to each pixel corresponding to each image point and the posture of the installed imaging unit. If at least one indefinite region is present as a result of the determination, a mask setting unit (20) sets a mask region that is a rectangular region having a straight line circumscribed on the at least one indefinite region and covers an end portion of the object image in which the indefinite region is present. The image generation unit generates, as a new object image, an image generated by performing masking for filling the set mask region with prescribed color on the object image.

Description

Image generating apparatus, image generating method, and program Cross-reference of related applications

This application is based on Japanese Patent Application No. 2015-167891 filed on August 27, 2015, and claims the benefit of its priority. Is incorporated herein by reference.

This disclosure relates to a technique for generating an image to be displayed on a display unit.

2. Description of the Related Art Conventionally, an image display device mounted on an automobile is known that includes an imaging unit, an image processing unit, and a display unit (see Patent Document 1). The imaging unit captures an image through a wide-angle lens and is installed so as to capture a predetermined area around the vehicle. The image processing unit corrects distortion of the image captured by the imaging unit and specifies a target image that is an area of the image to be displayed on the display unit. The display unit displays the target image specified by the image processing unit.

JP 2013-110712 A

In the image display device, an indefinite region where an image is not formed in the target image is generated depending on the installation posture of the imaging unit. This indefinite area is formed by correcting distortion of an image taken through a wide-angle lens and specifying an area having a certain range and size as a target image in the corrected image.

Since this indefinite area is a semicircular area formed at the periphery of the image, there arises a problem that the user who visually recognizes the image displayed on the display unit feels uncomfortable. In order to solve this problem, it is considered to mask an indefinite region of a target image in an image display device.

However, the area to be masked needs to be obtained in advance according to the indefinite area. Therefore, each time the posture where the image pickup unit is installed is changed, it is necessary to re-determine the area to be masked, which is troublesome.

This disclosure aims to reduce the labor for obtaining an area to be masked while masking an indefinite area in an image displayed on a display unit.

The first aspect of the present disclosure relates to an image generation apparatus including an image generation unit, an image display unit, an indeterminate region determination unit, and a mask setting unit.

The image generation unit performs a process of correcting distortion of the captured image captured by the imaging unit, and generates a target image that is the maximum range of the image to be displayed. The imaging unit referred to here is an imaging unit that captures an image through a wide-angle lens, and is installed in the vehicle so as to capture a predetermined area around the vehicle.

The image display unit displays the target image generated by the image generation unit in the display area of the display unit. Furthermore, the indeterminate area determination unit has at least one indeterminate area in which no image is formed in the target image based on image height data representing the distance from the imaging center of the imaging unit to each pixel and the attitude at which the imaging unit is installed. Determine if it exists.

The mask setting unit is a rectangular region having a straight line circumscribing the at least one indeterminate region when there is at least one indeterminate region as a result of the determination by the indeterminate region determining unit, and the indefinite region A mask area that covers the edge of the target image in which is present is set.

Further, the image generation unit generates, as a new target image, an image obtained by performing masking on the target image by painting the mask area set by the mask setting unit with a specified color.

According to such an image generation device, the presence / absence of an indeterminate area is determined based on the attitude at which the imaging unit is installed and the image height data. If the result of the determination is that an indefinite area exists, a mask area is set. it can. Since the setting of the mask area is automatically performed by the image generation apparatus, it is not necessary to obtain the mask area in advance for each installation posture of the imaging unit, and labor can be reduced.

In other words, according to the image generation device, it is possible to reduce the labor for obtaining the area to be masked while masking the indefinite area in the image displayed on the display unit.
The mask setting unit includes an image specifying unit that specifies a specific image area that is a rectangular area inscribed in each of the indefinite areas in the target image when a plurality of indefinite areas exist as a result of the determination in the indefinite area determination unit. It may be.

In this case, the mask setting unit may set a region outside the specific image region specified by the image specifying unit as a mask region.

According to such an image generation device, when there are a plurality of indefinite areas, the corrected image does not include the indefinite area, and the maximum range in the image to be displayed can be set as the specific image area. .

In the corrected image, an optimum mask area can be set by setting the area outside the specific image area as the mask area.

Further, the present disclosure may be made as an image generation method for generating a target image, or may be made as a program executed by a computer mounted on a vehicle.

According to the former image generation method, the same effect as that of the image generation apparatus can be obtained. If it is the latter program, it can be used by loading it into a computer from a recording medium as needed and starting it, or by acquiring it and starting it through a communication line as needed. Then, by causing the computer to execute each procedure, the computer can function as an image generation device.

The recording medium referred to here includes, for example, a computer-readable electronic medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), and a hard disk.
Note that the reference numerals in parentheses described in the “Claims” column indicate a corresponding relationship with specific means described in the embodiment described later as one aspect, and are within the technical scope of the present invention. It is not intended to limit.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. The drawing
FIG. 1 is a block diagram illustrating a schematic configuration of an image display device. FIG. 2 is a flowchart showing a processing procedure of image display processing. FIG. 3 is a flowchart showing the procedure of the initialization process. 4A and 4B are explanatory diagrams for explaining an overview of the initialization process. FIG. 4A is a diagram for explaining a target image in which one indefinite region exists. FIG. 4B is a diagram for explaining a target image in which the indefinite region is masked. It is a diagram to explain, 5A and 5B are diagrams for explaining an overview of the initialization process. FIG. 5A is a diagram for explaining a target image having a plurality of indeterminate regions. FIG. 5B is a diagram for explaining a specific image region to be displayed. And FIG. 6 is a flowchart showing a processing procedure of image generation processing. 7A and 7B are explanatory diagrams for explaining the outline of the display process. FIG. 7A is a diagram for explaining the process when a person is present in the area corresponding to the mask area. FIG. 7B is a process for the upper mask. FIG.

Embodiments of the present invention will be described below with reference to the drawings.

<Image display device>
An image display apparatus 1 shown in FIG. 1 is an apparatus mounted on an automobile, and is an apparatus that displays an image obtained by photographing the periphery of the host vehicle by performing image processing.

The image display device 1 includes an imaging unit 10, a storage unit 14, an image processing unit 20, and a display unit 30.

The imaging unit 10 is a known imaging device that generates an image obtained by projecting real space onto a rectangular plane composed of a plurality of pixels. This imaging apparatus includes an imaging element, an optical lens, and peripheral electronic circuits such as a power source.

The image sensor is a so-called CCD (charged-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor.
An optical lens is an optical element that converges or diverges light. In this embodiment, a wide-angle lens is used as the optical lens. The wide-angle lens mentioned here is a lens having a wide angle of view (for example, 60 degrees or more) and a short focal length. As an example of this wide-angle lens, a fish-eye lens (for example, a lens having an angle of view of 180 degrees or more) can be considered.

The imaging unit 10 is attached to the host vehicle so that the area around the host vehicle is a shooting area. The attitude data of the imaging unit 10 and the image height data of the imaging unit 10 are stored in the storage unit 14 described later in detail.

The attitude data referred to here is data representing the attitude of the imaging unit 10 attached to the vehicle, and is data obtained in advance for each imaging unit 10. This attitude data includes an x coordinate that is a coordinate along the full length direction of the vehicle, a y coordinate that is a coordinate along the vehicle width direction of the vehicle, and a z coordinate that is a coordinate along the vehicle height direction of the vehicle. Further, the posture data includes a yaw representing the inclination in the xy plane, a pitch representing the inclination in the xz plane, and a roll representing the inclination in the yz plane.

Further, the image height data referred to here is data representing the distance from the imaging center of the imaging unit 10 to each pixel corresponding to each image point, and is obtained in advance for each imaging unit 10. Note that the imaging center of the imaging unit 10 referred to here is a point where the imaging plane of the imaging unit 10 and the optical axis of the wide-angle lens intersect.

The storage unit 14 is a rewritable nonvolatile storage device. As an example of the storage unit 14, a hard disk drive, a flash memory, or the like can be considered.

The display unit 30 is a display device that displays an image based on a signal from the image processing unit 20. As an example of the display unit 30, a known liquid crystal display, a projector, or the like can be considered.

<Image processing unit>
The image processing unit 20 is an image processing apparatus configured around a known microcomputer having a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 24, and a CPU (Central Processing Unit) 26. The ROM 22 stores data and programs that need to retain the stored contents even when the power is turned off. The RAM 24 temporarily stores data. The CPU 26 executes processing according to a program stored in the ROM 22 or RAM 24. The ROM 22 stores a program that performs processing for correcting distortion of a captured image captured by the imaging unit 10, generates a target image that is an image to be displayed, and executes image display processing for displaying the target image on the display unit 30. ing. The image processing unit 20 is an example of an image generation device. The image processing unit 20 includes an image generation unit (including an enlargement unit), an image display unit (including a color information acquisition unit, a color determination unit, a color change unit, a person determination unit, and a release unit), an indefinite region determination unit, and This corresponds to a mask setting unit (including an image specifying unit).

<Image display processing>
The image display process is activated when a predefined activation command is input. The start command referred to here is a command for starting image display processing. The activation command may be, for example, a signal input to the image processing unit 20 when a specific switch is turned on.

When the image display process is activated, the image processing unit 20 first executes an initialization process for generating a correspondence map and mask data as shown in FIG. 2 (step S110). The correspondence map referred to here is a map for converting an image captured by the imaging unit 10 into a target image, and a map indicating which pixel in the captured image captured by the imaging unit 10 corresponds to which pixel in the target image. It is. This correspondence map is created so as to correct the distortion of the captured image. The target image referred to here is the maximum range of images to be displayed on the display unit 30. The target image is generated by an image generation process described later. Note that the mask data is data representing the position and size of a region to be masked in the target image.

Subsequently, in the image display process, the image processing unit 20 converts the captured image captured by the imaging unit 10 into a target image based on the correspondence map and the mask data generated in the initialization process, and displays the display unit 30. The image generation processing to be displayed on the screen is executed (step S120).

Further, in the image display process, the image processing unit 20 determines whether or not an end command that is a command to end the image display process is input (step S130). If the end command is not input as a result of the determination in step S130 (step S130: NO), the image processing unit 20 returns the image display process to step S120. That is, a target image based on an image newly captured by the imaging unit 10 is generated and displayed on the display unit 30.

On the other hand, if the end instruction is input as a result of the determination in step S130 (step S130: YES), the image processing unit 20 ends the image display process.

<Initialization process>
Next, the initialization process activated in step S110 of the image display process will be described.

When this initialization process is started, the image processing unit 20 first acquires image height data and orientation data from the storage unit 14 as shown in FIG. 3 (step S210).

Subsequently, in the initialization process, the image processing unit 20 creates a correspondence map based on the image height data and the posture data acquired in step S210, and stores it in the storage unit 14 (step S220). That is, in the image generation processing described later in detail, based on the correspondence map, the luminance value of each pixel in the captured image captured by the imaging unit 10 is displayed on the display unit 30 as the luminance value of each corresponding pixel in the target pixel. Is done.

Note that an image captured by the imaging unit 10 via the wide-angle lens is distorted by the distortion of the wide-angle lens. For this reason, as shown in FIG. 4A, the correspondence map has a maximum range that should be displayed on the display unit 30 in the image captured in the captured image while correcting distortion of the captured image as much as possible. It is created so as to generate a rectangular area as a target image. In addition, since the creation method of a correspondence map is well-known, detailed description here is abbreviate | omitted.

In the initialization process, the image processing unit 20 determines whether or not at least one indefinite region exists in the target image based on the correspondence map created in step S220 (step S230). The indefinite area here is an area that may be formed by correcting the distortion of the captured image captured by the imaging unit 10 via the wide-angle lens and specifying the area corresponding to the target image. As shown in FIG. 4A, the indefinite region here is a region that appears in a semicircular shape at the periphery of the target image, and is a region where no image is formed in the target image. This indefinite area is generally painted black when displayed on the display unit 30.

Then, as a result of the determination in step S230, if there is no undefined region in the target image based on the correspondence map (step S230: NO), the image processing unit 20 ends the initialization process and generates an image. The image display process is shifted to the process.

On the other hand, if at least one indefinite region is formed in the target image based on the correspondence map as a result of the determination in step S230 (step S230: YES), the image processing unit 20 moves the initialization process to step S240. Let In step S240, the image processing unit 20 determines whether or not a plurality of indefinite areas are formed in the target image based on the correspondence map.

As a result of the determination in step S240, when a plurality of indefinite regions are formed in the target image based on the correspondence map (step S240: YES), the image processing unit 20 performs initialization processing to step S270 described later in detail. Transition. On the other hand, if the result of determination in step S240 is that there is a single indefinite region formed in the target image based on the correspondence map (step S240: NO), the image processing unit 20 moves the initialization process to step S250. .

In step S250, the image processing unit 20 determines a mask area that covers an indefinite area formed in the target image based on the correspondence map. As shown in FIG. 4B, the mask area referred to here is a rectangular area that covers the edge of the target image where an indefinite area formed in the target image based on the correspondence map exists. Specifically, in step S250, a straight line (that is, a tangent) circumscribing an indefinite region formed in the target image based on the correspondence map is obtained. This tangent is a straight line that is orthogonal to a side that is not parallel to the side of the target image from which the indefinite region does not protrude and that circumscribes the indefinite region.

In step S250, the image processing unit 20 sets the obtained tangent as one of the sides, and determines a rectangular area covering the indefinite area formed in the target image based on the correspondence map as a mask area.

Subsequently, in the initialization process, the image processing unit 20 stores the mask data indicating the position and size of the mask area determined in step S250 in the storage unit 14 (step S260). By this step S260, in the image generation process, the pixel corresponding to the mask area in the target image based on the correspondence map is displayed on the display unit 30 in the prescribed color. Thereafter, the image processing unit 20 ends the initialization process, and shifts the image display process to the image generation process.

By the way, as a result of the determination in step S240, in step S270, where a plurality of indefinite regions are formed in the target image based on the correspondence map (step S240: YES), the image processing unit 20 determines the specific image in the target image. Deriving a region.

In the case where a plurality of indefinite areas are formed in the target image based on the correspondence map, as shown in FIG. 5A, the indefinite areas are formed in either the upper part or the left and right in the target image based on the correspondence map. It may be a case where an indefinite region is formed in both the upper part and the lower part. That is, the case where a plurality of indefinite areas are formed in the target image based on the correspondence map may be any case as long as a plurality of indefinite areas are formed.

Also, the specific image area is a rectangular area inscribed in each of a plurality of indefinite areas in the target image.

In step S270 of the present embodiment, as illustrated in FIG. 5B, the image processing unit 20 specifies a rectangular area inscribed in each of the plurality of indefinite areas in the target image as the specific image area. At the same time, the image processing unit 20 calculates a height H ′ that is the vertical size in the specific image region and a width W ′ that is the horizontal size in the specific image region. The height H ′ in the specific image area and the width W ′ in the specific image area may be the number of pixels in the specific image area.

Subsequently, in the initialization process, the image processing unit 20 calculates a horizontal direction ratio (W / W ′) and a vertical direction ratio (H / H ′) (step S280). The horizontal direction ratio (W / W ′) referred to here is a ratio of the width W ′ that is the size in the horizontal direction in the specific image region having the width W that is the size in the horizontal direction of the target image. Further, the vertical direction ratio (H / H ′) referred to here is a ratio of the height H that is the size in the vertical direction of the specific image region to the height H that is the size in the vertical direction of the target image.

Further, in the initialization process, the image processing unit 20 determines whether or not the horizontal direction ratio (W / W ′) is larger than the vertical direction ratio (H / H ′) (step S290). If the result of determination in step S290 is that the horizontal ratio (W / W ′) is greater than the vertical ratio (H / H ′) (step S290: YES), the image processing unit 20 enlarges the specific image area. The magnification to be set is set to the vertical ratio (H / H ′) (step S300). Thereafter, the image processing unit 20 shifts the initialization process to step S320, which will be described in detail later.

On the other hand, if the result of determination in step S290 is that the horizontal ratio (W / W ′) is less than or equal to the vertical ratio (H / H ′) (step S290: NO), the image processing unit 20 selects a specific image area. The magnification to be enlarged is set to the horizontal ratio (W / W ′) (step S310). Thereafter, the image processing unit 20 shifts the initialization process to step S320.

In step S320, the image processing unit 20 determines a mask area that covers each indefinite area formed in the target image based on the correspondence map. Specifically, in step S320, the specific image area is enlarged according to the vertical ratio (H / H ′) set in step S300 or the horizontal ratio (W / W ′) set in step S310. Then, the outside of the enlarged specific image area is determined as a mask area. The enlargement of the specific image area is set so that the area center of the specific image area coincides with the display center of the display area of the display unit 30.

Further, the outside of the specific image area referred to here is an area in the display area of the display unit 30 where no pixel corresponding to the enlarged specific image area exists.

Subsequently, in the initialization process, the image processing unit 20 stores mask data representing the position and size of the mask area determined in step S320 in the storage unit 14 (step S330). By this step S330, in the image generation process, the pixel corresponding to the mask area in the target image based on the correspondence map is displayed on the display unit 30 in the prescribed color. Note that black is considered as an example of the prescribed color.

After that, the image processing unit 20 ends the initialization process, and shifts the image display process to the image generation process.

<Image generation processing>
Next, the image generation process activated in step S120 of the image display process will be described.

When this image generation processing is started, the image processing unit 20 first acquires a correspondence map and mask data from the storage unit 14 as shown in FIG. 6 (step S410).

Subsequently, the image processing unit 20 determines whether or not a mask area exists in the target image based on the correspondence map based on the correspondence map and the mask data acquired in Step S410 (Step S420). In step S420, the image processing unit 20 may determine that a mask area exists if the mask data has been acquired.

As a result of the determination in step S420, if there is no mask area in the target image based on the correspondence map (step S420: NO), the image processing unit 20 shifts the image generation processing to step S490 described later in detail. .

On the other hand, if the result of determination in step S420 is that there is a mask area in the target image based on the correspondence map (step S420: YES), the image processing unit 20 shifts the image generation processing to step S430. In step S <b> 430, the image processing unit 20 acquires peripheral information representing a situation around the host vehicle based on the captured image from the imaging unit 10. This surrounding information includes whether or not there is a person around the host vehicle, the position of the person when the person is present, and the like.

In step S430 of the present embodiment, the image processing unit 20 may acquire peripheral information by performing image processing on the captured image. As an example of the image processing referred to here, well-known template matching for specifying the presence or absence of a person and the position of the person when the person exists can be considered.

In the image generation process, the image processing unit 20 determines whether or not there is a person at a position corresponding to the mask area of the target image based on the peripheral information acquired in step S430 (step S440).

If the result of determination in step S440 is that there is a person at a position corresponding to the mask area of the target image (step S440: YES), the image processing unit 20 increases the first count value by one (step S440). S450). The first count value referred to here is a first count that counts the presence of a person at a position corresponding to the mask area of the target image based on the captured image captured by the imaging unit 10 during a preset time. This is the counter value. After that, the image processing unit 20 shifts the image generation processing to step S460 described later in detail.

On the other hand, if the result of determination in step S440 is that there is no person at the position corresponding to the mask area of the target image (step S440: NO), the image processing unit 20 does not execute step S450, but executes step S460. Migrate to

In step S460, the image processing unit 20 determines whether or not the first count value is equal to or greater than a first threshold value defined in advance. The first threshold value is defined as a predetermined number of captured images, and is smaller than the number of captured images that can be captured at a predetermined time (that is, an upper limit value of the first count value). Is a small number). That is, in step S460, it is determined whether or not it can be considered that a human being reflected in a captured image area corresponding to the mask area is a continuous number of predetermined captured images.

As a result of the determination in step S460, if the first count value is less than the first threshold value (step S460: NO), the image processing unit 20 shifts the image generation processing to step S480 described later in detail. On the other hand, if the result of determination in step S460 is that the first count value is greater than or equal to the first threshold value (step S460: YES), the image processing unit 20 masks the target image in the mask area corresponding to the position where the person exists. Is set to be canceled (step S470). The masking referred to here is a process of painting the mask area with a prescribed color. The specified color referred to here is, for example, black.

That is, in S440 to S470 of the image generation process, as shown in FIG. 7A, based on the captured image captured by the imaging unit 10, it is determined whether or not a person is present at a position corresponding to the mask area. This is determined each time a captured image is captured. When it is determined that a person is present at a position corresponding to the mask area continuously for a predetermined number of photographed images, masking for the mask area corresponding to the position where the person exists is cancelled. Set as follows. After that, the image processing unit 20 shifts the image generation processing to step S500 described in detail later.

Note that, as a result of the determination in step S460, in step S480, where the first count value is less than the first threshold (step S460: NO), the image processing unit 20 seems to maintain masking in the mask region. Keep the setting on. Thereafter, the image processing unit 20 shifts the image generation processing to step S500.

By the way, as a result of the determination in step S420, in step S490, where the mask area does not exist in the target image based on the correspondence map (step S420: NO), the image processing unit 20 initializes the first count value. . The initialization is to return to the initial value, and the initial value is “0”, for example.

The image processing unit 20 then shifts the image generation process to step S500. In step S500, the image processing unit 20 determines whether or not a mask region (hereinafter referred to as “upper mask”) exists above the target image based on the correspondence map (step S500).

If the result of determination in step S500 is that there is no upper mask (step S500: NO), the image generation processing is shifted to step S570 described later in detail. On the other hand, if the result of determination in step S500 is that there is an upper mask (step S500: YES), the image processing unit 20 shifts the image generation processing to S510.

In step S510, the image processing unit 20 is a color that is information relating to the color of each pixel constituting the indefinite surrounding area set around the indeterminate area included in the upper mask based on the captured image from the imaging unit 10. Get information. The indefinite surrounding area here is an area adjacent to the indefinite area included in the upper mask in the upper mask, as shown in FIG. The color information referred to here includes hue, brightness, and saturation. As an example of the color information, luminance values in the three primary colors of “red”, “blue”, and “green” can be considered.

In the image generation process, the image processing unit 20 determines whether or not the color of the indefinite surrounding area is uniform based on the color information acquired in Step S510 (Step S520). In this step S520, the distribution of color information is obtained, and if the distribution of the color information is within a specified range defined as a range where the color can be regarded as uniform, the color of the indefinite surrounding region is uniform. Can be determined. Note that the method of determining whether or not the color of the indefinite surrounding region is uniform is not limited to this, and other methods may be used.

If the result of determination in step S520 is that the color of the indefinite surrounding area is uniform (Sstep 520: YES), the image processing unit 20 increases the second count value by one (step S530). The second count value here is a value of a second counter that counts that the color of the indefinite surrounding area is uniform based on the captured image captured by the imaging unit 10 during a preset time. . Thereafter, the image processing unit 20 shifts the image generation processing to step S540 described later in detail.

On the other hand, if the result of determination in step S520 is that the color of the indefinite surrounding area is not uniform (step S520: NO), the image processing unit 20 proceeds to step S540 without executing step S530. .

In step S540, the image processing unit 20 determines whether or not the second count value is equal to or greater than a second threshold value defined in advance (step S540). The second threshold value is defined as a predetermined number of photographed images, and is smaller than the number of photographed images that can be photographed at a predefined time (ie, an upper limit value of the second count value). Is a small number). That is, in step S540, it is determined whether or not it can be considered that a predetermined number of photographed images are continuous if the color in the indefinite surrounding region is uniform.

If the result of determination in step S540 is that the second count value is less than the second threshold value (step S540: NO), the image processing unit 20 shifts the image generation processing to step S560 described in detail later. On the other hand, if the result of determination in step S540 is that the second count value is greater than or equal to the second threshold value (step S540: YES), the image processing unit 20 releases the upper mask and the indeterminate covered by the upper mask. The area is displayed in a specific color which is a color determined to be uniform (step S550). That is, when “sky” is reflected in the indefinite surrounding area, the indefinite area is displayed in a specific color (for example, blue) that is uniform with the sky.

The image processing unit 20 then shifts the image generation processing to step S580. In step S560, the process proceeds to the case where the second count value is less than the second threshold (step S540: NO), the image processing unit 20 maintains the setting so that masking in the upper mask is maintained. Thereafter, the image processing unit 20 shifts the image generation processing to step S580.

By the way, as a result of the determination in step S500, in step S570, where the upper mask does not exist (S500: NO), the image processing unit 20 initializes the second count value. Thereafter, the image processing unit 20 shifts the image generation processing to step S580. In step S580, the image processing unit 20 generates a target image according to the correspondence map acquired in step S410 and each content set in the main image generation process.

Specifically, in step S580, when there is a single indefinite area, the image processing unit 20 generates an image subjected to masking in which the mask area is filled with a specified color as a new target image. When there are a plurality of indefinite areas, the image processing unit 20 masks the outside of the image obtained by enlarging the specific image area inscribed in each of the plurality of indefinite areas at a smaller ratio of the vertical ratio and the horizontal ratio. An image subjected to masking as a region is generated as a new target image. In this case, the enlargement of the specific image area is performed by matching the area center of the specific image area with the display center of the display area.

In step S580, when a person is reflected in the mask area, the image processing unit 20 generates a target image in which masking for the mask area is canceled. In step S580, if the image processing unit 20 represents that the color in the indefinite surrounding area is uniform, the image processing unit 20 cancels the mask area for the indefinite area corresponding to the indefinite surrounding area. At the same time, the image processing unit 20 generates, as a target image, an image in which an indefinite area corresponding to the indeterminate surrounding area is filled with a specific color that is determined to be uniform.

On the other hand, if it does not indicate that the color in the indefinite surrounding area is uniform, the image processing unit 20 generates, as a target image, an image in which the indefinite area corresponding to the indefinite surrounding area is masked.

In subsequent step S590, the image processing unit 20 outputs an image signal representing the target image generated in step S580 to the display unit 30. The display unit 30 that has acquired the image signal displays the target image generated in step S580 of the image generation process. Thereafter, the image processing unit 20 ends the main image generation process, and returns the process to step S110 of the image display process.

[Effect of the embodiment]
As described above, according to the image display processing according to the present embodiment, the presence / absence of an indefinite region in the target image is determined based on the attitude and image height data in which the imaging unit 10 is installed. If an area exists, a mask area that covers the indefinite area can be set. This mask area setting is automatically performed by the image processing unit 20 by image display processing, so that it is not necessary to perform an experiment for each installation posture of the imaging unit 10, thereby reducing labor.

That is, according to the image display device 1, it is possible to reduce the labor for obtaining the area to be masked while masking the indefinite area in the image displayed on the display unit 30. In particular, in the image display process, when there are a plurality of indefinite areas, the target image does not include the indefinite area and the maximum range in the target image is set as the specific image area. According to the image display process, since the area outside the specific image area is set as the mask area, an optimal mask area corresponding to the captured image can be set.

Further, in the image display processing, a specific image area is enlarged at a smaller ratio of the vertical ratio and the horizontal ratio, and an image subjected to masking using the outside of the enlarged specific image area as a mask area is a new target. It is generated as an image. According to such an image generation device, it is possible to enlarge a specific image area and display all images formed in the specific image area. As a result, according to the image display device 1, it is possible to display the widest possible range with high visibility for the image reflected in the image captured by the imaging unit 10.

Also, in the image display process, when the target image is displayed, the display center of the display area is matched with the image center of the target image (that is, the specific image area) displayed on the display unit. As a result, according to the image display process, the image displayed on the display unit 30 can be made less uncomfortable.

Further, in the image display processing, if the color in the indefinite surrounding area is uniform, the mask area set in the indefinite area corresponding to the indefinite surrounding area is canceled and the indefinite area is made uniform with the color of the indefinite surrounding area. Displayed in various colors. Thereby, in the image display apparatus 1, for example, when an indefinite area exists in the upper part of the image and “sky” is reflected in the upper part, the indefinite area can be displayed in a uniform color with the sky. . As a result, according to the image display device 1, the visibility of the image displayed on the display unit 30 can be improved.

The image display process indicates that the color in the indefinite surrounding area is uniform, but the indefinite area corresponding to the indefinite surrounding area is displayed in a specific color when a predetermined number of captured images are continuous. indicate. Therefore, in the image display device 1, when the color in the indefinite surrounding area becomes uniform, it is possible to avoid displaying the indefinite area corresponding to the indefinite surrounding area in a uniform color. As a result, according to the image display device 1, it is possible to prevent the indeterminate area from being masked and being displayed in a color in the indeterminate surrounding area from being switched each time a captured image is captured by the imaging unit 10. .

In the image display process, if a person is reflected in the mask area, the masking execution in the mask area is cancelled. For this reason, according to the image display process, in the target image, the person reflected in the image can be reliably displayed on the display unit 30. Thereby, the user who visually recognizes the image displayed on the display unit 30 can recognize that there is a person around the own vehicle.

In addition, in the image display process, the masking execution cancellation in the mask area where the person is reflected is performed when the person is reflected and the predetermined number of photographed images are consecutive. For this reason, according to the image display process, it is possible to avoid that the masking is released when a person appears in the mask area. As a result, according to the image display device 1, it is possible to prevent the undefined area from being masked and the masking being released from switching each time an image is captured by the imaging unit 10.

[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

(1) In step S320 of the initialization process, the specific image area is enlarged and the outside of the enlarged specific image area is used as a mask area. However, in step S320, the specific image area is maintained without being enlarged. The area outside the mask area in which the size is maintained may be determined as the mask area.

(2) In step S460 of the image generation process, it is determined whether the first count value is greater than or equal to the first threshold value, so that a person is reflected in the captured image area corresponding to the mask area. It has been determined whether or not a predetermined number of photographed images can be regarded as being continuous. However, in step S460, a method of determining whether or not it can be considered that a human being reflected in the area of the captured image corresponding to the mask area is considered to be a predetermined number of consecutive captured images. It is not limited to. In other words, any method can be used as long as it is possible to determine whether or not it can be considered that a person is reflected in the area of the photographed image corresponding to the mask area as a continuous number of photographed images. It may be.

Furthermore, in the above-described embodiment, when it can be considered that a person appears in the captured image area corresponding to the mask area is a predetermined number of consecutive captured images, in step S470. Although the masking execution has been cancelled, the condition for canceling the masking execution is not limited to this. For example, when it is determined that a person is reflected in the area of the captured image corresponding to the mask area in one captured image, the execution of masking may be canceled immediately.

(3) In step S540 of the image generation process, it is determined whether or not the second count value is equal to or larger than the second threshold value. It was determined whether or not the images can be regarded as continuous. However, the method of determining in step S540 whether or not the color in the indeterminate surrounding region is uniform can be regarded as a predetermined number of continuously captured images is not limited to this. That is, any method may be used as long as it is possible to determine whether or not a uniform color in the indefinite surrounding area can be regarded as a pre-defined number of captured images being continuous.

Furthermore, in the above-described embodiment, when the color in the indefinite surrounding region is uniform, it can be considered that the predetermined number of photographed images are continuous, and the upper mask is uniform in step S550. Although a process of displaying in an observable color (hereinafter referred to as a color correction process) has been executed, the condition for executing this color correction process is not limited to this. For example, when it is determined that the color in the indefinite surrounding area is uniform in one photographed image, the color correction process may be executed immediately.

(4) In the above embodiment, in the upper mask, the area adjacent to the indefinite area included in the upper mask is set as the indefinite surrounding area, but the indefinite surrounding area is not limited to this. That is, the indefinite surrounding area may be an area adjacent to the indefinite area formed on the left and right of the target image, or may be an area adjacent to the indefinite area formed in the lower part of the target image.

For example, in the former case, when “buildings” are reflected in the indefinite area and the indeterminate surrounding area, the indefinite area can be displayed in a uniform color with the building. As a result, according to the image display device 1, the visibility of the image displayed on the display unit 30 can be improved.

(5) By the way, in the image display process in the above embodiment, the initialization process is executed every time the image display process is started. In the image display process, the image display process is started first. The initialization process may be executed at the timing, and when the image display process is activated for the second time or later, the initialization process may be omitted.

In addition, the aspect which abbreviate | omitted a part of structure of the said embodiment is also embodiment of this invention. Further, an aspect configured by appropriately combining the above embodiment and the modification is also an embodiment of the present invention. Moreover, all the aspects which can be considered in the limit which does not deviate from the essence of the invention specified by the wording described in the claims are the embodiments of the present invention.

For example, in the above-described embodiment, the image processing unit 20 which is an example of an image generation apparatus stores a program in a ROM which is a semiconductor memory corresponding to a non-transitional tangible recording medium, and this program corresponds to a computer processor. Each function of the image generation apparatus is realized by executing the CPU, but the program is stored in a non-transitional tangible recording medium other than the ROM (for example, non-volatile memory other than the ROM). The configuration may be executed by the processor. In this case, in the image processing unit 20, a program stored in a non-transitional tangible recording medium such as a nonvolatile memory is executed by the processor, so that a method corresponding to the program (for example, an image generation procedure, an image display, etc. An image generation method having a procedure, an indeterminate region determination procedure, and an image mask setting procedure may be executed.

Also, some or all of the functions of the image processing unit 20 may be configured by hardware using one or a plurality of integrated circuits (ie, ICs). Further, each function of the image processing unit 20 is provided by software recorded on a non-transitional tangible recording medium such as a nonvolatile memory and a computer that executes the software, or only software, only hardware, or a combination thereof. May be.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims (11)

1. An imaging unit (10) that captures an image through a wide-angle lens, and corrects distortion of a captured image captured by an imaging unit installed in the vehicle so as to capture a predetermined area around the vehicle. An image generation unit (20, S580) that performs distortion correction and generates a target image that is the maximum range of an image to be displayed;
   An image display unit (20, S590) for displaying the target image generated by the image generation unit in a display area of the display unit;
   At least one indefinite region in which no image is formed in the target image based on image height data representing the distance from the imaging center of the imaging unit to each pixel corresponding to each image point and the attitude at which the imaging unit is installed. An indeterminate area determination unit (20, S230, S240) that determines whether there are two,
   As a result of the determination by the indeterminate area determination unit, if there is at least one indeterminate area, the target is a rectangular area having a straight line circumscribing the at least one indeterminate area, and the indefinite area exists. A mask setting unit (20, S250, S320) for setting a mask region covering the edge of the image,
   The image generation unit (20) generates, as the new target image, an image in which the target image is subjected to masking that fills the mask area set by the mask setting unit with a specified color. .
2. The mask setting unit specifies an image specification that specifies a specific image region that is a rectangular region inscribed in each of the indefinite regions in the target image when there are a plurality of the indefinite regions as a result of the determination by the indefinite region determination unit. The image generation apparatus according to claim 1, further comprising: a unit (20, S270), wherein an area outside the specific image area specified by the image specifying unit is set as a mask area.
3. The image generation unit includes a vertical ratio that is a ratio of the size of the target image in the vertical direction to the size of the specific image region in the vertical direction, and the size of the specific image region in the horizontal direction of the target image. An enlargement unit (20, S320) for enlarging the specific image region at a small ratio in a horizontal ratio that is a ratio of sizes in the horizontal direction, and the mask region outside the specific image region enlarged by the enlargement unit The image generation apparatus according to claim 2, wherein the image subjected to the masking is generated as a new target image.
4. The image generating apparatus according to claim 3, wherein the enlargement unit enlarges the specific image region by matching a region center of the specific image region with a display center of the display region.
5. The image display unit
   A color information acquisition unit (20, S510) that acquires color information that is information relating to the color of the indefinite surrounding area set around the indefinite area;
   A color determination unit (20, S520) for determining whether or not the color information acquired by the color information acquisition unit indicates that the color in the indefinite surrounding region is uniform;
   As a result of the determination by the color determination unit, if the color information indicates that the color in the indefinite surrounding region is uniform, the mask setting unit sets the indefinite region corresponding to the indefinite surrounding region. And a color changing unit (20, S550) for canceling the mask area and displaying the indefinite area corresponding to the indefinite surrounding area with the specific color that is determined to be uniform. The image generation apparatus according to claim 4.
6. The color information acquisition unit acquires color information of the indeterminate surrounding region every time a captured image is captured by the imaging unit,
   The color determination unit determines whether or not the color information is uniform in the indefinite surrounding region every time color information is acquired by the color information acquisition unit,
   As a result of the determination by the color determination unit, the image display unit indicates that the color information indicates that the color in the indefinite surrounding region is uniform, and a predetermined number of photographed images are consecutive. The image generation apparatus according to claim 5, wherein when it can be regarded as a thing, an indefinite area corresponding to the indefinite surrounding area is displayed in the specific color.
7. The image display unit generates an indeterminate area corresponding to the indefinite surrounding area if the color information does not indicate that the color in the indeterminate surrounding area is uniform as a result of the determination in the color determining unit. The image generation apparatus according to claim 5, wherein the image generation apparatus displays the image in a state where the image is masked in the unit.
8. The image display unit
   A person determination unit (20, S440) for determining whether a person is reflected in the mask area set by the mask setting unit;
   8. A release unit (20, S 470) that cancels masking for the mask area if a person is reflected in the mask area as a result of the determination by the person determination unit. 8. The image generation apparatus as described in any one of Claims.
9. The person determination unit determines whether or not a person is reflected in the mask area each time a captured image is captured by the imaging unit.
   The canceling unit cancels the masking for the mask area when it is determined that the person is reflected as a result of the determination by the person determining unit as a pre-defined number of captured images. The image generation apparatus according to claim 8.
10. An imaging unit (10) that captures an image through a wide-angle lens, and corrects distortion of a captured image captured by an imaging unit installed in the vehicle so as to capture a predetermined area around the vehicle. An image generation procedure (S580) for performing distortion correction and generating a target image that is the maximum range of the image to be displayed;
    An image display procedure (S590) for displaying the target image generated in the image generation procedure in the display area of the display unit;
    At least one indefinite region in which no image is formed in the target image based on image height data representing the distance from the imaging center of the imaging unit to each pixel corresponding to each image point and the attitude at which the imaging unit is installed. Undefined area determination procedure (S230, S240) for determining whether or not there are two,
    As a result of the determination in the indefinite area determination procedure, when there is at least one indefinite area, the target is a rectangular area having a straight line circumscribing the at least one indeterminate area, and the indefinite area exists. A mask setting procedure (S250, S320) for setting a mask area covering the edge of the image,
    The image generation procedure generates, as the new target image, an image obtained by performing masking for painting the mask area set by the mask setting procedure with a specified color on the target image.
11. A program executed by a computer mounted on a vehicle,
    An imaging unit (10) that captures an image through a wide-angle lens, and corrects distortion of a captured image captured by an imaging unit installed in the vehicle so as to capture a predetermined area around the vehicle. An image generation procedure (S580) for performing distortion correction and generating a target image that is the maximum range of the image to be displayed;
    An image display procedure (S590) for displaying the target image generated in the image generation procedure in the display area of the display unit;
    At least one indefinite region in which no image is formed in the target image based on image height data representing the distance from the imaging center of the imaging unit to each pixel corresponding to each image point and the attitude at which the imaging unit is installed. Undefined area determination procedure (S230, S240) for determining whether or not there are two,
    As a result of the determination in the indefinite area determination procedure, when there is at least one indefinite area, the target is a rectangular area having a straight line circumscribing the at least one indeterminate area, and the indefinite area exists. Causing the computer to execute a mask setting procedure (S250, S320) for setting a mask region covering the edge of the image;
    In the image generation procedure, a program that generates, as the new target image, an image that is masked to fill the mask area set in the mask setting procedure with a specified color with respect to the target image.

Images