JP7467000B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method, and is particularly suitable for use in an image processing device that synthesizes and displays images captured by a side camera and a rear camera, and in an image processing method using the image processing device.

Conventionally, there is known an image processing device that displays video based on images captured by a side camera installed in a vehicle, and that superimposes video based on images captured by a rear camera in areas of the video that are blind spots caused by the vehicle body (hereinafter referred to as "blind spot areas"), providing a video that makes it appear as if the rear of the vehicle is being filmed through the vehicle body (see, for example, Patent Document 1).

JP 2015-74436 A

In conventional image processing devices, video based on images captured by the rear camera is displayed in the blind spot area, meaning that no side camera images are provided for the blind spot area. This causes a problem in that even if an object that the user should recognize is captured by the side camera in the blind spot area, the user cannot recognize the object from the video provided by the image processing device.

The present invention was made to solve such problems, and aims to enable the user to recognize specific objects captured by the side camera in the blind spot area from the video images provided.

In order to solve the above problems, the present invention provides a composite video in which a region of a video based on an image captured by a side camera that corresponds to a blind spot of the side camera is supplemented with a video based on an image captured by a rear camera, and the composite video is made such that a portion of the blind spot region of the composite video is partially visible to the video based on the image captured by the side camera, or when a predetermined condition is met, a portion or all of the blind spot region is temporarily made visible to the video based on the image captured by the side camera.

According to the present invention configured as described above, instead of a state in which a video based on an image captured by the rear camera is displayed throughout the entire blind spot area without exception, it is possible to make a video based on an image captured by the side camera visible partially or temporarily in the blind spot area, so that a specific object captured by the side camera in the blind spot area can be included in the video, and the user can recognize the specific object based on the video provided.

1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment of the present invention; 4 is a diagram showing the installation positions and imaging ranges of a rear camera and a right side camera. FIG. FIG. 13 is a diagram showing an example of a rear shot image. FIG. 13 is a diagram showing an example of a right side photographed image. FIG. 13 is a diagram showing an example of a right-side composite image in a normal mode. FIG. 13 is a diagram showing an example of a right-side composite image in a special mode. 5 is a flowchart showing an example of the operation of an image processing unit according to the first embodiment of the present invention. FIG. 13 is a diagram showing an example of an image displayed on a right monitor. FIG. 11 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a second embodiment of the present invention.

An embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the functional configuration of an image processing device 1 according to this embodiment. The image processing device 1 is a device capable of displaying a scene equivalent to the scene that can be seen when the user looks at the right side mirror MR or the left side mirror ML (FIG. 2) of the vehicle, including parts that cannot be seen when looking at these mirrors because they are blocked by the vehicle body. Hereinafter, a vehicle equipped with the image processing device 1 will be referred to as the "own vehicle".

As shown in FIG. 1, the image processing device 1 is connected to a right monitor 2R, a left monitor 2L, a rear camera 3, a right side camera 4R, and a left side camera 4L. The right monitor 2R is a display panel (e.g., a liquid crystal display panel, an organic EL display panel, etc.) provided at the right end of the dashboard of the vehicle (at a position inside the vehicle close to the right side mirror MR). The left monitor 2L is a display panel provided at the left end of the dashboard of the vehicle. The right monitor 2R and the left monitor 2L correspond to the "display device" in the claims. Hereinafter, when there is no need to distinguish between the right monitor 2R and the left monitor 2L, they are expressed as "monitor 2". Of course, the monitor 2 can be seen by people other than the driver, but for convenience of explanation, in this embodiment, it is the driver who sees the monitor 2. The rear camera 3 is an imaging device provided at the rear of the vehicle. The right side camera 4R is an imaging device provided on the right side mirror MR of the right side door of the front seat of the vehicle. The left side camera 4L is a photographing device mounted on the left side mirror ML of the front left door of the vehicle.

Figure 2 is a diagram for explaining the installation positions and shooting ranges of the rear camera 3 and the right side camera 4R. In the following explanation, when there is no need to distinguish between the right side camera 4R and the left side camera 4L, they will be referred to as "side cameras 4." In addition, in the following explanation, when the terms "front, back, left, and right" are used in relation to the vehicle body, they will follow the front, back, left, and right indicated by the arrows in Figure 2.

As shown in FIG. 2, the rear camera 3 is provided at the rear end of the vehicle, in the center of the vehicle's width direction, and captures images with its optical axis facing toward the rear of the vehicle. In FIG. 2, the symbol HB shows a schematic representation of the capturing range of the rear camera 3. Hereinafter, the capturing range of the rear camera 3 will be referred to as the "rear camera capturing range HB." The rear camera 3 is composed of a fisheye camera with a fisheye lens. The rear camera 3 captures images at a predetermined interval, generates captured images, and outputs them. Hereinafter, the captured images generated and output by the rear camera 3 will be referred to as "rear captured images." FIG. 3 shows an example of a rear captured image.

As shown in FIG. 2, the right side camera 4R captures images with its optical axis oriented so that it is slightly tilted toward the right as it moves rearward relative to the rear of the vehicle. In FIG. 2, the symbol HR shows a schematic representation of the image capturing range of the right side camera 4R. The right side camera 4R is an image capturing device that captures images using a standard lens, and the horizontal angle of view of its image capturing range is smaller than the horizontal angle of view of the rear camera image capturing range HB. Hereinafter, the image capturing range of the right side camera 4R will be referred to as the "right side camera image capturing range HR."

As shown in FIG. 2, the right side camera 4R's right side camera shooting range HR forms a blind spot range DH that is blocked by the vehicle's body (corresponding to the "range that is a blind spot caused by the vehicle body" in the claims). The vehicle body acts as an obstruction in the blind spot range DH, and the scenery is not captured. The rear camera 3's rear camera shooting range HB includes the blind spot range DH. In other words, the rear camera 3 can capture scenery in the blind spot range DH. The right side camera 4R captures images at a predetermined interval, generates captured images, and outputs them. Hereinafter, the captured images generated and output by the right side camera 4R are referred to as "right side captured images."

Figure 4 shows an example of a right side image generated by the right side camera 4R in the same environment as the environment when the rear image of Figure 3 was generated. As shown in Figure 4, a blind spot area DR in which an image of the vehicle's body is recorded is formed on the left side of the right side camera image. The blind spot area DR is an area corresponding to the blind spot area DH (Figure 2) of the right side camera shooting range HR of the right side camera 4R. Since the right side camera 4R is fixedly installed at a specific position of the vehicle, the appearance (position, shape) of the blind spot area DR in the entire area of the right side camera shooting image is constant. As shown in Figure 4, a fuel filler 5 is provided at a predetermined position on the right side of the vehicle body of the vehicle in this embodiment. The fuel filler 5 is used to explain the second embodiment.

The left side camera 4L is a photographing device with the same capabilities as the right side camera 4R, and is installed in the same manner as the right side camera 4R on the left side mirror ML attached to the left side door of the front seat. The left side camera 4L takes photographs at a predetermined cycle, generates photographed images, and outputs them. Hereinafter, the photographed images generated and output by the left side camera 4L will be referred to as "left side photographed images", and the right side photographed images and left side photographed images will be collectively referred to as "side photographed images".

As described above, the side camera 4 is installed on the side of the vehicle, captures images toward the rear of the vehicle with a part of the side of the vehicle's body included in its capture range, and outputs a side capture image based on the capture results. The rear camera 3 is installed on the rear of the vehicle, and can capture at least the range of the side camera 4's capture range that is a blind spot due to the vehicle body, and outputs a rear capture image based on the capture results.

As shown in FIG. 2, the vehicle according to this embodiment has side doors on the left and right sides of the front and rear, and each side door has a window 6.

As shown in FIG. 1, the image processing device 1 has, as its functional configuration, an image acquisition unit 10, a protrusion detection unit 11, and an image processing unit 12. Each of the above-mentioned functional blocks 10 to 12 can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each of the above-mentioned functional blocks 10 to 12 is actually configured with a computer's CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as the RAM, ROM, hard disk, or semiconductor memory.

The image acquisition unit 10 inputs and acquires rear-view images from the rear camera 3 at a predetermined cycle. The image acquisition unit 10 also inputs and acquires right-side images from the right-side camera 4R at a predetermined cycle. The image acquisition unit 10 also inputs and acquires left-side images from the left-side camera 4L at a predetermined cycle. The image acquisition unit 10 outputs the right-side and left-side images to the protrusion detection unit 11 at a predetermined cycle, and outputs the rear, right-side, and left-side images to the image processing unit 12 at a predetermined cycle.

The protrusion detection unit 11 inputs the right side and left side captured images from the image acquisition unit 10 at a predetermined cycle, and detects whether an object is protruding from the window 6 of the side door (hereinafter referred to as a "protruding state") based on the input right side and left side captured images. Note that "an object protruding from the window 6" means that some object is protruding from inside the vehicle to outside the vehicle through the window 6. Below, the process executed by the protrusion detection unit 11 for the window 6 formed on the right rear side door (hereinafter referred to as the "right rear window 6") will be described.

The protrusion detection unit 11 continuously monitors whether the right rear window 6 is open or not. The right rear window 6 being open means that the window glass of the right rear window 6 is open. The protrusion detection unit 11 monitors whether the right rear window 6 is open or not by communicating with a vehicle system that monitors the status of various mechanisms of the vehicle. However, any method of monitoring may be used, and as an example, the protrusion detection unit 11 may be configured to monitor based on input from an optical or mechanical sensor that outputs a detection value that differs depending on the status of the right rear window 6.

When the right rear window 6 is not open, no object protrudes from the right rear window 6, and the protrusion detection unit 11 determines that there is no protrusion. On the other hand, when the right rear window 6 is open, the protrusion detection unit 11 analyzes the right side captured images input at a predetermined interval and determines whether or not an object has protruded from the right rear window 6. Note that objects that protrude from the right rear window 6 may be parts of the body of a passenger in the rear seat of the vehicle (such as a face or hand), or an object held by a passenger.

When determining whether an object has protruded from the right rear window 6, the protrusion detection unit 11 executes the following process. That is, a three-dimensional coordinate system is defined in advance for the space shown in the right side photographed image, and an object existing in the space can be defined as a three-dimensional area in this three-dimensional coordinate system. Furthermore, the three-dimensional area of the right rear window 6 in the three-dimensional coordinate system is set in advance. The protrusion detection unit 11 then continuously analyzes the right side photographed image input at a predetermined interval, and monitors whether there is an object that has protruded from inside the vehicle to outside the vehicle through the three-dimensional area of the right rear window 6. This monitoring is performed by identifying the three-dimensional area of the object in the three-dimensional coordinate system using existing object recognition technology and position recognition technology, and then comparing the three-dimensional area of the object with the three-dimensional area of the right rear window 6.

The protrusion detection unit 11 detects a protruding state when an object protrudes from the right rear window 6, and detects a non-protruding state when an object does not protrude. Usually, the period during which a protruding state is detected is a chunk of time spanning multiple periods. The above describes the processing executed by the protrusion detection unit 11 for the right rear window 6, but the protrusion detection unit 11 also executes similar processing for the other windows 6, and detects when each window 6 is in a protruding state.

The image processing unit 12 inputs the rear, right side and left side captured images from the image acquisition unit 10 at a predetermined cycle, and displays a moving image based on these images on the monitor 2. When displaying the moving image, the image processing unit 12 displays the moving image in normal mode while a protrusion state is detected by the protrusion detection unit 11, and displays the moving image in special mode while a protrusion state is not detected. The processing of the image processing unit 12 in each of the normal mode and special mode will be explained below. In the following explanation, the image processing unit 12 displaying a moving image on the monitor 2 may be expressed as "the image processing unit 12 displays a moving image."

<Normal mode>
First, the process of the image processing unit 12 in the normal mode will be described. As described above, the image processing unit 12 displays a moving image in the normal mode while the protrusion detection unit 11 does not detect a protruding state (= while an object does not protrude from the window 6). In the normal mode, the image processing unit 12 displays a composite moving image on the monitor 2, in which a blind spot area DR (FIG. 4) corresponding to the blind spot of the side camera 4 out of the area of the moving image based on the side photographed image (hereinafter referred to as "side moving image") is supplemented with a part of the moving image based on the rear photographed image (hereinafter referred to as "rear moving image"). Hereinafter, the process of the image processing unit 12 in the normal mode will be described using as an example a case where a composite moving image is displayed on the right monitor 2R based on the right side photographed image of the right side camera 4R and the rear photographed image of the rear camera 3.

The image processing unit 12 executes a composite image generation process for generating a right-side composite image based on the input rear-view image and right-side image every time the image processing unit 12 receives the input rear-view image and right-side image at a predetermined cycle. FIG. 5 shows an example of a right-side composite image generated by the composite image generation process based on the rear-view image of FIG. 3 and the right-side image of FIG. 4. For ease of explanation, a dashed line is drawn in FIG. 5 at the boundary between the blind spot area DR and other areas. Referring to FIGS. 3 to 5, the right-side composite image generated by the composite image generation process is an image in which the blind spot area DR of the side image (the right-side image in this example) is supplemented with an image based on the rear-view image (a supplementary image to be described later), and the composite image generation process is a process for generating such a right-side composite image at each cycle.

In the composite image generation process, first, the image processing unit 12 converts the input rear photographed image into a complementary image using a right-side lookup table stored in a memory unit (not shown). The right-side lookup table is a mapping table used to convert the rear photographed image into a "photographed image generated by the right-side virtual camera virtually photographing the blind spot range DH while being see-through from the vehicle body." The right-side virtual camera is a virtual photographing device with the same photographing direction and photographing range as the right side camera 4, and is conceptually used when converting a specified photographed image into the viewpoint of the right side camera 4.

The right-side lookup table has correspondence information that associates the positions of all dots that make up the blind spot DR in the virtual image captured by the right-side virtual camera with the positions of the corresponding dots in the rear-view image on a one-to-one basis. Note that the appearance (position, shape) of the blind spot DR in the entire right-side image is constant, and the positions of all dots that make up the blind spot DR in the virtual image captured by the right-side virtual camera are specified in advance.

The correspondence information of the right side lookup table is appropriately set in advance, with a view to converting the viewpoint of the rear camera 3 to the viewpoint of the right side virtual camera, reflecting the position and angle of the rear camera 3 and the right side virtual camera in the vehicle coordinate system, the specifications of the rear camera 3 (projection method, angle of view, distortion (lens distortion), resolution, etc.), the specifications of the right side virtual camera, the aspect of the projection surface used for viewpoint conversion, and other elements. The image processing unit 12 uses the right side lookup table to generate a complementary image in which the pixel values of the dots constituting the blind spot area DR of the virtual captured image of the right side virtual camera are replaced by the pixel values of the corresponding dots in the rear captured image.

After generating the complementary image, the image processing unit 12 generates mask information. The mask information is information that defines the transmittance of each of the dots that constitute the complementary image when the complementary image is superimposed on the blind spot area DR of the right side image. In this embodiment, the transmittance ranges from "0" to "1". If the pixel value of a dot in the right side image is R, the image value of a dot in the complementary image that is superimposed (blended) on the dot is H, the pixel value of the dot after superimposition is X, and the transmittance is α, then "X = (1 - α) * H + α * R" holds. As shown in the formula, when the transmittance is "1", the pixel value of the dot after superimposition is the pixel value of the corresponding dot in the right side image, and conversely, when the transmittance is "0", the pixel value of the dot after superimposition is the pixel value of the corresponding dot in the complementary image.

In normal mode, the image processing unit 12 generates mask information in which the transmittance of all dots that make up the complementary image is "0". After generating the mask information, the image processing unit 12 superimposes each dot of the complementary image on each dot that makes up the blind spot DR of the right side captured image with the corresponding transmittance based on the mask information, to generate a right side composite image. In normal mode, the transmittance of all dots is "0", so the right side composite image is an image in which the blind spot DR of the right side captured image is replaced by the complementary image (see Figure 5).

The above processing is the composite image generation processing executed at a predetermined cycle in response to the input of the right side captured image and the rear captured image. The image processing unit 12 loads the right side composite image in an image buffer such as a VRAM at a predetermined cycle and displays it on the right monitor 2R. As a result, the right monitor 2R displays a moving image (composite moving image) that reflects the capture results of the rear camera 3 and the right side camera 4R in real time, with the blind spot area DR of the moving image (side moving image) based on the right side captured image being supplemented by the moving image (rear moving image) based on the rear captured image.

The above describes the operation of the image processing unit 12 when displaying a composite moving image on the right monitor 2R in normal mode, but the image processing unit 12 also displays a composite moving image on the left monitor 2L by performing similar processing based on the left side captured image and the rear captured image.

<Special mode>
Next, a description will be given of the processing of the image processing unit 12 in the special mode. As described above, the image processing unit 12 displays a moving image in the special mode while the protrusion detection unit 11 detects a protruding state (i.e., while an object is protruding from the window 6).

In special mode, the image processing unit 12 executes the composite image generation process in a different way than in normal mode. That is, in the composite image generation process, the image processing unit 12 generates a complementary image in the same way as in normal mode, but for the mask information, the transmittance for all dots is set to "1". As a result, the composite moving image displayed on the monitor 2 shows the side moving image without the blind spot area DR being supplemented by the rear moving image. Figure 6 shows an example of an image of the composite moving image displayed on the right monitor 2R in special mode. In the composite moving image illustrated in Figure 6, a hand is sticking out from the right rear window 6.

As described above, the image processing device 1 according to this embodiment has a function of temporarily making the entire blind spot area DR visible as a video based on the side-captured image when a predetermined condition is met when displaying a composite video image. By having such a function, the state in which the rear video image is displayed in the entire blind spot area DR continues without exception, but the side video image can be temporarily made visible in the blind spot area DR, so that a predetermined object captured by the side camera 4 in the blind spot area DR can be included in the video image and the user can recognize the predetermined object based on the provided video image.

In particular, in this embodiment, when the protrusion detection unit 11 detects that an object has protruded through the window 6, the image processing device 1 displays a side video image in the blind spot region DR while the protrusion is detected. Here, it is assumed that the driver will want to be aware of an object protruding through the window 6. This is because it becomes possible to take some kind of measure against the situation in which an object is protruding through the window 6. For example, it becomes possible to warn a child sitting in the back seat when they accidentally put their hand out.

On the other hand, if a rear moving image were always displayed for the blind spot region DR of the moving image displayed on monitor 2, the driver would basically not be able to recognize that an object is protruding from the window 6 from the moving image displayed on monitor 2. However, if the image of the object protruding from the window 6 extends beyond the blind spot region DR, the image of the object will be displayed for the portion beyond the blind spot region DR. However, in such a situation, only a portion of the tip of the portion protruding from the window 6 is displayed, and a considerable amount of the object has already protruded beyond the window 6, so it is not possible to quickly and accurately recognize that an object is protruding from the window 6.

Based on this, according to the configuration of this embodiment, when an object protrudes from the window 6, the moving image displayed on the monitor 2 displays the side moving image in the blind spot area DR, i.e., the blind spot range DH captured by the side camera 4 is displayed, so the driver can quickly and accurately recognize that an object is protruding from the window 6 through the moving image displayed on the monitor 2. In particular, according to the configuration of this embodiment, the moving image is displayed on the monitor 2 in the special mode only during the period when an object is protruding from the window 6 (= the period when the protrusion state is detected by the protrusion detection unit 11), so that the moving image is not displayed in the special mode for an unnecessarily long period of time, which impairs user convenience.

Next, an example of the operation of the image processing device 1 according to this embodiment will be described using a flowchart. FIG. 7 is a flowchart showing the operation of the image processing device 1. As shown in FIG. 7, the image acquisition unit 10 of the image processing device 1 acquires a side image and a rear image (step SA1). Next, the image processing unit 12 displays on the monitor 2 a composite video image in which the blind spot area DR in the area of the side video image is supplemented with the rear video image based on the side image and the rear image acquired by the image acquisition unit 10 (step SA2). As described above, when displaying the composite video image in step SA2, the image processing unit 12 has a function of temporarily making the entire blind spot area DR visible to the side video image when the protrusion detection unit 11 detects a protrusion state (when a predetermined condition is met).

<First Modification of First Embodiment>
Next, a modified example of the first embodiment will be described. In the first embodiment, the protrusion detection unit 11 detected that an object was protruding from the window 6. On the other hand, the protrusion detection unit 11 in this modified example detects that an object is located near the open window 6, and the image processing unit 12 displays a moving image on the monitor 2 in a special mode when the protrusion detection unit 11 detects that an object is located near the open window 6. Note that the location of an object near the open window 6 can be detected by a method similar to the method for detecting that an object is protruding in the first embodiment.

Here, it is assumed that the driver will want to be aware of any situation in which an object approaches the open window 6 and may protrude, even if the object is not protruding from the window 6. This is because the driver will be able to take the necessary measures and steps in advance. One example of a situation in which an object approaches the open window 6 and may protrude is when a child sitting in the back seat brings his or her face close to the open window 6. With the configuration of the above modified example, the driver can recognize that an object is approaching the open window 6 and recognize the object through the moving image displayed on the monitor 2.

<Second Modification of First Embodiment>
Next, a second modified example of the first embodiment will be described. In the first embodiment, in the special mode, the image processing unit 12 sets the transmittance of each dot of the complementary image in the mask information to "0". In this regard, the image processing unit 12 according to this modified example sets the transmittance of each dot of the complementary image in the mask information to a transmittance (for example, "0.5") such that both the side photographed image and the complementary image are visible when the complementary image is superimposed on the blind spot area DR in the special mode. Even with the configuration of this modified example, the side moving image is visible for the blind spot area DR in the protruding state, and the same effect as the first embodiment can be achieved. Furthermore, according to this modified example, the driver can also check the contents of the rear moving image for the blind spot area DR when a moving image is displayed in the special mode.

<Third Modification of First Embodiment>
Next, a third modified example of the first embodiment will be described. In this modified example, the protruding state is detected by the protrusion detection unit 11, and when the composite moving image is displayed in the special mode, the image processing unit 12 marks the image of the object protruding from the window 6 in the composite moving image. The marking is performed, for example, by displaying an image of a rectangular frame surrounding the object, displaying an image of a line along the outer edge of the object, or displaying a mark to call attention. According to the configuration of this modified example, the driver can reliably and immediately recognize that an object is protruding from the window 6 by looking at the moving image displayed on the monitor 2.

<Fourth Modification of First Embodiment>
Next, a fourth modified example of the first embodiment will be described. In the first embodiment, in the special mode, the image processing unit 12 displays a moving image based on the side-captured image in the entire blind spot area DR. In this regard, the image processing unit 12 according to this modified example executes the following processing. The processing of the image processing unit 12 according to this modified example will be described below using the case where a composite moving image is displayed on the right monitor 2R as an example.

When the protrusion detection unit 11 detects a protruding state and a moving image is displayed in a special mode, the image processing unit 12 according to this modified example executes the following process in the composite image generation process for generating a composite image. That is, the image processing unit 12 identifies the area of the image of an object protruding from the window 6 (hereinafter referred to as a "protruding object") from the area of the right side captured image. In this embodiment, the image processing unit 12 identifies the area of the part of the protruding object that protrudes beyond the window 6 to the outside of the vehicle. However, the type of area to be identified for the protruding object is not limited to this, and for example, the part of the protruding object that protrudes outside the vehicle and the part that is continuous with this part and extends into the vehicle may be identified. Hereinafter, the area identified by the image processing unit 12 is referred to as the "protruding object area."

Then, the image processing unit 12 generates a complementary image and identifies an area of the generated complementary image that corresponds to the protruding object area. Next, the image processing unit 12 sets the transmittance of each pixel belonging to the area corresponding to the protruding object area of the complementary image to "1" for the mask information, and sets the transmittance of the area other than the area to "0". Next, the image processing unit 12 generates a right-side composite image by combining the complementary image and the right-side photographed image according to the mask information. In the right-side composite image synthesized here, the side photographed image is displayed in the blind spot area DR for the protruding object area (the area where the image of the protruding object extends in the side photographed image), and the complementary image is displayed in the other areas. The above processing is the composite image generation processing according to this modified example. The monitor 2 displays a composite video based on the composite image generated in this way. Figure 8 shows an example of an image displayed on the right monitor 2R in the same situation as in Figure 6.

As described above, in this modified example, the side video is displayed in a visible state in the area of the protruding object in the blind spot area DR of the video displayed on the monitor 2, so the driver can recognize that an object is protruding from the window 6 based on the video displayed on the monitor 2. In particular, in this modified example, video based on the side captured image is not displayed for the entire blind spot area DR, but the side video is displayed only for the area of the protruding object, preventing the display of the rear video from being stopped for an unnecessarily wide area.

Note that this modified example may be adapted from the first modified example, in which the protrusion detection unit 11 detects that an object is located near the open window 6. Alternatively, the second modified example may be adapted so that the image processing unit 12 changes the transmittance of each pixel in the protruding object area of the mask information from "1" to a transmittance (e.g., "0.5") that makes both the side video image and the rear video image visible. Alternatively, the third modified example may be adapted so that the image processing unit 12 marks the area of the protruding object in the video image.

<Other Modifications of the First Embodiment>
In the first embodiment, the protrusion detection unit 11 detects the protruding state by analyzing the side photographed image, but the protruding state may be detected by a method other than the method exemplified in the first embodiment. As an example, a transmission type or reflection type optical sensor that detects the passing of an object through the window 6 may be provided in the vehicle, and the protrusion detection unit 11 may detect the protruding state based on the detection value of the optical sensor. In the first embodiment, the image processing unit 12 adjusts the transmittance when the complementary image is superimposed when displaying a moving image in the special mode, so that the rear moving image is displayed in the blind spot area DR. In this regard, instead of adjusting the transmittance when the complementary image is superimposed, the complementary image may not be superimposed.

Second Embodiment
Next, a second embodiment will be described. In the following description of the second embodiment, the same elements as those in the first embodiment are given the same reference numerals, and detailed description thereof will be omitted. FIG. 9 is a block diagram showing an example of the functional configuration of an image processing device 1A according to this embodiment. As is clear from a comparison of FIG. 9 with FIG. 1, the image processing device 1A according to this embodiment includes an image acquisition unit 10A instead of the image acquisition unit 10, a specific state detection unit 15 instead of the protrusion detection unit 11, and an image processing unit 12A instead of the image processing unit 12.

The specific state detection unit 15 detects whether each of the four windows 6 is open or closed. Furthermore, the specific state detection unit 15 detects whether the fuel filler opening 5 is open or closed. The fuel filler opening 5 being open means that the lid covering the fuel filler opening 5 is open. The specific state detection unit 15 communicates with the vehicle system described above and detects the open/closed state of each window 6 and the fuel filler opening 5. However, any detection method may be used, and for example, the specific state detection unit 15 may be configured to use various sensors for detection, or may be configured to detect by analyzing images captured by a side camera.

When the specific state detection unit 15 detects that the window 6 is open or detects that the fuel filler 5 is open, the image processing unit 12A displays a moving image in the special mode. More specifically, when at least one of the two right windows 6 and the fuel filler 5 (as described above, the fuel filler 5 is provided on the right side of the vehicle) is open, the image processing unit 12A displays a moving image on the right monitor 2R in the special mode. When at least one of the two left windows 6 is open, the image processing unit 12A displays a moving image on the left monitor 2L in the special mode. As a result, for example, when the right rear window 6 is open, the moving image is displayed on the right monitor 2R in the special mode, and a side moving image is displayed for the blind spot area DR of the moving image.

The configuration of this embodiment provides the following effects. That is, when the window 6 is open, an object may protrude from the window 6, and it is assumed that the driver would like to be able to constantly check the situation around the window 6. With this in mind, this embodiment allows the driver to constantly check the situation around the window 6 through the monitor 2 when the window 6 is open. Furthermore, the fuel filler cap 5 should normally be closed, and it is better to make the driver aware of this when the fuel filler cap 5 is open. And according to this embodiment, when the fuel filler cap 5 is open, a moving image is displayed in a special mode, so that the state in which the fuel filler cap 5 is open is displayed as an image, and the driver can immediately recognize from the moving image that the fuel filler cap 5 is open.

Next, an example of the operation of the image processing device 2A according to this embodiment will be described with reference to the flowchart of FIG. 7. As shown in FIG. 7, the image acquisition unit 10A of the image processing device 1A acquires a side image and a rear image (step SA1). Next, the image processing unit 12A displays on the monitor 2 a composite video image in which the blind spot area DR in the side video image area is supplemented with the rear video image based on the side image and the rear image acquired by the image acquisition unit 10A (step SA2). As described above, the image processing unit 12A according to this embodiment has a function of temporarily making the entire blind spot area DR visible to the side video image when the specific state detection unit 15 detects that the window 6 is open or the fuel filler opening 5 is open (when a predetermined condition is met) when displaying the composite video image in step SA2.

<First Modification of Second Embodiment>
Next, a first modified example of the second embodiment will be described. In the second embodiment, the image processing device 1A does not include the specific state detection unit 15. When displaying the moving image, the image processing unit 12A executes the following process. That is, regardless of the state of the window 6 and the fuel filler opening 5, the side moving image is continuously displayed in the region of the window 6 and the fuel filler opening 5 in the blind spot region DR of the moving image. Since the region of the window 6 and the region of the fuel filler opening 5 in the side photographed image are constant, the image processing unit 12A sets the transmittance of the region of the window 6 and the region of the fuel filler opening 5 to "1" when superimposing the complementary image in the composite image generation process, thereby continuously displaying the side moving image in the region of the window 6 and the region of the fuel filler opening 5.

According to the configuration of this modified example, the driver can always check the situation near the window 6 and the state of the fuel filler opening 5 by visually checking the moving image displayed on the monitor 2. Note that, in this modified example, the second modified example of the first embodiment may be applied, and the image processing unit 12A may set the transmittance of the window 6 area and the fuel filler opening 5 area to a transmittance (e.g., "0.5") that allows both the side moving image and the rear moving image to be visible, rather than setting the transmittance to "1" for the window 6 area and the fuel filler opening 5 area.

<Other Modifications of the Second Embodiment>
In the second embodiment, when the image processor 12A displays a moving image in the special mode, the image processor 12A may be configured to display the side moving image only in the area of the open member, rather than displaying the side moving image in the entire blind spot area DR. For example, when displaying a moving image on the right monitor 2R, if the right rear window 6 is open and the front right window 6 and the fuel filler opening 5 are closed, the image processor 12A may be configured to display the side moving image only in the area of the right rear window 6.

In the second embodiment, the targets for which the specific state detection unit 15 monitors the state are all windows 6 and the fuel filler opening 5, but the targets are not limited to this example. For example, the targets may be only the windows 6 (excluding the fuel filler opening 5), or may be some of the windows 6. The targets may also be members other than the windows 6 and the fuel filler opening 5 (for example, an antenna provided on the vehicle body).

The second embodiment and the first embodiment may also be combined. That is, the image processing device 1A may be further configured to include the protrusion detection unit 11 according to the first embodiment, and the image processing unit 12A may be configured to make the side moving image visible in the blind spot area DR not only when the specific state detection unit 15 detects that a specific member is in a specific state, but also when the protrusion detection unit 11 detects a protruding state.

Although the embodiments of the present invention (including modified examples) have been described above, each of the above embodiments merely shows an example of a concrete implementation of the present invention, and the technical scope of the present invention should not be interpreted in a limiting manner based on them. In other words, the present invention can be implemented in various forms without departing from its gist or main characteristics.

For example, in each of the above embodiments, the rear camera 3 is configured as a fisheye camera and the side camera 4 is configured as a standard camera, but the type of camera that constitutes the rear camera 3 and the side camera 4 is not limited to the examples of the above embodiments.

In addition, in each of the above embodiments, the display devices that display the composite image are the right monitor 2R and the left monitor 2L. However, the display devices that display the moving images are not limited to the right monitor 2R and the left monitor 2L, and may be a display panel provided in the center of the dashboard or in the center console of the vehicle, a display panel provided in the right side mirror MR and the left side mirror ML, a display panel provided in the rearview mirror, etc. The display device may be a mobile terminal.

The installation position of the side camera 4 on the vehicle is not limited to the positions exemplified in the above embodiments. For example, the side camera 4 may be installed on the side of the vehicle, close to the front end of the vehicle.

1, 1A Image processing device 2L Left monitor (display device)
2R Right monitor (display device)
3 Rear camera 4L Left side camera (side camera)
4R Right side camera (side camera)
5 Fuel filler cap 6 Window 10, 10A Image acquisition unit 11 Protrusion detection unit 12, 12A Image processing unit 15 Specific state detection unit DH Blind spot range DR Blind spot area

Claims (12)

an image acquisition unit that acquires a side image based on the image capture result of a side camera that is provided on a side portion of the vehicle and captures images toward the rear of the vehicle with a capture range that includes a part of the side of the vehicle body including the window of a side door of the vehicle, and a rear image based on the image capture result of a rear camera that is provided on the rear of the vehicle and can capture at least the blind spot of the vehicle body within the capture range of the side camera;
an image processing unit that displays on a display device a composite moving image in which a blind spot area corresponding to a blind spot of the side camera among areas of a moving image based on the side photographed image and the rear photographed image acquired by the image acquisition unit is supplemented with a moving image based on the rear photographed image ;
a protrusion detection unit that detects that an object has protruded from the window in an open state or that an object has been positioned in the vicinity of the window in an open state,
The image processing unit includes:
When the composite moving image is displayed, if the protrusion detection unit detects that an object protrudes from the window or that an object is located in the vicinity of the window, the moving image based on the side photographed image is made visible at least for the region of the object in the blind spot region.
13. An image processing device comprising: 2. The image processing device according to claim 1, characterized in that, when the protrusion detection unit detects that an object has protruded from the window or that an object has been positioned in the vicinity of the window, the image processing unit makes only the area of the object in the blind spot area visible in a moving image based on the side-captured image. 3. The image processing apparatus according to claim 1 , wherein the image processing unit marks an area of an object that protrudes from the window or is located in the vicinity of the window. an image acquisition unit that acquires a side image based on the image capture result of a side camera that is provided on a side portion of the vehicle and captures images toward the rear of the vehicle with a capture range that includes a part of the side of the vehicle body including the window of a side door of the vehicle, and a rear image based on the image capture result of a rear camera that is provided on the rear of the vehicle and can capture at least the blind spot of the vehicle body within the capture range of the side camera;
an image processing unit that displays on a display device a composite moving image in which a blind spot area corresponding to a blind spot of the side camera among areas of a moving image based on the side photographed image and the rear photographed image acquired by the image acquisition unit is supplemented with a moving image based on the rear photographed image;
a specific state detection unit that detects whether the window is in an open state;
the image processing unit, when the specific state detection unit detects that the window is open, makes a moving image based on the side-captured image visible at least in the region of the window in the blind spot area. The captured image of the side camera further includes a fuel filler opening formed on a side surface of the vehicle body of the vehicle,
The specific state detection unit further detects that the fuel filler opening is in an open state,
The image processing device according to claim 4, characterized in that, when the specific state detection unit detects that the fuel filler opening is open, the image processing unit makes a moving image based on the side-captured image visible for at least the area of the fuel filler opening in the blind spot area. an image acquisition unit that acquires a side image based on the image capture result of a side camera that is provided on a side portion of the vehicle and captures images toward the rear of the vehicle with a capture range including a part of the side of the vehicle body including a fuel filler port formed on the side of the vehicle body, and a rear image based on the image capture result of a rear camera that is provided on the rear of the vehicle and can capture at least a blind spot due to the vehicle body within the capture range of the side camera;
an image processing unit that displays on a display device a composite moving image in which a blind spot area corresponding to a blind spot of the side camera among areas of a moving image based on the side photographed image and the rear photographed image acquired by the image acquisition unit is supplemented with a moving image based on the rear photographed image;
A specific state detection unit that detects that the fuel filler opening is in an open state,
The image processing device is characterized in that, when the specific state detection unit detects that the fuel filler opening is open, the image processing unit makes a moving image based on the side-captured image visible for at least the area of the fuel filler opening in the blind spot area. an image acquisition unit that acquires a side image based on the image capture result of a side camera that is provided on a side portion of the vehicle and captures images toward the rear of the vehicle with a capture range including a part of the side of the vehicle body including a fuel filler port formed on the side of the vehicle body, and a rear image based on the image capture result of a rear camera that is provided on the rear of the vehicle and can capture at least a blind spot due to the vehicle body within the capture range of the side camera;
an image processing unit that displays on a display device a composite moving image in which a blind spot area corresponding to a blind spot of the side camera among areas of a moving image based on the side photographed image and the rear photographed image acquired by the image acquisition unit is supplemented with a moving image based on the rear photographed image,
The image processing device according to claim 1, wherein the image processing unit continuously makes a moving image based on the side-photographed image visible for an area of the fuel filler opening in the blind spot area. The side camera's imaging range further includes the side door windows of the vehicle,
The image processing device according to claim 7 , wherein the image processing unit further makes a moving image based on the side-photographed image visible continuously in the region of the window or an upper region of the window in the blind spot region. a first step in which an image acquisition unit of an image processing device acquires a side photographed image based on the photographing result of a side camera provided on a side portion of a vehicle, the side camera photographing the rearward direction of the vehicle with a photographing range including a part of the side of the vehicle body including a side door window , and a rear photographed image based on the photographing result of a rear camera provided on a rear portion of the vehicle, the rear camera being capable of photographing at least a blind spot due to the vehicle body within the photographing range of the side camera;
a second step of displaying on a display device a composite moving image in which a blind spot area of a moving image based on the side photographed image, which corresponds to a blind spot of the side camera, among areas of a moving image based on the side photographed image, is supplemented with a moving image based on the rear photographed image, based on the side photographed image and the rear photographed image acquired by the image acquisition unit;
When a protrusion detection unit detects that an object has protruded from the window in an open state or that an object has been positioned in the vicinity of the window in an open state during display of the composite moving image in the second step, the image processing unit makes the moving image based on the side-photographed image visible for at least the region of the object in the blind spot region when the protrusion detection unit detects that an object has protruded from the window in an open state or that an object has been positioned in the vicinity of the window.
13. An image processing method comprising: a first step in which an image acquisition unit of an image processing device acquires a side photographed image based on the photographing result of a side camera provided on a side portion of a vehicle, the side camera photographing the rearward direction of the vehicle with a photographing range including a part of the side of the vehicle body including a side door window, and a rear photographed image based on the photographing result of a rear camera provided on a rear portion of the vehicle, the rear camera being capable of photographing at least a blind spot due to the vehicle body within the photographing range of the side camera;
a second step of displaying on a display device a composite moving image in which a blind spot area of a moving image based on the side photographed image, which corresponds to a blind spot of the side camera, among areas of a moving image based on the side photographed image, is supplemented with a moving image based on the rear photographed image, based on the side photographed image and the rear photographed image acquired by the image acquisition unit;
In the second step, when a specific state detection unit that detects that the window is in an open state detects that the window is in an open state, the image processing unit makes a moving image based on the side-captured image visible for at least an area of the window in the blind spot area.
13. An image processing method comprising: a first step in which an image acquisition unit of an image processing device acquires a side photographed image based on the photographing result of a side camera provided on a side portion of a vehicle, the side camera photographing the rearward direction of the vehicle with a photographing range including a part of the side of the vehicle body including a fuel filler opening formed on the side of the vehicle body, and a rear photographed image based on the photographing result of a rear camera provided on the rear of the vehicle, the rear camera being capable of photographing at least a blind spot due to the vehicle body within the photographing range of the side camera;
a second step of displaying on a display device a composite moving image in which a blind spot area of a moving image based on the side photographed image, which corresponds to a blind spot of the side camera, among areas of a moving image based on the side photographed image, is supplemented with a moving image based on the rear photographed image, based on the side photographed image and the rear photographed image acquired by the image acquisition unit;
In the second step, when a specific state detection unit that detects that the fuel filler opening is in an open state detects that the fuel filler opening is in an open state, the image processing unit makes a moving image based on the side photographed image visible for at least the region of the fuel filler opening in the blind spot region.
13. An image processing method comprising: a first step in which an image acquisition unit of an image processing device acquires a side photographed image based on the photographing result of a side camera provided on a side portion of a vehicle, the side camera photographing the rearward direction of the vehicle with a photographing range including a part of the side of the vehicle body including a fuel filler opening formed on the side of the vehicle body, and a rear photographed image based on the photographing result of a rear camera provided on the rear of the vehicle, the rear camera being capable of photographing at least a blind spot due to the vehicle body within the photographing range of the side camera;
a second step of displaying on a display device a composite moving image in which a blind spot area of a moving image based on the side photographed image, which corresponds to a blind spot of the side camera, among areas of a moving image based on the side photographed image, is supplemented with a moving image based on the rear photographed image, based on the side photographed image and the rear photographed image acquired by the image acquisition unit;
In the second step, the image processing unit continuously displays a moving image based on the side photographed image in a state in which the area of the fuel filler opening in the blind spot area is visible.
13. An image processing method comprising: