JP2021118435A - Image processing device and image processing method - Google Patents

Abstract

To provide an image processing technique capable of improving the appearance of a virtual viewpoint image.SOLUTION: An image processing device comprises: an image generation unit which on the basis of an image captured with a camera installed on a mobile object, generates a virtual viewpoint image showing the circumference of the mobile object viewed from a virtual viewpoint; an adjustment unit which adjusts the region of a mask to mask the mobile object reflected in the virtual viewpoint image, on the basis of the captured image; and a superimposition unit which superimposes the mask on the virtual viewpoint image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing technique for generating a virtual viewpoint image showing the periphery of a moving body as seen from a virtual viewpoint, based on a captured image taken by a camera mounted on the moving body.

Conventionally, various image processing techniques have been developed to generate a virtual viewpoint image showing the periphery of a moving body viewed from a virtual viewpoint based on an image taken by a camera mounted on the moving body (for example, patent documents). 1). The virtual viewpoint image described above is used, for example, to support the driving of a moving body.

In the image processing device disclosed in Patent Document 1, the area in which the vehicle is reflected in the captured image taken by the camera mounted on the vehicle is painted in black to hide the vehicle reflected in the image. ing. This is to prevent the body of the vehicle reflected in the image from being mistaken for an obstacle existing near the vehicle.

International Publication No. 00/64175 (Fig. 37 and Fig. 38)

Patent Document 1 describes that the reflection area of the body of a vehicle can be uniquely determined in advance if the specifications and orientation of the camera and the shape of the vehicle on which the camera is mounted are determined.

However, if the reflection area of the vehicle body is uniquely determined in advance, for example, when the user attaches aero parts or the like after the vehicle is delivered, or when the shape of the bumper or the like changes due to aging or an accident. When the orientation (posture) of the camera with respect to the vehicle changes due to aging or an accident, the body of the vehicle reflected in the image may not be completely hidden.

In view of the above problems, an object of the present invention is to provide an image processing technique capable of improving the appearance of a virtual viewpoint image.

The image processing apparatus according to the present invention includes an image generation unit that generates a virtual viewpoint image showing the periphery of the moving body as seen from a virtual viewpoint based on a captured image taken by a camera mounted on the moving body, and the virtual image generating unit. A configuration including an adjusting unit that adjusts the area of the mask that hides the moving object reflected in the viewpoint image based on the captured image, and a compositing unit that superimposes the mask on the virtual viewpoint image (No. 1). 1 configuration).

In the image processing apparatus having the first configuration, the adjusting unit adjusts the mask area based on a plurality of captured images taken at different times while the moving body is moving (second configuration). ) May be.

In the image processing apparatus having the first or second configuration, the adjusting unit adjusts the mask area based on the position of the end of the white line reflected in the captured image (third configuration). It may be.

In the image processing apparatus having any of the first to third configurations, when changing the mask area, the adjusting unit executes the change of the mask area after obtaining approval by a user operation (a configuration in which the mask area is changed. 4th configuration) may be used.

In the image processing apparatus having any of the first to fourth configurations, the adjusting unit may be configured to return the mask area to the initial setting in response to a reset request by a user operation (fifth configuration). good.

The image processing apparatus having any of the first to fifth configurations may be configured to include a notification unit for notifying an abnormality when at least a part of the mask region exceeds a predetermined range (sixth configuration). ..

The image processing method according to the present invention includes an image generation step of generating a virtual viewpoint image showing the periphery of the moving body viewed from a virtual viewpoint based on a captured image taken by a camera mounted on the moving body, and the virtual image generation step. A configuration including an adjustment step of adjusting the area of the mask that hides the moving object reflected in the viewpoint image based on the captured image, and a composition step of superimposing the mask on the virtual viewpoint image (the first). 7 configuration).

According to the image processing technique according to the present invention, the appearance of the virtual viewpoint image can be improved.

Block diagram showing the configuration of the vehicle peripheral monitoring system Schematic diagram showing a vehicle in which four cameras are arranged Flow chart showing the operation of the image processing device Diagram for explaining the method by which the image generator generates a virtual viewpoint image Enlarged view of the part where the image of the vehicle is placed in the composite image Schematic diagram showing a captured image taken by a back camera Schematic diagram showing other captured images taken with a back camera Flow chart showing a modified example of the operation of the image processing device Flowchart showing other variations of the operation of the image processing device

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, a vehicle will be described as an example of a moving body, but the moving body is not limited to a vehicle. Vehicles include a wide range of vehicles having wheels, such as automobiles, trains, and automatic guided vehicles. Examples of moving objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight from the driver's seat to the steering wheel is referred to as the "forward direction". Further, the direction in which the vehicle travels straight from the steering wheel to the driver's seat is defined as the "rear direction". Further, the direction from the right side to the left side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "left direction". Further, the direction from the left side to the right side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "right direction".

<System configuration>
FIG. 1 is a block diagram showing a configuration of a vehicle peripheral monitoring system SYS1 according to an embodiment of the present invention. The vehicle peripheral monitoring system SYS1 is installed in each vehicle. The vehicle peripheral monitoring system SYS1 has a function of monitoring the peripheral condition of the vehicle equipped with the system. As shown in FIG. 1, the vehicle monitoring system SYS1 includes an image processing system SYS2 and a display device 3.

The image processing system SYS2 has a function of photographing the surroundings of the vehicle, generating an image, and outputting the generated image to the display device 3. The image processing system SYS2 also has a function of detecting a detection target based on an image obtained by photographing the periphery of the vehicle. In the present embodiment, the detection target includes a parking frame. It should be noted that this is an example, and the detection target may include a moving body or the like. The detected parking frame information is used, for example, by a driving support device (not shown) that provides parking support or the like. The image processing system SYS2 includes a photographing unit 1 for photographing the surroundings of the vehicle and an image processing device 2.

The photographing unit 1 includes four cameras 11 to 14 mounted on the vehicle. That is, the image processing system SYS2 includes cameras 11 to 14 mounted on the vehicle. FIG. 2 is a schematic view showing a vehicle 4 in which four cameras 11 to 14 are arranged. In the present embodiment, the vehicle 4 is an automobile. The number of cameras included in the photographing unit 1 is an example, and the number of cameras may be changed as appropriate. Each of the cameras 11 to 14 is connected to the image processing device 2 by wire or wirelessly, and transmits the captured image to the image processing device 2.

As shown in FIG. 2, the camera 11 is provided at the front end of the vehicle 4. Therefore, the camera 11 is also referred to as a front camera 11. The optical axis 11a of the front camera 11 is along the front-rear direction of the vehicle 4 in a plan view. The front camera 11 photographs the front direction of the vehicle 4. The camera 14 is provided at the rear end of the vehicle 4. Therefore, the camera 14 is also referred to as a back camera 14. The optical axis 14a of the back camera 14 is along the front-rear direction of the vehicle 4 in a plan view. The back camera 14 captures the rear direction of the vehicle 4. The front camera 11 and the back camera 14 are preferably mounted at the center of the left and right sides of the vehicle 4, but may be slightly deviated from the center of the left and right sides in the left-right direction.

The camera 12 is provided on the left door mirror 41 of the vehicle 4. Therefore, the camera 12 is also referred to as a left side camera 12. When the own vehicle is a so-called door mirrorless vehicle, the left side camera is attached around the rotation axis (hinge portion) of the left side door without passing through the door mirror. The optical axis 12a of the left side camera 12 is along the left-right direction of the vehicle 4 in a plan view. The left side camera 12 captures the left direction of the vehicle 4. The camera 13 is provided on the right door mirror 42 of the vehicle 4. Therefore, the camera 13 is also referred to as a right side camera 13. When the own vehicle is a so-called door mirrorless vehicle, the right side camera is attached around the rotation axis (hinge portion) of the right side door without passing through the door mirror. The optical axis 13a of the right side camera 13 is along the left-right direction of the vehicle 4 in a plan view. The right side camera 13 photographs the vehicle 4 in the right direction.

As the lens of each camera 11-14, for example, a fisheye lens is used. The horizontal angle of view θ of each camera 11 to 14 is 180 degrees or more. Therefore, by using the four cameras 11 to 14, it is possible to take a picture of the entire circumference of the vehicle 4 in the horizontal direction.

The image processing device 2 acquires a photographed image photographed by the photographing unit 1 and processes the photographed image. Further, the image processing device 2 outputs the processed image obtained by processing the captured image to the display device 3. Further, the image processing device 2 processes the captured image and detects a detection target (for example, a parking frame) from the captured image. The image processing device 2 is arranged at a predetermined position of the vehicle 4. That is, the image processing device 2 is used in a vehicle. The details of the image processing device 2 will be described later.

The display device 3 includes a display such as a liquid crystal display having an operation function such as a touch panel, and displays a display image obtained by processing by the image processing system SYS2. The display device 3 is provided at a position where the occupant (typically, the driver) of the vehicle 4 can visually recognize the display screen of the display. For example, the display device 3 is installed on the instrument panel of the vehicle 4.

<Configuration of image processing device>
The image processing device 2 is configured as an ECU (Electronic Control Unit). As shown in FIG. 1, the image processing device 2 includes an image generation unit 21, a control unit 22, a synthesis unit 23, a communication unit 24, and a storage unit 25.

The image generation unit 21 processes the captured image captured by the photographing unit 1 to generate a display image. In the present embodiment, the image generation unit 21 is configured as a hardware circuit capable of performing various image processing. In the present embodiment, the image generation unit 21 generates a virtual viewpoint image showing the periphery of the vehicle 4 as seen from the virtual viewpoint, based on the images taken by the cameras 11 to 14 mounted on the vehicle 4. The details of the method for generating the virtual viewpoint image will be described later.

After the mask is superimposed by the compositing unit 23, the virtual viewpoint image is output to the display device 3 by the communication unit 24 and displayed on the display screen of the display device 3.

The control unit 22 controls the entire image processing device 2. The control unit 22 is a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). In the present embodiment, the control unit 22 includes an image control unit 221, a display control unit 222, an adjustment unit 223, and a detection unit 224. Each of these units included in the control unit 22 is a function realized by, for example, the CPU performing arithmetic processing according to a program stored in the ROM.

The image control unit 221 controls the image processing executed by the image generation unit 21 and the image processing executed by the composition unit 23. For example, the image control unit 221 instructs the image generation unit 21 various parameters and the like necessary for generating a virtual viewpoint image, and causes the composition unit 23 to generate a composite image. Instruct various necessary parameters.

The display control unit 222 mainly controls the display device 3 to display the image information obtained by the processing of the image processing device 2. For example, the display control unit 222 controls when the composite image generated by the synthesis unit 23 is output to the display device 3.

The adjustment unit 223 adjusts the area of the mask that hides the vehicle reflected in the virtual viewpoint image based on the captured image.

The detection unit 224 detects the parking frame line that divides the parking area of the vehicle reflected in the captured image, and detects the area between the detected two parking frames as the parking frame. The parking frame detection process is carried out as appropriate.

The communication unit 24 communicates with the display device 3. The communication unit 24 outputs, for example, the composite image generated by the composite unit 23 to the display device 3. Further, the communication unit 24 receives a signal transmitted from the operation unit (touch panel or the like) when the driver or the like operates the display device 3.

The storage unit 25 is, for example, a non-volatile memory such as a flash memory, and stores various types of information. The storage unit 25 stores, for example, a program as firmware and various data for the image generation unit 21 to generate a virtual viewpoint image. In addition, the storage unit 25 stores various data required for the adjusting unit 223 and the detecting unit 224 to execute the process.

<Configuration of image processing device>
FIG. 3 is a flowchart showing the operation of the image processing device 2.

First, according to the instruction of the image control unit 221, the image generation unit 21 executes an image generation process for generating a virtual viewpoint image showing the state of the surroundings of the vehicle 4 as seen from the virtual viewpoint. (Step S10). The details of the method for generating the virtual viewpoint image will be described later.

Next, the adjustment unit 223 executes an adjustment process for adjusting the area of the mask that hides the vehicle reflected in the virtual viewpoint image based on the captured image (step S20). The details of the method for adjusting the mask will be described later.

Finally, according to the instruction of the image control unit 221, the compositing unit 23 executes the compositing process of superimposing the mask on the virtual viewpoint image (step S30). The details of the composite image obtained by executing the composite processing will be described later. After step S30, a parking frame detection process, a communication process for communicating with the display device 3, and the like are executed as appropriate.

<Virtual viewpoint image generation process>
A method in which the image generation unit 21 generates a virtual viewpoint image showing the state of the surroundings of the vehicle 4 as seen from the virtual viewpoint will be described. FIG. 4 is a diagram for explaining a method in which the image generation unit 21 generates a virtual viewpoint image. The image generation unit 21 uses a virtual three-dimensional projection surface TS to generate a virtual viewpoint image, thereby generating a virtual viewpoint image having a sense of reality close to reality.

The front camera 11, the left side camera 12, the right side camera 13, and the back camera 14 simultaneously acquire four captured images P11 to P14 indicating the front, left side, right side, and rear of the vehicle 4. .. These four captured images P11 to P14 include data on the entire circumference of the vehicle 4. The image generation unit 21 acquires these four captured images P11 to P14. The solid-painted portion in the four photographed images P11 to P14 is the body of the vehicle 4 reflected in the photographed images.

The image generation unit 21 projects the data (pixel values) contained in these four captured images P11 to P14 onto the projection surface TS which is a three-dimensional curved surface in a virtual three-dimensional space. The projection surface TS has, for example, a substantially hemispherical shape (bowl shape), and its central portion (bottom portion of the bowl) is defined as the position of the vehicle 4. The correspondence between the position of each pixel included in the captured images P11 to P14 and the position of each pixel on the projection surface TS is determined in advance. The table data showing this correspondence is stored in the storage unit 25. The value of each pixel of the projection surface TS can be determined based on this correspondence and the value of each pixel included in the captured images P11 to P14. Images captured by the two cameras are projected in the area of the projection surface TS corresponding to the shaded area shown in FIG. In the area of the projection surface TS corresponding to the shaded area shown in FIG. 2, for example, the images captured by the two cameras may be blended in a predetermined ratio.

Next, the image generation unit 21 sets the virtual viewpoint VP for the three-dimensional space under the control of the image control unit 221. The virtual viewpoint VP is defined by the viewpoint position and the line-of-sight direction. The image generation unit 21 can set a virtual viewpoint VP at an arbitrary viewpoint position and an arbitrary line-of-sight direction in a three-dimensional space. The image generation unit 21 cuts out a necessary area on the projection surface TS according to the set virtual viewpoint VP. As a result, a virtual viewpoint image viewed from an arbitrary virtual viewpoint VP is generated.

For example, as shown in FIG. 4, assuming a virtual viewpoint VPa with the viewpoint position directly above the vehicle 4 and the line-of-sight direction directly below, the virtual viewpoint image (overhead image) that gives a bird's-eye view of the vehicle 4 and the surroundings of the vehicle 4 is obtained. Using FIG. 5, a composite image CPa obtained by superimposing the image of the mask and the vehicle 4 described later can be obtained. Further, assuming a virtual viewpoint VPb in which the viewpoint position is the left rear of the vehicle 4 and the line-of-sight direction is the front of the vehicle 4, the virtual viewpoint image showing the vehicle 4 and the periphery of the vehicle 4 as seen from the left rear of the vehicle 4 is masked. And a composite image CPb in which the image of the vehicle 4 is superimposed can be obtained.

The image of the vehicle 4 shown in the virtual viewpoint image CP is prepared in advance as data such as a bitmap and stored in the storage unit 25. At the time of generating the composite image, the data of the image of the vehicle 4 having the shape corresponding to the viewpoint position and the line-of-sight direction of the virtual viewpoint VP of the virtual viewpoint image is read out, and the mask is already superimposed. The image of the vehicle 4 is superimposed on the viewpoint image.

FIG. 5 is an enlarged view showing a portion of the composite image in which the image of the vehicle 4 is arranged. Note that FIG. 5 assumes a virtual viewpoint image using the above-mentioned virtual viewpoint VPa. As shown in FIG. 5, the mask 5 is superimposed on the virtual viewpoint image. In FIG. 5, the mask 5 is a shaded portion. In the present embodiment, the area of the mask 5 is an area where the body of the vehicle 4 may be reflected. Specifically, the outer edge (boundary) of the mask 5 is rectangular as shown in FIG. The mask 5 is displayed in black or the like on the display device 3. As a result, it is possible to prevent the vehicle reflected in the captured image from being mistaken for an obstacle existing near the vehicle and the user who sees the composite image.

The initial setting of the area of the mask 5 is determined by the body shape of the vehicle 4 and the mounting positions of the cameras 11 to 14. Therefore, the initial setting of the area of the mask 5 can be set for each vehicle type or each grade, for example. In the present embodiment, the initial setting of the area of the mask 5 is set in consideration of the mounting error and the calibration error of the cameras 11 to 14. Specifically, the initial setting of the area of the mask 5 is set so that the body image of the vehicle 4 is not displayed in the composite image even when the range in which the body of the vehicle 4 is reflected is maximized due to an error. The data regarding the initial setting of the area of the mask 5 is stored in the storage unit 25. The mask 5 is superimposed on the virtual viewpoint image according to the viewpoint position and the line-of-sight direction of the virtual viewpoint VP of the virtual viewpoint image when the composite image is generated.

<Mask area adjustment process>
The adjustment unit 223 may perform the adjustment process every time the image processing device 2 is activated, for example, and performs the adjustment process every time the driver or the like instructs the display device 3 to perform the adjustment process. You may.

The adjustment unit 223 adjusts the area of the mask 5 based on the captured image. Specifically, the adjustment unit 223 estimates the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image based on the captured image, the mask 5 covers the entire reflection estimation area of the body, and the body. The area of the mask 5 is adjusted so as not to cover the area other than the reflection estimation area as much as possible. Therefore, if the reflection estimation area of the body is wide, the area of the mask 5 is also wide, and if the reflection estimation area of the body is narrow, the area of the mask 5 is also narrow.

Therefore, for example, when the user attaches aero parts or the like after the vehicle is delivered, the shape of the bumper or the like changes due to aging or an accident, the orientation (posture) of the camera with respect to the vehicle changes due to aging or an accident. Even when the area is changed, the vehicle reflected in the virtual viewpoint image can be hidden after the area of the mask 5 is adjusted by the adjustment unit 223, so that the appearance of the virtual viewpoint image can be improved.

Further, as described above, in the initial setting of the area of the mask 5, the body image of the vehicle 4 is not displayed in the composite image even when the range in which the body of the vehicle 4 is reflected is maximized due to an error. According to the image processing device 2, after the region of the mask 5 is adjusted by the adjusting unit 223, the region of the mask 5 can be optimized in each vehicle. That is, the appearance of the virtual viewpoint image can be improved as compared with the initial setting.

Further, according to the image processing device 2, since the area of the mask 5 can be optimized, the initial setting of the area of the mask 5 can be set to a common setting instead of being set separately for each vehicle type or grade, for example. can. Further, according to the image processing device 2, since the area of the mask 5 can be optimized, the initial setting of the area of the mask 5 can be roughly set, and the mask can be roughly set without using detailed data on the body shape of the vehicle. The initial setting of the area of 5 becomes possible.

The adjusting unit 223 adjusts the area of the mask 5 while keeping the outer edge (boundary) of the mask 5 rectangular in the bird's-eye view image.

Further, in the area of the projection surface TS corresponding to the shaded area shown in FIG. 2, the image generation unit 21 is not the reflection estimation area of the body in the image taken from one camera, but the body in the image taken from the other camera. In the part that is the reflection estimation area of, the above-mentioned blending process is not performed, and only the image captured by one of the cameras is projected.

The adjusting unit 223 may estimate the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image, for example, based on a plurality of captured images taken at different times while the vehicle is traveling. Since the body of the vehicle 4 is always stationary with respect to the camera 11 and is reflected at the same pixel position in the captured image, the pixel value does not change much between the plurality of captured images and the shape does not change, whereas the scenery is the camera. Since it is moving with respect to 11 to 14, the pixel value changes drastically between a plurality of captured images, and the shape also changes. Therefore, the difference between the two frames of the captured images taken by the camera 11 at different times while the vehicle is running, the difference between the two frames of the captured images taken by the camera 12 at different times while the vehicle is running, and the difference between the two frames of the captured images taken by the camera 12 while the vehicle is running. There is no change in shape or pixel value from the difference between the two frames of images taken by the camera 13 at different times and the difference between the two frames of images taken by the camera 14 at different times while the vehicle is running. The area can be estimated as the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image. According to this estimation method, it is possible to estimate the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image even if there is no white line on the road surface. Since mud and the like adhering to the lens of the camera are also stationary with respect to the camera, for example, the adjusting unit 223 can distinguish between the lens adhering matter and the body of the vehicle 4 depending on the hue and the location in the captured image. desirable. Specifically, as shown in P11 to P14 of FIG. 4, the body of the vehicle 4 is projected along the lower side of the captured image. Therefore, when the stationary region is along the lower side of the captured image, the vehicle It can be estimated to be the body of 4.

Since there may be pixels in which the body of the vehicle 4 and the scenery cannot be distinguished from each other only by the difference between the captured images of 2 frames, the dispersion of the pixel values is calculated for each pixel of the captured images of 3 frames or more, and the variance is calculated. If it is more than a predetermined value, it may be classified as a landscape, and if the variance is less than a predetermined value, it may be classified as a body. This improves the estimation accuracy.

Further, the image processing device 2 acquires information such as the speed of the vehicle, the position of the vehicle, and the brightness around the vehicle, and uses only the captured images taken under the same conditions for the estimation process. Further, the estimation accuracy is improved.

The adjusting unit 223 may estimate the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image, for example, based on the end position of the white line reflected in the captured image. This estimation method utilizes the fact that when the body of the vehicle 4 hides a part of the white line in the captured image, the position of the end of the white line changes according to the area where the body of the vehicle 4 is reflected. .. For example, in the captured images shown in FIGS. 6 and 7 captured by the back camera 14, a part of the white line WL1 is hidden by the region R1 in which the body of the vehicle 4 is reflected, so that the white line WL1 on the captured image The end position E1 is the boundary between the region R1 and the white line WL1. Since the shape of the region R1 is different between the captured image shown in FIG. 6 and the captured image shown in FIG. 7, the end position E1 of the white line WL1 on the captured image is also different from each other. That is, since there is a correlation between the end position E1 of the white line WL1 and the shape of the region R1 on the captured image, a virtual viewpoint based on the end position of the white line reflected in the captured image as described above. It is possible to estimate the area in which the body of the vehicle 4 is reflected in the image. According to this estimation method, it is possible to estimate the area in which the body of the vehicle 4 is reflected in the virtual viewpoint image even when the vehicle is not traveling. In this case as well, the image processing device 2 acquires information such as the position of the vehicle and the brightness around the vehicle, and uses only the captured images taken under the same conditions for the estimation processing to obtain the estimation accuracy. Is improved.

Further, when the area of the mask 5 is changed, the adjustment unit 223 may change the area of the mask 5 after the driver or the like performs an operation of indicating the approval of the change to the display device 3. good. This makes it possible to prevent the area of the mask 5 from being adjusted against the will of the driver or the like. For example, the driver or the like may not approve the adjustment of the area of the mask 5 when the aero parts or the like are not attached or the accident is not remembered.

Further, when the driver or the like performs an operation of indicating the approval of the reset request change to the display device 3, the adjustment unit 223 may return the area of the mask 5 to the initial setting according to the operation. As a result, even if a problem occurs due to the adjustment of the area of the mask 5, the original state can be restored, which improves convenience.

Further, when at least a part of the area of the mask 5 exceeds a predetermined range as a result of the adjustment by the adjusting unit 223, the control unit 22 may function as a notification unit for notifying the display device 3 of an abnormality. As a result, the driver or the like can be made aware of the misalignment of the cameras 11 to 14 that the driver or the like is not aware of. In this case, for example, as shown in the flowchart shown in FIG. 8, steps S21 and S22 may be added between steps S20 and S30. Note that, unlike the flowchart shown in FIG. 8, steps S21 and S22 may be added after step S30. In step S21, the control unit 22 determines whether or not at least a part of the area of the mask 5 exceeds a predetermined range. When at least a part of the area of the mask 5 exceeds a predetermined range, the control unit 22 executes a notification process for notifying the display device 3 of an abnormality (step S22). On the other hand, if at least a part of the area of the mask 5 does not exceed the predetermined range, the notification process of step S22 is not executed.

Further, the control unit 22 may display the result of comparing the latest adjustment result of the adjustment unit 223 with the previous adjustment result on the display device 3. As a result, it is possible to notify the driver or the like whether or not the area of the mask 5 has been changed. In this case, for example, as shown in the flowchart shown in FIG. 9, steps S23 and S24 may be added between steps S20 and S30. Note that, unlike the flowchart shown in FIG. 9, steps S23 and S24 may be added after step S30. In step S23, the control unit 22 executes a comparison process for comparing the latest adjustment result of the adjustment unit 223 with the previous adjustment result. Then, in step S24, the control unit 22 executes a display process of displaying the result of comparing the latest adjustment result of the adjustment unit 223 with the previous adjustment result on the display device 3. If there is no change between the latest adjustment result of the adjustment unit 223 and the previous adjustment result, the display process of step S24 may not be executed.

<Notes>
The configurations of the embodiments and examples in the present specification are merely examples of the present invention. The configurations of the embodiments and modifications may be appropriately changed without exceeding the technical idea of the present invention. Moreover, a plurality of embodiments and modifications may be carried out in combination to the extent possible.

2 Image processing device 4 Vehicle 11 Front camera 12 Left side camera 13 Right side camera 14 Back camera 21 Image generation unit 23 Synthesis unit 223 Adjustment unit

Claims (7)

An image generator that generates a virtual viewpoint image showing the periphery of the moving body as seen from the virtual viewpoint based on the captured image taken by the camera mounted on the moving body.
An adjustment unit that adjusts the area of the mask that hides the moving object reflected in the virtual viewpoint image based on the captured image, and
A compositing unit that superimposes the mask on the virtual viewpoint image,
An image processing device. The image processing apparatus according to claim 1, wherein the adjusting unit adjusts a region of the mask based on a plurality of captured images taken at different times while the moving body is moving. The image processing apparatus according to claim 1 or 2, wherein the adjusting unit adjusts an area of the mask based on the position of an end portion of a white line reflected in the captured image. The image processing apparatus according to any one of claims 1 to 3, wherein the adjusting unit executes the change of the mask area after obtaining approval by a user operation when changing the mask area. The image processing apparatus according to any one of claims 1 to 4, wherein the adjusting unit returns the mask area to the initial setting in response to a reset request by a user operation. The image processing apparatus according to any one of claims 1 to 5, further comprising a notification unit that notifies an abnormality when at least a part of the mask area exceeds a predetermined range. An image generation process that generates a virtual viewpoint image showing the periphery of the moving body as seen from the virtual viewpoint based on the captured image taken by the camera mounted on the moving body.
An adjustment step of adjusting the area of the mask that hides the moving object reflected in the virtual viewpoint image based on the captured image, and
A compositing step of superimposing the mask on the virtual viewpoint image and
An image processing method.

Images