JP2023029064A - Image processing device, image processing method, image capturing device, and mobile vehicle - Google Patents

Abstract

To allow for providing viewpoint modified image with less distortion.SOLUTION: An image processing device 14 provided herein comprises an image interface 15 and a processor 18. The processor 18 estimates depth at multiple points in a captured image. The processor 18 converts the depth and image coordinates at the multiple points into three-dimensional coordinates corresponding to each of the multiple points. The processor 18 generates a viewpoint modified image having a viewpoint different from that of the captured image based on the three-dimensional coordinates.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image processing device, an image processing method, an imaging device, and a moving body.

A technique has been proposed in which an image captured by a camera is converted from a virtual viewpoint above the vehicle into a bird's-eye coordinate and displayed on a display means (see Patent Document 1).

JP 2003-125394 A

However, in the above display, distortions that differ from reality may occur.

Accordingly, an object of the present disclosure, which has been made in view of the problems of the conventional technology as described above, is to provide an image processing device, an image processing method, an imaging device, and a moving body that can provide a viewpoint-changed image with reduced distortion. to do.

In order to solve the above-mentioned problems, the image processing device according to the first aspect includes:
an image interface for acquiring a captured image;
estimating depths at a plurality of locations in the captured image; transforming the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations; and determining the captured image based on the three-dimensional coordinates. and a processor that generates a viewpoint-changed image with a different viewpoint.

The image processing method according to the second aspect includes:
obtaining a captured image;
estimating depths at a plurality of locations in the captured image; transforming the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations; and determining the captured image based on the three-dimensional coordinates. and generating a viewpoint-changed image with a different viewpoint.

The imaging device according to the third aspect is
an imaging unit that generates an image signal by imaging;
an imaging unit that generates an image signal by imaging;
estimating depths at a plurality of locations in a captured image corresponding to the image signal, converting the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations, and based on the three-dimensional coordinates and a processor that generates a viewpoint-changed image having a viewpoint different from that of the captured image.

A moving object according to the fourth aspect is
the main body;
an imaging unit that is fixed at a predetermined position on the main body in a predetermined posture and that generates an image signal by imaging;
estimating depths at a plurality of locations in a captured image corresponding to the image signal, converting the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations, and based on the three-dimensional coordinates and a processor that generates a viewpoint-changed image having a viewpoint different from that of the captured image.

According to the image processing device, image processing method, imaging device, and moving object according to the present disclosure configured as described above, a viewpoint-changed image with reduced distortion can be provided.

1 is an external view conceptually illustrating the position of an imaging device including an image processing device according to the present embodiment on a moving object; FIG. 2 is a block diagram showing a schematic configuration of the imaging device of FIG. 1; FIG. 3 is a diagram showing a viewpoint-changed image created by the processor of FIG. 2; FIG. FIG. 3 is a diagram illustrating a process in which the processor in FIG. 2 synthesizes a plurality of viewpoint-changed images to generate a single viewpoint-changed image; FIG. 3 is a flowchart for explaining processing executed by a processor in FIG. 2; FIG.

An embodiment of an image processing apparatus to which the present disclosure is applied will be described below with reference to the drawings.

In the present disclosure, at least one imaging device, each including an image processing device, is provided at different positions on the body 12 of the moving object 11 . In this embodiment, four imaging devices 10a, 10b, 10c (see FIGS. 1 and 4) and 10d (see FIG. 4) are provided. The imaging devices 10a, 10b, 10c, and 10d are attached so as to be able to image the surroundings of the moving body 11. FIG. In the following description, the description common to the imaging devices 10a, 10b, 10c, and 10d may be referred to as the imaging device 10. FIG.

The mobile object 11 may include a vehicle, an aircraft, or the like that can move on a road surface. The moving body 11 may have an automatic driving function. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, utility vehicles, and fixed-wing aircraft that travel on runways, and the like. Motor vehicles may include, for example, cars, trucks, buses, motorcycles, trolleybuses, and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction, and the like. Industrial vehicles may include, for example, forklifts, golf carts, and the like. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, lawn mowers, and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, mobile cranes, tippers, road rollers, and the like. Vehicles may include those that are powered by humans. Vehicle classification is not limited to the above example. For example, automobiles may include road-drivable industrial vehicles. Multiple classifications may contain the same vehicle. Aircraft may include, for example, fixed-wing aircraft, rotary-wing aircraft, and the like.

As shown in FIG. 2 , the imaging device 10 includes an imaging section 13 and an image processing device 14 .

The image capturing unit 13 captures an image of the surroundings of the moving body 11 to generate a captured image, and provides the captured image to the image processing device 14 . As shown in FIGS. 1 and 4, the imaging devices 10a, 10b, 10c, and 10d may be fixed at predetermined positions on the main body 12 of the moving body 11 in predetermined postures.

In the present embodiment, the imaging device 10a may be positioned near the roof in the vertical direction of the main body 12 and in the front and center in the width direction. The imaging device 10 b may be positioned below the left side mirror of the main body 12 . The imaging device 10 c may be positioned below the right side mirror of the main body 12 . The imaging device 10d may be positioned in the vicinity of the roof in the vertical direction of the main body 12, in the rear, in the center in the width direction.

The imaging devices 10a and 10d may be fixed so that their optical axes face the front and rear of the main body 12, respectively, and are horizontal when the moving body 11 is stationary. The imaging devices 10b and 10c may be fixed so that their optical axes face the outside in the width direction of the main body 12 and are horizontal when the moving body 11 is stationary.

The vertical direction of the imaging devices 10a, 10b, 10c, and 10d is determined. may be fixed so as to overlap in the vertical direction.

The imaging device 10 a generates a captured image by capturing an image of the front periphery of the main body 12 . The imaging device 10 b generates a captured image by capturing an image of the left side periphery of the main body 12 . The imaging device 10 c generates a captured image by capturing an image of the right side periphery of the main body 12 . The imaging device 10d generates a captured image by imaging the rear periphery of the main body 12 .

As shown in FIG. 4, part of the field of view Fa of the imaging device 10a may overlap with part of the field of view Fb of the imaging device 10b and part of the field of view Fc of the imaging device 10c. Part of the field of view Fd of the imaging device 10d may overlap with part of the field of view Fb of the imaging device 10b and part of the field of view Fc of the imaging device 10c.

As shown in FIG. 2, image processing device 14 includes image interface 15 and processor 18 . The image processing device 14 may further include a storage section 16 and a transmission section 17 .

The image interface 15 receives various information and commands from an external device 19 of the image processing device 14 . The image interface 15 acquires a captured image from the imaging unit 13, for example.

The storage unit 16 includes arbitrary storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory). The storage unit 16 stores various programs that cause the processor 18 to function and various information that the processor 18 uses.

The storage unit 16 may store a coordinate conversion formula or a conversion table from two-dimensional image coordinates and depths to three-dimensional coordinates based on the position of the imaging unit 13 . The three-dimensional coordinates may have the longitudinal direction, width direction, and vertical direction of the main body 12 as axes. The image coordinates may have the vertical direction and the horizontal direction of the captured image of the imaging unit 13 as axes. The depth is the distance from the imaging unit 13 of the moving body 11 to the object point corresponding to each pixel of the captured image. A conversion formula or a conversion table may be created based on a predetermined position and a predetermined posture for fixing the imaging unit 13 to the main body 12 .

The transmission unit 17 transmits information generated by image processing by the processor 18 to be described later to the external device 19 .

Processor 18 includes one or more processors and memory. The processor may include a general-purpose processor that loads a specific program to perform a specific function, and a dedicated processor that specializes in specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). A PLD may include an FPGA (Field-Programmable Gate Array). Processor 18 may be either a System-on-a-Chip (SoC), in which one or more processors work together, and a System In a Package (SiP).

Processor 18 estimates depth at multiple locations in a single captured image corresponding to the image it receives. Processor 18 may perform depth estimation by any method. Processor 18 may perform depth estimation using, for example, a depth estimation model constructed by machine learning. The processor 18 may estimate, for example, the depth DEPTH_Y of the object point corresponding to each pixel of the captured image from the imaging unit 13 of the moving object 11 . Instead of performing depth estimation, the processor 18 may use depth detected using a ranging sensor such as a LiDAR, an ultrasonic sensor, or the like.

Processor 18 transforms the depth and image coordinates at multiple locations into three-dimensional coordinates corresponding to each of the multiple locations. Specifically, the processor 18 uses the image coordinates (u, v) of each of the plurality of locations, in other words, each pixel, and the depth estimated for each of the plurality of locations, to convert to three-dimensional coordinates. For example, the processor 18 calculates a three-dimensional coordinate unit vector (ex, ey, ez) from the image coordinates using a conversion formula. The processor 18 calculates a scale (actual distance) parameter K by dividing the estimated depth DEPTH_Y by the depth direction component coordinate ey in the three-dimensional coordinate unit vector. K indicates the ratio between the unit vector of the three-dimensional coordinates and the position vector. The processor 18 calculates three-dimensional coordinates by multiplying all components (ex, ey, ez) of the unit vector by K.

The processor 18 generates a viewpoint-changed image whose viewpoint is different from that of the captured image based on the three-dimensional coordinates. The processor 18 associates the pixel value of each pixel with the calculated three-dimensional coordinates. The processor 18 transforms the calculated three-dimensional coordinates into two-dimensional coordinates of the viewpoint-changed image viewed from a specific direction corresponding to the viewpoint-changed image. The processor 18 generates the viewpoint-changed image by arranging the pixel values associated with the three-dimensional coordinates in the two-dimensional coordinates of the viewpoint-changed image. The viewpoint of the viewpoint-changed image, for example, faces vertically downward from vertically above the moving object 11 as shown in FIGS.

The processor 18 displays a portion hidden by the subject 30 in the captured image (hereinafter referred to as an “invisible portion”) in the viewpoint-changed image in a clear manner. Specifically, as described above, the processor 18 moves each pixel forming the captured image to the corresponding position in the top view image. The processor 18 recognizes an area Ra, which is generated in a part of the top-view image and in which pixels constituting the image do not exist, as an invisible portion. The processor 18 makes the region Ra identified as the invisible portion clearly indicated by coloring or the like.

The processor 18 may display an enlarged view of the subject 30 that produces the region Ra identified as the invisible portion as viewed from above the front. Specifically, referring to FIG. 3, the processor 18 enlarges and displays the portion closest to the moving body 11 in the area Ra and the vicinity 31 of the portion. A subject 30 appears in the vicinity 31 of the relevant portion. Therefore, the driver can see the magnified display and recognize the details of the subject 30 .

The processor 18 may generate a plurality of viewpoint-changed images from each of the plurality of captured images. A plurality of viewpoint-changed images may have the same viewpoint. Processor 18 may combine the multiple viewpoint-changed images to generate a single viewpoint-changed image shown in FIG.

An example of the process by which the processor 18 generates a single view-changed image is shown below. As described above, the processor 18 generates four viewpoint-changed images of the same viewpoint from the captured images of the four imaging devices 10a, 10b, 10c, and 10d. The processor 18 creates a composite image using pixels of at least one of the two partially overlapping viewpoint-changed images among the four viewpoint-changed images. When using the pixels of the two viewpoint-changed images for the overlapping portion, the processor 18 may determine the average value, weighted average value, or the like of the pixel values of the two pixels as the pixel value of the pixel.

Even in an image obtained by synthesizing a plurality of viewpoint-changed images as shown in FIG. 4, there is a region R' in which there are no pixels constituting the image. The processor 18 recognizes the region R' as an invisible portion, and makes it visible by coloring or the like in the top view image.

In this embodiment, the imaging range of one of the imaging devices 10a, 10b, 10c, and 10d overlaps the imaging range of the other imaging devices. By overlapping the imaging ranges while the imaging positions are different, the processor 18 can interpolate the part located behind the subject in the captured image of one imaging device based on the other captured image. Specifically, in the synthesized image of a plurality of viewpoint-changed images, the processor 18 performs synthesis as described above, so that the part located behind the subject 30 in the image captured by the imaging device 10a is replaced by the image captured by the other imaging device 10b. Complementation may be performed based on the captured image. In the viewpoint-changed image corresponding to the imaging device 10a, the region Rao, which is a part of the region Ra identified as the invisible portion, is complemented by the viewpoint-changed image corresponding to the imaging device 10b. Further, a partial region Rbo of the region Rf recognized as an invisible portion in the viewpoint-changed image corresponding to the imaging device 10b is complemented by the viewpoint-changed image corresponding to the imaging device 10a. The processor 18 may cause the regions Rao and Rbo to be highlighted, for example by coloring differently than the invisible portion R'.

The processor 18 generates a top front view of the subject 30 that produces the region R' identified as the invisible portion, as shown in FIG. Specifically, the processor 18 enlarges and displays the portion closest to the moving object 11 and the vicinity 31 of the portion in the region R'.

The processor 18 may recognize the type of road surface from the captured image. Road surfaces are, for example, roadways and sidewalks. Processor 18 may indicate to the user a region Rf indicating a sidewalk, such as the horizontal stripes in FIGS. Recognizing the type of road surface may be done by any method. The processor 18 may perform recognition using, for example, road surface height information obtained from the three-dimensional coordinates described above. In general, there is a height difference between the carriageway and the non-carriageway surface. The non-vehicular road surface is generally higher than the roadway. The processor 18 may take advantage of the difference in elevation between vehicular and non-vehicle road surfaces to improve the accuracy of road surface type recognition.

Next, processing for generating a viewpoint-changed image, which is executed by the processor 18 in this embodiment, will be described using the flowchart of FIG. The processor 18 starts the process of generating a viewpoint-changed image each time an image of one frame is acquired from the imaging unit 13, for example.

In step S100, the processor 18 causes the imaging unit 13 to capture an image. The processor 18 acquires the captured image generated by the imaging unit 13 via the image interface 15 . In step S101, the processor 18 performs depth estimation on the received captured image. In step S102, the processor 18 performs three-dimensional coordinate transformation using the depth estimated in step S101 for each pixel of the captured image. In step S103, the processor 18 generates a viewpoint-changed image whose viewpoint is different from that of the captured image based on the three-dimensional coordinates. In step S104, the processor 18 recognizes the type of road surface from the captured image.

In step S105, the processor 18 generates a plurality of viewpoint-changed images for each of the plurality of captured images, and combines the plurality of viewpoint-changed images to generate a single viewpoint-changed image. At step S106, the processor 18 identifies invisible portions in the captured image. Further, the processor 18 colors the regions identified as invisible in the viewpoint-changed image.

In a configuration in which a viewpoint-changed image is generated based only on two-dimensional image coordinates, a tall object may be distorted and displayed large. On the other hand, the image processing device 14 of the present embodiment estimates depths at a plurality of locations in a captured image, and converts the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations. , a viewpoint-changed image having a viewpoint different from that of the captured image is generated based on the three-dimensional coordinates. With such a configuration, the image processing device 14 generates a viewpoint-changed image based on the height of the object. Therefore, the image processing device 14 can provide a viewpoint-changed image with reduced distortion.

Further, the image processing apparatus 14 of the present embodiment generates an image in which the viewpoint-changed image is displayed so as to indicate that it is an invisible portion, that is, that it is a portion hidden by the subject in the captured image. With such a configuration, the image processing device 14 can make the driver pay attention to the relevant portion.

In addition, the image processing apparatus 14 of the present embodiment complements the portion located behind the subject in one captured image among the plurality of captured images based on the plurality of captured images other than the one captured image. . Therefore, the image processing device 14 can complement each other with the viewpoint-changed images with reduced distortion, and can create an image with less invisible regions with reduced distortion.

Further, the image processing apparatus 14 of the present embodiment generates a plurality of the viewpoint-changed images for each of the plurality of captured images, and combines the plurality of viewpoint-changed images to generate a single viewpoint-changed image. With such a configuration, the image processing device 14 can generate a viewpoint-changed image that displays a wider area.

Although the present invention has been described with reference to the drawings and examples, it should be noted that various variations and modifications will be readily apparent to those skilled in the art based on this disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention.

10a, 10b, 10c, 10d Imaging device 11 Moving body 12 Main body 13 Imaging unit 14 Image processing device 15 Image interface 16 Storage unit 17 Transmission unit 18 Processor 19 External device 30 Subject 31 Enlarged portion Fa, Fb, Fc, Fd Field of view R1, R2, R' Area recognized as invisible area Rao, Rbo Complemented area Rf Area showing sidewalk

Claims (7)

an image interface for acquiring a captured image;
estimating depths at a plurality of locations in the captured image; transforming the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations; and determining the captured image based on the three-dimensional coordinates. An image processing device comprising: a processor that generates a viewpoint-changed image with a different viewpoint. The image processing device according to claim 1,
The processor generates an image displayed in the viewpoint-changed image so as to indicate a portion hidden by a subject in the captured image. The image processing device according to claim 1 or 2,
The captured image acquired by the image interface includes a plurality of captured images that partially overlap each other and are captured from different positions,
The image processing device, wherein the processor interpolates a portion located behind a subject in one captured image among the plurality of captured images based on the plurality of captured images other than the one captured image. In the image processing device according to claim 3,
The processor generates a plurality of the viewpoint-changed images for each of the plurality of captured images, and combines the plurality of viewpoint-changed images to generate a single viewpoint-changed image. obtaining a captured image;
estimating depths at a plurality of locations in the captured image; transforming the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations; and determining the captured image based on the three-dimensional coordinates. and generating a viewpoint-changed image with a different viewpoint. an imaging unit that generates an image signal by imaging;
estimating depths at a plurality of locations in a captured image corresponding to the image signal, converting the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations, and based on the three-dimensional coordinates and a processor that generates a viewpoint-changed image whose viewpoint is different from that of the captured image. the main body;
an imaging unit that is fixed at a predetermined position on the main body in a predetermined posture and that generates an image signal by imaging;
estimating depths at a plurality of locations in a captured image corresponding to the image signal, converting the depths and image coordinates at the plurality of locations into three-dimensional coordinates corresponding to the plurality of locations, and based on the three-dimensional coordinates a processor that generates a viewpoint-changed image having a viewpoint different from that of the captured image.

Images