JP7293862B2 - Mobile display system, image processing device and image processing program - Google Patents

Description

The present invention relates to a mobile object display system, an image processing apparatus, and an image processing program, for example, a mobile object display system that synthesizes images captured by a plurality of cameras attached to a mobile object and displays a synthesized image viewed from a virtual viewpoint. It can be applied to

For example, Patent Document 1 and Patent Document 2 are known as technologies for synthesizing images captured by a plurality of cameras attached to a moving body such as a vehicle and displaying the synthesized image when viewed from a virtual viewpoint on a display unit such as a monitor. and so on.

In the technique described in Patent Document 1, in order to assist the driver in confirming safety when driving a vehicle, for example, a virtual viewpoint is set above the own vehicle, and the pixels of the images captured by each camera and the pixels of the synthesized image are combined. Using a mapping table that describes the correspondence between , images captured by each camera are mapped to generate a composite image. The virtual viewpoint can be freely set and switched, and a mapping table corresponding to the position of the virtual viewpoint is used. In addition, by synthesizing an illustration image in the area where the own vehicle exists in the composite image, an illustration image showing the own vehicle and obstacles existing around it can be displayed. It is possible to assist safety confirmation in such a case.

The technique described in Patent Document 2 avoids discontinuity in the joint area of the image areas when combining images captured by a plurality of cameras, and displays an easy-to-see image. Specifically, when mapping the two-dimensional captured images of each camera onto a three-dimensional projection plane shape, conversion information when converting to projection image data viewed from the viewpoint or line-of-sight direction, and a method of synthesizing overlapping portions of image regions. is constantly changed according to vehicle state information and the like.

As described above, the conventional techniques such as Patent Document 1 and Patent Document 2 are images showing the situation around the vehicle, and can display, for example, obstacles around the vehicle, white lines for parking, and the like.

International publication WO00/64175 JP 2010-128951 A

However, for example, when used for a vehicle running on a road surface, the actual image of the contact area between the road surface and the tire, or for example, when used for a work vehicle such as a bulldozer that scrapes off earth and sand, the earth and sand and the blade part. There is a problem that the actual image of the contact area (work contact area) cannot be projected as a composite image.

Therefore, when a plurality of cameras are attached to a mobile body that is running or driving, images captured by the plurality of cameras are synthesized, and a synthesized image viewed from a virtual viewpoint is displayed, the moving body and the moving plane are separated from each other. There is a demand for a mobile object display system, an image processing device, and an image processing program capable of projecting an actual image of the contact area of the user.

In order to solve such a problem, a moving body display system according to a first aspect of the present invention includes (1) a plurality of cameras attached to a moving body for imaging the surroundings of the moving body, and (2) a plurality of cameras. (3) a synthetic peripheral image generator that generates a synthetic peripheral image when viewed from a changeable virtual viewpoint based on the captured image; , a synthetic moving body image generating unit for generating a synthetic moving body image including a contact area vicinity image including a contact area between a moving plane of the moving body and the moving body, and a predetermined moving body model; and (4) a synthetic moving body and a display unit for displaying the synthetic moving object image generated by the image generation unit.

An image processing apparatus according to a second aspect of the present invention provides: (1) a synthesized peripheral image viewed from a changeable virtual viewpoint based on a plurality of captured images obtained from a plurality of cameras that capture images of the surroundings of a moving object; and (2) a contact area vicinity image including a contact area between the moving body and the moving body, which is included in the images captured by all or some of the plurality of cameras. , and a synthetic moving body image generation unit that generates a synthetic moving body image including a predetermined moving body model.

An image processing program according to a third aspect of the present invention provides a computer with: (1) synthesis when viewed from a changeable virtual viewpoint based on a plurality of captured images obtained from a plurality of cameras capturing the surroundings of a moving body; (2) a contact area including a contact area between a plane of movement of a moving body and a contact area of the moving body, which is included in images captured by all or some of the plurality of cameras; It is characterized by functioning as a synthetic moving body image generation unit that generates a synthetic moving body image including a neighborhood image and a predetermined moving body model.

According to the present invention, when a plurality of cameras are attached to a mobile body that is running or driving, images captured by the plurality of cameras are synthesized, and a synthesized image viewed from a virtual viewpoint is displayed, the moving body and the , can project a real image of the contact area with the moving plane.

1 is a configuration diagram showing the overall configuration of a mobile display system according to an embodiment; FIG. 1 is a plan view of a vehicle showing a method of installing a plurality of cameras in an embodiment; FIG. It is a left side view of vehicles showing a method of installing a plurality of cameras in an embodiment. It is a figure of the back part of vehicles which shows the installation method of a plurality of cameras in an embodiment. It is a figure which shows an example of the captured image imaged with the camera which concerns on embodiment. 7 is a flowchart showing the operation of moving object display processing in the moving object display system according to the embodiment; FIG. 10 is an explanatory diagram illustrating a method of adjusting boundary angles in a synthesized image according to the embodiment; FIG. 10 is an explanatory diagram for explaining a method of setting a mask area according to the embodiment; FIG. 4 is an explanatory diagram illustrating pasting of a vehicle model to a mask area according to the embodiment and conventional technology; FIG. 9A is a plan view and a left side view of a vehicle showing a method of installing a plurality of cameras in a modified embodiment; It is a figure which shows an example of the captured image imaged with the camera which concerns on modified embodiment.

(A) Main Embodiments Hereinafter, embodiments of a mobile display system, an image processing apparatus, and an image processing program according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of Embodiment (A-1-1) Configuration of Mobile Display System FIG. 1 is a configuration diagram showing the overall configuration of a mobile display system according to this embodiment.

In FIG. 1, a mobile object display system 1 according to this embodiment has a plurality of (four in FIG. 1) cameras 11 (11-1 to 11-4), an image processing device 12, and a display unit 13.

The moving object display system 1 synthesizes captured images captured by a plurality of cameras 11-1 to 11-4 attached to a moving object, and displays a bird's-eye synthesized image viewed from a virtual viewpoint on a display unit 13. indicate.

As will be described later, after the final synthetic image is generated, the position of the virtual viewpoint can be set or changed to any position in the coordinate system of the synthetic image. Therefore, it is possible to display a bird's-eye synthetic image including the state of the running moving body itself from the set virtual viewpoint.

Here, the type of mobile body is not particularly limited, and can be applied to a wide variety of things. can do. More specifically, the automobile may be an automobile capable of traveling on a general road, or a vehicle construction machine such as a bulldozer, an excavator, a wheel loader, a road roller, or a vehicle such as a tractor. Agricultural machines and the like (hereinafter, vehicles including vehicle-type construction machines and vehicle-type agricultural machines are also referred to as "working vehicles"). Also, the vessel may be a general vessel such as a yacht or a cruiser, or a special vessel such as a fire boat. Furthermore, the mobile body may be a manned mobile body in which a person rides in and operates the mobile body, or a remote-controlled (unmanned) mobile body in which the mobile body is operated remotely. can be When the present invention is applied to a vehicle, the plane of movement is the road surface or the like on which the vehicle is running. When the present invention is applied to a ship, the surface of the water on which the ship floats is the moving plane, and the image near the contact area is an image including the hull and the water surface near the waterline of the ship. In this embodiment, the moving object is a vehicle such as an automobile, and is described as being remotely operable.

The cameras 11 - 1 to 11 - 4 are attached to automobile vehicles (hereinafter referred to as “vehicles”) as moving bodies, capture images of the surroundings of the vehicles, and provide the captured images to the image processing device 12 .

The number of cameras 11-1 to 11-4 is not limited to four. Two or three cameras 11 may be used, or five or more cameras 11 may be used. In this embodiment, it is assumed that images are taken in the front direction, the rear direction, the right direction, and the left direction of the vehicle, and four cameras 11-1 to 11-4 are attached to the vehicle in order to take images in each of these four directions. Illustrate the case.

For the cameras 11-1 to 11-4, for example, a camera using a fisheye lens or a camera using a wide-angle lens can be applied. More specifically, each camera 11-1 to 11-4 has an angle of view of about 100° or more, so that each camera 11 captures the surroundings of the vehicle. can be captured with sufficient coverage.

In addition, in order to actually generate a composite image by synthesizing the captured images of the respective cameras 11, it is necessary to adjust the installation position of the camera 11 attached to the vehicle, the imaging angle of the camera 11, and the like. Further, when synthesizing the captured images of each camera 11, it is necessary to calibrate the internal parameters and external parameters of each camera 11 in order to accurately synthesize images at the joints of the images.

The image processing device 12 receives each captured image from each of the plurality of cameras 11-1 to 11-4, synthesizes the two-dimensional captured images from each of the cameras 11-1 to 11-4, and produces a three-dimensional image. It generates a composite image.

The hardware for performing the processing of the image processing device 12 can be, for example, an integrated circuit device having a computer, ROM, RAM, EEPROM, etc., and the computer stores a processing program (for example, an image processing program) stored in the ROM. etc.), various functions of the image processing apparatus 12 are realized.

As shown in FIG. 1 , the image processing device 12 has a synthetic peripheral image generator 121 and a synthetic image generator 122 .

Synthetic peripheral image generation unit 121 acquires images (captured images) showing the vehicle peripherals captured by a plurality of cameras 11-1 to 11-4, synthesizes the captured images, and produces an omnidirectional image of the vehicle peripherals. to generate a first composite image of .

Here, various image processing methods can be applied to the method of generating the first synthetic image by the synthetic peripheral image generation unit 121 . For example, a three-dimensional model is prepared in advance, and the captured images (two-dimensional images) captured by a plurality of cameras 11-1 to 11-4 are coordinate-transformed into a three-dimensional model, resulting in a three-dimensional image. can be used to generate the first composite image. More specifically, for example, a three-dimensional model formed on the spherical surface of a downwardly convex hemisphere is used. The curved surface angle of the three-dimensional model of the hemispherical surface may be appropriately set according to the first synthesized image to be synthesized. A mesh is formed in the three-dimensional model, and coordinate values (eg, X-axis value, Y-axis value, and Z-axis value) of each pixel in the coordinate system of the three-dimensional model are set. In addition, for the image captured by each camera, the coordinate values (x-axis value, y-axis value) of the pixel in the coordinate system of the camera image are set. Each pixel of the image captured by each camera is mapped to the corresponding pixel in the three-dimensional model by projectively transforming the coordinates of each pixel of the dimensional model. By doing so, each pixel of the images captured by the cameras 111-1 to 111-4 is mapped to each pixel of the three-dimensional model to generate the first composite image.

Synthetic image generation unit 122 generates a contact area neighborhood image including a contact area between a traveling vehicle portion and a moving plane on which the vehicle is traveling, in the first synthetic image generated by synthetic peripheral image generation unit 121. It has a contact area neighborhood image generator 122a.

Here, the contact area vicinity image is intended to be an image including an image area including a portion (contact portion) where the contact target and a portion of the moving body are in contact with each other and the vicinity thereof. Further, a contact target refers to a target that a moving body or a part of the moving body is in contact with or may contact. The contact target includes, for example, a moving body that is moving or a moving plane that a part of the moving body is in contact with, and an object that a work vehicle as a moving body acts on as work work.

In this embodiment, a case where the contact target is a moving plane will be described as an example. By plane of motion is intended the plane associated with the moving object in motion, as exemplified below.

For example, when a vehicle traveling on a road is a moving object, the road surface on which the vehicle is traveling is a moving plane, and the road surface, etc., and the tires, etc., of the traveling vehicle are in contact with each other. The region including the portion where the contact is located and the portion near the contact region becomes the contact region vicinity image. Further, for example, when a ship running on water is used as a moving body, the surface of the water on which the ship is running becomes a movement plane, and the hull near the waterline of the ship floating on the water surface is in contact with the water surface. A region including the neighboring portion becomes the contact region neighboring image.

Further, for example, in the case where the moving body is a work vehicle such as a road roller or a bulldozer that performs work work, the road surface on which the work vehicle such as the road roller or bulldozer is running and/or the work vehicle that performs the work work The object to be worked on (for example, when a bulldozer pushes through earth and sand, the earth and sand that is the object to be worked on) is the contact object. The area including the portion where the contact area is located becomes the image near the contact area. When the work vehicle is a moving body, whether the contact target is a moving plane such as a road surface and/or the work target can be determined appropriately according to the purpose. Depending on the purpose, it is also possible to appropriately determine which part of the object and the part of the moving body that contacts it (that is, the image near the contact area).

For example, when remotely controlling a vehicle-type construction machine such as a road roller or a bulldozer, it is possible to visualize the construction work, such as the vehicle leveling the road surface or excavating. It is required that the operator can operate remotely while grasping it. Further, for example, not limited to this, in the driving operation of a vehicle, it may be necessary to operate the vehicle after confirming the details of how the vehicle is acting on the moving plane. is desired to display an image including the contact area between the moving plane and the vehicle and its vicinity.

However, according to the conventional technology, although a plurality of 11-1 to 11-4 mounted on a vehicle can capture an image of a moving plane (for example, the road surface), the captured image including the moving plane is a little away from the own vehicle. It is a moving plane (for example, a road surface) that exists at a distant position, and it is not possible to capture a portion where the moving plane and a part of the vehicle are in contact. Therefore, the composite image obtained by synthesizing the respective captured images can also show a moving plane at a position slightly away from the vehicle, but the contact area including the portion where the moving plane and a part of the vehicle are in contact with each other. Images cannot be displayed.

Therefore, in this embodiment, the synthetic image generation unit 122 secures a contact area vicinity image including a contact portion where the moving plane and a part of the vehicle are in contact, and then generates a synthetic image.

Further, the composite image generation unit 122 has a mask area determination unit 122b and a vehicle model formation unit 122c. In the first synthetic image generated by the synthetic peripheral image generation unit 121, the mask area determination unit 122b effectively functions the area of the contact area vicinity image described above, and then, among the areas where the own vehicle exists, An area to be hidden in the final synthesized image is determined as a mask area. In addition, the vehicle model forming unit 122c determines the mask area determined by the mask area determining unit 122b by pasting, for example, a vehicle model provided in advance by CG (computer graphics) or the like. The composite image generation unit 122 provides the display unit 13 with the second composite image finally generated in this way.

The display unit 13 can employ, for example, a display, and outputs (displays) the composite image generated by the composite image generation unit 122 .

Here, the composite image generated by the image processing device 12 may be provided to the display unit 13 through a line such as a cable, or may be provided to the display unit 13 through a communication line. good. More specifically, for example, the display unit 13 may be mounted on a vehicle, provided near the image processing device 12, and the synthesized image may be provided to the display unit 13 via a cable or the like. Thereby, the driver of the vehicle can be shown a composite image of the surroundings of the vehicle. Further, for example, when a composite image is given to the display unit 13 through a communication line such as a wired line or a wireless line, the composite image is displayed on the display unit 13 installed remotely from the vehicle. In this case, communication means for transmitting/receiving the synthesized image is required, but since this communication means is out of the scope of the present invention, detailed description thereof is omitted here.

(A-1-2) Camera Mounting Method Next, an installation example for mounting a plurality of cameras 11-1 to 11-4 on a vehicle will be described in detail with reference to the drawings.

Below, regarding the vehicle 2 traveling on the road surface, the state of contact between the road surface 3 and the tires 21-1 and 21-3 on the right side of the vehicle 2, and the road surface 3 and the tire 21 on the left side of the vehicle 2 A description will be made on the assumption that a composite image (second composite image) capable of capturing contact with 21-2 and 21-4 is generated.

In other words, the state in which the road surface 3 and the tires 21-1 to 21-4 of the vehicle 2 are in contact with each other can be displayed, and the objects included in the contact area vicinity image are the road surface 3 and the tires 21-1 to 21-4. 21-4 is illustrated.

FIG. 2 is a plan view of a vehicle showing how the cameras 11-1 to 11-4 are installed in the embodiment, and FIG. 3 shows the installation method of the cameras 11-1 to 11-4 in the embodiment. 4 is a left side view of the vehicle shown, and FIG. 4 is a view of the rear part of the vehicle showing a method of installing a plurality of cameras 11-1 to 11-4 in the embodiment. An arrow A shown in FIGS. 2 to 4 indicates the front of the vehicle 2. As shown in FIG.

As shown in FIGS. 2 to 4, of the cameras 11-1 to 11-4, the camera 11-1 is provided in front of the vehicle 2, the camera 11-2 is provided in the rear of the vehicle 2, and the camera 11- 3 is provided on the right side of the vehicle 2, and the camera 11-4 is provided on the left side of the vehicle 2. FIG.

In FIG. 4, the camera 11-3 on the right side of the vehicle 2 is provided relatively to the front side of the vehicle 2, and the camera 11-4 on the left side of the vehicle 2 is provided relatively to the rear side of the vehicle. However, this can be adjusted according to the images of the respective cameras 11-3 and 11-4, and is not limited to such arrangement.

Of the cameras 11-1 to 11-4, the camera 11-3 captures the state of contact between the tires 21-1 and 21-3 of the vehicle 2 and the road surface 3, and the camera 11-4 captures the tires of the vehicle. 21-2 and the tire 21-4, and the state of contact with the road surface 3 are imaged.

The installation positions of the cameras 11-1 to 11-4 on the vehicle 2 are not particularly limited as long as they are positions capable of imaging the state of contact between the tires 21-1 to 21-4 of the vehicle 2 and the road surface 3. In the embodiment, the case where the extension members 22-1 to 22-4 are installed on the roof of the vehicle 2 is exemplified. Specifically, the extension members 22-1 to 22-4 extending outward of the vehicle 2 are attached to the roof of the vehicle 2, and the camera 11-1 is attached to the tip side of the extension members 22-1 to 22-4. Fix ~11-4.

At this time, each of the cameras 11-1 to 11-4 fixed to the tip side of each of the extension members 22-1 to 22-4 has an angle (angle of view) related to the imaging range of each of the cameras 11-1 to 11-4. may be adjusted. That is, in this example, at least the camera 11-3 installed on the right side of the vehicle 2 captures an image including the road surface 3 and the tires 21-1 and 21-3, and the camera installed on the left side of the vehicle 2 11-4 is desired to capture an image including road surface 3 and tires 21-2 and 21-4.

Therefore, as illustrated in FIG. 4, when installing the cameras 11-1 to 11-4, at least the camera 11-3 (or the camera 11-4) is positioned on the road surface 3, The outward protrusion distance and installation of the camera are such that an image including the tires 21-1 and 21-3 (or the tires 21-2 and 21-4) and the contact area R can be constantly captured while the vehicle is running. The angles are adjusted so that the angles of view of the cameras 11-3 and 11-4 can be adjusted.

FIG. 5 is a diagram showing an example of a captured image captured by the camera 11-3 according to the embodiment.

As shown in FIG. 5, the vehicle 2 in FIG. 3 includes a contact area R1 between the road surface 3 and the tire 21-1 and a contact area R2 between the road surface 3 and the tire 21-3.

More specifically, the rectangular portion existing on the lower side of the captured image illustrated in FIG. 5 is an image corresponding to the right side of the vehicle 2, and above the image of the right side of the vehicle 2 , images of tires 21-1 and 21-3 in contact with the road surface 3 are shown.

As shown in FIG. 5, as the vehicle 2 travels, the tire 21-1 and the tire 21-3 reflected in the contact areas R1 and R2 are shown rotating. It is possible to recognize how the tires 21 - 1 and 21 - 3 , which are on the road surface 3 , act on the road surface 3 .

Although FIG. 5 shows an example of an image captured by the camera 11-3 installed on the right side of the vehicle, the image captured by the camera 11-4 installed on the left side of the vehicle 2 also includes the image of the contact area R. be able to

(A-2) Operation of Embodiment Next, the operation of mobile object display processing in the mobile object display system 1 according to this embodiment will be described with reference to the drawings.

FIG. 6 is a flowchart showing the operation of mobile object display processing in the mobile object display system 1 according to the embodiment.

[Step S101]
First, a plurality of cameras 11-1 to 11-4 are attached to the vehicle 2, and some of the cameras 11-1 to 11-4, cameras 11-3 and 11-4, 21-4 is adjusted (step S101).

In this example, extension members 22 (22-1 to 22-4) are fixed to the front, rear, right and left sides of the roof of the vehicle 2 so as to extend outward from the vehicle 2, and the extension members The corresponding cameras 11 (11-1 to 11-4) are fixed on the tip side of 22-1 to 22-4. By providing the extension members 22-1 to 22-4 in this manner, the cameras 11-1 to 11-4 can capture and image the state of contact between the road surface 3 and the tires 21, and the like.

In this embodiment, the extension member 22 is provided so that the road surface 3 as a moving plane and the tires 21 are reflected within the imaging angle of view of the camera 11 as an example. However, since the angle of view of the camera 11 is 100° or more as in a fish-eye camera or a wide-angle camera, the road surface 3 as a moving plane and the tires 21 are displayed in the image of the camera 11 without providing the extension member 22. If included, the extension member 22 may be omitted. On the contrary, for example, when the mobile body is a work vehicle, and the driving operation is performed while watching the state of road surface leveling or excavation, the extension member 22 is provided and the tip of the extension member 22 is provided. The extension member 22 functions effectively in that the camera 11 can be fixed to the side and installed at a position where the camera 11 can look into the contact portion between the contact target and the work site that can contact the contact target. do. In particular, although automobiles traveling on general roads are subject to regulations regarding loads such as long objects, it is possible to provide the extension member 22 to the working vehicle, so when the mobile body is a working vehicle, it is particularly useful. It is valid.

Also, in step S101, the camera angles of the cameras 11 (11-1 to 11-4) provided on the tip side of the extension members 22 (22-1 to 22-4) are adjusted. In this example, the cameras 11-3 and 11-4 on the right and left sides of the vehicle 2 capture contact conditions between the road surface 3 and the tires 21-1 to 21-4. Adjust the camera angle of In addition, in the synthesized image (second synthesized image), the camera angles of the front camera 11-1 and the rear camera 11-2, which do not capture the contact situation, are also adjusted in order to create an image without distortion at the joint of the camera-captured images. It is desirable to

[Step S102]
The contact area R is identified by confirming whether or not the contact area R is captured in the images captured by the cameras 11-3 and 11-4 on the right and left sides of the vehicle 2 (step S102). For example, it is confirmed whether or not the target contact regions R1 and R2 are reflected in the captured image of the camera 11-3 as illustrated in FIG. 5, and specified.

For example, the position coordinate values of the target contact area R may be specified in the captured image of the camera 11-3 capturing the contact area R (R1 and R2). More specifically, in the captured image, the values of the position coordinates (coordinate values) of the four corners of the rectangular area including the contact area R are specified, or the outer circumference (boundary line) of the contact area R is specified along the image area. It is also possible to specify an area by using By doing so, an area image including the contact area R can be specified in the coordinate system of the two-dimensional image by the camera 11 .

[Step S103]
Predetermined calibration processing is performed when generating a composite image (first composite image) in composite peripheral image generation unit 121 (step S103).

In the predetermined calibration process, when synthesizing images captured by a plurality of cameras 11-1 to 11-4 to generate a three-dimensional synthetic image (first synthetic image) around the vehicle, two This is processing for associating the coordinate values of pixels in the dimensional coordinate system with the coordinate values of pixels in the 3D coordinate system of the 3D model. As a result, it is possible to suppress image distortion that may occur at a joint, which is a boundary between a given image area and an adjacent image area, in the synthesized image.

[Step S104]
Next, the captured images of the plurality of cameras 11-1 to 11-4 are input to the combined peripheral image generation unit 121, and the combined peripheral image generation unit 121 converts each pixel of the four captured images (two-dimensional image data) into Each pixel of a three-dimensional model (mesh shape model) formed on the spherical surface of a hemisphere is mapped to generate a first three-dimensional synthetic image (step S104).

[Step S105]
The first synthetic image generated by the synthetic peripheral image generation unit 121 is provided to the synthetic image generation unit 122, and the contact area vicinity image generation unit 122a of the synthetic image generation unit 122 adjusts the boundary angle between the captured images. Then, a contact area vicinity image is generated (step S105).

The contact area vicinity image generation unit 122a adjusts the boundary angle between a captured image and an adjacent captured image for the four captured images so that the contact area vicinity image is displayed in the first synthesized image. .

FIG. 7 is an explanatory diagram illustrating a method of adjusting boundary angles in a synthesized image according to the embodiment.

Here, for ease of explanation, it is assumed that the position of the virtual viewpoint V is at a position extending upward from the central position of the vehicle 2 as shown in FIG. 7(A). In this case, a composite image of the vehicle 2 viewed from above from the position of the virtual viewpoint V is a diagram such as that illustrated in FIG. 7B. Note that in FIG. 7B, the boundary angle X between adjacent images of the front image, the rear image, the right image, and the left image is set so that the influence of the distortion of the outer peripheral portion of the lens of a fisheye camera, a wide-angle camera, or the like is A case of about 45° is illustrated for the purpose of reduction.

However, since the images capturing the contact area R are the right camera 11-3 and the left camera 11-4, when the boundary angle X is a constant value, the contact area vicinity image including the contact area R is , it may not be reflected correctly. For example, when part of the contact area R shown in the left image is included in the boundary between the front image and the left image, the image of the contact area R is distorted or part of the contact area R is missing. In some cases, the image quality of the image near the contact area cannot be ensured.

Therefore, in this embodiment, as shown in FIG. 7C, in the composite image, the boundary angle between the images captured by the cameras 11-3 and 11-4 that capture the contact area R is large. The generation unit 121 adjusts the boundary angle of each captured image.

For example, in step S102, in the image captured by the camera 11-4, the values of the position coordinates (coordinate values) of the four corners of the rectangular area defining the image area including the contact area R, the outer circumference of the contact area R (boundary line ) are specified. Further, by using the three-dimensional model, the coordinate values of the pixels of the two-dimensional captured image are projected and transformed into the coordinate values of the pixels of the three-dimensional composite image, so that the composite peripheral image generation unit 121 can perform three-dimensional composite In the coordinate system of the image, it is possible to recognize the coordinate values indicating the image area corresponding to the image area including the contact area specified in the captured image (that is, the area of the image near the contact area). That is, the combined peripheral image generator 121 can recognize the coordinate values of the contact area vicinity image in the three-dimensional coordinate system.

Therefore, based on the coordinate values indicating the area of the contact area vicinity image in the three-dimensional coordinate system of the synthesized image, if the area of the contact area vicinity image exists in the vicinity of the boundary, the synthetic surrounding image generation unit 121 Adjust so that the boundary angle of the captured image of the camera becomes large.

In other words, using the coordinate values indicating the area of the contact area vicinity image, the boundary angle of the captured image including the contact area vicinity image is set to be large, and conversely, the boundary angle of the captured image that does not include the contact area vicinity image is set. Set the angle to be small.

Note that the adjustment of the boundary angle of each captured image may be made manually by an operator or the like so that the image near the contact area is reflected in the synthesized image, for example. In that case, the coordinate values after adjustment (the coordinate values of the contact area vicinity image in the coordinate system of the combined image may be used, or the coordinate values including the contact area in the coordinate system of the captured image may be used). may be stored as set values. As a result, even when the position of the virtual viewpoint is changed, it is possible to display the contact area vicinity image with a different perspective in the composite image after the change when viewed from the virtual viewpoint after the change. The virtual viewpoint can be changed by providing the display unit 13 with an operation function as a touch panel or by preparing an operation interface (not shown).

[Steps S106, S107]
The mask area determination unit 122b of the synthetic image generation unit 122 effectively retains the area of the contact area vicinity image in the first synthetic image, and then, out of the area where the own vehicle exists, the non-mask area in the final synthetic image. The area to be displayed is set as a mask area (step S106).

Then, the vehicle model forming unit 122c pastes, for example, a vehicle model prepared in advance by CG or the like to the mask area determined by the mask area determining unit 122b (step S107).

Here, the vehicle model may be previously formed by CG or the like and held. The vehicle model may be, for example, an illustration image or an actual image. Alternatively, for example, a skeleton model may be used as part or all of a vehicle model such as an illustration image or an actual image. In this way, by using the skeleton for the portion of the vehicle model that is close to the image near the contact area, it is possible to clearly show the image near the contact area in the synthesized image. As a result, the viewer can reliably grasp the state of contact, for example, between the road surface 3 and the tire 21, which is displayed in the image near the contact area. Furthermore, for example, a surface model or a solid model may be used as a three-dimensional model of the own vehicle using CAD, CAM, or the like.

FIG. 8 is an explanatory diagram illustrating a method of setting a mask area according to the embodiment.

Since the cameras 11-1 to 11-4 do not capture the entire view of the vehicle, when the images captured by the cameras 11-1 to 11-4 are combined, the hatched portion M shown in FIG. A region that should be hidden (including, for example, a region without a captured image) is generated by projective transformation to the first composite image. Therefore, finally, the hatched portion M is set as a mask area (hereinafter referred to as a "mask area M") in order to generate a composite image with high image quality viewed from a virtual viewpoint.

Here, in this embodiment, the right camera 11-3 and the left camera 11-4 image contact portions between the road surface (moving plane) 3 as a contact target and the tires 21-1 to 21-4. Therefore, the size of the mask area M is generally smaller than the size of the area where the own vehicle exists, and the mask area M exists generally inside the area where the own vehicle exists. It is desirable that the mask area M includes at least an area to be hidden and excludes the contact area R. FIG. 8 exemplifies a case where the mask region M has a rectangular shape to simplify the explanation, but the mask region M is not limited to a rectangular shape. The mask area M may be set by setting the coordinate values that define the mask area M, or by designating the outer circumference (boundary line) of the mask area M, in consideration of the position of the contact area.

FIG. 9 is an explanatory diagram for explaining pasting of a vehicle model to a mask area according to the embodiment and the conventional technique.

FIG. 9(A) shows the relationship between the mask area and the vehicle model according to the prior art, and FIG. 9(B) shows the relationship between the mask area and the vehicle model according to this implementation liquid.

As shown in FIG. 9A, in the conventional case, since there are many areas to be hidden in the synthesized image, the mask area M is also large. Since the vehicle model 51 is attached to the composite image so as to cover the mask area M, the vehicle model 51 for masking the mask area M also becomes large, and for example, the area in contact with the road surface 3 can be viewed. It is not possible.

On the other hand, as shown in FIG. 9B, according to this embodiment, the size of the mask area M is generally smaller than the size of the area where the own vehicle exists, and the mask area M is generally It comes to exist inside the existence area of the self-vehicle. Therefore, even when the vehicle model 52 is attached to the mask area M, the contact area vicinity image R is displayed as the actual image in the composite image, so that the viewer can view the contact area vicinity image R as the actual image. can.

[Steps S108 and S109]
The second synthesized image generated by the image processing device 12 is provided to the display section 13 (step S108), and the second synthesized image is displayed on the display section 13 (step S109).

The second synthesized image generated by the image processing device 12 includes a bird's-eye view image including the running vehicle (vehicle model 52) viewed from a changeable virtual viewpoint, the road surface 3, and the tires of the vehicle 2. and a contact area vicinity image. Therefore, the vehicle 2 running on the road surface 3 can be viewed from above, and the contact state of a portion (tire) of the vehicle 2 that is in contact with the road surface 3 can be confirmed.

(A-3) Effect of Embodiment As described above, according to this embodiment, a plurality of cameras are attached to a mobile body that is running or driving, and images captured by the plurality of cameras are combined to generate a virtual image from a virtual viewpoint. A real image of the contact area between the moving object and the moving plane can be displayed when displaying the composite image as viewed.

(B) Other Embodiments Although various modified embodiments have been mentioned in the embodiments described above, the present invention is also applicable to the following modified embodiments.

(B-1) In the above-described embodiment, the moving object is a vehicle traveling on the road surface, and the actual image of the contact point between the road surface and the tire is displayed as a composite image. In this case, the camera on the right side and the camera on the left side of the vehicle are configured to image the contact portion between the tire and the road surface, but the present invention is not limited to this.

For example, as shown in FIG. 10, the present invention can also be applied to a remotely operated road roller 2A, in which a plurality of cameras are attached to generate a composite image. It is assumed that the remote operator remotely operates the road roller 2A while viewing the composite image (second composite image) through the display unit. A detailed description of the system for remotely controlling the road roller 2A is omitted. In this example, it is assumed that the front camera 111 captures an image of the contact area R between the roller portion 25, which is a working portion provided in front of the running road roller 2A, and the road surface 3. FIG.

11 is an image captured by the front camera 11-1, which is an image of the contact area R between the road surface 3 and the roller portion 25. The image captured by the front camera 11-1 illustrated in FIG. It is used as a partial image of the first synthesized image and the second synthesized image described in the above embodiment.

By generating the first synthetic image and the second synthetic image using the captured image of the front camera 11-1 illustrated in FIG. 11, the first synthetic image and the second synthetic image are generated. includes a bird's-eye view image including the traveling road roller 2A and an image near the contact area between the roller portion 25 of the road roller 2A and the road surface 3, as seen from a virtual viewpoint. Therefore, the road roller 2A running on the road surface 3 can be viewed from a virtual perspective, and the details of the work of the road roller 2A for leveling the road surface 3 can be viewed.

Since the contact area vicinity image R illustrated in FIG. 11 is included in the composite image, the road surface 31 that has already been leveled by the roller unit 25 and the road surface 3 that has not yet been leveled among the road surfaces 3 to be leveled by the road roller 2A. can be confirmed through the composite image. Therefore, it is possible to check the details of the leveling condition of the road surface 3 by the roller portion 25 of the road roller 2A. In other words, while confirming the accuracy of the running condition, running direction, etc. of the road roller 2A running in a bird's-eye view from a virtual viewpoint, it is possible to simultaneously confirm the detailed conditions of the work performed by the road roller 2A, and perform remote control. A person can remotely control the road roller 2A.

Specifically, by projecting a composite image in which the road roller 2A is also projected from a changeable virtual viewpoint, the running direction of the running road roller 2A can be viewed from above. In addition, by projecting the image near the contact area synthesized with the final synthesized image, for example, the road roller 2A runs straight without meandering, or the road roller 2A repeats the same part. The remote operator can check whether the ground is not leveled or whether there is a leak in ground leveling, and can remotely operate the road roller 2A.

Further, by performing the same processing as in the above-described embodiment, for example, when the moving body is a ship, an image near the contact area (actual image) between the ship and the quay can be obtained, and the state of the ship berthed can be confirmed. can be

Alternatively, for an unmanned flying object such as an aircraft or a drone, an image (real image) near the contact area between the landing gear and the runway or landing floor can be obtained by performing the processing of the present invention on the area of the landing gear. , departure and arrival, etc. may be confirmed.

(B-2) In the above-described embodiments, various techniques can be widely applied to image processing for synthesizing a plurality of captured images to generate a synthesized image. For example, the techniques disclosed in Patent Literature 1 and Patent Literature 2 and their improved techniques can be applied.

Reference Signs List 1 mobile object display system 11 (11-1 to 11-4) camera 12 image processing device 121 synthetic peripheral image generator 122 synthetic image generator 122a near contact area image generator 122b... mask area determination unit, 122c... vehicle model formation unit, 13... display unit, 2 and 2A... moving object, R... contact area vicinity image.

Claims (6)

a plurality of cameras attached to a moving object for imaging the surroundings of the moving object;
a synthetic peripheral image generation unit that generates a synthetic peripheral image when viewed from a changeable virtual viewpoint based on the images captured by the plurality of cameras;
a contact area vicinity image including a contact area between the movement plane of the moving body and the moving body, which is included in the captured images of all or part of the plurality of cameras, and a predetermined moving body model; A synthetic moving object image generation unit that generates a synthetic moving object image including
and a display unit that displays the synthetic moving object image generated by the synthetic moving object image generation unit. the mobile body is a work vehicle,
The mobile object display system according to claim 1, wherein the contact area is a contact area between a work site of the work vehicle and the ground. 2. A moving body display system according to claim 1, wherein said moving body is a vehicle or a ship. the plurality of cameras are fisheye cameras;
4. The moving body display system according to claim 1, wherein said synthetic peripheral image generating section generates an all-around image centered on said moving body as said synthetic peripheral image. a synthetic peripheral image generation unit that generates a synthetic peripheral image when viewed from a changeable virtual viewpoint based on a plurality of captured images acquired from a plurality of cameras that capture images of the surroundings of a moving object;
a contact area vicinity image including a contact area between the movement plane of the moving body and the moving body, which is included in the captured images of all or part of the plurality of cameras, and a predetermined moving body model; A synthetic moving object image generation unit that generates a synthetic moving object image including
An image processing device comprising: the computer,
a synthetic peripheral image generation unit that generates a synthetic peripheral image when viewed from a changeable virtual viewpoint based on a plurality of captured images acquired from a plurality of cameras that capture images of the surroundings of a moving object;
a contact area vicinity image including a contact area between the movement plane of the moving body and the moving body, which is included in the captured images of all or part of the plurality of cameras, and a predetermined moving body model; An image processing program characterized by functioning as a synthetic moving object image generating unit that generates a synthetic moving object image including a moving object.

Images