WO2016075954A1

WO2016075954A1 - Display device and display method

Info

Publication number: WO2016075954A1
Application number: PCT/JP2015/055957
Authority: WO
Inventors: 拓良柳
Original assignee: 日産自動車株式会社
Priority date: 2014-11-14
Filing date: 2015-02-27
Publication date: 2016-05-19
Also published as: JP6500909B2; JPWO2016075954A1; WO2016075810A1

Abstract

Provided is a display device 100 comprising a control device 10 that executes: an image acquisition function that obtains a captured image captured by a camera 40 mounted in a vehicle V; an information acquisition function that obtains position information for the vehicle V; an image generation function that uses the position information for the vehicle, generates a position display image indicating the position at which a traveling body is present in the captured image, and generates a video including this position display image and the captured image; and a display function that displays the generated video.

Description

Display device and display method

The present invention relates to a display device and a display method for stereoscopically displaying an image around a moving object.

Referring to this type of device, there is known a moving body display device that projects an image of an object around the moving body and an image of the moving body onto a three-dimensional coordinate system (Patent Document 1).

JP 2012-138660 A

However, there is a problem in that it is difficult to grasp the positional relationship between the moving object and the object simply by displaying the image of the moving object as it is.

The problem to be solved by the present invention is to display an image in which the position of the moving object can be easily grasped.

The present invention solves the above problem by displaying a video including a position display image indicating the position of the moving object.

According to the present invention, since the position of the moving body can be expressed by displaying the position display image, it is possible to display an image in which the position of the moving body can be easily grasped.

It is a block diagram of a display system provided with the display apparatus which concerns on this invention. 2A and 2B are diagrams showing an example of the installation position of the camera of this embodiment. It is a figure which shows an example of a solid coordinate system. It is a figure which shows an example of the position display image of the own vehicle in the solid coordinate system shown to FIG. 3A. It is a figure which shows the other example of a solid coordinate system. It is a figure which shows the other example of the position display image of the own vehicle in the solid coordinate system shown to FIG. 4A. It is a figure which shows the example of a display of the position display image of the own vehicle and another vehicle. It is a figure which shows the 1st example of the 1st aspect of a position display image. It is a figure which shows the 2nd example of the 1st aspect of a position display image. It is a figure which shows the 3rd example of the 1st aspect of a position display image. It is a figure which shows the 4th example of the 1st aspect of a position display image. It is a figure which shows the 5th example of the 1st aspect of a position display image. It is a figure which shows the 6th example of the 1st aspect of a position display image. It is a figure which shows the example of the 2nd aspect of a position display image. It is a figure which shows the 1st example of the 3rd aspect of a position display image. It is a figure which shows the 2nd example of the 3rd aspect of a position display image. It is a figure which shows the 3rd example of the 3rd aspect of a position display image. It is a figure which shows the 4th example of the 3rd aspect of a position display image. It is a flowchart figure which shows the control procedure of the display apparatus which concerns on this embodiment. FIG. 10 is a flowchart showing a subroutine showing a procedure of reference point setting processing shown in FIG. 9. It is a figure for demonstrating the other example of a display of a position display image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the display device according to the present invention is applied to a display system 1 mounted on a moving body will be described as an example. The display system 1 of the present embodiment displays a video for grasping the moving body and the surroundings of the moving body on a display viewed by an operator of the moving body.

FIG. 1 is a block configuration diagram of a display system 1 including a display device 100 according to the present embodiment. As shown in FIG. 1, the display system 1 of this embodiment includes a display device 100 and a mobile device 200. Each device of the display device 100 and the mobile device 200 includes a wired or wireless communication device (not shown), and exchanges information with each other.

The moving body to which the display system 1 of the present embodiment is applied includes a vehicle, a helicopter, a submarine explorer, an airplane, an armored vehicle, a train, a forklift, and other devices having a moving function. In the present embodiment, a case where the moving body is a vehicle will be described as an example. Note that the moving body of the present embodiment may be a manned machine on which a human can be boarded, or an unmanned machine on which a human is not boarding.

Note that the display system 1 of the present embodiment may be configured as a device mounted on a moving body, or may be configured as a portable device that can be brought into the moving body. In addition, a part of the configuration of the display system 1 according to the present embodiment may be mounted on a moving body, and another configuration may be mounted on a device physically different from the moving body, and the configuration may be distributed. In this case, the mobile body and another device are configured to be able to exchange information.

As shown in FIG. 1, the mobile device 200 of this embodiment includes a camera 40, a controller 50, a sensor 60, a navigation device 70, and a display 80. Each of these devices is connected by a LAN (CANＣＡController 体 Area Network) or other LAN mounted on a mobile body, and can exchange information with each other.

The camera 40 of the present embodiment is provided at a predetermined position of a vehicle (an example of a moving body; the same applies hereinafter). The number of cameras 40 provided in the vehicle may be one or plural. The camera 40 mounted on the vehicle images the vehicle and / or the surroundings of the vehicle, and sends the captured image to the display device 100. The captured image in the present embodiment includes a part of the vehicle and a video around the vehicle. The captured image data is used for calculation processing of the positional relationship with the ground surface around the vehicle and generation processing of the image of the vehicle or the surroundings of the vehicle.

FIGS. 2A and 2B are diagrams illustrating an example of the installation position of the camera 40 mounted on the host vehicle V. FIG. In the example shown in the figure, the host vehicle V includes a right front camera 40R1, a right center camera 40R2, a right rear camera 40R3, a left front camera 40L1 of the host vehicle V, and a left side. Six cameras 40 of a center camera 40L2 and a left rear camera 40L3 are installed. In addition, the arrangement position of the camera 40 is not specifically limited, The imaging direction can also be set arbitrarily. Each camera 40 sends a captured image to the display device 100 at a command from the control device 10 to be described later or at a preset timing.

In the present embodiment, the captured image captured by the camera 40 is used for generating a video, and for detecting an object and measuring a distance to the object. The captured image of the camera 40 of the present embodiment includes an image of an object around the vehicle.
The target object in the present embodiment includes other vehicles, pedestrians, road structures, parking lots, signs, facilities, and other objects existing around the host vehicle V. The object in the present embodiment includes the ground surface around the moving body. In the present embodiment, the “ground surface” is a term indicating a concept including the surface of the earth and the surface of the earth's crust (land). The term “surface” of the present embodiment refers to a land surface, a sea surface, a river or river surface, a lake surface, a seabed surface, a road surface, a parking lot surface, a port surface, or two of these. Includes faces that contain more than one. In addition, in the case where the moving body is present indoors in a facility (building) such as a warehouse or a factory, the term “ground surface” in the present embodiment has a meaning including the surface of a structure such as a floor surface or a wall surface of the facility. . The term “surface” used in the description here is a (tangible) surface exposed to the camera 40 during imaging.

The camera 40 of this embodiment includes an image processing device 401. The image processing apparatus 401 extracts features such as an edge, a color, a shape, and a size from the captured image data of the camera 40, and identifies an attribute of the target object included in the captured image from the extracted features. The image processing apparatus 401 stores in advance the characteristics of each target object, and identifies the target object included in the captured image by pattern matching processing. The method for detecting the presence of an object using captured image data is not particularly limited, and a method known at the time of filing this application can be used as appropriate. The image processing device 401 calculates the distance from the own vehicle to the object from the position of the feature point extracted from the data of the captured image of the camera 40 or the change over time of the position. When calculating the distance, the image processing apparatus 401 uses imaging parameters such as the installation position of the camera 40, the optical axis direction, and imaging characteristics. The method for measuring the distance to the object using the captured image data is not particularly limited, and a method known at the time of filing the present application can be used as appropriate.

In the present embodiment, the distance measuring device 41 may be provided as means for acquiring data for calculating the positional relationship with the host vehicle V. The distance measuring device 41 may be used together with the camera 40 or may be used instead of the camera 40. The distance measuring device 41 detects a target existing around the host vehicle V and measures the distance between the target and the host vehicle V. That is, the distance measuring device 41 has a function of detecting an object around the host vehicle V. The distance measuring device 41 sends distance measurement data up to the measured object to the display device 100.

The distance measuring device 41 may be a radar distance measuring device or an ultrasonic distance measuring device. A ranging method known at the time of filing of the present application can be used. The number of distance measuring devices 41 that install the distance measuring devices 41 on the host vehicle V is not particularly limited. Also, the installation position of the distance measuring device 41 that installs the distance measuring device 41 in the host vehicle V is not particularly limited. The distance measuring device 41 may be provided at a position corresponding to the installation position of the camera 40 shown in FIG. 2 or in the vicinity thereof, or may be provided in front of or behind the host vehicle V. When the moving body is a helicopter, airplane, submarine spacecraft, or the like that moves in the height direction, the camera 40 and / or the distance measuring device 41 may be provided on the bottom side of the body.

The controller 50 of this embodiment controls the operation of the moving object including the host vehicle V. The controller 50 centrally manages each piece of information related to the operation of the moving body, including detection information of the sensor 60 described later.

The sensor 60 of the present embodiment includes a speed sensor 61 and a longitudinal acceleration sensor 62. The speed sensor 61 detects the moving speed of the host vehicle V. The longitudinal acceleration sensor 62 detects the acceleration in the longitudinal direction of the host vehicle V.

The navigation device 70 of the present embodiment includes a position detection device 71 including a GPS (Global Positioning System) 711, map information 72, and road information 73. The navigation device 70 obtains the current position of the host vehicle V using the GPS 711 and sends it to the display device 100. The map information 72 of the present embodiment is information in which points are associated with roads, structures, facilities, and the like. The navigation device 70 has a function of referring to the map information 72, obtaining a route from the current position of the host vehicle V detected by the position detection device 71 to the destination, and guiding the host vehicle V.

The road information 73 of this embodiment is information in which position information and road attribute information are associated with each other. The road attribute information includes road attributes such as that each road is an overtaking lane / not an overtaking lane, an uphill lane / not an uphill lane. The navigation device 70 refers to the road information 73 and, at the current position detected by the position detection device 71, the lane adjacent to the road on which the host vehicle V is traveling is an overtaking lane (a lane having a relatively high traveling speed). It is possible to obtain information on whether or not there is an uphill lane (a lane having a relatively low traveling speed). The control device 10 can predict the vehicle speed of the other vehicle from the detected attribute information of the road on which the other vehicle travels.

The display 80 of the present embodiment displays an image of the host vehicle V and the surroundings of the host vehicle V generated from an arbitrary virtual viewpoint generated by the display device 100 described later. In the present embodiment, the display system 1 in which the display 80 is mounted on a moving body will be described as an example. However, the display 80 may be provided on the portable display device 100 side that can be brought into the moving body.

Next, the display device 100 of this embodiment will be described. The display device 100 of this embodiment includes a control device 10.

As shown in FIG. 1, the control device 10 of the display device 100 according to the present embodiment is stored in a ROM (Read Only Memory) 12 in which a program for displaying a moving body and surrounding images is stored, and in the ROM 12. The CPU (Central Processing Unit) 11 serving as an operation circuit for realizing the functions of the display device 100 and the RAM (Random Access Memory) 13 functioning as an accessible storage device are provided. The control device 10 may include a Graphics / Processing / Unit that executes image processing.

The control device 10 of the display device 100 according to the present embodiment realizes an image acquisition function, an information acquisition function, an image generation function, and a display function. The control apparatus 10 of this embodiment performs each function by cooperation of the software for implement | achieving the said function, and the hardware mentioned above.

Hereinafter, each function realized by the control device 10 will be described.

First, the image acquisition function of the control device 10 will be described. The control device 10 according to the present embodiment acquires captured image data captured by the camera 40. The display device 100 acquires captured image data from the mobile device 200 using a communication device (not shown).

Next, the information acquisition function of the control device 10 will be described. The control device 10 acquires various types of information from the mobile device 200 using a communication device (not shown). The control apparatus 10 acquires the current position information of the host vehicle V as a moving body. The control device 10 acquires the current position detected by the GPS 711 of the navigation device 70.

The control device 10 acquires position information of an object existing around the host vehicle V as a moving body. The acquired position information of the object is used for setting processing of the position of the virtual light source described later. The control device 10 calculates the distance from the host vehicle V to the target object from the captured image of the camera 40 as position information of the target object with respect to the host vehicle V. The control device 10 may use the imaging parameter of the camera 40 for the calculation process of the position information of the object. The control device 10 may acquire the position information of the object calculated by the image processing device 401.

The control device 10 acquires the speed of the object. The control device 10 calculates the speed of the object from the change with time of the position information of the object. The control device 10 may calculate the speed of the object based on the captured image data. The control device 10 may acquire the speed information of the object calculated by the image processing device 401.

The control device 10 acquires attribute information of the lane on which the target object travels. The control device 10 refers to the road information 73 of the navigation device 70 and acquires the attribute information of the lane on which the object travels. Specifically, the control device 10 refers to the map information 72 or the road information 73 and identifies a road and a traveling lane that include the acquired position of the object. The control device 10 refers to the road information 73 and acquires attribute information associated with the travel lane of the identified object. The control device 10 calculates the positional relationship between the position of the host vehicle V detected by the GPS 711 and the symmetrical object, and considers the positional relationship, and determines the attribute of the traveling lane of the target object from the traveling lane attribute of the own vehicle V. May be judged. For example, in the case where there is a rule that the right lane is an overtaking lane, the other vehicle as the object travels on the right lane of the own vehicle V, and the traveling lane of the own vehicle is a non-overtaking lane. In this case, it can be determined that the traveling lane of the other vehicle is an overtaking lane.

The control device 10 acquires the acceleration in the traveling direction of the host vehicle V that is a moving body. The control device 10 acquires the acceleration in the traveling direction of the host vehicle V from the host vehicle V. The control device 10 acquires the longitudinal acceleration detected by the longitudinal acceleration sensor 62. The control device 10 may calculate the acceleration from the speed detected by the speed sensor 61. The control device 10 may calculate acceleration from a change in position information of the host vehicle V detected by the GPS 711.

Next, the image generation function of the control device 10 will be described.
The control device 10 according to the present embodiment uses the position information of the moving body to generate a video including a position display image indicating the position of the moving body in the captured image. Specifically, the control device 10 of the present embodiment sets a reference point using the position information of the moving body, and displays a position display image and a captured image that indicate the position of the moving body observed from the reference point in the captured image. Generate a video containing The position display image of the present embodiment is an image that represents the position where the moving object is present. The location of the moving object may be positively expressed by a diagram, a shadow, or the like, or may be passively expressed by a lack or change of the background.

The position display image of the present embodiment may include a diagram image indicating the position of the moving object in the captured image that is observed from a preset reference point. The control device 10 uses the position information of the moving body to set a viewpoint from which the moving body is viewed as a reference point, and includes a diagram image indicating the position of the moving body when the moving body is viewed from this viewpoint. Generate a display image. In this example, the reference point is a viewpoint for observing the moving body. The reference point may be set based on the position where the moving object exists. The reference point may be set based on the position of the projection plane described later. The diagram image of the position display image may be a graphic image imitating the outer shape of the moving body or an icon image reminiscent of the moving body. The representation mode of the diagram image is not particularly limited. The form of lines such as line thickness, color, broken line, and double line of the diagram image is not limited. The hue, brightness, saturation, color tone (tone), pattern, gradation, and size (enlargement / reduction) of the diagram image are not particularly limited. The control apparatus 10 of this embodiment calculates | requires the presence position of the mobile body in a captured image using the positional information on a mobile body, and produces | generates the position display image containing the diagram image which shows the presence position of a mobile body.

The position display image of the present embodiment may include a background image indicating the position of the moving object observed from a preset reference point. The control device 10 uses the position information of the moving body to set a viewpoint from which the moving body is viewed as a reference point, and when the moving body is viewed from this viewpoint, a position display including a background image indicating the position of the moving body Generate an image. In the present embodiment, the position of the moving object is expressed by a background image. In this example, the reference point is a viewpoint for observing the moving body. The reference point may be set based on the position where the moving object exists. The reference point may be set based on the position of the projection plane described later. The representation mode of the background image is not particularly limited. You may include the image defect area | region which shows the presence position of a moving body in a background image. You may include the diagram which shows the presence position of a moving body in a background image. The mode of lines such as line thickness, color, broken line, and double line of the background image is not limited. The hue, brightness, saturation, color tone (tone), pattern, gradation, and size (enlargement / reduction) of the background image are not particularly limited. The control apparatus 10 of this embodiment calculates | requires the presence position of the moving body in a captured image using the positional information on a moving body, and produces | generates the position display image containing the background image which shows the presence position of a moving body.

The position display image of the present embodiment may include a shadow image indicating the position of the moving object observed from a preset reference point. In the present embodiment, the position of the moving object is expressed by the shadow image of the moving object. In this example, the reference point is a light source that irradiates light to the moving body. Specifically, the control device 10 according to the present embodiment sets a virtual light source as a reference point using position information of the moving body, and imitates a shadow that is generated when light is emitted from the virtual light source to the moving body. A position display image including a shadow image indicating the existence position of is generated. A shadow image imitating a shadow that is generated when light is emitted from a virtual light source to a moving object indicates the position of the moving object that is observed from a preset reference point. The expression form of the shadow image is not particularly limited. The form of the line such as the thickness, color, broken line, double line, etc. of the shadow image is not limited. The hue, brightness, saturation, color tone (tone), pattern, gradation, and size (enlargement / reduction) of the shadow image are not particularly limited. The position display image is displayed at the position where the moving body exists in the coordinate system of the captured image, which is obtained using the actual position information of the moving body. The control device 10 according to the present embodiment sets a virtual light source according to the position information of the host vehicle V, and generates a shadow image imitating a shadow that is generated when the host vehicle V is irradiated with light from the virtual light source.

The control device 10 according to the present embodiment displays the image obtained by projecting the captured image on the three-dimensional coordinate system including the position display image indicating the position of the moving object. The displayed video includes part or all of the captured image and the position display image.

The control device 10 generates a position display image indicating the position where the moving object is present. The position display image includes a diagram image, a background image, and a shadow image.
The control device 10 sets a viewpoint as a reference point according to the acquired position information of the host vehicle V, and generates a diagram image and a background image imitating a video when a moving object and an object are viewed from the reference point. . In addition, the control device 10 sets a virtual light source as a reference point according to the acquired position information of the host vehicle V, and generates a shadow image imitating a shadow generated when light is emitted from the virtual light source to the moving object or the object. Generate. The shadow image may be an image approximated to the shadow itself, or may be an image obtained by deforming the shadow itself in order to display the position and orientation of the host vehicle V.
The position display image including the diagram image, the background image, and the shadow image does not strictly correspond to the current shape of the host vehicle V, and is an image that can indicate the position where the host vehicle V exists. Since the position only needs to be known, it is not necessary to imitate the shape of the host vehicle V or the like. However, a position display image imitating the shape of the host vehicle V or the like may be generated so that the traveling direction of the host vehicle V can be understood. The shadow image is an image that looks like a shadow, not an actual shadow. A pattern or color may be added to the position display image. The hue, brightness, saturation, color tone (tone), pattern, and gradation of the position display image can be changed according to the illuminance outside the moving body, the brightness of the captured image, and the like.
The position display image including the diagram image, the background image, and the shadow image is not limited to the shape corresponding to the current shape of the host vehicle V, and may be an image showing the movable range of the host vehicle V. For example, the position display image of the present embodiment may be an image showing the movable range of the door when the door of the vehicle V that is currently closed is released. The mode of the position display image is not limited, and is appropriately designed according to information desired to be shown to the user. You may use the shadow mapping technique known at the time of application for the production | generation of a position display image (a diagram image, a background image, a shadow image).

FIG. 3A is a diagram illustrating an example of a cylindrical solid coordinate system S1. In the three-dimensional coordinate system S1 shown in FIG. 3A, the host vehicle V shows a state of being placed on the plane G0. FIG. 3B is a diagram illustrating a display example of the position display image SH including the diagram image, the background image, and the shadow image in the three-dimensional coordinate system S1 illustrated in FIG. 3A. The position display image SH including the diagram image, background image, or shadow image in this example is projected on the plane G0. FIG. 4A is a diagram illustrating an example of a spherical solid coordinate system S2. In the three-dimensional coordinate system S2 shown in FIG. 4A, the host vehicle V shows a state placed on the plane G0. FIG. 4B is a diagram illustrating a display example of the position display image SH including the diagram image, the background image, or the shadow image of the host vehicle V in the three-dimensional coordinate system S2 illustrated in FIG. 4A. The position display image SH is projected on the plane G0.

The position display image SH shown in FIGS. 3B and 4B is a shadow image SH imitating a shadow generated when light is emitted from the virtual light source LG set in the three-dimensional coordinate systems S1 and S2 to the host vehicle V. The position display image SH shown in FIG. 3B and FIG. 4B is a diagram image showing the existence position of the host vehicle V when the host vehicle V is viewed from the viewpoint LG set in the three-dimensional coordinate systems S1 and S2. Although not shown, the captured image is projected onto the three-dimensional coordinate system S (S1, S2) of FIGS. 3B and 4B. The position display image SH is the presence of the host vehicle V viewed from the viewpoint LG by changing the background image of the captured image projected on the three-dimensional coordinate systems S1 and S2 (for example, by shifting the relative position of the background image). A position may be indicated. As shown in FIGS. 3B and 4B, the presence of the host vehicle V and the direction of the host vehicle V can be expressed by the position display image SH of the host vehicle V. Thereby, even if the image of the surrounding object is projected onto the three-dimensional coordinate system S1, S2, an image in which the positional relationship between the host vehicle V and the object can be easily grasped can be displayed.
Note that the shape of the three-dimensional coordinate system S of the present embodiment is not particularly limited, and may be a bowl shape disclosed in Japanese Patent Application Laid-Open No. 2012-138660.

The control device 10 of the present embodiment sets a reference point according to the acquired position information of an object such as another vehicle, and generates a position display image indicating the position of the object observed from this reference point. The position display image includes the above-described diagram image, background image, and shadow image. The control device 10 according to the present embodiment uses the acquired position information of the target object such as the other vehicle VX to set a viewpoint for viewing the target object as a reference point, and the mobile object when the mobile object is viewed from this viewpoint. A diagram image or background image indicating the location of the image is generated. The control device 10 of the present embodiment sets a virtual light source as a reference point according to the acquired position information of the target object such as the other vehicle VX, and creates a shadow that is generated when the target object is irradiated with light from the virtual light source. A simulated shadow image is generated. In the present embodiment, a position display image (a diagram image, a background image, a shadow image) indicating the position where the moving object is present is projected onto the three-dimensional coordinate systems S1 and S2. The reference point for generating the diagram image or the background image may be set according to a reference position (center of gravity, driver's seat position, etc.) set in advance on the moving body, or a predetermined point in the three-dimensional coordinate system S1, S2. The position may be set. In the embodiment, the reference point is set according to the height of the headrest of the driver's seat of the moving body.

FIG. 5 includes a position display image (line image, background image, shadow image) SH of the own vehicle V, a position display image SH1 of the other vehicle VX1 as an object, and a position display image SH2 of the other vehicle VX2. It is a figure which shows an example of an image | video. The control device 10 of the present embodiment displays a position display image SH as a position display image indicating the position of the moving body including the host vehicle V and / or other vehicles VX1, VX2 on the projection plane SQ. The position display image SH is an image showing the position of the moving body when the moving body is observed from a reference point set at a position corresponding to the position of the moving body. The position display image SH may be a shadow image imitating a shadow when a moving body is irradiated with light from a virtual light source. The position of the reference point LG including the viewpoint and the virtual light source may be set according to the reference position θ of the moving object. The reference position θ can be arbitrarily set according to the center of gravity position, the center position, and the like of the moving body. The reference point (viewpoint, virtual light source) LG for the host vehicle V, the other vehicle VX1, and the other vehicle VX2 may be one point, or may be set for each of the host vehicle V, the other vehicle VX1, and the other vehicle VX2. The position of the reference point (viewpoint, virtual light source) LG for the host vehicle V and the position of the reference point (viewpoint, virtual light source) LG for the other vehicle VX1 (VX2) may be the same or different. May be. In this way, the positional relationship between the host vehicle V and the target object is displayed by also displaying the position display images (line diagram image, background image, shadow image) of the target object such as the other vehicle VX as well as the host vehicle V. Can display images that are easy to grasp.

As shown in FIG. 5, the control device 10 of the present embodiment sets a projection surface SQ for projecting a position display image (line image, background image, shadow image) SH along the direction in which the host vehicle V moves. . The control device 10 of the present embodiment displays a position display image (a diagram image, a background) when the other vehicle VX1 traveling in the adjacent lane Ln1 adjacent to the traveling lane Ln2 in which the host vehicle V travels (moves) is observed from the reference point. Image, shadow image) SH1 is generated.
As described above, the position display image (line diagram image, background image, shadow image) SH1 of the other vehicle VX1 (target object) traveling in the adjacent lane Ln1 is converted into the position display image (line diagram image, background image, By projecting onto the common projection plane SQ together with the shadow image (SH), it is possible to present an image that makes it easy to grasp the positional relationship between the host vehicle V and the other vehicle V1. Further, by setting the projection plane SQ along the traveling direction of the host vehicle V, it is possible to present an image in which the distance between the host vehicle V and the other vehicle VX1 can be easily recognized. On the projection surface SQ, a captured image, a position display image (line diagram image, background image, shadow image) SH of the host vehicle V, and a position display image (line diagram image, background image) of another vehicle VX1 (target object), A shadow image SH1 is projected.
Further, by setting the projection plane SQ so as to be substantially orthogonal (intersect at 90 degrees) with the road surface of the lane Ln2 on which the host vehicle V is traveling, the position of the host vehicle V is at a position where the driver of the host vehicle V can easily see. The display image SH and the position display image SH1 of the other vehicle VX1 can be displayed. That is, the driver | operator of the own vehicle V can show the image | video which can recognize the positional relationship of the own vehicle V and the other vehicle VX1 correctly from a driver's seat.

The projection position when the diagram image SH (or background image SH) is projected onto the projection plane SQ indicates the own vehicle V (or other vehicle VX1) so as to indicate the position of the own vehicle V (or other vehicles VX1, VX2). , VX2) is determined according to an arbitrary reference position θ. The position on the XZ coordinate of the arbitrary reference position θ of the host vehicle V (or other vehicles VX1, VX2) shown in FIG. 5 is preferably the same as the position on the XZ coordinate of the projection plane SQ.

When the position of the host vehicle V shown in FIG. 5 is the same, the diagram image SH, the background image SH, and the shadow image SH are displayed at substantially the same position on the projection plane SQ. Similarly, when the position of the other vehicle SH2 is the same, the diagram image SH2, the background image SH2, and the shadow image SH2 are displayed at substantially the same position on the projection plane SQ.

Hereinafter, the mode of the position display image including the shadow image SH, the diagram image SH, and the background image SH will be described with reference to FIGS. 6A-6F, FIGS. 7, and 8A-8D. In addition, area | region Rd1 of each figure is the background image Rd1 corresponding to road structures, such as a travel path of a mobile body, a travel path adjacent to it, and a guardrail / roadside zone. A region Rd2 (region on the upper side of the region Rd1) in each figure is a background image Rd2 corresponding to a tree, a building, a sign, or the like on the road side of the traveling path of the moving body. Background image Rd1 and background image Rd2 are displayed on projection plane SQ. The projection plane SQ may include both the background images Rd1 and Rd2 or any one of them.

FIGS. 6A to 6F are views showing aspects of a position display image including a shadow image and a position display image including a diagram image SH imitating the shape of the vehicle. When the moving body is a vehicle, the shadow image also approximates the shape of the vehicle. Here, in order to avoid overlapping explanation, a position display image including a shadow image and a position display image including a diagram image SH representing a shape of a vehicle will be described with reference to the same drawing.

FIG. 6A shows a position display image in which the shadow area of the shadow image SH6a is shown as an opaque area (transmittance is less than a predetermined value). FIG. 6A shows a position display image in which the region in the figure of the diagram image SH6a is shown as an opaque region (low transmittance).
FIG. 6B shows a position display image in which the shadow area of the shadow image SH6b is shown as a semi-transparent area (transmittance is a predetermined value or more). FIG. 6B shows a position display image in which the in-figure region of the diagram image SH6b is shown as a semi-transparent region (transmittance is a predetermined value or more).
FIG. 6C shows a position display image in which the brightness of the shadow area (or diagram area) of the shadow image (or diagram image) SH6c is expressed higher than the brightness of the area other than the shadow image (or diagram image). . When the periphery of the moving body is dark, the brightness of the background images Rd1 and Rd2 is low. When the captured image is included in the position display image, if the brightness of the shadow image (or diagram image) SH6a is lowered as shown in FIG. 6A, the drawing position of the shadow image (or diagram image) SH6a may become unclear. There is. For this reason, in this embodiment, when the brightness around the moving object is less than a predetermined value, the brightness of the shadow area (or diagram area) of the shadow image (or diagram image) SH6c is the shadow image (or diagram area). The shadow image (or diagram) in which the brightness is adjusted by adjusting the brightness of the shadow region (or diagram region) of the shadow image (or diagram image) SH6c so as to be higher than the brightness of the region other than the diagram image). Image) A position display image including SH6c is shown.

FIG. 6D shows a position display image in which the shadow region (or diagram region) of the shadow image (or diagram image) SH6d is colored. Since colors cannot be expressed in the drawings attached to the application, it is shown for convenience that the shadow image (or diagram image) SH6d is colored by hatching. The hue of the color of the shadow area (or diagram area) is not limited. The brightness and saturation are not limited.
FIG. 6E shows a position display image including the outline of the shadow region (or diagram region) of the shadow image (or diagram image) SH6e. The inward extension (inner side) surrounded by the outline of the shadow area (or diagram area) shown in FIG. 6E is transparent or translucent. Of course, the inward extension (inner side) surrounded by the outline may be opaque, or a color or pattern may be added.
FIG. 6F shows a position display image obtained by blurring a part of the shadow region (or diagram region) of the shadow image (or diagram image) SH6f. In the shadow area (or diagram area) shown in FIG. 6F, gradation is given so that the brightness decreases from the inside toward the outside. Thereby, the outline of the shadow area (or diagram area) can be clearly shown. In addition, the object on the other side of the moving object can be shown by increasing the inner transparency. Thereby, the presence position of a moving body and the presence of the object of the other side (side) of a moving body can be confirmed simultaneously.

FIG. 7 is a diagram illustrating an aspect of a position display image including a background image.
FIG. 7 shows the shadow image (or diagram image) SH7a as translucent or transparent, and the height (z direction) of the road background image Rd1a that is transparently displayed inside the shadow image (or diagram image) SH7a. This is an example in which (position) is changed. The height in the Z-axis direction of the road background image Rd1a transparently displayed inside the shadow image (or diagram image) SH7a is lower than the height of the other road background images Rd1b and Rd1c in the Z-axis direction. As a result, the height in the Z-axis direction of the road background images Rd1a, Rd1b, and Rd1c that should be continuous changes at the front end and the rear end of the shadow image (or diagram image) SH7a. The user can recognize the presence position (position in the x direction) of the moving body based on the change position of the height position of the road background image Rd1 shown in the position display image.
Further, FIG. 7 shows the shadow image (or diagram image) SH7a as translucent or transparent, and the horizontal position of the background image Rp1a of the utility pole transparently displayed inside the shadow image (or diagram image) SH7a ( This is an example in which the position in the X direction is changed. The position in the X-axis direction of the background image Rp1a of the utility pole that is transparently displayed so as to overlap with the shadow image (or diagram image) SH7a is shifted to the right in the figure from the position in the X-axis direction of the background image Rp1b of the other utility pole ing. As a result, the positions of the utility pole background images Rp1a and Rp1b in the X-axis direction that should be continuous change at the boundary of the shadow image (or diagram image) SH7a. From the change position of the horizontal position of the electric pole background image Rp1 shown in the position display image, the user can recognize the presence position (position in the X-axis direction) of the moving body.
Although not shown, a region (outlined extension line) in which a part of the background image is missing may be formed along the extension of the shadow image (or diagram image) SH7a. In other words, the outline region along the extension of the shadow image (or diagram image) SH7a may be added to the background image Rd1 and the background image Rd2.

FIGS. 8A to 8D are diagrams showing aspects of a position display image including a diagram. FIGS. 6A to 6F show examples of graphics imitating the outline of the moving body, but FIGS. 8A to 8D show an aspect of a position display image including an axis indicating a coordinate position and a graphic.
FIG. 8A is an example in which the position of the moving object is indicated by a circular diagram image SH8a. The position of the moving body on the projection plane SQ can be indicated by the displayed position of the circle. The diagram image SH8a illustrated in FIG. 8A further includes a vertical line SH8az along the Z-axis direction of the projection surface SQ and a horizontal line SH8ax along the X-axis direction of the projection surface SQ. The vertical line SH8az and the horizontal line SH8ax are substantially orthogonal at the center point SH8a0. The center point SH8a0 corresponds to a reference position such as the center of gravity of the moving object. The vertical line SH8az and the horizontal line SH8ax may be graduated. In the example shown in FIG. 8A, the XZ coordinates are graduated with reference to the center of the radius of the circular diagram image SH8a. Thereby, the user can grasp | ascertain specifically the positional relationship of the external field and moving body which are projected on the projection surface SQ.
FIG. 8B is an example showing the position of the moving object using coordinate axes SH8bx and SH8bx that are substantially orthogonal to each other at the reference point SH8b0. The center point SH8b0 corresponds to a reference position such as the center of gravity of the moving body. The position of the moving object on the projection plane SQ can be indicated by the position of the center point and the coordinate axis.
FIG. 8C is an example showing the position of the moving object using two vertical lines SH8c1 and SH8c2 along the Z direction of the projection plane SQ. The position of the vertical line SH8c1 corresponds to the position of the front end (or rear end) of the moving body. The position of the vertical line SH8c2 corresponds to the position of the rear end (or front end) of the moving body. The position of the moving body on the projection plane SQ can be indicated by the positions of the two vertical lines SH8c1 and SH8c2 in the X-axis direction.
FIG. 8D is an example showing the location of the moving object using a rectangular area SH8d. One end of the region SH8d in the X-axis direction is defined by a vertical line SH8d1, and the other end is defined by a vertical line SH8d2. The height of the vertical line SH8d1 in the Z-axis direction is not limited. In this example, the height of the vertical line SH8d1 in the Z-axis direction is the same as the height of the projection surface SQ in the Z-axis direction. The position of the vertical line SH8d1 in the X-axis direction corresponds to the position of the front end (or rear end) of the moving body. The position of the vertical line SH8d2 in the X-axis direction corresponds to the position of the rear end (or front end) of the moving body. The position of the moving body on the projection plane SQ can be indicated by the positions of the two vertical lines SH8d1 and SH8d2 in the X direction.

The control device 10 according to the present embodiment generates a video including a position display image indicating the position of the target in the captured image, using the position information of the target existing around the moving body acquired by the information acquisition function. . A position display image of an object such as another vehicle includes a diagram image, a background image, and a shadow image, similarly to the position display image of the host vehicle (moving body). About these, in order to avoid the overlapping description, the description mentioned above is used.

As in the position display image shown in FIG. 5, the control device 10 of the present embodiment observes the preceding other vehicle VX2 traveling in front of the traveling lane in which the host vehicle V travels (moves) from the reference point, and from the reference point. A position display image (a diagram image, a background image, a shadow image) of the observed preceding other vehicle VX2 is generated.

Incidentally, as shown in FIG. 5, when the host vehicle V tries to change the lane from the lane Ln2 to the lane Ln1 along the route indicated by the arrow Rn, the driver of the host vehicle V is rearward on the adjacent lane Ln1. Pay attention to the behavior of the other vehicle VX1 traveling in front of the vehicle and the other vehicle VX2 traveling in front of the traveling lane Ln2 of the host vehicle V. In such a scene, the control device 10 of the present embodiment uses the position display image SH of the host vehicle V, the position display image SH1 of the other vehicle VX1, and the position display image SH2 of the other vehicle VX2 on a common projection plane SQ. Project. Thereby, the driver of the own vehicle V can correctly grasp the positional relationship between the own vehicle V, the other vehicle VX1, and the other vehicle VX2 from the positional relationship with the position display images SH, SH1, and SH2. The timing at which the host vehicle tries to change the lane from the lane Ln2 to the lane Ln1 is determined from the steering angle of the host vehicle, the blinker operation, the braking operation, and the like.

In this case, the projection position of the position display image such as the shadow image SH is changed according to the acceleration / deceleration of the host vehicle V. Specifically, when the host vehicle V is accelerating, the control device 10 shifts the position of the position display image (line image, background image, shadow image) SH forward. On the other hand, when the host vehicle V is decelerating, the control device 10 shifts the position backward to the position display image (line image, background image, shadow image) SH. Thereby, according to the situation of the own vehicle V, the position display image SH which can grasp | ascertain the positional relationship with the own vehicle V and the other vehicle VX can be displayed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

Further, the projection position of the position display image (line image, background image, shadow image) is changed according to the difference between the speed of the lane in which the host vehicle V travels and the speed of the host vehicle V. The speed of the lane flow may be an average speed of another vehicle VX traveling in the same lane as the host vehicle V is traveling, or a legal speed of the lane. Specifically, the control device 10 determines the position when the difference (positive value) between the vehicle speed of the host vehicle V and the flow speed of the lane is large, that is, when the host vehicle V is approaching or overtaking the preceding other vehicle. The position of the position display image SH of the display image (line diagram image, background image, shadow image) is shifted forward. On the other hand, the control device 10 has a small difference between the vehicle speed of the host vehicle V and the flow speed of the lane, or a large difference (negative value), that is, the host vehicle V is approached by the other vehicle behind or overtakes the rear vehicle. The position display image SH is shifted backward. Thereby, a position display image (a diagram image, a background image, a shadow image) SH that makes it easy to grasp the positional relationship between the host vehicle V and the other vehicle VX according to the relative situation between the host vehicle V and the other vehicle VX. Can be displayed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

The control device 10 according to the present embodiment has an area of a position display image (a diagram image, a background image, a shadow image) SH of an object having a relatively high speed of the object including the other vehicle VX and a relatively high speed. The position display images SH1 and SH2 are generated so as to be larger than the area of the position display image such as the position display image SH of the low object. The control device 10 of the present embodiment acquires the vehicle speed P1 of the other vehicle VX1 and the vehicle speed P2 of the other vehicle VX2, and compares the vehicle speeds P1 and P2. The control device 10 generates the position display image SH of the other vehicle VX with a high vehicle speed so that the area is larger than the position display image SH of the other vehicle VX with a low vehicle speed. In this example, when the vehicle speed P1 of the other vehicle VX1 is higher than the vehicle speed P2 of the other vehicle VX2, as shown in FIG. 5, the area of the position display image SH1 of the other vehicle VX1 is set to the position display image SH2 of the other vehicle VX2. Larger than the area. By displaying the large position display image SH2 with a large area, it is possible to alert the driver to the other vehicle VX traveling at high speed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

Although not particularly limited, the control device 10 of the present embodiment may increase the area of the position display image (line image, background image, shadow image) SH of the other vehicle VX as the speed of the other vehicle VX increases. . A driver using this system can predict the speed of the other vehicle VX from the size of the position display image SH. The speed of the other vehicle VX may be an absolute speed or may be a relative speed with respect to the vehicle speed of the host vehicle V. Thereby, the position display image SH of the other vehicle V having a high degree of approach to the host vehicle V can be displayed in a large size. Of course, depending on the speed of the other vehicle V with respect to the host vehicle V, the hue, brightness, saturation, color tone (tone), pattern, gradation, and size (enlargement / reduction) of the position display image may be changed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

In addition, when the control device 10 of the present embodiment acquires attribute information that the lane Ln1 in which the other vehicle VX travels is an overtaking lane, the lane Ln2 in which the other vehicle VX is traveling is not in the overtaking lane. The position display image SH is generated so that the area of the position display image (line image, background image, shadow image) SH of the other vehicle VX traveling on the overtaking lane is larger than that obtained. This is because the speed of the other vehicle VX traveling on the overtaking lane can be predicted to be higher than the speed of the other vehicle VX traveling on the non-overtaking lane. The attribute information that the lane is an overtaking lane or is not an overtaking lane is acquired from the map information 72 and / or road information 73 of the navigation device 70. The method for identifying and acquiring the lane attribute information is not particularly limited, and a method known at the time of filing can be used as appropriate.

Further, the control device 10 according to the present embodiment changes the area of the position display image (line image, background image, shadow image) SH according to the driving skill of the driver of the moving object, for example, the driver of the host vehicle V. Also good. For example, when the skill of the pilot is low, the control device 10 generates the position display image SH so that the area of the position display image SH is larger than when the skill of the pilot is high. Thereby, each position of the own vehicle V and the other vehicle V and those positional relationships can be shown in an easy-to-understand manner to a pilot with low skill. The operator's skill may be input by the operator himself / herself, or may be determined based on experience such as the number of operations and distance. The driver's skill may be determined from the pilot's past operation history. For example, the operation history of a pilot with high skills is compared with the operation history of individual pilots. If the difference is large, it is determined that the operation skill is low, and if the difference is small, the operation skill is Judged to be high. Taking vehicle driving (driving) as an example, the driving skill of the vehicle can be determined based on the acceleration operation, the timing of the steering operation, and the steering amount when the travel lane is changed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

In the example shown in FIG. 5, lane Ln1 is an overtaking lane, and lane Ln2 is not an overtaking lane (uphill lane). Therefore, the position display image (line diagram image, background image, shadow image) SH1 of the other vehicle VX1 traveling on the lane Ln1 is more position display image (line diagram image, background image) of the other vehicle VX2 traveling on the lane Ln2. , Shadow image) The area is larger than SH2 and the position display image SH of the host vehicle V. As described above, the size of the position display image SH is controlled by the actual vehicle speed by changing the size of the position display image SH according to the attribute of the lane in which the host vehicle V and the other vehicle VX travel. In addition, it is possible to alert the driver to the other vehicle VX traveling at high speed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

Furthermore, when the control device 10 of the present embodiment determines that the host vehicle V is in an acceleration state from the acquired acceleration of the host vehicle V, the control device 10 determines the position of the reference point LG in the traveling direction of the host vehicle V (in the drawing). Shift to the opposite side (arrow F ′ direction in the figure) of arrow F direction. Thereby, the projection position of the position display image SH can be shifted to the traveling direction (the direction of arrow F in the figure) of the host vehicle V (the direction of arrow F in the figure). Further, when the shadow image SH is used as the position display image, the control device 10 according to the present embodiment determines that the host vehicle V is in an acceleration state from the acquired acceleration of the host vehicle V, and then the virtual light source LG. Is shifted to the opposite side (arrow F ′ direction in the figure) to the traveling direction of the host vehicle V (arrow F direction in the figure). In the example illustrated in FIG. 5, when it is determined that the host vehicle V is in an acceleration state, a reference point (viewpoint, virtual light source) LG is set to a rear position, for example, a reference point (viewpoint, virtual light source) LG2 ( Shift to the position indicated by the broken line. Thus, in the acceleration state, the position of the reference point (viewpoint, virtual light source) LG is shifted backward, and the projection position of the position display image (line image, background image, shadow image) SH is shifted forward. Thus, it is possible to display the position display image SH suitable for the determination status of the driver. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

When the control device 10 according to the present embodiment determines from the acquired acceleration that the host vehicle V is in a decelerating state, the control device 10 determines the position of the reference point (viewpoint, virtual light source) LG in the traveling direction side of the host vehicle V (see FIG. Shift in the direction of the middle arrow F). Thereby, when it is determined that the host vehicle V is in a decelerating state, the projection position of the position display image SH can be shifted to the side opposite to the traveling direction of the host vehicle V (the direction of the arrow F ′ in the figure). Further, when the shadow image SH is used as the position display image, the control device 10 of the present embodiment determines the position of the virtual light source LG when the host vehicle V determines from the acquired acceleration that the vehicle V is in a decelerating state. Shift to the traveling direction side of vehicle V (in the direction of arrow F in the figure). In the example shown in FIG. 5, when it is determined that the host vehicle V is in a decelerating state, the reference point (viewpoint, virtual light source) LG is set to a forward position, for example, the position of the reference point LG1 (displayed with a broken line). Shift to. As described above, in the deceleration state, the position of the reference point LG is shifted forward, and the projection position of the position display image (line diagram image, background image, shadow image) SH is shifted backward, thereby determining the driver's judgment status. The position display image suitable for can be displayed. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

The setting position of the reference point (viewpoint, virtual light source) LG is not limited, but may be the same position as the virtual viewpoint in the projection processing. As a result, the position of the virtual viewpoint viewing the host vehicle V can be recognized from the shape of the position display image (line diagram image, background image, shadow image) SH, and the positional relationship between the host vehicle V and the surroundings can be easily understood. Further, the reference point LG may be arranged at infinity. In this case, since parallel projection can be performed, it becomes easy to grasp the positional relationship between the host vehicle V and the object (other vehicle VX, etc.) from the position display image SH.

The setting position of the virtual viewpoint is not particularly limited. Further, the position of the virtual viewpoint may be changeable according to the user's designation. Further, the surface on which the position display image SH is projected (represented) may be a road surface of a road on which the host vehicle travels, or may be a projection surface set as shown in FIG. Further, when the moving body is a helicopter, a position display image (line diagram image, background image, shadow image) for displaying the position may be projected on the ground surface below the helicopter. When the moving body is a ship, the shadow information may be projected on the sea surface.

Finally, the display function of the control device 10 of this embodiment will be described. The control device 10 projects the captured image data acquired from the camera 40 onto the three-dimensional coordinate system S or the projection plane SQ, and generates images of the host vehicle V and surrounding objects from the set virtual viewpoint. Then, the control device 10 displays the generated video on the display 80. The display 80 may be mounted on the host vehicle V and configured as the mobile device 200 or may be provided on the display device 100 side. The display 80 may be a display for a two-dimensional image or a display that displays a three-dimensional image in which the positional relationship in the depth direction of the screen can be visually recognized.

The video to be displayed in the present embodiment includes a position display image (a diagram image, a background image, a shadow image) SH of the host vehicle V. Further, the display image may include position display images (line diagram image, background image, shadow image) SH1 and SH2 of the other vehicle VX as an object. Of course, the displayed image may include both the position display image SH of the host vehicle V and the position display images SH1 and SH2 of the other vehicle VX. In this process, a shadow image SH, a diagram image SH, and a background image SH can be used as the position display image SH.

Even if the icon image V ′ (see FIGS. 3A, 3B, 4A, and 4B) indicating the host vehicle V prepared in advance is superimposed and displayed on the video displayed by the display device 100 of the present embodiment. Good. The icon image V ′ of the vehicle may be created and stored in advance based on the design of the host vehicle V. In this manner, by superimposing the icon image V ′ of the host vehicle V on the video, the relationship between the position and orientation of the host vehicle V and the surrounding video can be shown in an easily understandable manner.

Hereinafter, based on the flowchart of FIG. 9, the first operation of the control device 10 of the present embodiment will be described.
In step S <b> 1, the control device 10 acquires a captured image captured by the camera 40. In step S2, the control device 10 acquires the current position of the host vehicle V and the position of the object including the other vehicle VX. In this process, the host vehicle V is an example of a “moving body”, and the other vehicle VX is an example of an “object”.

In step S3, the control device 10 sets a reference point using the position information of the moving body (the host vehicle V, the other vehicle VX). One reference point may be set based on the host vehicle V, or a plurality of reference points may be set for each of the host vehicle V and the other vehicle VX. The reference point is a point (viewpoint) for observing a moving object that is a target of a diagram image, a point (viewpoint) for observing a moving object that is a target of a background image, and a light that is irradiated to a moving object that is a target of a shadow image Includes points (light sources).

An example of the method for setting the reference point (viewpoint, virtual light source) (step S3) will be described based on the flowchart of FIG.
As shown in FIG. 10, in step S <b> 11, the control device acquires the acceleration of the host vehicle V. In step S12, the control device 10 determines whether the host vehicle V is in an acceleration state or a deceleration state based on the acceleration. If it is in the accelerated state, the process proceeds to step S13. In step S13, by shifting the reference point (viewpoint, virtual light source) to the rear side, the projection position of the position display image including the diagram image SH, the background image SH, and the shadow image SH is shifted to the front side in the traveling direction. . In other words, the position of the position display image projected on the projection plane when the host vehicle V is in the acceleration state is more than the position of the position display image projected on the projection plane when the host vehicle V is not in the acceleration state. It is located on the front side in the traveling direction of V. On the other hand, if it is determined in step S14 that the host vehicle V is in a decelerating state, the process proceeds to step S15. In step S15, by shifting the reference point (viewpoint, virtual light source) to the front side, the projection position of the position display image including the diagram image SH and the background image SH is shifted to the rear side in the traveling direction. That is, the position of the reference point that is set when the host vehicle V is in the deceleration state is more forward of the traveling direction of the host vehicle V than the position of the reference point that is set when the host vehicle V is not in the deceleration state. To position. The position of the position display image projected on the projection plane when the host vehicle V is in a deceleration state is greater than the position of the position display image projected on the projection plane when the host vehicle V is not in a deceleration state. Located on the rear side in the direction of travel. The projection position of the position display image is a position in the coordinate system of the projection plane set in a later process.

Referring back to FIG. 9, in step S4, the control device 10 sets a projection plane. The projection plane may be a three-dimensional coordinate system shown in FIGS. 3A, 3B, 4A, and 4B, or may be a two-dimensional coordinate system like the projection plane SQ shown in FIG. Furthermore, as shown in FIG. 11 described later, a plurality of projection planes may be set.

In subsequent step S5, the control device 10 generates a diagram image SH, a background image SH, or a shadow image SH of the host vehicle V. When the other vehicle VX exists around the host vehicle V, the diagram images SH1, SH2, background images SH1, SH2 or shadow images SH1, SH2 of the other vehicles VX1, VX2 are generated.

In step S6, the control device 10 executes a process of projecting the captured image on the set projection plane SQ, and generates a display image.

Finally, in step S7, the control device 10 displays the generated video on the display 80.

Hereinafter, based on FIG. 11, another example of the display mode of the position display image of the host vehicle V that is a moving body will be described. The position display image described here includes a shadow image SH, a diagram image SH, and a background image SH. In this example, a case where the position display image is a shadow image SH will be described as an example. Instead of the shadow image SH, a diagram image SH and a background image SH may be used.
The shadow image SH (or diagram image SH) of the position display image of this example includes an image indicating the movable range of the movable member of the host vehicle V. The background image SH includes a contour line of the movable member of the host vehicle V and a diagram (background defect region) indicating the movable range of the working member. In the example shown in FIG. 11, the projection plane showing the shadow image SH of the host vehicle V includes a first projection plane SQs along the vehicle length direction of the host vehicle V and a second projection plane SQb along the vehicle width direction. On the first projection plane SQs, a position display image (shadow image, diagram image, background image) SHs showing a movable member when observed from a reference point (viewpoint, virtual light source) LG set on the side of the vehicle is displayed. Project. The position display image includes a shadow image imitating the shadow of the movable member when light is emitted from the virtual light source LG. On the second projection plane SQb, a position display image (shadow image, diagram image, background image) SHb showing a movable member when observed from a reference point (viewpoint, virtual light source) LG set in front of the vehicle is projected. To do. The position display image projects a shadow image SHb simulating a shadow when light is emitted from the virtual light source LG.

The position display image (shadow image, diagram image, background image) SH of this example includes an image indicating the movable range of the movable member of the host vehicle V. The own vehicle V of this example is a hatchback type vehicle, and has a side door and a back door (hatch door). The side door of the host vehicle V opens and closes sideways, and the back door of the host vehicle V opens and closes rearward. The side door and the back door of the host vehicle V are movable members of the host vehicle V that is a moving body.

The control device 10 assumes a case where the occupant opens the back door in order to carry in or out the luggage from the rear loading platform, and displays a position display image (shadow image, line drawing image, background image) indicating the movable range of the back door. ) Is generated. As shown in FIG. 11, the position display image SHs projected on the first projection surface SQs includes a back door portion Vd3. The back door portion Vd3 represents the rear extension (movable range) of the host vehicle V when the back door is opened. The shadow image indicating the movable range of the back door is projected on the left and right sides of the host vehicle V or on the placement surface (parking surface / road surface) of the host vehicle V. The shadow image may be projected on a wall surface or floor surface that actually exists.

The control device 10 assumes a case where the side door is opened so that an occupant can get on and off from the seat or carry in / out the luggage, and a position display image (shadow image, diagram image) indicating the movable range of the side door. , Background image). As shown in FIG. 11, the position display image (shadow image, diagram image, background image) SHb projected on the second projection plane SQb includes side door portions Vd1 and Vd2. Expresses the lateral extension (movable range) of the side door when the side door is opened. The position display image indicating the movable range of the side door is projected forward and / or rearward of the host vehicle V.
Note that the control device 10 may set a projection plane to project a position display image (shadow image, diagram image, background image), and project the position display image onto the projection plane.

Thus, by generating and displaying the position display image SH including the image indicating the movable range of the movable member of the host vehicle V, the driver of the host vehicle V considers the work after the host vehicle V is parked. And the parking position of the own vehicle V can be determined. Furthermore, by superimposing the captured images of the surrounding objects together with the position display image SH, it is possible to park at a position where the work after parking is not hindered while avoiding the surrounding objects.

In this example, the mode of the position display image (shadow image, diagram image, background image) SH will be described in the case where the moving body is the host vehicle V, that is, the door of the host vehicle V is movable. The mode of the position display image SH is not limited to this. For example, when the moving body is a forklift, a position display image (a shadow image, a diagram image, a background image) is generated in consideration of the movable range of the forklift body and the lift equipment. When the moving body is a helicopter, a position display image (shadow image, diagram image, background image) is generated in consideration of the rotation range of the helicopter body and the rotor blades. When the moving body is an airplane, a position display image (a shadow image, a diagram image, a background image) is generated in consideration of the installation range of the airplane main body and ancillary equipment such as a passenger trap. When the mobile body is a submarine spacecraft, it is generated in consideration of the installation range of the platform provided as necessary. The generated position display image is displayed on the display 80.

Incidentally, when the moving body is a forklift, a position display image (shadow image, diagram image, background image) indicating the operating range of the lift device is displayed, for example, on the surrounding ground surface (for example, the floor of a facility such as a warehouse or a factory). By displaying on the surface or wall surface, it can be confirmed in advance whether or not the forklift can be operated without interfering with other devices and facilities. By displaying a position display image (shadow image, diagram image, background image) indicating the rotation range of the rotor blade of the helicopter on, for example, the surrounding ground surface (for example, the ground surface), the helicopter can be landed You can search for a place from the sky. For example, by displaying a shadow image indicating the rotation range of the rotor blade on the ground surface (for example, a cliff surface) around the helicopter, the attitude of the helicopter can be confirmed from the sky. By displaying a shadow image indicating the installation range of the main body of the airplane and the attached equipment on the ground surface (for example, the ground surface) of the airplane, it is possible to search from the sky for a place having an area where the airplane can make an emergency landing.

The position display image (shadow image, diagram image, background image) SH may store a basic pattern prepared in advance, and the control device 10 may read it from the memory as necessary.

Since the present invention is configured as described above and operates as described above, the following effects can be obtained.

[1] The display device 100 according to the present embodiment sets a reference point using position information of a moving body such as the host vehicle V and the other vehicle VX, and the host vehicle V and the other vehicle observed from the reference point in the captured image. A position display image SH indicating the position where the VX exists is generated, and an image including the position display image SH and part or all of the captured image is displayed. As described above, since the video including the position display image SH indicating the position where the host vehicle V is present and the captured image can be displayed, a video in which the positional relationship of the host vehicle V with respect to the surroundings can be easily understood can be displayed.

[2] The display device 100 of the present embodiment generates a diagram image SH as a position display image indicating the position of the host vehicle V in the captured image, and includes part or all of the diagram image SH and the captured image. Display video. As described above, the diagram image SH indicating the location of the host vehicle V and the image including a part or all of the captured image can be displayed, so that the image in which the positional relationship of the host vehicle V with respect to the surroundings can be easily understood can be displayed.

[3] The display device 100 according to the present embodiment generates a background image SH as a position display image indicating the position of the host vehicle V in the captured image, and displays a video including the background image SH and a part or all of the captured image. Display. As described above, the background image SH indicating the position of the host vehicle V and the image including a part or all of the captured image can be displayed, so that it is possible to display an image in which the positional relationship of the host vehicle V with respect to the surroundings can be easily understood.

[4] The display device 100 of the present embodiment generates a shadow image SH as a position display image indicating the position of the host vehicle V in the captured image, and displays an image including the shadow image SH. The display device 10 according to the present embodiment sets a virtual light source as a reference point using position information of moving bodies such as the host vehicle V and the other vehicle VX, and generates shadows when the host vehicle V is irradiated with light from the virtual light source. Is generated, and a video including part or all of the shadow image SH and a captured image around the host vehicle V is displayed. Thus, since the presence of the host vehicle V and the direction of the host vehicle V can be expressed by the shadow image SH of the host vehicle V, it is possible to display an image in which the positional relationship between the host vehicle V and surrounding objects can be easily understood.

[5] The display device 100 according to the present embodiment generates a position display image SH indicating the position of an object including the other vehicle VX, and displays an image including the position display image SH and a part or all of the captured image. . As described above, since the position display image SH indicating the position of the other vehicle VX as the object and the image including part or all of the captured image can be displayed together, the positional relationship between the host vehicle V and the other vehicle VX can be displayed. Easy-to-understand video can be displayed.

[6] The display device 100 according to the present embodiment generates a diagram image SH as a position display image indicating the position of an object including the other vehicle VX, and displays part or all of the diagram image SH and the captured image. Display the video that contains it. As described above, since the diagram image SH indicating the position of the other vehicle VX as the object and the image including part or all of the captured image can be displayed together, the positional relationship between the host vehicle V and the other vehicle VX can be grasped. Can be displayed easily.

[7] The display device 100 according to the present embodiment generates a background image SH as a position display image indicating the position where an object including the other vehicle VX is present, and includes a part or all of the background image SH and the captured image. Is displayed. As described above, the background image SH indicating the position of the other vehicle VX as the object and the image including a part or all of the captured image can be displayed together, so that the positional relationship between the host vehicle V and the other vehicle VX can be grasped. Easy video can be displayed.

[8] The display device 100 of the present embodiment generates a shadow image SH as a position display image indicating the position of an object including the other vehicle VX in the captured image, and displays an image including the shadow image SH. The display device 100 according to the present embodiment generates a shadow image SH imitating a shadow generated when light is irradiated to an object including the other vehicle VX, and displays an image including the shadow image SH and a part or all of the captured image. Display. Thus, since the positional relationship between the other vehicle VX and the host vehicle V can be expressed by the shadow images SH1 and SH2 of the other vehicle VX, the positional relationship between the host vehicle V and the surrounding object such as the other vehicle VX can be expressed. Easy-to-understand video can be displayed.

[9] The display device 100 of the present embodiment sets a projection plane SQ for projecting a position display image along the direction in which the host vehicle V moves, and is adjacent to the travel lane Ln2 in which the host vehicle V travels (moves). A position display image for displaying the position of the other vehicle VX1 traveling in the adjacent lane Ln1 is generated. By projecting the position display image SH1 of the other vehicle VX1 (target) traveling in the adjacent lane Ln1 onto the common projection plane SQ together with the position display image SH of the host vehicle V, the position of the host vehicle V and the other vehicle V1. It is possible to present video that makes it easy to understand the relationship. Moreover, since the video includes a part or all of the captured image around the host vehicle V, the positional relationship is easier to grasp. In addition, by setting the projection surface SQ along the traveling direction of the host vehicle V, it is possible to present an image in which the distance between the host vehicle V and the other vehicle VX1 can be easily recognized.

[10] The display device 100 according to the present embodiment has an area of a position display image (a diagram image, a background image, or a shadow image) SH of an object having a relatively high speed, and an object having a relatively low speed. The position display images (line image, background image, or shadow image) SH1 and SH2 of the object are generated so as to be larger than the area of the position display image (line image, background image, or shadow image) SH. In this way, the position display image (line image, background image, or shadow image) of the relatively high-speed object is displayed relatively large, so that an image that calls attention to the high-speed object is displayed. Can be displayed.

[11] The display device 100 according to the present embodiment changes the size of the position display image (a diagram image, a background image, or a shadow image) SH according to the attribute of the lane on which the host vehicle V and the other vehicle VX travel. As a result, the driver's attention to the other vehicle VX traveling at a high speed can be alerted in the same way as when the size of the position display image (line image, background image, or shadow image) SH is controlled by the actual vehicle speed.

[12] In the acceleration state, the display device 100 according to the present embodiment shifts the position of the reference point for observing the moving object or the object to the rear to thereby display the position display image (the diagram image, the background image, or the shadow image). ) It is possible to display an image suitable for the judgment situation of the driver that shifts the SH projection position forward.
In particular, when the position display image is a shadow image, the display device 100 of the present embodiment shifts the position of the virtual light source LG of the shadow image SH backward in the acceleration state, resulting in the shadow image SH as a result. The projected position can be shifted forward, and a shadow image suitable for the driver's judgment situation can be displayed.

[13] When the display device 100 according to the present embodiment is in a decelerating state, the position display image (a diagram image, a background image, or a shadow image) is obtained by shifting the position of a reference point for observing a moving object or object forward. ) The SH projection position can be shifted backward to display an image suitable for the driver's judgment status.
In particular, when the position display image is a shadow image, the display device 100 according to the present embodiment shifts the position of the virtual light source LG forward when the vehicle is in a decelerating state, thereby resulting in the projection position of the shadow image SH. It is possible to display a shadow image suitable for the judgment situation of the driver by shifting backward.

[14] The display device 100 according to the present embodiment generates and displays a position display image including a shadow image SH, a diagram image SH, and a background image SH indicating the movable range of the movable member of the host vehicle V, thereby displaying the position display image. The driver of the vehicle V can determine the parking position of the host vehicle V in consideration of the work after the host vehicle V is parked.

[15] The display device 100 according to the present embodiment causes the display method of the present embodiment to execute the above-described effect.

The embodiment described above is described for easy understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

That is, in this specification, the display system 1 including the display device 100 as one embodiment of the display device according to the present invention will be described as an example, but the present invention is not limited to this.

In this specification, the display device 100 including the control device 10 including the CPU 11, the ROM 12, and the RAM 13 is described as an embodiment of the display device according to the present invention, but the present invention is not limited to this.

Further, in this specification, as one aspect of a display device having an image acquisition unit, an information acquisition unit, an image generation unit, and a display unit according to the present invention, an image acquisition function, an information acquisition function, and an image generation The display device 100 including the control device 10 that executes the function and the display function will be described as an example, but the present invention is not limited to this.

DESCRIPTION OF SYMBOLS 1 ... Display system 100 ... Display apparatus 10 ... Control apparatus 11 ... CPU
12 ... ROM
13 ... RAM
DESCRIPTION OF SYMBOLS 200 ... Mobile body apparatus 40 ... Camera 41 ... Distance measuring device 50 ... Controller 60 ... Sensor 61 ... Speed sensor 62 ... Front-back acceleration sensor 70 ... Navigation apparatus 71 ... Position detection apparatus 711 ... GPS
72 ... Map information 73 ... Road information 80 ... Display

Claims

Image acquisition means for acquiring a captured image captured by a camera mounted on a moving body;
Information acquisition means for acquiring position information of the mobile body;
A reference point is set using the position information of the moving body, a position display image indicating the position of the moving body observed from the reference point in the captured image is generated, and the position display image, the captured image, Image generating means for generating video including
Display means for displaying the video generated by the image generation means;
A display device.
The image generation means sets the viewpoint of viewing the moving object as the reference point using the position information of the moving object, and indicates the position of the moving object when the moving object is viewed from the viewpoint The display device according to claim 1, wherein the position display image including a diagram image is generated.
The image generation means sets the viewpoint of viewing the moving object as the reference point using the position information of the moving object, and indicates the position of the moving object when the moving object is viewed from the viewpoint The display device according to claim 1, wherein the position display image including a background image is generated.
The image generating means sets the virtual light source as the reference point using the position information of the moving body, and the existence of the moving body imitating a shadow that is generated when light is emitted from the virtual light source to the moving body The display device according to claim 1, wherein the position display image including a shadow image indicating a position is generated.
The information acquisition means acquires position information of an object existing around the moving body,
The image generation means sets a reference point using the position information of the object, generates a position display image indicating the position of the object observed from the reference point in the captured image, and displays the position display The display device according to any one of claims 1 to 4, wherein an image including an image and the captured image is generated.
The image generation means sets the viewpoint of viewing the object as the reference point using the position information of the object, and indicates the position of the object when the object is viewed from the viewpoint The display device according to claim 5, wherein the position display image including a diagram image is generated.
The image generation means sets the viewpoint of viewing the object as the reference point using the position information of the object, and indicates the position of the object when the object is viewed from the viewpoint The display device according to claim 5, wherein the position display image including a background image is generated.
The image generation means sets a virtual light source as the reference point using the acquired position information of the object, and imitates a shadow generated when light is emitted from the virtual light source to the object The display device according to claim 5, wherein the position display image including a shadow image indicating a presence position of the image is generated.
The image generation means sets a projection plane for projecting the position display image along a direction in which the moving body moves, and the presence of the object moving in an adjacent lane adjacent to the moving lane in which the moving body moves The display device according to any one of claims 5 to 8, wherein a position display image indicating a position is generated.
The information acquisition means acquires the speed of the object,
The area of the position display image indicating the position of the object having the relatively low acquired speed is the area of the area of the position display image indicating the position of the object having the relatively high acquired speed. The display device according to any one of claims 5 to 9, wherein the position display image is generated so as to be larger than an area.
The information acquisition means acquires attribute information of a lane in which the object travels,
The image generation means, when acquiring attribute information that the lane on which the object travels is an overtaking lane, than when acquiring attribute information that the lane on which the object is traveling is not an overtaking lane, The display device according to any one of claims 5 to 9, wherein the position display image is generated so that an area of a region of the position display image indicating a position of an object traveling on the overtaking lane is increased.
The information acquisition means acquires acceleration in the traveling direction of the moving body,
12. The image generating unit according to claim 1, wherein when the moving body determines that the moving body is in an accelerated state from the acquired acceleration, the position of the reference point is shifted to the opposite side of the moving direction of the moving body. A display device according to claim 1.
The information acquisition means acquires acceleration in the traveling direction of the moving body,
The image generation means shifts the position of the reference point toward the traveling direction of the moving body when the moving body determines that the moving body is in a decelerating state from the acquired acceleration. The display device according to item.
The display device according to any one of claims 1 to 13, wherein the position display image of the movable body includes an image indicating a movable range of a movable member of the movable body.
A display method executed by a display device used for a mobile body,
The display device includes an image acquisition unit, an information acquisition unit, an image generation unit, and a display unit,
The image acquisition means executes a step of acquiring a captured image captured by a camera mounted on a moving body,
The information acquisition means executes a step of acquiring position information of the moving body,
The image generation means sets a reference point using the acquired position information of the moving body, generates a position display image indicating the position of the moving body observed from the reference point in the captured image, and Executing a step of generating an image including a position display image and the captured image;
A display method in which the display means executes a step of displaying the video.