JP5412979B2 - Peripheral display device - Google Patents

Description

  The present invention relates to a peripheral display device, and more particularly, to a peripheral display device that displays a bird's-eye view of a surrounding state of a mobile object capable of self-running.

  Two-dimensional images around the vehicle are individually picked up by a plurality of imaging means, and a plurality of overhead images are created with the road surface as a projection plane by converting each image to a viewpoint, and the plurality of overhead images are arranged around the vehicle. Various techniques for obtaining a bird's-eye view of the vicinity of the host vehicle by combining them are disclosed.

  For example, Patent Document 1 discloses a configuration that includes a video selection unit and switches to an image of a camera that displays the region when a specific region of the displayed composite overhead image is selected.

JP 2006-321419 A

  The technique disclosed in Patent Document 1 described above is such that when a specific region of the overhead view image is selected, the image is actually switched to the image actually displayed by the camera that displays the region. The section is only to select the direct video from each camera to obtain the synthesized overhead image, and the direct video from each camera is a partial video, not a bird's-eye view, For users, the situation is difficult to grasp and is not always satisfactory.

  The present invention has been made to solve the above-described problems, and can easily obtain an overhead image viewed from an arbitrary viewpoint position in the display of an overhead image around a mobile object such as an automobile that can be self-propelled. An object is to provide a peripheral display device.

According to a first aspect of the peripheral display device of the present invention, a plurality of cameras that acquire peripheral image data in a self-propelled movable body, and first three-dimensional image data is calculated based on the image data. A dimensionalization calculation unit; a coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system based on the moving body to form second 3D image data; An integration unit that generates integrated data obtained by integrating the three-dimensional image data of 2 and the data of the three-dimensional model of the moving object, and a virtual camera virtually arranged outside the moving object in the moving object coordinate system An image generation unit that generates overhead image data obtained by observing the integrated data, a viewpoint position change unit that changes the viewpoint position of the virtual camera, and a virtual camera posture that determines the posture of the virtual camera whose viewpoint position has been changed. With the decision part The virtual camera attitude determination unit, when changing the viewpoint position of the virtual camera, a virtual camera line-of-sight direction vector set as a vector connecting the changed viewpoint position and the gazing point to be watched by the virtual camera; Using the virtual camera upward direction vector associated with any axial direction of the moving body coordinate system, the virtual camera faces the direction of the virtual camera line-of-sight direction vector and the upward direction of the overhead image data is the virtual direction The orientation of the virtual camera is determined so as to match the direction of the camera upward vector, and the integration unit generates the integrated data based on the orientation of the virtual camera and the changed viewpoint position.

  In a second aspect of the peripheral display device according to the present invention, the plurality of cameras are a plurality of stereo cameras.

  In a third aspect of the peripheral display device according to the present invention, the position of the gazing point is defined by gazing point position information set in advance on the three-dimensional model of the moving body.

  A fourth aspect of the peripheral display device according to the present invention includes a gazing point position information changing unit that changes gazing point position information that defines the position of the gazing point.

  The fifth aspect of the peripheral display device according to the present invention further includes a driving status grasping unit that outputs driving status data related to the driving status of the mobile body to the gazing point location information changing unit, and the gazing point location information is: It is changed based on the driving situation data.

  In a sixth aspect of the peripheral display device according to the present invention, the driving status data includes driving state data including forward and backward movement of the moving body, and the gazing point position information is included in the driving state data. Will be changed based on.

  In a seventh aspect of the peripheral display device according to the present invention, the driving situation data includes steering angle information for controlling straight travel and right / left turn of the moving body, and the gazing point position information is included in the steering angle information. Continuously changed based on.

  According to an eighth aspect of the peripheral display device of the present invention, the presence / absence of an obstacle on the road surface is determined based on image data obtained by the plurality of cameras, and information on the relative position to the obstacle is obtained. An obstacle determination unit that outputs to the gaze position information change unit as obstacle data is further provided, and the gaze position information is changed based on the obstacle data.

  In a ninth aspect of the peripheral display device according to the present invention, the gazing point position information change unit calculates a region including the position of the obstacle based on the obstacle data, and sets the center position of the region. The gaze point position information is changed so that the closest gaze point is selected.

  In a tenth aspect of the peripheral display device according to the present invention, based on image data obtained by the plurality of cameras, the environmental situation around the moving body is grasped, and the gazing point position information is changed as environmental situation data. An environmental condition grasping unit for outputting to the part, and the gazing point position information is changed based on the environmental condition data.

  An eleventh aspect of the peripheral display device according to the present invention has data indicating whether the environmental status data is nighttime or daytime obtained based on the brightness of the image data, and the gaze point position The information is changed between nighttime and daytime.

  In a twelfth aspect of the peripheral display device according to the present invention, the viewpoint position changing unit virtually sets a hemisphere centered on the three-dimensional model of the moving body, and calculates the latitude and longitude on the hemisphere. The viewpoint position of the virtual camera is changed by designating.

  In a thirteenth aspect of the peripheral display device according to the present invention, the viewpoint position changing unit changes the viewpoint position in a direction in which the virtual camera moves away from and a direction in which the virtual camera moves away from the three-dimensional model of the moving body.

  According to the first aspect of the peripheral display device according to the present invention, it is possible to arbitrarily change the viewpoint position of the virtual camera, and in the display of the overhead image around the moving body, the overhead image viewed from the arbitrary viewpoint position is displayed. It can be obtained easily.

  According to the second aspect of the peripheral display device of the present invention, it is possible to acquire three-dimensional data by acquiring image data around a moving object using a plurality of stereo cameras.

  According to the third aspect of the peripheral display device of the present invention, the position of the gazing point of the virtual camera is defined by the gazing point position information set in advance on the three-dimensional model of the moving object. Even when the position is arbitrarily changed, the position of the gazing point is fixed, so it is easy to obtain the attitude of the virtual camera.

  According to the fourth aspect of the peripheral display device of the present invention, since the gazing point position information changing unit for changing the gazing point position information defining the position of the gazing point of the virtual camera is provided, the gazing point of the virtual camera is moved. It is possible to change according to the driving situation of the body, and it is possible to provide an easy-to-see overhead image for the user.

  According to the fifth aspect of the peripheral display device according to the present invention, the point-of-gaze position information is changed based on the driving situation data, so that an overhead image suitable for the driving situation of the mobile object can be provided.

  According to the sixth aspect of the peripheral display device of the present invention, the point-of-gaze position information is changed based on the driving state data including the forward and backward movements of the moving body, so that the moving body moves backward and forwards. It is possible to provide an easy-to-see overhead image in each case.

  According to the seventh aspect of the peripheral display device of the present invention, since the gazing point position information is continuously changed based on the steering angle information of the moving body, the movement is performed when the moving body turns right or left. It is possible to provide a bird's-eye view image that changes as the body progresses.

  According to the eighth aspect of the peripheral display device of the present invention, since the point-of-gaze position information is changed based on the obstacle data, an overhead image including the obstacle can be provided.

  According to the ninth aspect of the peripheral display device of the present invention, an overhead image near the obstacle can be provided.

  According to the tenth aspect of the peripheral display device according to the present invention, since the point-of-gaze position information is changed based on the environmental situation data, it is possible to provide an overhead view image that matches the environmental situation of the mobile object.

  According to the eleventh aspect of the peripheral display device according to the present invention, the point-of-gaze position information is changed between the case of nighttime and the case of daytime. It is possible to provide an overhead image suitable for each of the above.

  According to the twelfth and thirteenth aspects of the peripheral display device according to the present invention, it is possible to provide a practical method in changing the viewpoint position of the virtual camera in the viewpoint position changing unit.

It is a figure which shows the example of arrangement | positioning of a stereo camera. It is a block diagram which shows the structure of embodiment of the peripheral display apparatus which concerns on this invention. It is a figure explaining the model for setting the viewpoint position of a virtual camera. It is a figure explaining the other model for setting the viewpoint position of a virtual camera. It is a figure which shows typically the determination method of a virtual camera gaze direction vector. It is a figure explaining the setting method of a virtual camera upward direction vector. It is a figure explaining the change operation | movement of the gazing point position of a virtual camera. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows an example of a gaze point position. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows an example of a gaze point position. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows an example of a gaze point position. It is a figure which shows an example of a gaze point position. It is a figure which shows an example of a gaze point position. It is a figure which shows an example of a gaze point position. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows an example of a gaze point position. It is a figure which shows the bird's-eye view image from a virtual camera. It is a figure which shows the other example of arrangement | positioning of a stereo camera.

<Embodiment>
<Device configuration>
FIG. 1 is a diagram showing an arrangement example of stereo cameras in a vehicle VC equipped with a peripheral display device according to the present invention.

  In FIG. 1, a vehicle VC is equipped with four stereo cameras that acquire front, left, and right images. That is, a stereo camera SC1 that acquires a front image, a stereo camera SC2 that acquires a right image with respect to the front, a stereo camera SC3 that acquires a rear image, and a stereo camera SC4 that acquires a left image with respect to the front. I have.

  By using such a system, it is possible to acquire image data around the host vehicle and change it to an overhead view seen from above the host vehicle by changing the viewpoint, but in the following description, A case where images acquired by the stereo cameras SC1 and SC4 are used will be described as an example.

  The actual arrangement position of each camera differs depending on the vehicle type. For example, the stereo camera SC1 is arranged at the position of the room mirror, and the stereo cameras SC2 and SC4 are arranged at the positions of the right and left side mirrors, respectively. Stereo camera SC3 is arranged at the position of the rear bumper. The above arrangement is an example, and the arrangement of the stereo camera is not limited to this.

  FIG. 2 is a block diagram showing the configuration of the peripheral display device 100 according to the present invention. As shown in FIG. 2, the peripheral display device 100 is output from the stereo cameras SC <b> 1 to SC <b> 4, a display image generation unit 80 that performs image processing of image data obtained by the stereo cameras SC <b> 1 to SC <b> 4, and the display image generation unit 100. An image display unit 90 such as an in-vehicle monitor that displays processed image data and viewpoint position changing means 70 that changes the viewpoint position of the overhead image are provided as main components.

  The display image generation unit 80 includes three-dimensional calculation units 11, 21, 31, and 41 that calculate three-dimensional image data based on the two-dimensional image data obtained by the stereo cameras SC1 to SC4, and three-dimensional calculation. 3D expressing the position of the stereo camera in the vehicle coordinate system (moving body coordinate system) in the 3D image data (first 3D image data) calculated by each of the units 11, 21, 31 and 41 By applying the transformation matrix, the coordinate transformation units 12 and 22 that transform the three-dimensional image data expressed in the stereo camera coordinate system into the three-dimensional image data (second three-dimensional image data) expressed in the car coordinate system. , 32 and 42, three-dimensional image data expressed in the automobile coordinate system output from the coordinate conversion units 12, 22, 32 and 42, and the three-dimensional model of the host vehicle stored in the storage unit 7. 3D image data in the measurement space (integrated) with the vehicle as the origin of the vehicle coordinate system and the 3D image data expressed in the vehicle coordinate system arranged around the vehicle coordinate system. Data) and an image generation unit 6 that generates the three-dimensional image data generated by the integration unit 5 as overhead image data observed by a camera (virtual camera) arranged at a virtual position. Yes.

  Further, the virtual camera posture determining unit 8 that determines the posture of the virtual camera according to the information on the viewpoint position of the virtual camera from the viewpoint position changing unit 70, and the virtual camera posture and the viewpoint position determined by the virtual camera posture determining unit 8 And a storage unit 51 for storing virtual camera information about integrated data to be integrated by the integration unit 5, for example, viewpoint position and orientation information, and driving of the host vehicle provided outside the peripheral display device 100. The driving situation data output means 200 for outputting the situation data is received, the driving situation of the own vehicle is grasped, the driving state (forward / reverse / stop) of the own vehicle, speed, steering angle information (straight / (Right turn / left turn) etc., driving status grasping section 3 that outputs driving situation data, environmental situation grasping section 4 that grasps the environmental situation around the vehicle and outputs environmental situation data, and stereo Based on the image data obtained by the cameras SC1 to SC4, an obstacle determination unit 9 that determines the presence or absence of an obstacle on the road surface and outputs the obstacle data, a driving condition grasping part 3, an environmental condition grasping part 4, and A gazing point position information changing unit 10 that receives the output of the obstacle determination unit 9 and the gazing point position information stored in the storage unit 7 and changes the gazing point position information of the virtual camera is provided.

  Here, the obstacle determination unit 9 receives the three-dimensional image data expressed in the car coordinate system corresponding to the image data obtained by each stereo camera from the integration unit 5, and detects the road surface from the three-dimensional image data. When an object having a height greater than or equal to a predetermined value with respect to the road surface is detected, the fact that it is determined as an obstacle is output, and information on the relative position of the vehicle relative to the obstacle is output.

  The environmental condition grasping unit 4 receives the three-dimensional image data expressed in the car coordinate system corresponding to the image data obtained by each stereo camera from the integrating unit 5 and determines whether it is nighttime or daytime from the brightness value. In addition, information about the weather (sunny weather, rain, fog) is output as environmental status data from the image.

<Operation>
<Basic operation>
Next, the basic operation of the peripheral display device 100 having the above-described configuration will be described.

  The stereo cameras SC1 to SC4 are both taken simultaneously from different viewpoints by the two cameras, and each of the three-dimensional calculation units 11, 21, 31 and 41 corresponds to the corresponding stereo camera based on the principle of triangulation. The feature points of the two images obtained in the above are associated with each other (corresponding point search), and each corresponding point is obtained from parameters such as parallax, focal length, and base line length for each of the pair of corresponding points obtained as a result. The three-dimensional image data is calculated by obtaining the three-dimensional position for each set.

  Since the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41 is expressed in the stereo camera coordinate system, coordinates are used for conversion into the three-dimensional image data expressed in the car coordinate system. Coordinate conversion is performed in the conversion units 12, 22, 32 and 42.

  This coordinate transformation is executed by applying predetermined three-dimensional transformation matrices Mat1, Mat2, Mat3, and Mat4 to the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41, respectively. The

  Here, as shown in FIG. 2, the three-dimensional transformation matrices Mat1 to Mat4 are stored in the storage units 14, 24, 34, and 44 associated with the coordinate transformation units 12, 22, 32, and 42, respectively. .

  The three-dimensional image data expressed in the vehicle coordinate system output from the coordinate conversion units 12, 22, 32, and 42 is given to the integration unit 5, and the data of the three-dimensional model of the host vehicle stored in the storage unit 7 Integrated. At this time, the integration is executed based on the viewpoint position and orientation information of the virtual camera included in the virtual camera information stored in the storage unit 51.

  The image generation unit 6 receives the three-dimensional integrated data output from the integration unit 5, generates overhead image data and supplies it to the image display unit 90.

<Viewpoint position change operation>
Next, the operation of changing the viewpoint position of the virtual camera in the peripheral display device 100 according to the present invention will be described.

  In the peripheral display device 100, a virtual camera viewpoint of the overhead image displayed on the in-vehicle monitor by the driver or the like accessing the viewpoint position changing means 70 by an operation key of the in-vehicle monitor or a cursor operation on the operation screen. The position can be changed arbitrarily.

  FIG. 3 shows one model for setting the viewpoint position of the virtual camera. As shown in FIG. 3, a hemisphere HS centered on a three-dimensional model of the host vehicle VC is virtually set, and a virtual camera CX is set on the surface of the hemisphere HS. FIG. 3 shows the X, Y, and Z directions of the automobile coordinate system. The direction perpendicular to the bottom surface (or ceiling surface) of the vehicle is defined as the Z-axis direction, and each direction is perpendicular to the Z-axis direction. The direction formed is the X-axis direction and the Y-axis direction.

  The in-vehicle monitor displays the hemisphere HS, the host vehicle VC, and the virtual camera CX, and the driver or the like moves the virtual camera CX with, for example, the up / down / left / right cursor keys on the surface of the hemisphere HS of the virtual camera CX. The latitude (for example, an angle around the Z axis of the vehicle coordinate system) and the longitude (for example, an angle with respect to the Z axis of the vehicle coordinate system) are changed. In this case, the Z-axis is an axis in the Z-axis direction with the center point PC of the host vehicle VC as the origin.

  Thereby, the viewpoint position of the virtual camera CX can be set to an arbitrary position on the surface of the hemisphere HS. Although the virtual camera CX can be moved anywhere on the surface of the hemisphere HS on the monitor, it displays a bird's-eye view image from the relationship between the position of the stereo cameras SC1 to SC4 (FIG. 1) and the angle of view. In some cases, it may not be possible, but in such a case, a process such as displaying the fact on the monitor may be executed.

  FIG. 4 shows another model for setting the viewpoint position of the virtual camera. In addition to the up / down / left / right cursor keys, a zoom-in / zoom-out key is prepared. When the zoom-in key is pressed, the virtual camera CX moves closer to the host vehicle VC. When the zoom-out key is pressed, the virtual camera CX It moves in a direction away from the host vehicle VC.

  In FIG. 4, the point (gaze point) that the virtual camera CX is gazing at is the center point PC of the host vehicle VC, and the virtual camera VC moves in the direction along the Z axis with the center point PC as the origin. However, the present invention is not limited to this.

  Further, it can be used in combination with the model shown in FIG. 3, and if the virtual camera CX is configured so that the radius of the hemisphere HS becomes smaller or larger each time the zoom-in / zoom-out is performed, the zoom-in is performed. In this state, the virtual camera CX can be moved to an arbitrary position on the surface of the hemisphere HS, or the virtual camera CX can be moved to an arbitrary position on the surface of the hemisphere HS in a zoomed-out state.

  As described above, in the peripheral display device 100, the user can arbitrarily change the viewpoint position of the virtual camera, so that the overhead image viewed from an arbitrary viewpoint position can be easily obtained in the display of the overhead image around the host vehicle. .

<Determination of posture>
The virtual camera attitude determination unit 8 that has received information on the viewpoint position of the virtual camera from the viewpoint position changing means 70 determines the attitude of the virtual camera.

  To determine the attitude of the virtual camera, it is necessary to determine a virtual camera line-of-sight direction vector and a virtual camera upward direction vector that serves as a reference for the direction of the virtual camera.

  The virtual camera line-of-sight direction vector is set as a vector connecting the viewpoint position of the virtual camera and the gazing point of the virtual camera.

  FIG. 5 schematically shows a method of determining the virtual camera line-of-sight direction vector. When the gazing point of the virtual camera is the center point PC of the three-dimensional model of the host vehicle VC, the center point PC is the origin. A virtual camera CX1 existing on the Z axis and a virtual camera CX2 existing on an axis that forms a predetermined angle with respect to the Z axis are shown.

  In the virtual cameras CX1 and CX2, the direction toward the center point PC (that is, the gazing point) is the direction of the virtual camera viewing direction vector.

  Next, a method for setting the virtual camera upward vector will be described. As shown in FIG. 6, the virtual camera upward direction vector V1 is on a plane including the virtual camera viewing direction vector V2 and the vehicle coordinate system Z-axis direction vector VZ, and is orthogonal to the virtual camera viewing direction vector V2. , And an acute angle with the vehicle coordinate system Z-axis direction vector VZ.

  When this method is employed, taking the virtual camera CX2 as an example, the upward direction vector of the virtual camera is represented by a vector V21 in FIG. 5, and the vector V21 is directed away from the front of the vehicle.

  Further, the virtual camera upward vector may be set by replacing the vehicle coordinate system Z-axis direction vector VZ shown in FIG. 6 with the vehicle coordinate system X-axis direction vector.

  When this method is employed, taking the virtual camera CX2 as an example, the virtual camera upward direction vector is represented by a vector V22 in FIG. In this case, the X-axis direction corresponds to the front direction of the vehicle, and the vector V22 faces the front direction of the vehicle.

  Further, when the visual line direction of the virtual camera is nearly parallel to the vehicle coordinate system Z-axis direction vector, the virtual camera upward direction vector may be approximated by the vehicle coordinate system X-axis direction vector.

  When this method is adopted, if the virtual camera CX1 is taken as an example, the virtual camera upward direction vector is represented by a vector V11 in FIG. In this case, the X-axis direction corresponds to the front direction of the vehicle, and the vector V11 is parallel to the X-axis direction and faces the front portion of the vehicle.

  Further, when the line-of-sight direction of the virtual camera is close to parallel to the vehicle coordinate system X-axis direction vector, the virtual camera upward direction vector may be approximated by the vehicle coordinate system Z-axis direction vector.

  The virtual camera posture determination unit 8 that has determined the virtual camera line-of-sight direction vector and the virtual camera upward direction vector by the method described above determines the posture of the virtual camera based on both vectors.

  That is, taking the virtual camera CX1 in FIG. 5 as an example, the virtual camera line-of-sight direction vector is represented by a vector along the Z axis, and the virtual camera upward direction vector is represented by a vector V11. In such a case, the virtual camera CX1 is determined to be in a posture in which the own vehicle VC is looked down.

  Information on the attitude and position of the virtual camera determined by the virtual camera attitude determination unit 8 is stored as virtual camera information in the storage unit 51, and is read when the integration unit 5 generates integrated data.

  Note that, as described above, the position information of the gazing point is, for example, the position coordinates when the center point PC of the three-dimensional model of the host vehicle VC is the gazing point, but there are a plurality of other locations. Are stored in the storage unit 7 as gazing point position information, and read out as necessary in the process of determining the attitude of the virtual camera in the virtual camera attitude determining unit 8. In FIG. 2, the storage unit 7 once gives the gazing point position information changing unit 10, and the gazing point position information changing unit 10 gives the virtual camera posture determining unit 8. This is related to the gaze point changing operation described below.

<Gaze position change operation>
The peripheral display device 100 has a function that allows the user to arbitrarily change the viewpoint position of the virtual camera and automatically changes the point of interest of the virtual camera as the vehicle moves. Hereinafter, the operation of changing the point of gaze position of the virtual camera will be described with reference to FIGS.

<Gaze point location information>
The position information of the gazing point of the virtual camera is information on which part of the three-dimensional model of the host vehicle VC the gazing point should be gazed at, and is represented by the coordinate value of a specific point of the three-dimensional model.

  For example, in FIG. 7, three virtual cameras C1, C2, and C3 are assumed, and the gazing points of the respective cameras are represented as VP1, VP2, and VP3.

  That is, the gazing point VP1 corresponds to the center point of the host vehicle VC, and the virtual camera C1 is present directly above the host vehicle VC and is in a posture of looking down the host vehicle VC.

  The overhead view image obtained by the virtual camera C1 is an image of the host vehicle VC viewed from directly above. This image is shown in FIG.

  The gazing point VP2 corresponds to a coordinate point around the driver's seat of the host vehicle VC, and the virtual camera C2 exists obliquely upward in the front direction of the host vehicle VC, and the posture in which the front portion of the host vehicle VC is viewed obliquely from above. It is in.

  The bird's-eye view image obtained by the virtual camera C2 is an image obtained by viewing the front portion of the host vehicle VC from obliquely above. This image is shown in FIG.

  The gazing point VP2 corresponds to a coordinate point around the driver's seat of the host vehicle VC, and the virtual camera C2 exists obliquely upward in the front direction of the host vehicle VC, and the posture in which the front portion of the host vehicle VC is viewed obliquely from above. It is in.

  The bird's-eye view image obtained by the virtual camera C2 is an image obtained by viewing the front portion of the host vehicle VC from obliquely above. This image is shown in FIG.

  The gazing point VP3 corresponds to a coordinate point at the rear of the host vehicle VC, and the virtual camera C3 is present obliquely upward in the rear direction of the host vehicle VC and is in an attitude of viewing the rear portion of the host vehicle VC from obliquely above.

  The bird's-eye view image obtained by the virtual camera C3 is an image obtained by viewing the rear portion of the host vehicle VC obliquely from above. This image is shown in FIG.

  Note that the viewpoint position and orientation information of the virtual cameras C <b> 1 to C <b> 3 described above is stored as virtual camera information in the storage unit 51 as described above, and is read out as necessary for integration processing in the integration unit 5. It has a configuration.

  One of the features of the peripheral display device 100 is that the gazing point position of the virtual camera is automatically changed in accordance with the movement of the host vehicle. In this case, the gazing point position stored in the storage unit 7 is used. The information cannot be handled, and the gazing point position information changed by the gazing point position information changing unit 10 is required.

<Example 1 of changing gaze point position information>
Hereinafter, the change of the gazing point position information will be described while explaining the operation of automatically changing the gazing point position of the virtual camera according to the movement of the host vehicle.

  FIG. 11 is a diagram illustrating an example of a gazing point position when the host vehicle VC is moving forward, using the gazing point position information set in the vehicle front part of the three-dimensional model of the host vehicle VC. A close display image is generated. FIG. 11 shows an example in which the gazing point position information of the gazing point VP1 at the center is used among the three gazing points set in the front part of the vehicle.

  FIG. 12 shows a display image generated using the gazing point position information of the gazing point VP1, and the display screen DP of the image display unit 90 (FIG. 2) is closer to the front part of the three-dimensional model of the host vehicle VC. An overhead image is shown.

  FIG. 13 is a diagram illustrating an example of the position of the point of sight when the host vehicle VC is moving backward, and using the point of sight position information set at the rear of the vehicle of the three-dimensional model of the host vehicle VC, Generate a display image. FIG. 13 shows an example in which the gazing point position information of the gazing point VP3 in the center is used among the three gazing points set in the front part of the vehicle.

  FIG. 14 shows a display image generated using the gazing point position information of the gazing point VP3. The display screen DP of the image display unit 90 (FIG. 2) shows an overhead view of the three-dimensional model of the host vehicle VC near the rear of the vehicle. An image is shown.

  As described above, in the peripheral display device 100, the gazing point is automatically changed according to the forward and backward movements of the host vehicle, and the overhead image corresponding to the changed gazing point is displayed in the image display unit 90.

  As described with reference to FIG. 2, the driving situation grasping unit 3 receives the driving situation data from the driving situation data output means 200 outside the peripheral display device 100 and moves forward / backward. By grasping the situation such as / stop and giving the information of the grasped driving situation to the gazing point position information changing unit 10, the gazing point position information is appropriately changed according to the driving situation.

  The point-of-gaze position information changing unit 10 may be configured to change the point-of-gaze position information by software that calculates optimal point-of-gaze position information in accordance with the driving situation. The configuration may be such that the gazing point position information is changed with reference to a table in which the gazing point position information is associated.

  The gazing point position information changed by the gazing point position information changing unit 10 is given to the virtual camera posture determination unit 8 and is used to determine the posture of the virtual camera as described above.

<Example 2 of changing gaze point position information>
It is also possible to grasp the driving situation of the host vehicle using the steering angle information (straight forward / right turn / left turn). In this case, it is possible to change the gazing point position as shown in FIGS. It becomes.

  FIG. 15 is a diagram illustrating a gazing point position when the host vehicle VC is moving forward. In this case, of the three gazing points set in the front part of the three-dimensional model of the host vehicle VC, The example which uses the gaze point position information of the gaze point VP1 is shown.

  On the other hand, FIG. 16 is a diagram showing the position of the gazing point immediately after the host vehicle VC starts a left turn, and is a situation where the steering angle is small (the steering wheel is not turned too much). In this case, a new gazing point VP is set between the central gazing point VP1 and the left gazing point VP2 set at the front of the vehicle of the three-dimensional model of the host vehicle VC. The gazing point position information of the gazing point VP can be obtained by the gazing point position information changing unit 10 by calculation that complements the gazing point position information of the gazing points VP1 and VP2 on both sides.

  FIG. 17 is a diagram illustrating the position of the point of gaze at the end of the left turn of the host vehicle VC, in which the steering angle is large (the steering wheel is largely turned). In this case, an example is shown in which the gazing point position information of the left gazing point VP2 set in the front part of the three-dimensional model of the host vehicle VC is used.

  As described above, in the peripheral display device 100, it is possible to continuously change the position of the gazing point in accordance with the driving situation of the host vehicle. As another example, when the host vehicle is slow, the host vehicle VC is used. An example of using the gazing point position information of the gazing point set in the vehicle center of the three-dimensional model and using the gazing point position information of the gazing point closer to the front of the vehicle as the speed increases is given.

<Example 3 of changing gaze point position information>
In the peripheral display device 100, in addition to the function of automatically changing the position of the gazing point of the virtual camera according to the movement of the host vehicle as described above, the gazing point of the virtual camera is also used when an obstacle is detected. It has a function to change the position automatically. Hereinafter, this function will be described.

  FIG. 18 is a diagram illustrating an example of the position of the point of sight when the obstacle OB is detected behind the vehicle while the host vehicle VC is moving backward, and is set at the rear part of the three-dimensional model of the host vehicle VC. The example which uses the gaze point position information of the left gaze point VP4 among the two gaze points is shown.

  As shown in FIG. 18, an obstacle OB exists on the left rear side of the host vehicle VC, and the obstacle determination unit 9 determines this with respect to the road surface based on image data obtained by each stereo camera. When detected as an object having a height higher than the value, the fact that the obstacle OB exists and the relative position information of the obstacle OB with respect to the host vehicle VC are output to the gazing point position information changing unit 10. The gazing point position information changing unit 10 changes the gazing point position information based on the relative position information with respect to the obstacle OB.

  In such a case, the gazing point position information changing unit 10 calculates a region including the position of the obstacle OB, and changes the gazing point position information so that the gazing point closest to the center position of this region is selected.

  In the example of FIG. 18, a broken-line area surrounding the obstacle OB is an area including the obstacle OB obtained by calculation, and a cross mark at the center is the center position of the area. As a result, the watch point position information is changed so that the watch point VP4 is selected.

  FIG. 19 shows a display image generated using the gazing point position information of the gazing point VP4. The display screen DP of the image display unit 90 (FIG. 2) shows the vicinity of the left rear part of the three-dimensional model of the host vehicle VC. A bird's-eye view image is shown.

  The driver who sees such a bird's-eye view image can perform operations such as changing the backward path and stopping so as to avoid the obstacle OB.

  FIG. 20 is a diagram illustrating an example of a gazing point position when the garage wall WL is detected while the host vehicle VC is moving backward, and is set at the rear part of the three-dimensional model of the host vehicle VC. The example which uses the gaze point position information of the gaze point VP3 of the center part among the three gaze points is shown.

  As shown in FIG. 20, a wall WL exists so as to surround the rear part of the host vehicle VC, and this is detected by the obstacle determination unit 9, and the fact that the wall WL exists and the wall WL with respect to the host vehicle VC Is output to the gazing point position information changing unit 10. The gazing point position information changing unit 10 changes the gazing point position information based on the relative position information with respect to the wall WL.

  In such a case, the gazing point position information changing unit 10 calculates a region including the position of the wall WL, and changes the gazing point position information so that the gazing point closest to the center position of this region is selected.

  In the example of FIG. 20, a broken line region surrounding the wall WL is a region including the wall WL obtained by calculation, and a cross mark at the center is the center position of the region. As a result, the gazing point position information is changed so that the gazing point VP3 is selected.

  FIG. 21 shows a display image generated using the gazing point position information of the gazing point VP3. The display screen DP of the image display unit 90 (FIG. 2) shows the vicinity of the vehicle rear part of the three-dimensional model of the host vehicle VC. An overhead image is shown.

  The driver who has seen such a bird's-eye view image can perform a backward movement so as not to contact the wall WL.

<Example 4 of changing gaze point position information>
In addition to the function of automatically changing the position of the gazing point of the virtual camera when an obstacle is detected as described above, the peripheral display device 100 has its own vehicle grasped by the environment situation grasping unit 4. It also has a function of automatically changing the position of the gazing point of the virtual camera based on the surrounding situation. Hereinafter, this function will be described.

  As described above, the environmental condition grasping unit 4 (FIG. 2) determines whether it is nighttime or daytime, and sets parameters regarding the weather (sunny weather, rain, fog) state as a gazing point position information changing unit. 10 is provided.

  In the point-of-gaze position information changing unit 10, when receiving information from the environmental condition grasping unit 4, for example, at night, the point-of-gaze position is set so that the point of sight is set in the direction in which the light is illuminated in the traveling direction of the host vehicle. Change information. Thereby, a visually meaningful image can be preferentially displayed.

  Further, when the gazing point position information changing unit 10 receives information from the environmental condition grasping unit 4 such as raining or in the fog, the state-of-the-art note in the vehicle traveling direction of the three-dimensional model. Change gaze point position information so that the viewpoint is selected. Thereby, the image of the advancing direction can be displayed in as wide a range as possible.

<Modification 1>
FIG. 1 shows an example in which one stereo camera is arranged in each of the four directions of the vehicle VC. However, as shown in FIG. 22, two stereo cameras may be arranged in each of the four directions of the vehicle VC. good.

  That is, as shown in FIG. 22, stereo cameras SC1 and SC2 that acquire the left and right images of the vehicle VC are arranged, and the stereo camera SC3 that acquires the images of the front side and the rear side on the right side with respect to the front side. SC4 is arranged, stereo cameras SC5 and SC6 for obtaining the left and right images at the rear are arranged, and stereo cameras SC7 and SC8 for obtaining the images at the rear and the front on the left side with respect to the front are arranged.

  By using such a system, it is possible to acquire peripheral image data with fewer blind spots.

<Modification 2>
In the above description, by using stereo cameras SC1 to SC4 (or stereo cameras SC1 to SC8), two-dimensional image data including three-dimensional information can be obtained. Based on the two-dimensional image data, 3D Although the configuration in which the three-dimensional image data is calculated by the dimensionalization calculation units 11, 21, 31, and 41 is shown, it is possible to obtain the three-dimensional image data from the two-dimensional image data obtained by the monocular camera.

  In other words, by performing projective transformation processing on 2D image data obtained by a monocular camera, it is possible to convert an image taken from a certain viewpoint into an image viewed from a different viewpoint. By providing the calculation units 11, 21, 31 and 41, 3D image data can be obtained from 2D image data having no 3D information.

  Further, instead of a stereo camera, for example, using a TOF (time-of-flight measurement) type three-dimensional distance measuring camera using infrared rays, an image of the target area and distance information can be obtained. Based on the data, Three-dimensional image data can be calculated by the three-dimensional calculation units 11, 21, 31 and 41.

  A TOF type three-dimensional distance measuring camera has a configuration in which each of a plurality of pixels constituting a CMOS sensor or a CCD sensor measures a distance to an object in a predetermined space.

DESCRIPTION OF SYMBOLS 3 Driving condition grasping part 4 Situation grasping part 5 Integration part 8 Virtual camera attitude | position determination part 9 Obstacle determination part 10 Gaze point position information change part 70 View point position change part SC1-SC4 Stereo camera

Claims (13)

A plurality of cameras that acquire peripheral image data in a self-propelled moving object;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system based on the moving body to obtain second 3D image data;
An integration unit that generates integrated data obtained by integrating the second three-dimensional image data and the data of the three-dimensional model of the moving body;
An image generation unit that generates overhead image data obtained by observing the integrated data with a virtual camera virtually disposed outside the moving body in the moving body coordinate system;
A viewpoint position changing unit for changing the viewpoint position of the virtual camera;
A virtual camera posture determination unit that determines the posture of the virtual camera whose viewpoint position has been changed,
The virtual camera posture determination unit
When changing the viewpoint position of the virtual camera, a virtual camera line-of-sight direction vector set as a vector connecting the changed viewpoint position and the gazing point to be watched by the virtual camera , and any axis of the moving body coordinate system The virtual camera upward direction vector associated with the direction is used so that the virtual camera faces the direction of the virtual camera line-of-sight direction vector and the upward direction of the overhead image data matches the direction of the virtual camera upward direction vector. Determine the attitude of the virtual camera;
The integration unit
A peripheral display device that generates the integrated data based on an attitude of the virtual camera and the changed viewpoint position.   The peripheral display device according to claim 1, wherein the plurality of cameras are a plurality of stereo cameras. The position of the gaze point is
The peripheral display device according to claim 1, which is defined by gaze point position information set in advance on the three-dimensional model of the moving body.   The peripheral display device according to claim 1, further comprising a gazing point position information changing unit that changes gazing point position information that defines a position of the gazing point. A driving condition grasping unit that outputs driving condition data relating to the driving condition of the moving object to the gazing point position information changing unit;
The gazing point position information is
The peripheral display device according to claim 4, wherein the peripheral display device is changed based on the driving situation data. The driving situation data is
Having data on the driving state including forward and backward movement of the moving body;
The gazing point position information is
The peripheral display device according to claim 5, wherein the peripheral display device is changed based on the data of the operation state. The driving situation data is
Steering angle information for controlling straight travel and right / left turn of the moving body,
The gazing point position information is
The peripheral display device according to claim 5, which is continuously changed based on the steering angle information. Based on the image data obtained by the plurality of cameras, the presence / absence of an obstacle on the road surface is determined, and information on the relative position to the obstacle is output to the gazing point position information changing unit as obstacle data. An obstacle determination unit;
The peripheral display device according to claim 4, wherein the gazing point position information is changed based on the obstacle data. The gazing point position information changing unit
9. The area including the position of the obstacle is calculated based on the obstacle data, and the gazing point position information is changed so that the gazing point closest to the center position of the area is selected. Peripheral display device. Based on the image data obtained by the plurality of cameras, further comprising an environmental situation grasping unit that grasps the environmental situation around the moving body and outputs the environmental situation data to the gazing point position information changing unit,
The peripheral display device according to claim 4, wherein the gazing point position information is changed based on the environmental situation data. The environmental status data is
Having data on whether it is nighttime or daytime obtained based on the brightness of the image data;
The peripheral display device according to claim 10, wherein the gazing point position information is changed between night time and daytime. The viewpoint position changing unit
The periphery according to claim 1, wherein a hemisphere centered on the three-dimensional model of the moving body is virtually set, and the viewpoint position of the virtual camera is changed by specifying latitude and longitude on the hemisphere. Display device. The viewpoint position changing unit
The peripheral display device according to claim 1, wherein the viewpoint position is changed in a direction in which the virtual camera moves away from and a direction in which the virtual camera moves away from the three-dimensional model of the moving body.

Images