JP2010268369A - Peripheral display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a peripheral display device reduced in time spent for display of an overhead image by reducing data to be transferred from an image measurement part to an image processing part, in a peripheral display device for displaying an overhead image around a self-propelled mobile object. <P>SOLUTION: The image measurement part 10 includes: a three-dimensionalization calculation part 11 for calculating three-dimensional image data based on two-dimensional image data obtained by a stereo camera SC1; a coordinate conversion part 12 for converting the three-dimensional image data to three-dimensional image data expressed by an automobile coordinate system by applying, into the three-dimensional image data, a three-dimensional conversion matrix for expressing the position and attitude of the stereo camera in the automobile coordinate system; and an image generation part 13 for combining the three-dimensional image data with data of a three-dimensional model of the own vehicle, and generating, by using the own vehicle as an origin of the automobile coordinate system, overhead image data with the three-dimensional image data, expressed by the automobile coordinate system, arranged around it. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a peripheral display device, and more particularly, to a peripheral display device that displays a bird's-eye view of a surrounding state of a mobile object capable of self-running.

  Two-dimensional images around the vehicle are individually picked up by a plurality of imaging means, and a plurality of overhead images are created with the road surface as a projection plane by converting each image to a viewpoint, and the plurality of overhead images are arranged around the vehicle. Various techniques for obtaining a bird's-eye view of the vicinity of the host vehicle by combining them are disclosed.

  For example, Patent Document 1 discloses a method of providing a stereo camera that captures the rear of a vehicle and creating a display image by changing an image synthesis method according to the reliability of stereo measurement.

  Further, in Patent Document 2, visibility is created by creating a bird's-eye view image including a pseudo image (an image compressed in the height direction) considering the height information of an object using three-dimensional information measured by a stereo camera. A technique for improving the above is disclosed.

JP 2004-56778 A JP 2006-333209 A

  In order to display an image around the vehicle three-dimensionally, a three-dimensional calculation for acquiring three-dimensional information including distance information based on a two-dimensional image acquired by a stereo camera is required. Although it is necessary to perform drawing based on the information, generally, an overhead image is generated by integrating processing of three-dimensional information corresponding to each stereo camera by an image processing unit.

  However, the three-dimensional information has a large amount of data, and it takes time to transfer data from each image measurement unit including a stereo camera to the image processing unit, and also requires time for integration processing. .

  The present invention has been made to solve the above-described problems. In a peripheral display device that displays a bird's-eye view image around a mobile object such as an automobile, it is transferred from the image measurement unit to the image processing unit. An object of the present invention is to provide a peripheral display device that reduces data and reduces the time spent on displaying a bird's-eye view image.

  A first aspect of the peripheral display device according to the present invention is a stereo camera that acquires peripheral image data in a self-propelled movable body, and a three-dimensional image that calculates first three-dimensional image data based on the image data. And the second three-dimensional image data obtained by converting the coordinate system of the first three-dimensional image data into a moving body coordinate system that is viewed from a virtual camera virtually arranged outside the moving body. A plurality of image measuring units, a plurality of image measuring units each including an image generating unit that generates overhead image data based on the second three-dimensional image data output from the coordinate converting unit, and the plurality of images And an image processing unit that integrates the bird's-eye view image data output from each measurement unit into one image to form a display image.

  In a second aspect of the peripheral display device according to the present invention, the image generation unit synthesizes the second three-dimensional image data and the data of the three-dimensional model of the moving object, and uses the moving object as an origin. The second three-dimensional image data is arranged in the vicinity thereof and used as the overhead image data.

  In a third aspect of the peripheral display device according to the present invention, the image processing unit uses the overhead image data output from the plurality of image measurement units and the data of the three-dimensional model of the moving body as moving body coordinates. An image integration unit that integrates the moving object data represented by the system and generates integrated data in which the moving object is the origin and the overhead image data is arranged around the origin.

  In a fourth aspect of the peripheral display device according to the present invention, each of the plurality of image measurement units includes a plurality of the stereo cameras, and the three-dimensional calculation unit and the coordinate conversion unit include the plurality of stereo cameras. A series of processing systems that are uniquely associated with each of the image processing units, and the image generation unit receives the second three-dimensional image data output from each of the processing systems in common and receives overhead image data. Is generated.

  A fifth aspect of the peripheral display device according to the present invention is a stereo camera that acquires peripheral image data in a self-propelled mobile body, and a three-dimensional image that calculates first three-dimensional image data based on the image data. A conversion calculation unit, a conversion unit that generates second 3D image data including virtual camera data in which a virtual camera virtually disposed outside the moving body is represented in a stereo camera coordinate system, and the second A plurality of image measurement units each including an image generation unit that generates overhead image data based on the three-dimensional image data and the first three-dimensional image data, and the plurality of image measurement units respectively output An image processing unit that integrates the overhead image data into a single image to generate a display image.

  In a sixth aspect of the peripheral display device according to the present invention, the second 3D image data includes moving body data in which a data of a 3D model of the moving body is represented in a stereo camera coordinate system, and the image generation The unit synthesizes the first three-dimensional image data, the virtual camera data, and the moving body data, uses the moving body as an origin, and arranges the first three-dimensional image data around the moving body, so that the overhead view is obtained. Let it be image data.

  In a seventh aspect of the peripheral display device according to the present invention, the image processing unit converts the overhead image data output from each of the plurality of image measurement units and the data of the three-dimensional model of the moving body into stereo camera coordinates. An image integration unit that integrates the moving object data represented by the system and generates integrated data in which the moving object is the origin and the overhead image data is arranged around the origin.

  In an eighth aspect of the peripheral display device according to the present invention, the overhead image data includes distance information from the virtual camera for each constituent pixel.

  In a ninth aspect of the peripheral display device according to the present invention, when the image processing unit integrates the overhead image data into one image, the distance information is compared with respect to overlapping pixel data, A process of adopting pixel data having a value closer to the distance from the virtual camera is performed.

  In a tenth aspect of the peripheral display device according to the present invention, when the image data acquired by the stereo camera is image data of a wall surface, the stereo image of the stereo camera at the time of capturing the image data is acquired. When the direction of the wall surface calculated based on the line-of-sight direction and the distribution of the image data is facing away from the virtual camera, the area corresponding to the image data on the wall surface is expressed as translucent. Process.

  In an eleventh aspect of the peripheral display device according to the present invention, when the image data acquired by the stereo camera is not within the visual field of the virtual camera, the stereo camera Processing that does not generate overhead image data is performed.

  According to the first and second aspects of the peripheral display device according to the present invention, since the overhead image data is transferred from the plurality of image measurement units to the image processing unit, the three-dimensional information having a large amount of data is transferred. In comparison, the transfer time can be reduced, and the time required for the integration process can also be reduced.

  According to the third aspect of the peripheral display device of the present invention, it is possible to further reduce the amount of data transferred from the image measurement unit to the image processing unit.

  According to the fourth aspect of the peripheral display device of the present invention, the image generation unit can be shared, and the system configuration can be simplified.

  According to the fifth and sixth aspects of the peripheral display device according to the present invention, since the overhead image data is transferred from the plurality of image measuring units to the image processing unit, the three-dimensional information having a large amount of data is transferred. In comparison, the transfer time can be reduced, and the time required for the integration process can also be reduced.

  According to the seventh aspect of the peripheral display device of the present invention, it is possible to further reduce the amount of data transferred from the image measurement unit to the image processing unit.

  According to the eighth aspect of the peripheral display device of the present invention, it is possible to use distance information to determine which pixel is not to be adopted depending on the perspective from the virtual camera during the integration process.

  According to the ninth aspect of the peripheral display device of the present invention, it is possible to select overlapping pixel data by using the distance information.

  According to the tenth aspect of the peripheral display device of the present invention, the moving body can be confirmed even when the moving body is hidden by the wall.

  According to the eleventh aspect of the peripheral display device of the present invention, the amount of data transferred from the image measurement unit to the image processing unit can be further reduced.

It is a figure which shows the example of arrangement | positioning of a stereo camera. It is a block diagram which shows the structure of Embodiment 1 of the peripheral display apparatus which concerns on this invention. It is a figure which shows typically the stereo camera coordinate system in each stereo camera, and the motor vehicle seat system of the own vehicle. It is a figure which shows typically the virtual camera which image | photographs the three-dimensional image data represented by a motor vehicle coordinate system. It is a figure which shows typically the positional relationship of a virtual camera with respect to the coordinate system in each stereo camera. It is a flowchart explaining the process which employ | adopts the pixel data which has a near value from the virtual camera using distance information. It is a block diagram which shows the structure of the modification of Embodiment 1 of the peripheral display apparatus which concerns on this invention. It is a block diagram which shows the structure of the modification of Embodiment 1 of the peripheral display apparatus which concerns on this invention. It is a block diagram which shows the structure of Embodiment 2 of the peripheral display apparatus which concerns on this invention. It is a block diagram which shows the structure of the modification of Embodiment 2 of the peripheral display apparatus which concerns on this invention. It is a figure which shows the example which has arrange | positioned the virtual camera above the own vehicle. It is a figure which shows the imaging | photography range of each stereo camera. It is a figure which shows the two-dimensional image image | photographed with each stereo camera. It is a figure which shows the bird's-eye view image produced | generated in the image generation part. It is a figure which shows the example which has arrange | positioned the virtual camera above the side surface of the own vehicle. It is a figure which shows the bird's-eye view image produced | generated in the image generation part. It is a figure which shows the example which has arrange | positioned the virtual camera above the side surface of the own vehicle. It is a figure which shows the two-dimensional image image | photographed with each stereo camera. It is a figure which shows the bird's-eye view image produced | generated in the image generation part. It is a figure which shows the bird's-eye view image produced | generated in the image generation part. It is a figure which shows the integrated bird's-eye view image. It is a figure which shows the integrated bird's-eye view image. It is a figure which shows the integrated bird's-eye view image. It is a figure which shows the integrated bird's-eye view image. It is a figure which shows the bird's-eye view image produced | generated in the image generation part. It is a figure which shows the other example of arrangement | positioning of a stereo camera.

<Embodiment 1>
<Apparatus configuration 1>
FIG. 1 is a diagram showing an arrangement example of stereo cameras in a vehicle VC equipped with a peripheral display device according to the present invention.

  In FIG. 1, a vehicle VC is equipped with four stereo cameras that acquire front, left, and right images. That is, a stereo camera SC1 that acquires a front image, a stereo camera SC2 that acquires a right image with respect to the front, a stereo camera SC3 that acquires a rear image, and a stereo camera SC4 that acquires a left image with respect to the front. I have.

  By using such a system, it is possible to acquire image data around the host vehicle, and change the viewpoint to an overhead view viewed from above the host vehicle.

  The actual arrangement position of each camera differs depending on the vehicle type. For example, the stereo camera SC1 is arranged at the position of the room mirror, and the stereo cameras SC2 and SC4 are arranged at the positions of the right and left side mirrors, respectively. Stereo camera SC3 is arranged at the position of the rear bumper. The above arrangement is an example, and the arrangement of the stereo camera is not limited to this.

  FIG. 2 is a block diagram showing the configuration of the peripheral display device 100 according to the present invention. As shown in FIG. 2, the peripheral display device 100 includes three-dimensional images output from the image measuring units 10, 20, 30, and 40 including the stereo cameras SC 1, SC 2, SC 3, and SC 4, and the image measuring units 10 to 40, respectively. An image processing unit 50 that integrates data to generate a display image, a storage unit 7 that stores data of a three-dimensional model of the host vehicle, and a storage unit that stores virtual camera information including information on the viewpoint position of the virtual camera 8 as a main configuration.

  The image measurement unit 10 includes a three-dimensional calculation unit 11 that calculates three-dimensional image data based on the two-dimensional image data obtained by the stereo camera SC1, and the three-dimensional image data calculated by the three-dimensional calculation unit 11. Expressed in the stereo camera coordinate system by applying a 3D transformation matrix that represents the position and orientation of the stereo camera in the car coordinate system (moving body coordinate system) to the (first 3D image data) A coordinate converter 12 that converts the three-dimensional image data into three-dimensional image data (second three-dimensional image data) expressed in the car coordinate system, and 3 expressed in the car coordinate system output from the coordinate converter 12 The three-dimensional image data and the data of the three-dimensional model of the host vehicle stored in the storage unit 7 are combined, and the host vehicle is set as the origin of the automobile coordinate system. Arrangement And an image generating unit 13 for generating the overhead image data. The coordinate conversion unit 12 is connected to a storage unit 14 that stores a three-dimensional conversion matrix Mat1.

  The image measurement unit 20 has the same configuration, and includes a stereo camera SC2, a three-dimensional calculation unit 21, a coordinate conversion unit 22, an image generation unit 23, and a storage unit 24.

  The image measurement unit 30 has the same configuration, and includes a stereo camera SC3, a three-dimensional calculation unit 31, a coordinate conversion unit 32, an image generation unit 33, and a storage unit 34.

  The image measurement unit 40 has the same configuration, and includes a stereo camera SC4, a three-dimensional calculation unit 41, a coordinate conversion unit 42, an image generation unit 43, and a storage unit 44.

  The image processing unit 50 integrates the overhead image data output from the image generation units 13, 23, 33, and 43 of the image measurement units 10 to 40 into one image to generate three-dimensional image data (integrated data). And an image display unit 6 that displays the three-dimensional image data generated by the image integration unit 5 as a display image.

<Basic operation>
Next, the basic operation of the peripheral display device 100 having the above-described configuration will be described.

  The stereo cameras SC1 to SC4 are both taken simultaneously from different viewpoints by the two cameras, and each of the three-dimensional calculation units 11, 21, 31 and 41 corresponds to the corresponding stereo camera based on the principle of triangulation. Each pixel of the two images obtained in step 1 is associated with each other (corresponding point search), and each corresponding point set is obtained from parameters such as parallax, focal length, and base line length. The three-dimensional image data is calculated by obtaining the three-dimensional position, and is output together with the texture image corresponding to the three-dimensional image data.

  Since the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41 is expressed in the stereo camera coordinate system, coordinates are used for conversion into the three-dimensional image data expressed in the car coordinate system. Coordinate conversion is performed in the conversion units 12, 22, 32 and 42.

  In FIG. 3, the stereo camera coordinate system in each of stereo camera SC1-SC4 and the motor vehicle coordinate system of the own vehicle VC are shown typically.

  In FIG. 3, the stereo camera coordinate system is represented by x-axis and y-axis, the automobile coordinate system is represented by X-axis and Y-axis, and the stereo camera coordinate system is different for each of the stereo cameras SC1 to SC4. .

  FIG. 4 schematically shows a virtual camera VCA that captures three-dimensional image data represented in the vehicle coordinate system. FIG. 5 shows a virtual camera for each coordinate system of the stereo cameras SC1 to SC4. The positional relationship of camera VCA is shown typically.

  That is, FIG. 5A shows the positional relationship of the virtual camera VCA with respect to the coordinate system of the stereo camera SC1, and the direction indicated by the arrow AR is the vehicle front direction. .

  Similarly, part (b) of FIG. 5 shows the positional relationship of virtual camera VCA with respect to the coordinate system of stereo camera SC2, and part (c) of FIG. The positional relationship of the virtual camera VCA with respect to the coordinate system of the camera SC3 is shown. FIG. 5D shows how the virtual camera VCA is relative to the coordinate system of the stereo camera SC4. It shows whether there is a proper positional relationship.

  This coordinate transformation is executed by applying predetermined three-dimensional transformation matrices Mat1, Mat2, Mat3, and Mat4 to the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41, respectively. The

  The three-dimensional image data expressed in the vehicle coordinate system output from the coordinate conversion units 12, 22, 32, and 42 is given to the image generation units 13, 23, 33, and 43, respectively.

  The image generation units 13, 23, 33, and 43 combine the three-dimensional image data expressed in the automobile coordinate system, the texture image, and the data of the three-dimensional model of the host vehicle given from the storage unit 7, and Is used as the origin of the vehicle coordinate system, and overhead image data in which three-dimensional image data expressed in the vehicle coordinate system is arranged around the origin is generated.

  At this time, distance information from the virtual camera may be provided for each pixel of the generated image. This distance information can be used to determine which pixel is not to be used depending on the distance from the virtual camera during the integration processing in the image integration unit 5.

  As described above, in the peripheral display device 100, the image measuring units 10, 20, 30, and 40 are provided with the image generating units 13, 23, 33, and 43, respectively, and the overhead image data is generated respectively, and the data is stored. Since transfer processing is performed to the image processing unit 50 and integration processing is performed, transfer time can be reduced and time required for integration processing can be reduced as compared with the case of transferring three-dimensional information having a large amount of data.

  In addition, since the image generation units 13, 23, 33, and 43 generate overhead image data only for images captured by the corresponding stereo cameras SC1 to SC4, for pixels outside the imaging range of each stereo camera in the overhead image. An invalid value is set assuming that there is no valid image data.

  Further, in the image generation units 13, 23, 33 and 43, when the measurement data measured (captured) by the stereo camera is, for example, data obtained by measuring a wall surface, the line-of-sight direction of the stereo camera at the time of measurement of the measurement data When the surface of the measurement data calculated based on the distribution of the measurement data is facing away from the virtual camera, when converting the area corresponding to the measurement data into an image, Displaying the wall surface on the side viewed from the virtual camera, or performing processing to make the image displayed as the wall surface translucent, for example, by thinning out the data reflected on the pixel value on the wall surface viewed from the virtual camera It is also possible.

  In addition, it is assumed that overhead image data is not generated for a stereo camera in which no measurement data exists within the visual field of the virtual camera.

  The overhead image data generated by the image generation units 13, 23, 33 and 43 is given to the image integration unit 5 and integrated into one image. At this time, the virtual camera information stored in the storage unit 8 is added to the virtual camera information. Integration is executed based on the viewpoint position information of the included virtual camera.

  The three-dimensional image data integrated by the image integration unit 5 is given to the image display unit 6 and displayed as an overhead image viewed from the virtual camera.

  Here, in the image integration unit 5, when a specific effective pixel in the finally obtained display image is duplicated in the newly obtained overhead image data before integration, the overlapping pixel has. The distance information from the virtual camera may be compared with the distance information from the virtual camera of the specific effective pixel in the display image, and the processing of adopting the data of the pixel having a value closer to the distance from the virtual camera may be performed. good. A flowchart of this process is shown in FIG.

  In FIG. 6, in step S1, a specific pixel Pr (x, y) in the finally obtained display image is acquired.

  Next, in step S2, the pixel Pn (x, y) corresponding to the pixel Pr is converted from the four bird's-eye view image data before integration (data obtained by coordinate-converting each image obtained by the four stereo cameras into a virtual camera image). ) Exists. Here, n represents the number of the four bird's-eye view image data, and is any number from 1 to 4.

  If it is determined in step S2 that the pixel Pn (x, y) corresponding to the pixel Pr exists, one of the pixels Pn (x, y) is selected, and the pixel Pn (x, y) is invalid in step S3. It is determined whether or not it is a pixel.

  On the other hand, if it is determined in step S2 that the pixel Pn (x, y) corresponding to the pixel Pr does not exist, the process ends.

  In step S3, if the pixel Pn (x, y) is an invalid pixel, the process returns to step S2, and whether or not there is a pixel Pn (x, y) corresponding to the pixel Pr in the other overhead image data. Determine.

  On the other hand, when it is determined in step S3 that the pixel Pn (x, y) is an effective pixel, in step S4, the distance information from the virtual camera of the pixel Pr and the distance information from the virtual camera of the pixel Pn If the distance from the virtual camera of the pixel Pn is determined to be closer than a predetermined threshold, the pixel value of the pixel Pr is replaced with the pixel value of the pixel Pn, and the distance information of the pixel Pr is Replace with the distance information of the pixel Pn.

  On the other hand, if the difference between the distance information of the pixel Pr and the distance information of the pixel Pn is smaller than a predetermined threshold value in step S4, the pixel value and the distance information are not replaced, and the processes in and after step S2 are repeated. .

  By performing such processing, it is possible to select overlapping pixel data.

<Apparatus configuration 2>
Instead of the peripheral display device 100 described above, the configuration of the peripheral display device 100A shown in FIG. 7 can be adopted. In addition, in the peripheral display device 100A shown in FIG. 7, the same components as those in the peripheral display device 100 shown in FIG.

  In the peripheral display device 100 shown in FIG. 2, three-dimensional image data expressed in an automobile coordinate system in each of the image generation units 13, 23, 33, and 43 of the image measurement units 10, 20, 30, and 40, The data of the three-dimensional model of the host vehicle stored in the storage unit 7 is synthesized with the host vehicle data expressed in the car coordinate system, and the host vehicle is set as the origin of the car coordinate system and is expressed in the car coordinate system around it. Although described as generating overhead image data in which three-dimensional image data is arranged, in the peripheral display device 100A shown in FIG. 7, synthesis with the data of the three-dimensional model of the host vehicle is provided in the image processing unit 50A. The image generation unit 9 performs the configuration.

  That is, as shown in FIG. 7, the data of the three-dimensional model of the host vehicle stored in the storage unit 7 is provided to the image generation unit 9 provided in the image processing unit 50 </ b> A. In this configuration, the data of the three-dimensional model of the host vehicle is used as overhead image data, and the image integration unit 5 provided in the image processing unit 50A performs integration with the overhead image data generated by the image generation units 13, 23, 33, and 43. It has become. The virtual camera information is also given to the image generation unit 9.

  By adopting such a configuration, it is possible to further reduce the amount of data transferred from the image measurement unit to the image processing unit.

<Apparatus configuration 3>
Instead of the peripheral display device 100 shown in FIG. 2, the configuration of the peripheral display device 100B shown in FIG. 8 can be adopted. In the peripheral display device 100B shown in FIG. 8, the same components as those in the peripheral display device 100 shown in FIG.

  In the peripheral display device 100 shown in FIG. 2, the image measuring units 10, 20, 30, and 40 have image generating units 13, 23, 33, and 43, respectively, and are two-dimensional obtained by stereo cameras S <b> 1 to S <b> 4. Each of the image data has been described as generating overhead image data. However, in the peripheral display device 100B shown in FIG. 8, the two-dimensional image data obtained by a plurality of stereo cameras are combined into a single overhead image data. It has a configuration to generate.

  That is, as shown in FIG. 8, peripheral display device 100B is output from image measurement unit 10A including stereo cameras SC1 and SC2, image measurement unit 20A including stereo cameras SC3 and SC4, and image measurement units 10A and 20A, respectively. Image processing unit 50A that integrates three-dimensional image data to generate a display image, storage unit 7 that stores data of a three-dimensional model of the host vehicle, and virtual camera information that includes information on the viewpoint position of the virtual camera. A storage unit 8 is provided as a main configuration.

  The image measurement unit 10A includes a three-dimensional calculation unit 11, a coordinate conversion unit 12, and a storage unit 14 that form a series of processing systems for processing two-dimensional image data from the stereo camera SC1, and two from the stereo camera SC2. An image for generating overhead image data based on the outputs of the three-dimensional calculation unit 21, the coordinate conversion unit 22 and the storage unit 24, and the coordinate conversion units 12 and 22 constituting a series of processing systems for processing the dimensional image data. And a generation unit 131.

  The same applies to the image measurement unit 20A. The three-dimensional calculation unit 21, the coordinate conversion unit 32, and the storage unit 34 that form a series of processing systems for processing the two-dimensional image data from the stereo camera SC3, and the stereo camera SC4. Based on the outputs of the three-dimensional calculation unit 41, the coordinate conversion unit 42 and the storage unit 44, and the coordinate conversion units 32 and 42 which constitute a series of processing systems for processing the two-dimensional image data from And an image generation unit 231 for generation.

  By adopting such a configuration, it is possible to share the image generation unit and simplify the system configuration.

  Further, in the peripheral display device 100B shown in FIG. 8, the composition with the data of the three-dimensional model of the host vehicle is performed in the image generation unit 9 provided in the image processing unit 50A.

  That is, as shown in FIG. 8, the data of the three-dimensional model of the host vehicle stored in the storage unit 7 is provided to the image generation unit 9 provided in the image processing unit 50 </ b> A. In 231 and 231, three-dimensional image data expressed in the car coordinate system is generated as overhead image data, and each is supplied to the image generation unit 9 provided in the image processing unit 50 </ b> A.

  Then, the composition of the overhead image data and the data of the three-dimensional model of the host vehicle is performed in the image generation unit 9.

  By adopting such a configuration, it is possible to further reduce the amount of data transferred from the image measurement unit to the image processing unit.

<Embodiment 2>
<Apparatus configuration 1>
FIG. 9 is a block diagram showing the configuration of the peripheral display device 200 according to the present invention. As shown in FIG. 2, the peripheral display device 100 includes three-dimensional images output from the image measurement units 101, 201, 301, and 401 and the image measurement units 101 to 401 including stereo cameras SC 1, SC 2, SC 3, and SC 4, respectively. An image processing unit 50 that integrates data to generate a display image, a storage unit 7 that stores data of a three-dimensional model of the host vehicle, and a storage unit that stores virtual camera information including information on the viewpoint position of the virtual camera 8 as a main configuration.

  The image measuring unit 101 is expressed by a three-dimensional calculation unit 11 that calculates three-dimensional image data based on two-dimensional image data obtained by the stereo camera SC1, and an automobile coordinate system (moving body coordinate system). Receives virtual camera information (stored in the storage unit 8) and data of a three-dimensional model of the host vehicle (stored in the storage unit 7), and represents a three-dimensional transformation matrix (representing the position and orientation of each stereo camera in the automobile coordinate system) By applying Mat11), virtual camera data expressing the position and orientation of the virtual camera in the stereo camera coordinate system (FIG. 5) and own vehicle data expressing the three-dimensional model of the own vehicle in the stereo camera coordinate system are generated. Parameter conversion unit 15 to perform, virtual camera data and host vehicle data expressed in a stereo camera coordinate system output from the parameter conversion unit 15, and a three-dimensional calculation unit 1 It synthesizes the three-dimensional image data represented in the stereo camera coordinate system output from, and an image generating unit 13 for generating the overhead image data captured from a virtual camera is represented by the stereo camera coordinate system.

  The parameter conversion unit 15 is connected to a storage unit 14 that stores a three-dimensional conversion matrix Mat11.

  The image measurement unit 201 has the same configuration, and includes a stereo camera SC2, a three-dimensional calculation unit 21, a parameter conversion unit 25, an image generation unit 23, and a storage unit 24 that stores a three-dimensional conversion matrix Mat21.

  The image measurement unit 301 has the same configuration, and includes a stereo camera SC3, a three-dimensional calculation unit 31, a parameter conversion unit 35, an image generation unit 33, and a storage unit 34 that stores a three-dimensional conversion matrix Mat31.

  The image measurement unit 401 has the same configuration, and includes a stereo camera SC4, a three-dimensional calculation unit 41, a parameter conversion unit 45, an image generation unit 43, and a storage unit 44 that stores a three-dimensional conversion matrix Mat41.

  The image processing unit 50 integrates the overhead image data output from the image generation units 13, 23, 33, and 43 of the image measurement units 101 to 401 into one image and generates three-dimensional image data (integrated data). And an image display unit 6 that displays the three-dimensional image data generated by the image integration unit 5 as a display image.

  By adopting such a configuration, each of the image generation units 13, 23, 33, and 43 generates overhead image data and transfers the data to the image processing unit 50 for integration processing. Compared to the case of transferring three-dimensional information, the transfer time can be reduced, and the time required for the integration process can also be reduced.

<Basic operation>
Next, the basic operation of the peripheral display device 200 having the above-described configuration will be described. Basically, it is the same as the basic operation of the peripheral display device 100 shown in FIG. 2, but is output from the three-dimensional calculation units 11, 21, 31 and 41 in the image generation units 13, 23, 33 and 43. A bird's-eye view taken from a virtual camera expressed in the stereo camera coordinate system by synthesizing the three-dimensional image of the host vehicle coordinate-converted into the stereo camera coordinate system by the coordinate converters 12, 22, 32, and 42 with the three-dimensional image data The difference is that image data is generated.

  The positional relationship of the virtual camera VCA when the stereo camera coordinate system of each of the stereo cameras SC1 to SC4 is centered is shown in FIG. 5. In each of the image measuring units 101 to 401, the stereo camera coordinate system is the same. Since the obtained three-dimensional image data and the three-dimensional model of the host vehicle are synthesized, the image measuring unit 101 is a virtual that faces the front of the host vehicle VC, for example, as shown in part (a) of FIG. Overhead image data photographed by the camera VCA is obtained. Similarly, in the image measuring unit 102, for example, as shown in part (b) of FIG. 5, overhead image data captured by the virtual camera VCA oriented in a direction perpendicular to the front direction of the host vehicle VC is obtained. In the image measurement unit 103, for example, as shown in part (c) of FIG. 5, overhead image data photographed by the virtual camera VCA facing in the direction opposite to the front direction of the host vehicle VC is obtained, and image measurement is performed. In the unit 104, for example, as shown in FIG. 5D, overhead image data captured by the virtual camera VCA oriented in a direction perpendicular to the front direction of the host vehicle VC is obtained, and these are processed by image processing. By integrating in the unit 50, an overhead image viewed from a virtual camera similar to the peripheral display device 100 can be obtained.

<Apparatus configuration 2>
Instead of the peripheral display device 200 described above, the configuration of the peripheral display device 200A shown in FIG. 10 can be adopted. Note that in the peripheral display device 200A shown in FIG. 10, the same components as those of the peripheral display device 100 shown in FIG.

  In the peripheral display device 200 shown in FIG. 9, the three-dimensional model of the host vehicle expressed in the stereo camera coordinate system in the image generation units 13, 23, 33, and 43 of the image measurement units 101, 201, 301, and 401, respectively. The own vehicle data expressed in the stereo camera coordinate system and the three-dimensional image data expressed in the stereo camera coordinate system output from the three-dimensional calculation unit 11 are synthesized and expressed in the stereo camera coordinate system. In the peripheral display device 200A shown in FIG. 10, combining with the data of the three-dimensional model of the host vehicle is an image provided in the image processing unit 50A. The integration unit 5 performs the configuration.

  That is, as shown in FIG. 10, the data of the three-dimensional model of the host vehicle stored in the storage unit 7 is given to the image generation unit 9 provided in the image processing unit 50A and expressed in the stereo camera coordinate system. In the image generation units 13, 23, 33, and 43, the three-dimensional image data expressed in the stereo camera coordinate system is generated as overhead image data, and the image generation unit 50A is provided with the image processing unit 50A. Each of the bird's-eye view image data and the own vehicle data is integrated in the image integration unit 5. The virtual camera information is also provided to the image generation unit 9.

  By adopting such a configuration, it is possible to further reduce the amount of data transferred from the image measurement unit to the image processing unit.

<Overhead image by virtual camera>
Variations of the overhead image obtained by the peripheral display devices of the first and second embodiments described above will be described below.

<Example 1 of an overhead image>
FIG. 11 shows a situation where the virtual camera VCA is arranged above the host vehicle VC and the wall WL is close to the left side with respect to the front of the host vehicle VC. In such a situation, each imaging range of the stereo cameras SC1 to SC4 is shown in FIG.

  In FIG. 12, the shooting range of stereo camera SC1 is shown as region R1, the shooting range of stereo camera SC2 is shown as region R2, the shooting range of stereo camera SC3 is shown as region R3, and the shooting range of stereo camera SC4 is shown as region R4. Show.

  And the two-dimensional image image | photographed with stereo camera SC1-SC4 can be represented as an image shown by FIG.

  That is, as shown in part (a) of FIG. 13, the front end of the wall WL is reflected in the image taken by the stereo camera SC1, and as shown in part (b) of FIG. The wall WL is not shown in the image taken by the camera SC2. Further, as shown in part (c) of FIG. 13, the rear end of the wall WL is reflected in the image photographed by the stereo camera SC3. As shown in part (d) of FIG. A continuous wall WL is shown in the image taken by the camera SC4.

  If the peripheral display device 100 shown in FIG. 2 is taken as an example, three-dimensional image data is calculated based on the two-dimensional image data of such an image, and the three-dimensional image data is converted into a stereo camera in the automobile coordinate system. By applying a three-dimensional transformation matrix that expresses the position and orientation, the three-dimensional image data expressed in the stereo camera coordinate system is converted into the three-dimensional image data expressed in the car coordinate system. By combining with the data of the three-dimensional model of the host vehicle, overhead image data is generated in which the host vehicle is the origin of the automobile coordinate system and the three-dimensional image data expressed in the automobile coordinate system is arranged around the origin.

  FIG. 14 shows overhead images generated by the image generation units 13, 23, 33, and 43 of the peripheral display device 100.

  As described above, the image generation units 13, 23, 33, and 43 generate overhead image data only for images captured by the corresponding stereo cameras SC <b> 1 to SC <b> 4. An invalid value is set for the other pixels. Accordingly, in the overhead image data for the image captured by the stereo camera SC1, the pixels other than the region R1 indicating the imaging range of the stereo camera SC1 are invalid pixels, as shown in part (a) of FIG. Shown in black.

  Similarly, in the bird's-eye view image data for the image captured by the stereo camera SC2, pixels other than the region R2 indicating the imaging range of the stereo camera SC2 are invalid pixels as shown in part (b) of FIG. Shown in black.

  In addition, in the overhead image data for the image captured by the stereo camera SC3, pixels other than the region R3 indicating the imaging range of the stereo camera SC3 are invalid pixels, as shown in part (c) of FIG. Shown in black.

  In addition, in the overhead image data for the image captured by the stereo camera SC4, pixels other than the region R4 indicating the imaging range of the stereo camera SC4 are invalid pixels, as shown in the part (d) of FIG. Shown in black.

  By transferring such overhead image data to the image processing unit 50, the amount of data to be transferred can be reduced.

<Example 2 of an overhead image>
FIG. 15 shows a situation in which the virtual camera VCA is disposed outside the right side with respect to the front of the host vehicle VC, and the wall WL is close to the left side with respect to the front of the host vehicle VC. ing. In such a situation, the images taken by the stereo cameras SC1 to SC4 (FIG. 1) are the same as those shown in FIG. 13, but for example, the image generation units 13, 23, 33 of the peripheral display device 100 and The overhead view image generated in 43 can be represented as an image shown in FIG.

  That is, in the overhead image data for the image captured by the stereo camera SC1, the pixels other than the region R1 indicating the imaging range of the stereo camera SC1 are invalid pixels as shown in part (a) of FIG. Shown in black. Since the region R1 includes the wall surface of the wall WL, the wall surface is shown as an effective pixel.

  Similarly, in the bird's-eye view image data for the image captured by the stereo camera SC2, pixels other than the region R2 indicating the shooting range of the stereo camera SC2 are invalid pixels as shown in part (b) of FIG. Shown in black. Note that the wall R is not included in the region R2.

  In the bird's-eye view image data for the image captured by the stereo camera SC3, as shown in part (c) of FIG. 16, pixels other than the region R3 indicating the imaging range of the stereo camera SC3 are invalid pixels and are filled in black. Is shown. Note that since the region R3 includes the wall surface of the wall WL, the wall surface is shown as an effective pixel.

  In the bird's-eye view image data for the image photographed by the stereo camera SC4, as shown in the part (d) of FIG. 16, the pixels other than the region R4 indicating the photographing range of the stereo camera SC4 are invalid pixels and are blacked out. Is shown. Since the region R4 includes the wall surface of the wall WL, the wall surface is shown as an effective pixel.

<Example 3 of an overhead image>
FIG. 17 shows a situation where the virtual camera VCA is arranged at the upper rear side of the host vehicle VC and the wall WL is close to the left and right of the front of the host vehicle VC. In such a situation, images taken by stereo cameras SC1 to SC4 (FIG. 1) are shown in FIG.

  That is, as shown in part (a) of FIG. 18, the front end portions of the left and right walls WL are reflected in the image taken by the stereo camera SC <b> 1, and are shown in part (b) of FIG. 18. As described above, the continuous right wall WL is shown in the image taken by the stereo camera SC2. Further, as shown in FIG. 18 (c), the image captured by the stereo camera SC3 includes the rear end portions of the left and right walls WL, as shown in FIG. 18 (d). In the image taken by the stereo camera SC4, the continuous left wall WL is shown.

  Here, as described above, in the image generation units 13, 23, 33, and 43, when the measurement data measured (captured) by the stereo camera is, for example, data obtained by measuring a wall surface, the measurement data is measured. When the direction of the surface on which the measurement data exists calculated based on the viewing direction of the stereo camera at the time and the distribution of the measurement data is facing away from the virtual camera, that is, the virtual camera VCA is shown in FIG. In the case of the position shown in FIG. 18, when converting the area corresponding to the measurement data into an image, it is possible to perform processing for displaying the wall surface on the side viewed from the virtual camera. Taking the peripheral display device 100 shown in FIG. 2 as an example, the image data can be represented as an overhead image shown in FIG.

  That is, in the bird's-eye view image data for the image captured by the stereo camera SC1, the pixels other than the region R1 indicating the imaging range of the stereo camera SC1 are invalid pixels as shown in part (a) of FIG. Shown in black. Note that since the region R1 includes the wall surface of the left wall WL, the wall surface is shown as an effective pixel.

  Similarly, in the bird's-eye view image data for the image captured by the stereo camera SC2, pixels other than the region R2 indicating the shooting range of the stereo camera SC2 are invalid pixels as shown in part (b) of FIG. Shown in black. Since the region R2 includes the wall surface of the continuous right wall WL, the wall surface is an effective pixel, and the wall surface is displayed on the side viewed from the virtual camera.

  As shown in FIG. 19 (c), the overhead image data for the image captured by the stereo camera SC3 is a pixel that is not a pixel other than the region R3 indicating the imaging range of the stereo camera SC3, and is blacked out. Is shown. In addition, since the wall surface of the right wall WL is included in the region R3, the wall surface is shown as an effective pixel.

  In the bird's-eye view image data for the image captured by the stereo camera SC4, pixels other than the region R4 indicating the shooting range of the stereo camera SC4 are invalid pixels as shown in FIG. Is shown. In addition, since the wall surface of the right wall WL is included in the region R4, the wall surface is shown as an effective pixel.

<Example 4 of an overhead image>
Further, as described above, in the image generation units 13, 23, 33, and 43, when the measurement data measured (captured) by the stereo camera is, for example, data obtained by measuring a wall surface, FIG. 16 shows a case where the orientation of the surface on which the measurement data exists calculated based on the viewing direction of the stereo camera and the distribution of the measurement data is facing away from the virtual camera, that is, the virtual camera VCA is shown in FIG. When the area corresponding to the measurement data is converted into an image, the wall surface viewed from the virtual camera is displayed as a wall surface by thinning out the data to be reflected in the pixel value. 18 can be translucent, the image data shown in FIG. 18 can be obtained as an overhead image shown in FIG. 20 if the peripheral display device 100 shown in FIG. 2 is taken as an example. It can be expressed Te.

  That is, in the overhead image data for the image captured by the stereo camera SC1, the pixels other than the region R1 indicating the imaging range of the stereo camera SC1 are invalid pixels as shown in part (a) of FIG. Shown in black. Note that since the region R1 includes the wall surface of the left wall WL, the wall surface is shown as an effective pixel.

  Similarly, in the bird's-eye view image data for the image captured by the stereo camera SC2, pixels other than the region R2 indicating the shooting range of the stereo camera SC2 are invalid pixels as shown in part (b) of FIG. Shown in black. In addition, since the wall surface of the right side wall WL is included in the region R2, the wall surface is regarded as an effective pixel, and the wall surface is displayed on the side viewed from the virtual camera. The image displayed as the wall surface is displayed as being translucent.

  In the bird's-eye view image data for the image captured by the stereo camera SC3, pixels other than the region R3 indicating the imaging range of the stereo camera SC3 are invalid pixels as shown in FIG. Is shown. In addition, since the wall surface of the right wall WL is included in the region R3, the wall surface is shown as an effective pixel.

  In the bird's-eye view image data for the image captured by the stereo camera SC4, pixels other than the region R4 indicating the imaging range of the stereo camera SC4 are invalid pixels as shown in FIG. Is shown. In addition, since the wall surface of the right wall WL is included in the region R4, the wall surface is shown as an effective pixel.

  The overhead images obtained by integrating the overhead images of Examples 1 to 4 described above are shown in FIGS. That is, FIG. 21 is a view obtained by integrating the bird's-eye view images shown in the parts (a) to (d) of FIG. 14, and FIG. 22 is a bird's-eye view shown in the parts (a) to (d) of FIG. FIG. 23 is a diagram in which the overhead images shown in (a) to (d) of FIG. 19 are integrated, and FIG. 24 is a diagram in which (a) to (d) of FIG. It is the figure which integrated the bird's-eye view image shown to the part.

<Example 5 of an overhead image>
As shown in FIG. 11, in the situation where the virtual camera VCA is arranged above the host vehicle VC and the wall WL is close to the left side with respect to the front of the host vehicle VC, the stereo cameras SC1 to SC1. The image taken in SC4 (FIG. 1) is the same as that shown in FIG. 13. For example, the overhead image generated in the image generation units 131 and 231 of the peripheral display device 100B shown in FIG. 25 can be represented as an image.

  In other words, the overhead image data for the images taken by the stereo cameras SC1 and SC2 is invalid for the pixels other than the regions R1 and R2 indicating the shooting ranges of the stereo cameras SC1 and SC2, as shown in part (a) of FIG. It is a pixel and is shown in black.

  Similarly, as shown in FIG. 25 (b), the overhead image data for the images captured by the stereo cameras SC3 and SC4 is the pixels other than the regions R3 and R4 indicating the imaging ranges of the stereo cameras SC3 and SC4. It is an invalid pixel and is shown in black.

<Modification>
FIG. 1 shows an example in which one stereo camera is arranged in four directions of the vehicle VC. However, as shown in FIG. 26, two stereo cameras may be arranged in four directions of the vehicle VC. good.

  That is, as shown in FIG. 26, stereo cameras SC1 and SC2 that acquire the left and right images of the vehicle VC are arranged, and the stereo camera SC3 that acquires images on the right front side and rear side with respect to the front side. SC4 is arranged, stereo cameras SC5 and SC6 for obtaining the left and right images at the rear are arranged, and stereo cameras SC7 and SC8 for obtaining the images at the rear left side and the front side of the left side with respect to the front are arranged.

  By using such a system, it is possible to acquire peripheral image data with fewer blind spots.

5 Image integration unit 11, 21, 31, 41 Three-dimensional calculation unit 12, 22, 32, 42 Coordinate conversion unit 13, 23, 33, 43 Image generation unit 15, 25, 35, 45 Conversion unit 10-40 Image measurement Part 50, 50A Image processing part SC1 to SC4 Stereo camera

Claims (11)

A stereo camera that acquires image data of the surroundings of a self-propelled moving object;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is viewed from a virtual camera that is virtually arranged outside the moving body to obtain second 3D image data. When,
A plurality of image measurement units each including an image generation unit that generates overhead image data based on the second three-dimensional image data output from the coordinate conversion unit;
A peripheral display device comprising: an image processing unit that integrates the bird's-eye image data output from the plurality of image measurement units into one image to form a display image. The image generation unit
The second three-dimensional image data and the data of the three-dimensional model of the moving object are combined, the moving object is set as the origin, and the second three-dimensional image data is arranged around the origin, and the overhead image data and The peripheral display device according to claim 1. The image processing unit
The bird's-eye view image data output from each of the plurality of image measuring units and the moving object data representing the data of the three-dimensional model of the moving object in a moving object coordinate system, the moving object as an origin, The peripheral display device according to claim 1, further comprising an image integration unit that generates integrated data in which the overhead image data is arranged in the periphery. Each of the plurality of image measuring units is
A plurality of stereo cameras;
The three-dimensional calculation unit and the coordinate conversion unit are uniquely associated with each of the plurality of stereo cameras to form a series of processing systems,
The peripheral display device according to claim 1, wherein the image generation unit generates the overhead image data by commonly receiving the second three-dimensional image data output from each of the processing systems. A stereo camera that acquires image data of the surroundings of a self-propelled moving object;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A conversion unit that generates second three-dimensional image data including virtual camera data in which a virtual camera virtually disposed outside the moving body is represented in a stereo camera coordinate system;
A plurality of image measurement units each including an image generation unit that generates overhead image data based on the second three-dimensional image data and the first three-dimensional image data;
A peripheral display device comprising: an image processing unit that integrates the bird's-eye image data output from the plurality of image measurement units into one image to form a display image. The second three-dimensional image data is
Including mobile object data representing the data of the three-dimensional model of the mobile object in a stereo camera coordinate system;
The image generation unit
The first three-dimensional image data, the virtual camera data, and the moving body data are synthesized, the moving body is set as an origin, and the first three-dimensional image data is arranged around the moving body data, and the overhead image data The peripheral display device according to claim 5. The image processing unit
The overhead image data output from each of the plurality of image measuring units and the moving body data representing the data of the three-dimensional model of the moving body in a stereo camera coordinate system are integrated, and the moving body is used as an origin, The peripheral display device according to claim 5, further comprising an image integration unit that generates integrated data in which the overhead image data is arranged around the periphery.   The peripheral display device according to claim 1, wherein the overhead image data includes distance information from the virtual camera for each constituent pixel. The image processing unit
When integrating the bird's-eye view image data into a single image, the distance information is compared with respect to overlapping pixel data, and pixel data having a closer distance from the virtual camera is used. The peripheral display device according to claim 8. The image generation unit
When the image data acquired by the stereo camera is image data of a wall surface, the direction of the wall surface calculated based on the line-of-sight direction of the stereo camera at the time of capturing the image data and the distribution of the image data 6. The peripheral display device according to claim 1 or 5, wherein, when facing away from the virtual camera, processing for expressing a region corresponding to the image data on the wall surface as translucent is performed. The image generation unit
6. The peripheral according to claim 1 or 5, wherein when the image data acquired by the stereo camera does not exist within the field of view of the virtual camera, the stereo camera performs processing that does not generate the overhead image data. Display device.

Images