JP5299101B2 - Peripheral display device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a circumference display device capable of reducing calculation costs when bird's-eye view images around a self-propelled mobile object such as a vehicle are acquired. <P>SOLUTION: The circumference display device is provided with: a three-dimensionalization calculation part 11 which calculates three-dimensional image data; a coordinate conversion part 12 which converts three-dimensional image data expressed in a stereo camera coordinate system into three-dimensional image data expressed in a vehicle coordinate system by adapting a three-dimensional conversion matrix expressing a position of a stereo camera in the vehicle coordinate system to the three-dimensional image data calculated by the three-dimensionalization calculation part 11; an integration part 5 which integrates the three-dimensional image data expressed in the vehicle coordinate system with data of a three-dimensional model of a self-vehicle stored in a storage part 7 to generate three-dimensional image data; and an image generation part 6 which generates the three-dimensional image data generated by the integration part 5 into bird's-eye view image data observed by a camera arranged at a virtual position. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a peripheral display device, and more particularly, to a peripheral display device that displays a bird's-eye view of a surrounding state of a mobile object capable of self-running.

  Two-dimensional images around the vehicle are individually picked up by a plurality of imaging means, and a plurality of overhead images are created with the road surface as a projection plane by converting each image to a viewpoint, and the plurality of overhead images are arranged around the vehicle. Various techniques for obtaining a bird's-eye view of the vicinity of the host vehicle by combining them are disclosed.

  For example, Patent Document 1 relates to a virtual reality system that three-dimensionally conveys a sense of the surrounding environment acquired in real time to a user, performs three-dimensional measurement from a set of images that have changed in time series, and has a box-shaped three-dimensional periphery. A technique is disclosed in which texture mapping is performed on a wall surface of a model and three-dimensional display is performed.

JP 2004-287808 A

  In order to display an image around the vehicle three-dimensionally, a three-dimensional calculation for acquiring three-dimensional information including distance information based on a two-dimensional image acquired by a stereo camera is required. Although it is necessary to perform drawing based on information, these processes take time, and there is a limit to hardware resources to perform with a system mounted on a vehicle, and the calculation cost cannot be ignored. 1 does not disclose a device for reducing the calculation cost.

  The present invention has been made to solve the above-described problems, and provides a peripheral display device that can reduce the calculation cost when acquiring an overhead view image of a mobile object such as a car that can be self-propelled. Objective.

A first aspect of the peripheral display device according to the present invention calculates a first three-dimensional image data based on a plurality of stereo cameras that acquire peripheral image data in a mobile object capable of self-propelling and the image data. A three-dimensional calculation unit and a coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second three-dimensional image data And an integration unit that generates integrated data obtained by integrating the second 3D image data and the data of the 3D model of the moving body, and an image generation unit that generates overhead image data from the integrated data. The three-dimensional calculation unit associates feature points of two images with each other with respect to the image data obtained by each of the plurality of stereo cameras, and determines a measurement point on the image from the parallax obtained as a result. Three-dimensional position Calculation target restriction information for reducing the processing target data in at least one of the three-dimensionalization calculation unit, the coordinate conversion unit, and the integration unit is acquired by calculating the first three-dimensional image data. Yes, and the calculated target restriction information is given to the three-dimensional calculation unit is information to limit the calculation target of the three-dimensional position of the measurement point.

In a second aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a predetermined measurement area defined on a two-dimensional image obtained by each of the plurality of stereo cameras, The data of the measurement points included in the predetermined measurement area is not subject to calculation of the three-dimensional position.

In a third aspect of the peripheral display device according to the present invention, the predetermined measurement area corresponds to a measurement area that overlaps each other in each of the plurality of stereo cameras.

In a fourth aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a threshold value set for the parallax amount at the measurement point, and the parallax amount does not exceed the threshold value. The point data is not subject to calculation of the three-dimensional position.

A fifth aspect of the peripheral display device according to the present invention calculates a first three-dimensional image data based on a plurality of stereo cameras that acquire peripheral image data in a mobile object capable of self-propelling and the image data. A three-dimensional calculation unit and a coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second three-dimensional image data And an integration unit that generates integrated data obtained by integrating the second 3D image data and the data of the 3D model of the moving body, and an image generation unit that generates overhead image data from the integrated data. The three-dimensional calculation unit associates feature points of two images with each other with respect to the image data obtained by each of the plurality of stereo cameras, and determines a measurement point on the image from the parallax obtained as a result. Three-dimensional position Calculation target restriction information for reducing the processing target data in at least one of the three-dimensionalization calculation unit, the coordinate conversion unit, and the integration unit is acquired by calculating the first three-dimensional image data. The calculation target restriction information is given to the three-dimensional calculation unit and is information on a predetermined measurement region defined on a three-dimensional image obtained by each of the plurality of stereo cameras, the data of the measurement points included in the predetermined measurement area, have such a calculation target of the three-dimensional position.

A sixth aspect of the peripheral display device according to the present invention calculates a first three-dimensional image data based on a plurality of stereo cameras that acquire peripheral image data in a self-propelled movable body and the image data. A three-dimensional calculation unit and a coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second three-dimensional image data And an integration unit that generates integrated data obtained by integrating the second 3D image data and the data of the 3D model of the moving body, and an image generation unit that generates overhead image data from the integrated data. The three-dimensional calculation unit associates feature points of two images with each other with respect to the image data obtained by each of the plurality of stereo cameras, and determines a measurement point on the image from the parallax obtained as a result. Three-dimensional position Calculation target restriction information for reducing the processing target data in at least one of the three-dimensionalization calculation unit, the coordinate conversion unit, and the integration unit is acquired by calculating the first three-dimensional image data. The calculation target restriction information is provided to the coordinate conversion unit, and is information for reducing the target of coordinate conversion in the first three-dimensional image data.

In a seventh aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a predetermined measurement area defined on a three-dimensional image obtained by each of the plurality of stereo cameras, The data of the measurement points included in the predetermined measurement area is not subject to the coordinate conversion.

In an eighth aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a predetermined measurement region defined on a three-dimensional image represented by the moving body coordinate system, The measurement point data included in the measurement area is not subject to the coordinate conversion.

In a ninth aspect of the peripheral display device according to the present invention, the predetermined measurement region sets a boundary at a position ahead of each of the plurality of stereo cameras by a predetermined distance, and the boundary and the angle of view of each stereo camera are set. It corresponds to an area other than the area defined by.

A tenth aspect of the peripheral display device according to the present invention calculates a first three-dimensional image data based on a plurality of stereo cameras that acquire peripheral image data in a mobile object capable of traveling, and the image data. A three-dimensional calculation unit and a coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second three-dimensional image data And an integration unit that generates integrated data obtained by integrating the second 3D image data and the data of the 3D model of the moving body, and an image generation unit that generates overhead image data from the integrated data. The three-dimensional calculation unit associates feature points of two images with each other with respect to the image data obtained by each of the plurality of stereo cameras, and determines a measurement point on the image from the parallax obtained as a result. Three-dimensional Calculation target restriction information for acquiring the first three-dimensional image data by calculating a position and reducing processing target data in at least one of the three-dimensional calculation unit, the coordinate conversion unit, and the integration unit The calculation target restriction information is provided to the integration unit, and the information that limits the region to be generated as the overhead image data by the image generation unit in the integrated data.

In an eleventh aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a predetermined measurement area defined on a three-dimensional image represented by the moving body coordinate system, The measurement point data included in the measurement area is discarded.

In a twelfth aspect of the peripheral display device according to the present invention, is the image taken by each stereo camera based on the image data obtained by the plurality of stereo cameras being an image taken in a normal state? The image forming apparatus further includes an abnormality detection unit that determines whether the image is captured in an abnormal state, and discards the image data for an image determined to be abnormal by the abnormality detection unit.

In a thirteenth aspect of the peripheral display device according to the present invention, the abnormality detection unit counts the number of pixels having a pixel value equal to or larger than a predetermined value for the image data acquired by the plurality of stereo cameras, and determines in advance. When the number of pixels equal to or greater than the threshold value is counted, the image data is determined to be abnormal.

According to the first aspect of the peripheral display device of the present invention, since the calculation target restriction information for reducing the processing target data in at least one of the three-dimensional calculation unit, the coordinate conversion unit, and the integration unit is included, the movement The calculation cost when acquiring a bird's-eye view image around the body can be reduced. Further, since the calculation target of the three-dimensional position among the measurement points is limited in the three-dimensional calculation unit based on the calculation target restriction information, it is possible to reduce the target data of the three-dimensional calculation and reduce the calculation cost. it can.

According to the second aspect of the peripheral display device according to the present invention, the calculation target restriction information is information on a predetermined measurement region defined on a two-dimensional image obtained by each of a plurality of stereo cameras. For example, it is possible to uniquely set the limited area based on the angle of view of each stereo camera, and the limited area can be set relatively easily.

According to the third aspect of the peripheral display device according to the present invention, since the measurement areas that overlap each other in each of the plurality of stereo cameras are set as the restriction areas, more practical settings are possible.

According to the fourth aspect of the peripheral display device according to the present invention, the data of the measurement point whose parallax amount does not exceed the threshold value is not subject to calculation of the three-dimensional position, and therefore needs to be grasped when the moving body is in operation. It is possible to reduce the calculation cost by reducing the data to be calculated while leaving the data for the measurement points.

According to the fifth aspect of the peripheral display device of the present invention, the measurement point data included in the predetermined measurement region is not the processing target data after the calculation of the three-dimensional position. Can be reduced.

According to the sixth aspect of the peripheral display device of the present invention, since the coordinate conversion target is reduced in the coordinate conversion unit based on the calculation target restriction information, the coordinate conversion target data is reduced, and the calculation cost is reduced. can do.

According to the seventh aspect of the peripheral display device of the present invention, the measurement point data included in the predetermined measurement region is not subject to coordinate conversion, and therefore the calculation cost associated with coordinate conversion can be reduced.

According to the eighth aspect of the peripheral display device of the present invention, the data of the measurement point included in the predetermined measurement area defined on the three-dimensional image represented by the moving object coordinate system is subjected to coordinate conversion. Since it is not a target, the calculation cost associated with coordinate transformation can be reduced.

According to the ninth aspect of the peripheral display device of the present invention, it is possible to uniquely set a predetermined measurement region.

According to the tenth aspect of the peripheral display device of the present invention, based on the calculation target restriction information, the integration unit limits a region that should be generated as overhead image data by the image generation unit in the integration data. Calculation costs associated with three-dimensional drawing can be reduced.

According to the eleventh aspect of the peripheral display device according to the present invention, the data of the measurement points included in the predetermined measurement region defined on the three-dimensional image represented by the moving body coordinate system is discarded. Calculation costs associated with three-dimensional drawing can be reduced.

According to the twelfth aspect of the peripheral display device of the present invention, the image data is discarded for the image determined to be abnormal by the abnormality detection unit, and therefore processing such as three-dimensional calculation is performed for the abnormal image data. Therefore, the calculation cost can be reduced.

According to the thirteenth aspect of the peripheral display device according to the present invention, the abnormality detection unit counts the number of pixels having a pixel value equal to or greater than a predetermined value for image data acquired by a plurality of stereo cameras, and determines the predetermined value. When the number of pixels equal to or greater than the threshold value is counted, the image data is determined to be abnormal, so that it is possible to cope with an abnormal state in which the number of saturated pixels is extremely large.

It is a figure which shows the example of arrangement | positioning of a stereo camera. It is a block diagram which shows the structure of embodiment of the peripheral display apparatus which concerns on this invention. It is a figure which shows typically the duplication of the measurement area | region of a stereo camera. It is a figure which shows the example of a calculation object restriction | limiting area | region typically. It is a figure which shows the example of a calculation object restriction | limiting area | region typically. It is a figure which shows the example of a calculation object restriction | limiting area | region typically. It is a figure which shows the example of a calculation object restriction | limiting area | region typically. It is a figure which shows typically the example of the calculation object restriction | limiting area | region in an integration part. It is a flowchart explaining operation | movement of embodiment of the peripheral display apparatus which concerns on this invention. It is a figure which shows the calculation object restriction | limiting area | region corresponding to the three-dimensional image data of a stereo camera coordinate system. It is a figure which shows the characteristic of parallax typically. It is a block diagram which shows the structure of the modification of embodiment of the periphery display apparatus which concerns on this invention. It is a flowchart explaining the operation | movement of the modification of embodiment of the periphery display apparatus which concerns on this invention. It is a figure which shows the other example of arrangement | positioning of a stereo camera.

<Embodiment>
<Device configuration>
FIG. 1 is a diagram showing an arrangement example of stereo cameras in a vehicle VC equipped with a peripheral display device according to the present invention.

  In FIG. 1, a vehicle VC is equipped with four stereo cameras that acquire front, left, and right images. That is, a stereo camera SC1 that acquires a front image, a stereo camera SC2 that acquires a right image with respect to the front, a stereo camera SC3 that acquires a rear image, and a stereo camera SC4 that acquires a left image with respect to the front. I have.

  By using such a system, it is possible to acquire image data around the host vehicle and change it to an overhead view seen from a predetermined direction of the host vehicle by changing the viewpoint. Now, a case where images acquired by the stereo cameras SC1 and SC4 are used will be described as an example.

  The actual arrangement position of each camera differs depending on the vehicle type. For example, the stereo camera SC1 is arranged at the position of the room mirror, and the stereo cameras SC2 and SC4 are arranged at the positions of the right and left side mirrors, respectively. Stereo camera SC3 is arranged at the position of the rear bumper. The above arrangement is an example, and the arrangement of the stereo camera is not limited to this.

  FIG. 2 is a block diagram showing the configuration of the peripheral display device 100 according to the present invention. As shown in FIG. 2, the peripheral display device 100 is output from the stereo cameras SC <b> 1 to SC <b> 4, a display image generation unit 80 that performs image processing of image data obtained by the stereo cameras SC <b> 1 to SC <b> 4, and the display image generation unit 100. And an image display unit 90 for displaying processed image data.

  The display image generation unit 80 includes three-dimensional calculation units 11, 21, 31, and 41 that calculate three-dimensional image data based on the two-dimensional image data obtained by the stereo cameras SC1 to SC4, and three-dimensional calculation. 3D expressing the position of the stereo camera in the vehicle coordinate system (moving body coordinate system) in the 3D image data (first 3D image data) calculated by each of the units 11, 21, 31 and 41 By applying the transformation matrix, the coordinate transformation units 12 and 22 that transform the three-dimensional image data expressed in the stereo camera coordinate system into the three-dimensional image data (second three-dimensional image data) expressed in the car coordinate system. , 32 and 42, three-dimensional image data expressed in the automobile coordinate system output from the coordinate conversion units 12, 22, 32 and 42, and the three-dimensional model of the host vehicle stored in the storage unit 7. 3D image data in the measurement space (integrated) with the vehicle as the origin of the vehicle coordinate system and the 3D image data expressed in the vehicle coordinate system arranged around the vehicle coordinate system. Data) and an image generation unit 6 that generates the three-dimensional image data generated by the integration unit 5 as overhead image data observed by a camera arranged at a virtual position.

  The stereo cameras SC1 to SC4 are both taken simultaneously from different viewpoints by the two cameras, and each of the three-dimensional calculation units 11, 21, 31 and 41 corresponds to the corresponding stereo camera based on the principle of triangulation. The feature points of the two images obtained in step 1 are associated with each other (corresponding point search), and the three-dimensional position for each pair of corresponding points is obtained from parameters such as parallax, focal length, and baseline length obtained as a result. The three-dimensional image data is calculated by obtaining.

  In the three-dimensional calculation units 11, 21, 31, and 41, in calculating three-dimensional image data, in order to reduce the target data of the three-dimensional calculation, the data located in the preset calculation target restriction area is set to three. Take measures not to be subject to dimensional calculation.

  That is, the stereo cameras SC1 to SC4 mounted as shown in FIG. 1 are arranged so that the measurement areas of the stereo cameras overlap each other to some extent in order to prevent measurement omission.

  FIG. 3 is a diagram schematically showing the overlap of the measurement areas of the stereo cameras SC1 and SC4. The measurement area R1 determined by the angle of view of the stereo camera SC1 and the measurement area R4 determined by the angle of view of the stereo camera SC4 are: They overlap each other in the diagonally left front of the host vehicle VC. The three-dimensional image data in the overlapping area DR includes redundant data.

  Therefore, for each stereo camera, a region on the image corresponding to the overlapping region DR is set in advance as a calculation target restriction region, and in the three-dimensional calculation, measurement data in the calculation target restriction region is used. On the other hand, by performing the process of not performing the three-dimensional calculation, the target data of the three-dimensional calculation is reduced, and the calculation cost is reduced.

  Therefore, as shown in FIG. 2, each of the three-dimensional calculation units 11, 21, 31 and 41 has storage units 13, 23, 33 and 43 each storing information (limitation information) on the calculation target restriction region. Therefore, the restriction information is given. For convenience, the restriction information stored in each of the storage units 13, 23, 33, and 43 is referred to as restriction information 1, restriction information 2, restriction information 3, and restriction information 4.

  In FIG. 2, the storage units 13, 23, 33, and 43 are shown as being independent from each other, but each of them is constituted by a ROM (Read Only Memory) or one of storage areas of a hard disk storage device. Also good.

  Here, an example of the calculation target restricted area will be described with reference to FIGS. In the following description, the measurement areas R1 and R4 of the stereo cameras SC1 and SC4 shown in FIG. 3 will be described as an example, and the vehicle OV is taken.

  First, a first example will be described with reference to FIGS. 4 and 5. 4 shows an image taken by the stereo camera SC1, and FIG. 5 shows an image taken by the stereo camera SC4.

  As shown in FIG. 3, the vehicle OV is located in front of the host vehicle VC so that a part of the vehicle OV is included in the overlapping region DR of the measurement regions R1 and R4. Although it is shown, only a part of it is shown in FIG.

  In the case of the overlapping region DR as shown in FIG. 3, the triangular region shown by a broken line at the upper right corner in FIG. 5 corresponds to the overlapping region DR viewed from the stereo camera SC4. Become. Therefore, among the measurement region R4 of the stereo camera SC4, this triangular region is set as the calculation target restricted region LMR, and the pixel data included in this region is not subjected to three-dimensional processing, thereby reducing the calculation cost. it can. A region other than the calculation target restricted region LMR is set as a calculation target region OR, and pixel data included in this region is subjected to a three-dimensional process.

  In FIG. 4, the calculation target restricted region LMR is not set, and the entire region is the calculation target region OR. Since the overlapping region DR is excluded from the measurement region R4 of the stereo camera SC4 as the calculation target restricted region LMR, the image data does not overlap.

  Next, a second example will be described with reference to FIGS. 6 shows an image taken by the stereo camera SC1, and FIG. 7 shows an image taken by the stereo camera SC4. 6 and 7, the position of the vehicle OV is the same as in the examples of FIGS. 4 and 5.

  Assuming an overlapping region DR as shown in FIG. 3, in FIG. 6, the region obtained by combining two triangular regions surrounded by a broken line at the upper left corner is an overlap seen from the stereo camera SC1. This corresponds to the region DR. In FIG. 7, a region obtained by combining two triangular regions surrounded by a broken line at the upper right corner portion corresponds to the overlapping region DR viewed from the stereo camera SC4.

  6 and 7, a triangular area including each corner is set as a calculation target restriction area LMR, and a triangular area adjacent to the calculation target restriction area LMR is an overlap area DR but a calculation target area. The overlapping calculation area DOR is substantially treated as the calculation target area OR.

  In this way, by processing a part of the overlapping region DR as the calculation target region, it is possible to reduce the possibility of missing necessary image data.

  In addition, as a method of dividing the overlapping region DR into the calculation target restricted region LMR and the overlapping calculation region DOR, a method of dividing the overlapping region DR viewed from each stereo camera so as to have a predetermined area ratio is adopted. Can do.

  In the examples of FIGS. 6 and 7, the calculation target restriction region LMR and the overlap calculation region DOR are set to be the same in area ratio, but the present invention is not limited to this, and the overlap region DR What is necessary is just to set suitably according to a position and a magnitude | size.

  Further, in the integration unit 5, among the three-dimensional image data generated by the integration unit 5, an area to be generated as overhead image data by the image generation unit 6 is limited in advance, thereby accompanying the three-dimensional drawing. Calculation cost can be reduced.

  An example of this area is shown in FIG. In FIG. 8, the measurement areas (including overlapping areas) determined by the respective angles of view of the stereo cameras SC1 to SC4 are shown in the three-dimensional image data expressed in the automobile coordinate system. A boundary is set at a position ahead of a predetermined distance from the closed region, a closed region surrounded by the boundary is set as a drawing target region VOR, and the other region is set as a calculation target restricted region LMR. In the example of FIG. 8, the drawing target area VOR is represented as a rectangular area surrounding the host vehicle VC as a center.

  In FIG. 2, such information on the drawing target area VOR and the calculation target restriction area LMR is provided to the integration unit 5 as restriction information 5, and the integration part 5 performs only the 3D image data included in the drawing target area VOR. Is generated to the image generation unit 6 to generate overhead image data.

  As shown in FIG. 2, the restriction information 5 is stored in the independent storage unit 53, but this may also be configured by one of storage areas of a ROM or a hard disk storage device, similar to the storage unit 13 or the like. .

<Operation>
Next, on the premise of the above description, the operation of the peripheral display device 100 will be described using the flowchart shown in FIG.

  When the peripheral display device 100 starts operating, image data from the stereo cameras SC1 to SC4 starts to be acquired (step S1).

  In each of the three-dimensional calculation units 11, 21, 31 and 41, the feature points (measurement points) of the two images respectively obtained by the two cameras of the corresponding stereo cameras are associated with each other. A point search is performed (step S2). This is a conventional technique.

  The measurement points (corresponding points) associated with each other by the corresponding point search are included in the calculation target restriction region LMR set by the restriction information 1 to 4 stored in the storage units 13, 23, 33 and 43. It is determined whether or not there is (step S3).

  If it is determined that the measurement point is included in the calculation target restricted region LMR, the process proceeds to step S9 to discard the data of the measurement point, and then proceeds to step S8.

  On the other hand, when it is determined that the measurement point is not included in the calculation target restriction region LMR, the process proceeds to step S4, and the parameter of each corresponding point is determined from parameters such as parallax, focal length, and baseline length with respect to the measurement point. Three-dimensional image data is calculated by performing a three-dimensional calculation for obtaining a three-dimensional position for the set.

  Since the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41 is expressed in the stereo camera coordinate system, coordinates are used for conversion into the three-dimensional image data expressed in the car coordinate system. Coordinate conversion is performed in the conversion units 12, 22, 32, and 42 (step S5).

  This coordinate transformation is executed by applying predetermined three-dimensional transformation matrices Mat1, Mat2, Mat3, and Mat4 to the three-dimensional image data output from the three-dimensional calculation units 11, 21, 31, and 41, respectively. The

  Here, as shown in FIG. 2, the three-dimensional transformation matrices Mat1 to Mat4 are stored in the storage units 14, 24, 34, and 44 associated with the coordinate transformation units 12, 22, 32, and 42, respectively. .

  The three-dimensional image data expressed in the car coordinate system output from the coordinate conversion units 12, 22, 32 and 42 is given to the integration unit 5 and integrated with the data of the three-dimensional model of the host vehicle. At the time of processing, it is determined whether or not each measurement point (converted to the vehicle coordinate system) is included in the drawing target area VOR as shown in FIG. 8, for example (step S6).

  If it is determined that the measurement point is included in the drawing target area VOR, the image data of the measurement point is given to the image generation unit 6 to generate overhead image data (step S7).

  On the other hand, when it is determined that the measurement point is not included in the drawing target region VOR, the process proceeds to step S9, the data of the measurement point is discarded, and then the process proceeds to step S8.

  After the bird's-eye view image data is generated by the image generation unit 6, in step S 8, it is confirmed whether or not the processing after step S 3 has been completed for all measurement points, and when the processing has been completed for all measurement points. The overhead image data is output to the image display unit 90, and the series of operations ends. On the other hand, if the processing for all the measurement points has not been completed, the processing after step S3 is repeated for the new measurement points.

  Note that the determination in step S8 may be performed by the image generation unit 6, may be performed by the integration unit 5, or may be performed by a control unit (not shown) that controls the display image generation unit 80.

  As described above, the display image generation unit 80 of the peripheral display device 100 determines whether or not the measurement points measured by each stereo camera are included in the preset calculation target restriction region, In the three-dimensional calculation in the three-dimensional calculation units 11, 21, 31 and 41, the measurement data in the calculation target restriction region is processed so that the three-dimensional calculation is not performed. The target data of the dimensionalization calculation can be reduced, and the calculation cost can be reduced.

  Further, in the integration unit 5, among the three-dimensional image data generated by the integration unit 5, an area to be generated as the overhead image data by the image generation unit 6 is limited in advance as a drawing target area. It is possible to reduce the calculation cost associated with the dimension drawing.

<Modification 1>
As described above, in the display image generation unit 80 of the peripheral display device 100, when the three-dimensional calculation is performed by the three-dimensional calculation units 11, 21, 31, and 41, the measurement data in the calculation target restricted region is processed. Although the configuration in which the three-dimensional calculation is not performed is shown, the restriction information 1 to 4 that defines the calculation target restriction region corresponds to the two-dimensional image data of the stereo camera coordinate system. Instead, the restriction information 1 to 4 may be set as corresponding to the three-dimensional image data of the stereo camera coordinate system.

  FIG. 10 shows an example of the calculation target restricted area corresponding to the three-dimensional image data of the stereo camera coordinate system.

  FIG. 10 shows the measurement area of the stereo camera SC1 when viewed from above the host vehicle VC. A boundary is set at a position ahead of the stereo camera SC1 by a predetermined distance, and the angle of view of the boundary and the stereo camera SC1 is shown. The region defined by the above is a calculation target region OR, and the other region is a calculation target restricted region LMR.

  Then, as a result of the three-dimensional calculation by the three-dimensional calculation units 11, 21, 31, and 41, a measure is taken that the measurement points included in the calculation target restricted region LMR are not output as three-dimensional image data. Thereby, the calculation cost in the processing after the three-dimensional calculation can be reduced.

<Modification 2>
In the display image generation unit 80 of the peripheral display device 100, when the three-dimensional calculation is performed by the three-dimensional calculation units 11, 21, 31, and 41, the three-dimensional calculation is performed on the measurement data in the calculation target restricted area. The corresponding point obtained by the corresponding point search is located on the screen shot by each stereo camera. It was judged by what.

  However, by using the parallax used for the corresponding point search, it is possible to more easily define the corresponding points for which the three-dimensional calculation is not performed.

  That is, the parallax used for the corresponding point search becomes smaller as the distance from the camera increases. FIG. 11 schematically shows the parallax characteristics.

  In FIG. 11, the light receiving portions of the two cameras constituting the stereo camera are shown as sensor surfaces SF1 and SF2, and the focal positions by the lens systems of the two cameras are shown as the focal points FP1 and FP2.

  Here, when the corresponding point closest to the camera is the corresponding point P1, the next farthest corresponding point is the corresponding point P2, and the farthest corresponding point is the corresponding point P3, the corresponding point P1. , P2 and P3 are represented by L1, L2 and L3, respectively, and L1 is the largest and L3 is the smallest.

  When the threshold value of the parallax amount is set for the corresponding points P1 to P3 having such a magnitude relationship with the parallax amount, the distance is determined corresponding to the threshold value, and this distance is referred to as an assumed limit distance VLM. In FIG. 11, the assumed limit distance VLM is set so as to correspond to the distance of the corresponding point P2, and the parallax amount corresponding to the assumed limit distance VLM is L2.

  The three-dimensional calculation by the three-dimensional calculation units 11, 21, 31 and 41 is not performed on the corresponding point (P3 in FIG. 11) having the parallax amount L2 as a threshold and having a parallax amount not exceeding the threshold. By taking measures, 3D calculation is not required for corresponding points at a long distance that do not need to be grasped during driving of the vehicle, and as a result, data is left for corresponding points that need to be grasped during driving. However, the calculation cost can be reduced by reducing the data to be calculated.

<Modification 3>
In the display image generation unit 80 of the peripheral display device 100, when the three-dimensional calculation units 11, 21, 31, and 41 perform the three-dimensional calculation, the measurement data in the calculation target restricted region (FIGS. 4 to 7) Although the configuration in which the three-dimensional calculation is not performed is shown, the three-dimensional calculation in the three-dimensional calculation units 11, 21, 31 and 41 is performed for all measurement data, and the three-dimensional calculation unit 11 is performed. , 21, 31 and 41, the coordinate conversion units 12, 22, 32 and 42 do not perform coordinate conversion on the three-dimensional image data in the calculation target restricted area. You may take it.

  In this case, the calculation target restricted region may be set as corresponding to the three-dimensional image data of the stereo camera coordinate system as described with reference to FIG.

  Moreover, although the example which sets the drawing object area | region VOR (FIG. 8) in the integration part 5 was shown, the same area | region is set also in the coordinate transformation parts 12, 22, 32, and 42, and the three-dimensional image data in the said area | region A configuration may be adopted in which coordinate transformation is performed only for the.

  By adopting these methods, it is possible to reduce the coordinate conversion process and reduce the calculation cost.

<Modification 4>
The display image generation unit 80 described with reference to FIG. 2 determines whether the image captured by each stereo camera is an image captured in a normal state or an image captured in an abnormal state. It is good also as a structure which further has a detection part. FIG. 12 is a block diagram illustrating a configuration of a peripheral display device 100A including a display image generation unit 80A having the abnormality detection unit. In the peripheral display device 100A shown in FIG. 12, the same components as those in the peripheral display device 100 shown in FIG.

  Peripheral display device 100A has a configuration in which abnormality detection units 10, 20, 30, and 40 are connected to signal lines that provide outputs of stereo cameras SC1 to SC4 to three-dimensional calculation units 11, 21, 31, and 41, respectively. It has become.

  Each abnormality detection unit, for example, counts the number of pixels having a pixel value equal to or greater than a predetermined value (a value close to the saturation value of the pixel) for image data captured by a corresponding stereo camera, and is equal to or greater than a predetermined threshold value. When the number of pixels is counted, processing for determining that the image data is abnormal is performed. This process can cope with an abnormality in which the number of saturated pixels becomes extremely large.

  As another example, the time change of each pixel value is measured for image data captured by a stereo camera, the number of pixels that have not changed for a predetermined time or more is counted, and the number of unchanged pixels is determined. When the ratio with respect to the total number of pixels is equal to or greater than a predetermined threshold value, processing for determining the image data as abnormal is performed. This processing can cope with a case where dirt is attached to the lens of the camera.

  As another example, when searching for corresponding points for image data captured by a stereo camera, a similarity search is performed in which a pixel expression parameter (error value) is evaluated to search for a similarity, and the similarity is a predetermined value. When the above number of pixels is equal to or less than a predetermined threshold value, processing for determining that the image data is abnormal is performed. This processing can cope with an abnormality in which the number of saturated pixels becomes extremely large or a stain on the camera lens.

  The operation of the peripheral display device 100A having such functions and having the abnormality detection units 10, 20, 30, and 40 will be described with reference to the flowchart shown in FIG.

  When the peripheral display device 100A starts operating, image data from the stereo cameras SC1 to SC4 starts to be acquired (step S11).

  In each of the three-dimensional calculation units 11, 21, 31 and 41, the feature points (measurement points) of the two images respectively obtained by the two cameras of the corresponding stereo cameras are associated with each other. A point search is performed (step S12). This is a conventional technique.

  Further, the abnormality detection units 10, 20, 30 and 40 detect the presence or absence of abnormality in the image data captured by the corresponding stereo cameras SC1 to SC4 by the above-described method (step S13).

  If no abnormality is detected in the image data, the process from step S14 is performed. In addition, since the process of step S14-S20 is the same as the process of step S3-S9 shown in FIG. 9, description is abbreviate | omitted.

  On the other hand, if an abnormality in the image data is detected in step S13, the image data is discarded in step S21, and the operations in step S11 and subsequent steps are repeated.

  As described above, in the peripheral display device 100A, when the image data acquired by the stereo cameras SC1 to SC4 is captured in an abnormal state, it can be determined, and processing for discarding the abnormal image data is performed. By doing so, it is possible to prevent the abnormal image data from being subjected to processing such as three-dimensional calculation, so that the calculation cost can be reduced.

<Modification 5>
FIG. 1 shows an example in which one stereo camera is arranged in each of the four directions of the vehicle VC. However, as shown in FIG. 14, two stereo cameras may be arranged in each of the four directions of the vehicle VC. good.

  That is, as shown in FIG. 14, stereo cameras SC1 and SC2 that acquire the left and right images of the vehicle VC are arranged, and the stereo camera SC3 that acquires the images of the front side and the rear side on the right side with respect to the front side. SC4 is arranged, stereo cameras SC5 and SC6 for obtaining the left and right images at the rear are arranged, and stereo cameras SC7 and SC8 for obtaining the images at the rear and the front on the left side with respect to the front are arranged.

  By using such a system, it is possible to acquire peripheral image data with fewer blind spots.

5 Integration unit 6 Image generation unit 10, 20, 30, 40 Anomaly detection unit 11, 21, 31, 41 Three-dimensional calculation unit 12, 22, 32, 42 Coordinate conversion unit SC1 to SC4 Stereo camera

Claims (13)

A plurality of stereo cameras that acquire peripheral image data in a self-propelled movable body;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second 3D image data;
An integration unit that generates integrated data obtained by integrating the second three-dimensional image data and the data of the three-dimensional model of the moving body;
An image generation unit that generates overhead image data from the integrated data,
The three-dimensional calculation unit
For the image data obtained by each of the plurality of stereo cameras, the feature points of two images are associated with each other, and the three-dimensional position of the measurement point on the image is calculated from the parallax obtained as a result. Obtaining the first three-dimensional image data;
The three-dimensional calculation unit, have a calculated target restriction information for reducing in at least one processing target data of the coordinate transformation unit and the integration unit,
The calculation target restriction information is
A peripheral display device, which is information given to the three-dimensional calculation unit, and is information that limits a calculation target of the three-dimensional position among the measurement points . The calculation target restriction information is
Information on a predetermined measurement area defined on a two-dimensional image obtained by each of the plurality of stereo cameras,
The peripheral display device according to claim 1, wherein the measurement point data included in the predetermined measurement region is not subject to calculation of the three-dimensional position . The predetermined measurement area is:
The peripheral display device according to claim 2, which corresponds to a measurement region that overlaps each other in each of the plurality of stereo cameras. The calculation target restriction information is
Information on a threshold value set for the parallax amount of the measurement point,
The data of the measurement points parallax amount does not exceed the threshold value, no calculation target of the three-dimensional position, a peripheral display device according to claim 1, wherein. A plurality of stereo cameras that acquire peripheral image data in a self-propelled movable body;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second 3D image data;
An integration unit that generates integrated data obtained by integrating the second three-dimensional image data and the data of the three-dimensional model of the moving body;
An image generation unit that generates overhead image data from the integrated data,
The three-dimensional calculation unit
For the image data obtained by each of the plurality of stereo cameras, the feature points of two images are associated with each other, and the three-dimensional position of the measurement point on the image is calculated from the parallax obtained as a result. Obtaining the first three-dimensional image data;
Calculation target restriction information for reducing processing target data in at least one of the three-dimensional calculation unit, the coordinate conversion unit, and the integration unit;
The calculation target restriction information is
Given to the three-dimensional calculation unit,
Information on a predetermined measurement region defined on a three-dimensional image obtained by each of the plurality of stereo cameras,
Wherein the data of said measurement point is included in a predetermined measurement region, not processed data after the calculation of the three-dimensional position, peripheral display device. A plurality of stereo cameras that acquire peripheral image data in a self-propelled movable body;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second 3D image data;
An integration unit that generates integrated data obtained by integrating the second three-dimensional image data and the data of the three-dimensional model of the moving body;
An image generation unit that generates overhead image data from the integrated data,
The three-dimensional calculation unit
For the image data obtained by each of the plurality of stereo cameras, the feature points of two images are associated with each other, and the three-dimensional position of the measurement point on the image is calculated from the parallax obtained as a result. Obtaining the first three-dimensional image data;
Calculation target restriction information for reducing processing target data in at least one of the three-dimensional calculation unit, the coordinate conversion unit, and the integration unit;
The calculation target restriction information is given to the coordinate conversion unit,
The first of the three-dimensional image data, which is information to reduce the target coordinate transformation, peripheral display device. The calculation target restriction information is
Information on a predetermined measurement region defined on a three-dimensional image obtained by each of the plurality of stereo cameras,
The peripheral display device according to claim 6 , wherein the data of the measurement points included in the predetermined measurement area is not subject to the coordinate conversion . The calculation target restriction information is
Information on a predetermined measurement area defined on a three-dimensional image represented by the moving body coordinate system,
The peripheral display device according to claim 6 , wherein the data of the measurement points included in the predetermined measurement area is not subject to the coordinate conversion. The predetermined measurement area is:
8. A boundary is set at a position ahead of each of the plurality of stereo cameras by a predetermined distance, and corresponds to an area other than an area defined by the boundary and an angle of view of each stereo camera. Item 9. The peripheral display device according to any one of items 8 to 9 . A plurality of stereo cameras that acquire peripheral image data in a self-propelled movable body;
A three-dimensional calculation unit that calculates first three-dimensional image data based on the image data;
A coordinate conversion unit that converts the coordinate system of the first three-dimensional image data into a moving body coordinate system that is looked down from a predetermined direction of the moving body to obtain second 3D image data;
An integration unit that generates integrated data obtained by integrating the second three-dimensional image data and the data of the three-dimensional model of the moving body;
An image generation unit that generates overhead image data from the integrated data,
The three-dimensional calculation unit
For the image data obtained by each of the plurality of stereo cameras, the feature points of two images are associated with each other, and the three-dimensional position of the measurement point on the image is calculated from the parallax obtained as a result. Obtaining the first three-dimensional image data;
Calculation target restriction information for reducing processing target data in at least one of the three-dimensional calculation unit, the coordinate conversion unit, and the integration unit;
The calculation target restriction information is given to the integration unit,
Wherein among the integrated data is information to limit the area to be generated as the bird's eye image data in the image generation unit, peripheral display device. The calculation target restriction information is
Information on a predetermined measurement area defined on a three-dimensional image represented by the moving body coordinate system,
The peripheral display device according to claim 10 , wherein data of the measurement points included in the predetermined measurement area is discarded . Based on the image data acquired by the plurality of stereo cameras, it is determined whether an image captured by each stereo camera is an image captured in a normal state or an image captured in an abnormal state. An anomaly detection unit that
The peripheral display device according to any one of claims 1 , 5, 6, and 10 , wherein the image data is discarded for an image determined to be abnormal by the abnormality detection unit . The abnormality detection unit
Regarding the image data acquired by the plurality of stereo cameras, the number of pixels having a pixel value equal to or greater than a predetermined value is counted, and if the number of pixels equal to or greater than a predetermined threshold is counted, the image data is determined to be abnormal. to the peripheral display device according to claim 12.

Images