JP7047306B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to an information processing apparatus, an information processing method, and a program.

In order to map the position of an object included in an image captured by a camera arranged so as to look down from diagonally above onto an image (referred to as a bird's-eye view map in this specification) from directly above, a projective transformation is performed. Transformation parameters such as matrices are used. For example, when a plurality of objects are installed in a real space corresponding to a bird's-eye view map, the conversion parameter is specified from the correspondence between the position coordinates in the captured image of the plurality of objects and the position coordinates in the bird's-eye view map of the plurality of objects. It is possible to do. Obtaining conversion parameters in this way is called calibration.

It is sufficient to perform the above calibration once if the camera arrangement is not changed. Further, Patent Document 1 below discloses a technique for calibrating camera parameters using images captured before and after the camera arrangement when the camera arrangement is changed.

Japanese Unexamined Patent Publication No. 2017-78923

However, the technique described in Patent Document 1 targets the change in the posture (angle) of the camera for calibration, and cannot cope with the change in the camera position. Therefore, when changing the position of the camera or adding a camera for the purpose of changing or enlarging the range to be converted into a bird's-eye view map, as described above, the position coordinates in the captured image of a plurality of objects and the bird's-eye view map of the plurality of objects. It is necessary to recalibrate from the correspondence with the position coordinates in. At this time, in order to obtain the position coordinates in the bird's-eye view map of the plurality of objects, it is necessary to measure in the real space or install the objects at the predetermined positions in the real space, which is costly.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is a new and improved information processing apparatus and information processing capable of suppressing the cost related to calibration. To provide methods and programs.

In order to solve the above problems, according to a certain viewpoint of the present invention, the captured image coordinates which are the position coordinates in the captured image captured by the camera and the bird's-eye view map coordinates which are the position coordinates in the bird's-eye view map corresponding to the real space. Corresponding to the arrangement of the camera, the calibration unit for specifying the conversion parameter for converting the captured image coordinate to the bird's-eye view map coordinate, and the converted image are used to obtain the captured image coordinate. The coordinate conversion unit includes a coordinate conversion unit that converts the bird's-eye view map coordinates and a target object identification unit that specifies the captured image coordinates of the target object, and the coordinate conversion unit sets a first conversion parameter corresponding to the first camera arrangement. Using, the captured image coordinates of the target object in the first captured image captured by the first camera arrangement are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map, and the calibration unit is used. Correspondence between the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map. Based on this, the second conversion parameter corresponding to the second camera arrangement is specified , the target object identification unit detects the captured image coordinates of the target object candidate from the captured image, and the detected target object. Provided is to specify the candidate's captured image coordinates as the captured image coordinates of the target object based on the determination of the operator .

Further, in order to solve the above problems, according to another viewpoint of the present invention, the captured image coordinates which are the position coordinates in the captured image captured by the camera and the bird's-eye view which is the position coordinates in the bird's-eye view map corresponding to the real space. Based on the correspondence with the map coordinates, the calibration unit corresponding to the arrangement of the camera and specifying the conversion parameters for converting the captured image coordinates to the bird's-eye view map coordinates, the captured image using the conversion parameters. It is provided with a coordinate conversion unit that converts the coordinates into the bird's-eye view map coordinates, and a correction unit that corrects the conversion parameters specified by the calibration unit using a reference object whose bird's-eye view map coordinates are known. The coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to map the captured image coordinates of the target object in the first captured image captured by the first camera arrangement to the bird's-eye view map. Converted to the bird's-eye view map coordinates of the target object in the above, and the calibration unit is the captured image of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement. An information processing apparatus is provided that specifies a second conversion parameter corresponding to the second camera arrangement based on the correspondence between the coordinates and the bird's-eye view map coordinates of the target object in the bird's-eye view map.

The information processing apparatus may further include a correction unit that corrects the conversion parameter specified by the calibration unit using a reference object whose bird's-eye view map coordinates are known.

The correction unit has known reference coordinates that are known bird's-eye view map coordinates of the reference object, and the bird's-eye view map coordinates of the reference object obtained by the coordinate conversion unit converting the captured image coordinates of the reference object. It may have a distance specifying unit for specifying a distance from the conversion reference coordinates, and may correct the conversion parameter based on the distance.

The correction unit may further include a determination unit for determining whether or not the distance is within a predetermined range, and may correct the conversion parameter according to the result of the determination by the determination unit.

When the determination unit determines that the distance is not within the predetermined range, the correction unit obtains the bird's-eye view map coordinates of the target object based on the difference between the known reference coordinates and the conversion reference coordinates. The coordinate correction unit for correction may be further provided, and the conversion parameter may be corrected based on the bird's-eye view map coordinates of the target object corrected by the coordinate correction unit.

The number of reference objects required for the correction unit to correct the conversion parameter may be less than the number of target objects required for the calibration unit to specify the second conversion parameter.

Further, in order to solve the above problem, according to another viewpoint of the present invention, the captured image coordinates which are the position coordinates in the captured image captured by the camera and the bird's-eye view which is the position coordinates in the bird's-eye view map corresponding to the real space. Based on the correspondence with the map coordinates, the calibration unit, the sensor information management unit, and the conversion parameters that correspond to the arrangement of the camera and specify the conversion parameters for converting the captured image coordinates to the bird's-eye view map coordinates are used. The coordinate conversion unit includes a coordinate conversion unit that converts the captured image coordinates into the bird's-eye view map coordinates, and the coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to arrange the first camera. The captured image coordinates of the target object in the first captured image captured in 1 are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map, and the calibration unit is different from the first camera arrangement. Based on the correspondence between the captured image coordinates of the target object in the second captured image captured by the second camera arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map, the second camera arrangement is adopted. The corresponding second conversion parameter is specified, and the sensor information management unit determines that the number of the target objects included in both the first captured image and the second captured image is less than a predetermined threshold value. Based on the bird's-eye view map coordinates of the first target object and the sensing of the position sensor in the bird's-eye view map obtained by converting the captured image coordinates of the first target object in the first captured image. Based on the position coordinates of the first target object in the sensor coordinate system corresponding to the position sensor obtained and the position coordinates of the second target object in the sensor coordinate system obtained based on the sensing of the position sensor. Then, the bird's-eye view map coordinates of the second target object in the bird's-eye view map are acquired, and the calibration unit obtains the captured image coordinates of the second target object in the second captured image and the above-mentioned in the bird's-eye view map. An information processing apparatus is provided that specifies a second conversion parameter based on the correspondence of the second target object with the bird's-eye view map coordinates.

Further, in order to solve the above problem, according to another viewpoint of the present invention, the captured image coordinates which are the position coordinates in the captured image captured by the camera and the bird's-eye view which is the position coordinates in the bird's-eye view map corresponding to the real space. Based on the correspondence with the map coordinates, the image pickup is performed by using the calibration unit that corresponds to the arrangement of the camera and specifies the conversion parameter for converting the captured image coordinates to the bird's-eye view map coordinates, and the conversion parameter. A coordinate conversion unit that converts image coordinates into the bird's-eye view map coordinates is provided, and the coordinate conversion unit is imaged with the first camera arrangement using the first conversion parameter corresponding to the first camera arrangement. The captured image coordinates of the target object in the first captured image are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map, and the calibration unit is a second camera different from the first camera arrangement. The second camera arrangement corresponds to the second camera arrangement based on the correspondence between the captured image coordinates of the target object in the second captured image captured by the arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map. The transformation parameters are specified, and the transformation parameters are provided with an information processing apparatus including a projection transformation matrix or a perspective projection transformation matrix.

Further, in order to solve the above problems, according to another viewpoint of the present invention, the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map. Based on this, the calibration unit that specifies the conversion parameters for converting the captured image coordinates to the bird's-eye view map coordinates according to the arrangement of the cameras, and the converted image coordinates are converted to the bird's-eye view map coordinates by using the conversion parameters. A first information processing method executed by an information processing apparatus including a coordinate conversion unit for converting to a target object and a target object identification unit for specifying the captured image coordinates of the target object , the first one corresponding to the first camera arrangement. Using the conversion parameters, the captured image coordinates of the target object in the first captured image captured by the first camera arrangement are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map. Based on the correspondence between the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map. Then, the second conversion parameter corresponding to the second camera arrangement is specified, the captured image coordinates of the target object candidate are detected from the captured image, and the captured image coordinates of the detected target object candidate are detected. Is specified as the captured image coordinates of the target object based on the determination of the operator, and an information processing method is provided.

Further, in order to solve the above problems, according to another viewpoint of the present invention, the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map. Based on the calibration function that specifies the conversion parameter for converting the captured image coordinates to the bird's-eye view map coordinates according to the arrangement of the cameras, the captured image coordinates are converted to the bird's-eye view map coordinates using the conversion parameters. It is a program for realizing a computer with a coordinate conversion function for converting to and a target object specifying function for specifying the captured image coordinates of the target object, and the coordinate conversion function corresponds to a first camera arrangement. Using the conversion parameter of 1, the captured image coordinates of the target object in the first captured image captured by the first camera arrangement are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map. The calibration function includes the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map of the target object in the bird's-eye view map. The second conversion parameter corresponding to the second camera arrangement is specified based on the correspondence with the coordinates, and the target object identification function detects and detects the captured image coordinates of the target object candidate from the captured image. A program is provided that identifies the captured image coordinates of the target object candidate as the captured image coordinates of the target object based on the determination of the operator . Further, in order to solve the above problems, according to another viewpoint of the present invention, the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map. Based on the calibration function that specifies the conversion parameter for converting the captured image coordinates to the bird's-eye view map coordinates according to the arrangement of the cameras, the captured image coordinates are converted to the bird's-eye view map coordinates using the conversion parameters. It is a program for realizing on a computer a coordinate conversion function for converting to a camera and a correction function for correcting the conversion parameter specified by the calibration function using a reference object whose bird's-eye view map coordinates are known. The coordinate conversion function uses the first conversion parameter corresponding to the first camera arrangement to obtain the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converting to the bird's-eye view map coordinates of the target object in the bird's-eye view map, the calibration function of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement. A program is provided that identifies a second conversion parameter corresponding to the second camera arrangement based on the correspondence between the captured image coordinates and the bird's-eye view map coordinates of the target object in the bird's-eye view map. Further, in order to solve the above problems, according to another viewpoint of the present invention, the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map. Based on this, the calibration function for specifying the conversion parameter for converting the captured image coordinates to the bird's-eye view map coordinates corresponding to the arrangement of the camera, and the conversion parameter are used to convert the captured image coordinates to the bird's-eye view map. It is a program for realizing a coordinate conversion function for converting into coordinates on a computer, and the coordinate conversion function uses the first conversion parameter corresponding to the first camera arrangement and the first camera arrangement. The captured image coordinates of the target object in the first captured image captured in 1 are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map, and the calibration function is different from the first camera arrangement. Based on the correspondence between the captured image coordinates of the target object in the second captured image captured by the second camera arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map, the second camera arrangement is adopted. A corresponding second transformation parameter is specified, the transformation parameter being provided programmatically comprising a projection transformation matrix or a perspective projection transformation matrix.

As described above, according to the present invention, it is possible to suppress the cost related to calibration.

It is a block diagram which shows the structural example of the information processing apparatus 8 which concerns on the comparative example of embodiment of this invention. It is explanatory drawing for demonstrating the method for obtaining the correspondence between the captured image coordinate, and the bird's-eye view map coordinate. It is a block diagram which shows the structural example of the information processing apparatus 1 which concerns on 1st Embodiment of this invention. It is a flowchart which shows the operation outline which concerns on the same embodiment. It is a flowchart which shows the detailed operation of the initial calibration. It is explanatory drawing for demonstrating the initial calibration. It is explanatory drawing for demonstrating the initial calibration. It is a flowchart which shows the detailed operation of the recalibration which concerns on the same embodiment. It is explanatory drawing for demonstrating the recalibration which concerns on the embodiment. It is explanatory drawing for demonstrating the recalibration which concerns on the embodiment. It is a block diagram which shows the structural example of the information processing apparatus 2 which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the operation outline which concerns on the same embodiment. It is a flowchart which shows the detailed operation of the correction of a conversion parameter. It is explanatory drawing for demonstrating the correction of a conversion parameter. It is explanatory drawing for demonstrating the correction of a conversion parameter. It is a block diagram which shows the structural example of the information processing apparatus 3 which concerns on 3rd Embodiment of this invention. It is a flowchart which shows an example of the operation which concerns on the recalibration which concerns on the same embodiment. It is explanatory drawing for demonstrating the recalibration which concerns on the embodiment. It is explanatory drawing for demonstrating the recalibration which concerns on the embodiment. It is explanatory drawing which shows the hardware composition of the information processing apparatus 1000 which concerns on embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to particularly distinguish each of the plurality of components having substantially the same functional configuration, only the same reference numerals are given.

<< 1. Background >>
In explaining the embodiment of the present invention, first, the background leading to the creation of the embodiment of the present invention will be described.

At construction sites, commercial facilities, and the like, the position of an object such as a person is detected or tracked from an image captured by a camera. In order for humans to see and grasp it and use it for various analyzes, it is desirable to obtain the position coordinates in the bird's-eye view map that corresponds to the real space and is viewed from directly above. However, since it is difficult to arrange the camera so as to look down from directly above, a camera arranged so as to look down from diagonally above is often used.

Therefore, the position coordinates of the object obtained from the captured image captured by the camera arranged so as to look down from diagonally above are mapped to the bird's-eye view map (converted to the position coordinates in the bird's-eye view map). ing. Transformation parameters such as a projective transformation matrix and a perspective projection transformation matrix are used for such mapping.

For example, assuming that the projective transformation matrix from the captured image to the bird's-eye view map is H and the position coordinates in the captured image are G (x, y), the position coordinates F (x, y) in the bird's-eye view map are obtained by the following equation (1). It is possible.

The conversion parameters from the captured image to the bird's-eye view map (for example, the projective transformation matrix) are the position coordinates in the captured image (hereinafter, may be referred to as captured image coordinates) and the position coordinates in the bird's-eye view map (hereinafter referred to as bird's-eye view map coordinates). It can be obtained by using multiple correspondences with (may be). Obtaining the conversion parameters from the captured image to the bird's-eye view map in this way is hereinafter referred to as bird's-eye view map calibration, or simply calibration.

As described above, in order to perform calibration, it is necessary to prepare a plurality of correspondences between the captured image coordinates and the bird's-eye view map coordinates. Therefore, for example, using a plurality of objects installed in the real space corresponding to the bird's-eye view map, a plurality of correspondences between the captured image coordinates and the bird's-eye view map coordinates are prepared and calibrated.

Hereinafter, the object used for calibration in this way is referred to as a target object. The target object is not limited to the object newly installed for calibration, and for example, an object that already exists in the real space can be used as the target object.

Here, with reference to FIGS. 1 and 2, an example of an information processing apparatus that performs calibration using a plurality of target objects in this way will be described as a comparative example of an embodiment of the present invention.

FIG. 1 is a block diagram showing a configuration example of the information processing apparatus 8 according to this comparative example. As shown in FIG. 1, the information processing apparatus 8 according to this comparative example includes a control unit 810, an image acquisition unit 860, an operation unit 870, an output unit 880, and a storage unit 890. The information processing device 8 according to this comparative example is an information processing device that obtains the bird's-eye view map coordinates of an object reflected in the captured image by performing calibration for obtaining conversion parameters from the captured image to the bird's-eye view map and using the projective transformation matrix. be. Hereinafter, the schematic configuration of the information processing apparatus 8 will be described.

The control unit 810 controls each configuration of the information processing apparatus 8. Further, as shown in FIG. 1, the control unit 810 according to this comparative example also functions as an object position specifying unit 811, a calibration unit 813, and a coordinate conversion unit 815.

The object position specifying unit 811 specifies the position coordinates (captured image coordinates) of the object reflected in the captured image provided by the image acquisition unit 860 for acquiring the captured image in the captured image. The method of specifying the captured image coordinates by the object position specifying unit 811 is not particularly limited, but for example, the captured image coordinates may be specified based on the user operation using the operation unit 870, or the captured image coordinates are specified by the image processing technique. May be done.

The object position specifying unit 811 provides the captured image coordinates to the calibration unit 813 or the coordinate conversion unit 815. For example, when the work related to the calibration is performed, the object position specifying unit 811 provides the captured image coordinates of the target object to the calibration unit 813. Further, when it is desired to obtain the bird's-eye view map position of the object reflected in the captured image after the calibration is completed, the object position specifying unit 811 provides the captured image coordinates of the object to the coordinate conversion unit 815.

The calibration unit 813 specifies a conversion parameter for converting the captured image coordinates into the bird's-eye view map coordinates based on the correspondence between the captured image coordinates and the bird's-eye view map coordinates. For example, when the transformation parameter is a projective transformation matrix, it is possible to specify the projective transformation matrix using the above equation (1) if the correspondence between the four or more sets of captured image coordinates and the bird's-eye view map coordinates is obtained. Is. For example, the calibration unit 813 specifies (calculates) a 3 × 3 projective transformation matrix (an example of transformation parameters) by solving simultaneous equations obtained from the correspondence between four or more sets of captured image coordinates and the bird's-eye view map coordinates. ) May.

Here, a method for obtaining a correspondence between the captured image coordinates and the bird's-eye view map coordinates will be described with reference to FIG. FIG. 2 is an explanatory diagram for explaining a method for obtaining a correspondence between the captured image coordinates and the bird's-eye view map coordinates.

FIG. 2 shows the real space R100 seen from directly above, and the camera 9 and the target objects T101 to T104 are installed in the real space. Further, as shown in FIG. 2, a real space coordinate system with the point OR as the origin is set in the real space _R100 .

The image captured by the camera 9 is, for example, the captured image G100 shown in FIG. The position coordinates (captured image coordinates) P101 to P104 of the target objects T101 to T104 in the captured image G100 are specified by the object position specifying unit 811 as described above.

Further, the bird's-eye view map corresponding to the real space R100 is like the bird's-eye view map M100 shown in FIG. The bird's-eye view map _M100 has an origin _OM corresponding to the origin OR of the real space R100, and each position in the real space R100 and each position in the bird's-eye view map M100 correspond to each other at a predetermined scale. Therefore, in order to obtain the position coordinates (overhead map coordinates) of the target objects T101 to T104 in the bird's-eye view map M100, for example, the real space positions of the target objects T101 to T104 may be known.

The target objects T101 to T104 may be installed at a predetermined position where the real space position is known, for example. Alternatively, when the target objects T101 to T104 are installed at positions where the real space position is unknown, the real space positions of the target objects T101 to T104 may be measured by a well-known positioning method. The positioning method may be, for example, a manual method or an automatic method using a positioning device such as a GPS receiver. Information on the real space position of the target object may be input via a communication unit (not shown), or may be input by an operation of the operation unit 870 by a user (operator).

Information on the real space position of the target object thus obtained is provided to the calibration unit 813, and the calibration unit 813 has the bird's-eye view map coordinates F101 to F104 of the target objects T101 to T104 based on the real space position. Is possible. The user may specify the bird's-eye view map coordinates from the real space position, and in such a case, the user may operate the operation unit 870 to directly input the bird's-eye view map coordinates.

As described above, the calibration unit 813 can use the correspondence between the captured image coordinates P101 to P104 and the bird's-eye view map coordinates F101 to F104 for calibration. The conversion parameter specified by the calibration unit 813 is stored in the conversion parameter DB891 stored in the storage unit 890.

The explanation will be continued by returning to FIG. The coordinate conversion unit 815 performs coordinate conversion that converts the captured image coordinates provided by the object position specifying unit 811 into the bird's-eye view map coordinates by using the conversion parameters. The coordinate conversion unit 815 performs such coordinate conversion using the conversion parameters stored in the conversion parameter DB891 stored in the storage unit 890, that is, the conversion parameters obtained by the calibration of the calibration unit 813 described above. May be good. The bird's-eye view map coordinates of the object obtained by the coordinate conversion of the coordinate conversion unit 815 are output from, for example, the output unit 880.

The schematic configuration of the information processing apparatus 8 according to this comparative example has been described above. As described above, in order to perform calibration by the information processing apparatus 8, it is necessary to obtain the correspondence between the captured image coordinates and the bird's-eye view map coordinates, and in order to obtain the bird's-eye view map coordinates, the real space position of the target object is determined. Need to get.

In order to obtain the real space position of the target object, it is necessary to install it at a predetermined position as described above, perform manual positioning, or use a positioning device such as a GPS receiver. When a positioning device is used, the manufacturing cost of the target object increases. In some cases, it may not be possible to use positioning equipment indoors or underground.

On the other hand, setting the target object in a predetermined position or manually positioning the target object leads to an increase in the work cost related to calibration. Assuming that the camera arrangement is not changed, it is sufficient to perform calibration once, but it is also possible that the camera arrangement is changed or a new camera is added. For example, at a construction site, it is conceivable to change the arrangement of cameras when the construction range moves, or to add cameras when the construction range becomes wide. Also, in commercial facilities, it is conceivable to change the layout of cameras or add cameras depending on the floor conditions.

When the camera arrangement is changed or the camera is added in this way, it is necessary to calibrate each time the camera arrangement is changed or the camera is added. Therefore, setting the target object at a predetermined position or manually positioning the target object each time the camera is rearranged or added, greatly increases the work cost.

Therefore, the inventor of the present invention has come to create an embodiment of the present invention with the above circumstances as the first point of view. The information processing apparatus according to each embodiment of the present invention described below stores the bird's-eye view map coordinates of the target object obtained in the calibrated camera arrangement, and is used for calibration in the new camera arrangement. It is possible to suppress. Hereinafter, each embodiment of the present invention that realizes such an effect will be described in detail in sequence.

<< 2. Detailed explanation of each embodiment >>
<2-1. First Embodiment>
(overview)
First, the outline of the first embodiment of the present invention will be described. The information processing apparatus according to the present embodiment is a target object in a captured image captured with a camera arrangement in which conversion parameters are specified, that is, a camera arrangement that has already been calibrated (hereinafter, may be referred to as a first camera arrangement). Converts the captured image coordinates of the above to the bird's-eye view map coordinates. The information processing apparatus according to the present embodiment temporarily stores the bird's-eye view map coordinates obtained in this way, and for example, changes the camera arrangement or adds a camera. The information processing apparatus according to the present embodiment is a captured image of a target object in a captured image captured by a camera arrangement after changing the camera arrangement or after adding a camera (hereinafter, may be referred to as a second camera arrangement). Calibration is performed based on the correspondence between the coordinates and the stored bird's-eye view map coordinates.

By performing the calibration in this way, it is possible to suppress the cost related to the calibration in the second camera arrangement. Hereinafter, a configuration example of the information processing apparatus according to the present embodiment will be described with reference to FIG.

(Configuration example)
FIG. 3 is a block diagram showing a configuration example of the information processing apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 3, the information processing apparatus 1 according to the present embodiment includes a control unit 10, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90.

The control unit 10 controls each configuration of the information processing device 1. Further, as shown in FIG. 3, the control unit 10 according to the present embodiment also functions as an object position specifying unit 11, a calibration unit 13, and a coordinate conversion unit 15. The functions of the object position specifying unit 11, the calibration unit 13, and the coordinate conversion unit 15 of the control unit 10 will be described later.

The image acquisition unit 60 is an interface that is connected to a camera (not shown) and acquires an image captured by the camera. The image acquisition unit 60 may be connected to a plurality of cameras to acquire captured images from the plurality of cameras. The image acquisition unit 60 provides the acquired captured image to the control unit 10.

The information processing device 1 itself may include a camera. In such a case, the information processing apparatus 1 may include an image pickup unit including a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like in place of or in addition to the image acquisition unit 60.

The operation unit 70 receives an input by a user (operator) of the information processing device 1. For example, the operation unit 70 may be realized by a mouse, a keyboard, a touch panel, a button, a switch, a lever, a dial, or the like.

The output unit 80 is, for example, a display, and displays and outputs the position of an object on the bird's-eye view map according to the purpose. For example, if the object is stationary, the position of the object may be displayed and output on the bird's-eye view map, and if the object is moving, its trajectory may be displayed and output based on the bird's-eye view map coordinates of each frame. good.

The storage unit 90 stores programs and data used for the operation of the information processing device 1. Further, the storage unit 90 stores the conversion parameter DB 91 and the position coordinate DB 93 as shown in FIG.

The conversion parameter DB 91 stores conversion parameters for converting the captured image coordinates into the bird's-eye view map coordinates provided by the calibration unit 13 described later. Further, the position coordinate DB 93 stores the bird's-eye view map coordinates of the target object provided by the coordinate conversion unit 15 described later. The storage unit 90 according to the present embodiment is different from the storage unit 890 shown in FIG. 1 in that the storage unit 90 according to the present embodiment stores the position coordinate DB 93.

The overall configuration of the information processing apparatus 1 according to the present embodiment has been described above. Subsequently, the functions of the object position specifying unit 11, the calibration unit 13, and the coordinate conversion unit 15 of the control unit 10 included in the information processing device 1 will be described.

The object position specifying unit 11 specifies the captured image coordinates of an object (including a target object) reflected in the captured image provided by the image acquisition unit 60. The object position specifying unit 11 may specify the captured image coordinates of the object based on the user operation using the operation unit 870, or may specify the captured image coordinates by an image processing technique. For example, when specifying the captured image coordinates by image processing technology, the object position specifying unit 11 detects an object from the captured image by using a technique such as template matching, and the lowest position in the region of the object in the captured image. The coordinates may be specified as the captured image coordinates of the object.

Further, the object position specifying unit 11 may detect the captured image coordinates of the target object candidate from the captured image, and output (for example, display) the captured image coordinates of the detected target object candidate to the output unit 80. The user (operator) who has confirmed the output of the output unit 80 may determine the target object from the target object candidates via the operation unit 70. Then, the object position specifying unit 11 may specify the captured image coordinates of the determined target object candidate as the captured image coordinates of the target object based on the determination of the operator via the operation unit 70.

With such a configuration, the captured image coordinates of the target object can be specified with higher accuracy. If there is an error in specifying the captured image coordinates of the target object, the calibration accuracy may be significantly affected. Therefore, the calibration accuracy can be improved by specifying the captured image coordinates of the target object with high accuracy. ..

The calibration unit 13 specifies a conversion parameter for converting the captured image coordinates into the bird's-eye view map coordinates based on the correspondence between the captured image coordinates and the bird's-eye view map coordinates. The transformation parameter specified by the calibration unit 13 may be, for example, a projective transformation matrix. The calibration unit 13 can specify the projection transformation matrix, for example, in the same manner as the calibration unit 813 according to the comparative example shown in FIG.

When the calibration unit 13 according to the present embodiment calibrates in the camera arrangement when the camera is first installed, for example, the real space position of the target object or the bird's-eye view map coordinates are obtained by the user operation via the operation unit 70. It may be entered.

However, when the bird's-eye view map coordinates of the target object are stored in the position coordinate DB 93 of the storage unit 90, the calibration unit 13 according to the present embodiment has a bird's-eye view of the target object stored (stored) in the position coordinate DB 93. Calibrate using map coordinates. The bird's-eye view map coordinates of the target object stored in the position coordinate DB 93 are the bird's-eye view map coordinates obtained by the coordinate conversion performed by the coordinate conversion unit 15 described later.

The coordinate conversion unit 15 performs coordinate conversion for converting the captured image coordinates provided by the object position specifying unit 11 into the bird's-eye view map coordinates by using the conversion parameters. The coordinate conversion unit 15 performs such coordinate conversion using the conversion parameters stored in the conversion parameter DB 91 stored in the storage unit 90, that is, the conversion parameters obtained by the calibration of the calibration unit 13 described above. May be good. The bird's-eye view map coordinates of the object obtained by the coordinate conversion of the coordinate conversion unit 15 are output from, for example, the output unit 80.

Further, the coordinate conversion unit 15 according to the present embodiment uses the first conversion parameter corresponding to the first camera arrangement to capture the target object in the first captured image captured by the first camera arrangement. The image coordinates are converted into the bird's-eye view map coordinates of the target object in the bird's-eye view map. Further, the coordinate conversion unit 15 provides the bird's-eye view map coordinates obtained in this way to the storage unit 90 and stores them in the position coordinate DB 93.

With this configuration, the calibration unit 13 uses the bird's-eye view map coordinates of the target object stored in the position coordinate DB 93 as described above when performing calibration in the second camera arrangement different from the first camera arrangement. It becomes possible. That is, the calibration unit 13 has the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map coordinates of the target object in the bird's-eye view map. Based on the correspondence, the second conversion parameter corresponding to the second camera arrangement can be specified.

According to this configuration, when the calibration is performed in the second camera arrangement, it is not necessary to perform the positioning of the target object in the real space, so that the work cost is suppressed.

(Operation example)
The configuration example of the information processing apparatus 1 according to the present embodiment has been described above. Subsequently, an operation example of this embodiment will be described with reference to FIGS. 4 to 10.

FIG. 4 is a flowchart showing an outline of the operation according to the present embodiment. As shown in FIG. 4, first, calibration (hereinafter referred to as initial calibration) for obtaining conversion parameters in the camera arrangement when the camera is first installed is performed (S10). Details of the initial calibration in step S10 will be described later with reference to FIGS. 5 to 7.

Subsequently, when the camera arrangement is changed or added (YES in S12), the calibration for obtaining the conversion parameters in the camera arrangement after the camera arrangement is changed or after the camera is added (hereinafter referred to as recalibration) is performed. It is done (S14). The detailed operation of the recalibration in step S14 will be described later with reference to FIGS. 8 to 10.

On the other hand, when neither the camera arrangement is changed nor the addition is performed (NO in S12), the operation ends. Although not shown, between step S10 and step S12, as described above, a process of displaying and outputting the position of an object such as a person reflected in the captured image on the bird's-eye view map may be included. ..

The outline of the operation according to the present embodiment has been described above. Subsequently, with reference to FIGS. 5 to 7, the initial calibration of step S10 shown in FIG. 4 will be described in more detail. FIG. 5 is a flowchart showing the detailed operation of the initial calibration. 6 and 7 are explanatory views for explaining the initial calibration.

FIG. 6 shows the real space R1 and the captured image G1 captured by the camera 5A installed in the real space R1. Further, FIG. 7 shows a real space R1 and a bird's-eye view map M1 corresponding to the real space R1. In the bird's-eye view map M1, the area MA indicates an area corresponding to the imaging range of the camera 5A. Further, the real space R1 shown in FIG. 6 is a plan view seen from directly above, and the real space R1 shown in FIG. 7 is a side view seen from the side.

As shown in FIG. 5, first, the target objects T1 to T4 are installed in the real space R1 (S100). Subsequently, the real space positions of the target objects T1 to T4 are measured, and the bird's-eye view map coordinates F1 to F4 corresponding to the real space positions of the target objects T1 to T4 are input by user operation via, for example, the operation unit 70. (S102).

Subsequently, the image acquisition unit 60 acquires the captured image G1 captured by the camera 5A (S104). The target objects T1 to T4 may be removed after the captured image G1 is acquired. Subsequently, the object position specifying unit 11 identifies the captured image coordinates P1 to P4 of the target objects T1 to T4, respectively (S106).

Subsequently, the calibration unit 13 specifies the conversion parameter based on the correspondence between the captured image coordinates P1 to P4 and the bird's-eye view map coordinates F1 to F4 (S108). The conversion parameter specified in step S108 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S110).

The initial calibration in step S10 has been described above. Subsequently, the recalibration of step S14 shown in FIG. 4 will be described in more detail with reference to FIGS. 8 to 10. FIG. 8 is a flowchart showing a detailed operation of the recalibration according to the present embodiment. 9 and 10 are explanatory views for explaining the recalibration according to the present embodiment.

Note that FIGS. 9 and 10 show an example in which the recalibration is performed first after the initial calibration is performed, that is, an example in which the camera arrangement is changed or added first. ing. When the camera arrangement is changed, the camera 5A and the camera 5B shown in FIGS. 9 and 10 may be the same camera 5, the camera 5A indicates the camera 5 before the arrangement change, and the camera 5B , It may be understood that it indicates the camera 5 after the arrangement change. Further, when a camera is added, the camera 5A and the camera 5B are different cameras, and the camera 5B is added.

Since the calibration in the camera arrangement of the camera 5A has already been completed in step S10, the camera arrangement of the camera 5A is referred to as the above-mentioned first camera arrangement, and the camera arrangement of the camera 5B is referred to as the above-mentioned second camera arrangement. ..

In FIG. 9, the real space R10, the first captured image G11 captured by the camera 5A installed in the real space R10 in the first camera arrangement, and the second camera 5B installed in the real space R10 are shown. A second captured image G12 captured in the camera arrangement is shown. Further, FIG. 10 shows the real space R10, and the bird's-eye view map M11 and the bird's-eye view map M12 corresponding to the real space R10. In the bird's-eye view map M11, the area MA indicates the area corresponding to the image pickup range of the camera 5A, and the area MB in the bird's-eye view map M12 indicates the area corresponding to the image pickup range of the camera 5B. Further, the real space R10 shown in FIG. 9 is a plan view seen from directly above, and the real space R10 shown in FIG. 10 is a side view seen from the side. Further, the bird's-eye view map M11 and the bird's-eye view map M12 shown in FIG. 10 have the same coordinate system as the bird's-eye view map M1 shown in FIG. 7.

As shown in FIG. 8, first, the target objects T11 to T14 are installed in the real space R10 (S140). Subsequently, the image acquisition unit 60 acquires the first captured image G11 captured by the camera 5A in the first camera arrangement (S142). Subsequently, the object position specifying unit 11 identifies the captured image coordinates P111 to P114 of the target objects T11 to T14 in the first captured image G11 (S144).

Subsequently, the coordinate conversion unit 15 converts the captured image coordinates P111 to P114 into the bird's-eye view map coordinates F11 to F14, respectively, using the first conversion parameter stored (stored) in the conversion parameter DB 91 of the storage unit 90. (S146). The first conversion parameter used in step S146 is a conversion parameter corresponding to the camera arrangement (first camera arrangement) of the camera 5A, and is, for example, the conversion parameter stored in step S110 shown in FIG. You may.

The bird's-eye view map coordinates F11 to F14 obtained in step S146 are stored (stored) in the position coordinate DB 93 of the storage unit 90 (S147).

Subsequently, the arrangement of the cameras is changed or the cameras are added (S148). Subsequently, the image acquisition unit 60 acquires the second captured image G12 captured by the camera 5B in the second camera arrangement (S150). Subsequently, the object position specifying unit 11 identifies the captured image coordinates P121 to P124 of the target objects T11 to T14 in the second captured image G12 (S152).

Subsequently, the calibration unit 13 performs a second conversion based on the correspondence between the captured image coordinates P121 to P124 specified in step S152 and the bird's-eye view map coordinates F11 to F14 stored in the position coordinates DB93 in step S147. Specify the parameter (S154). The second conversion parameter specified in step S154 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S156).

The recalibration of step S14 has been described above. As described with reference to FIG. 4, the recalibration in step S14 is repeated until the cameras are not rearranged or the cameras are not added. Therefore, the camera arrangement (second camera arrangement) of the camera 5B described with reference to FIGS. 8 to 10 and the specified second conversion parameter will be the first when the recalibration is performed next time. It is treated as the camera arrangement and the first conversion parameter.

Further, as shown in FIG. 10, the above-mentioned target objects T11 to T14 are arranged so as to be included in the imaging range of both the camera 5A and the camera 5B. When the camera arrangement is changed or the camera is added after the target objects T11 to T14 are arranged as shown in FIG. 8, the camera 5B is included in the imaging range of the camera 5B. It suffices if the installation is done.

(supplement)
The first embodiment of the present invention has been described above. According to the present embodiment, the bird's-eye view map coordinates of the target object obtained in the calibrated first camera arrangement are used for the calibration in the second melody arrangement, so that the calibration in the second camera arrangement can be performed. It is possible to suppress such costs. Although the example in which the number of target objects is 4 has been described above, the number of target objects may differ depending on the conversion parameters and the number required for the coordinate conversion method. Further, the number of target objects may be larger than the number required for the conversion parameters and the coordinate conversion method, and such a configuration may enable more accurate calibration.

<2-2. Second embodiment>
(overview)
In the first embodiment described above, an example in which recalibration is repeatedly performed each time a camera is rearranged or a camera is added has been described. However, if the number of repetitions of recalibration increases, the error between the bird's-eye view map coordinates of the target object obtained by the coordinate conversion and the bird's-eye view map coordinates corresponding to the real space position of the target object may accumulate and become large. be.

Therefore, an example of performing correction using an object whose position in the real space is known, that is, an object whose bird's-eye view map coordinates are known (hereinafter referred to as a reference object) will be described as a second embodiment.

(Configuration example)
FIG. 11 is a block diagram showing a configuration example of the information processing apparatus 2 according to the second embodiment of the present invention. As shown in FIG. 11, the information processing apparatus 2 according to the present embodiment includes a control unit 20, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90. Of the components shown in FIG. 11, the components having substantially the same functional configuration as those shown in FIG. 3 are designated by the same reference numerals, and thus the description thereof will be omitted.

The control unit 20 controls each configuration of the information processing device 2. Further, as shown in FIG. 11, the control unit 20 according to the present embodiment also functions as an object position specifying unit 21, a calibration unit 23, a coordinate conversion unit 25, and a correction unit 27.

The object position specifying unit 21 specifies the captured image coordinates of the object reflected in the captured image provided by the image acquisition unit 60, similarly to the object position specifying unit 11 described with reference to FIG. However, the captured image coordinates specified by the object position specifying unit 21 are different from the object position specifying unit 11 in that the object position specifying unit 21 includes the captured image coordinates of the reference object. The object position specifying unit 21 may specify the captured image coordinates of the reference object by the same method as the method for specifying the captured image coordinates of the target object.

Similar to the calibration unit 13 described with reference to FIG. 3, the calibration unit 23 converts the captured image coordinates into the bird's-eye view map coordinates based on the correspondence between the captured image coordinates and the bird's-eye view map coordinates. Identify the parameters. However, in addition to the function of the calibration unit 13, the calibration unit 23 has a function of specifying conversion parameters as correction candidates by using the corrected bird's-eye view map coordinates of the target object provided by the correction unit 27 described later. Have.

Similar to the coordinate conversion unit 15 described with reference to FIG. 3, the coordinate conversion unit 25 performs coordinate conversion for converting the captured image coordinates provided by the object position specifying unit 21 into the bird's-eye view map coordinates by using the conversion parameters. Has the function to do. However, in addition to the function of the coordinate conversion unit 15, the coordinate conversion unit 25 has a function of converting the captured image coordinates of the object into the bird's-eye view map coordinates according to the control of the correction unit 27 described later. The coordinate conversion unit 25 according to the present embodiment converts the captured image coordinates of the reference object into the bird's-eye view map coordinates by using the conversion parameters stored in the conversion parameter DB 91 under the control of the correction unit 27 described later. Further, the coordinate conversion unit 25 according to the present embodiment converts the captured image coordinates of the reference object into the bird's-eye view map coordinates by using the conversion parameters as correction candidates according to the control of the correction unit 27 described later.

The correction unit 27 corrects the conversion parameters specified by the calibration unit 23 using a reference object whose bird's-eye view map coordinates are known. As shown in FIG. 11, the correction unit 27 includes a distance specifying unit 271, a determination unit 273, a coordinate correction unit 275, and a correction control unit 277.

The distance specifying unit 271 has known bird's-eye view map coordinates of the reference object (hereinafter referred to as known reference coordinates), and the coordinate conversion unit 25 converts the captured image coordinates of the reference object to obtain the bird's-eye view map coordinates of the reference object. Specify the distance to (hereinafter referred to as conversion reference coordinates). The distance specified by the distance specifying unit 271 may be, for example, the Euclidean distance between the known reference coordinates and the conversion reference coordinates. However, the distance specified by the distance specifying unit 271 is not limited to the Euclidean distance.

The determination unit 273 determines whether or not the distance specified by the distance specifying unit 271 is within a predetermined range.

The coordinate correction unit 275 is based on the difference between the known reference coordinates and the conversion reference coordinates, that is, the vector from the conversion reference coordinates to the known reference coordinates when the determination unit 273 determines that the distance is not within a predetermined range. Then, the bird's-eye view map coordinates of the target object are corrected. For example, the coordinate correction unit 275 targets the target object so that the vector from the uncorrected bird's-eye view map coordinates to the corrected bird's-eye view map coordinates is the same as or substantially the same as the vector from the conversion reference coordinates to the known reference coordinates. The bird's-eye view map coordinates of the object may be corrected. That is, the coordinate correction unit 275 may obtain the corrected bird's-eye view map coordinates of the target object by adding the difference between the known reference coordinates and the conversion reference coordinates to the bird's-eye view map coordinates before the correction of the target object.

The correction control unit 277 controls the operations of the distance specifying unit 271, the determination unit 273, and the coordinate correction unit 275 described above. Further, the correction control unit 277 controls a part of the operation of the calibration unit 23 and the coordinate conversion unit 25 in order to correct the conversion parameters.

For example, the correction control unit 277 controls the coordinate conversion unit 25 to convert the captured image coordinates of the reference object into the conversion reference coordinates using the conversion parameters stored in the conversion parameter DB 91. Further, the correction control unit 277 controls the distance specifying unit 271 and causes the distance to be specified by using the conversion reference coordinates obtained by the coordinate conversion unit 25.

Further, the correction control unit 277 controls the determination unit 273 to determine whether or not the distance specified by the distance specifying unit 271 is within a predetermined range. Then, the correction control unit 277 corrects the conversion parameter according to the determination result of the determination unit 273.

For example, the correction control unit 277 controls the coordinate correction unit 275 to correct the bird's-eye view map coordinates of the target object when the determination unit 273 determines that the distance is not within a predetermined range. Further, the bird's-eye view map coordinates of the target object corrected by the coordinate correction unit 275 are provided to the calibration unit 23, and the conversion parameters to be correction candidates are specified.

The correction control unit 277 repeats the above-mentioned series of processes until the determination unit 273 determines that the distance is within the predetermined range, and finally the correction candidate is such that the distance is within the predetermined range. The conversion parameter to be obtained may be stored in the position coordinate DB 93 of the storage unit 90 as the corrected conversion parameter.

In this repetition, when the coordinate conversion unit 25 converts the captured image coordinates of the reference object into the conversion reference coordinates in the second and subsequent processes, it is specified by the calibration unit 23 in the previous process as a conversion parameter. The conversion parameters that are candidates for correction are used.

Moreover, there may be a plurality of reference objects. As the number of reference objects increases, it may be possible to perform more accurate correction. However, if at least one reference object is present, the correction unit 27 can correct the conversion parameters described above. That is, the number of reference objects required for the correction unit 27 to correct the conversion parameter is smaller than the number of target objects (at least 4 or more) required for the calibration unit 23 to specify the second conversion parameter. good. Therefore, it is possible to perform the above-mentioned correction at a lower work cost than positioning the bird's-eye view map coordinates of all the target objects.

(Operation example)
The configuration example of the information processing apparatus 2 according to the present embodiment has been described above. Subsequently, an operation example of this embodiment will be described with reference to FIGS. 12 to 15.

FIG. 12 is a flowchart showing an outline of the operation according to the present embodiment. As shown in FIG. 12, first, initial calibration is performed to obtain conversion parameters in the camera arrangement when the camera is first installed (S20). Since the initial calibration in step S20 is the same as the initial calibration in the first embodiment described with reference to FIGS. 5 to 7, detailed description thereof will be omitted.

Subsequently, when the camera arrangement is changed or added (YES in S22), recalibration is performed to obtain the conversion parameters in the camera arrangement after the camera arrangement is changed or after the camera is added (S24). Since the recalibration in step S24 is the same as the recalibration in the first embodiment described with reference to FIGS. 8 to 10, detailed description thereof will be omitted.

Subsequently, in step S26, it is determined whether or not to perform the correction. The determination as to whether or not to perform the correction may be performed based on, for example, a user operation via the operation unit 70, or may be automatically performed based on a predetermined condition. For example, correction may be performed when the reference object is reflected in the captured image when the recalibration in step S24 is performed, or when the number of repetitions of the recalibration in step S24 reaches a predetermined number. Correction may be made.

If it is determined that the correction is not performed (NO in step S26), the process returns to step S22. On the other hand, when it is determined to perform correction (YES in step S26), the correction unit 27 corrects the second conversion parameter obtained by the recalibration in step S24 (S28). The detailed operation of step S28 will be described later with reference to FIGS. 13 to 15.

On the other hand, when neither the arrangement of the cameras is changed nor the addition is performed (NO in S22), the operation ends. Although not shown, a process of displaying and outputting the position of an object such as a person reflected in the captured image on the bird's-eye view map may be included between steps S20 and S22.

The outline of the operation according to the present embodiment has been described above. Subsequently, the correction of step S28 shown in FIG. 12 will be described in more detail with reference to FIGS. 13 to 15. FIG. 13 is a flowchart showing a detailed operation of correction of conversion parameters. 14 and 15 are explanatory diagrams for explaining the correction of conversion parameters.

In FIGS. 14 and 15, the camera arrangement of the camera 5A is the above-mentioned first camera arrangement, and the camera arrangement of the camera 5B is the above-mentioned second camera arrangement. Further, when the camera arrangement is changed in the recalibration in step S20, the camera 5A and the camera 5B shown in FIGS. 14 and 15 may be the same camera 5, and the camera 5A is arranged. It may be understood that the camera 5 before the change is shown and the camera 5B shows the camera 5 after the rearrangement change. Further, when a camera is added in the recalibration in step S20, the camera 5A and the camera 5B are different cameras, and the camera 5B is a camera to which the camera 5B is added.

FIG. 14 shows the real space R20, the captured image G21 captured by the camera 5A installed in the real space R20 in the first camera arrangement, and the second camera arrangement by the camera 5B installed in the real space R20. The captured image G22 captured is shown. Further, FIG. 15 shows the real space R20, and the bird's-eye view map M21 and the bird's-eye view map M22 corresponding to the real space R20. In the bird's-eye view map M21, the area MA indicates the area corresponding to the image pickup range of the camera 5A, and the area MB in the bird's-eye view map M22 indicates the area corresponding to the image pickup range of the camera 5B. Further, the real space R20 shown in FIG. 14 is a plan view seen from directly above, and the real space R20 shown in FIG. 15 is a side view seen from the side.

Further, the target objects T21 to T24 have already been arranged in the recalibration of step S24 before the operation of step S28 described below is performed. Further, in the recalibration in step S24, the captured image coordinates P211 to P214 of the target objects T21 to T24 in the captured image G21 and the captured image coordinates P221 to P224 of the target objects T21 to T24 in the captured image G22 have already been specified. .. Further, in the recalibration in step S24, the bird's-eye view map coordinates F21 to F24 of the target objects T21 to T24 have already been obtained by the conversion using the first conversion parameter.

As shown in FIG. 13, first, the known bird's-eye view map coordinates C25 (known reference coordinates) corresponding to the real space position of the reference object B20 are input by a user operation via, for example, the operation unit 70 (S282). The known reference coordinates may be stored in the storage unit 90 in advance.

Subsequently, the object position specifying unit 21 identifies the captured image coordinates P220 of the reference object B20 in the captured image G22 (S283).

Subsequently, the coordinate conversion unit 25 converts the captured image coordinates P220 of the reference object B20 into the bird's-eye view map coordinates F25 (conversion reference coordinates) using the conversion parameters (S284). When step S284 is performed first, the coordinate conversion unit 25 may use the second conversion parameter stored in the conversion parameter DB 91 of the storage unit 90 as the conversion parameter. Further, when step S284 is performed for the second time or later, the coordinate conversion unit 25 may use a conversion parameter as a correction candidate specified in step S292 described later as a conversion parameter.

Subsequently, the distance specifying unit 271 of the correction unit 27 specifies the distance between the input bird's-eye view map coordinates C25 (known reference coordinates) and the bird's-eye view map coordinates F25 (conversion reference coordinates) obtained by the conversion (S286). ..

Subsequently, the determination unit 273 of the correction unit 27 determines whether or not the distance specified in step S286 is within a predetermined range (S288).

When it is determined that the distance is not within the predetermined range (NO in S288), the coordinate correction unit 275 of the correction unit 27 corrects the bird's-eye view map coordinates F21 to F24 of the target objects T21 to T24 (S290). For example, in step S290, in the coordinate correction unit 275, the vector from the bird's-eye view map coordinates F21 to F24 before the correction to the bird's-eye view map coordinates C21 to C25 after the correction is changed from the bird's-eye view map coordinates F25 (conversion reference coordinates) to the bird's-eye view map coordinates C25 ( The bird's-eye view map coordinates of the target object may be corrected so as to be the same as or substantially the same as the vector to the known reference coordinates).

Subsequently, the calibration unit 23 sets the correction candidate based on the correspondence between the captured image coordinates P221 to P224 already obtained in step S24 and the corrected bird's-eye view map coordinates C21 to C25 obtained in step S290. (S292).

According to the control by the correction control unit 277 of the correction unit 27, the above-mentioned processes of steps S284 to S292 are repeated until it is determined in step S288 that the distance is within a predetermined range. Then, when it is determined that the distance is within a predetermined range (YES in S288), the correction control unit 277 corrects the conversion parameter which is the correction candidate specified in the last step S292 performed at that time. As the second conversion parameter, it is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S294).

The correction of the conversion parameter in step S28 has been described above. In the examples shown in FIGS. 14 and 15, the reference object B20 is included in the image pickup range of the camera 5A, but the reference object B20 may be included in the image pickup range of the camera 5B and is not necessarily included in the camera 5A. It does not have to be included in the imaging range of.

(supplement)
The second embodiment of the present invention has been described above. According to the present embodiment, even if an error occurs due to repeated recalibration, the work cost is suppressed by correcting the conversion parameters using a number of reference objects smaller than the number of target objects. At the same time, it is possible to specify the conversion parameters with high accuracy.

In the second embodiment described above, when the correction control unit 277 determines that the correction cannot be performed, the correction control unit 277 controls the output unit 80 to notify the user that the correction is not possible. You may let me. Then, when it is notified that the correction is impossible, the calibration may be performed by the same method as the initial calibration described with reference to FIGS. 5 to 7.

For example, even if the processing of steps S284 to S292 shown in FIG. 13 is repeated a predetermined number of times or more, the correction control unit 277 cannot perform correction when it is not determined in step S288 that the distance is within the predetermined range. It may be determined that it is possible. Alternatively, the correction control unit 277 corrects when the distance specified by the distance specifying unit 271 is larger than a predetermined range (a range larger than the predetermined range used in the determination in step S288 shown in FIG. 13). May be determined to be impossible.

With this configuration, even if an uncorrectable error occurs due to repeated recalibration, by notifying the user to that effect, calibration can be performed in the same way as the initial calibration with high accuracy. It is possible to specify various conversion parameters.

<2-3. Third Embodiment>
(overview)
In the first embodiment and the second embodiment described above, a common target object is used in the first camera arrangement and the second camera arrangement. That is, the number of target objects included in both the first captured image captured by the first camera arrangement and the second captured image captured by the second camera arrangement is required to specify the conversion parameter. If the number was less than the above, recalibration could not be performed. Therefore, for example, when there is no overlap between the image pickup range in the first camera arrangement and the image pickup range in the second camera arrangement, or when the overlap is very small, the first embodiment and the second embodiment It was not possible to perform such a recalibration.

Therefore, in the following, as the third embodiment, by using a position sensor such as LiDAR (Light Detection and Ranging), the first camera arrangement and the second camera arrangement do not use a common target object. An example of performing recalibration will be described.

(Configuration example)
FIG. 16 is a block diagram showing a configuration example of the information processing apparatus 3 according to the third embodiment of the present invention. As shown in FIG. 16, the information processing apparatus 3 according to the present embodiment includes a control unit 30, a sensor information acquisition unit 50, an image acquisition unit 60, an operation unit 70, an output unit 80, and a storage unit 90. Of the components shown in FIG. 16, the components having substantially the same functional configuration as those shown in FIG. 3 are designated by the same reference numerals, and thus the description thereof will be omitted.

The control unit 20 controls each configuration of the information processing device 3. Further, as shown in FIG. 16, the control unit 30 according to the present embodiment also functions as an object position specifying unit 31, a calibration unit 33, a coordinate conversion unit 35, and a sensor information management unit 39.

The object position specifying unit 31 specifies the captured image coordinates of the object reflected in the captured image provided by the image acquisition unit 60, similarly to the object position specifying unit 11 described with reference to FIG. However, in addition to the function of the object position specifying unit 11, the object position specifying unit 31 includes a first captured image captured by the first camera arrangement and a second captured image captured by the second camera arrangement. It has a function of specifying the number of target objects (hereinafter referred to as common target objects) included in both of them. For example, the number of common target objects may be specified by an image recognition method or by a user operation via the operation unit 70. The object position specifying unit 31 notifies the sensor information management unit 39 of the number of the specified common target objects.

Similar to the calibration unit 13 described with reference to FIG. 3, the calibration unit 33 converts the captured image coordinates into the bird's-eye view map coordinates based on the correspondence between the captured image coordinates and the bird's-eye view map coordinates. Identify the parameters.

However, the calibration unit 33 has a function of specifying conversion parameters according to the control of the sensor information management unit 39, which will be described later, in addition to the function of the calibration unit 13.

For example, the calibration unit 33 has a function of specifying conversion parameters for converting position coordinates (hereinafter referred to as sensor coordinates) in the sensor coordinate system corresponding to the position sensor into bird's-eye view map coordinates under the control of the sensor information management unit 39. Has. The sensor coordinate system corresponding to the position sensor is, for example, a coordinate system that differs only from the coordinate system of the bird's-eye view map in scale and the angle in the horizontal direction (Z-axis direction). Therefore, it is possible to specify the conversion parameter for converting the sensor coordinates to the bird's-eye view map coordinates by using the correspondence between the two or more sets of sensor coordinates and the bird's-eye view map coordinates. The calibration unit 33 has, for example, the bird's-eye view map coordinates in the bird's-eye view map obtained by converting the captured image coordinates of the target object (hereinafter referred to as the first target object) included in the first captured image, and the first The conversion parameter may be specified based on the correspondence with the sensor coordinates of the target object. The number of the first target objects is, for example, 2 or more from the above-mentioned conditions. Further, the calibration unit 33 may store (store) the conversion parameters for converting the sensor coordinates into the bird's-eye view map coordinates in the conversion parameter DB 91 of the storage unit 90.

Further, the calibration unit 33 controls the captured image coordinates of the target object (hereinafter referred to as the second target object) included in the second captured image and the second in the bird's-eye view map under the control of the sensor information management unit 39. The second conversion parameter is specified based on the correspondence with the bird's-eye view map coordinates of the target object. When the transformation parameter is a projective transformation matrix, the number of second target objects is 4 or more.

Similar to the coordinate conversion unit 15 described with reference to FIG. 3, the coordinate conversion unit 35 performs coordinate conversion for converting the captured image coordinates provided by the object position specifying unit 21 into the bird's-eye view map coordinates by using the conversion parameters. Has the function to do. However, in addition to the function of the coordinate conversion unit 15, the coordinate conversion unit 35 converts the sensor coordinates of the second target object included in the second captured image into the bird's-eye view map coordinates according to the control of the sensor information management unit 39 described later. It also has a function to provide to the sensor information management unit 39. The coordinate conversion unit 35 converts the sensor coordinates of the second target object into the bird's-eye view map coordinates specified by the calibration unit 33 and stored in the conversion parameter DB 91 when the sensor coordinates of the second target object are converted into the bird's-eye view map coordinates. Conversion parameters may be used.

The sensor information management unit 39 manages the sensor information acquired from the position sensor by the sensor information acquisition unit 50, which will be described later, and controls the execution of recalibration based on the sensor information. The sensor information includes, for example, the sensor coordinates of the target object. The sensor coordinates of the target object may be directly acquired from the position sensor by the sensor information acquisition unit 50, or may be specified based on the information acquired from the position sensor. For example, when the sensor information acquisition unit 50 includes only the position and shape information sensed by the sensor information acquired from the position sensor, the sensor information management unit 39 uses the sensing position and shape information to obtain the sensor coordinates of the target object. May be specified. The sensor information management unit 39 may have the output unit 80 output the sensor information and specify the sensor coordinates of the target object based on the user operation via the operation unit 70 by the user who has confirmed the sensor information.

The sensor information management unit 39 may execute recalibration based on the sensor information when the number of common target objects provided by the object position specifying unit 31 is less than the threshold value. The predetermined threshold value may be, for example, the number of pairs of the captured image coordinates required for the calibration unit 33 to specify the conversion parameter and the bird's-eye view map coordinates. For example, if the transformation parameter specified by the calibration unit 33 is a projective transformation matrix, the predetermined threshold value may be 4.

When the number of common target objects is less than the threshold value, the sensor information management unit 39 may control the calibration unit 33 and specify conversion parameters for converting the sensor coordinates to the bird's-eye view map coordinates. Further, the sensor information management unit 39 controls the coordinate conversion unit 35 to convert the sensor coordinates of the second target object included in the second captured image into the bird's-eye view map coordinates by using the conversion parameter, and the bird's-eye view map. The coordinates may be acquired from the coordinate conversion unit 35. Then, the sensor information management unit 39 controls the calibration unit 33, and sets the second conversion parameter based on the correspondence between the captured image coordinates of the second target object and the bird's-eye view map coordinates of the second target object. It may be specified.

With such a configuration, it is possible to suppress the work cost related to recalibration even when a sufficient number of common target objects do not exist.

The sensor information acquisition unit 50 is an interface that is connected to a position sensor (not shown) and acquires sensor information obtained by sensing the position sensor. The sensor information acquisition unit 50 may be connected to a plurality of position sensors to acquire sensor information from the plurality of position sensors. The sensor information acquisition unit 50 provides the acquired sensor information to the control unit 30.

The information processing device 3 itself may include a position sensor. In such a case, the information processing apparatus 3 may be provided with a position sensor such as LiDAR in place of or in addition to the sensor information acquisition unit 50.

(Operation example)
The configuration example of the information processing apparatus 3 according to the present embodiment has been described above. Subsequently, an operation example of this embodiment will be described. Since the operation outline of the present embodiment is substantially the same as the operation outline of the first embodiment described with reference to FIG. 4, the description of the operation outline of the present embodiment will be omitted. The difference between the operation of the present embodiment and the operation of the first embodiment is the difference in the operation related to the recalibration corresponding to step S14 shown in FIG. Therefore, in the following, the operation related to the recalibration according to the present embodiment will be described with reference to FIGS. 17 to 19.

FIG. 17 is a flowchart showing an example of the operation related to the recalibration according to the present embodiment. 18 and 19 are explanatory views for explaining the recalibration according to the present embodiment.

When the camera arrangement is changed, the camera 5A and the camera 5B shown in FIGS. 18 and 19 may be the same camera 5, and the camera 5A indicates the camera 5 before the arrangement change, and the camera 5B is used. However, it may be understood that the camera 5 is shown after the arrangement is changed. Further, when a camera is added, the camera 5A and the camera 5B are different cameras, and the camera 5B is added. That is, the camera arrangement of the camera 5A is the first camera arrangement, and the camera arrangement of the camera 5B is the second camera arrangement.

In FIG. 18, a second image is taken by the real space R30, the first captured image G31 captured by the camera 5A installed in the real space R30 in the first camera arrangement, and the camera 5B installed in the real space R30. The second captured image G32 captured in the camera arrangement is shown. Further, FIG. 19 shows the real space R30, and the bird's-eye view map M31 and the bird's-eye view map M32 corresponding to the real space R30. In the bird's-eye view map M31, the area MA indicates the area corresponding to the image pickup range of the camera 5A, and the area MB in the bird's-eye view map M32 indicates the area corresponding to the image pickup range of the camera 5B. Further, the real space R30 shown in FIG. 18 is a plan view seen from directly above, and the real space R30 shown in FIG. 19 is a side view seen from the side.

Further, as shown in FIG. 18, a position sensor 6 is installed in the real space R30. The position sensor 6 senses information of an object existing in the range D30 in the real space R30 by LiDAR or the like, and provides the information to the information processing apparatus 3.

As shown in FIG. 17, first, the target objects T31 to T36 are installed in the real space R30 (S302). Subsequently, the image acquisition unit 60 acquires the first captured image G31 captured by the camera 5A in the first camera arrangement (S304). Subsequently, the object position specifying unit 31 identifies the captured image coordinates P311 and P312 of the first target objects T31 and T32 in the first captured image G31 (S308).

Subsequently, the coordinate conversion unit 35 converts the captured image coordinates P311 and P312 into the bird's-eye view map coordinates F31 and F32, respectively, using the first conversion parameter stored (stored) in the conversion parameter DB 91 of the storage unit 90. (S312). The bird's-eye view map coordinates F31 and F32 obtained in step S312 are stored (stored) in the position coordinate DB 93 of the storage unit 90 (S313).

Subsequently, the arrangement of the cameras is changed or the cameras are added (S314). Subsequently, the image acquisition unit 60 acquires the second captured image G32 captured by the camera 5B in the second camera arrangement (S316). Subsequently, the object position specifying unit 31 identifies the captured image coordinates P323 to P326 of the second target objects T33 to T36 in the second captured image G32 (S318).

Subsequently, it is determined whether or not the number of common target objects included in both the first captured image G31 and the second captured image G32 is equal to or greater than a predetermined threshold value (S320). In the example shown in FIGS. 18 and 19, the number of common target objects is 0, which is less than a predetermined threshold value (4 in the above-mentioned example) (NO in S320), so the process proceeds to step S322.

In step S322, the sensor information acquisition unit 50 acquires sensor information from the position sensor 6 (S322). Subsequently, the calibration unit 33 follows the control of the sensor information management unit 39, and based on the correspondence between the sensor coordinates of the first target objects T31 and T32 included in the first captured image and the bird's-eye view map coordinates F31 and F32. The conversion parameter for converting the sensor coordinates to the bird's-eye view map coordinates is specified (S324). Further, the coordinate conversion unit 35 converts the sensor coordinates of the second target objects T33 to T36 included in the second captured image G32 into the bird's-eye view map coordinates F33 to F36, respectively, under the control of the sensor information management unit 39.

Subsequently, the calibration unit 33 uses the second conversion parameter based on the correspondence between the captured image coordinates P323 to P326 specified in step S318 and the bird's-eye view map coordinates F33 to F36 obtained by the coordinate conversion in step S326. (S332). The second conversion parameter specified in step S332 is stored (stored) in the conversion parameter DB 91 of the storage unit 90 (S334).

In the above description, the description has been given according to the examples shown in FIGS. 18 and 19, but if it is determined in step S320 that the number of common target objects is larger than the threshold value, the process proceeds to step S332. When it is determined in step S320 that the number of common target objects is larger than the threshold value, the series of processes shown in FIG. 17 is the first implementation described with reference to FIG. 8, except for the determination process in step S320. This is the same as the recalibration process related to the form.

(supplement)
The third embodiment of the present invention has been described above. According to the present embodiment, by using the position sensor, when there is not a sufficient number of common target objects included in the captured image before and after the camera arrangement is changed or between the existing camera and the added camera. Even if there is, it is possible to suppress the work cost related to the recalibration.

In the above, an example of performing recalibration based on sensor information when the number of common target objects is less than the threshold value has been described, but the present embodiment is not limited to such an example. For example, recalibration based on the sensor information described above may always be performed, or whether or not recalibration based on the sensor information may be performed may be switched by user operation via the operation unit 70.

It is also possible to combine the above-mentioned second embodiment and the third embodiment.

<< 3. Hardware configuration >>
Each embodiment of the present invention has been described above. Information processing such as the object position specifying process, the calibration process, and the coordinate conversion process described above is realized by the cooperation between the software and the hardware of the information processing device 1, the information processing device 2, and the information processing device 3. Hereinafter, the hardware configuration of the information processing apparatus 1000 will be described as an example of the hardware configuration of the information processing apparatus 1, the information processing apparatus 2, and the information processing apparatus 3 which are the information processing apparatus according to the embodiment of the present invention.

FIG. 20 is an explanatory diagram showing a hardware configuration of the information processing apparatus 1000 according to the embodiment of the present invention. As shown in FIG. 20, the information processing apparatus 1000 includes a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, a RAM (Random Access Memory) 1003, an input device 1004, and an output device 1005. , A storage device 1006 and a communication device 1007.

The CPU 1001 functions as an arithmetic processing device and a control device, and controls the overall operation in the information processing device 1000 according to various programs. Further, the CPU 1001 may be a microprocessor. The ROM 1002 stores programs, calculation parameters, and the like used by the CPU 1001. The RAM 1003 temporarily stores a program used in the execution of the CPU 1001 and parameters that are appropriately changed in the execution. These are connected to each other by a host bus composed of a CPU bus or the like. Mainly, the functions of the control unit 10, the control unit 20, the control unit 30, and the like are realized by the cooperation between the CPU 1001, the ROM 1002, the RAM 1003, and the software.

The input device 1004 includes an input means for the user to input information such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and an input control circuit that generates an input signal based on the input by the user and outputs the input signal to the CPU 1001. It is composed of such things. By operating the input device 1004, the user of the information processing device 1000 can input various data to the information processing device 1000 and instruct the processing operation. The input device 1004 corresponds to the operation unit 70.

The output device 1005 includes, for example, a liquid crystal display (LCD) device, an OLED device, and a display device such as a lamp. Further, the output device 1005 includes an audio output device such as a speaker and headphones. For example, the display device displays an captured image, a generated image, or the like. On the other hand, the voice output device converts voice data and the like into voice and outputs it. The output device 1005 corresponds to the output unit 80.

The storage device 1006 is a device for storing data. The storage device 1006 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like. The storage device 1006 stores programs and various data executed by the CPU 1001. The storage device 1006 corresponds to the storage unit 90.

The communication device 1007 is a communication interface composed of, for example, a communication device for connecting to a communication network. Further, the communication device 1007 may include a wireless LAN (Local Area Network) compatible communication device, an LTE (Long Term Evolution) compatible communication device, a wire communication device for wired communication, or a Bluetooth (registered trademark) communication device.

<< 4. Conclusion >>
As described above, according to the embodiment of the present invention, it is possible to suppress the cost related to the calibration associated with the change of the camera arrangement or the addition of the camera.

Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to these examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical ideas described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

For example, in the above embodiment, an example in which the transformation parameter for converting the captured image coordinates to the bird's-eye view map coordinates is a projective transformation matrix has been mainly described, but the present invention is not limited to such an example. For example, the conversion parameter for converting the captured image coordinates to the bird's-eye view map coordinates may include a perspective projection transformation matrix.

Further, each step in the above embodiment does not necessarily have to be processed in chronological order in the order described as a flowchart. For example, each step in the processing of the above embodiment may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

Further, according to the above embodiment, a computer for causing hardware such as the CPU 1001, ROM 1002, and RAM 1003 to exhibit the same functions as the configurations of the information processing device 1, the information processing device 2, and the information processing device 3 described above. Programs can also be provided. Also provided is a recording medium on which the computer program is recorded.

1, 2, 3 Information processing device 5 Camera 6 Position sensor 10, 20, 30 Control unit 11, 21, 31 Object position identification unit 13, 23, 33 Calibration unit 15, 25, 35 Coordinate conversion unit 27 Correction unit 39 Sensor Information management unit 50 Sensor information acquisition unit 60 Image acquisition unit 70 Operation unit 80 Output unit 90 Storage unit 271 Distance specification unit 273 Judgment unit 275 Coordinate correction unit 277 Correction control unit

Claims (12)

Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map corresponding to the real space, the captured image corresponds to the arrangement of the camera. A calibration unit that specifies conversion parameters for converting coordinates to the bird's-eye view map coordinates.
A coordinate conversion unit that converts the captured image coordinates to the bird's-eye view map coordinates using the conversion parameters, and
A target object identification unit that specifies the coordinates of the captured image of the target object is provided.
The coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to obtain the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the bird's-eye view map,
The calibration unit has the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map of the target object in the bird's-eye view map. Based on the correspondence with the coordinates, the second conversion parameter corresponding to the second camera arrangement is specified .
The target object identification unit detects the captured image coordinates of the target object candidate from the captured image, and captures the captured image coordinates of the detected target object candidate based on the determination of the operator. An information processing device that specifies as image coordinates . Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map corresponding to the real space, the captured image corresponds to the arrangement of the camera. A calibration unit that specifies conversion parameters for converting coordinates to the bird's-eye view map coordinates.
A coordinate conversion unit that converts the captured image coordinates to the bird's-eye view map coordinates using the conversion parameters, and
A correction unit for correcting the conversion parameter specified by the calibration unit using a reference object whose bird's-eye view map coordinates are known is provided.
The coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to obtain a bird's-eye view of the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the map,
The calibration unit has the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map of the target object in the bird's-eye view map. An information processing apparatus that specifies a second conversion parameter corresponding to the second camera arrangement based on the correspondence with the coordinates. The correction unit has known reference coordinates that are known bird's-eye view map coordinates of the reference object, and the bird's-eye view map coordinates of the reference object obtained by the coordinate conversion unit converting the captured image coordinates of the reference object. The information processing apparatus according to claim 2 , further comprising a distance specifying unit for specifying a distance between the conversion reference coordinates and the conversion reference coordinates, and correcting the conversion parameters based on the distance. 3. The correction unit further includes a determination unit that determines whether or not the distance is within a predetermined range, and corrects the conversion parameter according to the result of the determination by the determination unit. The information processing device described in. When the determination unit determines that the distance is not within the predetermined range, the correction unit obtains the bird's-eye view map coordinates of the target object based on the difference between the known reference coordinates and the conversion reference coordinates. The information processing apparatus according to claim 4 , further comprising a coordinate correction unit for correction, and correcting the conversion parameters based on the bird's-eye view map coordinates of the target object corrected by the coordinate correction unit. 3. The number of reference objects required for the correction unit to correct the conversion parameter is smaller than the number of target objects required for the calibration unit to specify the second conversion parameter. The information processing apparatus according to any one of 5 to 5 . Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map corresponding to the real space, the captured image corresponds to the arrangement of the camera. A calibration unit that specifies conversion parameters for converting coordinates to the bird's-eye view map coordinates.
Sensor information management department,
A coordinate conversion unit that converts the captured image coordinates into the bird's-eye view map coordinates using the conversion parameters is provided.
The coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to obtain a bird's-eye view of the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the map,
The calibration unit has the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map of the target object in the bird's-eye view map. Based on the correspondence with the coordinates, the second conversion parameter corresponding to the second camera arrangement is specified.
The sensor information management unit is the first in the first captured image when the number of the target objects included in both the first captured image and the second captured image is less than a predetermined threshold. Sensor coordinate system corresponding to the position sensor obtained based on the bird's-eye view map coordinates of the first target object in the bird's-eye view map obtained by converting the captured image coordinates of the target object and the sensing of the position sensor. The second target object in the bird's-eye view map based on the position coordinates of the first target object in the above and the position coordinates of the second target object in the sensor coordinate system obtained based on the sensing of the position sensor. Obtain the above-mentioned bird's-eye view map coordinates of
The calibration unit is based on the correspondence between the captured image coordinates of the second target object in the second captured image and the bird's-eye view map coordinates of the second target object in the bird's-eye view map. An information processing device that specifies conversion parameters. Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map corresponding to the real space, the captured image corresponds to the arrangement of the camera. A calibration unit that specifies conversion parameters for converting coordinates to the bird's-eye view map coordinates, and
A coordinate conversion unit that converts the captured image coordinates into the bird's-eye view map coordinates using the conversion parameters is provided.
The coordinate conversion unit uses the first conversion parameter corresponding to the first camera arrangement to obtain a bird's-eye view of the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the map,
The calibration unit has the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement, and the bird's-eye view map of the target object in the bird's-eye view map. Based on the correspondence with the coordinates, the second conversion parameter corresponding to the second camera arrangement is specified.
The transformation parameter is an information processing apparatus including a projection transformation matrix or a perspective projection transformation matrix. Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map, the captured image coordinates correspond to the camera arrangement, and the captured image coordinates are used as the bird's-eye view map coordinates. Calibration unit , which specifies the conversion parameters for conversion to
A coordinate conversion unit that converts the captured image coordinates to the bird's-eye view map coordinates using the conversion parameters, and
An information processing method executed by an information processing apparatus including a target object identification unit that specifies the captured image coordinates of the target object .
Using the first conversion parameter corresponding to the first camera arrangement, the captured image coordinates of the target object in the first captured image captured by the first camera arrangement are set to the target object in the bird's-eye view map. To convert to the above-mentioned bird's-eye view map coordinates,
Based on the correspondence between the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map coordinates of the target object in the bird's-eye view map. Then, to specify the second conversion parameter corresponding to the second camera arrangement,
The captured image coordinates of the target object candidate are detected from the captured image, and the detected captured image coordinates of the target object candidate are specified as the captured image coordinates of the target object based on the determination of the operator. , Including information processing methods. Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map, the captured image coordinates correspond to the camera arrangement, and the captured image coordinates are used as the bird's-eye view map coordinates. Calibration function , which specifies conversion parameters for conversion to
A coordinate conversion function that converts the captured image coordinates to the bird's-eye view map coordinates using the conversion parameters, and
It is a program for realizing a computer with a target object identification function for specifying the captured image coordinates of the target object .
The coordinate conversion function uses the first conversion parameter corresponding to the first camera arrangement to obtain the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the bird's-eye view map,
The calibration function includes the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map of the target object in the bird's-eye view map. Based on the correspondence with the coordinates, the second conversion parameter corresponding to the second camera arrangement is specified .
The target object identification function detects the captured image coordinates of the target object candidate from the captured image, and captures the captured image coordinates of the detected target object candidate based on the determination of the operator. A program that identifies as image coordinates . Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map, the captured image coordinates correspond to the camera arrangement, and the captured image coordinates are used as the bird's-eye view map coordinates. Calibration function, which specifies conversion parameters for conversion to
A coordinate conversion function that converts the captured image coordinates to the bird's-eye view map coordinates using the conversion parameters, and
It is a program for realizing a correction function for correcting the conversion parameter specified by the calibration function by using a reference object whose bird's-eye view map coordinates are known.
The coordinate conversion function uses the first conversion parameter corresponding to the first camera arrangement to obtain a bird's-eye view of the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the map,
The calibration function includes the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map of the target object in the bird's-eye view map. A program that specifies a second conversion parameter corresponding to the second camera arrangement based on the correspondence with the coordinates. Based on the correspondence between the captured image coordinates, which are the position coordinates in the captured image captured by the camera, and the bird's-eye view map coordinates, which are the position coordinates in the bird's-eye view map, the captured image coordinates correspond to the arrangement of the camera, and the captured image coordinates are used as the bird's-eye view map coordinates. Calibration function to specify conversion parameters for conversion to, and
It is a program for realizing a coordinate conversion function for converting the captured image coordinates into the bird's-eye view map coordinates by using the conversion parameters in a computer.
The coordinate conversion function uses the first conversion parameter corresponding to the first camera arrangement to obtain a bird's-eye view of the captured image coordinates of the target object in the first captured image captured by the first camera arrangement. Converted to the bird's-eye view map coordinates of the target object in the map,
The calibration function includes the captured image coordinates of the target object in the second captured image captured by the second camera arrangement different from the first camera arrangement and the bird's-eye view map of the target object in the bird's-eye view map. Based on the correspondence with the coordinates, the second conversion parameter corresponding to the second camera arrangement is specified.
The transformation parameter is a program including a projective transformation matrix or a perspective projection transformation matrix.

Images