WO2021049490A1 - Image registration device, image generation system, image registration method and image registration program - Google Patents

Abstract

This image registration device is connected communicatively to a distance measurement sensor (10) which generates a reflected light image, including distance information, by a light receiving element detecting light reflected by an object irradiated with light, and which generates a background light image, with the same coordinate system as the reflected light image, by said light receiving element detecting background light corresponding to the reflected light, and the image registration device is connected communicatively to a camera (20) which generates a camera image by a camera element detecting incident external light. This image registration device is provided with an image acquisition unit (41) which acquires a reflected light image, a background light image and a camera image, and an image processing unit (42) which, by specifying the relation between feature points in the background light image and feature points in the camera image, performs image registration of the reflected light image, which has the same coordinate system as the background light image, and the camera image.

Description

Image registration device, image generation system, image registration method and image registration program Cross-reference of related applications

This application is based on Patent Application No. 2019-164860 filed in Japan on September 10, 2019, and the contents of the basic application are incorporated by reference as a whole.

The disclosure according to this specification relates to an image registration device, an image generation system, an image registration method, and an image registration program.

Patent Document 1 discloses a distance measuring sensor. This distance measuring sensor can generate a reflected light image including distance information by detecting the reflected light reflected from an object by light irradiation by a light receiving element. Patent Document 2 discloses a camera. The camera can generate a high-resolution camera image by detecting the incident light from the outside by the camera element.

JP-A-2019-95452 JP-A-2018-69878

The reflected light image and the camera image can be processed by the application. However, the reflected light image and the camera image may have a deviation Δt in the detection timing. When the reflected light image and the object reflected in the camera image move during the deviation Δt, it becomes difficult to accurately associate and process the object reflected in the reflected light image and the object reflected in the camera image. .. Therefore, even if the application uses both the reflected light image and the camera image, the information of these images cannot be fully utilized, and therefore the processing accuracy cannot be sufficiently improved.

One of the purposes of the disclosure of this specification is to provide an image registration device, an image generation system, an image registration method, and an image registration program that enhance the processing accuracy of the application.

One of the embodiments disclosed here is that the light receiving element senses the reflected light image including the distance information by detecting the reflected light reflected from the object by the light irradiation, and the light receiving element senses the background light with respect to the reflected light. It is communicably connected to a ranging sensor that generates a reflected light image and a background light image that has the same coordinate system as the reflected light image, and the reflected light image and background are detected by the camera element by detecting incident light from the outside. An image registration device communicatively connected to a camera that produces a camera image with a higher resolution than an optical image.
An image acquisition unit that acquires a reflected light image, a background light image, and a camera image,
By specifying the correspondence between the feature points of the background light image and the feature points of the camera image, an image processing unit that performs image registration between the reflected light image having the same coordinate system as the background light image and the camera image, To be equipped.

According to such an aspect, image registration between the acquired reflected light image and the camera image is performed using the background light image having the same coordinate system as the reflected light image. That is, by comparing the feature points of the background light image whose properties are closer to those of the camera image than those of the reflected light image with the feature points of the camera image, it becomes easy to identify the correspondence between the feature points. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image can be remarkably improved.

In addition, another one of the disclosed aspects is an image generation system that generates an image to be processed by an application.
When the light receiving element senses the reflected light reflected from the object by light irradiation, the light receiving element senses the reflected light image including the distance information, and when the light receiving element senses the background light for the reflected light, the coordinate system becomes the same as the reflected light image. A distance measuring sensor that generates a background light image and
A camera that generates a camera image with a higher resolution than the reflected light image and the background light image by detecting the incident light from the outside by the camera element.
By specifying the correspondence between the feature points of the background light image and the feature points of the camera image, image registration between the reflected light image of the same coordinate system as the background light image and the camera image is performed to obtain the distance information. It includes an image processing unit that generates a composite image in which information of a camera image is integrated.

According to such an aspect, image registration between the reflected light image and the camera image is performed using the background light image having the same coordinate system as the reflected light image. That is, by comparing the feature points of the background light image whose properties are closer to those of the camera image than the reflected light image and the feature points of the camera image, it is easy to identify the correspondence between the feature points. With such an association, the coordinate system of the reflected light image and the coordinate system of the camera image can be accurately matched. Then, the distance information and the camera image information, which are information from different image generation sources of the distance measuring sensor and the camera, can be provided in the form of a composite image that can be easily processed by the application. Therefore, the processing accuracy of the application using both the reflected light image and the camera image can be remarkably improved.

Further, as another one of the disclosed aspects, the image registration method is an image generated by a distance measuring sensor, and distance information is obtained by detecting the reflected light reflected from an object by light irradiation by a light receiving element. To prepare a reflected light image including the reflected light image and a background light image having the same coordinate system as the reflected light image by the light receiving element detecting the background light with respect to the reflected light.
It is an image generated by the camera, and a camera image having a higher resolution than the reflected light image and the background light image is prepared by detecting the incident light from the outside by the camera element.
Detecting the feature points of the background light image and the feature points of the camera image, respectively,
Identifying the correspondence between the detected feature points of the background light image and the feature points of the camera image,
It includes making each pixel of one of the background light image and the camera image correspond to each pixel of the other based on the specific result of the correspondence relationship.

According to such an aspect, the feature points of the prepared background light image and the feature points of the camera image are detected, respectively. After that, the correspondence between the detected feature points of the background light image and the feature points of the camera image is specified. After that, each pixel of the background light image and the camera image is made to correspond to each pixel of the other based on the specific result of the correspondence relationship. In this way, the image registration between the reflected light image and the camera image is performed using the background light image which has the same coordinate system as the reflected light image and whose properties are closer to those of the camera image than the reflected light image. It becomes easy to identify the correspondence between. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image can be remarkably improved. Then, after specifying the correspondence relationship between the feature points, the coordinates of each pixel are made to correspond using the result, so that the processing amount can be suppressed or compared with the case where the correspondence relationship of each pixel is specified unnecessarily. Image registration with high accuracy can be performed while improving the processing speed.

Further, another one of the disclosed aspects is an image registration program that performs image registration between the image generated by the distance measuring sensor and the image generated by the camera.
In at least one processing unit
An image generated by a distance measuring sensor, in which the light receiving element senses a reflected light image including distance information by detecting the reflected light reflected from an object by light irradiation, and a background light with respect to the reflected light. The process of acquiring the reflected light image and the background light image that has the same coordinate system as the
A process of acquiring a camera image having a higher resolution than the reflected light image and the background light image by detecting the incident light from the outside in the image generated by the camera.
The process of detecting the feature points of the background light image and the feature points of the camera image, respectively.
Processing to identify the correspondence between the detected feature points of the background light image and the feature points of the camera image,
Based on the specific result of the correspondence relationship, the process of making each pixel of one of the background light image and the camera image correspond to each pixel of the other is executed.

According to such an aspect, the feature points of the acquired background light image and the feature points of the camera image are detected, respectively. After that, the correspondence between the detected feature points of the background light image and the feature points of the camera image is specified. After that, each pixel of the background light image and the camera image is made to correspond to each pixel of the other based on the specific result of the correspondence relationship. In this way, the image registration between the reflected light image and the camera image is performed using the background light image which has the same coordinate system as the reflected light image and whose properties are closer to those of the camera image than the reflected light image. It becomes easy to identify the correspondence between. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image can be remarkably improved. Then, after specifying the correspondence relationship between the feature points, the coordinates of each pixel are made to correspond using the result, so that the processing amount can be suppressed or compared with the case where the correspondence relationship of each pixel is specified unnecessarily. Image registration with high accuracy can be performed while improving the processing speed.

Note that the reference numerals in parentheses in the claims, etc., exemplify the correspondence with the parts of the embodiments described later, and are not intended to limit the technical scope.

It is a figure which shows the whole image of the image generation system and the operation support ECU of 1st Embodiment. It is a figure which shows the mounting state of the distance measuring sensor and the outside world camera of 1st Embodiment in a vehicle. It is a block diagram which shows the structure of the image processing ECU of 1st Embodiment. It is a figure for demonstrating the detection of the feature point in the background light image of 1st Embodiment. It is the figure which changed the background light image included in FIG. 4A into a line diagram. It is a figure for demonstrating the detection of the feature point in the camera image of 1st Embodiment. It is the figure which changed the camera image included in FIG. 5A into a diagram. It is a figure for demonstrating the identification of the correspondence relation of the feature point of 1st Embodiment. It is a figure which changed the included camera image and background light image shown in FIG. 6A into a line diagram. It is a figure for demonstrating the matching of coordinates of 1st Embodiment. It is a flowchart for demonstrating the processing of the image processing ECU of 1st Embodiment. It is a figure corresponding to FIG. 3 in the second embodiment.

Hereinafter, a plurality of embodiments will be described based on the drawings. In addition, duplicate description may be omitted by assigning the same reference numerals to the corresponding components in each embodiment. When only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other parts of the configuration. Further, not only the combination of the configurations specified in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined even if the combination is not specified. ..

(First Embodiment)
As shown in FIG. 1, the image registration device according to the first embodiment of the present disclosure is used for a vehicle 1 as a moving body, and is an image processing ECU (Electronic Control Unit) configured to be mounted on the vehicle 1. It is 30. The image processing ECU 30 constitutes an image generation system 100 together with a distance measuring sensor 10 and an external camera 20. The image generation system 100 of the present embodiment can generate peripheral monitoring image information that integrates the measurement results of the distance measuring sensor 10 and the external camera 20 and provide the peripheral monitoring image information to the driving support ECU 50 and the like.

The image processing ECU 30 is communicably connected to the communication bus of the vehicle-mounted network mounted on the vehicle 1. The image processing ECU 30 is one of a plurality of nodes provided in the vehicle network. In addition to the distance measuring sensor 10 and the external camera 20, the driving support ECU 50 and the like are connected to the communication bus of the vehicle-mounted network as nodes.

The operation support ECU 50 has a configuration mainly including a computer equipped with a processor, a RAM (RandomAccessMemory), a storage unit, an input / output interface, a bus connecting these, and the like. The driving support ECU 50 has at least one of a driving support function that assists the driver's driving operation in the vehicle 1 and a driving agency function that can act for the driver's driving operation. The driving support ECU 50 recognizes the surrounding environment of the vehicle 1 based on the peripheral monitoring image information acquired from the image generation system 100 by executing the program stored in the storage unit by the processor. The driving support ECU 50 realizes automatic driving or advanced driving support of the vehicle 1 according to the recognition result by executing the program stored in the storage unit by the processor.

Next, details of the distance measuring sensor 10, the external camera 20, and the image processing ECU 30 included in the image generation system 100 will be described in order.

The distance measuring sensor 10 is, for example, a SPAD LiDAR (Single Photon Avalanche Diode Light Detection And Ringing) arranged in front of the vehicle 1 or on the roof of the vehicle 1. The distance measuring sensor 10 can measure at least the front measurement range MA1 in the periphery of the vehicle 1.

The distance measuring sensor 10 has a configuration including a light emitting unit 11, a light receiving unit 12, a control unit 13, and the like. The light emitting unit 11 irradiates the light beam emitted from the light source toward the measurement range MA1 shown in FIG. 2 by scanning with a movable optical member (for example, a polygon mirror). The light source is, for example, a semiconductor laser (Laser diode), and emits a light beam in the near infrared region, which is invisible to the occupants and humans in the outside world, in response to an electric signal from the control unit 13.

The light receiving unit 12 collects the reflected light reflected from the object within the measurement range MA1 or the background light for the reflected light by, for example, a condensing lens, and causes the irradiated light beam to enter the light receiving element 12a.

The light receiving element 12a is an element that converts light into an electric signal by photoelectric conversion, and is a SPAD light receiving element that realizes high sensitivity by amplifying a detection voltage. For the light receiving element 12a, for example, in order to detect the reflected light in the near infrared region, a CMOS sensor in which the sensitivity in the near infrared region is set to be high with respect to the visible region is adopted. This sensitivity can also be adjusted by providing an optical filter in the light receiving unit 12. The light receiving element 12a has a plurality of light receiving pixels in an array so as to be arranged in a one-dimensional direction or a two-dimensional direction.

The control unit 13 is a unit that controls the light emitting unit 11 and the light receiving unit 12. The control unit 13 is arranged on a substrate common to, for example, the light receiving element 12a, and is mainly composed of a processor in a broad sense such as a microcomputer or an FPGA (Field-Programmable Gate Array). The control unit 13 realizes a scanning control function, a reflected light measurement function, and a background light measurement function.

The scanning control function is a function that controls optical beam scanning. The control unit 13 oscillates the light beam from the light source a plurality of times in a pulse shape at a timing based on the operating clock of the clock oscillator provided in the distance measuring sensor 10, and operates the movable optical member.

The reflected light measurement function is a function that reads out the voltage value based on the reflected light received by each light receiving pixel by using, for example, a rolling shutter method, and measures the intensity of the reflected light according to the timing of light beam scanning. In the measurement of the reflected light, the distance from the distance measuring sensor 10 to the object reflecting the reflected light can be measured by detecting the time difference between the emission timing of the light beam and the reception timing of the reflected light. By measuring the reflected light, the control unit 13 is image-like data in which the intensity of the reflected light and the distance information of the object reflecting the reflected light are associated with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1. A reflected light image can be generated.

The background light measurement function is a function that reads out the voltage value based on the background light received by each light receiving pixel at the timing immediately before measuring the reflected light and measures the intensity of the background light. Here, the background light means incident light incident on the light receiving element 12a from the measurement range MA1 in the outside world, which does not substantially include reflected light. The incident light includes natural light, display light incident from the display of the outside world, and the like. By measuring the background light, the control unit 13 can generate a background light image ImL which is image-like data in which the intensity of the background light is associated with the two-dimensional coordinates on the image plane corresponding to the measurement range MA1. ..

The reflected light image and the background light image ImL are sensed by the common light receiving element 12a and acquired from the common optical system including the light receiving element 12a. Therefore, the coordinate system of the reflected light image and the coordinate system of the background light image ImL can be regarded as the same coordinate system that coincide with each other. Furthermore, it can be said that there is almost no difference in measurement timing between the reflected light image and the background light image ImL (for example, less than 1 ns). Therefore, the reflected light image and the background light image ImL can be regarded as being synchronized.

For example, in the present embodiment, the integrated image data in which the data of three channels of the intensity of the reflected light, the distance of the object, and the intensity of the background light are stored corresponding to each pixel is used as the sensor image in the image processing ECU 30. Is output sequentially to.

The outside world camera 20 is, for example, a camera arranged on the vehicle interior side of the front windshield of the vehicle 1. The outside world camera 20 can measure at least the front measurement range MA2 of the outside world of the vehicle 1, and more specifically, the measurement range MA2 that overlaps at least a part of the measurement range MA1 of the distance measurement sensor 10.

The external camera 20 has a configuration including a light receiving unit 22 and a control unit 23. The light receiving unit 22 collects the incident light (background light) incident from the measurement range MA2 outside the camera by, for example, a light receiving lens, and causes the light receiving unit 22 to enter the camera element 22a.

The camera element 22a is an element that converts light into an electric signal by photoelectric conversion, and for example, a CCD sensor or a CMOS sensor can be adopted. The camera element 22a is set to have high sensitivity in the visible region with respect to the near infrared region in order to efficiently receive natural light in the visible region. The camera element 22a has a plurality of light receiving pixels (corresponding to so-called sub-pixels) in an array so as to be arranged in a two-dimensional direction. For example, red, green, and blue color filters are arranged on the light receiving pixels adjacent to each other. Each light receiving pixel receives visible light of a color corresponding to the arranged color filter. By measuring the intensity of red, the intensity of green, and the intensity of blue, the camera image ImC captured by the external camera 20 is a higher resolution image than the reflected light image and the background light image ImL, and is visible. It can be a color image of the area.

The control unit 23 is a unit that controls the light receiving unit 22. The control unit 23 is arranged on a substrate common to, for example, the camera element 22a, and is mainly composed of a processor in a broad sense such as a microcomputer or an FPGA. The control unit 23 realizes a shooting function.

The shooting function is a function for shooting the above-mentioned color image. The control unit 23 reads out the voltage value based on the incident light received by each light receiving pixel at the timing based on the operating clock of the clock oscillator provided in the external camera 20, for example, by using the global shutter method, and the intensity of the incident light. Is detected and measured. This clock oscillator is provided independently of the clock oscillator of the ranging sensor 10. The control unit 23 can generate a camera image ImC which is image-like data in which the intensity of the incident light is associated with the two-dimensional coordinates on the image plane corresponding to the measurement range MA2. Such camera image ImC is sequentially output to the image processing ECU 30.

The distance measuring sensor 10 and the external camera 20 operate based on different clock oscillators, and the measurement timing cycles (that is, frame rates) are not always the same, and are often different. Therefore, a deviation Δt in measurement timing may occur between the reflected light image and the background light image ImL and the camera image ImC. Δt can be 1000 times or more the deviation of the measurement timing between the reflected light image and the background light image ImL.

The image processing ECU 30 is an electronic control device that performs combined image processing of a reflected light image, a background light image ImL, and a camera image ImC. The image processing ECU 30 has a configuration mainly including a computer including a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, a bus connecting them, and the like. The processing unit 31 is hardware for arithmetic processing combined with the RAM 32. The processing unit 31 is configured to include at least one arithmetic core such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), and RISC (Reduced Instruction Set Computer). The processing unit 31 may be configured to further include an FPGA and an IP core having other dedicated functions. The RAM 32 may be configured to include a video RAM for image generation. The processing unit 31 executes various processes for realizing the functions of each functional unit, which will be described later, by accessing the RAM 32. The storage unit 33 is configured to include a non-volatile storage medium. The storage unit 33 stores various programs (image registration program, etc.) executed by the processing unit 31.

The image processing ECU 30 has a plurality of functional units for performing image registration by executing the image registration program stored in the storage unit 33 by the processing unit 31. Specifically, as shown in FIG. 3, functional units such as an image acquisition unit 41 and an image processing unit 42 are constructed in the image processing ECU 30.

The image acquisition unit 41 acquires the reflected light image and the background light image ImL from the distance measuring sensor 10 and also acquires the camera image ImC from the outside world camera 20. The image acquisition unit 41 sequentially provides the image processing unit 42 with the latest set of the reflected light image and the background light image ImC and the latest camera image ImC.

By specifying the correspondence between the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC, the image processing unit 42 sets the reflected light image having the same coordinate system as the background light image ImC and the camera image ImC. Perform image registration with. In particular, the image processing unit 42 of the present embodiment is a sensor image in which data of three channels of reflected light intensity, object distance, and background light intensity are stored, and a high-resolution image and a color image in the visible range. When the camera image ImC is input, a composite image in which data of four or more channels of reflected light intensity, object distance, background light intensity, and color information are stored is output. In the present embodiment, since the color information is composed of three channels of data of red intensity, green intensity, and blue intensity, the composite image is an image in which six channels of data are stored.

The image processing unit 42 has a feature point detection function, a correspondence relationship identification function, and a coordinate matching function. In image registration, the feature point detection function realizes the processing of the first phase, the correspondence identification function realizes the processing of the second phase after the first phase, and the coordinate matching function realizes the processing of the second phase. The processing of the third phase of is realized.

The feature point detection function is a function for detecting the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC, respectively. For example, corners can be adopted for the feature points FPa and FPb. For the detection of feature points FPa and FPb, various feature point detection methods using a feature point detector can be adopted. In particular, in this embodiment, a Harris corner detection method using Harris Corner Detectors 43a and 43b is adopted.

The Harris corner detectors 43a and 43b have the structure when the weighted sum of squares of the difference in intensity due to the movement of the pixel position in the evaluation target region is expressed by a structural tensor by approximation using Taylor expansion. The feature points FPa and FPb are detected by the eigenvalue of the tensor. In the Harris corner detectors 43a and 43b, whether the evaluation target region is a corner (corresponding to the feature points FPa and FPb), an edge, or a flat region is evaluated by the determinant of the structural tensor and the evaluation of the eigensum. Can be determined.

As shown in FIGS. 4A and 4B, the Harris corner detector 43a detects a plurality of feature points FPa of the background light image ImL. As shown in FIGS. 5A and 5B, the Harris corner detector 43b detects a plurality of feature points FPb of the camera image ImC. In FIGS. 4A to 5B, the feature points FPa and FPb are schematically represented by cross marks, but in reality, more feature points FPa and FPb are detected.

The Harris corner detectors 43a, 43b have one or more (two in this embodiment) parameters that affect the scale (in other words, parameters that are less invariant to the scale). For example, the first parameter is the size of the evaluation target area. The second parameter is the kernel size of the gradient detection filter (eg Sobel's gradient detection filter).

Since the resolution of the background light image ImC and the resolution of the camera image ImC are different for the parameters affecting such scale, the Harris corner detectors 43a and 43b usually have different numbers for the background light image ImL and the camera image ImC. Feature points FPa and FPb are detected. Therefore, even if there is an overlapping range between the measuring range MA1 of the distance measuring sensor 10 and the measuring range MA2 of the external camera 20, the same number of feature points FPa and FPb are detected for the overlapping range. Not necessarily.

The Harris corner detectors 43a and 43b are arranged one by one corresponding to the background light image ImC and the camera image ImC for convenience in FIG. 3, but are common to the background light image ImC and the camera image ImC ( It may be provided (by a common program). Of the processes of the Harris corner detectors 43a and 43b, only a highly versatile portion may be provided in common for the background light image ImC and the camera image ImC.

The correspondence relationship identification function specifies the correspondence relationship between the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC. In the present embodiment, in addition to not having the same number of detected feature points FPa and FPb, the positional relationship between the plurality of feature points FPa in the background light image ImC and the positional relationship between the plurality of feature points FPb in the camera image ImC. The fact that there is a possibility that the correspondences are different and that the feature points FPa and FPb that do not have a correspondence relation may be included increase the difficulty of identifying the correspondence relation.

That is, the light receiving element 12a of the distance measuring sensor 10 and the camera element 22a of the external camera 20 are arranged at different positions in the vehicle 1 as shown in FIG. 2, and the orientation of the arrangement may be different. As a result, as described above, the positional relationship at the plurality of feature points FPa in the background light image ImC and the positional relationship at the plurality of feature points FPb in the camera image ImC are different.

Further, in the case of being used for the vehicle 1 moving at high speed as in the present embodiment, the position of the object reflected in the background light image ImC and the position of the object reflected in the camera image ImC due to the deviation Δt of the measurement timing. Can be very different, and even an object can be reflected in only one side. Therefore, the positional relationship at the plurality of feature points FPa in the background light image ImC and the positional relationship at the plurality of feature points FPb in the camera image ImC are different, and the feature points FPa and FPb having no corresponding relationship are included. However, it is easy to occur.

In order to deal with the difficulty of identifying such a correspondence, first, the image processing unit 42 is a feature amount obtained from a peripheral region including each feature point FPa and FPb, and is highly invariant with respect to the scale. Use the features to identify the correspondence. Examples of the feature quantity having high invariance with respect to the scale include information on the direction of the edge, an average value or a ratio of some physical quantity in the peripheral region, and the like. In the present embodiment, as a feature amount that is highly invariant with respect to the scale, information on the extreme value for the degree of smoothing when a low-pass filter is applied to the peripheral region is adopted.

For example, the image processing unit 42 of the present embodiment detects the feature amount (hereinafter, SIFT feature amount) of SIFT (Scale-Invariant Feature Transform) by the SIFT feature amount detector (hereinafter, feature amount detector) 44a, 44b. Correspondence is specified using the detected SIFT features. The feature amount detectors 44a and 44b apply a Gaussian filter as the above-mentioned low-pass filter to the peripheral region including the feature points FPa and FPb detected by the Harris corner detectors 43a and 43b.

The feature detectors 44a and 44b change the weighting coefficient σ corresponding to the standard deviation of the Gaussian filter and search for a local extremum in the peripheral region. The feature detectors 44a and 44b set at least a part of the promising (excluding edges) σ of the σ in which the local extremum is found as the SIFT feature that is highly invariant with respect to the scale.

In this way, in the comparison between the SIFT feature amount individually corresponding to each feature point FPa of the background light image ImC and the SIFT feature amount individually corresponding to each feature point FPb of the camera image ImC, the matching accuracy between the feature points FPa and FPb is improved. Can be enhanced.

The feature amount detectors 44a and 44b are arranged one by one corresponding to the background light image ImC and the camera image ImC for convenience in FIG. 3, but are common to the background light image ImC and the camera image ImC ( It may be provided (by a common program). Of the processes of the feature amount detectors 44a and 44b, only a part having high versatility may be provided in common for the background light image ImC and the camera image ImC.

Secondly, the image processing unit 42 specifies the correspondence relationship in consideration of the difference in the position where the corresponding point on the image is reflected based on the relative position between the distance measuring sensor 10 and the external camera 20. Based on Epipolar Geometry, the image processing unit 42 uses the epipolar line projector 45 to display the epipolar line EL corresponding to the feature point FPb of the camera image ImC as a background light image, as shown in FIGS. 6A and 6B. Project to ImL. The epipolar line EL is a line where the epipolar plane and the image plane intersect. The epipolar plane is a plane that passes through the optical center of the ranging sensor 10, the optical center of the external camera 20, and the three-dimensional point of the subject corresponding to the feature point FPb of the camera image ImC.

Actually, the epipolar line projector 45 stores an E matrix (Essential matrix) defined based on the position of the light receiving element 12a and the position of the camera element 22a. The E matrix is a matrix for mapping a point on the camera image ImC to a line (that is, an epipolar line EL) on the background light image ImC.

Assuming that the background light image ImC and the camera image ImC are synchronized, the feature point FPb of the background light image ImL corresponding to a certain feature point FPb of the camera image ImC is an epipolar ray projector. It should be on the epipolar line EL by 45. However, in the present embodiment, there is a deviation Δt in the measurement timing between the background light image ImL and the camera image ImC, and the object reflected in the background light image ImL and the camera image ImC moves between the deviation Δt. there is a possibility.

Therefore, the image processing unit 42 narrows down the feature points FPa having a corresponding relationship by using the determination region JA which is a band-shaped region centered on the epipolar line EL and has a predetermined allowable width W. The permissible width W is set according to the amount of deviation assumed between the measurement timing of the background light image ImC and the measurement timing of the camera image ImC. Specifically, the image processing unit 42 uses the feature point FPa of the background light image ImL located inside the determination region JA as a candidate for a point corresponding to the feature point FPb of the camera image ImC which is the projection source of the epipolar line EL. Narrow down. Then, among the narrowed down feature points FPa, the feature point FPa with which the SIFT feature amount is closest to be determined is determined to be the corresponding point. In this way, the image processing unit 42 identifies a one-to-one individual correspondence relationship from each feature point FPb of the camera image ImC and each feature point FPa of the background light image ImC. If the numbers of feature points FPa and FPb detected by the Harris corner detectors 43a and 43b do not match between the background light image ImL and the camera image ImC, naturally, the corresponding points are not found in the other party's image. Feature points FPa and FPb appear, but these feature points FPa and FPb are not used for image registration as a result and are excluded from the subsequent processing.

The coordinate matching function is a function of making each pixel of the background light image ImC and the camera image ImC correspond to each pixel of the other based on the result of specifying the correspondence relationship between the feature points FPa and FPb. Specifically, as shown in FIG. 7, the image processing unit 42 non-linearly sets at least one of the background light image ImL and the camera image ImC based on the positional relationship between the pair of feature points FPa and FPb having a corresponding relationship. By smoothly distorting, the correspondence between the coordinate system of the background light image ImC and the coordinate system of the camera image ImC can be obtained.

The image processing unit 42 performs TPS (Thin Plate Spline) using, for example, a TPS model when matching coordinates. In the TPS model, TPS is performed with the coordinates of the corresponding feature points FPa and FPb as covariates. The TPS model identifies the correspondence between the background light image ImL and the camera image ImC of each pixel that does not correspond to the feature points FPa and FPb.

As a specific example for explaining the meaning of specifying the correspondence relationship of each pixel, the measurement timing of the background light image ImL is after Δt of the measurement timing of the camera image ImC, and the other vehicle in front is shifted Δt with respect to the vehicle 1. Consider the case of moving away from. In this case, the ratio of the distance between the feature points reflecting the other vehicle to the distance between the feature points reflecting the landscape in the background light image ImC is the ratio between the feature points reflecting the other vehicle to the distance between the feature points reflecting the landscape in the camera image ImC. Can be less than the ratio of intervals. Therefore, by non-linearly distorting the background light image ImL so that the area reflecting the other vehicle is enlarged with respect to the area reflecting the landscape in the background light image ImL, the coordinate system of the background light image ImL is captured by the camera. It can be adjusted to the coordinate system of the image ImC.

That is, the processing in the coordinate matching function corrects the deviation Δt of the measurement timing, and makes it possible to treat the background light image ImL and the camera image ImC in the same manner as the data synchronized with each other. As described above, the background light image ImL can be regarded as having the same coordinate system and synchronization with the reflected light image. As a result, the image processing unit 42 makes it possible to handle the reflected light image including the distance information and the camera image ImC, which is a high-resolution and color image, in the same manner as the data synchronized with each other. In the image registration of such a reflected light image and the camera image ImC, the background light image ImL functions like an adhesive for associating the two images.

Then, the image processing unit 42 can output the composite image which is the above-mentioned integrated image data by converting the coordinates corresponding to each pixel of the background light image ImC into the coordinates on the camera image ImC. .. Since this composite image has a common coordinate system for each channel, the processing of the application program (hereinafter referred to as the application) using the composite image is simplified to reduce the calculation load and improve the processing accuracy of the application. Can be made to.

In the present embodiment, the composite image output by the image processing unit 42 is provided to the driving support ECU 50 as peripheral monitoring image information. In the driving support ECU 50, object recognition using a composite image is performed by executing an object recognition program as an application for the processor to recognize the surrounding environment of the vehicle 1.

In this embodiment, object recognition using Semantic Segmentation is performed. In the storage unit of the driving support ECU 50, an object recognition model 51 mainly composed of a neural network is constructed as one component of the object recognition program. For this neural network, for example, a structure called SegNet in which an encoder and a decoder are combined can be adopted.

Next, the details of the image registration method for performing image registration between the reflected light image and the camera image ImC based on the image registration program will be described with reference to the flowchart of FIG. A series of image processing based on each step of this flowchart is performed, for example, at predetermined time intervals or every time the distance measuring sensor 10 or the external camera 20 generates a new image.

First, in S11, the image acquisition unit 41 acquires the latest reflected light image and background light image ImL from the ranging sensor 10, and acquires the latest camera image ImC from the outside camera 20. The image acquisition unit 41 provides these images to the image processing unit 42. After the processing of S11, the process proceeds to S12.

In S12, the image processing unit 42 detects the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC, respectively. After the processing of S12, the process proceeds to S13.

In S13, the image processing unit 42 specifies the correspondence between the feature point FPa of the background light image ImL detected in S12 and the feature point FPb of the camera image ImC. After the processing of S13, the process proceeds to S14.

In S14, the image processing unit 42 does not correspond to the feature points FPa and FPb between the background light image ImL and the camera image ImC because of the positional relationship between the feature points FPa and FPb whose correspondence is specified in S13. Coordinates of each pixel are associated (coordinate matching). After the processing of S14, the process proceeds to S15.

In S15, the image processing unit 42 converts the coordinate system of the background light image ImL and the reflected light image into the coordinate system of the camera image ImC, or vice versa, so that the reflected light image and the camera image ImC are combined. Complete image registration. A series of processes is completed by S15.

(Action effect)
The effects of the first embodiment described above will be described again below.

According to the image processing ECU 30 of the first embodiment, image registration between the acquired reflected light image and the camera image ImC is performed using the background light image ImL having the same coordinate system as the reflected light image. By comparing the feature point FPa of the background light image ImC whose properties are closer to those of the camera image ImC than the reflected light image and the feature point FPb of the camera image ImC, it becomes easy to identify the correspondence between the feature points FPa and FPb. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image ImC can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image ImC can be remarkably improved. it can.

Further, according to the first embodiment, in the image registration, the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC are detected, respectively. After that, the correspondence between the detected feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC is specified. Then, each pixel of the background light image ImC and the camera image ImC is made to correspond to each pixel of the other based on the specific result of the correspondence relationship. That is, since the correspondence between the feature points FPa and FPb is specified and then the coordinates of each pixel are made to correspond using the result, the processing amount is suppressed as compared with the case where the correspondence between each pixel is specified unnecessarily. Alternatively, highly accurate image registration can be performed while improving the processing speed.

Further, according to the first embodiment, in specifying the correspondence relationship, the difference in the position where the corresponding point on the image is reflected based on the relative position between the distance measuring sensor 10 and the outside world camera 20 is taken into consideration. By such consideration, it is possible to improve the accuracy of specifying the correspondence between the feature points FPa and FPb.

Further, according to the first embodiment, in the identification of the correspondence relationship, the epipolar line EL corresponding to the feature point FPb of the projection source among the background light image ImL and the camera image ImC is projected on the projection destination image. Then, it is determined that the feature point FPa of the projection destination located in the band-shaped determination region JA having a predetermined allowable width W along the epipolar line EL is a point corresponding to the feature point FPb of the projection source. By determining the allowable width W, it is possible to identify the correspondence between the feature points FPa and FPb, absorb errors such as the projection error between the background light image ImL and the camera image ImC, and improve the identification accuracy. ..

Further, according to the first embodiment, the allowable width W is set according to the amount of deviation assumed between the measurement timing of the background light image ImL and the measurement timing of the camera image ImC. Even if the objects that are reflected in the background light image ImL and the camera image ImC and constitute the feature points FPa and FPb move during the measurement timing deviation Δt, the determination region JA has an allowable width W according to the deviation amount. If the feature point FPa of the object is located in, the feature points FPa and FPb having a corresponding relationship are specified. Therefore, the accuracy in identifying the correspondence can be improved.

Further, according to the first embodiment, in specifying the correspondence between the feature point FPa of the background light image ImL and the feature point FPb of the camera image ImC, it is a feature amount obtained from the peripheral region including the feature points FPa and FPb. Therefore, the SIFT feature amount is used as a feature amount that is highly invariant with respect to the scale. By using SIFT features that are highly invariant to the scale, there is a difference in the detection levels (detection sensitivities) of the feature points FPa and FPb due to the high resolution of the camera image ImC with respect to the resolution of the background light image ImL. Even if it occurs, it is possible to suppress erroneous determination of the correspondence relationship. Therefore, the accuracy in identifying the correspondence can be improved.

Further, according to the image generation system 100 of the first embodiment, image registration of the reflected light image and the camera image ImC is performed using the background light image ImL of the same coordinate system as the reflected light image. That is, in comparison between the feature point FPa of the background light image ImC whose properties are closer to those of the camera image ImC than the reflected light image and the feature point FPb of the camera image ImC, it is easy to identify the correspondence between the feature points FPa and FPb. Become. With such an association, the coordinate system of the reflected light image and the coordinate system of the camera image ImC can be accurately matched. Then, the distance information and the camera image ImC information, which are information from different image generation sources of the distance measuring sensor 10 and the external camera 20, can be provided in the form of a composite image that can be easily processed by the application. Therefore, the processing system of the application using both the reflected light image and the camera image ImC can be remarkably enhanced.

Further, according to the image registration method of the first embodiment, the feature point FPa of the prepared background light image ImC and the feature point FPb of the camera image ImC are detected, respectively. After that, the correspondence between the detected feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC is specified. Then, each pixel of the background light image ImC and the camera image ImC is made to correspond to each pixel of the other based on the specific result of the correspondence relationship. In this way, the image registration between the reflected light image and the camera image ImC is performed using the background light image ImL which has the same coordinate system as the reflected light image and whose properties are closer to those of the camera image ImC than the reflected light image. It becomes easy to identify the correspondence between the feature points FPa and FPb. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image ImC can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image ImC can be remarkably improved. it can. Then, after the correspondence between the feature points FPa and FPb is specified, the coordinates of each pixel are made to correspond using the result, so that the processing amount is higher than the case where the correspondence of each pixel is specified unnecessarily. It is possible to carry out highly accurate image registration while suppressing the above or improving the processing speed.

Further, according to the image registration program of the first embodiment, the feature point FPa of the acquired background light image ImC and the feature point FPb of the camera image ImC are detected, respectively. After that, the correspondence between the detected feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC is specified. Then, each pixel of the background light image ImC and the camera image ImC is made to correspond to each pixel of the other based on the specific result of the correspondence relationship. In this way, the image registration between the reflected light image and the camera image ImC is performed using the background light image ImL which has the same coordinate system as the reflected light image and whose properties are closer to those of the camera image ImC than the reflected light image. It becomes easier to identify the correspondence between the feature points FPa and FPb. By such association, the coordinate system of the reflected light image and the coordinate system of the camera image ImC can be accurately matched, so that the processing accuracy of the application using both the reflected light image and the camera image ImC can be remarkably improved. it can. Then, after the correspondence between the feature points FPa and FPb is specified, the coordinates of each pixel are made to correspond using the result, so that the processing amount is higher than the case where the correspondence of each pixel is specified unnecessarily. It is possible to carry out highly accurate image registration while suppressing the above or improving the processing speed.

(Second Embodiment)
As shown in FIG. 9, the second embodiment is a modification of the first embodiment. The second embodiment will be described focusing on the points different from those of the first embodiment.

In the second embodiment, the functions of the image processing ECU 30 and the operation support ECU 50 of the first embodiment are integrated into one ECU, and the operation support ECU 230 is configured. Therefore, in the second embodiment, the operation support ECU 230 corresponds to the image registration device. Further, in the driving support ECU 230 of the second embodiment, it can be said that the image registration function constitutes a part for realizing the highly accurate peripheral recognition function. Therefore, the driving support ECU 230 provides the surrounding environment of the vehicle 1. It also corresponds to a peripheral environment recognition device that recognizes. The operation support ECU 230 has a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, and the like, similarly to the image processing ECU 30 of the first embodiment.

Similar to the image processing ECU 30 of the first embodiment, the operation support ECU 230 of the second embodiment has a plurality of functions by executing the image registration program and the object recognition program stored in the storage unit 33 by the processing unit 31. Has a part. Specifically, as shown in FIG. 9, functional units such as an image acquisition unit 41, an image processing unit 242, and an object recognition unit 48 are constructed in the driving support ECU 230.

The image acquisition unit 41 is the same as that of the first embodiment. The object recognition unit 48 uses the same object recognition model 48a as in the first embodiment to perform object recognition using semantic segmentation.

The image processing unit 242 of the second embodiment has a feature point detection function, a correspondence relationship identification function, and a coordinate matching function, as in the first embodiment. However, the first point is that the feature point detection function considers the ratio between the resolution of the background light image ImC and the resolution of the camera image ImC (hereinafter referred to as the resolution ratio), and that the SIFT feature amount is not used in the correspondence identification function. Different from the embodiment.

Specifically, the storage unit 33 of the driving support ECU 230 is provided with a sensor database (hereinafter, sensor DB) 243c. The sensor system DB243c stores information on various sensors and cameras mounted on the vehicle 1. This information includes information on the specifications of the light receiving element 12a of the distance measuring sensor 10 and information on the specifications of the camera element 22a of the external camera 20. The information regarding the specifications of the light receiving element 12a of the distance measuring sensor 10 includes the resolution information of the light receiving element 12a, and the information regarding the specifications of the camera element 22a of the external camera 20 includes the resolution information of the camera element 22a. included. From the information of these resolutions, the image processing unit 242 can grasp the resolution ratio.

The Harris corner detector 243a of the second embodiment has a scale parameter for detecting the feature point FPa of the background light image ImC and a scale parameter for detecting the feature point FPb of the camera image ImC based on the resolution ratio. Is different. Specifically, in the feature point detection of the camera image ImC, which has a high resolution with respect to the background light image ImL, the size of the evaluation target area as the scale parameter and the kernel of the gradient detection filter are compared with the case of the background light image ImL. At least one of the sizes is reduced. Then, the detection levels of the feature points FPa and FPb can be brought close to each other between the background light image ImL and the camera image ImC.

As a result, even if SIFT is not used in the correspondence relationship identification function, it becomes easy to specify the correspondence relationship between the feature point FPa of the detected background light image ImC and the feature point FPb of the camera image ImC. In other words, the correspondence can be identified with high accuracy.

According to the second embodiment described above, the Harris corner detectors 243a and 243b as feature point detectors for detecting the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC determine the scale parameters that affect the scale. Have. In such a configuration, the scale parameter used for detecting the feature point FPa of the background light image ImC and the feature point FPb of the camera image ImC are based on the ratio of the obtained background light image ImC resolution and the camera image ImC resolution. It is different from the scale parameter used to detect. In this way, even if the resolution of the camera image ImC is higher than the resolution of the background light image ImC, the detection levels of the feature points FPa and FPb can be brought close to each other between the background light image ImC and the camera image ImC. .. Since the feature points FPa and FPb detected at close levels can be compared with each other, the accuracy in specifying the correspondence can be improved.

(Other embodiments)
Although the plurality of embodiments have been described above, the present disclosure is not construed as being limited to those embodiments, and is applied to various embodiments and combinations within the scope of the gist of the present disclosure. Can be done.

Specifically, as a modification 1, the distance measuring sensor 10 and the external camera 20 may form an integrated sensor unit. Further, an image registration device such as the image processing ECU 30 of the first embodiment may be included as a component of this sensor unit.

As a modification 2 regarding the first embodiment, the image processing ECU 30 may include an object recognition unit 48 as in the second embodiment and recognize the surrounding environment of the vehicle 1. The analyzed information in which the image processing ECU 30 recognizes the surrounding environment of the vehicle 1 may be provided to the driving support ECU 50 having a driving support function or the like.

As a modification 3, the image processing unit 42 does not have to integrate the reflected light image, the background light image ImL, and the camera image ImC into a multi-channel composite image and output it. The image processing unit 42 outputs the reflected light image, the background light image ImL, and the camera image ImC as separate image data, and in addition to these image data, outputs coordinate correspondence data indicating the correspondence between the coordinates of each image. You may try to do it.

As a modification 4, the image processing unit 42 may output the image-registered reflected light image and the camera image ImC, and may not output the background light image ImC.

As a modification 5, the camera image ImC may be a grayscale image instead of a color image.

As a modification 6, the object recognition using the image-registered reflected light image and the camera image ImC does not have to be the object recognition using semantic segmentation. The object recognition may be, for example, object recognition using a bounding box.

As a modification 7, the image-registered reflected light image and the camera image ImC may be used for applications other than object recognition in the vehicle 1. For example, the distance measuring sensor 10 and the camera 20 may be installed in the conference room, and the image-registered reflected light image and the camera image ImC may be used in a communication application for video conferencing.

As a modification 8, information on the orientation for imparting rotational invariance may be added to the detected SIFT feature amount. The orientation information is useful, for example, in a situation where the inclination of the installation surface of the distance measuring sensor 10 and the inclination of the installation surface of the camera 20 are different.

As a modification 9, the epipolar line EL corresponding to the feature point FPb of the camera image ImC is projected onto the background light image ImL, or the epipolar line EL corresponding to the feature point FPa of the background light image ImC is projected onto the camera image ImC. When projecting to, an F matrix (Fundamental matrix) may be used instead of the E matrix. The F matrix is useful in situations where the ranging sensor 10 and camera 20 have not been calibrated.

As a modification 10, the image processing unit 42 may perform image registration on an image generated by the ranging sensor 10 and an image generated by the camera 20 as well as an additional image generated by a millimeter-wave radar or the like. Good.

As a modification 11, each function provided by the image processing ECU 30 can be provided by software and hardware for executing the software, only software, only hardware, or a combination thereof. Further, when such a function is provided by an electronic circuit as hardware, each function can also be provided by a digital circuit including a large number of logic circuits or an analog circuit.

As the modification 12, the form of the storage medium for storing the abnormality detection program or the like capable of realizing the above abnormality detection method may be changed as appropriate. For example, the storage medium is not limited to the configuration provided on the circuit board, but is provided in the form of a memory card or the like, is inserted into the slot portion, and is electrically connected to the control circuit of the image processing ECU 30. You may. Further, the storage medium may be an optical disk or a hard disk as a copy base of the program of the image processing ECU 30.

The control unit and its method described in the present disclosure may be realized by a dedicated computer constituting a processor programmed to execute one or a plurality of functions embodied by a computer program. Alternatively, the apparatus and method thereof described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the apparatus and method thereof described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. Further, the computer program may be stored in a computer-readable non-transitional tangible recording medium as an instruction executed by the computer.

Claims (10)

1. A reflected light image including distance information when the light receiving element (12a) senses the reflected light reflected from the object by light irradiation, and the reflected light image when the light receiving element senses the background light with respect to the reflected light. By being communicably connected to the ranging sensor (10) that generates a background light image (IML) having the same coordinate system, and by detecting incident light from the outside by the camera element (22a). An image registration device communicatively connected to a camera (20) that generates a camera image (ImC) having a higher resolution than the reflected light image and the background light image.
   An image acquisition unit (41) that acquires the reflected light image, the background light image, and the camera image, and
   By specifying the correspondence between the feature point (FPa) of the background light image and the feature point (FPb) of the camera image, the reflected light image having the same coordinate system as the background light image and the camera image An image registration device including an image processing unit (42) for performing image registration.
2. The image processing unit
   The feature points of the background light image and the feature points of the camera image are detected, respectively.
   The correspondence between the detected feature points of the background light image and the feature points of the camera image is specified.
   The image registration apparatus according to claim 1, wherein one pixel of the background light image and the camera image is associated with each other pixel based on the specific result of the correspondence relationship.
3. The image according to claim 2, wherein the image processing unit considers the difference in the position where the corresponding point on the image is reflected based on the relative position between the distance measuring sensor and the camera, and specifies the corresponding relationship. Registration device.
4. The image processing unit
   An epipolar line (EL) corresponding to a feature point of a projection source among the background light image and the camera image is projected onto the projection destination image, and a strip-shaped strip having a predetermined allowable width (W) along the epipolar line. The image registration apparatus according to claim 3, wherein the feature point of the projection destination located in the determination area (JA) is determined to be a point corresponding to the feature point of the projection source.
5. The image registration apparatus according to claim 4, wherein the allowable width is set according to an amount of deviation assumed between the measurement timing of the background light image and the measurement timing of the camera image.
6. The image processing unit
   In identifying the correspondence between the feature points of the background light image and the feature points of the camera image, the feature quantities obtained from the peripheral region including each of the feature points and which are highly invariant with respect to the scale are defined. The image registration apparatus according to any one of claims 2 to 5 to be used.
7. The image processing unit
   Using a feature point detector having parameters that affect the scale, the feature points of the background light image and the feature points of the camera image are detected, respectively.
   The ratio of the resolution of the background light image to the resolution of the camera image is grasped, and based on the ratio, the parameters used for detecting the feature points of the background light image and the feature points of the camera image are detected. The image registration apparatus according to any one of claims 2 to 5, which is different from the parameters used.
8. An image generation system that generates images to be processed by an application.
   The reflected light image including distance information when the light receiving element (12a) senses the reflected light reflected from the object by light irradiation, and the reflected light image when the light receiving element senses the background light with respect to the reflected light. A ranging sensor (10) that generates a background light image (IML) that has the same coordinate system as
   A camera (20) that generates a camera image (ImC) having a higher resolution than the reflected light image and the background light image by detecting incident light from the outside by the camera element (22a).
   By specifying the correspondence between the feature point (FPa) of the background light image and the feature point (FPb) of the camera image, the reflected light image having the same coordinate system as the background light image and the camera image An image generation system including an image processing unit (42) that performs image registration and generates a composite image in which the distance information and the information of the camera image are integrated.
9. An image generated by the distance measuring sensor (10), which includes a reflected light image including distance information when the light receiving element (12a) senses the reflected light reflected from an object by light irradiation, and a background light for the reflected light. To prepare a background light image having the same coordinate system as the reflected light image by detecting the light receiving element.
   An image generated by the camera (20), in which the camera element (22a) detects incident light from the outside, a camera image (ImC) having a higher resolution than the reflected light image and the background light image is prepared. To do and
   Detecting the feature points (FPa) of the background light image and the feature points (FPb) of the camera image, respectively,
   Identifying the correspondence between the detected feature points of the background light image and the feature points of the camera image, and
   An image registration method comprising making each pixel of one of the background light image and the camera image correspond to each pixel of the other based on the specific result of the correspondence relationship.
10. An image registration program that performs image registration between the image generated by the distance measuring sensor (10) and the image generated by the camera (20).
    In at least one processing unit (31),
    An image generated by the ranging sensor, which includes a reflected light image including distance information when the light receiving element (12a) senses the reflected light reflected from an object by light irradiation, and a background light for the reflected light. A process of acquiring a background light image (IML) having the same coordinate system as the reflected light image by being sensed by the light receiving element, and
    A process of acquiring a camera image (ImC) having a higher resolution than the reflected light image and the background light image by detecting the incident light from the outside in the image generated by the camera by the camera element (22a). When,
    A process of detecting a feature point (FPa) of the background light image and a feature point (FPb) of the camera image, respectively.
    Processing for specifying the correspondence between the detected feature points of the background light image and the feature points of the camera image, and
    An image registration program that executes a process of making each pixel of one of the background light image and the camera image correspond to each pixel of the other based on the specific result of the correspondence relationship.

Images