JP2019174308A - Imaging device, distance calculation method for imaging device, and distance calculation program for imaging device - Google Patents

Abstract

To provide an imaging device with which it is possible to measure the distance to a subject with high accuracy.SOLUTION: The imaging device comprises a light source, an imaging unit, a reference distance calculation unit, a storage unit, and a distance selection unit. The light source irradiates a subject with light. The imaging unit receives reflected light that is the light irradiated by the light source and reflected from the subject. The reference distance calculation unit calculates, by a combination of illumination light sources, among three or more light sources, except at least one, that irradiate light, a reference distance to the subject for each pixel in a combination of respective illumination light sources on the basis of a duration from when light is irradiated to when reflected light from the subject is received by the imaging unit, and generates a reference distance image. The storage unit associates a reference distance for each pixel in the reference distance image generated by the reference distance calculation unit with the received amount of the reflected light from the subject used for calculating the reference distance and stores them. The distance selection unit selects, for each pixel, as the distance to the subject in the pixel, on the basis of the reference distance, among the reference distances stored in the storage unit, for which effective distance is calculated and the received amount of light.SELECTED DRAWING: Figure 1

Description

Embodiments described herein relate generally to an imaging apparatus that acquires a distance image to a subject, a distance calculation method for the imaging apparatus, and a distance calculation program for the imaging apparatus.

In general, a three-dimensional camera such as a TOF (Time Of Flight) camera capable of measuring a distance to a subject is known. The TOF camera is a technique for measuring a distance based on a time until irradiation light hits a subject and returns to an imaging unit. However, if the subject has a shape with few planes, such as a thin cylinder, the irradiation light is diffused by the subject and the amount of reflected light received decreases.
There is a problem that the distance cannot be measured with sufficient accuracy. In addition, when the subject is specularly reflected locally, the reflected light from the specularly reflected part does not return to the imaging unit, or most of the irradiation light from the light source returns to the imaging unit and the imaging unit is saturated. There is a problem that the distance cannot be calculated. Such an example occurs in a case where the subject is uneven, such as vinyl, or the subject is covered with aluminum foil.

JP 2010-190675 A

The problem to be solved by the present invention is an imaging device capable of measuring the distance to a subject with high accuracy,
An object is to provide a distance calculation method for an imaging apparatus and a distance calculation program for the imaging apparatus.

According to the embodiment, the imaging apparatus includes three or more light sources, an imaging unit, a reference distance calculation unit, a storage unit, and a distance selection unit. Three or more light sources irradiate the subject with light. The imaging unit
The reflected light from the subject of the light irradiated by the light source is received. The reference distance calculation unit is a combination of lighting light sources in which light sources other than at least one light source among three or more light sources emit light, and from when the imaging unit receives reflected light from the subject. On the basis of this time, a reference distance to the subject is calculated for each pixel in each combination of lighting light sources to generate a reference distance image. The storage unit stores the reference distance for each pixel in the reference distance image generated by the reference distance calculation unit and the received light amount of the reflected light from the subject used to calculate the reference distance. The distance selection unit selects a predetermined reference distance for each pixel as a distance to the subject on the basis of the reference distance for which an effective distance is calculated from the reference distances stored in the storage unit and the received light amount. .

The figure which shows the front view which shows an example of the external appearance of the imaging device which concerns on embodiment, and the combination of lighting of a light source. FIG. 3 is a block diagram illustrating an example of a functional configuration of a control unit included in the imaging apparatus according to the embodiment. The perspective view which shows an example of the imaging condition using the imaging device which concerns on embodiment. FIG. 4 is a diagram illustrating an example of pixels of a reference distance image acquired by the imaging apparatus according to the embodiment. 4 is a diagram illustrating an example of data stored in a storage unit included in the imaging apparatus according to the embodiment. FIG. The flowchart which shows an example of the flow from which the control part with which the imaging device which concerns on embodiment is provided selects the distance to a to-be-photographed object.

Hereinafter, imaging device 1 concerning an embodiment is explained using a drawing.
FIG. 1A is a front view illustrating an example of the appearance of a TOF camera as the imaging apparatus 1 according to the embodiment. The imaging device 1 includes a plurality of light sources 2 a to 2 f that irradiate light on a subject and an imaging unit 3. The light sources are abbreviated as light sources 2 when it is not necessary to individually distinguish the six light sources 2a to 2f shown in FIG.

The imaging apparatus 1 is configured to be able to measure the distance to the subject for each pixel based on the time from when the light emitted from the light source 2 hits the subject until it is received by the imaging unit 3. In addition, the imaging device 1 can generate a distance image that represents the measured distance information for each pixel.

Three or more light sources 2 are installed around the imaging unit 3, and in FIG. 1A, light is emitted from six light sources 2 toward a subject. The light emitted from the light source 2 may be infrared or visible light.
Any light in a wavelength band that can be received by the imaging unit 3 may be used. Further, the light source 2 may include a plurality of LED light sources for each light source in order to increase the amount of light applied to the subject or the irradiation intensity.

The light source 2 lights up the light source 2 by the combination of the lighting light source which the light source 2 except the at least 1 light source 2 irradiates light among the six light sources 2. FIG. 1B is a diagram showing a combination of lighting of the light sources 2, and shows a combination of sequentially lighting the light sources 2 excluding at least one of the light sources 2a to 2f.

In addition, the installation position and the number of the light sources 2 are not limited to those in FIG. For example, in FIG. 1, the light source 2 is installed symmetrically around the imaging unit 3, but may be installed asymmetrically. When it is desired to increase the amount of light, a plurality of LEDs may be arranged in each of the light sources 2a to 2f.

The imaging unit 3 receives reflected light from the subject of light emitted from the light source 2 and captures a two-dimensional image. For example, the imaging unit 3 receives reflected light from the subject via an opening, a lens, and a light receiving unit (not shown).

The opening is a stop for adjusting the amount of light incident on the lens. The lens condenses light from the subject on the light receiving unit. The number of lenses is not limited to one, and a focus lens or a zoom lens may be configured by combining a plurality of lenses, for example. The light receiving unit is a light receiving element such as an image sensor.

Next, the control part with which the imaging device 1 is provided is demonstrated using FIG. FIG. 2 is a block diagram illustrating an example of a functional configuration included in the imaging device 1 according to the embodiment.

The control unit 4 includes a CPU (not shown) and comprehensively controls the operation of each component included in the imaging device 1. The control unit 4 includes a reference distance calculation unit 5, a two-dimensional image generation unit 6, a light emission control unit 7,
A storage unit 8, a distance selection unit 9, and a distance image output unit 10 are provided.

The reference distance calculation unit 5 performs a process of calculating a reference distance that is a distance to the subject based on the reflected light from the subject received by the imaging unit 3. That is, the reference distance calculation unit 5 uses a combination of lighting light sources that emit light from at least one of the three or more light sources 2 except for the light source 2, and then the imaging unit 3 starts from the subject. Based on the time until the reflected light is received, the reference distance to the subject is calculated for each pixel in each combination of lighting light sources, and a reference distance image is generated.

Further, for example, with respect to a pixel in which all of the light specularly reflected by the subject reaches the imaging unit 3, the received light amount of the imaging unit 3 is saturated and an effective reference distance may not be obtained. On the other hand, there may be a pixel in which the specularly reflected light is reflected in a direction other than the imaging unit 3 and the amount of light received by the imaging unit 3 is insufficient and an effective reference distance cannot be obtained. That is, the reference distance calculation unit 5 may not be able to calculate an effective distance and may set the distance to zero.

The two-dimensional image generation unit 6 generates a two-dimensional image based on the light received by the imaging unit 3. The reference distance to the subject and the reference distance image calculated by the reference distance calculation unit 5 are calculated for each pixel of the two-dimensional image generated by the two-dimensional image generation unit 6.

Since the two-dimensional image is generated by the two-dimensional image generation unit 6 based on the reflected light from the subject of the light emitted from the light source 2, it is a kind of image depending on the light source, such as a gray scale image or an infrared image. is there.

The light emission control unit 7 controls lighting and extinction of the six light sources 2 independently as shown in FIG. When the subject is imaged, the light source 2 is turned on according to the combination of the lighting light sources that the light sources 2 excluding at least one of the six light sources 2 emit light.

In addition, when measuring the reference distance to the subject, the control unit 4 may control each unit so as to perform imaging for all combinations of lighting sources, or for a specific combination of lighting sources. Then, each unit may be controlled so as to perform imaging.

Although not shown, the storage unit 8 includes a ROM (Read Only Memory) and a RAM (Random Access Me).
memory). The ROM is a non-volatile memory in which a control program, control data, and the like are stored in advance. The control program and control data stored in the ROM are incorporated in advance according to the specifications of the control unit 4. The ROM stores, for example, a program for controlling the circuit board of the control unit 4.

The RAM is a volatile memory. The RAM temporarily stores data being processed by the control unit 4. The ROM stores various application programs based on instructions from the control unit 4. The ROM may store data necessary for executing the application program, an execution result of the application program, and the like.

Further, the storage unit 8 stores the reference distance for each pixel in the reference distance image generated by the reference distance calculation unit 5 and the received light amount of the reflected light from the subject used to calculate the reference distance. To do. The storage unit 8 may store a two-dimensional image generated by the two-dimensional image generation unit 6. Further, the storage unit 8 stores a distance image output from a distance image output unit 10 described later.

The distance selection unit 9 sets a predetermined reference distance for each pixel as a distance to the subject based on the reference distance and the received light amount for which an effective distance among the reference distances stored in the storage unit 8 is calculated. select. For example, the distance selection unit 9 selects, as the distance to the subject in the pixel, the reference distance in which the largest amount of received light in each pixel is stored in each combination of lighting light sources. Alternatively, the distance selection unit 9 may select, as the distance to the subject in the pixel, the reference distance in which the most received light amount is stored in a predetermined range in each pixel in each combination of lighting light sources. good.

The distance image output unit 10 outputs the distance of each pixel selected by the distance selection unit 9 as a distance image.

Next, a method in which the reference distance calculation unit 5 calculates the reference distance for each pixel will be described with reference to FIGS. 3 and 4. First, FIG. 3 is a diagram illustrating an example of an imaging situation using the imaging apparatus 1 according to the embodiment. FIG. 3 illustrates a situation in which a rectangular parallelepiped object 12 is present in front of the imaging apparatus 1 and a fence 11 is provided between the imaging apparatus 1 and the object 12. It is assumed that the object 12 is positioned at an angle at which only one surface of the object 12 can be seen when viewed from the imaging device 1.

FIG. 4 is a diagram illustrating an example of pixels of the reference distance image acquired by the imaging device 1 according to the embodiment. 4 is an example of an imaged pixel in the imaging state of FIG. It is assumed that the pixels in FIG. 4 are imaged in a combination of lighting light sources in which the light sources 2b and 2e are turned on. In order to simplify the description, alphabets and numbers are assigned to the rows and columns of pixels shown in FIG. For example, the pixel located at the upper left is called A row and 1 column.

The numerical value described in each pixel means the distance to the subject, and the larger the number, the farther the distance to the subject. However, a pixel with a numerical value of 0 means that the distance could not be calculated. The value 0 means that the distance cannot be calculated, for example, when the distance to the subject is too far, when the amount of received light is too small, or when the amount of received light is too large, the imaging unit 3 is saturated. And so on.

In the pixel shown in FIG. 4, a numerical value 6 is shown in a rectangular range having the pixels of B row 4 column, B row 9 column, F row 4 column and F row 9 column excluding 5 column and 8 column. ing. This indicates the distance to the plane of the object 12, and the fifth and eighth rows indicate that the object 12 is blocked by the fence 11. The TOF camera is generally advantageous for measuring the distance to the plane, but the irradiation light is diffused for the distance to the fence 11, which is an edge, so that it is disadvantageous for the measurement. It is an extreme example illustrated.

Here, in FIG. 4, imaging is performed by a combination of lighting light sources in which the light sources 2 b and 2 e are turned on.
In addition to the light sources 2b and 2e, for example, when the light sources 2a and 2d are turned on, irradiation with light from the light source 2 with different positions and angles is given, so the distance to the fence 11 may be calculated. Therefore, even if it is not possible to calculate the distance to a subject of a pixel with a certain lighting source combination, the distance to the subject of that pixel can be calculated by using another lighting source combination, and the calculation result is complemented. Is possible.

In FIG. 1, since six light sources 2 are installed, combinations of lighting light sources that emit light from at least one of the six light sources 2 excluding at least one light source 2 are as shown in FIG. There are 63 ways. In this case, if the distance is calculated for all 63 ways, there are too many combinations.
On the contrary, it may be inconvenient. Therefore, the combination of the lighting light sources for calculating the reference distance may be set to be limited in advance. For example, the light sources 2b and 2e having a short distance from the imaging unit 3 may be necessarily turned on as long as the light sources 2 except for at least one light source 2 among the six light sources 2 are turned on. Further, the number of light sources 2 is not limited to six, but may be three or more.

Next, with reference to FIG. 5, the storage unit 8 included in the imaging device 1 according to the embodiment associates the reference distance with the received light amount of the reflected light from the subject used to calculate the reference distance. The data to be stored will be described. FIG. 5 is a diagram illustrating an example of data stored in the storage unit 8 included in the imaging apparatus 1 according to the embodiment.

FIG. 5 is a diagram illustrating an example of pixels of the reference distance image acquired by the imaging device 1 of FIG. 10 combinations of lighting light sources are shown as data in which reference distances and received light amounts are associated with pixels in the A row and the 11th column and the 1st to 7th columns of the B row. The amount of received light is shown as a ratio with 100% being the maximum.

As shown in FIG. 5, the Ath row and the eleventh column all have a received light amount of 10% or less, so the reference distance is “0” which is not effective because it is not an effective received light amount. Also, the first to third columns of the B row are not effective since the amount of received light is 10% or less, so the reference distance is “0” which is not effective. Next, the B row and the fourth column are all effective received light amounts. Among these combinations, the third and fourth combinations show the maximum amount of received light, and both are 90% received. For this reason, the reference distance of “6.0” stored in association with the maximum received light amount is selected as the distance in the Bth row and the fourth column.
Similarly, since the 90% received light amount is the maximum for the Bth row and the 6th column, the reference distance “6.0, which is the third to fifth combinations stored in association with the maximum received light amount. Is B line 6
It is selected as the distance of the column. In the Bth row and the 7th column, 90% of the received light amount is the maximum received light amount in the fourth and fifth combination, and therefore the reference distance “6.0” stored in association with this received light amount is B.
It is selected as the distance in row 7 column.

In the above embodiment, the reference distance stored in association with the maximum amount of received light is selected. However, if the amount of received light is too large, it is saturated and an accurate distance cannot be obtained. Is set to 15 to 89%, for example, in the above example, the received light amount of 88% is the largest received light amount within the predetermined range, so the reference distance “5.9” stored in association with it is stored. Is selected as the distance of the pixel.
Next, a method in which the control unit 4 included in the imaging apparatus 1 according to the embodiment calculates the distance to the subject will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of a flow in which the control unit 4 included in the imaging device 1 according to the embodiment calculates the distance to the subject.

First, the control unit 4 calculates a reference distance for all the set combinations of lighting sources (
Step S00). Next, the control unit 4 selects one arbitrary pixel (step S01), and among the reference distances of the pixels corresponding to the selected pixels of all the reference distances, there is an effective reference distance whose numerical value is not 0. Is determined (step S03). When an effective reference distance exists (step S03, Yes), the control unit 4 corresponds to the selected pixel, and a reference satisfying a predetermined condition among the effective reference distances excluding the numerical value 0 for which the distance cannot be calculated. The distance is selected as the distance to the subject (step S05). The predetermined condition is, for example, a reference distance in which the largest amount of received light is stored, or a reference distance in which the largest amount of received light is stored within a predetermined range in each pixel. The predetermined range is, for example, a range of 50% to 80% of the received light amount that the imaging unit 3 is saturated.

In step S03, when there is no effective reference distance (step S03,
No), the control unit 4 selects the numerical value 0 as the distance to the subject (step S07).

Next, the control unit 4 determines whether or not processing has been performed on all pixels (step S09). If it is determined that processing has been performed on all pixels (step S09, Yes), each selected pixel Is output as a distance image (step S10), and the process ends.

Further, when it is determined that processing has not been performed for all the pixels (step S09, No)
), The process is returned to step S01, and the selection process for the next pixel is executed.

By performing the operation according to the above flow, the imaging apparatus 1 according to the embodiment causes the light emitted from the light source 2 to be hardly diffused by the subject, and the received light amount is too large. The distance to the subject obtained by an optimal combination of lighting sources that do not saturate can be selected for each pixel. As a result, the distance to the subject can be measured with higher accuracy.

In the above embodiment, six light sources 2 have been described as an example. However, the number of light sources 2 may be three or more, and the number is not limited.

Moreover, the process in the imaging device 1 of the above-described embodiment can be executed by some software. For this reason, these programs are stored in the imaging apparatus 1 through a computer-readable storage medium storing several programs for executing the above-described processing procedures.
The above-described processing can be easily realized simply by installing it on the (TOF camera) and executing it. For example, the imaging device 1 downloads the above program via a network,
Stores downloaded programs and completes program installation. Alternatively, the imaging apparatus 1 can read the program from the information storage medium, store the read program, and complete the installation of the program.

Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Imaging device 2, 2a, 2b, 2d, 2e, 2f Light source 3 Imaging part 4 Control part 5 Reference distance calculation part 6 Dimensional image generation part 7 Light emission control part 8 Storage part 9 Distance selection part 10 Distance image output part 11 Fence 12 object

Claims (6)

Three or more light sources that irradiate the subject with light;
An imaging unit that receives reflected light from the subject of light emitted by the light source;
Of the three or more light sources, a combination of a lighting light source that emits light from at least one light source, and a time from when the imaging unit receives reflected light from the subject after irradiation with light. A reference distance calculating unit that calculates a reference distance to the subject for each pixel in each combination of the lighting light sources and generates a reference distance image;
The reference distance for each pixel in the reference distance image generated by the reference distance calculator;
A storage unit for storing the received light amount of reflected light from the subject used to calculate the reference distance in association with each other;
A predetermined reference distance is selected for each pixel as a distance to the subject on the basis of the reference distance for which an effective distance is calculated among the reference distances stored in the storage unit and the received light amount. A distance selector;
An imaging apparatus comprising: The distance selection unit selects, as the distance to the subject in the pixel, the reference distance in which the most received light amount in each pixel is stored in each combination of the lighting light sources.
The imaging device according to claim 1. The distance selection unit selects, as the distance to the subject in the pixel, the reference distance in which the most received light amount is stored in a predetermined range in each pixel in each combination of the lighting light sources.
The imaging device according to claim 1. A distance image output unit that outputs the distance of each pixel selected by the distance selection unit as a distance image;
The imaging device according to any one of claims 1 to 3, further comprising: Receiving light reflected from the subject by three or more light sources that irradiate the subject with light;
Of the three or more light sources, a combination of lighting light sources that emit light from at least one light source, based on the time from irradiating light to receiving reflected light from the subject, A reference distance image is calculated by calculating a reference distance to the subject for each pixel in the combination of the lighting light source,
Storing the reference distance for each pixel in the generated reference distance image in association with the amount of reflected light received from the subject used to calculate the reference distance;
A predetermined reference distance is selected as a distance to the subject in the pixel for each pixel based on the reference distance for which an effective distance is calculated from the stored reference distances and the received light amount.
The distance calculation method of an imaging device. A program for controlling a TOF camera having three or more light sources,
To the TOF camera,
Receiving light reflected from the subject by three or more light sources that irradiate the subject with light;
Of the three or more light sources, a combination of lighting light sources that emit light from at least one light source, based on the time from receiving light to receiving reflected light from the subject, A procedure for calculating a reference distance to the subject for each pixel in the combination of the lighting light source and generating a reference distance image;
A procedure for storing the reference distance for each pixel in the generated reference distance image and the received light amount of the reflected light from the subject used to calculate the reference distance in association with each other;
A procedure for selecting a predetermined reference distance as a distance to the subject in the pixel for each pixel based on the reference distance for which an effective distance is calculated from the stored reference distances and the received light amount;
A procedure for outputting the distance of each selected pixel as a distance image;
The distance calculation program of the imaging device that executes

Images