JP2021152543A - Vehicular processing system - Google Patents

Abstract

To provide a vehicular processing system that is able to acquire the size of a subject with a monocular camera.SOLUTION: According to an embodiment, a processor includes a size calculation unit and a determination unit. Using a distance indicated by each of pixel values, the size calculation unit projects onto a three-dimensional space each of pixels of a distance map, the distance map being acquired together with an image captured at one time by a single imaging optical system, in which the pixel value of each of pixels is formed as an image indicating a distance and information indicating the distance of a subject included in the image is mapped in correspondence with the image. The size calculation unit calculates the size of the subject from a distance between the pixels projected on the three-dimensional space. Based on the size of the subject, the determination unit determines a driving method for a vehicle.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to vehicle processing systems.

A technique for acquiring the size of a subject captured by a stereo camera (compound eye camera), for example, the length between two specified points on the subject is known.

Japanese Unexamined Patent Publication No. 2011-232330

An object to be solved by the present invention is to provide a vehicle processing system capable of acquiring the size of a subject with a monocular camera.

According to the embodiment, the vehicle processing system includes a size calculation unit and a determination unit. The size calculation unit is a distance map acquired together with an image by one imaging by a single imaging optical system, and a subject whose pixel value of each pixel is generated as an image indicating a distance and included in the image. Each pixel of the distance map, in which the information indicating the distance to is mapped in correspondence with the image, is projected onto the three-dimensional space using the distance indicated by each pixel value, and is projected onto the three-dimensional space. The size of the subject is calculated from the distance between the pixels. The determination unit determines the driving method of the vehicle based on the size of the subject.

The figure which shows an example of the functional block of the processing apparatus of 1st Embodiment. The figure which shows an example of the hardware composition of the processing apparatus of 1st Embodiment. The block diagram which shows one configuration example of the image pickup apparatus of 1st Embodiment. The figure which shows one configuration example of the filter of 1st Embodiment. The figure which shows an example of the transmittance characteristic of the filter region of 1st Embodiment. The figure for demonstrating the light ray change and the shape of a blur by a color aperture of 1st Embodiment. The figure which shows an example of the blurring function of the reference image of 1st Embodiment. The figure which shows an example of the blurring function of the target image of 1st Embodiment. The figure which shows an example of the blur correction filter of 1st Embodiment. The figure which shows an example of the image and the distance map of 1st Embodiment. The flowchart which shows an example of the processing flow of the processing system of 1st Embodiment. The figure which shows one output example of the size of a subject by the processing system of 1st Embodiment. The figure which shows one output example of the moving distance of a subject by the processing system of 1st Embodiment. The figure for demonstrating a general motion capture system. The figure which shows one further output example of the size of the subject by the processing system of 1st Embodiment. The figure which shows one further output example of the size of the subject by the processing system of 1st Embodiment. The figure for demonstrating one application example to the moving body of the processing apparatus of 1st Embodiment. The figure for demonstrating one further application example to the moving body of the processing apparatus of 1st Embodiment. The figure for demonstrating one application example to the automatic door system of the processing apparatus of 1st Embodiment. The figure which shows one modification of the functional block of the processing apparatus of 1st Embodiment. The figure which shows an example of the functional block of the processing apparatus of 2nd Embodiment. The figure for demonstrating one application example to the robot of the processing apparatus of 2nd Embodiment. The figure which shows an example of the functional block of the processing apparatus of 3rd Embodiment. The figure for demonstrating one application example to the moving body of the processing apparatus of 3rd Embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
(First Embodiment)
First, the first embodiment will be described.
FIG. 1 is a diagram showing an example of a functional block of the processing device of the first embodiment.
The processing device 1 calculates the size of the subject from the distance map (distance image) 2B obtained by imaging, and outputs the calculated size of the subject. The processing device 1 may further calculate the size of the subject by using the image 2A as well. The size of the subject may be displayed on the display unit 3 at the same time as the image 2A, for example. The details of the image 2A and the distance map 2B will be described later, but the image 2A and the distance map 2B may be acquired directly from, for example, an image pickup device that generates the image 2A and the distance map 2B, or the image 2A and the distance map 2B. It may be obtained from a server connected via a network that stores the map 2B. The display unit 3 is, for example, a liquid crystal display, a touch screen display in which a touch panel is arranged on the liquid crystal display, or the like.

The processing device 1 may form a processing system together with an imaging device (which generates an image 2A and a distance map 2B) and a display unit 3. The processing system is realized as, for example, a recording device such as a camera or a drive recorder, a smartphone having a camera function, a personal computer having a camera function, a surveillance system, a moving body such as a vehicle, a flying object, or a robot having a camera function. obtain.

As shown in FIG. 1, the processing device 1 has a size calculation unit 11 and an output information generation unit 12.
The size calculation unit 11 is a processing unit that has a function of calculating the size of the subject on the image 2A from the distance map 2B. The size calculation unit 11 may further calculate the size of the subject using the image 2A. The output information generation unit 12 is a processing unit that has a function of generating and outputting output information based on the size of the subject. The output information is, for example, information for displaying on the display unit 3 at the same time as the image 2A.

FIG. 2 is a diagram showing an example of the hardware configuration of the processing device 1.
As shown in FIG. 2, the processing device 1 has a CPU 101, a RAM 102, a non-volatile memory 103, an input / output unit 104, and a communication unit 105, and also has a CPU 101, a RAM 102, a non-volatile memory 103, an input / output unit 104, and communication. It has a bus 106 that connects the parts 105 to each other.

The CPU 101 loads the computer program stored in the non-volatile memory 103 into the RAM 102 and executes it to drive each processing unit of the processing device 1 including the size calculation unit 11 and the output information generation unit 12 shown in FIG. It is a processor that realizes it. Here, an example is shown in which each processing unit of the processing device 1 is realized by one CPU 101, that is, a single processor, but each processing unit of the processing device 1 may be realized by a plurality of processors. Each processing unit of the processing device 1 may be realized by a dedicated electronic circuit. The RAM 102 is a storage medium used as a main storage device, and the non-volatile memory 103 is a storage medium used as an auxiliary storage device.

The input / output unit 104 is a module that executes input / output such as inputting an image 2A and a distance map 2B from an imaging device, inputting an instruction from a user, and outputting a display screen to the display unit 3. be. The instruction from the user may be input according to the operation of the keyboard, the pointing device, the operation button, or the like, and when the display unit 3 is a touch screen display, the instruction is input according to the touch operation on the touch screen display. You may. The communication unit 105 is a module that executes communication with an external device via a network, wireless communication with an external device existing in the vicinity, and the like. The image 2A and the distance map 2B may be acquired by the communication unit 105.

Here, the details of the image 2A and the distance map 2B will be described.
FIG. 3 is a block diagram showing a configuration example of an imaging device that generates an image 2A and a distance map 2B.
As shown in FIG. 3, the image pickup apparatus 100 includes a filter 110, a lens 120, an image sensor 130, and an image processing unit (processing apparatus) 140. In FIG. 3, the arrows from the filter 110 to the image sensor 130 indicate the light path. Further, the arrow from the image sensor 130 to the image processing unit 140 indicates the path of the electric signal. The image processing unit 140 includes the image acquisition unit 141, the distance calculation unit 142, and the second output information generation unit 143, in addition to the size calculation unit 11 and the output information generation unit 12 described above.

The image sensor 130 receives light transmitted through the filter 110 and the lens 120, and converts the received light into an electric signal (photoelectric conversion) to generate an image. For the image sensor 130, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) is used. The image sensor 130 includes, for example, an image sensor (first sensor 131) that receives red (R) light, an image sensor (second sensor 132) that receives green (G) light, and blue (B). ), It has an image sensor (third sensor 33) that receives the light. Each image sensor receives light in the corresponding wavelength band and converts the received light into an electric signal. A color image (image 2A) can be generated by A / D converting this electric signal. It is also possible to generate an R image, a G image, and a B image by using the electric signals of each of the red, green, and blue image pickup elements. That is, a color image, an R image, a G image, and a B image can be generated at the same time. In other words, the image pickup apparatus 100 can obtain a color image, an R image, a G image, and a B image by one imaging. The filter 110, the lens 120, and the image sensor 130 form a single optical system.

The filter 110 has two or more color filter regions. Each color filter region has a shape that is asymmetric with respect to the optical center of the image pickup apparatus. A part of the wavelength region of the light transmitted through one color filter region and a part of the wavelength region of the light transmitted through the other color filter region overlap, for example. The wavelength region of light transmitted through one color filter region may include, for example, the wavelength region of light transmitted through another color filter region.

FIG. 4 shows a configuration example of the filter 110. The filter 110 is composed of, for example, a first filter region 111 and a second filter region 112, which are color filter regions of two colors. The center of the filter 110 coincides with the optical center 113 of the image pickup apparatus 100. The first filter region 111 and the second filter region 112 each have a shape that is astigmatic with respect to the optical center 113. Further, for example, the filter areas 111 and 112 do not overlap, and the two filter areas 111 and 112 constitute the entire area of the filter area. In the example shown in FIG. 4, the first filter region 111 and the second filter region 112 each have a semicircular shape in which the circular filter 110 is divided by a line segment passing through the optical center 113. The first filter region 111 is, for example, a yellow (Y) filter region, and the second filter region 112 is, for example, a cyan (C) filter region. Hereinafter, it will be described assuming that the filter 110 shown in FIG. 4 is used.

For example, by arranging the filter 110 shown in FIG. 4 in the opening of the camera, a color opening which is a structural opening in which the opening is divided into two colors is formed. The image sensor 130 produces an image based on the light rays that pass through this color aperture. A lens 120 may be arranged between the filter 110 and the image sensor 130 on the optical path of the light incident on the image sensor 130. A filter 110 may be arranged between the lens 120 and the image sensor 130 on the optical path of the light incident on the image sensor 130. When a plurality of lenses 120 are provided, the filter 110 may be arranged between the two lenses 120.

Light in the wavelength band corresponding to the second sensor 132 passes through both the yellow first filter region 111 and the cyan second filter region 112. Light in the wavelength band corresponding to the first sensor 131 passes through the yellow first filter region 111 and does not pass through the cyan second filter region 112. Light in the wavelength band corresponding to the third sensor 133 passes through the cyan second filter region 112 and does not pass through the yellow second filter region 112.

When light in a certain wavelength band passes through a filter or a filter region, the filter or the filter region transmits light in the wavelength band with a high transmittance, and the filter or the filter region attenuates the light in the wavelength band (that is, , The decrease in the amount of light) is extremely small. Also, the fact that light in a certain wavelength band does not pass through the filter or filter area means that the light is blocked by the filter or filter area, for example, the filter or filter area transmits light in that wavelength band with low transmittance. However, it means that the attenuation of light in the wavelength band by the filter or the filter region is extremely large. For example, a filter or filter region attenuates light by absorbing light in a certain wavelength band.

FIG. 5 is a diagram showing an example of the transmittance characteristics of the first filter region 111 and the second filter region 112. As shown in FIG. 5, in the transmittance characteristic 151 of the first filter region 111 of yellow, light in the wavelength band corresponding to the R image and the G image is transmitted with high transmittance, and the light in the wavelength band corresponding to the B image is transmitted. Almost no light is transmitted. Further, in the transmittance characteristic 152 of the second filter region 112 of cyan, the light in the wavelength band corresponding to the B image and the G image is transmitted with a high transmittance, and the light in the wavelength band corresponding to the R image is almost transmitted. Not.

Therefore, the light in the wavelength band corresponding to the R image is transmitted only through the first filter region 111 of yellow, and the light in the wavelength band corresponding to the B image is transmitted only through the second filter region 112 of cyan. The shape of the blur on the R image and the B image changes according to the distance d to the subject, and more specifically, according to the difference between the distance d and the focusing distance df. Further, since the filter regions 111 and 112 have shapes that are asymmetric with respect to the optical center, the shape of the blur on the R image and the B image is such that the subject is in front of or behind the focusing distance df. It depends on whether or not. That is, the shape of the blur on the R image and the B image is biased.

With reference to FIG. 6, the light ray change due to the color aperture in which the filter 110 is arranged and the shape of the blur will be described.
When the subject 200 is deeper than the focusing distance df (d> df), the image captured by the image sensor 130 is blurred. The blur function (PSF: Point Spread Function) indicating the shape of the blur of this image is different for the R image, the G image, and the B image, respectively. That is, the blur function 161R of the R image shows the shape of the blur biased to the left side, the blur function 161G of the G image shows the shape of the blur without bias, and the blur function 161B of the B image shows the shape of the blur biased to the right. Shown.

Further, when the subject 200 is at the focusing distance df (d = df), almost no blur occurs in the image captured by the image sensor 130. The blur function indicating the shape of the blur in this image is almost the same in the R image, the G image, and the B image. That is, the blur function 162R of the R image, the blur function 162G of the G image, and the blur function 162B of the B image show an unbiased shape of the blur.

Further, when the subject 200 is in front of the focusing distance df (d <df), the image captured by the image sensor 130 is blurred. The blur function indicating the shape of the blur in this image is different for the R image, the G image, and the B image, respectively. That is, the blur function 103R of the R image shows the shape of the blur biased to the right side, the blur function 163G of the G image shows the shape of the blur without bias, and the blur function 163B of the B image shows the shape of the blur biased to the left. Shown.

The image processing unit 140 of the image pickup apparatus 100 uses such characteristics to calculate the distance to the subject.
The image acquisition unit 141 acquires a G image whose blur function shows an unbiased shape as a reference image. Further, the image acquisition unit 141 acquires one or both of the R image and the B image showing the shape of the blur in which the blur function is biased as the target image. The target image and the reference image are images captured at the same time by one imaging device.

The distance calculation unit 142 calculates the distance to the subject by obtaining a blur correction filter that, when added to the target image, has a higher correlation with the reference image among the plurality of blur correction filters. Further, the distance calculation unit 142 generates a distance map from the calculated distance. The plurality of blur correction filters are functions that add different blurs to the target image. Here, the details of the distance calculation process by the distance calculation unit 142 will be described.

The distance calculation unit 142 generates a corrected image in which the blur shape of the target image is corrected by adding different blurs to the target image based on the acquired target image and the reference image. Here, the distance calculation unit 142 generates a correction image in which the blur shape of the target image is corrected by using a plurality of blur correction filters created on the assumption that the distance to the subject is an arbitrary distance, and the correction image. The distance to the subject is calculated by finding the distance at which the correlation between the image and the reference image becomes higher.

The blurring function of the captured image is determined by the aperture shape of the imaging device 100 and the distance between the position of the subject and the focus position. FIG. 7 is a diagram showing an example of a blur function of the reference image. As shown in FIG. 7, since the aperture shape through which the wavelength region corresponding to the second sensor 132 is transmitted is a circular shape that is point-symmetrical, the shape of the blur indicated by the blur function changes before and after the focus position. However, the width of the blur changes depending on the size of the distance between the subject and the focus position. The blur function showing the shape of such a blur can be expressed as a Gaussian function in which the width of the blur changes depending on the size of the distance between the position of the subject and the focus position. The blur function may be expressed as a pillbox function in which the width of the blur changes depending on the distance between the position of the subject and the focus position.

FIG. 8 is a diagram showing an example of a blur function of the target image. The center of each figure (x ₀ , y ₀ ) = (0, 0). As shown in FIG. 8, the blurring function of the target image (for example, R image) is blurred by the light attenuation in the first filter region 111 at x> 0 when the subject is d> df farther from the focus position. It can be expressed as a Gaussian function that attenuates the width of. Further, when the subject is d <df closer than the focus position, it can be expressed as a Gaussian function in which the width of the blur is attenuated by the light attenuation in the first filter region 111 at x <0.

Further, by analyzing the blur function of the reference image and the blur function of the target image, it is possible to obtain a plurality of blur correction filters for correcting the blur shape of the target image to the blur shape of the reference image.
FIG. 9 is a diagram showing an example of a blur correction filter. The blur correction filter shown in FIG. 9 is a blur correction filter when the filter 110 shown in FIG. 4 is used. As shown in FIG. 9, the blur correction filter passes through the center point of the line segment at the boundary between the first filter region 111 and the second filter region 112, and is distributed on a straight line (near the straight line) orthogonal to this line segment. do. The distribution is a mountain-like distribution as shown in FIG. 9, in which the peak point (position on a straight line, height) and the way of spreading from the peak point are different for each assumed distance. The blur shape of the target image can be corrected to various blur shapes assuming an arbitrary distance by using a blur correction filter. That is, it is possible to generate a corrected image assuming an arbitrary distance.

The distance calculation unit 142 obtains the distance at which the blurred shape of the generated corrected image and the reference image most closely approximates or matches from each pixel of the captured image. The degree of coincidence of the blurred shapes may be determined by calculating the correlation between the corrected image and the reference image in a rectangular region of an arbitrary size centered on each pixel. The existing similarity evaluation method may be used to calculate the degree of coincidence of the blurred shapes. The distance calculation unit 142 calculates the distance to the subject for each pixel by obtaining the distance at which the correlation between the corrected image and the reference image is highest.

For example, existing similarity evaluation methods include SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross-Correlation), ZNCC (Zero-mean Normalized Cross-Correlation), Color Alignment Measure, etc. You can use it. In the present embodiment, the Color Alignment Measure is used, which utilizes the fact that the color components of the natural image have a characteristic of having a locally linear relationship. In the Color Alignment Measure, an index showing the correlation is calculated from the variance of the color distribution in the local boundary region centered on the target pixel of the captured image.

In this way, the distance calculation unit 142 generates a corrected image in which the blur shape of the target image corresponding to the filter area is corrected by the blur correction filter assuming the distance, and the correlation between the generated corrected image and the reference image is improved. The distance to the subject is calculated by finding the higher distance.

Further, the distance calculation unit 142 generates a distance map from the calculated distance. The distance map is generated, for example, as an image in which the pixel value of each pixel indicates a distance. For example, a value indicating a color having a long wavelength (red) to a value indicating a color having a short wavelength (purple) are assigned from the front to the back of the focusing position. As described above, in the distance map, the information indicating the distance to the subject is mapped corresponding to the area of the image, and the pixel value is used as the information indicating the distance to the subject. Since the distance map generated as an image can be displayed, it is possible to confirm the positional relationship in the depth direction between a plurality of subjects, for example, by color. The second output information generation unit 12 generates and outputs output information including the distance map generated by the distance calculation unit 142.

FIG. 10 is a diagram showing an example of the image 2A and the distance map 2B generated by the image pickup apparatus 100.
In FIG. 10, (A) is a display example of the image 2A, and (B) is a display example of the distance map 2B. The distance map in which the pixel value of each pixel indicates a color with a longer wavelength as it is located closer to the in-focus position and a value indicating a color with a shorter wavelength as it is located deeper than the in-focus position is shown in the figure. As shown in (B) of 10, the subject located in the foreground is displayed as an image in which a reddish color is arranged, and the subject located in the back is displayed as an image in which a purplish color is arranged.

Next, the details of the processing of the processing apparatus 1 that acquires the image 2A and the distance map 2B and executes various processing will be described.
If the focusing distance when the image 2A is taken is known, the length of the subject on the captured image and the actual length of the subject are obtained from the ratio of the distance from the optical center to the image center and the distance from the optical center to the object. You can get the ratio of the optics. Further, as described above, in the distance map 2B, since the pixel value indicates the distance, each pixel can be projected (mapped) on the real space (three-dimensional space). The processing device 1, more specifically, the size calculation unit 11 projects (maps) each pixel onto a real space (three-dimensional space), so that, for example, a subject corresponding to two points designated on the image 2A. Get the size of. Further, the output information generation unit 12 generates, for example, output information for displaying the acquired size of the subject on the image 2A, that is, output information for displaying the size of the subject and the image at the same time. , Output to display unit 3. The image 2A is used for recognizing the subject and designating a portion to be measured on the subject when acquiring the size of the subject. In other words, the size calculation unit 11 given the two points to be measured on the image 2A can acquire the size of the subject only by the distance map 2B, and does not require the image 2A. In the following, a case where the size of the subject is acquired by using the image 2A and the distance map 2B will be described.

FIG. 11 is a flowchart showing an example of the processing flow of the processing system including the processing device 1.
First, an image is captured by the imaging device 100 (step A1). The image pickup apparatus 100 generates the image 2A (step A2) and the distance map 2B (step A3). The image 2A and the distance map 2B generated by the image pickup apparatus 100 are acquired by the processing apparatus 1.

The processing device 1 displays, for example, the image 2A on the display unit 3, and receives an instruction to specify any one of a plurality of subjects reflected in the image 2A, an instruction to specify two points on the image 2A, and the like. .. At the same time, this instruction requests the acquisition of the size of the subject. When the acquisition of the size of the subject is requested, the processing device 1 projects each pixel of the distance map 2B onto the three-dimensional space using the distance indicated by each pixel value (step A4). For example, when receiving an instruction to specify a certain subject, the processing device 1 acquires the width and length of the subject from the distance (actual distance) between the pixels projected on the three-dimensional space (step A5). .. The processing device 1 outputs the actual size of the subject to the display unit 3 in order to display it at the same time as the image 2A by, for example, superimposing it on the image 2A (step A6).

FIG. 12 is a diagram showing an example of one output of the size of a subject by a processing system including the processing device 1. It is assumed that this processing system is realized as, for example, an imaging device that captures moving images (30 images per second) at a frame rate of 30 fps.
The processing device 1 sequentially displays the captured images 2A on the display unit 3, and for example, when a certain position on the image 2A is specified, the processing device 1 analyzes the image 2A and recognizes a subject including the specified position. Using the distance map 2B corresponding to the image 2A, at least the pixel in which the subject is captured is projected onto the three-dimensional space among the pixels of the image 2A. The analysis of the image 2A for recognizing the subject including the specified position is not limited to a specific method, and various known methods can be applied. The processing device 1 may acquire, for example, a predetermined direction of the subject or a size between two designated points from the distance between pixels projected on the three-dimensional space. The defined direction and the specified two points may be different for each subject. The processing device 1 may acquire the maximum length or the minimum length of the subject. The size may be displayed on the display unit 3 by superimposing it on the image 2A, for example. For example, the size may be displayed in the vicinity of the subject in the image 2A. Alternatively, when one point in the subject is specified when the subject is specified, the maximum distance among the distances of the line segments from one end to the other end of the subject passing through the one point may be acquired. Further, for example, the size may be determined by recognizing what the subject is and determining the distance between two points according to the type. The processing device 1 may be provided with an interface for setting the portion for which the size should be acquired for each type. After displaying the size of the subject, for example, when a predetermined operation is performed, the processing device 1 returns the display by the display unit 3 to a form in which the latest image 2A is sequentially displayed.

In FIG. 12, during the shooting of a soccer game, the height a2 of the player is on the image 2A as the instruction is given to specify one player a1 from the plurality of players (subjects) shown in the image 2A. An example of being displayed in the vicinity of player a1 is shown.
As described above, according to the processing device 1, it is possible to acquire the size of a moving object such as a player in a match from the image 2A and the distance map 2B acquired by the monocular camera.

Although FIG. 12 shows an example of acquiring and presenting the size of the designated subject, the size of all the subjects reflected in the image 2A may be acquired and presented. In this case, the size of a specified part of the subject in the image 2A may be displayed in a pop-up on the image 2A. In addition to displaying the image 2A on top of each other, for example, the image 2A is reduced and displayed, a separate window is opened next to the reduced image 2A, and the size acquired in this separate window is placed in a list format. Therefore, it may be presented at the same time as the image 2A.

Further, according to the processing device 1, not only the size of the subject but also the moving distance of the subject can be acquired. FIG. 13 is a diagram showing an output example of the moving distance of the subject by the processing system including the processing device 1. It is assumed that this processing system is also realized as an imaging device for capturing moving images.

FIG. 13 shows an example in which the flight distance of a tee shot during a golf game, and more specifically, the movement distance b3 of the golf ball b1 from the tee position b2, is displayed on the image 2A in real time.
If the subject (here, the golf ball b1) for which the moving distance should be acquired and its initial position are clear, the processing device 1 may use the image 2A for each frame of the image captured every 1/30 second, for example. By projecting each pixel, at least the pixel in which the subject appears, into the three-dimensional space based on the distance map 2B, the distance between the initial position and the current position, that is, the moving distance of the subject can be acquired in real time. can. The area to be imaged may be moved so as to track the subject. Further, FIG. 13 illustrates an image 2A that captures a direction that substantially coincides with the direction in which the golf ball b1 moves, but the image 2A is not limited to this and is captured from a position where the subject can be continuously captured. All you need is.

In this way, according to the processing device 1, it is possible to acquire the moving distance of a moving object.
Further, since the moving distance of the subject can be acquired, this processing device 1 can be applied to a motion capture system.

Generally, in a motion capture system, as shown in FIG. 14, sensors c1 and the like for measuring motion are attached to various parts of a subject. On the other hand, the processing system including the processing device 1 does not require such preparation and can measure the movement of the subject. The processing device 1 projects each pixel of the image 2A into a three-dimensional space based on the distance map 2B for each frame of the image captured every 1/30 second, so that the subject is in 1/30 second increments. Can measure the movement of.

As described above, according to this processing device 1, it is possible to realize motion capture.
Further, the function of the processing device 1 can be realized as, for example, a measuring tool which is one of the applications installed on a smartphone having a camera function (which can generate a distance map). For example, if you want to know the size of a product displayed at a store, you can get the size of the product by taking a picture of the product even if you do not carry a tape measure. It is assumed that the touch screen display of the smartphone corresponds to the display unit 3.

For example, suppose you want to measure different sizes of chairs on display at a furniture store. In this case, first, after activating the measuring tool, the image 2A of the chair as shown in FIG. 15 is captured by the camera function of the smartphone. The measuring tool displays the image 2A on the touch screen display of the smartphone. Further, the measuring tool projects each pixel of the image 2A onto the three-dimensional space based on the distance map 2B.

The user can know the distance between two points by designating one end and the other end. When it is desired to measure the width of the backrest of the chair, the user performs a touch operation on the touch screen display, for example, by touching one end (d1) of the backrest in the lateral direction and then touching the other end (d2). As a method of designating two points on the image 2A, various methods can be adopted. For example, a bar may be displayed on a touch screen display, and the bar may be expanded or contracted so that the tip and end of the bar correspond to both ends (d1, d2) in the lateral direction of the backrest. When two points on image 2A are specified, the measuring tool obtains the actual size between the specified two points using the coordinates of the two pixels projected on the three-dimensional space in the three-dimensional space. Then, for example, the size is displayed on the touch screen display of the smartphone so as to be superimposed on the image 2A (d3).

Further, the distance measured when two points are specified on the image 2A is not limited to the linear distance between the two points. For example, when two points on the curved outer surface of the subject are specified, the outer peripheral distance along the curved outer surface can be acquired. The outer peripheral distance is, for example, the length of the shortest line connecting two points along a curved outer surface. The outer peripheral distance can be obtained by summing the distances between adjacent pixels on the line connecting the two points. The measuring tool may include a first mode for measuring the linear distance between two points and a second mode for measuring the outer peripheral distance between the two points. Based on the user's mode setting through the input unit, either the linear distance or the outer peripheral distance can be acquired and displayed. In addition, the measuring tool may acquire and display both the linear distance and the outer peripheral distance. A third mode for measuring the linear distance and the outer peripheral distance between two points is further provided, and when the third mode is set, both the linear distance and the outer peripheral distance may be acquired and displayed.

Further, for example, a reference may be provided together with a measuring tool, or a size-standardized daily necessities may be used as a reference for imaging together with an article whose size is desired to be measured (d4). By acquiring the reference size from the image 2A, it is possible to perform calibration that absorbs individual differences in the camera functions of the smartphone. The correction value for calibration may be given to the measuring tool in advance as a parameter at the time of shipment of the smartphone, for example.

As described above, according to the function of the processing device 1, the size of various articles such as the products displayed at the store can be acquired from the image 2A and the distance map 2B acquired by the monocular camera. The tool can be realized.
Further, this processing device 1 can be applied to a monitoring system.

FIG. 16 is an image 2A captured by a surveillance camera (which can generate a distance map 2B) installed to monitor a pedestrian on a passage in a commercial facility, and is an image 2A captured by a processing device 1 for a pedestrian (a distance map 2B can be generated). An example of image 2A in which e1, e2, e3) are recognized and the heights (e11, e21, e31) of each pedestrian acquired by the processing device 1 are superimposed and displayed is shown. For example, when a security guard finds a person who seems to be a suspicious person on the image 2A, he / she can immediately obtain the height, which is one of the influential information indicating the characteristics of the person.

Further, for example, when an article that seems to be a knife is recognized, if the blade length exceeds a predetermined length instead of simply performing some processing based on the recognition, a warning display by the display unit 3 or a voice is displayed. If the output unit is connected, the warning sound may be output by the voice output unit. Alternatively, when a predetermined article including a knife is recognized, the length of a predetermined portion of the article may be acquired and displayed. When no suspicious point is found in the pedestrian or the article carried by the pedestrian, the display unit 3 may display to that effect, or the audio output unit may output a voice to that effect.

Further, the processing device 1 does not always recognize a pedestrian or an article carried by a pedestrian (such as the above-mentioned knife) and acquire the size, but is special when the user specifies a position on the image 2A. The mode may be switched to, and during the period of the special mode, the subject including the specified position may be recognized, tracked, and the size may be acquired and displayed. The switching from the special mode to the normal mode may be performed, for example, when the target subject disappears from the image 2A or when a predetermined operation is performed.

As described above, according to the processing device 1, the size of the pedestrian or the article carried by the pedestrian can be acquired from the image 2A and the distance map 2B acquired by the monocular camera, and a warning is given according to the acquired size. It is possible to realize a monitoring system that executes processing.
Further, the processing device 1 can be applied to a support system that supports the driving and maneuvering of a moving body such as an automobile.

For example, as shown in FIG. 17, it is assumed that the moving body is an automobile and the image pickup device 100 is mounted so as to image the traveling direction thereof. Further, it is assumed that a step f1 exists in the traveling direction of the automobile.
In such a case, the processing device 1 measures the step f1 from the image 2A and the distance map 2B generated by the image pickup device 100, determines whether or not the automobile can be overcome, and displays the determination result on the display unit 3. Present to the driver. If the vehicle cannot be overcome, a warning sound may be output from the sound output unit of the automobile.

Further, it is assumed that the gate g1 exists in the traveling direction of the automobile, for example, as shown in FIG. In such a case, the processing device 1 measures the width g2 between the subject g1-1 and the subject g1-2 from the image 2A and the distance map 2B generated by the image pickup device 100, and whether or not the automobile can pass through. Is determined, and the determination result is presented to the driver by the display unit 3. If the vehicle cannot pass through, a warning sound may be output from a sound output unit provided in the automobile.

As described above, according to the processing device 1, it is possible to perform processing for supporting the driving and maneuvering of the moving body from the image 2A and the distance map 2B acquired by the monocular camera. Considering that the processing device 1 is mounted on an automobile, the processing device 1 may be realized as a recording device such as a drive recorder.

Further, the processing device 1 can be applied to an automatic door system.
For example, as shown in FIG. 19, an automatic door in which a revolving door continues to rotate at a constant speed is assumed. In the case of revolving doors, it is difficult for passers-by to know how large the door can pass. Therefore, for example, the size of the luggage (subject) of a passerby is acquired from the image 2A and the distance map 2B generated by the image pickup device 100, and when the size exceeds the size that can pass through the revolving door, the sound output unit provided on the revolving door is used. Output a warning voice. When applied to the automatic door system, the processing device 1 may include an audio output unit 3-2 instead of the display unit 3 of FIG. 1, as shown in FIG. Both the display unit 3 and the audio output unit 3-2 may be provided. The same applies to the monitoring system and support system described above. Since the processing device 1 can track the movement of the subject, it is possible to adaptively acquire the size and determine whether or not the passage is possible only when the subject is moving toward the revolving door. Therefore, even if a subject exceeding the size that can be passed crosses in front of the revolving door, it is possible to prevent an erroneous output of a warning sound.

When an obstacle in the traveling direction of the moving body such as the step f1 in FIG. 17 or the gate g1 in FIG. 18 moves, the processing device 1 may acquire information on the obstacle at any time. The information on the obstacle is, for example, the shape of the obstacle and the width of the portion where the obstacle and the path through which the moving body passes overlap. The determination unit 13 can determine whether or not the vehicle can pass according to the time change of the obstacle. For example, when the shape of the obstacle changes after determining that the obstacle can be passed, the determination unit 13 may determine that the obstacle cannot be passed. Alternatively, if the shape of the obstacle changes after determining that the obstacle cannot be passed, the determination unit 13 may determine that the obstacle can be passed.

As described above, according to the processing device 1, it is possible to perform processing for preventing an accident at the automatic door from the image 2A and the distance map 2B acquired by the monocular camera.
Further, since the processing device 1 can acquire the size of the subject from the image 2A and the distance map 2B acquired by the monocular camera, the weight of the imaging device 100 can be reduced as compared with, for example, a compound eye camera. In addition, cost reduction can be achieved. Weight reduction is an important matter when mounted on a flying object such as a drone having a small maximum load capacity, and in this respect, the image pickup device 100, which is a monocular camera, is more suitable than a compound eye camera. By mounting the image pickup device 100 on a flying object such as a drone, the processing device 1 can be applied to a support system that supports, for example, inspection work of a structure. Further, in comparison with the compound eye camera, the image pickup device 100, which is a monocular camera, does not cause parallax, which is a problem in the compound eye camera, so that the size acquisition accuracy can be improved. It is also possible to calculate the size using the image acquired by the compound eye camera and the distance map. Since the three-dimensional shape of the subject can be acquired based on the image acquired by the compound eye camera, it is possible to obtain the distance between any two points on the surface of the subject.

For example, a function for acquiring position information such as a GPS [global positioning system] receiver and an altitude sensor is mounted on a drone together with an image pickup device 100, and the drone is flown around a structure to be inspected to image an image of the outer surface of the structure. 2A and distance map 2B are acquired and recorded in association with the position information. For example, when a defective portion on the outer surface of a structure is found on image 2A, its position can be specified, and its scale and shape can be confirmed.

Alternatively, based on the three-dimensional information about the structure to be inspected, the drone is flown to acquire the image 2A and the distance map 2B of the outer surface of the structure, and record the image 2A and the distance map 2B in association with the three-dimensional information. Also in this case, for example, when a defective portion on the outer surface of the structure is found on the image 2A, the position can be specified and the scale and shape can be confirmed.

In addition, the distance map 2B of the image 2A captured during the previous inspection is compared with the distance map 2B of the image 2A captured during the current inspection, and if a difference of more than a predetermined value is detected, the position is determined. By displaying the corresponding image 2A in an identifiable form, for example, a missing bolt can be found without overlooking. Alternatively, by comparing the distance map 2B of the image 2A captured after the earthquake and before the earthquake with the distance map 2B of the image 2A captured after the earthquake, the damage to the structure can be accurately described. Can be grasped.

In addition, the processing device 1 can be applied to various uses by flying the drone and acquiring the image 2A and the distance map 2B. For example, it can be applied to an application such as flying a drone along an electric wire, acquiring an image 2A of the electric wire and a distance map 2B, and investigating how many meters of the electric wire are stretched in what state.

(Second Embodiment)
Next, the second embodiment will be described. Hereinafter, the same reference numerals will be used for the same configurations as those in the first embodiment, and duplicate description of the same configurations will be omitted.
FIG. 21 is a diagram showing an example of a functional block of the processing device of the second embodiment.

The processing device 1 calculates the size of the subject on the image 2A from the acquired image 2A and the distance map (distance image) 2B, and controls the driving of the driving unit 4 of the moving body based on the calculated size of the subject. .. In the present embodiment, the output information is a control signal that controls the movement of a part or all of the processing system.
As shown in FIG. 21, the processing device 1-2 includes a size calculation unit 11, a determination unit 13, and a moving body control unit 14.

The size calculation unit 11 is a processing unit that has a function of calculating the size of the subject on the image 2A from the image 2A and the distance map 2B. The determination unit 13 is a processing unit that has a function of determining how to drive the drive unit 4 based on the size of the subject. The mobile body control unit 14 is a processing unit that has a function of controlling the drive of the drive unit 4 based on the determination of the determination unit 13. The hardware configuration of the processing device 1-2 is the same as that of the processing device 1 of the first embodiment, and each processing unit of the processing device 1-2 is also realized by, for example, a single processor or a plurality of processors. A display unit and / or an audio output unit may be connected to the processing device 1-2. The display unit and / or the audio output unit is connected to, for example, the determination unit 13.

The processing device 1-2 may form a processing system together with the imaging device (which generates the image 2A and the distance map 2B) and the driving unit 4. The processing system can be realized, for example, as a moving body such as a vehicle, a flying object, or a robot having a camera function.
First, an example of processing when the processing device 1-2 is applied to a support system that supports driving and maneuvering of a moving body such as an automobile will be described.

For example, it is assumed that the moving body is an automobile and the image pickup device 100 is mounted so as to image the traveling direction thereof. Further, as in the case described in the first embodiment, it is assumed that a step f1 exists in the traveling direction of the automobile as shown in FIG.
In such a case, the size calculation unit 11 of the processing device 1-2 measures the step f1 from the image 2A and the distance map 2B generated by the image pickup device 100. The determination unit 13 determines whether or not the vehicle can overcome the step f1, and if it cannot overcome the step f1, for example, sends a signal to the moving body control unit 14 for stopping the vehicle or changing the traveling direction of the vehicle. Send. Upon receiving this signal, the mobile control unit 14 controls the drive of the drive unit 4 so as to stop the vehicle or change the traveling direction of the vehicle, for example.

Further, as in the case described in the first embodiment, it is assumed that the gate g1 exists in the traveling direction of the automobile as shown in FIG. The size calculation unit 11 measures the width g2 of the gate g1 from the image 2A and the distance map 2B generated by the image pickup apparatus 100. The determination unit 13 determines whether or not the vehicle can pass through the width g2, and if it cannot pass, for example, transmits a signal for stopping the vehicle or changing the traveling direction of the vehicle to the moving body control unit 14. do. Upon receiving this signal, the mobile control unit 14 controls the drive of the drive unit 4 so as to stop the vehicle or change the traveling direction of the vehicle, for example.

Alternatively, when the determination unit 13 determines that the width g2 of the gate g1 is a size that the automobile can pass through by folding the side mirror of the automobile, the determination unit 13 transmits a signal for folding the side mirror to the moving body control unit 14. You may try to do it. The mobile control unit 14 that has received this signal controls the drive of the drive unit 4 so as to fold the side mirrors.

Next, an example of processing when this processing device 1-2 is applied to an automatic door system will be described.
For example, as in the case described in the first embodiment, as shown in FIG. 19, an automatic door in which the revolving door continues to rotate at a constant speed is assumed. The size calculation unit 11 acquires, for example, the size of a passerby's luggage (subject) from the image 2A and the distance map 2B generated by the image pickup apparatus 100. The determination unit 13 determines whether or not the size can pass through the revolving door, and if it exceeds the size that can pass through the revolving door, for example, transmits a signal for stopping the rotation of the automatic door to the moving body control unit 14. do. The mobile control unit 14 that has received this signal controls the drive of the drive unit 4 so as to stop the rotation of the automatic door, for example.

Next, an example of processing when this processing device 1-2 is applied to a robot will be described. Here, as shown in FIG. 21, for example, it is assumed that the robot has a robot arm as a drive unit 4 that picks up an object h1 transported on a transport line and sorts it according to its size.

The size calculation unit 11 acquires the size of the object h1 from the image 2A and the distance map 2B generated by the image pickup apparatus 100. The determination unit 13 first determines the size of the object h1 that can be picked up, and if it is the size that can be picked up, secondly determines the sorting destination. If the size is not pickable (including both smaller and larger than the permissible range), the mobile control unit 14 drives the drive unit 4 to perform an operation other than picking up the object h1. May be controlled, a warning may be displayed on the display unit, or a warning voice may be output from the voice output unit. Further, the determination unit 13 has a size that can be picked up, and when determining the sorting destination, the determination unit 13 transmits a signal for carrying the object h1 to the sorting destination to the mobile control unit 14. The mobile body control unit 14 that has received this signal controls the drive of the drive unit 4 so as to move the object h1 to the instructed place.

Alternatively, it is also possible to arrange a plurality of robots along the transport line and acquire the size of the object h1 so that each of them picks up only the object h1 having a predetermined range of sizes.
The robot can be realized not only for industrial use but also for home use such as a cleaning robot that moves autonomously to clean the floor. In the case of a cleaning robot, by applying this processing device 1-2, for example, it is determined whether or not the size of the dust can pass through the suction port, and if the size is such that the dust may clog the suction port when sucked. It is possible to control such as pausing the suction and passing the place or changing the movement route. In addition, in devices that move autonomously such as cleaning robots, a self-position estimation technology called SLAM (Simultaneous localization and mapping) has recently attracted attention, but from image 2A and distance map 2B to the subject. The processing device 1-2 capable of acquiring the distance of can be applied to the self-position estimation by this SLAM.

(Third Embodiment)
Next, the third embodiment will be described. Hereinafter, the same reference numerals will be used for the same configurations as those of the first embodiment or the second embodiment, and duplicate description of the same configurations will be omitted.
FIG. 23 is a diagram showing an example of a functional block of the processing device of the third embodiment.

The processing device 1 calculates the size of the subject on the image 2A from the acquired image 2A and the distance map (distance image) 2B, and executes communication with the obstacle 5 based on the calculated size of the subject. ..
As shown in FIG. 23, the processing device 1-3 includes a size calculation unit 11, a determination unit 13, and a signal transmission unit 15.

The size calculation unit 11 is a processing unit that has a function of calculating the size of the subject on the image 2A from the image 2A and the distance map 2B. The determination unit 13 is a processing unit that has a function of determining how to drive the drive unit 4 based on the size of the subject. The mobile control unit 14 is a processing unit that has a function of executing communication with the obstacle 5 based on the judgment of the determination unit 13. The hardware configuration of the processing devices 1-3 is the same as that of the processing device 1 of the first embodiment and the processing device 1-2 of the second embodiment, and each processing unit of the processing device 1-3 is also, for example, single or single. It is realized by multiple processors.

In addition, the processing devices 1-3 may form a processing system together with an imaging device (which generates an image 2A and a distance map 2B). The processing system can be realized, for example, as a moving body such as a vehicle, a flying object, or a robot having a camera function.
Now, it is assumed that this processing device 1-2 is applied to a support system that supports the driving and maneuvering of a moving body such as an automobile, and that the image pickup device 100 is mounted so as to image the traveling direction thereof. .. Further, as shown in FIG. 24, it is assumed that while traveling on a certain road, another automobile (obstacle j1) is stopped on the road.

In such a case, the size calculation unit 11 of the processing device 1-3 measures the width j2 of the space beside the other automobile (obstacle j1) from the image 2A and the distance map 2B generated by the image pickup device 100. .. The determination unit 13 determines whether or not the vehicle can pass through the width j2, and if it cannot pass, transmits a signal to the signal transmission unit 15 for urging another vehicle (obstacle j1) to move. Upon receiving this signal, the signal transmission unit 15 outputs a signal prompting the other automobile (obstacle j1) to move.

Alternatively, when another vehicle (obstacle j1) is traveling on the road as an oncoming vehicle, the size of the other vehicle (obstacle j1) or the size of the space beside the other vehicle (obstacle j1) is acquired. If it is determined that they cannot pass each other on the road, it is possible to prevent contact accidents by notifying other automobiles (obstacle j1) as soon as possible. The determination unit 13 may acquire the size of another automobile (obstacle j1) or the size of the space beside the other automobile (obstacle j1) at any time. For example, the determination unit 13 may be trying to get off the passenger (driver only) of another vehicle (obstacle j1) when the door of the other vehicle (obstacle j1) is opened and the width is increased. However, a signal for calling attention of a passenger may be included) may be output from the signal transmission unit 15 to another automobile (obstacle j1). By acquiring the size of another car (obstacle j1) or the size of the space beside the other car (obstacle j1) at any time, the door of the other car (obstacle j1) is closed and they pass each other. It is possible to adapt to the case where the door of another automobile (obstacle j1) is opened and the person cannot pass each other.

As described above, according to the first to third embodiments, the size of the subject can be acquired by the monocular camera.
Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

1,1-2,1-3 ... Processing device, 2A ... Image, 2B ... Distance map (distance image), 3 ... Display unit, 4 ... Drive unit, 5 ... Obstacle, 11 ... Size calculation unit, 12 ... Output Information generation unit, 13 ... Judgment unit, 14 ... Moving object control unit, 15 ... Signal transmission unit, 100 ... Imaging device, 110 ... Filter, 111 ... First filter area, 112 ... Second filter area, 113 ... Optical Center, 120 ... lens, 130 ... image sensor, 131 ... first sensor, 132 ... second sensor, 133 ... third sensor, 140 ... image processing unit, 141 ... image acquisition unit, 142 ... distance calculation unit, 143 ... Output unit.

Claims (8)

A distance map acquired together with an image by a single imaging with a single imaging optical system, in which the pixel value of each pixel is generated as an image indicating the distance, and information indicating the distance to the subject included in the image. Projects each pixel of the distance map mapped in correspondence with the image onto the three-dimensional space using the distance indicated by each pixel value, and from the distance between the pixels projected on the three-dimensional space. A size calculation unit that calculates the size of the subject,
A judgment unit that determines the driving method of the vehicle based on the size of the subject,
A processing system for vehicles equipped with. The vehicle processing system according to claim 1, wherein the distance map is obtained by using the blur function included in the image. The image and the distance map according to claim 1 or 2 obtained by the single optical system capable of producing an image including a first wavelength component having a symmetric blur function and a second wavelength component having an asymmetric blur function. The vehicle processing system described. The vehicle processing system according to any one of claims 1 to 3, wherein the determination unit can determine to transmit a signal for urging another vehicle to move. The determination unit acquires the size of another vehicle or the size of the space beside the other vehicle on the road, and determines whether or not the vehicle can pass the other vehicle on the road. The vehicle processing system according to any one of claims 1 to 4. When the determination unit determines that the vehicle cannot pass the other vehicle on the road, the determination unit informs the other vehicle that the vehicle and the other vehicle cannot pass each other on the road. The vehicle processing system according to claim 5, wherein it is determined as the driving method of the vehicle to notify the vehicle. The vehicle processing system according to any one of claims 1 to 6, wherein the determination unit acquires the size of the step and determines whether or not the vehicle can overcome the step. The determination unit obtains the size between the first subject and the second subject, and determines whether or not the vehicle can pass between the first subject and the second subject. Item 6. The vehicle processing system according to any one of Items 1 to 7.

Images