JP7008308B2 - Image processing device, ranging device, image pickup device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, a distance measuring device, an imaging device, an image processing method and a program.

Various methods for detecting moving objects in images are being studied. Patent Document 1 discloses a method of detecting a moving object using a plurality of frames constituting a moving image. In the method of Patent Document 1, the difference A between the frame Fr ₀ and the frame Fr ₁ and the difference B between the frame Fr ₁ and the frame Fr ₂ are calculated for three consecutive frames Fr ₀ , Fr ₁ , and Fr ₂ . By acquiring the logical sum of the difference A and the difference B, a moving object in the image can be detected.

Japanese Unexamined Patent Publication No. 2004-171431

In the method for detecting a moving object as described in Patent Document 1, it is necessary to use a plurality of temporally continuous images such as a frame of a moving image for processing. Therefore, it is not possible to extract a moving object from one image. Can not.

The present invention has been made in view of the above-mentioned problems, and provides an image processing device, a distance measuring device, an image pickup device, an image processing method and a program capable of extracting a moving object from one image. The purpose.

According to one aspect of the present invention, a first acquisition unit for acquiring an instantaneous velocity distribution of an object in a first region in space and a second acquisition unit for acquiring an image including at least a part of the first region. And an image processing apparatus characterized by having a detection unit for detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution.

According to another aspect of the present invention, a step of acquiring an instantaneous velocity distribution of an object in a first region in space, a step of acquiring an image including at least a part of the first region, and the instantaneous velocity. Provided is an image processing method comprising: a step of detecting a second region indicating a moving object included in the image based on a distribution.

According to still another aspect of the present invention, a step of acquiring an instantaneous velocity distribution of an object in a first region in space and a step of acquiring an image including at least a part of the first region are obtained by a computer. , A program comprising the step of detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution and the execution of the step is provided.

According to the present invention, it is possible to provide an image processing device, a distance measuring device, an image pickup device, an image processing method and a program capable of extracting a moving object from one image.

It is a schematic diagram which shows the schematic structure of the object detection system including the image processing apparatus which concerns on 1st Embodiment. It is a block diagram which shows the hardware configuration example of the image processing apparatus which concerns on 1st Embodiment. It is a functional block diagram of the image processing apparatus which concerns on 1st Embodiment. It is a functional block diagram of the ranging device which concerns on 1st Embodiment. It is a functional block diagram of the image pickup apparatus which concerns on 1st Embodiment. It is a sequence diagram which shows the outline of the processing performed by the image processing apparatus, the distance measuring apparatus and the image pickup apparatus which concerns on 1st Embodiment. It is a flowchart which shows the outline of the process performed by the distance measuring apparatus which concerns on 1st Embodiment. It is a graph which shows the principle of distance measurement. It is a graph which shows the principle of instantaneous velocity measurement. It is a graph which shows the principle of instantaneous velocity measurement. It is a flowchart which shows the outline of the processing performed by the image processing apparatus which concerns on 1st Embodiment. It is a schematic diagram which shows the outline of the trimming process. It is a block diagram which shows the schematic structure of the object detection system which concerns on 2nd Embodiment. It is a block diagram which shows the schematic structure of the object detection system which concerns on 3rd Embodiment. It is a functional block diagram of the image processing apparatus which concerns on 4th Embodiment.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. Similar elements or corresponding elements may be designated by the same reference numerals in the drawings, and the description thereof may be omitted or simplified.

[First Embodiment]
FIG. 1 is a schematic diagram showing a schematic configuration of an object detection system 1 including an image processing device 10 according to the first embodiment. The object detection system 1 includes an image processing device 10, a distance measuring device 20, and an image pickup device 30.

The rangefinding device 20 is, for example, a LiDAR (Light Detection and Ranging) device, and can acquire the distribution of the distance, the instantaneous speed, and the like from the rangefinding device 20 in a predetermined range. The image pickup device 30 is, for example, a digital camera, and can acquire an image in a predetermined range. With these functions, the object detection system 1 can detect the object 2.

The image processing device 10 is, for example, a computer, and processes an image acquired by the image pickup device 30 based on the information acquired by the distance measuring device 20. The image processing device 10 and the distance measuring device 20 and the image processing device 10 and the image pickup device 30 are communicably connected by wire or wirelessly.

The object detection system 1 may further include a control device (not shown) that collectively controls the image processing device 10, the distance measuring device 20, and the image pickup device 30, and a computer functioning as the image processing device 10 performs the control function. You may have. Further, the object detection system 1 may be configured by a plurality of devices connected so as to be communicable, or may be configured as one device.

FIG. 2 is a block diagram showing a hardware configuration example of the image processing apparatus 10 according to the first embodiment of the present invention. The image processing device 10 may be, for example, a computer integrally configured with the distance measuring device 20 and the image pickup device 30. Further, the image processing device 10 may be a computer configured as a device different from the distance measuring device 20 and the image pickup device 30.

The image processing device 10 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and an HDD (Hard Disk Drive) 104 as a computer that performs calculation, control, and storage. Further, the image processing device 10 includes a communication I / F (interface) 105, a display device 106, and an input device 107. The CPU 101, RAM 102, ROM 103, HDD 104, communication I / F 105, display device 106, and input device 107 are connected to each other via the bus 108. The display device 106 and the input device 107 may be connected to the bus 108 via a drive device (not shown) for driving these devices.

In FIG. 2, each part constituting the image processing device 10 is shown as an integrated device, but some of these functions may be provided by an external device. For example, the display device 106 and the input device 107 may be external devices different from the parts constituting the functions of the computer including the CPU 101 and the like.

The CPU 101 performs a predetermined operation according to a program stored in the ROM 103, the HDD 104, and the like, and also has a function of controlling each part of the image processing device 10. The RAM 102 is composed of a volatile storage medium and provides a temporary memory area necessary for the operation of the CPU 101. The ROM 103 is composed of a non-volatile storage medium and stores necessary information such as a program used for the operation of the image processing device 10. The HDD 104 is a storage device composed of a non-volatile storage medium and storing image data, programs, and the like.

The communication I / F 105 is a communication interface based on a standard such as Wi-Fi (registered trademark) and 4G, and is a module for communicating with other devices. The display device 106 is a liquid crystal display, an OLED (Organic Light Emitting Diode) display, or the like, and is used for displaying images, characters, interfaces, and the like. The input device 107 is a keyboard, a pointing device, or the like, and is used for the user to operate the image processing device 10. Examples of pointing devices include mice, trackballs, touch panels and the like. The display device 106 and the input device 107 may be integrally formed as a touch panel.

The hardware configuration shown in FIG. 1 is an example, and devices other than these may be added or some devices may not be provided. Further, some devices may be replaced with other devices having similar functions. Further, some functions of the present embodiment may be provided by other devices via a network, or the functions of the present embodiment may be distributed and realized by a plurality of devices. For example, the HDD 104 may be replaced with an SSD (Solid State Drive) using a semiconductor memory, or may be replaced with a cloud storage.

FIG. 3 is a functional block diagram of the image processing apparatus 10 according to the present embodiment. The image processing device 10 includes a first acquisition unit 121, a second acquisition unit 122, a detection unit 123, a coordinate setting unit 124, a trimming processing unit 125, and a storage unit 126.

The CPU 101 realizes the functions of the first acquisition unit 121, the second acquisition unit 122, the detection unit 123, the coordinate setting unit 124, and the trimming processing unit 125 by loading the program stored in the ROM 103 or the like into the RAM 102 and executing the program. do. The processing performed in each of these parts will be described later. The CPU 101 realizes the function of the storage unit 126 by controlling the HDD 104. The storage unit 126 stores information such as the distance acquired from the distance measuring device 20, the distribution of the instantaneous velocity, and the image acquired from the imaging device 30.

FIG. 4 is a functional block diagram of the distance measuring device 20 according to the present embodiment. FIG. 4 shows a block diagram of an FMCW (Frequency Modulated Continuous Wave) type LiDAR device as an example of the configuration of the distance measuring device 20. The LiDAR device is a device that acquires reflected light intensity distribution, distance distribution, etc. by projecting laser light such as infrared light, visible light, and ultraviolet light and scanning a predetermined range by repeating the operation of acquiring reflected light. Is. Further, the FMCW method is applied to the distance measuring device 20 shown in FIG. Therefore, the distance measuring device 20 can further acquire the instantaneous velocity distribution by measuring the frequency change due to the Doppler effect. LiDAR is sometimes called a laser radar. Further, since the Doppler effect is used for measuring the frequency change, the instantaneous velocity distribution indicates the instantaneous velocity in the direction from the object 2 toward the distance measuring device 20 or in the direction away from the distance measuring device 20. Is desirable.

The distance measuring device 20 includes a reflecting mirror unit 201, LD (Laser Diode) 204, PD (Photodiode) 205, 206, filters 207, 208, 209, modulator 210, demodulators 211, 212, triangular wave generator 213, and amplitude. It has a period comparator 214, 215, a motor control unit 216, and a calculation unit 217. The reflector unit 201 has a reflector 202 and a motor 203.

The triangular wave generator 213 generates a triangular wave in which the voltage repeatedly increases and decreases with time. The triangular wave generated by the triangular wave generator 213 is output to the modulator 210. Further, the triangular wave is also output to the amplitude / period comparator 214, 215 and the calculation unit 217 as a reference signal for referring to the amplitude / period and the like.

The modulator 210 includes a VCO (Voltage-Controlled Oscillator) and the like, and generates a frequency modulated wave corresponding to an input of a triangular wave-shaped voltage generated by the triangular wave generator 213. The generated frequency modulated wave is input to the filter 207. The filter 207 is a band-passing filter whose passband is the frequency of the frequency-modulated wave. The frequency-modulated wave that has passed through the filter 207 is input to the LD204. The LD204 produces a laser beam based on the input frequency modulated wave. The LD204 is, for example, a light emitting element for infrared communication that emits a laser beam having a wavelength in the near infrared region.

The laser beam emitted from the LD 204 is incident on the reflecting mirror unit 201. The reflecting mirror 202 in the reflecting mirror unit 201 reflects the incident laser light and changes the direction in which the laser light is projected. The motor 203 is, for example, a DC (Direct Current) motor with an encoder, and rotates the reflector 202. The reflector 202 is rotationally driven by the motor 203 so that the laser beam can be scanned within a predetermined range. A part of the laser light is incident on the PD 205 as reference light, and the other part is projected to the outside of the distance measuring device 20.

When the laser beam projected to the outside of the distance measuring device 20 is reflected by the object 2 and is incident on the distance measuring device 20 again, the reflected light is incident on the PD 206. Assuming that the distance between the object 2 and the distance measuring device 20 is r, the reflected light has an optical path longer than the reference light by 2r. Therefore, the time when the reflected light is incident on the PD206 is 2r / c later than the time when the reference light is incident on the PD205, where c is the speed of light.

PD205 and 206 are, for example, photoelectric conversion elements for infrared communication that receive light having a wavelength similar to that of LD204 and convert it into electric charges. When light is incident on PD205 and 206, the change in voltage based on the generated charge is transmitted to the subsequent filters 208 and 209 as an electric signal. Similar to the filter 207, the filters 208 and 209 are bandpass filters having the frequency of the frequency modulated wave generated by the triangular wave generator 213 as the passband. The frequency-modulated wave that has passed through the filter 208 is input to the demodulator 211, and the frequency-modulated wave that has passed through the filter 209 is input to the demodulator 212.

The demodulators 211 and 212 include a PLL (Phase-Locked Loop) and the like, and demodulate the input frequency-modulated wave. Since the frequency-modulated wave is based on the triangular wave generated by the triangular wave generator 213, the signal demodulated by the demodulators 211 and 212 becomes a triangular wave. The triangular wave obtained by demodulation in the demodulator 211 is input to the amplitude / period comparator 214, and the triangular wave obtained by demodulation in the demodulator 212 is input to the amplitude / period comparator 215.

The amplitude / period comparators 214 and 215 include a mixer and the like that generate a beat signal. The amplitude / period comparator 214 compares the amplitude / period of the triangular wave output from the triangular wave generator 213 with the amplitude / period of the triangular wave output from the demodulator 211. The comparison result of the amplitude / period comparator 214 is output to the calculation unit 217. The amplitude / period comparator 215 compares the amplitude / period of the triangular wave output from the triangular wave generator 213 with the amplitude / period of the triangular wave output from the demodulator 212. The comparison result of the amplitude / period comparator 215 is output to the calculation unit 217. Here, the comparison result may be the difference or ratio of the amplitude / period of the two input signals.

The calculation unit 217 uses the signal based on the reference light output from the amplitude / period comparator 214 and the triangular wave output from the triangle wave generator 213, and is based on the reflected light output from the amplitude / period comparator 215. Performs an operation to correct the signal. As a result, the calculation unit 217 calculates the intensity of the reflected light, the distance between the distance measuring device 20 and the object 2, and the instantaneous speed of the object 2. The distance measuring device 20 scans the laser beam within a predetermined range to measure the intensity, distance, and instantaneous velocity of the reflected light, and images these as a two-dimensional reflected light intensity distribution, distance distribution, and instantaneous velocity distribution. Output to the processing device 10.

The information of the reference light output from the amplitude / period comparator 214 is also output to the motor control unit 216. The motor control unit 216 calculates the current position, rotation speed, etc. of the reflector 202 based on the information acquired from the amplitude / period comparator 214 and the information acquired from the encoder provided in the motor 203. The motor control unit 216 controls the rotation speed of the motor 203 to be increased or decreased based on information such as the current position of the reflector 202 and the rotation speed, thereby stabilizing the rotation speed of the reflector 202 at a predetermined value. To make it.

Filters 207, 208, 209, modulator 210, demodulators 211, 212, triangle wave generator 213, amplitude / period comparator 214, 215, motor control unit 216, and part or all of the calculation unit 217 are integrated circuits. It may be formed. Here, the integrated circuit may be an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

FIG. 5 is a functional block diagram of the image pickup apparatus 30 according to the present embodiment. FIG. 5 shows a block diagram of a digital still camera in which a lens and a main body are integrated as an example of the configuration of the image pickup apparatus 30. The image pickup apparatus 30 acquires an image of a predetermined range (predetermined angle of view) including the object 2 as digital data.

The image pickup device 30 includes a lens unit 301, an image pickup element 302, a signal processing unit 303, an image pickup device control unit 304, an external I / F 305, and a recording medium 306. The image pickup device control unit 304 controls the entire image pickup device 30. The lens unit 301 is an optical system that guides the light from the object 2 to the image pickup device 302, and includes one or more lenses. Further, the lens unit 301 may further include an optical member such as a diaphragm, a zoom lens, an optical filter, and a mechanical shutter.

The image pickup device 302 may be a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, or the like. A plurality of pixels are arranged two-dimensionally on the image pickup surface of the image pickup element 302. The image pickup device 302 generates an image signal based on an optical image having a predetermined angle of view projected on the image pickup surface.

The signal processing unit 303 may be a processor such as a DSP (Digital Signal Processor). The signal processing unit 303 performs processing such as correction and compression on the image signal generated by the image sensor 302 to generate image data. The generated image data is stored in a non-volatile recording medium 306 configured by a flash memory or the like. The external I / F 305 outputs the image data stored in the recording medium 306 to the image processing device 10.

FIG. 6 is a sequence diagram showing an outline of the processing performed by the image processing device 10, the distance measuring device 20, and the image pickup device 30 according to the first embodiment. The outline of the process will be described with reference to FIG.

In step S10, the ranging device 20 acquires the reflected light intensity distribution, the distance distribution, and the instantaneous velocity distribution within a predetermined range. Information on these distributions is sometimes collectively referred to as distribution information. Further, a predetermined range for acquiring distribution information may be referred to as a first region.

In step S20, the image pickup apparatus 30 captures an image within a range including at least a part of the range in which the reflected light intensity distribution, the distance distribution, and the instantaneous velocity distribution are acquired. Although it is shown in FIG. 6 that step S20 is performed after step S10, the order of step S10 and step S20 may be reversed, and these may be performed in parallel.

In step S30, the distance measuring device 20 transmits distribution information to the image processing device 10. In step S40, the image pickup apparatus 30 transmits an image to the image processing apparatus 10. Although it is shown in FIG. 6 that step S40 is performed after step S30, the order of step S30 and step S40 may be reversed, and these may be performed in parallel.

In step S50, the image processing apparatus 10 detects a moving object in the image based on at least the instantaneous velocity distribution. The range in the image containing the detected moving object is sometimes referred to as the second region.

In step S60, the image processing device 10 performs image processing on the second region. Here, as an example of image processing, there is a trimming process of cutting out a range including a second region from an image to generate a moving object image.

The processing performed in the distance measuring device 20 will be described more specifically with reference to FIGS. 7 to 10. FIG. 7 is a flowchart showing an outline of the processing performed by the distance measuring device 20 according to the first embodiment. FIG. 8 is a graph showing the principle of distance measurement. 9 and 10 are graphs showing the principle of instantaneous velocity measurement.

Steps S101 to S104 of FIG. 7 are processes corresponding to step S10 of FIG. Step S105 in FIG. 7 is a process corresponding to step S30 in FIG. In step S101, the ranging device 20 projects laser light in a frequency band such as near-infrared light. In step S102, the ranging device 20 receives the reflected light reflected from the object 2.

In step S103, the distance measuring device 20 calculates the reflected light intensity, the distance, and the instantaneous speed using a signal based on the reference light, the reflected light, and the like. Here, a method of calculating the distance and the instantaneous speed by the FMCW method will be described with reference to FIGS. 8 to 10.

First, a method of calculating the distance will be described with reference to FIG. The graph of FIG. 8 shows a part of the triangle wave based on the reference light, a part of the triangle wave based on the reflected light, and the time change of the frequency of these beats. From FIG. 8, the triangular wave based on the reference light has a frequency f ₀ at time t ₀ , and the frequency increases linearly with the passage of time. After that, at the time t ₀ + T when the time T has elapsed, the frequency becomes f ₀ + F. That is, the slope of the triangular wave based on the reference light is F / T. The value of the slope F / T of the triangular wave is known because it is determined by the triangular wave generated by the triangular wave generator 213. As described above, the triangular wave based on the reflected light is input with a time Δt (= 2r / c) later than the reference light. Therefore, as shown in FIG. 8, the reference light is laterally shifted by the time Δt. The waveform becomes. In FIG. 8, it is assumed that the object 2 is not moving and the waveform is not shifted due to the Doppler effect.

Since the time Δt is an extremely short time, it may be difficult to measure the time Δt itself with high accuracy, but the frequency Δf can be measured with relatively high accuracy by generating a beat with a mixer or the like. .. Therefore, a beat is generated by mixing a triangular wave based on the reference light and a triangular wave based on the reflected light, and the frequency of the beat is measured to obtain the difference Δf between the frequency of the reference light and the frequency of the reflected light. The ratio of Δf to Δt agrees with the ratio of F to T, as is clear from FIG. Therefore, Δt can be expressed as Δt = Δf / (F / T) by using Δf obtained from the beat and a known (F / T) value. Considering the above-mentioned relationship of Δt = 2r / c, the distance from the distance measuring device 20 to the object 2 is expressed by the following equation (1).

Therefore, distance measurement can be performed by projecting frequency-modulated light so that the frequency increases linearly with the passage of time, and measuring the beat frequencies of the signal based on the reflected light and the signal based on the reference light. ..

Next, a method of calculating the instantaneous velocity will be described with reference to FIGS. 9 and 10. 9 and 10 are graphs showing the graph shown in FIG. 8 over a wider time range so as to include one cycle of the triangular wave. FIG. 9 shows the time change of the frequency when the instantaneous velocity of the object 2 is zero. As can be understood from FIG. 9, the beat frequency is constant at Δf except near the apex of the triangular wave.

FIG. 10 shows the time change of the frequency when the object 2 is moving in the direction toward the distance measuring device 20. The frequency of the light projected from the distance measuring device 20 becomes high due to the Doppler effect when it is reflected by the object 2. As a result, the beat frequency repeats two types of values, Δf ₁ and Δf ₂ , when the frequency of the triangular wave rises and falls. Assuming that the frequency fluctuation amount due to the Doppler effect is Δf _d , it is expressed by Δf _d = (Δf ₂ -Δf ₁ ) / 2. Further, Δf in the above equation (1) is represented by Δf = (Δf ₁ + Δf ₂ ) / 2. Therefore, by acquiring Δf ₁ and Δf ₂ as the beat frequencies, the distance r can be calculated based on the equation (1) even when the object 2 is moving. Further, the instantaneous velocity of the object 2 can be calculated by inputting Δf _d into the formula of the Doppler effect of light. As described above, since the distance measuring device 20 of the present embodiment is a LiDAR device using the FMCW method, it is possible to acquire the distance distribution and the instantaneous velocity distribution. Further, the ranging device 20 of the present embodiment can acquire the reflected light intensity distribution based on the signal intensity based on the reflected light.

In step S104 of FIG. 7, the distance measuring device 20 determines whether or not the acquisition of the reflected light intensity distribution, the distance distribution, and the instantaneous velocity distribution within a predetermined range is completed. When these acquisitions are not completed (NO in step S104), the process proceeds to step S101, and the reflected light intensity, distance, and instantaneous velocity of different measurement points are measured by changing the position of irradiating the light. When these acquisitions are completed (YES in step S104), the process proceeds to step S105. As described above, in the loop from step S101 to step S104, scanning for acquiring the reflected light intensity distribution, the distance distribution, and the instantaneous velocity distribution is performed.

In step S105, the distance measuring device 20 transmits the reflected light intensity distribution, the distance distribution, and the instantaneous velocity distribution (distribution information) to the image processing device 10.

The processing performed in the image processing apparatus 10 will be described more specifically with reference to FIGS. 11 and 12. FIG. 11 is a flowchart showing an outline of the processing performed by the image processing apparatus 10 according to the first embodiment. FIG. 12 is a schematic diagram showing an outline of the trimming process.

Step S201 of FIG. 11 is a process corresponding to step S30 of FIG. 6 and step S105 of FIG. 7. Step S202 of FIG. 11 is a process corresponding to step S40 of FIG. Step S203 of FIG. 11 is a process corresponding to step S50 of FIG. Step S204 and step S205 of FIG. 11 are processes corresponding to step S60 of FIG.

In step S201, the first acquisition unit 121 of the image processing device 10 acquires distribution information in the first region in the space from the distance measuring device 20. The acquired distribution information is stored in the storage unit 126, and is appropriately read from the storage unit 126 and used in the subsequent processing. Here, the distribution information includes at least the instantaneous velocity distribution.

In step S202, the second acquisition unit 122 of the image processing device 10 acquires an image including at least a part of the first region from the distance measuring device 20. The acquired image is stored in the storage unit 126, and is appropriately read from the storage unit 126 and used in the subsequent processing. The order of steps S201 and S202 may be reversed, and they may be performed in parallel.

In step S203, the detection unit 123 of the image processing apparatus 10 detects a moving object region (second region) indicating a moving object included in the image based on the instantaneous velocity distribution. The process of step S203 will be described in more detail with reference to FIG.

The upper part of FIG. 12 shows the instantaneous velocity distribution 91 in the first region. The instantaneous velocity distribution 91 shows a two-dimensional distribution of the instantaneous velocity, and a portion having an instantaneous velocity larger than a predetermined threshold value is shown by hatching. From FIG. 12, it can be seen that the humanoid moving object 92 exists in the instantaneous velocity distribution 91. It should be noted that the determination of the portion having the instantaneous velocity here does not have to be based on the threshold value. For example, an edge extraction process using a differential value of the instantaneous velocity is performed, and the area surrounded by the edges is defined as a predetermined moment. It may be discriminated as a portion having a speed.

Image 93 is shown in the middle of FIG. 12. The image 93 is one frame of a still image or a moving image. When only the image 93 is referred to, it is difficult to determine in which region of the image 93 the moving object is located. For example, it is difficult to determine that the humanoid portion in the image 93 is a moving object based only on the information in the image 93. Therefore, the detection unit 123 detects a region corresponding to the moving object 92 from the image 93 with reference to the instantaneous velocity distribution 91. This detection is performed, for example, by searching the image 93 for a shape similar to the shape of the moving object 92 in the instantaneous velocity distribution 91. In this way, the humanoid moving object region 94 is detected in the image 93 as the region corresponding to the moving object 92.

In step S204, the coordinate setting unit 124 associates the coordinates of the moving object 92 in the instantaneous velocity distribution 91 with the coordinates of the moving object region 94 in the image 93. Thereby, for example, even if the position or size of the range in which the instantaneous velocity distribution 91 is acquired and the range in which the image 93 is acquired does not match, the range of the moving object region 94 in the image 93 is clarified. can do.

In step S205, the trimming processing unit 125 trims a part of the image 93 so as to include the moving object region 94 to generate the moving object image 95. The generated moving object image 95 is stored in the storage unit 126, and can be output to the outside from the image processing device 10 as needed. In this way, according to this process, it is possible to acquire the moving object image 95 including only the moving object region 94 or only the moving object region 94 and its vicinity.

According to the present embodiment, there is provided an image processing device 10 capable of extracting a moving object from one image without using a plurality of temporally continuous images such as a frame of a moving image for processing.

Here, the effect of extracting a moving object from one image will be described more specifically. In the method of detecting a moving object by comparing a plurality of images, the amount of information to be handled is large, so that the processing load can be large. A method of speeding up the processing by transmitting a plurality of images from the image pickup device to another processing device and performing a process of detecting a moving object by the other processing device is also conceivable. In this case, a plurality of images are transmitted. This can lead to an increase in communication load. On the other hand, in the present embodiment, the number of images used for processing can be one, and the number of images used for processing can be reduced. The instantaneous velocity distribution used in the processing of the present embodiment often has a smaller amount of information than the image. Therefore, the influence of the processing load and the communication load due to the above-mentioned factors is reduced.

When trimming a moving object, it is necessary to perform trimming with a margin so that the entire moving object is included. Therefore, the higher the accuracy of detecting the contour of the moving object, the smaller the size of the trimmed image. be able to. Since the method of the present embodiment directly measures the instantaneous velocity, the detection accuracy of the moving object can be improved as compared with the indirect method of estimating and detecting the moving object from the difference between a plurality of images. Therefore, the contour of the moving object can be detected with high accuracy, and the amount of information of the moving object image after trimming can be further reduced. Therefore, it is possible to reduce the communication load when transmitting the moving object image to another device.

In the detection of the moving object region 94 in step S203, the distance distribution may be further used in addition to the instantaneous velocity distribution 91. It is rare that the moving object region 94 and the other regions are equidistant, and usually there is a difference in the distance between them. Therefore, it may be possible to detect the contour of a moving object even in the distance distribution. Therefore, the detection accuracy can be further improved by detecting the moving object region 94 by integrating the information of the instantaneous velocity distribution 91 and the distance distribution.

Further, the reflected light intensity distribution may be further used in the detection of the moving object region 94 in step S203. It is unlikely that the moving object region 94 and the other regions happen to have the same reflectance, and usually there is a difference in the reflectance. Therefore, it may be possible to detect the contour of a moving object even in the reflected light intensity distribution. Therefore, the detection accuracy can be further improved by detecting the moving object region 94 by integrating the information of the instantaneous velocity distribution 91 and the reflected light intensity distribution.

In order to improve the image quality, it is desirable that the image used for processing is either a still image or a frame constituting a moving image in a format in which compression between frames is not performed. When one frame is taken out from a moving image such as MPEG4 (Moving Picture Experts Group 4), which is a format for compressing between frames, and used for this processing, image quality deterioration due to compression between frames may occur. On the other hand, when one frame is taken out from a still image or a moving image in a format in which compression between frames is not performed and used for this processing, this deterioration in image quality does not occur. The moving image in a format in which compression between frames is not performed may be, for example, Motion JPEG (Joint Photographic Experts Group).

[Second Embodiment]
As a second embodiment of the present invention, a configuration example in which the image processing device 10 is incorporated in the distance measuring device 40 will be described. The description of the elements common to the first embodiment will be omitted or simplified.

FIG. 13 is a schematic diagram showing a schematic configuration of the object detection system 1 according to the second embodiment. The object detection system 1 includes a distance measuring device 40 and an imaging device 30. The distance measuring device 40 includes an image processing device 10 and a measuring device 50. The measuring device 50 may be, for example, an FMCW type LiDAR device corresponding to the distance measuring device 20 described in the first embodiment. The image processing device 10 included in the distance measuring device 40 performs the same processing as the image processing device 10 of the first embodiment.

Even in the configuration in which the image processing device 10 is incorporated in the distance measuring device 40 of the present embodiment, the same effect as that of the first embodiment can be obtained. Therefore, there is provided a distance measuring device 40 capable of extracting a moving object from one image without using a plurality of temporally continuous images such as a frame of a moving image for processing.

[Third Embodiment]
As a third embodiment of the present invention, a configuration example in which the image processing device 10 is incorporated in the image pickup device 60 will be described. The description of the elements common to the first embodiment will be omitted or simplified.

FIG. 14 is a schematic diagram showing a schematic configuration of the object detection system 1 according to the third embodiment. The object detection system 1 includes a distance measuring device 20 and an imaging device 60. The image pickup device 60 includes an image processing device 10 and a camera unit 70 including an image pickup element. The camera unit 70 may be, for example, a digital still camera corresponding to the image pickup apparatus 30 described in the first embodiment. The image processing device 10 included in the image pickup device 60 performs the same processing as the image processing device 10 of the first embodiment.

Even in the configuration in which the image processing device 10 is incorporated in the image pickup device 60 of the present embodiment, the same effect as that of the first embodiment can be obtained. Therefore, there is provided an image pickup device 60 capable of extracting a moving object from one image without using a plurality of temporally continuous images such as a frame of a moving image for processing.

The system described in the above-described embodiment can also be configured as in the following fourth embodiment.

[Fourth Embodiment]
FIG. 15 is a functional block diagram of the image processing apparatus 800 according to the fourth embodiment. The image processing device 800 includes a first acquisition unit 821, a second acquisition unit 822, and a detection unit 823. The first acquisition unit 821 acquires the instantaneous velocity distribution of the object in the first region in the space. The second acquisition unit 822 acquires an image including at least a part of the first region. The detection unit 823 detects a second region showing a moving object included in the image based on the instantaneous velocity distribution.

According to the present embodiment, an image processing device capable of extracting a moving object from one image is provided.

[Modification Embodiment]
The present invention is not limited to the above-described embodiment, and can be appropriately modified without departing from the spirit of the present invention.

A processing method in which a program for operating the configuration of the embodiment is recorded in a storage medium so as to realize the functions of the above-described embodiment, the program recorded in the storage medium is read out as a code, and the program is executed in a computer is also described in each embodiment. Included in the category. That is, a computer-readable storage medium is also included in the scope of each embodiment. Further, not only the storage medium in which the above-mentioned program is recorded but also the program itself is included in each embodiment. Further, the one or more components included in the above-described embodiment may be a circuit such as an ASIC or FPGA configured to realize the function of each component.

As the storage medium, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD (Compact Disk) -ROM, a magnetic tape, a non-volatile memory card, or a ROM can be used. Further, not only the program recorded on the storage medium is executed by itself, but also the program that operates on the OS and executes the process in cooperation with other software and the function of the expansion board is also an embodiment. Is included in the category of.

The service realized by the functions of each of the above-described embodiments can also be provided to the user in the form of SaaS (Software as a Service).

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

Some or all of the above embodiments may also be described, but not limited to:

(Appendix 1)
The first acquisition unit that acquires the instantaneous velocity distribution of the object in the first region in space,
A second acquisition unit that acquires an image including at least a part of the first region, and
A detection unit that detects a second region indicating a moving object included in the image based on the instantaneous velocity distribution.
An image processing device characterized by having.

(Appendix 2)
The image processing apparatus according to Appendix 1, wherein the instantaneous velocity distribution is a distribution indicating the magnitude of the instantaneous velocity in the direction toward the measuring device for measuring the instantaneous velocity distribution or in the direction away from the measuring device.

(Appendix 3)
The image processing apparatus according to Appendix 1 or 2, wherein the detection unit detects the second region so that the instantaneous velocity in the instantaneous velocity distribution includes a region larger than a predetermined threshold value.

(Appendix 4)
The image processing apparatus according to Appendix 1 or 2, wherein the detection unit detects an edge of an instantaneous velocity in the instantaneous velocity distribution and detects a region including the edge as the second region.

(Appendix 5)
The image processing apparatus according to any one of Supplementary note 1 to 4, further comprising a coordinate setting unit that associates the coordinates of the first region with the coordinates of the image.

(Appendix 6)
The image processing according to any one of Supplementary note 1 to 5, further comprising a trimming processing unit for generating a moving object image by trimming a part of the image so as to include the second region. Device.

(Appendix 7)
The first acquisition unit further acquires a distance distribution indicating the distance from the measuring device for measuring the instantaneous velocity distribution to the subject.
The image processing apparatus according to any one of Supplementary note 1 to 6, wherein the detection unit further detects the second region based on the distance distribution.

(Appendix 8)
The first acquisition unit further acquires a reflected light intensity distribution indicating the intensity of the reflected light from the subject by the light projected on the first region.
The image processing apparatus according to any one of Supplementary note 1 to 7, wherein the detection unit further detects the second region based on the reflected light intensity distribution.

(Appendix 9)
The instantaneous velocity distribution is based on the difference due to the Doppler effect between the frequency of the projected light and the frequency of the reflected light measured by using an FMCW (Frequency Modulated Continuous Wave) type LiDAR (Light Detection and Ranging) device. The image processing apparatus according to any one of Supplementary note 1 to 8, wherein the image processing apparatus is to be acquired.

(Appendix 10)
The image processing apparatus according to any one of Supplementary note 1 to 9, wherein the image is a frame constituting a moving image in a format in which compression between frames is not performed.

(Appendix 11)
The image processing apparatus according to any one of Supplementary note 1 to 9, wherein the image is a still image.

(Appendix 12)
A measuring device that measures the instantaneous velocity distribution,
The image processing apparatus according to any one of Supplementary note 1 to 11 and the image processing apparatus.
A ranging device characterized by having.

(Appendix 13)
The measuring device is an FMCW type LiDAR device, and is characterized in that the instantaneous velocity distribution is measured based on the difference caused by the Doppler effect between the frequency of the projected light and the frequency of the reflected light. The range measuring device described.

(Appendix 14)
Appendix 12 or 13 is characterized in that the measuring device further measures a distance distribution indicating the distance from the measuring device to the subject and a reflected light intensity distribution indicating the intensity of the reflected light from the subject due to the projected light. The distance measuring device described in.

(Appendix 15)
An image sensor that generates an image based on incident light,
The image processing apparatus according to any one of Supplementary note 1 to 11 and the image processing apparatus.
An imaging device characterized by having.

(Appendix 16)
The step of acquiring the instantaneous velocity distribution of an object in the first region in space,
The step of acquiring an image including at least a part of the first region, and
A step of detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution, and a step of detecting the second region.
An image processing method characterized by having.

(Appendix 17)
On the computer
The step of acquiring the instantaneous velocity distribution of an object in the first region in space,
The step of acquiring an image including at least a part of the first region, and
A step of detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution, and a step of detecting the second region.
A program characterized by executing.

10,800 Image processing device 121, 821 First acquisition unit 122, 822 Second acquisition unit 123, 823 Detection unit 124 Coordinate setting unit 125 Trimming processing unit 126 Storage unit

Claims (17)

The first acquisition unit that acquires the instantaneous velocity distribution of the object in the first region in space,
A second acquisition unit that acquires an image including at least a part of the first region, and
A detection unit that detects a second region indicating a moving object included in the image based on the instantaneous velocity distribution.
Have,
The detection unit detects the second region by searching the image for a shape similar to the shape of a moving object having a predetermined instantaneous velocity included in the instantaneous velocity distribution.
An image processing device characterized by this. The image processing apparatus according to claim 1, wherein the instantaneous velocity distribution is a distribution indicating the magnitude of the instantaneous velocity in the direction toward the measuring device for measuring the instantaneous velocity distribution or in the direction away from the measuring device. .. The image processing apparatus according to claim 1 or 2, wherein the detection unit detects the second region so that the instantaneous velocity in the instantaneous velocity distribution includes a region larger than a predetermined threshold value. The image processing apparatus according to claim 1 or 2, wherein the detection unit detects an edge of an instantaneous velocity in the instantaneous velocity distribution and detects a region including the edge as the second region. The image processing apparatus according to any one of claims 1 to 4, further comprising a coordinate setting unit that associates the coordinates of the first region with the coordinates of the image. The image according to any one of claims 1 to 5, further comprising a trimming processing unit that generates a moving object image by trimming a part of the image so as to include the second region. Processing device. The first acquisition unit further acquires a distance distribution indicating the distance from the measuring device for measuring the instantaneous velocity distribution to the subject.
The image processing apparatus according to any one of claims 1 to 6, wherein the detection unit further detects the second region based on the distance distribution. The first acquisition unit further acquires a reflected light intensity distribution indicating the intensity of the reflected light from the subject by the light projected on the first region.
The image processing apparatus according to any one of claims 1 to 7, wherein the detection unit further detects the second region based on the reflected light intensity distribution. The instantaneous velocity distribution is based on the difference due to the Doppler effect between the frequency of the projected light and the frequency of the reflected light measured by using an FMCW (Frequency Modulated Continuous Wave) type LiDAR (Light Detection and Ranging) device. The image processing apparatus according to any one of claims 1 to 8, wherein the image processing apparatus is obtained. The image processing apparatus according to any one of claims 1 to 9, wherein the image is a frame constituting a moving image in a format in which compression between frames is not performed. The image processing apparatus according to any one of claims 1 to 9, wherein the image is a still image. A measuring device that measures the instantaneous velocity distribution,
The image processing apparatus according to any one of claims 1 to 11.
A ranging device characterized by having. The measuring device is an FMCW type LiDAR device, and is characterized in that the instantaneous velocity distribution is measured based on the difference caused by the Doppler effect between the frequency of the projected light and the frequency of the reflected light. The distance measuring device described in. 13. 13. The distance measuring device according to 13. An image sensor that generates an image based on incident light,
The image processing apparatus according to any one of claims 1 to 11.
An imaging device characterized by having. The step of acquiring the instantaneous velocity distribution of an object in the first region in space,
The step of acquiring an image including at least a part of the first region, and
A step of detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution, and a step of detecting the second region.
Have,
The second region is detected by searching the image for a shape similar to the shape of a moving object having a predetermined instantaneous velocity included in the instantaneous velocity distribution.
An image processing method characterized by that. On the computer
The step of acquiring the instantaneous velocity distribution of an object in the first region in space,
The step of acquiring an image including at least a part of the first region, and
A step of detecting a second region indicating a moving object included in the image based on the instantaneous velocity distribution, and a step of detecting the second region.
Is a program that executes
The second region is detected by searching the image for a shape similar to the shape of a moving object having a predetermined instantaneous velocity included in the instantaneous velocity distribution.
A program characterized by that.

Images