JP7406695B2 - Image processing device and image processing program - Google Patents

Description

The present invention relates to an image processing device and an image processing program.

When evaluating the health of a building immediately after a major earthquake occurs, it is important to perform a primary diagnosis to determine whether evacuation is necessary before conducting a detailed diagnosis of the building. In view of this, the inventors of the present invention have proposed in Patent Document 1 a system for diagnosing the health of a building that can perform a primary and simple diagnosis without relying on acceleration sensors or the like.

This system numerically evaluates the health of a building based on the rate of change in the building's natural frequency before and after an earthquake, which is obtained by analyzing video images taken of the building using a camera. can be converted into

When calculating the natural frequency of a target building, the natural frequency can be calculated by detecting minute vibration components in moving images using the techniques disclosed in Patent Documents 2 to 3 and Non-Patent Document 1. When calculating, if the imaging device itself exists in a microvibration environment, it is necessary to distinguish between the vibrations of the imaging device and the vibrations of the building.

If the time-frequency characteristics and spatial frequency characteristics of the vibration of the photographing device and the vibration of the building are sufficiently different, it becomes possible to separate them in the spatio-temporal frequency domain as described in Patent Document 1. However, if the vibration characteristics of the imaging device and the vibration characteristics of the building overlap in the spatio-temporal frequency domain, they cannot be simply separated, making it impossible to correctly evaluate the characteristics such as the natural frequency of the vibration of the building that is originally intended to be measured. There are cases.

On the other hand, there are some physical techniques to reduce the vibration of photographic equipment, such as increasing the weight of the entire photographing system including the photographing equipment and incidental equipment such as tripods, and introducing equipment such as anti-vibration rubber and springs. Anti-vibration measures can also be considered. However, this measure cannot be expected to have a dramatic effect in the frequency band of several Hz or less, which is included in ground vibrations.

Therefore, there is a need for software-based processing that can detect (and even remove) the vibration components of the imaging device. Note that such software measures are not limited to the use of diagnosing the health of buildings using moving photography as described in Patent Document 1, but also include the use of diagnosing the health of buildings using fixed photography. The present invention is generally useful in vibration analysis methods that involve only moving image processing based on spatiotemporal filtering of objects, as typified by Documents 2 and 3 and Non-Patent Document 1.

As a technology related to software-based vibration component detection and removal processing, Patent Document 4 discloses a digital image stabilization technology that is not mechanical or optical. With this technology, it is possible to remove fluctuations in images caused by the movement of the imaging device itself, in visual applications such as viewing and recording, and in matching applications between images such as panoramic composition and three-dimensional reconstruction.

Japanese Patent Application Publication No. 2018-136191 US Patent Application Publication No. 2014/0072190 US Patent No. 9324005 JP2019-004451A

J.G. Chen, A. Davis, N. Wadhwa, F. Durand, W.T. Freeman, and O. Buyukozturk, “Video Camera-based Vibration Measurement for Condition Assessment of Civil Infrastructure”, International Symposium Non-Destructive Testing in Civil Engineering (2015)

However, in visual applications, the main targets that must be detected and removed are fluctuation components that extend over several pixels (pixels), and the technology disclosed in Patent Document 4 is capable of detecting and removing ultrafine fluctuations on the sub-pixel level. There is no mention of detection, etc. In addition, although sub-pixel-level accuracy may be required for matching purposes, it is often based on geometric transformation based on the correspondence between multiple pixels, and it is often If a vibration component from the photographing device is mixed into a subject that does not appear to be moving at first glance, it may not necessarily function correctly.

The present disclosure has been made in view of the above circumstances, and aims to provide an image processing device and an image processing program that can accurately detect minute vibration components of a photographing device from a moving image. .

The image processing device according to the present invention according to claim 1 includes: an acquisition unit that acquires a moving image that includes a plurality of objects as subjects and that is obtained by photographing with a photographing device; a derivation unit that derives a physical quantity indicating the moving speed of the object for each divided area obtained by dividing each frame image into a plurality of divided areas between each frame image in a moving image; comprising: an extraction unit that extracts a divided area corresponding to the most frequently occurring physical quantity; and a identification unit that identifies a vibration component of the divided area extracted by the extraction unit as a vibration component of the imaging device ; The deriving unit performs a complex spatial filtering process on the moving image to generate a phase image, and calculates a phase difference between temporally adjacent frame images of the phase image and indicating a difference for each of the divided regions. Deriving the signal as the physical quantity, the identifying unit identifies a time-series signal of the representative value of the phase difference signal in the divided region extracted by the extracting unit as a vibration component of the imaging device, and The components include the amplitude and temporal frequency of the vibration of the imaging device .

According to the image processing device according to the present invention as set forth in claim 1, a physical quantity indicating the moving speed of an object for each divided region between each frame image in a moving image obtained by photographing with a photographing device is derived; By identifying the vibration component of the divided region corresponding to the physical quantity with the highest frequency among the physical quantities of each divided region as the vibration component of the photographing device, minute vibration components of the photographing device can be accurately detected from the video image. can do.

In the image processing device according to the present invention according to claim 1 , the deriving unit performs a complex spatial filtering process on the moving image to generate a phase image, and generates a temporally adjacent frame of the phase image. A phase difference signal indicating the difference between the images and for each of the divided regions is derived as the physical quantity.

According to the image processing device according to the present invention as set forth in claim 1 , a phase image is generated by performing complex spatial filtering processing on the moving image, and between temporally adjacent frame images of the phase image, Further, by deriving a phase difference signal indicating the difference between the divided regions as the physical quantity, it is possible to more easily derive the physical quantity indicating the moving speed of the object.

The image processing device according to the present invention according to claim 2 is the image processing device according to claim 1 , in which, when the phase image is a wrapped phase, the derivation unit The phase difference signal is derived after unwrapping the phase of the pixel.

According to the image processing device according to the present invention as set forth in claim 2 , when the phase image is a wrapped phase, the phase difference signal is processed after unwrapping the phase of each pixel of the phase image. By deriving , the phase difference signal can be derived with higher accuracy.

In the image processing device according to the present invention as set forth in claim 1 , the specifying unit converts the time-series signal of the representative value of the phase difference signal in the divided region extracted by the extracting unit into a vibration component of the photographing device. .

The image processing device according to the present invention according to claim 3 is the image processing device according to claim 1 or 2 , wherein the representative value of the phase difference signal is an average value of each of the phase difference signals. , or the median value, or the maximum value, or the minimum value.

According to the image processing device according to the present invention according to claims 1 and 3 , the time series signal of the representative value of the phase difference signal in the extracted divided region is identified as a vibration component of the photographing device. By doing so, the vibration component of the imaging device can be specified more easily than when the representative value is not applied.

The image processing device according to the present invention according to claim 4 is the image processing device according to any one of claims 1 to 3 , wherein the derivation unit The physical quantity is derived for only a part of the area.

According to the image processing device according to the fourth aspect of the present invention, the calculation load can be reduced by deriving the physical quantity only for a predetermined part of the moving image.

The image processing apparatus according to the present invention according to claim 5 is the image processing apparatus according to claim 4 , wherein the partial area is an area where the S/N ratio in the moving image is equal to or higher than a predetermined level. It is something that is.

According to the image processing device according to the present invention as set forth in claim 5 , by setting the partial area to an area where the S/N ratio in the moving image is equal to or higher than a predetermined level, the calculation load can be more effectively reduced. The vibration component can be detected with high accuracy while reducing the vibration.

The image processing device according to the present invention according to claim 6 is the image processing device according to any one of claims 1 to 5 , in which the divided area is divided into areas for each pixel in the moving image. It is a region or a region for each of a plurality of pixel groups.

According to the image processing device according to the present invention as set forth in claim 6 , by setting the divided region to be a region for each pixel in the moving image or a region for each group of a plurality of pixels, the processing can be performed more simply and with high efficiency. It is possible to accurately detect the vibration component of the photographing device.

The image processing program according to the present invention according to claim 7 acquires a moving image in which a plurality of objects are included as subjects and is obtained by photographing with a photographing device, and calculates the difference between each frame image in the acquired moving image. , deriving a physical quantity indicating the moving speed of the object for each divided region obtained by dividing each frame image into a plurality of divided regions, and extracting a divided region corresponding to the physical quantity with the highest frequency among the derived physical quantities for each divided region, An image processing program that causes a computer to execute a process of identifying a vibration component of the extracted divided region as a vibration component of the imaging device, the program performing a complex space filtering process on the moving image to generate a phase image. Then, a phase difference signal indicating a difference between temporally adjacent frame images of the phase image and for each divided region is derived as the physical quantity, and a time difference of the representative value of the phase difference signal in the extracted divided region is calculated. The series signal is identified as a vibration component of the photographing device, and the vibration component includes an amplitude and a temporal frequency of vibration of the photographing device .

According to the image processing program according to the present invention as set forth in claim 7 , a physical quantity indicating the moving speed of an object for each divided region between each frame image in a moving image obtained by photographing with a photographing device is derived; By identifying the vibration component of the divided region corresponding to the physical quantity with the highest frequency among the physical quantities of each divided region as the vibration component of the photographing device, minute vibration components of the photographing device can be accurately detected from the video image. can do.

As described above, according to the present invention, minute vibration components of the photographing device can be detected with high accuracy from a moving image.

FIG. 1 is a block diagram illustrating an example of a hardware configuration of an image processing device according to an embodiment. FIG. 1 is a block diagram illustrating an example of a functional configuration of an image processing device according to an embodiment. FIG. 1 is a schematic diagram showing an example of the configuration of a moving image database according to an embodiment. It is a flow chart which shows an example of vibration component identification processing concerning an embodiment. FIG. 2 is a front view showing an example of a moving image (representative image) according to the embodiment. It is a front view showing an example of a phase image (horizontal component) concerning an embodiment. It is a front view showing an example of a phase image (vertical component) concerning an embodiment. It is a graph provided for explanation of unwrapping processing concerning an embodiment. It is a graph showing an example of frequency distribution of a phase difference signal concerning an embodiment. FIG. 2 is a front view showing an example of a sample image (sine wave image) used in a verification experiment according to the embodiment. 7 is a graph showing an example of an average phase difference signal in a pixel group having a phase difference with a maximum frequency distribution in a verification experiment according to an embodiment. It is a graph which shows an example of the movement amount signal of a pixel used for explanation of the verification experiment based on embodiment. It is a graph which shows an example of a time frequency spectrum used for explanation of the verification experiment based on embodiment. It is a graph which shows an example of a time frequency spectrum used for explanation of the verification experiment based on embodiment.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In this embodiment, a case will be described in which the present invention is applied to an image processing apparatus that processes a moving image of a building in a state of slight movement under the influence of wind excitation or ground vibration.

First, the configuration of an image processing apparatus 10 according to the present embodiment will be described with reference to FIGS. 1 and 2. Note that examples of the image processing device 10 include information processing devices such as a personal computer and a server computer.

As shown in FIG. 1, the image processing device 10 according to the present embodiment includes a CPU (Central Processing Unit) 11, a memory 12 as a temporary storage area, a nonvolatile storage section 13, an input section 14 such as a keyboard and a mouse, It includes a display section 15 such as a liquid crystal display, a medium read/write device (R/W) 16, and a communication interface (I/F) section 18. The CPU 11, memory 12, storage section 13, input section 14, display section 15, medium reading/writing device 16, and communication I/F section 18 are connected to each other via a bus B1. The medium read/write device 16 reads information written in the recording medium 17 and writes information to the recording medium 17 .

The storage unit 13 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. A vibration component identification program 13A is stored in the storage unit 13 as a storage medium. The vibration component identification program 13A is created by setting the recording medium 17 on which the vibration component identification program 13A has been written into the medium reading/writing device 16, and reading the vibration component identification program 13A from the recording medium 17. It is stored in the storage unit 13. The CPU 11 reads the vibration component identification program 13A from the storage unit 13, expands it into the memory 12, and sequentially executes the processes included in the vibration component identification program 13A. The storage unit 13 also stores various databases such as a moving image database 13B and a complex spatial filter database 13C.

In the image processing device 10 according to the present embodiment, a photographing device 20 that photographs moving images is connected to the communication I/F section 18 . The photographing device 20 is for photographing so as to include a plurality of buildings at the time of photographing. Note that the photographing method using the photographing device 20 may be any method such as aerial photographing, moving photographing by a person on the ground, or fixed photographing using a tripod or a fixed member. Further, in the present embodiment, a photographing device that takes a color image is used as the photographing device 20, but the invention is not limited to this. For example, a photographing device that takes a monochrome image can be applied as the photographing device 20. It may also be a form.

Next, with reference to FIG. 2, the functional configuration of the image processing device 10 according to this embodiment will be described. As shown in FIG. 2, the image processing device 10 includes an acquisition section 11A, a derivation section 11B, an extraction section 11C, and a specification section 11D. By executing the vibration component identification program 13A, the CPU 11 of the image processing device 10 functions as an acquisition unit 11A, a derivation unit 11B, an extraction unit 11C, and an identification unit 11D.

The acquisition unit 11A according to the present embodiment acquires a moving image that includes a plurality of objects as subjects and that is obtained by photographing with the photographing device 20. Note that in this embodiment, a building is used as the object, but the object is not limited to this. For example, buildings other than buildings such as bridges and towers, natural objects such as mountains and trees, biological information such as pulsation, equipment such as air conditioning ducts and transformers, or combinations of multiple types of these can be applied as the above objects. It may also be in the form of

Further, the derivation unit 11B derives a physical quantity indicating the moving speed of the object for each divided region obtained by dividing each frame image into a plurality of parts between each frame image in the moving image acquired by the acquisition unit 11A. Note that in this embodiment, regions for each of a plurality of pixel groups are used as the divided regions, but the present invention is not limited to this. For example, a configuration may be adopted in which each pixel area in a moving image obtained by photographing with the photographing device 20 is applied as the divided area.

The derivation unit 11B according to the present embodiment performs a complex space filtering process on the moving image to generate a phase image, and calculates the difference between temporally adjacent frame images of the phase image and for each divided region. A phase difference signal indicating the difference is derived as the physical quantity. That is, the "phase" here is a variable representing the "position" of a signal, and the phase image has position information indicating the spatial position. Therefore, the phase difference signal, which is the difference in phase images between temporally adjacent frame images, represents a "displacement" that is a change in position in the time interval over which one frame passes. This is a variable that has a correlation with moving speed.

Furthermore, when the phase image has a wrapped phase, the derivation unit 11B according to the present embodiment derives the phase difference signal after performing an unwrapping process on the phase of each pixel of the phase image. Further, the deriving unit 11B according to the present embodiment derives the physical quantity only for a predetermined part of the moving image. Note that in this embodiment, an area in which the S/N ratio (Signal to Noise ratio) in the video image is equal to or higher than a predetermined level is used as the above-mentioned partial area, but the present invention is not limited to this. .

Furthermore, the extraction unit 11C according to the present embodiment extracts the divided area corresponding to the physical quantity with the highest frequency among the physical quantities for each divided area derived by the derivation unit 11B. Further, the identifying unit 11D according to the present embodiment identifies the vibration component of the divided region extracted by the extracting unit 11C as the vibration component of the imaging device 20.

The identifying unit 11D according to the present embodiment identifies the time series signal of the representative value of the phase difference signal in the divided region extracted by the extracting unit 11C as a vibration component of the imaging device 20. In this embodiment, the average value of the phase difference signal is used as the representative value of the phase difference signal, but the present invention is not limited to this. For example, instead of the average value, a median value, maximum value, or minimum value may be used as the representative value.

Next, with reference to FIG. 3, the moving image database 13B according to this embodiment will be described. As shown in FIG. 3, the moving image database 13B according to the present embodiment stores moving image information obtained by photographing a moving image with the photographing device 20 for each moving image ID (Identification) assigned in advance. There is. In this manner, in this embodiment, moving image information is captured in advance from the photographing device 20 and registered in the moving image database 13B, but the present invention is not limited to this. For example, a configuration may be adopted in which the image capturing device 20 constantly performs image capturing, and the moving image information obtained from the image capturing device 20 when a vibration of a predetermined level or higher occurs is used online, in real time or non-real time.

On the other hand, in the complex space filter database 13C according to this embodiment, information indicating a predetermined complex space filter (in this embodiment, a complex Gabor filter) is registered. However, the complex spatial filter is not limited to the complex Gabor filter, but is a complex spatial filter that is a set of spatial filters (real part filter, imaginary part filter) whose spatial phase characteristics differ by 90 degrees and whose spatial amplitude characteristics are equal. If available, other filters may be applied as complex space filters.

Next, the operation of the image processing device 10 according to this embodiment will be explained with reference to FIGS. 4 to 8. When the user inputs an instruction to start executing the vibration component identification program 13A through the input unit 14, the CPU 11 of the image processing device 10 executes the vibration component identification program 13A, thereby executing the program shown in FIG. Vibration component identification processing is executed. Here, in order to avoid confusion, a case will be described in which the moving image database 13B and the complex spatial filter database 13C have been constructed, and the moving image information to be processed is specified by the user.

At step 200 in FIG. 4, the acquisition unit 11A acquires moving image information specified by the user (hereinafter referred to as "processing target moving image information") by reading it from the moving image database 13B.

In step 202, the derivation unit 11B reads information indicating a complex space filter from the complex space filter database 13C, and performs complex space filtering on the processing target video information using the complex space filter (in this embodiment, a complex Gabor filter). Processing is performed to generate a phase image.

That is, first, the deriving unit 11B performs a complex space filtering process using the read complex space filter to extract the real part image I _re and the imaginary part from each frame image of the moving image indicated by the processing target moving image information. Calculate the image I _im . Next, the derivation unit 11B calculates the phase image I _θ by performing calculation according to the following equation (1) for each pixel.

I _θ = tan ⁻¹ (I _im /I _re ) (1)

The above processing is performed on all frame images of the moving image indicated by the processing target moving image information. For the complex spatial filtering process, either a method using convolution kernel convolution on the spatial domain or a method using filter product on the spatial frequency domain may be applied.

For example, if one of the moving images indicated by the processing target moving image information is the image shown in FIG. The phase image of the vertical component is as shown in FIG. 6B. Note that, for convenience, the image shown in FIG. 5 is a photograph of a structure created by the inventors of the present invention, rather than a building as a subject.

In step 204, the deriving unit 11B determines a region (hereinafter referred to as a "processing target region") in which the vibration component of the photographing device 20 is to be detected in the derived phase image _Iθ . In this embodiment, as a method for determining the processing target area, a method is applied in which a group of pixels having an S/N ratio of a predetermined level or higher is set as the processing target area. Here, as an example of a pixel group whose S/N ratio is a predetermined level or higher, in a moving image indicated by the processing target moving image information at the stage acquired by the acquisition unit 11A, a spatial first-order differential filter of a frame image (for example, This corresponds to an output image that has been subjected to a Sobel filter), a spatial second-order differential filter (for example, a Laplacian filter), or an edge detection process. If it is necessary to remove spike-like noise components and secure a cluster of regions, a median filter, dilation processing, and contraction processing are used together. The derivation unit 11B finally determines the processing target area by performing binarization using threshold processing.

Note that the method for determining the region to be processed is not limited to the above method, and for example, a region of interest designated in advance by the user may be determined as the region to be processed.

In step 206, the derivation unit 11B performs phase unwrapping processing on the phase image _Iθ obtained by the processing in step 202. That is, the phase information is generally wrapped in the range of -π to +π (ie, for example, π+π/4→-π/4). Therefore, in this embodiment, phase unwrap processing (phase connection processing) is performed. As phase unwrapping processing, for example, the Internet (URL: https://www.researchgate.net/publication/265151826), (URL: http://retrofocus28.blogspot.com/2013/12/phase-unwrapping_26.html) , (URL: https://jp.mathworks.com/help/dsp/ref/unwrap.html#f5-1119858) etc. can be applied. It goes without saying that if the derived phase image I _θ is not wrapped, there is no need to perform the process of step 206.

FIG. 7 shows an example of phase images before and after the phase connection by the phase unwrapping process in step 206 when the image to be processed is an image of a partial region of the image shown in FIG. 5. In the example shown in FIG. 7, the time-series changes in the phase images before phase connection are shown by broken lines, and the time-series changes in the phase images after phase connection are shown by solid lines.

As shown in FIG. 7, due to the phase connection, a signal that has been folded over +180 degrees to -180 degrees is continuously expressed over +180 degrees. However, the phase unwrapping process used in the example in Figure 7 does not assume that the phase rotates more than two times, so the folding of the signal where the value of the horizontal axis in Figure 7 exceeds +360 degrees around the 120th frame remains. It's still there. This problem can be resolved by subjecting the unwrapped signal to the phase unwrapping process again. However, although the present invention targets weak vibrations at the sub-pixel level, if phase wrapping occurs, it can be interpreted as a large movement that exceeds the sub-pixel level. It is also possible to use phase unwrap processing for the purpose of detecting that a group or image region is to be excluded from the processing target region.

In step 208, the deriving unit 11B indicates the difference between temporally adjacent frame images in each pixel of the processing target area determined by the processing in step 204 in the phase image I _θ obtained through the above processing. The above-described phase difference signal, which is a time-series signal, is calculated.

In step 210, the extraction unit 11C calculates the frequency distribution of the phase difference signal derived by the process in step 208 in an arbitrary frame. FIG. 8 shows an example of the frequency distribution in one frame obtained by the process of step 210. In addition, in the frequency distribution shown in Figure 8, since the phase difference between adjacent frames is calculated, extremely large phase differences are ignored and only the range from -90 degrees to +90 degrees is calculated in 1 degree increments. I am counting.

In step 212, the extraction unit 11C obtains, for all frames, the representative value of the phase difference signal in the pixel group having the phase difference in the range where the frequency is maximum in the calculated frequency distribution. Note that in this embodiment, the average value of the phase difference signals is used as the representative value, but the present invention is not limited to this. For example, as described above, the median value, maximum value, or minimum value of the phase difference signal may be used as the representative value.

In step 214, the identifying unit 11D identifies the time series signal of the representative value obtained by the process in step 212 as a vibration component of the imaging device 20, and stores information indicating the identified vibration component in a predetermined location in the storage unit 13. After storing it in the area, the main vibration component specifying process ends.

Next, with reference to FIGS. 9 to 12, a verification experiment regarding identification of the vibration component of the photographing device by the image processing device 10 according to the present embodiment will be described.

Here, a sine wave image of 100 pixels x 100 pixels with a spatial wavelength of 8 pixels shown in Fig. 9 is used as a moving image vibrated at an amplitude of 0.5 pixels and a temporal frequency of 2 Hz, which is likened to the vibration component of the imaging device. Verified. As the complex spatial filter, a Gaussian function type bandpass filter having a peak at spatial wavelength λ=8 pixels was used.

In this case, the average phase difference signal ΔI _{θ_ave} (t) in the pixel group where the frequency distribution has the maximum phase difference θ _max is as shown in FIG. Since this signal corresponds to the average angular velocity [rad/s] of the pixel group with the phase difference θ _max , the operation shown in the following formula was performed to convert it to the pixel movement amount signal d(t) [pixel]. . It is assumed that d(t) in this formula is a signal of the vibration component of the imaging device (see FIG. 11).

From the above, it can be assumed that the above vibration component exists as a vibration component of the photographing device in all pixels. Therefore, a signal A(t) obtained by extracting only one arbitrary pixel in the image and performing vibration detection, and a signal obtained by subtracting the vibration component d(t) of the photographing device from the signal A(t), The respective time-frequency spectra are shown in FIGS. 12A and 12B. It can be seen from FIG. 12A that the signal A(t) has a peak at 2 Hz. Since the verification moving image does not contain vibration components other than vibrations of the photographing device, it was confirmed from FIG. 12B that the vibration component of the photographing device was removed by the method according to the present embodiment.

As described above, according to the present embodiment, the acquisition unit 11A acquires a moving image in which a plurality of objects are included as subjects and is obtained by photographing with a photographing device, and the moving image acquired by the acquisition unit 11A. A deriving unit 11B that derives a physical quantity indicating the moving speed of the object for each divided area obtained by dividing each frame image into a plurality of parts between each frame image in the image, and a physical quantity for each divided area derived by the deriving unit 11B. , an extraction unit 11C that extracts a divided area corresponding to the most frequently occurring physical quantity, and a identification unit 11D that identifies the vibration component of the divided area extracted by the extraction unit 11C as a vibration component of the imaging device. There is. Therefore, minute vibration components of the photographing device can be detected with high accuracy from the moving image.

Further, according to the present embodiment, a phase image is generated by performing complex space filtering processing on a moving image, and the difference between temporally adjacent frame images of the phase image and for each of the divided regions is calculated. The phase difference signal shown is derived as the above-mentioned physical quantity. Therefore, the physical quantity indicating the moving speed of the object can be derived more easily.

Furthermore, according to the present embodiment, when the phase image has a wrapped phase, the phase difference signal is derived after unwrapping the phase of each pixel of the phase image. Therefore, the phase difference signal can be derived with higher accuracy.

Further, according to the present embodiment, the time series signal of the representative value of the phase difference signal in the extracted divided region is specified as being the vibration component of the photographing device. Therefore, the vibration component of the imaging device can be specified more easily than when the representative value is not applied.

Further, according to the present embodiment, the physical quantity is derived only for a predetermined part of the moving image. Therefore, calculation load can be reduced.

Further, according to the present embodiment, the above-mentioned part of the area is an area where the S/N ratio in the moving image is equal to or higher than a predetermined level. Therefore, vibration components can be detected with high accuracy while reducing the calculation load more effectively.

Furthermore, according to the present embodiment, the divided areas are areas for each pixel in a moving image or areas for each group of pixels. Therefore, the vibration component of the photographing device can be detected more simply and with high precision.

Note that in the above embodiment, a case has been described in which a phase image is derived for the entire moving image indicated by processing target moving image information, but the present invention is not limited to this. For example, if the method is limited to a convolution kernel, the process of step 204 in FIG. 4 is executed prior to the process of step 202, and the spatial information is limited to only the surrounding pixels necessary for convolution with a specific pixel of interest. It is also possible to perform the process up to calculating the phase difference signal, and then repeat the process of step 202 and step 208 while moving the pixel of interest to calculate the phase difference signal of all the necessary pixels. In this embodiment, intermediate processing information up to the final determination can be sequentially discarded during processing, making it possible to minimize the storage capacity required during execution of a series of processing.

Further, in the above embodiment, for example, the hardware structure of a processing unit that executes each process of the acquisition unit 11A, derivation unit 11B, extraction unit 11C, and identification unit 11D includes the following various types. A processor can be used. As mentioned above, the various processors mentioned above include the CPU, which is a general-purpose processor that executes software (programs) and functions as a processing unit, as well as circuit configurations such as FPGA (Field-Programmable Gate Array) after manufacturing. A programmable logic device (PLD), which is a processor that can be changed, and a dedicated electric circuit, which is a processor that has a circuit configuration specifically designed to execute a specific process, such as an ASIC (Application Specific Integrated Circuit) etc. are included.

The processing unit may be configured with one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). It may be composed of. Further, the processing section may be configured with one processor.

As an example of configuring the processing unit with one processor, first, as typified by computers such as clients and servers, one processor is configured with a combination of one or more CPUs and software, and this processor is There is a form that functions as a processing section. Second, there is a form of using a processor, such as a system on chip (SoC), in which the functions of the entire system including a processing section are realized by one IC (Integrated Circuit) chip. In this way, the processing section is configured as a hardware structure using one or more of the various processors described above.

Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit (circuitry) that is a combination of circuit elements such as semiconductor elements can be used.

10 Image processing device 11 CPU
11A Acquisition unit 11B Derivation unit 11C Extraction unit 11D Specification unit 12 Memory 13 Storage unit 13A Vibration component identification program 13B Moving image database 13C Complex spatial filter database 14 Input unit 15 Display unit 16 Media reading/writing device 17 Recording medium 18 Communication I/F unit 20 Photography device

Claims (7)

an acquisition unit that acquires a moving image that includes a plurality of objects as subjects and that is obtained by photographing with a photographing device;
a derivation unit that derives a physical quantity indicating a moving speed of the object for each divided region obtained by dividing each frame image into a plurality of regions between each frame image in the moving image acquired by the acquisition unit;
an extraction unit that extracts a divided area corresponding to the most frequently occurring physical quantity among the physical quantities for each divided area derived by the deriving unit;
a specifying unit that specifies a vibration component of the divided region extracted by the extraction unit as a vibration component of the photographing device;
Equipped with
The deriving unit performs a complex spatial filtering process on the moving image to generate a phase image, and calculates a phase difference between temporally adjacent frame images of the phase image and indicating a difference for each of the divided regions. Deriving the signal as the physical quantity,
The identifying unit identifies a time series signal of the representative value of the phase difference signal in the divided region extracted by the extracting unit as a vibration component of the imaging device,
The vibration component includes the amplitude and temporal frequency of vibration of the imaging device,
Image processing device. When the phase image is a wrapped phase, the derivation unit derives the phase difference signal after performing an unwrapping process on the phase of each pixel of the phase image.
The image processing device according to claim 1 . The representative value of the phase difference signal is an average value, a median value, a maximum value, or a minimum value of the phase difference signal, respectively.
The image processing device according to claim 1 or claim 2 . The derivation unit derives the physical quantity only from a predetermined part of the moving image.
The image processing device according to any one of claims 1 to 3 . The partial area is an area where the S/N ratio in the moving image is equal to or higher than a predetermined level;
The image processing device according to claim 4 . The divided area is an area for each pixel in the moving image, or an area for each group of pixels,
The image processing device according to any one of claims 1 to 5 . Obtaining a moving image containing multiple objects as subjects and obtained by shooting with a shooting device,
Deriving a physical quantity indicating the moving speed of the object for each divided area obtained by dividing each frame image into a plurality of parts between each frame image in the acquired moving image,
Among the derived physical quantities for each divided region, extract the divided region corresponding to the most frequently occurring physical quantity,
identifying a vibration component of the extracted divided region as a vibration component of the imaging device;
An image processing program for causing a computer to perform processing ,
A phase image is generated by performing a complex space filtering process on the moving image, and a phase difference signal indicating a difference between temporally adjacent frame images of the phase image and for each divided area is used as the physical quantity. Derive,
identifying a time series signal of a representative value of the phase difference signal in the extracted divided region as being a vibration component of the imaging device;
The vibration component includes the amplitude and temporal frequency of vibration of the imaging device,
Image processing program .