JP2023002445A - Image processing device, image processing method, and computer program - Google Patents

Abstract

To reduce gaps between a plurality of videos caused by remaining blurring of video signals from each camera unit.SOLUTION: An image processing device includes means for correcting blurs of each of a plurality of video signals obtained by a plurality of camera units, means for acquiring phase differences of remaining blurs after correcting the blurs of each of the plurality of video signals by the means for correcting blurs, and means for aligning phases of the remaining blurs of the plurality of video signals when the phase differences are equal to or greater than a predetermined value.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device, an image processing method, a computer program, and the like, and more particularly to an image processing device and the like that processes video signals from a plurality of camera units.

2. Description of the Related Art In recent years, an imaging device (image processing device) having a plurality of camera units in one housing, such as a multi-lens camera, has been used.
In this imaging apparatus, each camera unit can be pan-tilt driven manually or automatically, so that each camera unit can be brought close to each other for photographing.
When the camera units are brought close to each other, in order to reduce the overlapping of the photographing areas of the camera units, the photographing angle of view of each camera unit is set narrow by performing a zoom operation.

If the angle of view of each camera unit is narrowed, even the slightest shake will be noticeable. It will be effective to reduce
However, in the electronic blur correction, since the clipping range is set in advance, if the camera is installed in an installation location with a large amount of shaking, the clipping amount is insufficient, resulting in residual blurring.

When each camera unit has a residual blur after blur correction, if the residual blur amount of each camera unit is the same and the residual blur phase of each camera unit is the same, there is not much of a problem. That is, when the images of the respective camera units are displayed side by side, the joints between the images of the respective camera units appear naturally as if they were one connected image without being misaligned.

However, since each camera unit is a mechanical mechanism that can be manually or automatically pan-tilt driven, mechanical rattling and errors are likely to occur, and accordingly the phase of the remaining blur between each camera unit is different. end up
Even if each camera unit has the same amount of residual blurring, if the phase of the residual blurring of each camera unit differs, a problem arises when the images of the respective camera units are displayed side by side. In other words, the joints between the images of the camera units are misaligned with the adjacent images and look unnatural.

In Patent Document 1, a camera shake correction amount signal indicating the amount of camera shake detected by a first imaging apparatus is supplied to a second imaging apparatus, and a camera shake correction amount indicating the amount of camera shake detected by the second imaging apparatus is supplied. A signal is provided to the first imaging device. Then, in the first image pickup device and the second image pickup device, each camera shake correction is performed based on the camera shake amount that is the average of the camera shake amount detected by the self apparatus side and the camera shake amount detected by the other apparatus side. By performing, it is possible to capture a high-quality three-dimensional image with less three-dimensional motion sickness.

In Japanese Patent Laid-Open No. 2004-100000, the amount of background shake is calculated from the shake signal and the amount of motion, image blur correction is performed, and a plurality of captured images subjected to the image blur correction are combined to create a beautiful image with suppressed background blur. It is possible to acquire a wide range of panoramic images.

JP 2012-142837 A JP 2019-125880 A

However, in Patent Literature 1 disclosed in the above-mentioned Patent Literature, blur correction of each imaging device is performed using an average blur amount of the first imaging device and the second imaging device. However, the case where the phase of the remaining blur is different is not considered. Therefore, if the phases of the remaining blurring are different, the image will not be seen as a continuous image.
In the technique described in Patent Document 2, a plurality of shot images subjected to image blur correction are synthesized, but since the phase of the remaining blur after image blur correction is not considered, the joints of the synthesized panoramic images are It can become unnatural.

In view of the above problems, an object of the present invention is to provide an image processing apparatus and the like that can reduce the displacement of the joints of a plurality of images caused by residual blurring of image signals from a plurality of camera units. be.

The image processing device of the present invention is
blur correction means for respectively performing blur correction on a plurality of video signals obtained by a plurality of camera units;
phase difference acquisition means for acquiring a phase difference of a residual blur remaining after each of the plurality of video signals is subjected to blur correction by the blur correction means;
and phase adjusting means for adjusting the phases of the blur residue of the plurality of video signals when the phase difference is equal to or greater than a predetermined value.

According to the present invention, it is possible to reduce the deviation of the joints of a plurality of images caused by residual blurring of image signals from a plurality of camera units.

1 is a block diagram showing a configuration example of an image processing apparatus 1000 according to Embodiment 1; FIG. 4 is a flow chart showing processing in the first embodiment; FIG. 3 is a flowchart illustrating details of step S203 in FIG. 2; FIG. 3 is a block diagram showing a detailed configuration example of a blur correction amount calculation unit; FIG. FIG. 10 is a diagram showing an example of a waveform signal of residual blur; FIG. 10 is a diagram showing images of a first camera unit and a second camera unit connected side by side and displayed by a display unit 217; 10 is a flowchart showing processing in Example 2. FIG. 11 is a flow chart showing processing in Example 3. FIG. 10 is a flow chart showing processing in Example 4. FIG. 10 is a flow chart showing processing in Example 5. FIG. 14 is a flow chart showing processing in Example 6. FIG. 14 is a flow chart showing processing in Example 7. FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the present invention will be described below using examples with reference to the accompanying drawings. In each drawing, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified.
Also, in the embodiment, an example in which the image processing apparatus is applied to a network camera will be described. However, image processing devices include electronic devices such as digital still cameras, digital movie cameras, smart phones, personal computers, vehicle-mounted cameras, drone cameras, and robots.

FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 1000 according to the first embodiment.
Some of the functional blocks shown in FIG. 1 are executed by causing a computer (not shown), which is control means, included in the image processing apparatus 1000 to execute a computer program stored in a memory (not shown), which is a storage medium. Realized. However, some or all of them may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used.

The image processing apparatus 1000 in FIG. 1 has a first camera unit 100 and a second camera unit 200 . Since the functional blocks 101 to 112 of the first camera unit 100 and the functional blocks 201 to 212 of the second camera unit 200 have the same configuration, the functional blocks of the first camera unit will be mainly described below. do.
In FIG. 1, a lens group 101 of a first camera unit 100 is an optical system that converges light incident from a subject onto an imaging device 103 . The lens group 101 includes a focus lens for focusing on a subject, a zoom lens for adjusting the angle of view, and the like.

A subject image entering the camera through the lens group 101 passes through an optical filter 102 such as an infrared cut filter IRCF and enters an image sensor 103 .
A subject image formed through the lens group 101 passes through a color filter with a predetermined pattern arranged on the light receiving surface of the image sensor 103, is photoelectrically converted by each pixel of the image sensor 103, and is output as an analog image signal. be.

An AGC (Auto Gain Control) 104 adjusts the level of an image signal output from the image sensor 103 by gain control, and an A/D converter 105 converts the signal into a digital image signal.
The video signal processing unit 106 performs predetermined processing on the digital image signal from the A/D conversion unit 105, outputs a video signal composed of a luminance signal and a color signal, and generates various parameters for camera control.

Various parameters for camera control include, for example, parameters used for aperture control, focus control, and white balance control for color adjustment.
The exposure control unit 207 calculates luminance information in the shooting screen from luminance information as a parameter output from the video signal processing unit 106, and controls the diaphragm and AGC 104 so as to adjust the shooting image to desired brightness.

The optical control unit 108 extracts high frequency components from the video signal generated by the video signal processing unit 106 for focusing control. Then, the lens group 101 is controlled so that the focus evaluation value is maximized using the value of the high frequency component as focus information (focus evaluation value). The optical control unit 108 also performs focal length adjustment of the lens group 101 and insertion/removal control of the optical filter 102 according to the luminance level.

A blur correction amount calculation unit 109 calculates a blur correction amount by performing signal processing such as a digital filter on angular velocity information as a blur signal acquired from the angular velocity sensor 215 . Then, electronic blur correction is performed by changing the position (reading area) of the image from the image memory 409, which will be described later, in accordance with the calculated amount of blur correction. That is, the blur correction amount calculation unit 109 functions as blur correction means for executing a blur correction process for performing blur correction on each of a plurality of video signals obtained by a plurality of camera units using a blur signal for each video signal. are doing.

The image signal output unit 110 outputs the image subjected to the electronic blur correction in a predetermined format, and transmits the image to the display unit 217 via the communication unit 216 that performs wired or wireless communication. supply to
Here, the angular velocity sensor 215 functions as a shake sensor that detects shake and outputs a shake signal. However, the blur sensor may output a blur signal by calculating, for example, the direction of the average motion vector in the video signal and the amount of the average motion vector by the residual blur amount calculator 111 .
The display unit 217 may be separate from or integrated with the image processing apparatus 1000, and functions as display means for arranging and displaying images of a plurality of image signals after blur correction.

The residual blur amount calculation unit 111 calculates how much residual blur occurs in the image after the electronic blur correction output from the video signal output unit 110 . That is, for example, the average value of the motion vectors of the entire screen can be used as the residual blur amount.
A residual blur phase acquisition unit 112 samples the residual blur amount calculated by the residual blur amount calculation unit 111 at predetermined time intervals, stores the samples for a predetermined period, and generates a waveform signal of the residual blur amount. The functional blocks 201 to 212 of the second camera unit also perform the same operations as the functional blocks 101 to 112 of the first camera unit.

The residual blur phase difference calculation unit 213 calculates the residual blur waveform signal acquired by the residual blur phase acquisition unit 112 of the first camera unit and the waveform of the residual blur amount acquired by the residual blur phase acquisition unit 212 of the second camera unit. Compare signals. Thereby, the residual phase difference between the first camera unit and the second camera unit is calculated. Here, the blur residual phase difference calculation unit 213 is a phase difference acquisition unit that executes a phase difference acquisition step of acquiring a phase difference of the residual blur remaining after the blur correction is performed on each of the plurality of video signals by the blur correction unit. It is functioning.

The phase difference adjustment unit 214 changes the phase of the digital filter by how much, based on the phase difference between the first camera unit and the second camera unit, which is calculated by the phase difference calculation unit 213 for remaining blur. or Then, the phase of at least one of the digital filter used in the blur correction amount calculation unit 109 of the first camera unit and the digital filter used in the blur correction amount calculation unit 209 of the second camera unit is changed. .

In other words, the phase difference adjustment unit 214 functions as a phase adjustment unit that performs a phase adjustment step of matching phases of residual blurs of a plurality of video signals when the phase difference is equal to or greater than a predetermined value.
Although the phase of the digital filter (phase lead or phase lag filter) is changed in the first embodiment, a delay circuit may be provided to change the phase of the blur correction signal.

1 (for example, the first camera unit 100, the second camera unit 200, etc.) of the image processing apparatus shown in FIG. . However, they do not have to be built in the same housing, and may be composed of separate devices connected to each other via signal paths.
That is, the plurality of camera units and the like may be separate units, and the image processing apparatus 1000 may be a general-purpose computer, server, or the like that processes video signals from the plurality of camera units, for example.

FIG. 2 is a flow chart showing processing in the first embodiment. Hereinafter, referring to FIG. 2, phase matching of the residual blur of the first camera unit and the second camera unit according to the first embodiment of the present invention will be described. 2 is executed by causing a computer (not shown) included in the image processing apparatus 1000 as a control means to execute a computer program stored in a memory (not shown) as a storage medium. The same applies to the flow charts of FIGS. 3 and 7-12.

First, in step S201 in FIG. 2, the residual blur amounts of the first camera unit and the second camera unit are acquired. That is, how much residual blur occurs in the image after electronic blur correction is calculated using, for example, the amount of motion vector.
In the first embodiment, the amount of motion vector is used to calculate the amount of residual blur, but the amount of residual blur may be calculated using a gyro signal in combination.

In step S202, it is determined whether or not the residual blur amount calculated in step S201 is equal to or greater than a predetermined value. As a result of the determination, if the residual blur amount is not equal to or greater than the predetermined value (NO), the flow of FIG. 2 is terminated without matching the phases of the first camera unit and the second camera unit.
This is because when the remaining amount of blurring is very small, even if the phases of the first camera unit and the second camera unit are different, when the images of the respective camera units are displayed side by side and connected, the joint portion will not be shifted. This is because it is minute and does not stand out and does not look unnatural.

If the result of determination in step S202 is that the residual blur amount is equal to or greater than the predetermined value (YES), the residual blur phase difference between the first camera unit and the second camera unit calculated in step S201 is equal to or greater than the predetermined value. (step S203).
As a result of the determination in step S203, if the phase difference of the remaining blur is smaller than the predetermined value (NO), the flow of FIG. 2 ends without matching the phases of the first camera unit and the second camera unit.

This is because when the phase difference is minute, when the images of the respective camera units are displayed side by side in a connected manner, the gaps at the seams are minute and do not stand out and do not look unnatural.
As a result of determination in step S203, if the residual blur phase difference is equal to or greater than the predetermined value (YES), the residual blur phases of the first camera unit and the second camera unit are matched (step S204). As described above, in this embodiment, the phase adjusting means adjusts the phases of the remaining blurring of the plurality of video signals when the remaining blurring amounts of the plurality of video signals are equal to or greater than a predetermined value and the phase difference is equal to or greater than a predetermined value. Matching. However, even if the residual blur amounts of the plurality of video signals are not equal to or greater than the predetermined value, the residual blur phases of the plurality of video signals may be matched when the phase difference is equal to or greater than the predetermined value.

FIG. 3 is a flow chart explaining the details of step S203 in FIG.
Hereinafter, a detailed flow for aligning the phase of the residual blur will be described with reference to FIG. 3 .
First, in S301 of FIG. 3, the residual waveform signals of the first camera unit and the second camera unit are acquired.
By sampling the amount of residual blur of each camera unit at predetermined time intervals and storing it for a predetermined period of time, a waveform signal of the residual blur is generated. The predetermined time interval and the predetermined period are appropriately changed according to the imaging time, system frequency, and blur frequency.

In step S302, the waveform signal of the residual blur of the first camera unit and the waveform signal of the residual blur of the second camera unit acquired in step S301 are compared, and the waveform signal of the residual blur of the first camera unit and the residual blur of the second camera unit are compared. Calculate the phase difference of
In step S303, it is determined whether the phase difference calculated in step S302 is equal to or greater than a predetermined value. As a result of determination in step S303, if the residual blur phase difference is not equal to or greater than the predetermined value (NO), the flow of FIG. 3 ends without matching the phases of the first camera unit and the second camera unit.

As a result of determination in step S303, if the residual blur phase difference is equal to or greater than the predetermined value (YES), the phase of the blur correction signal of the blur correction amount calculator 109 of the first camera unit is changed (step S304).

FIG. 4 is a block diagram showing a detailed configuration example of the blur correction amount calculation unit.
The shake correction amount calculation units 109 and 209 in FIG. 1 acquire the shake generated in the image processing apparatus as an angular velocity signal from the angular velocity sensor 215, and the A/D converter 402 outputs the acquired angular velocity information as a digital signal.
The gain adjustment unit 403 adjusts the amplitude of the digital signal by multiplying the digital signal output from the A/D conversion unit 402 by a predetermined coefficient.

A phase lead filter 404 is a filter for leading the phase of a digital signal, and an HPF 405 performs filtering in a predetermined frequency band.
Phase delay filter 406 is a filter for delaying the phase of the signal filtered by HPF 405 . A focal length calculation unit 407 acquires focal length information of the lens group 101 from the optical control unit 108 and adjusts the magnitude of the signal so that the blur correction amount corresponds to the focal length.

An integration processing unit 408 integrates the signal calculated by the focal length calculation unit 407 using an LPF or the like to calculate a final blur correction amount.
An image memory 409 is a memory that temporarily stores the video signal from the video signal processing unit 106 .
A clipping position changing unit 410 changes the clipping position of the image stored in the image memory 409 based on the blur correction amount and blur direction information obtained from the integration processing unit 408 , thereby correcting blurring in the video. The shake-corrected image is supplied to the image signal output unit 110 in the subsequent stage.

In step S304 of FIG. 3, the phase of the blur correction signal of the first camera unit is changed by changing at least one of the phase lead filter 404 and the phase lag filter 406 of FIG. ing. However, as described above, a delay circuit or the like for changing the phase of the blur correction signal may be provided separately. As described above, in the first embodiment, by adjusting the phase of the blur signal for performing blur correction of at least one of the plurality of video signals, the phases of the remaining blurs of the plurality of video signals are matched.

In step S305, the residual phase difference between the first camera unit and the second camera unit after changing the phase of the blur correction signal of the digital filter of the first camera unit is calculated again. The calculation method is the same as the method used in steps S301 and S302.
Before changing the digital filter of the first camera unit in step S304, the residual blur phase difference of each camera unit calculated in step S305 is compared with the residual blur phase difference of each camera unit calculated in step S302. . Accordingly, it is determined whether the phase difference of each camera unit has decreased (step S306).

As a result of determination in step S306, if the phase difference of each camera unit has decreased (YES), nothing is done and the flow of FIG. 3 ends.
If the phase difference of each camera unit has not decreased as a result of determination in step S306 (NO), the process proceeds to step S307. Then, in step S307, the phase of the blur correction signal is changed by changing at least one of the phase lead filter and the phase lag filter in the blur correction amount calculator 209 of the second camera unit (step S307).

In the above, the phase of the blur correction signal of the first camera unit is changed first, but the phase of the blur correction signal of the second camera unit may be changed first.
As described above, in this embodiment, when the phase difference is equal to or greater than a predetermined value, the phase of one of the plurality of video signals is adjusted. further adjusts the phase of the residual blur of the other video signal among the plurality of video signals.

FIG. 5 is a diagram showing an example of a waveform signal of the remaining blur.
A waveform signal drawn with a dotted line in FIG. 5 indicates a waveform signal of residual blur of the first camera unit, and a waveform signal drawn with a solid line of FIG. 5 indicates a waveform signal of residual blur of the second camera unit. show.
By changing the digital filter of the first camera unit or the second camera unit, the phase shift of the waveform signal in each camera unit as shown in FIG. becomes a uniform waveform signal.

FIG. 6 is a diagram in which the images of the first camera unit and the second camera unit are displayed side by side by the display unit 217 . As shown in FIG. 6, if the phase difference of the residual blur of each camera unit is not matched, the joint portion is shifted as shown by the dotted line in the upper diagram of FIG. 6, resulting in an unnatural image.
On the other hand, according to the present embodiment, by matching the phase difference of the residual blur of each camera unit, it is possible to obtain a natural image without shifting the joint portion as shown in the lower diagram of FIG.

A second embodiment of the present invention will be described below with reference to FIG.
FIG. 7 is a flow chart showing the processing flow of the second embodiment. Steps S701 to S703, S705 and S706 are the same as steps S301 to S303, S305 and S306 in FIG.
First, in step S701 in FIG. 7, the waveform signals of the residual blur of the first camera unit and the second camera unit are acquired. A method for acquiring the waveform signal of the remaining blur may be the same as the method in step S301 of the flowchart in FIG. 3 of the first embodiment.

By comparing the residual blur waveform signals of the respective camera units acquired in step S701, the residual blur phase difference between the first camera unit and the second camera unit is calculated (step S702).
In step S703, it is determined whether the phase difference calculated in step S702 is equal to or greater than a predetermined value. As a result of determination in step S703, if the residual blur phase difference is not equal to or greater than the predetermined value (NO), the flow in FIG. 7 ends without matching the phases of the first camera unit and the second camera unit.

As a result of determination in step S703, if the phase difference of the residual blur is equal to or greater than the predetermined value (YES), the imaging timing of the first camera unit is changed (step S704).
In step S705, the residual phase difference between the first camera unit and the second camera unit after changing the imaging timing of the first camera unit is calculated again. The calculation method is the same as that used in steps S701 and S702.

By comparing the residual blur phase difference of each camera unit calculated in step S705 with the residual blur phase difference of each camera unit calculated in step S702, it is determined whether the phase difference of each camera unit has decreased. (Step S706).
As a result of determination in step S706, if the phase difference of each camera unit has decreased (YES), nothing is done and the process ends.

As a result of determination in step S706, if the phase difference of each camera unit has not decreased (NO), the imaging timing of the second camera unit is changed (step S707).
In the above description, the imaging timing of the first camera unit is changed first, but the imaging timing of the second camera unit may be changed first.
As described above, in the second embodiment, the phase adjusting means adjusts the phases of the residual blurs of the plurality of video signals by adjusting the imaging timing of at least one of the plurality of camera units.

FIG. 8 is a flow chart showing processing in the third embodiment. In FIG. 8, steps S801 to S803, S805, and S806 are the same as steps S701 to S703, S705, and S706 in FIG. 7, so description thereof is partially omitted.
In the third embodiment, the sampling timing of the gyro signal, which is the output of the angular velocity sensor, is changed in order to match the phases of the camera units.

That is, in FIG. 8, as a result of determining the phase difference of the residual blur between the first camera unit and the second camera unit in step S803, if the phase difference of the residual blur is equal to or greater than a predetermined value (YES), the process proceeds to step S804. move on. Then, in step S804, by changing the sampling timing for acquiring the gyro signal as the shake signal of the first camera unit, the phase difference between each camera unit is adjusted (step S804).

Furthermore, if the phase difference has not decreased in step S806, the sampling timing for acquiring the gyro signal of the second camera unit is changed in step S807.
As described above, in the third embodiment, by changing the sampling timing of the output of the angular velocity sensor as the blurring signal for performing blur correction of at least one of the plurality of video signals, the remaining phase of the blurring of the plurality of video signals can be adjusted. Matching.

FIG. 9 is a flow chart showing processing in the fourth embodiment. In FIG. 9, steps S901 to S903, S905, and S906 are the same as steps S801 to S803, S805, and S806 in FIG. 8, so description thereof is partially omitted.
In Example 4, the imaging time is changed in order to match the phase of each camera unit. Here, the imaging time may be changed by changing the shutter speed of a mechanical shutter, or may be changed by changing the accumulation time of the image sensor. Alternatively, both may be combined and changed.

In FIG. 9, as a result of determining the phase difference of the residual blur between the first camera unit and the second camera unit in step S903, if the phase difference of the residual blur is equal to or greater than a predetermined value (YES), the process proceeds to step S904. move on. Then, in step S904, by changing the imaging time of the first camera unit, the phase difference between each camera unit is matched (step S904).

Furthermore, if the phase difference has not decreased in step S906, then in step S907 the imaging time of the second camera unit is changed to match the phase difference between the camera units.
As described above, in the fourth embodiment, the phase adjusting means adjusts the phases of the residual blurs of the plurality of video signals by adjusting the imaging time of at least one of the plurality of camera units.

FIG. 10 is a flow chart showing processing in the fifth embodiment.
In Examples 1 to 4, only the phase of the blur correction signal, the imaging timing, and the imaging time were changed in order to match the phase difference of the residual blur of each camera unit. However, in the fifth embodiment, both the imaging timing and the phase of the blur correction signal are changed according to the phase difference of the residual blur.

First, in step S1001, the residual phase difference between the first camera unit and the second camera unit is calculated. Then, in step S1002, it is determined whether the calculated phase difference is greater than or equal to the first threshold (Th1).
As a result of determination in step S1002, if the phase difference is not equal to or greater than the first threshold (NO), the process proceeds to step S1005.

On the other hand, if the result of determination in step S1002 is that the phase difference is equal to or greater than the first threshold (YES), then in step S1003 the imaging timing of the first camera unit or the second camera unit is changed first.
After the imaging timing is changed in step S1003, the residual phase difference between the first camera unit and the second camera unit is calculated again in step S1004, and the process proceeds to step S1005.

In step S1005, it is determined whether the phase difference of the remaining blur is equal to or less than the first threshold, equal to or more than the second threshold (Th2), and less than the first threshold (Th1) (step S1005). Here, Th2<Th1.
If the result of determination in step S1005 is that the phase difference of the residual blur is greater than or equal to the second threshold and less than the first threshold (YES), the process proceeds to step S1006. Then, in step S1006, the phase of the blur correction signal is changed by the digital filter used in the blur correction of the first camera unit or the second camera unit.
If NO in step S1005, nothing is done and the flow of FIG. 10 ends.

FIG. 11 is a flow chart showing processing in the sixth embodiment.
In Examples 1 to 5, the case where the number of camera units for adjacent display is two has been described as an example, but in Example 6, when the number of camera units for adjacent display is three or more, which Determines whether the camera unit is mainly used for phase matching.
First, in step S1101, the number of camera units for adjacent display is obtained. It is determined whether or not the number of acquired camera units is 3 or more (step S1102), and as a result of determination, if the number of camera units is not 3 or more (NO), no processing is performed as it is and the flow of FIG. 11 is performed. exit.

If the result of determination in step S1102 is that the number of camera units is three or more (YES), it is determined whether there is a camera unit in which a person is captured in the image (step S1003).
If the result of determination in step S1103 is that there is a camera unit in which a person is captured in the image (YES), the phases of other camera units are adjusted so as to match the phase with the camera unit in which a person is captured in the image. (step S1104).

If the result of determination in step S1103 is that there is no camera unit in which a person is captured in the image, the process ends without performing any processing. In the sixth embodiment, when the images of three or more camera units are displayed adjacently, the phases of the other camera units are adjusted to the camera unit in which the person is captured. However, when a specific subject is designated in advance, the phase of another camera unit may be matched with the phase of the camera unit in which the specific subject is captured.

Furthermore, even when the images of two camera units are displayed adjacently, the phase of the other camera unit may be adjusted to the phase of the camera unit in which the specific subject is captured.
As described above, in this embodiment, when the phase difference is equal to or greater than a predetermined value, the phase adjustment means adjusts the phase of the residual blur of the other video signal in accordance with the phase of the residual blur of one of the plurality of video signals. adjusting the phase. Moreover, the residual blur phases of the other video signals are adjusted in accordance with the phase of the residual blur of the video signal in which the predetermined subject is shown among the plurality of video signals.

FIG. 12 is a flow chart showing processing in the seventh embodiment.
In the seventh embodiment, the frequency of shake (vibration) occurring in the camera unit is detected by the angular velocity sensor 215 or the like as a shake sensor, and the intensity of phase matching is changed according to the frequency.
First, in step S1201, frequency detection of blurring (vibration) occurring in the camera unit is performed based on a blurring signal obtained from the angular velocity sensor 215 or the like as a blurring sensor. In step S1201, it is determined whether the frequency of the shake (vibration) detected is equal to or less than a predetermined frequency (step S1202).
If the result of determination in step S1202 is that the frequency of the detected shaking (vibration) is equal to or less than the predetermined frequency (YES), the degree of phase matching is increased in step S1203. That is, the phases are made to match more.
On the other hand, if the frequency of the detected shake (vibration) is greater than the predetermined frequency as a result of determination in step S1202 (NO), the degree of phase matching is decreased in step S1204.

The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention. They are not excluded from the scope of the invention.

It should be noted that a computer program that implements the functions of the above-described embodiments for part or all of the control in this embodiment may be supplied to an image processing apparatus or the like via a network or various storage media. A computer (or CPU, MPU, etc.) in the image processing apparatus or the like may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

1000 image processing device 100 first camera unit 200 second camera unit 109 blur correction amount calculator 111 residual blur amount calculator 112 residual blur phase acquisition unit 213 residual blur phase difference calculator 214 phase difference adjuster 215 angular velocity sensor

Claims (17)

blur correction means for respectively performing blur correction on a plurality of video signals obtained by a plurality of camera units;
phase difference acquisition means for acquiring a phase difference of a residual blur remaining after each of the plurality of video signals is subjected to blur correction by the blur correction means;
and phase adjusting means for adjusting the phases of the blur residue of the plurality of video signals when the phase difference is equal to or greater than a predetermined value. 2. The image processing apparatus according to claim 1, wherein said phase adjusting means adjusts the phases of said plurality of video signals for said remaining blurring amount when the remaining blurring amount of said plurality of video signals is equal to or greater than a predetermined value. . 3. The image processing apparatus according to claim 1, wherein the phase difference obtaining means calculates the phase difference of the blur residue using a waveform signal of the blur residue. Furthermore, it has a blurring sensor that detects blurring and outputs a blurring signal,
4. The image processing according to claim 1, wherein said blur correction means performs blur correction using said blur signal obtained from said blur sensor for each of said plurality of video signals. Device. 5. The image processing apparatus according to claim 4, wherein said phase adjusting means adjusts the phase of said blur signal for performing blur correction of at least one of said plurality of video signals. 5. The image processing apparatus according to claim 4, wherein said phase adjusting means adjusts sampling timing of said blur signal for performing blur correction of at least one of said plurality of video signals. 7. The image processing apparatus according to any one of claims 1 to 6, wherein said phase adjusting means adjusts imaging timing of at least one of said plurality of camera units. 7. The image processing apparatus according to any one of claims 1 to 6, wherein said phase adjustment means adjusts the imaging time of at least one of said plurality of camera units. The phase adjustment means adjusts the phase of the residual blur of one of the plurality of video signals when the phase difference is equal to or greater than the predetermined value, and the residual blur is equal to or greater than the predetermined value after adjustment. The image processing apparatus according to any one of claims 1 to 8, further comprising adjusting the phase of the remaining image signal of another image signal among the plurality of image signals in some cases. When the phase difference is equal to or greater than the predetermined value, the phase adjustment means adjusts the phase of the residual blur of the other video signal in accordance with the phase of the residual blur of one of the plurality of video signals. 9. The image processing apparatus according to any one of claims 1 to 8, characterized in that: When the phase difference is equal to or greater than the predetermined value, the phase adjustment means adjusts the phase difference of the other video signal in accordance with the phase of the remaining video signal of the video signal showing a predetermined subject among the plurality of video signals. 11. The image processing apparatus according to claim 10, wherein the remaining phases are adjusted. Furthermore, it has a blurring sensor that detects blurring and outputs a blurring signal,
2. The image processing apparatus according to claim 1, wherein the degree of matching the phase difference is changed according to the frequency of the shake signal output from the shake sensor. 13. The image processing apparatus according to claim 12, wherein when the detected frequency is equal to or lower than a predetermined frequency, the degree of matching the phase difference is increased. 13. The image processing apparatus according to claim 12, wherein the degree of matching the phase difference is weakened when the detected frequency is higher than a predetermined frequency. The image processing apparatus according to any one of claims 1 to 11, wherein the plurality of camera units are built in the same housing. a blur correction step of performing blur correction on each of a plurality of video signals obtained by a plurality of camera units;
a phase difference acquisition step of acquiring a phase difference of a residual blur remaining after performing blur correction on each of the plurality of video signals by the blur correction step;
and a phase adjustment step of matching the phases of the blur residue of the plurality of video signals when the phase difference is equal to or greater than a predetermined value. A computer program for controlling each means of the image processing apparatus according to any one of claims 1 to 15 by a computer.

Images