JP2003058893A - Processor, image pickup device and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor which accurately detects the movement of an object or photograph picture frame of a camera on the basis of photographed image information having a small number of pixels, and an image pickup device in which the image processor can be packaged. SOLUTION: The image pickup device is provided with a detection part 26 for detecting the movement of an image from each of a plurality of detection areas in a picture inputted by an imaging device and determining the movement of an entire or partial picture from the movement in each of areas, and by decomposing the movement of the image into two axial directions (horizontal direction and vertical direction) and changing a method for selecting the detection area corresponding to each of axial directions, the detection area suited to the axial direction to detect the movement can be segmented. Thus, since a large quantity of information is taken in a moving direction corresponding to the direction of the movement, a detected moving amount becomes more highly reliable and even when using the imaging device having a comparatively small number of pixels, the movement of the object or image pickup picture frame of the image pickup device can be accurately detected from the image information.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus capable of detecting a motion of an image as a motion vector, an image pickup apparatus capable of mounting the image processing apparatus, and a control method thereof.

2. Description of the Related Art In an optical device such as a camera or an image pickup apparatus, the movement of a subject or the movement of a camera image frame (pan or blur) is detected based on image information input from an image pickup device at a certain time interval. Various techniques have been proposed so far. Also, as a preprocessing technology to detect the movement of the main subject, we also propose a technology that detects the movement of the camera from the information of the peripheral part of the screen and detects and calculates the movement of the main subject based on that information. Has been done.

However, in the conventional technique, an image pickup device having a very large number of pixels is required to accurately detect the movement of the subject or the image frame of the camera based on the image information. . In particular, when only some areas in the screen are used for calculation, the detection area alone must have a sufficient number of pixels for image motion detection. Therefore, a very large number of pixels is required for the entire screen.
Therefore, it is difficult to detect the movement of the subject or the movement of the image frame of the camera (panning or blurring) based on the image sensor having a small number of pixels. The present invention has been made in order to solve the above-mentioned problems of the prior art, and an object thereof is to accurately move a subject or a shooting image frame of a camera based on image information of a small number of pixels taken. An object of the present invention is to provide a detectable image processing device, an imaging device capable of mounting the image processing device, and a control method thereof.

An image pickup apparatus according to an embodiment of the present invention for achieving the above object has the following configuration. That is, the image pickup apparatus has an image pickup unit that continuously picks up a subject image and generates a plurality of images, and is information regarding the luminance of a pixel group that forms a rectangular area that is long in the vertical direction of the image frame of the image. First obtaining means for obtaining the vertical direction luminance information, and second obtaining means for obtaining the horizontal direction luminance information which is information relating to the luminance of a pixel group forming a horizontally long rectangular area of the image frame of the image. And a movement amount generation unit that generates information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information.

Here, for example, the vertically long rectangular area is a left end portion or a right end portion in the image frame, and the horizontally long rectangular area is an upper end portion in the image frame. Part or lower end part.

Here, for example, the movement amount detecting means is
The vertical movement amount of the image frame is calculated from the vertical direction luminance information, the horizontal movement amount of the image frame is calculated from the horizontal direction luminance information, and the image frame is calculated based on the calculated movement amount. It is preferable to generate a movement amount of

Here, for example, correction means for correcting a photographed image shift that occurs between the plurality of images based on the information regarding the movement amount of the image frame, and the correction images are used in the plurality of images. It is preferable to further include a detection unit that detects an existing region of an existing moving body.

A processor according to an embodiment of the present invention for achieving the above object has the following configuration. That is, a processor that generates information on the amount of movement of image frames that occur between the plurality of images based on a plurality of images that are input, and that is a pixel that forms a rectangular area that is long in the vertical direction of the image frames of the images. First acquisition means for acquiring vertical direction brightness information that is information about the brightness of a group, and horizontal brightness information that is information about the brightness of a pixel group that forms a rectangular region that is long in the horizontal direction of the image frame of the image. And a movement amount generation unit that generates information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information. Is characterized by.

Here, for example, the vertically long rectangular area is a left end portion or a right end portion in the photographing image frame, and the horizontally long rectangular area is an upper portion in the image frame. It is preferably an end or a lower end.

Here, for example, the movement amount detecting means is
The vertical movement amount of the photographing image frame is calculated from the vertical direction luminance information, the horizontal movement amount of the photographing image frame is calculated from the horizontal direction luminance information, and the movement amount is calculated based on the calculated movement amount. It is preferable to generate the movement amount of the photographing image frame.

[0008] Here, for example, a correction means for correcting the shift of the shooting image frame that occurs between the plurality of images based on the information regarding the movement amount of the shooting image frame, and the plurality of images using the corrected image. It is preferable to further include a detection unit that detects a region where a moving body exists in the data.

An image pickup apparatus control method according to an embodiment of the present invention for achieving the above object has the following configuration. That is, a method of controlling an image pickup apparatus having an image pickup step of continuously picking up a subject image and generating a plurality of images, which relates to the luminance of a pixel group forming a rectangular region long in the vertical direction of the image frame of the image. A first acquisition step of acquiring vertical luminance information that is information, and a second acquisition step of horizontal luminance information that is information regarding the luminance of a pixel group that forms a rectangular region that is long in the horizontal direction of the image frame of the image. An acquisition step, and a movement amount generation step of generating information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information. .

A processor according to an embodiment of the present invention for achieving the above object has the following configuration. That is, it is a control method of a processor that generates information on the amount of movement of image frames that occur between the plurality of images based on a plurality of input images, and a rectangular area that is long in the vertical direction of the image frames of the images. A first acquisition step of acquiring vertical direction brightness information that is information related to the brightness of a pixel group to be formed, and horizontal brightness that is information related to the brightness of a pixel group that forms a rectangular area that is long in the horizontal direction of the image frame of the image. A second acquisition step of acquiring information, and a movement amount generation step of generating information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information, It is characterized by having.

A program according to an embodiment of the present invention for achieving the above object has the following configuration. That is, it is characterized in that the information processing apparatus having the program code for realizing the above-described method for controlling the image pickup apparatus can be executed.

A storage medium according to an embodiment of the present invention for achieving the above object has the following configuration. That is, the above-mentioned program is stored.

A program according to an embodiment of the present invention for achieving the above object has the following configuration. That is, it is characterized in that an information processing apparatus having a program code for realizing the above-described processor control method can be executed.

A storage medium according to an embodiment of the present invention for achieving the above object has the following configuration. That is, the above-mentioned program is stored.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION An image processing apparatus as an example of a preferred embodiment according to the present invention, an image pickup apparatus equipped with this image processing apparatus, and a control method thereof will be described below with reference to the drawings.

In this embodiment, a processor is used as an image processing apparatus, and a so-called single-lens reflex type camera with interchangeable lenses is used as an image pickup apparatus. [Overall Configuration of Imaging Device: FIG. 1] FIG. 1 is a diagram illustrating an overall configuration of a camera, which is an imaging device, equipped with a processor that is an image processing device according to the present embodiment.

In FIG. 1, 10 is a camera body and 30 is an interchangeable lens.
Are exchangeable.

[Camera Body] First, the camera body 10 will be described. Reference numeral 11 is an optical axis of the taking lens, 12 is a film surface, 13 is a semi-transparent main mirror, and 14 is a first reflecting mirror. Both the main mirror 13 and the first reflecting mirror 14 jump up to the top during shooting, and deviate from the optical axis 11 of the shooting lens.

Reference numeral 15 is a paraxial image forming plane conjugate with the film surface 12 formed by the first reflecting mirror 14, and 16 is a second reflecting mirror.
Reference numeral 17 is an infrared cut filter, 18 is a diaphragm having two openings, 19 is a secondary imaging lens, and 20 is a focus detection sensor.

[Focus Detection Sensor: FIG. 2] The focus detection sensor 20 is composed of an accumulation type photoelectric conversion element in an area such as a CMOS or CCD. For example, as shown in FIG. 2, a plurality of light receiving sensor portions of the focus detection sensor 20 corresponding to the two openings of the diaphragm 18 form a pair of areas as shown in 20A and 20B. is doing.

In addition to the light receiving sensor units 20A and 20B, a signal storage unit (not shown), a peripheral circuit for signal processing (not shown), and the like are formed on the same chip as an integrated circuit.

The structure from the first reflecting mirror 14 to the focus detecting sensor 20 in FIG. 1 is disclosed in Japanese Patent Laid-Open No. 9-1849.
As described in detail in No. 65 and the like, it is possible to detect the focus by the image shift method at an arbitrary position within the photographing screen.

[Focus Detection Area: FIG. 3] In FIG. 3, since the focus detection is performed by the image shift method, FA1 is displayed in the photographing screen.
8 shows an example in which 25 points of focus detection areas of FA25 are provided.

Returning to the explanation of FIG. 1, 21 is a focusing plate having diffusivity, 22 is a penta prism, 23 is an eyepiece lens, 24 is a third reflecting mirror, 25 is a condenser lens, and 26 is a condenser lens.
Is a photometric sensor for obtaining information about the brightness of the subject.

[Photometric sensor: FIGS. 4 and 5] The photometric sensor 26 is composed of, for example, a storage photoelectric conversion element in an area such as a CMOS or CCD. For example,
As shown in FIG. 4, it is configured to have a plurality of light-receiving sensor portions divided into a lattice shape, and has a field of view substantially over the entire photographing screen. The plurality of divided light receiving sensor units shown in FIG. 4 are divided into P1, 1 to Pm, n, that is, m × n, as shown in FIG.

Further, in the photometric sensor 26, in addition to the light receiving sensor section, a signal storage section (not shown), a peripheral circuit for signal processing (not shown), and the like are formed as an integrated circuit on the same chip. .

Reference numeral 27 in FIG. 1 denotes a mount portion for mounting a photographing lens, and 28 denotes a contact portion for performing information communication with the photographing lens.

[Interchangeable Lens: FIG. 1] Next, the interchangeable lens 30
Will be described.

In FIG. 1, 31 is a diaphragm, 32 is a contact part for communicating information with the camera body, 33 is a mount part to be attached to the camera, and 34 to 37 are optical lenses constituting a taking lens.

[Structure of Electric Circuit of Imaging Device: FIG. 6] FIG.
An example of the configuration of the electric circuit of the imaging device is shown in FIG.

[Camera body: FIG. 6] First, the camera body 10
explain about.

41 is, for example, an ALU, ROM, RA inside
It is a control unit by a one-chip microcomputer having a built-in M or A / D converter, a timer function, a serial communication port, etc., and controls the entire camera mechanism and the like.
A specific control sequence of the control unit 41 will be described later.

The focus detecting sensor 20 and the photometric sensor 26 are the same as those shown in FIG. Output signals of the focus detection sensor 20 and the photometric sensor 26 are connected to an A / D converter input terminal of the control unit 41.

A shutter 42 is connected to an output terminal of the control unit 41 and controlled. A first motor driver 43 is connected to an output terminal of the control unit 41 and controlled, and drives a first motor 44 for feeding the film, driving the main mirror 13, and the like. Reference numeral 45 denotes a sensor that detects the attitude of the camera, and the output signal thereof is connected to the input terminal of the control unit 41. The control unit 41 uses the attitude detection sensor 4
By inputting the information of 5, it is possible to obtain information as to whether the image is held in the horizontal position or in the vertical position at the time of shooting.

Reference numeral 46 denotes the distance measuring sensor 20 under the condition of low illuminance.
Is an AF light source that projects infrared light or the like onto a subject when focus detection is performed by, and emits light in accordance with an output signal from the control unit 41. Reference numeral 47 is a light emitting unit that emits light when photographing when the brightness of the subject is insufficient, and is emitted by a flash or the like according to the output signal of the control unit 41.

Reference numeral 48 denotes a display device which is composed of a liquid crystal panel or the like and displays the number of shots, date information, shooting information and the like, and lighting of each segment is controlled according to the output signal of the control section 41. Reference numeral 49 denotes various switches including a release button and the like. Reference numeral 28 is the contact portion shown in FIG.
The input / output signals of one serial communication port are connected.

[Interchangeable lens: FIG. 6] Next, the interchangeable lens 30
Will be described.

51 is, for example, an ALU, ROM, R inside
It is a lens control unit by a one-chip microcomputer including an AM and a serial communication port. 52 is
A second motor driver, which is connected to an output terminal of the lens control unit 51 and is controlled to perform a second focus adjustment.
The motor 53 is driven.

Reference numeral 54 denotes a third motor driver, which is connected to the output terminal of the lens controller 51 and controlled to drive the third motor 55 for controlling the diaphragm 31 shown in FIG.

Reference numeral 56 denotes a distance encoder for obtaining the amount of extension of the focus adjustment lens, that is, information on the object distance, which is connected to the input terminal of the lens controller 51. Reference numeral 57 is a zoom encoder for obtaining focal length information at the time of shooting when the interchangeable lens 30 is a zoom lens, and is connected to the input terminal of the lens control unit 51.

Reference numeral 32 is the contact portion shown in FIG.
The input / output signals of the serial communication port of the lens controller 51 are connected.

When the interchangeable lens 30 is attached to the camera body 10, the respective contact points 28 and 32 are connected,
The lens control unit 51 can perform data communication with the control unit 41 of the camera body.

The optical information peculiar to the lens necessary for the control unit 41 of the camera body to perform focus detection and exposure calculation, and the information on the object distance or the focal length information based on the distance encoder 56 or the zoom encoder 57, , The lens controller 51 to the camera controller 4
1 is output by data communication.

The focus adjustment information and aperture information obtained as a result of the focus detection and the exposure calculation performed by the control unit 41 of the camera body are output from the control unit 41 of the camera body to the lens control unit 51 by data communication. The lens control unit 51 controls the second motor driver 52 according to the focus adjustment information, and controls the third motor driver 54 according to the aperture information.

[Operation of Control Unit] Next, the operation sequence of the control unit 41 of the camera body will be described with reference to the flowchart of FIG.

A power switch (not shown) is turned on to enable the controller 41 to operate, and the first release button (not shown) is activated.
When the stroke switch is turned on, the operation of the control unit 41 starts.

First, in step S101, the control section 41 outputs a control signal to the photometric sensor 26 to accumulate the first photometric signal.

Next, in step S102, the control section 41 controls each light receiving section P stored in the photometric sensor 26.
A / D conversion is performed while reading out the signals 1 to 1 to Pm and n, and the signals are stored in the RAM as information of the first accumulated signal.
At this time, various necessary data corrections such as shading are performed on each converted digital data.

Next, in step S103, the control unit 41 instructs the timer unit to wait until a predetermined time has elapsed, and when the predetermined time has elapsed, the process proceeds to step S104.

Next, in step S104, the control section 41 outputs a control signal to the photometric sensor 26 to accumulate the second photometric signal.

Next, in step S105, the control section 41 controls the light receiving sections P accumulated in the photometric sensor 26.
A / D conversion is performed while reading out the signals 1 to 1 to Pm and n, and stored in the RAM as second accumulated signal information. At this time, various necessary data corrections such as shading are performed on each converted digital data.

Next, in step S106, the control section 41 performs an exposure calculation by a known algorithm based on the information of the second accumulated signal. The brightness of the subject is obtained by the exposure calculation, and the shutter speed and aperture value that provide the proper exposure are determined. Also, it is determined whether or not the flash is to be emitted.

Next, in step S107, the control unit 41 detects the moving body area based on the information of the first accumulated signal and the information of the second accumulated signal. (A specific method for detecting this moving body area will be described later.) Next, step S10.
In 8, the control unit 41 selects an area where focus detection is to be performed from the focus detection areas FA1 to FA25 based on the moving body area detection result.

Next, in step S109, the control section 41 outputs a control signal to the focus detection sensor 20 to accumulate the signal.

Next, in step S110, the control section 41 performs A / D conversion while reading the signal accumulated in the focus detection sensor 20. Further, various necessary data corrections such as shading are performed on each converted digital data.

Next, in step S111, the control unit 41 inputs lens information and the like required for focus detection from the lens control unit 51, and based on this and digital data obtained from the focus detection sensor 20. Step S1
The focus state of the focus detection area selected at 08 is calculated. Further, the amount of lens movement for achieving focusing is calculated according to the calculated focus state.

Next, in step S112, the control section 41 outputs the calculated lens movement amount to the lens control section 51. In accordance with this, the lens controller 51 outputs a signal to the second motor driver 52 to drive the focus adjustment lens, and drives the second motor 53. As a result, the taking lenses 34 to 37 are in focus with respect to the subject. The controller 41 can obtain the distance information to the subject by obtaining the information about the subject distance based on the distance encoder 56 from the lens controller 51 after the focus state is achieved.

Next, in step S113, the control unit 41 waits for the second stroke switch of the shutter button to be turned on, and if not turned on, step S10.
Returning to step 1, when turned on, the process proceeds to step S114.

Next, in step S114, the control section 41 outputs a control signal to the first motor driver,
The first motor 44 is driven to flip up the main mirror 13 and the first reflection mirror 14.

Next, in step S115, the control section 41 outputs the aperture value information calculated in step S106 to the lens control section 51. According to this information, the lens control unit 51 outputs a signal to the third motor driver 54 so as to drive the diaphragm 31, and the third motor 55
To drive. As a result, the photographing lens is brought into a narrowed state.

Next, in step S116, the control unit 41 controls the shutter 42 according to the shutter speed calculated in step S106 to expose the film. When it is determined by the exposure calculation that the scene is to be shot using the light emitting unit 47, the light emitting unit 47 such as a strobe is caused to emit light.

Next, in step S117, the control section 41 outputs information to the lens control section 51 so as to open the diaphragm 31. The lens controller 5 according to this information
1 is the third motor driver 5 so as to drive the diaphragm 31
A signal is output to 4 to drive the third motor 55. As a result, the photographic lens is in the aperture open state.

Next, in step S118, a control signal is output to the first motor driver to drive the first motor 4
4 to drive the main mirror 13 and the first reflecting mirror 14
Bring down.

Next, in step S119, the control section 41 outputs a control signal to the first motor driver,
By driving the first motor 44 to wind the film, a series of photographing sequences is completed.

[Detailed Description of Moving Object Detection: FIG. 8] Next, the detailed content of the moving object detection executed in step S107 will be described with reference to the flowchart of FIG. It should be noted that the part that executes only the moving object detection may be integrated into one processor.

[Peripheral part of accumulated signal information: FIG. 9] First, in step S151, the one-dimensional projection information of the first accumulated signal information obtained in step S102 of FIG. 7 is calculated.

However, at this time, the peripheral portion of the first accumulated signal information is divided into blocks Ha to Hd (block group H) which are long in the horizontal direction, as shown in FIG. 9A. In each divided block, 1 in the horizontal direction (column direction)
Calculates dimensional projection information.

Here, the information of the peripheral portion except for the central portion shown in FIG. 9A is used because the projection information is based on the area of the background portion other than the main subject.

The peripheral part is divided into a plurality of blocks as shown in FIG.
This is because it is possible to more accurately calculate only the camera movement by excluding the block including the moving subject in step S153 later. Similarly, in the horizontal direction by the block group H of the second accumulated signal information obtained in step S105 of FIG.
The calculation is also performed by including the one-dimensional projection information (in the column direction).

[Example of accumulated signal information Photographing scene: FIG. 1
0] FIGS. 10 (a) and 10 (b) show an example of a shooting scene. FIG. 10 (a) shows a shooting scene at the time when the first signal accumulation is performed, and FIG. It is a shooting scene at the time when the signal accumulation of No. 2 is performed.

In FIGS. 10A and 10B, 71
Is a main subject, and the person is moving from the left side to the right side of the screen at the time point of FIG. 10B with respect to the time point of FIG. 72 is the sky, 73 is the ground, 74
Numerals to 76 are trees, which constitute the background in this shooting scene.

FR1 in FIG. 10 (a) is an image frame of the camera at the time when the first signal accumulation is carried out.
FR2 in (b) indicates the image frame of the camera when the second signal is stored.

That is, FIG. 10A and FIG.
It shows a state in which the photographing image frame moves from the left side of the screen to the right side according to the movement of the person 71, but also moves from the lower side to the upper side of the screen due to the influence of camera shake or the like.

[One-dimensional projection information calculated from accumulated signal information: FIG. 11] FIGS. 11A and 11B are shown in FIG.
An example of one-dimensional projection information calculated from a block having a peripheral portion of the first and second accumulated signal information obtained from the shooting scenes shown in (a) and (b) is shown.

FIG. 11A shows the one-dimensional projection information in the horizontal direction (column direction) of a block (for example, Ha) in the block group H, and the curve H1 is based on the first accumulated signal information. The curve H2 is based on the second accumulated signal information.

Next, in step S152, a correlation calculation is performed between the projection information by the block group H of the first accumulated signal information and the corresponding projection information by the block group H of the second accumulated signal information, and the left and right sides of the camera are calculated. The direction and amount of the directional movement (movement of the image frame) is calculated.

Specifically, by performing the correlation calculation between the curve H1 and the curve H2 shown in FIG. 11A, the second signal accumulation is performed at the time when the first signal accumulation is performed. It is possible to calculate the moving direction and the moving amount in the left-right direction of the image in the block at that time.

For example, if the curve H2 is moved to the left by two pixels with respect to the curve H1, the image frame of the camera is changed from the time when the first signal is stored to the time when the second signal is stored. This indicates that the pixel has moved to the right by two pixels.

Next, in step S153, Ha ~
The amount of movement in the left-right direction of the entire captured image frame is determined by selecting the mode value from the calculation result obtained by performing the correlation calculation of step S152 on the projection information of each block of Hd. As described above, Ha to Hd are blocks existing around the photographing screen, and the calculation result of step S152 calculated using the one-dimensional projected image thereof is mainly based on the information of the background which is a stationary object. Should indicate the amount of movement of the image frame.

However, depending on the block, a moving subject may be included or the contrast may be low, so that the amount of movement of the photographic image frame may not be accurately calculated. Therefore, in this embodiment, step S for each block is performed.
The motion amount of the photographing image frame is determined by selecting the mode value from the calculation result of 152. For example, blocks Ha, H
From the projection information of b and Hc, the image is two pixels on the left side,
That is, a calculation result that the image frame is moved to the right by two pixels is obtained, and the image is moved to the right by one pixel, that is, the image frame is moved to the left by one pixel from the projection information of the block Hd. Suppose that the result is obtained. In such a case, it is considered that the calculation results of the blocks Ha, Hb, and Hc are more reliable than the calculation result of the block Hd, and in the block Hd, the image captured by the camera is captured due to the fact that there is a subject that is actually moving. It is considered that the frame movement has not been calculated accurately. By selecting the mode value of the calculation result of these 4 blocks, the amount of movement of the image frame of the camera in the left-right direction is determined to be 2 pixels on the right side. Next, in step S154, in order to subsequently calculate the vertical movement of the camera, the peripheral portion in the first accumulated signal information obtained in step S102 is a block long in the vertical direction shown in FIG. 9B. Divide into Va to Vd (block group V), and in each block in the vertical direction (row direction)
The one-dimensional projection information of is calculated.

Similarly, the peripheral portion in the second accumulated signal information obtained in step S105 is shown in FIG.
Are divided into vertically long blocks Va to Vd (block group V) shown in FIG.
It also calculates the one-dimensional projection information in the (row direction).

FIG. 11B shows the one-dimensional projection information in the vertical direction (row direction) of a block (for example, Va) in the block group V, and the curve V1 is based on the first accumulated signal information. The curve V2 is based on the second accumulated signal information.

Next, in step S155, step S1
By performing the same correlation calculation as described in 52 on the block group V, the moving direction and the moving amount of the image in the block in the vertical direction are calculated.

In FIG. 11B, for example, the curve V1
On the other hand, if it is calculated that the curve V2 is moved downward by one pixel, the shooting frame of the camera is changed from the time when the first signal is stored to the time when the second signal is stored. Indicates that one pixel has moved upward.

Next, in step S156, Va ~
The vertical movement amount of the entire photographing image frame is determined by selecting the mode value from the calculation result obtained by performing the correlation calculation similar to step S152 on the projection information of each block of Vd. Here, the blocks Vb, Vc, Vd
From the projection information of the above, the calculation result that the image is moved downward by one pixel, that is, the captured image frame is moved upward by one pixel, is obtained, and from the projection information of the block Va, the image is not moved, that is, It is assumed that the result that the image frame does not move is obtained. By taking the mode value from the calculation result of the four blocks, it is determined that the vertical movement amount of the photographing image frame of the camera is one pixel on the upper side. Next, in step S157, based on the movements of the camera in the left-right direction and the up-down direction (movement of the photographing image frame) calculated above, first, the two-dimensional information of the second accumulated signal information with respect to the first accumulated signal information is obtained. The positional relationship is shifted, and then, on the shifted positional relationship, a difference value between each luminance information of the second accumulated signal information and each luminance information of the first accumulated signal information is calculated for the overlapping portion. To do.

[Calculation of difference value (two-dimensional information) from second and first accumulated signal information: FIG. 12] That is, in FIG. 12, a two-dimensional representation of the first accumulated signal information is represented by D.
Let FR1 be DFR2 that two-dimensionally represent the second accumulated signal information.

Here, the first accumulated signal information D is calculated according to the direction and amount of the camera movement (movement of the photographing frame) calculated in steps S153 and S156.
The second accumulated signal information DFR2 is shifted from the FR1 by 2 pixels on the right side and by 1 pixel on the upper side, and P1 and 3 of the first accumulated signal information DFR1 are overlapped with each other (hatched portion).
All the difference values of the brightness information are calculated for each pixel, such that the difference value between the brightness information of 1 and the brightness information of P2, 1 of the second accumulated signal information DFR2 is calculated.

Although there are positive and negative values in the difference value of the brightness information, it is treated as the absolute value of the difference in this embodiment.

[From the difference value (two-dimensional information), the projection information (1
Dimension): FIG. 13] Next, in step S158, the difference value information calculated in step S157 is 2
The dimensional information is calculated to be one-dimensional projection information.

Here, the difference value calculated in step S158 is the luminance information in the first accumulated signal information and the luminance information in the second accumulated signal information in the subject area where there is no movement such as the background portion. Is almost 0 because there is almost no error in the calculation of the direction and amount of the camera movement (movement of the image frame), so that it is almost 0.

However, if a moving subject exists in the screen like the person 71 in the scene described with reference to FIG. 10,
In the subject area, although the luminance information of the person 71 is used in the first accumulated signal information, the luminance information in the second accumulated signal information becomes a background area, which causes a difference in the luminance information. .

Therefore, as the projection information calculated in this step corresponding to the shooting scenes shown in FIGS. 10A and 10B, the data in the horizontal direction (column direction) is shown in FIG. 13A. The value of the curved line Hmv becomes high in the region where the person 71 has moved, as indicated by the curved line Vmv showing the data in the vertical direction (row direction) in FIG.

Next, in step S159, the moving object area (moving subject area) in the photographing screen is determined based on the projection data of the difference value calculated in step S158.

Horizontal direction (column direction) shown in FIG. 13 (a)
Within the screen, an area in which the projection image of No. 2 shows a high value of a predetermined value or more and an area in which the projection image of the vertical direction (row direction) shown in FIG. When the crossing area of is obtained, the moving body area in the photographing screen can be obtained.

When this step is completed, the process returns to step S108 in FIG. 7, but as the focus detection area selection method in this step, the area included in the moving object area is selected to perform auto-focusing well following a moving subject. Can be realized.

FA14 and FA1 are the focus detection areas included in the moving object area obtained in step S159.
There are several points such as 9, FA24. Here, from these several points, for example, the focus detection area FA
As a method of determining 14, various methods can be used.

For example, there is a method (for example, near distance priority) for performing the distance measurement at the above selected several points and determining the final focus area from the distance measurement information. There is a method of narrowing down to one point by an algorithm (for example, upper screen priority). Further, by repeatedly detecting the moving body region as described above at time intervals, it is possible to detect the moving amount and moving direction of the moving body.

For example, in the scene described with reference to FIG. 10, if the person 71 is moving across the shooting image frame from the left side to the right side, the projection information curve shown in FIG. This is because it can be detected that the portion having a large Hmv value moves from the left side to the right side.

In this way, the focus detection area FA14 can be selected and focused on a moving subject as illustrated in FIG.

In the present embodiment, the method for detecting the movement of the photographing frame of the camera due to the pan or blur by using the accumulated signal information of the photometric sensor in the camera for photographing on the photographic film has been described. Alternatively, it may be an image pickup device of an electronic still camera. Further, in the present embodiment, the movement of the image frame of the camera is calculated using the one-dimensional projection information of the accumulated signal information, but the method of calculating the movement is not limited to this, and the projection information is not used. Good.

Other Embodiments Even when the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus including one device (for example, a copying machine). Machine, facsimile machine, etc.).

Further, an object of the present invention is to supply a storage medium (or a recording medium) recording a program code of software for realizing the functions of the above-described embodiment to a system or apparatus, and to supply a computer of the system or apparatus ( Alternatively, by the CPU or MPU) reading and executing the program code stored in the storage medium,
It goes without saying that it will be achieved. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.
Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also based on the instruction of the program code,
An operating system (OS) running on the computer does some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, , The CPU provided in the function expansion card or the function expansion unit performs some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

When the present invention is applied to the above storage medium, the storage medium stores the program code corresponding to the above-described flowchart.

As described above, the image pickup apparatus of the present embodiment detects the movement of an image from each of a plurality of detection areas in the screen input by the image pickup element, and further detects the movement of each area from the whole or part of the screen. It is equipped with a detection unit that determines the movement. By dividing the movement of the image into two axial directions (horizontal direction and vertical direction) and changing the detection area selection method according to each axial direction, It is possible to cut out a detection area suitable for the axial direction of detection.

That is, when detecting a horizontal movement, a plurality of blocks which are long in the horizontal direction are used as detection areas, and when detecting a movement in the vertical direction, a plurality of blocks which are long in the vertical direction are detected. Used as.

As described above, by obtaining a large amount of information on the direction of movement in accordance with the direction of movement, the detected amount of movement becomes more reliable. Therefore, even when the image pickup device having a relatively small number of pixels is used, it is possible to accurately detect the movement of the subject or the image pickup image frame of the image pickup device from the image information.

As described above, according to the present invention,
(EN) Provided are an image processing device capable of accurately detecting a movement of a subject or a shooting image frame of a camera based on imaged image information of a small number of pixels, an imaging device capable of mounting the image processing device, and a control method thereof. it can.

[Brief description of drawings]

FIG. 1 is a cross-sectional view showing an arrangement of optical members of a camera of an embodiment according to the present invention.

FIG. 2 is a diagram showing an example of a configuration of a focus detection sensor.

FIG. 3 is a diagram showing an example of arrangement of focus detection areas.

FIG. 4 is a diagram showing an example of a configuration of a photometric sensor.

FIG. 5 is a diagram showing an example of screen division of a photometric sensor.

FIG. 6 is a block diagram showing an example of a configuration of electric circuits of a camera and an interchangeable lens.

FIG. 7 is a flowchart showing an example of the operation of the control unit of the camera.

FIG. 8 is a flowchart showing an example of detection of a moving body area of a control unit.

FIG. 9 is a diagram showing an example of a detection range.

FIG. 10 is a diagram showing an example of a shooting scene.

FIG. 11 is a diagram showing projection information of a peripheral portion.

FIG. 12 is a diagram illustrating a method of correcting a shift in a captured image frame between images.

FIG. 13A is a diagram in which projection information is obtained from horizontal difference information.

FIG. 13B is a diagram in which projection information is obtained from vertical difference information.

FIG. 13C is a diagram in which a moving body region is obtained from horizontal and vertical difference information.

FIG. 14 is a diagram showing an example in which a distance measuring point is selected from a plurality of moving object areas.

[Explanation of symbols]

10 camera body 20 Focus detection sensor 26 Photometric sensor 30 interchangeable lens 41 Camera control 45 Posture detection sensor 47 Light emitting part 51 Interchangeable lens controller 56 distance encoder 57 Zoom encoder

Claims (14)

[Claims] 1. An image pickup apparatus having an image pickup means for continuously picking up a subject image to generate a plurality of images, wherein the luminance of a pixel group forming a rectangular region long in a vertical direction of an image frame of the image is related. A first acquisition unit that acquires vertical brightness information that is information, and a second acquisition unit that acquires horizontal brightness information that is information related to the brightness of a pixel group that forms a rectangular area that is long in the horizontal direction of the image frame of the image. An acquisition unit; and a movement amount generation unit that generates information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information. Imaging device. 2. The vertically long rectangular area is a left end portion or a right end portion in the image frame, and the horizontally long rectangular area is an upper end portion or a lower end portion in the image frame. The imaging device according to claim 1, wherein the imaging device is a side end portion. 3. The movement amount detection means calculates a vertical movement amount of the image frame from the vertical direction luminance information, and calculates a horizontal movement amount of the image frame from the horizontal direction luminance information, The image pickup apparatus according to claim 1, wherein the moving amount of the image frame is generated based on the calculated moving amount. 4. A correction unit that corrects a captured image shift that occurs between the plurality of images based on information about the movement amount of the image frame, and a moving object that exists in the plurality of images using the corrected image. 4. The image pickup apparatus according to claim 1, further comprising: a detection unit that detects a region in which the image exists. 5. A processor for generating information on a movement amount of an image frame occurring between the plurality of images based on a plurality of input images, the rectangular region being long in a vertical direction of the image frame of the image. First acquisition means for acquiring vertical direction luminance information which is information relating to the luminance of the pixel group to be formed, and horizontal luminance which is information relating to the luminance of the pixel group which forms a horizontally long rectangular area of the image frame of the image. Second acquisition means for acquiring information; movement amount generation means for generating information regarding the movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information, A processor having. 6. The rectangular area elongated in the vertical direction is a left end portion or a right end portion in the photographing image frame, and the rectangular area elongated in the horizontal direction is an upper end portion in the image frame or The processor according to claim 5, wherein the processor is a lower end portion. 7. The movement amount detection means calculates a movement amount in the vertical direction of the photographing image frame from the vertical direction luminance information,
6. The horizontal movement amount of the photographing image frame is calculated from the horizontal direction luminance information, and the movement amount of the photographing image frame is generated based on each calculated movement amount. 6. The processor according to 6. 8. A correction unit that corrects a shooting image frame shift that occurs between the plurality of images based on information about the movement amount of the shooting image frame, and a correction unit that uses the corrected image in the plurality of image data. 8. The processor according to claim 5, further comprising: a detection unit that detects a region where a moving body that exists is present. 9. A method of controlling an image pickup apparatus having an image pickup step of continuously picking up a subject image to generate a plurality of images, the pixel group forming a rectangular area long in a vertical direction of an image frame of the image. A first acquisition step of acquiring vertical direction brightness information that is information related to the brightness of the image, and horizontal brightness information that is information related to the brightness of a pixel group that forms a rectangular area that is long in the horizontal direction of the image frame of the image. A second acquisition step, and a movement amount generation step of generating information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information. A method for controlling an image pickup device having a feature. 10. A control method for a processor, which generates information on a movement amount of an image frame occurring between the plurality of images based on a plurality of input images, wherein the image frame of the image is long in a vertical direction. A first acquisition step of acquiring vertical direction brightness information, which is information about the brightness of a pixel group forming a rectangular area, and information about the brightness of a pixel group forming a rectangular area that is long in the horizontal direction of the image frame of the image. A second acquisition step of acquiring horizontal direction luminance information; and a movement amount generation for generating information regarding a movement amount of the image frame between the plurality of images based on the vertical direction luminance information and the horizontal direction luminance information. A method of controlling a processor, comprising: 11. A program executable by an information processing apparatus having a program code for implementing the method for controlling an image pickup apparatus according to claim 9. 12. A storage medium storing the program according to claim 11. 13. A program executable by an information processing device, having a program code for implementing the method of controlling a processor according to claim 10. 14. A storage medium storing the program according to claim 13.