JP2006319578A - Depth direction movement determining device, blurring correction system having the same, and blurring correction method, program, computer readable record medium recording the program, and electronic apparatus equipped with the blurring correction system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem wherein a conventional electronic blurring correction technology performs only the detection of blurring and correction of an image in two dimensional direction such as up/down or left/right, but omits the detection of blurring and correction of an image in the depthwise direction, although various technologies, such as optical and electronic ones, are employed as blurring correction technology for a photographing device. <P>SOLUTION: A depthwise direction movement determining device 300 comprises a function for determining movement of an object before and behind in the depthwise direction at the time of photographing based on a local motion vector within an image. A blurring correction system performs image correction based on the determination result of the depthwise direction movement determining device. Also, the depthwise direction move determining method and the blurring correction method, a program for making a computer perform these methods, a computer readable record medium recording these programs, and an electronic apparatus equipped with the blurring correction system are offered. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for determining whether or not there is a movement of a subject in the depth direction relative to the imaging apparatus due to a shake of the imaging apparatus, and an image captured using the movement determination technique in the depth direction. The present invention relates to a technology for correcting information.

Usually, when photographing a subject with a photographing device such as a camera, a person supports the photographing device with his / her hand. Therefore, when photographing a subject, there is always a problem that the quality of a photographed image is degraded due to so-called “camera shake”. In order to solve this, conventionally, in addition to the method of shooting with a shooting device fixedly supported by a tripod or the like, shooting of blurring is performed by driving and controlling an optical system and an imaging device using a vibration gyroscope or an angular velocity sensor. Techniques for reducing the impact on the environment are disclosed. Also, in Patent Document 1 and Patent Document 2, the direction of the shake that occurred is detected by electronically comparing, on the software, the data of the captured images taken before and after the same subject. A method of detecting the direction of blurring, called electronic, and a camera shake compensation technique are also disclosed.
JP-A-7-177419 JP-A-5-110931

  However, for example, in the above-described conventional electronic camera shake detection and correction technology, only the shake of the subject in the two-dimensional direction, up and down or left and right, is detected in the detection of the shake direction. However, in reality, in addition to the blur in the two-dimensional direction, there is also a blur in the depth direction with respect to the photographing apparatus. That is, there is a problem that the conventional electronic camera shake detection and correction technology does not detect the shake and correct the image in consideration of the shake in the depth direction.

  In particular, the advancement of technology has made the photographic device smaller and lighter, making it possible to easily hold the photographic device with one hand and shoot, resulting in camera shake compared to conventional cameras that support and shoot with both hands. It is easy to do. In particular, when shooting with a camera with a camera-equipped mobile phone or a liquid crystal display monitor, camera shake in the depth direction is more noticeable than with conventional cameras that look into the viewfinder. This is because a conventional camera and a camera-equipped mobile phone have greatly different photographing postures such as a method of holding the photographing device. In other words, since it is necessary to shoot through the viewfinder with a conventional camera, since the photographic device is photographed in a posture that is relatively close to the body, the photographic device is easy to stabilize, and camera shake in the depth direction is relatively unlikely to occur. . However, in general, in a shooting posture such as a mobile phone with a camera that extends the arm and keeps the imaging device away from the body, it is easy for the extended arm to shake back and forth, so there is relatively little camera shake in the depth direction. It can be said that it is easy to occur.

  In order to solve the above-described problems, the present invention provides a depth direction movement determination device having a function of determining a back and forth movement of a subject in a depth direction at the time of shooting based on a local motion vector in an image. A camera shake correction system that performs image correction based on the determination result of the depth direction movement determination device is also provided.

  Specifically, the depth direction movement determination device of the present invention includes an image information acquisition unit that acquires image information, and local motion at a plurality of detection locations in the image based on the acquired plurality of pieces of image information. It is a determination device having a configuration of a motion vector detection unit that detects a vector, and a movement determination unit that determines forward and backward movement of the subject in the depth direction based on the detection result. The camera shake correction system according to the present invention is an apparatus having a configuration called the depth direction movement determination device and an image information correction unit that corrects image information based on a determination result from the depth direction movement determination device. In the present invention, the depth direction movement determination method and the camera shake correction method, a program for causing the computer to execute these methods, a computer-readable recording medium on which the program is recorded, and an electronic device including the camera shake correction system are also combined. To provide.

  In the depth direction movement determination and camera shake correction according to the present invention, it is possible to perform determination of movement in the depth direction due to shaking other than the hand, for example, shake caused by shaking of the ground during shooting, and image correction. Needless to say. In addition to camera shake, it is also possible to determine the subject's own back and forth movement.

  With the depth direction movement determination device of the present invention having the above-described configuration, it is possible to determine not only the movement of the subject in the two-dimensional direction of up / down / left / right, which has been conventionally performed, but also the movement in the depth direction that occurs during shooting. Is possible. In addition, the camera shake correction system of the present invention that uses this determination correctly detects the direction of the image blur caused by the camera shake in the depth direction as a blur in the depth direction, not in the vertical and horizontal directions. can do. In addition, it is possible to correct the captured image accurately based on the detection of the accurate blur direction.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to these embodiments, and can be implemented in various modes without departing from the scope of the invention. In the first embodiment, claims 1, 4, 6 and 8 will be mainly described. In the second embodiment, claims 2, 5, 7, and 8 will be mainly described. In the third embodiment, claims 3, 9, and 10 will be mainly described.

  First Embodiment <Outline of First Embodiment> FIG. 1 is a diagram for explaining the direction of “blurring” of a photographing apparatus that occurs during photographing with a camera. Note that (1) in FIG. 1 is a view of the subject A and the photographing apparatus B as viewed from above. Further, (2) of FIG. 1 is a view of the subject A and the photographing device B as seen from the side. First, as indicated by an arrow α in (1) of FIG. 1, the photographing apparatus B may be shaken in the left-right direction with respect to the subject A. On the other hand, as indicated by the arrow β in (2) of FIG. 1, the photographing apparatus B may be shaken in the vertical direction with respect to the subject A. However, the photographing apparatus B may be shaken back and forth in the depth direction with respect to the subject A as can be seen from the arrows γ in (1) and (2) of FIG. .

  FIG. 2 is a diagram illustrating an example of blurring in a captured image caused by blurring of the imaging apparatus described above. In this figure, for example, in moving image shooting, individual frame (frame) images in the vicinity on the time axis are superimposed. As shown in this figure, for example, when the photographing apparatus is shaken in the left-right direction as indicated by an arrow α in FIG. 1, it is photographed at a position originally indicated by a solid line in the photographed image as indicated by α in FIG. It can be seen that the subject to be shaken in the left and right positions as indicated by broken lines in the front and rear frames. Similarly, when there is a vertical shake indicated by an arrow β in FIG. 1, it can be seen that a vertical shake of the subject as indicated by β in FIG. 2 occurs. Also, when there is a blur in the three-dimensional direction indicated by the arrow γ in FIG. 1, the blur is captured in the form of enlargement / reduction of the subject image in the two-dimensional captured image, as indicated by γ in FIG. . Therefore, when this is reproduced as a moving image, the reproduced moving image is hard to see because the subject moves in a blurred manner.

  In view of the above, the camera shake direction determination apparatus in the conventional camera shake correction system detects the shake direction in the vertical and horizontal directions as indicated by the arrows α and β in FIG. Based on the detection result, the camera shake photographed image as indicated by α and β in FIG. 2 is corrected. However, the depth direction movement determination apparatus according to the present embodiment is characterized in that the direction of the camera shake can be accurately determined not only for that but also for the shake before and after the three-dimensional depth direction as indicated by the arrow γ. As a result, it is possible to perform accurate camera shake correction even for a photographed image as indicated by γ in FIG. 2, which has conventionally been difficult to correct because the direction of camera shake cannot be accurately detected.

    <Configuration of Embodiment 1> FIG. 3 is a diagram illustrating an example of functional blocks in the depth direction movement determination device of the present embodiment. As shown in this figure, the “depth direction movement determination device” (0300) of this embodiment includes an “image information acquisition unit” (0301), a “motion vector detection unit” (0302), and a “movement determination unit”. (0303).

  Note that the functional blocks of the apparatus described below can be realized as hardware, software, or both hardware and software. Specifically, if a computer is used, a CPU (Central Processing Unit), a buffer memory, a bus, a storage device such as a hard disk or a nonvolatile memory, a storage medium such as a CD-ROM or a DVD-ROM, and the like Hardware components such as media reading drives, transmission / reception ports for various communications and printing devices, other peripheral devices, driver programs and other application programs for controlling those hardware, interfaces used for information input, etc. Is mentioned. Specifically, the functions of the respective units are realized by sequentially executing the programs developed on the memory, by processing, storing, and outputting data on the memory and data input through the interface. Here, FIG. 19 shows an example of the hardware configuration of an imaging apparatus provided with the depth direction movement determination apparatus of the present invention. The function block of FIG. 3 and the hardware configuration of FIG. 19 will be described below together with an example of the function of this embodiment and a configuration for realizing it.

  Further, the present invention can be realized not only as an apparatus or a system but also as a method. In addition, a part of the invention can be configured as software. Furthermore, a software product used for causing a computer to execute such software and a recording medium in which the product is fixed to a recording medium are naturally included in the technical scope of the present invention (the same applies throughout the present specification). Is).

  The “image information acquisition unit” (0301) has a function of acquiring image information. “Image information” is image information represented by numerical values such as RGB values or YUV values in units of pixels, which are obtained by using a photoelectric conversion element such as a CCD image sensor (1902) or a CMOS image sensor, or a color filter. Etc. Note that a plurality of pieces of image information acquired by the image information acquisition unit need to be acquired with time transition for the same subject. This is because a plurality of pieces of image information shifted in time are necessary when detecting a motion vector described later. As shown in FIG. 4, “a plurality of pieces of image information acquired with time transition for the same subject” means, for example, in the case of shooting a moving image, the continuous (or non-displayed) constituting the moving image. A plurality of pieces of image information which are continuous (frame) frames can be mentioned.

  Alternatively, in the case of a still image, a method of capturing a plurality of pieces of image information by capturing an image of a subject that has been photographed at the same time immediately after that. In this way, by comparing continuous (or non-continuous) image information for the same subject in accordance with the time transition, it becomes possible to detect a motion vector as described later.

  These pieces of image information are encoded information, and may be information that can be displayed by decoding. In this case, if it is a moving image, it may be image information that is independently encoded using fractal encoding or the like, and inter-frame prediction that refers to temporally previous or / and subsequent image information. It may be image information encoded using.

  Image acquisition by the “image information acquisition unit” is performed by, for example, converting light from a subject passing through the lens (1901) in FIG. 19 by a CCD image sensor (1902), which is an assembly of CCD elements in pixel units, and converting the light. There is a method in which digital data indicating image information is generated and acquired by the A / D converter (1903) in accordance with the amount of the generated charge. The image information acquired in this way is stored in the temporary storage memory (1905) or the storage area of the storage device (1906) corresponding to the frame number in a storage area at a predetermined address. The frame number is a number assigned in accordance with the passage of time of image information acquisition. If the image information is moving image information, the frame number is information indicating the reproduction order. Further, if the image information is a still image, for example, the information indicates the shooting order of continuous shooting. As described above, the image information is stored at a predetermined address corresponding to the frame number, so that when the motion vector is detected by the arithmetic processing of the CPU as will be described later, the image information before and after the time is compared. The detection process can be performed.

  The “motion vector detection unit” (0302) has a function of detecting local motion vectors at a plurality of detection points in an image based on a plurality of pieces of image information acquired by the image information acquisition unit (0301). Have. FIG. 5 is a diagram for explaining an example of detection of this motion vector. As shown in FIGS. 5A and 5B, the image information acquired by the image information acquisition unit includes a predetermined grid area, for example, by size information or position information for designating a divided area. It is divided into such as. In the present embodiment, an example will be described in which local motion vectors at each detection location are detected using the predetermined five locations shown in FIG. Of course, as in the second embodiment described later, local motion vectors may be detected using all of the lattice regions (or regions having shapes other than the lattice shape) as detection locations.

  In the image of image information 1 in FIG. 5A, a grid area (0501) indicated by diagonal lines is one of the predetermined five detection points. And the detection of the local motion vector in this detection location (0501) is performed by the following processes, for example. That is, it is determined to which position the subject image in this detection location (0501) has moved in the image of the image information 2 in FIG.

  For this determination, an image matching processing technique such as block matching may be used. In block matching, first, a search range (0502) composed of a predetermined number of blocks around the block at the same position as the detection location (0501) is determined from the image of image information 2 in FIG. 5B. To do. A block (0503) that is most approximate to the subject image in the search location (0501) in FIG. 5A is detected from the search range using pixel values and distribution information thereof. Then, the local motion vector (0501n) at the detection location (0501) is detected using the movement amount as the horizontal and vertical components. Then, the motion vector is similarly detected for the other detection points, and the motion vector is comprehensively used in the movement determination unit described later, so that the depth direction movement determination device of the present embodiment can detect the depth of the subject. Determine movement in the direction.

  The detection of a local motion vector using the above block matching or the like is a process generally performed in encoding of a normal moving image, and is information included in the encoded moving image information. is there. Therefore, in the case of a photographing apparatus that records with encoded moving images, the motion vector detection unit in the encoding is also used as the motion vector detection unit in the depth direction movement determination device of the present invention, and is detected during the encoding. The motion vector may be reused. In this way, by reusing the motion vector at the time of encoding, the depth direction movement determination device of the present embodiment can be realized simply by adding another determination function to the conventional encoded moving image capturing device. can do. Therefore, the detection location is the same as a rectangular macroblock having a size of 8 × 8, 16 × 16, 4 × 8 pixels or the like that is generally used for motion vector detection in moving image encoding. Is desirable. Alternatively, it is desirable to use the same block as the outer edge of the contour of the subject extracted by the edge extraction process, which is similarly used for motion vector detection in moving image coding.

  However, of course, the shape of the detection portion is not limited to the above, and may be various shapes such as a hexagon and a circle. In addition, the image information referred to by block matching or the like when detecting a motion vector may be temporally subsequent or subsequent, or may be image information before that, or both. I do not care.

  In addition, the same motion vector detection process may be performed for the one detection point in time and further on the contrary, and the previous frame in time, and a plurality of motion vectors may be detected for one detection point. good. In this case, for example, another motion vector 0501m is further detected in addition to the motion vector 0501n at the detection location (0501).

  Further, as described above, the motion vector detection process is not limited to a process using a moving image. For still images, it is not possible to compare the previous and next frames unless they are taken continuously, but for example, a subject that has been shot still images is automatically shot immediately after that, and the multiple still images are stored in the buffer memory. Hold it once. Then, by comparing them by the same processing as described above, a motion vector including a time before and after photographing in a still image may be detected.

  For example, a method of detecting a motion vector by the “motion vector detection unit” is performed in the following procedure. As an example of the detection method, first, a motion vector detection program is read from a predetermined storage area of the storage device from the main memory by detecting an execution command or the like instructing execution of the movement direction determination mode in the depth direction movement determination device. The data is expanded in the storage area and sequentially executed by the CPU. By executing this program, a command for reading out a plurality of pieces of image information stored at a predetermined address in accordance with the frame number is sent out by the operation of the above-described image information acquisition unit. Image information is stored in a temporary storage memory. Subsequently, a plurality of pieces of image information stored in the temporary storage memory are compared based on the time relationship corresponding to the frame number by the logical operation processing of the CPU. Then, for example, a motion vector at a predetermined detection location is detected by the comparison process using the block matching. Then, the detected motion vector is associated with the detected location and stored again in a storage area of a predetermined address such as a temporary storage memory or a storage device.

  The “movement determination unit” (0303) has a function of determining the back-and-forth movement of the subject in the depth direction based on the detection result of the motion vector detection unit (0302). Note that “the subject moves back and forth in the depth direction” means that the subject moves back and forth in the depth direction with respect to the imaging apparatus. Therefore, this includes not only the subject but also the case where the photographing apparatus moves back and forth.

  FIG. 6 is a diagram for explaining an example of determination by the movement determination unit performed using the predetermined five locations as sample detection locations and using a motion vector at the sample detection locations. As shown in this figure, the five sample detection locations are grid-like regions (0601, 0602, 0603, 0604, 0605) indicated by hatching. Then, the respective motion vectors (0601n,..., 0605n) are detected for these five sample detection locations, and this movement determination unit determines whether the subject is moving back and forth in the depth direction. As an example of a specific determination method, a motion vector 0601n, a motion vector 0602n, ..., a motion vector 0605n,

として、数式

As a formula

により、これらサンプルの動きベクトルを合成する。そして合成された動きベクトルｆ（ａ）の成分が、図７で示すように所定の閾値ａ，ｂ以上Ａ，Ｂ以下であれば被写体は奥行方向の前後移動をしている、と判定する。逆に、合成動きベクトルｆ（ａ）の成分が所定の閾値Ａ，Ｂのいずれかを超えていた場合、少なくと上下、または左右方向への移動をしている、と判定する。また、合成動きベクトルｆ（ａ）の成分のいずれかが所定の閾値ａ，ｂよりも小さい場合はぶれなどによる被写体の動きはなかったと判定する、という具合である。

Thus, the motion vectors of these samples are synthesized. Then, if the component of the combined motion vector f (a) is a predetermined threshold value a, b or more and A or B or less as shown in FIG. 7, it is determined that the subject is moving back and forth in the depth direction. Conversely, if the component of the combined motion vector f (a) exceeds either of the predetermined thresholds A and B, it is determined that the movement is at least up and down or left and right. In addition, when any of the components of the combined motion vector f (a) is smaller than the predetermined thresholds a and b, it is determined that there has been no movement of the subject due to shaking or the like.

  The reason for such a determination is that, on a two-dimensional plane called an image, there is a premise that the movement width is shown smaller than the movement left and right or up and down. However, if there are many pairs of movements whose direction is paired, such as “up and down”, the combined motion vector component may be below the threshold even if there is only a large vertical movement in the above formula. It is possible. Therefore, in the above formula, it is preferable to calculate by taking the absolute value of the motion vector component of each sample.

  In the above determination, the blur at the center of the image may be used as the most important blur determination element during vector synthesis, and the motion vector at the center detection point may be weighted and combined to perform determination. . Of course, the determination method of the movement in the movement determination unit is not limited to the method using the synthesis of the motion vector components, but may be various methods such as a method of determining from the variation in the direction of the motion vector as described later in the second embodiment. May be realized. In this way, the depth direction movement determination apparatus according to the present embodiment can determine whether the subject is moving in the depth direction relative to the imaging apparatus using the local motion vector.

  For example, a method of determining the subject's depth movement in the depth direction by the “movement determination unit” may be performed in the following procedure. As an example of the determination method, first, by detecting the above-described local motion vector detection end information, the moving direction determination program is changed from a predetermined storage area of the storage device (1906) to the main storage area of the memory. And sequentially executed by the CPU (1904). By executing this moving direction determination program, the motion vector stored at a predetermined address for each of a plurality of detection points as described above is read onto the temporary storage memory (1905) and stored at each predetermined address. (Of course, a plurality of motion vectors may be stored at predetermined addresses of the temporary storage memory from the beginning, for example, when the motion vector detection process and the movement direction determination process are continuously performed). Subsequently, a plurality of motion vectors stored in the temporary storage memory are synthesized by arithmetic processing of the CPU. Then, information indicating the combined vector, which is a processing result by the CPU, is output to a temporary storage memory or the like and stored in a predetermined storage area.

  Also, by this moving direction determination program, predetermined value information for determination (threshold A, B, etc. in the above example) stored in another predetermined storage area of the storage device is read and stored in the temporary storage memory. The Then, the combined vector stored in the predetermined area of the temporary storage memory and the threshold indicated by the predetermined value information stored in another predetermined area are compared by the logical operation processing of the CPU. As a result of the comparison processing, for example, if a comparison result indicating that the combined vector is a predetermined threshold a, b or more and A or B or less is output by the CPU, it is determined that the subject has moved back and forth in the depth direction. In this method, information indicating that is generated. The determination information is stored in a temporary storage memory or a storage area of a predetermined address of the storage device.

  Then, using the determination result of the direction of movement made in this way, for example, a warning message such as “the camera is shaking in the front-rear direction” may be displayed on the display unit of the photographing apparatus. Specifically, for example, the above determination is performed every 1/30 seconds, and a warning regarding the direction of shake determined most frequently in 30 determinations per second is displayed.

    <Other Examples of Example 1> In the above, the embodiment in which the depth direction movement determination device is used to warn of the direction of shaking has been described. However, in addition to such an embodiment, the depth direction movement determination apparatus according to the present embodiment may be used for camera shake correction processing, for example, as described in Embodiment 3 described later, or may be used for other image processing, or There may be various usage forms such as “moving video moving horizontally” and “moving video moving back and forth”, such as using an index for classifying moving images based on moving attribute of moving images.

    <Processing Flow of First Embodiment> FIG. 8 is a flowchart illustrating an example of a processing flow in the depth direction movement determination device of the present embodiment. As shown in this figure, first, the n-th image information is acquired (step S0801), and then the n + 1-th image information is acquired (step S0802). Next, the n-th image information and the (n + 1) -th image information are compared for a plurality of detection locations in the image indicated by the n-th image information acquired in step S0801 (step S0803). Further, local motion vectors at a plurality of detection points in the image are detected based on the comparison result in step S0803 (step S0804). Finally, based on the detection result in step S0804, the back and forth movement of the subject in the depth direction is determined (step S0805). Based on the determination result, for example, the direction of camera shake is warned, and image correction is performed. Or an index for image classification can be assigned.

    <Simple Explanation of Effect of First Embodiment> As described above, it is possible to determine the movement of the forward and backward movement in the depth direction by the depth direction movement determination device of the present embodiment. Further, from the determination result, for example, a warning message about the direction of camera shake can be displayed. In addition, by incorporating it in the camera shake correction system of the present invention, which will be described later, blurring of a captured image caused by blurring of the photographing apparatus in the depth direction is correctly determined as blurring in the depth direction, not in the vertical and horizontal directions. It is possible to recognize and correct according to the direction of the accurate deviation. Alternatively, a classification index based on the direction of motion of the moving image can be added.

  << Example 2 >> <Outline of Example 2> The depth direction movement determination apparatus of this example also uses the local motion vector in the image detected by the motion vector detection unit in the same manner as in Example 1 to Is a device that determines movement in the depth direction. The feature point of the present embodiment is that the movement of the subject in the depth direction is determined based on how the motion vector at each detection location in the image shows in that direction. Thus, by determining the movement of the subject in the depth direction from the variation in the frequency of the direction of the motion vector, for example, the determination of the movement in the depth direction with higher accuracy than the determination method using sampling or the like of the first embodiment, for example. It can be performed.

  <Configuration of Second Embodiment> FIG. 9 is a diagram illustrating an example of functional blocks in the depth direction movement determination device of the present embodiment. As shown in this figure, the “depth direction movement determination device” (0900) of this embodiment includes an “image information acquisition unit” (0901), a “motion vector detection unit” (0902), and a “movement determination unit”. (0903). Since the “image information acquisition unit” and the “motion vector detection unit” have already been described in the first embodiment, the description thereof will be omitted. The feature point of the depth direction movement determination device of this embodiment is that the “movement determination unit” further includes “counting means” (0904), “variation determination means” (0905), and “variation dependent forward / backward movement determination means”. (0906).

  “Aggregating means” (0904) has a function of aggregating the frequency of motion vectors detected for each of a plurality of detection locations in a plurality of images. Examples of the “plurality of detection points” include the plurality of grid-like regions described with reference to FIG. 5 in the description of the motion vector detection in the first embodiment. In the present embodiment, unlike the first embodiment, an example will be described in which local motion vectors are used (aggregated) in all grid areas instead of the predetermined five locations to determine subject movement. Of course, the shape of the plurality of detection points is not limited to a lattice shape, and the number thereof is not limited.

  10, 11, and 12 are diagrams for explaining an example of totalization of the direction frequency of the motion vector. First, as shown in FIG. 10, the above-described motion vector detection unit detects a motion vector 1001n, a motion vector 1002n,..., A motion vector for each macro block of 16 × 16 pixels that is a plurality of detection locations in the image. 1030n was detected. Here, the motion vectors are divided into 8 directions (1 to 8 in the figure) and 9 directions without direction (0 in the figure) divided at a central angle of 45 degrees as shown in FIG. (Of course, the direction for counting the frequencies is not limited to nine directions).

  Then, as shown in FIG. 12, the detection frequency of the motion vector whose direction is direction 0 (no direction) is “3”, the detection frequency of the motion vector in direction 1 is “1”, and the motion vector in direction 2 Detection frequency is “4”, direction 3 motion vector detection frequency is “5”, direction 4 motion vector detection frequency is “4”, and direction 5 motion vector detection frequency is “3”. The direction 6 motion vector detection frequency is “4”, the direction 7 motion vector detection frequency is “4”, the direction 8 motion vector detection frequency is “2”, and so on. The frequency of the motion vectors of is summarized. In this way, the frequency for each direction is totaled for the motion vectors at a plurality of detection locations in the image.

  For example, a method of summing up the frequency of each motion vector by the “aggregating means” may be performed in the following procedure. As an example of the tabulation method, first, as described in the first embodiment, a local motion vector is detected by processing such as block matching by the CPU (1904) and stored in the temporary storage memory (1905) or the like. Is done. Then, by detecting the motion vector detection end information, this time, a program for summing up the frequency according to the direction of the motion vector is expanded from a predetermined storage area of the storage device (1906) to the main storage area of the memory and sequentially executed by the CPU. The By this frequency counting program, information indicating the direction-specific classification as shown in FIG. 11 is read from another predetermined storage area of the storage device, and the direction-specific classification information is stored in the temporary storage memory. Subsequently, the logical operation processing of the CPU compares the direction indicated by the motion vector stored in the temporary storage memory with the classified direction indicated by the direction-specific classification information. Then, by classifying the motion vector in association with the direction information indicated by the direction-specific classification information and storing it at a predetermined address of the storage device, the motion vector-direction classification processing is performed. Then, according to the classification, the CPU sums up the frequency of motion vectors for each direction. Then, information indicating the frequency of motion vectors collected in this way is stored in a temporary storage memory or a storage area of a predetermined address of the storage device in association with the direction.

  The “variation determination unit” (0905) has a function of determining whether the frequency variation for each direction aggregated by the aggregation unit (0904) is included in a predetermined variation range. As a method of determining the frequency variation for each direction, for example, a method using standard deviation can be cited. That is, the standard deviation is calculated on the basis of the frequency of motion vectors for each direction aggregated by the aggregation means as shown in FIG. If the standard deviation value is less than or equal to the predetermined value, there is no variation in the frequency of each direction (included in the predetermined variation range), that is, the motion vector directions at a plurality of detection points are almost evenly distributed in each of the nine directions. This is a method for determining that the detection has been made. More specifically, the average value of the frequency of each motion vector in the direction is 30/9, which is “3.3” (second decimal place, rounded off). Accordingly, the value of the standard deviation is “1.2”, and if the predetermined value is “1.6”, for example, the standard deviation is equal to or less than the predetermined value. It is determined that it is included in the variation range.

  Conversely, if the value of the standard deviation is greater than or equal to a predetermined value, the frequency of each direction varies (not included in the predetermined variation range), that is, the direction of the motion vector at a plurality of detection locations is out of 9 directions. It can be determined that the object is detected in a specific direction.

  In addition, the “predetermined value” used for determining whether or not the variation in frequency according to the direction is included in a predetermined variation range depends on, for example, the degree of the influence on the moving image or the like due to the blur in the depth direction. There is a method in which variation in frequency for each direction is collected as statistical data in advance and determined based on the statistical data. More specifically, the “predetermined value” is a factor that depends on hardware, such as a lens-specific depth of focus or an aperture value at the time of shooting, and the number of motion vector detection points and the number of frequency in each direction. Statistical data may be collected and determined based on both factors dependent on settings in the software.

  The determination of the variation in the frequency of each motion vector by the “variation determination unit” may be performed by the following procedure, for example. An example of the determination method is as follows. First, a program for determining variation in frequency of each direction of motion vectors based on detection of count end information of the frequency count program described above is a predetermined storage area of the storage device (1906). To the main storage area of the memory and sequentially executed by the CPU (1904). Based on the information indicating the direction frequency of the motion vector stored in the temporary storage memory as described above, the CPU performs arithmetic processing for calculating the standard deviation of the numerical value indicating the frequency based on the variation determination program. . A numerical value indicating the standard deviation is output by the arithmetic processing of the CPU and stored in the temporary storage memory. In addition, the information indicating the “predetermined value” is also read from the predetermined storage area of the storage device into the temporary storage memory by the variation determination program. Then, the standard deviation stored in the temporary storage memory is compared with a predetermined value by a logical operation process of the CPU. As a result, when the standard deviation is equal to or smaller than the predetermined value, information indicating a determination result that the variation in the frequency of the motion vector according to the direction is included in the predetermined variation range is generated. is there. Information indicating the determination result generated in this way is stored in a temporary storage memory or a storage area of a predetermined address of the storage device.

  As another variation determination method for the motion vector frequency, a method for determining that the difference between the maximum frequency and the minimum frequency is within a predetermined variation range if the difference between the maximum frequency and the minimum frequency is equal to or less than a predetermined value. Alternatively, the frequency of motion vectors in the direction indicating the maximum frequency out of the frequency by direction of the motion vectors collected as shown in FIG. 12 described above depends on what percentage of the total number of frequencies corresponds to each direction. A method for determining the variation in frequency is also mentioned. Specifically, if the total number of detected motion vectors is 30 as in the above example, and the direction with the highest frequency is 5 in the direction 7 among the frequency by direction of the aggregated motion vectors, The index indicating the variation is 5/30, that is, 17% rounded off to the third decimal place. Then, this variation determination means determines whether the index calculated in this way is included in a predetermined variation range, for example, 20%.

  In addition, as described above, in this variation determination means, using the “frequency standard deviation”, the “difference between the maximum frequency and the minimum frequency”, or the “ratio of the maximum frequency in the overall frequency”, the frequency of each motion vector by direction The reason why it is determined whether the variation of the subject is within a predetermined range is to determine the forward / backward movement of the subject in the depth direction by the following “variation-dependent forward / backward moving determination unit”.

  Specifically, the determination of the variation by the other variation determination program includes a method of performing the following procedure, for example. First, in the same manner as described above, a program for determining the variation in the frequency of each direction of the motion vector is developed in the main storage area of the memory and is sequentially executed by the CPU. The “predetermined value” is stored in the temporary storage memory. Then, each information indicating the direction frequency of the motion vector stored in the temporary storage memory is compared in magnitude by a logical operation process of the CPU, and a numerical value indicating the maximum frequency and a numerical value indicating the minimum frequency are extracted by the comparison process. Is done. Then, a difference value between the numerical value indicating the maximum frequency and the numerical value indicating the minimum frequency is calculated by the arithmetic processing of the CPU. Alternatively, what percentage of the numerical value indicating the overall frequency the numerical value indicating the maximum frequency is calculated by the arithmetic processing of the CPU. The numerical value indicating the “difference between the maximum frequency and the minimum frequency” or “the ratio of the maximum frequency to the overall frequency” output by the calculation processing of the CPU is stored in a predetermined storage area of the temporary storage memory or the storage device. The Then, the numerical value based on the direction-specific frequency stored in the temporary storage memory and the “predetermined value” are compared in magnitude by the logical operation processing of the CPU, and the degree of variation is determined from the comparison result. Condition.

  The “variation dependent forward / backward movement determination unit” (0906) determines that the subject has moved back and forth in the depth direction when the determination result of the variation determination unit (0905) is within a predetermined variation range. Have Here, the reason why the variation-dependent back-and-forth movement determination unit performs the back-and-forth movement determination based on the determination result of the variation determination unit will be described below with reference to FIG.

  FIG. 13 shows an example of how the frequency of the detected motion vector differs depending on the difference between the shake of the photographing apparatus in the left and right (up and down) direction and the forward and backward shake in the depth direction. It is a figure for demonstrating. For example, when the photographing apparatus shakes left and right (up and down), the image taken as shown in the figure moves to the left and right (up and down) as a whole. Then, as shown in FIG. 13 (1), the motion vectors at the respective detection points in the image indicate directions in approximately the same direction. As a result, the frequency of motion vectors in a specific direction increases, and in the determination by the variation determination means, it is determined that variation occurs in the frequency for each direction (not included in the predetermined variation range). is there. Therefore, when it is determined in the variation determination means that the variation in the frequency for each direction is not included in the predetermined variation range, the direction of the motion vector at the plurality of detection points is 9 in the variation dependent back-and-forth movement determination unit. It can be determined that the direction is detected in a specific direction among the directions.

  On the other hand, when the photographing apparatus is shaken back and forth in the depth direction, the motion vector is detected in a manner different from the movement on the two-dimensional axis in the vertical and horizontal directions. As shown in FIG. 13 (2), when the photographing device is shaken in the depth direction which is a three-dimensional direction, in the photographed image which is a two-dimensional plane, toward the focal point of the photographing device (usually the central point 1301) The subject moves in the opposite direction, and the detected motion vector is as shown by the arrow in FIG. That is, it can be said that the motion vectors detected in this way are not in the same direction, but vary according to the position of the detected part. Therefore, the frequency of the detected motion vector by direction is substantially the same, and the variation in the frequency of the motion vector by direction is likely to be included in the predetermined variation range. For this reason, the variation-dependent back-and-forth movement determination unit determines that the subject has moved back and forth in the depth direction when the determination result of the variation determination unit (0905) is within a predetermined variation range. To do.

  The determination of the subject's back-and-forth movement in the depth direction by the "variation-dependent back-and-forth movement determination means" is performed by, for example, detecting the determination end information of the above-described variation determination program, and then detecting the movement of the subject in the depth direction. The determination program is expanded from a predetermined storage area of the storage device (1906) to the main storage area of the memory, and is sequentially executed by the CPU (1904). Then, information indicating that the subject has moved back and forth in the depth direction or not has been generated from the information indicating the degree of variation in the direction frequency of the motion vector determined as described above, and the temporary storage memory Or stored in a predetermined area of the storage device.

  The “predetermined variation range” may be set in advance by a shipping manufacturer or the like in accordance with, for example, the number of detection locations and the number of distinct directions for counting the frequencies as described above. Alternatively, it can be set in advance by the user, and for this purpose, the depth direction movement determination apparatus of this embodiment may include a “variation range setting unit”. In addition, the depth direction movement determination device according to the present embodiment prevents erroneous determination that movement in the depth direction that occurs in a subject in a captured image during zooming due to camera shake correction processing, which will be described later, is due to hand movement, for example. In addition, a “zoom determination unit” may be provided that determines whether the zoom is a camera shake or a camera shake based on a comparison between the zoom speed and a motion speed calculated from the detected motion vector.

    <Processing Flow of Second Embodiment> FIG. 14 is a flowchart illustrating an example of a processing flow in the depth direction movement determination device of the present embodiment. As shown in this figure, first, the n-th image information is acquired (step S1401), and then the (n + 1) -th image information is acquired (step S1402). Next, the n-th image information and the (n + 1) -th image information are compared for a plurality of detection locations in the image indicated by the n-th image information acquired in step S1401 (step S1403). Furthermore, local motion vectors at a plurality of detection locations in the image are detected based on the comparison result in step S1403 (step S1404). Next, the frequency according to the direction of the motion vector detected for each of the plurality of detection points is totaled over a plurality of images (step S1405). Then, it is determined whether or not the variation in the frequency for each direction collected in step S1405 is included in a predetermined variation range (step S1406). Here, if the determination result is within a predetermined variation range, it is determined that the subject has moved back and forth in the depth direction (step S1407). On the other hand, if the determination result is not within a predetermined variation range, the subject is not moved back and forth in the depth direction. For example, in the case of determination processing in a camera shake correction system, correction is conventionally performed as camera shake in the vertical or horizontal direction. Process.

    <Simple Explanation of Effects of Second Embodiment> As described above, the variation in the frequency of the motion vector detected by the depth direction movement determination device of the present embodiment is determined, thereby moving back and forth in the depth direction with high accuracy. Can be determined. As a result, for example, it is possible to display a warning message for camera shake that moves back and forth in the depth direction with higher accuracy and to correct a camera shake moving image.

  Embodiment 3 <Overview of Embodiment 3> The camera shake correction system of the present embodiment is a camera shake correction system using the depth direction movement determination device described in Embodiment 1 and Embodiment 2. In other words, the camera shake correction system is capable of determining the camera shake in the depth direction in addition to the conventional camera shake correction function and correcting the camera shake in the depth direction based on the determination result.

    <Configuration of Third Embodiment> FIG. 15 is a diagram illustrating an example of functional blocks in the camera shake correction system of the present embodiment. As shown in this figure, the “camera shake correction system” (1500) of this embodiment includes a “depth direction movement determination device” (1501) and an “image information correction unit” (1502). Note that the “depth direction movement determination device” includes an “image information acquisition unit”, a “motion vector detection unit”, and a “movement determination unit” as described in the first embodiment. Alternatively, as described in the second embodiment, the “movement determination unit” further includes “aggregation means”, “variation determination means”, and “variation dependent forward / backward movement determination means”. However, the “image information acquisition unit”, “motion vector detection unit”, “movement determination unit”, “aggregation unit”, “variation determination unit”, and “variation dependent forward / backward movement determination unit” in these “depth direction movement determination devices” "Has already been described in Example 1 or Example 2, and the description thereof will be omitted.

  The “image information correction unit” (1502) has a function of correcting the image information acquired by the image information acquisition unit based on the determination result from the depth direction movement determination device (1501). FIG. 16 is a diagram for explaining an example of correction processing in the image information correction unit. As shown in FIG. 16 (a), three pieces of image information 1, 2, 3 that are temporally continuous constituting a moving image are acquired. These are read into a buffer memory or the like, where it is determined that there is a back-and-forth movement in the depth direction by the processing of the motion vector detection unit and the movement determination unit. Then, in this image information correction unit, for example, the average movement amount in pixel units of the motion vector at a plurality of detection points detected by the motion vector detection unit, and the pixels in the pixel region (effective for shooting) of the shot image From the ratio with the size, the reduction and enlargement magnification for reducing and enlarging the captured image are calculated. Then, the image information is corrected based on the calculated magnification. It should be noted that the determination as to the reduction magnification or the enlargement magnification is preferably made based on whether or not the direction of the detected motion vector is the center direction of the image.

  Alternatively, the processing may be such that the image information is not corrected when there is camera shake in the depth direction. In addition, in the case where image blur in the vertical or horizontal direction is determined in the image information temporally prior to the image information in which the camera shake in the depth direction is determined, the image information correction unit performs correction, but the correction target The correction based on the movement vector in the depth direction in the image information to be performed is not performed, and the correction is performed only in the vertical and horizontal directions based on the movement vector in the previous image information in terms of time. Also good.

  Of course, if it is determined by the processing of the motion vector detection unit or the movement determination unit that the camera shake is not in the depth direction but up and down or left and right, the movement amount of the subject is calculated from the detected motion vector, for example, It is preferable that this image information correction unit perform normal camera shake correction processing such as moving the effective pixel region by the amount of movement.

  The correction of the image information by the “image information correction unit” is performed as follows, for example. First, as described above, the color data and the position data for each pixel indicating the plurality of pieces of image information acquired by the image information acquisition unit are stored in a predetermined memory in the temporary storage memory (1904) of FIG. 19 corresponding to each pixel. Stored in the address. When the determination result information of the subject's back and forth movement based on the motion vector described in the first and second embodiments is output, a program for detecting and correcting the image information is expanded from the storage device to the main storage area of the memory. Are sequentially executed by the CPU. The pixel color data and position data are corrected by the arithmetic processing of the CPU with respect to the pixel data to be corrected from the determination result. Then, the same correction processing as described above is performed on the pixel to be corrected, and data of each pixel indicating the corrected image information is output. Then, the output information indicating each pixel is stored again in a predetermined area such as a storage area, or is drawn in a memory for image output and output to a display or the like.

    <Processing Flow of Third Embodiment> FIG. 17 is a flowchart illustrating an example of a processing flow in the depth direction movement determination device of the present embodiment. As shown in this figure, first, the n-th image information is acquired (step S1701), and then the (n + 1) -th image information is acquired (step S1702). Next, the n-th image information and the n + 1-th image information are compared for a plurality of detection locations in the image indicated by the n-th image information acquired in step S1701 (step S1703). Further, local motion vectors at a plurality of detection locations in the image are detected based on the comparison result in step S1703 (step S1704). Next, based on the detection result in step S1704, it is determined whether the subject is moving back and forth in the depth direction (step S1705). Based on the determination result, the n-th image information acquired in step S1701 is corrected (step S1706).

    <Simple Explanation of Effects of Third Embodiment> As described above, the camera shake correction system according to the present embodiment takes into account the camera shake in the depth direction, which has not been realized by the conventional camera shake correction system arm. Correction can be performed. Therefore, it is possible to avoid a situation in which the same processing as the correction in the vertical and horizontal directions is erroneously performed on an image photographed due to camera shake in the depth direction.

  << Example of Use of Camera Shake Correction System of Example 3 >> <Outline of Electronic Device Provided with Camera Shake Correction System of Example 3> Note that examples of the electronic device provided with the camera shake correction system include a wireless communication device with a camera, For example, a portable information terminal with a camera. If these electronic devices are, for example, wireless communication devices with a camera, taking pictures on the assumption that they are carried outdoors, there are many cases where shooting is likely to occur. Therefore, it is preferable that the camera shake correction system of the present embodiment can be used to perform various corrections corresponding to camera shake in the depth direction in addition to the top, bottom, left, and right. Alternatively, for example, in the case of a portable information terminal with a camera, it is often the case that the arm is slightly stretched and held with one hand as described above, and it is likely that camera shake due to back and forth movement in the depth direction is likely to occur. Therefore, the effect by providing the camera shake correction system of the present embodiment can be expected to be high.

    <Configuration of Electronic Device Provided with Camera Shake Correction System> FIG. 18 is a diagram for explaining an example of functional blocks in an electronic device provided with the camera shake correction system. As shown in this figure, the “electronic device” (1800) includes a “camera shake correction system” (1801), a “corrected image information acquisition unit” (1802), and a “corrected image information output unit” (1803). Have. As described in the third embodiment, the “camera shake correction system” includes an “image information acquisition unit”, a “motion vector detection unit”, a “movement determination unit”, and an “image information correction unit”. is doing. Alternatively, as described in the second embodiment, the “movement determination unit” further includes “aggregation means”, “variation determination means”, and “variation dependent forward / backward movement determination means”. However, the “image information acquisition unit”, “motion vector detection unit”, “movement determination unit”, “image information correction unit”, “aggregation unit”, “variation determination unit”, and “variation” in these “camera shake correction systems” Since the “dependent forward / backward movement determination means” has already been described in the first, second, and third embodiments, the description thereof will be omitted.

  The “corrected image information acquisition unit” (1802) has a function of acquiring image information corrected by the camera shake correction system. The corrected image information acquired here may be encoded image information.

  The “corrected image information output unit” (1803) has a function of outputting the corrected image information acquired by the corrected image information acquisition unit (1802). The output form of the corrected image information output unit is, for example, output of corrected image information for display on a display, output of corrected image information for recording on a recording medium, or other connection via a connection cable such as a network. For example, output of corrected image information to be transmitted to an external device. Of course, if it is encoded information, it is decoded and output as appropriate.

  Specifically, for example, if the corrected image information stored in the buffer memory or the storage device as described above is an analog monitor via the image output unit (1907) shown in FIG. 19, analog data is output by a D / A converter. If the monitor can be digitally displayed, it is output as digital data to the monitor (1908) for display.

    <Simple description of the effect of the electronic device provided with the above-described camera shake correction system> As described above, since these electronic devices are provided with the camera shake correction system, for example, when the photographing apparatus is carried outdoors or a portable information terminal with a camera. As described above, it is also possible to correct the camera shake in the depth direction that occurs when the arm is slightly stretched and held with one hand. Then, with this electronic device, an image with improved camera shake and improved quality can be output to a display device such as a display or recorded on a recording medium.

The figure for demonstrating the direction of the "blur" of the imaging device produced at the time of imaging | photography with a camera The figure showing an example of the blur in the picked-up image produced by the blur of an imaging device The figure showing an example of the functional block in the depth direction movement determination apparatus of Example 1. The figure for demonstrating an example of the several image information acquired with the time transition in the image information acquisition part of the depth direction movement determination apparatus of Example 1 with respect to the same subject. The figure for demonstrating an example of the detection of the motion vector in the motion vector detection part of the depth direction movement determination apparatus of Example 1. FIG. The figure for demonstrating an example of the determination by the sampling in the movement determination part of the depth direction movement determination apparatus of Example 1. FIG. Another figure for demonstrating an example of the determination by the sampling in the movement determination part of the depth direction movement determination apparatus of Example 1. FIG. The flowchart showing an example of the flow of the process in the depth direction movement determination apparatus of Example 1. The figure showing an example of the functional block in the depth direction movement determination apparatus of Example 2. The figure for demonstrating an example of the total of the frequency according to the direction of a motion vector in the totaling means of the depth direction movement determination apparatus of Example 2. FIG. Another figure for demonstrating an example of the total of the frequency according to the direction of a motion vector in the totaling means of the depth direction movement determination apparatus of Example 2. FIG. Further another figure for demonstrating an example of the total of the frequency according to the direction of a motion vector in the totaling means of the depth direction movement determination apparatus of Example 2. FIG. An example of the difference in the frequency of each detected motion vector according to the direction due to the difference in blurring in the depth direction movement determination apparatus according to the second embodiment in the vertical and horizontal directions of the image capturing apparatus or in the depth direction. Illustration to explain The flowchart showing an example of the flow of the process in the depth direction movement determination apparatus of Example 2. The figure showing an example of the functional block in the camera-shake correction system of Example 3. FIG. 10 is a diagram for explaining an example of correction processing in the image information correction unit of the camera shake correction system according to the third embodiment. 9 is a flowchart illustrating an example of a process flow in the camera shake correction system according to the third embodiment. FIG. 10 is a diagram for explaining an example of functional blocks in an electronic apparatus including the camera shake correction system according to the third embodiment. The figure shown about an example of the hardware constitutions of the imaging device provided with the depth direction movement determination apparatus of this invention

Explanation of symbols

0300 Depth direction movement determination device 0301 Image information acquisition unit 0302 Motion vector detection unit 0303 Movement determination unit 0904 Aggregation unit 0905 Variation determination unit 0906 Variation dependent forward / backward movement determination unit

Claims (10)

An image information acquisition unit for acquiring image information;
A motion vector detection unit that detects local motion vectors at a plurality of detection locations in the image based on the plurality of pieces of image information acquired by the image information acquisition unit;
A movement determination unit that determines the forward and backward movement of the subject in the depth direction based on the detection result in the motion vector detection unit;
A depth direction movement determination device having The movement determination unit
Aggregating means for aggregating the frequency according to the direction of the motion vector detected for each of the plurality of detection points over a plurality of images;
Variation determining means for determining whether the variation in the frequency for each direction aggregated by the aggregation means is included in a predetermined variation range;
A variation-dependent back-and-forth movement determination unit that determines that the subject has moved back and forth in the depth direction when the determination result by the variation determination unit is within a predetermined variation range;
The depth direction movement determination apparatus according to claim 1, comprising: Depth direction movement determination device according to claim 1 or 2,
An image information correction unit that corrects image information acquired by the image information acquisition unit based on a determination result from the depth direction movement determination device;
An image stabilization system. An image information acquisition step for acquiring image information;
Based on a plurality of pieces of image information acquired in the image information acquisition step, a motion vector detection step of detecting local motion vectors at a plurality of detection locations in the image;
A movement determination step for determining a back-and-forth movement of the subject in the depth direction based on the detection result in the motion vector detection step;
A depth direction movement determination method comprising: An image information acquisition step for acquiring image information;
Based on a plurality of pieces of image information acquired in the image information acquisition step, a motion vector detection step of detecting local motion vectors at a plurality of detection locations in the image;
A counting step of counting the frequency according to the direction of the motion vector detected for each of the plurality of detection points over a plurality of images;
A variation determination step for determining whether the variation in frequency for each direction aggregated in the aggregation step is included in a predetermined variation range;
A variation-dependent back-and-forth movement determination step for determining that the subject has moved back and forth in the depth direction when the determination result in the variation determination step is within a predetermined variation range;
A depth direction movement determination method comprising: An image information acquisition step for acquiring image information;
Based on a plurality of pieces of image information acquired in the image information acquisition step, a motion vector detection step of detecting local motion vectors at a plurality of detection locations in the image;
A movement determination step for determining a back-and-forth movement of the subject in the image based on a detection result in the motion vector detection step;
Depth direction movement determination program that causes a computer to execute. An image information acquisition step for acquiring image information;
Based on a plurality of pieces of image information acquired in the image information acquisition step, a motion vector detection step of detecting local motion vectors at a plurality of detection locations in the image;
A counting step of counting the frequency according to the direction of the motion vector detected for each of the plurality of detection points over a plurality of images;
A variation determination step for determining whether the variation in frequency for each direction aggregated in the aggregation step is included in a predetermined variation range;
A variation-dependent back-and-forth movement determination step for determining that the subject has moved back and forth in the depth direction when the determination result in the variation determination step is within a predetermined variation range;
Depth direction movement determination program that causes a computer to execute.   A computer-readable recording medium on which the depth direction movement determination program according to claim 6 or 7 is recorded. The image stabilization system of claim 3 is provided,
A corrected image information acquisition unit for acquiring image information corrected by the camera shake correction system;
A corrected image information output unit that outputs the corrected image information acquired by the corrected image information acquisition unit;
Electronic equipment having   The electronic device according to claim 9, wherein the electronic device is a wireless communication device with a camera or a portable information terminal with a camera.

Images