JP2018006992A - Imaging device and electronic component - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of accurately detecting a travel speed of a subject in a real world (inertia coordinate system).SOLUTION: The imaging device comprises: an imaging part for imaging a subject image, and for outputting an imaging signal; a subject position detection part for detecting the position of a subject in an imaging screen from the imaging signal; an attitude detection part for detecting an attitude of the imaging device; a field angle acquisition part for acquiring a field angle of the imaging part; a distance detection part for detecting a distance to the subject; and a speed calculation part for calculating a travel speed of the subject on the basis of the position and the attitude and the field angle and the distance in several times of imaging.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging device and an electronic component.

  A technique for detecting a main subject by detecting a motion of a subject in a moving image is known (for example, Patent Document 1). With the technique described in Patent Document 1, the moving speed of the subject in the real world (inertial coordinate system) cannot be detected.

JP 2010-9425 A

According to the first aspect, the imaging device includes an imaging unit that captures a subject image and outputs an imaging signal, a subject position detection unit that detects the position of the subject in the imaging screen from the imaging signal, and the attitude of the imaging device. A position detection unit for detecting, a field angle acquisition unit for acquiring a field angle of the imaging unit, a distance detection unit for detecting a distance to the subject, the position, the posture, and the field angle obtained by multiple imaging. A speed calculation unit that calculates the moving speed of the subject based on the distance.
According to the second aspect, the imaging device includes an imaging unit that captures a subject image and outputs an imaging signal, a subject position detection unit that detects the position of the subject in the imaging screen from the imaging signal, and the attitude of the imaging device. A main body that identifies a main subject from a plurality of the subjects by a posture detection unit to detect, a field angle acquisition unit that acquires a field angle of the imaging unit, and the position, the posture, and the field angle of a plurality of times of imaging. A subject specifying unit.
According to the third aspect, the electronic component includes an imaging signal input unit to which an imaging signal obtained by imaging a subject image is input, a subject position detection unit that detects the position of the subject in the imaging screen from the imaging signal, and an imaging device A posture detection unit that detects the posture of the subject, an angle-of-view acquisition unit that acquires an angle of view when the subject image is captured, a distance detection unit that detects a distance to the subject, and the position obtained by multiple imaging A speed calculator that calculates a moving speed of the subject based on the posture, the angle of view, and the distance.
According to the fourth aspect, the electronic component includes an imaging signal input unit that receives an imaging signal obtained by imaging a subject image, a subject position detection unit that detects the position of the subject in a shooting screen from the imaging signal, and an imaging device A plurality of the subjects by the posture detection unit that detects the posture of the image, the angle of view acquisition unit that acquires the angle of view when the subject image is captured, and the position, the posture, and the angle of view obtained by multiple imaging. A main subject specifying unit for specifying the main subject.

1 is a block diagram schematically showing the configuration of an imaging apparatus according to a first embodiment Diagram illustrating the definition of the coordinate system The top view which shows the positional relationship of an imaging device and a candidate subject in the time t0. The top view which shows the positional relationship of an imaging device and a candidate subject at the time t1 A plan view showing the positional relationship between the imaging device and the candidate subject at times t0 and t1 The top view which shows the positional relationship of an imaging device and a candidate subject in the time t0. The figure which shows typically the positional relationship of an imaging device and a candidate subject in the time t0 and t1. FIG. 2 is a block diagram schematically illustrating the configuration of an imaging apparatus according to a second embodiment. The figure which illustrates the display image of time t1 and t2 The top view which shows typically the positional relationship of an imaging device and a candidate subject at the time t1 and t2. The figure which illustrates the change (difference vector) of the vector of the face A direction

(First embodiment)
FIG. 1 is a block diagram schematically illustrating the configuration of the imaging apparatus 1 according to the first embodiment. The imaging device 1 has a camera body 2 and a lens barrel 3. The lens barrel 3 is a so-called interchangeable lens that can be attached to and detached from the camera body 2. Note that the lens barrel 3 may not be replaceable and may be configured integrally with the camera body 2.

  The lens barrel 3 includes an imaging optical system 30, a zoom encoder 31, a distance encoder 32, a lens driving unit 33, and a lens control unit 34. The lens barrel 3 illustrated in FIG. 1 is a so-called zoom lens. When the user operates a zoom ring (not shown), the focal length of the imaging optical system 30 is changed by a zoom mechanism (not shown).

  The zoom encoder 31 outputs a zoom signal corresponding to a change in the focal length of the imaging optical system 30 by a zoom mechanism (not shown) to the lens control unit 34. The lens drive unit 33 changes the focal position of the imaging optical system 30 by driving a focus lens (not shown) included in the imaging optical system 30. The distance encoder 32 outputs a distance signal corresponding to a change in the photographing distance (focal position) of the imaging optical system 30 to the lens control unit 34. The lens control unit 34 includes, for example, a CPU (not shown) and its peripheral circuits. The lens control unit 34 controls each part of the lens barrel 3.

  The camera body 2 includes an imaging unit 21, a posture / translation detection unit 22, a distance detection unit 23, a body control unit 24, and a display unit 25. The imaging unit 21 is an imaging device such as a CCD or CMOS. The imaging unit 21 captures a subject image and outputs an imaging signal to the body control unit 24. The posture / translation detection unit 22 detects the posture and translational motion of the imaging apparatus 1. The posture / translation detection unit 22 includes a triaxial angular velocity sensor 221, a triaxial acceleration sensor 222, a triaxial geomagnetic sensor 223, and a calculation unit 224.

  The triaxial angular velocity sensor 221 detects the angular velocity of the imaging device 1. The triaxial acceleration sensor 222 detects the acceleration of the imaging device 1. The triaxial geomagnetic sensor 223 is a so-called electronic compass and detects geomagnetism. The calculation unit 224 uses the information on the angular velocity detected by the triaxial angular velocity sensor 221, the information on the acceleration detected by the triaxial acceleration sensor 222, and the information on the geomagnetism detected by the triaxial geomagnetic sensor 223. Information on the posture and translational motion of 1 is calculated. The information regarding the translational motion is, for example, the speed or displacement of the translational motion. The calculation unit 224 outputs information related to the calculated posture and information related to translational motion to the body control unit 24.

  The distance detection unit 23 includes a distance image sensor using, for example, a ToF (Time of Flight) method, a Light Coding method, or the like, and detects the distance from the imaging device 1 to the subject. The distance detection unit 23 outputs distance information corresponding to each pixel of the imaging unit 21. Here, “distance information corresponding to each pixel” does not necessarily mean distance information for each pixel. For example, one piece of distance information may be output for every 2 × 2 total of four pixels. In addition, the imaging ranges of the imaging unit 21 and the distance detection unit 23 are not necessarily the same. It is sufficient that distance information corresponding to each pixel of the imaging unit 21 can be output at least. When the both imaging ranges are the same, it is desirable that the resolutions of the imaging unit 21 and the distance detection unit 23 have an integer multiple relationship. The frame rate of the distance image sensor of the distance detection unit 23 is desirably the same as the frame rate of the display image described later or an integer multiple of the frame rate of the display image.

  The body control unit 24 includes, for example, a CPU (not shown) and its peripheral circuits. The body control unit 24 controls each part of the camera body 2. The display unit 25 is a display device using a display element such as a liquid crystal panel. The body control unit 24 functionally includes an angle-of-view acquisition unit 241, a subject detection unit 242, a speed calculation unit 243, a main subject specification unit 244, a focus adjustment unit 245, an exposure control unit 246, and a white balance control unit 247. These units are implemented in software by the body control unit 24 executing a predetermined control program. In addition, you may comprise these each part with the electronic circuit which has an equivalent function.

  The angle of view acquisition unit 241 receives the focal length information and the shooting distance information of the imaging optical system 30 from the lens control unit 34 and acquires the angle of view of the imaging optical system 30. The lens control unit 34 generates shooting distance information from the output of the distance encoder 32. The lens control unit 34 generates focal length information from the output of the zoom encoder 31.

  The subject detection unit 242 detects the position of one or more subjects (candidate subjects that are candidates for the main subject) in the shooting screen from the imaging signal output from the imaging unit 21. The subject detection unit 242 detects candidate subjects by methods such as template matching, face detection, human body detection, and moving object detection based on interframe differences.

  The speed calculation unit 243 calculates a moving speed for each candidate subject from the position of the candidate subject obtained by a plurality of times of imaging, the orientation and translational motion of the imaging device 1, the angle of view of the imaging optical system 30, and the distance of the candidate subject. Although details will be described later, the speed calculation unit 243 calculates the movement speed in the inertial coordinate system, not the movement speed in the imaging screen or the relative movement speed with the imaging apparatus 1. For example, even when the user operates the zoom ring to change the zoom ratio of the imaging optical system or moves the imaging device 1 itself, the speed calculation unit 243 calculates the actual moving speed of the candidate subject.

  The main subject specifying unit 244 specifies one candidate subject as a main subject from one or more candidate subjects based on the moving speed of each candidate subject. The focus adjustment unit 245 performs automatic focus adjustment of the imaging optical system so that the main subject specified by the main subject specifying unit 244 is in focus. The exposure control unit 246 performs automatic exposure adjustment so that the main subject specified by the main subject specifying unit 244 is imaged with appropriate exposure. The white balance control unit 247 performs automatic white balance adjustment so that the main subject specified by the main subject specifying unit 244 is imaged with an appropriate white balance.

(Explanation of main subject automatic detection function)
The imaging apparatus 1 according to the present embodiment has a main subject automatic detection function based on the moving speed. The main subject automatic detection function will be described below.

  When the imaging apparatus 1 is in a power-on state, the imaging unit 21 repeatedly performs imaging at a predetermined cycle (for example, 1/60 second) and outputs an imaging signal. The body control unit 24 creates a display image (a so-called through image) based on the imaging signal and displays it on the display unit 25. When the user performs a predetermined imaging operation (for example, a pressing operation of a release switch (not shown)), the body control unit 24 causes the imaging unit 21 to perform imaging, and an image for storage is generated based on the imaging signal output from the imaging unit 21. Create and store in a storage medium (not shown). This imaging for creating the storage image is called main imaging.

  The display image may be an image having a smaller number of pixels than the storage image. For example, the number of pixels equal to the number of display pixels of the display unit 25 may be used, or the number of pixels slightly higher than the number of display pixels of the display unit 25 may be used. By doing so, it is possible to reduce the amount of calculation and power consumption required for creating a display image. Further, when creating a display image, the imaging unit 21 may output an imaging signal obtained by performing so-called thinning readout. That is, the imaging unit 21 may perform imaging for a display image using only a part of the imaging pixels instead of using all the imaging pixels included in the imaging unit 21.

  The user designates the moving speed of the main subject in advance before performing the main imaging. For example, an operation member (not shown) is operated to input to the body control unit 24 that a subject having a speed of 200 km / h or more is the main subject. The subject detection unit 242 detects main subject candidates from the display image. The speed calculation unit 243 calculates the moving speed of the candidate subject detected from the display image by a process described later. If there is a candidate subject with the moving speed specified by the user, the main subject specifying unit 244 specifies that the candidate subject is the main subject.

  When the main subject is specified by the main subject specifying unit 244, automatic focus adjustment by the focus adjustment unit 245, automatic exposure adjustment by the exposure control unit 246, and automatic white balance adjustment by the white balance control unit 247 are the main subjects. To be done. Thereby, since the setting suitable for the main imaging of the main subject is automatically set in the imaging device 1 without performing an operation other than designating the moving speed in advance, the user can concentrate on framing and the like. .

  The main subject specifying operation by the main subject specifying unit 244 described above is also performed during moving image capturing. At the time of moving image capturing, the processing described as being performed on the display image is performed on each frame of the moving image. That is, the subject detection unit 242, the speed calculation unit 243, and the main subject specifying unit 244 execute the above-described processing on each frame of the moving image in real time, and the main subject specifying unit 244 specifies the main subject in each frame. The Then, automatic focus adjustment by the focus adjustment unit 245, automatic exposure adjustment by the exposure control unit 246, and automatic white balance adjustment by the white balance control unit 247 are performed on the main subject. As a result, the setting suitable for moving image capturing of the main subject is automatically set in the imaging apparatus 1 without performing an operation other than specifying the moving speed in advance, so that the user can concentrate on framing and the like. .

(Description of posture / translation detection unit 22)
FIG. 2 is a diagram illustrating the definition of the coordinate system. In the following description, the coordinate system defined in FIG. 2 is used. As shown in FIG. 2B, the inertial coordinate system is defined by an eastward E axis, a northward N axis, and an upward U axis. In addition, as a coordinate system focused on the imaging apparatus 1, an imaging apparatus coordinate system defined by the X axis, the Y axis, and the Z axis of the imaging apparatus 1 shown in FIG.

  The triaxial angular velocity sensor 221, the triaxial acceleration sensor 222, and the triaxial geomagnetic sensor 223 are fixed to the imaging device 1. The calculation unit 224 of the posture / translation detection unit 22 determines the relative angle between the inertial coordinate system and the imaging device coordinate system based on the outputs of the triaxial acceleration sensor 222 and the triaxial geomagnetic sensor 223 when the imaging device 1 is stationary. The attitude of the imaging device 1 is calculated. The calculation unit 224 sets the posture of the imaging device 1 when it is stationary as the initial posture of the imaging device 1. In the following description, the initial posture of the imaging device 1 is represented by a quaternion q0 (q00, q01, q02, q03).

  Thereafter, the calculation unit 224 tracks the change in the attitude of the imaging device 1 from the output of the triaxial angular velocity sensor 221. For example, from the output of the triaxial angular velocity sensor 221, the rotated posture of the imaging device 1 is expressed by quaternions. Since the quaternion time derivative can be expressed by an angular velocity vector (roll, pitch, yaw), the differential equation can be solved with the initial posture q0 as an initial value to obtain the quaternion q (t) at each time. Thereby, the latest posture information of the imaging device 1 is always output from the posture / translation detection unit 22 to the body control unit 24.

  Note that the sampling frequency of the angular velocity necessary for obtaining the quaternion is preferably higher than the frame rate of the display image. Further, it is more desirable that the sampling frequency of the angular velocity is an integer multiple of the frame rate (because quaternions can be obtained at equal intervals to the image acquisition timing). For example, when the frame rate of the display image is 30 fps, the sampling frequency of the angular velocity is preferably 60 Hz or 120 Hz.

  The body control unit 24 defines the optical axis direction of the imaging device 1 in the inertial coordinate system based on the orientation of the imaging device 1 detected by the orientation / translation detection unit 22. For example, when a unit vector (1, 0, 0) in the optical axis direction (X axis) of the imaging device 1 at time t is converted into an inertial coordinate system by quaternion q (t), a vector in the optical axis direction of length 1 is obtained. It is done. This vector corresponds to a point on the spherical surface when a spherical surface having a radius of 1 centered on the imaging device 1 is defined.

  In addition, although the case where the initial orientation with respect to the inertial coordinate system of the imaging device 1 is known has been described, when the triaxial acceleration sensor 222 or the triaxial geomagnetic sensor 223 is not provided and the initial orientation of the imaging device 1 is unknown, The optical axis direction may be a relative direction based on the initial direction of the imaging device 1.

  The computing unit 224 further detects the translational motion of the imaging device 1 from the attitude of the imaging device 1 described above and the output of the triaxial acceleration sensor 222.

(Description of the speed calculation unit 243)
Hereinafter, processing for calculating the moving speed of the candidate subject in the inertial coordinate system, which is executed by the speed calculation unit 243, will be described. First, a case will be described in which the posture change of the imaging device 1 is limited to the yaw direction (Z-axis direction) and the movement of the candidate subject is limited to the EN direction (on the two-dimensional plane) of the inertial coordinate system.

  FIG. 3 is a plan view showing the positional relationship between the imaging device 1 and the candidate subject at time t0. In FIG. 3, the candidate subject for which the moving speed is calculated is a car. In the following description, this car is simply called a candidate subject. In the upper left of FIG. 3, a display image captured at time t0 is illustrated.

  In FIG. 3, the size of the display image in the long side direction is 2L, the horizontal coordinate in the display image of the candidate subject (for example, the center of the display image is 0, the left direction is negative, and the right direction is positive). The coordinate) is Yt0, and the half angle of view around the Z axis of the imaging apparatus 1 is θf. At this time, the following equation (1) holds for Yt0. An angle θt0 formed by the optical axis of the imaging optical system 30 and the direction of the candidate subject viewed from the imaging unit 21 is obtained by the following equation (2) obtained by modifying equation (1).

  Since the size of the display image in the long side direction is known (predetermined), L is known. Yt0 is detected by the subject detection unit 242. θf is acquired by the view angle acquisition unit 241. The speed calculation unit 243 applies these pieces of information to the above equation (2) and calculates θt0.

  The speed calculation unit 243 specifies distance information corresponding to the position of the candidate subject detected by the subject detection unit 242 from distance information corresponding to each pixel of the imaging unit 21 output by the distance detection unit 23. The distance information specified here indicates the distance Dt0 from the imaging device 1 to the candidate subject at time t0. The speed calculation unit 243 calculates an angle θc0 formed by the optical axis of the imaging optical system 30 and the E axis of the inertial coordinate system from the attitude of the imaging device 1 in the inertial coordinate system output from the attitude / translation detection unit 22. .

  FIG. 4 is a plan view showing the positional relationship between the imaging device 1 and the candidate subject at time t1 t seconds after time t0. In the upper left of FIG. 4, a display image captured at time t1 is illustrated as in FIG. Similar to the case of time t0, the speed calculation unit 243 calculates an angle θt1 formed by the optical axis of the imaging optical system 30 and the direction of the candidate subject viewed from the imaging unit 21 by the following equation (3). In the following equation (3), the horizontal coordinate in the display image of the candidate subject at time t1 is expressed as Yt1.

  Similarly to the case of time t0, the speed calculation unit 243 obtains the distance Dt1 from the imaging device 1 to the candidate subject at time t1 based on the output of the distance detection unit 23. Similarly to the case of time t0, the speed calculation unit 243 calculates an angle θc1 formed by the optical axis of the imaging optical system 30 and the E axis of the inertial coordinate system at time t1 based on the output of the posture / translation detection unit 22. The speed calculation unit 243 further calculates the translation Vc · t of the imaging device 1 performed during t seconds from the time t0 to the time t1 from the translational motion of the imaging device 1 detected by the posture / translation detection unit 22. Ask.

  FIG. 5 is a plan view showing the positional relationship between the imaging device 1 and the candidate subject at times t0 and t1. Assuming that the movement of the candidate subject made during t seconds from time t0 to time t1 is VT · t, the movement speed VT of the candidate subject in this movement is obtained by the following equation (4).

  In the above equation (4), VTE, VTN, and VTU are the moving speeds of candidate subjects in the E-axis direction, N-axis direction, and U-axis direction, respectively. VCE and VCN are moving speeds of the imaging apparatus 1 in the E-axis direction and the N-axis direction, respectively. The speed calculation unit 243 calculates the moving speed of the candidate subject by calculating the above equation (4).

  Next, a case where the posture change of the imaging apparatus 1 and the movement of the candidate subject are not limited will be described. FIG. 6 is a plan view showing the positional relationship between the imaging device 1 and the candidate subject at time t0. In FIG. 6, the candidate subject for which the moving speed is calculated is an airplane. In the following description, this airplane is simply called a candidate subject. In the upper left of FIG. 6, a display image captured at time t0 is illustrated.

  In FIG. 6, the size of the display image in the long side direction is 2LY, the size of the display image in the short side direction is 2LZ, the coordinates in the display image of the candidate subject (for example, the center of the display image is 0, and the left direction (Yt0, Zt0), the half angle of view around the Z axis of the imaging device 1 is θf, and the half angle of view around the Y axis of the imaging device 1 is Let φf. At this time, the following equation (5) holds for Yt0, and the following equation (6) holds for Zt0. A Z-axis rotation component θt0 of an angle formed by the optical axis of the imaging optical system 30 and the direction of the candidate subject viewed from the imaging unit 21 is obtained by the following equation (7) obtained by modifying the equation (5). A Y-axis rotation component φt0 of an angle formed by the optical axis of the imaging optical system 30 and the direction of the candidate subject viewed from the imaging unit 21 is obtained by the following equation (8) obtained by modifying equation (6).

  Since the size of the display image is known (predetermined), LY and LZ are known. Yt0 and Zt0 are detected by the subject detection unit 242. θf and φf are acquired by the field angle acquisition unit 241. The speed calculation unit 243 applies these pieces of information to the above equations (7) and (8), and calculates θt0 and φt0. Thereby, the direction of the candidate subject viewed from the imaging device 1 in the imaging device coordinate system is specified.

  The speed calculation unit 243 performs imaging in the inertial coordinate system from the orientation of the imaging device 1 in the inertial coordinate system output by the orientation / translation detection unit 22 and the direction of the candidate subject viewed from the imaging device 1 in the imaging device coordinate system. Device 1 → Calculates a candidate subject vector RT.

  When the attitude of the imaging device 1 viewed from the inertial coordinate system is represented by quaternions q0, q1, q2, and q3, the direction cosine matrix DCM from the inertial coordinate system to the imaging device coordinate system is represented by the following equation (9). .

  The subject direction unit vector NTC viewed from the imaging apparatus coordinate system is expressed by the following equation (10).

Coordinate transformation from the imaging device coordinate system to inertial coordinate system, since it is expressed by the transposed matrix DCM ^T direction cosine matrix DCM, subject direction unit vector NT viewed from the inertial coordinate system is obtained by the following equation (11).

  By multiplying the subject distance DT identified from the output of the distance detection unit 23 and the subject direction unit vector NT obtained by the equation (11), the imaging device 1 → the subject vector RT as seen from the inertial coordinate system is obtained. That is, the imaging device 1 → the subject vector RT viewed from the inertial coordinate system is obtained by the following equation (12).

  FIG. 7 is a diagram schematically illustrating the positional relationship between the imaging device 1 and the candidate subject at times t0 and t1. As described above, the posture / translation detection unit 22 can detect the translational motion of the imaging device 1, that is, the translation vector MC (t0 → t1) of the imaging device 1 illustrated in FIG. The speed calculation unit 243 obtains from the imaging device → subject vector RT calculated by the speed calculation unit 243 at time t0 and time t1, and the translation vector MC (t0 → t1) detected by the posture / translation detection unit 22, respectively. The moving speed VTt1 of the candidate subject in the inertial coordinate system is calculated by the following equation (13).

  As described above, the speed calculation unit 243 can calculate the speed vector indicating the movement speed of each candidate subject in the inertial coordinate system regardless of the movement of the candidate subject and the movement of the imaging apparatus 1.

According to the embodiment described above, the following operational effects can be obtained.
(1) The imaging unit 21 captures a subject image and outputs an imaging signal. The subject detection unit 242 detects the position of the subject in the imaging screen from the imaging signal. The posture / translation detection unit 22 detects the posture of the imaging apparatus 1. The angle of view acquisition unit 241 acquires the angle of view of the imaging unit 21. The distance detection unit 23 detects the distance to the subject. The speed calculation unit 243 calculates the moving speed of the subject based on the position, posture, angle of view, and distance obtained by multiple imaging. Since it did in this way, the moving speed of the to-be-photographed object in the real world (inertial coordinate system) can be detected correctly.

(2) The posture / translation detection unit 22 detects the translational motion of the imaging device 1. The speed calculation unit 243 calculates the moving speed of the subject based on the position, posture, angle of view, distance, and translational motion obtained by a plurality of imaging operations. Since it did in this way, even if it is a case where the imaging device 1 translates on the way, the moving speed of the to-be-photographed object in the real world (inertial coordinate system) can be detected correctly.

(3) The main subject specifying unit 244 specifies a main subject from a plurality of subjects based on the moving speed of the subject. Since it did in this way, a main subject can be specified accurately.

(4) The speed calculation unit 243 includes the movement of the imaging device 1 in the inertial coordinate system calculated from the posture and the translational motion, and the movement of the subject in the imaging device coordinate system calculated from the position and the distance (that is, the imaging device 1 and The movement speed of the subject is calculated based on the relative movement of the subject, in other words, the movement of the subject viewed from the imaging device 1. Since it did in this way, the moving speed of the to-be-photographed object in the real world (inertial coordinate system) can be detected correctly.

(5) The focus adjustment unit 245 performs automatic focus adjustment on the identified main subject. The exposure control unit 246 performs automatic exposure adjustment on the identified main subject. The white balance control unit 247 performs automatic white balance adjustment on the identified main subject. Since it did in this way, the imaging device with sufficient operativity which can image a main subject easily can be provided.

(6) The posture / translation detection unit 22 includes a triaxial angular velocity sensor 221 and a triaxial geomagnetic sensor 223. Since this is done, it is possible to accurately detect the moving speed of the subject in the real world (inertial coordinate system) regardless of the initial posture of the imaging apparatus 1.

(Second Embodiment)
FIG. 8 is a block diagram schematically showing the configuration of the imaging apparatus 1001 according to the second embodiment. Although the imaging device 1 according to the first embodiment specifies the main subject based on the moving speed of the subject, the imaging device 1001 specifies the main subject based on the direction of the subject instead of the moving speed. Hereinafter, the imaging apparatus 1001 according to the second embodiment will be described, but the same parts as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment, and description thereof will be omitted.

  The imaging device 1001 has a camera body 1002 and a lens barrel 3. The camera body 1002 includes an imaging unit 21, a posture detection unit 1022, a body control unit 1024, and a display unit 25. The posture detection unit 1022 is an alternative to the posture / translation detection unit 22 and detects the posture of the imaging apparatus 1. The posture detection unit 1022 includes a triaxial angular velocity sensor 221, a triaxial acceleration sensor 222, a triaxial geomagnetic sensor 223, and a calculation unit 1224. The calculation unit 1224 uses the angular velocity information detected by the triaxial angular velocity sensor 221, the acceleration information detected by the triaxial acceleration sensor 222, and the geomagnetic information detected by the triaxial geomagnetic sensor 223 to use the imaging device. The attitude of 1 is calculated. The calculation unit 1224 outputs information regarding the calculated posture to the body control unit 1024.

  The body control unit 1024 includes, for example, a CPU (not shown) and its peripheral circuits. The body control unit 1024 controls each part of the camera body 1002. The body control unit 1024 functionally includes an angle-of-view acquisition unit 241, a subject detection unit 242, a main subject specification unit 1244, a focus adjustment unit 245, an exposure control unit 246, and a white balance control unit 247. Each of these units is implemented in software by the body control unit 1024 executing a predetermined control program. In addition, you may comprise these each part with the electronic circuit which has an equivalent function.

  The main subject specifying unit 244 specifies one candidate subject as a main subject from one or more candidate subjects, based on the position of the candidate subject obtained by a plurality of times of imaging, the attitude of the imaging apparatus 1001, and the angle of view of the imaging optical system 30. .

(Explanation of main subject automatic detection function)
The imaging apparatus 1001 according to this embodiment has a main subject automatic detection function based on the correlation between the orientation of the imaging apparatus 1001 and the direction of the candidate subject. The main subject automatic detection function will be described below.

  The subject detection unit 242 detects candidate subjects from the display image by a method such as template matching, face detection, human body detection, or moving body detection based on interframe differences. In the following description, it is assumed that the candidate subject is a person and the subject detection unit 242 uses face detection to detect the person. The main subject specifying unit 1244 specifies one of the candidate subjects as the main subject by the process described below.

  FIG. 9A is a diagram illustrating a display image at time t1, and FIG. 9B is a diagram illustrating a display image at time t2. In FIG. 9, a total of three faces, face A, face B, and face C, are detected as candidate subjects from the display image. In the following description, the position of the detected candidate subject is represented by coordinates indicating one point on the display screen. The positions of the face A, the face B, and the face C are expressed as coordinates (xA, yA), (xB, yB), and (xC, yC). FIG. 3 shows an example in which yA = yB = yC = 0 in order to simplify the description.

  Pay attention to the movements of these three faces from time t1 to time t2. As seen from the position of the imaging device, face A moves to the right, face B moves to the left, and face C stops. Here, it is assumed that the optical axis of the imaging apparatus 1001 moves to the right from time t1 to time t2. Since the change in the angle of the optical axis is larger than the change in the direction of the face A viewed from the position of the imaging device 1001, the face A moves slightly to the left in the display image as shown in FIG. Further, the face B and the face C are also moved to the left in the display image.

  From the coordinates of the face in the display image and the angle of view information of the display image, the direction of the face viewed from the imaging device 1001 can be known. The orientation of the face in the inertial coordinate system with the imaging device 1001 as the origin can be determined from the orientation of the imaging device 1001 detected by the orientation detection unit 1022. Therefore, a point indicating the face direction in spherical coordinates with the imaging device 1001 as the origin is determined. When the unit spherical surface (radius 1) of the inertial coordinate system with the imaging device 1001 as the origin is defined, the directions of the face A, the face B, and the face C can be replaced with points on the unit spherical surface.

  10A is a plan view schematically showing the positional relationship between the imaging device 1001 and the candidate subject at time t1, and FIG. 10B is the positional relationship between the imaging device 1001 and the candidate subject at time t2. It is a top view which shows typically. FIGS. 10A and 10B are diagrams in which the unit spherical surface of the inertial coordinate system is cut along a horizontal plane passing through the center of the sphere, and the center is the position of the imaging device 1001.

  Since face C is stopped from time t1 to time t2, the vector indicating face C does not change, but the optical axis changes in the N direction, and the fan shape indicating the angle of view also shifts in the N direction. The face C appears to move to the left in the image.

  As described with reference to FIGS. 9 and 10, the face A, the face B, and the face C are obtained by using the display image repeatedly captured a plurality of times and the attitude angle of the image capturing apparatus 1001 when the display image is captured. Can be plotted on the unit sphere. Similarly, the optical axis direction of the imaging device 1001 can be plotted on the unit spherical surface of the inertial coordinate system.

  When imaging using the imaging apparatus 1001, the user changes the orientation of the imaging apparatus 1001 so that the optical axis of the imaging apparatus 1001 faces the direction of the main subject. The main subject specifying unit 1244 displays the direction of the detected candidate subjects (face A, face B, face C) and the direction of the optical axis of the imaging device 1001 on the unit spherical surface of the inertial coordinate system every time the display image is captured. It is calculated as a point. Thereby, direction data corresponding to the frame rate of the display image is calculated. Each of the plurality of direction data is indicated by a line graph on the spherical surface. The main subject specifying unit 1244 specifies a candidate subject (face) showing a motion correlated with the motion in the optical axis direction as the main subject.

  The main subject specifying unit 1244 uses the change on the spherical surface of each point in order to examine the correlation between the motion in the optical axis direction and the motion in the direction of the candidate subject (face). For example, a vector of size 1 corresponding to the direction of the face A in the n-th frame display image is VA (n). This vector represents from the origin to a point corresponding to the face A on the unit sphere. As shown in FIG. 11, the change (difference vector) ΔVA (n + 1) in the face A direction from frame number n to n + 1 is expressed by the following equation (14).

  Similarly, assuming that a vector of size 1 corresponding to the optical axis direction is VO (n), a change (difference vector) ΔVO (n + 1) in the optical axis direction from frame number n to n + 1 is expressed by the following equation (15). ).

  The main subject specifying unit 1244 obtains a normalized inner product of the difference vectors in order to examine the correlation between the change in the direction of the face A and the change in the optical axis direction. For example, the normalized inner product NIPA (n + 1) of the difference vector ΔVA (n + 1) in the face A direction and the difference vector ΔVO (n + 1) in the optical axis direction is calculated by the following equation (16).

  When the direction of ΔVA matches the direction of ΔVO, the normalized inner product NIPA is 1, and when they do not match, the value is smaller than 1 (the minimum value is −1).

  In the same manner as the main subject specifying unit 1244 calculates the normalized inner product NIPA of the difference vector ΔVA in the face A direction and the difference vector ΔVO in the optical axis direction, the difference vector for the face B and the optical axis and the face C and the optical axis are also obtained. The normalized dot products NIPB and NIPC are obtained. If the face A is a main subject, the direction of the face A and the optical axis direction are likely to move in a similar manner, so the direction of the difference vector is close. Therefore, when the sum of normalized inner products NIPA (SNIPA = ΣNPIA) is obtained, values higher than SNIPB and SNIPC are shown as time passes. The main subject specifying unit 1244 specifies the face A as the main subject when the sum SNIPA of the normalized dot product NIPA is higher than SNIPB and SNIPC.

  In the above description, an example of using a normalized inner product as a correlation evaluation value has been shown, but other evaluation values may be used. For example, the inner product IPA shown in the following equation (17), the inner product IPRA in consideration of the vector size ratio shown in the following equation (18), and the normalized inner product in consideration of the vector size ratio shown in the following equation (19) NIPRA or the like can be used as a correlation evaluation value.

  The inner product IPA shown in the above equation (17) has an advantage of easy calculation. The inner product IPRA in consideration of the vector size ratio shown in the above equation (18) has an advantage that the similarity in the direction of the vector and the similarity in the size can be considered. In the normalized inner product NIPRA in consideration of the ratio of vector sizes shown in the above equation (19), the similarity in vector direction and the similarity in size can be considered, and a value in the range of ± 1 is obtained. So there is an advantage that it is easy to handle.

  The processing after the main subject is specified by the main subject specifying unit 1244 is the same as that in the first embodiment, and thus the description thereof is omitted. Note that the points applicable to the time of moving image capturing are as described in the first embodiment.

According to the embodiment described above, the following operational effects can be obtained.
(1) The imaging unit 21 captures a subject image and outputs an imaging signal. The subject detection unit 242 detects the position of the subject in the imaging screen from the imaging signal. The posture detection unit 1022 detects the posture of the imaging device 1001. The angle of view acquisition unit 241 acquires the angle of view of the imaging unit 21. The main subject specifying unit 1244 specifies a main subject from a plurality of subjects based on a position, posture, and angle of view obtained by a plurality of imaging operations. Since it did in this way, a main subject can be specified accurately.

(2) The main subject specifying unit 1244 specifies the main subject by calculating the correlation between the orientation of the imaging device 1001 specified by the position and orientation. Since it did in this way, a main subject can be specified accurately.

(3) The focus adjustment unit 245 performs automatic focus adjustment on the identified main subject. The exposure control unit 246 performs automatic exposure adjustment on the identified main subject. The white balance control unit 247 performs automatic white balance adjustment on the identified main subject. Since it did in this way, the imaging device with sufficient operativity which can image a main subject easily can be provided.

(4) The posture detection unit 1022 includes a triaxial angular velocity sensor 221 and a triaxial geomagnetic sensor 223. Since this is done, the main subject can be accurately identified regardless of the initial posture of the imaging apparatus 1.

(Modification 1)
If the user does not translate the imaging device 1 during framing, the posture / translation detection unit 22 may not be able to detect translational motion. That is, the posture / translation detection unit 22 may not include a sensor necessary for detecting translational motion, such as the triaxial acceleration sensor 222.

(Modification 2)
The moving speed of the candidate subject calculated by the speed calculating unit 243 can be used for purposes other than specifying the main subject. For example, if the speed information of the subject is recorded in the EXIF information of the storage image (still image and moving image), a search based on the moving speed of the subject can be performed from a plurality of storage images.

(Modification 3)
In the second embodiment, the evaluation value (such as the sum of normalized inner products) may be weighted in the time direction. For example, when calculating the sum of normalized inner products, the past integrated value may be multiplied by a coefficient ρ (a value smaller than 1) to reduce the influence of the past value. Alternatively, a moving average of normalized inner products may be obtained. Thus, the weighting in the time direction speeds up the reaction to the change. In this way, it is possible to quickly follow the change of the main subject by the user.

(Modification 4)
In the second embodiment, the position of the candidate subject in the display image may be used for specifying the main subject. For example, since there is a high possibility that the main subject exists near the center of the screen, it can be used to identify the main subject. Specifically, the normalized inner product may be multiplied by a weight function that has a maximum value of 1 at the center of the screen and approaches 0 as the distance from the center of the screen increases. In this way, the main subject can be identified with higher accuracy by considering the position of the candidate subject in the display image.

(Modification 5)
In the second embodiment, when the imaging apparatus 1001 has a so-called image blur correction function, the result of image blur correction may be used to identify the main subject. For example, when the position of the face detected on the display image is represented by coordinates on the screen, the coordinates when there is no lens shift are obtained in consideration of the image shift amount due to image blur correction. By performing this correction, it is possible to determine the exact position of the face in the inertial coordinate system without being affected by the image shift due to the image blur correction, and it is possible to accurately identify the main subject.

(Modification 6)
The function for specifying the main subject and the speed calculation function for the candidate subject in each embodiment described above may be configured as an electronic component such as an IC package. In this case, an imaging signal input unit to which an imaging signal from the imaging unit 21 is input may be provided in the electronic component, and the various processes described above may be performed on the input imaging signal.

  In addition, in a device such as a so-called smartphone or tablet terminal, the main subject specifying function and the candidate subject speed calculation function in each of the above-described embodiments can be mounted.

DESCRIPTION OF SYMBOLS 1,1001 ... Imaging device, 2 ... Camera body, 3 ... Lens barrel, 21 ... Imaging part, 22 ... Attitude / translation detection part, 23 ... Distance detection part, 24, 1024 ... Body control part, 30 ... Imaging optical system 241 ... View angle acquisition unit 242 ... Subject detection unit 243 ... Speed calculation unit 244,1244 ... Main subject identification unit 245 ... Focus adjustment unit 246 ... Exposure control unit 247 ... White balance control unit 1022 ... Attitude detection unit

Claims (10)

An imaging unit that captures a subject image and outputs an imaging signal;
A subject position detector that detects the position of the subject in the imaging screen from the imaging signal;
An attitude detection unit for detecting the attitude of the imaging device;
An angle of view acquisition unit for acquiring an angle of view of the imaging unit;
A distance detector for detecting a distance to the subject;
A speed calculation unit that calculates the moving speed of the subject based on the position, the posture, the angle of view, and the distance obtained by imaging a plurality of times;
An imaging apparatus comprising: The imaging device according to claim 1,
A translational motion detection unit for detecting translational motion of the imaging device;
The speed calculation unit is an imaging device that calculates a moving speed of the subject based on the position, the posture, the angle of view, the distance, and the translational motion by a plurality of times of imaging. The imaging device according to claim 2,
An imaging apparatus further comprising a main subject specifying unit that specifies a main subject from a plurality of the subjects according to a moving speed of the subject. The imaging device according to claim 3.
The speed calculation unit is configured to detect the subject based on the movement of the imaging device in an inertial coordinate system calculated from the posture and the translational motion, and the movement of the subject in the imaging device coordinate system calculated from the position and the distance. Imaging device for calculating the moving speed of the camera. An imaging unit that captures a subject image and outputs an imaging signal;
A subject position detector that detects the position of the subject in the imaging screen from the imaging signal;
An attitude detection unit for detecting the attitude of the imaging device;
An angle of view acquisition unit for acquiring an angle of view of the imaging unit;
A main subject specifying unit for specifying a main subject from a plurality of the subjects based on the position, the posture, and the angle of view obtained by a plurality of times of imaging;
An imaging apparatus comprising: The imaging apparatus according to claim 5,
The main subject specifying unit specifies the main subject by calculating a correlation between the direction of the plurality of subjects obtained from the position, the posture, and the angle of view, and the orientation of the imaging device specified by the posture. An imaging device. In the imaging device according to any one of claims 3 to 6,
An imaging apparatus that performs at least one of automatic focus adjustment, automatic exposure adjustment, and automatic white balance adjustment on the main subject. In the imaging device according to any one of claims 1 to 7,
The posture detection unit is an imaging apparatus including at least one of a triaxial angular velocity sensor, a triaxial acceleration sensor, and a triaxial geomagnetic sensor. An imaging signal input unit to which an imaging signal obtained by imaging a subject image is input;
A subject position detector that detects the position of the subject in the imaging screen from the imaging signal;
An attitude detection unit for detecting the attitude of the imaging device;
An angle of view acquisition unit for acquiring an angle of view when the subject image is captured;
A distance detector for detecting a distance to the subject;
A speed calculation unit that calculates the moving speed of the subject based on the position, the posture, the angle of view, and the distance obtained by imaging a plurality of times;
With electronic components. An imaging signal input unit to which an imaging signal obtained by imaging a subject image is input;
A subject position detector that detects the position of the subject in the shooting screen from the imaging signal;
An attitude detection unit for detecting the attitude of the imaging device;
An angle of view acquisition unit for acquiring an angle of view when the subject image is captured;
A main subject specifying unit for specifying a main subject from a plurality of the subjects based on the position, the posture, and the angle of view obtained by a plurality of times of imaging;
With electronic components.

Images