JP2023119433A - System, apparatus, and control method - Google Patents

Abstract

To provide a system capable of automatically determining imaging settings of a camera according to a motion state of a subject by integrating the motion state of the subject with an analysis result of a camera and a sensing result of a mobile device.SOLUTION: A system includes: an imaging apparatus comprising imaging means configured to capture an image of a subject, subject motion detection means configured to detect motion of the subject using the image captured by the imaging means, and exposure control means configured to control exposure of the imaging means; and a sensor device comprising sensor means for acquiring information on the subject, and capable of being attached to the subject to be imaged, unlike the imaging means having transmission means for transmitting a sensing result of the sensor means to the imaging apparatus wirelessly. The imaging apparatus is configured to perform imaging while controlling exposure of the imaging means using the sensing result of the sensor means and a result of the subject motion detection means.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for controlling photography of a camera, and more particularly to a camera system, apparatus, and control method for photographing using a wearable device.

2. Description of the Related Art Techniques for controlling camera shooting based on sensing results from an acceleration sensor, a human sensor, or the like are known.

For example, Japanese Patent Application Laid-Open No. 2002-200000 discloses controlling automatic shooting of a remote camera according to the status of a portable device (wearable device).

JP 2016-072673 A

However, in the prior art disclosed in the above-mentioned patent document, the timing control of automatic shooting is mainly performed based on the sensing result, and the shooting setting of the camera is automatically performed including the movement state of the subject captured by the camera. not taken into consideration so far. Therefore, even if the timing of photographing can be controlled, there is a possibility that the photographed subject will be blurred if the movement of the subject is fast. Therefore, the object of the present invention is to make it possible to automatically determine the camera shooting settings according to the movement state of the subject by linking the analysis result of the camera and the sensing result of the portable device regarding the movement state of the subject. provide a system that

To achieve the above object, the present invention provides imaging means for imaging a subject, subject motion detection means for detecting movement of the subject using the image captured by the imaging means, and exposure control of the imaging means. It is equipped with an imaging device having exposure control means and a sensor means for acquiring information on a subject, and is worn on the subject being photographed, unlike the imaging means having a transmission means for wirelessly transmitting sensing results of the sensor means to the imaging device. and the imaging device uses the sensing result of the sensor means and the result of the subject motion detection means to control the exposure of the imaging means and perform imaging. characterized by

According to the present invention, it is possible to provide camera control capable of reducing subject blur by taking into consideration the movement of the subject more preferably.

A diagram showing a configuration example of the present invention A diagram showing an example of an appearance of the imaging system 100. Flowchart for explaining the operation of the camera 101 of the first embodiment Flowchart for explaining the operation of the camera 101 of the second embodiment Flowchart for explaining the operation of the wearable device 102 Diagram explaining block matching Flowchart for explaining the operation of determining shooting conditions for a preparatory shot image A diagram showing the movement of the subject Flowchart showing motion vector calculation processing Diagram for correcting the motion vector of the subject Diagram showing the relationship between subject motion vector and subject motion blur amount Diagram for explaining the process of correcting the exposure amount by multiplying the digital gain Flowchart for explaining the details of the main exposure process of the second embodiment Diagram for explaining the configuration of the electronic front curtain shutter Diagram explaining the operation of the electronic front curtain shutter Diagram for explaining how to perform exposure cut-off control Image diagram when there are multiple subjects Flowchart for explaining a case where there are a plurality of wearable devices 102 Flowchart for determining primary wearable device 102 Example of setting screen for setting priority of wearable device 102 FIG. 5 is a diagram for explaining priority determination items of the wearable device 102; FIG. 4 is a diagram for explaining a method of calculating the priority order of the wearable device 102;

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the claimed invention. In addition, although multiple features are described in the embodiments, not all of them are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted. In this embodiment, a smart phone, a wearable device such as a wristband type terminal, and a camera are linked, and the exposure control operation of the camera is realized automatically.

As a result, even in use cases such as unattended shooting and self-portraits, it is possible to perform exposure control so as to suppress the movement of the subject to a desired degree of blurring or less, thereby expanding the scenes in which auto can be used.

FIG. 1 is an example of an imaging system 100 described in this embodiment. The imaging system 100 in FIG. 1 is implemented by two devices, a camera 101 and a wearable device 102 . Examples of the wearable device 102 include a smart phone, a wristband type terminal, and the like, and the form thereof is not limited. FIG. 2 is a diagram showing the appearance of one embodiment of the present invention. FIG. 2A shows an imaging system 100 in which a digital camera 201 as a camera 101 and a smart phone 202 as a wearable device 102 are connected.

The control unit 112 is, for example, a CPU, and reads a control program for each block included in the camera 101 from a ROM 113 (to be described later), develops it in a RAM 114 (to be described later), and executes it. Thereby, the control unit 112 controls the operation of each block included in the camera 101 .

The ROM 113 is an electrically erasable/recordable non-volatile memory, and stores an operation program for each block provided in the camera 101, as well as parameters required for the operation of each block.

A RAM 114 is a rewritable volatile memory, and is used for developing programs executed by the control unit 112 and the like, temporarily storing data generated by operations of blocks included in the camera 101, and the like.

Communication unit 115 performs communication according to a predetermined wireless communication standard. For example, wireless communication standards include IEEE 802.11 standard Wi-Fi, Bluetooth (registered trademark), and NFC, and at least one of them should be supported.

The optical system 121 is composed of a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging surface of an imaging device 122, which will be described later.

The imaging element 122 is configured by, for example, a CCD or CMOS sensor. Each pixel of the imaging device 122 photoelectrically converts an optical image formed on the imaging surface of the imaging device 122 by the optical system 121, and outputs the obtained analog image signal to the A/D conversion unit 123 described later.

The A/D converter 123 converts the input analog image signal into digital image data, and the digital image data output from the A/D converter 123 is temporarily stored in the RAM 114 .

The image processing unit 124 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 114 . The image processing unit 124 also calculates a motion vector between recorded images and detects a subject. Details of these will be described later.

The recording unit 125 is a detachable memory card or the like, and records image data processed by the image processing unit 124 as a recorded image via the RAM 114 .

The pulse generation unit 126 supplies a scan clock (horizontal driving pulse) and a predetermined control pulse to the imaging device 122 when the non-imaging state is shifted to the imaging state. Of the scanning clocks generated by the pulse generating section 126, the clock for vertical scanning is input to the vertical drive modulation section 111, which will be described later.

The vertical drive modulation unit 111 modulates the vertical scanning clock among the scanning clock signals generated by the pulse generation unit 126 to a predetermined clock frequency, and inputs the modulated clock to the imaging device 122 . The vertical drive modulation unit 111 determines a scanning pattern for reset scanning performed for each line of the image sensor 122 consisting of a plurality of pixels. The line-by-line reset scanning of the image sensor 122 realizes the function of an electronic front curtain shutter.

The gyro sensor 119 is a motion detection sensor that detects angular velocity and determines the magnitude of camera shake.

The mechanical shutter 118 is composed of an open/close shutter mechanism that physically shields light (hereinafter, mechanical shutter). Also, the mechanical shutter 118 functions as a rear curtain (hereinafter referred to as a mechanical rear curtain) composed of a plurality of light shielding blades. The control unit 112 controls the exposure time (shutter speed) by adjusting the timing for starting the movement of the mechanical rear curtain. On the other hand, the function of the electronic front curtain is realized by sequentially reset-scanning the pixels of the image sensor 122 line by line at a predetermined timing.

The display unit 127 is a display device such as an LCD, and displays images stored in the RAM 114 and images recorded in the recording unit 125, and displays an operation user interface for accepting user instructions. In addition, the display unit 127 displays an image captured by the imaging device 122 for composition adjustment during preparatory shooting.

The configuration of the camera 101 has been described above.

Next, the configuration of the wearable device 102 will be explained using FIG. The wearable device 102 has a touch screen display 141, and a liquid crystal display 142 displays characters, images, icons, and the like. The touch screen 143 can be operated by detecting gesture operations.

The in-camera 134 includes a lens and an imaging device such as a CCD or CMOS that converts an optical image into an electrical signal. The in-camera 134 is a small camera module having AF (autofocus), aperture, and shutter speed adjustment functions. Then, the in-camera 134 images an object facing the touch screen display 141 .

The illuminance sensor 145 acquires illuminance information of a subject focused by the in-camera 134 and the out-camera 135, and uses it to adjust the exposure time and ISO sensitivity during shooting.

The control unit 138 is, for example, a CPU, and reads a control program for each block included in the wearable device 102 from the ROM 151 described later, develops it in the RAM 152 described later, and executes it. Thereby, the control unit 138 controls the operation of each block included in the wearable device 102 . The control unit 138 provides a camera function by controlling the touch screen 143, the switch 144, the in-camera 134, the illuminance sensor 145, the out-camera 135, the light 136, and the like.

The ROM 151 is an electrically erasable/recordable non-volatile memory, and stores an operation program for each block included in the wearable device 102 as well as parameters required for the operation of each block.

The RAM 152 is a rewritable volatile memory, and is used for expansion of programs executed by the control unit 112 and the like, temporary storage of data generated by operation of each block included in the wearable device 102, and the like.

The speaker 139 outputs a shutter sound and a warning sound during imaging when the switch 144 is turned on for sound output.

The connector 133 is used when connecting the wearable device 102 and an external device. For example, the connector 133 is connected with an AC adapter for charging a battery provided in a power supply module 132, which will be described later. Also, the connector 133 is used when inputting/outputting image data, audio data, etc. to/from a non-volatile memory connected from the outside. Note that the connector 133 may be a specially designed terminal such as a dock connector, or a general-purpose terminal such as a USB (Universal Serial Bus).

The out-camera 135 is a small camera module similar to the in-camera 134 . The out-camera 135 captures an object on the opposite side of the in-camera 134 . The light 136 is a light-emitting module and functions as a flash when the out-camera 135 takes an image.

The communication module 131 performs communication according to a predetermined wireless communication standard. For example, wireless communication standards include IEEE 802.11 standard Wi-Fi, Bluetooth (registered trademark), and NFC, and the communication module 131 may support at least one of them. Specific communication includes input/output of image data obtained by imaging and downloading of function addition program modules to the wearable device 102 . It is also used for transmitting information of a group of sensors (illuminance sensor 145, acceleration sensor 146, gyro sensor 147, depth sensor 148) to the camera 101, which will be described later.

The power module 132 includes a rechargeable battery to power the entire wearable device 102 . As the battery provided in the power supply module 132, for example, a lithium ion battery or a nickel metal hydride battery is used.

Acceleration sensor 146 detects the direction and magnitude of acceleration acting on wearable device 102 . The acceleration sensor 146 is capable of detection in three axes of the XYZ directions.

Gyro sensor 147 detects the angle and angular velocity of wearable device 102 .

A depth sensor 148 measures the distance from the camera to the subject being photographed. As a method for measuring distance, there is a method that measures the time it takes for infrared rays, light, ultrasonic waves, etc. to reflect off an object and bounce back. There are ways to get information.

FIG. 2 shows an example of the appearance of the imaging system 100. As shown in FIG. (a) to (c) of FIG. 2 show combinations of a camera 101 and a wearable device 102 that cooperate as an imaging system 100, respectively. The camera 101 and the wearable device 102 are connected wirelessly, for example, by Bluetooth (registered trademark), and cooperate with the imaging system 100 through communication. 201 denotes a digital camera as an example of the camera 101 . Reference numeral 202 denotes a smart phone as an example of the wearable device 102, and the smart phone often has a camera function. Therefore, like the in-camera 134 and the out-camera 135 mounted on the smartphone 204 in FIG. 2C, it is possible to use the camera 101 side instead of the wearable device 102 side by using the camera function. In addition, the wearable device 102 can also use a form such as a wristband type terminal 203 other than a smart phone.

The appearance and system configuration of the imaging system 100 have been described above.

(First embodiment)
Hereinafter, processing of the imaging system 100 according to the first embodiment (Example 1) of the present invention will be described with reference to the flowcharts of FIGS. 3 and 5. FIG. In the first embodiment, the wearable device 102 is worn by the subject to be photographed, and the camera 101 is configured as a remote camera arranged outside. Subject movement information is sensed through the wearable device 102, and the imaging conditions of the camera 101 are determined using the subject movement information acquired by the wearable device 102 as auxiliary information.

First, the operation of the wearable device 102 will be described with reference to FIG. It should be noted that the following processing is realized by the control section 138 controlling each section of the apparatus according to the program stored in the ROM 151 .

In step S501, the user first powers on the wearable device 102 . Then, the wearable device 102 performs a standby operation for receiving sensing signals of the wearable device 102 from sensors (acceleration sensor 146, gyro sensor 147, depth sensor 148, etc.) in order to detect subject movement information.

In step S502, the communication unit 115 of the camera 101 acquires the sensing signal acquired in step S501 at regular time intervals. For example, acceleration information of the wearable device 102 is obtained as the sensing signal. In order to acquire acceleration information, the output from the acceleration sensor 146 is periodically acquired at predetermined time intervals. As described above, it is possible to obtain the acceleration information of the subject at the site where the wearable device 102 is worn. Instead of using the acceleration sensor 146, it is also possible to indirectly acquire the acceleration information of the subject at the wearable device 102 attachment site by using a sensor capable of detecting the motion state of another subject. As an example, by obtaining a change in distance from the camera 101 to the wearable device 102 by the depth sensor 148, it is possible to calculate the motion speed and acceleration information of the subject per unit time.

In step S503, the user first wears the wearable device 102 at a specific position, such as a wrist that does not interfere with movement. The wearable device 102 senses that it is worn by a subject using a tactile sensor (not shown) or the like. Also, the wearable device 102 can be attached freely as long as the movement of the subject is visible. In that case, as a method of specifying the mounting site of the wearable device 102, the mounted device may be detected from the image data acquired by the camera 101 side. In addition, a method of setting the part of the wearable device 102 to be attached to the subject in advance, and recording the acceleration and speed of the specific part within a predetermined time in advance, and moving based on the actual movement. A technique for specifying a subject part is known (Japanese Patent No. 6325581). For example, acceleration information is obtained from the acceleration sensor 146 of the wearable device 102 in step S502. Therefore, the wearing site of the wearable device 102 can be specified by comparing the acceleration change recorded per predetermined time for each wearing site in advance with the acquired acceleration information.

In step S504, the communication module A131 of the wearable device 102 transmits the acceleration information obtained in step S502 and the mounting site information of the wearable device 102 to the camera 101 as subject movement information.

The above is the processing of the wearable device 102 . Next, the operation of the camera 101 will be explained using the flowchart of FIG. The processing of the flowchart in FIG. 3 is realized by the control unit 112 controlling each unit of the camera 101 in accordance with the program stored in the ROM 113 in the camera 101 of the imaging system 100 .

In step S<b>301 , the communication unit 115 of the camera 101 receives subject movement information transmitted from the wearable device 102 .

In step S302, the user uses the camera 101 to start preparatory photographing such as composition adjustment. During this preparatory shooting period, the camera 101 continuously captures images and displays the recorded images on the display unit 127 . The user adjusts the composition while viewing the displayed preparatory photographed image. It should be noted that the processes of steps S303, S304, and S305, which will be described later, are performed during the preparatory photographing period.

In step S303, the control unit 112 of the camera 101 determines shooting conditions for a preparatory shot image to be shot in order to detect the motion vector of the subject in the composition. Although details will be described later, the wearable device 102 is worn using the amount of subject movement in the composition when preparatory photography is performed under the initial photography conditions and the subject movement information transmitted from the communication module A 131 of the wearable device 102. Sets the shutter speed that minimizes subject blur in the area where the subject is (subject's focus area).

In step S304, the display unit 127 displays the subject in the composition and the shooting setting conditions (shutter speed, ISO sensitivity, F number, etc.).

In step S305, the control unit 112 of the camera 101 determines whether remote release has been activated. Remote release is control for transmitting a timing signal for starting exposure with the camera 101 through the wearable device 102 connected to the camera by the user. Activation of the remote release can be generated by outputting the gyro sensor 147 or pressing a release button (not shown) of the wearable device 102 when the user wearing the wearable device 102 performs a predetermined gesture. Alternatively, it is determined whether or not the photographer (user) has directly pressed the shutter button (not shown) of the camera 101 . In that case, the user presses the shutter button of the camera 101 in time with the shutter timing while looking at the subject displayed on the display unit 127 . If the shutter button of the camera 101 has been pressed, the process proceeds to main exposure processing in step S306. Conversely, if it is not the shutter timing, it is possible to redo the shooting settings by returning to step S301.

In step S<b>306 , the control unit 112 of the camera 101 performs exposure processing with the shooting settings made in the processing of the above steps in order to shoot an image, and records the completed image in the ROM 113 .

In this way, during preparatory photography, the user repeatedly sets the exposure time for actual photography while confirming the motion blur notification image displayed on the display unit 127 until the desired motion blur is achieved, and presses the shutter button at a photo opportunity. Press.

When the user presses the shutter button of the camera 102 in step S<b>305 , the camera 101 performs main photographing and records the main photographed image in the ROM 113 .

Next, the processing of step S303, which is a feature of the present invention, will be described with reference to the flowchart of FIG.

In step S701 of FIG. 7, the control unit 112 of the camera 101 sets initial imaging conditions, and the camera 101 continuously captures preparatory imaging images. The initial shooting conditions here mainly indicate the frame rate and shutter speed. And the highest frame rate and the highest shutter speed within the range that does not affect the processing of calculating the evaluation value used for controlling the automatic functions performed by general cameras such as AE (auto exposure) and AF (auto focus) control Set the speed. Also, the optical system 121 is controlled in accordance with the shutter speed so that even when the shutter speed is set to a high speed, shooting can be performed under appropriate exposure conditions. For example, by controlling the aperture of the lens of the optical system 121 and the ISO sensitivity setting of the camera 101, control (exposure control) is performed so that shooting can be performed under appropriate exposure conditions. The camera 101 shoots consecutive images in time series under the initial shooting conditions set as described above. It is desirable that the captured images are captured under conditions in which there is almost no accumulated blurring of the moving subject and the amount of movement of the subject between the captured consecutive images is as small as possible. However, under such conditions, the ISO sensitivity tends to increase, and there are also adverse effects such as the acquisition of high-sensitivity image data with a lot of noise. On the other hand, since the amount of movement of the subject is small, there is an advantage that even a fast subject can easily be captured.

In step S702, the image processing unit 124 of the camera 101 calculates the motion vector of the subject from the time-series consecutive pre-shot images shot in step S701.

First, the motion vector calculated from the preparatory photographed image will be described with reference to FIG. FIG. 8 is a diagram showing the motion of the subject. FIG. 8A shows an example of photographing a scene of a dog 801 running to the left and a dog 802 standing still. A motion vector represents the amount of horizontal movement and the amount of vertical movement of a subject between pre-captured images. An example of this motion vector is shown in FIG. 8(b).

FIG. 8(b) is a diagram showing an example of motion vectors of the preparatory photographed image of FIG. 8(a). In the example of FIG. 8B, a running dog 801 is detected as a motion vector in the left direction, and a stationary dog 802 and a fence in the background are detected as motion vectors 0, so the motion vectors are illustrated. not

Next, a motion vector calculation method will be described in detail with reference to FIGS. 9 and 6. FIG. FIG. 9 is a flowchart showing motion vector calculation processing. The following flowchart is realized by the control unit 112 controlling each unit of the camera 101 according to the program stored in the ROM 113 . In the present invention, the block matching method will be described as an example of the motion vector calculation method, but the motion vector calculation method is not limited to this example, and may be, for example, a gradient method.

In step S901 of FIG. 9, two preparatory photographed images temporally adjacent to each other are input to the image processing unit 124 of the camera 101 . Then, the image processing unit 124 sets the preparatory photographed image of the Mth frame as the reference frame, and sets the preparatory photographed image of the M+1th frame as the reference frame.

In step S902, the image processing unit 124 arranges a reference block 602 of N×N pixels in a reference frame 601 as shown in FIG.

In step S903, the image processing unit 124, as shown in FIG. 605.

In step S904, the image processing unit 124 performs a correlation operation between the reference block 602 of the reference frame 601 and the reference block 606 of N×N pixels at different coordinates existing within the search range 605 of the reference frame 603, and obtains the correlation value Calculate A correlation value is calculated based on the sum of inter-frame absolute differences for the pixels of the reference block 602 and the reference block 606 . That is, the coordinate with the smallest inter-frame difference absolute value sum is the coordinate with the highest correlation value. Note that the method of calculating the correlation value is not limited to the method of calculating the sum of the inter-frame absolute difference values, and may be a method of calculating the correlation value based on the inter-frame difference sum of squares or the normal cross-correlation value, for example. In the example of FIG. 6, it is assumed that reference block 606 exhibits the highest correlation.

In step S905, the image processing unit 124 calculates a motion vector based on the reference block coordinates indicating the highest correlation value found in step S904. In the example of FIG. 6, a motion vector is obtained based on the coordinates 604 corresponding to the center coordinates of the reference block 602 of the reference frame 601 and the center coordinates of the reference block 606 in the search range 605 of the reference frame 603 . That is, the inter-coordinate distance and direction from the same coordinates 604 to the center coordinates of the reference block 606 are obtained as motion vectors.

In step S<b>906 , the image processing unit 124 determines whether motion vectors have been calculated for all pixels of the reference frame 601 . If the image processing unit 124 determines in step S906 that motion vectors for all pixels have not been calculated, the process returns to step S902. Then, in step S902, a reference block 602 of N×N pixels is arranged in the reference frame 601 described above, centering on pixels for which motion vectors have not been calculated. . That is, the image processing unit 124 repeats the processing from step S902 to step S905 while moving the reference block 602 in FIG. 6 to calculate motion vectors of all pixels of the reference frame 601. As a unit for calculating a motion vector, it may be calculated in units of pixels or in units obtained by dividing an image into a predetermined number of divisions. A motion vector is calculated by performing the above processing between preparatory captured images whose imaging times are close to each other.

Subsequently, in step S703, the image processing unit 124 uses the subject motion information obtained from the wearable device 102 and the subject motion vector obtained by the camera 101 to calculate a vector corresponding to the main part of the subject.

The image processing unit 124 uses the subject motion information obtained by the wearable device 102 to specify the vector corresponding to the main part of the subject for which the user wants to reduce the blurring of the subject, and then corrects the motion vector of the corresponding main part of the subject. process. A specific description will be given with reference to FIG.

First, the image processing unit 124 finds a candidate motion vector group of the subject in order to specify the vector corresponding to the main part of the subject. This will be described with reference to FIG. 10(a). FIG. 10A is a diagram showing motion vector groups (1011, 1012, 1013, 1014) of subjects that are candidates for main parts. The main part information is information corresponding to the wearing part of the wearable device 102 of the subject movement information transmitted in step S504. The correspondence between the main part of the subject and the motion vector of the subject is realized by the image processing unit 124 selecting the motion vector of the subject corresponding to the main part from the preparatory photographed images for obtaining the motion vector of the subject. As a method for detecting the main part from the preparatory photographed image, a method such as a general subject recognition technique may be used. For example, when the wearing part of the wearable device 102 is the head of a dog, the image processing unit 124 selects the head of the dog on which the wearable device 102 is worn, within the range for which the motion vector of the subject in the preparatory photographed image is obtained. To detect. Then, the image processing unit 124 selects the motion vector group (1011, 1012, 1013, 1014) of the subject existing within a predetermined distance from the detected dog head region, and further selects the motion vector group (1011, 1012, 1013, 1014) from among them, The motion vector (1011) of the subject is detected and used as the motion vector of the subject of the main part.

Subsequently, processing for correcting the motion vector of the subject of the corresponding main part will be described in detail using FIGS. 10(b) and 10(c). The correction of the motion vector of the subject of the main part means that the motion vector of the subject is corrected using the acceleration information of the wearing part transmitted from the wearable device 102 with a high output update rate, in contrast to the motion vector calculation processing with a low output update rate. By correcting , the update rate of the motion vector of the subject is pseudo-improved, and the motion vector of the subject corresponding to the change in motion of the subject is obtained.

FIG. 10(b) is a diagram showing acquisition timings of subject motion vectors and subject motion information.

The motion vector of the subject is obtained by using two or more frames of preparatory shot images that are continuous in time series, and the amount of movement between the preparatory shot images to be used as the motion vector of the subject. For example, the motion vector 1031 of the subject cannot be obtained until the camera 101 obtains at least two frames of preparatory shot images 1021 and 1022 . Further, the motion vector 1032 of the next subject cannot be calculated until the preparatory shot image 1023 is acquired. If the motion of the subject suddenly changes during the blank period 1041 from the motion vector 1031 of the subject to the calculation of the motion vector 1032 of the subject, the update rate of the motion vector of the subject is slow. Subject movement cannot be detected correctly. On the other hand, the acceleration information as subject movement information detected by the wearable device 102 does not depend on the preparatory photographed image, and the movement of the device can be directly detected, so generally high-speed detection is possible (1051). .

Since a pre-captured image that can be acquired by a digital camera, which is the general camera 101, is about 120 fps even at high speed, the motion vector of the subject has an update rate of 120 fps or less. On the other hand, the output update rate of the acceleration sensor 146 mounted on a smartphone, which is a general wearable device 102, is 10 to 100 times or more the output update rate of the motion vector of the subject.

Therefore, by correcting the motion vector of the subject by the image processing unit 124 using the acceleration information detected by the wearable device 102, it is possible to obtain a highly accurate motion vector of the subject even during a period when the motion vector of the subject is not updated. can. Further, since the motion vector of the subject depends on the preparatory photographed image, there are cases where the motion vector of the subject cannot be obtained from a low-contrast subject or an image in which accumulated blurring or out-of-focus occurs. That is, subject motion information may not be obtained by the camera 101 alone, and the vector update rate may be slow. Therefore, it is effective to correct and update the motion vector of the subject using sensor information from the wearable device 102 with a high update rate.

Subsequently, correction processing of motion vectors of main parts will be described with reference to FIG. 10(c). FIG. 10(c) shows a motion vector 1061 of the subject of the main parts, a motion vector 1062 of the subject corrected when the motion of the subject slows down before updating the motion vector of the subject, and a motion vector 1062 of the subject corrected when the motion of the subject speeds up. It represents the motion vector 1063 of the object.

Since the motion vector of the main part of the image has angles and magnitudes in multiple directions, it is converted into the magnitude of the vector using Equation (1). A motion vector generally used for an image has directionality on two-dimensional coordinates, so by applying Equation 1, it can be converted to a vector magnitude as a scalar.

A method of correcting the motion vector of the subject is possible by performing gain processing corresponding to changes in acceleration of the main parts up to the time when the motion vector of the subject is updated. Therefore, if the amount of change in acceleration calculated using the acceleration information of the main part detected by the wearable device 102 is α (1 when the acceleration does not change), the correction of the motion vector of the subject is represented by Equation 2. be able to.
Subject motion vector after correction=α×Subject motion vector Equation 2

When the motion vector of the main part of the subject is corrected using the above formula 1, it becomes 1062 when the acceleration change amount α is smaller than 1, and conversely, when the acceleration change amount α is larger than 1, it becomes 1063 before correction. converted from the motion vector of the subject. By correcting the motion vector of the subject in this manner, the blurring amount of the main part is obtained so as to minimize the time difference from the motion of the subject in real time.

Subsequently, in step S704, under the instruction of the control unit 101, the blur amount estimating unit (not shown) of the camera 100 moves the subject using the motion vector of the main part of the subject calculated in the processing of the above step and the shutter speed set by the user. Estimate the amount of motion blur that occurs in The subject motion blur amount is calculated by the following formula using the shooting frame rate of the pre-shot image for calculating the motion vector of the subject, the shutter speed set in the camera 101 by the user, and the motion vector of the subject.
Subject motion blur amount = Subject motion vector*
(Frame rate (fps)/Shutter speed (s)) Equation 3

The relationship between the motion vector of the subject and the amount of motion blur of the subject with respect to Equation 3 will be described with reference to FIG. 11 . FIG. 11 is a diagram showing the relationship between the subject motion vector and the subject motion blur amount. For example, since the motion vector of the subject is calculated using the frames before and after the preparatory shot image updated at a frame rate of 60 fps, the update frame rate of the motion vector 1101 of the subject is also 60 fps. On the other hand, the subject motion blur amount corresponds to the shutter speed set by the user because it is the amount of blur caused by the subject moving during exposure. For example, when the user sets the shutter speed to 1/120 seconds, the interval between the frames before and after the preparatory photographed images for obtaining the motion vector of the subject is 1/60 seconds. Therefore, if the subject motion vector 1101 is 10 pix, the subject motion blur amount 1102 is 5 pix, which is half the length.

Subsequently, in step S705, the camera 101 compares the subject motion blur amount calculated in step S704 with the allowable amount of motion, and changes the shutter speed of the preparation shot image to be shot next so that the amount of subject motion blur is equal to or less than the allowable amount of motion. to change the shooting conditions for the preparatory shot image. The allowable amount of motion is the amount of motion blur that can be kept to an inconspicuous extent as motion blur when photographing at a predetermined shutter speed. The size of the allowable amount of motion is determined by the size of an imaging device such as a CCD or CMOS sensor, the number of pixels, and the resolution of a display. For example, assume that the image pickup device is APS-C, the number of pixels is 200,000 pixels, and the allowable amount of movement is 5 pixels or less in a full HD (1920×1080 pixels) PC display. In order for the camera 101 to capture the preparatory shot image so that the amount of motion blur is equal to or less than the allowable amount of motion, the shutter speed is determined using Equations 4 and 5 below.
n=subject motion blur amount/allowable motion amount Equation 4

At this time, if n obtained by Equation 4 is greater than 1, it indicates that there is a high possibility that subject blurring will occur when shooting at the currently set shutter speed, and if it is less than 1, subject blurring will occur. This means that the shutter speed is less likely to Therefore, an appropriate shutter speed at which the occurrence of subject blurring is reduced is calculated using Equation 5 below.
Updated shutter speed (s) ≤ set shutter speed (s) * 1/n Expression 5

To explain using specific numerical values, the motion blur amount 1102 in FIG. 11 is 5 pix, and the allowable motion amount is also 5 pix. Therefore, n=1 from Equation 4, and it can be seen that the currently set shutter speed is less affected by subject blurring. Therefore, according to Equation 5, the shutter speed as the shooting condition for the preparation shot image (if not changed, the shooting condition for actual exposure) should be set to an exposure time faster than 1/120 (in this case, 1/120 s (does not change from ). Also, when there is a sufficient amount of light for photographing, the shutter speed may be set faster than 1/250 in consideration of the ISO sensitivity and aperture.

An example in which the shutter speed is updated as the shooting condition for the preparatory shot image has been described above. In order to increase the accuracy of detecting the motion vector of the subject, the frame rate for capturing the preparatory captured image may be increased and the update rate for calculating the motion vector of the subject may be increased. The conditions for increasing the update rate are as follows.

Update frame rate (fps)≧Set frame rate (fps)*n Expression 6
The above frame rate and shutter speed are important shooting conditions for motion detection. In addition, in order to capture an image with proper brightness, the aperture value and ISO sensitivity are changed in accordance with the change of the frame rate and shutter speed, and control is performed so that the exposure value does not change.

Regarding the detailed processing of step S303, the processing of determining the shooting conditions of the pre-shot image using the processing of steps S701 to S705 in FIG. Ta.

The processing of the imaging system 100 according to the first embodiment has been described above. Specifically, subject motion information is sensed through the wearable device 102 . Then, the motion vector of the subject is updated using the subject motion information acquired by the wearable device 102 as auxiliary information, and the photographing conditions of the camera 101 are determined. According to the present invention, in cooperation with the wearable device 102, it is possible to increase the detection accuracy of the motion vector of the subject calculated by the camera 101 and set the shutter speed at which subject blurring is reduced. This enables the photographer to set the shutter speed and adjust the exposure for photography without touching the camera so that the movement of the subject desired by the photographer is suppressed to a desired blur or less. According to the present invention, it is possible to expand the use scene of automatic photographing.

In the first embodiment, the method of calculating the amount of motion blur by converting the motion vector of the subject in accordance with the shutter speed set by the user has been described. It should be noted that the calculation of the amount of motion blur does not have to be converted in accordance with the shutter speed. For example, by comparing the motion vector of the subject with a preset threshold value and changing the shutter speed to a value faster than the current setting value when the threshold value is exceeded, the same processing can be easily achieved. It is possible.

In the first embodiment, the method of identifying the main parts of the subject and selecting the motion vector of the subject in the main parts has been described. Note that the motion vector of the subject with the fastest motion may be selected from the motion vectors of the subject obtained from the preparatory photographed images.

In the first embodiment, the method of identifying the main parts of the subject and selecting the motion vector of the subject in the main parts has been described. Note that if the camera 101 is equipped with an acceleration sensor similar to the acceleration sensor 146 that is installed in the wearable device 102, a motion vector of the subject that is different from the motion of the acceleration sensor installed in the camera 101 is selected. You can Since the acceleration sensor mounted on the camera 101 can obtain motion information of the main body that moves the camera 101, by selecting a motion vector different from the motion vector, motion information other than the main moving subject can be discarded.

In the first embodiment, the method of identifying the main parts of the subject and selecting the motion vector of the subject in the main parts has been described. Of the calculated motion vectors of the subject, it is also possible to select the motion vector of the subject in the center of the angle of view when shooting with the camera 101 or the vicinity of the autofocus target area in the image. good.

In the first embodiment, the method of identifying the main parts of the subject and selecting the motion vector of the subject in the main parts has been described. Note that when the wearable device 102 is shown in the preparatory photographed image, the position of the wearable device 102 may be detected directly from the image.

Further, before selecting the motion vector of the subject of the main part, the motion vector of the subject obtained from the preparatory photographed image may be subjected to selection processing. For example, in calculations such as template matching performed in the process of obtaining the motion vector of the subject, correlation value calculations are performed. At this time, a vector whose correlation value is lower than a predetermined threshold value is determined to be a subject motion vector with low reliability (reliability equal to or lower than the threshold). As a result, it is possible to extract only the motion vector of the subject with a correlation value greater than or equal to the threshold value (reliability greater than or equal to the threshold value) and higher accuracy.

(Second embodiment)
Next, a second embodiment (Example 2) of the present invention will be described in detail with reference to the drawings. In the second embodiment, by performing exposure control based on the subject motion blur amount during the main exposure process, it is possible to obtain an image with a reduced motion blur amount of the subject. Note that processing on the wearable device 102 side of the imaging system 100 according to the second embodiment is the same as that according to the first embodiment, and a description thereof will be omitted. The operation performed by the control unit 112 of the camera 101, which characterizes the second embodiment, will be described with reference to the flowchart of FIG. Note that the following processing is realized by the control unit 112 controlling each unit of the camera 101 as the imaging system 100 according to the program stored in the ROM 113 . Also, the wearable device 102 is realized by the controller 138 controlling each part of the device according to the program recorded in the ROM 151 . The same steps as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

First, the photographing operation of the camera 101 will be described with reference to FIG. The processing from step S301 to step S305 in FIG. 4 is the same as the processing from step S301 to step S305 in FIG. The second embodiment differs from the first embodiment in the main exposure process in step S401.

In step S<b>401 , the camera 101 performs exposure interruption processing based on the amount of motion blurring with respect to the part of interest of the subject being exposed, and performs photography while suppressing the occurrence of subject blurring.

Next, the control of the main exposure process in step S401 performed by the control unit 112 of the camera 101 based on the amount of motion blur for the part of interest of the subject being exposed will be described in detail with reference to the flowchart of FIG.

In step S306 of FIG. 13, the camera 101 starts the main exposure process as in step S306 of the first embodiment.

14 and 15, the configuration of the electronic front curtain shutter and the shooting operation when performing the main exposure process will be described in detail.

FIG. 14 is a front view showing the imaging element 122 and the mechanical rear curtain 1403 as seen from the lens side along the optical axis direction. It shows a state in which the reset scanning performed by the image sensor 122 after the start of shooting and the running of the mechanical rear curtain 1403 are in progress. An arrow 1401 indicates the reset scanning operation direction (the running direction of the electronic front curtain 1407 ) and the running direction of the mechanical rear curtain 1403 . A state in which a mechanical rear curtain 1403 configured by the mechanical shutter 118 in FIG. Further, a reset line 1408 is a reset scanning line (reset line) performed by the image sensor 122, and corresponds to an end portion of the electronic front curtain 1407 as an operation of resetting the charge accumulation amount of the pixel to zero. A region 1406 formed by a slit between the reset line 1408 and the end portion 1405 of the mechanical rear curtain 1403 is controlled to move in the direction of the arrow 1401 as the electronic front curtain 1407 and the mechanical rear curtain 1403 run. do. The time from the passage of the reset line 1408, that is, the pixels being sequentially reset line by line in the direction of the arrow 1401, to the mechanical rear curtain 1403 blocking the light, is the charge accumulation time due to exposure of the pixels. In this manner, the reset line 1408 runs in the direction of the arrow 1401 to start charge accumulation for each line, so the start timing of charge accumulation differs for each line of the image sensor 122 as well.

The timing of charge accumulation will be described in detail with reference to FIG. FIG. 15(a) is a conceptual diagram of charge reset and read start, and FIG. 15(b) is a timing explanatory diagram of charge reset and read for each line. Lines 1501 to 1510 in FIG. 15(b) represent the timing of charge reset processing for each line, and after the line 1501 at the end reads the charge first, the operation of resetting the charge is performed. Conversely, the end line 1510 is the last line to be reset. In this way, the reset timing is controlled for each line. Since the reset timing is different for each line, the control unit 112 controls the charge storage time to be the same for each line, and the charge readout time for lines 1511 to 1520 to be the same exposure time for each line.

In step S1301 in FIG. 13, the control unit 112 corrects the subject motion vector calculated by the image processing unit 124 of the camera 101 immediately before the main exposure detected in step S303 in FIG. Correction of the motion vector of the subject is performed based on the output of the acceleration sensor 146 included in the subject motion information from the wearable device 102 during exposure. When the main exposure starts, unless the camera 101 has a plurality of image pickup elements 122, the preparatory photographed image cannot be photographed. Therefore, the motion vector of the subject cannot be updated during the main exposure. Therefore, here, the motion vector of the subject in the region of interest calculated until just before the main exposure is estimated. For the estimation, the acceleration information of the acceleration sensor 146 is used to correct the motion vector of the main part of the subject, which corresponds to the processing of step S703 in the first embodiment, and then the processing of converting it into the motion blur amount is performed. With the above processing, it is possible to estimate the motion blur amount even during the main exposure of the camera 101 .

In step S1302, the control unit 112 of the camera 101 determines whether the amount of motion blur estimated in step S1301 exceeds the allowable amount of motion. If the allowable motion amount is exceeded, the process proceeds to step S1303. Conversely, if the allowable blur amount is not exceeded, the process proceeds to step S1304.

In step S1304, the control unit 112 of the camera 101 determines whether the shooting conditions obtained in step S303 of FIG. 4 are satisfied. The shooting conditions mainly determine whether or not they affect accumulated blurring during exposure. This can be determined by whether or not the exposure time corresponding to the shutter speed set before the main exposure is satisfied. If the shooting conditions are satisfied, the process advances to step S1305 to end the exposure.

In step S1303, the control unit 112 of the camera 101 determines that subject blurring will occur in the image if the image sensor 122 is exposed to more light, and closes the mechanical shutter 118 earlier than the exposure time set in the shooting conditions. This blocks the light from entering the image sensor 122 . Subsequently, the process advances to step S1305 to end the exposure. A method of performing exposure termination control in step S1303 will be described in detail with reference to FIG.

FIG. 16 is a timing chart of processing in which the control unit 112 of the camera 101 controls the mechanical rear curtain and interrupts exposure when the subject motion blur amount exceeds the allowable motion amount (allowable threshold value) during the main exposure processing. is shown. By controlling the mechanical rear curtain and controlling the exposure time when subject blurring is likely to occur in the image during main exposure, the blurring of the subject can be reduced.

Lines 1501 to 1510 in FIG. 16 (under the same imaging conditions as in FIG. 15B) are reset lines. After the reset processing of the line 1501 is started, external light reaches the image sensor 122 and charge accumulation is started. If no accumulation blurring occurs in the image due to the movement of the main body of the camera 101 during exposure, charges are accumulated until the next reset processing lines 1511 to 1520 (under the same photographing conditions as in FIG. 15B). This time, a method of controlling charge accumulation when the subject motion blur amount exceeds the allowable motion amount during exposure will be described.

The control unit 112 executes reset processing of the head reset line 1501 in FIG. 16 and starts exposure. At this time, when it is detected that the subject motion blur amount exceeds the permissible threshold value at timing 1611 in FIG. Close the mechanical shutter 118 so that it does not reach. Due to the action of the mechanical shutter 118, the actual exposure time is between 1601 and 1610 in FIG. In other words, no charge is accumulated in the imaging device 122 during the period from 1601 to 1610 to lines 1511 to 1520 because no exposure is performed. Therefore, it is possible to prevent subject blur caused by the movement of the subject. Although the accumulation of electric charges is stopped by controlling the mechanical shutter 118, similar processing can be performed by causing the pulse generating section 126 to generate a reset pulse to interrupt the accumulation of electric charges for shutters not equipped with the mechanical shutter 118. .

Subsequently, in step S1306 of FIG. 13, the control unit 112 determines whether or not the set exposure time has come. If it is determined that the exposure time is still shorter than the set exposure time, the process proceeds to step S1307.

In step S1307, the control unit 112 multiplies the acquired image data by the difference exposure time as a digital gain so that the brightness of the image data becomes the brightness of the original exposure time for the insufficient exposure time when the image data is acquired. process. Note that the digital gain obtained from the difference in exposure time can be obtained from the following equation.

Digital gain = Exposure time set at the start of imaging / (Exposure time set at the start of imaging -
Time from the start of exposure until the mechanical shutter is released) Equation 7
By multiplying the image data by the digital gain calculated by Equation 7, the brightness of the image data is corrected to the brightness corresponding to the expected exposure time. When performing gain correction more strictly, the digital gain may be calculated for each horizontal line of the image data, and the image data may be multiplied by the digital gain for each horizontal line.

On the other hand, when the set exposure time is reached, the control unit 112 reads out the charge for each line, and performs charge reset processing from the line for which the readout is completed, thereby ending the exposure of the camera 101. Get image data.

Next, the correction processing for the insufficient exposure amount in step S1307 performed by the image processing unit 124 will be described in detail with reference to FIG. As an example, an image signal 1201 in FIG. 12 indicates an image signal when an image is captured with the target exposure amount, and an image signal 1202 indicates an image signal that does not reach the target exposure amount due to insufficient exposure time due to interrupted exposure. In FIG. 12, the horizontal axis represents subject brightness, and the vertical axis represents the image signal level at the subject brightness on the horizontal axis. For example, an image signal 1202 is an image signal obtained when an image is captured with an exposure time half that of 1201, which is the target exposure amount. The amount of exposure when the camera 101 takes an image is generally determined by the F number, ISO sensitivity and shutter speed (exposure time). Therefore, by interrupting the exposure halfway, the exposure amount becomes insufficient compared to the target exposure amount by the amount of the shortened exposure time. If the exposure time is halved, the exposure amount is halved. Therefore, by applying a double digital gain, the image signal 1202 can be adjusted to the same exposure amount as the image signal 1201, and the original target exposure amount can be obtained. By correcting the above processing for each line of the image pickup device 122, even when exposure is interrupted, it is possible to photograph the subject with the brightness of the original target exposure amount.

The method in which the imaging system 100 of the second embodiment controls exposure based on the degree of blurring of the subject during exposure has been described above using the flowchart of FIG. 13 . With these processes, even in situations where it is difficult to change the shutter speed during exposure, it is possible to obtain an image with reduced subject blur by controlling the exposure time.

(Third embodiment)
Next, a third embodiment (Example 3) of the present invention will be described in detail with reference to the drawings. In the third embodiment, a plurality of subjects to be photographed wear the wearable device 102 and the camera 101 is set outside. FIG. 17 shows an example of a scene in which wearable devices 102 are worn by multiple subjects. FIG. 17 shows an example in which six subjects A to F wear wearable devices 102 1701 to 1708 . Subject A wears 1701, subject B wears 1702, 1703, and 1704, subject C wears 1705, subject D wears 1706, subject E wears 1707, and subject F wears 1708 wearable devices 102 in this example. Subject motion information of each subject is sensed through the wearable device 102, and the photographing conditions of the camera 101 are determined using the acquired subject motion information as auxiliary information. Note that processing on the wearable device 102 side of the imaging system 100 according to the third embodiment is the same as that according to the first embodiment, and a description thereof will be omitted. The operation performed by the control unit 112 of the camera 101, which is the feature of the third embodiment, will be described with reference to the flowchart of FIG. Note that the following processing is realized by the control unit 112 controlling each unit of the camera 101 as the imaging system 100 according to the program stored in the ROM 113 . Also, the wearable device 102 is realized by the controller 138 controlling each part of the device according to the program recorded in the ROM 151 . The same steps as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

The photographing operation of the camera 101 is the same as the processing of steps S301 to S306 in FIG. 3 of the first embodiment, so the description is omitted. The processing of step S303, which is a feature of the third embodiment, will be described with reference to the flowchart of FIG. 18, but the description of the portions that perform the same processing as the processing of steps S701 to S705 of FIG. 7 of the first embodiment will be omitted. .

In step S701 of FIG. 18, the control unit 112 of the camera 101 reads out the initial shooting conditions from the ROM 113 of the camera 101, and successively takes preparatory shot images according to the conditions.

In step S702, the image processing unit 124 of the camera 101 calculates the motion vector of the subject from the time-series consecutive pre-shot images shot in step S701.

In step S1801, the image processing unit 124 of the camera 101 detects a subject from the time-series continuous pre-shot images shot in step S701. As a method for detecting a subject, a method such as a general subject detection technique is used. Examples include face/facial organ detection and head detection. Face/facial organ detection and head detection are methods of detecting areas in which a person's face, organs, and head exist from a photographed image by a method based on pattern recognition or machine learning.

In step S1802, the image processing unit 124 determines whether the number of subjects detected in step S1801 is one or multiple. If one person is determined, the process advances to step S1804, and if multiple people are determined, the process advances to step S1803.

In step S<b>1803 , since multiple subjects were detected in step S<b>1802 , the image processing unit 124 of the camera 101 detects the main subject among the persons wearing the wearable device 102 . As a main subject detection method, a general main subject detection method may be used. For example, in the preparatory photographed image, the person whose subject occupies the largest area in the angle of view or the person closest to the center of the preparatory photographed image is detected as the main subject. Alternatively, a configuration may be adopted in which a person registered in advance by the user in the ROM 113 of the camera 101 as the main subject is detected as the main subject.

In step S1804, the control unit 112 of the camera 101 determines whether the number of wearable devices 102 worn by the person detected as the main subject in step S1803 is one or plural. If it is determined to be one, the process advances to step S1806, and if it is determined to be plural, the process advances to step S1805. As a method of determining the number of wearable devices 102 , the user may register in advance the wearable devices 102 worn by each subject in the ROM 113 of the camera 101 . In addition, as described in step S503 of FIG. 5 in the first embodiment, the image processing unit 124 of the camera 101 specifies on which part of each subject detected in step S1801 the wearable device 102 is attached. Therefore, a method of determining the number of wearable devices 102 may be used. Also, a method of determining the number by communication between the communication unit 115 of the camera 101 and the communication module 131 of the wearable device 102 is possible.

In step S1805, it is determined in step S1804 that there are a plurality of wearable devices 102 worn by the person detected as the main subject in steps S1802 and S1803. Detect device 102 . A method for detecting the wearable device 102 will be described in detail with reference to the flowchart of FIG. 19 .

In step S1901, the control unit 112 of the camera 101 determines whether the priority of the wearable device 102 has been set in advance by the user. If it is determined that the priority is set, the process advances to step S1904, and if it is determined that the priority is not set, the process advances to step S1902.

As an example in which the priority is set in advance by the user, FIG. 20 shows an example of a priority setting screen of the wearable device 102 that can be set by the user on the display unit 127 of the camera 101 . The display unit 127 may be configured not only as a display unit but also as a touch panel that allows the user to input setting values. In FIG. 20 , 2000 is an example of a setting screen for setting the priority of the wearable device 102 . A subject ID 2001 indicates subjects A to F in the image diagram of FIG. 2002 is the wearable device 102 ID, which indicates the wearable devices 102 of 1701 to 1708 in the image diagram of FIG. 2003 indicates the priority of the wearable device 102 . In this setting screen example, the user sets the wearable device 102 of subject B 1704 to have the highest priority.

In step S1902, the control unit 112 of the camera 101 detects the wearable device 102 positioned within a predetermined area of the preparatory photographed image for a certain period of time. For example, when the angle of view of the preparatory photographed image is set to 100% as the predetermined area, the wearable that exists within a rectangular range of 90% centered on the central coordinates of the preparatory photographed image for a period of time equal to or longer than an arbitrarily set threshold. Detect device 102 .

In step S1903, the control unit 112 of the camera 101 calculates the priority in the priority calculation unit (not shown) of the wearable device 102 detected in step S1902. In the calculation of the priority, items for determining the priority are set in advance by the user, and the priority is calculated according to the setting contents. An example of the setting screen is shown in FIG. A setting screen 2100 is used to set items for determining the priority of the wearable device 102 . Item 2101 is the content of items for determining priority. There are items such as the area of the attachment site within the corner, the order of the distance between the center of the angle of view and the wearable device 102, the order of the distance between the center of the face and the wearable device 102, and the degree of face detection reliability. “Acceleration speed order” is an item to be set when it is desired to give a higher priority to the wearable device 102 having a higher acceleration speed order. “Wearing part” is an item for setting which part of the wearable device 102 to which the wearable device 102 is attached should have a higher priority among the parts of the head, body, hands, and feet. “Area of face within angle of view” is an item to be set when a wearable device 102 worn by a person having a large area ratio of a detected face within the angle of view is to be given higher priority. . “Area of attachment site within angle of view” is an item to be set when the wearable device 102 having a large area ratio of the head, body, hands, and feet within the angle of view is to be given higher priority. The “order of distance between the center of the angle of view and the wearable device 102” is set when the wearable device 102 having a short distance between the center coordinate of the angle of view and the center coordinate of each worn wearable device 102 is to be given higher priority. item. “Order of distance between face center and wearable device 102” is an item to be set when it is desired to give higher priority to wearable devices 102 having a short distance between face center coordinates and the center coordinates of each worn wearable device 102. . “Face detection reliability” is an item to be set when a wearable device 102 worn by a person whose face detection reliability is high is to be given a high priority. 2102 is an item used as a criterion for determining the priority of the main wearable device 102, and the item selected by the user for the item content of 2101 is checked.

A method of calculating the priority will be described with reference to FIG. For example, when items for determining priority are set as described with reference to FIG. 21, after calculating the score for each of the set items, the total value is calculated by adding each score. The wearable devices 102 are prioritized in descending order of total score, and the priority of each wearable device 102 is determined in the same manner as in FIG. 20 described in step S1901.

A method for calculating the score of the total value will be described with reference to FIG.

In FIG. 22(a), 2201 is a subject ID, which indicates subject A to subject F in the image diagram of FIG. 2202 is the wearable device 102 ID, which indicates the wearable device 102 of 1701 to 1708 in the image diagram of FIG. 2203 is a score value of "acceleration speed order", and the wearable device 102 with a higher speed rank has a higher score value. 2204 is a "wearing part", and since the hand is selected in FIG. 21, the score value of the wearable device 102 worn on the hand is high. 2205 is “order of distance between the center of the angle of view and the wearable device 102”, and the closer the distance between the center of the angle of view and the wearable device 102, the higher the score value. Reference numeral 2206 denotes “order of face detection reliability”, and the higher the reliability of face detection of each subject, the higher the score value. 2207 is the total score value, which is the sum of the score values calculated in 2203-2206. For example, consider the case where the main subject is detected as subject B in step S1803. A subject B wears three wearable devices 102 1702 , 1703 , and 1704 . Then, when the "wearing part" is set to the hand as an item for determining the priority, the total score value of the wearable device 102 of 1703 is the largest, and the priority is the highest. FIG. 22(b) shows the result of the priority when the priority is assigned based on the calculated total score value. In FIG. 22(b), 2208 is a subject ID, which indicates subject A to subject F in the image diagram of FIG. 2209 is the wearable device 102 ID, which indicates the wearable device 102 of 1701 to 1708 in the image diagram of FIG. 2210 indicates the priority of the wearable device 102 .

If subject B wears the wearable device 102 on both his right hand and his left hand, the score value of the "wearing site" will be the same 100, so the higher the score value of the other items, the higher the priority. get higher

In step S1904, the control unit 112 of the camera 101 tentatively determines the wearable device 102 with the highest priority as the main wearable device 102 (main sensor device) according to the priority calculated in step S1901 or step S1903. do.

In step S1905, the control unit 112 of the camera 101 determines whether the wearable device 102 tentatively determined in step S1904 is positioned within a predetermined region in the preparatory photographed image for a certain period of time. For example, when the angle of view of the preparatory photographed image is set to 100% as the predetermined area, if it exists within a rectangular range of 90% centered on the central coordinates of the preparatory photographed image for a period of time equal to or longer than an arbitrarily set threshold value. , the process proceeds to step S1907. If not, the process advances to step S1906.

In step S<b>1906 , the control unit 112 of the camera 101 tentatively determines the wearable device 102 with the next highest priority to the wearable device 102 tentatively determined in step S<b>1904 as the main wearable device 102 . After that, the process advances to step S1905 and repeats this determination until the process advances to step S1907 where the main wearable device 102 is determined. If, after repeating this determination, it is determined that none of the wearable devices 102 with priority determined in step S1901 or step S1903 is positioned within the predetermined region of the preparatory captured image for a certain period of time. , the camera 101 determines the wearable device 102 preset as the default as the main wearable device 102 in the next step S1907.

In step S<b>1907 , the control unit 112 of the camera 101 detects the wearable device 102 provisionally determined in step S<b>1904 or S<b>1906 as the main wearable device 102 .

Subsequently, in step S1806 of FIG. 18, the control unit 112 of the camera 101 determines the main wearable device 102 based on the detection result of the flow of FIG. 19 in order to acquire the information of the wearable device 102 in the next step. .

In step S703, similarly to step S703 of the first embodiment, the camera 101 uses the subject motion information obtained from the main wearable device 102 determined in step S1806 and the subject motion vector obtained by the camera 101 to determine the main parts of the subject. Calculate the vector corresponding to .

In step S704, as in step S704 of the first embodiment, under the instruction of the control unit 101, the blur amount estimation unit (not shown) of the camera 100 calculates the motion vector of the main part of the subject calculated in the processing of the above step, Estimates the amount of motion blur that occurs in the subject at the shutter speed set by the user.

In step S705, as in step S705 of the first embodiment, the camera 101 compares the subject motion blur amount calculated in step S704 with the allowable motion amount, and performs next shooting so that the subject motion blur amount is equal to or less than the allowable motion amount. The shutter speed of the preparatory photographed image is changed, and the photographing conditions of the preparatory photographed image are changed.

The processing of the imaging system 100 according to the third embodiment has been described above. Specifically, even in a scene with a plurality of subjects, the main wearable device 102 is determined from the plurality of worn wearable devices 102, the subject movement information is sensed through the main wearable device 102, and the wearable device 102 Using the acquired object motion information as auxiliary information, it is possible to update the motion vector of the object, determine the shooting conditions of the camera 101, and acquire an image with reduced object blur.

The items for determining the priority of the wearable device 102 are "acceleration speed order", "setting of wearing parts (head/body/hands/legs)", "area of face within angle of view", and "area of face within angle of view". area of the attachment site in the face”, “order of the distance between the center of the angle of view and the wearable device 102”, “order of the distance between the center of the face and the wearable device 102”, and “face detection reliability” are limited to these items. isn't it. For example, the shorter the "distance between the camera and the subject" or the "distance between the camera and the wearable device 102", the higher the score calculated and the higher the priority.

[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

REFERENCE SIGNS LIST 100 imaging system 101 camera 121 optical system 122 imaging device 123 A/D converter 124 image processor 125 recorder 126 pulse generator 127 display 118 mechanical shutter 119 gyro sensor 111 vertical drive modulator 112 controller 113 ROM
114 RAMs
115 Communication Department 102 Wearable Device 132 Power Module 131 Communication Module 133 Connector 135 Incamera 135 Out camera 136 Light 137 System Memory 138 Control Department 139 Speaker 141 Touch Screen Display 142 LCD Display 143 Touch Screen 144 Switch 145 illuminance sensor 147 Gyro Sensor 148 Depth Sensor 151 ROM
152 RAMs
146 acceleration sensor

Claims (15)

imaging means for imaging a subject;
subject movement detection means for detecting movement of the subject using the image captured by the imaging means;
an imaging device comprising: a receiving means for receiving sensing results transmitted from an external device; and an exposure control means for controlling exposure of the imaging means; a sensor means for acquiring subject information;
an imaging system comprising a sensor device attached to a subject to be photographed, which is different from the imaging means and which transmits a sensing result of the sensor means to the imaging device,
The imaging system, wherein the imaging device receives a sensing result of the sensor means, and performs exposure control of the imaging means using the sensing result of the sensor means and the result of the subject movement detection means. 2. The imaging system according to claim 1, wherein the sensing result of said sensor means is output as the sensing result of information relating to the movement of the subject wearing the sensor. The sensor means senses information related to the movement of the subject wearing the sensor device based on any of the amount of movement of the subject at the site where the sensor device is worn, the amount of change in the movement of the subject, and the change in the position of the subject. 3. The imaging system according to claim 1 or 2. 4. The imaging system according to any one of claims 1 to 3, wherein said sensor means further acquires information on a part where said sensor device is attached. the subject motion detection means calculates a motion vector of the subject using the image captured by the imaging means;
5. The imaging system according to claim 1, wherein said exposure control means controls exposure of said imaging device using said motion vector. the subject motion detection means selects the motion vector using the sensing result of the sensor means;
6. The imaging system according to claim 5, wherein said exposure control means controls exposure of said imaging device using said selected motion vector. a calculating means for calculating reliability of the accuracy of the motion vector;
7. The imaging system according to claim 5, wherein said exposure control means performs exposure control of said imaging device using a motion vector whose reliability is equal to or greater than a threshold. a correction means for correcting the motion vector based on the sensing result of the sensor means;
8. The imaging system according to claim 5, wherein said exposure control means controls exposure of said imaging device using said motion vector corrected by said correction means. Blur amount estimating means for estimating the blur amount of an image when photographed based on the calculation of the motion vector of the subject motion detecting means and the exposure time set by the exposure control means,
9. The imaging system according to any one of claims 5 to 8, wherein said exposure control means controls exposure of said imaging device based on the result of said blur amount estimation means. 10. The imaging system according to any one of claims 1 to 9, wherein the update of the output of said sensor means is faster than the update of the output of said subject motion detection means. 11. The imaging system according to any one of claims 1 to 10, wherein said exposure control means controls the amount of charge accumulation after said imaging means starts exposure. The imaging device has a determining means for determining a main sensor device from among the plurality of sensor devices when there are a plurality of the sensor devices,
12. The exposure control means controls the exposure of the imaging device using the sensing result of the main sensor device and the result of the subject movement detection means. The imaging system according to . The determination means comprises priority calculation means for calculating a priority indicating that the sensor device is a primary sensor device,
The priority calculation means determines that the acceleration of the sensor device is large, the distance from the center of the image is short, the distance from the camera is short, the area of the subject wearing the sensor device is large, and the subject is wearing the sensor device. 13. The imaging system according to claim 12, wherein the larger the area of the part, the higher the priority calculated. an imaging step of imaging a subject;
a subject movement detection step of detecting movement of the subject using the image captured in the image capturing step;
an imaging device control step comprising a receiving step of receiving a sensing result transmitted from an external device and an exposure control step of controlling exposure of the imaging means;
a sensing step for acquiring subject information;
A control method for an imaging system comprising a step of controlling a sensor device attached to a subject to be photographed, which is different from the imaging device, and a transmission step of transmitting the sensing result of the sensor means to the imaging device,
The imaging system, wherein the imaging device receives a sensing result of the sensor means, and executes an exposure control step of the imaging means using the sensing result of the sensing step and the result of the subject movement detection step. control method. A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 1 to 13.

Images