WO2024048069A1

WO2024048069A1 - Information processing device, information processing method, and program

Info

Publication number: WO2024048069A1
Application number: PCT/JP2023/024971
Authority: WO
Inventors: 壮平岡田
Original assignee: ソニーグループ株式会社
Priority date: 2022-08-30
Filing date: 2023-07-05
Publication date: 2024-03-07

Abstract

This information processing device comprises an adjustment unit for adjusting shutter speed and a value related to blur correction in an imaging device that comprises an electronic blur correction function of correcting a blur in image data, and a shutter speed control function.

Description

Information processing device, information processing method and program

The present technology relates to an information processing device, an information processing method, and a program.

In recent years, cameras have been equipped with a function to correct blurring that occurs in captured images due to hand movements of the user holding the camera or shaking of various devices equipped with the camera. This blur correction function is useful because images with less blur can be obtained even when the imaging device moves. Methods of blur correction include mechanical, optical, optical systems that can capture a wider angle of view than the output, and electronic methods that correct by cutting out the image sensor output according to the amount of blur. In recent years, imaging devices with a horizontal maintenance function have been realized using information from gyro sensors and acceleration sensors.

There are various methods for image stabilization. For example, a first image stabilization unit performs image stabilization processing on the image data, and a second image stabilization unit uses the image stabilization process output from the first image stabilization unit. A technology that realizes highly accurate image stabilization by performing control processing to remove the influence of the image stabilization process by the first image stabilization unit based on auxiliary information regarding the content of the process, and then executing the image stabilization process again. It has been proposed (Patent Document 1).

Japanese Patent Application Publication No. 2015-216510

As shown in Patent Document 1, various methods have been proposed for the blur correction function to further improve accuracy. In addition, the image corrected by the electronic blur correction function will be an image with motion blur when the imaging device is moving, and will be an image without motion blur when the imaging device is stationary, resulting in an output result. There is also a problem that the larger the amount of movement of the imaging device, the more unnatural the image becomes.

The present technology has been developed in view of these points, and aims to provide an information processing device, an information processing method, and a program that can suppress the sense of discomfort that occurs in captured images during electronic blur correction.

In order to solve the above-mentioned problems, a first technique is to use one of a value related to blur correction and a shutter speed in an imaging device that is equipped with an electronic blur correction function that corrects blur in image data and a shutter speed control function. This is an information processing device including an adjustment section that adjusts.

The second technology is an information processing method for adjusting either a value related to blur correction or a shutter speed in an imaging device equipped with an electronic blur correction function for correcting blur in image data and a shutter speed control function. be.

Furthermore, a third technology provides an information processing method for adjusting either a value related to blur correction or a shutter speed in an imaging device equipped with an electronic blur correction function for correcting blur in image data and a shutter speed control function. It is a program that is executed by a computer.

FIG. 2 is a diagram showing a first example for explaining the problem of the present technology. It is a figure which shows the 2nd example for demonstrating the problem of this technique. It is a figure which shows the 3rd example for demonstrating the problem of this technique. It is a figure which shows the 4th example for demonstrating the problem of this technique. FIG. 1 is a block diagram showing the configurations of an imaging device 10 and an information processing device 300 in a first embodiment. 1 is an external view showing the configuration of a lens head 100. FIG. FIG. 3 is an explanatory diagram of a blur correction method. 3 is a flowchart showing processing of the information processing device 300. It is a graph showing adjustment data in a 1st embodiment. FIG. 3 is an explanatory diagram when an image is captured by an imaging device 10 that performs processing by an information processing device 300. FIG. FIG. 2 is a block diagram showing the configurations of an imaging device 10 and an information processing device 300 in a second embodiment. It is a graph which shows the adjustment data in 2nd Embodiment. FIG. 3 is a block diagram showing the configurations of an imaging device 10 and an information processing device 300 in a third embodiment. It is a graph which shows the adjustment data in 3rd Embodiment. It is a block diagram showing the composition of imaging device 10 and information processing device 300 in a modification. It is a graph which shows the modification of the data for correction.

Embodiments of the present technology will be described below with reference to the drawings. Note that the explanation will be given in the following order.
<1. First embodiment>
[1-1. Issues solved by this technology]
[1-2. Configuration of imaging device 10 and information processing device 300]
[1-3. Processing by information processing device 300]
<2. Second embodiment>
[2-1. Configuration of imaging device 10 and information processing device 300 and processing by information processing device 300]
<3. Third embodiment>
[3-1. Configuration of imaging device 10 and information processing device 300 and processing by information processing device 300]
<4. Modified example>

<1. First embodiment>
[1-1. Issues solved by this technology]
First, we will explain the problems that this technology solves. FIG. 1 is a first example for explaining the problem of imaging a stationary subject P with an imaging device CAM without functioning a blur correction function (or without a blur correction function).

First, as shown in FIG. 1A, in a state where the subject P is located on the right side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces), imaging is performed without moving the imaging device CAM. Then, the subject P is located on the right side in the captured image, and no motion blur occurs. Motion blur refers to the movement of the subject P or the imaging device CAM, which causes the subject P in the captured image to blur in a flowing manner, or to cause an afterimage.

Furthermore, as shown in FIG. 1B, when imaging is performed while the imaging device CAM is panning clockwise from the state of FIG. 1A, the blur caused by the movement of the imaging device CAM is not corrected and compared to the state of FIG. 1A. The subject P is located on the left side in the captured image, and the composition has changed. Furthermore, even if the subject P is stationary, motion blur of the subject P occurs in the captured image due to the movement of the imaging device CAM.

Furthermore, as shown in FIG. 1C, when the imaging device CAM is panned clockwise by 90 degrees, the subject P will be located on the left side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces). When imaging is performed when the imaging device CAM is panned 90 degrees clockwise, no motion blur of the subject P occurs in the captured image. However, since the blur caused by the movement of the imaging device CAM is not corrected, the subject P is located on the left side in the captured image, and the composition has changed from the captured image in FIG. 1A.

In this case, if the image capture device CAM does not function (or does not have the image stabilization function), the composition of the captured image will change if the image capture device CAM moves, and furthermore, motion blur will occur in the subject P due to the movement of the image capture device CAM. Therefore, there is no sense of discomfort in the captured image.

FIG. 2 is a second example for explaining the problem of imaging a stationary subject P with the imaging device CAM using blur correction. Note that the imaging conditions such as the positions of the imaging device CAM and the subject P are the same as in the first example.

First, as shown in FIG. 2A, in a state where the subject P is located on the right side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces), imaging is performed without moving the imaging device CAM. Then, the subject P is located on the right side in the captured image.

Furthermore, as shown in FIG. 2B, when imaging is performed while the imaging device CAM is panning clockwise from the state of FIG. 2A, motion blur of the subject P occurs in the captured image due to the movement of the imaging device CAM. In addition, changes in the composition are suppressed by performing blur correction to cancel the movement of the imaging device CAM, and the subject P is located on the right side in the captured image, as in FIG. 2A. The length of the arrow b in FIG. 2B indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image due to blur correction.

Further, as shown in FIG. 2C, when the imaging device CAM is panned clockwise by 90 degrees, the subject P will be located on the left side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces). If you capture an image after panning the image capturing device CAM clockwise to 90 degrees, shake correction is applied to cancel the movement of the image capturing device CAM, suppressing changes in the composition, and subject P in the captured image. is located on the right side as in FIG. 2A. The length of the arrow c in FIG. 2C indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image due to blur correction.

When performing blur correction in this way, although the change in the composition of the captured image due to the movement of the imaging device CAM is suppressed by the blur correction, the motion blur of the subject P due to the movement of the imaging device CAM remains, so the captured image is blurred. This will cause a sense of discomfort.

FIG. 3 is a third example for explaining the problem in which blur correction is performed to eliminate the discomfort caused by the blur correction shown in FIG. 2, and the shutter is released at a faster shutter speed to capture an image. Note that the imaging conditions such as the positions of the imaging device 10 and the subject P are the same as in the first and second examples.

First, as shown in FIG. 3A, in a state where the subject P is located on the right side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces), imaging is performed without moving the imaging device CAM. Then, in the captured image, the subject P is located on the right side, and no motion blur occurs.

Furthermore, as shown in FIG. 3B, if an image is captured while the imaging device CAM is panning clockwise from the state shown in FIG. The motion blur of the subject P that occurs due to this can be suppressed. Furthermore, shake correction is performed to cancel the movement of the imaging device CAM, thereby suppressing changes in the composition, and the subject P is located on the right side in the captured image, as in FIG. 3A. The length of the arrow b in FIG. 3B indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image due to blur correction.

Furthermore, as shown in FIG. 3C, when the imaging device CAM is panned clockwise by 90 degrees, the subject P will be located on the left side of the imaging device CAM in the imaging direction of the imaging device CAM (the direction in which the lens faces). If you capture an image after panning the image capturing device CAM clockwise to 90 degrees, shake correction is applied to cancel the movement of the image capturing device CAM, suppressing changes in the composition, and subject P in the captured image. is located on the right side as in FIG. 3A. The length of the arrow c in FIG. 3C indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image due to blur correction.

By releasing the shutter at such a fast shutter speed, it is possible to suppress motion blur, and furthermore, it is possible to suppress changes in the composition through blur correction, and it is possible to eliminate the sense of discomfort in the captured image. However, there is a problem in that because the exposure time becomes shorter when the shutter is pressed, the captured image becomes darker than when the shutter is not pressed.

Furthermore, in order to solve the problem that the captured image becomes dark when the shutter is released, the brightness of the captured image can be corrected by adjusting the ND (Neutral Density), iris, gain, etc. However, since these image processes affect not only the brightness but also the image quality, there is a problem in that increasing the gain increases noise and deteriorates the image quality, as shown in FIG. Furthermore, there are cases where the image quality desired by the user is not achieved. Furthermore, when the imaging environment is dark, such as at night, adjusting the ND or iris is not effective in adjusting the image quality, so it is necessary to increase the gain, but there is also the problem that increasing the gain increases noise in the captured image.

[1-2. Configuration of imaging device 10 and information processing device 300]
Next, the configurations of the imaging device 10 and the information processing device 300 of the present technology will be described with reference to FIG. 5. The imaging device 10 includes a lens head 100, a camera body 200, and an information processing device 300.

As shown in FIG. 6, the lens head 100 includes a lens 101 including an optical lens for condensing light from a subject onto an image sensor and a focus lens for adjusting the focus, and a lens information storage section that holds information regarding the lens 101. 102, a lens drive driver (not shown) for moving the lens 101, and the like. An optical image of the subject obtained through the lens 101 is formed on an image sensor in the imaging unit 201.

The lens head 100 is constructed as a separate member from the camera body 200 and is attached to the camera body 200. Therefore, as shown in FIG. 6, it is possible to replace the lens head 100 attached to the camera body 200 with another lens head 100B. The lens head 100 is fixed to the camera body 200 by, for example, screws. Since the lens head 100 is replaceable, it can be traded alone as a product. If the lens head 100 is to be traded, it may be traded as a set with a coaxial cable and a board.

The lens information storage unit 102 is composed of, for example, an EPROM (Erasable Programmable Read Only Memory). The lens information storage unit 102 stores in advance various information regarding the lens 101, such as a lens center deviation value which is a unique value of each lens, lens shading correction data which is a fixed value of the lens type, distortion correction data, and lens type information. ing.

Since information regarding lenses is stored in the lens information storage section 102, the camera body 200 reads information regarding the lenses from the lens information storage section 102 after replacing the lens head 100 or when starting up the imaging device 10. Then, the camera body 200 recognizes the attached lens 101 based on the information, automatically changes to an operation mode suitable for the lens 101, and performs imaging.

Furthermore, as shown in FIG. 6, it is also possible to replace the lens 101 attached to the lens head 100 with another lens. FIG. 6 shows an example in which the lens 101 and the lens 101C attached to the lens head 100 are interchangeable, and furthermore, the lens 101B and the lens 101C attached to the lens head 100B are interchangeable. Various lenses such as a wide-angle lens, a narrow-angle lens, and a fisheye lens can be attached to the lens head 100.

A user can rewrite the information stored in the lens information storage section 102 by connecting the lens head 100 to a device such as a personal computer. Therefore, when the user replaces the lens 101, the user connects the lens head 100 to a personal computer or the like and rewrites the information in the lens information storage section 102 with information about the newly attached lens. Thereby, the camera body 200 can recognize a newly attached lens, automatically change to an operation mode suitable for the lens, and perform imaging.

The camera body 200 includes an imaging section 201, a sensor section 202, an exposure control section 203, a blur correction amount generation section 204, a blur correction amount adjustment section 205, a blur correction processing section 206, a development processing section 207, an output processing section 208, and a communication section. 209. An information processing device 300 is configured by an exposure control section 203, a blur correction amount generation section 204, a blur correction amount adjustment section 205, and a blur correction processing section 206.

The imaging unit 201 includes an imaging element and a shutter mechanism that photoelectrically converts incident light from a subject obtained through the lens 101 into an amount of charge to generate image data. The imaging unit 201 outputs the generated image data to the exposure control unit 203 and the blur correction processing unit 206. As the image sensor, a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or the like is used. The shutter mechanism drives the shutter based on the shutter speed acquired from the exposure control unit 203 and adjusts the exposure time. The shutter mechanism may be of any type, such as a mechanical shutter type or an electronic shutter type.

The sensor unit 202 is a variety of sensors that detect the angular velocity and acceleration of the movement of the imaging device 10. Examples of the sensor include an IMU (Inertial Measurement Unit), an inertial sensor (an acceleration sensor in two or three axis directions, an angular velocity sensor, a gyro sensor), and the like. A plurality of sensors may be used together as the sensor unit 202. The sensor unit 202 outputs sensing results to the blur correction amount generation unit 204. Note that the sensor unit 202 may not be included in the camera body 200 but may be configured as an external device and transmit sensing results to the camera body 200 by wired or wireless communication.

The exposure control unit 203 sets the shutter speed to adjust the exposure time based on user input and functions provided in the imaging device 10 in advance. In this way, the camera body 200 has a shutter speed control function. The exposure control unit 203 outputs the set shutter speed to the imaging unit 201 and the blur correction amount adjustment unit 205.

The blur correction amount generation unit 204 uses the sensing results obtained from the sensor unit 202 to generate a blur correction amount based on a general blur correction algorithm. The blur correction amount generation unit 204 outputs the blur correction amount to the blur correction amount adjustment unit 205. The amount of blur correction is the amount of movement of the cutting position of image data in electronic blur correction that corrects blur by cutting out image data of a predetermined size from the image data output from the imaging unit 201 according to the degree of blur. It is. The blur correction amount corresponds to the value related to blur correction in the claims.

The blur correction amount adjustment unit 205 holds adjustment data prepared in advance, and adjusts the blur correction amount based on the adjustment data and the shutter speed obtained from the exposure control unit 203. The blur correction amount adjustment unit 205 outputs the adjusted blur correction amount (referred to as an adjusted blur correction amount) to the blur correction processing unit 206.

The blur correction processing unit 206 performs blur correction processing using an electronic blur correction method that cuts out an image of a predetermined size from the image data based on the adjusted blur correction amount. As shown in FIG. 7, the blur correction processing unit 206 performs lens distortion removal and rotation processing on the input image data and rotation vector mesh, respectively, and pastes the image data onto the rotation vector mesh, for example, 4K ( Blurring is corrected by cutting out full HD (horizontal 1920 x vertical 1080) image data from image data (horizontal 3840 x vertical 2160). Note that the blur correction process may be performed using a general method, such as a method using sensing results from a gyro sensor or IMU as the sensor unit 202, or a method in which alignment with past frames is performed by image processing.

The development processing unit 207 performs predetermined image processing, such as gain correction processing, white balance processing, color adjustment processing, Debayer processing, and aspect conversion processing, on the image data subjected to blur correction processing by the blur correction processing unit 206. etc.

The output processing unit 208 performs output processing such as compression encoding and output format conversion on the image data output from the development processing unit 207. As the encoding method, for example, MPEG (Moving Picture Experts Group), JPEG (Joint Photographic Experts Group), etc. are applied. If the data is stored as uncompressed RAW, compression encoding is not required. Examples of the output format include MDMI (High-Definition Multimedia Interface) and SDI (Serial Digital Interface).

The communication unit 209 is a communication module that performs communication with other external devices, the Internet, and the like. The communication method may be either wired or wireless, and specifically, cellular communication, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), HDMI (registered trademark) (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc. The image data output from the output processing unit 208 is transmitted as a captured image to an external device such as a personal computer via the communication unit 209 and a network. The imaging device 10 can also transmit captured images to an external device in real time. Note that the imaging device 10 does not need to include the communication unit 209.

Further, although not shown, the imaging device 10 may include a storage medium, a display section, a control section, an input section, and the like.

The storage medium stores captured images, and is composed of a drive device for a portable recording medium such as a magnetic tape or an optical disk, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The storage medium may be an external device connected to imaging device 10 via a wired or wireless connection. The image data output from the output processing unit 208 may be stored as a captured image in a storage medium.

The display unit displays captured images, through images, GUI (Graphical User Interface), and the like. The display unit is realized by, for example, a monitor display composed of an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, or an EVF (Electronic View Finder). . Note that the display unit may be an external device connected to the imaging device 10 through a wired or wireless connection.

The control unit is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU controls the entire imaging device 10 and each part by executing various processes and issuing commands according to programs stored in the ROM.

The input unit is for the user to input various instructions and the like to the imaging device 10. When a user makes an input to the input unit, a control signal corresponding to the input is generated, and the imaging device 10 performs various processes corresponding to the control signal. Input units include a release button for issuing a release instruction, physical buttons for various operations, a touch panel, a touch screen integrated with a display unit, and the like.

The imaging device 10 and the information processing device 300 are configured as described above. The information processing device 300 may be configured as a dedicated device using hardware having that function, or the imaging device 10 may have the function as the information processing device 300 by executing a program, and the information processing method may be realized. The program may be installed in the imaging device 10 in advance, or may be downloaded or distributed on a storage medium, and installed by the user in the imaging device 10. Alternatively, the information processing device 300 may include the imaging unit 201.

[1-3. Processing by information processing device 300]
Next, processing of the information processing device 300 in the first embodiment will be described with reference to FIG. 8. In order to solve the above-mentioned problem, in the first embodiment, in the imaging device 10 equipped with an electronic blur correction function, the blur correction amount adjustment unit 205 adjusts the blur correction amount based on the shutter speed.

First, in step S11, the blur correction amount generation unit 204 generates a target amount of blur correction based on the sensing results received from the sensor unit 202. The blur correction amount generation unit 204 outputs the target amount of the blur correction amount to the blur correction amount adjustment unit 205. The target amount of blur correction amount is a value of 100% (maximum amount) of the blur correction amount calculated based on the sensing results.

Next, in step S12, the exposure control unit 203 sets the shutter speed. The exposure control section 203 outputs the shutter speed to the blur correction amount adjustment section 205.

Note that step S11 and step S12 may be performed in the reverse order, or may be performed simultaneously or almost simultaneously.

Next, in step S13, the blur correction amount adjustment unit 205 calculates the minimum amount of the blur correction amount by multiplying the target amount of the blur correction amount by a predetermined magnification (0x to 1x). The predetermined magnification may be set in advance, for example, at the time of designing or manufacturing the imaging device 10 or the information processing device 300, or may be set by the user.

Furthermore, the blur correction amount adjustment unit 205 adjusts the blur correction amount according to the shutter speed based on the blur correction amount (target amount and minimum amount), shutter speed, and adjustment data shown in FIG. 9 prepared in advance. adjust. The blur correction amount adjustment unit 205 outputs the adjusted blur correction amount to the blur correction processing unit 206.

Specifically, as shown in FIG. 9, when the shutter speed is equal to or higher than the first speed, the blur correction amount is increased to reach the target amount than when the shutter speed is less than the first speed. Adjust to. Further, when the shutter speed is less than or equal to a second speed that is smaller than the first speed, the blur correction amount is adjusted to be smaller than when the shutter speed is larger than the second speed so that it becomes the minimum amount. Furthermore, when the shutter speed is higher than the second speed and lower than the first speed, the blur correction amount is adjusted so as to approach the target amount as the shutter speed becomes faster. Note that the characteristics of the adjustment data shown in FIG. 9 may change not linearly but exponentially or quadratically, and may be adjusted according to the use case.

In order to strongly suppress the motion blur of the subject in the captured image, the shutter speed is increased, and in order to weakly suppress the motion blur of the subject, the shutter speed is slowed. Further, in order to strongly suppress changes in the composition, the adjusted blur correction amount is increased, and in order to weakly suppress changes in the composition, the adjusted blur correction amount is decreased.

In addition, in FIG. 9, the first speed of the shutter speed is "1/250" and the second speed is "1/200", but this is just an example, and the present technology is limited to these values. It's not a thing.

In this way, the blur correction amount adjustment unit 205 adjusts the blur correction amount according to the shutter speed set by the exposure control unit 203, and determines the adjusted blur correction amount.

Then, in step S14, the blur correction processing unit 206 performs blur correction processing on the image data based on the adjusted blur correction amount. The image data that has been subjected to blur correction processing is processed by a development processing section 207 and an output processing section 208, and is output as a captured image.

The processing of the information processing device 300 in the first embodiment is performed as described above. As described above, even though there is motion blur of the subject in the captured image, if the composition does not change due to blur correction, the captured image will feel unnatural. However, according to the first embodiment, when the shutter speed is slowed in order to weakly suppress motion blur, the adjusted blur correction amount approaches the minimum amount and becomes small, and changes in the composition are also suppressed weakly. Therefore, the sense of discomfort in the captured image can be suppressed.

This point will be explained with reference to FIG. Note that the imaging conditions such as the positions of the imaging device 10 and the subject P are the same as in the examples shown in FIGS. 1 to 4.

First, as shown in FIG. 10A, imaging is performed without moving the imaging device 10 in a state where the subject P is located on the right side of the imaging device 10 in the imaging direction of the imaging device 10 (the direction in which the lens faces). Then, the subject P is located on the right side in the captured image. This is similar to FIG. 2A.

Furthermore, as shown in FIG. 10B, if an image is captured while the imaging device 10 is being panned clockwise from the state of FIG. The motion blur of the subject P is suppressed. However, since the shutter speed is set slow, the motion blur is suppressed weakly, so the motion blur remains although it is suppressed compared to FIG. 2B.

In addition, since blur correction is performed to cancel the movement of the imaging device 10, the subject P, whose composition has changed due to the movement of the imaging device 10 and has moved to the left, is corrected so that it moves to the right. Change is suppressed. However, when the shutter speed is slowed down, the adjusted blur correction amount approaches the minimum amount and becomes smaller, so the change in composition is weakly suppressed, and the position of the subject P after blur correction in FIG. 10B is located on the left side compared to FIG. 2B. become.

The length of the arrow b' in FIG. 10B indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image by blur correction. The amount of movement of the subject P indicated by the length of the arrow b' is shorter than the amount of movement of the subject P due to suppressed changes in the composition in FIG. 2B, which is indicated by the arrow b. That is, in FIG. 10B, changes in the composition are suppressed weaker than in FIG. 2B.

Furthermore, as shown in FIG. 10C, when the imaging device 10 is panned clockwise by 90 degrees, the subject P will be located on the left side of the imaging device 10 in the imaging direction of the imaging device 10 (the direction in which the lens faces). When imaging is performed when the imaging device 10 is panned 90 degrees clockwise, blur correction is performed to cancel the movement of the imaging device 10. However, when the shutter speed is slowed down, the adjusted blur correction amount approaches the minimum amount and becomes smaller, so the change in composition is weakly suppressed, and the position of the subject P after blur correction in FIG. 10C is on the left side compared to FIG. 2C. become.

The length of the arrow c' in FIG. 10C indicates the amount of movement of the position of the subject P due to suppressing changes in the composition of the captured image by blur correction. The amount of movement of the subject P indicated by the length of the arrow c' is shorter than the amount of movement of the subject P due to suppressed changes in the composition in FIG. 2C, which is indicated by the arrow c. That is, in FIG. 10C, changes in the composition are suppressed weaker than in FIG. 2C.

In this way, when the shutter speed is slowed to suppress motion blur, the adjusted blur correction amount approaches the minimum amount and becomes small, so changes in the composition are also suppressed. Therefore, the sense of discomfort in the captured image can be suppressed.

On the other hand, if the shutter speed is increased in order to strongly suppress motion blur, the adjusted blur correction amount will approach the target amount and increase, which will also strongly suppress changes in the composition. Therefore, in this case as well, the sense of discomfort in the captured image can be suppressed.

Further, according to the first embodiment, for example, if the user does not want to shorten the exposure time (want to lengthen the exposure time) to prevent the captured image from becoming dark, the shutter speed is brought closer to the second speed. The adjusted blur correction amount approaches the minimum amount. Further, when the shutter speed is lower than the second speed, the adjusted blur correction amount becomes the minimum value. As a result, the operation approaches that of a normal imaging device 10 that does not have an electronic image stabilization function, so that it is possible to take a captured image with less discomfort.

In this way, it is possible to obtain an image with appropriate brightness and image quality that exhibits both "motion blur" and "change in composition" due to the movement of the imaging device 10 when capturing an image using the imaging device 10 equipped with a blur correction function. can.

The present technology is useful when the movement or vibration of the imaging device 10 is large because it can correct blur caused by movement of the imaging device 10 and suppress changes in composition while suppressing the sense of discomfort in the captured image. For example, the present technology is useful when mounted on a moving motorcycle to capture images of the front and rear of the moving motorcycle, the driver's face, etc. Note that the moving object may be other than a motorcycle, such as a bicycle, a car, a drone, a ship, or a person. Further, the imaging device 10 is not limited to one that is mounted on a moving object, but may be a digital camera, a single-lens reflex camera, a camcorder, a professional camera, a professional imaging device, etc. that is held and used by a person.

<2. Second embodiment>
[2-1. Configuration of imaging device 10 and information processing device 300 and processing by information processing device 300]
Next, a second embodiment of the present technology will be described with reference to FIG. 11. Regarding the configurations of the imaging device 10 and the information processing device 300, only the processing units that perform processing different from those in the first embodiment will be described.

The blur correction amount generation unit 204 uses the sensing results obtained from the sensor unit 202 to generate a blur correction amount based on a general blur correction algorithm. The blur correction amount generation unit 204 outputs the blur correction amount to the exposure adjustment unit 210 and the blur correction processing unit 206.

The exposure adjustment unit 210 determines the adjusted shutter speed by adjusting the shutter speed based on the blur correction amount obtained from the blur correction amount generation unit 204 and the adjustment data shown in FIG. 12 prepared in advance. The exposure adjustment section 210 outputs the adjusted shutter speed to the imaging section 201. The imaging unit 201 performs shutter control based on the adjusted shutter speed. The second embodiment is characterized in that the exposure adjustment section 210 adjusts the shutter speed based on the amount of blur correction in this way.

In the second embodiment, the lower limit value of the shutter speed (that is, the maximum exposure time) is determined according to the amount of blur correction, thereby placing constraints on the exposure control circuit. First, the lower limit value of the shutter speed is calculated from the blur correction amount based on the adjustment data shown in FIG. The shutter speed is adjusted using this lower limit value as the shutter speed limit value. The limit on the vertical axis (lower limit of shutter speed) is determined by the capability value of the imaging device 10. In the case of videos, the speed is usually determined by the frame rate. On the other hand, the lower limit value is basically a value determined in advance by the camera designer, but may be determined by the user.

Specifically, as shown in FIG. 12, when the amount of blur correction is equal to or greater than the first threshold, the shutter speed is made slower than when the amount of blur correction is less than the first threshold to reach the lower limit value. Adjust as follows. Further, when the amount of blur correction is less than or equal to a second threshold value which is smaller than the first threshold value, the shutter speed is adjusted to be faster than when the amount of blur correction is greater than the second threshold value so that it is unlimited. Further, when the amount of blur correction is greater than or equal to the second threshold and less than or equal to the first threshold, the shutter speed is adjusted so as to approach the lower limit as the amount of blur correction increases. Note that the characteristics of the adjustment data shown in FIG. 12 may change not linearly but exponentially or quadratically, and may be adjusted according to the use case.

The blur correction processing unit 206 performs blur correction processing using an electronic blur correction method that cuts out an image of a predetermined size from the image data based on the blur correction amount obtained from the blur correction amount generation unit 204. The blur correction method is the same as in the first embodiment. The image data that has been subjected to blur correction processing is processed by a development processing section 207 and an output processing section 208, and is output as a captured image.

Other configurations and processing of the imaging device 10 and the information processing device 300 are the same as in the first embodiment.

The processing of the information processing device 300 in the second embodiment is performed as described above. According to the second embodiment, for example, when the amount of blur correction is made small in order to weakly suppress changes in composition, the shutter speed is slowed and motion blur is suppressed weakly. Therefore, similar to the first embodiment, it is possible to suppress the sense of discomfort in the captured image.

Additionally, if you increase the amount of blur correction to strongly suppress changes in the composition, the shutter speed will become faster and motion blur will be strongly suppressed. Therefore, similar to the first embodiment, it is possible to suppress the sense of discomfort in the captured image. In this way, prioritize the amount of image stabilization, and if a large amount of image stabilization is required, increase the shutter speed and shorten the exposure time, and if a small amount of image stabilization is required, slow the shutter speed and take the necessary amount of exposure. It is also possible to set the time to be reserved. In this case, when the imaging device 10 moves, the exposure time changes and the brightness changes, so it is desirable to adjust the captured image to a constant brightness using ND, iris, gain, etc.

<3. Third embodiment>
[3-1. Configuration of imaging device 10 and information processing device 300 and processing by information processing device 300]
Next, a third embodiment of the present technology will be described with reference to FIG. 13. Regarding the configurations of the imaging device 10 and the information processing device 300, only the processing units that perform processing different from those in the first embodiment will be described.

The exposure control unit 203 sets the shutter speed to adjust the exposure time based on user input and functions provided in the imaging device 10 in advance. The exposure control unit 203 outputs the set shutter speed to the imaging unit 201 and the blur correction setting value adjustment unit 211.

The blur correction setting value adjustment unit 211 allows the blur correction amount generation unit 204 to generate the blur correction amount based on the shutter speed acquired from the exposure control unit 203 and adjustment data shown in FIG. 14 prepared in advance. Adjust the blur correction settings used for calculations. The blur correction setting value corresponds to the value related to blur correction in the claims. Examples of the blur correction setting value include a correction tracking speed and a cutoff frequency. However, the blur correction setting value is not limited to these, and may be any value as long as it is used in calculations for generating the blur correction amount. The third embodiment is characterized in that the blur correction setting value adjustment section 211 adjusts the blur correction setting value based on the shutter speed in this way.

Specifically, as shown in FIG. 14, when the shutter speed is equal to or higher than the first speed, the blur correction setting value is increased and adjusted to the target amount. Furthermore, when the shutter speed is equal to or lower than the second speed, which is smaller than the first speed, the blur correction setting value is decreased and adjusted to the minimum amount. Furthermore, when the shutter speed is equal to or higher than the second speed, the camera shake correction setting value is adjusted so as to approach the target amount as the shutter speed becomes faster. Note that the characteristics of the adjustment data shown in FIG. 14 may change not linearly but exponentially or quadratically, and may be adjusted according to the use case.

The blur correction amount generation unit 204 uses the sensing result obtained from the sensor unit 202 and the blur correction setting value obtained from the blur correction setting value adjustment unit 211 to generate a blur correction amount based on a general blur correction algorithm. generate. When the blur correction setting value is adjusted to a low value, the correction amount generated by the blur correction amount generation unit 204 also becomes small, and when the blur correction setting value is adjusted to a high value, the blur correction amount also becomes large. . The blur correction amount generation unit 204 outputs the blur correction amount to the blur correction processing unit 206.

The blur correction processing unit 206 performs blur correction processing using an electronic blur correction method that cuts out an image of a predetermined size from the image data based on the amount of blur correction. The blur correction method is the same as in the first embodiment. The image data that has been subjected to blur correction processing is processed by a development processing section 207 and an output processing section 208, and is output as a captured image.

The other configurations of the imaging device 10 and the information processing device 300 are the same as in the first embodiment.

The processing of the information processing device 300 in the third embodiment is performed as described above. According to the third embodiment, the present technology can be applied not only to the amount of camera shake correction itself but also to adjustment of various setting values for generating the amount of camera shake correction. By adjusting the blur correction setting value based on the shutter speed and determining the adjusted blur correction amount based on the blur correction setting value, it is possible to suppress the sense of discomfort in the captured image as in the first embodiment. .

<4. Modified example>
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical idea of the present technology are possible.

In the first to third embodiments described above, the imaging device 10 has the function of the information processing device 300, and all processing is performed within the imaging device 10. Not limited.

As shown in FIG. 15, the processing by the information processing device 300 may be divided between the imaging device 10 and the external device 20. In the example of FIG. 15, the imaging device 10 includes an exposure control section 203 and a blur correction amount generation section 204, as shown in FIG. 15A. Further, as shown in FIG. 15B, the external device 20 includes a blur correction amount adjustment section 205 and a blur correction processing section 206. Examples of the external device 20 include a server, a personal computer, a smartphone, a tablet terminal, and a camera.

The imaging device 10 superimposes the shutter speed set by the exposure control unit 203 and the blur correction amount generated by the blur correction amount generation unit 204 on image data as metadata, and transmits the metadata to the external device 20. Setting the shutter speed and generating the blur correction amount are the same as in the first embodiment.

Then, the external device 20 that has received the image data through the communication unit 21 outputs the image data to the blur correction processing unit 206. Further, the shutter speed and the blur correction amount as metadata superimposed on the image data are output to the blur correction amount adjustment unit 205.

The blur correction amount adjustment unit 205 adjusts the blur correction amount in the same manner as in the first embodiment, and outputs the adjusted blur correction amount to the blur correction processing unit 206. Then, the blur correction processing unit 206 performs blur correction processing on the image data based on the adjusted blur correction amount. The image data subjected to the blur correction process is stored in, for example, the storage medium 22 as a captured image. Note that the captured image may be transmitted from the external device 20 to another device.

Note that in this modification, the blur correction amount generation unit 204 is configured to be included in the external device 20, and the imaging device 10 superimposes the sensing result acquired by the sensor unit 202 on image data as metadata and transmits it to the external device 20. You may also do so. In this case, the blur correction amount generation unit 204 in the external device 20 generates the blur correction amount.

Next, a second modification example in which fail-safe is executed will be explained. If there is any abnormality in the sensing results output from the sensor unit 202 or if the sensor unit 202 malfunctions, the blur correction processing unit 206 performs blur correction processing by cutting out a predetermined range based on the center of the image data as a fail-safe measure. I do. Thereby, even if there is an abnormality in the sensing result, the output of the captured image will not be interrupted, and for example, it is possible to avoid problems with broadcasting using the captured image.

Cutting out a predetermined range based on the center of the image data can be realized, for example, by generating the blur correction amount as 0 in the blur correction amount generation unit 204. By setting the blur correction amount to 0, even if the blur correction amount is adjusted by the blur correction amount adjustment unit 205, the result is 0 even if 0 is multiplied by the magnification (0 to 1 times).As a result, the center is used as the reference. It will be cut out.

In this way, the determination as to whether or not to execute failsafe may be made by the blur correction amount generation unit 204 that receives the sensing results from the sensor unit 202, or may be made by a control unit included in the imaging device 10. Alternatively, a processing unit dedicated to the determination may perform the determination.

Further, the adjustment data for adjusting the blur correction amount is not limited to that shown in FIG. 9, and may have a characteristic having hysteresis, for example, as shown in FIG. 16. For example, a plurality of adjustment data may be prepared for each imaging environment (indoor, outdoor, etc.), and the adjustment data to be used may be selected depending on the imaging environment. Alternatively, a plurality of adjustment data may be prepared for each type of subject P, and the adjustment data to be used may be selected depending on the subject P. For example, adjustment data preselected by the user from a plurality of adjustment data may be used, or the control unit (not shown) of the imaging device 10 may use the image data transmitted from the imaging unit 201 to create an imaged scene ( Alternatively, the adjustment data may be selected based on the recognition result.

Further, in the embodiment, the lens head 100 including the lens 101 is described as being replaceable with respect to the camera body 200, but the present technology relates to an imaging device in which the lens 101 and the camera body 200 are integrally configured. is also applicable.

Additionally, this technology can be applied to any device that has a camera function, such as a smartphone, personal computer, tablet terminal, portable game machine, or wearable device.

Furthermore, the present technology is not limited to still images, but can also be applied to frame images that make up a moving image.

The present technology can also have the following configuration.
(1)
An information processing device that includes an adjustment unit that adjusts either a value related to blur correction or the shutter speed in an imaging device that has an electronic blur correction function that corrects blur in image data and a shutter speed control function.
(2)
The information processing device according to (1), wherein the value related to blur correction is a blur correction amount indicating a movement amount of a cutting position of image data in electronic blur correction.
(3)
The information processing device according to (2), wherein the adjustment section is a blur correction amount adjustment section that adjusts the blur correction amount based on the shutter speed.
(4)
The information processing device according to (3), further comprising a blur correction amount generation unit that generates a target amount of the blur correction amount based on a motion sensing result of the imaging device.
(5)
The information processing device according to (4), wherein the blur correction amount adjustment unit determines the minimum amount of the blur correction amount based on the target amount of the blur correction amount.
(6)
The information processing device according to (4) or (5), wherein the blur correction amount adjustment unit adjusts the blur correction amount to the target amount when the shutter speed is equal to or higher than a first speed.
(7)
The blur correction amount adjustment unit adjusts the blur correction amount to the minimum amount when the shutter speed is less than or equal to a second speed that is a smaller value than the first speed (5) or (5) 6) The information processing device according to item 6).
(8)
The blur correction amount adjustment unit adjusts the blur correction amount from the minimum amount to approach the target amount as the shutter speed approaches the second speed from the first speed (5) to (7). ).
(9)
The information processing device according to (2), wherein the adjustment section is an exposure adjustment section that adjusts the shutter speed based on the blur correction amount.
(10)
The information processing device according to any one of (1) to (9), wherein the value related to blur correction is a blur correction setting value used to calculate the blur correction amount.
(11)
The information processing device according to (10), wherein the blur correction setting value is a tracking speed of blur correction.
(12)
The information processing device according to (10), wherein the blur correction setting value is a cutoff frequency.
(13)
The information processing device according to any one of (10) to (12), wherein the adjustment section is a blur correction setting value adjustment section that adjusts the blur correction setting value based on the shutter speed.
(14)
The information processing device according to any one of (10) to (13), including a blur correction amount generation unit that generates a blur correction amount based on the blur correction setting value.
(15)
The information processing device according to any one of (1) to (14), including a blur correction processing section that performs blur correction processing on the image data.
(16)
The information processing device according to any one of (1) to (15), including an imaging unit that generates the image data.
(17)
In the information processing device according to (15), when there is an abnormality in the motion sensing result of the imaging device, the blur correction processing unit performs blur correction processing by cutting out a predetermined range based on the center of the image data. .
(18)
The information according to any one of (1) to (17), wherein the adjustment unit adjusts either the value related to blur correction or the shutter speed based on the adjustment data selected from a plurality of adjustment data. Processing equipment.
(19)
An information processing method for adjusting either one of the value related to the blur correction and the shutter speed in an imaging device that includes an electronic blur correction function for correcting blur in image data and a shutter speed control function.
(20)
A program that causes a computer to execute an information processing method for adjusting either a value related to the blur correction or the shutter speed in an imaging device having an electronic blur correction function for correcting blur in image data and a shutter speed control function.

10... Imaging device 201... Imaging section 204... Shake correction amount generation section 205... Shake correction amount adjustment section 206... Shake correction processing section 210... Exposure adjustment section 211... Blur correction setting value adjustment unit 300...information processing device

Claims

An information processing device that includes an adjustment unit that adjusts either a value related to blur correction or the shutter speed in an imaging device that has an electronic blur correction function that corrects blur in image data and a shutter speed control function.
2. The information processing apparatus according to claim 1, wherein the value related to blur correction is a blur correction amount indicating a movement amount of a cutting position of image data in electronic blur correction.
The information processing apparatus according to claim 2, wherein the adjustment section is a blur correction amount adjustment section that adjusts the blur correction amount based on the shutter speed.
The information processing apparatus according to claim 3 , further comprising a blur correction amount generation unit that generates a target amount of the blur correction amount based on a motion sensing result of the imaging device.
The information processing apparatus according to claim 4, wherein the blur correction amount adjustment section determines the minimum amount of the blur correction amount based on the target amount of the blur correction amount.
The information processing device according to claim 4, wherein the blur correction amount adjustment unit adjusts the blur correction amount to the target amount when the shutter speed is a first speed or higher.
The blur correction amount adjustment unit adjusts the blur correction amount to the minimum amount when the shutter speed is less than or equal to a second speed that is a smaller value than the first speed. information processing equipment.
The blur correction amount adjustment unit adjusts the blur correction amount so that it approaches the target amount from the minimum amount as the shutter speed approaches the second speed from the first speed. Information processing device.
The information processing apparatus according to claim 2, wherein the adjustment section is an exposure adjustment section that adjusts the shutter speed based on the blur correction amount.
The information processing apparatus according to claim 1, wherein the value related to blur correction is a blur correction setting value used to calculate the blur correction amount.
The information processing apparatus according to claim 10, wherein the blur correction setting value is a tracking speed of blur correction.
The information processing apparatus according to claim 10, wherein the blur correction setting value is a cutoff frequency.
The information processing apparatus according to claim 10, wherein the adjustment section is a blur correction setting value adjustment section that adjusts the blur correction setting value based on the shutter speed.
The information processing apparatus according to claim 10, further comprising a blur correction amount generation unit that generates a blur correction amount based on the blur correction setting value.
The information processing apparatus according to claim 1, further comprising a blur correction processing section that performs blur correction processing on the image data.
The information processing device according to claim 1, further comprising an imaging unit that generates the image data.
The information processing device according to claim 15, wherein when there is an abnormality in the motion sensing result of the imaging device, the blur correction processing unit performs blur correction processing by cutting out a predetermined range based on the center of the image data. .
The information processing apparatus according to claim 1 , wherein the adjustment unit adjusts either the value related to blur correction or the shutter speed based on the adjustment data selected from a plurality of adjustment data.
An information processing method for adjusting either one of the value related to blur correction and the shutter speed in an imaging apparatus that includes an electronic blur correction function for correcting blur in image data and a shutter speed control function.
A program that causes a computer to execute an information processing method for adjusting either a value related to the blur correction or the shutter speed in an imaging device having an electronic blur correction function for correcting blur in image data and a shutter speed control function.