WO2023106118A1 - 情報処理装置、情報処理方法、およびプログラム - Google Patents
情報処理装置、情報処理方法、およびプログラム Download PDFInfo
- Publication number
- WO2023106118A1 WO2023106118A1 PCT/JP2022/043456 JP2022043456W WO2023106118A1 WO 2023106118 A1 WO2023106118 A1 WO 2023106118A1 JP 2022043456 W JP2022043456 W JP 2022043456W WO 2023106118 A1 WO2023106118 A1 WO 2023106118A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- map
- information
- subject
- depth information
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- H04N23/671—Focus control based on electronic image sensor signals in combination with active ranging signals, e.g. using light or sound signals emitted toward objects
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
- G03B13/32—Means for focusing
- G03B13/34—Power focusing
- G03B13/36—Autofocus systems
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B15/00—Special procedures for taking photographs; Apparatus therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Definitions
- the present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable an appropriate subject to be a predetermined processing target.
- Patent Document 1 Devices with shooting functions such as digital still cameras and smartphones are usually equipped with an autofocus function.
- This technology has been developed in view of such a situation, and enables an appropriate subject to be subject to predetermined processing.
- An information processing apparatus provides first depth information indicating a distance to a subject included in an input image based on the output of a distance sensor, and an inference model for inferring the distance to the subject from the input image.
- a processing target determination unit that determines a processing target in a predetermined process for the input image using depth information for processing target determination selected from the second depth information based on the output when input.
- first depth information indicating the distance to the subject included in the input image based on the output of the distance sensor
- the depth information for determining the processing object selected from the second depth information based on the output of is determined.
- FIG. 10 is a diagram illustrating an example of focus target determination processing; It is a block diagram which shows the hardware structural example of an imaging device.
- FIG. 4 is a block diagram showing a functional configuration example of a control unit when the present technology is applied to autofocus control processing; 4 is a flowchart for explaining autofocus control processing of the imaging device; 7 is a flowchart for explaining focus target determination processing of the imaging device;
- FIG. 10 is a diagram showing a modified example 1 of focus target determination processing;
- FIG. 10 is a diagram showing a modified example 2 of focus target determination processing;
- FIG. 13 is a diagram showing a modified example 3 of focus target determination processing;
- FIG. 4 is a block diagram showing a functional configuration example of a control unit when the present technology is applied to tracking control processing; 4 is a flowchart for explaining tracking control processing of an imaging device;
- FIG. 10 is a block diagram showing a functional configuration example of a control unit when the present technology is applied to background blurring control processing; 4 is a flowchart for explaining background blurring control processing of the imaging device;
- Focus target determination process2 Configuration of photographing device 3 . 4. Selection of depth information for determining processing object; 4. Operation of imaging device; Modified example 1 of focus target determination processing 6. Modified example 2 of focus target determination processing 7. Modified example 3 of focus target determination processing 8. Tracking target determination process 9 . Emphasis target determination process 10. Others 11. Configuration combination example
- FIG. 1 is a diagram illustrating an example of focus target determination processing in an imaging device according to an embodiment of the present technology.
- a photographing device is a device having a photographing function, such as a smartphone, digital still camera, or surveillance camera.
- the imaging device has an autofocus function.
- Auto-focusing in an imaging device is performed by appropriately determining the focus target, which is the subject to be focused, from among the subjects included in the input image using the input image, etc., without performing phase difference detection. is done.
- Shooting is performed in a focused state on the determined focus target.
- FIG. 1 shows focus target determination processing for an input image P1 including subjects O1 and O2.
- the subject O1 that appears substantially in the center of the input image P1 is a flower in a vase, and the subject O2 that appears large on the left side is a box-shaped subject.
- the object O2 is in front, and the object O1 is behind the object O2.
- a wall surface is shown behind the subject O1 and the subject O2 placed on the floor or the like.
- the focus target is determined using a preview image including such a subject as an input image.
- the input image P1 shown at the left end of FIG. 1 and the input image P2 shown at the right end of FIG. 1 are the same image including the same subject.
- different numbers are assigned to the input image P1 and the input image P2.
- the first map is a ToF map in which the value of each pixel is the distance to the subject at the position corresponding to each pixel of the input image P1 measured by a ToF (Time Of Flight) sensor.
- the shading on the ToF map in Fig. 1 basically indicates the difference in the distance to the subject at the position corresponding to each pixel.
- the region of pixels in which the object O2 located in the foreground is shown is shown in light color.
- the second map is an MSD (Main Subject Detection) map in which the value of each pixel is the main subject degree, which is the degree of main subject-likeness of the subject at the position corresponding to each pixel of the input image P1.
- MSD Main Subject Detection
- the degree of the main subject is obtained by performing main subject detection (MSD).
- the main subject detection is performed using, for example, an inference model that receives the input image P1 and outputs the degree of the main subject.
- An inference model for main subject detection which is generated in advance by machine learning using a plurality of images in which main subject degrees of subjects included in the images are set, is prepared in the photographing device.
- the shading on the MSD map in Fig. 1 indicates the difference in the main subject degree of the subject at the position corresponding to each pixel.
- the area of pixels in which the object O1 located in the center is shown is shown in light color. In the case of photographing a subject such as that shown in the input image P1 in the composition, it is considered that the user will pay attention to the flower subject O1 in the center.
- the third map is an AiDepth map in which the value of each pixel is the distance to the subject at the position corresponding to each pixel, based on the output when the input image P1 is input to the inference model generated by machine learning.
- An inference model for distance inference which is generated in advance by machine learning using a plurality of images in which distances to subjects included in the images are set, is prepared in the photographing device.
- the shading on the AiDepth map in Figure 1 indicates the difference in the distance to the subject at the position corresponding to each pixel. Similar to the ToF map, the lighter the color of the pixel, the closer the distance to the object corresponding to that pixel, and the darker the pixel, the closer the distance to the object corresponding to that pixel is far away.
- depth information indicating the distance to the subject there are two types of depth information: a ToF map based on the output of the ToF sensor and an AiDepth map based on the output of the inference model generated by machine learning. generated.
- a ToF + MSD composite map is generated by combining the ToF map and the MSD map.
- an AiDepth+MSD combined map is generated by combining the AiDepth map and the MSD map.
- Each synthesizing process is performed, for example, by adding pixel values of corresponding pixels of two maps to be synthesized.
- the subject O1 is determined as the focus target as indicated by a thick line on the input image P2.
- the ToF+MSD composite map and the AiDepth+MSD composite map is selected based on the reliability of the AiDepth map, and the focus target is determined using the selected composite map.
- the subject O1 is a subject captured in an area having a higher value than the subject O2, so the subject O1 is the focus target. has been determined as In the imaging device, focus control is performed to bring the object O1 into focus.
- the focus target is determined by an algorithm that prioritizes the shortest distance based only on information about the distance to the subject, the subject closest to the camera is always determined as the focus target. is not necessarily the most appropriate focus target.
- determining the focus target using the composite map as shown in FIG. 1 means determining the focus target using both information on the distance to the subject and information on the main subject degree of the subject.
- determining the focus target using both the distance information to the subject and the subject's main subject degree information it is possible to focus on a more appropriate subject that takes into account the subject's type, size, position in the image, etc. It can be determined as a target. Since it is considered that the user will pay attention to the subject O1 of the flower in the center, it can be said that setting the subject O1 as the focus target determines a more appropriate subject as the focus target.
- learning of the inference model used for generating the MSD map is performed in consideration of the subject type, size, position within the image, and the like.
- Determining the processing target using either one of the two composite maps means determining the processing target using either one of the two depth information, the ToF map or the AiDepth map, while considering the degree of the main subject. corresponds to
- FIG. 2 is a block diagram showing a hardware configuration example of the imaging device 1. As shown in FIG.
- the imaging device 1 includes a control unit 11, an optical system 12, an image sensor 13, a lens driver 14, a microphone 15, a distance sensor 16, a display 17, an operation unit 18, a speaker 19, a storage unit 20, and a communication unit 21.
- the control unit 11 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like.
- the control unit 11 executes a predetermined program and controls the overall operation of the imaging device 1 according to user's operations.
- the optical system 12 includes a photographing lens for condensing light from a subject onto the image sensor 13, a driving mechanism for moving the photographing lens for focusing and zooming, a shutter mechanism, an iris mechanism, and the like. . These are driven under the control of the lens driver 14 . Incident light from a subject reaches an image sensor 13 as a photographing device via an optical system 12 .
- the image sensor 13 photoelectrically converts incident light from a subject into a charge amount, and outputs it as image data. Image data output from the image sensor 13 is output to the control unit 11 .
- a CCD Charge Coupled Device
- CMOS Complementary Metal Oxide Semiconductor
- the optical system 12 and image sensor 13 function as an imaging unit.
- the lens driver 14 is composed of, for example, a microcomputer, and controls the operation of the drive mechanism, shutter mechanism, iris mechanism, etc. of the optical system 12 under the control of the control unit 11 . As a result, adjustment of exposure time (shutter speed), adjustment of aperture value (F value), and the like are performed.
- the microphone 15 outputs audio data such as collected sound to the control unit 11 .
- the distance sensor 16 is composed of a sensor such as a ToF sensor that can detect the distance in the depth direction of the subject.
- the distance sensor 16 measures the distance to the subject included in the input image and outputs sensor data to the control unit 11 .
- the display 17 is configured by an LCD or the like, and displays various information such as menu screens and images being captured according to control by the control unit 11 .
- the operation unit 18 is composed of operation buttons, a touch panel, etc. provided on the surface of the housing of the photographing device 1 .
- the operation unit 18 outputs to the control unit 11 information indicating the details of the user's operation.
- the speaker 19 outputs sound based on the audio signal supplied from the control unit 11.
- the storage unit 20 is composed of a flash memory or a memory card inserted into a card slot provided in the housing.
- the storage unit 20 stores various data such as image data supplied from the control unit 11 .
- the communication unit 21 performs wireless or wired communication with an external device.
- the communication unit 21 transmits various data such as image data supplied from the control unit 11 to a computer, a smartphone, or the like.
- FIG. 3 is a block diagram showing a functional configuration example of the control unit 11. As shown in FIG.
- the control unit 11 includes a ToF map generation unit 31, an MSD map generation unit 32, an AiDepth map generation unit 33, a map synthesis unit 34, a focus target determination unit 35, and a lens drive control unit 36. .
- At least part of the configuration shown in FIG. 3 is realized by executing a predetermined program by the CPU that configures the control unit 11 .
- Sensor data supplied from the ToF sensor forming the distance sensor 16 is input to the ToF map generator 31
- image data supplied from the image sensor 13 is input to the MSD map generator 32 and AiDepth map generator 33 .
- the ToF map generator 31 generates a ToF map based on the output of the ToF sensor.
- the output of the ToF sensor is sensor data indicating the distance (Depth) to the subject included in the input image.
- a ToF map is a depth map in which the distance to a subject at a position corresponding to each pixel of an input image is recorded as the pixel value of each pixel.
- the depth information generated by the ToF map generation unit 31 can be information other than map-like information. For example, instead of information indicating the distances to all subjects included in the input image, information indicating the distances to some subjects may be generated as the depth information.
- the information used by the ToF map generation unit 31 to generate depth information is not limited to the output of the ToF sensor, and may be the output of another distance sensor.
- the ToF map generation unit 31 When the depth information is generated using the output of another distance sensor, the ToF map generation unit 31 generates depth information based on the output of the other distance sensor, which is different from the ToF sensor. That is, it can be said that the ToF map generation unit 31 generates depth information indicating the distance to the subject included in the input image based on the outputs of various distance sensors.
- the ToF map generated by the ToF map generation unit 31 is supplied to the map synthesis unit 34.
- the MSD map generator 32 generates an MSD map based on the input image.
- the MSD map is a map in which the pixel value of each pixel is the main subject degree, which is a value indicating the likelihood of the main subject degree of the subject at the position corresponding to each pixel of the input image.
- the main subject degree information generated by the MSD map generation unit 32 can be information other than map-like information. For example, instead of information indicating the main subject degrees of all the subjects included in the input image, information indicating the main subject degrees of some subjects may be generated as the main subject degree information.
- the MSD map generation unit 32 generates main subject degree information indicating the main subject degree of the subject included in the input image.
- Such an MSD map may be generated, for example, based on a rule base based on the results of image analysis, or may be generated using an inference model obtained by machine learning as described above.
- an inference model for example, machine learning is performed using multiple images labeled with information indicating which subject is the main subject as training data.
- An inference model for main subject detection is generated.
- the MSD map generation unit 32 for example, inputs the input image to an inference model for main subject detection, and generates an MSD map based on the output of the inference model.
- the MSD map generated by the MSD map generation unit 32 is supplied to the map synthesis unit 34.
- the AiDepth map generation unit 33 generates an AiDepth map based on the input image.
- the AiDepth map is a depth map in which the distance to the subject at the position corresponding to each pixel of the input image is recorded as the pixel value of each pixel.
- the AiDepth map generator 33 prepares an inference model that receives an input image captured by the image sensor 13 and outputs the distance to the subject included in the input image.
- the AiDepth map generator 33 inputs an input image captured by the image sensor 13 to an inference model generated by machine learning to generate an AiDepth map.
- the depth information generated by the AiDepth map generation unit 33 can be information other than map-like information. For example, instead of information indicating the distances to all subjects included in the input image, information indicating the distances to some subjects may be generated as the depth information.
- the AiDepth map generation unit generates AiDepth information based on the output when the input image is input to an inference model that infers the distance to the subject.
- the MSD map generated by the AiDepth map generation unit 33 is supplied to the map synthesis unit 34.
- the map synthesizing unit 34 generates a ToF+MSD synthetic map by synthesizing the ToF map and the MSD map. Also, the map synthesizing unit 34 generates an AiDepth+MSD synthetic map by synthesizing the AiDepth map and the MSD map.
- the synthesizing process by the map synthesizing unit 34 generates one synthesized map from the two maps to be synthesized by, for example, calculating one pixel value from the pixel values of the corresponding pixels of the two maps to be synthesized. It is done as if For example, the pixel values of the corresponding pixels in the two maps to be synthesized may be added to generate the synthesized map, or the pixel values of the corresponding pixels in the two maps to be synthesized may be weighted. may be added to generate a composite map. When generating a composite map by adding, the closer the distance to the subject, which is the pixel value of the ToF map and the AiDepth map, the more likely it is suitable as a focus target.
- the pixel values may be converted so that the closer the distance, the larger the value, and then added to the pixel values of the MSD map.
- the map synthesizing unit 34 may generate, for example, a ToF+MSD synthetic map and an AiDepth+MSD synthetic map by different synthesizing processes. In this way, the map synthesizing unit 34 generates synthetic information that is two synthetic maps, the ToF+MSD synthetic map and the AiDepth+MSD synthetic map.
- composite information is generated by simply adding values indicating corresponding information of depth information and main subject degree information, or performing weighted addition.
- the distance and the main subject degree for the same subject are the values indicated by the corresponding depth information and main subject degree information.
- the ToF+MSD composite map and AiDepth+MSD composite map generated by the map synthesizer 34 are supplied to the focus target determiner 35 .
- the focus target determination unit 35 determines the focus target based on either the ToF+MSD combined map or the AiDepth+MSD combined map. For example, the focus target determination unit 35 selects either one of the ToF+MSD composite map and the AiDepth+MSD composite map as a map used for determining the focus target in the autofocus control process for the input image, and selects the selected map. is used to determine the focus target.
- the selection of the map is performed by the focus target determining unit 35 using either the ToF map or the AiDepth map as depth information for determining the focus target in the autofocus control process for the input image. This is the process of selecting as depth information. That is, when the focus target determining unit 35 selects either one of the ToF+MSD combined map and the AiDepth+MSD combined map, it means selecting depth information for processing target determination.
- the synthesized information generated by the map synthesizing unit 34 is information other than map-like information
- one of the two pieces of synthesized information is selected.
- a focus target is determined by the focus target determination unit 35 using the selected combined information. Determining the focus target using the synthesis information corresponds to determining the focus target based on the main subject information and the depth information for determining the processing target.
- the method of determining the focus target based on the main subject information and the depth information for determining the processing target is not limited to the method described above.
- a method of selecting a subject with the highest degree of main subject based on main subject information as a focus target from subjects selected based on depth information for determining a processing target may be used.
- the focus target determination unit 35 determines the subject corresponding to the area having the predetermined value as the focus target. For example, in the selected map, a subject corresponding to an area having a value equal to or greater than the threshold is determined as a focus target. A subject corresponding to a region indicates a subject appearing in that region.
- Information about the focus target determined by the focus target determination unit 35 is supplied to the lens drive control unit 36 .
- the lens drive control unit 36 drives the lens driver 14 to perform autofocus control so as to focus on the focus target indicated by the information output from the focus target determination unit 35 .
- selection conditions for the depth information for determining the processing object will be described.
- conditions for the case where the focus target determination unit 35 selects map-like information as the processing target determination depth information will be described. It is also applicable to selection of depth information for determining a processing target in the case of information of .
- the focus target determining unit 35 selects depth information for determining a processing target based on the reliability of the AiDepth map, for example.
- the depth information for determining the processing target is determined based on the comparison result between the reliability of the AiDepth information and the threshold.
- the reliability of the AiDepth map is an index of the certainty of the distance indicated by the AiDepth map.
- the reliability of the AiDepth map is an index that indicates whether the input image is an image that the inference model for distance inference is good at or not good at inferring the distance to the subject. Depending on how the subject is photographed, there are images that are good at inferring the distance using an inference model and images that are not so good at it.
- the focus target determination unit 35 determines the focus target using the AiDepth+MSD composite map obtained by combining the AiDepth maps. At this time, the focus target determining unit 35 selects the AiDepth map as depth information for determining the processing target.
- the focus target determination unit 35 determines the focus target using the ToF+MSD composite map combined with the ToF map. At this time, the focus target determining unit 35 has selected the ToF map as the depth information for determining the processing target.
- the reliability of the AiDepth map is, for example, a value based on the output when an input image is input to an inference model for reliability inference that infers the reliability of the AiDepth map.
- An inference model for reliability inference that is generated in advance by machine learning using a plurality of images for which the reliability of the AiDepth map is set is prepared in the imaging device 1 .
- the reliability of the AiDepth map is, for example, the AiDepth map generated when an image was input to the inference model for distance inference, the correct depth map of the same image, is determined based on the difference between
- the reliability of the AiDepth map may be, for example, a value calculated based on a value obtained by comparing the values of corresponding pixels in the two maps pixel by pixel and averaging the absolute values of the errors.
- the correct depth map may be, for example, an AiDepth map generated using an inference model for distance inference that requires a larger number of parameters and operations than the inference model for distance inference prepared in the imaging device 1.
- AiDepth where each pixel value is the distance to the subject at the position corresponding to each pixel, based on the output when an image is input to an inference model for distance inference using such a huge number of parameters. map is used as the correct depth map.
- a correct depth map may be, for example, a depth map based on the output of a distance sensor.
- a value that is an index of the reliability of the AiDepth map may be used to select the depth information for determining the processing target.
- Selecting the depth information for determining the processing target based on the value that is the index of the reliability of the AiDepth map is equivalent to selecting the depth information for determining the processing target based on the reliability of the AiDepth map.
- selecting the depth information for determining the processing target based on the comparison result between the value that is the reliability index of the AiDepth map and the threshold means This corresponds to selecting depth information for determination.
- the value that is an indicator of the reliability of the AiDepth map is, for example, the average absolute value of the error between the AiDepth map generated when an image is input to an inference model for distance inference and the correct depth map for the same image. It may be a converted value.
- the reliability of the AiDepth map is higher than the threshold, and when the average absolute value of the error is higher than the threshold, the reliability of the AiDepth map is It corresponds to when it is lower than the threshold.
- the focus target determination unit 35 determines the processing target based on the reliability of the depth information output by the AiDepth map generation unit 33. Determining depth information can be selected.
- the reliability of the depth information output by the AiDepth map generation unit 33 is generated, for example, using an inference model for reliability inference as described above.
- the depth information for determining the processing object is selected based on the reliability of the depth information output by the AiDepth map generation unit 33.
- the depth information output by the AiDepth map generation unit 33 can be selected as the depth information for determining the processing target. , it is possible to determine an appropriate subject as a focus target.
- the focus target determination unit 35 is an example of a processing target determination unit.
- the processing target determination unit determines a processing target in predetermined processing on the input image.
- the focus target determination unit 35 is an example of the processing target determination unit in which the predetermined process is the autofocus control process and the processing target is the focus target.
- the focus target determining unit 35 selects depth information for determining a processing target, for example, based on the reliability of the ToF map.
- the depth information for determining the processing target is determined based on the comparison result between the reliability of the ToF information and the threshold.
- the reliability of the ToF map is a measure of the likelihood of the distances indicated by the ToF map.
- the focus target determination unit 35 determines the focus target using the ToF+MSD composite map obtained by combining the ToF maps. At this time, the focus target determining unit 35 has selected the ToF map as the depth information for determining the processing target.
- the focus target determination unit 35 determines the focus target using the AiDepth+MSD composite map obtained by combining the AiDepth maps. At this time, the focus target determining unit 35 selects the AiDepth map as depth information for determining the processing target.
- the reliability of the ToF map is calculated, for example, based on the ratio (error rate) of pixels in error among all pixels on the ToF map.
- An error pixel is, for example, a pixel that does not have a value or has a value of 0 as distance information.
- Distance information such as a pixel whose brightness is higher than the threshold or a pixel whose brightness is lower than the threshold becomes an error. For example, the higher the error rate, the less reliable the ToF map.
- the focus target determination unit 35 determines the processing target based on the reliability of the depth information output by the ToF map generation unit 31. Determining depth information can be selected.
- the depth information for determining the processing target is selected based on the reliability of the depth information output by the ToF map generation unit 31.
- the depth information output by the ToF map generation unit 31 can be selected as the depth information for determining the processing target. , it is possible to determine an appropriate subject as a focus target.
- the reliability of the depth information output by the ToF map generation unit 31 or the AiDepth map generation unit 33 may be generated, for example, by the ToF map generation unit 31 or the AiDepth map generation unit 33 that generated the depth information. However, it may be generated by a reliability determination unit (not shown in FIG. 3) of the control unit 11 that is different from the ToF map generation unit 31 or the AiDepth map generation unit 33 .
- the reliability of depth information is input to the map synthesizing unit 34 in association with the depth information. Further, the reliability of the depth information is input from the map synthesizing unit 34 to the focus target determining unit 35 in association with the synthetic information generated by synthesizing the depth information and the main subject information. For example, the reliability of depth information may be directly input to the focus target determination unit 35 from a configuration that generates the reliability of depth information without going through the map synthesizing unit 34 .
- the focus target determining unit 35 selects depth information for determining a processing target based on, for example, the brightness of the subject included in the input image. . As the value indicating the brightness of the subject included in the input image decreases, the accuracy of the AiDepth map decreases.
- the focus target determination unit 35 determines the focus target using the AiDepth+MSD composite map obtained by combining the AiDepth maps. At this time, the focus target determining unit 35 selects the AiDepth map as depth information for determining the processing target.
- the focus target determination unit 35 determines the focus target using the ToF+MSD composite map, which is a composite of the ToF maps, when, for example, the value indicating the brightness of the subject included in the input image is lower than the threshold. At this time, the focus target determining unit 35 has selected the ToF map as the depth information for determining the processing target.
- the brightness of the subject included in the input image may be detected using an illuminance sensor (not shown in FIG. 2) of the imaging device 1, or may be detected using the brightness of the input image.
- the brightness of the input image is specified based on the luminance of each pixel.
- the depth information for determining the processing target is selected based on the brightness of the subject included in the input image.
- the depth information output by the ToF map generation unit 31 can be selected when the accuracy of the depth information output by the AiDepth map generation unit 33 is low. Therefore, it is possible to determine a subject that is subject to focus as a focus target.
- the focus object determination unit 35 selects processing object determination depth information based on, for example, the shooting mode when the input image was shot.
- a predetermined shooting mode is set according to various situations such as the brightness of the subject.
- the focus target determination unit 35 determines that the ToF A focus target is determined using the ToF+MSD composite map in which maps are composited. At this time, the focus target determining unit 35 has selected the ToF map as the depth information for determining the processing target.
- the focus target determination unit 35 determines that the shooting mode when the input image was shot is not the predetermined shooting mode set when the value indicating the brightness of the subject included in the input image is low.
- the focus target is determined using the AiDepth+MSD composite map in which the AiDepth map is composited.
- the focus target determining unit 35 selects the AiDepth map as depth information for determining the processing target.
- Predetermined shooting modes that are set when the value indicating the brightness of the subject included in the input image is low include, for example, night view mode, dark place mode, and night mode.
- the depth information for determining the processing object is selected based on the shooting mode when the input image was shot.
- the accuracy of the depth information output by the AiDepth map generation unit 33 is low, the depth information output by the ToF map generation unit 31 can be selected. It becomes possible to determine an appropriate subject as a focus target.
- Depth information for determining the processing target may be selected by combining the above conditions. That is, it is possible to select the depth information for determining the processing object based on at least one of the reliability of the AiDepth map, the reliability of the ToF map, the brightness of the subject, and the shooting mode. be.
- step S1 the control unit 11 acquires an input image.
- the input image is a preview image obtained by photographing with the photographing device 1 .
- Acquisition of an input image by the control unit 11 is synonymous with reception of image data output by the image sensor 13 .
- step S2 the ToF map generation unit 31 generates a ToF map based on sensor data indicating the measurement result of the ToF sensor.
- step S3 the MSD map generator 32 generates an MSD map based on the input image.
- step S4 the AiDepth map generator 33 generates an AiDepth map based on the input image.
- steps S2 to S4 are not only performed in the order shown in FIG. 4, but may be performed in any order. Alternatively, the processes of steps S2 to S4 may be performed simultaneously.
- step S5 the map synthesizing unit 34 generates a ToF+MSD synthetic map by synthesizing the ToF map and the MSD map, and generates an AiDepth+MSD synthetic map by synthesizing the AiDepth map and the MSD map.
- step S6 the focus target determination unit 35 performs focus target determination processing.
- the focus target determination process will be described later with reference to the flowchart of FIG.
- step S7 the lens drive control unit 36 drives the lens driver 14 based on the focus target indicated by the information output from the focus target determination unit 35, thereby performing autofocus control so that the focus target is focused. conduct.
- the above processing is repeated, for example, each time an input image is acquired.
- the photographing device 1 can perform autofocus control processing with an appropriate subject as the focus target.
- step S6 of FIG. 4 The focus target determination process performed in step S6 of FIG. 4 will be described with reference to the flowchart of FIG.
- the focus target determination unit 35 acquires the reliability of the AiDepth map. Acquisition of the reliability of the AiDepth map by the focus target determination unit 35 is synonymous with the focus target determination unit 35 receiving information indicating the reliability of the AiDepth map output from the AiDepth map generation unit 33 .
- the focus target determining unit 35 receiving the information indicating the reliability of the AiDepth map output from the AiDepth map generating unit 33 is merely receiving the information indicating the reliability of the AiDepth map directly from the AiDepth map generating unit 33. It also includes receiving via the map synthesizing unit 34 .
- step S12 the focus target determination unit 35 determines whether the reliability of the AiDepth map is equal to or greater than a threshold.
- step S13 the focus target determination unit 35 determines the focus target using the AiDepth+MSD combined map. That is, the focus target determination unit 35 selects the AiDepth map as depth information for processing target determination, and determines the focus target based on the AiDepth map and the MSD map. The focus target determination unit 35 determines, as a focus target, a subject corresponding to an area having a value equal to or greater than a threshold in the AiDepth+MSD composite map.
- step S14 the focus target determination unit 35 determines the focus target using the ToF+MSD combined map. That is, the focus target determination unit 35 selects the ToF map as depth information for processing target determination, and determines the focus target based on the ToF map and the MSD map. The focus target determination unit 35 determines, as a focus target, a subject corresponding to an area having a value equal to or greater than a threshold in the ToF+MSD composite map.
- the focus target in the input image is determined. After the focus target is determined in step S13 or S14, the process returns to step S6 in FIG. 4 and the subsequent processes are performed.
- FIG. 6 is a diagram illustrating another example of focus target determination processing in the imaging device 1 according to an embodiment of the present technology. Explanations overlapping with the explanations described above will be omitted as appropriate. The same applies to other figures to be described later.
- the input image P1 used for the focus target determination process is the same image as the input image P1 described with reference to FIG. Note that the input image P1 shown on the left end of FIG. 1 and the input image P3 shown on the right end are the same image including the same subject. For convenience of explanation, different numbers are assigned to the input image P1 and the input image P3.
- the focus target is determined based on the depth information selected from the two depth information. That is, processing such as map synthesis is not performed.
- a ToF map and an AiDepth map are generated corresponding to the input image P1.
- either the ToF map or the AiDepth map is selected as the depth information for determining the processing target.
- the depth information for determining the processing object is, for example, at least one of the reliability of the AiDepth map or ToF map, the brightness of the subject captured in the input image P1, and the shooting mode of the shooting device 1. selected based on
- the object O2 which is closer to the object O1 than the object O1, is the focus object, as indicated by the thick line on the input image P3. is determined as
- the depth map selected from the ToF map and AiDepth map is used to determine the focus target by an algorithm that prioritizes short distances.
- the focus target may be determined using, for example, an algorithm that determines a subject at a predetermined distance as the focus target, other than the short-distance priority algorithm.
- focus control is performed to bring the subject O2 into focus.
- the focus target is determined using the processing target determination depth information selected based on various conditions from the two depth information. Thereby, it is possible to select the depth information with higher accuracy or more certainty from the two pieces of depth information. Based on such highly accurate or probable information, it is possible to determine an appropriate subject as a focus target.
- FIG. 7 is a diagram illustrating another example of focus target determination processing in the imaging device 1 according to an embodiment of the present technology.
- the input image P1 used for the focus target determination process is the same image as the input image P1 described with reference to FIG.
- the input image P1 shown at the left end of FIG. 1 and the input image P4 shown at the right end of FIG. 1 are the same image including the same subject.
- the input image P1 and the input image P4 are given different numbers.
- the focus target is determined based on the depth information and the main subject information. That is, a process of selecting one depth information from two depth information is not performed.
- a ToF map and an MSD map are generated corresponding to the input image P1.
- a ToF + MSD composite map is generated by combining the ToF map and the MSD map.
- the subject O1 captured in the region having a higher value than the subject O2 in the ToF+MSD composite map is determined as the focus target. be done.
- focus control is performed to bring the subject O1 into focus.
- Determining the focus target using the composite map as shown in FIG. 7 means determining the focus target using both information on the distance to the subject and information on the main subject degree of the subject. By determining the focus target using both the distance information to the subject and the subject's main subject degree information, it is possible to focus on a more appropriate subject that takes into account the subject's type, size, position in the image, etc. It can be determined as a target.
- FIG. 8 is a diagram illustrating another example of focus target determination processing in the imaging device 1 according to an embodiment of the present technology.
- the input image P1 used for the focus target determination process is the same image as the input image P1 described with reference to FIG. Note that the input image P1 shown on the left end of FIG. 1 and the input image P5 shown on the right end are the same image including the same subject. For convenience of explanation, different numbers are assigned to the input image P1 and the input image P5.
- the focus target is determined using the depth information based on the output when the input image P1 is input to the inference model for distance inference. That is, map synthesis, processing for selecting one depth information from two depth information, and the like are not performed.
- an AiDepth map is generated corresponding to the input image P1.
- the subject O2 which is closer to the subject O1 than the subject O1, is determined as the focus target, as indicated by the thick line on the input image P5.
- the AiDepth map is used to determine the focus target by a short distance priority algorithm.
- the focus target may be determined using, for example, an algorithm that determines a subject at a predetermined distance as the focus target, other than the short-distance priority algorithm.
- focus control is performed to bring the subject O2 into focus.
- the focus target is determined using the depth information based on the output when the input image is input to the inference model for distance inference.
- the photographing apparatus 1 is not provided with a distance sensor, it is possible to determine an appropriate subject as a focus target.
- the present technology is applicable not only to autofocus control processing but also to tracking control processing.
- the tracking target determination process is performed by using an input image or the like to appropriately determine a tracking target, which is a subject to be tracked, from subjects included in the input image.
- Angle of view control is performed based on the determined tracking target.
- FIG. 9 is a block diagram showing a functional configuration example of the control unit 11 of the photographing device 1 when the present technology is applied to tracking control processing.
- the control unit 11 includes a ToF map generation unit 41, an MSD map generation unit 42, an AiDepth map generation unit 43, a map synthesis unit 44, a tracking target determination unit 45, and an angle of view control unit 46.
- the ToF map generation unit 41, the MSD map generation unit 42, the AiDepth map generation unit 43, and the map synthesis unit 44 are respectively the ToF map generation unit 31, the MSD map generation unit 32, the AiDepth map generation unit 33, and the map synthesis unit in FIG. Since it has the same function as that of 34, the explanation is omitted.
- the tracking target determination unit 45 determines the tracking target based on either the ToF+MSD combined map or the AiDepth+MSD combined map. For example, the tracking target determination unit 45 selects either one of the ToF+MSD combined map and the AiDepth+MSD combined map, and uses the selected map to determine the tracking target.
- the synthetic map selected by the tracking target determination unit 45 is a map used for determining the tracking target in the tracking control process for the input image.
- the tracking target determination unit 45 can select depth information for processing target determination.
- the depth information for determining the processing object is selected from the ToF map and the AiDepth map based on, for example, the reliability of the AiDepth map or ToF map, the brightness of the subject included in the input image, or the shooting mode of the imaging device 1. be done.
- the tracking target determination unit 45 is an example of a processing target determination unit that defines a predetermined process as a tracking control process and a processing target as a tracking target.
- the angle-of-view control unit 46 controls the angle of view so that the tracking target is included in the angle of view by performing PTZ driving and cropping processing based on the tracking target output from the tracking target determination unit 45 .
- steps S21 to S25 is the same as the processing of steps S1 to S5 in FIG. 4, respectively. That is, after a preview image or the like is obtained as an input image, a ToF map, an MSD map, and an AiDepth map are generated. Also, a ToF+MSD composite map is generated by combining the ToF map and the MSD map, and an AiDepth+MSD composite map is generated by combining the AiDepth map and the MSD map.
- the tracking target determination unit 45 performs tracking target determination processing.
- the tracking target determination process is a process for determining the tracking target instead of the focus target by the same process as the focus target determination process described with reference to the flowchart of FIG.
- step S27 the angle-of-view control unit 46 controls the angle of view so that the tracking target is included in the angle of view by performing PTZ driving and cropping processing based on the tracking target determined by the tracking target determination unit 45. .
- the above processing is repeated, for example, each time an input image is acquired.
- the imaging device 1 can perform tracking control processing with an appropriate subject as the tracking target.
- the present technology is applicable not only to autofocus control processing but also to background blur control processing.
- the background blurring control process is performed by appropriately determining an emphasis target, which is a subject to be emphasized, from subjects included in the input image using an input image or the like. Blur processing is performed based on the determined emphasis target.
- the background blurring control process is a process of making the emphasis target stand out by blurring the area corresponding to the background of the emphasis target. That is, it can be said that the emphasis target is the subject that is not subjected to the blurring process, and is the subject that is emphasized by the background blurring control process.
- FIG. 12 is a block diagram showing an example of the functional configuration of the control unit 11 of the photographing apparatus 1 when the present technology is applied to background blurring control processing.
- the control unit 11 includes a ToF map generation unit 51, an MSD map generation unit 52, an AiDepth map generation unit 53, a map synthesis unit 54, an emphasis target determination unit 55, and an image processing unit 56.
- the ToF map generation unit 51, the MSD map generation unit 52, the AiDepth map generation unit 53, and the map synthesis unit 54 are similar to the ToF map generation unit 31, the MSD map generation unit 32, the AiDepth map generation unit 33, and the map synthesis unit 34 in FIG. Since it has the same function, the explanation is omitted.
- the emphasis target determination unit 55 determines the emphasis target based on either the ToF+MSD composite map or the AiDepth+MSD composite map. For example, the emphasis target determination unit 55 selects either one of the ToF+MSD composite map and the AiDepth+MSD composite map as the map used to determine the emphasis target in the blur control process for the input image, and selects the selected map. is used to determine the object to be emphasized.
- the emphasis target determination unit 55 can select depth information for processing target determination.
- the depth information for determining the processing object is selected from the ToF map and the AiDepth map based on, for example, the reliability of the AiDepth map or ToF map, the brightness of the subject included in the input image, or the shooting mode of the imaging device 1. be done.
- the emphasis target determination unit 55 is an example of a processing target determination unit that designates a background blurring control process as a predetermined process and a processing target as an enhancement target.
- the image processing unit 56 performs blurring processing based on the enhancement target output from the enhancement target determination unit 55.
- the blurring process is a process of separating the area to be emphasized from the background area and blurring the background area.
- the separation of the area to be emphasized and the background area is performed, for example, based on the image recognition results.
- the region to be emphasized and the background region can be separated more accurately.
- depth information such as depth information based on parallax between a plurality of lenses and depth information for determining a processing target
- the region to be emphasized and the background region can be separated more accurately.
- depth information such as depth information for determining a processing target is used to change the amount of blur according to the distance to the subject, thereby improving the appearance of the photograph.
- steps S31 to S35 is the same as the processing of steps S1 to S5 in FIG. 4, respectively. That is, after a preview image or the like is obtained as an input image, a ToF map, an MSD map, and an AiDepth map are generated. Also, a ToF+MSD composite map is generated by combining the ToF map and the MSD map, and an AiDepth+MSD composite map is generated by combining the AiDepth map and the MSD map.
- the emphasis target determination unit 55 performs emphasis target determination processing.
- the emphasis target determination process is a process of determining a focus target instead of a focus target by the same processing as the focus target determination process described with reference to the flowchart of FIG. 5 .
- step S37 blurring processing is performed based on the emphasis target determined by the emphasis target determination unit 55.
- the blurring process is a process of separating the area to be emphasized from the background area and blurring the background area. The above processing is repeated, for example, each time an input image is acquired.
- the photographing device 1 can perform blur processing with an appropriate subject as an emphasis target.
- the process of determining the processing target in the above-described predetermined processing, including the focus target determination process, may be performed in an external device different from the imaging device 1 that captures the input image.
- the external device has the same configuration as the focus target determination unit 35, tracking target determination unit 45, and emphasis target determination unit 55 described with reference to FIGS. Both the imaging device 1 and the external device are information processing devices.
- Embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.
- this technology can take the configuration of cloud computing in which a single function is shared by multiple devices via a network and processed jointly.
- each step described in the flowchart above can be executed by a single device, or can be shared and executed by a plurality of devices.
- one step includes multiple processes
- the multiple processes included in the one step can be executed by one device or shared by multiple devices.
- An information processing apparatus comprising: a processing target determination unit that determines a processing target in predetermined processing on the input image using depth information for processing target determination selected from information.
- the processing target determining unit selects the depth information for determining the processing target based on the reliability of the second depth information.
- the processing target determination unit selects the processing target determination depth information based on a comparison result between the reliability of the second depth information and a threshold value.
- the reliability of the second depth information is a value based on an output when the input image is input to an inference model that infers the reliability of the depth information. .
- the processing target determination unit selects the processing target determination depth information based on brightness of a subject included in the input image.
- the processing target determination unit selects the second depth information as the processing target determination depth information when a value indicating brightness of the subject is equal to or greater than a threshold.
- the information processing apparatus selects the depth information for determining the processing target based on the reliability of the first depth information.
- the processing target determination unit selects the processing target determination depth information based on a shooting mode when the input image was captured.
- the processing target determination unit selects the first depth information as the processing target determination depth information when the shooting mode is one of a dark place mode, a night view mode, and a night mode.
- the processing target determination unit determines the processing target based on main subject degree information indicating a main subject degree, which is a degree of main subject-likeness of a subject included in the input image, and the processing target determination depth information.
- main subject degree information indicating a main subject degree, which is a degree of main subject-likeness of a subject included in the input image, and the processing target determination depth information.
- the information processing apparatus according to any one of (1) to (9).
- the main subject degree information is a main subject degree map in which the main subject degree is the pixel value of each pixel, and the processing target determination depth information is the pixel value of each pixel representing the distance to the subject of the input image.
- the processing target determination unit determines the processing target based on a composite map generated by combining the main subject intensity map and the processing target determination depth map. .
- the processing target determination unit determines, as the processing target, a subject corresponding to an area having a predetermined value in the composite map.
- the predetermined process is an autofocus control process
- the process target is a focus target that is a subject to be focused by the autofocus control process. processing equipment.
- the information processing apparatus according to any one of (1) to (13), wherein the predetermined processing is background blur control processing, and the processing target is an emphasis target that is a subject to be emphasized by the background blur control processing.
- the predetermined processing is tracking control processing, and the processing target is a tracking target that is a subject to be tracked in the tracking control processing.
- the information processing apparatus according to any one of (1) to (16), further comprising a photographing unit that photographs the input image.
- Each of the first depth information and the second depth information is a depth map in which the pixel value of each pixel is a value indicating the distance to a subject at a position corresponding to each pixel of the input image.
- the information processing device First depth information indicating the distance to the subject included in the input image based on the output of the distance sensor, and second depth information based on the output when the input image is input to an inference model for inferring the distance to the subject.
- to the computer First depth information indicating the distance to the subject included in the input image based on the output of the distance sensor, and second depth information based on the output when the input image is input to an inference model for inferring the distance to the subject.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Optics & Photonics (AREA)
- Studio Devices (AREA)
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US18/714,449 US12671898B2 (en) | 2021-12-09 | 2022-11-25 | Information processing device, information processing method, and program |
| DE112022005859.5T DE112022005859T5 (de) | 2021-12-09 | 2022-11-25 | Informationsverarbeitungsvorrichtung, informationsverarbeitungsverfahren und programm |
| CN202280079880.9A CN118369929A (zh) | 2021-12-09 | 2022-11-25 | 信息处理设备、信息处理方法和程序 |
| JP2023566225A JPWO2023106118A1 (cg-RX-API-DMAC7.html) | 2021-12-09 | 2022-11-25 |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2021-200038 | 2021-12-09 | ||
| JP2021200038 | 2021-12-09 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2023106118A1 true WO2023106118A1 (ja) | 2023-06-15 |
Family
ID=86730447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2022/043456 Ceased WO2023106118A1 (ja) | 2021-12-09 | 2022-11-25 | 情報処理装置、情報処理方法、およびプログラム |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US12671898B2 (cg-RX-API-DMAC7.html) |
| JP (1) | JPWO2023106118A1 (cg-RX-API-DMAC7.html) |
| CN (1) | CN118369929A (cg-RX-API-DMAC7.html) |
| DE (1) | DE112022005859T5 (cg-RX-API-DMAC7.html) |
| WO (1) | WO2023106118A1 (cg-RX-API-DMAC7.html) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH095611A (ja) * | 1995-06-14 | 1997-01-10 | Fuji Photo Optical Co Ltd | 測距装置 |
| JP2016059051A (ja) * | 2015-10-23 | 2016-04-21 | リコーイメージング株式会社 | 撮像装置および距離情報取得方法 |
| JP2019124537A (ja) * | 2018-01-15 | 2019-07-25 | キヤノン株式会社 | 情報処理装置及びその制御方法及びプログラム、並びに、車両の運転支援システム |
| JP2019159914A (ja) * | 2018-03-14 | 2019-09-19 | キヤノンメディカルシステムズ株式会社 | 信号復元装置、信号復元方法、信号復元プログラム、モデル学習方法、およびモデル学習プログラム |
| WO2020059565A1 (ja) * | 2018-09-18 | 2020-03-26 | パナソニックIpマネジメント株式会社 | 奥行取得装置、奥行取得方法およびプログラム |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4500434B2 (ja) | 2000-11-28 | 2010-07-14 | キヤノン株式会社 | 撮像装置及び撮像システム、並びに撮像方法 |
| US20070237514A1 (en) * | 2006-04-06 | 2007-10-11 | Eastman Kodak Company | Varying camera self-determination based on subject motion |
| US10407047B2 (en) * | 2015-12-07 | 2019-09-10 | Magna Electronics Inc. | Vehicle control system with target vehicle trajectory tracking |
| US20210233115A1 (en) * | 2018-10-17 | 2021-07-29 | Firefly Systems Inc. | Vehicle-mounted dynamic content delivery systems |
| JP7400222B2 (ja) * | 2019-06-14 | 2023-12-19 | マツダ株式会社 | 外部環境認識装置 |
| JP7363118B2 (ja) * | 2019-06-14 | 2023-10-18 | マツダ株式会社 | 外部環境認識装置 |
| US10783559B1 (en) * | 2019-10-06 | 2020-09-22 | Bao Tran | Mobile information display platforms |
| JP2023178092A (ja) * | 2022-06-03 | 2023-12-14 | キヤノン株式会社 | 情報処理装置、撮像装置、情報処理方法および撮像装置の制御方法 |
-
2022
- 2022-11-25 JP JP2023566225A patent/JPWO2023106118A1/ja active Pending
- 2022-11-25 DE DE112022005859.5T patent/DE112022005859T5/de active Pending
- 2022-11-25 US US18/714,449 patent/US12671898B2/en active Active
- 2022-11-25 WO PCT/JP2022/043456 patent/WO2023106118A1/ja not_active Ceased
- 2022-11-25 CN CN202280079880.9A patent/CN118369929A/zh active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH095611A (ja) * | 1995-06-14 | 1997-01-10 | Fuji Photo Optical Co Ltd | 測距装置 |
| JP2016059051A (ja) * | 2015-10-23 | 2016-04-21 | リコーイメージング株式会社 | 撮像装置および距離情報取得方法 |
| JP2019124537A (ja) * | 2018-01-15 | 2019-07-25 | キヤノン株式会社 | 情報処理装置及びその制御方法及びプログラム、並びに、車両の運転支援システム |
| JP2019159914A (ja) * | 2018-03-14 | 2019-09-19 | キヤノンメディカルシステムズ株式会社 | 信号復元装置、信号復元方法、信号復元プログラム、モデル学習方法、およびモデル学習プログラム |
| WO2020059565A1 (ja) * | 2018-09-18 | 2020-03-26 | パナソニックIpマネジメント株式会社 | 奥行取得装置、奥行取得方法およびプログラム |
Also Published As
| Publication number | Publication date |
|---|---|
| DE112022005859T5 (de) | 2024-10-02 |
| CN118369929A (zh) | 2024-07-19 |
| US20250024145A1 (en) | 2025-01-16 |
| US12671898B2 (en) | 2026-06-30 |
| JPWO2023106118A1 (cg-RX-API-DMAC7.html) | 2023-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107948519B (zh) | 图像处理方法、装置及设备 | |
| JP5048468B2 (ja) | 撮像装置およびその撮像方法 | |
| CN103595979B (zh) | 图像处理设备、图像拍摄设备及图像处理方法 | |
| JP4497211B2 (ja) | 撮像装置、撮像方法及びプログラム | |
| JP5390707B2 (ja) | 立体パノラマ画像合成装置、撮像装置並びに立体パノラマ画像合成方法、記録媒体及びコンピュータプログラム | |
| JP6512810B2 (ja) | 撮像装置および制御方法とプログラム | |
| US8854528B2 (en) | Imaging apparatus | |
| JP6838994B2 (ja) | 撮像装置、撮像装置の制御方法およびプログラム | |
| US8648961B2 (en) | Image capturing apparatus and image capturing method | |
| JP2011205282A (ja) | 撮像装置及び画像処理方法 | |
| JP2015012480A (ja) | 画像処理装置及び画像処理方法 | |
| JP5263310B2 (ja) | 画像生成装置、撮像装置、画像生成方法 | |
| US12333752B2 (en) | Image processing apparatus and method, and image capturing apparatus and control method thereof, and storage medium | |
| JP2020107956A (ja) | 撮像装置、撮像方法およびプログラム | |
| US10412321B2 (en) | Imaging apparatus and image synthesis method | |
| JP4710983B2 (ja) | 画像合成装置、撮像装置、画像合成方法 | |
| US20150264249A1 (en) | Image processing apparatus and image processing method | |
| US10447937B2 (en) | Image processing apparatus, imaging apparatus, image processing method, and storage medium that perform image processing based on an image processing parameter set for a first object area, and information on a positional relationship between an object in a second object area and an object in the first object area | |
| CN106878604A (zh) | 一种基于电子设备的图像生成的方法和电子设备 | |
| KR101630307B1 (ko) | 디지털 촬영 장치, 그 제어 방법, 및 컴퓨터 판독가능 저장매체 | |
| JP2020046475A (ja) | 画像処理装置およびその制御方法 | |
| US10943328B2 (en) | Image capturing apparatus, method for controlling same, and storage medium | |
| WO2013005477A1 (ja) | 撮像装置、立体画像撮像方法及びプログラム | |
| US20250024145A1 (en) | Information processing device, information processing method, and program | |
| JP2014131188A (ja) | 撮像装置、その制御方法、および制御プログラム |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22904049 Country of ref document: EP Kind code of ref document: A1 |
|
| ENP | Entry into the national phase |
Ref document number: 2023566225 Country of ref document: JP Kind code of ref document: A |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 18714449 Country of ref document: US |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 202280079880.9 Country of ref document: CN |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 22904049 Country of ref document: EP Kind code of ref document: A1 |
|
| WWG | Wipo information: grant in national office |
Ref document number: 18714449 Country of ref document: US |