Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/71—Circuitry for evaluating the brightness variation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/72—Combination of two or more compensation controls
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/74—Circuitry for compensating brightness variation in the scene by influencing the scene brightness using illuminating means
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/70—Circuitry for compensating brightness variation in the scene
  • H04N23/75—Circuitry for compensating brightness variation in the scene by influencing optical camera components

Definitions

• the technology of the present disclosure relates to an imaging support device, an imaging device, an imaging support method, and a program.
• JP 2014-176065A discloses an imaging device.
• the imaging device described in Japanese Unexamined Patent Publication No. 2014-176065 includes a pixel circuit that outputs a pixel signal having a signal level according to the exposure amount by non-destructive readout, and a photoelectric circuit in which a plurality of pixel circuits are arranged in a two-dimensional matrix. A conversion unit is provided. Further, the imaging device described in Japanese Patent Application Laid-Open No.
• 2014-176065 includes a row decoder that resets a plurality of pixel circuits on a row-by-row basis and selects a plurality of pixel circuits that output pixel signals on a row-by-row basis; It includes a plurality of A/D converters that perform A/D conversion to generate pixel data, and an image data generation section.
• the image data generation unit generates, in each of the plurality of pixel circuits, pixel data generated at a first point in time after reset, and pixel data generated at a point in time when a first exposure time has elapsed from the first point in time.
• the first image data is generated by calculating the difference between the images.
• the image data generation unit also generates pixel data generated at a second time point that is later than the first time point, and a point in time when a second exposure time that is longer than the first exposure time has elapsed from the second time point.
• the second image data is generated by calculating the difference between the pixel data and the pixel data generated in .
• JP 2022-007039A discloses an imaging device.
• the imaging device described in Japanese Unexamined Patent Publication No. 2022-007039 includes a light source and an imaging element that images a subject using reflected light that is light emitted from the light source and reflected from the subject. Furthermore, the imaging device described in Japanese Patent Application Laid-open No. 2022-007039 controls the direction in which light is emitted from the light source, so that the part of the optical path of reflected light traveling from the subject to the image sensor is not emitted from the light source.
• the apparatus includes a control unit that reduces the portion through which the emitted light, which is the emitted light, passes through the traveling region. The control unit controls the light emission timing of the light source and the exposure timing of the image sensor to obtain an image of the subject captured by the image sensor.
• Japanese Patent Laid-Open No. 2021-027409 discloses a circuit configured to set an upper limit value of the exposure time and determine the exposure time of the imaging device within a range below the upper limit time based on the exposure control value of the imaging device.
• a control device is disclosed.
• One embodiment of the technology of the present disclosure generates an image with less uneven brightness for an image sensor even when a difference in brightness occurs in one direction in the photoelectric conversion region due to light incident on the photoelectric conversion region.
• An imaging support device, an imaging device, an imaging support method, and a program are provided.
• a first aspect of the technology of the present disclosure includes a processor that controls an exposure time for a photoelectric conversion area of an image sensor having a photoelectric conversion area in which a plurality of pixels are arranged two-dimensionally, and the processor controls the exposure time of a photoelectric conversion area.
• the photoelectric conversion area extends from the dark side to the bright side of the plurality of divided areas divided along one direction. This is an imaging support device that performs control to shorten exposure time.
• a second aspect according to the technology of the present disclosure is the imaging support device according to the first aspect, in which the segmented area is an area in which pixels are arranged linearly in a direction intersecting one direction.
• a third aspect of the technology of the present disclosure is that the incident light includes reflected light obtained when auxiliary light emitted from a lighting device used in imaging using an image sensor is reflected by a subject.
• This is an imaging support device according to a certain first aspect or second aspect.
• a fourth aspect of the technology of the present disclosure is that when a mechanical shutter and/or an electronic shutter is used in imaging using an image sensor, the processor controls the mechanical shutter and/or the electronic shutter to control the photoelectric conversion region.
• An imaging support device according to any one of the first to third aspects, which controls exposure time.
• the processor controls exposure start timing and/or exposure end timing for the plurality of segmented areas, thereby changing the segmentation from the dark side to the bright side of the plurality of segmented areas.
• An imaging support device according to any one of the first to fourth aspects, in which the exposure time is shortened over a region.
• a sixth aspect of the technology of the present disclosure is that the processor delays the exposure start timing for the plurality of segmented areas from the darker segmented area to the brighter segmented area, and
• This is an imaging support device according to a fifth aspect that performs control to match exposure end timings for divided regions.
• a seventh aspect of the technology of the present disclosure is the imaging support device according to the sixth aspect, in which the processor performs control to match the exposure end timings for a plurality of segmented areas using a global shutter method.
• An eighth aspect of the technology of the present disclosure is that the processor matches the exposure start timings of the plurality of divided regions, and adjusts the exposure end timing of the plurality of divided regions to the bright side of the plurality of divided regions.
• This is an imaging support device according to a fifth aspect, which performs slowing control from the area to the dark side segmented area.
• a ninth aspect according to the technology of the present disclosure is the imaging support device according to the eighth aspect, in which the processor performs control to match the exposure start timings for a plurality of segmented areas using a global shutter method.
• a tenth aspect of the technology of the present disclosure is that the processor delays the exposure start timing for the plurality of segmented areas from the darker segmented area to the brighter segmented area, and The exposure end timing is delayed from the dark side to the bright side of the multiple divided areas, and the exposure time is shortened from the dark side to the bright side of the multiple divided areas.
• An eleventh aspect of the technology of the present disclosure is that the exposure time for each segmented area from the dark side segmented area to the bright side segmented area of the plurality of segmented areas is determined according to the first degree of difference;
• the degree of difference is the degree of difference between the signal level of the first reference region in the photoelectric conversion region and the plurality of signal levels obtained from the plurality of segmented regions when the exposure time for the photoelectric conversion region is made shorter than the first reference exposure time.
• a twelfth aspect according to the technology of the present disclosure is the imaging support device according to the eleventh aspect, in which the first reference exposure time is less than the time at which the signal level of the first reference area is saturated.
• a thirteenth aspect of the technology of the present disclosure is such that the exposure time for each segmented area from the darker segmented area to the brighter segmented area of the plurality of segmented areas is determined according to the second degree of difference;
• the degree of difference is the degree of difference between the signal level of the second reference region in the photoelectric conversion region and the plurality of signal levels obtained from the plurality of segmented regions when the exposure time for the photoelectric conversion region is the first exposure time,
• the exposure time for each segmented area from the dark side segmented area to the bright side segmented area of the plurality of segmented areas is adjusted to a time in which the plurality of signal levels fall within a reference range.
• a fourteenth aspect according to the technology of the present disclosure is that when the imaging range is changed in imaging using an image sensor, from the dark side to the bright side of the plurality of divided areas before the imaging range is changed.
• the exposure time for each divided area over the area is determined according to the third degree of difference, and the third degree of difference is a third criterion within the photoelectric conversion area when the exposure time for the photoelectric conversion area is the second exposure time. It is the degree of difference between the signal level of a region and multiple signal levels obtained from multiple segmented regions, and it is the degree of difference between the signal level of a region and multiple signal levels obtained from multiple segmented regions, and the segmentation from the dark side segmented area to the bright side segmented area of multiple segmented areas after the imaging range is changed.
• the exposure time for each region is determined according to the second reference exposure time and the third degree of difference determined for the third reference region. This is such an imaging support device.
• a fifteenth aspect of the technology of the present disclosure is that when a flash is used in synchronization with the timing when imaging using an image sensor is performed, the division from the dark side to the bright side of the plurality of divided areas is provided.
• the exposure time for each area is determined according to the fourth degree of dissimilarity, and the fourth degree of dissimilarity is the third exposure time determined according to the flash for the exposure time for the photoelectric conversion area.
• the imaging support device according to any one of the first to tenth aspects, which is the degree of difference between the signal level of the fourth reference region and the plurality of signal levels obtained from the plurality of segmented regions.
• a 16th aspect according to the technology of the present disclosure is according to the 15th aspect, in which, when the aperture is adjusted in imaging using an image sensor, the third exposure time is determined according to the values of the flash and the aperture. It is an imaging support device.
• a seventeenth aspect of the technology of the present disclosure provides that when the exposure time for the photoelectric conversion area is determined according to the moving speed of the focal plane shutter, the second exposure time in the photoelectric conversion area when the photoelectric conversion area is exposed for a fourth exposure time is Any one of the first to sixteenth aspects, wherein the moving speed is determined based on the result of regression analysis based on the signal level of the five reference areas and the plurality of signal levels obtained from the plurality of segmented areas.
• 1 is an imaging support device according to one embodiment.
• An eighteenth aspect according to the technology of the present disclosure is an imaging device including the imaging support device according to any one of the first to seventeenth aspects and an image sensor.
• a nineteenth aspect of the technology of the present disclosure is to expose a photoelectric conversion area of an image sensor having a photoelectric conversion area in which a plurality of pixels are arranged two-dimensionally, and to perform photoelectric conversion by light incident on the photoelectric conversion area. Control that shortens the exposure time from the dark side to the bright side of multiple divided areas in which the photoelectric conversion area is divided along one direction when a difference in brightness occurs in one direction.
• This is an imaging support method including performing.
• a 20th aspect of the technology of the present disclosure provides a computer that controls an exposure time for a photoelectric conversion area of an image sensor having a photoelectric conversion area in which a plurality of pixels are arranged two-dimensionally.
• the exposure time is shortened from the dark side to the bright side of the multiple divided areas in which the photoelectric conversion area is divided along one direction.
• FIG. 2 is a front view showing an example of an imaging system according to first to sixth embodiments.
• 7 is a conceptual diagram showing an example of a manner in which a subject is imaged by an imaging device included in an imaging system according to a first to a sixth embodiment;
• FIG. FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device included in the imaging system according to the first embodiment.
• FIG. 2 is a conceptual diagram showing an example of a manner in which the exposure time of each pixel column of the photoelectric conversion region of the image sensor included in the imaging device according to the first embodiment is controlled.
• 7 is a flowchart illustrating an example of the flow of exposure time control processing according to the first embodiment.
• FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device included in the imaging system according to the first embodiment.
• FIG. 2 is a conceptual diagram showing an example of a manner in which the exposure time of each pixel column of the photoelectric conversion region of the image sensor included in the imaging device according to the
• FIG. 7 is a conceptual diagram illustrating a first modified example of the manner in which the exposure time of each pixel column of the photoelectric conversion region of the image sensor included in the imaging device according to the first embodiment is controlled.
• FIG. 7 is a conceptual diagram showing a second modified example of the manner in which the exposure time of each pixel column of the photoelectric conversion region of the image sensor included in the imaging device according to the first embodiment is controlled.
• FIG. 2 is a block diagram showing an example of the configuration of a controller included in an imaging device according to second to fifth embodiments.
• 7 is a flowchart illustrating an example of the flow of exposure time determination processing according to the second embodiment.
• 12 is a flowchart illustrating an example of the flow of exposure time determination processing according to the third embodiment.
• FIG. 12 is a flowchart illustrating an example of the flow of exposure time determination processing according to the fourth embodiment. This is a continuation of the flowchart shown in FIG. 11A. 13 is a flowchart illustrating an example of the flow of exposure time determination processing according to the fifth embodiment. It is a flowchart which shows the modification of the flow of exposure time determination processing based on 5th Embodiment.
• FIG. 12 is a block diagram illustrating an example of the configuration of a controller included in an imaging device according to a sixth embodiment. 12 is a flowchart illustrating an example of the flow of shutter speed determination processing according to the sixth embodiment.
• FIG. 7 is a conceptual diagram showing an example of processing contents of a processor according to a sixth embodiment.
• CPU is an abbreviation for "Central Processing Unit”.
• GPU is an abbreviation for “Graphics Processing Unit.”
• TPU is an abbreviation for “Tensor Processing Unit”.
• HDD is an abbreviation for “Hard Disk Drive.”
• SSD is an abbreviation for “Solid State Drive.”
• RAM is an abbreviation for "Random Access Memory.”
• NVM is an abbreviation for "Non-Volatile Memory.”
• ASIC is an abbreviation for “Application Specific Integrated Circuit.”
• FPGA is an abbreviation for "Field-Programmable Gate Array.”
• PLD is an abbreviation for “Programmable Logic Device”.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• CCD is an abbreviation for “Charge Coupled Device”.
• SoC is an abbreviation for "System-on-a-chip.”
• UI is an abbreviation for “User Interface”.
• EL is an abbreviation for "Electro Luminescence”.
• vertical means not only completely vertical but also perpendicular to a degree that is generally accepted in the technical field to which the technology of the present disclosure belongs and does not go against the spirit of the technology of the present disclosure. It refers to vertical in the sense of including the error of.
• orthogonality refers to not only complete orthogonality, but also orthogonality that is generally accepted in the technical field to which the technology of the present disclosure belongs, to the extent that it does not go against the spirit of the technology of the present disclosure. Refers to orthogonality in the sense of including the error of.
• match refers to not only a complete match but also a match that is generally accepted in the technical field to which the technology of the present disclosure belongs, and to the extent that it does not go against the spirit of the technology of the present disclosure. Refers to agreement in the sense of including errors.
• an imaging system 10 includes a moving body 12 and an imaging device 14.
• the imaging system 10 is connected to a communication device (not shown) for wireless communication, and various information is exchanged wirelessly between the imaging system 10 and the communication device.
• the operation of imaging system 10 is controlled by a communication device.
• An example of the moving body 12 is an unmanned moving body.
• an unmanned aircraft for example, a drone
• the moving body 12 may be a vehicle. Examples of the vehicle include a vehicle with a gondola, an aerial work vehicle, a bridge inspection vehicle, and the like.
• the moving body 12 may be a slider, a cart, or the like on which the imaging device 14 can be mounted.
• the moving object 12 may be a person. The person in this case refers to, for example, a worker who carries around the imaging device 14 and operates the imaging device 14 to survey and/or inspect land and/or infrastructure.
• the moving body 12 includes a main body 16 and a plurality of propellers 18 (four propellers in the example shown in FIG. 1).
• the moving body 12 flies or hovers in a three-dimensional space by controlling the rotation of the plurality of propellers 18.
• An imaging device 14 is attached to the main body 16.
• the imaging device 14 is attached to the top of the main body 16.
• the imaging device 14 may be attached to a location other than the top of the main body 16 (for example, the bottom of the main body 16).
• the imaging system 10 has an X axis, a Y axis, and a Z axis defined.
• the X-axis is an axis along the front-back direction of the moving body 12
• the Y-axis is an axis along the left-right direction of the moving body 12
• the Z-axis is an axis along the vertical direction, that is, the X-axis and This is an axis perpendicular to the Y axis.
• the direction along the X axis will be referred to as the X direction
• the direction along the Y axis will be referred to as the Y direction
• the direction along the Z axis will be referred to as the Z direction.
• one direction of the X-axis ie, the front of the moving body 12
• the other direction of the X-axis ie, the rear of the moving body 12
• the -X direction one direction of the X-axis
• the +Y direction one direction of the Y-axis (that is, the right side of the moving body 12 when viewed from the front)
• the +Y direction one direction of the Y-axis
• the direction of (that is, the left side of the moving body 12 when viewed from the front) is defined as the ⁇ Y direction. Furthermore, one direction of the Z-axis (ie, above the moving body 12) is defined as the +Z direction, and the other direction of the Z-axis (ie, below the moving body 12) is defined as the -Z direction.
• the imaging device 14 includes an imaging device main body 20 and lighting equipment 22.
• the imaging device 14 is an example of an "imaging device” according to the technology of the present disclosure
• the lighting device 22 is an example of a "lighting device” according to the technology of the present disclosure.
• the imaging device main body 20 includes an imaging lens 24 and an image sensor 26.
• An example of the imaging lens 24 is an interchangeable lens.
• an example of the image sensor 26 is a CMOS image sensor.
• the imaging lens 24 may be a non-interchangeable lens.
• CMOS image sensor is cited as an example of the image sensor 26, but this is just an example, and the imaging lens 24 may be of another type of image sensor (for example, a CCD image sensor). good.
• the imaging lens 24 has an optical axis OA that coincides with the X-axis.
• the center of the image sensor 26 is located on the optical axis OA of the imaging lens 24.
• the imaging lens 24 takes in object light, which is light indicating the object 27 , and forms an image of the taken object 27 on the image sensor 26 .
• the image sensor 26 receives object light and images the object 27 by photoelectrically converting the received object light.
• the lighting equipment 22 is arranged on the +Y direction side with respect to the imaging device main body 20.
• the lighting device 22 is used for imaging using the image sensor 26 and emits auxiliary light 28 .
• the auxiliary light 28 is light for compensating for a lack of light amount when the image sensor 26 takes an image, and is irradiated onto the subject 27 side.
• the reflected light obtained when the auxiliary light 28 emitted from the lighting equipment 22 is reflected by the subject 27 is received by the image sensor 26 and photoelectrically converted.
• a captured image 29 is generated by the image sensor 26.
• the captured image 29 becomes brighter than when the auxiliary light 28 is not irradiated onto the subject 27 side.
• the auxiliary light 28 is an example of "auxiliary light" according to the technology of the present disclosure.
• the imaging device 14 moves in the same direction as the flight direction of the moving object 12 (+Y direction in the example shown in FIG. 1), and captures the subject 27 at each of a plurality of designated positions (for example, a plurality of waypoints). Take an image.
• Examples of the subject 27 imaged by the imaging device 14 include land and/or infrastructure.
• Examples of infrastructure include road facilities (e.g., bridges, road surfaces, tunnels, guardrails, traffic lights, and/or windbreak fences), waterway facilities, airport facilities, port facilities, water storage facilities, gas facilities, power supply facilities, and medical facilities. Examples include equipment and/or firefighting equipment.
• the imaging device 14 irradiates auxiliary light 28 from the illumination device 22 toward the subject 27, and in this state images an imaging range 36 within the subject 27.
• the imaging range 36 is a range determined from the angle of view set for the imaging device main body 20.
• the image sensor 26 includes a photoelectric conversion element 30.
• the photoelectric conversion element 30 has a photoelectric conversion region 32.
• the photoelectric conversion area 32 is formed by a plurality of pixels 34.
• the plurality of pixels 34 are arranged two-dimensionally (ie, in a matrix).
• Each pixel 34 includes a color filter, a photodiode, a capacitor, and the like.
• the pixel 34 receives subject light, generates an analog image signal by performing photoelectric conversion on the received subject light, and outputs the generated analog image signal.
• the analog image signal is an electrical signal having a signal level corresponding to the amount of light received by the photodiode of the pixel 34.
• the photoelectric conversion region 32 is an example of a "photoelectric conversion region” according to the technology of the present disclosure
• the plurality of pixels 34 is an example of "a plurality of pixels” according to the technology of the present disclosure.
• the number of pixels 34 in the photoelectric conversion area 32 is smaller than in reality for convenience of illustration, but the actual number of pixels in the photoelectric conversion area 32 is, for example, several million to several. It's 10,000,000.
• the photoelectric conversion region 32 has a plurality of pixel columns 33.
• the plurality of pixel columns 33 are obtained by dividing the photoelectric conversion region 32 into a plurality of columns along the +Y direction.
• the pixel row 33 is an area in which a plurality of pixels 34 are linearly arranged in the Z direction, which is a direction intersecting the Y direction.
• the pixel row 33 is formed by a plurality of pixels 34 arranged at equal intervals along the Z direction.
• the plurality of pixel columns 33 are arranged at equal intervals along the Y direction.
• Analog image signals are read out from the photoelectric conversion region 32 in units of pixel columns 33 from the +Y direction side to the -Y direction side.
• the plurality of pixel columns 33 are an example of "the plurality of segmented areas" according to the technology of the present disclosure.
• the incident light to the photoelectric conversion region 32 refers to light including reflected light obtained by the auxiliary light 28 emitted from the lighting device 22 being reflected by the subject 27, for example.
• the illumination device 22 disposed on the +Y direction side of the imaging device main body 20 irradiates the imaging range 36 located in front of the imaging device main body 20 with the auxiliary light 28.
• a difference in brightness occurs in the range 36 along the Y direction.
• a difference in brightness occurs in the imaging range 36 such that the side closer to the lighting equipment 22 is brighter and the side farther from the lighting equipment 22 is darker.
• the difference in brightness that occurs in the imaging range 36 is also reflected in the photoelectric conversion area 32.
• the photoelectric conversion region 32 gradually becomes darker from the +Y direction side to the ⁇ Y direction side due to the incident light on the photoelectric conversion region 32.
• the Y direction is an example of "one direction" according to the technology of the present disclosure.
• the exposure time control process is performed by the imaging device 14.
• Exposure time control processing is performed to control a plurality of pixel columns 33 (for example, all pixel columns 33 ) is a process in which control is performed to shorten the exposure time from the pixel row 33 on the dark side to the pixel row 33 on the bright side. This will be explained in more detail below.
• the imaging device main body 20 includes an image sensor 26, a controller 38, an image memory 40, a UI device 42, a shutter driver 44, an actuator 46, a focal plane shutter 48, and a photoelectric conversion element driver 51.
• the imaging lens 24 includes an optical system 52 and an optical system driving device 54.
• the image sensor 26 includes a signal processing circuit 56.
• a controller 38 Connected to the input/output interface 50 are a controller 38, an image memory 40, a UI device 42, a shutter driver 44, a photoelectric conversion element driver 51, an optical system drive device 54, and a signal processing circuit 56.
• the controller 38 includes a processor 58, an NVM 60, and a RAM 62. Input/output interface 50, processor 58, NVM 60, and RAM 62 are connected to bus 64.
• the controller is an example of an "imaging support device” and a “computer” according to the technology of the present disclosure
• the processor 58 is an example of a "processor” according to the technology of the present disclosure.
• the processor 58 has a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for executing image processing.
• the processor 58 may be one or more CPUs with integrated GPU functionality, or may be one or more CPUs without integrated GPU functionality. Further, the processor 58 may include a multi-core CPU or a TPU.
• the NVM 60 is a nonvolatile storage device that stores various programs, various parameters, and the like.
• Examples of the NVM 60 include flash memory (eg, EEPROM) and/or SSD. Note that the flash memory and the SSD are merely examples, and may be other non-volatile storage devices such as an HDD, or a combination of two or more types of non-volatile storage devices.
• the RAM 62 is a memory in which information is temporarily stored, and is used by the processor 58 as a work memory.
• the processor 58 reads the necessary program from the NVM 50 and executes the read program on the RAM 62.
• the processor 58 controls the entire imaging device 14 according to a program executed on the RAM 62.
• the optical system 52 within the imaging lens 24 includes a zoom lens 52A, a lens shutter 52B, and an aperture 52C.
• Optical system 52 is connected to optical system driver 54, which operates optical system 52 (e.g., zoom lens 52A, lens shutter 52B, and aperture 52C) under the control of processor 58.
• the focal plane shutter 48 is arranged between the optical system 52 and the photoelectric conversion region 32.
• the focal plane shutter 48 has a leading curtain and a trailing curtain.
• the leading and trailing curtains of the focal plane shutter are mechanically connected to an actuator 46.
• Actuator 46 has a power source (eg, a solenoid).
• the shutter driver 44 is connected to the actuator 46 and controls the actuator 46 according to instructions from the processor 58.
• the actuator 46 generates power under the control of the shutter driver 44 and selectively applies the generated power to the leading and trailing curtains of the focal plane shutter 48 . control opening and closing.
• the leading and trailing curtains of the focal plane shutter 48 move from the +Y direction side to the ⁇ Y direction side.
• the leading curtain starts moving before the trailing curtain.
• the front curtain is fully closed before the start of exposure, and when the exposure start timing arrives, it is fully opened by moving from the +Y direction side to the -Y direction side.
• the trailing curtain is fully opened before the start of exposure, and when the exposure start timing arrives, it is fully closed by moving from the +Y direction side to the -Y direction side.
• the exposure time of the pixel row 33 is controlled by the width of the gap created between the front curtain and the rear curtain and the shutter speed of the focal plane shutter 48.
• a photoelectric conversion element driver 51 is connected to the photoelectric conversion element 30.
• the photoelectric conversion element driver 51 supplies an imaging timing signal that defines the timing of imaging performed by the photoelectric conversion element 30 to the photoelectric conversion element 30 according to instructions from the processor 58 .
• the photoelectric conversion element 30 performs reset, exposure, and output of an analog image signal according to the imaging timing signal supplied from the photoelectric conversion element driver 51. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.
• the subject light incident on the imaging lens 24 is imaged onto the photoelectric conversion region 32 by the imaging lens 24.
• the photoelectric conversion element 30 photoelectrically converts the subject light received by the photoelectric conversion area 32 under the control of the photoelectric conversion element driver 51, and converts an electrical signal corresponding to the amount of the subject light into an analog image signal indicating the subject light. It is output to the signal processing circuit 56.
• the signal processing circuit 56 reads analog image signals from the photoelectric conversion element 30 frame by frame and for each pixel column 33 using a line exposure sequential readout method.
• the signal processing circuit 56 generates a captured image 29 by digitizing the analog image signal input from the photoelectric conversion element 30, and stores the generated captured image 29 in the image memory 40.
• the processor 58 acquires the captured image 29 from the image memory 40 and executes various processes using the acquired captured image 29.
• the UI device 42 includes a display device and a reception device.
• Examples of the display device include an EL display or a liquid crystal display.
• Examples of the reception device include a touch panel, hard keys, and/or a dial.
• the processor 58 operates according to various instructions received by the UI device 42.
• the processor 58 also displays the results of various processes on a UI device.
• the lighting equipment 22 includes a light source 66 and a light source driver 68.
• a light source driver 68 is connected to the light source 66 .
• Light source driver 68 is connected to input/output interface 50 and controls light source 66 according to instructions from processor 58.
• the light source 66 emits visible light (here, white light as an example) under the control of the light source driver 68.
• the visible light emitted from the light source 66 is irradiated onto the subject 27 (see FIGS. 1 and 2) as auxiliary light 28.
• An exposure time control program 70 is stored in the NVM 60.
• the processor 58 reads the exposure time control program 70 from the NVM 60 and executes the read exposure time control program 70 on the RAM 62.
• the exposure time control process is realized by the processor 58 executing the exposure time control program 70 on the RAM 62.
• the exposure time control program 70 is an example of a "program" according to the technology of the present disclosure.
• the focal plane shutter 48 and the lens shutter 52B are used in imaging using the image sensor 26.
• the processor 58 resets the photoelectric conversion element 30 before starting exposure, and outputs an analog image signal from each pixel column 33 after exposure.
• the processor 58 controls the exposure time for the photoelectric conversion region 32 by controlling the focal plane shutter 48 and the lens shutter 52B.
• the focal plane shutter 48 and the lens shutter 52B are an example of a "mechanical shutter" according to the technology of the present disclosure.
• the processor 58 controls the exposure start timing and the exposure end timing for the plurality of pixel columns 33 (here, as an example, all the pixel columns 33), so that the The exposure time is shortened toward the pixel row 33 on the bright side.
• the dark side of the plurality of pixel columns 33 refers to the side in the +Y direction
• the bright side of the plurality of pixel columns 33 refers to the side in the ⁇ Y direction.
• the exposure time of each pixel column 33 is determined in advance based on the characteristics of the brightness difference appearing in the photoelectric conversion region 32. There are various ways to decide the exposure time, and how to decide according to the technology of the present disclosure will be explained in the second embodiment and thereafter.
• Each pixel column 33 is exposed for a predetermined exposure time.
• the exposure start timing and exposure end timing for the plurality of pixel columns 33 are determined by the processor 58 according to the exposure time determined in advance for each pixel column 33.
• the processor 58 adjusts the exposure start timing for the plurality of pixel columns 33 as a control for shortening the exposure time from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33. Control is performed such that the exposure timing is delayed from the dark side pixel column 33 to the bright side pixel column 33 of 33, and the exposure end timings for the plurality of pixel columns 33 are made to coincide with each other.
• the processor 58 first sets the position of the mechanical shutter to the initial position.
• the initial position of the focal plane shutter 48 when the exposure time control process is performed refers to a position where the rear curtain is fully open and the front curtain is fully closed.
• the initial position of the lens shutter 52B when the exposure time control process is performed refers to the position where the lens shutter 52B is fully opened.
• the processor 58 fully opens the front curtain of the focal plane shutter 48 by moving the front curtain of the focal plane shutter 48 from the dark side to the bright side of the plurality of pixel columns 33 at the first shutter speed.
• the exposure start timing for the plurality of pixel columns 33 is defined based on the first shutter speed.
• the first shutter speed is determined by the processor 58 according to a predetermined exposure time for each pixel column 33.
• the first shutter speed can be calculated using an equation in which the exposure time applied to the pixel row 33 is an independent variable and the first shutter speed is a dependent variable, or the exposure time applied to the pixel row 33 is It is derived from a table in which the first shutter speed is associated with the first shutter speed.
• the processor 58 When the front curtain of the focal plane shutter 48 is fully opened as described above, the processor 58 performs control to match the exposure end timings of the plurality of pixel columns 33 using a global shutter method using the lens shutter 52B. That is, the processor 58 operates the lens shutter 52B at the timing when the front curtain of the focal plane shutter 48 is fully opened. As a result, the lens shutter 52B is fully closed and the object light is blocked by the lens shutter 52B, so that the exposure of the photoelectric conversion area 32 is completed.
• the processor 58 controls the photoelectric conversion element 30 to output an analog image signal from each pixel column 33 to the signal processing circuit 56.
• FIG. 5 shows an example of the flow of the exposure time control process executed by the processor 58.
• the flow of the exposure time control process shown in FIG. 5 is an example of the "imaging support method" according to the technology of the present disclosure.
• step ST100 the processor 58 determines whether the timing for the image sensor 26 to start imaging has arrived.
• a first example of the timing at which imaging is started is the condition that the imaging system 10 reaches a predetermined position (for example, a waypoint) and the photoelectric conversion region 32 directly faces the imaging range 36. It will be done.
• a second example of the timing at which imaging is started is a condition in which an instruction to start imaging is given to the imaging device 14 from the outside (for example, a communication device).
• step ST100 if the timing for starting imaging by the image sensor 26 has not arrived, the determination is negative and the exposure time control process moves to step ST114. In step ST100, when the timing for starting imaging by the image sensor 26 has arrived, the determination is affirmative and the exposure time control process moves to step ST102.
• step ST102 the processor 58 resets the photoelectric conversion element 30. After the process of step ST102 is executed, the exposure time control process moves to step ST104.
• step ST104 the processor 58 moves the front curtain of the focal plane shutter 48 from the dark side to the bright side of the plurality of pixel columns 33 at the first shutter speed. After the process of step ST104 is executed, the exposure time control process moves to step ST106.
• step ST106 the processor 58 determines whether or not the first pixel column 33 to the last pixel column 33 have been exposed.
• the case where the first pixel column 33 to the last pixel column 33 are exposed refers to the case where the front curtain of the focal plane shutter 48 is fully closed.
• step ST106 if the first pixel column 33 to the last pixel column 33 have not been exposed, the determination is negative and the determination in step ST106 is performed again.
• step ST106 if the first pixel column 33 to the last pixel column 33 have been exposed, the determination is affirmative and the exposure time control process moves to step ST108.
• step ST108 the processor 58 operates the lens shutter 52B.
• the lens shutter 52B is fully closed and the object light is blocked by the lens shutter 52B, so that the exposure of the photoelectric conversion area 32 is completed.
• the exposure time becomes shorter in order from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33.
• the exposure time control process moves to step ST110.
• step ST110 the processor 58 controls the photoelectric conversion element 30 to output an analog image signal from each pixel column 33 to the signal processing circuit 56.
• step ST100 the exposure time control process moves to step ST112.
• step ST112 the processor 58 returns the position of the mechanical shutter to its initial position by controlling the mechanical shutter. After the process of step ST112 is executed, the exposure time control process moves to step ST114.
• step ST114 the processor 58 determines whether the conditions for terminating the exposure time control process are satisfied.
• a first example of the condition for terminating the exposure time control process is that the imaging system 10 has captured images at all predetermined positions (for example, all waypoints).
• a second example of the condition for terminating the exposure time control process is that an instruction to terminate the exposure time control process is given to the imaging device 14 from the outside (for example, a communication device).
• step ST114 if the conditions for terminating the exposure time control process are not satisfied, the determination is negative and the exposure time control process moves to step ST100. In step ST114, if the conditions for ending the exposure time control process are satisfied, the determination is affirmative and the exposure time control process ends.
• the imaging range 36 within the subject 27 is imaged by the image sensor 26, from the lighting equipment 22 arranged on the +Y direction side of the imaging device main body 20, the imaging range 36 located in front of the imaging device main body 20 Since the auxiliary light 28 is irradiated onto the imaging range 36, a difference in brightness occurs in the imaging range 36 along the Y direction. That is, within the imaging range 36, there is a difference in brightness such that the side closer to the lighting equipment 22 is brighter and the side farther from the lighting equipment 22 is darker. The difference in brightness that occurs in the imaging range 36 is also reflected in the photoelectric conversion area 32. That is, due to the incident light on the photoelectric conversion region 32, the photoelectric conversion region 32 gradually becomes darker from the +Y direction side to the ⁇ Y direction side.
• the imaging system 10 when a difference in brightness occurs in the photoelectric conversion area 32 along the Y direction due to light incident on the photoelectric conversion area 32, a plurality of pixel columns 33 in the photoelectric conversion area 32 Control is performed to shorten the exposure time from the pixel column 33 on the dark side to the pixel column 33 on the bright side.
• the image sensor 26 can generate a captured image 29 with less uneven brightness. .
• a column in which a plurality of pixels 34 are linearly arranged in the Z direction, which is a direction intersecting the Y direction is used as the pixel column 33.
• a difference in brightness occurs in the photoelectric conversion region 32 along the Y direction due to incident light on the photoelectric conversion region 32
• pixels from the dark side pixel row 33 of the plurality of pixel rows 33 in the photoelectric conversion region 32 to the bright side pixel Control is performed to shorten the exposure time over column 33. Therefore, even if a difference in brightness occurs along the Y direction, which is a direction that crosses the plurality of pixel rows 33, the image sensor 26 can generate a captured image 29 with less uneven brightness.
• the imaging system 10 Furthermore, in the imaging system 10 according to the first embodiment, light including reflected light obtained when the auxiliary light 28 emitted from the lighting device 22 is reflected by the subject 27 is incident on the photoelectric conversion region 32. This causes a difference in brightness in the photoelectric conversion region 32 along the Y direction.
• control is performed to shorten the exposure time from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33 in the photoelectric conversion region 32.
• light including reflected light obtained when the auxiliary light 28 emitted from the lighting device 22 is reflected by the subject 27 is incident on the photoelectric conversion area 32, so that the light is directed toward the photoelectric conversion area 32 along the Y direction. Even if there is a difference in brightness or darkness, the image sensor 26 can generate a captured image 29 with less uneven brightness.
• a mechanical shutter is used to Control is performed to shorten the exposure time from the dark pixel row 33 to the bright pixel row 33 of the plurality of pixel rows 33 . Therefore, even if a difference in brightness occurs along the Y direction in the photoelectric conversion area 32 due to light incident on the photoelectric conversion area 32 of the imaging device 14 that is not equipped with an electronic shutter, uneven brightness will be caused to the image sensor 26. It is possible to generate fewer captured images 29.
• the processor 58 controls the exposure start timing and the exposure end timing for the plurality of pixel columns 33, thereby controlling the dark side pixel column 33 of the plurality of pixel columns 33.
• the exposure time is shortened from the bright side to the pixel row 33 on the bright side.
• the processor 58 delays the exposure start timing for the plurality of pixel columns 33 from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33, and Control is performed to match the exposure end timing. Therefore, compared to the case where the incident light to the photoelectric conversion region 32 is adjusted, the amount of exposure can be easily reduced from the dark pixel row 33 to the bright pixel row 33 of the plurality of pixel rows 33.
• the processor 58 performs control to match the exposure end timings of the plurality of pixel columns 33 using a global shutter method using the lens shutter 52B. Therefore, the exposure end timings for the plurality of pixel rows 33 can be made to coincide more easily than when control is performed to match the exposure end timings using only the rolling shutter method.
• the exposure end timings for a plurality of pixel rows 33 are made to coincide with each other using a global shutter method using a mechanical shutter (for example, a global shutter method using a lens shutter 52B).
• a mechanical shutter for example, a global shutter method using a lens shutter 52B
• the technology of the present disclosure is not limited to this.
• a fully electronic shutter i.e., an electronic shutter
• the image sensor 26 as a global shutter together with the lens shutter 52B or in place of the lens shutter 52B, it is possible to synchronize the exposure end timings for the plurality of pixel columns 33. You can also do this.
• the technology of the present disclosure is not limited to this.
• an electronic front curtain shutter instead of the front curtain of the focal plane shutter 48, an electronic front curtain shutter may be used.
• the electronic front curtain shutter may be moved from the dark side to the bright side at the first shutter speed with the front curtain and rear curtain of the focal plane shutter 48 fully open.
• the exposure start timing for the plurality of pixel columns 33 is set at a plurality of times.
• the embodiment has been described in which the processor 58 performs control to slow down the exposure from the dark side pixel column 33 to the bright side pixel column 33 of the pixel column 33 and to match the exposure end timing for a plurality of pixel columns 33.
• the technology of the present disclosure is not limited thereto.
• the processor 58 makes the exposure start timings of the plurality of pixel columns 33 coincide, and the exposure end timing of the plurality of pixel columns 33 on the bright side of the plurality of pixel columns 33. Control may be performed to slow down the pixel row 33 to the pixel row 33 on the dark side.
• the exposure start timings for the plurality of pixel columns 33 may be made to match in the same manner as the exposure end timings are made to match in the first embodiment. That is, the processor 58 may synchronize the exposure start timings for the plurality of pixel columns 33 using a global shutter method using a mechanical shutter (for example, a global shutter method using the lens shutter 52B). Further, the processor 58 may synchronize the exposure start timings for the plurality of pixel columns 33 by using a fully electronic shutter by the image sensor 26 as a global shutter.
• FIG. 3 an example is shown in which the trailing curtain of the focal plane shutter 48 moves from the +Y direction side to the ⁇ Y direction side.
• the moving direction of the focal plane shutter 48 is reversed. That is, the focal plane shutter 48 is arranged in such a direction that the rear curtain of the focal plane shutter 48 is moved from the -Y direction side to the +Y direction side.
• the processor 58 controls the rear curtain of the focal plane shutter 48 to delay the exposure end timing for the plurality of pixel columns 33 from the bright pixel column 33 to the dark pixel column 33 of the plurality of pixel columns 33. Control is performed to move the plurality of pixel columns 33 from the bright side to the dark side at the first shutter speed.
• the exposure start timings for the plurality of pixel columns 33 are made to match, and, Even if control is performed to delay the exposure end timing for the plurality of pixel columns 33 from the bright side pixel column 33 to the dark side pixel column 33 of the plurality of pixel columns 33, the same effect as in the first embodiment is performed. effect can be obtained.
• the rear curtain of the focal plane shutter 48 is moved at the first shutter speed from the bright side to the dark side of the plurality of pixel rows 33 at the first shutter speed, but the technology of the present disclosure is limited to this. Not done.
• the exposure end timing for the column 33 may be delayed from the brighter pixel column 33 to the darker pixel column 33 of the plurality of pixel columns 33.
• the processor 58 delays the exposure start timing for the plurality of pixel columns 33 from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33,
• the exposure end timing for the plurality of pixel columns 33 may also be controlled to be delayed from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33.
• the processor 58 performs control to shorten the exposure time from the dark side to the bright side of the plurality of pixel columns 33.
• the processor 58 changes the exposure start timing for the plurality of pixel columns 33 from the dark side pixel column 33 to the bright side pixel column 33 in the same manner as in the first embodiment. Control is performed to slow down the pixel row 33. That is, the processor 58 moves the front curtain of the focal plane shutter 48 from the dark side to the bright side of the plurality of pixel rows 33 at the first shutter speed, or operates the electronic front curtain shutter.
• the processor 58 adjusts the exposure end timing for the plurality of pixel columns 33 from the pixel column 33 on the bright side to the pixel column 33 on the dark side of the plurality of pixel columns 33 in the same manner as in the example shown in FIG. Control to slow down.
• a focal plane shutter 48 for ending exposure is used.
• the focal plane shutter 48 for ending exposure is a focal plane shutter in which the rear curtain moves from the -Y direction side to the +Y direction side with the front curtain fully open.
• the processor 58 performs control to move the rear curtain of the focal plane shutter 48 for ending exposure from the bright side to the dark side of the plurality of pixel columns 33 at a second shutter speed.
• the exposure end timing for the plurality of pixel columns 33 is defined based on the second shutter speed.
• the second shutter speed is determined by the processor 58 according to a predetermined exposure time for each pixel column 33.
• the second shutter speed can be calculated using an equation in which the exposure time applied to the pixel row 33 is an independent variable and the second shutter speed is a dependent variable, or the exposure time applied to the pixel row 33 is The second shutter speed is derived from the associated table.
• the front curtain of the focal plane shutter 48 moves from the dark side to the bright side of the plurality of pixel columns 33 at the first shutter speed
• the rear curtain of the focal plane shutter 48 for ending exposure moves from the dark side to the bright side of the plurality of pixel columns 33.
• the exposure time is shortened from the dark pixel row 33 to the bright pixel row 33 of the plurality of pixel rows 33 by moving the focal plane shutter 48, but the technology of the present disclosure is limited to this. Not done. For example, even with a line exposure sequential readout method using a fully electronic shutter, it is possible to shorten the exposure time from the dark pixel row 33 to the bright pixel row 33 of the plurality of pixel rows 33.
• an exposure time determination program 72 is stored in the NVM 60.
• the processor 58 reads the exposure time determination program 72 from the NVM 60 and executes the read exposure time determination program 72 on the RAM 62.
• the processor 58 performs an exposure time determination process (see FIG. 9) according to an exposure time determination program 72 executed on the RAM 62.
• FIG. 9 shows an example of the flow of the exposure time determination process according to the second embodiment performed by the processor 58.
• the second embodiment will be described on the assumption that the auxiliary light 28 is irradiated from the illumination device 22 to the subject 27 side.
• the front curtain and the rear curtain of the focal plane shutter 48 are moved along the Y direction (for example, a plurality of pixel columns).
• the explanation will be given on the assumption that the photoelectric conversion area 32 is exposed by moving the photoelectric conversion area 33 from the bright side to the dark side.
• step ST200 the processor 58 starts exposing the photoelectric conversion region 32 by resetting the photoelectric conversion element 30 and activating the focal plane shutter 48. After the process of step ST200 is executed, the exposure time determination process moves to step ST202.
• step ST202 the processor 58 determines whether a reference exposure time has elapsed since the process in step ST200 was executed.
• the reference exposure time is an example of a "first reference exposure time" according to the technology of the present disclosure.
• An example of the reference exposure time is an exposure time determined according to an instruction given from the outside (for example, a communication device) or an instruction received by the UI device 42.
• step ST202 if the reference exposure time has not elapsed since the processing in step ST200 was executed, the determination is negative and the determination in step ST202 is performed again. If the reference exposure time has elapsed since the process in step ST200 was executed, the determination is affirmative and the exposure time determination process moves to step ST204.
• steps ST200 to ST204 an example is given in which the exposure to the photoelectric conversion region 32 is controlled by the processor 58 using a mechanical shutter, but this is just an example.
• exposure to the photoelectric conversion region 32 may be controlled by the processor 58 using an electronic shutter.
• step ST204 the processor 58 ends the exposure of the photoelectric conversion region 32 by fully closing the rear curtain of the focal plane shutter 48. After the process of step ST204 is executed, the exposure time determination process moves to step ST206.
• step ST206 the processor 58 causes each pixel column 33 to output an analog image signal. After the process of step ST206 is executed, the exposure time determination process moves to step ST208.
• the processor 58 acquires the representative signal level of the reference pixel column among all the pixel columns 33.
• the reference pixel row is an example of a "first reference area within the photoelectric conversion area" according to the technology of the present disclosure.
• the reference pixel column refers to, for example, the center pixel column 33 among all the pixel columns 33. Note that although the central pixel column 33 among all the pixel columns 33 is illustrated here as the reference pixel column, this is just an example, and other predetermined pixel columns 33 Alternatively, it may be a plurality of pixel columns 33 located at the center of the photoelectric conversion region 32.
• a pixel column 33 located on the brighter side than the reference pixel column among all the pixel columns 33 for example, a pixel column 33 located in the -Y direction of all the pixel columns 33
• An example of this is the pixel row 33) located at one end of the pixel row 33).
• the representative signal level refers to the average value of the signal levels of analog image signals output from all pixels 34 included in the pixel row 33.
• the average value is given here, this is just an example, and instead of the average value, a statistical value such as a median value or a mode value may be applied.
• step ST210 the processor 58 determines whether the representative signal level obtained in step ST208, that is, the representative signal level of the reference pixel column is saturated.
• the representative signal level of the reference pixel array is saturated, it means that the representative signal level of the reference pixel array is a signal level at which an image area based on the reference pixel array in the captured image 29 is blown out.
• step ST210 if the representative signal level of the reference pixel row is not saturated, the determination is negative and the exposure time determination process moves to step ST214.
• step ST210 if the representative signal level of the reference pixel row is saturated, the determination is affirmative and the exposure time determination process moves to step ST212.
• step ST212 the processor 58 sets the reference exposure time to half the time. After the process of step ST212 is executed, the exposure time determination process moves to step ST200. Note that by performing the processing in steps ST200 to ST212, the reference exposure time is adjusted to a time shorter than the time at which the representative signal level of the reference pixel column is saturated.
• step ST214 the processor 58 acquires the representative signal level of each pixel column 33 (for example, each of all the pixel columns 33). After the process of step ST214 is executed, the exposure time determination process moves to step ST216.
• step ST216 the ratio of the representative signal level (hereinafter also simply referred to as "ratio") between the representative signal level of each pixel column 33 and the representative signal level of the reference pixel column is calculated for each pixel column 33.
• the ratio refers to the ratio of the representative signal level of the pixel column 33 to the representative signal level of the reference pixel column.
• the ratio calculated in step ST216 is an example of the "first degree of difference" according to the technology of the present disclosure.
• step ST2108 the processor 58 determines the exposure time for each pixel column 33 based on the ratio calculated for each pixel column 33. For example, in the photoelectric conversion region 32, the processor 58 increases the exposure time of the pixel rows 33 whose ratio value is less than 1, and shortens the exposure time of the pixel rows 33 whose ratio value exceeds 1. decide.
• the exposure time is determined by an arithmetic expression in which the ratio is an independent variable and the exposure time of the pixel row 33 is a dependent variable, or by being derived from a table in which the ratio and the exposure time of the pixel row 33 are associated. .
• the exposure time of each pixel column 33 is determined in advance, but the exposure time determined in step ST218 may be applied as the exposure time of each pixel column 33. After the process of step ST218 is executed, the exposure time determination process ends.
• the exposure time for each pixel column 33 from the dark side pixel column 33 to the bright side pixel column 33 among the plurality of pixel columns 33 in the photoelectric conversion area 32 is , is determined by the ratio of the representative signal level between the representative signal level of each pixel column 33 and the representative signal level of the reference pixel column.
• the ratio is determined for each pixel column 33.
• Each ratio is a value representing the degree of difference between the representative signal level of the reference pixel column and each representative signal level obtained from each pixel column 33 when the exposure time for the photoelectric conversion region 32 is made shorter than the reference exposure time.
• a suitable exposure time for each pixel column 33 from the dark side pixel column 33 to the bright side pixel column 33 for example, the brightness unevenness in the captured image 29 (or an exposure time that does not cause overexposure) can be determined.
• the reference exposure time is adjusted to be less than the time at which the signal level of the analog image signal output from the reference pixel array in the photoelectric conversion area 32 is saturated. be done.
• the exposure time of the reference pixel array in the photoelectric conversion area 32 is set to be less than the time at which the signal level of the analog image signal output from the reference pixel array is saturated.
• the exposure time of each pixel column 33 is determined based on the exposure time of the reference pixel column. As a result, each pixel column 3 In contrast, it is possible to set an exposure time such that each pixel column 33 is unlikely to be overexposed.
• the ratio is illustrated, but this is just an example, and instead of the ratio, it is the difference between the representative signal level of the reference pixel row and the representative signal level of the pixel row 33. Good too.
• the difference between the representative signal level of the reference pixel column and the representative signal level of the pixel column 33 is an example of the "first degree of difference" according to the technology of the present disclosure.
• the reference exposure time is set to half the time when the representative signal level of the reference pixel row in the photoelectric conversion area 32 is saturated. This is just an example. The extent to which the reference exposure time is reduced when the representative signal level of the reference pixel row in the photoelectric conversion region 32 is saturated depends on the instructions given by the user, etc., and/or the imaging conditions of the imaging device 14, etc. You can decide accordingly.
• the exposure time for each pixel column 33 is Although the example of the determined form has been described, the technology of the present disclosure is not limited thereto.
• the exposure times of some pixel rows 33 may be estimated using an interpolation method. In this case, first, two or more ratios are calculated from the representative signal level of the reference pixel column and the representative signal level of one or more pixel columns 33 other than the reference pixel column. Next, the exposure time of each corresponding pixel column 33 is determined based on two or more ratios. Then, the exposure times of the remaining pixel rows 33 may be estimated from the plurality of determined exposure times using an interpolation method.
• FIG. 10 shows an example of the flow of the exposure time determination process according to the third embodiment performed by the processor 58.
• the third embodiment will be described on the assumption that the auxiliary light 28 is irradiated from the illumination device 22 to the subject 27 side.
• the front curtain and the rear curtain of the focal plane shutter 48 are moved along the Y direction (for example, a plurality of pixel columns). The explanation will be given on the assumption that the photoelectric conversion area 32 is exposed by moving the photoelectric conversion area 33 from the bright side to the dark side.
• the processes from step ST300 to step ST308 are the same as the processes from step ST200 to step ST208 shown in FIG. Furthermore, in the exposure time determination process shown in FIG. 10, the processes from step ST310 to step ST314 are the same as the processes from step ST214 to step ST218 shown in FIG.
• the reference exposure time used in the process of step ST302 is an example of the "first exposure time” according to the technology of the present disclosure.
• the reference pixel column used in the process of step ST308 is an example of the "second reference area within the photoelectric conversion area” according to the technology of the present disclosure.
• the ratio calculated in step ST312 is an example of the "second degree of difference” according to the technology of the present disclosure.
• the exposure time of the reference pixel column among the plurality of exposure times determined in step ST314 is an example of the "first exposure time” according to the technology of the present disclosure.
• step ST316 the processor 58 exposes each pixel column 33 with the exposure time determined for each pixel column 33 in step ST314. After the process of step ST316 is executed, the exposure time determination process moves to step ST318.
• step ST3108 the processor 58 obtains the representative signal level of each of all the pixel columns 33. After the process of step ST318 is executed, the exposure time determination process moves to step ST320.
• step ST320 the processor 58 determines whether each of all representative signal levels obtained in step 318 falls within the reference range.
• the reference range used in the process of step ST320 is an example of the "reference range" according to the technology of the present disclosure.
• a first example of the reference range is a range of signal levels in the captured image 29 that does not have crushed blacks or blown out highlights.
• a second example of the reference range is a range of signal levels within the captured image 29 that does not cause overexposure.
• step ST320 if each of all the representative signal levels acquired in step 318 is not within the reference range, the determination is negative and the exposure time determination process moves to step ST308. In step ST320, if each of all the representative signal levels acquired in step 318 falls within the reference range, the determination is affirmative and the exposure time determination process moves to step ST320. Note that by executing the processes from step ST308 to step ST320, the exposure time of each of all pixel columns 33 is adjusted to a time in which each of all representative signal levels falls within the reference range.
• step ST322 the processor 58 finalizes the exposure time determined for each pixel column 33 in step ST314.
• the exposure time of each pixel column 33 is determined in advance, but the exposure time determined in step ST322 may be applied as the exposure time of each pixel column 33.
• the exposure time for each pixel column 33 from the dark side pixel column 33 to the bright side pixel column 33 among the plurality of pixel columns 33 in the photoelectric conversion region 32 is , is determined in the same manner as in the second embodiment. Then, the exposure time for each pixel column 33 is adjusted so that each representative signal level obtained by exposing each pixel column 33 with the exposure time for each pixel column 33 falls within the reference range.
• a suitable exposure time for each pixel column 33 from the dark side pixel column 33 to the bright side pixel column 33 for example, if the brightness unevenness in the captured image 29
• An exposure time that does not cause overexposure, an exposure time that does not cause underexposure, or an exposure time that does not cause underexposure can be determined.
• FIGS. 11A and 11B show an example of the flow of the exposure time determination process according to the fourth embodiment performed by the processor 58.
• the fourth embodiment will be described on the assumption that the auxiliary light 28 is irradiated from the illumination device 22 to the subject 27 side.
• the front curtain and the rear curtain of the focal plane shutter 48 are moved along the Y direction (for example, a plurality of pixel columns). The explanation will be given on the assumption that the photoelectric conversion area 32 is exposed by moving the photoelectric conversion area 33 from the bright side to the dark side.
• the fourth embodiment will be described on the premise that the exposure time determination process is performed by the processor 58 when the imaging target is changed from the imaging range 36 to another imaging range in imaging using the image sensor 26. .
• the processes from step ST400 to step ST422 are the same as the processes from step ST300 to step ST322 shown in FIG.
• the reference exposure time used in step ST402 and the exposure time of the reference pixel column among the plurality of exposure times determined in step ST414 are an example of the "second exposure time” according to the technology of the present disclosure.
• the reference pixel array used in the process of step ST408 is an example of the "third reference area within the photoelectric conversion area” according to the technology of the present disclosure.
• the ratio calculated in step ST412 is an example of the "third degree of difference" according to the technology of the present disclosure.
• step ST424 shown in FIG. 11B the processor 58 determines whether the imaging target by the imaging device 14 has been changed from the imaging range 36 to an imaging range other than the imaging range 36.
• step ST424 if the imaging target by the imaging device 14 has not been changed from the imaging range 36 to an imaging range other than the imaging range 36, the determination is negative and the exposure time determination process moves to step ST438.
• step ST424 if the imaging target by the imaging device 14 is changed from the imaging range 36 to an imaging range other than the imaging range 36, the determination is affirmative and the exposure time determination process moves to step ST426.
• step ST426 to step ST430 is the same as the processing from step ST400 to step ST404 shown in FIG. 11A.
• step ST432 the processor 58 obtains the representative signal level of the reference pixel column among all the pixel columns 33.
• the exposure time determination process moves to step ST434.
• the reference pixel array used in step ST432 is an example of the "third reference area within the photoelectric conversion area" according to the technology of the present disclosure.
• step ST434 the processor 58 updates the exposure time of the reference pixel array based on the representative signal level of the reference pixel array.
• the exposure time of the current reference pixel row (for example, the exposure time of the reference pixel row among the exposure times of all the pixel rows 33 determined in step ST414) is the first with respect to the exposure time of the current reference pixel row. It is updated by being multiplied by a coefficient.
• the first coefficient used here is, for example, the ratio of the representative signal level obtained in step ST432 to the representative signal level obtained in step ST408.
• the exposure time determination process moves to step ST436. Note that the exposure time updated by executing the process of step ST434 is an example of "the second reference exposure time determined for the third reference area" according to the technology of the present disclosure.
• step ST436 the processor 58 updates the exposure time of each pixel column 33 by multiplying the latest exposure time of the reference pixel column by the ratio calculated for each pixel column 33.
• the latest exposure time of the reference pixel row refers to the exposure time updated and obtained in step ST434.
• the calculated ratio for each pixel row 33 refers to the ratio calculated for each pixel row 33 in step ST412.
• updating the exposure time for each pixel column 33 means updating the exposure time determined for each pixel column 33 in step ST416, or re-updating the exposure time updated for each pixel column 33 in the previous step ST434. refers to After the process of step ST436 is executed, the exposure time determination process moves to step ST438.
• step ST4308 the processor 58 determines whether the conditions for terminating the exposure time determination process are satisfied.
• a first example of the condition for terminating the exposure time determination process is that the imaging system 10 has captured images at all predetermined positions (for example, all waypoints).
• a second example of the condition for terminating the exposure time determination process is that an instruction to terminate the exposure time determination process is given to the imaging device 14 from the outside (for example, a communication device).
• step ST4308 if the conditions for terminating the exposure time determination process are not satisfied, the determination is negative and the exposure time determination process moves to step ST424. In step ST438, if the conditions for terminating the exposure time determination process are satisfied, the determination is affirmative and the exposure time determination process is terminated.
• the pixel row on the dark side of the plurality of pixel rows 33 in the photoelectric conversion region 32 The exposure time for each pixel column 33 from 33 to the bright side pixel column 33 is determined in the same manner as in the second and third embodiments.
• the pixels are set such that each representative signal level obtained by exposing each pixel column 33 with the exposure time for each pixel column 33 falls within the reference range. The exposure time for each row 33 is adjusted.
• the current exposure time set for the reference pixel array is updated based on the representative signal level of the reference pixel array.
• the exposure time for each pixel row 33 is updated by multiplying the latest exposure time of the reference pixel row by the calculated ratio for each pixel row 33.
• a suitable exposure time for each pixel row 33 (for example, an exposure time that does not cause uneven brightness in the captured image 29, an exposure time that does not cause overexposure, or an exposure time that does not cause underexposure) can be easily determined.
• FIG. 12 shows an example of the flow of the exposure time determination process according to the fifth embodiment performed by the processor 58.
• the front curtain and rear curtain of the focal plane shutter 48 are moved from the dark side to the bright side of the plurality of pixel rows 33 from the fully closed state.
• the explanation will be made on the assumption that the photoelectric conversion area 32 is exposed to light by movement.
• step ST500 the processor 58 sets a flash exposure time determined according to the flash.
• the flash exposure time is an example of the "third exposure time” according to the technology of the present disclosure.
• An example of the flash exposure time is an exposure time predetermined for a guide number.
• step ST502 the processor 58 causes the lighting equipment 22 to emit a flash. Then, the processor 58 activates the focal plane shutter 48 to expose the photoelectric conversion region 32 to light for the flash exposure time. After the process of step ST502 is executed, the exposure time determination process moves to step ST504.
• step ST504 to step ST512 is the same as the processing from step ST408 to step ST414 shown in FIG. 11A.
• the representative signal level acquired in step ST506 is an example of the "signal level of the fourth reference region” according to the technology of the present disclosure.
• the representative signal level acquired for each pixel column 33 in step ST508 is an example of "a plurality of signal levels obtained from a plurality of divided regions” according to the technology of the present disclosure.
• the ratio calculated in step ST510 is an example of the "fourth degree of difference" according to the technology of the present disclosure.
• the exposure time of each pixel column 33 is determined in advance, but when a flash is used in synchronization with the timing of imaging using the image sensor 26, the exposure time of each pixel column 33 is determined in advance.
• the exposure time determined in step ST512 may be applied as the exposure time.
• each pixel column 33 is exposed to light during the flash exposure time, so that the pixel column Using the ratio obtained for each pixel column 33, the exposure time for each pixel column 33 is determined in the manner described in the second to fourth embodiments. Therefore, even if a flash is used in synchronization with the timing at which an image is captured using the image sensor 26, the pixel rows 33 on the dark side to the bright side among the plurality of pixel rows 33 in the photoelectric conversion area 32 A suitable exposure time for each pixel row 33 can be determined.
• the exposure time of the reference pixel row is determined based on the representative signal level obtained when a flash is emitted, but the technology of the present disclosure is not limited to this.
• the exposure time obtained for the reference pixel row may be updated before imaging using a flash is performed.
• the exposure time of the current reference pixel row (for example, the exposure time of the reference pixel row among the exposure times of all the pixel rows 33 determined in step ST414) is It is updated by multiplying the exposure time by a second coefficient.
• the second coefficient used here is, for example, the ratio of the representative signal level obtained in step ST506 to the representative signal level obtained in step ST408.
• the exposure time for each pixel column 33 is determined based on the representative signal level obtained when a flash is emitted, but the technology of the present disclosure is not limited to this.
• a flash is used in synchronization with the timing at which imaging is performed using the image sensor 26, information is obtained for each pixel column 33 before imaging using the flash is performed, in a manner similar to step ST434 shown in FIG. 11B.
• the determined exposure time (for example, the exposure time determined in step ST414 shown in FIG. 11A) may be updated.
• the exposure time updated for each pixel column 33 in this way is used as the exposure time for each pixel column 33 when imaging using a flash is performed.
• the technology of the present disclosure is also applicable to a case where the aperture 52C is adjusted in imaging using the image sensor 26. It is.
• the F value is adjusted at the timing when imaging with flash is performed, as in Flashmatic, for example, as shown in FIG. 13, exposure time determination is performed in which step ST500A is applied instead of step ST500 Processing is performed by processor 58.
• step ST500A of the exposure time determination process shown in FIG. 13 the processor 58 sets a flash exposure time determined according to the flash guide number and the F value of the aperture 52C. Even in this case, the same effects as in the fifth embodiment can be obtained.
• shutter speed determination processing is a process for determining the moving speed of the focal plane shutter 48 (hereinafter also referred to as "shutter speed") when the exposure time for each pixel row 33 is determined according to the moving speed of the focal plane shutter 48.
• shutter speed the moving speed of the focal plane shutter 48
• the same reference numerals are given to the constituent elements explained in each of the above embodiments, and the explanation thereof will be omitted, and only the different parts from the above embodiments will be explained.
• a shutter speed determination program 74 is stored in the NVM 60.
• the processor 58 reads the shutter speed determination program 74 from the NVM 60 and executes the read shutter speed determination program 74 on the RAM 62.
• the processor 58 performs shutter speed determination processing (see FIG. 15) according to the shutter speed determination program 74 executed on the RAM 62.
• FIG. 15 shows an example of the flow of the shutter speed determination process according to the sixth embodiment performed by the processor 58.
• the sixth embodiment will be described on the assumption that the auxiliary light 28 is irradiated from the illumination device 22 to the subject 27 side.
• the front curtain and the rear curtain of the focal plane shutter 48 are moved along the Y direction (for example, a plurality of pixel columns). The explanation will be given on the assumption that the photoelectric conversion area 32 is exposed by moving the photoelectric conversion area 33 from the bright side to the dark side.
• the processes from step ST600 to step ST608 are the same as the processes from step ST400 to step ST408 shown in FIG. 11A.
• the reference exposure time used in step ST602 is an example of the "fourth exposure time” according to the technology of the present disclosure.
• the reference pixel column used in step ST608 is an example of the "fifth reference region within the photoelectric conversion region” according to the technology of the present disclosure.
• the representative signal level acquired in step ST608 is an example of the "signal level of the fifth reference region" according to the technology of the present disclosure.
• step ST610 the processor 58 obtains the representative signal level of the first pixel column and the representative signal level of the second pixel column.
• the first pixel column refers to, for example, the pixel column 33 located at the end in the ⁇ Y direction among all the pixel columns 33 (see FIG. 16).
• the second pixel column refers to, for example, the pixel column 33 located at the end in the +Y direction among all the pixel columns 33 (see FIG. 16).
• the representative signal level of the first pixel column and the representative signal level of the second pixel column are examples of "a plurality of signal levels obtained from a plurality of divided regions" according to the technology of the present disclosure.
• step ST612 the processor 58 calculates a regression line 76 regarding the representative signal levels of the first and second pixel columns based on the representative signal level of the reference pixel column (that is, the representative signal level obtained in step ST608). (See Figure 16). After the process of step ST612 is executed, the shutter speed determination process moves to step ST614.
• step ST614 the processor 58 determines the shutter speed of the focal plane shutter 48 based on the slope of the regression line 76 calculated in step ST612.
• the shutter speed of the focal plane shutter 48 is determined by deriving the shutter speed of the focal plane shutter 48 from the arithmetic expression 78.
• Arithmetic expression 78 is an arithmetic expression that uses the slope of regression line 76 as an independent variable and the shutter speed of focal plane shutter 48 as a dependent variable.
• the shutter speed determined in step ST614 in this way is an example of the "focal plane shutter movement speed" according to the technology of the present disclosure.
• the shutter speed of the focal plane shutter 48 may be derived from a table in which the slope of the regression line 76 and the shutter speed of the focal plane shutter 48 are correlated.
• the results of regression analysis based on the representative signal level of the reference pixel column, the representative signal level of the first pixel column, and the representative signal level of the second pixel column (here As an example, the shutter speed of the focal plane shutter 48 is determined based on the slope of the regression line 76. Therefore, the shutter speed of the focal plane shutter 48 is the shutter speed necessary to realize a suitable exposure time for each pixel column 33 from the dark side pixel column 33 to the bright side pixel column 33 of the plurality of pixel columns 33. can be determined.
• regression analysis is performed based on the representative signal level of the reference pixel column, the representative signal level of the first pixel column, and the representative signal level of the second pixel column. is merely an example, and regression analysis may be performed based on representative signal levels of four or more pixel columns 33.
• the exposure time control program 70, the exposure time determination program 72, and the shutter speed determination program 74 will be referred to as an "imaging support program” unless it is necessary to explain them separately. Further, in the following, for convenience of explanation, the exposure time control process, the exposure time determination process, and the shutter speed determination process will be referred to as “imaging support processing” unless it is necessary to explain them separately.
• the photoelectric conversion region 32 may be divided into units of a plurality of pixel columns 33, and in this case, the representative signal level may be acquired for each of the plurality of pixel columns 33.
• the attitude of the imaging device main body 20 may be changed so that the orientation of the image sensor 26 is rotated by 90 degrees around the X-axis.
• the attitude of the imaging device main body 20 may be changed so that the longitudinal direction of the pixel row 33 (that is, the direction in which the plurality of pixels 34 are arranged) is orthogonal to the direction in which brightness differences occur within the photoelectric conversion region 32.
• a rotation mechanism that changes the attitude of the image sensor 26 by rotating the image sensor 26 around the optical axis OA may be incorporated into the imaging device main body 20.
• the posture of the image sensor 26 can be adjusted so that the longitudinal direction of the pixel row 33 is in the direction in which a difference in brightness occurs within the photoelectric conversion region 32. You can do it in a perpendicular position.
• the lighting device 22 is integrated with the imaging device main body 20, but this is just an example, and the lighting device 22 can be separated from the imaging device main body 20 and mounted on a moving object. It may be attached to 12.
• the auxiliary light 28 is irradiated from the periphery of the imaging device main body 20 to the subject 27, and a difference in brightness and darkness occurs in the photoelectric conversion region 32 along one direction, so the exposure time is By performing the control process, the exposure time determination process, and the shutter speed determination process, the same effects as in each of the above embodiments can be obtained.
• the imaging support process is performed by the processor 58 of the controller 38 included in the imaging device main body 20, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure is not limited to this, and the imaging support process is performed.
• the device may be provided outside the imaging device main body 20. Examples of devices provided outside the imaging device main body 20 include at least one server and/or at least one personal computer that are communicably connected to the imaging device main body 20. Further, the imaging support processing may be performed in a distributed manner by a plurality of devices.
• the imaging support program may be stored in a portable non-temporary storage medium such as an SSD or a USB memory.
• the imaging support program stored in the non-temporary storage medium is installed in the controller 38 of the imaging device main body 20.
• the processor 58 executes imaging support processing according to the imaging support program.
• the imaging support program is stored in a storage device such as another computer or a server device connected to the imaging device main body 20 via a network, and the imaging support program is downloaded in response to a request from the imaging device main body 20. It may be installed in the controller 38.
• imaging support program it is not necessary to store the entire imaging support program in another computer or storage device such as a server device connected to the imaging device main body 20, or in the NVM 60, but it is possible to store a part of the imaging support program. Good too.
• processors can be used as hardware resources for executing imaging support processing.
• the processor include a CPU, which is a general-purpose processor that functions as a hardware resource for performing imaging support processing by executing software, that is, a program.
• the processor include a dedicated electric circuit such as an FPGA, PLD, or ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
• Each processor has a built-in or connected memory, and each processor uses the memory to execute imaging support processing.
• the hardware resources that execute the imaging support processing may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or (a combination of a CPU and an FPGA). Furthermore, the hardware resource that executes the imaging support process may be one processor.
• one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource for executing imaging support processing.
• a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with “and/or”, the same concept as “A and/or B" is applied.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Studio Devices (AREA)
• Transforming Light Signals Into Electric Signals (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=89941457

Family Applications (1)

Country Status (3)

Citations (2)

Family Cites Families (3)

• 2023
  • 2023-06-22 JP JP2024541435A patent/JPWO2024038677A1/ja active Pending
  • 2023-06-22 WO PCT/JP2023/023218 patent/WO2024038677A1/ja not_active Ceased
• 2024
  • 2024-12-24 US US19/001,354 patent/US20250142219A1/en active Pending

Patent Citations (2)

Also Published As

Similar Documents

Legal Events