WO2023002826A1 - 撮像システム、及び、それを備えた移動体 - Google Patents

撮像システム、及び、それを備えた移動体 Download PDF

Info

Publication number
WO2023002826A1
WO2023002826A1 PCT/JP2022/026044 JP2022026044W WO2023002826A1 WO 2023002826 A1 WO2023002826 A1 WO 2023002826A1 JP 2022026044 W JP2022026044 W JP 2022026044W WO 2023002826 A1 WO2023002826 A1 WO 2023002826A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
imaging
distance
blur correction
imaging system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/026044
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智司 松井
範一 勝山
祐輔 松本
祐介 足立
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Priority to JP2023536671A priority Critical patent/JPWO2023002826A1/ja
Publication of WO2023002826A1 publication Critical patent/WO2023002826A1/ja
Priority to US18/413,646 priority patent/US12432450B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B5/00—Adjustment of optical system relative to image or object surface other than for focusing
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B5/00—Adjustment of optical system relative to image or object surface other than for focusing
    • G03B5/04—Vertical adjustment of lens; Rising fronts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/685—Vibration or motion blur correction performed by mechanical compensation
    • H04N23/687—Vibration or motion blur correction performed by mechanical compensation by shifting the lens or sensor position
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20201—Motion blur correction

Definitions

  • the present disclosure relates to an imaging system that corrects blur according to the movement of a mobile object, and a mobile object equipped with the same.
  • Inspection efficiency can be greatly improved by capturing images of infrastructure equipment while moving by a mobile object and detecting defective locations by image processing of the captured images instead of visual inspection by humans.
  • image processing of the captured images instead of visual inspection by humans.
  • blurring occurs in the captured image.
  • Patent Document 1 uses a saccade mirror technique to correct blur caused by camera movement during exposure. Blurring is reduced by irradiating light on an object to be imaged, and reflecting the light reflected by the object by a mirror that rotates for a predetermined exposure time and making the reflected light enter the camera.
  • the present disclosure provides an imaging system that suppresses deterioration of blur correction even when imaging while moving, and a moving body equipped with the same.
  • An imaging system is an imaging system installed in a mobile body, and includes an imaging device that captures an image of at least a part of a structure around the mobile body as an imaging target, and an imaging device that captures images while the mobile body is moving. It has a blur correction amount setting unit that sets the blur correction amount based on the subject distance from the imaging device to the imaging target, and the blur during imaging is corrected using the set blur correction amount. and a blur correction device for correcting. A first image is acquired by the imaging device while being subjected to blur correction by the blur correction device with a blur correction amount set based on the first distance as the object distance.
  • the imaging device After the first image, the imaging device acquires a second image while performing blur correction by the blur correction device with a blur correction amount set based on the second distance as the object distance.
  • the imaging system includes an image processing device that calculates pixel shift amounts of common feature portions in the first image and the second image.
  • the image processing device calculates a third distance as the subject distance of a third image captured after the second image, based on the pixel movement amount and the movement amount of the moving body.
  • the mobile object of the present disclosure includes the imaging system described above.
  • the imaging system of the present disclosure and the moving body equipped with the same, it is possible to provide an imaging system and a moving body equipped with the same that suppress degradation of blur correction even in imaging while moving.
  • FIG. 1 shows a vehicle with an imaging system 1 is a diagram showing the configuration of an imaging system according to an embodiment;
  • FIG. Explanatory diagram for explaining blur correction of an imaging system Explanatory diagram explaining general subject distance
  • Explanatory diagram showing subject distance in the embodiment Explanatory diagram for explaining an image captured without blur correction during exposure
  • 4 is a flowchart showing blurring imaging processing according to the embodiment; Graph showing the relationship between movement speed change, exposure time timing, and movement blur correction angle Graph showing effect of embodiment
  • a moving object is a vehicle 3 such as an automobile, and a case where the imaging system 1 is attached to the top of the vehicle 3 will be described as an example.
  • FIG. 1 is a diagram for explaining an imaging system 1.
  • FIG. FIG. 2 is a block diagram showing the internal configuration of the imaging system 1.
  • a vehicle 3 is traveling in a tunnel 5, for example.
  • holes 5b and cracks 5c are generated in the wall surface 5a in the tunnel 5.
  • FIG. 1 is a diagram for explaining an imaging system 1.
  • FIG. 2 is a block diagram showing the internal configuration of the imaging system 1.
  • a vehicle 3 is traveling in a tunnel 5, for example.
  • holes 5b and cracks 5c are generated in the wall surface 5a in the tunnel 5.
  • the object to be imaged by the imaging system 1 is at least part of the structure around the vehicle 3, and is an object that moves relatively according to the movement speed of the vehicle 3 as the vehicle 3 moves.
  • the imaging target area 9 is an area to be acquired as an image in this imaging target.
  • the side surface and bottom surface of an overpass, a utility pole, and an electric wire may be captured. This makes it possible to detect holes, cracks, floats, peelings, seams, tilts of utility poles, and flexures of electric wires in the imaging target by image processing from the acquired images.
  • the vehicle 3 includes a speed detection device 3a that detects the moving speed of the vehicle 3.
  • the speed detection device 3a is, for example, a vehicle speed sensor that detects the moving speed from the rotation speed of the axle of the vehicle 3. As shown in FIG.
  • the imaging system 1 is installed on the upper surface of the vehicle 3.
  • the imaging system 1 is fixed so as to capture an image of a structure above the vehicle 3, for example, the wall surface 5a of the tunnel 5 in FIG. may be installed to capture the
  • the imaging system 1 includes an imaging device 11, a blur correction device 13, an image processing device 14, and a storage unit 15.
  • the imaging device 11 captures an image of structures around the vehicle 3 , and captures an image of the wall surface 5 a of the tunnel 5 when the vehicle 3 travels through the tunnel 5 , for example.
  • the imaging device 11 includes a camera body 21 , a lens 23 as an optical system lens, a shutter 24 , an imaging element 25 and a camera control section 27 .
  • the camera body 21 has a replaceable lens 23 attached thereto, and houses an imaging device 25 and a camera control section 27 .
  • An imaging element 25 is arranged at the position of the focal length F of the lens 23 .
  • the imaging device 25 is, for example, a solid-state imaging device such as a CCD image sensor, a CMOS image sensor, or an infrared image sensor.
  • the camera body 21 is arranged on the vehicle 3 so that the orientation of the lens 23 is parallel to the moving direction of the vehicle 3 .
  • the camera body 21 is arranged so that the lens 23 faces the front or rear of the vehicle 3 .
  • the camera body 21 and the lens 23 may be integrated, for example, like a video camera, and a blur correction mechanism 31 is arranged outside the camera body 21 and the lens 23 that are integrated.
  • the camera control unit 27 opens the shutter 24 while receiving the exposure instruction signal from the correction processing unit 33 .
  • the shutter 24 may be configured to open and close a plurality of blade diaphragms, or may be an electronic shutter.
  • the blur correction device 13 corrects the optical path of light incident on the imaging system 1 so that blurring of the image of the imaging target area 9 is reduced even if the imaging device 11 captures images while the vehicle 3 is moving.
  • the blur correction device 13 includes a blur correction mechanism 31 and a correction processing section 33 .
  • the blur correction mechanism 31 corrects the optical path of the reflected light L1 of ambient light reflected by the imaging target area 9 in accordance with the movement of the vehicle 3 .
  • the blur correction mechanism 31 includes a mirror 41 and a mirror driver 43 that drives the mirror 41 to rotate.
  • the blur correction mechanism 31 is a pan-tilt mechanism that drives a lens barrel, in which the lens 23 and the imaging element 25 are integrated, around a rotation axis, for example, in the pan direction and the tilt direction. good too.
  • the pan-tilt mechanism has a drive unit that rotates the integrated lens 23 and imaging element 25 .
  • the driving part is, for example, a motor.
  • the blur correction mechanism 31 may have a tilt function for vertically rotating the camera body 21 and the lens 23 and a pan function for horizontally rotating the camera body 21 and the lens 23 . Further, the blur correction mechanism 31 may be a mechanism that rotates the entire imaging device 11, or may have an optical system lens driving mechanism and an imaging device driving mechanism. In this way, when blur correction is performed without using a mirror, the lens 23 and the camera body 21 are installed so that the orientation of the lens 23 and the moving direction of the vehicle 3 are perpendicular.
  • the mirror 41 is rotatably arranged so as to face the lens 23 .
  • the mirror 41 is rotatable, for example, in both forward and reverse clockwise directions, and the rotatable angular range may be less than 360 degrees or greater than or equal to 360 degrees. good too.
  • the mirror 41 totally reflects ambient light reflected by the imaging target toward the imaging device 25 of the imaging device 11 .
  • the mirror drive unit 43 rotates the mirror 41 from the initial angle to the instructed angle for correcting the imaging blur, rotates the mirror 41 to the instructed angle, and then returns it to the initial angle.
  • the mirror driver 43 is, for example, a motor.
  • the rotation angle of the mirror 41 is limited by the mechanical restrictions of the mirror driving section 43, and the mirror 41 can be rotated up to the maximum swing angle of the mirror 41 determined by this restriction.
  • FIG. 3A is an explanatory diagram for explaining blur correction of the imaging system 1.
  • FIGS. 4A and 4B show images captured without blur correction
  • FIG. 4A shows an image at the start of imaging
  • FIG. 4B shows an image at the end of imaging.
  • the imaging system 1 located at position A moves to position B together with the vehicle 3 after the exposure time Tp.
  • Image pickup is started at position A, and an image ImA that can be acquired at this timing is shown in FIG. 4(a).
  • the hole 5b of the imaging target area 9 is imaged in the image ImA.
  • the image ImA is a dark image and not sharp because the exposure time is not long enough.
  • the imaging target area 9 will relatively move in the direction opposite to the moving direction of the vehicle 3. Therefore, the image ImAa in which the hole 5b relatively moves as shown in FIG. becomes.
  • the pixel movement amount P is detected as the amount of blurring.
  • the image captured by the imaging device 11 while the vehicle 3 is moving is a blurred image.
  • the moving direction side end of the mirror 41 rotates the mirror 41 in a direction that offsets the relative movement of the imaging target during the exposure time.
  • the system 1 can image the same imaging target area 9 in the captured image during the exposure time, and can acquire an image with significantly reduced blurring.
  • the mirror 41 is rotated clockwise in FIG. 1 so that the end portion of the mirror 41 on the moving direction side turns around the imaging target side during the exposure time. By rotating the mirror 41, the pixel movement amount P is corrected to zero in the image ImAa.
  • the blur correction amount during imaging changes according to the amount of pixel movement of the feature points included in the imaging target from the start to the end of imaging.
  • the pixel movement amount is proportional to the object magnification M in addition to the vehicle movement amount.
  • the subject magnification M is inversely proportional to the subject distance from the imaging device 25 to the imaging target. Therefore, when the object distance changes, the pixel movement amount P also changes. Unless the blur correction amount is set according to the change in the object distance, the accuracy of the blur correction during imaging decreases.
  • FIG. 3B is an explanatory diagram for explaining a general object distance D.
  • FIG. 3C is an explanatory diagram showing the subject distance D in the embodiment.
  • the subject distance D[m] is the distance from the principal point Lp of the lens Ln arranged between the imaging target Sb, which is the subject, and the imaging device 25 to the imaging target Sb. Also, the distance from the principal point Lp of the lens Ln to the imaging device 25 is the focal length F of the lens Ln.
  • the principal point Lp of the lens Ln is not necessarily positioned at the center of the lens Ln depending on the lens shape of the lens Ln, or if the lens Ln is composed of a plurality of lenses. may be located.
  • the imaging range Sa as the subject size that fits within the angle of view is determined by the ratio of similar shapes of the two triangular regions Sq1 and Sq2. .
  • the parallel light L from the shooting range a is incident on the lens Ln, it is condensed on the imaging device 25 .
  • the image movement distance for each imaging interval and the movement distance of the vehicle 3 for each imaging interval are used.
  • the imaging interval may be a time interval or a distance interval.
  • the image is captured by exposure in synchronization with the pulse update of the vehicle speed signal output from the speed detection device 3a, so the image is captured at equal distance intervals.
  • the image may be captured at a constant frame rate based on the moving speed output from the speed detection device 3a.
  • the correction processing unit 33 controls the blur correction mechanism 31 for blur correction.
  • the correction processing unit 33 includes a blur correction amount setting unit 51 , a vehicle movement amount calculation unit 53 , a pixel movement amount calculation unit 55 , a subject magnification calculation unit 57 , a subject distance calculation unit 59 , and a pixel resolution calculation unit 61 . , provided.
  • the correction processing unit 33 is a circuit that can be realized with a semiconductor element or the like.
  • the correction processing unit 33 can be configured by, for example, a microcomputer, CPU, MPU, GPU, DSP, FPGA, or ASIC.
  • the function of the correction processing unit 33 may be configured only by hardware, or may be realized by combining hardware and software.
  • the correction processing unit 33 reads data and programs stored in the storage unit 15 and performs various arithmetic processing, thereby realizing predetermined functions.
  • the blur correction amount setting unit 51 sets the mirror swing angle of the mirror 41 during imaging. ⁇ is calculated in the following flow.
  • a vehicle movement amount La of the vehicle 3 moving during the exposure time Tp from the start of imaging to the end of imaging is calculated from the moving speed V1 and the exposure time Tp by the following equation (1).
  • La [mm] V1 [km/h] x 106 x Tp [ms]/(602 x 103) ( 1 )
  • the movement amount P of pixels on the image sensor 25 during the exposure time Tp from the start of imaging to the end of imaging is calculated from the vehicle movement amount La of the vehicle 3 and the subject magnification M by the following equation (2).
  • P [mm] La [mm] x M (2) formula
  • the mirror swing angle ⁇ is half the size of the motion blur correction angle ⁇ , it is calculated by the following equation (4).
  • ⁇ ⁇ /k (4)
  • k is the mirror swing angle ⁇ as the mechanism swing angle of the driving mechanism and the movement blur correction angle as the optical correction angle for correcting the light incident on the lens 23
  • k 2
  • the conversion coefficient k 2.
  • the blur correction amount setting unit 51 calculates the mirror swing angle ⁇ of the mirror 41 .
  • the imaging device 11 can receive light from the same imaging target region 9 during the exposure time. It is possible to suppress the occurrence of movement blur in captured images.
  • the focal length F in formulas (2) and (3) is a value determined by the lens 23, and the subject magnification M is a value determined by the focal length F and the subject distance.
  • the pixel movement amount qa [px] in pixel unit and the pixel movement amount qb [mm] in length unit which is the pixel movement amount by which the feature point moves during the imaging interval, and the pixel movement amount qb [mm]
  • a distance D [m] a vehicle movement amount Lb [mm] by which the vehicle 3 moves during the imaging interval, a pixel pitch s [ ⁇ m/px] of the captured image, and a vertical pixel number Vpx [px] of the captured image.
  • the focal length F [mm] of the lens 23, the pixel pitch s [ ⁇ m/px] of the captured image, and the number of vertical pixels Vpx [px] of the captured image are predetermined values.
  • the pixel movement amount qa in pixel units and the pixel movement amount qb in length units are simply referred to as pixel movement amount q.
  • FIG. 4 shows feature points that move during imaging
  • FIG. 2 shows an image ImC of the points.
  • the image processing device 14 performs image matching processing on the two captured images ImB and ImC captured in different frames, for example, to calculate the pixel movement amount q of the feature point such as the hole 5b between the captured frames.
  • the pixel movement amount q may be one point in the captured images ImB and ImC, or may be an average value of respective pixel movement amounts q1, q2, . . . qn at a plurality of points.
  • the image processing device 14 is a circuit that can be realized with a semiconductor element or the like.
  • the image processing device 14 can be configured by, for example, a microcomputer, CPU, MPU, GPU, DSP, FPGA, or ASIC.
  • the functions of the image processing device 14 may be configured only by hardware, or may be realized by combining hardware and software.
  • the image processing device 14 reads data and programs stored in the storage unit 15 and performs various arithmetic processing, thereby realizing predetermined functions.
  • an image is captured by exposure in synchronization with the pulse update of the vehicle speed signal output from the speed detection device 3a. Accordingly, instead of detecting the moving speed of the vehicle 3, the moving amount of the vehicle 3 can be directly calculated from the vehicle speed signal.
  • the vehicle movement amount calculator 53 calculates the vehicle movement amount Lb of the vehicle 3 between the two captured images.
  • the subject magnification calculator 57 calculates the subject magnification M using the following equation (6).
  • the subject distance calculator 59 calculates the subject distance D2 using the following equation (7).
  • D F/M/10 3 (7)
  • the vertical imaging range R satisfies the following relational expression (8).
  • R Vpx ⁇ s/10 3 ⁇ D/F (8)
  • the pixel resolution calculator 61 can associate the pixel unit of the captured image with the length unit of the captured image from the calculated pixel resolution of the captured image, and may measure the length of the crack in the captured image.
  • the captured image converted into units may be stored in the storage unit 15 .
  • the storage unit 15 is a storage medium that stores programs and data necessary for realizing the functions of the correction processing unit 33 .
  • the storage unit 15 can be implemented by, for example, a hard disk (HDD), SSD, RAM, DRAM, ferroelectric memory, flash memory, magnetic disk, or a combination thereof.
  • the operation unit 7 is an input device for the user to give instructions to the correction processing unit 33 .
  • the operation unit 7 may be an input device dedicated to the imaging system 1 or may be a mobile terminal such as a smartphone. When a mobile terminal is used as the operation unit 7, data is transmitted and received between the operation unit 7 and the correction processing unit 33 by wireless communication.
  • the user may use the operation unit 7 to indicate whether the imaging target area is an indoor dark area such as a tunnel or an outdoor bright area such as a slope of a mountain. In the case of capturing images at regular time intervals, the operation unit 7 may be used to instruct the frame rate Tf. Further, the operation unit 7 has a display unit, and displays a warning or the like from the imaging system 1 to notify the user.
  • FIG. 6 is a flowchart showing imaging processing performed by the imaging system 1 .
  • FIG. 7 is a graph showing the relationship between changes in moving speed, exposure time timing, captured images, and vehicle speed update flags.
  • FIG. 7(a) is a graph showing the moving speed of the vehicle 3 that changes over time.
  • FIG. 7B is a graph showing timing of exposure time for each frame.
  • FIG. 7(c) is an explanatory diagram showing an image captured for each frame.
  • FIG. 7(d) is a graph showing the timing at which the vehicle speed update flag rises.
  • the imaging process shown in FIG. 6 is started, for example, when an instruction to start imaging is given from the operation unit 7 while the vehicle 3 is moving.
  • a blur correction amount setting unit 51 of the correction processing unit 33 calculates a moving blur correction angle ⁇ , which is a blur correction amount, based on a predetermined distance da1 stored in the storage unit 15 as the subject distance D1.
  • Set (step S1) Since the subject distance D2 has not been calculated from the captured image immediately after the start of imaging, a predetermined distance da1 may be used or, for example, a value input from the operation unit 7 may be used.
  • the set or updated object distance value is D1
  • the calculated object distance value is D2.
  • the correction processing unit 33 calculates the mirror swing angle ⁇ based on the set motion blur correction angle ⁇ , and calculates the mirror rotation speed based on the calculated mirror swing angle ⁇ and the set exposure time Tp. .
  • the correction processing unit 33 further rotates the mirror 41 at the calculated mirror rotation speed, and the mirror 41 starts rotating from a predetermined initial angle. As a result, blur correction during imaging by the imaging device 11 is performed (step S2). At the same time, the correction processing section 33 continues to send a Hi signal instructing exposure to the camera control section 27 for the exposure time Tp.
  • the camera control unit 27 acquires the first frame image Im0 by opening the shutter 24 and exposing while receiving the Hi signal, and the correction processing unit 33 captures the image from the speed detection device 3a. Movement information of the vehicle 3 between frames is acquired (step S3).
  • the correction processing section 33 continues to send a Low signal instructing to stop exposure to the camera control section 27 . While the camera control unit 27 is receiving the Low signal, the shutter 24 is closed, and the correction processing unit 33 causes the mirror driving unit 43 to rotate the mirror 41 in the reverse direction to return the mirror 41 to the initial angle. Note that the mirror drive unit 43 may rotate the mirror 41 forward to return the mirror 41 to the initial angle.
  • the correction processing unit 33 determines whether or not the number of acquired captured images is two or more (step S4), and if it determines that the number of acquired captured images is not two or more (No in step S4). , the process returns to step S1, and the imaging process of the next frame, for example, the image Im1 of the second frame is started.
  • the blur correction amount setting unit 51 sets the moving blur correction angle ⁇ , which is the amount of blur correction, based on the same distance da1 as in the first frame as the subject distance D1. Thereafter, imaging processing similar to that for the first frame is performed, and the imaging device 11 captures the first image Im1 of the second frame shown in FIG.
  • the captured image Im ⁇ b>1 is stored in the storage unit 15 .
  • the pixel shift amount The calculation unit 55 calculates the pixel movement amount qb between two or more images, and the vehicle movement amount calculation unit 53 calculates the distance from the start of imaging of the current imaging frame to the start of imaging of the next imaging frame based on the acquired movement information.
  • a vehicle movement amount Lb by which the vehicle 3 has moved is calculated (step S5).
  • the calculated vehicle movement amount Lb is stored in the storage unit 15 in association with the captured image.
  • the subject distance calculator 59 calculates the subject distance D2 based on the pixel movement amount qb between two or more captured images and the vehicle movement amount Lb (step S6).
  • the image Im1 and the image Im0 each capture a common feature portion g1, and the image processing device 14 detects the pixel movement amount q1 between the image Im0 and the image Im1 by image matching processing. Based on this pixel movement amount q1, the subject distance calculator 59 calculates a distance ds2 as a new subject distance D2.
  • the subject distance calculator 59 updates the subject distance D1 to the subject distance D2 (step S7).
  • the subject distance D1 calculated based on the captured image of the nth frame is updated as the subject distance D1 of the (n+1)th frame.
  • step S8 the imaging system 1 finishes imaging while the vehicle 3 is moving.
  • step S8 the process returns to step S1 again to capture an image in the next frame.
  • the third frame image Im2 is captured as the next frame.
  • the blur correction amount setting unit 51 sets the movement blur correction angle ⁇ , which is the blur correction amount, based on the second distance ds2 different from that in the second frame as the subject distance D1.
  • the imaging process similar to that for the second frame is performed, and the imaging device 11 captures the second image Im2 of the third frame shown in FIG.
  • the captured image Im ⁇ b>2 is stored in the storage unit 15 .
  • the image processing device 14 captures a first image Im1 captured based on a first distance (eg, ds1) and a second image Im1 captured based on a second distance (eg, ds2).
  • a pixel movement amount q2 of a common characteristic portion g1 imaged with the second image Im2 is detected.
  • the subject distance calculator 59 calculates a distance ds3 as a new subject distance D2.
  • the movement blur correction angle ⁇ can be accurately set according to the changing subject distance, and the blur correction accuracy can be improved. can.
  • the subject distance D1 changes.
  • the blur correction accuracy of the captured image is reduced, so the movement amount calculation accuracy by image processing is also reduced, and the calculation accuracy of the subject distance D is also reduced.
  • the improvement of blur correction accuracy and the improvement of the calculation accuracy of the subject distance D are mutually related, so it is not possible to improve the accuracy of one after securing the accuracy of the other.
  • image processing it is possible to detect the pixel movement amount even if the blur does not become zero. Therefore, for example, as shown in FIG. Images Im1 and Im2 that have undergone blur correction are captured using .
  • a blur-corrected image Im3 is captured using the calculated second distance ds2, and a third distance ds3 as the subject distance is calculated by image processing based on the images Im2 and Im3.
  • the blur-corrected image Im4 with improved precision can be captured. Therefore, each time the subject distance D is updated, the calculated subject distance approaches the true value Dt, whereby an image with improved blur correction accuracy can be captured. Accuracy can also be improved.
  • the imaging system 1 includes the imaging device 11 having the imaging device 25 that captures an imaging target that is at least a part of the surroundings of the vehicle 3 away from the vehicle 3, and the vehicle that calculates the amount of movement of the vehicle 3.
  • a movement amount calculation unit 53 is provided, and a blur correction amount is set based on a subject distance D1 from the imaging device 11 to an imaging target for blurring in the moving direction when the imaging device 11 captures an image while the vehicle 3 is moving, and a blur correction device 13 that corrects blur during imaging using the set blur correction amount.
  • the image processing device 14 calculates the pixel movement amount q of the common characteristic portion g1 in the second image Im2 captured respectively.
  • the subject distance calculation unit 59 of the blur correction device 13 generates a first image Im1 with the pixel movement amount q and the first distance ds1 as the subject distance D1, and a second image Im2 with the second distance ds2 as the subject distance D1.
  • a third distance ds3 is calculated as the subject distance D2 of a third image Im3 captured after the second image Im2 based on the vehicle movement amount Lb of the vehicle 3 at the imaging interval of .
  • the blur correction device 13 updates the subject distance D1 to the third distance ds3, and sets the blur correction amount when capturing the third image Im3 based on the updated subject distance D1. Since the blur correction amount is updated based on the updated subject distance D1, the third image Im3 with improved blur correction can be captured.
  • the first distance ds1 and the second distance ds2, which are the subject distances when the first image Im1 and the second image Im2 are captured, are the pixels of the two previously captured images, respectively. It may be a value detected based on the amount of movement and the amount of movement of the moving object between the imaging intervals of the two images.
  • the subject distance D1 of the initial image Im0 and the subject distance D1 of the second image Im2 may be the same predetermined value ds1.
  • the first distance ds1 and the second distance ds2 are based on the pixel movement amounts q1 and q2 of the images Im1 and Im2 captured before the third image Im3 and the vehicle movement amount Lb of the vehicle 3. is the detected value.
  • the blur correction device 13 may update the average value of the object distance calculated two frames before and the object distance calculated one frame before as the object distance D1. For example, as the subject distance D2 when capturing the fourth image Im4, the average value of the distance ds2 calculated two frames before and the distance ds1 calculated one frame before is calculated, and the calculated subject distance D2 is the average value. may be updated as the object distance D1.
  • the amount of pixel movement of feature points obtained by image matching of two captured images is used to calculate the subject distance of the third image in three consecutive images, but the present invention is not limited to this. If the image matching processing of the two captured images is delayed from the imaging cycle, the object distance may be calculated with a delay of several frames.
  • the image processing device 14 outputs a warning signal to, for example, the operation unit 7, a speaker, or an image display device when the detected second distance exceeds a predetermined range. good too. This allows the driver of the vehicle 3 to recognize that the second distance exceeds the predetermined range.
  • a warning signal is output when the object distance is shorter or longer than expected.
  • the subject magnification M increases and the pixel movement amount also increases. As a result, the swing angle of the mirror required for blur correction increases, and may exceed the maximum swing angle. In addition, if the subject distance is longer than expected, the subject magnification M becomes small, and the pixel resolution [mm/px], which is the actual size per pixel, becomes rough, and the required inspection accuracy may not be obtained. As described above, if the second distance exceeds the predetermined range, deterioration of blur correction cannot be reduced or necessary inspection accuracy cannot be ensured. Can be retested. For example, if the cause is that the vehicle 3 meanders and the subject distance has changed, the vehicle 3 is re-traveled to perform the inspection again. Also, if there is no problem with the running of the vehicle 3 but the cause is a change on the subject side, replacement of the lens 23 can be urged.
  • the imaging device 11 and the blur correction mechanism 31 are separate bodies, but this is not the only option.
  • the imaging device 11 may have a housing 22 that connects a lens 23 and a camera body 21 , and the housing 22 may house the blur correction mechanism 31 .
  • the movement blur correction angle ⁇ corresponding to the movement amount P of the pixel is calculated for blur correction, and the mirror 41 is rotated by the mirror swing angle ⁇ corresponding to the movement blur correction angle ⁇ .
  • the imaging element 25 and the lens 23 may be driven in translation instead of the moving blur correction angle ⁇ .
  • the lens 23 of the imaging device 11 is directed directly toward the imaging target region 9 (see FIG. 1), and the blur correction mechanism 31B is directed toward the imaging target region 9 , an imaging device driving mechanism 71 for driving the imaging device 25 in translation, and an optical system lens driving mechanism 73 for driving the lens 23 in translation.
  • the imaging element drive mechanism 71 and the optical system lens drive mechanism 73 are, for example, translational drive mechanisms composed of pinion gears, racks, and motors.
  • the amount of translational driving by which the imaging element driving mechanism 71 drives the imaging element 25 is equal to the amount of movement P of the pixel
  • the amount of translational driving by which the optical system lens driving mechanism 73 drives the lens 23 is the amount of movement P of the pixel multiplied by a constant m. is the value
  • the constant m is a number of 0 or more.
  • the amount of translation changes with respect to the amount of lens driving. Further, instead of driving the lens 23 and the imaging element 25 individually, the imaging apparatus 11 as a whole may be driven translationally.
  • the imaging system 1 may include a speed detection device that detects the moving speed of the imaging system 1 .
  • the speed detection device may use a GPS (Global Positioning System) system or an acceleration sensor.
  • the imaging system 1 images the walls above and on the side of the vehicle 3, but the imaging system 1 is not limited to this.
  • the imaging system 1 may image the road surface below the vehicle 3 . Potholes, cracks, ruts, etc. occurring on the road surface can be detected by image processing from the captured image.
  • the mobile body is not limited to the vehicle 3, and may be a vehicle such as a train or a motorcycle that travels on the ground, a ship that travels on the sea, or an aircraft that flies in the sky, such as an airplane or a drone.
  • the imaging system 1 images the bottom of bridge piers and bridge girders, and structures constructed along the shore. If the moving object is a train, the deflection of the overhead wire can be detected by imaging the overhead wire.
  • the vehicle movement amount calculation unit 53 calculates the movement amount of the vehicle 3 as a moving object, when the moving object is something other than a vehicle, the vehicle movement amount calculation unit 53 calculates the movement amount of the moving object. .
  • an image is captured by light reflected by the imaging target area 9 from ambient light, but this is not the only option.
  • Light may be emitted from the moving object toward the imaging target region 9, and an image of the reflected light of the emitted light may be picked up.
  • An imaging system of the present disclosure is an imaging system installed in a mobile body, and is an imaging device having an imaging element that captures an image of an imaging target that is at least part of the surroundings of the mobile body and is remote from the mobile body. and a moving object movement amount calculating unit that calculates the amount of movement of the moving object, and the blur in the moving direction when the image capturing device captures an image while the moving object is moving is calculated based on the subject distance from the image capturing device to the image capturing object. and a blur correction device that sets a blur correction amount and corrects blur during imaging using the set blur correction amount.
  • the imaging system includes an image processing device that calculates a pixel shift amount of a characteristic portion common to each of the first image captured by the imaging device and the second image captured after the first image.
  • the blur correction device is based on a pixel shift amount and a shift amount of a moving object between imaging intervals of a first image with the first distance as the subject distance and a second image with the second distance as the subject distance. Based on this, the third distance is calculated as the subject distance of the third image captured after the second image.
  • the subject distance can be detected even while the moving object is moving, so blur correction can be performed according to changes in the subject distance, and the accuracy of blur correction can be improved.
  • the blur correction device updates the subject distance to the third distance, and sets the blur correction amount when capturing the third image based on the updated subject distance.
  • the first distance and the second distance which are the subject distances when the first image and the second image are captured, are respectively forward It is a value detected based on the amount of pixel movement between the two captured images and the amount of movement of the moving object between the two images.
  • the first distance and the second distance are the same predetermined value immediately after the start of imaging.
  • the blurring correction device uses the second distance calculated when capturing the second image and the third distance calculated when capturing the third image. is updated as the subject distance when capturing the third image.
  • the subject distance changes when the ratio between the pixel movement amount and the moving object movement amount changes.
  • the image processing device calculates the pixel resolution of the second image based on the second distance, and calculates the second image based on the calculated pixel resolution. converts the unit of the size of a predetermined pixel in the image from the pixel unit to the length unit.
  • the image processing device calculates the second pixel movement amount of the characteristic portion common to the second image and the third image.
  • the blur correction device calculates a fourth distance as the object distance based on the second pixel movement amount and the movement amount of the moving object between the imaging intervals of the second image and the third image.
  • the blur correction device updates the subject distance to a fourth distance, and sets a blur correction amount when capturing a fourth image to be captured after the third image based on the updated subject distance.
  • the image processing device outputs a warning signal when the detected second distance exceeds a predetermined range.
  • the blur correction device includes a mirror that totally reflects ambient light reflected by an image pickup target toward the image pickup device, and rotates the mirror. and a mirror drive unit, and the blur correction amount corresponds to the rotation angle of the mirror.
  • the imaging device has an optical system lens integrated with the imaging device
  • the blur correction device has an optical system lens integrated with the imaging device.
  • a drive unit for rotating the element, and the blur correction amount corresponds to the rotation angle of the drive unit.
  • the imaging device has an optical system lens focused on the imaging device, and the blur correction device combines the optical system lens and the imaging device.
  • a translational drive mechanism is provided for translational driving in a direction within the imaging plane of the imaging device, and the blur correction amount corresponds to the translational amount of the optical system lens.
  • a mobile object according to the present disclosure includes the imaging system according to any one of (1) to (13). As a result, while the mobile body is moving, the imaging system can capture an image of the surroundings of the mobile body with reduced blurring.
  • the imaging system described in the present disclosure is realized by hardware resources such as processors, memories, and cooperation with programs.
  • the present disclosure is applicable to an imaging system installed on a moving mobile object.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
PCT/JP2022/026044 2021-07-21 2022-06-29 撮像システム、及び、それを備えた移動体 Ceased WO2023002826A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023536671A JPWO2023002826A1 (https=) 2021-07-21 2022-06-29
US18/413,646 US12432450B2 (en) 2021-07-21 2024-01-16 Imaging system and moving body provided with same

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021120716 2021-07-21
JP2021-120716 2021-07-21

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/413,646 Continuation US12432450B2 (en) 2021-07-21 2024-01-16 Imaging system and moving body provided with same

Publications (1)

Publication Number Publication Date
WO2023002826A1 true WO2023002826A1 (ja) 2023-01-26

Family

ID=84979972

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/026044 Ceased WO2023002826A1 (ja) 2021-07-21 2022-06-29 撮像システム、及び、それを備えた移動体

Country Status (3)

Country Link
US (1) US12432450B2 (https=)
JP (1) JPWO2023002826A1 (https=)
WO (1) WO2023002826A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026004723A1 (ja) * 2024-06-26 2026-01-02 パナソニックＩｐマネジメント株式会社 撮像システム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003207813A (ja) * 2002-01-16 2003-07-25 Ricoh Co Ltd 撮像装置
JP2005318568A (ja) * 2004-03-30 2005-11-10 Fuji Photo Film Co Ltd 画像補正装置、および画像補正方法
JP2007020109A (ja) * 2005-07-11 2007-01-25 Olympus Imaging Corp 電子的ぶれ補正装置
JP2012128356A (ja) * 2010-12-17 2012-07-05 Nikon Corp ブレ補正装置及び光学機器
WO2020189224A1 (ja) * 2019-03-15 2020-09-24 ソニーセミコンダクタソリューションズ株式会社 撮像装置、固体撮像素子、カメラモジュール、駆動制御部、および撮像方法

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7982772B2 (en) * 2004-03-30 2011-07-19 Fujifilm Corporation Image correction apparatus and image correction method for correcting image blur using a mobile vector
KR102282458B1 (ko) * 2015-03-23 2021-07-27 한화테크윈 주식회사 영상안정화장치 및 방법
JP5729622B2 (ja) 2013-10-22 2015-06-03 国立大学法人 東京大学 ブラーレス画像撮像システム
JP6592335B2 (ja) * 2015-11-05 2019-10-16 キヤノン株式会社 像ブレ補正装置及び方法
JP2020118789A (ja) * 2019-01-22 2020-08-06 キヤノン株式会社 撮像装置、撮像システム、撮像装置の制御方法、およびプログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003207813A (ja) * 2002-01-16 2003-07-25 Ricoh Co Ltd 撮像装置
JP2005318568A (ja) * 2004-03-30 2005-11-10 Fuji Photo Film Co Ltd 画像補正装置、および画像補正方法
JP2007020109A (ja) * 2005-07-11 2007-01-25 Olympus Imaging Corp 電子的ぶれ補正装置
JP2012128356A (ja) * 2010-12-17 2012-07-05 Nikon Corp ブレ補正装置及び光学機器
WO2020189224A1 (ja) * 2019-03-15 2020-09-24 ソニーセミコンダクタソリューションズ株式会社 撮像装置、固体撮像素子、カメラモジュール、駆動制御部、および撮像方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026004723A1 (ja) * 2024-06-26 2026-01-02 パナソニックＩｐマネジメント株式会社 撮像システム

Also Published As

Publication number Publication date
US20240276099A1 (en) 2024-08-15
JPWO2023002826A1 (https=) 2023-01-26
US12432450B2 (en) 2025-09-30

Similar Documents

Publication Publication Date Title
KR102515482B1 (ko) 카메라 패닝 또는 모션에서 배경 블러링을 생성하는 시스템 및 방법
CN104954668A (zh) 变焦倍率算出装置及其控制方法、摄像装置以及移动体
US12141947B2 (en) Image processing method, electronic device, and computer-readable storage medium
WO2019129061A1 (zh) 一种镜头模组和镜头模组的控制方法
US12149826B2 (en) Image processing method and device, electronic device, and computer-readable storage medium
JP2010250156A (ja) 電子カメラ
WO2023002826A1 (ja) 撮像システム、及び、それを備えた移動体
US12581190B2 (en) Imaging system and moving body provided with same
JP4134869B2 (ja) 撮像装置
CN118573989A (zh) 具有水平摇摄和/或俯仰摇摄操作的自校准功能的安防摄像头
JP2019092036A (ja) 撮像装置及び制御方法
US20250350836A1 (en) Imaging system
JPWO2023243656A5 (https=)
JP3929778B2 (ja) 撮像装置
WO2023243656A1 (ja) 撮像システム、及び、それを備えた移動体
WO2023234356A1 (ja) 撮像システム、及び、それを備えた移動体
JP4639278B2 (ja) ジンバルを用いたディジタルカメラによる撮影方法
WO2026004723A1 (ja) 撮像システム
WO2023234360A1 (ja) 撮像システム、及び、それを備えた移動体
JPWO2023234356A5 (https=)
JP2015200737A (ja) 撮像装置
CN115720294B (zh) 视频防抖方法及装置、存储介质、视频采集设备
JPWO2023234360A5 (https=)
WO2024145837A1 (en) Imaging apparatus, imaging-apparatus control method, and computer program product
KR20110100961A (ko) 감시 카메라

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2023536671

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22845762

Country of ref document: EP

Kind code of ref document: A1