WO2021033250A1 - 画像処理装置、撮像装置、システム、画像処理方法およびプログラム - Google Patents

画像処理装置、撮像装置、システム、画像処理方法およびプログラム Download PDF

Info

Publication number
WO2021033250A1
WO2021033250A1 PCT/JP2019/032342 JP2019032342W WO2021033250A1 WO 2021033250 A1 WO2021033250 A1 WO 2021033250A1 JP 2019032342 W JP2019032342 W JP 2019032342W WO 2021033250 A1 WO2021033250 A1 WO 2021033250A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
time stamp
event signal
inverse filter
motion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/032342
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
宏昌 長沼
林　正和
洋輔 栗原
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Interactive Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Interactive Entertainment Inc filed Critical Sony Interactive Entertainment Inc
Priority to CN201980099252.5A priority Critical patent/CN114245983B/zh
Priority to US17/634,300 priority patent/US11954866B2/en
Priority to EP19942090.2A priority patent/EP4020962B1/en
Priority to JP2021541371A priority patent/JP7197715B2/ja
Priority to KR1020227003456A priority patent/KR102916853B1/ko
Priority to PCT/JP2019/032342 priority patent/WO2021033250A1/ja
Publication of WO2021033250A1 publication Critical patent/WO2021033250A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
    • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/00127—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
    • H04N1/00132—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture in a digital photofinishing system, i.e. a system where digital photographic images undergo typical photofinishing processing, e.g. printing ordering
    • H04N1/00167—Processing or editing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/00127—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture
    • H04N1/00132—Connection or combination of a still picture apparatus with another apparatus, e.g. for storage, processing or transmission of still picture signals or of information associated with a still picture in a digital photofinishing system, i.e. a system where digital photographic images undergo typical photofinishing processing, e.g. printing ordering
    • H04N1/00169—Digital image input
    • H04N1/00172—Digital image input directly from a still digital camera or from a storage medium mounted in a still digital camera
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
    • H04N1/32101—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
    • H04N1/32106—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title separate from the image data, e.g. in a different computer file
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/71—Circuitry for evaluating the brightness variation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/04—Synchronising
    • H04N5/06—Generation of synchronising signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/91—Television signal processing therefor
    • H04N5/92—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
    • H04N5/9201—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving the multiplexing of an additional signal and the video signal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20201—Motion blur correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
    • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
    • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
    • H04N2201/3212—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image
    • H04N2201/3215—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image of a time or duration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection

Definitions

  • the present invention relates to an image processing device, an imaging device, a system, an image processing method and a program.
  • An event-driven vision sensor is known in which pixels that detect changes in the intensity of incident light generate signals asynchronously in time.
  • the event-driven vision sensor is advantageous in that it can operate at low power and high speed compared to a frame-type vision sensor that scans all pixels at predetermined cycles, specifically, an image sensor such as a CCD or CMOS. Is. Techniques for such an event-driven vision sensor are described in, for example, Patent Document 1 and Patent Document 2.
  • the present invention provides an image processing device, an imaging device, a system, an image processing method, and a program capable of obtaining an advantageous effect by using an event-driven vision sensor in combination with a frame-type vision sensor. The purpose.
  • a first time stamp given to an image captured by a synchronous scan an event signal generated in response to a change in light intensity at one or more pixels of the image, and.
  • a motion estimation unit that estimates the movement of the subject in the image based on the second time stamp that is given to the event signal and synchronizes with the first time stamp, and an inverse filter generation unit that generates an inverse filter based on the motion.
  • An image processing apparatus includes a filter application unit that applies an inverse filter to an image.
  • Another aspect of the invention includes a first vision sensor that captures an image by synchronous scanning and gives the image a first time stamp, and a sensor associated with one or more pixels of the image.
  • an imaging device comprising a second vision sensor that generates an event signal when the sensor detects a change in light intensity and gives the event signal a second time stamp that is synchronized with the first time stamp. ..
  • a first vision sensor that captures an image by synchronous scanning and gives the image a first time stamp and a sensor associated with one or more pixels of the image.
  • a second vision sensor including a second vision sensor, which generates an event signal when the sensor detects a change in light intensity and gives the event signal a second time stamp that synchronizes with the first time stamp.
  • a motion estimation unit that estimates the movement of the subject in the image based on the event signal and the second time stamp, an inverse filter generator that generates an inverse filter based on the motion, and a filter application unit that applies the inverse filter to the image.
  • a system including an image processing device is provided.
  • the senor comprises a step of capturing an image by synchronous scanning and giving the image a first time stamp and a sensor associated with one or more pixels of the image.
  • An image processing method is provided that includes a step of estimating the movement of the subject in the image based on the stamp, a step of generating an inverse filter based on the movement, and a step of applying the inverse filter to the image.
  • a first time stamp given to an image captured by a synchronous scan an event signal generated in response to a change in light intensity at one or more pixels of the image.
  • a function to estimate the movement of the subject in the image based on the second time stamp synchronized with the first time stamp given to the event signal a function to generate an inverse filter based on the movement, and an inverse filter.
  • a program is provided to realize the function applied to the image on the computer.
  • FIG. 1 is a block diagram showing a schematic configuration of a system according to an embodiment of the present invention.
  • the system 10 includes an RGB camera 100, an EDS (Event Driven Sensor) 200, and an image processing device 300.
  • the RGB camera 100 is an example of a first vision sensor that captures an image by synchronous scanning, and includes an image sensor 110 and a processing circuit 120 connected to the image sensor 110.
  • the image sensor 110 captures an RGB image 101 by synchronously scanning all pixels, for example, at a predetermined cycle or at a predetermined timing according to a user operation.
  • the processing circuit 120 converts, for example, the RGB image 101 into a format suitable for storage and transmission. Further, the processing circuit 120 gives a time stamp 102 to the RGB image 101.
  • the processing circuit 120 gives the RGB image 101 a time stamp 102 indicating at least one of the start or end of scanning by the image sensor 110. For example, in the case of a still image, the time from the start to the end of scanning is also called the exposure time.
  • the EDS 200 is an example of a second vision sensor that generates an event signal when the sensor detects a change in light intensity, and includes a sensor 210 constituting a sensor array and a processing circuit 220 connected to the sensor 210. ..
  • the sensor 210 includes a light receiving element and generates an event signal 201 when an intensity change of incident light, more specifically, a change in brightness is detected. Since the sensor 210 that has not detected the change in the intensity of the incident light does not generate the event signal 201, the event signal 201 is generated asynchronously in time in the EDS 200.
  • the event signal 201 output via the processing circuit 220 includes identification information of the sensor 210 (for example, the position of a pixel), the polarity of the luminance change (increasing or decreasing), and the time stamp 202.
  • the time stamp 102 given to the RGB image 101 and the time stamp 202 given to the event signal 201 are synchronized.
  • the time stamp 102 can be synchronized with the time stamp 202 by providing the RGB camera 100 with the time information used to generate the time stamp 202 in the EDS 200.
  • the time information for generating the time stamps 102 and 202 is independent for the RGB cameras 100 and the EDS 200
  • the time when a specific event (for example, a change in the subject over the entire image) occurs is used as a reference.
  • the time stamp 102 and the time stamp 202 can be synchronized after the fact.
  • the senor 210 of the EDS 200 is associated with one or a plurality of pixels of the RGB image 101 by the calibration procedure of the RGB camera 100 and the EDS 200 executed in advance, and the event signal 201 is the RGB image 101. It is generated according to the change in light intensity in one or more pixels of.
  • FIG. 2 is a diagram for schematically explaining calibration between a camera and a sensor according to an embodiment of the present invention.
  • the common calibration pattern 21 is imaged by the RGB camera 100 and the EDS200 (in the case of the EDS200, the calibration pattern can be imaged by blinking the entire region of the calibration pattern 21 using, for example, the light source 22).
  • the sensor 210 can be associated with one or more pixels of the RGB image 101 by calculating the corresponding parameters between the cameras and sensors from the internal and external parameters of the RGB camera 100 and the EDS 200, respectively.
  • an imaging device that combines an RGB camera 100 and an EDS 200 that have been subjected to the above-mentioned time stamp synchronization and calibration procedures in advance may be provided.
  • FIG. 3 is a diagram for explaining an example of matching between an image and an event signal in one embodiment of the present invention.
  • the RGB image 101 captured by the RGB camera 100 and the event signal 201 output from the EDS 200 at the time corresponding to the scanning of the RGB image 101 are arranged at the pixel positions. ..
  • the event signal 201 is associated with one or more pixels of the RGB image 101 as shown in FIG. That is, the event signal 201 can be superimposed on the RGB image 101.
  • the image processor 300 is implemented by, for example, a computer having a communication interface, a processor, and a memory, and the processor operates according to a program stored in the memory or received via the communication interface. It includes the functions of the motion estimation unit 310, the inverse filter generation unit 320, and the filter application unit 330 realized by the above. The functions of each part will be further described below.
  • the motion estimation unit 310 estimates the motion of the subject in the RGB image 101 based on the time stamp 102 given to the RGB image 101, the event signal 201, and the time stamp 202 given to the event signal 201.
  • the motion estimation unit 310 can estimate the motion of the subject in the RGB image 101 based on the event signal 201 without referring to the RGB image 101 itself.
  • the motion estimation unit 310 is a motion region in which the movement of the subject has occurred in the RGB image 101 from the position change in the time series of the pixels indicating that the luminance change has occurred due to the event signal 201, for example, as in the example described later. And a motion vector indicating the movement of the subject in the RGB image 101 can be estimated.
  • the motion estimation unit 310 estimates motion based on an event signal 201 having a time stamp 202 included in the time from the start to the end of the scan for capturing the RGB image 101.
  • the time from the start to the end of the scan is specified by, for example, two time stamps given to the RGB image 101.
  • the time from the start to the end of the scan can be specified. Can be done.
  • an inverse filter generated based on the movement of the subject generated during the time from the start to the end of scanning the influence of blurring generated in the RGB image 101 due to the movement of the subject is reduced. can do.
  • the inverse filter generation unit 320 generates the inverse filter 321 based on the movement of the subject in the RGB image 101 estimated by the motion estimation unit 310.
  • the inverse filter causes the RGB image 101 to be an image of the original subject by causing a change (filter) opposite to the change (filter) from the original image of the subject caused by the movement of the subject in the RGB image 101. It is a filter intended to be close to.
  • the filter application unit 330 applies the inverse filter 321 to the RGB image 101 to obtain an output image 331. As will be described later, the filter application unit 330 separately provides the RGB image 101 with a filter for compensating for the change caused in the RGB image 101 by applying the inverse filter 321 (for example, filling a blank area by stretching the background). May be applied.
  • FIGS. 4A-4D are diagrams for explaining motion estimation and an inverse filter based on an event signal in one embodiment of the present invention.
  • the subject obj is captured in the RGB image 101. Since the subject OBj moved during the scan by the image sensor 110, blurring occurred in the RGB image 101 as shown in FIG. 4A, and the image of the subject obj was stretched.
  • the event signal 201 immediately after the start of the scanning of the RGB image 101 is an event occurs in the pixel group P 1, which is moved slowly (pixel group P 2, ... , P n-1 ), indicating that an event occurred in the pixel group P n immediately before the end of the scan.
  • the motion estimation unit 310 estimates the motion region R and the motion vector V in the RGB image 101 as shown in FIG. 4C.
  • the motion region R is an region including the pixel groups P 1 , P 2 , ..., P n-1 , P n in which an event occurred between the start and the end of the scan (blurred and stretched).
  • the motion vector V is a vector for moving each pixel in the pixel group P n corresponding to the end point of the motion of each pixel of the pixel group P 1 corresponding to the start of the movement.
  • the motion estimation unit 310 uses the RGB image 101 to estimate the motion region R and the motion vector V. You do not have to refer to it.
  • the inverse filter generation unit 320 When the motion region R and the motion vector V are estimated as described above, the inverse filter generation unit 320 generates an inverse filter 321 that obtains an output image 331 as shown in FIG. 4D by applying it to the RGB image 101. ..
  • the inverse filter 321 acts in a limited way on the pixels in the motion region R to cancel the motion vector V. Specifically, the inverse filter 321 moves the pixels in the motion region R by the vector ⁇ kV obtained by multiplying the inverse vector ⁇ V of the motion vector V by the coefficient k.
  • the coefficient k is gradually increased so as to be 0 at the start point (pixel group P 1 ) of the motion vector V and 1 at the end point (pixel group P n), for example.
  • the stretched image of the subject obj is compressed to the original size, and the influence of the blur caused by the movement of the subject obj is reduced.
  • the motion estimation unit 310 of the image processing apparatus 300 estimates the motion of the subject in the RGB image 101 from the event signal 201. Since the event signal 201 is generated only when a change in light intensity is detected in one or a plurality of pixels of the RGB image 101, for example, the pixels of a plurality of RGB images 101 that are continuous in time are compared with each other to move. The processing speed can be increased as compared with the case of estimating. Further, since the inverse filter 321 generated by the inverse filter generation unit 320 acts only on the motion region R of the RGB image 101, for example, when the filter is applied to the entire RGB image 101 including the region other than the motion region R. In comparison, the occurrence of artifacts can be suppressed.
  • the system 10 described in the above example may be mounted in a single device, or may be mounted in a plurality of devices in a distributed manner.
  • the RGB image 101 acquired by the RGB camera 100 and the event signal 201 acquired by the EDS 200 are stored in the memory together with the time stamps 102 and 202, and the motion estimation by the image processing device 300 and the inverse filter 321 as post-processing. And the application of the inverse filter 321 may be performed.
  • the processing up to the generation of the inverse filter 321 in the image processing apparatus 300 may be executed, and the inverse filter 321 may be saved together with the RGB image 101.
  • the output image 331 may be generated by applying the inverse filter 321 to the RGB image 101 according to, for example, a user operation.
  • an inverse filter 321 is generated that compresses the image of the subject obj that is stretched due to blurring toward the start point side of the motion vector V, but in other examples, the subject is An inverse filter 321 that compresses the image of obj to the end point side of the motion vector V or to the intermediate point may be generated. Further, the inverse filter 321 generated based on the event signal 201 is not limited to the above example, and for example, various filters known as filters for reducing the influence of blurring caused by the movement of the subject are used. be able to.
  • the motion estimation unit 310 estimates the motion region R of the RGB image 101, and the inverse filter generation unit 320 generates the inverse filter 321 that acts exclusively on the motion region R, so that the RGB image, for example, The occurrence of artifacts can be suppressed as compared with the case where the filter is applied to the entire 101.
  • FIG. 5 is a flowchart showing an example of an image processing method according to an embodiment of the present invention.
  • the RGB camera 100 captures the RGB image 101 (step S101), and at the same time, the EDS 200 generates the event signal 201 (step S102).
  • the step S102 for generating the event signal 201 is executed only when the sensor 210 associated with one or a plurality of pixels of the RGB image 101 detects a change in light intensity.
  • a time stamp 102 is given to the RGB image 101 (step S103), and a time stamp 202 is given to the event signal (step S104).
  • the motion estimation unit 310 estimates the motion of the subject in the RGB image 101 based on the time stamp 102 of the RGB image 101, the event signal 201, and the time stamp 202 of the event signal 201 (step S105).
  • the inverse filter generation unit 320 generates an inverse filter 321 based on the estimated motion (step S106), and the filter application unit 330 applies the inverse filter 321 to the RGB image 101 (step S107).
  • 10 ... system, 100 ... RGB camera, 101 ... RGB image, 102 ... time stamp, 110 ... image sensor, 120 ... processing circuit, 201 ... event signal, 202 ... time stamp, 210 ... sensor, 220 ... processing circuit, 300 ... Image processing device, 310 ... motion estimation unit, 320 ... inverse filter generation unit, 321 ... inverse filter, 330 ... filter application unit, 331 ... output image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
PCT/JP2019/032342 2019-08-20 2019-08-20 画像処理装置、撮像装置、システム、画像処理方法およびプログラム Ceased WO2021033250A1 (ja)

Priority Applications (6)

Application Number Priority Date Filing Date Title
CN201980099252.5A CN114245983B (zh) 2019-08-20 2019-08-20 图像处理装置、成像装置、系统、图像处理方法
US17/634,300 US11954866B2 (en) 2019-08-20 2019-08-20 Image processing apparatus using movement estimation, imaging apparatus, system, image processing method, and program
EP19942090.2A EP4020962B1 (en) 2019-08-20 2019-08-20 Image processing device, image capture device, system, image processing method, and program
JP2021541371A JP7197715B2 (ja) 2019-08-20 2019-08-20 画像処理装置、システム、画像処理方法およびプログラム
KR1020227003456A KR102916853B1 (ko) 2019-08-20 2019-08-20 화상 처리 장치, 촬상 장치, 시스템, 화상 처리 방법 및 프로그램
PCT/JP2019/032342 WO2021033250A1 (ja) 2019-08-20 2019-08-20 画像処理装置、撮像装置、システム、画像処理方法およびプログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/032342 WO2021033250A1 (ja) 2019-08-20 2019-08-20 画像処理装置、撮像装置、システム、画像処理方法およびプログラム

Publications (1)

Publication Number Publication Date
WO2021033250A1 true WO2021033250A1 (ja) 2021-02-25

Family

ID=74660624

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/032342 Ceased WO2021033250A1 (ja) 2019-08-20 2019-08-20 画像処理装置、撮像装置、システム、画像処理方法およびプログラム

Country Status (6)

Country Link
US (1) US11954866B2 (https=)
EP (1) EP4020962B1 (https=)
JP (1) JP7197715B2 (https=)
KR (1) KR102916853B1 (https=)
CN (1) CN114245983B (https=)
WO (1) WO2021033250A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023531039A (ja) * 2020-06-23 2023-07-20 プロフジー 画像強調方法、画像強調装置、および画像強調システム

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7851119B2 (ja) * 2021-12-24 2026-04-24 キヤノン株式会社 撮像装置及びその制御方法及びプログラム
KR20240126282A (ko) 2023-02-13 2024-08-20 삼성전자주식회사 비전 센서, 이의 이벤트 신호 처리 방법 및 이를 포함하는 이미지 처리 장치
WO2024194944A1 (ja) * 2023-03-17 2024-09-26 株式会社ソニー・インタラクティブエンタテインメント 情報処理装置、システム、情報処理方法、情報処理プログラム、およびコンピュータシステム
WO2024239224A1 (en) * 2023-05-23 2024-11-28 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Image processing device, image processing method, and program

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017535999A (ja) * 2014-09-30 2017-11-30 クアルコム，インコーポレイテッド フレームベースとイベントベースのハイブリッド方式を使用するセンサアーキテクチャ
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003209735A (ja) 2002-01-15 2003-07-25 Hitachi Ltd 監視装置及びブレ補正機能付蓄積型カメラ
KR101880998B1 (ko) 2011-10-14 2018-07-24 삼성전자주식회사 이벤트 기반 비전 센서를 이용한 동작 인식 장치 및 방법
US9549100B2 (en) * 2015-04-23 2017-01-17 Microsoft Technology Licensing, Llc Low-latency timing control
EP3354016A4 (en) * 2015-09-25 2019-08-21 Qualcomm Incorporated SYSTEMS AND METHOD FOR VIDEO PROCESSING
US10062151B2 (en) * 2016-01-21 2018-08-28 Samsung Electronics Co., Ltd. Image deblurring method and apparatus
US10434624B2 (en) 2016-07-07 2019-10-08 The Boeing Company Roto peening orbital drilling tool
US20180146149A1 (en) 2016-11-21 2018-05-24 Samsung Electronics Co., Ltd. Event-based sensor, user device including the same, and operation method of the same
JP7027737B2 (ja) 2017-09-05 2022-03-02 ソニーグループ株式会社 画像処理装置、および画像処理方法、並びにプログラム
CN108074222A (zh) * 2017-12-25 2018-05-25 清华大学 基于混合相机系统的高速运动物体去模糊方法及装置
CN108989606B (zh) * 2018-08-22 2021-02-09 Oppo广东移动通信有限公司 图像处理方法和装置、电子设备、计算机可读存储介质
US12142025B2 (en) * 2021-11-16 2024-11-12 Intrinsic Innovation Llc Polarization event cameras
US20230169665A1 (en) * 2021-12-01 2023-06-01 Intrinsic Innovation Llc Systems and methods for temporal autocalibration in a camera system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017535999A (ja) * 2014-09-30 2017-11-30 クアルコム，インコーポレイテッド フレームベースとイベントベースのハイブリッド方式を使用するセンサアーキテクチャ
US20180098082A1 (en) * 2016-09-30 2018-04-05 Intel Corporation Motion estimation using hybrid video imaging system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4020962A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023531039A (ja) * 2020-06-23 2023-07-20 プロフジー 画像強調方法、画像強調装置、および画像強調システム
JP7767331B2 (ja) 2020-06-23 2025-11-11 プロフジー 画像強調方法、画像強調装置、および画像強調システム

Also Published As

Publication number Publication date
JPWO2021033250A1 (https=) 2021-02-25
CN114245983A (zh) 2022-03-25
US20220284593A1 (en) 2022-09-08
EP4020962A1 (en) 2022-06-29
KR102916853B1 (ko) 2026-01-23
EP4020962A4 (en) 2023-05-10
US11954866B2 (en) 2024-04-09
JP7197715B2 (ja) 2022-12-27
KR20220050130A (ko) 2022-04-22
CN114245983B (zh) 2024-03-22
EP4020962B1 (en) 2025-04-23

Similar Documents

Publication Publication Date Title
JP7197715B2 (ja) 画像処理装置、システム、画像処理方法およびプログラム
JP4487191B2 (ja) 画像処理装置および画像処理プログラム
US8810663B2 (en) Methods for motion correction for high-dynamic-range imaging systems
JP2003078812A (ja) デジタル画像における動きによるぼけを低減する方法
CN102572245B (zh) 一种扩展图像动态范围的方法及装置
EP2323376B1 (en) Image sensor with shaking compensation
JP4831760B2 (ja) ３次元情報検出方法及びその装置
US20090087173A1 (en) Image capturing apparatus with movement compensation function and method for movement compensation thereof
CN102223484A (zh) 一种摄像机前端参数配置的方法和装置
CN103546730A (zh) 基于多摄像头的图像感光度增强方法
KR20100060140A (ko) 영상 처리 장치에서의 광역 역광 보정 영상 획득 장치 및 방법
JP2017169111A (ja) 撮像制御装置、および撮像制御方法、ならびに撮像装置
JP4250583B2 (ja) 画像撮像装置及び画像復元方法
JP2009182527A (ja) 撮像方法および撮像装置
JPWO2021033250A5 (https=)
JP2009134357A (ja) 画像処理装置、撮像装置、画像処理プログラム及び画像処理方法
JP7252352B2 (ja) 転送制御装置、システム、転送制御方法およびプログラム
CN118118795A (zh) Dvs与rgb图像融合的清晰图像获取方法、设备以及装置
WO2021033251A1 (ja) 画像処理装置、画像処理方法およびプログラム
CN110827230A (zh) 一种利用tof提升rgb图像质量的方法和装置
CN103905722A (zh) 图像处理装置以及图像处理方法
KR20210026535A (ko) 영상 처리 장치
US20250159369A1 (en) Object tracking apparatus, control method thereof, imaging system, and storage medium
CN110458860A (zh) 图像处理方法及图像处理装置
JP2003250093A (ja) 映像補正装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19942090

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021541371

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019942090

Country of ref document: EP

Effective date: 20220321

WWG Wipo information: grant in national office

Ref document number: 2019942090

Country of ref document: EP