WO2023080667A1 - Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia - Google Patents

Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia Download PDF

Info

Publication number
WO2023080667A1
WO2023080667A1 PCT/KR2022/017110 KR2022017110W WO2023080667A1 WO 2023080667 A1 WO2023080667 A1 WO 2023080667A1 KR 2022017110 W KR2022017110 W KR 2022017110W WO 2023080667 A1 WO2023080667 A1 WO 2023080667A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
exposure
exposure time
wdr
value
Prior art date
Application number
PCT/KR2022/017110
Other languages
English (en)
Korean (ko)
Inventor
박종선
백정원
이창민
Original Assignee
한화테크윈 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 한화테크윈 주식회사 filed Critical 한화테크윈 주식회사
Publication of WO2023080667A1 publication Critical patent/WO2023080667A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals

Definitions

  • the present specification relates to a WDR image processing apparatus and method in a surveillance camera.
  • the WDR function is a function that is used for efficient monitoring of areas where indoor and outdoor environments coexist or where dark and bright areas coexist outside the dynamic range of image sensors.
  • a WDR image is generated by synthesizing and correcting the outputs of image sensors (bright area image and dark area image) having nine different shutter information according to time series.
  • an object of one embodiment of the present specification is to provide an image processing apparatus and method capable of improving the quality of a WDR image by controlling a shutter value and a gain value of an image sensor when motion afterimage is detected.
  • a high-speed shutter is controlled to reduce the disparity between a short-exposure image and a long-exposure image, and a gain value of an image sensor is controlled to control a high-speed shutter.
  • An object of the present invention is to provide an image processing device and method capable of compensating for brightness according to the application of a shutter.
  • an object of the present invention is to provide an image processing apparatus and method capable of obtaining an image without motion afterimage even in a monitoring area in which the brightness difference of the monitoring area is large and the speed change of the subject is flexible.
  • An image processing device includes an image sensor configured to acquire a first image during a first exposure time and acquire a second image during a second exposure time shorter than the first exposure time; and a processor generating a Wide Dynamic Range (WDR) image by synthesizing the first image and the second image, wherein the processor detects motion blur of an object included in the WDR image.
  • WDR Wide Dynamic Range
  • the processor controls a shutter value of the image sensor so that a third shutter value is applied so that the first image is acquired during a third exposure time by reducing the first exposure time.
  • the processor controls a gain value of the image sensor so that a third gain value greater than the first gain value corresponding to the first exposure time is applied.
  • the processor applies a deep learning-based YOLO (You Only Look Once) algorithm to input the third shutter value and the third gain value to the image sensor when the degree of motion of the object classified in the WDR image is greater than or equal to a threshold value can be controlled as much as possible.
  • a deep learning-based YOLO You Only Look Once
  • the processor may train a neural network model that takes the movement speed of an object included in the WDR image as an input and outputs a shutter value and a gain value to be controlled according to the movement speed of the object, and stores the trained neural network model in a storage unit.
  • the processor recognizes an object included in the WDR image through a deep learning algorithm, and when the recognized object is an object of interest, controls the shutter value and gain value according to the output of the neural network model to be applied to the image sensor.
  • the processor may change the reduction range of the exposure time and the increase range of the gain value based on at least one of the type of the object and the movement speed of the object.
  • An image processing device includes acquiring a long exposure image for a first exposure time and a short exposure image for a second exposure time through an image sensor; generating a wide dynamic range (WDR) image by synthesizing the long exposure image and the short exposure image;
  • WDR wide dynamic range
  • a reduction in the first exposure time is set to minimize the disparity between the long-exposure image and the short-exposure image, thereby increasing the shutter value. resetting; and applying the reset shutter value to the image sensor.
  • the image processing method may include resetting a gain value of the image sensor to compensate for brightness according to a decrease in the first exposure time; and controlling to apply the reset shutter value and gain value to the image sensor.
  • an image without motion afterimages can be obtained even in a monitoring area in which the brightness difference of the monitoring area is large and the speed change of the subject is flexible.
  • FIG. 1 is a diagram for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to an embodiment of the present specification.
  • Figure 2 is a schematic block diagram of a surveillance camera according to an embodiment of the present specification.
  • FIG. 3 is a diagram for explaining an AI device (module) applied to analysis of surveillance camera images according to an embodiment of the present specification.
  • FIG. 4 is a flowchart of an image processing method of a monitoring camera according to an embodiment of the present specification.
  • FIG. 5 is a functional block diagram illustrating a WDR image processing process according to an embodiment of the present specification.
  • FIG. 6 is a diagram for explaining a motion blur phenomenon that occurs when synthesizing conventional frame WDR images.
  • FIG. 7 is a diagram for explaining WDR image synthesis using line interleaving according to an embodiment of the present specification.
  • FIGS. 8A and 8B are views for explaining an example in which a WDR image processing process according to an embodiment of the present specification is applied to low-speed shutter and high-speed shutter operations, respectively.
  • FIG 9 illustrates an example to which a WDR image processing process according to an embodiment of the present specification is applied.
  • FIG. 1 is a diagram for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to an embodiment of the present specification.
  • an image management system 10 may include a photographing device 100 and an image management server 200 .
  • the photographing device 100 may be an electronic device for photographing disposed at a fixed location in a specific place, may be an electronic device for photographing that may move automatically or manually along a certain path, or may be moved by a person or robot. It may be an electronic device for photography.
  • the photographing device 100 may be an IP camera used by connecting to the wired or wireless Internet.
  • the photographing device 100 may be a PTZ camera having pan, tilt, and zoom functions.
  • the photographing device 100 may have a function of recording or taking a picture of an area to be monitored.
  • the photographing device 100 may have a function of recording sound generated in the area to be monitored.
  • the photographing device 100 may have a function of generating a notification or recording or taking a picture when a change, such as motion or sound, occurs in the area to be monitored.
  • the image management server 200 may be a device that receives and stores an image captured by the photographing device 100 and/or an image obtained by editing the corresponding image.
  • the video management server 200 may analyze the received data to correspond to the purpose. For example, the image management server 200 may detect an object using an object detection algorithm to detect an object in an image.
  • An AI-based algorithm may be applied to the object detection algorithm, and an object may be detected by applying a pre-learned artificial neural network model.
  • the video management server 200 may store various learning models suitable for video analysis purposes.
  • a model capable of acquiring the movement speed of the detected object may be stored.
  • the learned models may include a shutter speed corresponding to the moving speed of the object or a learning model that outputs a gain value of a sensor for compensating for brightness according to the control of the shutter speed.
  • the learned models detect the intensity of motion blur according to the moving speed of the object analyzed through the AI object recognition algorithm, and the optimal shutter speed and/or sensor gain value for the shooting environment that causes the detected intensity of motion blur is determined. It may be implemented through a model learned to be output.
  • the video management server 200 may analyze the received video to generate meta data and index information for the meta data.
  • the image management server 200 may analyze image information and/or sound information included in the received image together or separately to generate metadata and index information for the corresponding metadata.
  • the image management system 10 may further include an external device 300 capable of wired/wireless communication with the photographing device 100 and/or the image management server 200.
  • the external device 300 may transmit an information provision request signal requesting provision of all or part of the video to the video management server 200 .
  • the external device 300 is the video management server 200, as a result of image analysis, whether or not there is an object, the moving speed of the object, a shutter speed adjustment value according to the moving speed of the object, a noise removal value according to the moving speed of the object, and a sensor gain value.
  • An information provision request signal requesting the like may be transmitted.
  • the external device 300 may transmit an information provision request signal requesting metadata obtained by analyzing an image to the image management server 200 and/or index information for the metadata.
  • the image management system 10 may further include a communication network 400 that is a wired/wireless communication path between the photographing device 100 , the image management server 200 , and/or the external device 300 .
  • the communication network 400 may include, for example, wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, and satellite communication.
  • wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, and satellite communication.
  • LANs Local Area Networks
  • WANs Wide Area Networks
  • MANs Metropolitan Area Networks
  • ISDNs Integrated Service Digital Networks
  • wireless LANs Code Division Multiple Access
  • CDMA Code Division Multiple Access
  • Bluetooth Code Division Multiple Access
  • Figure 2 is a schematic block diagram of a surveillance camera according to an embodiment of the present specification.
  • FIG. 2 is a block diagram showing the configuration of a camera shown in FIG. 1 .
  • the camera 200 is described as an example of a network camera that generates the image analysis signal by performing an intelligent video analysis function, but the operation of the network, surveillance, and camera system according to an embodiment of the present invention is necessarily limited to this. it is not going to be
  • the camera 200 includes an image sensor 210 , an encoder 220 , a memory 230 , an event sensor 240 , a processor 240 , and a communication interface 250 .
  • the image sensor 210 performs a function of capturing an image by photographing a monitoring area, and may be implemented with, for example, a Charge-Coupled Device (CCD) sensor, a Complementary Metal-Oxide-Semiconductor (CMOS) sensor, or the like.
  • CCD Charge-Coupled Device
  • CMOS Complementary Metal-Oxide-Semiconductor
  • the encoder 220 performs an operation of encoding an image acquired through the image sensor 210 into a digital signal, which is, for example, H.264, H.265, MPEG (Moving Picture Experts Group), M-JPEG (Motion Joint Photographic Experts Group) standards, etc. may be followed.
  • a digital signal which is, for example, H.264, H.265, MPEG (Moving Picture Experts Group), M-JPEG (Motion Joint Photographic Experts Group) standards, etc.
  • the memory 230 may store video data, audio data, still images, metadata, and the like.
  • the metadata includes object detection information (movement, sound, intrusion into a designated area, etc.) captured in the surveillance area, object identification information (person, car, face, hat, clothing, etc.), and detected location. It can be data containing information (coordinates, size, etc.).
  • the still image is generated together with the metadata and stored in the memory 230, and may be generated by capturing image information for a specific analysis region among the image analysis information.
  • the still image may be implemented as a JPEG image file.
  • the still image may be generated by cropping a specific region of image data determined to be an identifiable object among image data of the surveillance region detected in a specific region and during a specific period, which is the metadata. can be transmitted in real time.
  • the communication unit 240 transmits the video data, audio data, still images, and/or metadata to the video receiving/searching device 300 .
  • the communication unit 240 may transmit video data, audio data, still images, and/or metadata to the video receiving device 300 in real time.
  • the communication interface 250 may perform at least one communication function among wired and wireless local area network (LAN), Wi-Fi, ZigBee, Bluetooth, and near field communication.
  • the AI processor 250 is for artificial intelligence image processing, and applies a deep learning-based object detection algorithm learned as an object of interest from an image acquired through a surveillance camera system according to an embodiment of the present specification. .
  • the AI processor 250 may be implemented as a module with the processor 260 that controls the entire system or as an independent module.
  • Embodiments of the present specification may apply a You Only Lock Once (YOLO) algorithm in object detection.
  • YOLO is an AI algorithm that is suitable for surveillance cameras that process real-time video because of its fast object detection speed.
  • the YOLO algorithm resizes one input image and passes through a single neural network only once, indicating the position of each object. It outputs the bounding box and the classification probability of what the object is. Finally, one object is recognized (detection) once through non-max suppression.
  • the object recognition algorithm disclosed in this specification is not limited to the aforementioned YOLO and can be implemented with various deep learning algorithms.
  • the learning model for object recognition applied in the present specification may be a model trained by defining camera performance, motion speed information of an object recognizable without motion blur in a surveillance camera, etc. as learning data.
  • input data may be the moving speed of the object
  • output data may be a shutter speed optimized for the moving speed of the object as output data.
  • the learned model may be input data that is the moving speed of an object in a WDR image generated by synthesizing a long-exposure image and a short-exposure image, and output data is a shutter speed and/or sensor optimized for the moving speed of the object.
  • Gain values can be defined as output data.
  • the learned model may be a shutter value and a gain value of a long-exposure image when motion blur of a specific object is detected as input data in the WDR image, and output data may be used to minimize motion blur of the specific object.
  • it may be a shutter value and/or a sensor gain value to be changed in the long exposure image acquisition process.
  • FIG. 3 is a diagram for explaining an AI device (module) applied to analysis of surveillance camera images according to an embodiment of the present specification.
  • the AI device 20 may include an electronic device including an AI module capable of performing AI processing or a server including an AI module.
  • the AI device 20 may be included as a configuration of at least a portion of a monitoring camera or video management server and may be provided to perform at least a portion of AI processing together.
  • AI processing may include all operations related to a surveillance camera or a control unit of a video management server.
  • a surveillance camera or a video management server may perform AI processing on the acquired video signal to perform processing/determination and control signal generation operations.
  • the AI device 20 may be a client device that directly uses AI processing results or a device in a cloud environment that provides AI processing results to other devices.
  • the AI device 20 is a computing device capable of learning a neural network, and may be implemented in various electronic devices such as a server, a desktop PC, a notebook PC, and a tablet PC.
  • the AI device 20 may include an AI processor 21, a memory 25 and/or a communication unit 27.
  • the AI processor 21 may learn a neural network using a program stored in the memory 25 .
  • the AI processor 21 may learn a neural network for recognizing data related to surveillance cameras.
  • the neural network for recognizing the related data of the surveillance camera may be designed to simulate the structure of the human brain on a computer, and may include a plurality of network nodes having weights that simulate the neurons of the human neural network. there is.
  • a plurality of network modes may transmit and receive data according to a connection relationship, respectively, so as to simulate synaptic activity of neurons that transmit and receive signals through synapses.
  • the neural network may include a deep learning model developed from a neural network model.
  • a plurality of network nodes may exchange data according to a convolution connection relationship while being located in different layers.
  • Examples of neural network models are deep neural networks (DNN), convolutional deep neural networks (CNN), recurrent Boltzmann machines (RNNs), restricted Boltzmann machines (RBMs), deep trust It includes various deep learning techniques such as deep belief networks (DBN) and deep Q-network, and can be applied to fields such as computer vision, voice recognition, natural language processing, and voice/signal processing.
  • DNN deep neural networks
  • CNN convolutional deep neural networks
  • RNNs recurrent Boltzmann machines
  • RBMs restricted Boltzmann machines
  • DNN deep belief networks
  • Q-network deep Q-network
  • the processor performing the functions described above may be a general-purpose processor (eg, CPU), or may be an AI-only processor (eg, GPU) for artificial intelligence learning.
  • a general-purpose processor eg, CPU
  • an AI-only processor eg, GPU
  • the memory 25 may store various programs and data necessary for the operation of the AI device 20 .
  • the memory 25 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SDD).
  • the memory 25 is accessed by the AI processor 21, and reading/writing/modifying/deleting/updating of data by the AI processor 21 can be performed.
  • the memory 25 may store a neural network model (eg, the deep learning model 26) generated through a learning algorithm for data classification/recognition according to an embodiment of the present invention.
  • the AI processor 21 may include a data learning unit 22 that learns a neural network for data classification/recognition.
  • the data learning unit 22 may learn criteria regarding which training data to use to determine data classification/recognition and how to classify and recognize data using the training data.
  • the data learning unit 22 may acquire learning data to be used for learning and learn the deep learning model by applying the obtained learning data to the deep learning model.
  • the data learning unit 22 may be manufactured in the form of at least one hardware chip and mounted on the AI device 20 .
  • the data learning unit 22 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or manufactured as a part of a general-purpose processor (CPU) or a graphics-only processor (GPU) for the AI device 20. may be mounted.
  • the data learning unit 22 may be implemented as a software module.
  • the software module When implemented as a software module (or a program module including instructions), the software module may be stored in a computer-readable, non-transitory computer readable recording medium (non-transitory computer readable media). In this case, at least one software module may be provided by an Operating System (OS) or an application.
  • OS Operating System
  • the data learning unit 22 may include a training data acquisition unit 23 and a model learning unit 24 .
  • the training data acquisition unit 23 may acquire training data required for a neural network model for classifying and recognizing data.
  • the model learning unit 24 may learn to have a criterion for determining how to classify predetermined data by using the acquired training data.
  • the model learning unit 24 may learn the neural network model through supervised learning using at least some of the learning data as a criterion.
  • the model learning unit 24 may learn the neural network model through unsupervised learning in which a decision criterion is discovered by self-learning using learning data without guidance.
  • the model learning unit 24 may learn the neural network model through reinforcement learning using feedback about whether the result of the situation judgment according to learning is correct.
  • the model learning unit 24 may train the neural network model using a learning algorithm including error back-propagation or gradient decent.
  • the model learning unit 24 may store the learned neural network model in memory.
  • the model learning unit 24 may store the learned neural network model in a memory of a server connected to the AI device 20 through a wired or wireless network.
  • the data learning unit 22 further includes a training data pre-processing unit (not shown) and a learning data selection unit (not shown) to improve the analysis result of the recognition model or save resources or time required for generating the recognition model. You may.
  • the learning data pre-processing unit may pre-process the acquired data so that the acquired data can be used for learning for situation determination.
  • the learning data pre-processing unit may process the acquired data into a preset format so that the model learning unit 24 can use the acquired learning data for learning for image recognition.
  • the learning data selector may select data necessary for learning from among the learning data acquired by the learning data acquisition unit 23 or the training data preprocessed by the preprocessor.
  • the selected training data will be provided to the model learning unit 24.
  • the data learning unit 22 may further include a model evaluation unit (not shown) to improve the analysis result of the neural network model.
  • the model evaluation unit inputs evaluation data to the neural network model, and when an analysis result output from the evaluation data does not satisfy a predetermined criterion, it may cause the model learning unit 22 to learn again.
  • the evaluation data may be predefined data for evaluating the recognition model.
  • the model evaluator may evaluate that the predetermined criterion is not satisfied when the number or ratio of the evaluation data for which the analysis result is inaccurate among the analysis results of the learned recognition model for the evaluation data exceeds a preset threshold. there is.
  • the communication unit 27 may transmit the AI processing result by the AI processor 21 to an external electronic device.
  • external electronic devices may include surveillance cameras, Bluetooth devices, self-driving vehicles, robots, drones, AR devices, mobile devices, home appliances, and the like.
  • the AI device 20 shown in FIG. 3 has been functionally divided into an AI processor 21, a memory 25, a communication unit 27, etc., but the above-mentioned components are integrated into one module and the AI module Note that it can also be called
  • one or more of a surveillance camera, an autonomous vehicle, a user terminal, and a server is an artificial intelligence module, a robot, an augmented reality (AR) device, a virtual reality (VT) device, and a 5G It may be associated with a device related to the service.
  • AR augmented reality
  • VT virtual reality
  • 5G It may be associated with a device related to the service.
  • FIG. 4 is a flowchart of an image processing method of a monitoring camera according to an embodiment of the present specification.
  • the image processing method shown in FIG. 4 may be implemented through a processor or control unit included in the monitoring camera system, monitoring camera device, and monitoring camera device described with reference to FIGS. 1 to 3 .
  • the image processing method is described on the premise that various functions can be controlled through the processor 260 of the surveillance camera 200 shown in FIG. 2, but the present specification is not limited thereto. put
  • the monitoring camera 200 obtains a monitoring camera image.
  • the surveillance camera image may include a video.
  • the surveillance camera 200 acquires a first image for a first exposure time and acquires a second image for a second exposure time (S400).
  • the second exposure time is defined as a time shorter than the first exposure time, and the first image is defined as a long-exposure image and the second image is defined as a short-exposure image.
  • the processor may obtain a second image after acquiring the first image. Accordingly, a time interval (TI) may occur between acquiring the first image and acquiring the second image, and in an embodiment of the present specification, in order to minimize motion blur, the time interval (TI)
  • the image sensor can be controlled to minimize .
  • the processor 260 may generate a wide dynamic range (WDR) image by synthesizing the first image and the second image.
  • the first image may be an image obtained by applying a first gain value during the first exposure time (or first shutter value).
  • the second image may be an image obtained by applying a second gain value during the second exposure time (or second shutter value). More specifically, the first image is a long exposure time image, and a relatively bright area within the image may be saturated, and an object included in a relatively dark area may be clearly displayed.
  • the second image is a short exposure time image, and an object included in a relatively dark area in the image may not be visible.
  • the processor 260 may generate a WDR image by fusing the first image and the second image (S410).
  • the brightness of the saturated region of the first image is relatively dark, and the brightness of the dark region of the second image is adjusted so that objects included in the dark region can be viewed under conditions similar to the contrast of the real environment.
  • the first exposure time for acquiring the first image is different from the second exposure time for acquiring the second image.
  • the positions of the objects displayed in the first image and the second image may be different from each other, and the WDR image synthesis result Motion blur may occur with respect to the object.
  • the processor 260 may determine whether motion afterimage (motion blur) of the object exists as a result of the WDR image synthesis (S420).
  • the processor 260 may determine motion blur based on sharpness of a specific object shown in the WDR synthesized image.
  • the processor 260 may perform location information of a specific object in the first image and a specific object in the second image when the time difference between the first exposure time and the second exposure time is equal to or greater than a predetermined difference (e.g. For example, based on coordinate information), it is possible to determine whether the specific object has motion blur.
  • the processor 260 may additionally determine motion blur when a time interval between the acquisition of the first image and the acquisition of the second image differs by more than a predetermined time.
  • the processor 260 may determine whether a specific object of the WDR image has motion blur by applying an AI-based object recognition algorithm applied herein. Also, as an example, the processor 260 may determine motion blur of a specific object in the WDR image after synthesizing the WDR image based on the first exposure time before synthesizing the WDR image.
  • the first exposure time is the time required to acquire a long-exposure image. As the long-exposure time increases, the parallax with the short-exposure image increases. A motion blur phenomenon may be highly likely to occur frequently.
  • the processor 260 may reset the shutter value for acquiring the first image so that the time difference between the first exposure time and the second exposure time is minimized by reducing the first exposure time (S430).
  • the processor 260 may control a third shutter value to be applied to the image sensor so that the first image is acquired during the third exposure time by reducing the first exposure time.
  • the third exposure time (corresponding to the third shutter value) is longer than the second exposure time, and may be a shutter value that secures a certain level of brightness compared to a short exposure. Since the WDR image is intended to expand the dynamic range of brightness, the third shutter value that minimizes the time difference between the first exposure time and the second exposure time is the brightness of the image due to the second exposure time (second shutter value). Compared to , it may be a shutter value at which a certain level or more of brightness can be secured.
  • the third shutter value may be set as a minimum shutter value for obtaining a long-exposure image according to an illuminance value within a view angle range of the monitoring camera. Accordingly, the third shutter value may be adaptively changed according to an illuminance value of a surrounding environment where the monitoring camera is located.
  • the processor 260 may control the reset third shutter value to be applied to the image sensor (S440). Thereafter, motion blur of an object in a WDR synthesized image may be reduced due to a decrease in parallax between a long-exposure image and a short-exposure image acquired through a surveillance camera.
  • a long exposure image may be obtained by increasing a gain value of an image sensor instead of applying a third shutter value to a long exposure image. That is, the processor 260 may simultaneously control the shutter value and the gain of the image sensor so that a third gain greater than the first gain corresponding to the first exposure time is applied.
  • the present specification classifies an object in a WDR image by applying a deep learning-based YOLO (You Only Look Once) algorithm, and the degree of movement (eg, movement speed of the object) for the classified object is If it is equal to or greater than the threshold value, the third shutter value and the third sensor gain value may be input to the image sensor.
  • a deep learning-based YOLO You Only Look Once
  • the processor 260 takes the movement speed of the object as an input value, and sets a shutter value and/or a sensor gain value to be controlled according to the movement speed of the object. You can train a neural network model with output.
  • the processor 260 may control the trained neural network model to be stored in a storage unit of a surveillance camera or a server (200 in FIG. 1). Accordingly, when motion blur exists in the WDR image, the surveillance camera may receive information on a long exposure time capable of removing motion blur and a sensor gain value from the server 200 through the communication unit. In addition, the surveillance camera may analyze the WDR image and receive the long exposure value and the sensor gain value from the AI module built in the surveillance camera when there is motion blur.
  • the processor 260 may recognize an object included in the WDR synthesized image through a deep learning algorithm, and determine whether the recognized object is a predetermined object of interest. If the object included in the WDR image is an object that can be shaken by wind, such as a tree branch, the shutter value change operation according to the present specification does not need to be performed. That is, when the object of interest is designated as a human and the AI-based object recognition result determines that the object included in the WDR image (or long-exposure image and short-exposure image before synthesizing the WDR image) is a human, resetting according to the output of the neural network model. The shutter value and the sensor gain value may be controlled to be applied to the image sensor.
  • the neural network model that outputs the reset shutter value and gain value adaptively adjusts the amount of reduction in exposure time and/or the amount of increase in sensor gain value based on at least one of the type of object or the moving speed of the object. It can be a neural network model that has been trained to change. For example, when a surveillance camera needs to recognize the license plate of a driving vehicle, since the movement of the vehicle is very fast, set the reduction of the long exposure time as large as possible to minimize the parallax with the short exposure to minimize the possibility of motion blur. , it is possible to reset the sensor gain increment corresponding to the long exposure time decrease. In addition, for example, when the average movement speed of an object photographed by a surveillance camera is relatively slow, noise caused by an increase in sensor gain can be minimized by reducing the range of decrease in the long exposure time and the range of increase in the gain value.
  • the neural network model for resetting the shutter value and/or gain value applied in the present specification may be a neural network model trained based on the type of object and the movement speed of the object.
  • FIG. 5 is a functional block diagram illustrating a WDR image processing process according to an embodiment of the present specification.
  • the image sensor acquires a long exposure image 501 and a short exposure image 502 and transmits them to the ISP.
  • the ISP generates a WDR image by synthesizing the long exposure image 501 and the short exposure image 502 .
  • the ISP may embed an AI module, and the ISP may determine whether a motion afterimage for a specific object exists in the synthesized WDR image and calculate a shutter value for minimizing the motion afterimage.
  • the shutter value may be a long exposure shutter value capable of minimizing a time difference between a long exposure image and a short exposure image.
  • the long-exposure shutter value has an exposure time shorter than the initial shutter value, so that the parallax with the short-exposure image is reduced.
  • the AI module may increase a sensor gain value for acquiring a long-exposure image due to a decrease in a long-exposure shutter value.
  • the image sensor may reacquire a long exposure image and a short exposure image based on the reset shutter value and gain value, and the ISP may generate a WDR image from the reacquired two images and store the WDR image in the storage unit.
  • FIG. 6 is a diagram for explaining a motion blur phenomenon that occurs when synthesizing conventional frame WDR images.
  • (a) of FIG. 6 in the case of a frame-based WDR image, a short-exposure image 601a of a first frame, a long-exposure image 601b of a second frame, and a short-exposure image 602a of a second frame , Since the long exposure images 602b of the second frame do not partially overlap each other, it is common that the short exposure starts after the long exposure ends. Accordingly, when the long exposure time is long, the disparity between the long exposure image and the short exposure image is relatively large, and in the process, there is a high possibility that motion blur may occur according to the movement or movement speed of an object.
  • (b) of FIG. 6 is an example of blur caused by parallax between a long-exposure image and a short-exposure image in the frame-based WDR image.
  • FIG. 7 is a diagram for explaining WDR image synthesis using line interleaving according to an embodiment of the present specification.
  • each line of the image sensor can be output with two consecutive long and short exposures.
  • FIGS. 8A and 8B are views for explaining an example in which a WDR image processing process according to an embodiment of the present specification is applied to low-speed shutter and high-speed shutter operations, respectively.
  • the 8A is an example in which a slow shutter is applied in an interleaving-based WDR image synthesis process.
  • the first image (long exposure image, 801a) is applied with the first gain value (Gain 1) during the first exposure time (Exposure 1 period), and the second image (short exposure image, 801b) is applied with the second exposure time (Exposure 1 period). 2 period), the second gain value (Gain 2) is applied.
  • the first exposure time is a slow shutter speed
  • motion blur is more likely to occur when a moving object exists in the WDR image when synthesizing the WDR image due to the relatively parallax 810 between the first image and the second image.
  • the parallax 820 is reduced, so that motion blur can be minimized even if a moving object exists during WDR image synthesis.
  • whether or not the moving object exists is determined by applying an AI-based object recognition algorithm to classify the object to determine whether the object requires a WDR image processing process according to an embodiment of the present specification.
  • the surveillance camera resets the shutter value and the gain value and inputs them to the image sensor to reduce the size of motion blur.
  • the processor determines the first exposure time Exposure 1 in a state in which a time interval (time interval, 811, 821) for acquiring the second image after acquiring the first image is minimized. ' period) and can be controlled to minimize motion blocks.
  • FIG. 9 illustrates an example to which a WDR image processing process according to an embodiment of the present specification is applied.
  • (a) of FIG. 9 is an example of motion blur appearing in a moving subject in a WDR image when the WDR image processing method disclosed herein is not applied in an indoor environment
  • (b) is an example of WDR image processing disclosed herein. This is an example of an image with motion blur reduced by applying the technology.
  • an image sensor device includes a sensor that acquires a first image during a first exposure time and acquires a second image during a second exposure time that is shorter than the first exposure time; and a control unit controlling a shutter speed of the sensor, wherein the control unit, in a state in which a time interval between obtaining the first image and obtaining the second image is minimized, The shutter speed of the sensor is controlled so that the time difference between the first exposure time and the second exposure time is minimized by reducing the first exposure time.
  • the control unit may transmit a control signal for controlling the shutter speed to the sensor when detecting motion blur of an object included in a WDR image generated by synthesizing the first image and the second image.
  • the controller may control a third shutter value to be applied to the sensor so that the first image is acquired during a third exposure time by reducing the first exposure time.
  • the control unit may control the gain value of the sensor so that a third gain value greater than the first gain value corresponding to the first exposure time is applied.
  • the motion blur is determined as a case where the degree of motion of the object classified in the WDR image by applying a deep learning-based YOLO (You Only Look Once) algorithm is greater than or equal to a threshold value, and the control unit determines that the motion blur is detected.
  • the third shutter value and the third gain value can be controlled to be input to the sensor.
  • An embodiment according to the present specification may include a method for controlling the above-described image sensor device.
  • the control method of the image sensor device may be implemented by a control unit of the image sensor device, or may be implemented in the form of an image sensor module or image sensor chip including an image sensor.
  • the image sensor module (image sensor chip) may include a device including a module that is connected to the image sensor through a PCB and provides a predetermined control signal to the image sensor.
  • the control signal may be a signal generated through an image signal processor (ISP) for an image sensor.
  • ISP image signal processor
  • the control method of the image sensor device may include acquiring a long-exposure image for a first exposure time through a sensor and acquiring a short-exposure image for a second exposure time shorter than the first exposure time; and in a state in which a time interval for acquiring the second image after acquiring the first image is minimized, the first exposure time is reduced so that a time difference between the first exposure time and the second exposure time is minimized. resetting the shutter speed of the sensor; and applying the reset shutter value to the sensor.
  • the control method of the image sensor device may include resetting a gain value of the sensor to compensate for brightness according to a decrease in the first exposure time; and controlling to apply the reset shutter value and gain value to the sensor.
  • the image sensor includes a sensor that acquires a first image during a first exposure time based on a predetermined control signal and acquires a second image during a second exposure time shorter than the first exposure time.
  • the control signal is a signal received when motion blur is detected with respect to an object included in a wide dynamic range (WDR) image generated by synthesizing the first image and the second image.
  • WDR wide dynamic range
  • the sensor based on the control signal, reduces the first exposure time in a state in which a time interval for acquiring the second image after acquiring the first image is minimized, thereby reducing the first exposure time and the second image. 2
  • the first image and the second image are acquired so that a time difference between exposure times is minimized.
  • the senor may obtain the first image and the second image by varying a shutter speed of the sensor based on the control signal.
  • the senor obtains the first image according to a third shutter value corresponding to a third exposure time by reducing the first exposure time based on the control signal, and a first gain value corresponding to the first exposure time.
  • the first image may be obtained by applying a larger third gain value.
  • Another embodiment of the present specification may include a method of controlling an image sensor.
  • the control method of the image sensor includes obtaining a first image for a first exposure time through an image sensor and acquiring a second image for a second exposure time shorter than the first exposure time, and Receiving a control signal, and acquiring the first image and the second image by varying a shutter speed of the image sensor based on the control signal.
  • the control signal is a signal received when motion blur is detected with respect to an object included in a wide dynamic range (WDR) image generated by synthesizing the first image and the second image, and the control signal In a state in which the time interval for obtaining the second image after acquiring the first image is minimized, the first exposure time is reduced so that the time difference between the first exposure time and the second exposure time is minimized.
  • the first image and the second image are obtained as much as possible.
  • control method of the image sensor comprises acquiring the first image according to a third shutter value corresponding to a third exposure time by reducing the first exposure time based on the control signal, and corresponding to the first exposure time. and obtaining the first image by applying a third gain value greater than the first gain value.
  • the above-described present invention can be implemented as computer readable code on a medium on which a program is recorded.
  • the computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Therefore, the above detailed description should not be construed as limiting in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Studio Devices (AREA)

Abstract

L'invention concerne un dispositif de traitement d'image pour caméra de surveillance. Lorsqu'une image WDR est générée par synthèse d'une image à longue exposition et d'une image à courte exposition et qu'il est reconnu qu'un flou de mouvement est généré dans un objet inclus dans l'image WDR au moyen d'un algorithme de reconnaissance d'objets basée sur l'intelligence artificielle (IA), un dispositif de traitement d'image pour caméra de surveillance selon un mode de réalisation de la présente invention peut réduire le flou de mouvement de l'image WDR par réduction de la durée d'exposition de l'image à longue exposition et réduction au minimum d'une différence de durée entre les deux images. Dans la présente invention, un ou plusieurs éléments parmi une caméra de surveillance, un véhicule autonome, un terminal utilisateur et un serveur peuvent être associés à un module d'intelligence artificielle, à un robot, à un dispositif de réalité augmentée (RA), à un dispositif de réalité virtuelle (RV), à un dispositif relatif à un service 5G, etc.
PCT/KR2022/017110 2021-11-04 2022-11-03 Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia WO2023080667A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020210150640A KR20230064959A (ko) 2021-11-04 2021-11-04 Ai 기반 객체인식을 통한 감시카메라 wdr 영상 처리
KR10-2021-0150640 2021-11-04

Publications (1)

Publication Number Publication Date
WO2023080667A1 true WO2023080667A1 (fr) 2023-05-11

Family

ID=86241896

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2022/017110 WO2023080667A1 (fr) 2021-11-04 2022-11-03 Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia

Country Status (2)

Country Link
KR (1) KR20230064959A (fr)
WO (1) WO2023080667A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102644294B1 (ko) * 2023-07-20 2024-03-07 주식회사 모토웨이 다차원 인체 센싱용 카메라 임베디드 시스템 및 이를 활용하는 초소형 광각 카메라

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140111858A (ko) * 2013-03-12 2014-09-22 삼성테크윈 주식회사 이미지 처리 장치 및 방법
WO2018057103A1 (fr) * 2016-09-22 2018-03-29 Qualcomm Incorporated Procédé et système de génération sensible au mouvement d'une image à plage dynamique élevée
JP2018207413A (ja) * 2017-06-09 2018-12-27 キヤノン株式会社 撮像装置
JP2019016893A (ja) * 2017-07-05 2019-01-31 キヤノン株式会社 画像処理装置およびその制御方法及びプログラム
JP2020096280A (ja) * 2018-12-12 2020-06-18 キヤノン株式会社 撮像装置、撮像装置の制御方法、およびプログラム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20140111858A (ko) * 2013-03-12 2014-09-22 삼성테크윈 주식회사 이미지 처리 장치 및 방법
WO2018057103A1 (fr) * 2016-09-22 2018-03-29 Qualcomm Incorporated Procédé et système de génération sensible au mouvement d'une image à plage dynamique élevée
JP2018207413A (ja) * 2017-06-09 2018-12-27 キヤノン株式会社 撮像装置
JP2019016893A (ja) * 2017-07-05 2019-01-31 キヤノン株式会社 画像処理装置およびその制御方法及びプログラム
JP2020096280A (ja) * 2018-12-12 2020-06-18 キヤノン株式会社 撮像装置、撮像装置の制御方法、およびプログラム

Also Published As

Publication number Publication date
KR20230064959A (ko) 2023-05-11

Similar Documents

Publication Publication Date Title
WO2020246834A1 (fr) Procédé de reconnaissance d'objet dans une image
WO2020130309A1 (fr) Dispositif de masquage d'image et procédé de masquage d'image
WO2020253618A1 (fr) Procédé et dispositif de détection de scintillement vidéo
WO2021167374A1 (fr) Dispositif de recherche vidéo et système de caméra de surveillance de réseau le comprenant
US8798369B2 (en) Apparatus and method for estimating the number of objects included in an image
WO2021045599A1 (fr) Procédé d'application d'effet bokeh sur une image vidéo et support d'enregistrement
WO2023080667A1 (fr) Traitement d'image wdr de caméra de surveillance par reconnaissance d'objets basée sur l'ia
WO2013047954A1 (fr) Dispositif et procédé de capture d'image pour stabiliser des images par utilisation d'un mouvement global obtenu à partir d'un point caractéristique dans l'arrière-plan
WO2022158819A1 (fr) Procédé et dispositif électronique pour déterminer une saillance de mouvement et un style de lecture vidéo dans une vidéo
WO2023158205A1 (fr) Élimination de bruit d'une image de caméra de surveillance au moyen d'une reconnaissance d'objets basée sur l'ia
US20220006960A1 (en) Camera analyzing images on basis of artificial intelligence, and operating method therefor
US20240048672A1 (en) Adjustment of shutter value of surveillance camera via ai-based object recognition
KR102664027B1 (ko) 인공지능에 기반하여 영상을 분석하는 카메라 및 그것의 동작 방법
WO2023149603A1 (fr) Système de surveillance par images thermiques utilisant une pluralité de caméras
WO2022050622A1 (fr) Dispositif d'affichage et son procédé de commande
WO2022080680A1 (fr) Procédé et dispositif de retouche d'image basés sur une intelligence artificielle
WO2023172031A1 (fr) Génération d'images de surveillance panoramique
WO2022019355A1 (fr) Méthode de diagnostic de maladie utilisant un réseau neuronal entraîné à l'aide d'une image biométrique multiphase, et système de diagnostic de maladie la mettant en œuvre
WO2021066275A1 (fr) Dispositif électronique et procédé de commande de celui-ci
WO2022225105A1 (fr) Élimination de bruit pour une image de caméra de surveillance au moyen d'une reconnaissance d'objet basée sur l'ia
WO2023210856A1 (fr) Procédé de réglage de mise au point automatique et dispositif d'appareil de prise de vues l'utilisant
WO2024010133A1 (fr) Serveur d'apprentissage de bruit d'image et dispositif de réduction de bruit d'image utilisant l'apprentissage automatique
WO2024062971A1 (fr) Dispositif, procédé et programme de traitement d'informations
WO2022245177A1 (fr) Procédé et dispositif électronique d'obtention d'image reconstruite
WO2023277472A1 (fr) Procédé et dispositif électronique permettant de capturer une image d'un objet pour identifier un animal de compagnie

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22890398

Country of ref document: EP

Kind code of ref document: A1