WO2023100665A1 - 画像処理装置、画像処理方法、及びプログラム - Google Patents

画像処理装置、画像処理方法、及びプログラム Download PDF

Info

Publication number
WO2023100665A1
WO2023100665A1 PCT/JP2022/042662 JP2022042662W WO2023100665A1 WO 2023100665 A1 WO2023100665 A1 WO 2023100665A1 JP 2022042662 W JP2022042662 W JP 2022042662W WO 2023100665 A1 WO2023100665 A1 WO 2023100665A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unit
correction
parameter
captured image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/042662
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
裕也 山下
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/713,210 priority Critical patent/US20250014248A1/en
Priority to JP2023564867A priority patent/JPWO2023100665A1/ja
Publication of WO2023100665A1 publication Critical patent/WO2023100665A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/60—Creating or editing images; Combining images with text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/60—Extraction of image or video features relating to illumination properties, e.g. using a reflectance or lighting model
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Definitions

  • the present disclosure relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device, an image processing method, and a program that can improve the accuracy of correction of a captured image.
  • Patent Literature 1 discloses a program for automatically editing a moving image that designates a template.
  • the present disclosure has been made in view of such circumstances, and is intended to improve the accuracy of correction for a captured image.
  • An image processing apparatus corrects a captured image in a first unit, and corrects the captured image based on a relationship between the captured image and an intermediate corrected image obtained by correction in the first unit.
  • the image processing apparatus includes a processing unit that estimates a correction parameter for an image and corrects the captured image in a second unit larger than the first unit using the estimated parameter.
  • an image processing device corrects a captured image in a first unit, and the relationship between the captured image and an intermediate corrected image obtained by correction in the first unit is corrected.
  • the image processing method estimates parameters for correction of the photographed image based on the parameters, and corrects the photographed image in a second unit larger than the first unit using the estimated parameters.
  • a program causes a computer to correct a captured image in a first unit, and based on the relationship between the captured image and an intermediate corrected image obtained by correction in the first unit, the A program for estimating a correction parameter for a captured image and using the estimated parameter to correct the captured image in a second unit larger than the first unit.
  • the captured image is corrected in the first unit, and the captured image and the intermediate corrected image obtained by the correction in the first unit are A correction parameter for the photographed image is estimated based on the relationship of , and the photographed image is corrected in a second unit larger than the first unit using the estimated parameter.
  • the image processing device may be an independent device, or may be an internal block forming one device.
  • FIG. 1 is a diagram illustrating a configuration example of an embodiment of a video production system to which the present disclosure is applied;
  • FIG. It is a block diagram which shows the structural example of a camera.
  • It is a block diagram which shows the structural example of a cloud server.
  • It is a block diagram which shows the structural example of a terminal device.
  • FIG. 4 is a diagram showing an overall flow representing the flow of action creation service;
  • FIG. 10 is a diagram showing an example of a motion production algorithm;
  • FIG. 7 is a diagram showing an example of correction processing performed in automatic quality correction;
  • 4 is a block diagram showing a functional configuration example of a processing unit in the cloud server;
  • FIG. FIG. 4 is a diagram showing an overview of a first example of correction processing;
  • FIG. 10 is a diagram showing an example of a correction algorithm using a trained model;
  • FIG. 10 is a diagram showing an example of editing parameters;
  • FIG. 10 is a diagram showing an example of an image processed in the first example of correction processing;
  • 4 is a flowchart for explaining the flow of a first example of correction processing;
  • FIG. 10 is a diagram showing an overview of a second example of correction processing;
  • FIG. 4 is a diagram showing examples of brightness and contrast parameters;
  • FIG. 4 is a diagram showing examples of brightness and contrast parameters;
  • FIG. 10 is a diagram showing an overview of a second example of correction processing;
  • 9 is a flowchart for explaining the flow of a second example of correction processing;
  • FIG. 10 is a diagram illustrating another configuration example of a processing unit that executes a second example of correction processing;
  • FIG. 20 is a flowchart for explaining the flow of a second example of correction processing by the processing unit in FIG. 19;
  • FIG. 20 is a flowchart
  • FIG. 1 is a diagram showing a configuration example of an embodiment of a moving image production system to which the present disclosure is applied.
  • the movie production system 1 in FIG. 1 is a system for producing movies from images taken by a user.
  • a video production system 1 is composed of a camera 10 , a cloud server 20 and a terminal device 30 .
  • the camera 10 is a digital camera capable of shooting moving images and still images.
  • the camera 10 is not limited to a digital camera, and may be a device having a photographing function such as a smart phone or a tablet terminal.
  • the camera 10 shoots an image of a subject according to a user's operation, and records the resulting shot image.
  • Captured images include moving images and still images.
  • moving images include moving images and still images.
  • produced moving images when it is necessary to distinguish between moving images as captured images and moving images automatically produced by a moving image production service, the latter will be referred to as produced moving images.
  • the captured image captured by the camera 10 is transmitted to the cloud server 20.
  • the camera 10 can transmit the captured image to the cloud server 20 via the network 40-1.
  • a memory card such as a flash memory or wireless communication such as a wireless LAN (Local Area Network)
  • the terminal device 30 can be transferred to the network 40- 2, the captured image may be transmitted to the cloud server 20.
  • the networks 40-1 and 40-2 include communication lines such as the Internet and mobile phone networks.
  • the networks 40-1 and 40-2 may be the same network or different networks.
  • the network 40-1 and the network 40-2 will be referred to as the network 40 when there is no need to distinguish between them.
  • the cloud server 20 is a server that provides a video production service that produces (automatically produces) production videos from captured images through the network 40 .
  • the cloud server 20 is an example of an image processing device to which the present disclosure is applied.
  • the cloud server 20 receives captured images captured by the camera 10 via the network 40 .
  • the cloud server 20 produces a produced moving image by performing processing such as editing on the captured image, and transmits the produced moving image to the terminal device 30 via the network 40 .
  • the terminal device 30 is a device such as a PC (Personal Computer), a tablet terminal, or a smartphone.
  • the terminal device 30 performs processing such as settings related to the video production service and editing of the produced video according to the user's operation.
  • the terminal device 30 receives the production video transmitted from the cloud server 20 via the network 40 .
  • the terminal device 30 records the produced moving image in the terminal and outputs it to the outside.
  • FIG. 2 is a block diagram showing a configuration example of the camera 10 of FIG.
  • the camera 10 includes a lens system 111, an imaging unit 112, a camera signal processing unit 113, a recording control unit 114, a display unit 115, a communication unit 116, an operation unit 117, a camera control unit 118, and a memory unit 119. , a driver unit 120 , a sensor unit 121 , a sound input unit 122 and a sound processing unit 123 .
  • the lens system 111 takes in incident light (image light) from a subject and causes it to enter the imaging unit 112 .
  • the imaging unit 112 has a solid-state imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and converts the amount of incident light imaged on the imaging surface of the solid-state imaging device by the lens system 111 into an electric signal in pixel units. It is converted and output as a pixel signal.
  • CMOS Complementary Metal Oxide Semiconductor
  • the camera signal processing unit 113 is composed of a DSP (Digital Signal Processor), a frame memory for temporarily recording image data, and the like.
  • the camera signal processing unit 113 performs various kinds of signal processing on the image signal output from the imaging unit 112, and outputs image data of the captured image obtained as a result. In this manner, the lens system 111, the imaging section 112, and the camera signal processing section 113 constitute an imaging system.
  • the recording control unit 114 records the image data of the captured image captured by the imaging system in a storage medium including a memory card such as a flash memory.
  • a display unit 115 is composed of a liquid crystal display, an organic EL display, or the like, and displays a captured image captured by the imaging system.
  • the communication unit 116 is composed of a communication module or the like compatible with a predetermined communication method such as wireless communication including wireless LAN and cellular communication (for example, 5G (5th Generation)). Data is transmitted to other devices including the cloud server 20 via the network 40 or the like.
  • the operation unit 117 includes an operation system such as physical buttons and a touch panel, and issues operation commands for various functions of the camera 10 according to user's operations.
  • the camera control unit 118 is composed of processors such as a CPU (Central Processing Unit) and a microprocessor, and controls the operation of each unit of the camera 10 .
  • the memory unit 119 records various data under the control of the camera control unit 118 .
  • the driver unit 120 drives the lens system 111 to achieve autofocus, zooming, etc., under the control of the camera control unit 118 .
  • the sensor unit 121 senses spatial information, time information, etc., and outputs a sensor signal obtained as a result of the sensing.
  • the sensor unit 121 includes various sensors such as a gyro sensor and an acceleration sensor.
  • the sound input unit 122 is composed of a microphone or the like, collects sounds such as the user's voice or environmental sounds, and outputs the resulting sound signal.
  • the sound processing unit 123 performs sound signal processing on the sound signal output from the sound input unit 122 .
  • the sound signal from the sound processing unit 123 is input to the camera signal processing unit 113, processed in synchronization with the image signal under the control of the camera control unit 118, and recorded as the sound (audio) of the moving image.
  • FIG. 3 is a block diagram showing a configuration example of the cloud server 20 of FIG.
  • a CPU 211 a CPU (Read Only Memory) 212 and a RAM (Random Access Memory) 213 are interconnected by a bus 214 .
  • An input/output I/F 215 is further connected to the bus 214 .
  • An input unit 216 , an output unit 217 , a storage unit 218 and a communication unit 219 are connected to the input/output I/F 215 .
  • the input unit 216 supplies various input signals to each unit including the CPU 211 via the input/output I/F 215 .
  • the input unit 216 is composed of a keyboard, mouse, microphone, and the like.
  • the output unit 217 outputs various information according to control from the CPU 211 via the input/output I/F 215 .
  • the output unit 217 is composed of a display, a speaker, and the like.
  • the storage unit 218 is configured as an auxiliary storage device such as a semiconductor memory or HDD (Hard Disk Drive).
  • the storage unit 218 records various data and programs under the control of the CPU 211 .
  • the CPU 211 reads and processes various data from the storage unit 218 and executes programs.
  • the communication unit 219 is composed of a communication module that supports wireless communication such as wireless LAN or cellular communication (eg 5G), or wired communication.
  • the communication unit 219 communicates with other devices including the camera 10 and the terminal device 30 via the network 40 under the control of the CPU 211 .
  • the configuration of the cloud server 20 shown in FIG. 3 is an example, and image processing may be performed by providing a dedicated processor such as a GPU (Graphics Processing Unit).
  • a dedicated processor such as a GPU (Graphics Processing Unit).
  • FIG. 4 is a block diagram showing a configuration example of the terminal device 30 of FIG.
  • the CPU 311 , ROM 312 and RAM 313 are interconnected by a bus 314 .
  • An input/output I/F 315 is further connected to the bus 314 .
  • An input unit 316 , an output unit 317 , a storage unit 318 and a communication unit 319 are connected to the input/output I/F 315 .
  • the input unit 316 supplies various input signals to each unit including the CPU 311 via the input/output I/F 315 .
  • the input section 316 has an operation section 321 .
  • the operation unit 321 includes a keyboard, mouse, microphone, physical buttons, touch panel, and the like. The operation unit 321 is operated by a user and supplies an operation signal corresponding to the operation to the CPU 311 .
  • the output unit 317 outputs various information according to control from the CPU 311 via the input/output I/F 315 .
  • the output section 317 has a display section 331 and a sound output section 332 .
  • the display unit 331 is composed of a liquid crystal display, an organic EL display, and the like.
  • a display unit 331 displays a captured image, an editing screen, and the like under the control of the CPU 311 .
  • the sound output unit 332 is composed of a speaker, a headphone connected to an output terminal, or the like. The sound output unit 332 outputs a sound according to the sound signal under the control of the CPU 311 .
  • the storage unit 318 is configured as an auxiliary storage device such as a semiconductor memory.
  • the storage unit 318 may be configured as an internal storage, or may be an external storage such as a memory card.
  • a storage unit 318 records various data and programs under the control of the CPU 311 .
  • the CPU 311 reads and processes various data from the storage unit 318 and executes programs.
  • the communication unit 319 is composed of a communication module compatible with a predetermined communication method such as wireless communication such as wireless LAN or cellular communication (eg 5G), or wired communication.
  • the communication unit 319 communicates with other devices including the cloud server 20 via the network 40 under the control of the CPU 311 .
  • the configuration of the terminal device 30 shown in FIG. 4 is an example, and image processing may be performed by providing a dedicated processor such as a GPU.
  • captured images captured by the camera 10 are uploaded to the cloud server 20, and processing such as editing using the captured images is performed by the cloud server 20. is produced.
  • the terminal device 30 setting and editing related to the produced moving image are performed, and the completed produced moving image is output.
  • the cloud server 20 is installed in a data center or the like, but is not limited to one server, and may be composed of a plurality of servers to provide the moving image production service.
  • FIG. 5 is a diagram showing the overall flow of the moving picture production service provided by the cloud server 20 in the moving picture production system 1. As shown in FIG. The overall flow shown in FIG. 5 is divided into before shooting, during shooting, during editing, and review by concerned parties, with the direction from left to right being the direction of time.
  • project creation S1
  • automatic transfer settings for shot images S2 are performed.
  • the user inputs a project name and creates a project for managing information relating to the production of the produced moving image.
  • the automatic transfer destination of the photographed image photographed by the camera 10 is set.
  • the location information URL (Uniform Resource Locator), etc.) of the cloud server 20 is set as the automatic transfer destination.
  • Project creation and automatic transfer settings can be performed from the camera 10 or the terminal device 30 by the user's operation. Alternatively, it may be set from another device such as a smartphone owned by the user.
  • the video is automatically produced (S3).
  • the cloud server 20 produces produced moving images using captured images transferred from the camera 10 .
  • processes such as automatic selection, automatic trimming, and automatic quality correction are performed using captured images.
  • the video is additionally edited (S4).
  • the produced moving image is additionally edited by the terminal device 30 according to the user's operation. This additional editing is not essential, and if the user determines that there is no need to edit the produced moving image, there is no need to perform additional editing.
  • the video is output and shared (S5).
  • the terminal device 30 In outputting and sharing the moving image, the terminal device 30 outputs the produced moving image additionally edited as necessary in a predetermined format and shares it with the relevant parties. As a result, the produced moving image is reviewed by the relevant parties, and delivered after being appropriately corrected according to the review results.
  • FIG. 6 is a diagram showing an example of a video production algorithm used when producing a video in the automatic video production (S3) of FIG.
  • the video production algorithm includes processes such as automatic selection, automatic trimming, and automatic quality correction, and the production video is produced by performing these processes.
  • captured images (clips) including moving images and still images are grouped (grouped) by scene, and the appropriate captured image is selected for each scene.
  • the trimming range of a moving image (clip) is specified by an in point (start point) and an out point (end point).
  • start point an in point
  • end point an out point
  • a clip represents a photographed image captured by a device such as the cloud server 20 .
  • various corrections are applied to the captured image (clip) to improve the quality. For example, exposure correction (brightness correction) between multiple clips, color correction (color correction) between multiple clips, noise reduction of sound, camera shake correction, video effects such as panning and zooming, and audio level equalization. done.
  • automatic selection, automatic trimming, and automatic quality correction are exemplified as video production algorithms, but other processing may be added as long as it is necessary to produce the production video. .
  • FIG. 7 is a diagram showing an example of correction processing performed in automatic quality correction.
  • FIG. 7 illustrates a case where color correction of an input image including a photographed image is performed as correction processing.
  • target image B that is the target of correction (exemplar: target color, for example) is used.
  • a corrected corrected image D is output.
  • the input image A is corrected in two steps, pixel-by-pixel correction and image-frame-by-frame correction.
  • An input image A is a photographed image such as a moving image or a still image photographed by the camera 10 .
  • the input image A may be a photographed image subjected to processing such as automatic selection or automatic trimming.
  • the target image B may be a photographed image photographed by the camera 10, or may be an image other than the photographed image (for example, an image prepared in advance).
  • FIG. 8 is a block diagram showing a functional configuration example of the processing unit 200 in the cloud server 20.
  • the processing unit 200 is realized by executing a program such as a video production program by a processor such as the CPU 211 or GPU.
  • the processing unit 200 may be implemented as a dedicated circuit.
  • the input image A including the photographed image is corrected in two stages, the first unit and the second unit.
  • the first unit is pixel units and the second unit is image frame units.
  • the processing unit 200 has a pixel-by-pixel correction unit 251 , a parameter estimation unit 252 and an image frame correction unit 253 .
  • the input image A and the target image B are input to the pixel-by-pixel correction unit 251 .
  • the pixel-by-pixel correction unit 251 uses the input image A and the target image B to perform pixel-by-pixel correction, and supplies the resulting intermediate corrected image C to the parameter estimation unit 252 .
  • corrections such as color correction and brightness correction of the input image A are performed.
  • the input image A or the input image A and the target image B, and the intermediate corrected image C from the pixel-by-pixel correction unit 251 are input to the parameter estimation unit 252 .
  • the parameter estimating unit 252 uses the input image A and the intermediate corrected image C to estimate an editing parameter P, which is a correction parameter for the entire area (entire image frame) of the input image A, and supplies it to the image frame correcting unit 253 .
  • the parameter estimator 252 estimates the editing parameter P using the input image A, the target image B, and the intermediate corrected image C, and supplies it to the image frame corrector 253 .
  • the input image A and the editing parameter P from the parameter estimation unit 252 are input to the image frame correction unit 253 .
  • the image frame correction unit 253 uses the editing parameter P to correct the input image A in image frame units, and outputs a corrected image D obtained as a result.
  • correction such as color correction and brightness correction of the input image A is performed as correction corresponding to the pixel unit correction.
  • the input image A is corrected in pixel units by the pixel unit correction unit 251 and corrected in image frame units using the editing parameter P by the image frame correction unit 253 .
  • the correction is performed in two steps. By performing such a two-step correction, it is possible to improve the accuracy of the correction for the captured image. For example, by performing two-stage correction, failure of correction can be suppressed as compared with the case where only the previous-stage correction is performed on a pixel-by-pixel basis.
  • FIG. 200 Two patterns of a first example and a second example will be described below as two-step correction processing performed by the processing unit 200.
  • FIG. 9 is a diagram showing an overview of a first example of correction processing executed by the processing unit 200 of FIG. 9 will be described with reference to FIGS. 10 to 12 as appropriate.
  • the input image A11 such as the captured image is corrected in two stages. That is, in the first example of the correction process, the editing parameter P11 is estimated from the intermediate corrected image C11 obtained by pixel-by-pixel correction using the input image A11 and the target image B11, and the input image A11 is corrected for each image frame.
  • FIG. 9 illustrates a case where color correction is performed as correction processing for an input image A11 such as a captured image.
  • the input image A11 and the target image B11 are input to the pixel-by-pixel correction unit 251 .
  • the pixel-by-pixel correction unit 251 uses the target image B11 to color-correct the input image A11 on a pixel-by-pixel basis, and supplies the resulting intermediate corrected image C11 to the parameter estimation unit 252 .
  • a learned model 261 learned by machine learning can be used as the pixel-by-pixel correction process.
  • an intermediate corrected image C11 can be obtained as a result of correction of the output.
  • the learned model 261 can use a DNN (Deep Neural Network) that has been trained with an image as learning data as an input and an image after color correction as an output.
  • DNN Deep Neural Network
  • a well-known technique can be used for such a DNN-based correction algorithm.
  • the input image A11 can be converted into the hue of the target image B11.
  • the learned model 261 can be used for all or part of the process. For example, processing using the trained model 261 and image processing for the input image A11 and the target image B11 may be combined. Alternatively, if the trained model 261 is not used in pixel-by-pixel correction processing, image processing, rule-based processing, or the like may be used.
  • the input image A11 and the intermediate corrected image C11 from the pixel-by-pixel correction unit 251 are input to the parameter estimation unit 252 .
  • the parameter estimation unit 252 estimates the editing parameter P11 from the correspondence relationship between the input image A11 and the intermediate corrected image C11, and supplies it to the image frame correction unit 253.
  • the editing parameter P11 is a correction parameter for the entire area (entire image frame) of the input image A11.
  • the editing parameters P11 can include parameters used in general image editing software. For example, parameters used in image editing processing such as tone curve and level correction can be used.
  • the tone curves 271R, 271G, and 271B for each RGB value give an image You can adjust the color tone of the entire frame.
  • the parameter estimating section 252 can estimate a parameter related to color tone adjustment using this tone curve as an editing parameter P11.
  • the input image A11 and the editing parameter P11 from the parameter estimation unit 252 are input to the image frame correction unit 253 .
  • the image frame correction unit 253 uses the editing parameter P11 to color-correct the input image A11 in units of image frames, and outputs a corrected image D11 obtained as a result.
  • the image frame correction unit 253 recorrects the input image A11 using parameters (tone curve parameters, etc.) suitable for general image editing software as the editing parameters P11. That is, in the pixel-by-pixel correction process, there is a possibility that the correction may be partially broken in the image frame. By doing so, it is possible to suppress the failure of the correction.
  • an intermediate corrected image C11 obtained by pixel-by-pixel correction using an input image A11 and a target image B11 is such that the image frame as a whole is appropriately corrected.
  • the correction is broken in some parts (for example, the hand part). Therefore, by correcting the input image A11 in units of image frames using the editing parameter P11 through correction in two stages, a corrected image D11 in which correction failure is suppressed can be obtained.
  • the image frame correction unit 253 is not limited to specific image editing software, and performs correction processing using image editing software of each company. be able to. Further, since the image frame correction unit 253 only corrects the tone curve using the editing parameter P11 by image editing software, it is possible to correct only the tint without causing an overall failure.
  • step S111 the pixel-by-pixel correction unit 251 performs pixel-by-pixel correction (for example, color correction) using the input image A11 and the target image B11 input thereto to obtain an intermediate corrected image C11.
  • pixel-by-pixel correction for example, color correction
  • step S112 the parameter estimation unit 252 estimates the editing parameter P11 from the correspondence relationship between the input image A11 and the intermediate corrected image C11. For example, a tone curve parameter is estimated as the editing parameter P11.
  • step S113 the image frame correction unit 253 performs correction (for example, color correction) for each image frame on the input image A11 using the editing parameter P11 to obtain a corrected image D11.
  • correction for example, color correction
  • general image editing software performs a correction process for color adjustment using tone curve parameters as editing parameters P11.
  • tone curve parameters as editing parameters P11.
  • the input image A11 (captured image, etc.) is corrected in the first unit (pixel unit) using the target image B11, and the input image A11 and the input image A11 are corrected.
  • An editing parameter P11 (a parameter such as a tone curve) is estimated based on the relationship with the intermediate corrected image C11 obtained from the target image B11. image frame unit).
  • the photographed image as the input image A11 has a different color tone depending on the subject and light conditions at the time of photographing.
  • the moving image production system 1 applies the first example of the correction process at the time when the photographed images to be used for producing the produced moving image are determined. , it is possible to evenly match the color tone of the captured image of the object. As a result, it is possible to reduce the sense of incongruity when the user views the produced moving image, and improve the completeness of the produced moving image.
  • correction processing is not automatically performed, a user who has knowledge of editing performs this manually, which takes time and effort. With this correction processing, a user with knowledge of editing can save labor through automation, while a user without knowledge of editing can do things that have not been possible until now.
  • FIG. 14 is a diagram showing an overview of a second example of correction processing executed by the processing unit 200 of FIG. The description of FIG. 14 will be made with reference to FIGS. 15 to 17 as appropriate.
  • the editing parameters P21 and P23 are estimated from the intermediate corrected images C21 and C21 obtained by pixel-by-pixel correction using the input image A21 and the target image B21 separately. Using the parameters P21 and P23, the input image A21 is corrected for each image frame.
  • FIG. 14 illustrates a case where brightness correction is performed as correction processing for an input image A21 such as a captured image.
  • the pixel-by-pixel correction unit 251-1 performs brightness correction on the input image A21 input thereto on a pixel-by-pixel basis, and supplies the intermediate corrected image C21 obtained as a result to the parameter estimation unit 252-1.
  • automatic correction such as AE (Automatic Exposure) correction is performed.
  • This automatic correction can use a learned model learned by machine learning.
  • an intermediate corrected image C21 can be obtained as a corrected result of the output.
  • the trained model can use a DNN that has been trained with an image as learning data as an input and an image after brightness correction as an output.
  • a well-known technique can be used for such a DNN-based correction algorithm.
  • pixel-by-pixel correction processing all or part of the processing can be processed using a learned model. For example, processing using a trained model and image processing for the input image A21 may be combined. Alternatively, image processing, rule-based processing, or the like may be used in pixel-by-pixel correction processing when a learned model is not used.
  • the input image A21 and the intermediate corrected image C21 from the pixel-by-pixel correction unit 251-1 are input to the parameter estimation unit 252-1.
  • the parameter estimation unit 252-1 estimates the editing parameter P21 from the correspondence relationship between the input image A21 and the intermediate corrected image C21, and supplies it to the image frame correction unit 253-1.
  • the editing parameter P21 is a correction parameter for the entire area (entire image frame) of the input image A21.
  • the editing parameters P21 can include parameters used in general image editing software.
  • parameters such as brightness and contrast can be used as the editing parameter P21.
  • the horizontal axis and the vertical axis are input and output, and the relationship between the solid line and the broken line in the figure indicates that the correspondence relationship between the brightness of the input and output can be adjusted by uniformly adding an offset.
  • the relationship between the solid line and the broken line in the figure indicates that the inclination of the correspondence relationship between the input and output contrasts can be adjusted.
  • the pixel-by-pixel correction unit 251-2 performs brightness correction on a pixel-by-pixel basis on the input target image B21, and supplies the intermediate corrected image C22 obtained as a result to the parameter estimation unit 252-2.
  • the pixel-by-pixel correction unit 251-2 performs the same pixel-by-pixel correction process as the pixel-by-pixel correction unit 251-1.
  • the target image B21 and the intermediate corrected image C22 from the pixel-by-pixel correction unit 251-2 are input to the parameter estimation unit 252-2.
  • the parameter estimator 252-2 estimates the editing parameter P22 from the correspondence relationship between the target image B21 and the intermediate corrected image C22.
  • Parameter estimating section 252-2 estimates parameters related to brightness in the same manner as parameter estimating section 252-1.
  • the parameter estimation unit 252-2 further estimates an editing parameter P23 for correcting in the opposite direction to the editing parameter P22, and supplies it to the image frame correction unit 253-2.
  • an image frame correction unit 254 is provided after the parameter estimation units 252-1 and 252-2, and the image frame correction unit 254 performs correction using the editing parameters P21 and P22. Assume that it is done.
  • the image frame correction unit 254-1 performs brightness correction on the input image A21 using the editing parameter P21 to obtain a corrected image D21.
  • the image frame correction unit 254-2 performs brightness correction using the editing parameter P22 to obtain a corrected image D22.
  • the corrected image D21 and the corrected image D22 obtained by such correction are images with similar brightness.
  • the second correction processing is performed using the relationship that the corrected images D21 and D22 obtained by the correction processing have similar brightness. example is done.
  • the input image A21 and the editing parameter P21 from the parameter estimation unit 252-1 are input to the image frame correction unit 253-1.
  • the image frame corrector 253-1 performs brightness correction on the input image A21 using the editing parameter P21, and supplies the resulting corrected image E21 to the image frame corrector 253-2.
  • the corrected image E21 from the image frame correction section 253-1 and the editing parameter P23 from the parameter estimation section 252-2 are input to the image frame correction section 253-2.
  • the image frame correction unit 253-2 performs brightness correction (inverse conversion) on the corrected image E21 using the editing parameter P23, and outputs the resulting corrected image F21.
  • the brightness correction using the editing parameter P23 is performed after the brightness correction using the editing parameter P21.
  • the brightness of the image is brought closer to the target value.
  • the input image A21 is corrected in image frame units using the editing parameters P21 and P23, thereby suppressing the failure of the correction.
  • a corrected image F21 can be obtained.
  • the editing parameter P21 and the editing parameter P23 parameters suitable for general image editing software, such as parameters related to brightness, are used. Correction processing can be performed by image editing software of each company.
  • step S131 the pixel-by-pixel correction unit 251-1 performs automatic AE correction (for example, brightness correction) on the input image A21 input thereto to obtain an intermediate corrected image C21.
  • automatic AE correction for example, brightness correction
  • step S132 the parameter estimation unit 252-1 estimates the editing parameter P21 from the correspondence relationship between the input image A21 and the intermediate corrected image C21. For example, as the editing parameter P21, a parameter related to brightness such as "brightness+0.2" is estimated.
  • step S136 When the processing of the input image A21 in steps S131 and S132 is completed, the process proceeds to step S136.
  • step S133 the pixel-by-pixel correction unit 251-2 performs automatic AE correction (for example, brightness correction) on the input target image B21 to obtain an intermediate corrected image C22.
  • automatic AE correction for example, brightness correction
  • step S134 the parameter estimation unit 252-2 estimates the editing parameter P22 from the correspondence relationship between the target image B21 and the intermediate corrected image C22. Also, in step S135, the parameter estimator 252-2 estimates an editing parameter P23 that corrects in the opposite direction to the editing parameter P22.
  • steps S133 to S135 When the processing for the target image B21 in steps S133 to S135 is completed, the processing proceeds to step S136. Steps S131 and S132 and steps S133 to S135 can be processed in parallel.
  • step S136 the image frame correction unit 253-1 performs correction (for example, brightness correction) on the input image A21 input thereto using the editing parameter P21 to obtain a corrected image E21.
  • a corrected image E21 is obtained by correcting the input image A21 using the editing parameter P21 of "brightness+0.2".
  • step S137 the image frame correction unit 253-2 performs correction (for example, brightness correction) on the corrected image E21 using the editing parameter P23 to obtain a corrected image F21.
  • the final corrected image F21 is obtained by correcting the corrected image E21 using the editing parameter P23 of "brightness+0.1".
  • the input image A21 (photographed image, etc.) is corrected in the first unit (pixel unit), and the input image A21 and the intermediate corrected image obtained from the input image A21 are Editing parameters P21 (parameters related to brightness, etc.) are estimated based on the relationship with C21, and target image B21 is corrected in the first unit (pixel unit) to obtain target image B21 and target image B21.
  • Editing parameters P22 (parameters related to brightness, etc.) are estimated based on the relationship with the intermediate corrected image C22.
  • an editing parameter P23 for performing correction in the direction opposite to the correction using the estimated editing parameter P22 is estimated, and the estimated editing parameter P21 and editing parameter P23 are used to convert the input image A21 (such as a photographed image) into a third image. Corrected in units of 2 (image frame units). By performing such a two-step correction, it is possible to suppress the failure of the correction and improve the accuracy of the correction.
  • the captured image as the input image A11 is in a different state in terms of brightness, etc., depending on the subject and light conditions at the time of capturing.
  • the moving image production system 1 applies the second example of the correction process at the time when the photographed images to be used for producing the produced moving image are determined. , the brightness of the captured image of the object can be uniformed.
  • FIG. 19 is a diagram showing another configuration example of the processing unit 200 that executes the second example of correction processing.
  • the configuration shown in FIG. 19 has a configuration in which a parameter integrating section 255 is provided after the parameter estimating section 252-1 and the parameter estimating section 252-2.
  • the parameter integrating section 255 receives the editing parameter P21 from the parameter estimating section 252-1 and the editing parameter P23 from the parameter estimating section 252-2.
  • the editing parameter P21 is a parameter estimated from the correspondence relationship between the input image A21 and the intermediate corrected image C21.
  • the editing parameter P23 is a parameter for performing correction in the direction opposite to the correction using the editing parameter P22 estimated from the correspondence relationship between the target image B21 and the intermediate corrected image C22.
  • the parameter integration unit 255 estimates the editing parameter P24 by integrating the editing parameter P21 and the editing parameter P23, and supplies it to the image frame correction unit 253.
  • the input image A21 and the editing parameter P24 from the parameter integration unit 255 are input to the image frame correction unit 253 .
  • the image frame correction unit 253 performs brightness correction on the input image A21 using the editing parameter P24, and outputs a corrected image F21 obtained as a result.
  • steps S151 and S152 similar to steps S131 and S132 in FIG. 18, the input image A21 is automatically corrected, and the editing parameter P21 is estimated from the correspondence relationship between the input image A21 and the intermediate corrected image C21.
  • steps S153 to S155 similarly to steps S133 to S135 in FIG. 18, the target image B21 is automatically corrected, and the editing parameter P22 estimated from the correspondence relationship between the target image B21 and the intermediate corrected image C22 is corrected in the opposite direction. is estimated as an editing parameter P23.
  • the parameter integration unit 255 estimates the editing parameter P24 by integrating the editing parameter P21 and the editing parameter P23. For example, if a parameter of "brightness + 0.2" is estimated as the editing parameter P21 and a parameter of "brightness + 0.1" is estimated as the editing parameter P23, then these parameters are integrated to obtain an editing parameter of "brightness + 0.3". A parameter P24 is estimated.
  • step S157 the image frame correction unit 253 performs correction using the editing parameter P24 on the input image A21 input thereto to obtain a corrected image F21.
  • the final corrected image F21 is obtained by correcting the input image A21 using the editing parameter P24 of "brightness+0.3".
  • the correction process is performed more efficiently than the configuration in which the correction process is performed for each of the two editing parameters P21 and P23. can be combined into one, the processing load can be reduced.
  • the processing unit 200 of the cloud server 20 executes the correction processing, but the processing may be executed by a device other than the cloud server 20.
  • the processing unit of the terminal device 30 may have functions corresponding to the processing unit 200 to perform all or part of the correction processing.
  • Programs executed by computers can be provided by being recorded on removable recording media such as package media, for example. Also, the program can be provided via wired or wireless transmission media such as LAN, Internet, and digital satellite broadcasting.
  • the program can be installed in the storage unit via the input/output I/F by loading the removable recording medium into the drive. Also, the program can be received by the communication unit and installed in the storage unit via a wired or wireless transmission medium. In addition, the program can be pre-installed in the ROM or storage unit.
  • processing performed by the computer according to the program does not necessarily have to be performed in chronological order according to the order described as the flowchart.
  • processing performed by a computer according to a program includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).
  • the program may be processed by one computer (processor), or may be processed by a plurality of computers in a distributed manner. Furthermore, the program may be transferred to and executed on a remote computer.
  • the term “automatic” means that a device such as the cloud server 20 performs processing without the user’s direct operation
  • the term “manual” means that the user directly It means that processing is performed through a similar operation.
  • the effects described in this specification are merely examples and are not limited, and other effects may be provided.
  • a system means a set of multiple components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, both a plurality of devices housed in separate enclosures and connected via a network and a single device housing a plurality of modules within a single enclosure are systems.
  • the present disclosure can be configured as follows.
  • An image processing apparatus comprising a processing unit that corrects the captured image in a second unit larger than the first unit using the estimated parameter.
  • the first unit is a pixel unit, the second unit is an image frame unit;
  • the processing unit is correcting the captured image in the first unit using a target image as a correction target;
  • the image processing device according to (2), wherein the parameter is estimated based on a relationship between the captured image and the intermediate corrected image obtained from the captured image and the target image.
  • the processing unit is correcting the captured image in the first unit; estimating a first parameter for correction of the captured image based on a relationship between the captured image and a first intermediate corrected image obtained by correcting the captured image; correcting a target image to be corrected in the first unit; estimating a second parameter for correction of the captured image based on the relationship between the target image and a second intermediate corrected image obtained by correcting the target image; Estimate a third parameter that performs correction in the opposite direction to the correction using the estimated second parameter,
  • the image processing device according to (2), wherein the captured image is corrected in the second unit using the estimated first parameter and third parameter.
  • the processing unit is correcting the captured image in the second unit using the estimated first parameter; The image processing device according to (8), wherein the captured image corrected using the first parameter is corrected in the second unit using the estimated third parameter. (10) The processing unit is Estimate a fourth parameter that integrates the first parameter and the third parameter, The image processing device according to (8), wherein the captured image is corrected in the second unit using the estimated fourth parameter. (11) The image processing device according to any one of (1) to (10), wherein the processing unit corrects color or brightness of the captured image.
  • the captured image captured by a camera operated by a user and configured as a server for processing the captured image received via a network The image processing apparatus according to any one of (1) to (11), wherein a moving image produced based on the corrected captured image is transmitted to a terminal device operated by a user via a network.
  • the image processing device correcting the captured image in the first unit; estimating a correction parameter for the captured image based on the relationship between the captured image and the intermediate corrected image obtained by the first unit correction; An image processing method comprising correcting the captured image in a second unit larger than the first unit using the estimated parameter.
  • the computer correcting the captured image in the first unit; estimating a correction parameter for the captured image based on the relationship between the captured image and the intermediate corrected image obtained by the first unit correction;
  • a program functioning as a processing unit that corrects the captured image in a second unit larger than the first unit using the estimated parameter.
  • Video production system 10 camera, 20 cloud server, 30 terminal device, 40-1, 40-2, 40 network, 200 processing unit, 211 CPU, 251, 251-1, 251-2 pixel unit correction unit, 252, 252-1, 252-2 parameter estimator, 253, 253-1, 253-2 image frame corrector, 255 parameter integration unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Studio Devices (AREA)
PCT/JP2022/042662 2021-12-01 2022-11-17 画像処理装置、画像処理方法、及びプログラム Ceased WO2023100665A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US18/713,210 US20250014248A1 (en) 2021-12-01 2022-11-17 Image processing apparatus, image processing method, and program
JP2023564867A JPWO2023100665A1 (https=) 2021-12-01 2022-11-17

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-195415 2021-12-01
JP2021195415 2021-12-01

Publications (1)

Publication Number Publication Date
WO2023100665A1 true WO2023100665A1 (ja) 2023-06-08

Family

ID=86612021

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/042662 Ceased WO2023100665A1 (ja) 2021-12-01 2022-11-17 画像処理装置、画像処理方法、及びプログラム

Country Status (3)

Country Link
US (1) US20250014248A1 (https=)
JP (1) JPWO2023100665A1 (https=)
WO (1) WO2023100665A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005136852A (ja) * 2003-10-31 2005-05-26 Canon Inc 画像処理方法、画像処理装置および画像処理プログラム
JP2010231668A (ja) * 2009-03-27 2010-10-14 Infocom Corp アニメーション自動作成システム
CN111311517A (zh) * 2020-02-26 2020-06-19 福州大学 基于抠图的颜色校正优化方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006301675A (ja) * 2005-04-15 2006-11-02 Noritsu Koki Co Ltd 画像処理装置及び画像処理方法
WO2009098741A1 (ja) * 2008-02-04 2009-08-13 Panasonic Corporation 撮像装置、集積回路及び撮像方法
JP6303270B2 (ja) * 2012-05-18 2018-04-04 株式会社リコー ビデオ会議端末装置、ビデオ会議システム、映像の歪み補正方法および映像の歪み補正プログラム
EP2873237B1 (en) * 2012-07-13 2022-03-30 Koninklijke Philips N.V. Improved hdr image decoding methods and devices
CN102970549B (zh) * 2012-09-20 2015-03-18 华为技术有限公司 图像处理方法及装置
CN111886859B (zh) * 2018-03-22 2022-09-23 凸版印刷株式会社 颜色对应信息生成系统及方法、计算机可读取的记录介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005136852A (ja) * 2003-10-31 2005-05-26 Canon Inc 画像処理方法、画像処理装置および画像処理プログラム
JP2010231668A (ja) * 2009-03-27 2010-10-14 Infocom Corp アニメーション自動作成システム
CN111311517A (zh) * 2020-02-26 2020-06-19 福州大学 基于抠图的颜色校正优化方法

Also Published As

Publication number Publication date
US20250014248A1 (en) 2025-01-09
JPWO2023100665A1 (https=) 2023-06-08

Similar Documents

Publication Publication Date Title
CN104702851B (zh) 使用嵌入式数据的强大自动曝光控制
US9137447B2 (en) Imaging apparatus that generates an image including an emphasized in-focus part of a captured image
CN110060213A (zh) 图像处理方法、装置、存储介质及电子设备
US20180007292A1 (en) Imaging device, imaging method, and image processing device
WO2021104394A1 (zh) 图像处理方法及装置、电子设备、存储介质
JP2015149691A (ja) 画像補正装置、画像補正方法、及び、撮像装置
CN110675458A (zh) 用于摄像头标定的方法、装置及存储介质
US20080094484A1 (en) Image capture methods and systems
CN113810593A (zh) 图像处理方法、装置、存储介质及电子设备
JP2025510585A (ja) 露光撮像方法、装置、機器、記憶媒体及びプログラム製品
US10091415B2 (en) Image processing apparatus, method for controlling image processing apparatus, image pickup apparatus, method for controlling image pickup apparatus, and recording medium
CN112087569B (zh) 一种摄像机及摄像机启动方法、装置
CN110431840B (zh) 图像处理装置、方法以及存储介质
US20180270450A1 (en) Image processing device, image processing method, and picture transmission and reception system
CN112584008B (zh) 图像处理设备、图像处理方法、摄像设备和存储介质
JP6032912B2 (ja) 撮像装置、その制御方法及びプログラム
WO2023100665A1 (ja) 画像処理装置、画像処理方法、及びプログラム
TW202207693A (zh) 資料產生方法、學習方法、推定方法、資料產生裝置及程式
JP6083959B2 (ja) 伝送装置、その制御方法、および制御プログラム
US20160088225A1 (en) Method and technical equipment for imaging
TW201801515A (zh) 影像產生方法及影像拍攝裝置
JP2021114180A (ja) 画像処理装置、画像処理方法、およびプログラム
US9659222B2 (en) Vehicle event data recorder and operation method thereof
WO2022219985A1 (ja) 情報処理方法、情報処理装置、プログラム
JP5975705B2 (ja) 撮像装置およびその制御方法、画像処理装置および画像処理のための方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22901098

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023564867

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18713210

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22901098

Country of ref document: EP

Kind code of ref document: A1