WO2022244351A1 - 画像処理装置、画像処理方法、プログラム - Google Patents

画像処理装置、画像処理方法、プログラム Download PDF

Info

Publication number
WO2022244351A1
WO2022244351A1 PCT/JP2022/006239 JP2022006239W WO2022244351A1 WO 2022244351 A1 WO2022244351 A1 WO 2022244351A1 JP 2022006239 W JP2022006239 W JP 2022006239W WO 2022244351 A1 WO2022244351 A1 WO 2022244351A1
Authority
WO
WIPO (PCT)
Prior art keywords
moire
image
information
processing
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/006239
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
遼太 宮澤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/556,954 priority Critical patent/US20240221139A1/en
Priority to JP2023522232A priority patent/JP7761045B2/ja
Publication of WO2022244351A1 publication Critical patent/WO2022244351A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/91—Television signal processing therefor
    • H04N5/92—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/40—Picture signal circuits
    • H04N1/409—Edge or detail enhancement; Noise or error suppression
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H04N5/145—Movement estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/147—Scene change detection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Definitions

  • This technology relates to an image processing device, an image processing method, and a program, and particularly to the technical field of moiré that occurs in images.
  • Japanese Patent Laid-Open No. 2002-200002 proposes a method of preparing two optical systems with different resolutions, detecting moire from the difference between the two, and reducing the moire. Further, Japanese Patent Application Laid-Open No. 2002-200002 discloses a technique for detecting and reducing moire from the difference between two frames with different cutoff frequencies using a variable optical low-pass filter.
  • the difference between the two images also includes high-frequency components as real images that are not moiré, and only the low-frequency part is the difference that is moiré.
  • moire folded back to low frequencies can be detected, but the difference between moiré and true high frequency components cannot be discriminated in the high frequency part. For this reason, there is a trade-off relationship between maintaining the sense of resolution in the high-frequency area and eliminating moire.
  • this technology proposes a method that can detect moire by distinguishing it from patterns in the real image, regardless of the moire frequency.
  • the image processing device detects pixel regions with different movements among pixel regions where the same movement as that of an object is assumed to occur as a change in position within a frame between images taken at different times. and a moire detection unit that generates moire detection information.
  • Images at different times may be, for example, a current image and an image one to several frames before. If there is a change in the position of a certain subject between frames at different times, that is, if there is motion, and the pixel region is supposed to show the same motion as the motion, but the motion is different, The pixel area is determined as moire.
  • FIG. 1 is a block diagram of an imaging device according to an embodiment of the present technology
  • FIG. 1 is a block diagram of an information processing device according to an embodiment
  • FIG. 1 is a block diagram of a configuration example of an image processing apparatus according to an embodiment
  • FIG. 4 is a flowchart of moire detection processing according to the embodiment
  • 4 is a flowchart of moire reduction processing according to the embodiment
  • FIG. 10 is a block diagram of another configuration example of the image processing apparatus according to the embodiment
  • 4 is a flowchart of a first example of moiré detection processing according to the embodiment
  • FIG. 4 is an explanatory diagram of a concept of moire detection according to the embodiment
  • FIG. 4 is an explanatory diagram of a concept of moire detection according to the embodiment
  • FIG. 4 is an explanatory diagram of a concept of moire detection according to the embodiment
  • FIG. 4 is an explanatory diagram of a concept of moire detection according to the embodiment
  • FIG. 4 is an explanatory diagram of a concept
  • the “movement” of a subject in an image means that the intra-frame position of all or part of the subject changes between images at different times.
  • a movement of part or all of a so-called moving subject such as a human, an animal, or a machine, thereby changing the position of all or part of the subject within a frame, is expressed as “movement” in the present disclosure. It is a mode.
  • a change in the position within a frame of a stationary subject such as a landscape or still life due to a change in the imaging direction such as panning or tilting of an imaging device (camera) is also an aspect of "movement”.
  • image does not matter whether it is a still image or a moving image at the stage of recording.
  • the imaging device captures an image of one frame at each time at a predetermined frame rate, and as a result, one frame is recorded as a still image, or a moving image is recorded with continuous frames. Both are assumed.
  • the image processing device is installed as an image processing unit in an imaging device (camera) or an information processing device that performs image editing.
  • the imaging device and the information processing device themselves equipped with these image processing units can also be considered as the image processing device.
  • This imaging device 1 includes an image processing unit 20 that performs moiré detection processing, and the image processing unit 20 or the imaging device 1 including the image processing unit 20 is considered as an example of the image processing device of the present disclosure. can be done.
  • the imaging apparatus 1 includes, for example, a lens system 11, an imaging element section 12, a recording control section 14, a display section 15, a communication section 16, an operation section 17, a camera control section 18, a memory section 19, an image processing section 20, a buffer memory 21, It has a driver section 22 , a sensor section 23 and a connection section 24 .
  • the lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like.
  • the lens system 11 guides the light (incident light) from the object and converges it on the imaging element section 12 .
  • the lens system 11 can be provided with an optical low-pass filter for reducing moire, for example, by using a birefringent plate.
  • an optical low-pass filter for reducing moire
  • the image processing unit 20 detects and reduces moire that cannot be completely removed with an optical low-pass filter. Note that moire detection and reduction by the image processing unit 20 are effective even when no optical low-pass filter is provided.
  • the imaging device unit 12 is configured by having an imaging device (image sensor) 12a such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type.
  • an imaging device image sensor
  • CMOS Complementary Metal Oxide Semiconductor
  • CCD Charge Coupled Device
  • CDS Correlated Double Sampling
  • AGC Automatic Gain Control
  • the image processing unit 20 is configured as an image processing processor such as a DSP (Digital Signal Processor), for example.
  • the image processing unit 20 performs various kinds of signal processing on the digital signal (captured image signal) from the image sensor unit 12, that is, RAW image data.
  • the image processing unit 20 performs lens correction, noise reduction, synchronization processing, YC generation processing, color reproduction/sharpness processing, and the like.
  • synchronization processing color separation processing is performed so that the image data for each pixel has all of the R, G, and B color components.
  • demosaic processing is performed as color separation processing.
  • YC generation process a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data.
  • processing for adjusting gradation, saturation, tone, contrast, etc. is performed as so-called image creation.
  • the image processing unit 20 performs such signal processing, that is, signal processing generally called development processing, to generate image data in a predetermined format.
  • resolution conversion and file formation processing may be performed.
  • compression encoding for recording and communication for example, compression encoding for recording and communication, formatting, generation and addition of metadata are performed on the image data to generate a file for recording and communication.
  • an image file in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), HEIF (High Efficiency Image File Format), YUV422, or YUV420 is generated as a still image file.
  • MP4 format which is used for recording MPEG-4 compliant moving images and audio.
  • An image file of RAW image data that has not undergone development processing may be generated.
  • the image processing section 20 has signal processing functions as a moire detection section 31 and a moire reduction section 32 .
  • the moiré detection unit 31 detects pixel regions in which different movements appear among pixel regions in which the same movement as that of an object is assumed to occur as a change in position within a frame between images at different times. , to generate moiré detection information Sdt (see FIG. 3).
  • the moire reduction unit 32 performs moire reduction processing based on the detection information Sdt. Details of these signal processing functions will be described later. Note that the moire reduction unit 32 may not be provided in the image processing unit 20 in some cases.
  • the buffer memory 21 is formed by, for example, a D-RAM (Dynamic Random Access Memory).
  • the buffer memory 21 is used for temporary storage of image data during the above-described development process in the image processing section 20 .
  • the buffer memory 21 may be a memory chip separate from the image processing unit 20, or may be configured in an internal memory area such as a DSP that configures the image processing unit 20.
  • FIG. 1 A block diagram illustrating an exemplary computing system.
  • the recording control unit 14 performs recording and reproduction on a recording medium such as a non-volatile memory.
  • the recording control unit 14 performs processing for recording image files such as moving image data and still image data on a recording medium, for example.
  • a recording control unit 14 may be configured as a flash memory built in the imaging device 1 and its writing/reading circuit.
  • the recording control unit 14 may be configured by a card recording/reproducing unit that performs recording/reproducing access to a recording medium detachable from the imaging apparatus 1, such as a memory card (portable flash memory, etc.).
  • the recording control unit 14 may be implemented as an HDD (Hard Disk Drive) or the like as a form incorporated in the imaging device 1 .
  • HDD Hard Disk Drive
  • the display unit 15 is a display unit that provides various displays to the user, and is a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display arranged in the housing of the imaging device 1, for example. Due to the display panel and viewfinder.
  • the display unit 15 executes various displays on the display screen based on instructions from the camera control unit 18 .
  • the display unit 15 displays a reproduced image of image data read from the recording medium by the recording control unit 14 .
  • the display unit 15 is supplied with the image data of the picked-up image whose resolution has been converted for display by the image processing unit 20, and the display unit 15 responds to an instruction from the camera control unit 18, based on the image data of the picked-up image. may be displayed.
  • a so-called through image (monitoring image of the subject), which is an image captured while confirming the composition or recording a moving image, is displayed.
  • the display unit 15 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) on the screen based on instructions from the camera control unit 18 .
  • GUI Graphic User Interface
  • the communication unit 16 performs wired or wireless data communication and network communication with external devices. For example, still image files and moving image files including captured image data and metadata are transmitted and output to an external information processing device, display device, recording device, playback device, or the like.
  • the communication unit 16 performs communication via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, etc. on the network. can.
  • the imaging device 1 communicates with, for example, a PC, a smartphone, a tablet terminal, or the like via the communication unit 16, such as Bluetooth (registered trademark), Wi-Fi (registered trademark) communication, NFC (Near field communication), etc.
  • the imaging device 1 and other equipment may be able to communicate with each other through wired connection communication. Therefore, the imaging device 1 can transmit image data and metadata to an information processing device 70 (to be described later) or the like by using the communication unit 16 .
  • the operation unit 17 collectively indicates an input device for a user to perform various operation inputs. Specifically, the operation unit 17 indicates various operators (keys, dials, touch panels, touch pads, etc.) provided on the housing of the imaging device 1 . A user's operation is detected by the operation unit 17 , and a signal corresponding to the input operation is sent to the camera control unit 18 .
  • the camera control unit 18 is configured by a microcomputer (arithmetic processing unit) having a CPU (Central Processing Unit).
  • the memory unit 19 stores information and the like that the camera control unit 18 uses for processing.
  • a ROM Read Only Memory
  • RAM Random Access Memory
  • flash memory and the like are comprehensively illustrated.
  • the memory section 19 may be a memory area built into a microcomputer chip as the camera control section 18, or may be configured by a separate memory chip.
  • the camera control unit 18 controls the entire imaging apparatus 1 by executing programs stored in the ROM of the memory unit 19, flash memory, or the like.
  • the camera control unit 18 controls the shutter speed of the image sensor unit 12, instructs various signal processing in the image processing unit 20, performs image capturing operations and image recording operations according to user operations, reproduces recorded image files, It controls operations of necessary parts such as operations of the lens system 11 such as zoom, focus and aperture adjustment in the lens barrel.
  • the camera control unit 18 also detects operation information from the operation unit 17 and controls the display of the display unit 15 as user interface operations.
  • the camera control unit 18 also controls the communication operation with an external device by the communication unit 16 .
  • the RAM in the memory unit 19 is used as a work area for the CPU of the camera control unit 18 to perform various data processing, and is used for temporary storage of data, programs, and the like.
  • the ROM and flash memory (non-volatile memory) in the memory unit 19 are used for storing an OS (Operating System) for the CPU to control each unit and content files such as image files.
  • the ROM and flash memory in the memory unit 19 are used to store application programs for various operations of the camera control unit 18 and the image processing unit 20, firmware, various setting information, and the like.
  • the driver unit 22 includes, for example, a motor driver for the zoom lens drive motor, a motor driver for the focus lens drive motor, a motor driver for the motor of the aperture mechanism, and the like. These motor drivers apply drive currents to the corresponding drivers in accordance with instructions from the camera control unit 18 to move the focus lens and zoom lens, open and close the diaphragm blades of the diaphragm mechanism, and the like.
  • the sensor unit 23 comprehensively indicates various sensors mounted on the imaging device.
  • an IMU intial measurement unit
  • an acceleration sensor detects acceleration. be able to.
  • a position information sensor, an illuminance sensor, a range sensor, etc. may be mounted.
  • Various information detected by the sensor unit 23, such as position information, distance information, illuminance information, IMU data, etc. is supplied to the camera control unit 18, and together with date and time information managed by the camera control unit 18, metadata is sent to the captured image.
  • the camera control unit 18 can generate metadata for each frame of an image, associate it with the frame of the image, and cause the recording control unit 14 to record it on the recording medium together with the image. Further, the camera control unit 18 can associate metadata generated for each image frame with the image frame, for example, and cause the communication unit 16 to transmit the metadata together with the image data to the external device.
  • connection unit 24 performs communication with a so-called pan-tilter, a tripod, or the like, which performs panning and tilting with the imaging device 1 mounted.
  • the connection unit 24 can input operation information such as the direction and speed of panning and tilting from a pan-tilter or the like, and transmit the operation information to the camera control unit 18 .
  • the information processing device 70 is a device such as a computer device capable of information processing, particularly image processing.
  • the information processing device 70 is assumed to be a personal computer (PC), a mobile terminal device such as a smart phone or a tablet, a mobile phone, a video editing device, a video reproducing device, or the like.
  • the information processing device 70 may be a computer device configured as a server device or an arithmetic device in cloud computing.
  • the information processing device 70 includes an image processing unit 20 that performs moire detection and moire reduction. It can be considered as an example of a device.
  • the CPU 71 of the information processing device 70 executes various programs according to a program stored in a non-volatile memory unit 74 such as a ROM 72 or an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from the storage unit 79 to the RAM 73. process.
  • a non-volatile memory unit 74 such as a ROM 72 or an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from the storage unit 79 to the RAM 73. process.
  • the RAM 73 also appropriately stores data necessary for the CPU 71 to execute various processes.
  • the image processing unit 20 has functions as the moire detection unit 31 and the moire reduction unit 32 described in the imaging device 1 described above.
  • the moiré detector 31 and the moiré reducer 32 as the image processor 20 may be provided as functions within the CPU 71 .
  • the image processing unit 20 may be realized by a CPU, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (artificial intelligence) processor, or the like, which is separate from the CPU 71 .
  • the CPU 71 , ROM 72 , RAM 73 , nonvolatile memory section 74 and image processing section 20 are interconnected via a bus 83 .
  • An input/output interface 75 is also connected to this bus 83 .
  • the input/output interface 75 is connected to an input section 76 including operators and operating devices.
  • various operators and operation devices such as a keyboard, mouse, key, dial, touch panel, touch pad, remote controller, etc. are assumed.
  • a user's operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .
  • a microphone is also envisioned as input 76 .
  • a voice uttered by the user can also be input as operation information.
  • the input/output interface 75 is connected integrally or separately with a display unit 77 such as an LCD or an organic EL panel, and an audio output unit 78 such as a speaker.
  • the display unit 77 is a display unit that performs various displays, and is configured by, for example, a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like.
  • the display unit 77 displays images for various types of image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71 . Further, the display unit 77 displays various operation menus, icons, messages, etc., ie, as a GUI (Graphical User Interface), based on instructions from the CPU 71 .
  • GUI Graphic User Interface
  • the input/output interface 75 may be connected to a storage unit 79 made up of an HDD, a solid-state memory, etc., and a communication unit 80 made up of a modem or the like.
  • the storage unit 79 can store data to be processed and various programs.
  • the storage unit 79 stores image data to be processed, detection information Sdt by moiré detection processing, or an image subjected to moiré reduction processing. It is also assumed that data and the like are stored.
  • the storage unit 79 may also store programs for moire detection processing and moire reduction processing.
  • the communication unit 80 performs communication processing via a transmission line such as the Internet, and communication by wired/wireless communication with various devices, bus communication, and the like.
  • the communication unit 80 performs communication with the imaging device 1, for example, reception of captured image data, metadata, and the like.
  • a drive 81 is also connected to the input/output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately loaded.
  • Data files such as image files and various computer programs can be read from the removable recording medium 82 by the drive 81 .
  • the read data file is stored in the storage unit 79 , and the image and sound contained in the data file are output by the display unit 77 and the sound output unit 78 .
  • Computer programs and the like read from the removable recording medium 82 are installed in the storage unit 79 as required.
  • software for the processing of the present embodiment can be installed via network communication by the communication unit 80 or via the removable recording medium 82.
  • the software may be stored in advance in the ROM 72, the storage unit 79, or the like.
  • Image processing configuration and processing overview> The image processing unit 20 in the imaging device 1 and the information processing device 70 described above will be described.
  • a subject having a frequency component exceeding the Nyquist frequency of the imaging element 12a of the imaging apparatus 1 causes moire as aliasing distortion.
  • Moire is basically prevented by cutting frequencies above the Nyquist frequency using an optical low-pass filter in front of the image sensor 12a. Therefore, as post-processing, a moiré portion (pixel region) is detected from the motion of the subject and the moiré portion is blurred to reduce the moiré.
  • the imaging device 1 when the subject is stationary with respect to the imaging device 1, it is difficult to determine whether it is a moiré pattern or a true pattern, and the image does not become too obtrusive. On the other hand, if there is movement in the subject in the image, the real pattern moves in the same direction and at the same speed. .
  • FIG. 3 shows a configuration example for moire detection and moire reduction in the image processing unit 20 .
  • Image data Din indicates image data to be subjected to moire detection processing, and is image data that is sequentially input for each frame, for example.
  • the image data Din of the frame at each time is input to the memory 30, the moire detection section 31, and the moire reduction section 32, respectively.
  • the image data Din may be RAW image data input to the image processing unit 20 in the case of the imaging device 1, for example.
  • the image data may be image data that has undergone partial or complete development processing.
  • the memory 30 for example, in the case of the imaging device 1, a storage area of the buffer memory 21 inside or outside the image processing unit 20 is used. In the case of the information processing device 70, for example, the storage area of the RAM 73 may be used. Any storage area may be used as the memory 30 here.
  • the moiré detection unit 31 detects pixel regions in which different movements appear among pixel regions in which the same movement as that of an object is assumed to occur as a change in position within a frame between images at different times. , moire detection processing for generating moire detection information. Therefore, the image data Din is input as the current frame image (current image DinC), and the image data Din stored in the memory 30 is read after one frame period and input as the past image DinP. Note that the past image DinP is not necessarily read after one frame period. For example, the past image DinP may be read after two frame periods or after several frame periods.
  • the moire detection unit 31 only needs to be able to compare the current image DinC with the past image DinP, and the time difference between the current image DinC and the past image DinP used for the comparison is set as an appropriate time for moire detection. good.
  • the moiré detection unit 31 uses the current image DinC and the past image DinP to perform moiré detection processing as shown in FIG. 4 each time image data of a frame as the current image DinC is input.
  • the moire detector 31 detects motion in the image. For example, the movement of an object within an image is detected. Alternatively, there is a case of detecting a substantially uniform movement of the entire subject in the image.
  • step S102 the moiré detection unit 31 detects pixel areas that are moving differently from the pixel areas that are assumed to have the same movement as the movement detected in step S101.
  • the “different motions” refer to, for example, motions with different directions of motion (directions of changes in position within frames) and speeds (displacement amounts between frames).
  • a region in which the same movement as that of the subject is assumed is an area within the outline of the moving subject.
  • motion may be detected by panning or the like of the imaging device 1 itself. It becomes a pixel area in which the same movement is assumed.
  • the moire detection unit 31 performs a process of comparing the current image DinC with the previous image DinP and detecting a portion where a different movement appears in a pixel area where the same movement as that of the subject is assumed.
  • the moiré detection unit 31 generates moiré detection information Sdt based on the detection results of different motions.
  • the detection information Sdt may include moire presence/absence information indicating whether moire is present in the current image DinC.
  • the detection information Sdt may include area information indicating a pixel region where moiré occurs in the current image DinC.
  • the detection information Sdt may include both moiré presence/absence information and area information.
  • the detection information Sdt generated by the moire detection unit 31 in FIG. 3 through the above processing is supplied to the moire reduction unit 32 .
  • Image data Din is input to the moire reduction unit 32 as a target for moire reduction processing.
  • the moire reduction unit 32 performs moire reduction processing, specifically, for example, LPF (low-pass filter) processing, on the image data Din based on the detection information Sdt.
  • the detection information Sdt is the moiré detection result for the frame that the moiré detection unit 31 has determined as the current image DinC. will be done.
  • the moire reduction unit 32 performs moire reduction processing, for example, as shown in FIG. 5, on each frame of the image data Din.
  • the moiré reduction unit 32 acquires the detection information Sdt corresponding to the frame of the image data Din to be processed at the present time.
  • the moire reduction unit 32 refers to the detection information Sdt to determine whether the current frame to be processed is an image in which moire occurs. If the detection information Sdt contains moiré presence/absence information, it can be determined based on this information. Even if the detection information Sdt is only area information, occurrence of moire can be determined by whether or not the corresponding portion is indicated as the area information.
  • the moire reduction unit 32 proceeds to step S203, and reduces moire by performing LPF processing on the image data Din of the frame to be processed.
  • LPF processing is used, BPF (band pass filter) processing for filtering a specific frequency band may also be used. If it is determined that moire has not occurred, the moire reduction unit 32 ends the processing of FIG. 5 for the image data Din of the processing target frame without performing step S203.
  • Image data Dout in which moire is reduced (including elimination) is obtained by being processed by the moire reduction unit 32 as described above. For example, by performing development processing on this image data Dout, an image with reduced moire can be displayed.
  • the image processing unit 20 a configuration as shown in FIG. 6 is also conceivable. That is, this is a configuration example in which the moire reduction unit 32 is not provided.
  • the moire detector 31 generates the detection information Sdt as described above.
  • the camera control unit 18 of the imaging device 1 and the CPU 71 in the information processing device 70 use this detection information Sdt as metadata associated with the frame of the image data Din.
  • the camera control unit 18 can cause the recording control unit 14 to record metadata on the recording medium in association with each frame of the image data Dout.
  • the camera control unit 18 can cause the image data Dout and metadata associated with each frame thereof to be transmitted from the communication unit 16 to an external device.
  • each frame of the image data is associated with the moiré detection information Sdt.
  • the device for example, the information processing device 70 that receives the image file containing the image data and metadata, can perform the moire reduction processing as shown in FIG.
  • the detection information acquired in step S201 is read from the metadata corresponding to the frame targeted for moire reduction processing.
  • the CPU 71 of the information processing device 70 can also use the detection information Sdt as metadata associated with the frame of the image data Din, similarly to the camera control unit 18 described above.
  • the metadata associated with each frame of the image data Dout may be recorded on the recording medium in the storage unit 79 or the like, or the image data Dout and the metadata associated with each frame may be transmitted from the communication unit 80 to the external device. can be sent.
  • Moire detection processing example> Specific moire detection processing examples (first example, second example, and third example) by the moire detection unit 31 will be described below. Each example is an example of processing performed by the moiré detection unit 31 that inputs the current image DinC and the past image DinP as shown in FIG.
  • FIG. A first example is an example in which object recognition processing is performed on a subject in an image, and a pixel region where the movement of each object does not match the movement inside the object is determined as moire.
  • FIG. 7 is a flow chart showing a first example of the moire detector 31. As shown in FIG.
  • step S110 the moiré detection unit 31 performs object recognition processing on the image, and sets the target subject based on the recognition result.
  • the moiré detection unit 31 performs object recognition processing such as a person, an animal, and an object by semantic segmentation processing, pattern recognition processing, or the like, for example, on the subject in the current image DinC.
  • object recognition processing such as a person, an animal, and an object by semantic segmentation processing, pattern recognition processing, or the like, for example, on the subject in the current image DinC.
  • object A the subjects recognized as objects of some kind will be referred to as object A, object B, object C, and the like.
  • an object to be subjected to motion detection is specified and set as a target subject.
  • the moire detection unit 31 can consider an object that is estimated to be the same individual as the object recognized in the object recognition processing for the previous image DinP as a target subject for motion detection.
  • the object in the past image DinP can be determined, for example, based on the object recognition result when the moiré detection unit 31 treated the frame as the current image DinC in the past.
  • step S110 one or more objects are set as the target subject based on the object recognition processing for the image.
  • the moire detector 31 proceeds from step S111 to step S114.
  • the moiré detection unit 31 proceeds from step S111 to step S112 to detect the motion of each of the one or more target subjects. That is, for each target subject, the in-frame position in the past image DinP is compared with the in-frame position in the current image DinC to detect movement. For example, object A, which is the target subject, is moving leftward on the screen at a speed of "1", and object B is moving upward on the screen at a speed of "3". to detect Note that there may be cases where motion is not detected for a certain target subject.
  • the moiré detection unit 31 detects pixel areas of the object whose movement is detected among the objects set as the respective target subjects, in which the movement is different from that of the object. For example, in the pixel area of the object A as the target subject, that is, in each pixel corresponding to the outline of the subject as the object A, a portion where the motion different from the motion detected for the object A occurs is detected. Specifically, when it is detected that the object A is "moving in the left direction of the screen at a speed of "1"", among the pixels in the pixel area recognized as the object A, Detect pixels that are not detected to be moving at speed "1". Such one or more pixel regions are defined as pixel regions exhibiting different motions.
  • step S114 the moiré detection unit 31 generates detection information Sdt based on detection of pixel regions exhibiting different motions. For example, when pixel regions exhibiting different motions are detected, moire presence/absence information indicating "with moire" is generated as the detection information Sdt. If no pixel area showing different motion is detected, moiré presence/absence information indicating "no moiré" is generated as the detection information Sdt. Alternatively, as the detection information Sdt, area information, which is information specifying pixel regions exhibiting different motions, is generated. If there is no pixel area showing different motion, area information indicating that the corresponding area does not exist is generated. If the process proceeds from step S111 to step S114, moiré presence/absence information indicating "no moiré" is generated as the detection information Sdt, or area information in which the corresponding area does not exist is generated.
  • FIG. Assume that an object 50 exists as a subject in the past image DinP and the current image DinC in FIG. Suppose that a pattern 51 of vertical stripes (indicated by hatched and non-hatched stripes in the figure) is seen on the image of this object 50 . A motion on the image is detected by comparing the past image DinP and the current image DinC. That is, between frames at different times, the object 50 is detected to move to the left at a certain speed.
  • the pattern 51 is determined to be a pattern actually attached to the object 50 rather than a moiré pattern.
  • the pattern 52 is determined to be moire.
  • FIG. 10 shows an object 55 as a subject, and it is assumed that the motion of this object 55 is not detected by comparing the past image DinP and the current image DinC. However, it is assumed that some movement is detected with respect to the pattern 53 inside the object 50 . In this case, there is a high possibility that the pattern 53 is actually moving.
  • step S113 it is possible to reduce erroneous detection of moire by detecting pixel regions in which movement is different within only the target subject whose movement has been detected.
  • a second example of moire detection processing will be described with reference to FIG.
  • the second example is an example of performing moiré detection using the result of overall motion detection without object recognition when uniform motion information is obtained in advance that the subject in the image moves uniformly. .
  • step S121 of FIG. 11 the moiré detection unit 31 checks whether or not there is prior information that all subjects move uniformly, that is, whether or not there is uniform motion information, and branches the process.
  • the prior information as uniform motion information may be, for example, the setting of the shooting mode by the user, or may be information about performing panning or tilting.
  • the information when processing image data Din captured in the past, the information may be information indicating that the shooting mode, panning, or the like was performed when the image data Din was captured.
  • the target is a subject that does not move.
  • the subject in the screen changes according to the movement of the imaging device 1, and is expected to move uniformly.
  • the user will perform panning, such as when the panorama shooting mode is set, the subject is expected to move uniformly.
  • Information to the effect that a panning operation or a tilting operation will be performed by a pan-tilter or the like to which the imaging device 1 is attached can also be considered as one type of advance information indicating that a motionless subject is moving within an image.
  • the moiré detection unit 31 proceeds to other processing from step S121.
  • the processing of the first example in FIG. 7 may be performed.
  • the moiré detection process is not executed.
  • the moiré detection unit 31 proceeds to step S122 and first sets feature points in the image. For example, in the current image DinC, one point or a plurality of points such as a point where a clear edge is detected or a point showing a characteristic shape is selected and set as a feature point.
  • step S123 the moiré detection unit 31 compares the intra-frame positions of feature points in the past image DinP and the current image DinC to detect uniform movement (direction and speed) of the subject in the image.
  • step S124 the moiré detection unit 31 compares the past image DinP and the current image DinC to detect pixels exhibiting a movement different from the above uniform movement.
  • the pixel region in which the same motion as the subject is assumed is the entire frame. Therefore, among the pixels of the entire frame, the pixels exhibiting motion different from the uniform motion are determined, and the region of such pixels is detected. In other words, a portion in which movement in a direction or speed different from that of the entire movement is locally detected.
  • step S125 the moire detection unit 31 generates detection information Sdt (moire presence/absence information, area information) based on the detection of pixel regions exhibiting movements different from uniform movements.
  • detection information Sdt oire presence/absence information, area information
  • a third example of moire detection processing will be described with reference to FIG.
  • the subject itself does not move and information on the movement of the tripod or pan tilter is obtained, or information on the movement of the imaging apparatus 1 itself is obtained as IMU data of the sensor unit 23, etc.
  • step S131 of FIG. 12 the moire detection unit 31 checks whether or not uniform motion information indicating that all subjects move uniformly is obtained as prior information, and branches the process.
  • the prior information as the uniform motion information is, for example, the setting of the shooting mode by the user. information, panorama shooting mode information, panning information using a pan tilter or the like, and the like.
  • the imaging apparatus 1 is mounted on a pan-tilter or the like so that the direction and speed of the panning or tilting movement can be detected, or the imaging apparatus 1 itself can be detected as IMU data from the sensor unit 23 . It is premised on being able to detect the direction and speed of movement of the robot.
  • the moiré detection unit 31 proceeds to other processing from step S131.
  • the processing of the first example in FIG. 7 may be performed, or the moire detection processing may not be performed.
  • the moiré detector 31 proceeds to step S132 to acquire motion information.
  • motion information For example, information on the pan-tilter imaging direction corresponding to each time point of the frame of the past image DinP and the frame of the current image DinC, IMU data corresponding to each frame, and the like are acquired. Uniform motion (direction and speed) of the entire subject can be detected from this information.
  • step S133 the moiré detection unit 31 compares the past image DinP and the current image DinC to detect pixels exhibiting a movement different from the uniform movement described above.
  • each subject moves in the same manner as the movement of the pan-tilter or the imaging device 1, so the pixel area in which the same motion as the subject is assumed is the entire frame. Therefore, among the pixels of the entire frame, the pixels exhibiting motion different from the uniform motion are determined, and the region of such pixels is detected. In other words, a portion where movement in a direction different from the movement of the imaging apparatus 1 such as panning or movement at a different speed can be seen locally is detected.
  • step S134 the moire detection unit 31 generates detection information Sdt (moire presence/absence information, area information) based on the detection of pixel regions exhibiting motions different from uniform motions.
  • detection information Sdt oire presence/absence information, area information
  • the movement of the pan-tilter or the imaging device 1 can be regarded as the uniform movement of the subject, and different movement portions can be detected as moire.
  • moiré is detected when there are pixel regions with different movements.
  • the thresholds for the direction difference and speed difference that determine "different movements" are set to values that do not detect minute differences, or are variable depending on the situation. For example, it is conceivable to change the threshold depending on the type of the recognized subject, or to change the threshold depending on the speed of movement of the imaging device 1 or the like.
  • FIG. A first example of moire reduction processing is shown in FIG. This is an example in which moiré presence/absence information is input as the detection information Sdt.
  • step S211 the moire reduction unit 32 acquires moire presence/absence information as the detection information Sdt.
  • step S212 the moiré presence/absence information confirms whether or not moiré occurs in the current processing target frame of the image data Din based on the moiré presence/absence information. If no moire occurs, the moire reduction process ends without doing anything for the current frame to be processed.
  • the moire reduction unit 32 proceeds to step S213 and performs LPF processing on the entire image data Din currently being processed. This makes it possible to obtain image data Dout in which moire is less noticeable.
  • FIG. This is an example in which area information is input as detection information Sdt.
  • step S221 the moire reduction unit 32 acquires area information indicating a pixel region in which moire is detected as the detection information Sdt.
  • step S222 the moiré presence/absence information confirms whether or not moiré occurs in the current processing target frame of the image data Din based on the area information. That is, it is checked whether or not one or more pixel regions are indicated in the area information. If no moire occurs, the moire reduction process ends without doing anything for the current frame to be processed.
  • the moire reduction unit 32 proceeds to step S223 and performs LPF processing on the pixel region indicated by the area information. Accordingly, it is possible to obtain the image data Dout in which the moire is reduced in the portion where the moire occurs.
  • FIG. This is also an example in which area information is input as the detection information Sdt, and it is an example in which a smoothing process is performed to smoothly change the resolution of an image at the boundary between a portion subjected to LPF processing and a portion not subjected to LPF processing.
  • step S231 the moire reduction unit 32 acquires area information indicating a pixel region in which moire is detected as the detection information Sdt.
  • step S222 the moiré presence/absence information confirms whether or not moiré occurs in the current processing target frame of the image data Din based on the area information. That is, it is checked whether or not one or more pixel regions are indicated in the area information. If no moire occurs, the moire reduction process ends without doing anything for the current frame to be processed.
  • the moire reduction unit 32 proceeds to step S223 and generates an LPF-processed image by performing LPF processing on the entire frame.
  • step S234 the moire reduction unit 32 sets the blend ratio of each pixel based on the area information.
  • the blend ratio is a mixing ratio of pixel values of an LPF-processed image and its original image (image not subjected to LPF processing).
  • the blend ratio of each pixel is set as follows. It is assumed that the shaded area AR1 in FIG. 16 is the area indicated by the area information that moire occurs. Areas AR2, AR3, and AR4 are set so as to surround the perimeter of this area AR1, and the rest is defined as area AR5.
  • the blend ratio between the LPF-processed image and the original image is set as follows. ⁇ AR1 ... 100:0 ⁇ AR2 ... 75: 25 ⁇ AR3...50:50 ⁇ AR4 ⁇ 25:75 ⁇ AR5 ... 0: 100
  • step S235 of FIG. 15 the moire reduction unit 32 synthesizes the LPF-processed image and the original image at the above blend ratios in the areas AR1 to AR5.
  • the pixels of the area AR1 are applied to the pixels of the LPF-processed image.
  • the pixel value of each pixel in the area AR1 is such that the pixel values of the LPF-processed image and the original image are synthesized at a ratio of 75:25.
  • Areas AR3 and AR4 are also synthesized at the above blend ratio.
  • the pixels of the original image are applied to the area AR5.
  • the image data resulting from the synthesis in this manner is image data Dout subjected to moire reduction.
  • the moire reduction processing in the first, second, and third examples described above is an example of performing LPF processing
  • adjusting the frequency characteristics for example, by increasing the cutoff frequency
  • the user may be allowed to adjust the cutoff frequency of the LPF processing.
  • the user can check the image after moire reduction processing while performing an operation to change the cutoff frequency, and adjust the resolution of the image and the state of moire reduction to the desired state. can be considered.
  • moiré that is reflected in high frequencies is detected and reduced by this method
  • moiré that is reflected in low frequencies is detected by other methods, such as detecting the difference between two images with different optical characteristics as moiré. It is also possible to reduce that portion by LPF processing.
  • the image processing unit 20 selects pixel regions having different motions among pixel regions in which the same motion as that of a subject is assumed to occur as a change in position within a frame between images at different times.
  • a moire detection unit 31 is provided for detecting and generating moire detection information Sdt.
  • the movement of the subject in the image that is, the change in the position of the subject within the frame between the images at different times includes the change due to the movement of the subject itself and the change due to the movement of the imaging device 1 such as panning.
  • the pixel area within the contour of the subject is a pixel area that is assumed to move the same as the subject.
  • the entire pixel area in the frame is a pixel area in which the same motion as the subject is assumed. Therefore, when an object is moving, if a pixel area in which the same movement as the object should occur shows a different movement, it is detected as moire rather than the original pattern of the object. can. By detecting moire from the state of motion, that is, from the state of change in position within a frame between images at different times, moire can be detected by distinguishing it from a real pattern, regardless of the frequency of moire.
  • the image processing unit 20 of the embodiment further includes a moire reduction unit 32 that performs moire reduction processing based on the detection information Sdt.
  • Moire reduction is performed by, for example, LPF processing based on detection information obtained by detecting moire from the motion state. As a result, regardless of the moire frequency, the moire can be reduced while being distinguished from the true pattern.
  • the moiré detection unit 31 detects the motion of the target subject as the target of detection processing based on the object recognition result in the image, and detects the motion different from the motion of the target subject in the pixel area of the target subject.
  • An example of detecting a pixel area and generating detection information Sdt has been given (first example of moire detection processing: see FIG. 7).
  • the moiré detection unit 31 detects the motion of feature points in the image to which uniform motion information indicating that the entire subject in the image moves uniformly, and detects the motion of the feature points.
  • An example of generating the detection information Sdt by detecting a pixel area having a motion different from the motion of moire (second example of moiré detection processing: see FIG. 11). For example, if the shooting mode selected by the user or information indicating that a panning operation or a tilting operation will be performed by a pan tilter to which the imaging device 1 is attached is provided as advance information, the moiré detection unit 31 detects It can be understood that a uniform movement appears for the entire subject.
  • the moiré detection unit 31 detects an image to which uniform motion information indicating that the entire subject in the image moves uniformly is different from the motion information indicating the motion of the imaging device at the time of imaging.
  • An example of detecting a moving pixel area and generating detection information Sdt has been given (third example of moire detection processing: see FIG. 12).
  • the movement of the imaging device 1 at the time of imaging is indicated by inputting information on the direction and speed of motion from a pan-tilter or the like, or by IMU data from the sensor unit 23, or the like.
  • the moire detection unit 31 can grasp that a uniform movement appears in the entire subject in the image, it should detect a movement that matches the movement information indicating the movement of the imaging device 1 for the entire subject. is. In this case, when a pixel area showing motion different from the motion information is detected, it can be determined as moire.
  • a process of detecting a pixel area whose movement differs from that of the target subject based on object recognition as shown in FIG. It is also conceivable to selectively use the process of detecting an area and the process of acquiring information on the movement of the entire subject and detecting pixel areas with different movements as shown in FIG.
  • the detection information Sdt may include moiré presence/absence information.
  • the moiré reduction unit 32 can perform moiré reduction processing only for frames in which moiré is detected. By not applying the moiré reduction process even to images in which moiré does not occur, it is possible to prevent the sense of resolution of the image from being unnecessarily impaired.
  • the detection information Sdt may include area information indicating a pixel region in which moire is detected.
  • the moiré reduction unit 32 can perform moiré reduction processing only on the pixel region where the moiré is detected. As a result, it is possible to prevent the moire reduction process from being performed on the image area where no moire occurs, and it is possible to reduce only the moire without impairing the resolution of the image.
  • FIG. 4 an example having a control unit such as the camera control unit 18 of the imaging device 1 and the CPU 71 of the information processing device 70 that associates the detection information Sdt with the image as metadata corresponding to the image has been described (FIGS. 1 and 2). , see FIG. 4).
  • the camera control unit 18 of the imaging device 1 uses the detection information Sdt detected for each frame by the moiré detection unit 31 as metadata associated with the frame of the image, and records it on a recording medium or transmits it to an external device. Therefore, even devices other than the imaging apparatus 1 can perform moire reduction processing using the detection information Sdt. This makes it possible to effectively use the moire detection result based on the motion comparison. Even if such processing is performed by the CPU 71 of the information processing device 70, moire reduction processing using the detection information Sdt can be performed in subsequent processing of the information processing device 70 or processing of other devices. become.
  • the moiré reduction unit 32 of the embodiment performs the moiré reduction processing by the LPF processing on the image (the first example (FIG. 13), the second example (FIG. 14), and the third example (FIG. 14) of the moiré reduction processing. 15)).
  • the moire reduction unit 32 is formed by an LPF, and performs LPF processing on the current image (image data Din) based on the detection information Sdt as shown in FIG. 3, thereby performing moire reduction when necessary. It will be.
  • the moire reduction unit 32 performs moire reduction processing by LPF processing on the pixel region indicated by the area information based on the detection information Sdt including the area information indicating the pixel region where the moire is detected.
  • LPF processing is performed only on the pixel region indicated by the area information.
  • the moire reduction unit 32 can perform the LPF process only on the pixel region where the moire occurs.
  • the moire reduction unit 32 performs moire reduction processing by LPF processing on the pixel region indicated by the area information based on the detection information Sdt including the area information indicating the pixel region where the moire is detected.
  • An example of performing a smoothing process that gradually changes the degree of reflection of the LPF process on the area around the pixel area indicated by the information has been given (third example of moire reduction process, see FIGS. 15 and 16).
  • area information is supplied as detection information Sdt
  • LPF processing is performed on the pixel region indicated by the area information, and when LPF processing is not performed on other regions, the smoothness of the image is lost at the boundary of the pixel region. can be. Therefore, the smoothing process as described with reference to FIGS. 15 and 16 is performed. As a result, it is possible to prevent the pixel area in which moire is detected from appearing unnatural.
  • the moire reduction unit 32 performs moire reduction processing by LPF processing on the entire image based on the detection information Sdt including the moire presence/absence information (first example of moire reduction processing, see FIG. 13). ).
  • the moire reduction unit 32 can perform moire reduction by performing LPF processing on the entire image in which moire occurs. In other words, it is possible not to perform LPF processing on an image in which moire does not occur.
  • the moire reduction unit 32 may switch the LPF process according to the content of the detection information Sdt. For example, if the detection information Sdt contains only moire presence/absence information, the entire image is subjected to LPF processing, and if the detection information Sdt contains area information, the pixel region indicated by the area information is subjected to LPF processing. It is conceivable to perform processing.
  • the moire reduction unit 32 performs moire reduction processing by LPF processing on an image and variably sets the cutoff frequency of the LPF processing. For example, by changing the cutoff frequency of the LPF process according to the user's operation or the situation, it is possible to perform the moire reduction process according to the user's idea or situation. For example, when the subject is stationary or the imaging device 1 is moving, the cutoff frequency may be lowered.
  • an arithmetic processing unit such as a CPU, DSP, GPU, GPGPU, AI processor, or a device including these. It is a program that allows That is, the program of the embodiment detects a pixel region having a different motion from among the pixel regions that are assumed to have the same motion as the subject that is moving as a change in the position within the frame between images at different times. is a program for causing an arithmetic processing unit to execute processing for generating moiré detection information Sdt. With such a program, the image processing device referred to in the present disclosure can be realized by various computer devices.
  • This program may also be a program that causes a CPU, DSP, GPU, GPGPU, AI processor, etc., or a device including these to execute the moire reduction processing shown in FIGS. 5, 13, 14, and 15.
  • programs can be recorded in advance in a HDD as a recording medium built in equipment such as a computer device, or in a ROM or the like in a microcomputer having a CPU.
  • the program may be a flexible disc, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disc, DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), magnetic disc, semiconductor It can be temporarily or permanently stored (recorded) in a removable recording medium such as a memory or memory card.
  • Such removable recording media can be provided as so-called package software.
  • it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
  • LAN Local Area Network
  • Such a program is suitable for widely providing the image processing apparatus of the present disclosure.
  • a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), etc.
  • these devices function as the image processing device of the present disclosure. be able to.
  • Moire detection information is generated by detecting pixel areas with different movements among pixel areas that are assumed to have the same movement as the subject that moves as changes in the position within the frame between images taken at different times.
  • An image processing device equipped with a moire detection unit for (2) The image processing apparatus according to (1) above, further comprising a moiré reduction unit that performs moiré reduction processing based on the detection information.
  • the moire detection unit is Detecting the movement of the object subject to detection processing based on the object recognition result in the image, The image processing apparatus according to (1) or (2) above, wherein a pixel area in which a movement different from that of the target subject is detected from among the pixel areas of the target subject, and the detection information is generated.
  • the moire detection unit is For an image given uniform motion information indicating that the entire subject in the image moves uniformly, detecting a pixel region in which the motion differs from that of the feature point, and generating the detection information.
  • the image processing device according to any one of 1) to (3).
  • the moire detection unit is For an image given uniform motion information indicating that the entire subject in the image moves uniformly, detecting a pixel region in which a motion different from the motion information indicating the motion of the imaging device at the time of imaging appears, The image processing apparatus according to any one of (1) to (4) above, which generates the detection information.
  • the image processing apparatus according to any one of (1) to (5) above, wherein the detection information includes moire presence/absence information.
  • the image processing device according to any one of (1) to (6) above, wherein the detection information includes area information indicating a pixel region in which moire is detected.
  • the image processing apparatus according to any one of (1) to (7) above, further comprising a control unit that associates the detection information with the image as metadata corresponding to the image.
  • the moire reduction unit is The image processing device according to (2) above, wherein the moire reduction process is performed by low-pass filtering the image.
  • the moire reduction unit is Based on the detection information including area information indicating the pixel area where the moire is detected, The image processing device according to (2) or (9) above, wherein the moire reduction process is performed by low-pass filter processing on the pixel area indicated by the area information.
  • the moire reduction unit is Based on the detection information including area information indicating the pixel area where the moire is detected, By performing the moire reduction process by low-pass filtering on the pixel area indicated by the area information, The image processing apparatus according to any one of (2), (9), and (10) above, wherein a smoothing process is performed to gradually change the degree of reflection of the low-pass filter process on the area around the pixel area indicated by the area information.
  • the moire reduction unit is Based on the detection information including moire presence/absence information, The image processing device according to any one of (2), (9), (10), and (11) above, wherein the moire reduction processing is performed by low-pass filtering on the entire image.
  • the moire reduction unit is While performing the moiré reduction processing by low-pass filtering the image,
  • the image processing apparatus according to any one of (2), (9), (10), (11), and (12) above, wherein a cutoff frequency for low-pass filtering is variably set.
  • Moire detection information is generated by detecting pixel areas with different movements among pixel areas that are assumed to have the same movement as the subject that moves as changes in the position within the frame between images taken at different times.
  • Image processing method is generated by detecting pixel areas with different movements among pixel areas that are assumed to have the same movement as the subject that moves as changes in the position within the frame between images taken at different times.
  • Imaging device 11 Lens system 12 Imaging element unit 12a Imaging element 18 Camera control unit 20 Image processing unit 21 Buffer memory 30 Memory 31 Moire detection unit 32 Moire reduction unit 70 Information processing device 71 CPU

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Quality & Reliability (AREA)
  • Studio Devices (AREA)
PCT/JP2022/006239 2021-05-17 2022-02-16 画像処理装置、画像処理方法、プログラム Ceased WO2022244351A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US18/556,954 US20240221139A1 (en) 2021-05-17 2022-02-16 Image processing apparatus, image processing method, and program
JP2023522232A JP7761045B2 (ja) 2021-05-17 2022-02-16 画像処理装置、画像処理方法、プログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-082995 2021-05-17
JP2021082995 2021-05-17

Publications (1)

Publication Number Publication Date
WO2022244351A1 true WO2022244351A1 (ja) 2022-11-24

Family

ID=84140211

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/006239 Ceased WO2022244351A1 (ja) 2021-05-17 2022-02-16 画像処理装置、画像処理方法、プログラム

Country Status (3)

Country Link
US (1) US20240221139A1 (https=)
JP (1) JP7761045B2 (https=)
WO (1) WO2022244351A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2025505906A (ja) * 2023-01-19 2025-03-05 北京小米移動軟件有限公司 撮像装置および撮像制御プログラム
WO2026053712A1 (ja) * 2024-09-06 2026-03-12 ソニーグループ株式会社 画像処理装置、画像処理方法、プログラム

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021246171A1 (ja) * 2020-06-04 2021-12-09 ソニーグループ株式会社 撮像装置、および画像処理方法、並びにプログラム
JP2025104764A (ja) * 2023-12-28 2025-07-10 キヤノン株式会社 処理装置、撮影システム、処理方法、及びプログラム

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011087269A (ja) * 2009-09-18 2011-04-28 Sony Corp 画像処理装置、撮像装置、および画像処理方法、並びにプログラム
JP2012129747A (ja) * 2010-12-14 2012-07-05 Canon Inc 画像投影装置及びその制御方法、プログラム
WO2016157299A1 (ja) * 2015-03-27 2016-10-06 三菱電機株式会社 撮像装置及び方法、操作装置及び方法、並びにプログラム及び記録媒体

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7164807B2 (en) * 2003-04-24 2007-01-16 Eastman Kodak Company Method and system for automatically reducing aliasing artifacts
US7447337B2 (en) * 2004-10-25 2008-11-04 Hewlett-Packard Development Company, L.P. Video content understanding through real time video motion analysis
US20090086024A1 (en) * 2007-10-02 2009-04-02 Sam Systems, Inc. System and method for improving video compression efficiency
JP2012010170A (ja) * 2010-06-25 2012-01-12 Nec Casio Mobile Communications Ltd 撮像装置、画像処理装置、ならびに、画像処理方法
WO2016040833A1 (en) * 2014-09-12 2016-03-17 Kiswe Mobile Inc. Methods and apparatus for content interaction
JP6269425B2 (ja) * 2014-10-02 2018-01-31 ソニー株式会社 情報処理装置および情報処理方法
CN109558794B (zh) * 2018-10-17 2024-06-28 平安科技（深圳）有限公司 基于摩尔纹的图像识别方法、装置、设备和存储介质
US11935185B2 (en) * 2020-03-18 2024-03-19 Intel Corporation Content based anti-aliasing for image downscale
JP2022030812A (ja) * 2020-08-07 2022-02-18 キヤノン株式会社 画像処理装置及び撮像装置及びそれらの制御方法及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011087269A (ja) * 2009-09-18 2011-04-28 Sony Corp 画像処理装置、撮像装置、および画像処理方法、並びにプログラム
JP2012129747A (ja) * 2010-12-14 2012-07-05 Canon Inc 画像投影装置及びその制御方法、プログラム
WO2016157299A1 (ja) * 2015-03-27 2016-10-06 三菱電機株式会社 撮像装置及び方法、操作装置及び方法、並びにプログラム及び記録媒体

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2025505906A (ja) * 2023-01-19 2025-03-05 北京小米移動軟件有限公司 撮像装置および撮像制御プログラム
JP7738647B2 (ja) 2023-01-19 2025-09-12 北京小米移動軟件有限公司 撮像装置および撮像制御プログラム
WO2026053712A1 (ja) * 2024-09-06 2026-03-12 ソニーグループ株式会社 画像処理装置、画像処理方法、プログラム

Also Published As

Publication number Publication date
JPWO2022244351A1 (https=) 2022-11-24
US20240221139A1 (en) 2024-07-04
JP7761045B2 (ja) 2025-10-28

Similar Documents

Publication Publication Date Title
JP7761045B2 (ja) 画像処理装置、画像処理方法、プログラム
JP7491297B2 (ja) 情報処理装置、情報処理方法、プログラム
JP7444162B2 (ja) 画像処理装置、画像処理方法、プログラム
JP6056702B2 (ja) 画像処理装置、画像処理方法、プログラム
CN113810604B (zh) 文档拍摄方法、电子设备和存储介质
CN115526787B (zh) 视频处理方法和装置
JP7405131B2 (ja) 画像処理装置、画像処理方法、プログラム
WO2017076000A1 (zh) 夜景拍照方法、装置及移动终端
JP2009159559A (ja) 撮影装置及びそのプログラム
CN115170554A (zh) 图像检测方法和电子设备
JP7586081B2 (ja) 画像処理装置、画像処理方法、プログラム
JP2010147637A (ja) 動画再生装置、動画再生方法、および、動画再生プログラム
JP4639965B2 (ja) 撮像装置、画像処理方法及びプログラム
JP7513081B2 (ja) 画像処理装置、画像処理方法、プログラム
JP2007324856A (ja) 撮像装置、撮像制御方法
JP4909063B2 (ja) 撮像装置及び画像記録方法
US20230109911A1 (en) Image processing apparatus, image processing method, and program
JP5332668B2 (ja) 撮像装置および被写体検出プログラム
US12493929B2 (en) Image processing apparatus, image processing method, and program
WO2022219985A1 (ja) 情報処理方法、情報処理装置、プログラム
JP2008271181A (ja) 撮像装置および撮像方法、再生装置および再生方法、撮像画像処理システム
JP7732458B2 (ja) 撮像装置、撮像方法
JP5223974B2 (ja) ビデオカメラ及びその制御方法
WO2023276249A1 (ja) 画像処理装置、撮像装置、画像処理方法
WO2026053712A1 (ja) 画像処理装置、画像処理方法、プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22804273

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023522232

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18556954

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22804273

Country of ref document: EP

Kind code of ref document: A1