WO2023163217A1 - 処理装置、処理方法、及び処理プログラム - Google Patents

処理装置、処理方法、及び処理プログラム Download PDF

Info

Publication number
WO2023163217A1
WO2023163217A1 PCT/JP2023/007339 JP2023007339W WO2023163217A1 WO 2023163217 A1 WO2023163217 A1 WO 2023163217A1 JP 2023007339 W JP2023007339 W JP 2023007339W WO 2023163217 A1 WO2023163217 A1 WO 2023163217A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
processor
subject
composition
width
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/007339
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智大 島田
哲郎 芦田
雄大 阿部
森人 二関
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to CN202380023843.0A priority Critical patent/CN118786681A/zh
Priority to JP2024503315A priority patent/JPWO2023163217A1/ja
Publication of WO2023163217A1 publication Critical patent/WO2023163217A1/ja
Priority to US18/772,281 priority patent/US20240373138A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping

Definitions

  • the present invention relates to a processing device, a processing method, and a processing program.
  • an imaging device that outputs a signal corresponding to an optical image projected on itself by imaging, an image moving means that moves the optical image on the imaging device, and an output signal based on the output signal of the imaging device.
  • face detection means for detecting the face of a person as a subject from the determination image and detecting the position and orientation of the face on the determination image; and moving the image based on the detected position and orientation of the face.
  • composition control means for controlling the means and generating a composition-adjusted image from the output signal of the image pickup device after the control.
  • Patent Document 2 a display means for displaying image data, an imaging means having a plurality of light receiving elements for imaging a subject and generating image data, an extracting means for extracting a characteristic part of the subject from the image data, and the above setting means for setting a partial predetermined angle of view range within a screen of the entire angle of view imaged by the imaging means displayed by the display means according to the extraction result of the extraction means;
  • a digital camera is described which is characterized by comprising output means for outputting only the image data within the predetermined angle of view range to the outside.
  • an image capturing unit that captures an image, an image processing unit that detects a specific portion of a target subject from a reference image captured by the image capturing unit, and an image captured by the image capturing unit include the above a first control unit for controlling a specific portion of the target subject to have a predetermined size; and a second control unit for controlling the imaging apparatus.
  • a processing device is a processing device comprising a processor and a memory, wherein the processor acquires first image data including a subject, and based on first information about the state of the subject, performs a first determining a first composition of the subject in a direction; and determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information regarding the state of the subject. , to generate the second image data based on the first composition and the second composition.
  • a processing method acquires first image data including a subject, determines a first composition of the subject in a first direction based on first information about the state of the subject, determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information about the state; determining a second composition based on the first composition and the second composition; It generates image data.
  • a processing program acquires first image data including a subject, determines a first composition of the subject in a first direction based on first information about the state of the subject, determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information about the state; determining a second composition based on the first composition and the second composition; It causes a processor to perform the steps of generating image data.
  • a processing device is a processing device comprising a processor and a memory, wherein the processor obtains first image data including a subject, and obtains first information about the state of the subject in the first image data. determining the first composition of the subject in the first direction based on, and determining the first composition for the width of the first image data in the first direction based on the first information, the first composition, and the probability distribution; A first ratio, which is a width ratio in one direction, is set, and the second image data is generated based on the first ratio.
  • FIG. 1 is a diagram showing a schematic configuration of an image management system 100;
  • FIG. FIG. 4 is a schematic diagram for explaining the direction of image data; 4 is a flowchart for explaining image editing processing by a processor 42;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing;
  • FIG. 4 is a schematic diagram for explaining image editing processing; It is a figure which shows an example of probability distribution. It is a figure which shows an example of probability distribution.
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 11 is a flowchart for explaining a modification of image editing processing by a processor 42;
  • FIG. 1 is a diagram showing a schematic configuration of an image management system 100 including an image processing device 4, which is one embodiment of the processing device of the present invention.
  • the image management system 100 includes one or a plurality of imaging devices 1 (three imaging devices 1a, 1b, and 1c in the example of FIG. 1) and a network such as the Internet or a LAN (Local Area Network). 2 , an image storage server 3 , and an image processing device 4 .
  • the imaging device 1 is arranged, for example, in a theme park, an event venue, or the like.
  • the three imaging devices 1 are arranged at different positions to capture an image of a subject such as a person or an animal present at the installation location.
  • the imaging device 1 includes an imaging device, an image processing circuit that processes a captured image signal obtained by imaging a subject with the imaging device to generate image data, and a communication interface that can be connected to a network 2. .
  • the imaging device 1 is configured by, for example, a digital camera or a smart phone.
  • Image data generated by the imaging device 1 is also referred to as image data captured by the imaging device 1 .
  • the tag of the image data generated by the imaging device 1 includes attribute information of the imaging device 1 that generated the image data and imaging conditions (settings) of the imaging device 1 that generated the image data for a modified example described later. zoom magnification, set aspect ratio, etc.) and the capability of the imaging device 1 that generated the image data (zoom magnification setting range, etc.).
  • the attribute information of the imaging device 1 is information that defines the purpose of what kind of image is to be captured.
  • the imaging device 1a is provided with attribute information Z1 indicating that the imaging conditions and installation location are determined so that the entire subject (whole body) is within the imaging range.
  • the imaging devices 1b and 1c are provided with attribute information Z2 indicating that the imaging conditions and installation locations are determined so that part of the subject (for example, the face) is largely within the imaging range.
  • the imaging device 1 transmits the generated image data to the image storage server 3 via the network 2 .
  • the image storage server 3 includes a processor, a communication interface connectable to the network 2, and storage such as HDD (Hard Disk Drive), flash memory, SSD (Solid State Drive), or EEPROM (Electrically Erasable and Programmable Read Only Memory). a device; This storage device may be a network storage device connected to the network 2 .
  • the processor of the image storage server 3 acquires the image data (hereinafter referred to as first image data) transmitted from the imaging device 1 and stores the acquired first image data in the storage device.
  • the image processing device 4 is a device for editing the first image data stored in the storage device of the image storage server 3 .
  • the image processing device 4 includes a communication interface 41 for connecting to the network 2, a processor 42, and a memory 43.
  • the memory 43 includes RAM (Random Access Memory) used as a work memory, and non-volatile memory such as HDD, flash memory, SSD, or EEPROM.
  • RAM Random Access Memory
  • Non-volatile memory in memory 43 is an example of a non-transitory storage medium readable by processor 42 .
  • a processing program is stored in this nonvolatile memory.
  • the processor 42 is a processor whose circuit configuration can be changed after manufacturing, such as a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to perform various functions, and an FPGA (Field Programmable Gate Array).
  • CPU Central Processing Unit
  • FPGA Field Programmable Gate Array
  • a programmable logic device (PLD) or a dedicated electric circuit that is a processor having a circuit configuration specially designed to execute specific processing such as an ASIC (Application Specific Integrated Circuit).
  • the processor 42 may be composed of one processor, or may be composed of a combination of two or more processors of the same type or different types (for example, multiple FPGAs or a combination of a CPU and an FPGA).
  • each processor does not have to be in the same device, and may be in another location connected via the network 2 .
  • the hardware structure of the processor 42 is an electric circuit that combines circuit elements such as semiconductor elements.
  • FIG. 2 shows image data IM.
  • the image data IM is two-dimensional image data having a rectangular outer shape.
  • a two-dimensional Cartesian coordinate plane is assumed whose coordinate axes are the mutually orthogonal X-axis and Y-axis, and one vertex of the image data IM is arranged at the origin O on the coordinate plane.
  • the origin O as a base point, the image data IM are arranged in the positive direction of the X-axis and the positive direction of the Y-axis.
  • the side facing the negative direction of the X axis is the left side
  • the side facing the positive direction of the X axis is the right side
  • the side facing the negative direction of the Y axis is the upper side
  • the positive side of the Y axis is The side facing in the direction of is defined as the bottom side.
  • the direction from the left side of the image data IM to the right side is described as the right direction
  • the opposite direction is described as the left direction
  • the right direction and the left direction are collectively described as the horizontal direction.
  • the direction from the upper edge to the lower edge of the image data IM is referred to as the downward direction
  • the opposite direction is referred to as the upward direction
  • the upward direction and the downward direction are collectively referred to as the vertical direction.
  • the horizontal direction matches the horizontal direction of the image data
  • the vertical direction matches the vertical direction of the image data.
  • the lateral direction constitutes the first direction.
  • the longitudinal direction constitutes the second direction.
  • the processor 42 acquires the first image data stored in the image storage server 3 from the image storage server 3, and based on the acquired first image data, trims the first image data to generate the second image data. perform image editing processing.
  • the processor 42 performs trimming so that a part (specifically, a face) of a subject such as a person or an animal included in the first image data is included in the second image data without interruption.
  • FIG. 3 is a flowchart for explaining image editing processing by the processor 42.
  • the processor 42 acquires first image data from the image storage server 3 (step S1), and executes subject detection processing on the acquired first image data (step S2).
  • the subject detection process is a process of detecting a specific subject such as a person or an animal from the first image data and further detecting a face area including a face in the subject.
  • the orientation of the face of the subject in the first image data is also detected. That is, the processor 42 can distinguish and detect whether the orientation of the face detected in the first image data is front facing, left facing, or right facing.
  • the orientation of the face F is detected as facing rightward.
  • the orientation of the face F is detected to be frontal.
  • the face F in the first image data IMc of FIG. 4 when the face F in the first image data IMc appears as a face facing leftward, it is detected that the face F faces leftward.
  • the front facing face F faces the direction orthogonal to both the X axis and the Y axis, and the left facing face F faces the negative direction of the X axis.
  • the right-facing face F faces the positive direction of the X-axis.
  • the face detection information included in the result of subject detection processing by the processor 42 is information specifying "the position of the face, the orientation of the face, and the size of the face" in the first image data.
  • the processor 42 extracts a rectangular area containing a face as the face area, and expresses the position and size of the face by the center position and size (the number of pixels in the vertical and horizontal directions) of the facial area in the first image data.
  • the position of the face includes a horizontal position (hereinafter also referred to as horizontal position) and a vertical position (hereinafter also referred to as vertical position).
  • the face size includes a horizontal size (hereinafter also referred to as width) and a vertical size (hereinafter also referred to as vertical width).
  • the processor 42 may extract a rectangular area including the whole body of the subject as the subject area, and acquire the position and size of the subject based on the center position and size of the subject area in the first image data.
  • the processor 42 acquires information on the horizontal position, orientation, horizontal width (number of pixels), vertical position, and vertical width (number of pixels) of the face (preferably the whole body) included in the first image data. do.
  • the information on the horizontal face position, the orientation of the face, and the horizontal width of the face constitute first information regarding the state of the subject included in the first image data, and the information on the vertical position and the vertical width of the face constitute the first information on the subject. It constitutes a second information different from the first information about the state.
  • the processor 42 determines the first horizontal position of the face F in the lateral direction of the second image data to be generated. A composition is determined (step S3). Further, the processor 42 calculates a second composition of the face F in the vertical direction of the second image data to be generated based on the vertical position and the vertical width (second information) of the face F detected from the first image data. Determine (step S4)
  • the distance on the image data described in this embodiment is defined by the number of pixels.
  • the right end of the horizontal trimming range in the first image data is defined as the right cut line R1, and the left end as the left cut line L1.
  • the upper end of the trimming range in the vertical direction in the first image data is defined as an upper cutoff line U1, and the lower end thereof is defined as a lower cutoff line D1.
  • the processor 42 determines that the area on the side facing the face F is larger than the area on the side opposite to the side facing the face F, with the face F as a boundary. Decide on a composition that will also increase. When the face F is oriented forward, the processor 42 determines a composition in which the areas on the right and left sides of the face F are even.
  • the processor 42 When determining the first composition based on the first image data IMa shown in FIG. 4, the processor 42, as shown in FIG. Px and the distance X1 between the left cut line L1 on the left side opposite to the facing side of the face F and the lateral position Px of the face F is set to a value that makes the distance X2 larger than the distance X1.
  • the processor 42 When determining the first composition based on the first image data IMb shown in FIG. 4, the processor 42, as shown in FIG.
  • the composition of the face F in the horizontal direction is determined by setting the ratio of the distance X1 between the cutting line L1 and the lateral position Px of the face F to a value that makes the distance X1 and the distance X2 the same.
  • the processor 42 When determining the first composition based on the first image data IMc shown in FIG. 4, the processor 42, as shown in FIG. Px and the distance X2 between the right cutoff line R1 on the right side opposite to the side facing the face F and the lateral position Px of the face F is set to a value that makes the distance X1 larger than the distance X2.
  • the processor 42 determines a composition in which the upper area is smaller than the lower area with the face F as a border, regardless of the orientation of the face F.
  • the processor 42 determines the distance between the right cutoff line R1 and the left cutout line L1 (hereinafter referred to as The upper limit value and the lower limit value of the transverse cut width) are determined (step S5).
  • the horizontal cropping width is synonymous with the horizontal width of the second image data to be generated.
  • the processor 42 calculates the distance between the upper cutoff line U1 and the lower cutout line D1 based on the determined second composition, the vertical position of the face F, and the vertical width of the face F. (hereinafter referred to as vertical cut width) is determined (step S6).
  • the vertical crop width is synonymous with the vertical width of the second image data to be generated.
  • the horizontal cutting width and the vertical cutting width are collectively referred to as the cutting width.
  • the processor 42 determines that the distance from the lateral position Px to the left edge of the first image data IMa is the distance from the lateral position Px to the right edge of the first image data IMa. and the distance from the horizontal position Px to the left end of the first image data IMa is (X1/X2) times or more of the distance from the horizontal position Px to the right end of the first image data IMa. X2)
  • the lower limit value is determined as follows when it is less than times.
  • the method of determining the upper limit value of the horizontal cropping width in the first image data IMc including the face F facing left is the same as the method of determining the upper limit value of the horizontal cropping width in the first image data IMa, so the description is omitted. do.
  • the processor 42 in the example of the first image data IMb including the face F facing the front, the processor 42, as shown in FIG. 11, the distance between the horizontal position Px and the right edge of the first image data IMb is greater than the distance between the horizontal position Px and the left edge of the first image data IMb.
  • the method of determining the upper limit is changed depending on whether it is smaller than .
  • the ratio of the horizontal width of the imaging area included in the second image data to the horizontal width of the imaging area included in the first image data is defined as the first ratio
  • the process of determining the upper limit of the transverse cropping width can be regarded as equivalent to the process of determining the upper limit of the first ratio.
  • the margin width which is the horizontal width (the number of pixels) of the margin area Dx, may be a predetermined value or a value selected by the user.
  • the method of determining the lower limit value of the horizontal clipping width in the first image data IMc including the face F facing left is as follows.
  • the distance between the right cutting line R1 and the left cutting line L1 is obtained as the lower limit of the horizontal cutting width.
  • the process of determining the lower limit value of the transverse cropping width can be regarded as equivalent to the process of determining the lower limit value of the first ratio.
  • the processor 42 determines that the distance from the vertical position Py to the upper end of the first image data is to the lower end of the first image data (Y2/Y1) times or more, and the distance from the vertical position Py to the upper end of the first image data IMa is greater than the distance from the vertical position Py to the first image data
  • the lower bound is determined as follows.
  • the ratio of the vertical width of the imaging area included in the second image data to the vertical width of the imaging area included in the first image data (for example, the vertical width of the second image data is the vertical width of the first image data) ) is defined as the second ratio
  • the process of determining the upper limit of the vertical crop width can be regarded as equivalent to the process of determining the upper limit of the second ratio.
  • the processor 42 determines whether the lower limit of vertical cutting width is a predetermined value or a value selected by the user.
  • the process of determining the lower limit of the vertical crop width can be considered equivalent to the process of determining the lower limit of the second ratio.
  • the processor 42 determines the upper and lower limit values of the transverse cut width (in other words, the upper and lower limit values of the first ratio) and the transverse cut width (in other words, the first ratio ) is a random variable, and the transverse cropping width is set (step S7).
  • the processor 42 determines the upper limit and lower limit of the vertical cut width (in other words, the upper limit and lower limit of the second ratio) and the vertical cut width (in other words, the above second ratio). 2 ratio) as a random variable, the vertical cropping width is set (step S8).
  • the memory 43 prestores the probability distribution DX and the probability distribution DY in the form of functions, for example.
  • FIG. 17 is a diagram showing an example of the probability distribution DX of the transverse cropping width.
  • the horizontal axis of FIG. 17 indicates the width of the horizontal clipping, and the vertical axis indicates the probability that the width of the horizontal clipping is selected.
  • the probability distribution DX stored in the memory 43 is the part indicated by the thick solid line in the figure.
  • the probability distribution DX is a normal distribution indicated by the dashed line in the figure, and the random variable with the maximum probability is the lower limit of the cross-cutting width, and 1.96 ⁇ ( ⁇ is the standard deviation) is added to this lower limit It is a truncated normal distribution in which only the range between the upper limit and the lower limit is extracted from the normal distribution in which the standard deviation ⁇ is determined so that the value is the upper limit of the cross-cutting width.
  • the processor 42 acquires the probability distribution DX exemplified in FIG. 17, and selects one value from between the determined upper limit value and lower limit value of the cross-cropping width according to the probability of the acquired probability distribution DX. do.
  • the probability distribution DX shown in FIG. 17 has a high probability that the processor 42 selects a value close to the lower limit.
  • step S8 similarly to the probability distribution DX, the random variable with the maximum probability becomes the lower limit of the vertical cut width, and the value obtained by adding 1.96 ⁇ ( ⁇ is the standard deviation) to this lower limit is the vertical cut width.
  • the vertical cut width is set based on the probability distribution DY, which is a truncated normal distribution that extracts only the range between the upper limit and the lower limit of the normal distribution in which the standard deviation ⁇ is determined so as to be the upper limit of .
  • the probability distribution DX and the probability distribution DY are not limited to those shown in FIG. 17, and various ones can be adopted.
  • the probability distribution DX and the probability distribution DY may have a high probability of selecting a value close to the upper limit, as indicated by the thick solid line in FIG.
  • the probability distribution DX and the probability distribution DY may be distributions in which the probability is constant for all values between the upper limit and the lower limit.
  • the processor 42 trims the first image data according to the horizontal cropping width and the vertical cropping width set as described above to generate second image data (step S9).
  • a specific example of the trimming method will be described below.
  • the processor 42 determines a position to the right of the lateral position Px by a distance obtained by the calculation of Xset ⁇ (X2/(X1+X2)) as the right cutoff line R1.
  • the processor 42 determines the left cutoff line L1 as a position that is a distance to the left from the horizontal position Px by the calculation of Xset ⁇ (X1/(X1+X2)).
  • the processor 42 determines the lower cut line D1 as a position that is downwardly away from the vertical position Py by a distance obtained by calculating Yset ⁇ (Y1/(Y1+Y2)).
  • the processor 42 determines the upper cutoff line U1 as a position separated from the vertical position Py by a distance obtained by calculating Yset ⁇ (Y2/(Y1+Y2)). Then, the area surrounded by these four lines is cut out from the first image data IMa and used as the second image data.
  • the first composition of the face F in the horizontal direction is determined based on the orientation of the face F
  • the second composition of the face F in the vertical direction is determined based on the vertical position of the face F.
  • the second image data is generated based on the horizontal cropping width and the vertical cropping width set based on the first composition and the second composition.
  • the processor 42 sets the horizontal cropping width and the vertical cropping width using the probability distribution in addition to the first composition and the second composition. Therefore, even when the second image data is generated from each of a large number of first image data containing similar subjects with similar compositions, the horizontal cropping width and the vertical cropping width are set at random, resulting in various horizontal cropping widths. Second image data of crop width and vertical crop width can be generated. Therefore, the variations of the second image data can be increased, and the degree of satisfaction of the person viewing or purchasing the second image data can be enhanced.
  • a plurality of probability distributions are stored in the memory 43, and the processor 42 selects one of the plurality of probability distributions stored in the memory 43 as the probability distribution used in steps S7 and S8 of FIG. good.
  • [Method A] to [Method D] below are examples of methods for selecting one of a plurality of probability distributions.
  • Processor 42 selects a probability distribution prespecified by the user. According to this method, it is possible to generate the second image data reflecting the intention of the user.
  • the processor 42 randomly selects one from a plurality of probability distributions, and stores in the memory 43 the usage history of the probability distributions used when generating the second image data in the past. If the processor 42 determines from this usage history that there is a probability distribution with a low usage frequency, it preferentially selects this probability distribution with a low usage frequency. Thus, processor 42 selects one from a plurality of probability distributions based on previously generated second image data. According to this method, the second image data can be generated by uniformly using a plurality of probability distributions, and the variation of the second image data can be increased.
  • the processor 42 selects one of the plurality of probability distributions based on the attributes of the subject included in the first image data (for example, whether the subject is an adult or a child). For example, suppose that an adult wants image data showing a large face, while a child wants image data showing the whole body. In this case, if the first image data includes an adult as a subject, processor 42 sets the cropping width using the probability distribution illustrated in FIG. On the other hand, if the first image data includes a child as the subject, the processor 42 sets the cropping width using the probability distribution illustrated in FIG.
  • the second image data can be generated with the face in a close-up state, and if the subject is a child, the second image data can be generated in a state in which many parts other than the child's face are captured. Second image data can be generated. In this way, it is possible to generate optimal second image data according to the attribute of the subject.
  • the processor 42 selects one of the plurality of probability distributions based on the attribute of the imaging device 1 that captured the first image data (whether for telephoto or wide angle).
  • the plurality of imaging devices 1 includes an imaging device 1 equipped with a telephoto lens and an imaging device 1 equipped with a wide-angle lens.
  • the processor 42 sets the cropping width using the probability distribution illustrated in FIG. 17 .
  • the processor 42 sets the cropping width using the probability distribution illustrated in FIG. 18 .
  • Second image data can be generated according to one's intention. Since the imaging apparatus 1 equipped with a wide-angle lens is intended to capture an image of the whole body of the subject, the probability distribution of FIG. The second image data can be generated as intended by the device 1 .
  • the processor 42 stores in the memory 43 the setting history of the horizontal cutting width and the vertical cutting width set in the past.
  • the processor 42 refers to the above setting history when setting the horizontal and vertical cutting widths in steps S7 and S8 of FIG.
  • the horizontal and vertical clipping widths are randomly selected and set from the variables according to the probability distribution. By doing so, it becomes easier to select a horizontal cropping width and a vertical cropping width that have not been selected for a long period of time, and it is possible to increase the variation of the second image data.
  • the aspect ratio of the second image data generated by trimming is not limited. If the aspect ratio of the second image data is limited, it is preferable to set the horizontal and vertical crop widths in consideration of this aspect ratio.
  • FIG. 19 is a flowchart for explaining a modified example of image editing processing by the processor 42.
  • steps S1 to S4 in FIG. 19 are the same as steps S1 to S4 in FIG. 3, description thereof is omitted.
  • step S3 the processor 42 derives provisional values of the lower limit and upper limit of the lateral cropping width based on the first composition, the lateral position of the face F, and the lateral width of the face F (step S21), and step S4. process.
  • the provisional value derived in step S21 is the same as the value determined in step S5 of FIG. In the following description, it is assumed that the upper limit value and the lower limit value of the transverse cropping width derived in step S21 are 1200 and 900, respectively.
  • step S4 the processor 42 derives temporary values for the lower and upper limits of the vertical cropping width based on the second composition, the vertical position of the face F, and the vertical width of the face F (step S23).
  • the provisional value derived in step S23 is the same as the value determined in step S6 of FIG. Below, the upper limit value of the vertical cropping width derived in step S23 is set to 400, and the lower limit value is set to 500.
  • the processor 42 derives the upper limit value and lower limit value of the horizontal cropping width that can satisfy the allowable aspect ratio from the allowable aspect ratio and the provisional values of the vertical cropping width (step S24).
  • the width is set (step S26). In the following description, it is assumed that the horizontal cropping width set in step S26 is 950. As shown in FIG.
  • the width is set (step S29).
  • the processor 42 trims the first image data according to the horizontal cropping width set in step S26 and the vertical cropping width set in step S29 to generate second image data (step S30).
  • the second image data having the aspect ratio desired by the user can be generated in various variations. .
  • Processor 42 may set the horizontal and vertical crop widths without using probability distributions. For example, processor 42 may set the transverse crop width and the vertical crop width to their respective lower bounds. By setting the cropping width to the lower limit, it is possible to generate the second image data in which the face F included in the first image data is maximized. Alternatively, the processor 42 may generate the second image data by setting the cropping width to a value between the lower limit and the upper limit that makes the size of the face F in the second image data equal to or larger than the threshold. . By doing so, the size of the face F in the second image data can be stabilized in a large state.
  • the processor 42 may determine the information used to determine the first composition based on the attribute (telephoto or wide angle) of the imaging device 1 that captured the first image data. For example, the processor 42 refers to attribute information included in the first image data in step S3 of FIG. Then, if the imaging device 1 having the attribute information is an imaging device set so that the proportion of the face occupying the imaging range is large (higher zoom magnification, use of a telephoto optical system, etc.), the processor 42 , the lateral position of the face, the lateral width of the face, and the orientation of the face detected in step S2 are used as first information to determine the first composition of the face in the lateral direction.
  • the imaging device 1 having the attribute information is an imaging device (low zoom magnification, wide-angle optical system, etc.) that is set so that the proportion of the face in the imaging range is small (the whole body of the subject is captured).
  • the processor 42 uses the subject's lateral position, subject's lateral width, and face orientation detected in step S2 as first information to determine the first composition of the face in the lateral direction.
  • the processor 42 may determine the information used to determine the second composition based on the attribute (telephoto or wide-angle) of the imaging device 1 that captured the first image data.
  • the processor 42 may determine the information used for determining the first composition based on the content of the first image data (in other words, the imaging scene when the first image data was captured). For example, in step S3 of FIG. 3, the processor 42 analyzes the first image data to determine the imaging scene when the first image data was captured. For example, as imaging scenes, assume a first scene in which the subject is playing sports and a second scene in which the subject is still.
  • the processor 42 determines the first composition of the face in the horizontal direction using the lateral position of the face, the lateral width of the face, and the direction of the face detected in step S2 as first information. do.
  • the processor 42 uses the horizontal position of the subject, the width of the subject, and the orientation of the face detected in step S2 as the first information to determine the first composition of the subject in the horizontal direction. to decide.
  • the processor 42 may determine the information used for determining the second composition based on the imaging scene of the first image data.
  • the second image data in which the face is enlarged and the second image data including the face and the body and in which the face is relatively small are generated. can be generated. Therefore, it is possible to generate appropriate second image data according to the imaging scene, and to provide the user with high-value second image data.
  • the processor 42 sets the cropping width based on the first image data, and trims the first image data according to the set cropping width to generate the second image data. did.
  • the processor 42 sets the cropping width based on the first image data captured at the first timing, and according to the cropping width, the first image data captured at the second timing after the first timing.
  • the second image data may be generated by trimming (corresponding to the third image data).
  • the imaging device 1 that captured the first image data captured at the first timing and the imaging device 1 that captured the first image data captured at the second timing may be the same or different. good.
  • the processor 42 divides the first image data captured at the first timing (obtained in step S1 in FIG. 3) and the first image data captured at the second timing (i.e., as trimming targets).
  • the cropping width may be set based on the attributes of the imaging device 1 that captures image data to be acquired in the future.
  • the processor 42 captures the first image data captured at the second timing (that is, the image data to be trimmed) based on the attributes of the imaging device 1. Then, one of the multiple probability distributions should be selected.
  • the processor 42 sets the cropping width based on the first image data, and trims the first image data according to the set cropping width to generate the second image data. did.
  • the second image data may be generated by causing the imaging device 1 that captured the first image data to perform imaging in the imaging range according to the cropping width set as described above.
  • the position of the subject in the imaging range is the position based on the first composition and the second composition (in other words, the same composition as the second image data generated by trimming). At least one of zoom magnification, optical axis position, and crop range is controlled.
  • the imaging device 1 can, for example, change the zoom magnification, change the position of the optical axis (the position in the plane perpendicular to the optical axis), and limit the recording range of the signal from the imaging device. Crop imaging is possible.
  • the processor 42 can capture image data having the same composition as the second image data generated by trimming the first image data with the set cutting width. , the zoom magnification, optical axis position, or cropping range of the imaging device 1 that captured the first image data is controlled, and the imaging device 1 is caused to perform imaging in the controlled state. By this imaging, it is possible to obtain second image data having a composition similar to that obtained by trimming the first image data.
  • the processor 42 detects the zoom magnification, optical axis position, or cropping range of an imaging device 1 (corresponding to a second imaging device) different from the imaging device 1 (corresponding to a first imaging device) that captured the first image data. may be controlled and the second image data may be obtained by causing the other imaging device 1 to perform imaging in the controlled state.
  • the seventh modification can be combined with the first modification (processing in FIG. 19).
  • the processor 42 determines the horizontal cropping width that can satisfy the set aspect ratio from the provisional values of the vertical cropping width and the aspect ratio of the imaging device 1 that is the imaging source of the second image data. Derive upper and lower bounds.
  • the lower limit of the horizontal cut width is determined as 900 by satisfying both the condition of 600 to 1050 and the condition of 900 to 1200 (900 to 1050).
  • the upper width limit is determined as 1050.
  • the horizontal clipping width is set according to the probability distribution between these upper and lower limits.
  • the set aspect ratio of the imaging device 1 that captures the second image data is 3:2
  • the vertical cropping width is changed according to this set aspect ratio. , is set to 2/3 times the set value of the transverse crop width.
  • the processor 42 After setting the horizontal cropping width, the processor 42 captures the second image data based on the horizontal cropping size set in the imaging device 1 that captures the second image data and the set horizontal cropping width. The zoom magnification of the original imaging device 1 is determined.
  • the processor 42 determines a value obtained by dividing the width imaging size by the set value of the horizontal cropping width as the zoom magnification.
  • the processor 42 causes the imaging device 1 to perform imaging according to this zoom magnification, thereby obtaining second image data whose composition substantially matches that of the second image data generated by trimming according to the set value of the cutout width. be able to.
  • cropping is performed based on the first image data, the probability distribution, and the zoom performance and imaging conditions (set zoom magnification and imaging size) of the imaging device 1 that is the imaging source of the second image data. You can decide the width. Detailed operation will be described below.
  • the upper limit value and the lower limit value of the horizontal cropping width are adjusted.
  • the zoom performance is 1 ⁇ or more and 4 ⁇ or less
  • the set value of the zoom magnification is 1.6 times
  • the set value of the horizontal imaging size is 1600 pixels.
  • the processor 42 determines the lower limit of the transverse cropping width from the condition (1000 to 1200) that satisfies both the derived condition of 1000 to 4000 and the condition of 900 to 1200 determined in step S5 of FIG. Adjust the value to 1000 and adjust the upper limit of the vertical crop width to 1200. Then, the processor 42 sets the horizontal clipping width according to the probability distribution between the adjusted upper and lower limits.
  • the processor 42 After setting the horizontal cropping width, the processor 42 captures the second image data based on the horizontal cropping size set in the imaging device 1 that captures the second image data and the set horizontal cropping width. The zoom magnification of the original imaging device 1 is determined.
  • the processor 42 determines a value obtained by dividing the width imaging size by the set value of the horizontal cropping width as the zoom magnification.
  • the processor 42 concurrently performs the processing of step S8 in FIG. 3 to set the vertical cropping width.
  • the processor 42 then obtains the ratio of the set value of the vertical cropping width to the set value of the horizontal cropping width (referred to as the aspect ratio).
  • the processor 42 causes the imaging device 1 to perform imaging according to the determined zoom magnification, and also adjusts the cropping range such that the vertical width of the image data to be captured becomes the value obtained by multiplying the horizontal width by the vertical/horizontal ratio. set.
  • the composition of the image data obtained by imaging with the imaging device 1 can be substantially matched with the second image data generated by trimming according to the cutout width set by the processor 42 .
  • the processor 42 acquires the orientation of the subject included in the first image data by performing subject detection processing on the first image data. However, if the same subject is captured by an imaging device 1 different from the imaging device 1 that captured the first image data, subject detection for the image data acquired from the other imaging device 1 may be used. It may be acquired based on the result of processing. Alternatively, on the premise that the subject wears a device that transmits beacon information, when the first image data is captured, the imaging device 1 stores the received beacon information in association with it. Then, the processor 42 may acquire the orientation information of the subject included in the first image data based on the beacon information corresponding to the acquired first image data.
  • a processing device comprising a processor and a memory, The above processor Acquiring first image data including a subject; determining a first composition of the subject in a first direction based on first information about the state of the subject; determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information regarding the state of the subject; A processing device that generates second image data based on the first composition and the second composition.
  • the processing apparatus includes information regarding the orientation of the subject in the first image data, The processing device, wherein the second information includes information about the position of the subject in the first image data.
  • a ratio of a width in the first direction of the imaging region included in the second image data to a width in the first direction of the imaging region included in the first image data is defined as a first ratio;
  • the processing apparatus derives a lower limit value of the settable first ratio based on the first composition, the first information, and a margin width set next to the subject in the first direction. , processing equipment.
  • the processing apparatus stores a plurality of the probability distributions; A processing device, wherein the processor sets the first ratio based on one of the plurality of probability distributions stored in the memory.
  • the processing apparatus generating the second image data based on the first image data or third image data different from the first image data; A processing device that selects one of the plurality of probability distributions based on an attribute of an imaging device that captures the first image data or the third image data.
  • the processing apparatus according to any one of (4) to (11), The processor is further configured to set the first ratio based on an acceptable aspect ratio of the second image data.
  • the processing apparatus according to any one of (4) to (11), The above processor generating the second image data by causing a first imaging device that has captured the first image data or a second imaging device that is different from the first imaging device to perform imaging; Further, the processing device sets the first ratio based on the set aspect ratio of the first imaging device or the second imaging device.
  • the processing apparatus according to any one of (4) to (11), The above processor generating the second image data by causing a first imaging device that has captured the first image data or a second imaging device that is different from the first imaging device to perform imaging; Further, the processing device sets the first ratio based on imaging conditions of the first imaging device or the second imaging device.
  • the processing apparatus sets the first ratio to a value at which the size of the subject in the second image data is greater than or equal to a threshold value, and generates the second image data.
  • the processing apparatus according to any one of (1) to (17),
  • the first image data can be obtained from a plurality of imaging devices,
  • the processing apparatus according to any one of (1) to (18),
  • the first image data can be obtained from a plurality of imaging devices, A processing device that determines the first information and the second information used for composition determination based on the content of the first image data.
  • the processing apparatus according to any one of (1) to (19), The processor trims the first image data based on the first composition and the second composition to generate the second image data.
  • (21) Acquiring first image data including a subject; determining a first composition of the subject in a first direction based on first information about the state of the subject; determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information regarding the state of the subject; A processing method of generating second image data based on the first composition and the second composition.
  • (22) Acquiring first image data including a subject; determining a first composition of the subject in a first direction based on first information about the state of the subject; determining a second composition of the subject in a second direction different from the first direction based on second information different from the first information regarding the state of the subject; A processing program that causes a processor to execute a step of generating second image data based on the first composition and the second composition.
  • a processing device comprising a processor and a memory, The above processor Acquiring first image data including a subject; determining a first composition of the subject in a first direction based on first information about the state of the subject in the first image data; setting a first ratio, which is a ratio of the width in the first direction to the width in the first direction of the first image data, based on the first information, the first composition, and the probability distribution; A processing device for generating second image data based on the first ratio.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Image Processing (AREA)
PCT/JP2023/007339 2022-02-28 2023-02-28 処理装置、処理方法、及び処理プログラム Ceased WO2023163217A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202380023843.0A CN118786681A (zh) 2022-02-28 2023-02-28 处理装置、处理方法及处理程序
JP2024503315A JPWO2023163217A1 (https=) 2022-02-28 2023-02-28
US18/772,281 US20240373138A1 (en) 2022-02-28 2024-07-15 Processing apparatus, processing method, and processing program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022030052 2022-02-28
JP2022-030052 2022-02-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/772,281 Continuation US20240373138A1 (en) 2022-02-28 2024-07-15 Processing apparatus, processing method, and processing program

Publications (1)

Publication Number Publication Date
WO2023163217A1 true WO2023163217A1 (ja) 2023-08-31

Family

ID=87766292

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/007339 Ceased WO2023163217A1 (ja) 2022-02-28 2023-02-28 処理装置、処理方法、及び処理プログラム

Country Status (4)

Country Link
US (1) US20240373138A1 (https=)
JP (1) JPWO2023163217A1 (https=)
CN (1) CN118786681A (https=)
WO (1) WO2023163217A1 (https=)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011135527A (ja) * 2009-12-25 2011-07-07 Nikon Corp デジタルカメラ
JP2014143673A (ja) * 2012-12-26 2014-08-07 Canon Inc 画像処理装置、撮像装置、画像処理方法、及び記録媒体

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011135527A (ja) * 2009-12-25 2011-07-07 Nikon Corp デジタルカメラ
JP2014143673A (ja) * 2012-12-26 2014-08-07 Canon Inc 画像処理装置、撮像装置、画像処理方法、及び記録媒体

Also Published As

Publication number Publication date
CN118786681A (zh) 2024-10-15
US20240373138A1 (en) 2024-11-07
JPWO2023163217A1 (https=) 2023-08-31

Similar Documents

Publication Publication Date Title
US10762653B2 (en) Generation apparatus of virtual viewpoint image, generation method, and storage medium
CN111641775B (zh) 多摄变焦控制方法、装置及电子系统
US10970915B2 (en) Virtual viewpoint setting apparatus that sets a virtual viewpoint according to a determined common image capturing area of a plurality of image capturing apparatuses, and related setting method and storage medium
US10893251B2 (en) Three-dimensional model generating device and three-dimensional model generating method
CN102959943B (zh) 立体全景图像合成装置和方法、以及图像捕获装置
JP5592006B2 (ja) 三次元画像処理
CN103916582B (zh) 一种图像处理方法及装置
US20110018969A1 (en) Image reproducing apparatus and image reproducing method
JP2013254432A (ja) 画像処理装置及び画像処理方法
JP2018107793A (ja) 仮想視点画像の生成装置、生成方法及びプログラム
US9619886B2 (en) Image processing apparatus, imaging apparatus, image processing method and program
US20200226824A1 (en) Systems and methods for 3d scanning of objects by providing real-time visual feedback
US11127141B2 (en) Image processing apparatus, image processing method, and a non-transitory computer readable storage medium
JP2013153375A (ja) 画像処理装置、画像処理方法および記録媒体
KR20250135351A (ko) 스마트한 파노라마 이미지를 획득하기 위한 시스템 및 방법
JP2018113683A (ja) 画像処理装置、画像処理方法及びプログラム
EP3245616A1 (en) Event triggered by the depth of an object in the field of view of an imaging device
US11184545B2 (en) Display control apparatus, imaging apparatus, control method for display apparatus, and non-transitory computer readable medium
US10979684B2 (en) Image processing apparatus, information processing method, and program for associating an element in a background shape model with an image capturing apparatus
CN111917941B (zh) 摄像机画面处理方法及摄像机
JP6245892B2 (ja) 画像処理装置およびその制御方法、プログラム
JP5889022B2 (ja) 撮像装置、画像処理装置、画像処理方法及びプログラム
US20230209182A1 (en) Automatic photography composition recommendation
US12069398B2 (en) Image processing apparatus, method, and non-transitory computer-readable storage medium for superimposing images
CN116456191A (zh) 图像生成方法、装置、设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23760215

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024503315

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202380023843.0

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23760215

Country of ref document: EP

Kind code of ref document: A1