WO2023171120A1 - 情報処理装置、情報処理方法、およびプログラム - Google Patents

情報処理装置、情報処理方法、およびプログラム Download PDF

Info

Publication number
WO2023171120A1
WO2023171120A1 PCT/JP2023/000665 JP2023000665W WO2023171120A1 WO 2023171120 A1 WO2023171120 A1 WO 2023171120A1 JP 2023000665 W JP2023000665 W JP 2023000665W WO 2023171120 A1 WO2023171120 A1 WO 2023171120A1
Authority
WO
WIPO (PCT)
Prior art keywords
subject
information processing
control unit
cut out
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/000665
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
圭一 吉岡
和俊 河村
佑輝 中居
正俊 福田
建 齊藤
智和 酒井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/835,353 priority Critical patent/US20250148656A1/en
Priority to JP2024505921A priority patent/JPWO2023171120A1/ja
Priority to CN202380024994.8A priority patent/CN118805378A/zh
Priority to DE112023001333.0T priority patent/DE112023001333T5/de
Publication of WO2023171120A1 publication Critical patent/WO2023171120A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/268—Signal distribution or switching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G06T2207/20132—Image cropping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a program.
  • Patent Document 1 listed below discloses a technique related to appropriate editing of live distributed content.
  • the present disclosure proposes an information processing device, an information processing method, and a program that can reduce the burden of acquiring captured images of a subject.
  • a captured image acquired from one or more imaging devices that capture an image of a target space is analyzed, one or more subjects to be cut out from the captured image are determined, and control is performed to cut out the determined subject.
  • An information processing device including a control unit is provided.
  • the processor analyzes a captured image obtained from one or more imaging devices that capture an image of a target space, determines one or more subjects to be cut out from the captured image, and selects the determined subject.
  • An information processing method is provided that includes controlling the extraction of information.
  • the computer analyzes a captured image acquired from one or more imaging devices that capture an image of a target space, determines one or more subjects to be cut out from the captured image, and selects the determined subject.
  • a program is provided that functions as a control unit that performs control to extract.
  • FIG. 1 is a diagram illustrating an overview of a distribution system according to an embodiment of the present disclosure.
  • FIG. 1 is a block diagram showing an example of the configuration of a content generation device according to the present embodiment. It is a diagram showing an example of a position adjustment screen 400 displayed on the display unit of the content generation device according to the present embodiment.
  • FIG. 3 is a diagram showing an example of a cutout image display screen according to the present embodiment.
  • FIG. 3 is a diagram illustrating cutting out of a subject located in a region of interest according to the present embodiment. It is a figure explaining the cutting range by this embodiment.
  • FIG. 6 is a diagram illustrating a cropping range when a plurality of subjects are included according to the present embodiment.
  • FIG. 1 is a block diagram showing an example of the configuration of a content generation device according to the present embodiment. It is a diagram showing an example of a position adjustment screen 400 displayed on the display unit of the content generation device according to the present embodiment.
  • FIG. 3 is a diagram showing an
  • FIG. 6 is a diagram illustrating switching of a captured image to be cut out due to movement of a subject according to the present embodiment.
  • FIG. 3 is a diagram illustrating designation of a recognition area according to the present embodiment.
  • FIG. 2 is a block diagram showing an example of the configuration of a distribution switching device according to the present embodiment.
  • 3 is a flowchart illustrating an example of the flow of operation processing of the content generation device according to the present embodiment.
  • FIG. 7 is a diagram illustrating another method of using a cutout image according to an application example of the present embodiment.
  • FIG. 1 is a diagram illustrating an overview of a distribution system according to an embodiment of the present disclosure.
  • the distribution system according to the present embodiment includes cameras 10a to 10d (an example of an imaging device) that image a stage S (an example of a target space) of an event venue V, content of distribution candidates (specifically, an image ), and a distribution switching device 30 that switches content to be distributed.
  • the event venue V may be a facility with a stage S and audience seats, or may be a recording room (recording studio).
  • the cameras 10a to 10c are installed at the event venue V and can image each area of the stage S. Although the angles of view of the cameras 10a to 10c are different, images are taken in a state where they partially overlap, as shown in FIG.
  • the captured images captured by the cameras 10a to 10c are output to the content generation device 20, and are used in the content generation device 20 to cut out the subject.
  • the cameras 10a to 10c may be, for example, 4K cameras, 8K cameras, or 16K cameras.
  • the resolution of the cameras 10a to 10c is not particularly limited, it is desirable that the resolution be such that when a subject is cut out from a captured image, a cutout image that is suitable for viewing and viewing can be obtained.
  • the cameras 10a to 10c may be installed side by side on the audience seat side of the stage S.
  • the number of cameras 10 is not particularly limited. The number of cameras 10 may be one or more.
  • a camera 10d whose field of view includes the entire stage S may be further provided.
  • the captured image (overhead image of the stage S) captured by the camera 10d is not used for cutting out by the content generation device 20, but is output to the distribution switching device 30.
  • the camera 10d may be, for example, an HD (High Definition) camera.
  • the resolution of the camera 10d is not particularly limited, but may be, for example, lower than the resolution of the cameras 10a to 10c that acquire captured images used to cut out the subject.
  • a plurality of cameras may be installed to acquire captured images that are not used to cut out the subject. For example, a camera that images the entire stage S from a direction different from that of the camera 10d may be further installed.
  • the content generation device 20 is an information processing device that performs control to cut out one or more subjects from each image captured by the cameras 10a to 10c and generate one or more cutout images of the subject as distribution candidate content.
  • the content generation device 20 transmits the cut out image to the distribution switching device 30.
  • SDI Serial Digital Interface
  • the content generation device 20 performs cutting for the number of image outputs (specifically, the number of SDI outputs).
  • the distribution switching device 30 is a device that controls switching (selection) of images to be distributed to a distribution destination (specifically, a viewer terminal).
  • a plurality of images such as a cutout image output from the content generation device 20 and a captured image captured by the camera 10d, can be input to the distribution switching device 30.
  • the distribution switching device 30 selects an image to be output (distributed) from among the plurality of input images, and outputs it to the distribution destination. Further, the distribution switching device 30 appropriately switches (newly selects) images to be distributed. Switching (selection) may be performed arbitrarily by an operator (for example, a switcher), or may be performed automatically.
  • the distribution system it is possible to reduce the burden of acquiring captured images of a subject and reduce the number of people required for imaging. For example, by automatically cutting out an arbitrary subject from images captured by a plurality of cameras 10a to 10c installed in the event venue V shown in FIG. It can be obtained as appropriate. Even when a large number of subjects are on the stage, the workload can be reduced by automatically determining the subject to be cut out.
  • FIG. 2 is a block diagram showing an example of the configuration of the content generation device 20 according to this embodiment.
  • the content generation device 20 includes a communication section 210, a control section 220, an operation input section 230, a display section 240, and a storage section 250.
  • the content generation device 20 is used, for example, by a director who directs the entire event.
  • the communication unit 210 includes a transmitting unit that transmits data to an external device by wire or wirelessly, and a receiving unit that receives data from the external device.
  • the communication unit 210 uses, for example, wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication) 5G (fifth generation mobile communication system)), etc., to communicate with the cameras 10a to 10c and the distribution switching device 30.
  • the communication unit 210 can also function as a transmitting unit that transmits (outputs) the subject cutout image to the distribution switching device 30.
  • SDI output may be used.
  • Image output may be performed separately from data transmission performed using the LAN or the like.
  • the control unit 220 functions as an arithmetic processing device and a control device, and controls overall operations within the content generation device 20 according to various programs.
  • the control unit 220 is realized by, for example, an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the control unit 220 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate. Further, the control unit 220 may include a GPU (Graphics Processing Unit).
  • the control unit 220 also functions as a display position adjustment unit 221, a cutout processing unit 222, and an output control unit 223.
  • the display position adjustment unit 221 displays a plurality of captured images, which are obtained from cameras 10a to 10c, which are a plurality of imaging devices arranged on the audience seat side of the stage S, and which have partially overlapping angles of view on the display unit 240.
  • a process of displaying the plurality of captured images side by side in an overlapping state and a process of accepting adjustment of the overlapping position of the plurality of captured images are performed.
  • Such adjustments may be made by an operator (for example, a director) in a preparatory stage before the start of the event.
  • the cameras 10a to 10c are placed on the audience seat side so that they can image the entire stage S. For example, in the example shown in FIG.
  • each camera 10 may be set to partially overlap with the angle of view (imaging range) of the adjacent camera 10.
  • the left end of the imaging range of the camera 10b located at the center overlaps the right end of the imaging range of the camera 10a located on the left
  • the right end of the imaging range of the camera 10b located at the center is located on the right. It is set to overlap with the left end of the imaging range of the camera 10c.
  • the display position adjustment section 221 displays the captured images of the cameras 10a to 10c side by side on the display section 240. A detailed explanation will be given below with reference to FIG. 3.
  • FIG. 3 is a diagram showing an example of a position adjustment screen 400 displayed on the display unit 240 of the content generation device 20 according to the present embodiment.
  • a captured image 401 captured by the camera 10a a captured image 402 captured by the camera 10b
  • a captured image 403 captured by the camera 10c are displayed side by side.
  • the position adjustment screen 400 includes an operation screen for controlling the display position, display size, and transparency of each of the captured images 401 to 403.
  • the operator (for example, the director) of the content generation device 20 can move the display position of each captured image 401 to 403 vertically and horizontally, enlarge/reduce the display size, or make the captured image transparent so that the subject can be photographed.
  • the display position adjustment unit 221 receives an input of a display position adjustment operation, and stores the adjustment results (the display position and display size of each captured image) in the storage unit 250.
  • the adjustment result may be at least information on the overlapping position of each captured image (which region of the imaging range overlaps with which region of the imaging range of which camera).
  • the present disclosure is not limited thereto, and the display position adjustment unit 221 may perform the adjustment automatically. Alternatively, the operator may be asked to confirm the automatically adjusted results.
  • the cutout processing unit 222 analyzes captured images obtained from one or more imaging devices (for example, cameras 10a to 10c) that image a target space (for example, stage S), and extracts one or more subjects to be cut out from the captured images. is determined, and control is performed to cut out the determined subject. Such cutout processing may be continuously performed from the start of event distribution (start of imaging). Specifically, this is performed for each frame.
  • start of imaging start of imaging
  • the cutout processing unit 222 analyzes the captured images 401 to 403 and identifies the subject through object recognition.
  • the subject may be a human, an animal, an object, etc., but in this embodiment, a human performing on a stage is assumed.
  • the cutout processing unit 222 may perform face detection to identify the subject.
  • the cropping processing unit 222 determines a subject that satisfies a predetermined condition among the specified subjects as a cropping target, and performs cropping.
  • the image cut out by the cutout processing unit 222 (cutout image; captured image of the subject) is outputted to the distribution switching device 30 and the display unit 240 by the output control unit 223.
  • the output control unit 223 can control output (transmission) of one or more cutout images from the communication unit 210 to the distribution switching device 30 and output (display) them to the display unit 240. Further, the output control unit 223 may output the cutout image to the distribution switching device 30 and may also transmit a distribution switching control signal to the distribution switching device 30. For example, a signal (information used to control distribution switching in the distribution switching device 30) indicating a cut-out image with a high distribution priority, such as a singing subject or a subject in an attention area, may be transmitted.
  • FIG. 4 is a diagram showing an example of a cutout image display screen 410 according to the present embodiment.
  • a cutout image display screen 410 shown in FIG. 4 is displayed on the display unit 240 of the content generation device 20 during event distribution.
  • the director can intuitively know the subject specified by the system and the image (cutout image) that is preferentially cut out by the system and output (SDI output) to the distribution switching device 30.
  • SDI output system and output
  • the cutout image display screen 410 displays each of the captured images 401 to 403 obtained from the cameras 10a to 10c, and the cutout images 501 to 501 cut out from each of the captured images 401 to 403. 505 is displayed.
  • Corresponding SDI output numbers are assigned to the cutout images 501 to 505.
  • the cutout images 501 to 505 are SDI output to the distribution switching device 30.
  • the captured images 401 to 403 displayed on the cutout image display screen 410 are displayed side by side with some parts overlapping according to the results adjusted in advance by the display position adjustment unit 221.
  • Each of the captured images 401 to 403 shown in FIG. 4 includes subjects P1 to P9, and the result of face detection for each subject is clearly indicated by a frame line (a frame line surrounding the face). This allows the director to intuitively understand that the subject is being recognized by the system. Further, the frame line of the subject determined to be cut out may be highlighted.
  • the SDI output number associated with the cutout image of the subject is also displayed on the frame line of the subject determined to be the cutout target. This allows the director to intuitively understand which subject has been determined by the system to be cropped, and the cropped image of the determined subject.
  • the cropping processing unit 222 determines a subject to be cropped that satisfies a predetermined condition and performs the cropping, and the "predetermined condition" includes, for example, performing a predetermined action.
  • the cutout processing unit 222 preferentially determines a subject recognized to be performing a predetermined action as a cutout target.
  • the cutout processing unit 222 may recognize a predetermined motion by analyzing the captured image. Further, the cutout processing unit 222 may recognize a predetermined motion based on sensing data other than the captured image.
  • the cutout processing unit 222 determines a singing subject to be cut out as a subject that satisfies a predetermined condition. If the subject is an idol group or the like with a large number of people, the cutout processing unit 222 preferentially determines the singing subject to be cut out. This is because at a music concert, it is important to follow the person singing with a camera.
  • the cutout processing unit 222 analyzes the captured image to estimate the skeleton of the subject, and determines that the subject is singing if the subject lifts a hand holding a hand microphone. Furthermore, the extraction processing unit 222 determines whether a sound source is present ( If the microphone is turned on), it is determined that the user is singing. Further, the cutout processing unit 222 determines that the subject is singing when movement of the microphone is detected based on information from an acceleration sensor or the like provided in the microphone of the subject. The cutout processing unit 222 also performs image recognition of the captured image, and determines that the subject is singing if the subject's mouth is open.
  • the cutout processing unit 222 determines that the subject is singing if the subject is in a predetermined position at a predetermined timing (preset from the singing ratio and standing position) based on the position information of the subject on the stage.
  • the positional information of the subject on the stage is obtained by a sensor possessed by the subject (for example, a UWB (Ultra-Wide Band) positional information tag) or image recognition.
  • UWB Ultra-Wide Band
  • an example of the "predetermined condition" is that the object is located in the region of interest.
  • the cropping processing unit 222 determines the region of interest and determines a subject located in the region of interest as a subject to be cropped, as a subject that satisfies a predetermined condition. This is because, at a music concert or the like, a region of interest (an area that is desired to be noticed in terms of presentation) may be temporarily created.
  • the cutout processing unit 222 recognizes the movement of each subject by, for example, skeletal estimation, and determines an area where there is movement (it may be an area where the amount of movement is greater than other areas).
  • FIG. 5 is a diagram illustrating cutting out of a subject located in a region of interest according to this embodiment.
  • a captured image 404 acquired from the camera 10 any one of 10a to 10c
  • other subjects P12 and P13 are stationary, while only a specific group (subjects P10 and P11) is If they are moving, the cutout processing unit 222 determines the subjects P10 and P11 as a group to be cut out, and cuts them out from the captured image 404 (a cutout image 506 is generated).
  • an example of the "predetermined condition" is to be located at the center of the stage. This is because, at music concerts, etc., the subject of interest is often located at the center of the stage.
  • the cutout processing unit 222 determines, as a subject that satisfies a predetermined condition, a subject located at the center on the stage to be cut out.
  • the cropping processing unit 222 can crop a range that includes one subject (single cropping) or a range that includes multiple subjects (group cropping). As described with reference to FIG. 5, group cutting may be performed, for example, when cutting out based on a region of interest.
  • the cutout processing unit 222 cuts out the subject (generates cutout images) by the number of cutouts corresponding to the number of images output to the distribution switching device 30.
  • the number of image outputs is, for example, the number of SDI outputs, and can be defined in advance.
  • the cropping processing unit 222 may preferentially determine the subject identified from the captured image as the cropping target. When the number of identified subjects is equal to or greater than the number of cutouts, the cutout processing unit 222 preferentially cuts out subjects that satisfy the conditions according to each of the predetermined conditions described above. Further, the cropping processing unit 222 may determine the subject to be cropped by combining each of the above-mentioned predetermined conditions. For example, when the number of identified subjects is greater than or equal to the above-mentioned number of cutouts, and all the subjects are singing, the cutout processing unit 222 may preferentially determine a subject close to the center to be cut out. Further, if the cropping processing unit 222 can identify the subject and input popularity information of each subject, the cropping processing unit 222 may preferentially determine the popular subject as the cropping target.
  • the clipping processing unit 222 may determine a fixed position on the stage to be clipped. For example, at the start, transition, or end of a music concert, there may be some time before a subject appears on the stage. In this case, the cutout processing unit 222 preferentially cuts out an image at a fixed position, such as the center on the stage or the appearance position of the subject on the stage (which may be set in advance).
  • the subject to be cut out can also be arbitrarily specified by the operator (for example, the director) of the content generation device 20.
  • the operator specifies a subject to be cut out.
  • the designation method is not particularly limited, for example, the designation may be performed by touching the subject in each of the captured images 401 to 403 displayed on the cutout image display screen 410. Alternatively, the display of the frame surrounding the subject's face may be moved to the face of another subject by dragging and dropping.
  • the cutout processing unit 222 cuts out a range that includes at least the subject's face. Further, the cropping processing unit 222 may crop the image in a range that includes at least the subject's face and that is close to (enlarged to) the resolution limit value (resolution at a level that can withstand viewing). The resolution limit value may be set in advance. Further, the cutout processing unit 222 may further cut out a range that includes at least the subject's hand. When considering the choreography of a subject, it may be desirable to cut out a range that includes at least the face and hands.
  • the cropping processing unit 222 may also determine the cropping range (whether to include only the face, hands, upper body only, whole body, etc.) based on the skeletal estimation of the subject. For example, when the cutout processing unit 222 recognizes through skeletal estimation that the hands are moving significantly during choreography, etc., the cutout processing unit 222 may set the cutout range to include the hands.
  • the cropping processing unit 222 may perform cropping in a range that includes a predetermined margin above the top of the body of the subject (to be cropped).
  • the top of the body is the highest part of the person, usually the head, and when the hand is raised, the hand.
  • FIG. 6 is a diagram illustrating the cutout range according to this embodiment.
  • the cutout processing unit 222 acquires (generates) a cutout image 507 in a range including a margin h above the head, which is the top of the subject P.
  • the cropping processing unit 222 crops the subject to be cropped, including at least the face, in a range enlarged to the resolution limit value, it is assumed that other nearby subjects may enter the cropping range. Ru.
  • the cropping processing unit 222 temporarily includes a subject whose body falls within the cropping range more than half or whose body falls into the cropping range to the extent that it can be recognized by bone estimation, into the cropping target, and selects a subject that fits within the height of all the subjects. Make a cut. A specific example will be explained with reference to FIG.
  • FIG. 7 is a diagram illustrating the cropping range when multiple subjects are included according to the present embodiment.
  • the cutout processing unit 222 acquires (generates) cutout images 508 in a range including the margin h above the top of the body (the head of the subject P17) of all the subjects. This makes it possible to avoid cutting out an image in which the head is unnaturally cut.
  • Such adjustment of the cropping range when a plurality of subjects are included can also be applied to the case of group cropping described above.
  • the height of the cropping range will not be adjusted for distribution. It is also possible to keep the subject determined to be the cropping target when it is selected. In addition, when the cropped image is selected for distribution by the distribution switching device 30 (programmed out) and the number of subjects decreases from the cropping range (the subject is temporarily determined to be cropped), the cropping processing unit 222 (in the case that the subject has moved out of the cropping range), the height of the cropping range may not be changed. This maintains the quality of the image during program out.
  • the present embodiment is not limited to this, and the cropping range may be adjusted only to the subject determined to be cropped, without taking into consideration even if the subject enters the image. .
  • the cropping processing unit 222 may apply smoothing to the movement direction of the cropping range between frames so that the movement of the subject in continuous cropped images (cutout video consisting of a plurality of frames) looks natural.
  • types of smoothing include an average value of movement amounts for frames in a certain period, a weighted average, and the like.
  • the cropping processing unit 222 takes the average value of the coordinate positions of the subject determined to be the cropping target, and can reduce the amount of movement of the cropping range (without being affected by small movements of the subject).
  • the cutout processing unit 222 may perform cutting in a range that includes a large margin in the line of sight direction (direction of the face). good. As a result, it is possible to obtain a cropped image with a sophisticated composition that provides depth and guides the viewer's line of sight.
  • the cropping processing unit 222 may also crop a range that includes a plurality of subjects (group cropping) and crop a range that includes only one subject included in the plurality of subjects (single cropping). That is, both group cropping and individual cropping may be performed simultaneously on one cropping target subject. With this, for example, when the distribution switching device 30 switches between a group cutout image and an individual cutout image, it can be expected to make the viewer feel a sense of dynamism and give a sense of being at a music concert or the like.
  • the cropping processing unit 222 performs cropping from one of the captured images.
  • the cutout processing unit 222 needs to continue tracking (continue cutting out) the subject to be cut out. For this reason, when a subject to be cut out (also referred to as a tracking target) moves across multiple captured images, the cutout processing unit 222 switches the captured image to be cut out and continues tracking when it enters an overlapping area. It may be possible to do so.
  • the cutout processing unit 222 determines whether the first captured image and the second captured image are different from each other.
  • the source image to be cut out is switched at the overlapping part.
  • FIG. 8 is a diagram illustrating switching of a captured image to be cut out due to movement of a subject according to the present embodiment.
  • the cutout processing unit 222 switches the cutout source of the subject from the captured image 402 to the captured image 401 when the subject P1 enters the overlap region E between the captured image 402 and the captured image 401.
  • the zoom ratio appears to have changed on the cutout image output to the distribution switching device 30.
  • we can distinguish and identify the characteristics of the subject to be tracked color of clothing, hairstyle, etc.
  • a combination of depth sensors to track the subject's movement. It is conceivable to identify the direction by comparing it.
  • by combining a positioning sensor for example, by having the subject carry an identifiable tag), it is also possible to determine and identify the position of the subject.
  • the cutout processing unit 222 is not limited to tracking the subject, but may also cut out a predetermined area (preset) on the stage (fixed position cutout). Specifically, the cropping processing unit 222 determines one or more subjects located in a predetermined area on the stage to be cropped, and performs cropping in a range that includes the subject. Then, the cutout processing unit 222 does not track the subject even if the subject moves out of the predetermined area.
  • FIG. 9 is a diagram illustrating designation of a recognition area according to this embodiment.
  • captured images 401 to 403 are displayed side by side in a partially overlapping state.
  • a rectangular recognition frame D is displayed on the captured images 401 to 403.
  • the operator (for example, the director) of the content generation device 20 can adjust the position and size of the recognition frame D (for example, so as not to include the audience or the back screen) and specify the recognition area.
  • the cutout processing unit 222 calculates the coordinate position of the specified recognition frame D, and sets a recognition area (image analysis area) in each of the captured images 401 to 403, as shown in the lower part of FIG.
  • the cutout processing unit 222 performs image analysis within the recognition area and identifies the subject. Note that the adjustment of the recognition frame D is not limited to manual adjustment, and may be performed automatically by the content generation device 20.
  • the operation input unit 230 accepts operation input from an operator and outputs input information to the control unit 220.
  • the display unit 240 also displays various operation screens and the screens described in FIGS. 3, 4, and 9.
  • the display unit 240 may be a display panel such as a liquid crystal display (LCD) or an organic EL (electro luminescence) display.
  • the operation input section 230 and the display section 240 may be provided integrally.
  • the operation input unit 230 may be a touch sensor stacked on the display unit 240 (eg, a panel display).
  • the storage unit 250 is realized by a ROM (Read Only Memory) that stores programs, calculation parameters, etc. used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the configuration of the content generation device 20 has been specifically described above, the configuration of the content generation device 20 according to the present disclosure is not limited to the example shown in FIG. 2.
  • the content generation device 20 may have a configuration that does not include the operation input section 230 and the display section 240.
  • the content generation device 20 may be realized by a plurality of devices.
  • at least some of the functions of the content generation device 20 may be realized by a server.
  • FIG. 10 is a block diagram showing an example of the configuration of the distribution switching device 30 according to this embodiment.
  • the distribution switching device 30 includes a communication section 310, a control section 320, an operation input section 330, a display section 340, and a storage section 350.
  • the operator of the distribution switching device 30 may be a switcher whose position is to switch distribution images.
  • the communication unit 310 includes a transmitting unit that transmits data to an external device by wire or wirelessly, and a receiving unit that receives data from the external device.
  • the communication unit 310 uses, for example, wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication) 5G (fifth generation mobile communication system)), etc., to communicate with the content generation device 20 and the distribution destination.
  • SDI may be used for the communication unit 210 to input the subject cutout image from the content generation device 20.
  • the Internet may be used for the communication unit 210 to transmit (distribute) the image to the distribution destination.
  • Control unit 320 functions as an arithmetic processing device and a control device, and controls overall operations within the distribution switching device 30 according to various programs.
  • the control unit 320 is realized by, for example, an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor. Further, the control unit 320 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the control unit 320 also functions as a switching unit 321 and a distribution control unit 322.
  • the switching unit 321 switches (selects) images to be distributed (programmed out) to a distribution destination (viewer terminal). Specifically, the switching unit 321 selects one image to be distributed from among the plurality of cut-out images outputted from the content generation device 20 via SDI. The distribution control unit 322 then controls distribution of the selected image from the communication unit 310 to the distribution destination.
  • the switching unit 321 may select images to be automatically distributed according to a control signal from the content generation device 20.
  • the content generation device 20 sends a signal that designates five cut-out images of five subjects and two cut-out images of two people whose singing behavior has been recognized as images with high distribution priority. is input.
  • the switching unit 321 randomly selects one of the two cut-out images (the image of the singing subject) designated as an image with a high distribution priority.
  • the content generation device 20 may set a higher distribution priority for the subjects closer to the center, and the switching unit 321 may select accordingly.
  • the distribution priority may be set high also for the subject in the attention area. For presentation purposes, if there is a cutout image of the subject in the attention area, the switching unit 321 may always select it (as an image to be distributed).
  • the switching unit 321 also switches the image to be distributed (switches to the next cut-out image of the singing subject).
  • switching (selection) of distribution images by the switching unit 321 is performed automatically as described above, but is not limited to this, and the switching unit 321 can perform switching operations by an operator (for example, a switcher) of the distribution switching device 30.
  • the control unit 320 may display a plurality of cutout images (candidates for distribution images) output from the content generation device 20 on the display unit 340, and allow the operator to arbitrarily select one of the cutout images (candidates for distribution images).
  • the display unit 340 may also display information regarding the cropped subject (such as popularity, number of followers, center, etc.) and recommend it to the operator.
  • the switching unit 321 may adjust the timing of switching the distributed images to the tempo (BPM; Beats Per Minute) of the music that the subject is singing.
  • the switching unit 321 can extract BPM from the input sound source (sound collected by a subject's microphone, etc.).
  • the switcher may input the BPM by touching the touch panel display (the operation input section 330 and the display section 340 are integrated) in accordance with the rhythm (touching at regular intervals in accordance with the melody).
  • the switching unit 321 may be switched in accordance with the timing at which the switching button is pressed by the operator.
  • the image to be switched can be automatically selected by the switching unit 321.
  • the candidates for the distribution image also include an overhead image obtained from the camera 10d, but the priority is low. Therefore, the bird's-eye view image of the camera 10d may be selected as the distribution image, for example, when no one is singing or when there is no subject on the stage (at the beginning and end of a song, etc.).
  • the operation input unit 330 accepts operation input by an operator and outputs input information to the control unit 220.
  • the display unit 340 also displays various operation screens and delivery image candidates (cut out images).
  • the display unit 340 may be a display panel such as a liquid crystal display (LCD) or an organic EL (electro luminescence) display.
  • the operation input section 330 and the display section 340 may be provided integrally.
  • the operation input unit 330 may be a touch sensor stacked on the display unit 340 (eg, a panel display).
  • the storage unit 350 is realized by a ROM (Read Only Memory) that stores programs, calculation parameters, etc. used in the processing of the control unit 320, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the configuration of the distribution switching device 30 has been specifically described above, the configuration of the distribution switching device 30 according to the present disclosure is not limited to the example shown in FIG. 10.
  • the distribution switching device 30 may have a configuration that does not include the operation input section 330 and the display section 340. Further, the distribution switching device 30 may be realized by a plurality of devices.
  • FIG. 11 is a flowchart showing an example of the flow of operation processing of the content generation device 20 according to the present embodiment.
  • control unit 220 of the content generation device 20 controls the camera 10 (10a to 10c) to start photographing (step S103). Distribution can be started when the camera 10 starts photographing.
  • the content generation device 20 acquires captured images from each of the cameras 10a to 10c (step S106).
  • the cutout processing unit 222 of the content generation device 20 analyzes each captured image (step S109) and identifies the subject.
  • the cutout processing unit 222 determines the number of subjects to be cut out from each captured image (step S112). Note that a group including a plurality of subjects (a subject group to be cut out) is added as 1.
  • the cutout processing unit 222 cuts out the subject for the number of cuts (step S115). That is, the cutout processing unit 222 acquires (generates) a cutout image from the captured image.
  • the output control unit 223 displays one or more cut-out images on the display unit 240 (step S118). Further, the output control unit 223 transmits (SDI output) one or more cutout images to the distribution switching device 30 (step S121). The distribution switching device 30 selects an image to be distributed from one or more cut-out images.
  • steps S106 to S121 are performed for each frame until the shooting (distribution) is completed (step S124).
  • the distribution switching device 30 can perform distribution in real time.
  • FIG. 11 An example of the flow of operation processing of the content generation device 20 according to the present embodiment has been described above. Note that the operational processing shown in FIG. 11 is an example, and some of the processing may be performed in a different order or in parallel, or some of the processing may not be performed.
  • FIG. 12 is a diagram illustrating another method of using a cutout image according to an application example of this embodiment.
  • the output control unit 223 of the content generation device 20 may display the cutout images side by side on a multi-screen on a back screen 600 provided on the stage, as shown in FIG. It may be displayed not only on the back screen 600 but also on other large displays installed at the venue.
  • the display priority can be determined based on the singing, the attention area, the center, etc., as described above.
  • the output control unit 223 may always display cut-out images of all subjects on a multi-screen. In addition, in order to prevent the display position of each subject from being scattered on the multi-screen, the output control unit 223 displays the cutout image of the newly identified subject again in the same display after LOST the subject (if tracking fails or is lost). It may also be displayed at the location. Note that it does not have to depend on the output resolution. There may be irregular resolutions of LED displays installed at the venue, such as HD, 4K, or 8K.
  • the output control unit 223 acquires information indicating a cutout image that has been selected for distribution (programmed out) from the distribution switching device 30, and displays the information in real time on the display screen shown in FIG.
  • the cutout image selected for distribution may be highlighted. This allows the director to easily understand the video currently being distributed.
  • one or more computer programs for causing hardware such as a CPU, ROM, and RAM built in the content generation device 20 and distribution switching device 30 described above to exhibit the functions of the content generation device 20 and distribution switching device 30. can also be created. Also provided is a computer readable storage medium storing the one or more computer programs.
  • Information comprising a control unit that analyzes captured images obtained from one or more imaging devices that capture images of a target space, determines one or more subjects to be cut out from the captured images, and performs control to cut out the determined subjects. Processing equipment.
  • the information processing device according to (1) wherein the control unit cuts out a range that includes at least the face of the subject.
  • the control unit preferentially determines a subject that satisfies a predetermined condition as a subject to be cut out.
  • the control unit determines a singing subject to be cut out as a subject that satisfies the predetermined condition.
  • control unit determines a subject located in a region of interest to be cut out as a subject that satisfies the predetermined condition.
  • control unit determines, as a subject that satisfies the predetermined condition, a subject located at a center on a stage, which is the target space, to be cut out.
  • control unit determines a fixed position on the stage to be cut out when the number of subjects is insufficient for a predetermined number of cutouts.
  • control unit performs cutting for a number of images corresponding to the number of output images.
  • the control unit performs cropping in a range including a plurality of subjects and cropping in a range including one subject included in the plurality of subjects. information processing equipment.
  • the information processing device according to any one of (1) to (13), wherein the control unit performs cropping in a range that includes one or more subjects located in a predetermined area on the stage.
  • the captured images are a plurality of captured images with partially overlapping angles of view, which are obtained from a plurality of imaging devices arranged on the audience side of the stage, The information processing device according to any one of (1) to (14), wherein the control unit displays the plurality of captured images side by side in a partially overlapping state, and receives adjustment of the overlapping position.
  • the control unit described in (15) above performs control to output the plurality of cut out images to a device that switches distribution images and control to display them on a display unit together with the plurality of captured images arranged side by side. information processing equipment.
  • the control unit controls whether the subject to be cut out moves between the first captured image and the second captured image.
  • the information processing device according to (15) or (16), wherein the captured image to be cut out is switched at a portion where the images overlap.
  • the processor Information processing that includes analyzing captured images obtained from one or more imaging devices that capture images of a target space, determining one or more subjects to be cut out from the captured images, and performing control to cut out the determined subjects.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Studio Devices (AREA)
PCT/JP2023/000665 2022-03-11 2023-01-12 情報処理装置、情報処理方法、およびプログラム Ceased WO2023171120A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US18/835,353 US20250148656A1 (en) 2022-03-11 2023-01-12 Information processing device, information processing method, and program
JP2024505921A JPWO2023171120A1 (https=) 2022-03-11 2023-01-12
CN202380024994.8A CN118805378A (zh) 2022-03-11 2023-01-12 信息处理设备、信息处理方法及程序
DE112023001333.0T DE112023001333T5 (de) 2022-03-11 2023-01-12 Informationsverarbeitungsvorrichtung, Informationsverarbeitungsverfahren und Programm

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-037841 2022-03-11
JP2022037841 2022-03-11

Publications (1)

Publication Number Publication Date
WO2023171120A1 true WO2023171120A1 (ja) 2023-09-14

Family

ID=87936719

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/000665 Ceased WO2023171120A1 (ja) 2022-03-11 2023-01-12 情報処理装置、情報処理方法、およびプログラム

Country Status (5)

Country Link
US (1) US20250148656A1 (https=)
JP (1) JPWO2023171120A1 (https=)
CN (1) CN118805378A (https=)
DE (1) DE112023001333T5 (https=)
WO (1) WO2023171120A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240080566A1 (en) * 2022-09-02 2024-03-07 OnstageAI, INC. System and method for camera handling in live environments
WO2025192068A1 (ja) * 2024-03-14 2025-09-18 株式会社Ｊｖｃケンウッド 映像配信装置、映像配信システム、および映像配信プログラム
WO2025234222A1 (ja) * 2024-05-07 2025-11-13 キヤノン株式会社 画像処理装置、撮像装置、制御方法およびプログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009245404A (ja) * 2008-04-01 2009-10-22 Fujifilm Corp 画像処理装置および方法並びにプログラム
JP2015050695A (ja) * 2013-09-03 2015-03-16 カシオ計算機株式会社 動画生成システム、動画生成方法及びプログラム
JP2018117312A (ja) * 2017-01-20 2018-07-26 パナソニックＩｐマネジメント株式会社 映像配信システム、ユーザ端末装置および映像配信方法
JP2021057660A (ja) * 2019-09-27 2021-04-08 キヤノン株式会社 撮像制御装置、撮像装置、及び撮像制御方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110463208A (zh) 2017-03-24 2019-11-15 索尼公司 内容处理装置、内容处理方法以及程序

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009245404A (ja) * 2008-04-01 2009-10-22 Fujifilm Corp 画像処理装置および方法並びにプログラム
JP2015050695A (ja) * 2013-09-03 2015-03-16 カシオ計算機株式会社 動画生成システム、動画生成方法及びプログラム
JP2018117312A (ja) * 2017-01-20 2018-07-26 パナソニックＩｐマネジメント株式会社 映像配信システム、ユーザ端末装置および映像配信方法
JP2021057660A (ja) * 2019-09-27 2021-04-08 キヤノン株式会社 撮像制御装置、撮像装置、及び撮像制御方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240080566A1 (en) * 2022-09-02 2024-03-07 OnstageAI, INC. System and method for camera handling in live environments
WO2025192068A1 (ja) * 2024-03-14 2025-09-18 株式会社Ｊｖｃケンウッド 映像配信装置、映像配信システム、および映像配信プログラム
WO2025234222A1 (ja) * 2024-05-07 2025-11-13 キヤノン株式会社 画像処理装置、撮像装置、制御方法およびプログラム

Also Published As

Publication number Publication date
CN118805378A (zh) 2024-10-18
US20250148656A1 (en) 2025-05-08
JPWO2023171120A1 (https=) 2023-09-14
DE112023001333T5 (de) 2025-01-02

Similar Documents

Publication Publication Date Title
WO2023171120A1 (ja) 情報処理装置、情報処理方法、およびプログラム
CN111066315B (zh) 一种被配置为处理和显示图像数据的装置、方法及可读媒体
JP5594850B2 (ja) 代替現実システム制御装置、代替現実システム、代替現実システム制御方法、プログラム、および記録媒体
US7248294B2 (en) Intelligent feature selection and pan zoom control
CN107852476B (zh) 动画播放装置、动画播放方法、动画播放系统以及动画发送装置
JPWO2017119034A1 (ja) 撮影システム、撮影方法およびプログラム
US20210349620A1 (en) Image display apparatus, control method and non-transitory computer-readable storage medium
CN112804585A (zh) 一种在直播过程中实现产品智能展示的处理方法及装置
US11211097B2 (en) Generating method and playing method of multimedia file, multimedia file generation apparatus and multimedia file playback apparatus
JP4414708B2 (ja) 動画表示用パーソナルコンピュータ、データ表示システム、動画表示方法、動画表示プログラムおよび記録媒体
US12610153B2 (en) Information processing apparatus, information processing method, and program
US20240107150A1 (en) Zone-adaptive video generation
CN112887620A (zh) 视频拍摄方法、装置及电子设备
JP2020102687A (ja) 情報処理装置、画像処理装置、画像処理方法、及びプログラム
CN119383447A (zh) 图像处理装置、图像处理方法、系统、计算机程序产品、存储介质和计算机实现的方法
JPWO2018062538A1 (ja) 表示装置およびプログラム
WO2023189079A1 (ja) 画像処理装置、および画像処理方法、並びにプログラム
JP2019220783A (ja) 情報処理装置、システム、情報処理方法及びプログラム
US20230396873A1 (en) Information processing device, information processing method, and program
CN112236740A (zh) 热图展示装置以及热图展示用程序
JP2004518161A (ja) カメラ動き制御基準を決定する方法及び装置
JP2004289779A (ja) 移動体撮像方法、移動体撮像システム
KR20180089639A (ko) 수술 영상촬영 및 처리 시스템
KR101816208B1 (ko) 멀티 앵글 기반 가상 현실 융합 디스플레이 장치 및 방법
JP2006229467A (ja) フォトムービー作成装置及びフォトムービー作成プログラム、並びに被写体認識方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23766301

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024505921

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 18835353

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 202380024994.8

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 112023001333

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23766301

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18835353

Country of ref document: US