JP7261261B2 - Information processing device, switching system, program and method - Google Patents

Description

The present invention relates to an information processing device, switching system, program and method.

In recent years, the number of pixels in cameras has increased, and it has become possible to acquire high-quality images such as 4K and 8K. Techniques have been proposed for generating moving images that mimic camera work such as close-ups, pans, and tilts without the need for special equipment by cutting out a portion of high-quality video (for example, patent Reference 1).

JP 2016-28539 A

However, the above technique requires the user to specify the subject to be cut out in the video.

Furthermore, the user has to specify the size and position of the image to be clipped by the camera work, and it has not been possible to automatically clip the video of the desired camera work. In particular, close-up images include various shots such as full shots, bust shots, and waist shots, but such advanced camera work could not be performed automatically.

Accordingly, it is an object of the present invention to provide an information processing apparatus, a switching system, a program, and a method capable of automatically generating various camerawork images.

One aspect of the present invention includes an acquisition unit that acquires a captured image captured by a camera, a specific person designation reception unit that designates a specific person as a subject, and a part of the captured image that is trimmed to produce a virtual image. A virtual camerawork designation reception unit that designates a virtual camerawork that realizes a video effect by operating a camera, a person recognition unit that recognizes a person in the captured image, and a person in the image using the captured image. a skeleton determination unit that determines a skeleton; a specific person in the video is identified using the recognition result of the person recognition unit; A person identification unit that identifies the positional relationship of parts of a specific person on a captured image, a composition information storage unit that stores composition information in which a composition is determined for each virtual camera work, and a photograph of the parts of the specific person. a trimming frame size determination unit that determines a size of a trimming frame for trimming a portion of the photographed video using the positional relationship on the video and composition information corresponding to the specified virtual camera work; and the specific person. Trimming for controlling the position of the captured image of the trimming frame corresponding to the specified virtual camera work, using the positional relationship of the part on the captured image and the composition information corresponding to the specified virtual camera work The video processing device includes a frame control unit and a trimming unit that trims the video within the trimming frame from the captured video and outputs the trimmed video as a virtual camerawork video by the virtual camerawork.

One aspect of the present invention includes at least two or more video processing devices, a display unit, and a switching unit, and the video processing device includes an acquisition unit that acquires a captured video captured by a camera and a subject. a specific person designation reception unit that designates a specific person; a virtual camera work designation reception unit that designates virtual camera work that trims a part of the captured image and realizes a video effect by operating a virtual camera; A person recognition unit that recognizes a person in a captured image, a skeleton determination unit that uses the captured image to determine the skeleton of the person in the image, and a recognition result of the person recognition unit that identifies the person in the image. a person identification unit that identifies a person and uses the skeleton of the specific person among the skeletons determined by the skeleton determination unit to specify the positional relationship of the parts of the specific person on the captured image; using a composition information storage unit storing composition information in which the composition is determined, the positional relationship of the specific person's body part on the captured image, and the composition information corresponding to the specified virtual camera work, a trimming frame size determining unit that determines the size of a trimming frame for trimming a portion of the captured image; a positional relationship of the specific person's body parts on the captured image; and composition information corresponding to the specified virtual camera work. a trimming frame control unit for controlling the position of the photographed video in the trimming frame corresponding to the specified virtual camera work, and trimming the video within the trimming frame from the photographed video, and trimming the trimmed video as a virtual camera work image by the virtual camera work, the display unit displays the captured image and at least two or more of the virtual camera work images, and the switching unit The switching system outputs one virtual camerawork image specified by a user, out of at least two virtual camerawork images.

One aspect of the present invention includes an acquisition process for acquiring a captured image captured by a camera, a specific person designation reception process for receiving a designation of a specific person to be a subject, and a part of the captured image that is trimmed to produce a virtual image. a virtual camerawork specification acceptance process for accepting a virtual camerawork specification that realizes a video effect by operating a camera; a person recognition process for recognizing a person in the captured image; a skeleton determination process for determining a skeleton of a person; identifying a specific person in the video using a recognition result of the person recognition process; , person identification processing for identifying the positional relationship of the parts of the specific person on the captured image, the positional relationship of the parts of the specific person on the captured image, and the composition information in which the composition is determined for each of the virtual camerawork trimming frame size determination processing for determining the size of a trimming frame for trimming a portion of the captured video using composition information corresponding to the specified virtual camera work; trimming frame control processing for controlling the position of the captured image in the trimming frame corresponding to the specified virtual camera work using the positional relationship and composition information corresponding to the specified virtual camera work; A program for causing a computer to perform a trimming process of trimming a video within the trimming frame from a video and outputting the trimmed video as a virtual camerawork video by the virtual camerawork.

One aspect of the present invention includes an acquisition process for acquiring a captured image captured by a camera, a specific person designation process for receiving a designation of a specific person to be a subject, and a part of the captured image that is trimmed to produce a virtual image. A virtual camera work specification process for accepting a virtual camera work specification for realizing a video effect by operating a camera, a person recognition process for recognizing a person in the captured image, and a person in the image using the captured image. a skeleton determination process for determining a skeleton; a specific person in the video is identified using a recognition result of the person recognition process; Person identification processing for identifying the positional relationship of a specific person's whole body in the captured image, the positional relationship of the specific person's whole body in the captured image, and the composition information in which the composition of the subject is determined for each of the virtual camerawork trimming frame size determination processing for determining the size of a trimming frame for trimming a portion of the captured video using composition information corresponding to the specified virtual camera work; trimming frame control processing for controlling the position of the captured image in the trimming frame corresponding to the specified virtual camera work using the positional relationship and composition information corresponding to the specified virtual camera work; A trimming process for trimming the video within the trimming frame from the video and outputting the trimmed video as a virtual camerawork video by the virtual camerawork; Display processing for displaying at least two or more virtual camerawork images corresponding to virtual camerawork, and outputting one virtual camerawork image specified by a user among the at least two or more displayed virtual camerawork images. A program that causes a computer to perform a switching process.

In one aspect of the present invention, a computer obtains a captured image captured by a camera, receives designation of a specific person as a subject, trims a part of the captured image, and obtains a virtual image obtained by operating the camera. Receiving designation of virtual camera work that realizes an effect, recognizing a person in the captured image, determining the skeleton of the person in the image using the captured image, and using the recognition result of the person, the image identify a specific person in the system, use the skeleton of the specific person among the determined skeletons to specify the positional relationship of the parts of the specific person on the captured video, and identify the positional relationship of the parts of the specific person on the captured video Using the positional relationship and the composition information corresponding to the designated virtual camera work out of the composition information in which the composition is determined for each virtual camera work, the size of the trimming frame for trimming the part of the captured image is determined. and using the positional relationship of the parts of the specific person on the captured video and the composition information corresponding to the specified virtual camera work, the captured video of the trimming frame corresponding to the specified virtual camera work. and trimming the image within the trimming frame from the captured image, and outputting the trimmed image as a virtual camerawork image by the virtual camerawork.

In one aspect of the present invention, a computer obtains a captured image captured by a camera, receives designation of a specific person as a subject, trims a part of the captured image, and obtains a virtual image obtained by operating the camera. Receiving designation of virtual camera work that realizes an effect, recognizing a person in the captured image, determining the skeleton of the person in the image using the captured image, and using the recognition result of the person, the image identifying a specific person in the system, using the skeleton of the specific person among the determined skeletons, specifying the positional relationship of the specific person on the whole-body shot video, and identifying the specific person's whole-body shot video A trimming frame for trimming a part of the photographed image by using the positional relationship and the composition information corresponding to the designated virtual camera work out of the composition information in which the composition of the subject is determined for each virtual camera work. determining the size of the trimming frame corresponding to the specified virtual camera work using the positional relationship of the specific person's whole body shot video and the composition information corresponding to the specified virtual camera work; controlling the position of the captured image, trimming the image within the trimming frame from the captured image, outputting the trimmed image as a virtual camera work image by the virtual camera work, and combining the captured image with at least two At least two or more virtual camera work images corresponding to a specific person or at least two or more virtual camera works are displayed, and one virtual camera designated by the user among the displayed at least two or more virtual camera work images This is a switching method for outputting camerawork video.

The present invention can automatically generate a wide variety of camerawork videos.

FIG. 1 is a block diagram showing the overall configuration of the first embodiment. FIG. 2 is a block diagram of the video processing device 2. As shown in FIG. FIG. 3 is a diagram for explaining face recognition. FIG. 4 is a diagram for explaining skeleton determination. FIG. 5 is a diagram for explaining how the person identification unit 26 identifies the positional relationship of the parts of a specific person on the captured image. FIG. 6 is a block diagram of the video processing device 2 configured by a computer system. FIG. 7 is a diagram for explaining the display unit of the tablet terminal (computer 100). FIG. 8 is an overall operation flowchart of the video processing device 2. FIG. 9 is a diagram for explaining the overall operation of the video processing device 2. As shown in FIG. FIG. 10 is a diagram for explaining the first trimming frame. FIG. 11 is a diagram for explaining the second trimming frame. FIG. 12 is a diagram for explaining the third trimming frame. FIG. 13 is a diagram for explaining the fourth trimming frame. FIG. 14 is a diagram for explaining the operation of the first embodiment. FIG. 15 is a diagram for explaining the operation of the first embodiment. FIG. 16 is a diagram for explaining the operation of the first embodiment. FIG. 17 is a diagram for explaining the operation of the first embodiment. FIG. 18 is a diagram for explaining the operation of the first embodiment. FIG. 19 is a diagram for explaining the operation of the first embodiment. FIG. 20 is a diagram for explaining the operation of the first embodiment. FIG. 21 is a diagram for explaining the operation of the first embodiment. FIG. 22 is a diagram for explaining the operation of the first embodiment. FIG. 23 is a diagram for explaining the operation of the first embodiment. FIG. 24 is a diagram for explaining the operation of the first embodiment. FIG. 25 is a diagram for explaining the operation of the first embodiment. FIG. 26 is a diagram for explaining the operation of the first embodiment. FIG. 27 is a diagram for explaining the operation of the first embodiment. FIG. 28 is a diagram for explaining the operation of the modification of the first embodiment. FIG. 29 is a diagram for explaining the operation of the modification of the first embodiment. FIG. 30 is a diagram for explaining a modification of the first embodiment. FIG. 31 is a diagram for explaining the operation of the modification of the first embodiment. FIG. 32 is a diagram for explaining the operation of the modification of the first embodiment. FIG. 33 is a block diagram of the information processing device 2 of the modification of the first embodiment. FIG. 34 is a block diagram of the information processing device 2 of the second embodiment. FIG. 35 is a diagram for explaining the operation of the second embodiment. FIG. 36 is a diagram for explaining the operation of the second embodiment. FIG. 37 is a diagram for explaining the operation of the second embodiment. FIG. 38 is a block diagram of the switching system of the third embodiment. FIG. 39 is a diagram for explaining the operation of the third embodiment. FIG. 40 is a diagram for explaining the operation of the third embodiment. FIG. 41 is a diagram for explaining the operation of the third embodiment.

<First Embodiment>
A first embodiment will be described.

FIG. 1 is a block diagram showing the overall configuration of the first embodiment. In FIG. 1, 1 is a camera, 2 is a video processing device, and 3 is a display device.

A camera 1 is a camera for shooting a program. In principle, one camera 1 shoots a program with a wide angle of view so that all subjects (for example, performers of the program) can be captured. As will be described later, this embodiment generates virtual camerawork video by trimming a portion of video captured by camera 1 (hereinafter referred to as captured video). A 4K or 8K camera capable of capturing high-quality images is preferable, but is not limited to these.

The image processing device 2 inputs the image captured by the camera 1, and generates an image (hereinafter referred to as a virtual camera work image) obtained when camera work (hereinafter referred to as virtual camera work) specified by the user is performed on the subject. ) is output. Virtual camera work realizes video effects (for example, changing the composition and angle of view) obtained by operating a virtual camera. Pan (left and right), tilt (up and down), roll, zoom in, zoom out. A virtual camerawork video is a video obtained by specifying a virtual camerawork for a subject and trimming (cutting out) a video corresponding to the virtual camerawork from an input captured video.

The display device 3 is a display on which captured images and virtual camerawork images output from the image processing device 2 are output. However, the display device 3 may be a display having not only a display function but also a touch panel function like a tablet terminal.

Next, the video processing device 2 will be explained. FIG. 2 is a block diagram of the video processing device 2. As shown in FIG.

The video processing device 2 includes a captured video input unit 20, a specific person designation receiving unit 21, a virtual camera work designation receiving unit 22, a face image dictionary unit 23, a face recognition unit 24, a skeleton determination unit 25, a person A specifying unit 26 , a composition database 27 , a trimming frame size determining unit 28 , a trimming control unit 29 , and a trimming unit 30 are provided.

The captured image input unit 20 inputs the captured image captured by the camera 1 .

The specific person designation receiving unit 21 receives designation of a person who will be the subject of virtual camerawork from the user. The specific person designation reception unit 21 includes, for example, elements such as a touch operation pad, a mouse, and a keyboard that are directly operated by the user with a finger, as well as a microphone that acquires the user's voice, an acceleration sensor, an angular velocity sensor, an inclination sensor, and a geomagnetic sensor. It can also be realized by an element or the like for detecting motion or posture.

The person who is the subject of the virtual camerawork may be any person who appears in the captured image. In addition, the person who will be the subject of virtual camera work (hereinafter referred to as a specific person) can be specified by directly entering the name of the specific person, or by selecting from the names prepared in advance. , a method of designating a specific person by voice input, and a method of selecting and designating a person in a displayed photographed image by input means such as a touch panel, etc., but not limited to these.

The virtual camera work designation receiving unit 22 receives designation of virtual camera work from the user. The virtual camera work designation reception unit 22 includes, for example, elements such as a touch operation pad, a mouse, and a keyboard that are directly operated by the user's fingers, as well as a microphone that acquires the user's voice, an acceleration sensor, an angular velocity sensor, an inclination sensor, and a geomagnetic sensor. It can also be realized by an element or the like that detects motion or posture.

The virtual camera work can be specified by directly inputting the type of camera work, by selecting from preset camera works, or by voice input. is not limited to

The face image dictionary unit 23 is a storage unit in which image data of the face of a subject photographed by the camera 1, for example, image data of the face of a program performer is registered. The image data is, for example, data of facial features of a subject.

The face recognition unit 24 uses the face recognition dictionary 23 to recognize a person in the captured image. Although the type of recognition method does not matter, there are methods based on pattern matching, recognition based on algorithms obtained by machine learning, and the like. A typical machine learning method is deep learning, but it is not limited to this. The face recognition unit 24 uses these methods to recognize the person in the captured image. More specifically, in the case of a captured image showing three performers as shown in FIG. 3, the faces of the three performers are recognized. In the example of FIG. 3, person A, person B, and person C are recognized in the captured image.

The skeleton determination unit 25 determines the skeleton of the person appearing in the captured image. Skeleton determination methods include, but are not limited to, OpenPose, VisionPose, tf-pose-estimation, and the like. For example, in the case of a photographed video in which three performers are shown as shown in FIG. 4, the skeleton determination unit 25 determines the skeletons of the three performers.

The person identifying unit 26 identifies the specific person accepted by the specific person designation accepting unit 21 from the captured image using the result of the face recognition unit 24, and uses the result of skeleton determination by the skeleton determining unit 25 of the specific person. , to specify the positional relationship of the parts of a specific person on the captured image.

The identification of the positional relationship of the parts of the specific person on the captured image by the person identification unit 26 will be described. FIG. 5 is a diagram for explaining how the person identification unit 26 identifies the positional relationship of the parts of a specific person on the captured image. FIG. 5 shows the case where the specific person is person B, and shows the result of skeleton determination of the parts of person B. FIG. In FIG. 5, the entire length of the person B is defined from the top of the skeleton to the bottom of the skeleton of the legs. Also, in the vertical direction, the face of the person B in the vertical direction is defined from the vertex of the skeleton to the neck of the skeleton. The upper half of the body of the person B is defined from the neck portion of the skeleton to the portion corresponding to the hip joint of the skeleton in the vertical direction. In addition, in the vertical direction, the lower half of the body of the person B is defined from the portion corresponding to the hip joint of the skeleton to the tip of the foot. Also, the width of the whole body of the person B is defined as the maximum range occupied by the skeleton in the horizontal direction. In addition, the width of the face of the person B is defined as the maximum range of the facial skeleton in the lateral direction. In addition, the length from the tip of the right hand in the vertical direction to the elbow joint of the right arm is defined as the length of the vertical right forearm. The length from the tip of the right hand in the lateral direction to the elbow joint of the right arm is defined as the length of the lateral right forearm. The vertical length of the left forearm is defined as the length from the tip of the left hand in the vertical direction to the elbow joint of the left arm. The length from the left hand in the lateral direction to the elbow joint of the left arm is taken as the length of the lateral left forearm. Also, the vertical line passing through the neck of the skeleton is defined as the center line of the skeleton of the person B. FIG. Then, the person identification unit 26 identifies the position and length of each part of the specific person in correspondence with the pixel position of the captured image. The determination of the body part of the specific person by the person identification unit 26 is an example, and the body part to be determined may be changed as appropriate.

The images used for the processing by the face recognition unit 24, the skeleton determination unit 25, and the person identification unit 26 do not necessarily have to be captured images. It is also possible to use an image with a resolution lower than that of the captured image. By using an image with a resolution lower than that of the captured image, the processing speed of the face recognition unit 24, the skeleton determination unit 25, and the person identification unit 26 can be increased.

The composition database 27 is a database in which composition information for each type of virtual camerawork is stored. The composition information is information about the composition of the subject. Specifically, the composition information includes the size of the subject frame (e.g., vertical or horizontal length), the type of trimming frame to be used, the position of the trimming frame (e.g., positional relationship with the skeleton center line), the trimming frame control information for the movement of

A trimming frame size determination unit 28 determines the size of a trimming frame for cutting out (trimming) a subject from a photographed video from the specified virtual camera work, the skeleton of a specific person, and the composition information of the composition database 27.

The trimming control unit 29 controls the position of the trimming frame of the size determined by the trimming frame size determination unit 28 on the video.

The trimming unit 30 trims the image within the trimming frame arranged at a predetermined position on the image under the control of the trimming control unit 29 from the captured image.

Specifically, the video processing device 2 described above can be realized by a computer system having a processor that performs various kinds of arithmetic processing. FIG. 6 is a block diagram of the video processing device 2 configured by a computer system.

As shown in FIG. 6, the video processing apparatus 2 includes a processor 101, a memory (ROM or RAM) 102, a storage device (hard disk, semiconductor disk, etc.) 103, an input device (keyboard, mouse, touch panel, etc.) 104, and a communication device 105. It can be configured by a computer 100 having

The program stored in the storage device 103 is loaded into the memory 102 and executed by the processor 101, so that the video processing device 2 performs a photographed video input process 110, a specific person designation reception process 111, and a virtual camera work designation reception process 112. , face recognition processing 113, skeleton determination processing 114, person identification processing 115, trimming frame size determination processing 116, trimming control processing 117 and trimming processing 118 are realized. Here, the captured image input processing 110 corresponds to the captured image input unit 20, the specific person designation reception processing 111 corresponds to the specific person designation reception unit 21, and the virtual camera work designation reception processing 112 corresponds to the virtual camera work designation reception unit 22. , face recognition processing 113 corresponds to the face recognition unit 24, skeleton determination processing 114 corresponds to the skeleton determination unit 25, person identification processing 115 corresponds to the person identification unit 26, and trimming frame size determination processing 116 corresponds to The trimming control processing 117 corresponds to the trimming control unit 29 , and the trimming processing 118 corresponds to the trimming unit 30 . Also, the face image dictionary unit 23 and the composition database 27 correspond to the storage device 103 . The face image dictionary section 23 and the composition database 27 may be physically provided outside the computer 100 and connected to the computer 100 via a network such as a LAN.

Next, the operation of the video processing device 2 according to the first embodiment will be described.

First, the overall operation of the video processing device 2 will be described.

In the following description, a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrated is assumed. As shown in FIG. 7, the display unit (display device 3) of the tablet terminal (computer 100) has a display unit 40 that displays the shot image, a display unit 41 that displays the virtual camera work image, and a virtual A virtual camerawork designation reception unit 42 (virtual camerawork designation reception unit 22) for designating camerawork and a specific person designation reception unit 43 (specific person designation reception unit 21) for designating a specific person as a subject are displayed. shall be The virtual camera work that can be specified is long shot, up shot, bust shot, waist shot, tilt up, tilt down, and pan. Further, the specific person designation reception unit 43 (the specific person designation reception unit 21) displays the name of the person recognized by the face recognition unit 24. FIG. Instead of or in addition to the person's name, the person's face recognized by the face recognition unit 24 may be displayed.

FIG. 8 is an overall operation flowchart of the video processing device 2, and FIG. 9 is a diagram for explaining the overall operation of the video processing device 2. As shown in FIG.

The captured image input unit 20 inputs a captured image (Step 1). The captured image that has been input is displayed on the display unit 40 .

The face recognition unit 24 recognizes the person on the captured image (Step 2). In the example of FIG. 9, person A, person B, and person C are recognized, and as a result, in the example of FIG. there is

The skeleton determination unit 25 determines the skeleton of the person on the captured image (Step 3). In the example of FIG. 9, the skeletons of person A, person B, and person C are determined.

The user designates a specific person to be a subject by touching the person displayed in the specific person designation acceptance unit 43 while watching the captured image displayed on the display unit 40 (Step 4).

The person identification unit 26 identifies the designated specific person from the recognized persons, and determines the length and position of the part of the specific person in the captured image (Step 5).

The user designates a virtual camerawork by touching any one of the virtual camerawork displayed in the virtual camerawork designation receiving section 42 (Step 6).

The trimming frame size determination unit 28 reads the composition information of the designated virtual camera work from the composition database 27, and determines the trimming frame size using the composition information and the length and position of the specific person's part in the captured video ( Step 7).

The trimming control unit 29 reads the composition information of the virtual camera work, and uses the composition information and the length and position of the specific person's body part in the captured image to trim the trimming frame determined by the trimming frame size determination unit 28 on the captured image. is controlled (Step 8).

The trimming unit 30 trims the image within the trimming frame from the captured image (Step 9), outputs the trimmed image as a virtual camerawork image, and displays it on the display unit 41 (Step 10).

The outline of the operation of the video processing device 2 has been described above.

Next, specific operations of the trimming frame size determination unit 28, the trimming control unit 29, and the trimming unit 30 will be described.

Before describing specific operations, a trimming frame used for trimming will be described. In the first embodiment, the following first trimming frame, second trimming frame, third trimming frame and fourth trimming frame are used.

(1) First Trimming Frame The first trimming frame is used for close-up shots, bust shots, waist shots, etc., and is used for virtual camera work that does not require a margin at the bottom of the frame. A specific first trimming frame is shown in FIG. The features of the first trimming frame are as follows.
・Aspect ratio: 16:9
・Top margin: 0.1 to 0.15 times the length of the vertical frame (preferably 0.15 times the length)
・Left and right margins: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
Bottom Margin: None (2) Second Trimming Frame The second trimming frame is used for long shots and the like, and is used for virtual camera work that requires margins on the top, bottom, left, and right of the frame. A specific second trimming frame is shown in FIG. The characteristics of the second trimming frame are as follows.
・Aspect ratio: 16:9
・Top margin: 0.1 to 0.15 times the length of the vertical frame (preferably 0.15 times the length)
・Left and right margins: 0.05 to 0.1 times the length of the vertical frame (preferably 0.1 times the length)
・Bottom margin: 0.0.5 to 0.1 times the length of the vertical frame (preferably 0.1 times the length)
(3) Third trimming frame The third trimming frame is used for the first or last frame such as tilt-up or tilt-down, and is used for virtual camera work that does not require a margin at the top of the frame. . A specific third trimming frame is shown in FIG. The features of the third trimming frame are as follows.
・Aspect ratio: 16:9
- Top margin: none - Left and right margins: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
・Bottom margin: 0.2 to 0.25 times the length of the vertical frame (preferably 0.25 times the length)
(4) Fourth Trimming Frame The fourth trimming frame is used for frames in the middle of tilt-up and tilt-down, and is used for virtual camera work that does not require margins at the top and bottom of the frame. A specific fourth trimming frame is shown in FIG. The features of the fourth trimming frame are as follows.
・Aspect ratio: 16:9
- Top margin: none - Left and right margins: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
• Bottom Margin: None The features of the first trimming frame, the second trimming frame, the third trimming frame, and the fourth trimming frame are set in the trimming frame size determining section 28 .

Next, operations of the trimming frame size determination unit 28, the trimming control unit 29, and the trimming unit 30 assuming specific virtual camera work will be described.

1. Specific operation when the specific person is "person A (single)" and the virtual camera work is "close-up shot". , the specific person (subject of virtual camera work) accepted by the specific person specification accepting unit 21 is "Person A", and the virtual camera work specified by the virtual camera work specification accepting unit 22 is "close-up shot". Describe a case. Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 determines the skeleton of "person A" from the captured video as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "Person A" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the composition information corresponding to the specific person being "single person" and the virtual camera work being "close-up shot". Here, the composition information corresponding to the read "close-up shot" is as follows.
When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the length of the face in the vertical direction from the skeleton of the person A. Then, as shown in FIG. 15, a temporary frame of the subject frame whose length in the vertical direction is 1.1 times the length of the face is generated. A temporary frame is placed at the vertex and at a position where the vertical center line and the skeleton center line coincide.

Next, as shown in FIG. 15, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the maximum range of the horizontal skeleton of the person A is achieved without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads, from the composition database 27, the composition information related to control among the composition information corresponding to the specific person being "single person" and the virtual camera work being "close-up shot". Here, the read composition information is as follows.
・Arrange the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject match.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to convert the trimming frame of the size determined by the trimming frame size determination unit 28 into a trimming frame as shown in FIG. , and place it on the captured image.

In addition, since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move along with the movement. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and uses the trimmed video as a "close-up shot" of "person A", which is the designated virtual camerawork, as shown in FIG. Output as video. FIG. 17 shows a state in which virtual camerawork images are displayed on the display unit 41 of the tablet terminal (computer 100).

2. Specific operation when the specific person is "person A (single)" and the virtual camera work is "waist shot". The specific person (subject of the virtual camera work) accepted by the specific person specification accepting unit 21 is "Person A", and the virtual camera work specified by the virtual camera work specification accepting unit 22 is "West shot". Describe a case. Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 identifies the skeleton of "person A" from the captured video as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "Person A" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the composition information corresponding to the specific person being "single person" and the virtual camera work being "waist shot". Here, the composition information corresponding to the read "waist shot" is as follows.
・Assuming that the length of the face is 1, the length of the subject frame in the vertical direction is 1.7 times.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the length of the face in the vertical direction from the skeleton of the person A. Then, as shown in FIG. 18, a temporary frame of the subject frame whose length in the vertical direction is 1.7 times the length of the face is generated. A temporary frame is placed at the vertex and at a position where the vertical center line and the skeleton center line coincide.

Next, as shown in FIG. 18, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the maximum range of the horizontal skeleton of the person A is obtained without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads from the composition database 27 the composition information related to the control, among the composition information corresponding to the specific person being “single person” and the virtual camera work being “waist shot”. Here, the read composition information is as follows.
・Arrange the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject match.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to convert the trimming frame of the size determined by the trimming frame size determination unit 28 into a size as shown in FIG. , and place it on the captured image.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and converts the trimmed video to the "waist shot" of "person A", which is the designated virtual camera work, as shown in FIG. Output as video. The state in which the virtual camerawork image is displayed on the display unit 41 of the tablet terminal (computer 100) is omitted because the virtual camerawork image is only the image of the "person A"'s "waist shot".

3. Specific operation when the virtual camera work is "full shot" of "person A (single person)" In the following description, the image processing apparatus 2 is input with a shot image as shown in FIG. A case will be described where the specific person (subject of virtual camera work) accepted by the person designation accepting unit 21 is "person A" and the virtual camera work specified by the virtual camera work designation accepting unit 22 is "full shot". . Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 identifies the skeleton of "person A" from the captured video as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "Person A" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the composition information corresponding to the specific person being "single person" and the virtual camera work being "full shot". Here, the read composition information is as follows.
- The vertical length of a specific person's skeleton (whole body) is taken as the vertical length of the subject frame.
- The trimming frame uses the second trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the vertical length (whole body) of the skeleton of the person A from the skeleton. Then, as shown in FIG. 20, a temporary frame of the subject frame whose length in the vertical direction is the same as the length of the skeleton in the vertical direction of the person A is generated, and the upper side of the temporary frame is the skeleton of the person. A temporary frame is placed at the top vertex and at a position where the vertical center line and the center line of the skeleton coincide.

Next, as shown in FIG. 20, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the skeleton of the person A can be accommodated in the maximum range without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the second trimming frame so as to satisfy the composition information, as shown in FIG. The adjustment of the size of the second trimming frame ensures the minimum margin, and adjusts the size of the margin to be as preferable as possible within a range in which the margin of the second trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads, from the composition database 27, composition information related to control, among composition information corresponding to the specific person being “single person” and the virtual camera work being “full shot”. Here, the read composition information is as follows.
・Arrange the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject match.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to generate a trimming frame of the size determined by the trimming frame size determining unit 28 as shown in FIG. , and place it on the captured image.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and converts the trimmed video to a "full shot" of "person A", which is the designated virtual camera work, as shown in FIG. Output as video. It should be noted that the state in which the virtual camerawork image is displayed on the display unit 41 of the tablet terminal (computer 100) is only the "full shot" image of "person A", so a description thereof will be omitted.

4. Specific operation when the virtual camera work is "full shot" of "person A and person B (two people)" In the following description, the image processing apparatus 2 receives a photographed image as shown in FIG. , the specific persons (subjects of the virtual camera work) accepted by the specific person specification accepting unit 21 are "person A and person B (two people)", and the virtual camera work specified by the virtual camera work specification accepting unit 22 is is a "full shot". Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 uses the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25 to identify the skeletons of "person A" and "person B" from the captured image as shown in FIG. Identify. Also, the vertical skeleton centerlines of the skeletons of "Person A" and "Person B" are determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the vertical composition information among the composition information corresponding to the virtual camerawork "full shot" in which the specific persons are "two people". Here, the read composition information is as follows.
- The length in which all the skeletons of the specific person are included in the vertical direction is set as the vertical length of the subject frame.
• The vertical centerline of the subject frame is located in the middle of the skeletal centerlines of the two persons.
- The trimming frame uses the second trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the vertical length of the skeleton from the skeletons of "person A" and "person B". Then, as shown in FIG. 22, a temporary frame of a subject frame having a length in the vertical direction that is the same length as the length in which all the vertical skeletons of "Person A" and "Person B" are contained is generated, The temporary frame is installed at a position where the upper side of the temporary frame is positioned at the top vertex of the skeleton of the person and the center line in the vertical direction is in the middle of the skeleton center lines of the two people.

Next, as shown in FIG. 22, the trimming frame size determination unit 28 adjusts the horizontal size of the frame so that the skeletons of the persons A and B can be accommodated in the maximum range without changing the length of the frame in the vertical direction. Adjust length. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the second trimming frame so as to satisfy the composition information, as shown in FIG. The adjustment of the size of the second trimming frame ensures the minimum margin, and adjusts the size of the margin to be as preferable as possible within a range in which the margin of the second trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads from the composition database 27 the composition information related to the control among the composition information corresponding to the "full shot" which is the virtual camera work and the specific persons are "two people". Here, the read composition information is as follows.
- Arrange the trimming frame so that the longitudinal centerline of the trimming frame is positioned in the middle of the skeletal centerlines of the two persons.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to convert the trimming frame of the size determined by the trimming frame size determination unit 28 into the trimming frame as shown in FIG. , and place it on the captured image.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and converts the trimmed video into the specified virtual camera work "person A, person B (two people)" as shown in FIG. output as a “full-shot” image. When the virtual camerawork image is displayed on the display unit 41 of the tablet terminal (computer 100), the virtual camerawork image is a "full-shot" image of "person A and person B (two people)." only, so we omit it.

5. Specific operation when the virtual camera work is “person C (single person)” “tilt up” In the following description, the image processing device 2 is input with a shot image as shown in FIG. A case will be described where the specific person (subject of virtual camera work) accepted by the person designation accepting unit 21 is "person C" and the virtual camera work specified by the virtual camera work designation accepting unit 22 is "tilt up". . Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 identifies the skeleton of “person C” from the captured video as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "person C" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the composition information corresponding to the “tilt up” virtual camerawork in which the specific person is “single person”. Here, the read composition information is as follows.
When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
・The length of the subject frame in the vertical direction is the width of the whole body of the person (the maximum horizontal range occupied by the skeleton of the person).
- The third trimming frame is used as the frame used at the start of virtual camera work.
- The first trimming frame is used as the frame used at the end of the virtual camerawork.
- The frame used at the start and end of virtual camera work is the fourth trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the length of the width in the horizontal direction from the skeleton of the person C. FIG. Then, as shown in FIG. 24, a temporary frame is determined so that the horizontal width is the horizontal width of the person C, and the skeleton centerline of the person C and the centerline of the subject frame match. Furthermore, the trimming frame size determining unit 28 generates a subject frame in which the vertical length of the temporary frame is 1.1 times the length of the face.

The trimming frame size determining unit 28 adjusts the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. The widths of the third and fourth trimming frames are the same as the width of the first trimming frame after size adjustment. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads from the composition database 27 the composition information corresponding to the “tilt-up” virtual camera work in which the specific person is “single person”. Here, the read composition information is as follows.
- The third trimming frame is used as the frame used at the start of virtual camera work.
- The first trimming frame is used as the frame used at the end of the virtual camerawork.
- The frame used at the start and end of virtual camera work is the fourth trimming frame.
・Arrange the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the person match.
・After stopping the frame used at the start of the virtual camera work for 1 second, it moves to a spline (slow movement at the beginning and end, fast movement in the middle) along the skeleton centerline of the person.
- The frame used at the end of the virtual camerawork remains stationary.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to generate a trimming frame of the size determined by the trimming frame size determining unit 28 as shown in FIG. , and place it on the captured image.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame that moves from the captured video, and uses the trimmed video as shown in FIG. ” image. It should be noted that the state in which the virtual camerawork image is displayed on the display unit 41 of the tablet terminal (computer 100) is merely a "tilt-up" image of "Person C" who is the virtual camerawork. , omitted.

6. Specific operation when the virtual camera work is "pan" In the following description, the video processing device 2 is input with a photographed video as shown in FIG. , to explain the case of panning in a close-up shot. It is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

When the virtual camera work is "pan", it is necessary to specify the start point and end point. The specified contents of the start point and the end point are a specific person as a subject and a shot (long shot, close-up shot, bust shot, waist shot). In this example, the starting point is a close-up shot of "Person A", so after touching "Pan" in the virtual camera work designation receiving section 42 (virtual camera work designation receiving section 22), the starting point is touched, and " Touch Up (up shot). Subsequently, the user touches "Person A" on the specific person designation reception unit 43 (specific person designation reception unit 21). Next, since the end point is a close-up shot of "Person B", after touching "Pan" of the virtual camera work designation reception unit 42 (virtual camera work designation reception unit 22), the end point is touched, and "Up (Upshot)”. Subsequently, the user touches "Person B" of the specific person designation reception unit 43 (specific person designation reception unit 21). This specifies the start point and end point of the "pan".

First, the person identifying unit 26 and the trimming frame size determining unit 28 operate as follows for "person A" at the starting point.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 determines the skeleton of "person A" from the captured video as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "Person A" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determination unit 28 reads from the composition database 27 the composition information corresponding to the virtual camerawork "pan" in which the specific person at the starting point is "single person" and the shot at the starting point corresponds to "close-up shot". Here, the read composition information is as follows.
When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the length of the face in the vertical direction from the skeleton of the person A. Then, as shown in FIG. 15, a temporary frame of the subject frame whose length in the vertical direction is 1.1 times the length of the face is generated. A temporary frame is placed at the vertex and at a position where the vertical center line and the skeleton center line coincide.

Next, as shown in FIG. 15, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the maximum range of the horizontal skeleton of the person A is achieved without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

Next, the person identification unit 26 and the trimming frame size determination unit 28 operate as follows for "person B" at the end point.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 determines the skeleton of "person B" from the captured image as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "person B" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determination unit 28 reads from the composition database 27 the composition information corresponding to the virtual camerawork "pan" in which the specific person at the end point is "single person" and the shot at the end point is "close-up shot". Here, the read composition information is as follows.
When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determining unit 28 calculates the length of the face in the vertical direction from the skeleton of the person B. Then, as shown in FIG. 26, a temporary frame of the subject frame whose length in the vertical direction is 1.1 times the length of the face is generated. A temporary frame is placed at the vertex and at a position where the vertical center line and the skeleton center line coincide.

Next, as shown in FIG. 26, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the maximum range of the horizontal skeleton of the person A is obtained without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

When the sizes of the trimming frames at the start and end points are determined, the trimming control unit 29 performs virtual camerawork "pan" in which the specific person at the start and end points is "single" and the shot at the start and end points is a "close-up shot". Of the composition information, the composition information relating to control is read out from the composition database 27 . Here, the read composition information is as follows.
Arrange the trimming frame of the specific person specified at the start point so that the vertical center line of the trimming frame coincides with the vertical skeleton center line of the specific person specified at the start point.
Arrange the trimming frame of the specific person specified at the end point so that the vertical center line of the trimming frame coincides with the vertical skeleton center line of the specified person specified at the end point.
・The trimming frame used at the start point is used as the trimming frame between the start point and the end point.
・At the end point, the trimming frame of the specific person specified at the end point is used.
・ After stopping the trimming frame used at the start of virtual camera work for 1 second, move it to a spline (slow at the beginning and end, fast in the middle) from the position of the trimming frame at the start point to the position of the trimming frame just before the end point. do.
・The trimming frame used at the end of the virtual camera work remains stationary.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to convert the trimming frame of the size determined by the trimming frame size determining unit 28 into a trimming frame as shown in FIG. In addition, it is placed on the captured image and controlled.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and uses the trimmed video as a close-up pan from the start point "person A" to the end point "person B" as shown in FIG. Output as a corresponding virtual camera work video.

This completes the description of the first embodiment. It goes without saying that the numerical values of the trimming frame and the composition information described above are examples, and can be changed as appropriate.

Since the first embodiment is configured to recognize a person in a captured image, the user does not need to search for a specific person (object) in the captured image unlike the conventional technology, and thus saves time and effort. can be reduced.

Furthermore, in the first embodiment, the skeleton of a specific person (subject) is determined, the size and position of the part of the specific person on the video are specified, and the information and composition information are used to generate a virtual camerawork video. is generated, it is possible to generate various camera work images.

<Modification 1 of the first embodiment>
The virtual camerawork in the first embodiment described above does not take into consideration the direction of the face of a specific person. However, more natural camerawork can be performed by considering the orientation of a specific person's face. For example, when taking a close-up shot of a specific person, it is possible to obtain a better composition by widening the space in the direction in which the face of the specific person is facing. Therefore, in the modification of the first embodiment, information on the orientation of the face of a specific person is acquired from the face recognition unit 24, and virtual camera work is performed using composition information in consideration of the information on the orientation of the face of the specific person. An example of

In the following description, when the specific person is "Person B (single)" and the virtual camera work is "close-up shot", specific details of the trimming frame size determination unit 28, the trimming control unit 29, and the trimming unit 30 will be described. The explanation will focus on the operation. Then, a case where a photographed image as shown in FIG. 14 is input to the image processing device 2 will be described. Further, it is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

Using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25, the person identification unit 26 determines the skeleton of "person B" from the captured image as shown in FIG. Also, the vertical skeletal centerline of the skeleton of "Person B" is determined. The skeleton centerline is a line passing through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The face recognition unit 24 also outputs information on the orientation of the face of “Person B” to the trimming frame size determination unit 28 .

The trimming frame size determining unit 28 reads from the composition database 27 the composition information corresponding to the specific person being "alone", the virtual camera work being "up shot", and the face direction being "right". Here, the composition information corresponding to the read "close-up shot" is as follows.
When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the subject frame pass through the position of 0.65:0.35 from the left of the trimming frame.

The trimming frame size determining unit 28 calculates the length of the face in the vertical direction from the skeleton of the person B. Then, as shown in FIG. 28, a temporary frame of the subject frame whose length in the vertical direction is 1.1 times the length of the face is generated. A temporary frame is placed at the vertex and at a position where the vertical center line and the skeleton center line coincide.

Next, as shown in FIG. 28, the trimming frame size determining unit 28 adjusts the horizontal length of the temporary frame so that the maximum range of the horizontal skeleton of the person B is achieved without changing the vertical length of the temporary frame. to adjust. A frame generated in this manner is a subject frame.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted at a position where the vertical center line of the subject frame passes through a position of 0.65:0.35 from the left of the trimming frame, and after securing the minimum margin, The margin of the first trimming frame is adjusted so as to have a preferable size as long as it does not overlap the subject frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 reads, from the composition database 27, the composition information related to control among the composition information corresponding to the specific person being "single person" and the virtual camera work being "close-up shot". Here, the read composition information is as follows.
- Arrange the trimming frame so that the vertical skeletal centerline of the specific person passes through the trimming frame at a position of 0.65:0.35 from the left.

The trimming control unit 29 uses the skeleton position information (skeletal center line) and the composition information from the person identifying unit 26 to convert the trimming frame of the size determined by the trimming frame size determining unit 28 into a trimming frame as shown in FIG. , and place it on the captured image.

Note that since the captured video is a moving image, the person may move over time, and the skeletal centerline may also move. Although the position of the trimming frame may be controlled along with the movement of the skeleton centerline, even if the movement of the skeleton centerline is small, if the trimming frame is moved, the image may become difficult for the viewer to view. There is Therefore, if the movement range of the skeleton centerline is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and uses the trimmed video as a "close-up shot" of "person B", which is the designated virtual camera work, as shown in FIG. Output as video.

According to Modification 1 of the first embodiment, as shown in FIG. 27, compared to the "close-up shot" of "person B" that does not consider the orientation of the face, as shown in FIG. A “close-up shot” of “person B”, which takes into consideration the orientation of the person's face, can provide an image with a more ideal composition for human photography. That is, it is possible to obtain a virtual camerawork image that is closer to a camerawork image taken by a human being.

<Modification 2 of the first embodiment>
In the above-described first embodiment, an example of designating only a specific person as a subject has been described. However, the person identification unit 26 determines each part of the specific person. Therefore, in Modified Example 2 of the first embodiment, an example will be described in which not only a specific person but also parts of the specific person are specified, and camerawork with the parts of the specific person as subjects is realized.

As a specific example of Modified Example 2 of the first embodiment, the specific person is "Person B (one person)", the part of the specific person that is the subject is the "right hand", and the virtual camera work is "up". The case of "shot" will be explained. It is assumed that specific composition information read by each unit is stored in the composition database 27 in advance.

First, as a specific method of specifying a part, as shown in FIG. In addition, a site designation section 50 is provided. The part specifying section 50 is a diagram schematically showing the whole body of a person, and the part can be specified by touching the part of the subject. For example, when the specific person is "Person B (one person)" and the body part of the specific person is the "right hand", after touching "Person B" on the specific person designation accepting unit 43, By touching "Right Hand", "Person B" and "Right Hand" can be specified.

When the specific person, the part of the specific person to be the subject, and the virtual camerawork are specified, the person specifying unit 26 uses the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25 to determine the 5, the skeleton of "Person B" is determined from the captured image. Then, the coordinate information of these skeletons is output to the trimming frame size determining section 28 .

The trimming frame size determining unit 28 reads from the composition database 27 composition information corresponding to the specific person being "single person", the subject part being the "right hand", and the virtual camera work being "close-up shot". Here, the read composition information is as follows.
・The intersection of the vertical and horizontal center lines of the subject frame (the center of the subject frame) should be the tip of the right hand.
・The vertical length of the subject frame shall be twice the vertical length of the right forearm.
・The horizontal length of the subject frame is twice the length of the horizontal right forearm.
・The trimming frame uses the first trimming frame.
・Make the vertical center line of the trimming frame coincide with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the lengths of the right forearm in the vertical and horizontal directions from the skeleton of the person B. FIG. Then, as shown in FIG. 31, the vertical length is twice the length of the vertical right forearm, the horizontal length is twice the length of the horizontal right forearm, and the subject frame Adjust the subject frame so that the intersection of the vertical and horizontal center lines (the center of the subject frame) is the tip of the right hand.

The trimming frame size determining unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information, as shown in FIG. The size of the first trimming frame is adjusted so as to obtain a preferable size of the margin as long as the minimum margin is secured and the margin of the first trimming frame does not overlap the object frame. In this manner, the trimming frame size determining unit 28 determines the trimming frame size as shown in FIG.

The trimming control unit 29 stores the composition information related to control among the composition information corresponding to the specific person being "single person", the subject part being "right hand", and the virtual camera work being "close-up shot", to the composition database. 27. Here, the read composition information is as follows.
・Arrange the trimming frame so that the intersection of the vertical and horizontal center lines of the trimming frame is aligned with the tip of the right hand.

The trimming control unit 29 uses the skeleton position information (the coordinates of the tip of the right hand) and the composition information from the person specifying unit 26 to generate a trimming frame of the size determined by the trimming frame size determining unit 28 as shown in FIG. Place on the captured video as shown.

Since the captured image is a moving image, the person may move over time, and the coordinates of the tip of the right hand may also move along with the movement. Although the position of the trimming frame may be controlled according to the movement of the coordinates of the tip of the right hand, even if the movement of the coordinates of the tip of the right hand is small, if the trimming frame is moved, the image may be difficult for the viewer to view. could become Therefore, if the movement range of the coordinates of the tip of the right hand is within a predetermined distance range, control may be performed to maintain the position of the trimming frame.

The trimming unit 30 trims the video within the trimming frame from the captured video, and outputs the trimmed video as a "up shot" video of "person B's right hand" as shown in FIG.

In Modification 2 of the first embodiment, a specific part of a specific person (subject) is determined by using information on the position and length of each part determined using the skeleton of the specific person (subject) and composition information. It is possible to realize virtual camera work with a subject.

<Modification 3 of the first embodiment>
Modification 3 of the first embodiment will be described. FIG. 33 is a block diagram of the information processing device 2 of Modification 3 of the first embodiment.

Modification 3 of the first embodiment is an example in which a video effect is applied to a virtual camerawork video generated by trimming a captured video. Video effects include, for example, focus-in, in which the shooting frame itself does not change, but over time, an unfocused video changes to a subject-focused video, or vice versa, such as focus-out. There is

In order to give these video effects to the virtual camerawork video generated by trimming the captured video, the information processing device 2 of the third modification of the first embodiment further includes a video effect processing unit 31. .

The image effect processing unit 31 acquires information on image effects from the composition information corresponding to the specified virtual camera work from the composition database 27 . Then, the video effect processing unit 31 processes the video effects of the trimmed video output from the trimming unit 30, and outputs a virtual camerawork video.

<Second Embodiment>
A second embodiment will be described.

FIG. 34 is a block diagram of the information processing device 2 of the second embodiment.

The information processing apparatus 2 of the second embodiment further has a function of telopping or superimposing (superimposing) a CG (computer graphics) image or other edited video on the virtual camerawork video. In order to realize the function, the information processing apparatus 2 of the second embodiment includes an image database 50, an image selection unit 51, an adder, in addition to the configuration of the information processing apparatus 2 of the first embodiment. 52.

The image storage unit 50 stores CG images such as logos, edited images and the like (referred to as superimposed images).

The image selection unit 51 is a unit that receives, from the user, selection of a superimposed image to be superimposed on the virtual camerawork video from among the superimposed images stored in the image storage unit 50 . The image selection unit 51 displays, on the display device 3, a superimposition image that can be selected by the user from the superimposition images stored in the image storage unit 50 so that the user can select a superimposition image. Examples of display include identification information for identifying the superimposed image, a thumbnail image of the superimposed image, and the like. The image selection unit 51 also controls the position of the selected superimposed image on the virtual camerawork video and the size of the superimposed image. Position and size control are determined by the user's designation of the position and size on the virtual camerawork video.

The adder 52 adds (superimposes) the selected superimposed image on the virtual camerawork video, and outputs the virtual camerawork video on which the superimposed image is superimposed.

Next, the operation of the information processing device 2 according to the second embodiment will be described. In the following description, as in the first embodiment, a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrated is assumed.

As shown in FIG. 35, the display unit (display device 3) of the tablet terminal (computer 100) has a display unit 40 for displaying captured images, a display unit 41 for displaying virtual camera work images, and a virtual camera work image. In addition to a virtual camerawork designation reception unit 42 (virtual camerawork designation reception unit 22) that designates a specific person designation reception unit 43 (specific person designation reception unit 21) that designates a specific person to be a subject, a superimposed image A thumbnail image 44 (image selection portion 51) for selecting is displayed.

The user selects a superimposed image to be superimposed on the virtual camerawork video displayed on the display unit 41 from the thumbnail images 44 . As for the selection and placement method, as shown in FIG. 36, the thumbnail image to be selected is dragged onto the virtual camerawork video with a finger. Then, drop it at a desired position on the virtual camerawork video. After dropping the superimposed image onto the virtual camerawork video, the size of the superimposed image is determined by pinching in or pinching out the superimposed image as shown in FIG.

In this way, it is possible to generate an image in which the logo and the edited image are superimposed on the virtual camerawork image.

<Third Embodiment>
A third embodiment will be described.

A third embodiment is a switching system. FIG. 38 is a block diagram of the switching system of the third embodiment.

A switching system according to the third embodiment includes a plurality of information processing apparatuses 2 according to the first or second embodiment. Images taken by the camera 1 are input to each information processing device 2, and each information processing device 2 outputs one virtual camerawork image by virtual camerawork selected by the user.

The switching device 4 receives the virtual camerawork video from each information processing device 2, and outputs one virtual camerawork video as a line video according to the user's selection.

Next, the operation of the switching system of the third embodiment will be explained. In the following description, two information processing devices 2 are used, and a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrated as in the first embodiment is assumed. do. That is, the tablet terminal (computer 100) includes two information processing devices 2, a display device 3, and a switching device 4. FIG.

The display unit (display device 3) of the tablet terminal (computer 100) according to the third embodiment displays two types of virtual camerawork images output from the two information processing devices 2. The display unit 41-1 and 41-2 are different from the first and second embodiments. The display unit 41-1 is a display unit that displays line images, and the display unit 41-2 is a display unit that displays preview images that are candidates for line images. The display unit 41-1 displaying the line image has a thicker frame than the display unit 41-2, so that the line image can be easily recognized visually.

In order to obtain a virtual camerawork image that is a candidate for a line image, the user first touches (selects) the screen of the display unit 41-2 with a finger, and then uses the specific person designation reception unit 43 (specific person designation reception unit 21) to A specific person is designated, and virtual camerawork is designated by the virtual camerawork designation reception unit 42 (virtual camerawork designation reception unit 22). Then, the selected display unit 41-2 displays the virtual camera work image corresponding to the designated specific person and the virtual camera work. Here, when switching the virtual camera work image (preview image) displayed on the display unit 41-2 to the line image, by touching the display unit 41-2 to select it and then touching the "TAKE" button. , the virtual camera work image (preview image) displayed on the display unit 41-2 can be output as a line image. At this time, the virtual camera work image (preview image) displayed on the display unit 41-2 is not displayed on the display unit 41-2, and is displayed on the display unit 41-1 where the line image is displayed. become.

Furthermore, in order to obtain a virtual camerawork image that is a candidate for a line image, after touching (selecting) with a finger the screen of the display unit 41-2 where no virtual camerawork image (preview image) is displayed, a specific person is specified. A specific person is designated by the reception unit 43 (specific person designation reception unit 21), and virtual camerawork is designated by the virtual camerawork designation reception unit 42 (virtual camerawork designation reception unit 22). Then, the selected display unit 41-2 displays the virtual camera work image corresponding to the designated specific person and the virtual camera work.

FIG. 40 shows an example in which two types of virtual camerawork images are displayed on the display section 41-1 and the display section 41-2. The virtual camerawork image displayed on the display unit 41-1 is the line image, and the virtual camerawork image displayed on the display unit 41-2 is the preview image. When switching the line image from the virtual camerawork image displayed on the display unit 41-1 to the virtual camerawork image displayed on the display unit 41-2, the display unit 41-2 is touched to select. After that, by touching the "TAKE" button, the virtual camerawork image (preview image) displayed on the display unit 41-2 can be output as a line image. At this time, as shown in FIG. 41, the virtual camera work image (preview image) displayed on the display unit 41-2 is not displayed on the display unit 41-2, and the line image is displayed on the display unit 41. -1 will be displayed.

In the above description, an example of touching (selecting) the screen of the display unit 41-2 with a finger in order to obtain a virtual camerawork image that is a candidate for a line image has been described. This is in order to cope with the case where there are a plurality of display units for displaying preview images. However, as in this example, when there is only one display section for displaying a preview image, it is possible to adopt a configuration in which it is not necessary to select the display section.

According to the third embodiment, a switching system can be realized without requiring a large number of cameras and users (operators).

Although the present invention has been described above with reference to preferred embodiments, it is not necessary to include all the configurations of the embodiments, and not only can they be combined as appropriate, but the present invention is not necessarily limited to the above embodiments. However, it can be modified and implemented in various ways within the scope of its technical ideas.

1 Camera 2 Video processing device 3 Display device 4 Switching device 20 Captured video input unit 21 Specific person designation reception unit 22 Virtual camera work designation reception unit 23 Face image dictionary unit 24 Face recognition unit 25 Skeleton determination unit 26 Person identification unit 27 Composition database 28 trimming frame size determination unit 29 trimming control unit 30 trimming unit 31 image effect processing unit 50 image database 51 image selection unit 52 adder

Claims (13)

an acquisition unit that acquires an image ;
a specific person designation reception unit that designates a specific person;
a camerawork designation reception unit that designates camerawork;
a person recognition unit that recognizes a person in the video;
a skeleton determination unit that determines a skeleton of a person in the video using the video;
identifying a specific person in the video using the recognition result of the person recognition unit; using the skeleton of the specific person among the skeletons determined by the skeleton determination unit; a person identification unit that identifies a positional relationship;
Trimming frame size determination for determining a size of a trimming frame for trimming a portion of the video by using the positional relationship of the parts of the specific person on the video and the composition information corresponding to the specified camerawork. Department and
Trimming for controlling the position of the video in the trimming frame corresponding to the specified camerawork , using the positional relationship of the part of the specific person on the video and the composition information corresponding to the specified camerawork . a frame controller;
a trimming unit that trims a video within the trimming frame from the video and outputs the trimmed video as a camerawork video corresponding to the camerawork ;
A video processing device having A composition information storage unit storing composition information in which a composition corresponding to the camera work is determined,
The video processing device according to claim 1. at least one of the person recognition unit and the skeleton determination unit recognizes a person or determines a skeleton using an image with a resolution lower than that of the image ;
3. The video processing device according to claim 1 or 2. The person recognition unit determines the orientation of the face of the specific person,
The trimming frame control unit controls the position of the trimming frame in consideration of the orientation of the face of the specific person .
The video processing device according to any one of claims 1 to 3. Having a part designation reception unit that receives designation of the part of the specific person ,
The video processing device according to any one of claims 1 to 4. An image processing unit that processes the image output from the trimming unit and outputs a pseudo focus-in or focus-out image ,
The video processing device according to any one of claims 1 to 5. a superimposed image acquisition unit that acquires a superimposed image to be superimposed on the camerawork video;
a superimposing unit that superimposes a superimposed image on the camerawork video ;
7. The video processing device according to any one of claims 1 to 6, comprising: Having at least two video processing devices, a display unit, and a switching unit,
The video processing device is
an acquisition unit that acquires an image ;
a specific person designation reception unit that designates a specific person;
a camerawork designation reception unit that designates camerawork ;
a person recognition unit that recognizes a person in the video ;
a skeleton determination unit that determines a skeleton of a person in the video using the video ;
Identifying a specific person in the image using the recognition result of the person recognition unit, and using the skeleton of the specific person among the skeletons determined by the skeleton determination unit, the position of the part of the specific person on the image a person identification unit that identifies a relationship;
a trimming frame size determining unit that determines a trimming frame size for trimming a portion of the video by using the positional relationship of the specific person's body parts on the video and the composition information corresponding to the specified camerawork ; ,
Trimming for controlling the position of the video in the trimming frame corresponding to the specified camerawork , using the positional relationship of the part of the specific person on the video and the composition information corresponding to the specified camerawork . a frame controller;
a trimming unit that trims a video within the trimming frame from the video and outputs the trimmed video as a camerawork video corresponding to the camerawork ;
has
The display unit
displaying the image and at least two or more of the camerawork images ;
The switching unit is
Outputting one camerawork video specified by a user out of at least two camerawork videos;
switching system. The display unit distinguishes between camerawork images output as line images and other camerawork images, and displays them.
9. A switching system according to claim 8. an acquisition process for acquiring an image ;
Specific person designation reception processing for receiving designation of a specific person;
camerawork designation reception processing for receiving designation of camerawork ;
Person recognition processing for recognizing a person in the video ;
Skeleton determination processing for determining a skeleton of a person in the video using the video ;
Identifying a specific person in the image using the recognition result of the person recognition processing, and using the skeleton of the specific person among the skeletons determined by the skeleton determination processing, the position of the part of the specific person on the image Person identification processing to identify relationships;
Trimming frame size determination processing for determining the size of a trimming frame for trimming a portion of the video using the positional relationship of the specific person's parts on the video and the composition information corresponding to the specified camerawork. and,
controlling the position of the video of the trimming frame corresponding to the specified camerawork using the positional relationship of the parts of the specific person on the video and the composition information corresponding to the specified camerawork; trimming frame control processing;
A program for causing a computer to perform a trimming process of trimming the video within the trimming frame from the video and outputting the trimmed video as a camerawork video corresponding to the specified camerawork . an acquisition process for acquiring an image ;
Specific person designation processing for accepting designation of a specific person;
a camerawork specification process for accepting a camerawork specification ;
Person recognition processing for recognizing a person in the video ;
Skeleton determination processing for determining a skeleton of a person in the video using the video ;
A specific person in the video is identified using the recognition result of the person recognition process, and a position of the specific person's whole body on the video is determined using the skeleton of the specific person among the skeletons determined by the skeleton determination process. Person identification processing to identify relationships;
trimming frame size determination for determining a size of a trimming frame for trimming a portion of the video using positional relationship of the whole body of the specific person on the video and composition information corresponding to the specified camera work; processing;
Trimming for controlling the position of the video in the trimming frame corresponding to the specified camerawork , using the positional relationship on the video of the whole body of the specific person and the composition information corresponding to the specified camerawork frame control processing;
a trimming process of trimming the video within the trimming frame from the video and outputting the trimmed video as a camerawork video corresponding to the specified camerawork ;
display processing for displaying the image and at least two or more camerawork images corresponding to at least two or more specific persons or at least two or more camerawork images ; A switching process that outputs one camerawork video specified by
A program that makes a computer run The computer
get the video ,
Receiving the designation of a specific person ,
Receiving the designation of camera work ,
recognizing a person in the video ;
using the video to determine the skeleton of a person in the video;
Identifying a specific person in the image using the recognition result of the person, using the skeleton of the specific person among the determined skeletons to specify the positional relationship of the parts of the specific person on the image,
determining the size of a trimming frame for trimming a portion of the video using the positional relationship of the parts of the specific person on the video and the composition information corresponding to the camerawork ;
controlling the position of the video of the trimming frame corresponding to the specified camerawork using the positional relationship of the parts of the specific person on the video and the composition information corresponding to the specified camerawork; ,
trimming the video within the trimming frame from the video , and outputting the trimmed video as a camerawork video corresponding to the specified camerawork ;
video processing method. The computer
get the video ,
Receiving the designation of a specific person ,
Receiving the designation of camera work ,
recognizing a person in the video ;
using the video to determine the skeleton of a person in the video;
Identifying a specific person in the image using the recognition result of the person, using the skeleton of the specific person among the determined skeletons to specify the positional relationship of the whole body of the specific person on the image,
determining the size of a trimming frame for trimming a portion of the video using the positional relationship of the whole body of the specific person on the video and the composition information corresponding to the specified camera work ;
controlling the position of the video of the trimming frame corresponding to the specified camerawork using the positional relationship of the whole body of the specific person on the video and the composition information corresponding to the specified camerawork; ,
trimming the video within the trimming frame from the video , and outputting the trimmed video as a camerawork video corresponding to the specified camerawork ;
displaying the image and at least two or more camerawork images corresponding to at least two or more specific persons or at least two or more camerawork ;
outputting one camerawork video specified by the user out of at least two or more displayed camerawork videos ;
switching method.

Images