JP2021119686A - Information processing apparatus, switching system, program and method - Google Patents

Abstract

To automatically generate videos shot with various kinds of camera works.SOLUTION: A video processing apparatus according to the present invention has: an acquiring unit for acquiring an image captured by a camera; a specific-person designation receiving unit for designating a specific person; a virtual camera work designation receiving unit for designating a virtual camera work by trimming a part of the captured image; a person recognizing unit for recognizing a person; a skeleton frame determining unit for determining a skeleton frame of the person; a person specifying unit for specifying a positional relationship in a captured image of a part of the specific person by using the skeleton frame of the specific person by identifying a specific person based on the result of the person specifying unit; a composition information storing unit for storing composition information with respect to each virtual camera work; a trimming frame size determining unit for trimming a part of the captured image by using the positional relationship in the captured image of the part of the specific person and the composition information; a trimming control unit for controlling a positional relationship of the part of a specific person on the captured image and a position of the captured image of the trimmed frame; and a trimming unit for outputting the trimmed image as a virtual camera work image.SELECTED DRAWING: Figure 2

Description

The present invention relates to information processing devices, switching systems, programs and methods.

In recent years, the number of pixels in cameras has increased, and it has become possible to acquire high-quality images such as 4K and 8K. Then, a technique has been proposed in which a moving image that imitates camera work such as ups and pans and tilts is generated by cutting out a part of a high-quality image without requiring special equipment (for example, a patent). Document 1).

Japanese Unexamined Patent Publication No. 2016-28539

However, in the above technique, it is necessary for the user to specify the subject to be cut out in the video.

Further, it is necessary for the user to specify the size and position of the image to be cut out by the camera work, and it is not possible to automatically cut out the desired image of the camera work. In particular, there are various shots such as full shots, bust shots, and waist shots, but such advanced camera work could not be performed automatically.

Therefore, the present invention is to provide an information processing device, a switching system, a program and a method capable of automatically generating a wide variety of images of camera work.

One aspect of the present invention is a virtual acquisition unit that acquires a photographed image captured by a camera, a specific person designation reception unit that specifies a specific person to be a subject, and a part of the photographed image is trimmed. A virtual camera work designation reception unit that specifies a virtual camera work that realizes a video effect by operating a camera, a person recognition unit that recognizes a person in the captured image, and a person in the image using the captured image. The skeleton determination unit for determining the skeleton and the recognition result of the person recognition unit are used to identify a specific person in the image, and among the skeletons determined by the skeleton determination unit, the skeleton of the specific person is used. A person identification unit that specifies the positional relationship on the captured image of a specific person's part, a composition information storage unit that stores composition information for which a composition is defined for each virtual camera work, and an image of the specific person's part. A trimming frame size determination unit that determines the size of a trimming frame that trims a part of the captured image using the positional relationship on the image and the composition information corresponding to the specified virtual camera work, and the specific person. Trimming that controls the position of the captured image of the trimming frame corresponding to the specified virtual camera work by using the positional relationship of the portion on the captured image and the composition information corresponding to the specified virtual camera work. It is a video processing device having a frame control unit and a trimming unit that trims an image in the trimming frame from the captured image and outputs the trimmed trimmed image as a virtual camerawork image by the virtual camera work.

One aspect of the present invention includes at least two or more image processing devices, a display unit, and a switching unit, and the image processing device serves as an acquisition unit for acquiring a captured image captured by a camera and a subject. A specific person designation reception unit for designating a specific person, a virtual camera work designation reception unit for designating a virtual camera work that realizes a video effect by operating a virtual camera by trimming a part of the shot image, and the above. A person recognition unit that recognizes a person in a captured image, a skeleton determination unit that determines the skeleton of a person in the image using the captured image, and identification in the image using the recognition result of the person recognition unit. A person identification unit that identifies a person and uses the skeleton of the specific person among the skeletons determined by the skeleton determination unit to specify the positional relationship of the specific person's part on the captured image, and each of the virtual camera works. Using the composition information storage unit that stores the composition information for which the composition is defined, the positional relationship of the specific person's part on the captured image, and the composition information corresponding to the designated virtual camera work. The trimming frame size determining unit that determines the size of the trimming frame that trims a part of the captured image, the positional relationship of the specific person's part on the captured image, and the composition information corresponding to the specified virtual camera work. A trimming frame control unit that controls the position of the captured image of the trimming frame corresponding to the specified virtual camera work, and a trimmed image obtained by trimming the image in the trimming frame from the captured image. The display unit has a trimming unit that outputs the image as a virtual camera work image by the virtual camera work, the display unit displays the captured image and at least two or more of the virtual camera work images, and the switching unit is a switching unit. It is a switching system that outputs one virtual camera work image specified by a user among at least two or more virtual camera work images.

One aspect of the present invention is a virtual acquisition process for acquiring a captured image captured by a camera, a specific person designation reception process for accepting the designation of a specific person to be a subject, and trimming a part of the captured image to create a virtual image. A virtual camera work designation reception process that accepts a designation of a virtual camera work that realizes a video effect by operating a different camera, a person recognition process that recognizes a person in the shot video, and a person recognition process that recognizes the person in the shot video, and the shot video is used in the video. The skeleton determination process for determining the skeleton of a person and the recognition result of the person recognition process are used to identify a specific person in the image, and the skeleton of the specific person among the skeletons determined by the skeleton determination process is used. Of the person identification process for specifying the positional relationship of the specific person's part on the photographed image, the positional relationship of the specific person's part on the photographed image, and the composition information whose composition is determined for each virtual camera work. The trimming frame size determination process for determining the size of the trimming frame for trimming a part of the captured image using the composition information corresponding to the designated virtual camera work, and the captured image of the specific person's part. Using the positional relationship and the composition information corresponding to the specified virtual camera work, a trimming frame control process for controlling the position of the captured image of the trimming frame corresponding to the specified virtual camera work, and the shooting This is a program for causing a computer to perform a trimming process of trimming an image in the trimming frame from an image and outputting the trimmed image as a virtual camera work image by the virtual camera work.

One aspect of the present invention is a virtual acquisition process for acquiring a captured image captured by a camera, a specific person designation process for accepting the designation of a specific person as a subject, and a virtual trimming of a part of the captured image. Using the virtual camera work designation process that accepts the designation of the virtual camera work that realizes the image effect by operating the camera, the person recognition process that recognizes the person in the captured image, and the photographed image, the person in the image is The specific person in the image is identified by using the skeleton determination process for determining the skeleton and the recognition result of the person recognition process, and the skeleton of the specific person among the skeletons determined by the skeleton determination process is used. Of the person identification process that specifies the positional relationship of the whole body of a specific person on the captured image, the positional relationship of the specific person on the captured image of the whole body, and the composition information in which the composition of the subject is determined for each virtual camera work. The trimming frame size determination process for determining the size of the trimming frame for trimming a part of the captured image using the composition information corresponding to the designated virtual camera work, and the captured image of the whole body of the specific person. Using the positional relationship and the composition information corresponding to the specified virtual camera work, a trimming frame control process for controlling the position of the captured image of the trimming frame corresponding to the specified virtual camera work, and the shooting A trimming process that trims the image in the trimming frame from the image and outputs the trimmed image as a virtual camera work image by the virtual camera work, and the captured image and at least two or more specific persons or at least two or more. Display at least two or more virtual camera work images corresponding to virtual camera work Output one virtual camera work image specified by the user out of the display processing and the displayed at least two or more virtual camera work images. A program that causes a computer to perform switching processing.

In one aspect of the present invention, the computer acquires a photographed image taken by the camera, accepts the designation of a specific person as a subject, trims a part of the photographed image, and operates the image by a virtual camera. Accepts the designation of virtual camera work to realize the effect, recognizes the person in the captured image, determines the skeleton of the person in the image using the photographed image, and uses the recognition result of the person to determine the image. The specific person in the image is identified, the positional relationship of the specific person's part on the captured image is specified by using the skeleton of the specific person among the determined skeletons, and the specific person's part is on the photographed image. The size of the trimming frame for trimming a part of the captured image is determined by using the positional relationship and the composition information corresponding to the specified virtual camera work among the composition information whose composition is determined for each virtual camera work. The captured image of the trimming frame corresponding to the specified virtual camera work is used by determining the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work. This is a video processing method in which the position of the image is controlled, the image in the trimming frame is trimmed from the captured image, and the trimmed image is output as a virtual camera work image by the virtual camera work.

In one aspect of the present invention, the computer acquires a photographed image taken by the camera, accepts the designation of a specific person as a subject, trims a part of the photographed image, and operates the image by a virtual camera. Accepts the designation of virtual camera work to realize the effect, recognizes the person in the captured image, determines the skeleton of the person in the image using the photographed image, and uses the recognition result of the person to determine the image. The specific person in the image is identified, and the skeleton of the specific person is used among the determined skeletons to identify the positional relationship on the photographed image of the whole body of the specific person. A trimming frame that trims a part of the captured image using the positional relationship and the composition information corresponding to the specified virtual camera work among the composition information in which the composition of the subject is determined for each virtual camera work. The size is determined, and the trimming frame corresponding to the specified virtual camera work is used with the positional relationship on the captured image of the whole body of the specific person and the composition information corresponding to the specified virtual camera work. The position of the captured image is controlled, the image in the trimming frame is trimmed from the captured image, and the trimmed image is output as a virtual camera work image by the virtual camera work, and the photographed image and at least two or more images are output. At least two or more virtual camera work images corresponding to a specific person or at least two or more virtual camera works are displayed, and one of the displayed at least two virtual camera work images, one virtual specified by the user. This is a switching method that outputs camera work images.

The present invention can automatically generate a wide variety of camera work images.

FIG. 1 is a block diagram showing the overall configuration of the first embodiment. FIG. 2 is a block diagram of the video processing device 2. FIG. 3 is a diagram for explaining face recognition. FIG. 4 is a diagram for explaining the skeleton determination. FIG. 5 is a diagram for explaining the identification of the positional relationship on the captured image of the portion of the specific person by the person identification unit 26. FIG. 6 is a block diagram of a video processing device 2 configured by a computer system. FIG. 7 is a diagram for explaining a display unit of the tablet terminal (computer 100). FIG. 8 is an overall operation flowchart of the video processing device 2. FIG. 9 is a diagram for explaining the overall operation of the video processing device 2. FIG. 10 is a diagram for explaining the first trimming frame. FIG. 11 is a diagram for explaining a second trimming frame. FIG. 12 is a diagram for explaining a third trimming frame. FIG. 13 is a diagram for explaining a fourth trimming frame. FIG. 14 is a diagram for explaining the operation of the first embodiment. FIG. 15 is a diagram for explaining the operation of the first embodiment. FIG. 16 is a diagram for explaining the operation of the first embodiment. FIG. 17 is a diagram for explaining the operation of the first embodiment. FIG. 18 is a diagram for explaining the operation of the first embodiment. FIG. 19 is a diagram for explaining the operation of the first embodiment. FIG. 20 is a diagram for explaining the operation of the first embodiment. FIG. 21 is a diagram for explaining the operation of the first embodiment. FIG. 22 is a diagram for explaining the operation of the first embodiment. FIG. 23 is a diagram for explaining the operation of the first embodiment. FIG. 24 is a diagram for explaining the operation of the first embodiment. FIG. 25 is a diagram for explaining the operation of the first embodiment. FIG. 26 is a diagram for explaining the operation of the first embodiment. FIG. 27 is a diagram for explaining the operation of the first embodiment. FIG. 28 is a diagram for explaining the operation of the modified example of the first embodiment. FIG. 29 is a diagram for explaining the operation of the modified example of the first embodiment. FIG. 30 is a diagram for explaining a modified example of the first embodiment. FIG. 31 is a diagram for explaining the operation of the modified example of the first embodiment. FIG. 32 is a diagram for explaining the operation of the modified example of the first embodiment. FIG. 33 is a block diagram of the information processing apparatus 2 of the modified example of the first embodiment. FIG. 34 is a block diagram of the information processing apparatus 2 according to the second embodiment. FIG. 35 is a diagram for explaining the operation of the second embodiment. FIG. 36 is a diagram for explaining the operation of the second embodiment. FIG. 37 is a diagram for explaining the operation of the second embodiment. FIG. 38 is a block diagram of the switching system of the third embodiment. FIG. 39 is a diagram for explaining the operation of the third embodiment. FIG. 40 is a diagram for explaining the operation of the third embodiment. FIG. 41 is a diagram for explaining the operation of the third embodiment.

<First Embodiment>
The first embodiment will be described.

FIG. 1 is a block diagram showing the overall configuration of the first embodiment. In FIG. 1, 1 is a camera, 2 is a video processing device, and 3 is a display device.

The camera 1 is a camera that shoots a program. As a general rule, one camera 1 shoots a program with a wide angle of view so that all the subjects (for example, the performers of the program) can be captured. In the present embodiment, as will be described later, the camera 1 generates a virtual camera work image by trimming a part of the image captured by the camera 1 (hereinafter, referred to as a captured image). A 4K or 8K camera capable of capturing high-quality images is preferable, but the present invention is not limited thereto.

The image processing device 2 inputs an image captured by the camera 1 and performs an image obtained when a user-designated camera work (hereinafter referred to as virtual camera work) is performed on the subject (hereinafter, virtual camera work image). Is described) is output. Virtual camera work realizes video effects obtained by operating a virtual camera (for example, changing the composition and angle of view of the video), for example, up shot, bust shot, waist shot, full shot, etc. There are pan (left and right), tilt (up and down), roll, zoom in, zoom out, etc. The virtual camera work image is an image obtained by trimming (cutting out) an image corresponding to the virtual camera work from the input captured image by designating the virtual camera work for the subject.

The display device 3 is a display that outputs a captured image and a virtual camera work image output from the image processing device 2. However, the display device 3 may be a display having not only a display function but also a touch panel function such as a tablet terminal.

Next, the video processing apparatus 2 will be described. FIG. 2 is a block diagram of the video processing device 2.

The image processing device 2 includes a captured image input unit 20, a specific person designation reception unit 21, a virtual camera work designation reception unit 22, a face image dictionary unit 23, a face recognition unit 24, a skeleton determination unit 25, and a person. It includes a specific unit 26, a composition database 27, a trimming frame size determination unit 28, a trimming control unit 29, and a trimming unit 30.

The captured image input unit 20 inputs the captured image captured by the camera 1.

The specific person designation reception unit 21 receives from the user the designation of the person who is the subject of the virtual camera work. The specific person designation reception unit 21 includes, for example, elements such as a touch operation pad, a mouse, and a keyboard that the user directly operates with a finger, as well as a microphone that acquires the user's voice, an acceleration sensor, an angular velocity sensor, an inclination sensor, and a geomagnetic sensor. It can also be realized by an element that detects movement or posture.

The person who is the subject of the virtual camera work may be the person shown in the captured image. In addition, the person to be the subject of virtual camera work (hereinafter referred to as a specific person) can be specified by directly inputting the name of the specific person or by selecting from the names prepared in advance. , A method of designating a specific person by voice input, a method of selecting and designating a person in the displayed captured image by an input means such as a touch panel, and the like, but the present invention is not limited to these.

The virtual camera work designation reception unit 22 receives the designation of the virtual camera work from the user. The virtual camera work designation reception unit 22 includes not only elements directly operated by the user such as a touch operation pad, mouse, and keyboard, but also a microphone that acquires the user's voice, an acceleration sensor, an angular velocity sensor, an inclination sensor, and a geomagnetic sensor. It can also be realized by an element that detects movement or posture.

The virtual camera work can be specified by directly inputting the type of camera work, selecting from the camera work prepared in advance, and specifying the camera work by voice input. Not limited to.

The face image dictionary unit 23 is a storage unit in which image data of the face of a subject photographed by the camera 1, for example, image data of the face of a performer of a program is registered. The image data is data such as the feature amount of the face of the subject.

The face recognition unit 24 recognizes a person in the captured image by using the face recognition dictionary 23. The type of recognition method does not matter, but there are methods such as pattern matching and recognition by algorithms obtained by machine learning. Deep learning is a typical method of machine learning, but it is not limited to this. The face recognition unit 24 recognizes a person in the captured image by using these methods. More specifically, in the case of a photographed video showing three performers as shown in FIG. 3, the faces of the three performers are recognized. In the example of FIG. 3, the person A, the person B, and the person C in the captured image are recognized.

The skeleton determination unit 25 determines the skeleton of the person shown in the captured image. The skeleton determination method includes, for example, OpenPose, VisionPose, tf-pose-estimation, and the like, but is not limited to these. The skeleton determination unit 25 determines the skeletons of the three performers, for example, in the case of a photographed video showing the three performers as shown in FIG.

The person identification unit 26 identifies a specific person received by the specific person designation reception unit 21 from the captured video using the result of the face recognition unit 24, and uses the result of the skeleton determination by the skeleton determination unit 25 of the specific person. , Specify the positional relationship of the part of a specific person on the captured image.

The identification of the positional relationship on the captured image of the part of the specific person by the person identification unit 26 will be described. FIG. 5 is a diagram for explaining the identification of the positional relationship on the captured image of the portion of the specific person by the person identification unit 26. FIG. 5 shows the case where the specific person is the person B, and shows the result of the skeleton determination of the part of the person B. In FIG. 5, the whole body in the vertical direction of the person B is from the apex of the skeleton to the bottom of the skeleton of the foot. Further, in the vertical direction, the face from the apex of the skeleton to the neck of the skeleton is the face of the person B in the vertical direction. Further, in the vertical direction, the portion from the neck of the skeleton to the portion corresponding to the hip joint of the skeleton is the upper body of the person B. Further, in the vertical direction, the lower body of the person B is from the portion corresponding to the hip joint of the skeleton to the tip of the foot. Further, the maximum range occupied by the skeleton in the lateral direction is defined as the width of the whole body of the person B. Further, the maximum range occupied in the horizontal direction in the skeleton of the face is defined as the width of the face of the person B. The length from the right hand in the vertical direction to the elbow joint of the right arm is defined as the length of the vertical right forearm. The length from the right hand in the lateral direction to the elbow joint of the right arm is defined as the length of the lateral right forearm. The length from the left hand in the vertical direction to the elbow joint of the left arm is defined as the length of the vertical left forearm. The length from the left hand in the lateral direction to the elbow joint of the left arm is defined as the length of the lateral left forearm. Further, the perpendicular line passing through the neck of the skeleton is defined as the skeleton center line of the person B. Then, the person identification unit 26 specifies the position and length of each part of the specific person in correspondence with the pixel position of the captured image. The determination of the part of the specific person by the above person identification unit 26 is an example, and the part to be determined may be changed in a timely manner.

The image used for processing by the face recognition unit 24, the skeleton determination unit 25, and the person identification unit 26 does not necessarily have to be a captured image. It is also possible to use an image having a lower resolution than the captured image. By using an image having a resolution lower than that of the captured image, it is possible to obtain an effect that the processing speed of the face recognition unit 24, the skeleton determination unit 25, and the person identification unit 26 becomes faster.

The composition database 27 is a database in which composition information for each type of virtual camera work is stored. The composition information is information on what kind of composition the subject is projected on. Specifically, the composition information includes the size of the subject frame (for example, vertical or horizontal length), the type of trimming frame to be used, the position of the trimming frame (for example, the positional relationship with the skeleton center line), and the trimming frame. There is control information of the movement of.

The trimming frame size determination unit 28 determines the size of the trimming frame for cutting out (trimming) the subject from the captured image from the designated virtual camera work, the skeleton of the specific person, and the composition information of the composition database 27.

The trimming control unit 29 controls the position of the trimming frame of the size determined by the trimming frame size determination unit 28 on the image.

The trimming unit 30 trims the image in the trimming frame, which is controlled by the trimming control unit 29 and arranged at a predetermined position on the image, from the captured image.

Specifically, the video processing device 2 described above can be realized by a computer system having a processor that performs various arithmetic processes and the like. FIG. 6 is a block diagram of a video processing device 2 configured by a computer system.

As shown in FIG. 6, the image processing device 2 includes a processor 101, a memory (ROM or RAM) 102, a storage device (hard disk, semiconductor disk, etc.) 103, an input device (keyboard, mouse, touch panel, etc.) 104, and a communication device 105. It can be configured by the computer 100 having the computer 100.

In the image processing device 2, the program stored in the storage device 103 is loaded into the memory 102 and executed by the processor 101, whereby the captured image input process 110, the specific person designation reception process 111, and the virtual camera work designation reception process 112. , Face recognition process 113, skeleton determination process 114, person identification process 115, trimming frame size determination process 116, trimming control process 117, and trimming process 118 are realized. Here, the captured video input process 110 corresponds to the captured video input unit 20, the specific person designation reception process 111 corresponds to the specific person designation reception unit 21, and the virtual camera work designation reception process 112 corresponds to the virtual camera work designation reception unit 22. The face recognition process 113 corresponds to the face recognition unit 24, the skeleton determination process 114 corresponds to the skeleton determination unit 25, the person identification process 115 corresponds to the person identification unit 26, and the trimming frame size determination process 116 corresponds to. The trimming frame size determination unit 28 corresponds, the trimming control process 117 corresponds to the trimming control unit 29, and the trimming process 118 corresponds to the trimming unit 30. Further, the face image dictionary unit 23 and the composition database 27 correspond to the storage device 103. The face image dictionary unit 23 and the composition database 27 may be physically provided externally to the computer 100 and connected to the computer 100 via a network such as a LAN.

Next, the operation of the video processing device 2 in the first embodiment will be described.

First, the overall operation of the video processing device 2 will be described.

In the following description, a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrally configured will be described. Then, as shown in FIG. 7, the display unit (display device 3) of the tablet terminal (computer 100) includes a display unit 40 on which the captured image is displayed, a display unit 41 on which the virtual camera work image is displayed, and a virtual image. A virtual camera work designation reception unit 42 (virtual camera work designation reception unit 22) for designating camera work and a specific person designation reception unit 43 (specific person designation reception unit 21) for designating a specific person to be a subject are displayed. It shall be. The virtual camera work that can be specified is long shot, up shot, bust shot, waist shot, tilt up, tilt down, and pan. Further, the specific person designation reception unit 43 (specific person designation reception unit 21) shall display the name of the person recognized by the face recognition unit 24. In addition to or in addition to the name of the person, the face of the person recognized by the face recognition unit 24 may be displayed.

FIG. 8 is an overall operation flowchart of the video processing device 2, and FIG. 9 is a diagram for explaining the overall operation of the video processing device 2.

The captured image input unit 20 inputs a captured image (Step 1). The input captured image is displayed on the display unit 40.

The face recognition unit 24 recognizes a person on the captured image (Step 2). In the example of FIG. 9, the person A, the person B, and the person C are recognized, and as a result, in the example of FIG. 9, the recognized person A, the person B, and the person C are displayed as the specific person designation unit 43. There is.

The skeleton determination unit 25 determines the skeleton of a person on the captured image (Step 3). In the example of FIG. 9, the skeletons of the person A, the person B, and the person C are determined.

The user designates a specific person as a subject by touching the person displayed on the specific person designation reception unit 43 while viewing the captured image displayed on the display unit 40 (Step 4).

The person identification unit 26 identifies the designated specific person from the recognized person, and determines the length and position of the part of the specific person in the captured image (Step 5).

The user designates the virtual camera work by touching any of the virtual camera works displayed on the virtual camera work designation reception unit 42 (Step 6).

The trimming frame size determination unit 28 reads the composition information of the virtual camera work specified from the composition database 27, and determines the trimming frame size using the composition information and the length and position in the captured image of the part of the specific person (the trimming frame size determination unit 28). Step7).

The trimming control unit 29 reads out the composition information of the virtual camera work, and uses the composition information and the length and position in the captured image of the part of the specific person on the captured image of the trimming frame determined by the trimming frame size determining unit 28. Control the position in (Step 8).

The trimming unit 30 trims the image in the trimming frame from the captured image (Step 9), outputs the trimmed image as a virtual camera work image, and displays it on the display unit 41 (Step 10).

The above is the general operation of the video processing device 2.

Next, specific operations of the trimming frame size determining unit 28, the trimming control unit 29, and the trimming unit 30 will be described.

Prior to the description of the specific operation, the trimming frame used for trimming will be described. In the first embodiment, the following first trimming frame, second trimming frame, third trimming frame and fourth trimming frame are used.

(1) First Trimming Frame The first trimming frame is used for up shots, bust shots, waist shots, etc., and is used for virtual camera work that does not require a margin (margin) at the bottom of the frame. A specific first trimming frame is shown in FIG. The features of the first trimming frame are as follows.
・ Aspect ratio: 16: 9
-Upper margin: 0.1 to 0.15 times longer than the vertical frame length (preferably 0.15 times longer)
-Left and right margin: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
-Lower margin: None (2) Second trimming frame The second trimming frame is used for long shots, etc., and is used for virtual camera work that requires margins (margins) on the top, bottom, left, and right of the frame. A specific second trimming frame is shown in FIG. The features of the second trimming frame are as follows.
・ Aspect ratio: 16: 9
-Upper margin: 0.1 to 0.15 times longer than the vertical frame length (preferably 0.15 times longer)
-Left and right margin: 0.05 to 0.1 times the vertical frame length (preferably 0.1 times the length)
-Lower margin: 0.0.5 to 0.1 times the vertical frame length (preferably 0.1 times the length)
(3) Third trimming frame The third trimming frame is used for the first or last frame such as tilt-up or tilt-down, and is used for virtual camera work that does not require a margin at the top of the frame. .. A specific third trimming frame is shown in FIG. The features of the third trimming frame are as follows.
・ Aspect ratio: 16: 9
-Upper margin: None-Left and right margin: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
-Lower margin: 0.2 to 0.25 times longer than the vertical frame length (preferably 0.25 times longer)
(4) Fourth Trimming Frame The fourth trimming frame is used for a frame in the middle of tilt-up and tilt-down, and is used for virtual camera work that does not require a margin (margin) at the upper and lower parts of the frame. A specific fourth trimming frame is shown in FIG. The features of the fourth trimming frame are as follows.
・ Aspect ratio: 16: 9
-Upper margin: None-Left and right margin: 0.05 to 0.1 times the horizontal frame length (preferably 0.1 times the length)
-Lower margin: None The features of the first trimming frame, the second trimming frame, the third trimming frame, and the fourth trimming frame are set in the trimming frame size determination unit 28.

Next, the operations of the trimming frame size determination unit 28, the trimming control unit 29, and the trimming unit 30 assuming specific virtual camera work will be described.

1. 1. Specific operation when the specific person is "person A (one person)" and the virtual camera work is "up shot" In the following description, the captured image as shown in FIG. 14 is input to the image processing device 2. The specific person (subject of virtual camera work) received by the specific person designated reception unit 21 is "person A", and the virtual camera work designated by the virtual camera work designated reception unit 22 is "up shot". A case will be described. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 determines the skeleton of the “person A” from the captured image as shown in FIG. 15 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. In addition, the vertical skeleton centerline of the skeleton of "person A" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads the composition information corresponding to the “upshot” by the virtual camera work from the composition database 27 when the specific person is “one person”. Here, the composition information corresponding to the “up shot” read out is as follows.
-When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
-Use the first trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length of the face from the skeleton of the person A. Then, as shown in FIG. 15, a temporary frame of the subject frame whose vertical length is 1.1 times the length of the face is generated, and the upper side of the temporary frame is the skeleton of the human face. The temporary frame is installed at the apex and at the position where the vertical center line and the skeleton center line coincide with each other.

Next, as shown in FIG. 15, the trimming frame size determining unit 28 does not change the vertical length of the temporary frame, but has a horizontal length so as to be the maximum range of the skeleton of the person A in the horizontal direction. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 15, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

The trimming control unit 29 reads out the composition information related to the control from the composition database 27 among the composition information corresponding to the “upshot” by the virtual camera work in which the specific person is “one person”. Here, the composition information read out is as follows.
-Place the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject coincide with each other.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may move with the movement. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and the trimmed image is the “up shot” of the designated virtual camera work “person A” as shown in FIG. Output as video. Further, FIG. 17 shows a state in which the virtual camera work image is displayed on the display unit 41 of the tablet terminal (computer 100).

2. Specific operation when the specific person is "person A (one person)" and the virtual camera work is "west shot" In the following description, the captured image as shown in FIG. 14 is input to the image processing device 2. The specific person (subject of virtual camera work) received by the specific person designated reception unit 21 is "person A", and the virtual camera work designated by the virtual camera work designated reception unit 22 is "west shot". A case will be described. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 identifies the skeleton of the “person A” from the captured image as shown in FIG. 15 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. In addition, the vertical skeleton centerline of the skeleton of "person A" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads the composition information corresponding to the "west shot" by the virtual camera work from the composition database 27 when the specific person is "one person". Here, the composition information corresponding to the “west shot” read out is as follows.
-When the length of the face is 1, the length of the subject frame in the vertical direction is 1.7 times.
-Use the first trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length of the face from the skeleton of the person A. Then, as shown in FIG. 18, a temporary frame of the subject frame having a vertical length of 1.7 times the length of the face is generated, and the upper side of the temporary frame is the skeleton of the human face. The temporary frame is installed at the apex and at the position where the vertical center line and the skeleton center line coincide with each other.

Next, as shown in FIG. 18, the trimming frame size determining unit 28 does not change the vertical length of the temporary frame, but has a horizontal length so as to be the maximum range of the skeleton of the person A in the horizontal direction. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 18, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

The trimming control unit 29 reads out the composition information related to the control from the composition database 27 among the composition information in which the specific person is “one person” and the virtual camera work corresponds to the “west shot”. Here, the composition information read out is as follows.
-Place the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject coincide with each other.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and the trimmed image is the "west shot" of the designated virtual camera work "person A" as shown in FIG. Output as video. The state in which the virtual camera work image is displayed on the display unit 41 of the tablet terminal (computer 100) is omitted because the virtual camera work image is only the image of the "west shot" of the "person A".

3. 3. Specific operation when the virtual camera work is "full shot" of "person A (one person)" In the following explanation, the captured image as shown in FIG. 14 is input to the image processing device 2 and is specified. The case where the specific person (subject of virtual camera work) received by the person designation reception unit 21 is "person A" and the virtual camera work designated by the virtual camera work designation reception unit 22 is "full shot" will be described. .. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 identifies the skeleton of the “person A” from the captured image as shown in FIG. 15 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. In addition, the vertical skeleton centerline of the skeleton of "person A" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads the composition information corresponding to the “one person” and the virtual camera work “full shot” from the composition database 27. Here, the composition information read out is as follows.
-The vertical length (whole body) of the skeleton of a specific person is defined as the vertical length of the subject frame.
-Use the second trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length (whole body) of the skeleton from the skeleton of the person A. Then, as shown in FIG. 20, a temporary frame of the subject frame having a vertical length equal to the length of the vertical skeleton of the person A is generated, and the upper side of the temporary frame is the skeleton of the person. The temporary frame is installed at the top apex and at the position where the vertical center line and the center line of the skeleton coincide with each other.

Next, as shown in FIG. 20, the trimming frame size determining unit 28 has a horizontal length so as to be the maximum range in which the skeleton of the person A can be inserted without changing the vertical length of the temporary frame. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 20, the trimming frame size determination unit 28 adjusts the size of the second trimming frame so as to satisfy the composition information. The size of the second trimming frame is adjusted so that the margin of the second trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

The trimming control unit 29 reads out the composition information related to the control from the composition database 27 among the composition information corresponding to the “one person” and the virtual camera work “full shot”. Here, the composition information read out is as follows.
-Place the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the subject coincide with each other.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and the trimmed image is the “full shot” of the designated virtual camera work “person A” as shown in FIG. Output as video. The state in which the virtual camera work image is displayed on the display unit 41 of the tablet terminal (computer 100) is omitted because the virtual camera work image is only a "full shot" image of "person A".

4. Specific operation when the virtual camera work is "full shot" of "person A, person B (two people)" In the following description, the captured image as shown in FIG. 14 is input to the image processing device 2. The specific person (subject of virtual camera work) received by the specific person designated reception unit 21 is "person A, person B (two people)", and the virtual camera work designated by the virtual camera work designated reception unit 22. The case where is a "full shot" will be described. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 uses the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25 to obtain the skeletons of "person A" and "person B" from the captured video as shown in FIG. Identify. In addition, the vertical skeletal centerline of the skeletons of "person A" and "person B" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads out the composition information in the vertical direction from the composition database 27 among the composition information corresponding to the virtual camera work “full shot” in which the specific person is “two people”. Here, the composition information read out is as follows.
-The length in which the vertical direction of the skeleton of all specific persons is inserted is defined as the vertical length of the subject frame.
-The vertical centerline of the subject frame is located in the middle of the skeletal centerlines of the two people.
-Use the second trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length of the skeleton from the skeletons of "person A" and "person B". Then, as shown in FIG. 22, a temporary frame of the subject frame having a vertical length equal to the length including all the vertical skeletons of "person A" and "person B" is generated. The temporary frame is installed at a position where the upper side of the temporary frame is located at the top apex of the skeleton of the person and the center line in the vertical direction is in the middle of the center lines of the skeletons of the two people.

Next, as shown in FIG. 22, the trimming frame size determining unit 28 is arranged in the horizontal direction so as to have the maximum range in which the skeletons of the person A and the person B can be accommodated without changing the vertical length of the frame. Adjust the length. The frame generated in this way is the subject frame.

As shown in FIG. 22, the trimming frame size determination unit 28 adjusts the size of the second trimming frame so as to satisfy the composition information. The size of the second trimming frame is adjusted so that the margin of the second trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG. 23.

The trimming control unit 29 reads out the composition information related to control from the composition database 27 among the composition information corresponding to the virtual camera work “full shot” in which the specific person is “two people”. Here, the composition information read out is as follows.
-Place the trimming frame so that the vertical centerline of the trimming frame is located in the middle of the skeleton centerlines of the two people.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and displays the trimmed trimmed image as the designated virtual camera work “person A, person B (two people)” as shown in FIG. 23. Output as a "full shot" video of. When the virtual camera work image is displayed on the display unit 41 of the tablet terminal (computer 100), the virtual camera work image is a "full shot" image of "person A, person B (two people)". Only, so omit it.

5. Specific operation when the virtual camera work is "tilt up" of "person C (one person)" In the following explanation, the captured image as shown in FIG. 14 is input to the image processing device 2 and is specified. The case where the specific person (subject of virtual camera work) received by the person designation reception unit 21 is "person C" and the virtual camera work designated by the virtual camera work designation reception unit 22 is "tilt up" will be described. .. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 identifies the skeleton of the “person C” from the captured image as shown in FIG. 24 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. The vertical skeleton centerline of the skeleton of "person C" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads the composition information corresponding to the virtual camera work “tilt up” from the composition database 27 when the specific person is “one person”. Here, the composition information read out is as follows.
-When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
-The vertical length of the subject frame shall be the width of the whole body of the person (the maximum range occupied by the skeleton of the person in the horizontal direction).
-The third trimming frame is used as the frame used at the start of the virtual camera work.
-The first trimming frame is used as the frame used at the end of the virtual camera work.
-The fourth trimming frame is used as the frame used at the start and end of the virtual camera work.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the length of the lateral width from the skeleton of the person C. Then, as shown in FIG. 24, the temporary frame is determined so that the width is the width of the person C and the skeleton center line of the person C and the center line of the subject frame coincide with each other. Further, the trimming frame size determination unit 28 generates a subject frame in which the length of the temporary frame in the vertical direction is 1.1 times the length of the face.

As shown in FIG. 24, the trimming frame size determination unit 28 adjusts the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. The width of the third and fourth trimming frames shall be the same as the width of the first trimming frame after adjusting the size. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG. 24.

The trimming control unit 29 reads the composition information corresponding to the virtual camera work “tilt up” from the composition database 27 when the specific person is “one person”. Here, the composition information read out is as follows.
-The third trimming frame is used as the frame used at the start of the virtual camera work.
-The first trimming frame is used as the frame used at the end of the virtual camera work.
-The fourth trimming frame is used as the frame used at the start and end of the virtual camera work.
-Place the trimming frame so that the vertical center line of the trimming frame and the vertical skeleton center line of the person coincide with each other.
-After the frame used at the start of virtual camera work is stationary for 1 second, it moves to a spline (slow movement at the beginning and end and fast movement in the middle) along the skeleton centerline of the person.
-The frame used at the end of virtual camera work remains stationary.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame that moves from the captured image, and the trimmed trimmed image is “tilt up” of the designated virtual camera work “person C” as shown in FIG. Is output as a video. Since the virtual camera work image is displayed on the display unit 41 of the tablet terminal (computer 100), the virtual camera work image is only the "tilt up" image of the "person C" who is the virtual camera work. , Omitted.

6. Specific operation when the virtual camera work is "pan" In the following description, the captured image as shown in FIG. 14 is input to the image processing device 2, and the image is changed from "person A" to "person B". , The case of panning with an up shot will be described. It is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

When the virtual camera work is "pan", it is necessary to specify the start point and the end point. The specified contents of the start point and the end point are a specific person and a shot (long shot, up shot, bust shot, waist shot) to be the subject. In this example, since the start point is a close-up shot of "person A", touch "Pan" of the virtual camera work designated reception unit 42 (virtual camera work designated reception unit 22), and then touch the start point to " Touch "Up". Subsequently, "person A" of the specific person designated reception unit 43 (specific person designated reception unit 21) is touched. Next, since the end point is a close-up shot of "person B", touch "Pan" of the virtual camera work designated reception unit 42 (virtual camera work designated reception unit 22), then touch the end point, and then "Up". (Up shot) ”is touched. Subsequently, the "person B" of the specific person designated reception unit 43 (specific person designated reception unit 21) is touched. This specifies the start and end points of the "pan".

First, with respect to the starting point "person A", the person identification unit 26 and the trimming frame size determination unit 28 operate as follows.

The person identification unit 26 determines the skeleton of the “person A” from the captured image as shown in FIG. 15 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. In addition, the vertical skeleton centerline of the skeleton of "person A" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads from the composition database 27 the composition information corresponding to the virtual camera work “pan” in which the specific person at the start point is “one person” and the shot at the start point corresponds to the “up shot”. Here, the composition information read out is as follows.
-When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
-Use the first trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length of the face from the skeleton of the person A. Then, as shown in FIG. 15, a temporary frame of the subject frame whose vertical length is 1.1 times the length of the face is generated, and the upper side of the temporary frame is the skeleton of the human face. The temporary frame is installed at the apex and at the position where the vertical center line and the skeleton center line coincide with each other.

Next, as shown in FIG. 15, the trimming frame size determining unit 28 does not change the vertical length of the temporary frame, but has a horizontal length so as to be the maximum range of the skeleton of the person A in the horizontal direction. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 15, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

Next, with respect to the end point "person B", the person identification unit 26 and the trimming frame size determination unit 28 operate as follows.

The person identification unit 26 determines the skeleton of the “person B” from the captured image as shown in FIG. 26 by using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. The vertical skeleton centerline of the skeleton of "person B" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads from the composition database 27 the composition information corresponding to the virtual camera work “pan” in which the specific person at the end point is “one person” and the shot at the end point corresponds to the “up shot”. Here, the composition information read out is as follows.
-When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
-Use the first trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the vertical length of the face from the skeleton of the person B. Then, as shown in FIG. 26, a temporary frame of the subject frame whose vertical length is 1.1 times the length of the face is generated, and the upper side of the temporary frame is the skeleton of the human face. The temporary frame is installed at the apex and at the position where the vertical center line and the skeleton center line coincide with each other.

Next, as shown in FIG. 26, the trimming frame size determining unit 28 does not change the vertical length of the temporary frame, but has a horizontal length so as to be the maximum range of the skeleton of the person A in the horizontal direction. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 26, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

When the size of the trimming frame at the start point and the end point is determined, the trimming control unit 29 has a virtual camera work "pan" in which the specific person at the start point and the end point is "one person" and the shots at the start point and the end point correspond to the "up shot". Of the composition information of the above, the composition information related to control is read from the composition database 27. Here, the composition information read out is as follows.
-Arrange the trimming frame of the specific person specified at the start point so that the vertical center line of the trimming frame and the vertical skeleton center line of the specific person specified at the start point coincide with each other.
-Arrange the trimming frame of the specific person specified at the end point so that the vertical center line of the trimming frame and the vertical skeleton center line of the specific person specified at the end point coincide with each other.
-For the trimming frame from the start point to just before the end point, use the trimming frame used at the start point.
-At the end point, the trimming frame of the specific person specified at the end point is used.
・ After the trimming frame used at the start of virtual camera work is stopped for 1 second, it moves to a spline (slow movement at the beginning and end and fast movement in the middle) from the position of the trimming frame at the start point to the position of the trimming frame immediately before the end point. do.
-The trimming frame used at the end of virtual camera work remains stationary.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. 27. In addition, it is placed on the captured image and controlled.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and displays the trimmed trimmed image as an up-shot pan from the start point “person A” to the end point “person B” as shown in FIG. 27. Output as the corresponding virtual camera work video.

This is the end of the description of the first embodiment. It goes without saying that the above-mentioned trimming frame and numerical values of composition information are examples, and can be changed in a timely manner.

Since the first embodiment is configured to recognize a person in the captured image, it is not necessary for the user to search for a specific person (subject) in the captured image as in the conventional technique, which saves time and effort. Can be reduced.

Further, in the first embodiment, the skeleton of a specific person (subject) is determined, the size and position of the part of the specific person on the image are specified, and the information and the composition information are used to create a virtual camera work image. Is generated, so it is possible to generate images of various camera works.

<Modification 1 of the first embodiment>
In the virtual camera work in the first embodiment described above, the orientation of the face of a specific person is not taken into consideration. However, more natural camera work can be performed by considering the orientation of the face of a specific person. For example, when taking a close-up shot of a specific person, a better composition can be obtained by taking a wide space in the direction in which the face of the specific person is facing. Therefore, in the modified example of the first embodiment, the face recognition unit 24 acquires the information on the face orientation of the specific person, and the virtual camera work using the composition information in consideration of the information on the face orientation of the specific person. An example of is described.

In the following description, when the specific person is "person B (one person)" and the virtual camera work is "up shot", the trimming frame size determination unit 28, the trimming control unit 29, and the trimming unit 30 are concrete. The operation will be mainly explained. Then, a case where a captured image as shown in FIG. 14 is input to the image processing device 2 will be described. Further, it is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

The person identification unit 26 determines the skeleton of the “person B” from the captured image as shown in FIG. 28, using the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25. The vertical skeleton centerline of the skeleton of "person B" is also determined. The skeleton center line is a line that passes through the neck of the skeleton. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

Further, the face recognition unit 24 outputs information on the face orientation of the "person B" to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads composition information corresponding to "one person", virtual camera work "up shot", and face orientation "right direction" from the composition database 27. Here, the composition information corresponding to the “up shot” read out is as follows.
-When the vertical length of the face is 1, the vertical length of the subject frame is 1.1 times.
-Use the first trimming frame as the trimming frame.
-The vertical center line of the subject frame passes through the position of 0.65: 0.35 from the left of the trimming frame.

The trimming frame size determination unit 28 calculates the vertical length of the face from the skeleton of the person B. Then, as shown in FIG. 28, a temporary frame of the subject frame whose vertical length is 1.1 times the length of the face is generated, and the upper side of the temporary frame is the skeleton of the human face. The temporary frame is installed at the apex and at the position where the vertical center line and the skeleton center line coincide with each other.

Next, as shown in FIG. 28, the trimming frame size determining unit 28 does not change the vertical length of the temporary frame, but has a horizontal length so as to be the maximum range of the skeleton of the person B in the horizontal direction. To adjust. The frame generated in this way is the subject frame.

As shown in FIG. 28, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted after ensuring the minimum margin at the position where the vertical center line of the subject frame passes the position of 0.65: 0.35 from the left of the trimming frame. The margin of the first trimming frame is adjusted so as to be as preferable as possible within a range that does not overlap with the subject frame. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG. 28.

The trimming control unit 29 reads out the composition information related to the control from the composition database 27 among the composition information corresponding to the “upshot” by the virtual camera work in which the specific person is “one person”. Here, the composition information read out is as follows.
-The trimming frame is arranged so that the vertical skeleton center line of the specific person passes through the position of 0.65: 0.35 from the left of the trimming frame.

The trimming control unit 29 uses the position information (skeleton center line) of the skeleton from the person identification unit 26 and the composition information to show the trimming frame of the size determined by the trimming frame size determination unit 28 as shown in FIG. Place it on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the skeleton centerline may also move. The position of the trimming frame may be controlled as the skeleton centerline moves, but even if the skeleton centerline does not move much, moving the trimming frame may result in an image that is difficult for the viewer to view. There is. Therefore, when the range of movement of the skeleton center line is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and the trimmed image is the “up shot” of the designated virtual camera work “person B” as shown in FIG. 29. Output as video.

According to the first modification of the first embodiment, as illustrated in FIG. 27, as shown in FIG. 29, as compared with the “up shot” of the “person B” that does not consider the orientation of the face. The "up shot" of "person B" considering the orientation of the person's face can obtain an image with an ideal composition taken by a human. That is, it is possible to obtain a virtual camera work image that is closer to the camera work image taken by a human being.

<Modification 2 of the first embodiment>
In the first embodiment described above, an example of designating only a specific person as a subject designation has been described. However, the person identification unit 26 determines each part of the specific person. Therefore, in the second modification of the first embodiment, not only the specific person but also the part of the specific person is designated, and an example of realizing the camera work in which the part of the specific person is the subject will be described.

As a specific example of the modified example 2 of the first embodiment, the specific person is "person B (one person)", the part to be the subject of the specific person is "right hand", and the virtual camera work is "up". The case of "shot" will be described. It is assumed that the specific composition information read by each unit is stored in the composition database 27 in advance.

First, regarding a specific method of designating a part, as shown in FIG. 30, on a tablet terminal (computer 100) in which an information processing device 2 and a display device 3 are integrally configured, a specific person designation reception unit 43 is used. In addition, a site designation portion 50 is provided. The part designation unit 50 is a diagram schematically showing the whole body of a person, and the part can be designated by touching the part to be the subject. For example, when the specific person is "person B (one person)" and the part to be the subject of the specific person is "right hand", after touching "person B" of the specific person designation reception unit 43, the part designation unit 50 By touching the "right hand", the "person B" and the "right hand" can be specified.

When the designation of the specific person, the part of the specific person to be the subject, and the virtual camera work is completed, the person identification unit 26 uses the person recognized by the face recognition unit 24 and the skeleton determined by the skeleton determination unit 25 in the drawing. As shown in 5, the skeleton of "person B" is determined from the captured image. Then, the coordinate information of these skeletons is output to the trimming frame size determination unit 28.

The trimming frame size determination unit 28 reads the composition information corresponding to the “up shot” by the virtual camera work from the composition database 27, in which the specific person is “one person” and the part to be the subject is the “right hand”. Here, the composition information read out is as follows.
-The intersection of the vertical and horizontal center lines of the subject frame (the center of the subject frame) should be the tip of the right hand.
-The vertical length of the subject frame shall be twice the length of the vertical right forearm.
-The horizontal length of the subject frame shall be twice the length of the horizontal right forearm.
-Use the first trimming frame as the trimming frame.
-Match the vertical center line of the trimming frame with the vertical center line of the subject frame.

The trimming frame size determination unit 28 calculates the length of the right forearm portion in the vertical direction and the horizontal direction from the skeleton of the person B. Then, as shown in FIG. 31, the length in the vertical direction is twice the length of the vertical right forearm, the horizontal length is twice the length of the horizontal right forearm, and the subject frame. Adjust the subject frame so that the intersection of the vertical and horizontal center lines (the center of the subject frame) is the tip of the right hand.

As shown in FIG. 31, the trimming frame size determination unit 28 adjusts the size of the first trimming frame so as to satisfy the composition information. The size of the first trimming frame is adjusted so that the margin of the first trimming frame does not overlap with the subject frame and is as preferable as possible after ensuring the minimum margin. In this way, the trimming frame size determination unit 28 determines the size of the trimming frame as shown in FIG.

The trimming control unit 29 stores composition information related to control among composition information in which a specific person is "one person", a part to be a subject is "right hand", and virtual camera work corresponds to "up shot" in a composition database. Read from 27. Here, the composition information read out is as follows.
-Place the trimming frame so that the intersection of the vertical and horizontal center lines of the trimming frame coincides with the tip of the right hand.

The trimming control unit 29 uses the position information (coordinates of the tip of the right hand) of the skeleton from the person identification unit 26 and the composition information to display the trimming frame of the size determined by the trimming frame size determination unit 28 in FIG. 32. As shown, it is placed on the captured image.

Since the captured image is a moving image, the person may move with the passage of time, and the coordinates of the tip of the right hand may move with the movement. The position of the trimming frame may be controlled as the coordinates of the tip of the right hand move, but even if the coordinates of the tip of the right hand do not move much, moving the trimming frame makes it difficult for the viewer to view the image. May become. Therefore, when the range of movement of the coordinates of the tip of the right hand is within a predetermined distance range, the position of the trimming frame may be controlled to be maintained.

The trimming unit 30 trims the image in the trimming frame from the captured image, and outputs the trimmed trimmed image as an “up shot” image of the “right hand of the person B” as shown in FIG. 32.

In the second modification of the first embodiment, the position and length information of each part determined by using the skeleton of the specific person (subject) and the composition information are used to obtain the specific part of the specific person. It is possible to realize virtual camera work as a subject.

<Modification 3 of the first embodiment>
A modified example 3 of the first embodiment will be described. FIG. 33 is a block diagram of the information processing device 2 of the modification 3 of the first embodiment.

In the third modification of the first embodiment, an image effect is given to the virtual camera work image generated by trimming the captured image. As for the image effect, for example, the image effect that the shooting frame itself does not change like focus-in, but the image that is not focused with time changes to the image that is focused from the subject, and vice versa, such as focus-out. There is.

In order to give these video effects to the virtual camera work video generated by trimming the captured video, the information processing device 2 of the modification 3 of the first embodiment further includes a video effect processing unit 31. ..

The video effect processing unit 31 acquires video effect information from the composition database 27 from the composition information corresponding to the designated virtual camera work. Then, the image effect processing unit 31 processes the image effect on the trimmed image output from the trimming unit 30, and outputs the virtual camera work image.

<Second embodiment>
A second embodiment will be described.

FIG. 34 is a block diagram of the information processing apparatus 2 according to the second embodiment.

The information processing device 2 of the second embodiment further includes a function of teloping or superimposing a CG (computer graphics) image, another edited image, or the like on a virtual camera work image. In order to realize the function, the information processing device 2 of the second embodiment includes an image database 50, an image selection unit 51, and an adder in addition to the configuration of the information processing device 2 of the first embodiment. 52 and.

The image storage unit 50 stores a CG image such as a logo, an edited image, and the like (described as a superimposed image).

The image selection unit 51 is a unit that receives from the user the selection of the superimposed image to be superimposed on the virtual camera work image among the superimposed images stored in the image storage unit 50. The image selection unit 51 displays the superimposed image that can be selected by the user among the superimposed images stored in the image storage unit 50 on the display device 3 for the user to select the superimposed image. Examples of the display include identification information for identifying the superimposed image, a thumbnail image of the superimposed image, and the like. The image selection unit 51 also controls the position on which the selected superimposed image is superimposed on the virtual camera work image and the size thereof. The control of the position and size is determined by the user's designation of the position and size on the virtual camera work image.

The adder 52 adds (superimposes) the selected superimposed image to the virtual camera work image, and outputs the virtual camera work image on which the superimposed image is superimposed.

Next, the operation of the information processing device 2 according to the second embodiment will be described. In the following description, as in the first embodiment, a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrally configured will be described.

As shown in FIG. 35, the display unit (display device 3) of the tablet terminal (computer 100) includes a display unit 40 on which a captured image is displayed, a display unit 41 on which a virtual camera work image is displayed, and a virtual camera work. In addition to the virtual camera work designated reception unit 42 (virtual camera work designated reception unit 22) and the specific person designation reception unit 43 (specific person designation reception unit 21) that designates a specific person as a subject, a superimposed image is displayed. The thumbnail image 44 (image selection unit 51) for selecting is displayed.

The user selects a superimposed image to be superimposed on the virtual camera work image displayed on the display unit 41 from the thumbnail image 44. As for the selection and arrangement method, as shown in FIG. 36, the selected thumbnail image is dragged onto the virtual camera work image with a finger. Then, it is dropped at a desired position on the virtual camera work image. The size of the superimposed image is determined by dropping the superimposed image on the virtual camera work image and then pinching in or out of the superimposed image as shown in FIG. 37.

In this way, it is possible to generate an image in which the logo and the edited image are superimposed on the virtual camera work image.

<Third embodiment>
A third embodiment will be described.

A third embodiment is a switching system. FIG. 38 is a block diagram of the switching system of the third embodiment.

The switching system of the third embodiment includes a plurality of information processing devices 2 of the first or second embodiment. The captured image of the camera 1 is input to each information processing device 2, and each information processing device 2 outputs one virtual camera work image by the virtual camera work selected by the user.

The switching device 4 inputs virtual camera work images from each information processing device 2, and one virtual camera work image is output as a line image according to the user's selection.

Next, the operation of the switching system of the third embodiment will be described. In the following description, it is assumed that two information processing devices 2 are used and a tablet terminal (computer 100) in which the information processing device 2 and the display device 3 are integrally configured as in the first embodiment is assumed. do. That is, the tablet terminal (computer 100) includes two information processing devices 2, a display device 3, and a switching device 4.

On the display unit (display device 3) of the tablet terminal (computer 100) of the third embodiment, the display unit 41-1 displaying two types of virtual camera work images output from the two information processing devices 2. It differs from the first embodiment and the second embodiment in that there are 41-2 and 41-2. The display unit 41-1 is a display unit on which a line image is displayed, and the display unit 41-2 is a display unit on which a preview image as a candidate for the line image is displayed. The display unit 41-1 on which the line image is displayed has a thicker frame line than the display unit 41-2, so that the line image can be easily visually recognized.

In order to obtain a virtual camera work image that is a candidate for a line image, the user first touches (selects) the screen of the display unit 41-2 with a finger, and then uses the specific person designation reception unit 43 (specific person designation reception unit 21). A specific person is designated, and the virtual camera work is designated by the virtual camera work designation reception unit 42 (virtual camera work designation reception unit 22). Then, the designated specific person and the virtual camera work image corresponding to the virtual camera work are displayed on the selected display unit 41-2. Here, when switching the virtual camera work image (preview image) displayed on the display unit 41-2 to the line image, touch the display unit 41-2 to select it, and then touch the "TAKE" button. , The virtual camera work image (preview image) displayed on the display unit 41-2 can be output as a line image. At this time, the virtual camera work image (preview image) displayed on the display unit 41-2 is not displayed on the display unit 41-2, but is displayed on the display unit 41-1 on which the line image is displayed. become.

Further, in order to obtain a virtual camera work image that is a candidate for a line image, a specific person is specified after touching (selecting) the screen of the display unit 41-2 on which the virtual camera work image (preview image) is not displayed. A specific person is designated by the reception unit 43 (specific person designation reception unit 21), and the virtual camera work is designated by the virtual camera work designation reception unit 42 (virtual camera work designation reception unit 22). Then, the designated specific person and the virtual camera work image corresponding to the virtual camera work are displayed on the selected display unit 41-2.

FIG. 40 shows an example in which two types of virtual camera work images are displayed on the display unit 41-1 and the display unit 41-2. The virtual camera work image displayed on the display unit 41-1 is a line image, and the virtual camera work image displayed on the display unit 41-2 is a preview image. Then, when switching the line image from the virtual camera work image displayed on the display unit 41-1 to the virtual camera work image displayed on the display unit 41-2, touch the display unit 41-2 to select. After that, by touching the "TAKE" button, the virtual camera work image (preview image) displayed on the display unit 41-2 can be output as a line image. At this time, as shown in FIG. 41, the virtual camera work image (preview image) displayed on the display unit 41-2 is not displayed on the display unit 41-2, and the line image is displayed on the display unit 41. It will be displayed at -1.

In the above description, in order to obtain a virtual camera work image that is a candidate for a line image, an example of first touching (selecting) the screen of the display unit 41-2 with a finger has been described. This is to support even when there are a plurality of display units for displaying the preview image. However, as in this example, when there is only one display unit for displaying the preview image, it is possible to configure the configuration so that the display unit does not have to be selected.

According to the third embodiment, the switching system can be realized without requiring a large number of cameras and users (operators).

Although the present invention has been described with reference to preferred embodiments, it is not necessary to provide the configurations of all the embodiments, and not only can they be combined in a timely manner, but the present invention is not necessarily limited to the above embodiments. It is not something that is done, and it can be transformed and implemented in various ways within the scope of its technical ideas.

1 Camera 2 Video processing device 3 Display device 4 Switching device 20 Shooting video input unit 21 Specific person designation reception unit 22 Virtual camera work designation reception unit 23 Face image dictionary unit 24 Face recognition unit 25 Skeletal judgment unit 26 Person identification unit 27 Composition database 28 Trimming frame size determination unit 29 Trimming control unit 30 Trimming unit 31 Video effect processing unit 50 Image database 51 Image selection unit 52 Adder

Claims (13)

The acquisition unit that acquires the captured video taken by the camera,
A specific person designation reception section that specifies a specific person to be the subject, and
A virtual camera work designation reception unit that specifies a virtual camera work that realizes a video effect by operating a virtual camera by trimming a part of the shot image.
A person recognition unit that recognizes a person in the captured image,
A skeleton determination unit that determines the skeleton of a person in the image using the captured image,
A specific person in the image is identified by using the recognition result of the person recognition unit, and the skeleton of the specific person among the skeletons determined by the skeleton determination unit is used on the captured image of the part of the specific person. The person identification part that identifies the positional relationship and
For each virtual camera work, a composition information storage unit that stores composition information for which composition is defined, and a composition information storage unit.
A trimming frame size that determines the size of a trimming frame for trimming a part of the captured image using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work. The decision department and
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Trimming frame control unit to control and
An image processing device having a trimming unit that trims an image in the trimming frame from the captured image and outputs the trimmed image as a virtual camera work image by the virtual camera work. The video processing device according to claim 1, wherein the virtual camera work includes at least predetermined upshots, pans, and tilts. The person according to claim 1 or 2, wherein at least one of the person recognition unit and the skeleton determination unit recognizes a person or determines the skeleton by using an image having a resolution lower than the resolution of the captured image. Video processing equipment. The person recognition unit determines the orientation of the face of the specific person, and determines the direction of the face of the specific person.
The image processing device according to any one of claims 1 to 3, wherein the trimming frame control unit controls the position of the trimming frame in consideration of the orientation of the face of the specific person. The video processing apparatus according to any one of claims 1 to 4, which has a part designation receiving unit for receiving the designation of a part of the specific person. The image processing apparatus according to any one of claims 1 to 5, further comprising an image processing unit that processes an image output from the trimming unit and outputs a pseudo focus-in or focus-out image. A superimposed image acquisition unit that acquires a superimposed image superimposed on the virtual camera work image, and
The video processing apparatus according to any one of claims 1 to 6, further comprising a superimposing portion for superimposing a superimposing image on the virtual camera work image. It has at least two or more video processing devices, a display unit, and a switching unit.
The video processing device
The acquisition unit that acquires the captured video taken by the camera,
A specific person designation reception section that specifies a specific person to be the subject, and
A virtual camera work designation reception unit that specifies a virtual camera work that realizes a video effect by operating a virtual camera by trimming a part of the shot image.
A person recognition unit that recognizes a person in the captured image,
A skeleton determination unit that determines the skeleton of a person in the image using the captured image,
A specific person in the image is identified by using the recognition result of the person recognition unit, and the skeleton of the specific person among the skeletons determined by the skeleton determination unit is used on the captured image of the part of the specific person. The person identification part that identifies the positional relationship and
For each virtual camera work, a composition information storage unit that stores composition information for which composition is defined, and a composition information storage unit.
A trimming frame size that determines the size of a trimming frame for trimming a part of the captured image using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work. The decision department and
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Trimming frame control unit to control and
It has a trimming unit that trims the image in the trimming frame from the captured image and outputs the trimmed image as a virtual camera work image by the virtual camera work.
The display unit
The captured image and at least two or more of the virtual camera work images are displayed, and the image is displayed.
The switching unit is
A switching system that outputs one virtual camera work image specified by a user out of at least two or more virtual camera work images. The switching system according to claim 8, wherein the display unit distinguishes between a virtual camera work image output as a line image and another virtual camera work image. The acquisition process to acquire the captured video taken by the camera, and
Specific person designation reception processing that accepts the designation of a specific person to be the subject,
Virtual camera work designation acceptance processing that accepts the designation of virtual camera work that realizes the video effect by operating the virtual camera by trimming a part of the shot video,
The person recognition process for recognizing the person in the captured image and
A skeleton determination process for determining the skeleton of a person in the image using the captured image, and
A specific person in the image is identified by using the recognition result of the person recognition process, and the skeleton of the specific person among the skeletons determined by the skeleton determination process is used on the captured image of the part of the specific person. Person identification processing to identify the positional relationship and
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work among the composition information whose composition is determined for each virtual camera work, the captured image Trimming frame sizing process to determine the size of the trimming frame to trim a part,
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Trimming frame control processing to control and
A program that causes a computer to perform a trimming process of trimming an image in the trimming frame from the captured image and outputting the trimmed image as a virtual camera work image by the virtual camera work. The acquisition process to acquire the captured video taken by the camera, and
Specific person designation processing that accepts the designation of a specific person to be the subject,
A virtual camera work designation process that accepts a virtual camera work designation that realizes a video effect by operating a virtual camera by trimming a part of the shot video, and
The person recognition process for recognizing the person in the captured image and
A skeleton determination process for determining the skeleton of a person in the image using the captured image, and
A specific person in the image is identified by using the recognition result of the person recognition process, and the skeleton of the specific person among the skeletons determined by the skeleton determination process is used on the captured image of the whole body of the specific person. Person identification processing to identify the positional relationship and
The shooting is performed using the positional relationship on the captured image of the whole body of the specific person and the composition information corresponding to the designated virtual camera work among the composition information in which the composition of the subject is determined for each virtual camera work. Trimming frame size determination process to determine the size of the trimming frame that trims a part of the image,
Using the positional relationship on the captured image of the whole body of the specific person and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Trimming frame control processing to control and
A trimming process in which the image in the trimming frame is trimmed from the captured image and the trimmed image is output as a virtual camera work image by the virtual camera work.
Display processing for displaying the captured image and at least two or more virtual camera work images corresponding to at least two or more specific persons or at least two or more virtual camera works and at least two or more virtual camera work images displayed. Of these, a program that causes a computer to execute a switching process that outputs one virtual camera work image specified by the user. Computer
Acquire the shot video taken by the camera and
Accepting the designation of a specific person to be the subject,
Accepts the designation of virtual camera work that trims a part of the shot image and realizes the image effect by operating the virtual camera.
Recognize the person in the shot video and
Using the captured image, the skeleton of the person in the image is determined.
The specific person in the image is identified by using the recognition result of the person, and the positional relationship of the part of the specific person on the captured image is specified by using the skeleton of the specific person among the determined skeletons.
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work among the composition information whose composition is determined for each virtual camera work, the photographed image Determine the size of the trimming frame to trim a part,
Using the positional relationship of the specific person's part on the captured image and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Control and
An image processing method in which an image in the trimming frame is trimmed from the captured image, and the trimmed image is output as a virtual camera work image by the virtual camera work. Computer
Acquire the shot video taken by the camera and
Accepting the designation of a specific person to be the subject,
Accepts the designation of virtual camera work that trims a part of the shot image and realizes the image effect by operating the virtual camera.
Recognize the person in the shot video and
Using the captured image, the skeleton of the person in the image is determined.
The specific person in the image is identified by using the recognition result of the person, and the positional relationship of the whole body of the specific person on the captured image is specified by using the skeleton of the specific person among the determined skeletons.
The shooting is performed by using the positional relationship on the captured image of the whole body of the specific person and the composition information corresponding to the designated virtual camera work among the composition information in which the composition of the subject is determined for each virtual camera work. Determine the size of the trimming frame to trim a part of the image,
Using the positional relationship on the captured image of the whole body of the specific person and the composition information corresponding to the specified virtual camera work, the position of the captured image of the trimming frame corresponding to the specified virtual camera work is determined. Control and
The image in the trimming frame is trimmed from the captured image, and the trimmed image is output as a virtual camera work image by the virtual camera work.
The captured image and at least two or more virtual camera work images corresponding to at least two or more specific persons or at least two or more virtual camera works are displayed.
A switching method that outputs one virtual camera work image specified by the user out of at least two or more virtual camera work images displayed.

Images