KR102037997B1 - Electronic apparatus and method for generating contents - Google Patents

Abstract

An electronic device is disclosed. The electronic device includes: a communication device which receives a captured image; a memory which stores the inputted captured image; and a processor which extracts a face image included in each of a plurality of frames included in the inputted captured image, clusters the extracted face image into a plurality of people, generates track information for each of the people based on the face image clustered into a same person, and generates content for each of the people by using the inputted captured image and the generated track information. It is possible to generate the content automatically.

Description

ELECTRONIC APPARATUS AND METHOD FOR GENERATING CONTENTS}

The present disclosure relates to an electronic device and a content generating method, and more particularly, to an electronic device and a content generating method capable of automatically generating content for each person in a captured image using AI technology.

Recently, various types of contents generated by ordinary individuals are being distributed, as well as contents directly provided by broadcasters through airwaves. In particular, recently, content (hereinafter referred to as direct cam video) photographing only a specific member of a singer has been actively distributed.

These direct cam videos are generally taken by the audience watching the singer's performance, so there is a lot of shaking and the quality of the video is not high.

Recently, broadcasters have introduced camera equipment directly and hired directors to provide direct cam images. However, there was a problem in that a high cost was required in that individual cameras and directors were required for each member.

Accordingly, an object of the present disclosure is to provide an electronic device and a content generation method capable of automatically generating content for each person in a captured image using AI technology.

An electronic device according to the present disclosure for achieving the above object is a communication device for receiving a photographed image, a memory for storing the input photographed image, and a face included in each of a plurality of frames included in the input photographed image. Extracting an image, clustering the extracted face image into a plurality of people, generating track information for each person based on the face image clustered with the same person, and using the input image and the generated track information It includes a processor for generating person-specific content.

In this case, the processor detects a face included in each of the plurality of frames using a pre-learned face recognition AI algorithm, and detects a face image corresponding to the detected face, coordinate information of the face image, and the face image. Size information can be extracted.

The processor may cluster the extracted face images into a plurality of persons by calculating a 128D feature vector for each of the extracted face images, calculating a mutual similarity of the calculated feature vectors.

In this case, the processor may input the calculated feature vector into a pre-learned clustering AI algorithm to cluster the extracted face image into a plurality of persons.

Meanwhile, the processor may generate face information for each person by connecting face coordinate information in a frame having face images clustered with the same person.

In this case, if there is no face image of a specific person in a preset frame, the processor may generate trajectory information using face coordinate information of a frame immediately before the face image.

In the meantime, if there is no face image of a specific person over a preset frame, the processor may generate track information of the specific person by using user trajectory information of another person in the frame section.

The processor may verify the generated trajectory information for each person.

In this case, the processor may calculate the movement speed for each section using the generated trajectory information, and determine that there is an error in the frame corresponding to the section having the movement speed more than a predetermined value.

Meanwhile, the electronic device may include a first region displaying a section in which a face of a specific person is detected among all sections of the input image, a second region displaying a specific frame of the input image and a detected face in the specific frame, and the The display apparatus may further include a display configured to display a user interface window including a third area displaying an extracted image corresponding to the specific person in a specific frame.

The processor may extract a vertical image for each of the plurality of frames included in the input photographed image for each person having a length longer than the horizontal length, and generate vertical content for each person by using the extracted vertical image. have.

In this case, the processor may identify the face position of the person in the frame by using the generated trajectory information, and extract an image area corresponding to the face size of the person as a vertical image based on the identified face position.

The processor may generate merged content by merging a plurality of person contents among the generated person contents.

On the other hand, the content generating method according to an embodiment of the present disclosure is the step of extracting a face image included in each of the plurality of frames included in the input image, clustering the extracted face image into a plurality of people, the same Generating trajectory information for each person based on a face image clustered with the person, and generating content for each person using the input image and the generated trajectory information.

In this case, the extracting of the face image may include detecting a face included in each of the plurality of frames using a pre-learned face recognition AI algorithm, and a face image corresponding to the detected face and coordinate information of the face image. And size information of the face image.

On the other hand, in the clustering step, a 128-dimensional feature vector may be calculated for each of the extracted face images, and the similarity of the calculated feature vectors may be calculated to cluster the extracted face images into a plurality of persons.

In the generating of the locus information, the locus information of each person may be generated by connecting face coordinate information in a frame having face images clustered into the same person.

On the other hand, the content generation method further comprises the step of calculating the movement speed for each section using the generated trajectory information, and verifying the generated trajectory information for each person by checking whether the calculated movement speed has a predetermined value or more. can do.

In the meantime, the content generation method includes a first region displaying a section in which a face of a specific person is detected among all sections of an input image, a second region displaying a specific frame of the input image and a detected face in the specific frame; The method may further include displaying a user interface window including a third area displaying an extracted image corresponding to the specific person in the specific frame.

Meanwhile, in a computer-readable recording medium including a program for executing a content generating method according to an embodiment of the present disclosure, the content generating method includes a face image included in each of a plurality of frames included in an input photographed image. Extracting an image, clustering the extracted face image into a plurality of persons, generating trajectory information for each person based on the face image clustered with the same person, and the input image and the generated trajectory information Generating the content for each person by using;

As described above, according to various embodiments of the present disclosure, the content for each person may be generated using only one captured image, thereby reducing costs and rapidly editing individual vertical images edited for each person appearing on the stage. You can create a batch. In addition, since the vertical image is produced by the second half editing, the direct cam image of the required person can be generated at any time without limiting the shooting target. In addition, it is possible to precisely track by person using AI face recognition technology, it is possible to create high-quality content. The vertical timeline enables faster inspection and correction of face position detection results than an NLE in the horizontal timeline.

1 is a view showing the configuration of a content generation system according to an embodiment of the present disclosure;
2 illustrates a detailed configuration of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining a content generation method using an AI technology according to an embodiment of the present disclosure;
4 is a view for explaining an image cropping method;
5 is a view for explaining a face detection method;
6 is a view for explaining a clustering method using the detected face image;
7 is a view for explaining an example of a user interface window;
8 is a view showing a comparison example of the content for each member generated according to the present disclosure and the conventional content,
9 is a diagram illustrating an example of content generated according to a further embodiment; and
10 is a diagram for describing a content generation method according to one embodiment of the present disclosure.

Terms used herein will be briefly described, and the present disclosure will be described in detail.

The terms used in the embodiments of the present disclosure selected general terms widely used as far as possible in consideration of functions in the present disclosure, but may vary according to the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in certain cases, there is also a term arbitrarily selected by the applicant, in which case the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meanings of the terms and the contents throughout the present disclosure, rather than simply the names of the terms.

Embodiments of the present disclosure may be variously modified and have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the scope to the specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the scope of the disclosed spirit and technology. In describing the embodiments, when it is determined that the detailed description of the related known technology may obscure the gist, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, "includes." Or "made up." And the like are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features or numbers, step, action, component, part, or It should be understood that they do not preclude the presence or possibility of adding these in advance.

In an embodiment of the present disclosure, the 'module' or 'unit' performs at least one function or operation, and may be implemented in hardware or software or in a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module except for 'modules' or 'units' that need to be implemented by specific hardware, and may be implemented as at least one processor.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present disclosure. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted for simplicity of explanation, and like reference numerals designate like parts throughout the specification.

Hereinafter, with reference to the drawings will be described in more detail with respect to the present disclosure.

1 is a diagram illustrating a configuration of a content generation system according to an exemplary embodiment.

Referring to FIG. 1, the content generation system 1000 according to an exemplary embodiment includes a photographing apparatus 10, an electronic apparatus 100, and a server 20.

The photographing apparatus 10 captures an image to generate a photographed image. The photographing apparatus 10 may be a digital camera, a camcorder, a mobile phone, a PMP, a webcam, a black box, or the like capable of capturing an image.

In addition, the photographing apparatus 10 may be disposed at a position where a plurality of persons may be photographed as a subject to perform a photographing operation. In this case, the photographing apparatus 10 may photograph only in a fixed direction, or may photograph by changing a direction in response to a change in movement of the person.

In addition, the photographing apparatus 10 may transmit the generated photographed image to the electronic device 100. Here, the captured image is a high quality video, and may have a resolution of 4K or more. The captured image may include a sound source such as sound. In some implementations, the sound may be directly received from an external device other than the photographing device (for example, a singer's microphone or an actual broadcast sound source).

The electronic device 100 may receive a captured image and generate content for each person included in the image based on the input captured image. In detail, the electronic device 100 may extract face images included in the image, cluster the extracted face images, generate trajectory information for each person, and generate content for each person based on the generated trajectory information. A detailed configuration and operation of the electronic device 100 will be described later with reference to FIG. 2.

The electronic device 100 may provide the generated person-specific content to the server 20. Here, the electronic device 100 and the server 20 may be connected to the Internet network, and the electronic device 100 and the photographing device 10 may be directly connected to an internal network network.

The server 20 receives the content for each person and stores the received content for each person. In addition, the server 20 may transmit or provide the stored content for each person in response to a request of an individual terminal device.

As described above, the content generation system 1000 according to the present exemplary embodiment may generate and provide a plurality of contents (ie, content for each person) using only the captured image generated by one photographing apparatus. In addition, it is possible to reduce costs in that only one shot is required to generate content for each person, and to produce content of higher quality more easily and quickly in terms of creating content for each person using AI technology. It is possible.

Meanwhile, in FIG. 1, the photographing apparatus 10, the electronic device 100, the server 20, and the electronic device 100 are illustrated as directly connected to each other. It may be connected in a form via configuration. In addition, although it has been described that the content generated by the electronic device 100 is provided to only one server 20, the electronic device 100 may also provide the plurality of servers 20.

2 illustrates a detailed configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device 100 may include a communication device 110, a memory 120, a display 130, a manipulation input device 140, and a processor 150. The electronic device 100 may be a PC, a notebook PC, a smartphone, a server, or the like capable of image processing.

The communication device 110 may be connected to the photographing device 10 or the server 20 and may transmit and receive content. Specifically, the communication device 110 is formed to connect the electronic device 100 to an external device, and is connected to a mobile device through a local area network (LAN) and an internet network, as well as a USB (Universal). Serial Bus) can also be connected via the port.

In addition, the communication device 110 may be connected to another electronic device through a router or a router connected to a public Internet network as well as a wired method, and may be connected to a router or a router with a wired method as well as Wi-Fi, Bluetooth, cellular communication, or the like. It can also be connected wirelessly. The cellular communication may include, for example, Long-Term Evolution (LTE), LTE Advance (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), and Wireless Broadband (WiBro). ), Or Global System for Mobile Communications (GSM).

The communication device 110 may receive a captured image from the photographing device 10. The captured image may have various image formats. The photographed image may be a photographed image of a singer on a stage, but is not limited thereto and may include various characters such as an opera stage, a play, and a sports image.

The communication device 110 may receive sound source content from a separate external device. The sound source content may be a sound source corresponding to the captured image, and may be one sound source or a plurality of sound sources for each person. That is, the communication device 110 may receive a plurality of voice contents for each of the plurality of personnel. In this case, the plurality of received sound source contents may be directly provided through a microphone for each singer member, or may be provided through a broadcasting system.

In addition, the communication device 110 may transmit the person-specific content generated by the electronic device 100 to the server 20.

The memory 120 is a component for storing O / S for driving the electronic device 100, software for generating content for each person, AI algorithm, data, and the like. The memory 120 may be implemented in various forms such as RAM, ROM, flash memory, HDD, external memory, memory card, and the like, but is not limited thereto.

The memory 120 stores the captured image. In detail, the memory 120 may store a captured image received through the communication device 110. In addition, the memory 120 may temporarily store a plurality of frames extracted from the captured image for image editing, and may store content for each person generated last. The captured image may be a video file such as AVI, MOV, or the like.

In addition, the memory 120 may store the extracted face image, trajectory information, and the like calculated during the content generation process.

The memory 120 may store an AI algorithm (or program) for face image detection and face image clustering. Here, the AI algorithm is an algorithm that operates in a Tensorflow-based machine learning framework, and may be facenet or chinese whispers.

The display 130 displays a user interface window for selecting a function supported by the electronic device 100. In detail, the display 130 may display a user interface window for selecting various functions provided by the electronic device 100. The display 130 may be a monitor such as an LCD, a CRT, an OLED, or the like, and may be implemented as a touch screen capable of simultaneously performing a function of the manipulation input device 140 to be described later.

The display 130 may display a user interface window for verifying or correcting the trajectory information for each person. The user interface window may include a first area displaying a section in which a face of a specific person is detected among all sections of the input image, a second frame displaying a detected frame in a specific frame and a specific frame, and a specific frame in a specific frame. It may include a third area for displaying the extracted image corresponding to the person. An example of a user interface window that can be displayed on the display 130 will be described later with reference to FIG. 7.

In addition, the display 130 may display a verification result for the trajectory information. The display 130 may display the input photographed image or may display the generated content for each person.

The manipulation input device 140 may receive a function selection of the electronic device 100 and a control command for the corresponding function from the user. In detail, the manipulation input apparatus 140 may receive a user control command for correcting the calculated trajectory information. The user control command may be input through the above-described user interface window.

The processor 150 performs control of each component in the electronic device 100. In detail, when a boot command is input from a user, the processor 150 may perform booting using an operating system stored in the memory 120.

When the captured image is input, the processor 150 extracts a plurality of frames included in the input captured image. In this case, the plurality of frames may be all frames included in the captured image, and may be frames extracted at predetermined time intervals (for example, 0.5 seconds). The extracted frame may have an image format such as BMP or JPG.

The processor 150 extracts a face image included in each of the plurality of frames included in the input captured image. In detail, the processor 150 may detect a face included in each of the plurality of frames by using an AI algorithm learned about the face detection operation. Here, the AI algorithm uses a tensorflow-based face detection algorithm, and various AI algorithms capable of face detection may be used.

When the face is detected in the frame, the processor 150 may extract a face image corresponding to the detected face region, and extract coordinates and size information of the face image. When a plurality of faces are detected in one frame, the processor 150 may extract the detected face-specific image, coordinates, and size information.

The processor 150 clusters the extracted face image into a plurality of persons. In detail, the processor 150 may calculate a 128-dimensional feature vector for each of the extracted face images. Subsequently, the processor 150 may cluster the extracted plurality of images into a plurality of persons by comparing the feature vector values of each image, that is, calculating the mutual similarity of the calculated feature vectors. In this case, the processor 150 may input the feature vector calculated in the clustering algorithm and cluster the extracted face image into a plurality of persons.

The processor 150 generates trajectory information for each person based on face images clustered into the same person. In detail, the processor 150 may generate face information for each person by connecting face coordinate information in a frame having face images clustered into the same person.

On the other hand, the singer on the stage may be located in a direction in which the face is not visible (for example, looking at the face downward or standing in the back of the audience). In order to prepare for such a case, if there is no face image of a specific person in the preset frame, the processor 150 may generate trajectory information by using face coordinate information of a frame immediately before the face image. In the implementation, the trajectory information of the frame with the face image may be used after the frame without the face image, and the average trajectory information immediately before and after the face image may be used.

On the other hand, the singer on the stage may go out of the stage for a while depending on the movement. In order to prepare for such a case, if there is no face image of a specific person over a preset frame, the processor 150 may generate the track information of the specific person using user trajectory information of another person in the frame section. In this case, the other person may be a person having the longest trajectory information. On the other hand, the implementation may also be implemented in the form of using the coordinate information of the face coordinates of the last frame in which the face image, not the trajectory of another person.

The processor 150 verifies the generated trajectory information for each person. In detail, the processor 150 may calculate the movement speed for each section using the generated trajectory information, and determine that there is an error in the frame corresponding to the section having the movement speed greater than or equal to a preset value. In this case, the processor 150 may control the display 130 to display a user interface window for correcting the trajectory of the frame region in which the error is confirmed, and may modify the trajectory according to a correction command input through the manipulation input device 140. have.

The processor 150 generates content for each person using the input photographed image and the generated trajectory information. In detail, the processor 150 may extract a vertical image having a length longer than the width of each of the plurality of frames included in the input captured image. The vertical image may have a resolution suitable for the vertical mode of the smartphone.

In this case, the processor 150 may identify the face position of the person in the frame by using the generated trajectory information and extract a vertical image corresponding to the face size of the person based on the identified face position. For example, the face size of the person may vary depending on whether they are located in front of or behind the stage. Correspondingly, the processor 150 may vary the size of the extraction region based on the confirmed face position so that the zoom-in / zoom-out operation of the image and the possible effects may occur. Even if the size of the region is variable, the aspect ratio of the extracted vertical image may be maintained at the same value. In this case, the ratio may correspond to the horizontal / vertical ratio of the smartphone.

On the other hand, in the implementation, the processor 150 allows the face of the person to be located in the center if possible, but moves the area of the extracted vertical image only when the position of the face is shifted more than a certain amount of margin. Can be extracted.

Meanwhile, when a face of a specific user is located at both ends of a frame and an image is extracted based on the face, a normal vertical image may not be extracted. In this case, the processor 150 may extract a vertical image filled with a black (or blank area) in an area without a frame image. Alternatively, the processor 150 may prevent the key frame, which is a reference for extracting the vertical image, from being located outside the frame, that is, only if the face of the person is not located in the center.

The processor 150 generates vertical content for each person using the extracted vertical image. In detail, the processor 150 may generate vertical content (ie, a video) by rendering (or encoding) a vertical image extracted for each person.

When receiving a plurality of sound source data, the processor 150 may generate a vertical video by merging the vertical image corresponding to the person and the sound source data corresponding to the person. Meanwhile, the mapping between the plurality of sound source data and the plurality of vertical contents may be performed by a user's selection, or may be automatically performed through voice recognition and face recognition.

As described above, the electronic device 100 according to the present embodiment may generate a plurality of contents (ie, content for each person) using only the captured image generated by one photographing apparatus and provide the same to the users. In addition, the electronic device 100 according to the present embodiment can reduce costs in that only one shot image is required to generate content for each person, and in that it generates content for each person by using AI technology. It is possible to produce higher quality content more easily and quickly.

Meanwhile, in FIGS. 1 and 2, the face image is extracted and used as a result of face detection. However, in the implementation, only the face coordinate information is used without using a face image extraction operation. It is also possible to perform clustering using the generated face coordinate information. The face coordinate information may utilize a pre-detected result, an arbitrarily set face coordinate, or face coordinate information detected by another face detection program.

3 is a diagram for describing a content generation method using AI technology according to an exemplary embodiment.

Referring to FIG. 3, a high quality video is input 310. The input video may be a video having 4K quality or higher. If the quality of the input video is 4K or more, the cropped image is smaller than or equal to the original image, and the video quality of each final person has the maximum original resolution.

In the following description, it is assumed that the input video is an image of photographing several singers simultaneously on the stage.

Since the dance singer moves continuously on the stage, when using the general shooting method, the photographer must continuously adjust the imaging direction of the camera in response to the moving singer's moving line. If the user directly uses the method of setting the crop region of the image in the input image, the user should adjust the crop region in response to the moving line of the dance singer.

However, the two existing modes, the photographer or the user, need to adjust the direction of the image and the crop area. In the case of a vigorous dance singer, since the position changes every frame unit, the user changes the crop area in response to the user's position change. It is impossible to change.

Accordingly, in the present disclosure, a crop region is automatically calculated using a deep learning based face detection technique. Hereinafter, a method of calculating a crop region for each singer using a deep learning based face detection technology will be described in detail.

First, when a high quality video is input, the frame for the input video is sampled (320). In detail, when the face detection operation is performed on all the frames, since the image processing time is considerably longer, only frames of a predetermined time unit may be preferentially sampled to improve the processing speed. For example, a frame of 0.5 seconds may be sampled.

On the other hand, in the implementation, for more accurate trekking, finer sampling than the above-described units may be performed. In addition, the above-described numerical values are merely exemplary, and may be used by calculating a time value that satisfies both precision and processing speed through experiments.

In operation 330, a face detection operation is performed on each of the sampled frames. In detail, faces included in each of the sampled frames may be detected by using a face detection algorithm based on deep learning. Through this process, the face image included in each frame may be extracted, and various information such as coordinates on the frame of the face image and the size of the face image may be generated.

In operation 340, clustering of the extracted face images is performed. First, a 128-dimensional feature vector for each of the extracted face images may be extracted and classified into a plurality of persons through an operation of calculating mutual similarity with respect to faces expressed as vector values. As a result, by connecting the face coordinates of the face image grouped with the same person, it is possible to calculate the motion trajectory data for each person.

On the other hand, the face of the person may be recognized differently depending on the direction to be taken by the camera. That is, different people may be bundled into the result separated by the above-described clustering process, and one person's face may be classified into a plurality of people.

That is, since there may be various errors in the trajectory data generated at first, a speed verification operation is performed to verify whether the face of another person is bound in the result classified as one person. For example, in the first frame, it is confirmed that a specific user is on the left side, and in the next frame, it is confirmed that the specific user is at the rightmost side. That is, moving faster than the physical moving speed may indicate that the bundle in the corresponding section is incorrect.

Therefore, using the calculated trajectory data, it is possible to calculate the moving speed of the person (or subject), and if the calculated moving speed exceeds the physical moving speed of the person, it can be recognized that there is an error.

When the first verification is completed, a key frame including track information for each person may be extracted (360). The key frame is information indicating an image region (ie, a crop region) to be extracted from the input image according to the person's trajectory information. The key frame may be determined based on the extracted face coordinates of the person in consideration of the size of the face and the position of the face. For example, when the size of a face of a specific frame is calculated to be large, the user is standing in front of another frame, so that the crop region may be determined by zooming out. Alternatively, when the face coordinates of a specific frame are lower than others, it may be considered that the person performs the sitting operation. The crop area may be determined by zooming in.

The crop region may have a vertical shape. That is, the vertical image is longer than the horizontal image, and may have a resolution that is easy for a user to view in the vertical grip state of the smartphone.

In order to secondaryly perform the above-described verification, a key frame modification operation is performed (370). For this operation, the electronic device 100 may display a user interface window for trajectory verification. An example of the displayed user interface window will be described later with reference to FIG. 7.

When key frame modification is completed, an image is extracted according to the modified key frame for each frame (380). In operation 390, a video for each person may be generated by encoding (or rendering) the extracted image and voice data (or other voice data provided from the outside) included in the input image. In this case, the number of generated videos may correspond to the number of persons classified in the previous clustering step. For example, when five people are classified in the previous clustering step, five images may be generated. Meanwhile, if five keyframes are adjusted to four in the keyframe modification process, four images corresponding to the adjusted number of keyframes may be generated.

On the other hand, when the user inputs that there are four people in the corresponding image, the electronic device 100 automatically calculates only the trajectories for the top four of the number of face images included in each person among the five people classified in the clustering step. You can also create four videos. The generated video may be transmitted to an external server or the like.

Meanwhile, in FIG. 3, the vertical image longer than the horizontal length is extracted and the vertical content is generated accordingly. However, in the implementation, the content having the same horizontal and vertical lengths may be generated. In addition to this, it is also possible to generate the content in the form of another polygon (for example, trapezoid, etc.).

4 is a view for explaining an image cropping method.

Referring to FIG. 4, four persons are included in one sampled frame 410. Through the AI face detection engine described above, crop areas for each of the four people included in the frame are set, and as shown, four crop images 420, 430, 440, and 450 are extracted.

Looking at each cropped image 420, 430, 440, 450, it can be seen that it is in the form of a vertical image and the face of the person has a central position. On the other hand, in order to have the face of the person in the center of the crop image, it is necessary to know where the face of the person in the frame is located. Hereinafter, a face detection operation will be described with reference to FIG. 5.

5 is a diagram for describing a face detection method.

Referring to FIG. 5, it can be seen that four faces 510, 520, 530, and 540 are detected in one frame 500. Such face detection may be performed using a face detection library operating on the Tensorflow framework.

On the other hand, such a face detection is performed by splitting the image frame by frame, and if there are several characters, the face coordinates are also returned. Therefore, only the front and rear frame detection values can be connected to track the movement of each person. none.

Therefore, in the present disclosure, in order to track the movement of the person, a clustering operation is performed. This will be described below with reference to FIG. 6.

6 is a diagram illustrating a clustering method using the detected face image.

Referring to FIG. 6, an example of the extracted face image is illustrated. Although only nine extracted images are illustrated in the illustrated example, the image to be clustered may be the number of faces of a person included in each frame to be sampled. For example, when a three-minute image is sampled in units of 0.5 seconds and four people are fixedly positioned in each image, 1440 face images may be used.

For each of these face images, a 128-dimensional feature vector is extracted. The feature vector is for characterizing a face and may be a value associated with a feature value (eye size, eye and nose angle, etc.) of the face.

When the feature vector is extracted, the extracted face image may be classified into a plurality of persons through an operation of calculating mutual similarity between the calculated vector values. When the face coordinates of the face images grouped by the same person are connected, the motion trace data of each person can be calculated.

7 is a diagram for explaining an example of a user interface window.

Referring to FIG. 7, the UI window 700 may include a first area 710, a second area 720, and a third area 730.

The first area 710 displays a section in which a face of a specific person is detected among all sections of the input image. The user may check a time section in which a face for a specific person is not detected or a face is detected through the first region. For example, the user may check only a section in which a face is not detected to confirm that a face is not detected. The detected time interval may be selected to determine whether face detection and classification are properly performed.

As shown, the first region 710 of the present disclosure is a vertical timeline. In other words, unlike the non-linear editing system (NLE) program in the horizontal timeline, continuous frame still images are arranged vertically based on the editing time point, and the mouse wheel can be used to quickly check whether the face position detection result is normal. It is a vertical timeline.

The second area 720 displays a specific frame of the input image and a detected face in the specific frame. In this case, as illustrated in FIG. 7, the second region may also display a frame image immediately before or after the specific frame. For example, consecutive frames may use three or more frames in total, before, and after the editing point. In addition, since the original image can be checked in real time and the face position can be checked and corrected, it is easy to identify the movement of the person.

The third area 730 is an area for displaying an extracted image corresponding to a specific person in a specific frame. The user may check the final result (vertical image) through the third area 730.

In an implementation, the user interface window 700 may include an area for selecting a person, and a plurality of first to third areas may be simultaneously displayed for each person. In addition, the implementation may be implemented in a form in which any one of the first region to the third region is omitted.

8 is a diagram illustrating a comparison example of content for each member generated according to the present disclosure with conventional content. Specifically, the first image is a result of conventional content and the second image is a result of content generated in accordance with the present disclosure.

Referring to FIG. 8, while the position of the person is biased to the right in the first image, the position of the person may be determined in the second image. This is because in the present embodiment, precise subject tracking is possible by face recognition through AI technology. In other words, since the face coordinates of the person are known, the image is generated accurately fixed at the top center of the image.

In particular, in the conventional direct photographing method, if a subject is missed in the photographing process, there is no method of restoring. However, the photographed image according to the present disclosure has an advantage that the subject is not missed because it photographs the foreground of the stage. In addition, since trekking is performed through AI technology, precise tracking is possible even when trekking a vigorous dance singer.

Meanwhile, in FIG. 8, the figure is described and described as generating content for each person by performing only precise trekking in the final image generation process. However, in the implementation, the person may be expressed as if the person is intentionally set by the user. It is also possible to create star content. That is, it is possible to calculate keyframes in various ways according to the detected face position, and in the implementation, it is also possible to calculate various contents accordingly. For example, users beginner, intermediate. If you select any of the advanced, you can output the results of trekking according to the selected class. In addition, when a plurality of grades are selected, the electronic device 100 may generate a plurality of contents for each person and grade.

In addition, the implementation may not only output content corresponding to one person but also generate and output merged content in which images of a plurality of people are merged. In addition, the player can detect the position of the hand rather than the face, and can output content for body parts other than the face. For example, if the image is about playing the piano, only the image corresponding to the piano player's hand may be output or the image of each of the player's face and hand may be merged and displayed.

In addition, as shown in FIG. 9, in the case of an ensemble using various musical instruments, a hand image of a first person playing a piano and a hand image of a second person playing a guitar may be merged and output.

10 is a diagram for describing a content generation method according to one embodiment of the present disclosure.

The face image included in each of the plurality of frames included in the input captured image is extracted (S1010). A face included in each of the plurality of frames may be detected using a previously learned face detection AI algorithm, and a face image corresponding to the detected face, coordinate information of the face image, and size information of the face image may be extracted.

In operation S1020, the extracted face image is clustered into a plurality of persons. In detail, a 128-dimensional feature vector may be calculated for each of the extracted face images, and the similarity of the calculated feature vectors may be calculated to cluster the extracted face images into a plurality of persons.

In operation S1030, the trajectory information for each person is generated based on the face images clustered with the same person. In more detail, trajectory information for each person may be generated by connecting face coordinate information in a frame having face images clustered with the same person.

In this case, the primary verification may be performed using the generated trajectory information. In detail, the movement speed for each section may be calculated using the generated trajectory information, and the generated movement trajectory information may be verified by checking whether the calculated movement speed has a predetermined value or more.

In addition, the second verification may be received from the user through the operation of displaying the generated trajectory information. In detail, a first region displaying a section in which a face of a specific person is detected among all sections of the input image, a second frame displaying a detected frame in a specific frame and a specific frame of the input image, and a specific person in a specific frame. A user interface window including a third area displaying a corresponding extracted image may be displayed. Accordingly, the user may check the generated trajectory through the displayed user interface window, and if the modification is needed during the checking process, the trajectory may be directly modified through the corresponding user interface window.

In operation S1040, content for each person is generated using the input photographed image and the generated trajectory information. The generated content may be transmitted to an external server.

Therefore, in the file transfer method according to the present embodiment, a plurality of contents (ie, content for each person) may be generated and provided to users by using only the captured image generated by one photographing apparatus. 10 may be executed on an electronic device having the configuration of FIG. 2 or may be executed on an electronic device having another configuration.

In addition, the content generation method as described above may be implemented as a program including an executable algorithm that may be executed on a computer, and the above-described program may be stored and provided in a non-transitory computer readable medium. Can be.

A non-transitory readable medium refers to a medium that stores data semi-permanently and is read by a device, not a medium storing data for a short time such as a register, a cache, a memory, and the like. Specifically, programs for performing the above-described various methods may be stored and provided in a non-transitory readable medium such as a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, or the like.

In addition, while the above has been shown and described with respect to preferred embodiments of the present disclosure, the present disclosure is not limited to the specific embodiments described above, the technical field to which the disclosure belongs without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications can be made by those skilled in the art, and such modifications should not be individually understood from the technical spirit or the prospect of the present disclosure.

1000: content generation system 10: imaging device
20: server 100: electronic device
110: communication device 120: memory
130: display 140: operation input device
150: processor

Claims (20)

In an electronic device,
A communication device receiving a captured image;
A memory for storing the input captured image; And
Extracting face images included in each of the plurality of frames included in the input photographed image, clustering the extracted face images into a plurality of persons, and generating track information for each person based on face images clustered into the same person And a processor configured to generate content for each person using the input photographed image and the generated trajectory information.
The processor,
And calculating the movement speed for each section by using the generated trajectory information, and verifying the generated trajectory information for each person by checking whether the calculated movement speed has a predetermined value or more. delete delete delete delete delete delete delete The method of claim 1,
A first area indicating a section in which a face of a specific person is detected among all sections of the input image, a second frame indicating a specific frame of the input image and a detected face in the specific frame, and the specific person in the specific frame And a display configured to display a user interface window including a third area displaying an extracted image corresponding to the image. The method of claim 9,
The first area,
The electronic device as a vertical timeline displaying a section in which the face of the specific person is detected in the vertical direction. The method of claim 1,
The processor,
And extracting a vertical image for each of the plurality of frames included in the input photographed image for each person having a vertical length longer than a horizontal length, and generating vertical content for each person using the extracted vertical image. The method of claim 11,
The processor,
The electronic device identifies a face position of a person in a frame by using the generated trajectory information, and extracts an image area corresponding to the face size of the person as a vertical image based on the identified face position. The method of claim 1,
The processor,
And generating merged content by merging a plurality of person contents among the generated person contents. In the content creation method,
Extracting a face image included in each of the plurality of frames included in the input captured image;
Clustering the extracted face images into a plurality of persons;
Generating trajectory information for each person based on face images clustered into the same person;
Calculating the movement speed for each section by using the generated locus information, and verifying the generated locus information by checking whether the calculated movement speed has a predetermined value or more; And
And generating content for each person using the input photographed image and the generated trajectory information. delete delete delete delete The method of claim 14,
A first area indicating a section in which a face of a specific person is detected among all sections of the input image, a second frame indicating a specific frame of the input image and a detected face in the specific frame, and the specific person in the specific frame And displaying a user interface window including a third area displaying an extracted image corresponding to. A computer readable recording medium comprising a program for executing a method of creating a content, the method comprising:
The content generation method,
Extracting a face image included in each of the plurality of frames included in the input captured image;
Clustering the extracted face images into a plurality of persons;
Generating trajectory information for each person based on face images clustered into the same person;
Calculating the movement speed for each section by using the generated locus information, and verifying the generated locus information by checking whether the calculated movement speed has a predetermined value or more; And
And generating content for each person by using the input photographed image and the generated trajectory information.

Images