KR102457176B1 - Electronic apparatus and method for generating contents - Google Patents

Abstract

Disclosed is an electronic device. The electronic device comprises: a communication device for receiving captured images in real time; and a processor for generating a face ID and coordinate information for each of a plurality of face regions included in the input captured images, and generating image content for each of a plurality of faces in real time using the generated face ID and coordinate information and the captured images. The processor detects a face included in an input captured image using a pre-trained face detection deep learning model, extracts a face image corresponding to the detected face and the coordinate information of the face image, extracts the unique value of the face image, and compares the extracted unique value with a pre-stored unique value for each person to determine the face ID of the face image. Therefore, provided are an electronic device and a method for generating content, wherein content can be generated automatically in real time for each person in a captured image.

Description

ELECTRONIC APPARATUS AND METHOD FOR GENERATING CONTENTS

The present disclosure relates to an electronic device and a content generating method, and more particularly, to an electronic device capable of automatically generating content for each person in a captured image in real time and a content generating method.

Recently, broadcasts are filmed using many cameras to show various viewpoints. For example, in order to take a one-shot of a performer with good reaction, not only a general camera such as a foreground, a libero, etc., but also a dedicated camera for each performer is used.

Since recent cameras support high resolution, various methods for generating individual images by editing images captured by one camera have been proposed.

However, the existing method cannot be used when only applicable to a specific program (eg, music broadcast), individual content creation is possible only by post-processing of a filmed image, or there are multiple people in the image. Alternatively, there are limitations such as being able to generate an individual image for only one of a plurality of people.

Broadcasters have various programs such as entertainment programs, current affairs programs, and news programs as well as music programs, and since programs such as news or live Internet broadcasting require real-time processing, it is possible to A content creation method was required.

Accordingly, an object of the present disclosure is to provide an electronic device capable of automatically generating content for each person in a captured image in real time and a content generating method.

An electronic device according to the present disclosure for achieving the above object is a communication device that receives a captured image in real time, generates face ID and coordinate information for each of a plurality of face regions included in the input captured image, and and a processor for generating image content for each of the plurality of faces in real time by using the generated face ID and coordinate information and the captured image, wherein the processor uses a pre-learned face detection deep learning model to receive the input Detects a face included in the captured image, extracts a face image corresponding to the detected face, coordinate information of the face image, extracts a unique value for the face image, and extracts the extracted unique value and a pre-stored person A face ID for the face image is determined by comparing the unique values of each star.

In this case, the processor extracts a frame image for the captured image in units of a preset time period, generates face ID and coordinate information for each of a plurality of face regions included in the extracted frame image, and Within a time period, image content for each of the plurality of faces may be generated in real time using the face ID and coordinate information.

In this case, the processor may perform a preset pre-processing operation on the extracted frame image, and generate face ID and coordinate information for each of a plurality of face regions included in the frame image on which the pre-processing operation is performed.

In this case, the pre-processing operation may include at least one of a process of converting an image file format to a [Channel, Width, Height] RGB format and a process of reducing the image resolution to a preset size.

On the other hand, the processor inputs the inputted captured image to the pre-learned face detection deep learning model, obtains list information about the face position and the probability of the face position, and a face having a lower probability than a preset value The faces in the captured image may be detected by filtering the positions and performing Non-Maximum Suppression (NMS) to integrate faces with overlapping positions.

Meanwhile, the processor may compare the immediately detected coordinate information with the coordinate information, and when the movement of the coordinate information is equal to or greater than a preset value, the processor may generate image content using the changed coordinate information.

Meanwhile, the processor may control the communication device to transmit the image content for each of the plurality of faces to different output ports.

Meanwhile, the processor may generate merged content by merging at least two of the generated image content for each of the plurality of faces.

On the other hand, the content generation method according to an embodiment of the present disclosure includes the steps of receiving a captured image in real time, detecting a face included in the input captured image, using a pre-learned face detection deep learning model, and generating a face image corresponding to a face, determining coordinate information of the face image, extracting a unique value for the face image, and comparing the extracted unique value with a pre-stored unique value for each person to obtain the face image determining a face ID for , and generating image content for each of the detected faces in real time using the face ID, the coordinate information, and the captured image.

In this case, the content creation method further includes extracting a frame image for the captured image in units of a preset time period, and determining the coordinate information of the face image includes: Generating face ID and coordinate information for each of the plurality of face regions, wherein the generating includes generating image content for each of the plurality of faces in real time using the face ID and coordinate information within the preset time period can do.

In this case, the extracting may include performing a preset pre-processing operation on the extracted frame image, and generating face ID and coordinate information for each of a plurality of face regions included in the frame image on which the pre-processing operation is performed. have.

In this case, the pre-processing operation may include at least one of a process of converting an image file format to a [Channel, Width, Height] RGB format and a process of reducing the image resolution to a preset size.

On the other hand, the step of determining the coordinate information of the face image includes inputting the input captured image to the pre-learned face detection deep learning model to obtain list information about the face position and the probability of the face position, Face positions having a lower probability than a preset value may be filtered, and faces in the captured image may be detected by integrating faces with overlapping positions by performing non-maximum suppression (NMS).

Meanwhile, in the generating step, the coordinate information detected immediately before may be compared with the coordinate information, and when the movement of the coordinate information is equal to or greater than a preset value, the image content may be generated using the changed coordinate information.

Meanwhile, the method for generating content may further include transmitting image content for each of the detected faces to different output ports.

Meanwhile, the method for generating content may further include generating merged content by merging at least two of the generated image content for each of the plurality of faces.

On the other hand, in a computer-readable recording medium including a program for executing a content generating method according to an embodiment of the present disclosure, the content generating method includes receiving a captured image in real time, deep learning for pre-learned face detection Detecting a face included in the input captured image using a model, generating a face image corresponding to the detected face, determining coordinate information of the face image, and extracting a unique value for the face image and determining a face ID for the face image by comparing the extracted unique value with a pre-stored unique value for each person. and generating video content in real time.

In this case, the determining of the coordinate information includes inputting the inputted captured image to the pre-learned face detection deep learning model, acquiring list information about the face position and the probability of the face position, and a preset value Face positions having a lower probability may be filtered, and faces in the captured image may be detected by integrating faces with overlapping positions by performing non-maximum suppression (NMS).

Meanwhile, in the generating, the coordinate information detected immediately before may be compared with the coordinate information, and when the movement of the coordinate information is equal to or greater than a preset value, the image content may be generated using the changed coordinate information.

Meanwhile, the method for generating content may further include transmitting each of the image content for each of the plurality of faces to different output ports.

As described above, according to various embodiments of the present disclosure, it is possible to generate a plurality of content for each person in real time with only one captured image, thereby reducing costs.

1 is a view showing the configuration of a content creation system according to an embodiment of the present disclosure;
2 is a diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining a content creation method according to an embodiment of the present disclosure;
4 is a flowchart for explaining the content creation method of FIG. 3 in more detail;
5 is a view for explaining a content creation method according to another embodiment of the present disclosure;
6 is a view comparing the configuration of a camera in an existing shooting environment and a camera configuration when a shooting environment according to the present disclosure is applied;
7 is a diagram illustrating an example of content generated according to the present disclosure;
8 is a view for explaining a content creation method according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in specific cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

Embodiments of the present disclosure may apply various transformations and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and it should be understood to include all transformations, equivalents and substitutions included in the spirit and scope of the disclosure. In describing the embodiments, if it is determined that a detailed description of a related known technology may obscure the subject matter, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this application, "includes." Or "consistent." The term such as is intended to designate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, but one or more other features or number, step, action, component, part or It should be understood that it does not preclude the possibility of the existence or addition of combinations thereof.

In an embodiment of the present disclosure, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a diagram illustrating the configuration of a content creation system according to an embodiment of the present disclosure.

Referring to FIG. 1 , a content creation system 1000 according to an embodiment of the present disclosure includes a photographing device 10 and an electronic device 100 .

The photographing apparatus 10 captures an image to generate a photographed image. The photographing device 10 may be a digital camera, camcorder, mobile phone, PMP, webcam, black box, or the like capable of capturing an image.

In addition, the photographing apparatus 10 may be disposed at a position capable of photographing a plurality of persons as subjects to perform a photographing operation. In this case, the photographing apparatus 10 may perform photographing only in a fixed direction, or may perform photographing by changing the direction in response to a change in the movement of the people.

On the other hand, when a plurality of people are located on the same line and the shooting distance is long, the focus may be slightly different between the center and the periphery in the captured image. In this case, the image quality of the person located in the center and the image quality of the person located in the vicinity may be different. In order to solve this problem, a plurality of persons are arranged to be located at the same distance from the photographing device 10 , or different image processing is performed for each area on the captured image, and content for each person is generated in a process to be described later. quality differences can be avoided.

In addition, the photographing apparatus 10 may transmit the generated photographed image to the electronic apparatus 100 . Here, the captured image is a high-quality video, and may have a resolution of 4K or higher (eg, 8K, 6K, 4K). In addition, the captured image may include a sound source such as sound. Meanwhile, in implementation, the sound may be directly received from an external device other than the photographing device (eg, a singer's microphone or an actual broadcast sound source).

The electronic device 100 may receive a captured image and generate content for each person included in the image based on the input captured image. Specifically, the electronic device 100 may detect a face included in an image, generate user recognition for the detected face and coordinate information on a face region, and generate content for each person using the generated coordinates. . A detailed configuration and operation of the electronic device 100 will be described later with reference to FIG. 2 .

In addition, the electronic device 100 may transmit the generated content for each person to the outside in real time. In this case, the electronic device 100 may transmit each of the generated content for each person to different output ports.

As described above, the content generating system 1000 according to the present embodiment can generate a plurality of content (ie, content for each person) using only a captured image generated by one photographing device and provide it to users. In addition, in creating content for each person, it is possible to reduce costs in that only one shot video is required, and in that content for each person is generated using deep learning technology, it is easier and faster to produce higher quality contents more easily and quickly. it is possible In addition, since a deep learning technology suitable for high-speed processing is used, it is possible to generate content for each person in real time even if a high-resolution image is used.

Meanwhile, in the description of FIG. 1 , the photographing device 10 and the electronic device 100 are illustrated in a form in which they are directly connected to each other. For example, an image captured by the photographing device 10 is stored in real time in a recorder or server that primarily stores the image, and it is also possible to generate content for each person by receiving the image from the recorder or server. .

Also, although it has been described that the electronic device 100 of FIG. 1 is implemented as a single device, the electronic device 100 may be implemented as a plurality of devices. For example, the first electronic device receives an image, provides frame information necessary for detecting a person to the second electronic device, detects a face using the frame information provided by the second electronic device, and generates coordinate information for each face It is also possible for a plurality of devices to divide and perform the above-described operation of the electronic device of the present application in the form of providing the information to the first electronic device and generating content for each person using the information provided by the first electronic device.

In addition, although it has been described in FIG. 1 that the technology according to the present disclosure is applied from the perspective of a broadcasting station, that is, in the process of producing content, the technology can also be used from the viewpoint of consuming (or watching) content. For example, while viewing video content including a plurality of people, the technology may be used to view only an image of a person area designated by the user in a one-shot format.

2 is a diagram illustrating a detailed configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the electronic device 100 may include a communication device 110 , a memory 120 , a display 130 , a manipulation input device 140 , and a processor 150 . Here, the electronic device 100 may be a PC capable of image processing, a notebook PC, a smart phone, a server, or the like.

The communication device 110 is connected to the photographing device 10 or the server 20, and may transmit/receive content. Specifically, the communication device 110 is formed to connect the electronic device 100 with an external device, and is connected to a mobile device through a local area network (LAN) and an Internet network, as well as a USB (Universal) device. It is also possible to connect through a serial bus) port.

In addition, the communication device 110 may be connected to other electronic devices via a router or router connected to a public Internet network as well as a wired method, and may use a wired method with the router or router as well as Wi-Fi, Bluetooth, cellular communication, etc. It can also be connected wirelessly. Here, the cellular communication is, for example, Long-Term Evolution (LTE), LTE Advance (LTE-A), 5G-Advanced, New Radio (NR), code division multiple access (CDMA), wideband CDMA (WCDMA), UMTS ( universal mobile telecommunications system), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM).

The communication device 110 may receive a captured image from the photographing device 10 . Such a captured image may have various image formats. The captured image may be an image of a plurality of participants (a host or panel, etc.), but is not limited thereto, and may be various, such as an opera stage, a play, a sports image, etc., as long as a plurality of people appear.

In addition, the communication device 110 may transmit content for each person generated by the electronic device 100 to the outside. Such a communication device 110 may have a plurality of input/output ports in order to input a captured image and transmit content for each person to the outside. For example, the above-described input/output port may be a serial digital interface (SDI) such as HDMI or DP. For example, the communication device 110 may have eight or more input/output ports. In this case, the four input/output ports may be used to receive an image of 8K resolution, and the remaining four input/output ports may individually receive a 4K image. can be sent to The number of such input/output ports assumes that the input video is 8K video, the output video is 4K, and the communication speed of one input/output port has the performance of 12G, but depends on the communication speed of the input/output port or the resolution of the video used. Accordingly, the number of input/output ports may be changed. In addition, for ease of explanation, the generation of four images for each person will be described on the assumption that four images for each person are generated. It is also possible to send

The memory 120 is a component for storing O/S for driving the electronic device 100 or software for generating content for each person, a deep learning model, data, and the like. The memory 120 may be implemented in various forms such as RAM, ROM, flash memory, HDD, external memory, memory card, etc., but is not limited thereto.

The memory 120 stores the captured image. Specifically, the memory 120 may store the captured image received through the communication device 110 . In addition, the memory 120 may temporarily store a plurality of frames extracted from a captured image for image editing, and may store the finally generated content for each person. Here, the captured image may be a moving picture file such as MP4, AVI, or MOV.

In addition, the memory 120 may store the extracted face image, face coordinates, and ID information for each person (eg, ID and facial feature value information for each ID) calculated in the content creation process.

In addition, the memory 120 may store a deep learning model (or program) for face image detection and face matching. Here, a deep learning model for detecting a face image detects a face region in an image, and generates coordinate information corresponding to the detected face region (eg, a vertex coordinate value of a minimum size rectangle including all face regions). It can be a model that In addition, in the face matching process, it may be a model capable of extracting a feature value using an image corresponding to a face region and comparing the extracted unique value to compare whether the face is the same.

The display 130 displays a user interface window for receiving a selection of a function supported by the electronic device 100 . Specifically, the display 130 may display a user interface window for receiving selections of various functions provided by the electronic device 100 . The display 130 may be a monitor such as LCD, CRT, or OLED, and may be implemented as a touch screen capable of simultaneously performing the functions of the manipulation input device 140 to be described later.

In addition, the display 130 may display an input captured image or may display generated content for each person.

The manipulation input device 140 may receive, from a user, a function selection of the electronic device 100 and a control command for the corresponding function. Specifically, the manipulation input device 140 may receive a user control command for correcting the calculated trajectory information. Such a user control command may be input through the user interface window described above.

The processor 150 controls each component in the electronic device 100 . Specifically, when a booting command is input by the user, the processor 150 may perform booting using the operating system stored in the memory 120 .

When a captured image is input, the processor 150 extracts a plurality of frames included in the input captured image. In this case, the plurality of frames may be all frames included in the captured image, and may be frames extracted in units of a preset time interval (eg, 0.5 seconds to 1 second). The extracted frame may have an image format such as BMP or base 64 encoded JPG.

In addition, the processor 150 may perform a face detection operation by using a frame extracted in units of a preset time interval from among a plurality of frames included in the input captured image. Specifically, a lot of resources are required for a job such as face detection, and the change in the position of a participant included in the image is not very large within seconds. For example, if the deep learning recognition cycle is short, image stuttering may occur, and if the cycle is too long, fast movement may not be tracked. Therefore, it is important to optimize the recognition period of the deep learning engine, and in the environment according to the present disclosure, a period between 0.5 second and 1 second satisfies the above-described condition. However, the above-described cycle is not possible according to an exemplary embodiment, and the above-described cycle may be changed according to the performance of the electronic device 100 and the photographing environment (eg, when the movement of a participant is large).

In addition, the processor 150 may perform pre-processing on the extracted frame. When recognition is performed using a large-resolution image, preprocessing may be performed to reduce the data size of the extracted frame in that it requires a lot of resources and may take a long processing time. Such pre-processing may include processing of aligning the decoded image in [Channel, Width, Height] RGB format, processing of converting to 960x544 size, and the like.

In addition, the processor 150 may perform face detection, person matching with respect to the detected face, and coordinate detection of the corresponding face using the pre-processed image. Such an operation will be described later in detail with reference to FIG. 3 .

In addition, the processor 150 generates content for each person using the captured image and previously generated person ID/coordinate information. Specifically, the processor 150 uses the coordinate information to include all image regions corresponding to the corresponding coordinate information, and crops the image to a predetermined horizontal size and vertical size based on the center point of the corresponding coordinate information, thereby providing content for each person. can create

In this case, the processor 150 may compare the immediately detected coordinate information with the coordinate information, and when the movement of the coordinate information is equal to or greater than a preset value, the processor 150 may generate image content using the changed coordinate information. Specifically, when the crop region is changed in response to a user's minute movement, there may be inconvenience in viewing, such as feeling a tremor in the generated content. Accordingly, the position of the cropped image may be changed only in the case of movement of a certain level or more. Such an operation may be referred to as smooth tracking.

In addition, the processor 150 may generate not only content including only one person in one image, but also content including two or more people, and the person created in an image different from the image used to generate the content for each person It is also possible to create an image in which the content is included in the PIP format.

Also, the processor 150 may perform image processing on the generated content for each person. For example, as described above, there may be a quality difference between the image of a person located in the center of the image and an image of a person located in the vicinity. can be additionally performed.

As described above, the electronic device 100 according to the present embodiment can generate a plurality of contents (ie, content for each person) using only a photographed image generated by one photographing device and provide it to users. In addition, the electronic device 100 according to the present embodiment can reduce costs in that only one captured image is required to generate content for each person, and generates content for each person using deep learning technology. It is possible to produce more high-quality content more easily and quickly.

On the other hand, in the illustration and description of FIGS. 1 and 2, it has been described that a face image is extracted and used from a frame as a result of face detection, but in implementation, only face coordinate information is used without a separate face image extraction operation, and the original frame It is also possible to perform clustering using the generated face coordinate information. The face coordinate information may utilize a pre-detected result, arbitrarily set face coordinates, or face coordinate information detected in another face detection program.

Meanwhile, in FIG. 2 , it has been illustrated and described that the electronic device 100 includes a display and a manipulation input device. However, when the electronic device 100 is configured as a server, the display and manipulation input device may be omitted. can In addition, the electronic device 100 may further include other components not shown in FIG. 2 , and it is also possible for a plurality of electronic devices to perform the functions of FIG. 2 by dividing the labor.

3 is a view for explaining a content creation method according to an embodiment of the present disclosure.

Referring to FIG. 3 , the processor 200 may include an I/O module 210 and a processing module 220 .

The I/O module 210 may receive an image from the outside. Specifically, the I/O module 210 may convert a frame included in an image provided from the outside into a preset file format (eg, base64 jpeg) and transmit it to the processing module 220 . In this case, the I/O module may transmit a preset file format for all frames, or may transfer only a frame of a preset cycle unit among a plurality of frames. Meanwhile, such a conversion operation may be performed by the pre-processing unit 221 to be described later.

The processing module 220 may perform face detection using the image provided from the image processing module 210 and generate content for each person based on the detected face. The processing module 220 may include a preprocessor 221 , a face recognition unit 223 , an eigenvalue extraction unit 225 , a face identification unit 227 , and a content generation unit 229 .

The preprocessor 221 may convert the input image into a preset format and a preset size. Here, the preset format may be [Channel, Width, Height] RGB format, and the preset size may be 960x544, but this may be changed according to the deep learning model used in the face recognition unit.

The face recognition unit 225 may recognize a face in the pre-processed image, generate a face image for the recognized face, and generate coordinate information corresponding to the generated face. Specifically, the face recognition unit 225 inputs the preprocessed image to the face recognition model, checks the face position and probability list, filters the low probability area among the checked list, and performs Non-maximum Suppression (NMS). By doing so, it is possible to integrate a portion having a wide overlapping area to generate a position of a face finally recognized as a face and coordinates for the position. The face recognition model used in the present disclosure can recognize a face in a short time of 15 ms, so it is possible to perform face recognition in units of frames (or in real time).

In addition, the face recognition unit 225 may generate a face image by cropping a corresponding part of the input image using the coordinates of the recognized face. In this case, since the previously generated face coordinates may be tightly determined based on the center, a face image may be generated by cropping with a free space (eg, 32px).

The eigenvalue extractor 225 may extract an eigenvalue (or feature value) of a face using the eigenvalue extraction deep learning model from the previously generated face image. Such a face eigenvalue may be a 512-sized vector.

The face identification unit 227 compares the pre-stored unique value for each person with the unique value generated by the unique value extraction unit 225 to determine a person ID corresponding to the current person-specific image. Specifically, the face identification unit 227 calculates a degree of similarity between the previously stored unique value for each person and the currently generated unique value, and based on the calculated similarity, that is, the face detected in the highest order from the previously stored person. can be matched to

In this case, the face identification unit 227 may remove the similarity including the matched foreign object and the face from the list whenever matching is performed once. For example, if the pre-stored people are A, B, and C, and the currently recognized three face images are 1, 2, and 3, the similarity between the face image 1 and the pre-stored A, B, C with respect to the first face image. is compared, and as a result of the comparison, when face image 1 is matched with pre-stored C, matching may be performed only with pre-stored persons A and B in the second face image matching process (ie, matching with face image 2).

If, in the matching process, the similarity distance exceeds 1.5, but there is no longer an existing person to match, the remaining person is regarded as a new person, a new ID is created for the new person, and a unique value for the ID is stored. can be put Also, since there is no pre-stored unique value in the first recognition process for the same input image, a new ID may be assigned to each generated face image, and a unique value for each new ID may be stored.

The content generating unit 229 may receive the person information including the person ID and the face coordinates generated by the face identification unit 227 , and may generate an image for each person using the provided person information and the image. In this case, the person information may be in a json format. For example, when three people are detected in the input image, the content generator 229 may generate content for each person corresponding to each of the three detected people. Also, the content generator 229 may generate image content including a plurality of people (eg, two people) among the three detected people.

In this case, the content generating unit 210 may generate content for each person using the current person information until the person information of the next cycle is received, and stores the image within the above-described preset cycle in a buffer, and saves the person information. For other frames (hereinafter, general frames) between two frames (hereinafter, referred to as face recognition frames) used for generation, content for each person may be generated by linearly interpolating the face coordinates recognized in the face recognition frame. In the case of using the linear interpolation as described above, it is possible to smooth the adjustment of the crop area even when the movement of the person is large.

On the other hand, there is a case in which the coordinate value of a specific person does not exist for a predetermined time with respect to the input image. For example, it may be a case in which three panelists participate so that the input image must include three faces, but one of the three people temporarily leaves the seat or temporarily participates because it is a guest.

For example, in the case of a guest, it is possible to create video content for the guest only in the area where a face for the guest is detected. A face for the corresponding panel may not be detected. In this case, even if a specific panel (ie, an ID for a specific panel) is not detected, the content generating unit 210 may continuously generate image contents for the corresponding person using the previously confirmed final coordinate information. Also, if the unconfirmed period is longer than a predetermined time, the individual content creation operation for the person may be terminated.

On the other hand, referring to FIG. 3 , in the present disclosure, face detection and facial feature point extraction are performed using two deep learning models individually, but in implementation, face detection and feature point extraction for the detected face are performed in one It can also be implemented in the form of using a deep learning model.

Meanwhile, the above-described operations may be performed in parallel (or pipeline), and such operations will be described below with reference to FIG. 4 .

4 is a flowchart for describing the content creation method of FIG. 3 in more detail.

Referring to FIG. 4 , first, a stream may be input in real time ( S410 ).

In addition, pre-processing may be performed for each frame constituting the received stream (S420). Since the pre-processing operation has been described above, a redundant description thereof will be omitted.

A deep learning pipeline is performed using the preprocessed image (S430). Specifically, in the corresponding step, an analysis operation on images corresponding to a plurality of frames may be performed in the form of a pipeline. For example, the pipeline may be composed of 4 stages, and in the first stage, face region detection may be performed on the image of the Nth frame. In the second step, an operation of generating an eigenvalue vector for each face for the N-1 th frame may be performed. In the third step, an operation of comparing the eigenvalues of people pre-stored for the N-2 th frame and the eigenvalues calculated for the N-2 th frame may be performed, and in the last fourth step, they are included in the N-3 th frame A list of characters having information such as a person ID and coordinate values for the corresponding person may be generated.

Meanwhile, although it has been described above that the above-described operations are performed in units of one frame, each of the above-described operations may be performed in parallel depending on the resource of the electronic device.

Then, a pipeline for each slot is performed (S440). Specifically, this operation can be performed in parallel with the operation of the deep learning pipeline described above. Specifically, a list of persons to be synthesized in a slot may be brought, and when the deep learning pipeline of the person is completed, the corresponding crop position may be updated, the updated crop position may be cropped, and post-processing may be performed. In addition, target characters cropped in the slot can be synthesized in order, and the completed slot can be stored.

Finally, individual output slots are fetched (S450). Specifically, it is possible to generate a multi-view output by concatenating individual outputs, or output individual contents by transmitting individual outputs.

5 is a view for explaining a content creation method according to another embodiment of the present disclosure.

Referring to FIG. 5 , the imaging system includes a plurality of electronic devices 100 - 1 and 100 - 2 and a recording device 50 .

Each of the plurality of electronic devices 100-1 and 100-2 may receive an image, generate content for each person included in the input image, and output the generated content for each person.

In this case, each of the plurality of electronic devices 100 - 1 and 100 - 2 receives a plurality of captured images, generates a scaled-up image having a higher resolution than each of the received plurality of captured images, and generates the generated scaled-up image. By using , content for each person may be output.

The recording apparatus 50 may receive and store a plurality of contents output from input/output ports of the plurality of electronic devices 100 - 1 and 100 - 2 .

Meanwhile, although the illustrated example shows that two cameras are used for four participants, it is possible to use one camera in implementation. In addition, in the above example, one electronic device has been illustrated and described as generating content for four people, but if the performance of the electronic device is capable of processing more content, it may be implemented with one electronic device. In addition, instead of two electronic devices, it is also possible to combine three or more electronic devices to generate content for nine or more participants in real time.

6 is a diagram comparing the configuration of a camera in a conventional photographing environment with a camera configuration in a case in which a photographing environment according to the present disclosure is applied.

Referring to FIG. 6 , the existing shooting environment is a camera configuration diagram 610 used at a shooting site of a program in which 6 performers participate, in which a total of 11 cameras are used. Specifically, the foreground, Libero, and Jimmy Zip cameras, two cameras for three-shot, and six cameras for one-shot are utilized.

On the other hand, when the shooting environment according to the present disclosure is applied ( 620 ), an image for each performer can be generated by using an image of a camera for a three-shot, and six cameras for a one-shot may not be used. That is, as shown on the right, it is possible to shoot using only five cameras. In addition, the foreground cam, Libero, and Jimmy's Zip are arranged as before, so there is no difficulty in dealing with events or unexpected situations.

It is possible to reduce the number of cameras required to produce one such program from 11 to 5.

In addition, when photographing is performed in a narrow place, when photographing is performed with individual cameras for each person as in the past, a problem in which the angles of the cameras overlap each other may occur. However, in such an environment, as in the present disclosure, if a plurality of participants are photographed with a single camera and an image for each person is extracted from the photographed image, the problem of overlapping angles as in the prior art can be prevented.

7 is a diagram illustrating an example of content generated according to the present disclosure.

Referring to FIG. 7 , when a high-resolution image 710 is input, images (or contents) 710 , 720 , and 730 for each person included in the image may be generated. In addition, the generated individual image may also generate an image 740 in a form including two or more persons, rather than a form including only one person.

In addition, broadcasting contents can be easily created using such individual images, and these individual images can be independently output, and are also operated in a form included in an image different from the current image, that is, the PIP form 760 to be finally broadcast. it might be

8 is a view for explaining a content creation method according to an embodiment of the present disclosure.

Referring to FIG. 8 , first, a captured image is received in real time (S810). Specifically, a frame image of the captured image may be extracted in units of a preset time period, and a preset pre-processing operation may be performed on the extracted frame image. Such pre-processing may include at least one of a process of converting an image file format to a [Channel, Width, Height] RGB format and a process of reducing the image resolution to a preset size.

Then, using the pre-learned face detection deep learning model, a face included in the input captured image is detected, a face image corresponding to the detected face is generated, and coordinate information of the face image is determined ( S820 ). Specifically, by inputting the input captured image to the pre-trained face detection deep learning model, to obtain list information about the face position and the probability of the face position, the face position having a lower probability than the preset value is filtered, By performing Non-Maximum Suppression (NMS), faces with overlapping positions are integrated, and faces in the captured image can be detected.

Then, a unique value for the face image is extracted, and a face ID for the face image is determined by comparing the extracted unique value with a pre-stored unique value for each person (S830).

Then, image content for each of the plurality of faces is generated in real time by using the face ID, coordinate information, and the captured image (S840). Specifically, within a preset time period, image content for each of a plurality of faces may be generated in real time by using the generated face ID and coordinate information.

In addition, image content for each of the plurality of faces may be transmitted to different output ports.

Also, the merged content may be generated by merging at least two of the generated image content for each of the plurality of faces.

Accordingly, in the file transmission method according to the present embodiment, it is possible to generate a plurality of contents (ie, content for each person) using only a photographed image generated by a single photographing device and provide it to users. The content creation method shown in FIG. 8 may be executed on the electronic device having the configuration of FIG. 2 and may also be executed on the electronic device having other configurations.

In addition, the content creation method as described above may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program is stored in a non-transitory computer readable medium to be provided. can

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the programs for performing the above-described various methods may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the disclosure belongs without departing from the gist of the present disclosure as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present disclosure.

1000: content creation system 10: imaging device
20: display device 100: electronic device
110: communication device 120: memory
130: display 140: operation input device
150: processor

Claims (20)

In an electronic device,
a communication device receiving a stream corresponding to a captured image in real time;
performing a pre-processing operation on a frame constituting the input stream, and generating face ID and coordinate information for each of a plurality of face regions included in the frame using the pre-processed frame image and a deep learning pipeline; A processor for generating image content for each of the plurality of faces in real time by using the generated face ID and coordinate information and the captured image;
The deep learning pipeline is
For the N-th frame, a face included in the input captured image is detected using a pre-learned face detection deep learning model, a face image corresponding to the detected face, and coordinate information of the face image are extracted,
Extracting the eigenvalue for the face image for the N-1th frame,
Comparing the extracted eigenvalue with respect to the N-2th frame and the pre-stored eigenvalue for each person,
An electronic device for generating a list of characters including face IDs and coordinate information included in the N-3 th frame with respect to the N-3 th frame. delete delete According to claim 1,
The pre-processing operation is
An electronic device comprising at least one of a process for converting an image file format to a [Channel, Width, Height] RGB format, and a process for reducing an image resolution to a preset size. According to claim 1,
The processor is
By inputting the inputted captured image to the pre-learned face detection deep learning model, list information about the face position and the probability of the face position is obtained, and the face position having a probability lower than a preset value is filtered, An electronic device for detecting a face in the captured image by performing non-maximum suppression (NMS) to integrate faces with mutually overlapping positions. According to claim 1,
The processor is
An electronic device that compares previously detected coordinate information with the coordinate information, and generates image content by using the changed coordinate information when the movement of the coordinate information is equal to or greater than a preset value. According to claim 1,
The processor is
An electronic device for controlling the communication device so that each of the image content for each of the plurality of faces is transmitted to different output ports. According to claim 1,
The processor is
An electronic device generating merged content by merging at least two of the generated image content for each of the plurality of faces. In the content creation method,
receiving a stream corresponding to a captured image in real time;
performing a pre-processing operation on frames constituting the input stream, and generating face ID and coordinate information included in the frame using the pre-processed frame image and a deep learning pipeline; and
generating video content for each detected face in real time using the face ID, the coordinate information, and the captured image;
The deep learning pipeline is
Using the face detection deep learning model previously learned for the Nth frame, detecting a face included in the input captured image, extracting a face image corresponding to the detected face, and coordinate information of the face image,
Extracting the eigenvalue for the face image for the N-1th frame,
Comparing the extracted eigenvalue with respect to the N-2th frame and the pre-stored eigenvalue for each person,
A content creation method for generating a list of characters including face ID and coordinate information included in the N-3th frame with respect to the N-3th frame. delete delete 10. The method of claim 9,
The pre-processing operation is
A content creation method comprising at least one of a process of converting an image file format to [Channel, Width, Height] RGB format, and a process of reducing the image resolution to a preset size. 10. The method of claim 9,
The deep learning pipeline is
By inputting the inputted captured image to the pre-learned face detection deep learning model, list information about the face position and the probability of the face position is obtained, and the face position having a probability lower than a preset value is filtered, A content creation method for detecting faces in the captured image by integrating faces that overlap each other by performing non-maximum suppression (NMS). 10. The method of claim 9,
The generating step is
A method for generating content by comparing the coordinate information detected immediately before and the coordinate information, and generating image content by using the changed coordinate information when the movement of the coordinate information is equal to or greater than a preset value. 10. The method of claim 9,
and transmitting the image content for each of the detected faces to different output ports. 10. The method of claim 9,
and generating merged content by merging at least two of the generated image content for each of the plurality of faces. A computer-readable recording medium comprising a program for executing a content creation method,
The content creation method includes:
receiving a stream corresponding to a captured image in real time;
performing a pre-processing operation on frames constituting the input stream, and generating face ID and coordinate information included in the frame using the pre-processed frame image and a deep learning pipeline; and
generating video content for each detected face in real time using the face ID, the coordinate information, and the captured image;
The deep learning pipeline is
Using the face detection deep learning model previously learned for the Nth frame, detecting a face included in the input captured image, extracting a face image corresponding to the detected face, and coordinate information of the face image,
Extracting the eigenvalue for the face image for the N-1th frame,
Comparing the extracted eigenvalue with respect to the N-2th frame and the pre-stored eigenvalue for each person,
A computer-readable recording medium for generating a character list including face ID and coordinate information included in the N-3th frame with respect to the N-3th frame. delete 18. The method of claim 17,
The generating step is
A computer-readable recording medium that compares the coordinate information detected immediately before and the coordinate information, and generates image content by using the changed coordinate information when the movement of the coordinate information is equal to or greater than a preset value. 18. The method of claim 17,
The content creation method includes:
The computer-readable recording medium further comprising a; transmitting each of the image content for each of the plurality of faces to different output ports.

Images