KR102118093B1 - Method for processing workflow of character information providing system based on face recognition - Google Patents

Abstract

The present invention relates to a method for processing the workflow of a character information providing system based on facial recognition which provides an optimal deep metadata management technique. According to an embodiment of the present invention, the method for processing the workflow of a character information providing system based on facial recognition comprises: a step in which a content registration unit copies an original content file to store the original content file in a storage, and registers and stores content metadata in an MAM server; a step in which the MAM server assigns a preprocessing job required for facial recognition and verification to a preprocessing server; a step in which the preprocessing server performs preprocessing including transcoding, cataloging, and frame extraction; a step in which a recognition server recognizes a sound source, a situation, and a character in an image when the MAM server uses preprocessed results to assign a recognition job to the recognition server; a step in which a manager terminal uses an authoring tool to check recognition results and correct the recognition results through verification; and a step in which the MAM server transfers results to a user terminal after verification is completed.

Description

Workflow processing method of face recognition based character information providing system {METHOD FOR PROCESSING WORKFLOW OF CHARACTER INFORMATION PROVIDING SYSTEM BASED ON FACE RECOGNITION}

The present invention relates to a technology for managing media assets.

Media Asset Management (MAM, hereinafter referred to as'MAM') refers to the management of media files such as video, audio, and images in a narrow sense, and Content Management System (CMS) in a broad sense. It can be said.

CMS refers to a system that manages content by including general data and metadata. The content referred to herein refers to an electronically documented data file, and refers to an electronic file that exists independently, such as a word file, photo image, video file, and sound source file. General data refers to data that is recorded in DB such as electronic payment data, system logs, emails, messenger texts, etc., but does not exist alone. Metadata refers to additional data for recognizing the content itself to the system. For example, the content file is stored in a separate storage, and the path information, format information, creation date, owner, title, and permission to use it are managed in the DB, and this information is called metadata.

According to one embodiment, it recognizes the characters, situations, places, sound sources, etc. in a video to provide deep metadata to the user, and obtains content to optimize the entire process from extracting and transmitting deep metadata We propose a workflow processing method for face recognition based character information providing system that provides deep meta management technology.

The workflow processing method of the facial recognition-based character information providing system according to an embodiment includes the steps of a content registration unit copying a content original file and storing it in a storage, and registering and storing content metadata in the MAM server, and the MAM server A) assigning a pre-processing task necessary for recognition and verification to the pre-processing server; and a pre-processing server performing pre-processing including transcoding, cataloging, and frame extraction, and the MAM server using the pre-processed results to recognize the task. When assigned to the recognition server, the recognition server recognizes the person, situation, and sound source in the video, and the administrator terminal checks the recognition result using the authoring tool and verifies and corrects it after verification is completed. , MAM server passing the result to the user terminal.

The pre-processing steps include: the MAM server assigning a transcoding operation to the transcoder, the transcoder converting the original image into a searchable image playable on the web, and the MAM server assigns a cataloging operation to the cataloger , Catalogger analyzes the original video and divides the scene based on the transition point to output a shot image, and the MAM server assigns a frame extraction operation to the frame extractor, and the frame extractor extracts the frame image Each step constituting the pre-processing operation is performed in parallel at the same time, and can be communicated using the TCP protocol between the MAM server and the transcoder of the pre-processing server, the cataloger and the frame extractor.

Recognizing a person, a situation, and a sound source in an image includes: a MAM server assigning a character feature extraction task to a person recognizer, and a person recognizer calling a person recognition server for extracting features for recognition of characters; , Extracting the feature for recognition of the character using the gallery image of the character that the called character recognition server is pre-stored, and the character recognizer confirms the feature extraction status in the character recognition server and registers the confirmation result to the MAM server Step, the MAM server assigns a person recognition task to the person recognizer, the person recognizer calls the person recognition server for person recognition, and the called person recognition server uses the frame extraction image and the character feature file. The method may include recognizing a person, providing a representative profile recommendation picture, a person clustering image, a feature file for each frame, and a recognition result file, and storing the recognition result by the MAM server.

In the step of recognizing the person, situation, and sound source in the video, the MAM server assigns a sound source recognition task to the sound source recognizer, and the sound source recognizer extracts the wav file from the storage and uses the extracted wav file to record the sound source to the sound source recognition server. Invoking the recognition, the sound source recognition server recognizes the sound source using the sound source recognition library and registers the sound source recognition result to the MAM server, and the MAM server corrects the misrecognized result from the recognized result in a preset unit and is effective It may include the step of merging the results and storing the final result by the MAM server.

The step of merging valid results includes providing sound recognition results in scene units through the sound recognition page of the storyboard, and displaying sound source warnings for items with duplicate timecodes, and deleting duplicate sound sources, or It may include a step of providing an editing screen to merge with each other.

Recognizing a person, a situation, and a sound source in the video includes: a MAM server assigning a context recognition task to the context recognizer; and a context recognizer calling the context recognition server while delivering the frame extraction image to the context recognition server, The context recognition server may include receiving a frame extraction image and recognizing objects, events, places, landmarks, and videos through context recognition, and providing a recognition result.

The verification and correction step provides a web-based authoring tool that can check and manage registered content and deep meta recognition results, and the authoring tool includes a content management page, a storyboard page, a verification page, and a character management page. And an administrator page.

The content management page may include a content and metadata registration screen, a content confirmation screen, a metadata confirmation screen, a pre-processing command input screen, and a recognition command input screen.

The character management page may include an image screen for a profile recommended by the recognition server, a clustering image screen grouping the same people, and a search screen to search for and provide a person related to the input portrait.

The storyboard page is a storyboard-shot screen that checks shot extractions generated through cataloging, creates and manages scenes based on shots, checks sound source recognition results in units of scenes, corrects misrecognized results, and duplicates timecode It may include a storyboard-sound recognition screen for deleting or merging deleted items, a storyboard-object screen for checking object recognition results and editing the misrecognized results.

The verification page may include a character display screen corresponding to a reproduction time point, a sound source display screen, an additional information confirmation screen of the character, a character appearance point guidance screen, and a related content guidance screen at the ending point.

The face recognition based character information providing system according to an embodiment may recognize a character, a situation, a place, and a sound source in an image and provide deep metadata to a user. At this time, it is possible to provide an optimal deep meta management technology that integrates the entire process from obtaining content to extracting and transmitting deep metadata.

The face recognition-based character information providing system provides an optimal content management solution using MAM technology. In addition, the person recognition engine, the context recognition engine, and the sound source recognition engine are integrated for automatic processing, and the convenience of use is increased by linking engines of various platforms with each other using the Open Application Programming Interface (Open API). Order. Furthermore, it provides an authoring tool that can manage content and recognition tasks, and a player that can verify the final deep metadata.

1 is a diagram showing the configuration of a facial recognition-based character information providing system according to an embodiment of the present invention;
2 is a diagram showing a table showing functions and input data and output data of the components of the facial recognition-based character information providing system;
3 is a view showing the configuration of a MAM server and a client according to an embodiment of the present invention,
4 is a diagram showing an example of a workflow setting of the MAM server according to an embodiment of the present invention,
5 is a view showing a workflow processing flow of the MAM server according to an embodiment of the present invention,
6 is a diagram illustrating an example of metadata setting of a MAM server according to an embodiment of the present invention;
7 is a view showing a metadata processing flow of the MAM server according to an embodiment of the present invention,
8 is a diagram illustrating an entire workflow processing process of a face recognition-based character information providing system according to an embodiment of the present invention;
9 is a diagram illustrating a pre-processing workflow processing process of a face recognition-based character information providing system according to an embodiment of the present invention;
FIG. 10 is a diagram illustrating a process for processing a person recognition workflow in a system for providing character information based on face recognition according to an embodiment of the present invention;
11 is a diagram showing a sound source recognition workflow processing process of a face recognition based character information providing system according to an embodiment of the present invention;
12 is a diagram illustrating a situation recognition workflow processing process of a face recognition based character information providing system according to an embodiment of the present invention;
13 is a diagram showing a content registration screen through a Web UI according to an embodiment of the present invention;
14 is a diagram illustrating a content registration screen through an ingest program for a watchfolder method according to an embodiment of the present invention;
15 is a diagram showing a content registration and preprocessing work process according to an embodiment of the present invention;
16 is a view showing a device configuration for person recognition according to an embodiment of the present invention,
17 is a view showing a person recognition process according to an embodiment of the present invention,
18 is a view showing a device configuration for sound source recognition according to an embodiment of the present invention,
19 is a diagram illustrating a sound source recognition process according to an embodiment of the present invention,
20 is a view showing a device configuration for situational awareness according to an embodiment of the present invention;
21 is a diagram illustrating a situation recognition process according to an embodiment of the present invention,
22 is a view showing a authoring tool screen according to an embodiment of the present invention;
23 is a diagram showing a content management screen of the authoring tool according to an embodiment of the present invention;
24 is a view showing a character management screen of the authoring tool according to an embodiment of the present invention;
25 is a view showing a storyboard (shot) management screen of the authoring tool according to an embodiment of the present invention;
26 is a view showing a storyboard (sound source recognition) management screen of the authoring tool according to an embodiment of the present invention;
27 is a view showing a storyboard (object chain) management screen of the authoring tool according to an embodiment of the present invention;
28 is a view showing a verification screen of the authoring tool according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

In the description of the embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted, and terms to be described later in the embodiments of the present invention These terms are defined in consideration of the function of the user, and may vary depending on the user's or operator's intention or custom. Therefore, the definition should be made based on the contents throughout this specification.

Combinations of each block in the accompanying block diagrams and steps in the flow charts may be performed by computer program instructions (execution engines), these computer program instructions being incorporated into a processor of a general purpose computer, special purpose computer, or other programmable data processing device. Since it can be mounted, the instructions executed through a processor of a computer or other programmable data processing device create a means to perform the functions described in each block of the block diagram or in each step of the flowchart.

These computer program instructions can also be stored in computer readable or computer readable memory that can be oriented to a computer or other programmable data processing device to implement a function in a particular way, so that computer readable or computer readable memory The instructions stored in it are also possible to produce an article of manufacture containing instructions means for performing the functions described in each block of the block diagram or in each step of the flowchart.

And since computer program instructions may be mounted on a computer or other programmable data processing device, a series of operational steps are performed on the computer or other programmable data processing device to create a process that is executed by the computer to generate a computer or other programmable It is also possible for instructions to perform the data processing apparatus to provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step can represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical functions, and in some alternative embodiments referred to in blocks or steps It should be noted that it is also possible for functions to occur out of sequence. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, and it is also possible that the blocks or steps are performed in the reverse order of the corresponding function as necessary.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the present invention are provided to more fully describe the present invention to those of ordinary skill in the art.

1 is a diagram showing the configuration of a facial recognition-based character information providing system according to an embodiment of the present invention, and FIG. 2 is a function of the components of the facial recognition-based character information providing system and input data and output data. It is a diagram showing a table.

The face recognition based character information providing system 1 according to an embodiment recognizes a character, a situation, a place, and a sound source in an image and provides deep metadata to the user. At this time, it provides optimal deep meta management technology that integrates the entire process from content acquisition to deep metadata extraction and transmission. Deep Meta Solution is an AI-based video analysis technology that provides detailed additional information about characters, situations, and sound sources in the video, and automatically classifies various scenes of the video to sound files according to the characteristics of the scene. , Provides a service that automatically adjusts the color of images.

More specifically, the facial recognition-based character information providing system provides an optimal content management solution by using a media asset management (MAM) technology. In addition, the person recognition engine, the context recognition engine, and the sound source recognition engine are integrated for automatic processing, and the convenience of use is increased by linking engines of various platforms with each other using the Open Application Programming Interface (Open API). Order. Furthermore, it provides an authoring tool that can manage content and recognition tasks, and a player that can verify the final deep metadata.

1 and 2, the facial recognition based character information providing system 1 includes a content providing unit 10, a MAM storage 11-1, a MAM DB 11-2, and a meta management server 12. , A pre-processing server 13, a face recognition server 14, a situation recognition server 15, a sound source recognition server 16, an administrator terminal 17, a result transmission unit 18 and a user terminal 19.

① The meta management server 12 obtains the original image and CMS metadata from the content providing unit 10 and stores and manages them in the MAM storage 11-1 and the MAM DB 11-2 (automatic registration). The meta management server 12 may include a MAM server and a MAM extension server (MamEx).

② The pre-processing server 13 includes a transcoder, a cataloger, a frame extractor, a person recognizer, a situation recognizer, and a sound source recognizer. The transcoder receives the original image, converts it into a streamable format, and outputs a search image. The cataloger receives the original video, splits the scene (shot), and outputs the shot image of the scene change point. The frame extractor receives the original image and outputs a frame image. The person recognizer works with the MAM server and the face recognition server 14, the situation recognizer works with the MAM server and the situation recognition server 15, and the sound source recognizer works with the MAM server and the sound source recognition server 16.

③ The face recognition server 14 recognizes the characters in the image and generates a profile recommendation image. The input data of the face recognition server 14 is an extracted frame image (for example, an image extracted in units of 10 frames), and output data through face recognition is a person clustering image, a representative profile recommendation image, a frame feature storage file, and the like. .

④ The situation recognition server 15 recognizes objects, events, places, and landmarks in the video. The input data of the context recognition server 15 is an extracted frame image (for example, an image extracted in units of 5 frames), and the output data through context recognition is an object, event, place, landmark, video recognition result file, and the like. .

⑤ The manager terminal 17 creates, manages, and verifies a scene based on a shot targeting the content that has been pre-processed and recognized. The input data of the manager terminal 17 is a shot image, and the output data is a time code. The time code is, for example, "hour:minute:second:frame". For verification, a verification page is provided to verify the results of the deep meta, and the verification page displays the characters and sound sources corresponding to the playback time, and provides the function to check the additional information of the characters. It provides data verification function by displaying meta information in real time.

⑥ The result transmission unit 18 transmits the recognition and verification results to the user terminal 19. The user terminal 19 includes a TV, a set-top box, and a mobile terminal.

3 is a diagram showing the configuration of a MAM server and a client according to an embodiment of the present invention.

The MAM server 2 includes a communication unit 26, a MAM engine, and a DB 20.

The communication unit 26 handles communication with the client 3. It supports TCP-based and SOAP protocols for communication. The TCP-based protocol/SOAP protocol can be configured as a process execution option. SDK for clients that support TCP-based protocols can be provided.

The MAM engine includes a workflow manager 21, an asset manager 22, an admin manager 23, a system manager 24, and an extended MAM manager ) (25).

The workflow management unit 21 manages the system workflow. At this time, the job allocation is controlled to allocate the job to the component server (pre-processing server / recognition engine interlocker) 30 suitable for the job type. A mediation application that downloads and processes a job from the MAM server 2 is called a component server, and the job that the server processes is defined as a component job. After transcoding, two component tasks, such as transmission, come together to form a requirement, and a sequence of component tasks that occur in this order is defined as a system workflow.

The asset management unit 22 manages metadata according to the established asset scheme, and provides CRUD (Create: Create, Read: Read, Update: Update, Delete: Delete) functions of metadata. . The asset schema means redefining the DB field that is the target of creation, inquiry, modification, and deletion operations on data. Metadata refers to additional data for recognizing the content itself to the system. For example, the content file is stored in a separate storage, and the pass information, format information, creation date, owner, title, usage rights, etc. to utilize it are managed in the DB, and this information is called metadata.

The admin management unit 23 provides management functions necessary for operating the system, such as departments, users, and authorities. The system management unit 24 supports a system setting function, and processes assets and workflow settings. The extended MAM management unit 25 is in charge of interworking with the MAM extension server 4, which is an extension module of the MAM server 2.

The DB 20 is in charge of interworking with the DB and search engine.

The client 3 includes a component server 30, an authoring tool 31, an administration tool/monitoring tool 32, and a metadata design tool (MAM Designer) 33. . The component server 30 is assigned and performed by the workflow management unit 21 of the MAM server 2, and the authoring tool 31 is connected to the asset management unit 22 of the MAM server 2, and is managed and monitored. The tool 32 is connected to the admin management unit 23 of the MAM server 2, and the metadata design tool 33 is connected to the system management unit 24 of the MAM server 2.

4 is a diagram illustrating an example of a workflow setting of a MAM server according to an embodiment of the present invention.

Referring to FIGS. 3 and 4, the MAM server 2 stores and manages all workflow settings in a work setting table 40 which is a schema table rather than a code level. However, pre-processing and post-processing that require qualification can be processed through the MAM extension server (MamEx). The job setting table 40 includes the job identifier 41, job XML 42, caption 43, start call (STARTCMSEXNAME) 44, and end call (ENDCMSEXNAME) 45 fields.

① The MAM server 2 delivers the work XML 42, which standardizes the items set at the time of task assignment, to the component server. ② The job XML 42 includes MamEx information 44 and 45 to be called before and after the job in the component server. ③ The job status information is updated in the field of the job setting table 40. The work status information includes a start time, an end time, a progress rate, a work status, and a cause of failure.

5 is a diagram showing a workflow processing flow of the MAM server according to an embodiment of the present invention.

1. Call workflow

The client 3 calls the workflow to the MAM server 2 (step 1). The MAM server 2 adds the called job content to the job setting table (step 1-1).

2. Task assignment

The MAM server 2 assigns the job by passing the job information to the component server 5 (step 2). At this time, the task management thread periodically checks the task setting table and allocates the waiting task to the processable component server 5, and delivers the task XML based on the task setting table. If there are multiple waiting component servers, the task is assigned by applying priority. The MAM server 2 updates information related to the start of the task after task assignment (step 2-1).

3. Pre-treatment

The component server 5 calls the MAM extension server (Start Job MamEx) 4 for the pre-processing job after the job is allocated (step 3). The MAM extension server 4 executes the pre-processing task required for the task (step 3-1), then completes the final task information (Job XML) and delivers it to the component server 5 (step 3-2).

4. Work processing

The component server 5 reports the progress to the MAM server 2 periodically (Step 4-1) while processing the operation (Step 4). The MAM server 2 updates the work progress related information (step 4-2).

5. Post-processing

The component server 5 calls the MAM extension server (End Job MamEx) 4 for the job including the job result and the product information (step 5). The MAM extension server 4 transmits the final work result to the component server 5 after the work post-processing (storage work product information storage, etc.) (step 5-1) (step 5-2).

6. Notification of work results

The component server 5 notifies the MAM server 2 of the final operation result received from the MAM extension server 4 (step 6). The MAM server 2 updates the task termination related information (step 6-2).

6 is a diagram illustrating an example of metadata setting of a MAM server according to an embodiment of the present invention.

3 and 6, the MAM server 2 stores and manages asset (metadata) settings using a metadata setting table, not a code level. Metadata can be designed through the metadata design tool (MAM Designer) 33. However, content that needs to be qualified can be processed through the MAM extension server (4).

① The metadata design tool (MAM Designer) 33 performs basic setting of the metadata setting table. Set the sequence and PK column for issuing the name and primary key of the metadata table (Primary Key: PK, hereinafter referred to as'PK').

② The metadata design tool (MAM Designer) 33 sets the MAM extension server 4 to be linked when an asset is created or deleted. In the case of creation, mainly the basic values (meta ID, file path, etc.) that need to be qualified are updated.

③ The metadata design tool (MAM Designer) 33 sets metadata items. For example, set the field type, caption and default value of metadata. Depending on the setting, metadata items of the authoring tool can be dynamically configured.

7 is a diagram illustrating a metadata processing flow of the MAM server according to an embodiment of the present invention.

1. Asset Schema Setting

The asset schema is set using the metadata design tool (MAM Designer) 33 (step 1). The client 3 retrieves asset schema information through the MAM server 2 (step 1-1), and then dynamically configures metadata items (step 1-2).

2. Asset Creation

The client 3 requests the MAM server 2 to create an asset based on the asset schema setting (step 2). The MAM server 2 creates assets and updates the default values (step 2-1). The MAM server 2 calls the MAM extension server 4 for the asset creation qualification (step 2-2), and the MAM extension server 4 performs the asset creation related qualification logic to the MAM server 2 Update additional default values (step 2-3). The MAM server 2 returns the new asset information to the client (steps 2-4).

3. Delete assets

When the client 3 requests to delete the asset from the MAM server 2 (step 3), the MAM server 2 first deletes the reference asset of the table used as the PK (step 3-1). For example, in the case of a video table, shots, face recognition, and sound source recognition tables are deleted. Then, the workflow being executed with the asset requested to be deleted is canceled (step 3-2). Subsequently, the MAM server 2 calls the MAM extension server 4 for eligibility for asset deletion (step 3-3), and the MAM extension server 4 performs eligibility logic related to asset deletion (for example, File deletion, etc.) (step 3-4), and the result of the asset deletion processing is returned to the MAM server 2 (step 3-5). The MAM server 2 returns the result of the asset deletion processing to the client 3 (steps 3-6).

8 is a diagram illustrating an entire workflow processing process of a face recognition-based character information providing system according to an embodiment of the present invention.

1. Content registration

The content register (Ingest Manager) 6 copies the content original file (step 1-1) and stores it in the storage 11-1, and registers and stores the content metadata in the MAM server 2 (step 1- 2).

2. Pre-treatment work

The MAM server 2 allocates a pre-processing task required for recognition and verification to the pre-processing server 13 (step 2-1). The pre-processing server 13 performs pre-processing such as transcoding, cataloging, and frame extraction. Transcoding is to convert to a streamable format using the original image, cataloging is to extract the transition point, and frame extraction is to save the frame of the image as an image file. The extracted frame image is input to the recognition server 7. Subsequently, the pre-processing server 13 stores the pre-processing results in the storage 11-1 (step 2-1), and notifies the MAM server 2 of the completion of the operation (step 2-2).

3. Recognition work

The MAM server 2 assigns the recognition task to the recognition server 7 using the pre-processed result. The recognition server 7 performs person recognition, situation recognition, and sound source recognition. Character recognition is performed through person recognition, profile recommendation images are generated, and character clustering is performed. Recognize objects, events, places and landmarks in the video through context recognition. Recognize background music through sound source recognition. Subsequently, the recognition server 7 stores the recognition result in the storage 11-1 (step 3-1), and notifies the MAM server 2 of the completion of the operation (step 3-2).

4. Data verification

The administrator terminal 8 checks the recognition result using the authoring tool and corrects it after verification (step 4).

5. Sending results

After verification is completed, the MAM server 2 delivers the result to the user terminal 19 (step 5).

9 is a diagram illustrating a pre-processing workflow processing process of a face recognition-based character information providing system according to an embodiment of the present invention.

1. Transcoding

The MAM server 2 assigns a transcoding operation to the transcoder 130 (step 1), and the transcoder 130 converts the original image ts into a searchable image mp4 playable on the web. At this time, the input data of the transcoder 130 is an original image, and the output data is a search image (mp4, 640×360). The transcoder 130 generates the search image mp4 (step 1-1) and stores it in the storage 11-1, registers the search image information to the MAM server 2 (step 1-2), and completes Is informed (steps 1-3). In the case of the streaming source supply and demand method, the transcoding operation can be omitted.

2. Cataloging

The MAM server 2 allocates a catalogging operation to the catalog 131 (step 2), and the cataloger 131 analyzes the original image and divides the scene based on the scene change point. At this time, the input data is an original image, and the output data is a shot image (jpg, 320×180). The cataloger 131 stores the shot image jpg in the storage 11-1 (step 2-1), registers the shot information to the MAM server 2 (step 2-2), and notifies completion. (Step 2-3).

3. Frame extraction

The MAM server 2 allocates a frame extracting operation to the frame extractor 132 (step 3), and the frame extractor 132 extracts a frame image to be used for character recognition and context recognition. For example, an image is extracted in units of 10 frames for person recognition, and an image is extracted in units of 5 frames for context recognition. The input data is an original image, and the output data is a frame image (jpg, original image size). Transcoding, catalogging and frame extraction can be performed simultaneously in parallel. The frame extractor 132 stores the frame image jpg in the storage 11-1 (step 3-1), registers the frame extraction information to the MAM server 2 (step 3-2), and notifies the completion. (Step 3-3).

The MAM server 2 and the transcoder 130 of the pre-processing server, the catalog 131, and the frame extractor 132 may communicate using a TCP protocol.

FIG. 10 is a diagram illustrating a process of processing a person recognition workflow in a face recognition based character information providing system according to an embodiment of the present invention.

1. Character feature extraction

When the MAM server 2 assigns the character feature extraction operation to the person recognizer 133 (step 1), the person recognizer 133 calls the person recognition server 14 for extracting features for recognition of the characters ( Step 1-1). The person recognition server 14 extracts features for recognition of the characters using the constructed gallery image of the characters, and stores the extracted feature files in the storage 11-1 (steps 1-2). The person recognizer 133 checks the feature extraction state with the person recognition server 14 (steps 1-3), registers the verification result with the MAM server 2, and notifies completion (steps 1-4). The input data of the person recognition server 14 is a gallery image of the character, and the output data is a feature storage file of the character. If all the features of the characters are extracted, the feature extraction operation of the characters can be omitted.

2. Person recognition work

The person recognition server 14 is used to link the person recognition server 14 and the MAM server 2. When the MAM server 2 assigns a person recognition task to the person recognizer 133 (step 2), the person recognizer 133 calls the person recognition server 14 for person recognition (step 2-1). The person recognition server 14 recognizes a person using the frame extraction image and the character feature file, and outputs a representative profile recommendation picture, a person clustering image, a feature file for each frame, and a recognition result file. The person recognition server 14 stores the frame feature file in the storage 11-1 (step 2-2), stores the person clustering file in the storage 11-1 (step 2-3), and recommends a representative profile The file is stored in the storage 11-1 (step 2-4), and the person recognition result file is stored in the storage 11-1 (step 2-5). The person recognizer 133 checks the person recognition state in the person recognition server 14 (steps 2-6). The person recognizer 133 checks the recognition result file in the storage 11-1 (step 2-7), registers the result with the MAM server 2, and notifies completion (step 2-8).

11 is a diagram illustrating a sound source recognition workflow processing process of a facial recognition-based character information providing system according to an embodiment of the present invention.

1. Sound source operation

The entire image is cut in units of a preset unit, for example, 7 seconds, and stored in the storage 11-1 as a wav file. The MAM server 2 assigns a sound source recognition task to the sound source recognizer 134 (step 1). Then, the sound source recognizer 134 extracts the wav file from the storage 11-1 (step 1-1). The sound source recognizer 134 interlocks the sound source recognition server 16 and the MAM server 2 and calls the sound source recognition server 16 to the sound source recognition server 16 using a wav file (step 1-2). The sound source recognition server 16 recognizes the sound source and the sound source recognizer 134 registers the sound source recognition result in the MAM server 2 (steps 1-3). The sound source recognition call (step 1-2) and sound source recognition result registration (step 1-3) are repeatedly performed. After the operation is completed, the sound source recognizer 134 deletes the wav file and notifies the MAM server 2 of completion (steps 1-4).

2. Merging sound source recognition results

The MAM server 2 corrects the misrecognized result from the result recognized in a preset unit, for example, 7 seconds, and merges valid results (step 2). For example, items with duplicate time codes are deleted or merged with each other.

12 is a diagram illustrating a situation recognition workflow processing process of a face recognition based character information providing system according to an embodiment of the present invention.

1. Situational awareness work

The MAM server 2 assigns a situation recognition task to the situation recognizer 135 (step 1). The context recognizer 135 that links the context recognition server 15 and the MAM server 2 calls the context recognition server 15 while passing the frame extraction image to the context recognition server 15 (step 1-1). The context recognition server 15 receives the frame extraction image, recognizes objects, events, places, landmarks, and videos through context recognition, and stores the context recognition result file in the storage 11-1 (steps 1-2) ). The situation recognizer 135 checks the situation recognition result file in the storage 11-1 (step 1-3), registers the situation recognition result with the MAM server 2, and notifies the completion (step 1-4).

13 is a diagram illustrating a content registration screen through a Web UI according to an embodiment of the present invention, and FIG. 14 is an ingest program for a watch folder method according to an embodiment of the present invention This is a diagram showing a content registration screen.

13 and 14, a Web UI, a watch folder, and a streaming encoding method are provided for content registration, and a registration method is selected according to the situation of the operating system.

15 is a diagram showing a content registration and pre-processing process according to an embodiment of the present invention.

15, the facial recognition-based character information providing system registers content and automatically executes a pre-processing operation after content registration is completed. Pre-processing tasks include frame image extraction, cataloging, and streaming video generation. Preprocessing tasks can be performed in parallel at the same time. In the case of content such as streaming encoding, the streaming video generation process is omitted. After the frame image extraction operation is completed, the recognition operation is automatically executed.

16 is a diagram illustrating a device configuration for person recognition according to an embodiment of the present invention.

Referring to FIG. 16, person recognition is divided into a pre-processing task through the person recognizer 133 and a person recognition task through the person recognition server 14. The person recognizer 133 includes a GUI 1330, a person recognition server 14, and an MAM linker 1332 and a recognition server linker 1334 to link the MAM server 2 with each other. The MAM linker 1332 may be connected to the MAM server 2 using TCP/IP, and the recognition server linker 1334 may be connected to the person recognition server 14 using Restful. The person recognition server 14 includes a Restful API 140 and a face recognition library 142. The pre-processing and recognition tasks are automatically performed through the MAM server 2.

17 is a diagram illustrating a person recognition process according to an embodiment of the present invention.

Referring to FIGS. 16 and 17, after content registration is completed, the face recognizer 133 extracts a frame image from the original image (for example, extracts one image per 10 frames) and uses the stored person gallery to characterize Extracts features for recognition. The face recognizer 133 starts the face recognition operation as the extracted frame image and person feature values are transmitted to the face recognition server 14. The face recognition server 14 recognizes the characters and generates a representative profile recommendation image, a character clustering image, and a frame feature storage file. The result of the recognition work is stored in the storage and DB through the MAM server 2.

18 is a diagram illustrating a device configuration for sound source recognition according to an embodiment of the present invention.

Referring to FIG. 18, the sound source recognizer 134 includes a GUI 1340, a MAM interlocker 1342, and a sound source recognition library 1344. The MAM interlocker 1342 is linked to the MAM server 2 and the sound source recognizer 134 through TCP/IP. The sound source recognizer 134 recognizes music in the image using the sound source recognition library 1344.

19 is a diagram illustrating a sound source recognition process according to an embodiment of the present invention.

18 and 19, the sound source recognizer 134 extracts a WAV file from the original image for sound source recognition. Since the section for reproducing the sound source in the content is unknown, a WAV file is generated in all sections of the content in a preset unit (for example, in units of 7 seconds). It performs sound source recognition using a WAV file and stores the primary result in the DB. Sound source recognition is performed using a sound source recognition library 1344. The MAM server 2 selects valid recognition results based on the first stored result (result reprocessing) and stores the final result. Sound source recognition is automatically performed through the MAM server (2).

20 is a diagram illustrating a device configuration for situational awareness according to an embodiment of the present invention.

Referring to FIG. 20, the situation recognition task is divided into a pre-processing task through the situation recognizer 135 and a situation recognition task through the situation recognition server 15. The context recognizer 135 includes a GUI 1350, a context recognition server 15, and an MAM linker 1352 and a recognition server linker 1354 to link the MAM server 2 with each other. The MAM linker 1352 may be connected to the MAM server 2 using TCP/IP, and the recognition server linker 1354 may be connected to the context recognition server 15 using Restful. The context awareness server 15 includes a Restful API 150 and a context awareness library 152. The pre-processing and recognition tasks are automatically performed through the MAM server 2.

21 is a diagram illustrating a situation recognition process according to an embodiment of the present invention.

20 and 21, after content registration is completed, the situation recognizer 135 extracts a frame image from the image through a pre-processing operation (for example, extracts one image per 5 frames). Then, the extracted frame image is transmitted to the situation recognition server 15 to start recognition. All recognition tasks are processed sequentially in one task. The result of the recognition operation of the situation recognition server 15 (object recognition result, event recognition result, place recognition result, landmark recognition result, video recognition result) is stored in the storage and DB through the MAM server 2.

Hereinafter, an authoring tool of an administrator who can manage content and recognition tasks will be described later with reference to FIGS. 22 to 28.

22 is a view showing a authoring tool screen according to an embodiment of the present invention.

Referring to FIG. 22, the facial recognition-based character information providing system provides a web-based authoring tool that can check and manage registered content and deep meta recognition results. The authoring tool screen includes a content management page 2200, a storyboard page 2210, a verification page 2220, a person management page 2230, a work history page, and an administrator page, and can be displayed through the manager screen. .

The content management page according to an embodiment includes a content and metadata registration screen, a content confirmation screen, a metadata confirmation screen, a preprocessing command input screen, and a recognition command input screen.

The character management page according to an embodiment includes a profile image screen recommended by the recognition server, a clustering image screen grouping the same people, and a search screen to search for and provide people related to the input portrait.

The storyboard page according to one embodiment checks the shot extraction contents generated through cataloging and checks the storyboard-shot screen, the sound source recognition result in a scene unit, and corrects the misrecognized result based on the shot. It includes a storyboard-sound recognition screen for deleting or merging items with duplicate time codes, and a storyboard-object screen for checking object recognition results and editing misrecognized results.

The verification page according to an embodiment includes a character display screen corresponding to a reproduction time point, a sound source display screen, an additional information confirmation screen of a character, a character appearance point guidance screen, and a related content guidance screen at the ending point.

23 is a diagram illustrating a content management screen of an authoring tool according to an embodiment of the present invention.

Referring to FIG. 23, content can be registered and viewed 2300 using a page for managing registered content, management status for each content may be checked 2310, and metadata may be checked 2320. In addition, pre/post processing commands 2330 and 2340 may be executed. Examples of the preprocessing command 2330 include a transcoding request, a cataloging request, a frame extraction request, and a person feature extraction request command. Examples of the post-processing command 2340 include a face recognition request, a sound source recognition request, and an object recognition request.

24 is a diagram illustrating a character management screen of an authoring tool according to an embodiment of the present invention.

Referring to FIG. 24, the content whose face recognition has been completed may change the profile picture through the character menu of the authoring tool and add a picture to the gallery. In character management, you can change your profile picture and add a gallery in 3 ways. As a first step, a list of images 2410 for a profile recommended by the face recognition server is provided. In step 2, a clustering image list 2420 grouped between the same people is provided. In step 3, a list of people 2430 similar to the input portrait is searched and provided. In each step, you can change your profile picture and add a gallery.

25 is a view showing a storyboard (shot) management screen of the authoring tool according to an embodiment of the present invention.

Referring to FIG. 25, the content of which the pre-processing and recognition tasks have been completed can be checked through the storyboard and the results of the shot extraction and recognition. The shot page provides a function to check (2500, 2510) shot extracts generated through catalogging and add, delete, move, and edit shots (2520) based on the shot. In the shot editing function, it is possible to change the shot start point in units of frames (2530).

26 is a view showing a storyboard (sound recognition) management screen of the authoring tool according to an embodiment of the present invention.

Referring to FIG. 26, the sound source recognition page of the storyboard provides a function for checking (2600, 2610) sound source recognition results in a scene unit and easily correcting the misrecognized results. For example, a sound source warning is displayed for items with duplicated timecode (2620), and duplicate sound sources can be deleted or merged with each other (2630), and a function for initializing edits is provided.

27 is a view showing a storyboard (object chain type) management screen of the authoring tool according to an embodiment of the present invention.

Referring to FIG. 27, the object recognition page of the storyboard provides a summary of object recognition results (2700), and provides a function to easily edit (2710) misrecognized results. You can check the summary and frame image of the recognition results for each recognized object class.

28 is a view showing a verification screen of the authoring tool according to an embodiment of the present invention.

Referring to FIG. 28, the verification page is a page capable of verifying deep meta results. The verification page provides a character display screen, a sound source display screen, an additional information confirmation screen of the character, a character appearance point guidance screen, and a related content guidance screen at the ending point corresponding to the playing time. For example, it provides a function to display the character corresponding to the playback time (2800), display the sound source (2810), and display (2820) the information about the character's reciprocal character. And it provides effective data verification function by displaying deep metadata in real time during video playback, such as guiding the appearance point of the character (2830) and guiding the related content at the ending point (2840).

So far, the present invention has been focused on the embodiments. Those skilled in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (11)

The content registration unit copies the content original file and stores it in storage, and registers and stores the content metadata in a media asset management (MAM, hereinafter referred to as'MAM') server;
Assigning a preprocessing task necessary for the recognition task and verification to the preprocessing server by the MAM server;
A pre-processing server performing pre-processing including transcoding, cataloging, and frame extraction;
When the MAM server assigns the recognition task to the recognition server using the pre-processed result, the recognition server recognizes a person, a situation, and a sound source in the image;
A step in which the administrator terminal checks the recognition result using an authoring tool and verifies and corrects the result; And
After verification is completed, the MAM server delivers the results to the user terminal;
It includes,
Steps to verify and correct
Provides a web-based authoring tool to check and manage registered content and deep meta recognition results,
The authoring tool provides a content management page, a storyboard page, a verification page, a character management page, and a manager page, and the workflow processing method of the face recognition based character information providing system. The method of claim 1, wherein performing the pre-treatment
A MAM server assigning a transcoding operation to the transcoder, and the transcoder converting the original image into a searchable image playable on the web;
The MAM server allocates a cataloging operation to the cataloger, and the cataloger analyzes the original image and divides the scene based on a scene change point to output a shot image; And
MAM server assigns a frame extracting task to the frame extractor, and the frame extractor extracts the frame image; It includes,
Each step of the pre-processing work is performed in parallel at the same time,
A workflow processing method of a face recognition-based character information providing system, characterized in that the communication between the MAM server and the pre-processing server's transcoder, cataloger, and frame extractor is performed using a TCP protocol. The method of claim 1, wherein the step of recognizing the person, situation and sound source in the image is
Assigning, by the MAM server, a character feature extraction task to the person recognizer;
A person recognizer calling a person recognition server for feature extraction for recognition of characters;
Extracting a feature for recognition of the character by using the gallery image of the character stored by the called character recognition server;
A step in which the person recognizer checks the feature extraction status in the person recognition server and registers the verification result in the MAM server;
Assigning a person recognition task to the person recognizer by the MAM server;
A person recognizer calling a person recognition server for person recognition;
The called person recognition server recognizes the person using the frame extraction image and the character feature file, and provides a representative profile recommendation picture, a person clustering image, a feature file for each frame, and a recognition result file; And
Storing the recognition result by the MAM server;
A workflow processing method of a face recognition-based character information providing system comprising a. The method of claim 1, wherein the step of recognizing the person, situation and sound source in the image is
Assigning a sound source recognition task to the sound source recognizer by the MAM server;
A sound source recognizer extracting a wav file from the storage and calling the sound source recognition server to the sound source recognition server using the extracted wav file;
A sound source recognition server recognizing a sound source using the sound source recognition library and registering the sound source recognition result to the MAM server;
Correcting a misrecognized result from the result recognized by the MAM server in a preset unit and merging the valid results; And
MAM server storing the final result;
A workflow processing method of a face recognition-based character information providing system comprising a. The method of claim 4, wherein the step of merging the valid results is
Providing sound source recognition results in a scene unit through a sound source recognition page of the storyboard; And
Providing an editing screen for displaying a sound source warning for the items with duplicate time codes and deleting the duplicated sound sources or merging each other;
A workflow processing method of a face recognition-based character information providing system comprising a. The method of claim 1, wherein the step of recognizing the person, situation and sound source in the image is
Assigning a context-aware task to the context-aware MAM server;
The situation recognizer transmitting the frame extraction image to the situation awareness server and calling the situation awareness server; And
Receiving a frame extraction image by the context recognition server, recognizing objects, events, places, landmarks and videos through context recognition, and providing recognition results;
A workflow processing method of a face recognition-based character information providing system comprising a. delete According to claim 1,
The content management page includes a content and metadata registration screen, a content confirmation screen, a metadata confirmation screen, a pre-processing command input screen, and a recognition command input screen. According to claim 1,
The character management page includes a face image-based character recognition characterized by including an image screen for a profile recommended by the recognition server, a clustering image screen grouping the same people, and a search screen to search for and provide people related to the input portrait photo. Information processing system workflow processing method. According to claim 1,
The storyboard page is a storyboard-shot screen that checks shot extractions generated through cataloging, creates and manages scenes based on shots, checks the sound source recognition results in units of scenes, corrects misrecognized results, and duplicates the timecode It includes a storyboard-sound recognition screen for deleting or merging deleted items, and a storyboard-object screen for viewing object recognition results and editing misrecognized results. Workflow processing method. According to claim 1,
The verification page is based on face recognition, characterized in that it includes a character display screen, a sound source display screen, an additional information confirmation screen of the character, a character appearance point guidance screen, and a related content guidance screen at the ending point corresponding to the playing time. Workflow processing method of person information providing system.

Images