KR20110085160A - Stenography input system and method for conference using face recognition - Google Patents

Abstract

PURPOSE: A face recognition conference shorthand system and method are provided to stored and confirm video and audio signals related to a speaker in the conference and to match the video and audio signal with speaker related information. CONSTITUTION: A camera unit(130) comprises one or more cameras collecting an image which includes one or more participants attending in the conference. A microphone(120) comprises one or more microphones collecting an audio signal which at least one speakers among participants utters. According to user input and set conference schedule information, a control unit(160) collects so as to collect the audio and video signals of the speaker and to store the audio and video signal as a conference record.

Description

Stenography Input System And Method For Conference Using Face Recognition

The present invention relates to a shorthand technology for meetings, to collect and output video and audio signals for at least one speaker during a conference, and to store and output each speaker, and to store and output person information of the speaker together. A facial recognition conference shorthand system and method.

The video conferencing system considered for effective communication between remote users has been proved to be effective through the development of video compression technology and network, and the frequency of use is increasing. Recently, various functions such as face recognition and speaker analysis have been combined in such a video conference system.

However, the majority of general meetings that occur in real life do not provide a system for recording or taking notes separately, so they only rely on the memo of attendees. As a result, it is difficult to clearly know what the participants have been saying during the meeting or after the meeting, and it is difficult to accurately recognize and disseminate the arguments and conclusions discussed at the meeting.

Accordingly, an object of the present invention is to provide a face recognition conference shorthand system and method for supporting a shorthand of a speaker during a meeting or playback of a recorded video.

It is another object of the present invention to provide a facial recognition conference shorthand system and method capable of recording a speaker-related video and remarks generated during a meeting so that they can be used as data for future decision-making or business progress.

Face recognition conference shorthand system according to a preferred embodiment of the present invention as described above includes the configuration of the camera unit, the microphone unit and the control unit. The camera unit includes at least one camera that collects an image including at least one participant who attended a meeting, and the microphone unit includes at least one microphone that collects an audio signal spoken by at least one speaker of the participants. do. The controller controls to collect the speaker's audio signal and the speaker's video according to a user input and preset conference schedule information, and store the same as a conference record.

The present invention also provides a method of identifying a position of a speaker to perform or perform a speech among at least one participant who attends a meeting, collecting an image of the speaker and an audio signal corresponding to the speaker's speech, and collecting the speech. Disclosed is a conference shorthand method comprising the step of storing an image of a speaker and an audio signal of the speaker. The method of shorthand meeting may include storing sample information for face recognition of the attendees, storing person information of the attendees, recognizing the speaker's image, and confirming the speaker's person information, and confirming the confirmed person information. It may include the step of storing with the video and audio signal of the speaker.

The determining of the location of the speaker may include outputting an entire image including the participant to a display unit, instructing a specific participant among the participants included in the entire image output to the display unit as a speaker, and pre-stored schedule information. Confirming, designating a specific participant as a speaker according to the schedule information, acquiring an entire image including the participant, and recognizing a face image of each participant included in the entire image to recognize the image; The method may include at least one of instructing an attendee as a speaker, collecting an audio signal through microphones assigned to the attendee, and determining an attendee at the location where the audio signal is collected as the speaker.

According to the face recognition meeting shorthand system and method of the present invention, the present invention supports to store and confirm video and audio signals related to a speaker during a meeting, and to match various speaker related information, thereby storing various information related to a meeting, Can be managed and searched.

In addition, the present invention can support to accurately shorten the speech of the speaker during the meeting or during playback of the recorded video recording. In other words, when multiple speakers speak in a general meeting, it is difficult to identify which speaker is speaking because of the similar voice or appearance, or the authenticity, nuance, identification of the speaker, and confirmation of the pronunciation, for example, in a foreign-terminology terminology. In some cases, it is difficult to be shorthanded, and it is easy to shorten by recognizing the participant's face during the meeting and distinguishing the current speaker. To help.

1 is a view showing an example of a conference situation according to an embodiment of the present invention,
2 is a block diagram schematically showing a configuration of a face recognition conference shorthand system according to an embodiment of the present invention;
3 is a view showing in more detail the configuration of the control unit of FIG.
4 is a diagram illustrating an example of a screen interface provided for a conference recording support process of the present invention;
5 is a flowchart illustrating a conference shorthand method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, only parts necessary for understanding the operation according to the embodiment of the present invention will be described, and the description of other parts will be omitted so as not to disturb the gist of the present invention.

The terms or words used in the specification and claims of the present invention described below should not be construed as being limited to ordinary or dictionary meanings, and the inventors should use the concept of terms to explain their own invention in the best way. Based on the principle that it can be properly defined, it should be interpreted as meaning and concept corresponding to the technical idea of the present invention. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

1 is a view showing an example of a meeting situation that can be supported by the face recognition meeting shorthand system according to an embodiment of the present invention.

Referring to FIG. 1, at the start of a meeting, a plurality of attendees are each seated in a meeting seat to perform a certain type of meeting. At this time, each participant is preferably seated according to the meeting schedule information. Meanwhile, the face recognition conference shorthand system of the present invention includes a camera unit 130 and a microphone unit 120, and stores, outputs, and outputs video and audio signals collected by the camera unit 130 and the microphone unit 120. It may include a manageable electronic device. Accordingly, the microphone may be provided in plural so as to be arranged for each seat arrangement within the conference hall. In this way, the plurality of microphones 120 may be provided with an ID number to distinguish the audio information collected at each position. Here, the face recognition conference shorthand system of the present invention has been described as including a plurality of microphone units 120, but the present invention is not limited thereto. That is, in the face recognition conference shorthand system of the present invention, only one microphone may be disposed in the conference hall.

On the other hand, the camera unit 130 is preferably disposed at an angle capable of taking a picture of the participants attending the conference room. For example, when six attendees are seated around a table, the camera unit 130 may be arranged to have an angle at which each of the six attendees may photograph their face. In this case, a case in which the face of each participant cannot be photographed according to the shape of the table may occur. Accordingly, the face recognition meeting shorthand system of the present invention may support the capturing of each participant's face by arranging a plurality of cameras at a specific point. In this case, when a plurality of cameras are arranged, the face recognition conference shorthand system may assign ID numbers to the cameras and check which cameras the collected video signals are received.

Meanwhile, in the case of the conference room shown in FIG. 1, participants are arranged in parallel around a rectangular table, but the present invention is not limited thereto. That is, the environment to which the face recognition meeting shorthand system of the present invention can be applied is not limited to the form of the conference hall or the devices provided in the conference hall, and the front face or the side face of the attendees who attended the conference may be respectively photographed. Any environment may be applicable as long as the camera and the microphone device capable of collecting the voices of the attendees may be properly disposed. In addition, since the camera unit 130 and the microphone unit 120 support a plurality of operations, the face recognition conference shorthand system of the present invention may substantially support a general conference process without any limitation.

Hereinafter, the configuration of the facial recognition meeting shorthand system of the present invention disposed in the conference room to support management of various conference records discussed in the conference room will be described in more detail with reference to FIG. 2.

2 is a block diagram schematically illustrating a configuration of a face recognition conference shorthand system according to an exemplary embodiment of the present invention.

Referring to FIG. 2, the face recognition conference shorthand system 100 of the present invention includes an input unit 110, a microphone unit 120, a camera unit 130, a display unit 140, a storage unit 150, and a controller 160. It may include the configuration of). The microphone unit 120 and the camera unit 130 may be divided and distributed in a specific point or location of the conference hall. Accordingly, the face recognition conference shorthand system 100 may further include a communication interface capable of receiving video and audio signals from the microphone unit 120 and the camera unit 130. In this case, the microphone unit 120 and the camera unit 130 may also include a configuration capable of forming a communication channel with a communication interface of the face recognition conference shorthand system 100. For example, the microphone unit 120 and the camera unit 130 may be connected to the control unit 160 of the face recognition shorthand system 100 by wire, such as a cable. In addition, the microphone unit 120 and the camera unit 130 may be wirelessly connected to the controller 160 of the face recognition shorthand system 100. That is, the communication interface of the face recognition conference shorthand system 100 may be implemented in various communication interfaces such as wired and wireless according to the intention of an operator who operates the face recognition conference shorthand system 100 or designers configuring the system. . In addition, the face recognition conference shorthand system 100 may further include an audio processor including a speaker for outputting the collected audio signal.

On the other hand, in the face recognition meeting shorthand system 100 of the present invention having the above-described configuration, when the meeting starts, the activation of the microphone unit 120 and the camera unit 130 according to the meeting schedule information stored in the storage unit 150 Activation can be controlled. For example, when a participant is scheduled to speak at a location designated as the first seat in the meeting schedule information, the face recognition conference shorthand system 100 may use a microphone included in the microphone unit 120 disposed at the first seat. While controlling to activate, the camera unit 130 may be controlled to collect an image corresponding to the face of the attendee seated in the first seat. Here, the face recognition conference shorthand system 100 supports outputting the entire image of the entire conference hall, and separates the participant and the surrounding background through image recognition of the captured image, or separates the face and the surrounding background of the participant. Can be supported. When the conference organizer or the conference manager designates a specific participant or a specific participant's face by using the input unit 110 and the display unit 140, the face recognition conference shorthand system 100 includes a microphone disposed at the participant's location. The controller 120 may be activated to collect and store the audio signal spoken by the attendee and the video signal related to the attendee. To this end, the face recognition conference shorthand system 100 separates the participant or the participant's face from the entire image and the surrounding background, and then assigns the participant or participant's face on the display unit 140 to the microphone unit 120. Can be controlled to link Then, when the administrator designates a specific object, the face recognition conference shorthand system 100 may control to activate a microphone corresponding to the designated object. In this case, the face recognition conference shorthand system 100 may refer to meeting schedule information for matching a corresponding object and a microphone, and may also change a feature value of a specified object related to speech through image recognition, for example, a mouth shape. Determine if there is a change. As described above, the face recognition conference shorthand system 100 of the present invention, when various participants in the conference room speaks according to the meeting schedule information and the designation of the conference manager, the audio signal corresponding to the video signal and the contents of the speaker. The signal can be collected and stored. In addition, the face recognition conference shorthand system 100 of the present invention, when a particular participant wants to speak, the audio signal collected from the microphone assigned to the speaker is automatically detected and the video signal corresponding to the speaker is automatically collected. can do. In addition, the face recognition conference shorthand system 100 may determine which participant is speaking by recognizing the face of the participant, and accordingly automatically activates the microphone assigned to the participant who is speaking, and then displays the image of the participant. Signal and audio signals can be collected. Hereinafter, each configuration of the face recognition conference shorthand system 100 will be described in more detail.

The input unit 110 includes a plurality of input keys and function keys for receiving numeric or character information according to a user's operation and setting various functions. The function keys may include a direction key, a side key, and an accelerator key set for performing a specific function. In addition, the input unit 110 transmits a key signal input in relation to a user setting and a function control of the camera unit 130 and the microphone unit 120 to the control unit 160. In particular, the input unit 110 of the present invention generates various input signals for controlling the image capturing operation. For example, the input unit 110 may include an input signal corresponding to a shutter function of the camera unit 130, an input signal for searching, storing, and deleting various images displayed on the display unit 140, a zoom in function, or a zoom out function. An input signal corresponding to the input signal, an input signal corresponding to various photographing option functions provided by the camera unit 130, and the like are generated and transmitted to the controller 160. The photo shooting option function includes a change in illuminance, a change in resolution, a filter insertion, an image preprocessing providing function, and the like when taking a photo. The input unit 110 may receive a shorthand according to a user's key selection. The input unit 110 may be implemented in various forms such as a keyboard, a keypad, a mouse, and a voice input device. In addition, when the display unit 140 is manufactured in the form of a touch screen, the display unit 140 may support a function as the input unit 110.

The microphone unit 120 is configured to include at least one microphone disposed in the conference room to collect the comments of the attendees. At least one microphone included in the microphone unit 120 may be a USB microphone, and in some cases, a microphone having various characteristics, for example, a specific directivity, may be applied. The microphones included in the microphone unit 120 may be activated and deactivated under the control of the controller 160. For example, it is assumed that six microphones are included in the microphone unit 120, and each microphone is disposed at a specific position to collect an audio signal according to a participant's speech. Then, the six microphones may be given a predetermined ID, respectively, and the activation time may be different according to the control of the controller 160. That is, the controller 160 may sequentially change the time points at which the microphones are activated according to the meeting schedule information. Accordingly, the microphones included in the microphone 120 may be sequentially activated. In addition, the activation time of the microphones may be changed by an input signal input from the input unit 110, and the specific microphone performs image recognition on the image signal collected by the camera unit 130 as described above. As a result it can be activated.

The camera unit 130 includes at least one camera for capturing an image corresponding to at least one of all the attendees or the attendees. The camera unit 130 captures a predetermined image and receives the data as a data signal, and transmits the data signal to the controller 160. The data signal photographed by the camera unit 130 may be temporarily stored or semi-permanently stored according to a user selection along with the audio signal collected by the microphone unit 120. The camera unit 130 may capture an image of the subject and transmit the captured image to the display unit 140 as a preview screen. The camera unit 130 may focus on the subject image in response to a specific image of the subject image. Substantially, the camera unit 130 is controlled to photograph the entire conference hall, a specific person in the conference hall, or a face of a specific person under the control of the controller 160, and transmits the image signal collected in this process to the controller 160. . According to the control of the controller 160, a specific person or a face of a specific person may be focused and photographed, and the focused image may also be transmitted to the controller 160.

The display unit 140 may provide information input by a user or a menu including meeting schedule information, a video signal collected by the camera unit 130, a screen corresponding to the audio signal collected by the microphone unit 120, and various menus. Display information screens. In particular, the display unit 140 includes an image captured by the camera unit 130, person information corresponding to the result of the image recognition through the captured image, information of the speakers collected during the meeting time, and a speaking time and contents of the speakers. You can output an icon indicating to select and confirm. The display unit 140 may provide an image of a subject currently projected by the camera unit 130 as a preview screen. The preview screen function is a function of displaying an image of a subject incident through a lens before the camera unit 130 actually captures an image. Based on the preview function, the display unit 140 may display the image captured by the camera unit 130 before storing the image. In addition, the display unit 140 may display a confirmation message for storing and deleting the user's image. In this process, the display unit 140 may output an image signal of the speaker photographed by the camera unit 130, information corresponding to a result of recognizing the photographed speaker, and output the corresponding information in a list form. Can also be. In addition, the display unit 140 may output a voice waveform corresponding to the audio signal collected by the microphone unit 120. The screen interface of the display unit 140 will be described in more detail with reference to the accompanying drawings.

The storage unit 150 temporarily or semi-permanently stores an image acquired when capturing an image, including an application program required for a function operation according to an embodiment of the present invention, and temporarily stores the captured image before storing / deleting the image. It is responsible for the temporary storage function for the continuous picture in the buffering function and the photographing mode in which a specific image is selected from among a plurality of continuous images. The storage unit 150 may largely include a program area and a data area.

The program area stores an operating system (OS) for booting the face recognition conference shorthand system 100. The program area may include an application required to support the function if the face recognition conference shorthand system 100 supports other options such as a camera function, a sound playback function, an image or video playback function, and the like. Save it. In particular, the program area of the present invention includes a meeting recording support application (meeting recording support.app) for shorthand meetings.

The meeting recording support application (meeting recording support.app) controls the activation time of at least one microphone included in the microphone unit 120 according to preset meeting schedule information and a user input signal, and controls the camera unit 130. Controls activation of at least one camera included and focusing of the camera, and controls to store or output the image and audio signals collected based on the microphone unit 120 and the camera unit 130. . To this end, the meeting recording support application (meeting recording support.app) includes a routine for checking meeting schedule information, a routine for controlling the microphone unit 120 according to the meeting schedule information, and a user input, the meeting schedule information, and a user. A routine for controlling image capturing by the camera unit 130 according to an input, a routine for activating an image recognition algorithm for recognizing a captured image, a routine for retrieving person information of a captured image based on the recognized image data, A routine for controlling to integrate and store at least one of an audio signal collected by the microphone unit 120, an image signal collected by the camera unit 130, and the person information, and at least one of the audio signal, image signal, and person information It may include a routine for outputting one to the display unit 140.

The image recognition algorithm is an algorithm for recognizing an image of a subject, and is an algorithm for determining which image is compared by comparing a shape, an operation state, and the like of the subject with previously stored image samples. For example, when the subject image is an image related to the face of the person, the image recognition algorithm extracts the position of the eye, the position of the nose and the position of the leaf from the face, and the change of the distance and the change of position between the respective positions. Based on the expression, the face state of the current person may be determined based on the expression.

In particular, the image recognition algorithm of the present invention, when using at least one camera in a system for recording a speaker during a meeting, finds a camera where the speaker is located and recognizes a face collected based on the camera. In this case, the conference recording support application may determine the position of the camera by using the principle of performing the location check through the microphone unit 120 and reducing the size of the voice signal according to the distance.

In order to perform the image recognition algorithm, the conference recording support application may perform a face region extraction process and a normalization process. In more detail, the conference recording support application first performs histogram smoothing of the entire image to extract the face region from the image obtained from the camera at the location where the speaker is estimated to be present. In addition, the conference recording support application detects a plurality of face regions existing in the image by using a Haar-like feature value. The face area is normalized by extracting only the face object after deleting the surrounding background based on the face color information. Normalization assumes that the face object is an ellipse, and then obtains the center of gravity and the long axis to correct the face from tilting. Based on the bilinear interpolation, the face objects of various sizes are enlarged / reduced to a certain size. The face object normalization process consists of a face image object, finding the center of gravity of the face region, moving the coordinates of the region, finding the covariance array, finding the Eigen values, finding the Eigen vector, finding the slope (angle), rotating the face region, zooming in and out of the face region. It may include the process of acquiring the face image object.

The conference recording application application verifies the identity by using feature values of the normalized face object during face recognition. In this case, the conference recording support application may use the midpoint of both eyes, the midpoint of the nose, and the position of the upper lip as feature values. In addition, the conference recording support application may determine that the minimum distance is a similar face using the Euclidean distance in the face comparison process, and the feature value is used in the face comparison coordinate system by applying the feature value. Distortion of the face area due to change or noise, and up, down, left and right rotation of the face according to free posture can be reflected. The conference recording support application may use a ratio between feature values as face comparison coordinates.

For example, in two-dimensional coordinates, the nose X-axis position Xnose, the left eye X-axis position Xleye, the right eye X-axis position Xreye, the nose Y-axis position Ynose, the eye Y-axis position Yeye, Assuming the Y-axis position of the mouth is Ymouse, we can define dx = | Xnose-Xleye | / | Xnose-Xreye |, dy = | Ynose-Yeye | / | Ynose-Ymouse |.

Then, the conference recording support application may calculate the distance between the sample image collected in advance, that is, the face template (dx _t , dy _t ) and the facial feature values (dx, dy) obtained during the actual conference recording, through Equation 1 below. have.

Since the feature value for the coordinate value is larger in the case of the left and right rotation than the up and down rotation, it is necessary to consider the left and right rotation of the face when extracting the feature value. Techniques are used to extract the edges and use them to find the horizontal feature lines of the eyebrows, eyes, nose, and mouth. In this case, the conference recording application application divides the face area into two parts of the left and right sides in consideration of the left and right rotation of the face, collects the edges, integrates the collected edges by overlapping the middle area, and selects a representative value. .

The meeting recording support application uses a representative value of the nose to determine the face left and right rotation. The meeting recording support application assumes that the position of the nose is located at the horizontal midpoint of all face regions in the face image facing front, and determines that the face is rotated as the nose representative value is shifted from side to side. Do the work. The conference recording support application calculates an angle for shifting the representative value of the nose biased to one side and converts the representative value of both eyes using this value. This can be done on the assumption that the feature points of the nose and eyes are on the same circumference on the face. In the face recognition process, in order to correct such an error, it is preferable to obtain a face comparison coordinate and calculate a distance by using an average of feature values obtained from an image of 4 frames or more.

Meanwhile, when the distance between the plurality of template coordinates and the facial feature value coordinates is measured equally, the conference recording support application displays the templates existing in each quadrant in the order of 3-2-4-1 quadrants based on the facial feature value coordinates. You can choose. This is to first select that the error rate of the vertical rotation is lower than the left and right rotation and the difference between the feature values is small.

The conference recording application application performs speaker recognition after the face recognition process as described above, and uses the feature value and the face object obtained in the face recognition process. At this time, due to the nature of the speaker recognition problem, several frames that are continuous in time are required, and it is preferable that the number of frames per second used for the determination is guaranteed to be a certain amount or more.

The recognition process first calculates the variance of the edge scatter plot of the lip area under the nose and measures the rate of change. The face object with the measured change rate value for the predetermined time is recognized as the speaker, and the face object having a small change in the circumference of the mouth on a continuous frame may be determined as not the speaker and may be excluded from the speaker recognition object.

The process of calculating the rate of change is first determined as the sum of the number of points where each line meets the edges, based on the horizontal line, based on the edge scatter map around the entrance.

Where n is the length of the mouth region, i is the index of each horizontal line, f _i is the number of points where the i-th horizontal line meets the edges, and m is an average value. Meanwhile, the average value m may be the same value as in Equation 3 below.

The scatter change rate may be obtained through Equation 4 below, and may be included in the speaker recognition object based on the T value.

When the speaker is recognized, the conference recording support application may control to adjust the position by using the pan, tilt, and zoom modules of the camera unit 130 to position the speaker in the image.

The data area stores conference recording data generated according to the operation of the face recognition conference shorthand system 100. That is, the data area may automatically store a conference record including an audio signal collected by the microphone unit 120 and an image signal collected by the camera unit 130. The data area may store sample information corresponding to a speaker and person information of a speaker to identify a speaker through the image recognition. The sample information is provided for image recognition, and the person information of the speaker may be output to the display unit 140 or separately stored together with the audio signal and the image signal. The sample information may include image feature value information from various aspects such as the front face, the right side, and the left side of the participant for image recognition with higher accuracy. In addition, the conference recording data may store shorthand information input through the input unit 110 while the user views the conference video being recorded in real time or reproduces the recorded conference video.

The controller 160 controls the overall operation of the face recognition meeting shorthand system 100 and the signal flow between internal blocks of the face recognition meeting shorthand system 100. In particular, the controller 160 of the present invention can control data generation, storage, and output by checking schedule information necessary during a conference, collecting audio signals and video signals of a speaker, and recognizing images. In this process, the controller 160 controls the camera unit 130 to individually recognize the speaker's face in the entire image from which several speakers are released and focus on the speaker's face. After the meeting starts, the user clicks on a specific face area using the input unit 110 on the display unit 140 on which the face of each speaker focused by the camera unit 130 is output, or automatically according to preset meeting schedule information. When an individual speaker is selected, the video and audio of the selected individual speaker may be separately displayed and the timeline and the audio waveform of the speaker may be displayed on the display unit 140 of the entire image. In this case, the speaker image separation display may be manually or automatically classified by the user using the input unit 110. The control unit 160 which is automatically classified controls to automatically output to the display unit 140 a voice waveform obtained by analyzing an image of a speaker and an audio signal of a speaker according to the meeting schedule information. At this time, the controller 160 separates and displays the current speaker's image, and when the speaker finishes speaking, the display unit 140 removes the audio waveform corresponding to the video signal and the audio signal corresponding to the speaker, and the conference schedule information. According to the control, the image of the next speaker is collected and output to the display unit 140. When selecting a face of a specific speaker in an image recorded and reproduced in real time, or selecting a face of a specific speaker from among all the recorded images, the controller 160 displays only the voice and the image of the selected speaker. Can be controlled to output through. A detailed configuration of the controller 160 will be described in more detail with reference to FIG. 3.

On the other hand, the controller 160 recognizes the speaker's face each time the number of speakers increases, and controls to store and output the separated speaker's video and audio waveforms. It is possible to control operations such as changing the background color or highlighting so that the output is distinguished from the output of video and audio waveforms of other speakers. Here, the image related to the speaker may include an image of the speaker, a speaker name, a speech start time, a speech end time, and the like.

As described above, the face recognition conference shorthand system 100 according to an exemplary embodiment of the present invention applies a web camera to a camera applied to the camera unit 130, and uses a USB microphone as a microphone applied to the microphone unit 120. If applied, it can be implemented based on relatively low installation cost. In addition, each module of the face recognition conference shorthand system 100 may be independently configured to actively cope with changes in equipment or changes in the environment. In addition, the face recognition conference shorthand system 100 of the present invention may be implemented to make the conference recording more user-oriented based on face recognition information and speaker information in the recording process of the image. In addition, the face recognition conference shorthand system 100 of the present invention extracts feature values used in face recognition to be robust to face rotation as described in the above-described image recognition algorithm, and recognizes participants having various postures in a meeting of free situation. Can be supported. The face recognition conference shorthand system 100 of the present invention recognizes a speaker through a speaker position analysis, facial region extraction and normalization, face recognition, and speaker recognition process, and recognizes an image signal corresponding to the speaker and an audio signal spoken by the speaker. Can be collected.

3 is a view showing in more detail the configuration of the controller 160 of the face recognition conference shorthand system 100 according to an embodiment of the present invention.

Referring to FIG. 3, the controller 160 of the present invention includes a conference scheduling unit 161, an audio collecting unit 163, an image recognition and collection unit 165, and a conference record storage and output unit 167. .

The conference scheduling unit 161 is a component that manages conference scheduling such as collecting, storing, and outputting video and audio signals required for conference progress. The conference scheduling unit 161 checks conference schedule information stored in the storage unit 150, analyzes an input signal input by a user through the input unit 110, or analyzes the location of the speaker, or collects an audio signal. Determine the position of the microphone or determine the position of a specific speaker by performing the face image recognition of the participants in the entire image collected by the camera unit 130, accordingly using the microphone unit 120 and the camera unit 130 It is configured to control to collect an image of the speaker and to collect an audio signal corresponding to the speech. The conference scheduling unit 161 recognizes the speaker's person information by recognizing the speaker's image collected by the camera unit 130, checks the person's person information previously stored in the storage unit 150, and then checks the speaker's person's information. It may be controlled to search for the corresponding person information. Thereafter, the conference scheduling unit 161 may transmit corresponding data, that is, audio signal, video signal, and person information, to the conference record storage and output unit 167. In this case, the conference scheduling unit 161 may request the image recognition and collection unit 165 to activate the camera unit 130 and then request to obtain an image of all participants in the conference hall. When the image of all participants is obtained, the conference scheduling unit 161 may transfer the entire image to the display unit 140 by transferring the obtained image to the conference record storage and output unit 167. When a user input indicating a specific speaker is displayed among all images output to the display unit 140, the conference scheduling unit 161 controls to collect an audio signal by activating a microphone assigned to the corresponding speaker, and the speaker. Collect the image of the user, and recognize the collected image can be controlled to collect the speaker's person information. When there is no user input or separate conference schedule information, the conference scheduling unit 161 determines the location of the speaker to be spoken according to the conference schedule information, and activates the microphone of the corresponding location and focuses the speaker image of the location. Can be controlled. When the microphone does not collect an audio signal for a preset time or an input signal corresponding to the transfer of the voice is generated, the conference scheduling unit 161 activates the microphone located at the location of the next speaker according to the conference schedule information. Or to activate the microphone in the transferred position.

The audio collection unit 163 controls the microphone unit 120 according to the control of the conference scheduling unit 161 and controls to collect the speech of the speaker in which the microphone is arranged by using the activated microphone. The audio collector 163 manages ID information and location information of at least one of the microphones included in the microphone 120 to determine which microphone to activate, and accordingly microphone activation and audio signal. You will control the collection. In this case, the ID information and the location information may be included in the conference schedule information. Accordingly, the audio collector 163 may determine the microphone activation according to the conference schedule information stored in the storage unit 150. In addition, when the user generates an input signal for activating a microphone disposed at a specific position in a manner of instructing a specific speaker in the entire image output to the display unit 140, the audio collector 163 inputs the corresponding signal. You can control to activate the microphone assigned to the location based on the signal. To this end, the conference scheduling unit 161 controls to map the positions of the speakers and the arrangement positions of the microphones in the entire image disposed on the display unit 140 when the entire image collection and display unit 140 is output, and maps the mapping information. The audio collecting unit 163 may transmit the audio collection unit 163.

The image recognition and collection unit 165 controls the camera unit 130 according to the control of the conference scheduling unit 161 and controls to collect an image of a speaker disposed at a specific location using an activated camera. to be. In addition, the image recognition and collection unit 165 may perform image recognition on the collected image and determine who is the speaker using the sample information stored in the storage unit 150. To this end, the meeting scheduling unit 161 may acquire person information about the meeting participants in advance, and obtain face recognition information of each attendee in advance so that the person information on the face recognized during the meeting may be retrieved. Can support

The conference record storage and output unit 167 receives the image collected by the image recognition and collection unit, the audio signal collected by the microphone, and the person information under the control of the conference scheduling unit 161, and displays the corresponding signal and information. ) To control the output. The conference record storage and output unit 167 may control to automatically store corresponding information in the storage unit 150 according to a preset schedule, or control to store the information in the storage unit 150 according to a user input. Alternatively, the meeting record storage and output unit 161 may collect an audio signal of the speaker and an image of the speaker according to a user input and preset meeting schedule information, and store and reproduce the same as the meeting record. The information output by the conference record storage and output unit 167 will be described in more detail with reference to FIG. 4.

4 is an exemplary view illustrating a screen interface provided by the face recognition meeting shorthand system 100 according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the face recognition meeting shorthand system 100 collects an image including all attendees who participated in the conference by activating the camera unit 130 as shown in a screen 401, and the collected image is an entire image. As 141, the display unit 140 may be output to a predetermined area of the display unit 140. In this process, the face recognition conference shorthand system 100 may perform image recognition on the entire image 141, extract a face region of the attendees, and give a link function to the extracted face region. That is, when the user indicates the extracted face area, the face recognition meeting shorthand system 100 controls to focus the face area of the participant, and controls to search for the participant information by recognizing the focused face area as an image. In addition, it may support a link function that controls to activate the microphone assigned to the participant's location. Alternatively, the face recognition meeting shorthand system 100 may automatically recognize the speaker through face recognition and perform a corresponding function when at least one of the participants starts to speak.

Accordingly, when the user clicks on the face area of the first speaker 143 located at the top right of the entire image 141, the face recognition conference shorthand system 100 may select the face area of the first speaker 143. The display may be enlarged and output on one side of the bottom of the display unit 140. When the first speaker 143 finishes speaking, the face recognition conference shorthand system 100 focuses the face of the newly designated second speaker 145, and displays the face of the focused second speaker 145. Can be controlled to output to one side. In this case, the face recognition conference shorthand system 100 may remove the image corresponding to the first speaker 143 from the display unit 140 and output only the image of the second speaker 145.

Meanwhile, the face recognition conference shorthand system 100 may control to automatically focus the speaker's face corresponding to the speaking order according to the meeting schedule information even if the user does not instruct the specific speaker. That is, when the first speaker 143 speaks according to the meeting schedule information, the face recognition conference shorthand system 100 controls to focus on the face of the first speaker 143 and to enlarge the output to one side of the display unit 140. can do. The face recognition meeting shorthand system 100 may enlarge the face image on the second speaker 145 according to the meeting schedule information when the preset time of the preset first speaker 143 has elapsed or there is no speech for a predetermined time. The display unit 140 may control the output to the display unit 140. In this case, the face recognition conference shorthand system 100 may maintain the image of the first speaker 143 as the image of the previous speaker without removing the image from the display 140, and may be removed from the display 140 according to a setting. There is also. When the next speaker is selected, the image of the first speaker 143 may be removed, and the second speaker 145 may be output to the display unit 140 as an image of the previous speaker.

In addition, when there is no separate meeting schedule information, the face recognition conference shorthand system 100 recognizes the faces of each participant in the entire image 141 but recognizes in detail the image of the part of the mouth to see which participants are speaking. You can also judge. In addition, the face recognition conference shorthand system 100 may enlarge the face of the participant who speaks and output the face to the display unit 140 and activate the microphone assigned to the corresponding position. In this process, the face recognition conference shorthand system 100 may control to activate the microphone in advance, and determine the speaker according to the collection of the audio signals collected by the microphone.

Next, the face recognition conference shorthand system 100 may store the speaker information in a list form according to a user manipulation or environment setting, and output the stored list form information as shown on a screen 403. To this end, the face recognition meeting shorthand system 100 has a meeting name as a category, and a speaking time including face image information of the speakers who have spoken among the participants who attended the meeting, and a start time and an end time of the speaking. A list item including information, person information of a speaker, and an audio file 147 of a speaker may be generated and managed. The audio file 147 may include at least one of a speech waveform of the speaker and contents of the speaker's speech. When the administrator who operates the face recognition conference shorthand system 100 clicks the corresponding audio file 147, the face recognition conference shorthand system 100 may output the speaker's statement through the audio processor. Accordingly, the audio waveform may be output to the display unit 140.

In the above, each configuration, role, and function of the face recognition conference shorthand system 100 according to an exemplary embodiment of the present invention have been described, and the screen interface supported by the system has been described. Hereinafter, a conference shorthand method based on the face recognition conference shorthand system 100 will be described in detail with reference to the accompanying drawings.

5 is a flowchart illustrating a conference shorthand method according to an embodiment of the present invention.

Referring to FIG. 5, the face recognition meeting shorthand system 100 first checks meeting schedule information in step 501. That is, the face recognition conference shorthand system 100 determines the location of which speaker is speaking in advance by checking the meeting schedule information, and activates the microphone assigned to the corresponding location, while the camera unit 130 of the You can adjust the focusing to the appropriate position. In this case, if there is no meeting schedule information, the face recognition meeting shorthand system 100 controls to collect the whole image, recognizes the face of the participants in the whole image, and searches for the speaker who performs the speech, or the microphone unit 120 According to the collected audio signal, an attendee who is seated at a location where the audio signal is collected may be determined as a speaker.

Next, when schedule information exists, the face recognition conference shorthand system 100 may control to collect audio and video signals according to the schedule information in step 503 and perform image recognition on the collected images. If there is no schedule information, in step 501, the face recognition conference shorthand system 100 may collect an audio signal corresponding to the determined speaker's statement and an image of the speaker and perform image recognition on the collected image.

Next, the face recognition conference shorthand system 100 may store the conference record based on the information collected in step 505 and the image recognized person information, and control the output to be output through the display unit and the audio processor. At this time, the face recognition conference shorthand system 100 may collect the speaker's audio signal and the speaker's video according to a user input and preset conference schedule information, and store and reproduce the same as a conference record. In this process, the face recognition conference shorthand system 100 may control the display unit 140 to output an image corresponding to the speaker's face, an audio waveform image corresponding to the collected audio signal, and person information of the speaker. . In addition, the user may perform shorthand through the input unit 110 while watching a meeting video being recorded in real time or while playing the recorded meeting video.

Meanwhile, the face recognition conference shorthand system 100 may check whether audio reception occurs from an unscheduled microphone as in step 507 during the conference according to schedule information. In this case, if there is an audio reception, the face recognition conference shorthand system 100 branches to step 509 to control the collection of video and audio signals of the corresponding area and perform image recognition on the collected video signals. Thereafter, the face recognition conference shorthand system 100 may branch to step 505 to repeatedly perform the following process.

On the other hand, if there is no audio reception in step 507, the process proceeds to step 511 and confirms whether or not the end of the meeting, if there is no input signal for instructing the end of the meeting, branching to step 501 can be controlled to repeatedly perform the following process. have.

On the other hand, the embodiments of the present invention disclosed in the specification and drawings are merely presented specific examples for clarity and are not intended to limit the scope of the present invention. It is apparent to those skilled in the art that other modifications based on the technical idea of the present invention can be carried out in addition to the embodiments disclosed herein.

100: Face recognition shorthand system
110: input unit
120: microphone
130: camera unit
140: display unit
150: storage unit
160: control unit

Claims (13)

In the face recognition shorthand system,
A camera unit including at least one camera for collecting an image including at least one attendee who attended a meeting;
A microphone unit including at least one microphone for collecting an audio signal spoken by at least one speaker among the attendees;
A controller configured to collect an audio signal of the speaker and an image of the speaker according to a user input and preset meeting schedule information and store the same as a conference record;
Face recognition conference shorthand system comprising a. The method of claim 1,
A display unit configured to output at least one of an image of a speaker collected by the camera unit, an audio waveform of the audio signal, and an entire image including all the attendees;
Face recognition conference shorthand system further comprising. The method of claim 2,
An input unit configured to generate an input signal for instructing a speaker who performs speech among all images output to the display unit;
Face recognition conference shorthand system further comprising. The method of claim 3,
The display unit
An area in which the entire image is output;
An area for enlarging and outputting a speaker's face indicated by the input unit of the entire image;
Face recognition conference shorthand system comprising a. The method of claim 1,
A storage unit for storing the conference record and storing person information of the speaker;
Face recognition conference shorthand system further comprising. The method of claim 5,
The control unit
Determining at least one of the speaker recognized according to the preset meeting schedule information, the speaker instructed according to the user input, the entire image including the attendees, the speaker is recognized, the audio signal and the video signal of the determined speaker A meeting scheduling unit configured to control to collect and control to collect person information of the determined speaker;
An audio collector configured to collect an audio signal by activating a microphone assigned to the speaker's position determined by the conference scheduling unit;
An image recognition and collection unit configured to collect a speaker image by allocating focusing to a speaker determined according to the conference scheduling unit;
A meeting record storage and output unit configured to output and store at least one of the audio signal, the image, and the person information;
Face recognition conference shorthand system comprising a. The method of claim 6,
The meeting record storage and output unit
A facial recognition conference shorthand system, characterized in that the audio signal of the speaker and the speaker's video are collected according to the user input and preset meeting schedule information, and are simultaneously recorded and stored as a conference record. Identifying the location of a speaker to speak or to speak among at least one participant in the meeting;
Collecting an image of the speaker and an audio signal corresponding to the speaker's statement;
Storing the collected speaker's image and the speaker's audio signal;
Face recognition meeting shorthand method comprising a. The method of claim 8,
The process of checking the location of the speaker
Outputting the entire image including the participant to a display unit, and instructing a specific participant among the participants included in the entire image output to the display unit as a speaker;
Identifying previously stored schedule information and designating a specific participant as a speaker according to the schedule information;
Acquiring a full image including the participant, and instructing a participant who is recognized as an image to perform a speech by recognizing a face image of each participant included in the full image;
Collecting an audio signal with microphones assigned to the attendees, and determining an attendee at the location where the audio signal is collected as a speaker;
Face recognition meeting shorthand method comprising the step of at least one of. The method of claim 8,
Storing sample information for face recognition of the participants;
Storing person information of the participants;
Recognizing the speaker's image to identify the speaker's person information and storing the identified person's information together with the speaker's image and audio signal;
Face recognition meeting shorthand method, characterized in that it further comprises. The method of claim 8,
Outputting at least one of an image of a speaker collected by the camera unit, an audio waveform of the audio signal, and an entire image including all the attendees;
Face recognition meeting shorthand method, characterized in that it further comprises. The method of claim 11,
When a specific speaker is instructed while the entire image is output, enlarging and outputting a face of the indicated speaker;
Face recognition meeting shorthand method, characterized in that it further comprises. The method of claim 8,
The storing process,
Collecting the speaker's audio signal and the speaker's video according to a user input and preset conference schedule information, and storing and playing the same as a conference record;
Face recognition meeting shorthand method, characterized in that it further comprises.

Images