KR102368233B1 - Flatform system for providing video communication based on plug-in and method for providing video communication voting using the same - Google Patents

Abstract

A plug-in-based video communication platform system according to one technical aspect of the present invention, includes: a user application which is installed on a user terminal and provides a language conversion service to a user through the user terminal; and a service server which provides the language conversion service between a plurality of user applications by interworking with a plurality of user applications which are interlocked with each other for video communication, obtains voice information provided by any one user application in order to provide the language conversion service, converts the obtained voice information into text information, and provides at least one of the converted text information or sign language motion information corresponding to the converted text information to another user application.

Description

A plug-in-based video communication platform system and a method for providing video communication using the same

The present invention relates to a plug-in-based video communication platform system and a method for providing video communication using the same.

With the development of Internet technology and the deepening of the pandemic environment, various video communication technologies are being developed.

However, since the conventional video communication technologies are only at the level of a simple video call, there is a problem in that communication and utilization are still difficult for the weak in communication, such as the disabled, the elderly, or in an environment where the language is different.

Accordingly, there is a need for video communication that enables fast communication regardless of country or language for all users around the world as well as socially vulnerable groups such as the visually impaired and the elderly.

Laid-open Patent Publication No. 10-2015-0045335 (published date: April 28, 2015)

One technical aspect of the present invention is to solve the problems of the prior art, and to provide fast communication regardless of country or language for all users of the world between them and non-disabled as well as socially vulnerable groups such as the hearing impaired and the elderly. It is to provide a plug-in-based video communication platform system that can provide, and a video communication providing method using the same.

In addition, one technical aspect of the present invention is a plug-in-based video communication that can conveniently perform video communication of contents such as education, games, healthcare, etc. to anyone, regardless of age or gender, by applying various solutions in a plug-in method. It is to provide a platform system and a method for providing video communication using the same.

The above objects and various advantages of the present invention will become more apparent from preferred embodiments of the present invention by those skilled in the art.

One technical aspect of the present invention proposes a plug-in-based video communication platform system. The plug-in-based video communication platform system is installed in a user terminal and interlocks with a user application that provides a language conversion service to a user through the user terminal and a plurality of user applications that are interlocked for video communication. A language conversion service between user applications is provided, and in order to provide the language conversion service, voice information provided by any one user application is obtained, the obtained voice information is converted into text information, and converted text information or converted text information is obtained. and a service server that provides at least one of sign language motion information corresponding to text information to another user application.

In an embodiment, the user application receives at least one plug-in solution using a language conversion service or gesture recognition from the service server, installs it in the user terminal, and uses the installed plug-in solution to obtain a language from the service server. A conversion service or a video communication service using motion recognition may be provided.

In an embodiment, the service server provides at least one plug-in solution using the language conversion service or the motion recognition to the user application, and is generated in the user terminal by interworking with a plug-in solution installed in the user application It is possible to provide the language conversion service or the motion recognition for the video.

In an embodiment, the service server converts voice information of a video provided by the first user application into text, generates sign language motion information corresponding to the converted text information, and the converted text information or the sign language A language translation providing unit providing language motion information to at least one second user application interworking with the first user application, and acquiring the first and second moving images provided by the first user application, the first moving image and the second moving image 2 It may include a motion recognition providing unit that recognizes a human motion for each of the moving images and calculates a matching rate based on the degree of similarity between the recognized motions.

In an embodiment, the service server stores at least one plug-in solution based on a language conversion service or gesture recognition, and provides the at least one plug-in solution to the user application according to a request of the user application and may further include a plug-in solution providing unit that works with a plug-in solution installed in the user application to work with a language conversion service or a plug-in solution based on gesture recognition.

In an embodiment, the plug-in solution converts voice information into text information in conjunction with the plurality of user applications, and generates and provides sign language motion information corresponding to the text information according to a setting according to a setting. Solution, based on the similarity of the motion between the two images by recognizing the motion of the trainer in the video provided by the solution, the trainer user application, and recognizing the motion of another user in the video provided in the video provided by the other user application that operates in conjunction with the trainer user application Recognizes the motion of a dancer in a healthcare plug-in solution that provides training video communication by calculating the matching rate with It may include at least one of entertainment plug-in solutions that provide dance video communication by calculating a matching rate based on the similarity.

In an embodiment, the language translation providing unit may include a translation module for checking a language set in the user application and translating the text information according to a set language, and recognizes voice information of a video provided by the user application and converts it into text It may include a STT (Speech to Text) conversion module and a TTM (Text to Motion) conversion module for generating sign language motion information corresponding to the text information.

In one embodiment, the language translation providing unit, based on a set language, a TTS (Text to Speech) conversion module that generates and provides machine voice information corresponding to the text information and a sign language in a video provided by the user application It may further include at least one of an MTT (Motion to Text) conversion module that recognizes a motion and generates and provides text information corresponding to the recognized sign language motion.

Another technical aspect of the present invention proposes a method for providing video communication. The video communication providing method is a video communication providing method performed in a service server that provides video communication in conjunction with a user application installed in a user terminal, and the plurality of users by interworking with a plurality of user applications interworking with each other for video communication The steps of providing a language conversion service between applications, providing the user application with the language conversion service or at least one plug-in solution using the motion recognition, and interworking with a plug-in solution installed in the user application, in the user terminal It may include providing the language conversion service or the motion recognition for the generated video.

In one embodiment, the step of providing the language conversion service between the plurality of user applications includes converting voice information of a video provided by the first user application into text, and receiving sign language motion information corresponding to the converted text information. It may include generating and providing the converted text information or the sign language motion information to at least one second user application interworking with the first user application.

In an embodiment, the step of providing the language conversion service or the motion recognition includes: obtaining a first video and a second video provided from a first user application; for each of the first video and the second video The method may include recognizing a human motion and providing motion recognition by calculating a matching rate based on a degree of similarity between each recognized motion.

In an embodiment, the plug-in solution converts voice information into text information in conjunction with the plurality of user applications, and generates and provides sign language motion information corresponding to the text information according to a setting according to a setting. Solution, based on the similarity of the motion between the two images by recognizing the motion of the trainer in the video provided by the solution, the trainer user application, and recognizing the motion of another user in the video provided in the video provided by the other user application that operates in conjunction with the trainer user application Recognizes the motion of a dancer in a healthcare plug-in solution that provides training video communication by calculating the matching rate with It may include at least one of entertainment plug-in solutions that provide dance video communication by calculating a matching rate based on the similarity.

The means for solving the above-described problems do not enumerate all the features of the present invention. Various means for solving the problems of the present invention may be understood in more detail with reference to specific embodiments in the following detailed description.

According to one embodiment of the present invention, there is an effect that can provide fast communication regardless of country or language for all users around the world between these and non-disabled people as well as socially vulnerable groups such as the hearing impaired and the elderly.

In addition, according to an embodiment of the present invention, by applying various solutions in a plug-in method, there is an effect that can conveniently perform video communication of contents such as education, games, health care, etc. to anyone, regardless of age or gender.

1 is a diagram illustrating the configuration of a plug-in-based video communication platform system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an embodiment of the user terminal shown in FIG. 1 .
FIG. 3 is a block diagram for explaining an embodiment of the user application shown in FIG. 2 .
4 is a diagram illustrating an exemplary computing operating environment of a service server according to an embodiment of the present invention.
5 is a block diagram for explaining an embodiment of a service server according to the present invention.
6 is a block diagram illustrating an embodiment of the language conversion providing unit shown in FIG. 5 .
FIG. 7 is a block diagram illustrating an embodiment of the gesture recognition providing unit shown in FIG. 5 .
8 is a diagram for explaining an example of gesture recognition according to an embodiment of the present invention.
9 to 11 are diagrams for explaining examples of plug-in solutions according to an embodiment of the present invention.
12 is a flowchart illustrating a plug-in-based video communication providing method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

However, the embodiment of the present invention may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. In addition, the embodiments of the present invention are provided in order to more completely explain the present invention to those of ordinary skill in the art.

That is, the above-described objects, features and advantages will be described later in detail with reference to the accompanying drawings, and accordingly, a person of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps.

In addition, various components and sub-components thereof are described below in order to describe the system according to the present invention. These components and sub-components thereof may be implemented in various forms, such as hardware, software, or a combination thereof. For example, each element may be implemented as an electronic configuration for performing a corresponding function, or may be software itself operable in an electronic system or implemented as a functional element of such software. Alternatively, it may be implemented with an electronic configuration and corresponding driving software.

The various techniques described herein may be implemented with hardware or software, or a combination of both, where appropriate. As used herein, terms such as "Unit", "Server" and "System" likewise refer to computer-related entities, i.e. hardware, a combination of hardware and software, software or It can be treated as equivalent to software at the time of execution. In addition, each function executed in the system of the present invention may be configured in units of modules, and may be recorded in one physical memory, or may be recorded while being dispersed between two or more memories and recording media.

Although various flowcharts are disclosed to describe the embodiments of the present invention, this is for convenience of description of each step, and each step is not necessarily performed according to the order of the flowchart. That is, each step in the flowchart may be performed simultaneously with each other, performed in an order according to the flowchart, or may be performed in an order opposite to the order in the flowchart.

Hereinafter, various embodiments of a plug-in-based video communication platform system and a plug-in-based video communication providing method using the same according to the present invention will be described.

1 is a diagram for explaining a plug-in-based video communication platform system according to an embodiment of the present invention.

Referring to FIG. 1 , a plug-in-based video communication platform system (hereinafter, referred to as a 'video communication platform system') includes a plurality of user terminals 100 and a service server 300 .

The user possesses the user terminal 100 , and the user may be provided with a video communication service through the user terminal 100 . For example, a user may receive a user application from the service server 300 , install it on the user terminal 100 , and execute the installed user application to receive a video communication service function.

The user may be provided with a language conversion service or a motion recognition-based video communication service by using the user terminal 100 in which the application is executed.

The service server 300 may provide a user application to a user to enable installation, and may provide a language conversion service between a plurality of user applications by interworking with a plurality of user applications that are interlocked for video communication.

The service server 300 obtains voice information provided by any one user application in order to provide a language conversion service, converts the acquired voice information into text information, and corresponds to the converted text information or the converted text information. At least one of sign language motion information may be provided to another user application.

The service server 300 may provide a service using a plug-in solution. That is, the service server 300 provides at least one plug-in solution using a language conversion service or gesture recognition to a user application, and interworks with a plug-in solution installed in the user application for a video generated in the user terminal 100. It may provide a language conversion service or gesture recognition.

Hereinafter, each component of the plug-in-based video communication platform system according to the present invention will be described in more detail with reference to FIGS. 2 to 10 .

FIG. 2 is a block diagram illustrating an embodiment of the user terminal shown in FIG. 1 .

Referring to FIG. 2 , the user terminal 100 is a portable terminal carried by a user, and various types of terminals are applicable. For example, the user terminal 100 is a smart phone (smartphone), a smart pad (smartpad), a tablet PC (Tablet PC), PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS(Personal Handyphone System), PDA(Personal Digital Assistant), IMT(International Mobile Telecommunication)-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(W-Code Division Multiple Access), Wibro(Wireless Broadband Internet) ) includes a portable computing device that is portable by a user, such as a terminal, and capable of wireless communication.

The user terminal 100 may include a communication unit 110 , an input/output unit 120 , a memory 130 , a control unit 140 , and a user application 150 .

The communication unit 110 may connect to a wireless network and establish a communication channel with the service server 300 . The input/output unit 120 provides an input means and an output means to the user. For example, in the case of a smart phone, the input/output unit may be a touch screen.

The memory 220 stores data and programs necessary for the user terminal 100 to operate. For example, the memory unit 220 may include a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD). memory, etc.), magnetic memory, magnetic disk, optical disk, RAM (Random Access Memory: RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory: ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), It may include a storage medium such as a programmable read-only memory (PROM). In addition, the memory 220 may include a form of web storage that performs a storage function on the Internet.

The controller 140 controls other components of the user terminal 100 . For example, the controller 140 may be a processing unit of the user terminal 100 .

The user application 150 is a software application running in the user terminal 100 . That is, it may be a software function driven by the memory 330 and the controller 340 .

The user application 150 may provide a video communication service to the user in conjunction with the service server 300 . That is, the user application 150 may provide a video communication service by interworking with the controller 140 to obtain at least some rights to the communication unit 110 and the input/output unit 120 .

The user application 150 may provide a language conversion service to the user through the user terminal.

The user application 150 receives at least one plug-in solution using a language conversion service or gesture recognition from the service server 300 and installs it in the user terminal, and uses the installed plug-in solution to obtain a language from the service server 300 A conversion service or a video communication service using motion recognition may be provided.

The user application 150 can be implemented in various program types. For example, it may be implemented as software running on a computing device, or it may be implemented based on a web browser interworking with a web service provided by a service server.

FIG. 3 is a block diagram for explaining an embodiment of the user application shown in FIG. 2 .

Referring to FIG. 3 , the user application 150 may include an application platform 151 and at least one plug-in solution 152 to 154 .

The application platform 151 provides a list of a plurality of plug-in solutions provided by the service server 300 to the user, and provides customized installation of the plug-in solution according to the user's selection.

That is, the application platform 151 functions as a main platform, and a user can easily install and delete a plug-in solution by using the application platform 151 . Accordingly, users can easily install various plug-in solutions to suit their needs, enabling customization of user applications.

As an example of the plug-in solution, it includes a first plug-in that provides a language conversion service or a second plug-in solution that provides gesture recognition, and when the plug-in solution is installed, the plug-in solution works in conjunction with the service server It may provide the functionality of the in-solution, such as a language conversion service or gesture recognition.

Since the user application 150 operates in conjunction with the service server 300 , this can be more easily understood with reference to the following description of the service server 300 .

Hereinafter, the service server will be described in more detail with reference to FIGS. 4 to 11 .

4 is a diagram illustrating an exemplary computing operating environment of a service server according to an embodiment of the present invention.

4 is intended to provide a general and simplified description of a suitable computing environment in which embodiments of the service server 300 may be implemented. Referring to FIG. 4 , a computing device is illustrated as an example of the service server 300 . do.

The computing device may include at least a processing unit 303 and a system memory 301 .

A computing device may include a plurality of processing units that cooperate in executing a program. Depending on the exact configuration and type of computing device, system memory 301 may be volatile (eg, RAM), non-volatile (eg, ROM, flash memory, etc.), or a combination thereof. The system memory 301 includes a suitable operating system 302 for controlling the operation of the platform, such as the WINDOWS operating system from Microsoft Corporation. System memory 301 may include one or more software applications, such as program modules, applications, and the like.

The computing device may include additional data storage devices 304 such as magnetic disks, optical disks, or tape. Such additional storage may be removable storage and/or fixed storage. Computer readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for stored information such as computer readable instructions, data structures, program modules, or other data. The system memory 301 and the storage 304 are only examples of computer-readable storage media. A computer readable storage medium may include RAM (RAM), ROM (ROM), EEPROM, flash memory or other memory technology, CD-ROM, DVD or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage device, or any desired It may include, but is not limited to, any other medium that stores information and that can be accessed by computing device 300 .

input devices 305 of computing devices, such as keyboards, mice, pens, voice input devices, touch input devices, and comparable input devices. Output devices 306 may include, for example, displays, speakers, printers, and other types of output devices. Since these devices are widely known in the art, detailed descriptions thereof will be omitted.

Computing devices may include communication devices 307 that allow the device to communicate with other devices over networks, such as wired and wireless networks, satellite links, cellular links, local area networks, and comparable mechanisms, such as in distributed computing environments, for example. there is. Communication device 307 is one example of a communication medium, which may include computer readable instructions, data structures, program modules, or other data therein. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

The service server 300 may be described as a functional configuration (eg, a software module or a set of instructions) implemented in such a computing environment. In this regard, various embodiments of the service server will be described in more detail with reference to FIGS. 5 to 11 .

5 is a block diagram for explaining an embodiment of a service server according to the present invention.

Referring to FIG. 5 , the service server 300 may include a main platform 310 , a language translation providing unit 320 , a gesture recognition providing unit 330 , and a plug-in solution providing unit 340 .

The main platform 310 interworks with a user application to provide a video communication service. For example, the main platform 310 provides a user application, user subscription processing, user login authentication, communication session connection with a user application, acquisition of information provided by the user application, and other components of the service server 300 . It is possible to provide a function for interworking with the user application, such as providing the element to the element and providing the processed data thereto to the user application.

The language translation providing unit 320 may provide a language translation function for data provided by a user application.

The language translation providing unit 320 may provide a translation function between user applications according to a set country or language.

The language translation providing unit 320 may provide a sign language function. For example, the language translation providing unit 320 may convert voice information of a video provided by the first user application into text and generate sign language motion information corresponding to the converted text information. The language translation providing unit 320 may provide the converted text information or sign language motion information to at least one second user application interworking with the first user application.

The gesture recognition providing unit 330 may provide a gesture recognition function to a moving picture driven by a user application, for example, a moving image captured by the user application, a moving image, or the like.

The motion recognition providing unit 330 acquires the first video and the second video provided by the first user application, recognizes a human motion with respect to each of the first video and the second video, and determines the similarity between the recognized motions. Based on this, the matching rate can be calculated and provided.

The plug-in solution providing unit 340 may provide a plug-in solution to a user application.

The plug-in solution providing unit 340 stores at least one plug-in solution based on a language conversion service or gesture recognition. The plug-in solution providing unit 340 may provide at least one plug-in solution to the user application according to the request of the user application. The plug-in solution providing unit 340 may operate in conjunction with a language conversion service or a plug-in solution based on gesture recognition in conjunction with a plug-in solution installed in a user application.

The plug-in solution providing unit 340 may include individual plug-in interlocking units 351 to 35n for interworking with each plug-in solution.

For example, as a plug-in solution, there may be an Edutech plug-in solution, a healthcare plug-in solution, an entertainment plug-in solution, etc. to be described later. The plug-in interlocking units 351 to 35n may be configured such that the interlocking unit is configured for each plug-in solution, respectively. That is, the Edutech plug-in interworking unit interlocks between user terminals driving the Edutech plug-in solution, the healthcare plug-in interlocks interlocks between user terminals driving the healthcare plug-in solution, and the entertainment plug-in interlocks the entertainment plug-in. Interworking between user terminals running the solution can be provided.

The plug-in interworking units 351 to 35n may request a language translation or gesture recognition from the language translation providing unit 320 or the gesture recognition providing unit 330, and serve as a channel for providing the processing result to the user application. can

The plug-in interworking units 351 to 35n may provide other information required by the plug-in solution, for example, streaming information of a streaming video, information such as an operation name provided by a trainer.

These plug-in solutions do not belong to the basic installation configuration of the user application, but are program modules that can be installed upon user's request, and are various solutions developed based on the language conversion service or motion recognition provided by the service server. Such a plug-in solution can be developed and registered by a separate third-party development entity rather than the operating entity of the service server 300, and users can receive various solutions through this plug-in solution, and Third-party development entities may generate revenue through the sale or operation of the solution.

9 to 11 are diagrams for explaining examples of plug-in solutions according to an embodiment of the present invention. Examples of plug-in solutions will be described with reference to FIGS. 9 to 11 .

9 shows an example of an Edutech plug-in solution for providing subtitle, translation, and sign language functions.

The Edutech plug-in solution shown in FIG. 9 can provide real-time captions by converting voice information into text information in conjunction with a plurality of user applications. In addition, according to the sign language setting, sign language motion information (sign language) corresponding to the text information may be generated and provided. Edutech plug-in solution can provide machine translation function through language translation setting.

10 shows an example of a healthcare plug-in solution that provides healthcare between a trainer and a user.

The healthcare plug-in solution shown in FIG. 10 recognizes the motion of a trainer in a video provided by the trainer user application, and recognizes the motion of another user in a video provided by another user application that operates in conjunction with the trainer user application Thus, it is possible to provide training video communication by calculating a matching rate based on the similarity of motions between the two images.

The healthcare plug-in solution may provide a motion name provided by a trainer, and may calculate and provide a degree of posture matching, a timing matching, and a balance matching between the trainer's motion and the user's motion.

11 shows an example of an entertainment plug-in solution based on video streaming.

The entertainment plug-in solution recognizes the motion of a dancer in the reference video streaming (YouTube, etc.) to provide dance video communication.

The entertainment plug-in solution provides streaming information (eg, singer, song name, etc.) of a streaming video, and may calculate and provide the degree of matching posture, timing, and balance between the dancer's motion and the user's motion.

Hereinafter, various embodiments of the language translation and gesture recognition functions of the service server 300 will be described with reference to FIGS. 6 to 8 .

6 is a block diagram illustrating an embodiment of the language conversion providing unit shown in FIG. 5 .

Referring to FIG. 6 , the language conversion providing unit 320 includes a speech to text (STT) conversion module 510, a text to motion (TTM) conversion module 520, a translation module 530, and a text to speech (TTS) conversion module. It may include at least one of a conversion module 540 and a motion to text (MTT) conversion module 550 .

The translation module 530 may provide a translation function between different languages. The translation module 530 may check a language set in the user application and translate text information according to the set language. To this end, the translation module 530 may include a text-based machine translation function.

The speech to text (STT) conversion module 510 may convert voice information into text information.

The speech to text (STT) conversion module 510 may recognize voice information of a video provided by a user application and convert it into text. The converted text information may be used as a subtitle in a plug-in solution or may be provided to the translation module 530 and used for a translation function.

The text to motion (TTM) conversion module 520 may generate a sign language based on the text.

The text to motion (TTM) conversion module 520 may generate sign language motion information corresponding to the text information. To this end, the text to motion (TTM) conversion module may include a database for sign language motion and output sign language motion information corresponding to the input text. For example, the sign language motion information may be formed by continuously providing a sign language video corresponding to each word of the text.

The text to speech (TTS) conversion module 540 may convert text into speech.

The Text to Speech (TTS) conversion module 540 may generate and provide machine voice information corresponding to text information based on a set language (or country). To this end, the text to speech (TTS) conversion module 540 may include a machine voice information database corresponding to each text for each language.

The Motion to Text (MTT) conversion module 550 may convert sign language into text.

The Motion to Text (MTT) conversion module 550 may recognize a sign language motion in a video provided by a user application, and may generate and provide text information corresponding to the recognized sign language motion. To this end, the Motion to Text (MTT) conversion module 550 may include an image recognition function for recognizing a hand object in an image and outputting text in response to the recognized shape of the hand object.

FIG. 7 is a block diagram illustrating an embodiment of the gesture recognition providing unit shown in FIG. 5 .

Referring to FIG. 7 , the motion recognition providing unit 330 may include an object recognition module 610 , a motion recognition module 620 , and a motion similarity determining module 630 .

The object recognition module 610 may recognize a human object in a video provided by a user application. That is, the object recognition module 610 may recognize an object corresponding to the appearance of a person in the video, where the appearance of the person is based on the whole body of the person. To this end, the object recognition module 610 includes an image recognition function for a moving picture, however, since the image object recognition technology can be applied in various ways, it is not limited to a specific technology (algorithm) here.

The gesture recognition module 620 may extract the posture of the human object recognized by the object recognition module 610 . The posture of the human object extracted by the motion recognition module 620 may be displayed as a skeletal structure as in the example shown in FIG. 8 .

In one embodiment, the motion recognition module 620 may include a core recognition module 621 and a posture extraction module 622 .

The core recognition module 621 may recognize the head and body from the human object recognized by the object recognition module 610 and recognize it as a core. That is, the core recognition module 621 may recognize the outlines of the head and torso in the human object, and generate the head and torso skeletons in response to the recognized outlines of the head and torso. The trunk skeleton may include a shoulder skeleton, a vertebral skeleton, and a pelvic skeleton, as illustrated in FIG. 8 .

The posture extraction module 622 may identify limbs in a human object based on the head and torso bones recognized by the core recognition module 621 and generate arm and leg skeletons for the limbs. As illustrated in FIG. 8 , the arm bones may be connected one by one to both ends of the shoulder bone, and the leg bones may be connected to both ends of the pelvic bone one by one.

As described above, since the head and torso are located in the central part of the body and their appearance can be differentiated from other body parts, the motion recognition module 620 preferentially identifies the skeleton of the head and torso through the core recognition module 621 . and to generate the skeleton of the arm and leg through the posture extraction module 622 based on it. Through this, unlike the method of analyzing all motions at once, by recognizing the limbs after core-priority recognition, it is possible to quickly process human motion recognition and efficiently recognize with a small amount of resources.

The motion similarity determination module 630 may determine the motion similarity rate by comparing the skeleton structures generated by the motion recognition module 620 with each other.

The motion similarity determination module 630 may calculate a posture matching degree of motion by comparing the first skeletal structure of the first moving image and the second skeletal structure of the second moving image to be compared with each other. That is, the motion similarity determining module 630 may determine the similarity by calculating the inclination of the core skeleton and the angle between each skeleton in the skeleton structure. That is, the inclination of each individual of the core skeleton, that is, the head skeleton, the shoulder bone, the vertebral skeleton, and the pelvic skeleton is calculated, and the angle between each skeleton - for example, the angle of the head skeleton and the vertebral skeleton, the upper arm skeleton and the lower arm skeleton Angle, etc. can be calculated respectively, and the degree of similarity can be calculated by comparing them with each other. The calculated similarity may be used as the posture matching degree and the balance matching degree.

In an embodiment, the motion similarity determining module 630 may further calculate the timing coincidence. That is, the motion similarity determination module 630 may calculate the degree of coincidence of timings at which such similar postures occur in two videos that satisfy similar postures—that is, postures having a coincidence rate greater than or equal to a certain level.

In the above, various embodiments of a plug-in-based video communication platform system have been described with reference to FIGS. 1 to 11 .

Hereinafter, a plug-in-based video communication providing method according to an embodiment of the present invention will be described.

Since the plug-in-based video communication providing method to be described below is performed based on the plug-in-based video communication platform system described with reference to FIGS. 1 to 11, more Easy to understand.

12 is a flowchart illustrating a plug-in-based video communication providing method according to an embodiment of the present invention.

Referring to FIG. 12 , the service server 300 may provide at least one plug-in solution using a language conversion service or gesture recognition to a user application ( S1110 ).

The service server 300 may provide a language conversion service or motion recognition for a video generated in the user terminal in conjunction with a plug-in solution installed in the user application (S1120).

In one embodiment for step S1120, the service server 300, converting the voice information of the video provided by the first user application into text, generating sign language motion information corresponding to the converted text information, and The step of providing the converted text information or sign language motion information to at least one second user application interworking with the first user application may be performed.

In an embodiment of step S1120, the service server 300 recognizes a human motion with respect to each of the steps of acquiring the first video and the second video provided from the first user application, and the first video and the second video, respectively. and calculating a matching rate based on the degree of similarity between each recognized motion and providing motion recognition may be performed.

Here, the plug-in solution converts voice information into text information by interworking with a plurality of user applications, and generates and provides sign language motion information corresponding to the text information according to the setting. Edutech plug-in solution, trainer user application By recognizing the motion of the trainer in the video provided in A healthcare plug-in solution that provides training video communication and a reference video streaming recognize a dancer's motion, recognize another user's motion in a video provided by another user application, and match the rate based on the similarity of motion between the two videos It may include at least one of the entertainment plug-in solutions that provide dance video communication by calculating .

The present invention described above is not limited by the above-described embodiments and the accompanying drawings, but is limited by the claims described below, and the configuration of the present invention may vary within the scope without departing from the technical spirit of the present invention. Those of ordinary skill in the art to which the present invention pertains can easily recognize that it can be changed and modified.

100: user terminal
300 : service server
110: communication unit 120: input/output unit
130: memory 140: control unit
150: user application
301: system memory 302: operating system
303 processing unit 304 storage device
305: input device 306: output device
307: communication device
310: main platform 320: language translation provider
330: motion recognition providing unit 340: plug-in solution providing unit
510: STT conversion module 520: TTM conversion module
530: translation module 540: TTS conversion module
550: MTT conversion module
610: object recognition module 620: motion recognition module
630: motion similarity rate determination module

Claims (12)

a user application installed in a user terminal and providing a language conversion service to a user through the user terminal; and
Provides a language conversion service between the plurality of user applications by interworking with a plurality of user applications that are interlocked for video communication, and obtains and obtains voice information provided by any one user application to provide the language conversion service A service server that converts the converted voice information into text information, and provides at least one of the converted text information or sign language motion information corresponding to the converted text information to another user application,
The user application is
A first plug-in providing a language conversion service or a second plug-in solution providing gesture recognition is installed in the user terminal according to a user's selection, and the installed first or second plug-in solution is used from the service server To be provided with a language conversion service or a video communication service using gesture recognition,
The service server,
The first plug-in for providing a language conversion service to the user application or the second plug-in solution for providing motion recognition is provided, and the video generated by the user terminal is linked with the plug-in solution installed in the user application. to provide the language conversion service or the motion recognition for
Motion recognition for obtaining a first video and a second video provided from a first user application, recognizing a human motion with respect to each of the first video and the second video, and calculating a matching rate based on the similarity between the recognized motions including a provider;
The plug-in solution is
an edutech plug-in solution that converts voice information into text information in conjunction with the plurality of user applications, and generates and provides sign language motion information corresponding to the text information according to settings;
Recognizes the motion of the trainer in the video provided by the trainer user application, recognizes the motion of another user in the video provided in the video provided by another user application that operates in conjunction with the trainer user application, and the matching rate based on the similarity of the motion between the two images A healthcare plug-in solution that provides training video communication by calculating , and calculates and provides the name of the trainer's motion, the degree of posture matching between the trainer's motion and the user's motion, the degree of timing matching, and the degree of balance matching; and
Recognizes the motion of a dancer in video streaming, recognizes the motion of another user in a video provided by another user application, calculates a match rate based on the similarity of motion between the two videos, and provides dance video communication, see above It includes an entertainment plug-in solution that calculates and provides streaming information of video streaming, posture matching between the dancer's motion and the user's motion, timing matching, and balance matching information,
The user application is
An application platform that provides a list of a plurality of plug-in solutions provided by the service server to the user, and provides customized installation of plug-in solutions according to the user's selection
A plug-in-based video communication platform system that includes.
delete delete According to claim 1, wherein the service server,
Converts voice information of a video provided by a first user application into text, generates sign language motion information corresponding to the converted text information, and combines the converted text information or sign language motion information with the first user application Language translation providing unit that provides at least one second user application that is interworking
A plug-in-based video communication platform system comprising a.
The method of claim 4, wherein the service server,
Storing at least one plug-in solution based on a language conversion service or motion recognition, providing the at least one plug-in solution to the user application according to a request of the user application, and a plug-in solution installed in the user application a plug-in solution provider that works with a language conversion service or a plug-in solution based on gesture recognition in conjunction with;
A plug-in-based video communication platform system, characterized in that it further comprises.
delete The method of claim 4, wherein the language translation providing unit,
a translation module for checking a language set in the user application and translating the text information according to the set language;
STT (Speech to Text) conversion module for recognizing the voice information of the video provided by the user application and converting it into text; and
a text to motion (TTM) conversion module for generating sign language motion information corresponding to the text information;
A plug-in-based video communication platform system comprising a.
The method of claim 7, wherein the language translation providing unit,
a Text to Speech (TTS) conversion module that generates and provides machine voice information corresponding to the text information based on a set language; and
an MTT (Motion to Text) conversion module for recognizing sign language motion in the video provided by the user application, and generating and providing text information corresponding to the recognized sign language motion;
further comprising at least one of
A plug-in-based video communication platform system featuring a.
A method for providing video communication performed in a service server that provides video communication in conjunction with a user application installed in a user terminal, comprising:
providing the user application with at least one plug-in solution using a language conversion service or gesture recognition; and
Comprising the step of providing the language conversion service or the motion recognition for the video generated in the user terminal in conjunction with the plug-in solution installed in the user application,
The step of providing the language conversion service or motion recognition comprises:
acquiring a first video and a second video provided by a first user application;
recognizing a human motion with respect to each of the first video and the second video; and
Comprising the step of providing motion recognition by calculating a matching rate based on the degree of similarity between each recognized motion,
The plug-in solution is
an edutech plug-in solution for converting voice information into text information in conjunction with the user application, and generating and providing sign language motion information corresponding to the text information according to a setting;
Recognizes the movement of the trainer in a video provided by the trainer user application, recognizes the movement of another user in a video provided by another user application that operates in conjunction with the trainer user application, and recognizes the match rate based on the similarity of the motion between the two images A healthcare plug-in solution that provides training video communication by calculating , and calculates and provides the name of the trainer's motion, the degree of posture matching between the trainer's motion and the user's motion, the degree of timing matching, and the degree of balance matching; and
Recognizes the motion of a dancer in the reference video streaming, recognizes the motion of another user in the streaming video provided by another user application, calculates a match rate based on the similarity of the motion between the two images to provide dance video communication, Includes an entertainment plug-in solution that calculates and provides information on streaming information of reference video streaming, posture matching between the dancer's motion and the user's motion, timing matching, and balance matching information,
The user application is
An application platform that provides a list of a plurality of plug-in solutions provided by the service server to the user, and provides customized installation of plug-in solutions according to the user's selection
A plug-in-based video communication providing method comprising a.
10. The method of claim 9, wherein the step of providing the language conversion service or gesture recognition comprises:
converting audio information of a video provided by a first user application into text;
generating sign language motion information corresponding to the converted text information; and
providing the converted text information or the sign language motion information to at least one second user application interworking with the first user application;
A plug-in-based video communication providing method comprising a. delete delete

Images