KR102258991B1 - Sign-language service providing system - Google Patents

Abstract

The present invention relates to a sign language service providing system capable of providing sign language services by using augmented reality (AR) and application programs. According to the present invention, the sign language service providing system comprises: a wearable information display device expressing sign language broadcast including at least one of a sign language image using AR and a caption image; an application program module interlocked with the wearable information display device and outputting the sign language broadcast to the wearable information display device in real-time; and a sign language service providing server connected to the application program module, storing sign language broadcast prepared in advance corresponding to a broadcast program and a sign language broadcast schedule in which the previously produced sign language broadcast and the broadcast program are matched, searching for the stored sign language broadcast schedule when a sign language broadcast of a broadcast program is requested from the application program module, providing the corresponding sign language broadcast to the application program module when the sign language broadcast of the broadcast program exists, performing sync linkage between the broadcast program and the sign language broadcast, storing a sign language broadcast prepared in advance corresponding to a movie and a movie viewing schedule in which the sign language broadcast and the movie are matched, and providing the corresponding sign language broadcast by using the stored movie viewing schedule when the sign language broadcast corresponding to a movie is requested from the application program module. The application module is installed in at least one of a user terminal or wearable information display device using a sign language service.

Description

Sign language service provision system {SIGN-LANGUAGE SERVICE PROVIDING SYSTEM}

The present invention relates to a sign language service providing system, and more particularly, to a sign language service providing system capable of providing a sign language service using augmented reality and an application program.

In general, sign language is the official language of sign language mainly used for the hearing impaired (deaf and deaf). In Korea, a news program first started sign language broadcasting in 1993, and the official service of smart sign language broadcasting started in 2019, and in 2020 Sign language interpretation has been provided for all main news programs on terrestrial broadcasting since September.

The smart sign language broadcasting service, launched in 2019, can autonomously adjust the size of sign language and the position of closed captions, and it is transmitted through the broadcasting network and the Internet network and displayed on the TV screen provided in set-top boxes such as homes, government offices, workplaces, and schools. It is a service that can provide a simultaneous display method.

In this smart sign language broadcasting service, the existing sign language broadcasting service is small in size and fixed in the lower right corner, so it not only covers a part of the broadcasting image but also makes it difficult to see the sign language interpreter's expression or hand gestures. The position can be freely adjusted, the size can also be adjusted, and it can be turned off if you do not want to see it.

The smart sign language broadcasting service refers to a service in which a broadcasting image and a sign language broadcasting image are transmitted to the broadcasting network and the Internet network, respectively, and the receiver installed in the home synthesizes them and displays them on the TV screen at the same time.

However, since such smart sign language broadcasting service is currently provided by a designated operator, it is necessary to replace the receiver with a receiver capable of smart sign language broadcasting service or to change the platform if the user is subscribed to a platform that does not provide the service, and only receives terrestrial waves that are not subscribed to the platform directly. In the case of hearing impaired people, it is difficult to commercialize it because they have to subscribe to a platform that provides services for a fee.

In addition, currently, sign language broadcasting is mainly provided on news or current programs, but it is aimed at programs that do not have any problems even if the screen is slightly obscured by sign language images. In other words, hearing impaired people rarely have the opportunity to enjoy dramas or entertainment.

Accordingly, there is a demand for a sign language service that can selectively receive sign language broadcasting without replacing devices such as a separate platform or receiver, and does not restrict universal broadcasting access rights of the hearing impaired.

Patent Document 1. KR 10-2013-0067639 (Method for providing sign language broadcasting service and system therefor)

Accordingly, the present invention has been devised to solve the above problems, and an object of the present invention is to provide a sign language service by providing a wearable information display device and an application program for a hearing impaired person when watching a broadcast, thereby providing a sign language service. The purpose of the present invention is to provide a sign language service providing system in which sign language services can be provided without a separate subscription to the service or change of devices.

In addition, an object of the present invention is to provide a separate wearable information display device and application program to superimpose sign language broadcasting on a TV screen in augmented reality, so that not only can sign language broadcasting be easily recognized, but also sign language interpretation is unnecessary An object of the present invention is to provide a sign language service providing system that can solve the inconvenience of simultaneous viewing with viewers.

In addition, it is an object of the present invention to provide a real-time sign language translation service as well as pre-produced sign language broadcasting by providing a separate wearable information display device and application program, so that selective sign language broadcasting of various broadcasting programs is possible, and when watching a movie The goal is to provide a sign language service provision system that can provide a sign language service even in the world and can satisfy the universal broadcasting access rights of the hearing impaired.

First, to summarize the features of the present invention, the system according to one aspect of the present invention for achieving the above object is wearable information expressing sign language broadcasting including at least one of a sign language image and a caption image using augmented reality. Device; an application program module that is interlocked with the wearable information display device and outputs the sign language broadcast to the wearable information display device in real time; and a sign language broadcast, which is connected to the application program module, and includes a pre-produced sign language broadcast corresponding to the broadcast program, and programming information that matches the pre-produced sign language broadcast and the broadcast program, respectively, and stores the application program When a sign language broadcast of a broadcast program is requested from the module, the stored sign language broadcast schedule is searched and, if there is a sign language broadcast of the broadcast program, the corresponding sign language broadcast is provided to the application program module, but the broadcast program and the sign language broadcast are synchronized and, respectively, store a sign language broadcast produced in advance corresponding to the movie and a movie viewing schedule matching the sign language broadcasting and the movie, and when a sign language broadcasting corresponding to the movie is requested from the application program module, the stored movie viewing schedule is stored A sign language service providing server that provides a corresponding sign language broadcasting using a sign language service, wherein the application program module is installed in at least one of a user terminal using a sign language service or the wearable information display device. system can be provided.

The wearable information display device includes: a spectacle body made of a spectacle structure; a voice acquisition unit provided in the glasses body unit, receiving an externally generated voice and outputting a voice signal; an image acquisition unit provided in the glasses body portion and outputting an image signal by photographing the front thereof; a communication unit provided in the glasses body unit and performing data transmission/reception; a display unit for expressing sign language broadcasting in augmented reality on a lens provided on the front of the glasses body; and a switch unit for controlling the on/off of power, wherein the application program module is provided in the wearable information display device, and when a switch on signal of the switch unit is input, the pre-made sign language broadcast is reinforced. A service mode for executing at least one of a viewing mode including a broadcast viewing mode and a movie viewing mode and a service mode including a translation mode for generating and providing real-time sign language broadcasting when there is no pre-produced sign language broadcasting control unit; When the broadcast viewing mode is executed, the current broadcast channel is recognized by analyzing the motion of the currently watched broadcast screen from the video signal captured through the image acquisition unit, and a pre-made sign language broadcast matching the recognized current broadcast channel information is performed. A sign language broadcast requested by the sign language service providing server is controlled to be displayed in augmented reality on the display unit, and when the movie viewing mode is executed, when movie selection information is input, a sign language image corresponding to the movie is retrieved. and, when the movie selection information is not input, screen tracking through the image acquisition unit is performed to recognize a currently playing movie, and a sign language broadcast matching the recognized currently playing movie is searched for in augmented reality on the display unit. a viewing mode execution unit that controls to be displayed; When the translation mode is executed, a voice is recognized from the voice signal output from the voice acquisition unit, a sign language matching the recognized voice is searched for, and a sign language image is generated by connecting the searched sign language, and matching with the recognized voice A caption image is generated by searching for a word to be used and a sentence is constructed through a combination of the searched words, a sign language broadcast including at least one of the generated sign language image and the caption image is generated, and the generated sign language broadcast is augmented on the display unit Translation mode execution unit that controls to be expressed in reality; and a sign language broadcasting setting unit for adjusting at least one of a position, a size, and a transparency of a sign language broadcast displayed on the display unit.

The wearable information display device includes: a spectacle body made of a spectacle structure; a voice acquisition unit provided in the glasses body unit, receiving an externally generated voice and outputting a voice signal; an image acquisition unit provided in the glasses body portion and outputting an image signal by photographing the front thereof; a communication unit provided in the glasses body unit and performing data transmission/reception; a display unit for expressing sign language broadcasting in augmented reality on a lens provided on the front of the glasses body; and a switch unit for controlling on/off of the power source; at least, the application program module is provided in the user terminal, and provides a pre-made sign language broadcast in augmented reality, including a broadcast viewing mode and a movie viewing mode a service mode control unit executing at least one of a viewing mode and a service mode including a translation mode for generating and providing real-time sign language broadcasting when there is no pre-produced sign language broadcasting; When the broadcast viewing mode is executed, when a broadcast channel capable of sign language service is selected through the user interface screen of the user terminal, a pre-produced sign language broadcast of the selected broadcast channel is requested to the sign language service providing server, and the broadcast channel is selected In the absence of this, the current broadcasting channel is recognized by analyzing the movement of the currently viewed broadcasting screen from the image signal captured by the image acquisition unit, and a pre-fabricated sign language broadcast of the recognized current broadcasting channel is requested to the sign language service providing server. and control so that the received sign language broadcast is displayed in augmented reality on the display unit, and when the movie viewing mode is executed, when movie selection information is input, a sign language image corresponding to the movie is searched, and the movie selection information When is not input, screen tracking is performed through the image acquisition unit to recognize a currently playing movie, and a sign language broadcast matching the recognized currently screened movie is searched for and controlled to be displayed in augmented reality on the display unit. executive department; When the translation mode is selected, a voice is recognized from the voice signal output from the voice acquisition unit, a sign language matching the recognized voice is searched for, and a sign language image is generated by connecting the searched sign language, and matching with the recognized voice A caption image is generated by searching for a word to be used and a sentence is constructed through a combination of the searched words, a sign language broadcast including at least one of the generated sign language image and the caption image is generated, and the generated sign language broadcast is augmented on the display unit Translation mode execution unit that controls to be expressed in reality; a sign language broadcast setting unit for adjusting at least one of a position, size, and transparency of a sign language broadcast displayed on the display unit; and a communication unit for transmitting and receiving sign language broadcasting.

According to the system according to the present invention, by providing a sign language service only to the hearing impaired through a separate implementation device and an application program module, even when viewing simultaneously with general viewers who do not need sign language interpretation, it does not cause inconvenience to general viewers at all, It can increase the concentration of the audience.

In addition, according to the system according to the present invention, a sign language image is provided through a separate implementation device and an application program module, or a sign language image is generated and provided through real-time sign language translation, thereby guaranteeing universal broadcasting access for the hearing impaired to various broadcasting programs. In addition to being able to do this, it is possible to watch not only sign language movies but also general movies, expand video content for the hearing impaired, and furthermore, it has the effect of being used as a medium for cultural sharing in various fields.

The accompanying drawings, which are included as part of the detailed description to help the understanding of the present invention, provide embodiments for the present invention and explain the technical spirit of the present invention together with the detailed description.
1 is a diagram showing the overall configuration of a sign language service providing system according to an embodiment of the present invention.
FIG. 2 is a diagram showing a schematic configuration of the application program module shown in FIG. 1 .
3 is a diagram showing the overall configuration of a sign language service providing system according to another embodiment of the present invention.
FIG. 4 is a diagram showing a schematic configuration of the application program module shown in FIG. 3 .
5 is a diagram illustrating an example of use of a sign language service utilizing a sign language service providing system according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In this case, the same components in each drawing are indicated by the same reference numerals as possible. In addition, detailed descriptions of already known functions and/or configurations will be omitted. In the following, a portion necessary for understanding an operation according to various embodiments will be mainly described, and descriptions of elements that may obscure the subject matter of the description will be omitted. In addition, some elements of the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore, the contents described herein are not limited by the relative size or spacing of the components drawn in each drawing.

In describing the embodiments of the present invention, when it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout the present specification. The terms used in the detailed description are only for describing the embodiments of the present invention, and should not be limiting. Unless explicitly used otherwise, expressions in the singular form include the meaning of the plural form. In the present description, expressions such as "comprising" or "feature" are intended to refer to certain features, numbers, steps, actions, elements, some or combination thereof, and one or more It should not be construed to exclude the presence or possibility of other features, numbers, steps, actions, elements, any part or combination thereof.

In addition, terms such as first and second may be used to describe various components, but the components are not limited by the terms, and the terms are used to distinguish one component from other components. Is only used.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram showing the overall configuration of a sign language service providing system according to an embodiment of the present invention.

FIG. 2 is a diagram showing a schematic configuration of the application program module shown in FIG. 1 .

First, referring to FIG. 1, the sign language service providing system of the present invention includes an application program module 200 for executing sign language service provision, and performs sign language broadcasting including at least one image of a sign language image and a caption image. It includes a wearable information display device 100 that is displayed on the display unit in augmented reality, and a sign language service providing server 300 that provides a pre-made sign language broadcast.

The wearable information display apparatus 100 is a wearable smart device, and may be implemented in the form of a head-up display. For example, the wearable information display device 100 may have a structure of smart glasses in the form of a headset and may be implemented to implement augmented reality on a panel (lens) located in front of the eyes.

As an example, the wearable information display apparatus 100 may implement a glasses body having a basic structure of glasses, and the glasses body includes a front (front of glasses), an end piece (end of the front, and a temple that is a leg portion). part), rim, bridge, nose pad, tip (the part that touches the ear), temple (the entire leg of glasses), and hinge (the part that connects the front and temple).

That is, when the user wears the wearable information display device 100 and watches TV broadcast, the sign language broadcast is displayed on the display unit 130 corresponding to the front lens, so a separate sign language broadcast needs to be provided on the TV broadcast screen. there is no

The wearable information display device 100 and the sign service providing server 300 are connected through an Internet network using a network (wired Internet communication, wireless Internet communication such as WiFi, WiBro, wired and wireless networks such as mobile communication such as WCDMA, LTE) It can be implemented as much as possible.

In addition, the application program module 200 may be connected to the sign language service providing server 300 to receive a pre-made sign language broadcast, or may generate a sign language broadcast by itself by executing real-time sign language translation. The sign language broadcast is generated to include at least one image of a sign language image and a caption image, and may include only a sign language image, only a subtitle image, or both a sign language image and a subtitle image.

Specifically, referring to FIG. 2 , the wearable information display device 100 includes an application program module 200 , and a voice acquisition unit 110 and an image acquisition unit 120 in the glasses body having a structure of glasses. ), the display unit 130 and the communication unit 140 may be implemented to include at least.

The audio acquisition unit 110 is a device that collects analog audio signals from video objects such as TV broadcasts or movies and converts them into digital audio signals, and may use a microphone, but is not limited thereto. For example, input audio It may be desirable to use a directional microphone that improves the sensitivity of

The voice acquisition unit 110 is formed at the front of the eyeglass body portion and can be implemented to selectively receive only a sound coming from a specific direction, and can be used as a detachable device for easy replacement.

That is, the audio acquisition unit 110 receives the audio of a TV broadcast image or a movie image and converts it into an audio signal. The converted audio signal is used to generate a sign language image and a caption image when the real-time translation mode is executed.

The image acquisition unit 120 is a device that acquires an image by photographing a front view and capturing an image of a broadcast or a movie that is being viewed, and a camera module commonly used in electronic devices such as a smart phone may be used, but the present invention is limited thereto. It is not, and may be implemented to be detachable so that it can be replaced with a camera of another performance.

The image acquisition unit 120 is installed in the wearable information display device 100 so as to be able to photograph the front, acquires an image signal by shooting a TV broadcast image, and transfers the acquired image signal to the application program module 200 . output so that the application program module 200 is used to recognize the currently watched broadcast.

The display unit 130 may be implemented as a lens or an optical module for outputting sign language broadcasting including at least one of a caption image and a sign language image, and the sign language broadcasting is expressed in augmented reality.

The displayed sign language broadcast outputs sign language images linked in real time with the broadcast being watched or a video in which sign language images are connected. When sign language images are displayed, the sign language images are continuously output and connected, so it is displayed like a video performing natural sign language motions. can be In addition, even when the audio signal of the broadcast is automatically recognized and translated, the displayed sign language broadcast may be displayed as a moving picture in which sign language images corresponding to the audio signal are continuously connected.

The communication unit 140 performs an Internet connection to enable data transmission and reception, and enables mutual transmission and reception of information with other smart devices. The communication unit 140 may use communication technologies including various known technologies such as Wi-Fi, Bluetooth, and NFC.

And, the wearable information display device 100, although not shown in the drawing, a storage unit for storing a program for executing the application program module 200, a power supply unit for supplying power to each unit, and ON/OFF of the power A switch unit to control, a detachment detection unit for detecting whether the wearable information display device 100 is attached or detached, a sign language broadcasting setting unit for setting the position, size and transparency of the sign language broadcasting, an operation unit for performing operations of the sign language broadcasting setting unit, a power supply unit It may be configured to further include an input port for charging and downloading.

The storage unit (not shown) may be implemented as a built-in memory to store the application program module through downloading, the power supply unit (not shown) may be a rechargeable battery, and the switch unit and the operation unit (not shown) are pressure-type buttons can be implemented as

The detachable detection unit (not shown) detects the wearing and detachment of the wearable information display device 100, but detects this when the wearable information display device 100 is worn without the power ON operation of the switch unit, and for a predetermined time It is possible to control the operation of the switch unit so that the power is automatically turned on when the wear is maintained for a while. Conversely, the detachment detection unit (not shown) detects when the wearable information display device 100 is detached without the power OFF operation of the switch unit, and automatically powers on if the detachment is maintained for a certain period of time or re-wear is not detected. The operation of the switch unit can be controlled to be OFF.

On the other hand, the application program module 200 is interlocked with the wearable information display device 100 to display a pre-made sign language broadcast on the display unit 130 of the wearable information display device 100, or through real-time translation. The generated sign language broadcast is controlled to be displayed on the display unit 130 of the wearable information display device 100 .

To this end, the application program module 200 may include a service mode control unit 210 , a viewing mode execution unit 220 , and a translation mode execution unit 230 .

The service mode control unit 210 controls to execute at least one of the viewing mode and the translation mode when a switch on signal of the switch unit is input, but after executing the viewing mode first, if there is no pre-produced sign language broadcast, real-time sign language The service mode can be controlled to execute a translation mode that generates and provides a broadcast. Here, the viewing mode may include a broadcast viewing mode providing a sign language service when watching a broadcast and a movie viewing mode providing a sign language service when watching a movie.

This service mode control unit 210, by the user operating the operation unit (not shown) of the wearable information display device, the service mode is selected or the viewing mode is executed first, a search for a pre-made sign language broadcast is made, When there is no sign language broadcast produced, each execution unit can be controlled to execute the translation mode.

First, the viewing mode execution unit 220, when the broadcast viewing mode is executed, analyzes the movement of the currently viewed broadcasting screen from the image signal captured through the image acquisition unit 120 to recognize the current broadcasting channel, A pre-produced sign language broadcast matching the current broadcast channel is requested from the sign language service providing server 300 .

Then, the sign language service providing server 300 stores a sign language broadcast schedule prepared in advance corresponding to the broadcast program, and a sign language broadcast schedule including programming information matching the broadcast program and the pre-produced sign language broadcast, respectively. If there is a matching broadcast channel by inquiring the broadcast channel requested from the mode executor 220 in the sign language broadcast schedule, the matching broadcast channel and the matching sign language broadcast are searched, and the searched sign language broadcast is synchronized with the broadcast channel and provided.

In addition, the viewing mode execution unit 220, when the movie viewing mode is executed, when movie selection information is input, searches for a sign language image corresponding to the movie, and when the movie selection information is not input, the image acquisition unit 120 ) to recognize a currently playing movie by performing screen tracking, and request a sign language broadcast matching the recognized currently screened movie to the sign language service providing server 300 .

Then, the sign language service providing server 300 stores, respectively, a sign language broadcast prepared in advance corresponding to the movie, and a movie viewing schedule including the program information matching the sign language broadcast and the movie, and the viewing mode of the application program module. When a sign language broadcast corresponding to a movie is requested from the execution unit 220 , the stored movie viewing schedule may be inquired to find and provide the corresponding sign language broadcast. At this time, the sign language service providing server 300 may synchronize the sign language broadcasting and the movie screen by matching the movie start point analyzed in advance and the movie screen tracking using the image acquisition unit to match the automatically recognized movie start point. have.

Then, when the sign language broadcast synchronized with the requested broadcast channel is transmitted from the sign language service providing server 300 , the viewing mode execution unit 220 may stream the provided sign language broadcast to the display unit 130 in augmented reality. . That is, the sign language broadcast is expressed in augmented reality on the set area of the display unit 130 provided in the wearable information display device 100 .

In addition, when the sign language broadcast synchronized with the requested movie is transmitted from the sign language service providing server 300 , the viewing mode execution unit 220 may stream the provided sign language broadcast to the display unit 130 in augmented reality.

Meanwhile, the service mode control unit 210 may switch to the translation mode service when the sign language service providing server 300 fails to search for a sign language broadcast for the current broadcast. That is, the service mode control unit 210 may request a real-time sign language translation service from the translation mode execution unit 230 when a pre-produced sign language broadcast for the currently viewed broadcast is not searched.

The translation mode execution unit 230 includes a voice recognition unit 231 for recognizing a voice from a voice signal obtained through the voice acquisition unit 110, and a voice-sign language matching unit for searching for a sign language matching the recognized voice ( 232), a sign language image generating unit 234 for generating a sign language image by connecting sign words matching the recognized voice, a voice-word converting unit 235 for searching for a word matching the recognized voice, and A caption image generator 236 for generating a caption image by connecting words corresponding to audio, a sign language broadcast generator 237 for generating a sign language broadcast including at least one of the generated sign language image and the caption image; It includes a sign language broadcast playback unit 238 for streaming the sign language broadcast to the display unit 130 of the wearable information display device 100 in augmented reality.

Here, the sign language may be a three-dimensional sign language image or a moving image in which a plurality of three-dimensional sign language images are connected. The sign language image may be a moving image in which the recognized voice and the matching sign language are sequentially connected, and the subtitle image may be a moving image generated by connecting the recognized voice and matching words in a text format.

The voice recognition unit 231 receives a voice signal from the voice acquisition unit 110 , recognizes a voice from the input voice signal, and transmits it to the voice-sign language matching unit 232 .

The voice-sign language matching unit 232 includes a databased voice-sign language matching DB 233, sequentially compares the recognized voice with the voice stored in the voice-sign language matching DB 233, and compares the recognized voice with the recognized voice. If there is the same voice, a sign language corresponding to the same voice is called, and the called sign language is transmitted to the sign language image generating unit 234 .

The voice-sign language matching DB 233 stores sign language corresponding to each of a plurality of voices by matching them with a plurality of voices, and the matched sign language is a three-dimensional image for accurate expression of sign language or a plurality of three-dimensional images are connected It could be a video. Here, the three-dimensional image is a human-friendly three-dimensional avatar that is created in a human-like shape to provide natural movement, and is linked with the sign language to express non-resin language such as hand as well as mouth shape, expression, head and body orientation can be implemented to operate. As such, by implementing augmented reality so that the three-dimensional avatar expresses sign language, it is possible to promote a higher understanding of broadcasting for the hearing impaired.

The sign language generating unit 234 generates a sign language image by connecting the called sign language, and transmits the generated sign language image to the sign language broadcasting generating unit 237 .

In this case, the sign language image generating unit 234 generates a sign language image by contextually combining the voice recognized word and the called sign language with additional information for inferring a person's emotion. In this way, it is possible to create a sign language image that can be expressed naturally according to the situation.

For example, the additional information may include non-verbal information such as speed, tone, size, and tone of the voice for inferring the emotion of the voice language user.

Therefore, the sign language image generating unit 234, in order to obtain and analyze additional information from the image signal obtained from the image acquisition unit of the wearable information expression device 100, voice scaling modulation, face recognition, motion recognizing motion recognition, etc. Based on the extracted information, it is possible to classify and label what kind of emotion the voice language user has, and to digitize the size to generate additional information.

That is, the sign language image generating unit 234 may generate a sign language image by reflecting additional information in sign language operations continuously performed by the 2D or 3D model of the human shape.

Therefore, the type and size of emotions expressed by the actors in the screen, such as the speed, tone, size, tone, facial expression, and body movement, are reflected in the sign language image through the facial expression, hand movement speed, and gaze that the hearing impaired can visually recognize. , the hearing impaired can recognize the type and size of the actor's emotions on the screen.

The voice-word conversion unit 235 may perform a function of converting the recognized voice into a word corresponding thereto, and transmits the converted word to the caption image generating unit 236 . In this case, the voice-word conversion unit 235 may convert the recognized voice into text using a predefined sign language dictionary, and convert the voice into text using a voice-to-text conversion program. As a method of determining a word corresponding to a voice as described above, various well-known methods may be used. In addition, the language of the recognized voice may include languages of various countries such as Korean, English, Japanese, Chinese, German, and French, and the language of the word to be converted is similarly used in various languages such as Korean, English, Japanese, Chinese, German, French, etc. It may be implemented to select and set one of the languages of the country. Accordingly, it is possible to provide subtitle services in various foreign languages as well as Korean, so that it can be used for foreigners watching domestic broadcasts and movies.

The caption image generating unit 236 generates a caption image by connecting the transferred words, and transmits the generated caption image to the sign language broadcasting generation unit 237 . Here, the subtitle image may be in one of foreign languages as well as Korean.

The sign language broadcast generating unit 237 generates a sign language broadcast including at least one of the transmitted sign language image and the caption image. That is, a sign language broadcast may be generated only with a sign language image, with only a subtitle image, or may be generated to include both a sign language image and a subtitle image.

In addition, the sign language broadcast generation unit 237 synchronizes the generated sign language broadcast to the currently playing broadcast so that the TV broadcast and the sign broadcast can be synchronized.

The sign language broadcast reproducing unit 238 displays the generated sign language broadcast in augmented reality on the display unit 130 of the wearable information display device 100 .

On the other hand, the wearable information display device 100, although not shown, may further include a sign language broadcasting setting unit (not shown) capable of adjusting the position, size, and transparency of the sign language broadcasting. For example, by adjusting the position and size of sign language broadcasting, and lowering the transparency so that the broadcasting screen can be easily seen, the concentration of viewing and the understanding of broadcasting can be increased.

And, the sign language service providing server 300, when a sign language service is requested according to the viewing mode, a sign language broadcast search unit 310 that searches for a pre-made sign language broadcast corresponding to the currently watched broadcast from the sign language broadcast DB, and It includes a sync control unit 340 that synchronizes sign language broadcasting to the broadcast being viewed, and a communication unit 350 that is connected to the application program module 200 to transmit and receive data.

The sign language broadcasting search unit 310 includes a sign language broadcasting database 320 for storing pre-produced sign language broadcasting, and a sign language broadcasting schedule database ( 330), and inquires first whether there is a sign language broadcast matching the broadcast currently being watched by inquiring the sign language broadcast schedule, and if the sign language broadcast is in the schedule list, calls the matching sign language broadcast from the sign language broadcasting DB 320, and this It is transmitted to the sync adjuster 340 .

The sync adjustment unit 340 synchronizes the currently viewed broadcast and the called sign language broadcast and outputs it through the communication unit 350 to be displayed on the display unit 130 of the wearable information display device 100 .

Next, a case in which the application program module is installed in the user terminal will be described with reference to FIGS. 3 and 4 . The sign language service providing system of the present invention shown in FIGS. 3 and 4 is different from the sign language service providing system shown in FIGS. 1 and 2 in that the configuration in which the application program module 200 is provided in the user terminal 400 is different and different. Since all configurations are the same, a description of the same configuration may be omitted. In addition, the user terminal 400 may connect to an app store or a website of a service provider, download the application program module 200, and install and store it in advance so that it can be driven.

3 is a diagram showing the overall configuration of a sign language service providing system according to another embodiment of the present invention.

FIG. 4 is a diagram showing a schematic configuration of the application program module shown in FIG. 3 .

Referring to FIGS. 3 and 4 , the sign language service providing system according to the present invention includes a wearable information display device 100 , an application program module 200 provided in a user terminal 400 , and a sign language service providing server 300 . ) may be included.

Here, the wearable information display device 100 and the user terminal 400 may be implemented to be linked through short-range wireless communication (eg, Bluetooth, Zigbee, Wi-Fi, etc.).

The application program module 200 and the sign service providing server 300 may be implemented to be connected through an Internet network using a network (wired Internet communication, wireless Internet communication such as WiFi and WiBro, wired and wireless networks such as mobile communication such as WCDMA and LTE). have.

The wearable information display device 100 supports the implementation of augmented reality of sign language broadcasting. Here, sign language broadcasting may be expressed as an augmented reality implementation in the form of a two-dimensional or three-dimensional avatar.

Although not shown in the drawings, the wearable information display device 100 may be configured to include a voice acquisition unit, an image acquisition unit, a display unit, a communication unit, a power supply unit, a storage unit, an operation unit, a switch unit, and a detachment detection unit, The sign language service may be provided by interworking with the application program module 200 provided in the user terminal 400 through the communication unit.

Sign language service provides sign language broadcasts generated through real-time sign language translation when there is no pre-produced sign language broadcast of a broadcast program or a broadcast program, but sign language broadcast is a wearable information display device in the form of a 2D or 3D avatar It is a service that is expressed in augmented reality on a part of the display unit (not shown), such as the lens of ( 100 ). Here, the sign language broadcast may include at least one of a sign language image and a caption image.

The user terminal 200 includes an application program module 200 that interworks with the wearable information display device 100 to provide sign language broadcasting according to a sign language service.

The user terminal 200 may further include a communication unit 240 in the configuration shown in FIG. 2 , and may be connected to the sign language service providing server 300 through the communication unit 240 and perform real-time data transmission/reception. have.

The application program module 200, in order to support the sign language service using augmented reality, the service mode control unit 210, the viewing mode execution unit 220, the translation mode execution unit 230, the communication unit 240 and sign language broadcasting settings It may be configured to include a unit 250 .

The sign language service includes a viewing mode in which a pre-produced sign language broadcast is provided, and a translation mode in which a sign language broadcast is generated and provided according to a real-time broadcast. Here, the viewing mode may be divided into a broadcast viewing mode that provides a sign language service when watching a broadcast and a movie viewing mode that provides a sign language service when watching a movie. In the case of movie viewing, it may be provided for watching a movie when visiting a movie theater. . Such a sign language service may be streamed through the wearable information display device 100 and a real-time link.

First, the service mode control unit 210 of the application program module 200, when the switch on signal of the switch provided in the wearable information display device 100 is input, control to execute at least one of the viewing mode and the translation mode, but , after executing the viewing mode first, when there is no pre-produced sign language broadcast, it is possible to control the execution of the service mode to execute a translation mode that generates and provides a real-time sign language broadcast.

In addition, the service mode control unit 210 is provided with a manipulation unit (not shown) of the wearable information expression device 100, and when the service mode is selected by the user operating the manipulation unit, each execution unit to perform the selected service mode can be controlled

First, the viewing mode execution unit 220, when the broadcast viewing mode is executed, analyzes the movement of the currently viewed broadcasting screen from the image signal captured through the image acquisition unit 120 to recognize the current broadcasting channel, A pre-produced sign language broadcast matching the current broadcast channel is requested from the sign language service providing server 300 .

Then, the sign language service providing server 300 stores a sign language broadcast schedule prepared in advance corresponding to the broadcast program, and a sign language broadcast schedule including programming information matching the broadcast program and the pre-produced sign language broadcast, respectively. If there is a matching broadcast channel by inquiring the broadcast channel requested from the mode executor 220 in the sign language broadcast schedule, the matching broadcast channel and the matching sign language broadcast are searched, and the searched sign language broadcast is synchronized with the broadcast channel and provided.

In addition, the viewing mode execution unit 220, when the movie viewing mode is executed, when movie selection information is input, searches for a sign language image corresponding to the movie, and when the movie selection information is not input, the image acquisition unit 120 ) to recognize a currently playing movie by performing screen tracking, and request a sign language broadcast matching the recognized currently screened movie to the sign language service providing server 300 .

Then, the sign language service providing server 300 stores, respectively, a sign language broadcast prepared in advance corresponding to the movie, and a movie viewing schedule including the program information matching the sign language broadcast and the movie, and the viewing mode of the application program module. When a sign language broadcast corresponding to a movie is requested from the execution unit 220 , the stored movie viewing schedule may be inquired to find and provide the corresponding sign language broadcast. At this time, the sign language service providing server 300 may synchronize the sign language broadcasting and the movie screen by matching the movie start point analyzed in advance and the movie screen tracking using the image acquisition unit to match the automatically recognized movie start point. have.

Then, when the sign language broadcast synchronized with the requested broadcast channel is transmitted from the sign language service providing server 300 , the viewing mode execution unit 220 may stream the provided sign language broadcast to the display unit 130 in augmented reality. . That is, the sign language broadcast is expressed in augmented reality on the set area of the display unit 130 provided in the wearable information display device 100 . In this case, the sign language broadcast may be provided by being synchronized with the current broadcast channel in the sign language service providing server 300 in real time.

In addition, when the sign language broadcast synchronized with the requested movie is transmitted from the sign language service providing server 300 , the viewing mode execution unit 220 may stream the provided sign language broadcast to the display unit 130 in augmented reality.

Meanwhile, the service mode control unit 210 may switch to the translation mode service when the sign language service providing server 300 fails to search for a sign language broadcast for the current broadcast. That is, the service mode control unit 210 may request a real-time sign language translation service from the translation mode execution unit 230 when a pre-produced sign language broadcast for the currently viewed broadcast is not searched.

The translation mode execution unit 230 includes a voice recognition unit 231 , a voice-sign language matching unit 232 , a sign language image generation unit 234 , a voice-word conversion unit 235 , a subtitle image generation unit 236 and A sign language broadcast generating unit 237 may be included.

When the translation mode is executed, the voice recognition unit 231 interworks with the wearable information display device 100 to request a voice signal and receive it.

That is, the voice recognition unit 231 recognizes a voice from a voice signal transmitted from the voice acquisition unit of the wearable information expression device 100 , and transmits the recognized voice to the voice-sign language matching unit 232 .

The voice-sign language matching unit 232 includes a databased voice-sign language matching DB 233, sequentially compares the recognized voice with the voice stored in the voice-sign language matching DB 233, and compares the recognized voice with the recognized voice. If there is the same voice, a sign language corresponding to the same voice is called, and the called sign language is transmitted to the sign language image generating unit 234 . The sign language stored in the voice-sign language matching DB 233 may be a sign language image for a sign language operation matched with the voice or a moving image to which sign language images are connected.

The sign language image generation unit 234 generates a sign language image by connecting sign words called from the voice-sign language matching DB 233 . The sign language image may be a moving image in which sign language images are continuously connected.

Also, the sign language image generating unit 234 may be implemented to generate a sign language image by contextually combining additional information for inferring a person's emotion with the recognized voice and the matched sign language. In this way, it is possible to create a sign language image that can be expressed naturally according to the situation.

For example, the additional information may include non-verbal information such as speed, tone, size, and tone of the voice for estimating the emotion of the voice language user. This additional information is information for translating the emotions of the voice language user into non-signal signals of facial expressions and gestures when translating voice into sign language.

Therefore, the sign language image generating unit 234, in order to obtain and analyze additional information from the image signal obtained from the image acquisition unit of the wearable information expression device 100, voice scaling modulation, face recognition, motion recognizing motion recognition, etc. Based on the extracted information, it is possible to classify and label what kind of emotion the voice language user has, and to digitize the size to generate additional information for expressing non-significant words.

That is, the sign language image generating unit 234 may generate a sign language image by reflecting additional information in sign language operations continuously performed by the 2D or 3D model of the human shape.

Therefore, the type and size of the emotions expressed by the actors in the screen through the speed, tone, size, tone, facial expression, and body movements of the actors can be visually recognized by the deaf person, using non-responsible symbols such as facial expression, hand movement speed, and gaze. Since it is reflected in the image, the hearing impaired can recognize the type and size of the actor's emotions in the screen.

The voice-word conversion unit 235 converts the recognized voice into words in order to convert the voice recognized by the voice recognition unit 231 into a text format, and transfers the converted words to the caption image generator 236 . do.

The caption image generating unit 236 generates a caption image that connects the delivered words to sentence them and visually displays them in text format, and transmits the generated caption image to the sign language broadcast generation unit 237 .

The sign language broadcast generation unit 237 generates a sign language broadcast using the generated sign language image and caption image, and the sign language broadcast may include at least one of a sign language image and a caption image. For example, a sign language broadcast may include only a sign language image or only a subtitle image, and may also include both a sign language image and a subtitle image. Accordingly, it can be implemented to set using the user interface (UI) of the user terminal 400 when executing the application program module so that the simultaneous or individual expression of the subtitle image and the sign language image can be selected according to the user's request.

The sign language broadcast generation unit 238 may provide a sign language broadcast to the wearable information display device 100 through the communication unit 240 .

In addition, the sign language broadcasting setting unit 250 of the application program module 200 may adjust the sign language broadcasting size, position, and transparency of the augmented reality implementation screen. That is, the size, position, and transparency of sign language broadcasting can be adjusted so that the user does not have any inconvenience in viewing the broadcasting screen. Through this, it is easy to watch the broadcast screen and the sign language broadcast at the same time, and it is possible to concentrate more on the actual broadcast viewing.

In addition, the sign language service providing server 300, when a sign language service is requested according to the viewing mode, a sign language broadcast search unit 310 that searches for a pre-made sign language broadcast corresponding to the currently watched broadcast from the sign language broadcast DB 320 . and a sync control unit 340 that synchronizes sign language broadcasting to the currently viewed broadcast, and a communication unit 350 that is connected to the application program module 200 to transmit and receive data.

The sign language broadcasting search unit 310 stores a sign language broadcasting schedule including a sign language broadcasting DB 320 for storing pre-produced sign language broadcasting, and sign language broadcasting schedule information matching a TV broadcasting program and a pre-produced sign language broadcasting. A sign language broadcasting schedule DB 330 is provided, and if there is a matching broadcasting channel by inquiring the broadcasting channel requested from the viewing mode execution unit 220 in the sign language broadcasting schedule, the corresponding sign language broadcasting is retrieved from the sign language broadcasting DB 320. and transmits it to the sync control unit 340 .

The sync adjustment unit 340 synchronizes the currently viewed broadcast and the called sign language broadcast and outputs it through the communication unit 350 to be displayed on the display unit 130 of the wearable information display device 100 .

That is, in the viewing mode, the sync control unit 340 synchronizes the sign language broadcast called to the currently watched broadcast and transmits it to the application program module 200 through the communication unit 350 , and through this, the sign language broadcast to the currently watched broadcast This may be synchronized and displayed on the display unit 130 of the wearable information display device 100 . In this case, the sync adjuster 340 may support real-time sync adjustment when streaming sign language broadcasting.

Also, the sync control unit 340 may extract a TV broadcasting schedule and a real-time list such as over-the-air and IPTV, and perform sync between TV broadcasting and sign language broadcasting.

As described above, the sign language service providing system according to the present invention provides sign language broadcasting implemented in augmented reality separately from the broadcasting screen in front of the user's eyes through the user's wearable information display device, so that other viewers who do not need sign language broadcasting It is possible to watch the broadcast together without any inconvenience, and by setting the sign language broadcast to a location, size and transparency that the user can see, it is possible to increase the user's concentration on broadcast viewing and understanding of the broadcast content.

In addition, the sign language service providing system according to the present invention automatically processes the provision of sign language broadcasts generated through real-time translation as well as pre-produced sign language broadcasts, so that it is possible to watch broadcasts together with sign language broadcasts in real time without restriction of the broadcast program. have.

5 is a view showing an implementation example of a sign language service using a sign language service providing system according to an embodiment of the present invention, (a) is the location setting (A, B) of the augmented reality screen of the sign language broadcasting, ( b) shows the transparency settings (A, B) of the augmented reality screen of sign language broadcasting, and (c) shows examples displayed on the display unit of the wearable information display device, respectively.

As shown in (a) and (b) of Figure 5, the position (A, B) and transparency (C, D) of the augmented reality implementation screen of sign language broadcasting according to the present invention is changed according to the request of the hearing impaired user can be set by Through this, the hearing impaired users can most naturally recognize sign language broadcasting and increase the concentration of broadcasting viewing.

As shown in (c) of Figure 5, it can be displayed in the setting area of the display unit of the present invention wearable information display device.

As described above, the sign language service providing system according to the present invention can increase the concentration of broadcasting viewing and the understanding of broadcasting contents by setting the augmented reality realization screen to a position, size, and transparency convenient for a hearing impaired user to recognize.

In addition, by providing sign language broadcasting only to hearing impaired users by expressing sign language broadcasting on the display unit of the wearable information display device, other viewers who are not provided with sign language broadcasting can be viewed simultaneously without feeling uncomfortable.

Hereinafter, an embodiment in which the service mode of the sign language service providing system according to an embodiment of the present invention is implemented will be described.

[Example of broadcast viewing mode in service mode]

1. Example in case the application module is installed in the user terminal

The user may execute an application program module of the user terminal and search for currently viewable sign language broadcasts. At this time, the application program module provides a sign language broadcast schedule through a user interface screen, and waits for the user's selection.

When the user selects a sign language broadcasting from the sign language broadcasting schedule provided through the UI screen of the user terminal, the application module requests and receives the selected sign language broadcasting from the sign language service providing server, and displays it in augmented reality on the display unit of the wearable information display device do. At this time, the sign language service providing server provides the sign language broadcasting in a real-time streaming method, but by extracting the broadcasting schedule and the real-time broadcasting list, the currently viewed broadcasting and the sign language broadcasting are synchronized.

On the other hand, when there is no sign language broadcast currently viewable, the application program module may configure a UI screen for confirming whether to perform translation mode conversion and display it on the user terminal. When the user requests to change the translation mode, a sign language broadcast is generated according to the real-time broadcast, and then the sign language broadcast is controlled to be displayed on the display unit of the wearable information display device through a real-time link.

2. Example when the application module is installed in the wearable information display device

When the user turns on the wearable information display device and wears it, the application module recognizes the broadcast channel currently being viewed by analyzing the motion of the broadcast screen from the image captured by the image acquisition unit provided in the wearable information display device.

Then, the application program module requests sign language broadcasting of the recognized broadcasting channel to the sign language service providing server, and the sign language service providing server provides sign language broadcasting, but extracts the broadcasting schedule and real-time list of the currently watched broadcasting channel and sign language broadcasting. Perform sync linkage.

On the other hand, when there is no sign language broadcasting of the recognized broadcasting channel, after generating a sign language broadcasting in real time according to the real-time broadcasting, a translation mode service implemented in augmented reality on the display unit of the wearable information display device may be automatically performed.

[Example of movie viewing mode in service mode]

When a user visits a movie theater, a wearable information display device and an application program module are provided so that the user can watch movie images and sign language broadcasting together.

At this time, the wearable information display device may be provided by the cinema, and the application module that is connected to the sign language service providing server and provides the sign language service is provided through downloading on the user's mobile terminal or downloaded in advance to the wearable information display device. It can be installed via Receiving.

First, when the application module is installed in the user terminal, the user executes the application module, and when the user selects a movie through the UI screen according to the execution of the application module, sign language broadcasting corresponding to the movie selection information is provided. It may be retrieved from the providing server, and the searched sign language broadcast may be provided to the wearable information display device to be streamed together with the start of the movie.

At this time, the movie sync matches the sign language and the screen by automatically recognizing the movie starting point when tracking the screen with the pre-analyzed starting point and augmented reality glasses or mobile phone.

On the other hand, when the user does not select a movie, the movie screen is tracked through the image acquisition unit provided in the wearable information display device to recognize the currently screened movie and automatically synchronize the screen with the sign language broadcast.

In addition, the application program module can provide not only sign language broadcasting but also closed caption broadcasting simultaneously or individually.

In the case of subtitle service, it provides not only Korean but also various foreign language services, so it can be used not only for the deaf, but also for foreigners to watch domestic movies.

As described above, in the present invention, specific matters such as specific components, etc., and limited embodiments and drawings have been described, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Anyone with ordinary knowledge in the field to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the spirit of the present invention is limited to the described embodiments and should not be determined, and not only the claims to be described later, but also all technical ideas that are equivalent or equivalent to the claims are included in the scope of the present invention. Should be interpreted as.

100. Wearable information display device 200. Application module
300. Sign language service providing server 400. User terminal

Claims (3)

delete A wearable information display device for expressing sign language broadcasting including at least one of a sign language image and a caption image using augmented reality;
an application program module that is interlocked with the wearable information display device and outputs the sign language broadcast to the wearable information display device in real time; and
It is connected to the application program module, and stores a sign language broadcast prepared in advance corresponding to a broadcast program, a sign language broadcast schedule table including programming information matching the previously produced sign language broadcast and the broadcast program, respectively, and the application program module When a sign language broadcast of a broadcast program is requested from , the stored sign language broadcast schedule is searched and, if there is a sign language broadcast of the broadcast program, the corresponding sign language broadcast is provided to the application program module, but the broadcast program and the sign language broadcast are synchronized ,
A sign language broadcast prepared in advance corresponding to a movie and a movie viewing schedule matching the sign language broadcasting and the movie are stored, respectively, and when a sign language broadcasting corresponding to the movie is requested from the application program module, the stored movie viewing schedule is used A sign language service providing server that provides a corresponding sign language broadcast;
The wearable information display device includes: a spectacle body made of a spectacle structure;
a voice acquisition unit provided in the glasses body unit, receiving an externally generated voice and outputting a voice signal;
an image acquisition unit provided in the glasses body portion and outputting an image signal by photographing the front thereof;
a communication unit provided in the glasses body unit and performing data transmission/reception;
a display unit for expressing sign language broadcasting in augmented reality on a lens provided on the front of the glasses body; and
It includes at least a switch unit for controlling the on/off of the power.
The application module is provided in the wearable information display device,
When the switch-on signal of the switch unit is input, a pre-produced sign language broadcast is provided in augmented reality, but when there is no pre-made sign language broadcast and a viewing mode including a broadcast viewing mode and a movie viewing mode, a real-time sign language broadcast is generated. a service mode control unit executing at least one of the service modes including the provided translation mode;
When the broadcast viewing mode is executed, the current broadcast channel is recognized by analyzing the motion of the currently watched broadcast screen from the video signal captured through the image acquisition unit, and a pre-made sign language broadcast matching the recognized current broadcast channel information is performed. A sign language broadcast requested by the sign language service providing server is controlled to be displayed in augmented reality on the display unit, and when the movie viewing mode is executed, when movie selection information is input, a sign language image corresponding to the movie is retrieved. and, when the movie selection information is not input, screen tracking through the image acquisition unit is performed to recognize a currently playing movie, and a sign language broadcast matching the recognized currently playing movie is searched for in augmented reality on the display unit. a viewing mode execution unit controlling to be expressed;
When the translation mode is executed, a voice is recognized from the voice signal output from the voice acquisition unit, a sign language matching the recognized voice is searched for, and a sign language image is generated by connecting the searched sign language, and matching with the recognized voice A caption image is generated by searching for a word to be used and a sentence is constructed through a combination of the searched words, a sign language broadcast including at least one of the generated sign language image and the caption image is generated, and the generated sign language broadcast is augmented on the display unit Translation mode execution unit that controls to be expressed in reality; and
and a sign language broadcasting setting unit for adjusting at least one of a position, size, and transparency of a sign language broadcast displayed on the display unit. A wearable information display device for expressing sign language broadcasting including at least one of a sign language image and a caption image using augmented reality;
an application program module that is interlocked with the wearable information display device and outputs the sign language broadcast to the wearable information display device in real time; and
It is connected to the application program module, and stores a sign language broadcast prepared in advance corresponding to a broadcast program, a sign language broadcast schedule table including programming information matching the previously produced sign language broadcast and the broadcast program, respectively, and the application program module When a sign language broadcast of a broadcast program is requested from , the stored sign language broadcast schedule is searched and, if there is a sign language broadcast of the broadcast program, the corresponding sign language broadcast is provided to the application program module, but the broadcast program and the sign language broadcast are synchronized ,
A sign language broadcast prepared in advance corresponding to a movie and a movie viewing schedule matching the sign language broadcasting and the movie are stored, respectively, and when a sign language broadcasting corresponding to the movie is requested from the application program module, the stored movie viewing schedule is used A sign language service providing server that provides a corresponding sign language broadcast;
The wearable information display device includes: a spectacle body made of a spectacle structure;
a voice acquisition unit provided in the glasses body unit, receiving an externally generated voice and outputting a voice signal;
an image acquisition unit provided in the glasses body portion and outputting an image signal by photographing the front thereof;
a communication unit provided in the glasses body unit and performing data transmission/reception;
a display unit for expressing sign language broadcasting in augmented reality on a lens provided on the front of the glasses body; and
It includes at least a switch unit for controlling the on/off of the power.
The application module is provided in the user terminal,
Among the service modes that provide a pre-produced sign language broadcast in augmented reality, a viewing mode including a broadcast viewing mode and a movie viewing mode, and a translation mode that generates and provides real-time sign language broadcasts when there is no pre-produced sign language broadcast a service mode control unit executing at least one;
When the broadcast viewing mode is executed, when a broadcast channel capable of sign language service is selected through the user interface screen of the user terminal, a pre-produced sign language broadcast of the selected broadcast channel is requested to the sign language service providing server, and the broadcast channel is selected In the absence of this, the current broadcasting channel is recognized by analyzing the movement of the currently viewed broadcasting screen from the image signal captured by the image acquisition unit, and a pre-made sign language broadcast of the recognized current broadcasting channel is requested to the sign language service providing server. and control so that the received sign language broadcast is displayed in augmented reality on the display unit, and when the movie viewing mode is executed, when movie selection information is input, a sign language image corresponding to the movie is searched, and the movie selection information When is not input, screen tracking is performed through the image acquisition unit to recognize a currently playing movie, and a sign language broadcast matching the recognized currently screened movie is searched for and controlled to be displayed in augmented reality on the display unit. executive department;
When the translation mode is selected, a voice is recognized from the voice signal output from the voice acquisition unit, a sign language matching the recognized voice is searched for, and a sign language image is generated by connecting the searched sign language, and matching with the recognized voice A caption image is generated by searching for a word to be used and a sentence is constructed through a combination of the searched words, a sign language broadcast including at least one of the generated sign language image and the caption image is generated, and the generated sign language broadcast is augmented on the display unit Translation mode execution unit that controls to be expressed in reality;
a sign language broadcasting setting unit for adjusting at least one of a position, a size, and a transparency of a sign language displayed on the display unit; and
A sign language service providing system comprising a; a communication unit for transmitting and receiving sign language broadcasting.

Images