KR102585545B1 - Method for providing speech recognition based product guidance service using user manual - Google Patents

Abstract

A method of providing a voice-based product user manual guidance service is provided, including receiving content that can identify the product from the user terminal, checking whether the application for the guidance service is installed on the user terminal, and if the application is not installed, sending the product to the user terminal. Step of transmitting the URL to install the application, when the application is installed on the user terminal, extracting the unique identification code already mapped and stored to identify the product, and reading the product user manual previously mapped and stored with the unique identification code. It includes converting text in the step and product user manual into voice (Text To Speech) and transmitting it to the user terminal.

Description

Method of providing voice-based product user manual guidance service {METHOD FOR PROVIDING SPEECH RECOGNITION BASED PRODUCT GUIDANCE SERVICE USING USER MANUAL}

The present invention relates to a method of providing a voice-based product user manual guide service, and provides a platform that can convert text product user manuals into voice and output them.

The product user manual plays the role of a guide that provides all information necessary for the producer of a product or service to inform consumers of the steps to be taken for safe and efficient use. Under the Product Liability Act, defects that occur due to failure to properly comply with the obligations to explain, direct, or warn about the use of a product are called labeling defects. Manufacturers provide information to consumers to prevent accidents that may occur during use of the product. There is an obligation to provide materials such as manuals. As the importance of product user manuals is emphasized, the Korean Industrial Standards KS A ISO/IEC Guide 14:2003 and KS A ISO/IEC Guide 37:2003 provide national or international information regarding the information buyers require and expect about products or services. Relevant content is recommended so that those who establish standards can convey information on how to use the product correctly and safely. Both guidelines provide general principles for the content, method, format and writing of information and the use of a variety of delivery methods, including combinations of words, phrases, symbols, signs, figures, examples and visual and audible signals.

At this time, in order to provide product user manuals more intuitively, curating or voice guidance methods were researched and developed. In this regard, the prior art, Korea Patent Publication No. 2017-0002893 (published on January 9, 2017) and Korea Patent Publication No. Publication Patent No. 1997-0031674 (published on June 26, 1997) recognizes the QR code or smart tag displayed on the product and converts it into product data to provide the user with information about the product and content on how to use or install the product. To ensure that video, curating, and A/S diagnosis and A/S reception for product failures are processed through a unified process, list information by type for product composition, specification information such as shape, size, design, and color for each product, and identification for each product. It is linked to the manufacturer's server and the manufacturer's server, which consists of a product DB composed of available code information and a content DB that provides usage information and installation information for users to easily use based on the products registered in the product DB, and is linked to the manufacturer's server through voice or video communication with the user. It includes a curating function that enables two-way communication through a press process that includes A/S diagnosis and A/S reception for product failures in order to determine whether there is a breakdown at the user's product use or installation site and respond immediately. A guidance key is input so that the product's usage guidance message is output by voice from a phone equipped with a configuration that processes the product's usage guidance message as numerical data, a memory that stores the product's usage guidance message as numerical data, and a guidance key for outputting the usage guidance message. Each configuration is disclosed to read a product usage guide message from memory and output it.

However, in the case of the former of the above-mentioned configurations, even if the text user manual is changed to content, it is difficult to find content for the exact function desired by the user, and even if it is output by voice using the guidance key as in the latter, the user is You must know the guidance key exactly for what you are looking for in order to output a voice for it. For non-experts, it is difficult for ordinary people to understand the exact wording written in the product user manual, so not being able to find the keyword naturally results in not being able to find the content they are looking for, which is the result of the search. Accordingly, research and development of a platform that can help consumers obtain correct information through the principles provided in the guidelines to reduce the risk of damaging the product and thereby reduce inoperable conditions and inefficient operation is required. .

In one embodiment of the present invention, when a product is recognized using a product-recognizable identification code, the user terminal is controlled to access the page where the service can be provided, and the user terminal is ready to receive the service on the page. Then, the product user manual stored by mapping to the identification code can be searched and the product user manual can be output by voice, thereby increasing the readability and portability of the product user manual and extracting and delivering only the functions desired by the user. It is possible to provide a method of providing a voice-based product manual guidance service that allows users to extract and refine the parts of the product manual that they frequently look for, and to reduce costs such as phone inquiries and printing costs. However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention includes the steps of receiving content that can identify a product from a user terminal, checking whether an application for a guidance service is installed on the user terminal, and installing the application. If not installed, transmitting the URL to install the application to the user terminal. If the application is installed on the user terminal, extracting the unique identification code that has been mapped and stored to identify the product, and the unique identification code and the previously mapped and stored It includes the steps of reading the product user manual and converting the text in the product user manual into voice (Text To Speech) and transmitting it to the user terminal.

According to one of the above-described means for solving the problem of the present invention, when a product is recognized using an identification code that can recognize the product, access to a page where a service can be provided is controlled from the user terminal, and the service is provided from the page on the user terminal. When you are ready to receive the product, you can search the stored product manual mapped to the identification code and output the product manual by voice, thereby increasing the readability and portability of the product manual and extracting only the functions the user wants. It allows users to extract and refine parts of the product manual that they frequently look for, and it is possible to reduce costs such as phone inquiries and printing costs. Environmental resources can be saved by eliminating paper use. Since you can receive the updated version of the product user manual, you can refer to the explanation that suits your current situation.

1 is a diagram illustrating a system for providing a voice-based product manual guidance service according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a guidance service providing server included in the system of FIG. 1.
Figures 3 and 4 are diagrams for explaining an embodiment in which a voice-based product manual guidance service is implemented according to an embodiment of the present invention.
Figure 5 is an operation flowchart illustrating a method of providing a voice-based product manual guide service according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this does not mean excluding other components unless specifically stated to the contrary, but may further include other components, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms "about", "substantially", etc. used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. In addition, the components and 'parts' may be implemented to regenerate one or more CPUs within the device or secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating a system for providing a voice-based product manual guidance service according to an embodiment of the present invention. Referring to FIG. 1, the voice-based product user manual guidance service providing system 1 may include at least one user terminal 100, a guidance service providing server 300, and at least one artificial intelligence speaker 400. there is. However, since the voice-based product user manual guide service providing system 1 shown in FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1.

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one user terminal 100 may be connected to the guidance service providing server 300 through the network 200. In addition, the guidance service providing server 300 may be connected to at least one user terminal 100 and at least one artificial intelligence speaker 400 through the network 200. Additionally, at least one artificial intelligence speaker 400 may be connected to the information service providing server 300 through the network 200.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. This should be self-explanatory. In addition, whether each component is provided in singular or plural form may be changed depending on the embodiment.

At least one user terminal 100 registers the product with the guidance service providing server 300 using a web page, app page, program, or application related to the voice-based product manual guidance service and converts the product manual into voice. It may be a terminal that receives voice data streaming from the guidance service providing server 300. The user terminal 100 may be a terminal that registers the artificial intelligence speaker 400 on the information service providing server 300, and in this case, queries the contents of the product manual through voice through the artificial intelligence speaker 400. , It may be a terminal that receives a response by voice. Of course, it is obvious that the user terminal 100 can also selectively output the contents of the product manual through voice-based questions and answers.

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The guidance service providing server 300 may be a server that provides a voice-based product user manual guidance service web page, app page, program, or application. Additionally, the information service providing server 300 may be a server that collects product user manuals from at least one manufacturer or seller's server (not shown) or collects product user manuals through web crawling. In the latter case, since the copyright of the product user manual belongs to the manufacturer or seller, it can be collected and used after signing a licensing agreement for copyright use. Additionally, the information service providing server 300 may be a server that stores the product name, model number, model name, exterior photo, etc. for each product by mapping them to a unique identification code. Afterwards, when the product name or model number, etc., is input by text or voice from the user terminal 100, the information service providing server 300 extracts the unique identification code and extracts the product user manual previously mapped and stored to the unique identification code. This may be a server that streams voice conversion or previously converted audio data to the user terminal 100 in real time. In addition, when the artificial intelligence speaker 400 is registered through the user terminal 100, the information service providing server 300 can query and respond to the contents of the product manual based on voice through the artificial intelligence speaker 400. It may be a server that allows this. At this time, as the saying goes, “even if you sleep with a sickle,” the general public may not know the names of each part properly, so the information service providing server 300 uses AR content to identify members or parts (Members) that are currently explained in the product user manual. It may be a server that can tell you where the parts are through AR content. At this time, the audio data and the AR content can be synchronized and set so that when the AR content representing each part is displayed, the corresponding audio data is output. In addition, when the guidance service providing server 300 receives a voice utterance from the user terminal 100, it builds a corpus or a customer word dictionary using the wording included in the query and selects the correct keyword or search word. After mapping and saving, even if the customer's own word is input, that is, even if the customer utters the wrong name, the server can find the correct keyword and print the product user manual mapped to it.

Here, the guidance service providing server 300 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc.

At least one artificial intelligence speaker 400 may be an artificial intelligence device that asks and answers questions based on voice, with or without using a web page, app page, program, or application related to the voice-based product user manual guide service. At this time, the artificial intelligence speaker 400 may be a device that is linked with the guidance service providing server 300 through the user terminal 100 and receives and outputs a response to the user's inquiry from the guidance service providing server 300. there is.

Here, at least one artificial intelligence speaker 400 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one artificial intelligence speaker 400 may be implemented as a terminal that can connect to a remote server or terminal through a network. At least one artificial intelligence speaker 400 is, for example, a wireless communication device that guarantees portability and mobility, and is used for navigation, personal communication system (PCS), global system for mobile communications (GSM), and personal digital cellular (PDC). , PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband) It may include all types of handheld-based wireless communication devices such as Internet) terminals, smartphones, smartpads, and tablet PCs.

Figure 2 is a block diagram for explaining the guidance service providing server included in the system of Figure 1, and Figures 3 and 4 are an embodiment of the voice-based product manual guidance service according to an embodiment of the present invention. This is a drawing to explain.

Referring to FIG. 2, the information service providing server 300 includes an input unit 310, a transmission unit 320, a reading unit 330, a conversion unit 340, a database unit 350, and an automatic search unit 360. , may include a sharing unit 370 and a periodic alarm unit 380.

The guidance service providing server 300 or another server (not shown) operating in conjunction with an embodiment of the present invention uses a voice-based product with at least one user terminal 100 and at least one artificial intelligence speaker 400. When transmitting a manual guidance service application, program, app page, web page, etc., at least one user terminal 100 and at least one artificial intelligence speaker 400 are configured to use a voice-based product manual guidance service application, program, or app. You can install or open pages, web pages, etc. Additionally, a service program may be run on at least one user terminal 100 and at least one artificial intelligence speaker 400 using a script executed in a web browser. Here, a web browser is a program that allows the use of web (WWW: World Wide Web) services and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone). In addition, it can provide OS (Operating System) compatibility and can be composed of hybrid applications, and can also include Windows or Linux.

Referring to FIG. 2 , the input unit 310 may receive content identifying a product from the user terminal 100 . Content that can identify a product includes a one-dimensional identification code such as a barcode attached to the product or a two-dimensional identification code such as a QR code, an image taken of the product, the model name or name of the product, and the model name or name of the product. It may be any one or a combination of at least one of the audio data obtained by voice utterance. For example, if [LG Air Wash (product name)-HW500DAS 5L (model name)-12345678 (barcode)-QR code-audio data] is mapped and stored with a unique identification code, it will refer to LG Air Wash no matter what the user enters. You will be able to understand what you are doing. Of course, products can be identified in a variety of ways other than those described above, so they are not limited to those listed and are not excluded for reasons not listed.

At this time, users may often change the name of the product to a different name, or may call it their own word because they do not know the name of each member or part properly. For example, LG AirWash is the official name, but it is often called a similar name due to unfamiliarity or confusion, such as LG AirShower. In this case, no matter how much you type in LG Air Shower, no search results will appear. The biggest drawback of portal sites is also this keyword system. Obviously, all information can be found on portal sites, but wandering around without being able to find it is due to not knowing the exact keywords. Accordingly, when building a corpus by collecting all the names or keywords incorrectly queried by the user or building a dictionary of customer words, an accurate answer is provided because the correct keyword is already mapped even if the name is incorrectly referred to because the exact name is not known. I can do it. For example, continuing to refer to the above-mentioned example, it can be constructed in the form of [LG Air Shower (incorrect name) - LG Air Wash (correct name) - Unique Identification Code - Product User Manual], and [Correct Name - The correspondence relationship of [misnomer] can be one to many.

Even for the same product, the keywords searched by each person may be different, and because the foreign language notation is not established, different foreign language notations or abbreviations are used for the same product, even though a database corresponding to the product exists. We cannot rule out cases where it is not found. At this time, a corpus of similar keywords can be built. We decide to build a corpus of similar keywords (search terms) using Word2Vec based on unsupervised learning.

At this time, in the first profile process, text data is collected from web pages, the collected data is preprocessed through stopword removal and a morpheme analyzer, and representative keywords of the document are extracted based on the frequency of occurrence of words in the document. . Second, it is possible to build an unsupervised learning-based word embedding model that can calculate semantic similarity between words in a vector space of a specific dimension. For this purpose, unstructured text data is first collected, the collected data is preprocessed by removing stop words and using a morpheme analyzer, and then the noun text data is used to embed parameters such as learning count, number of words that appear with input data, and embedding. You can set the number of dimensions for

In the third keyword selection process, candidate keywords can be selected by calculating the semantic similarity between the website profile and candidate keywords through a word embedding model. Since the keywords extracted during the profile process may include keywords that do not contain topicality, keywords that do not contain topicality can be removed by selecting candidate keywords. Candidate keywords can be built in advance around search terms to find the same product, but are not limited to this. Lastly, in the filtering process, among the keywords selected in the keyword selection process, keywords that have high similarity to candidate keywords but are inconsistent with the document may be selected. Therefore, in order to remove inconsistent keywords among the selected candidate keywords, semantic similarity can be calculated for all candidate keyword pairs, the inconsistent keywords are removed, and the final keywords can be extracted and included as similar keywords.

Also, you can use a corpus of keywords. For example, let's say that a keyword called B is required for a search result called A. At this time, because people have different background knowledge or vocabulary, they may want to search for A through B', C, D, E, etc., rather than entering B exactly. At this time, words that lead to the keyword B, words that explain or are related to the keyword B, or words that have the same meaning but are different in vocabulary from the keyword B are placed in the corpus set of the keyword B.

At this time, corpus is a collection of linguistic data that collects texts in a computer-readable form for language research, and data that can reveal linguistic reality as a whole, based on criteria such as medium, time, space, and annotation level. There are various types, and it means a bundle of words that can be seen as one lump. This means that human language has been input through a technological method called computerization, and this individual language data forms a collection according to certain characteristics, which can be called a bundle. This kind of corpus construction is largely classified into two types. One is simple input, that is, a raw corpus that is computerized in its original form without going through a processing process, and annotations are attached to this raw corpus based on standards such as certain grammar or parts of speech. It is a tagged annotation corpus. Adding annotations like this depends on the purpose of how to use these raw corpus data. Ultimately, by constructing the same corpus data in both raw and processed form, the scope of purpose and use of various language studies is expanded and reproduced.

For example, a single search in which only one search term appears from the start to the end of the search behavior can be divided into a chain search that generates a bundle of multiple search terms. Chain search can be classified into search word elaboration, which corrects spelling or the form of the search word, and interest elaboration, which involves deepening or moving the topic. Search terms (keywords) used in chain search are scored along the axes of categorical and detailed attributes. Through this, properties can be divided into category search words, navigation search words, and instance search words.

For example, items with large categorical attributes such as [humidifier] or [air conditioner] can be classified as category search words, while items with large detailed attributes such as [LG Air Washer] and [LG Whisen Wall Mount] can be classified as instance search terms. You can. A navigation search word is a search word composed of two or more words that acts as an intermediate between a category search word and an instance search word. Search types that start with category search words are sequentially refined by moving from high category tags such as [humidifier] and [air conditioner] to navigation search words and instance search words, while search types that start with navigation search words start from the beginning with [ Starting with navigation tags such as [LG Air Washer] and [LG Whisen Wall Mount], the final instance search term is reached by going back and forth between navigation search term and instance search term. When using these attributes, the final result can be found at once using only the initial search term.

The input unit 310 stores the search term, result image data, and search log searched in the user terminal 100 as history data, distinguishes the start and end time of the search related to the search term, and records the search term up to the start and end point. Time-series changes in search terms can be collected as big data. Here, when a search term corresponding to the starting point is entered as a query, the results of the search term at the end point, that is, the part corresponding to the response in the product user manual, are output to learn and train the shortcut search. Through this, at least one artificial intelligence algorithm can be modeled.

For example, let's assume that the user owns an LG air washer, manufactured in 2012, and the model name is H-90NAW. At this time, the user owns the LG air washer, but does not know the product name because it was a gift from an acquaintance. The user wants to clean the filter, but doesn't know where the filter is. At this time, the user must find out how to clean the filter in the product user manual. Since the LG logo is visible and I know that it is a humidifier, and I received it in 2012, I used the search term [LG humidifier 2012]. As a result of this search, I found that the humidifier released by LG in 2012 was a natural evaporation humidifier. Accordingly, the user searches for his or her humidifier again using the search term [LG Humidifier 2012 Natural Vaporization].

The user found image search results similar to his or her humidifier, and found out through the image search results that the model name was H-90NAW. In this case, the time-series change in the search term changed from [LG, humidifier, 2012] -> [LG, humidifier, 2012, natural vaporization] -> [image search] -> [H-90NAW]. Now that you know the model name, you need to look up how to clean the filter. At this time, I searched for [H-90NAW, filter cleaning] and eventually found out how to clean the filter of my humidifier. At this time, since [LG, humidifier, 2012] is the initial search term, you can eliminate the [LG, humidifier, 2012, natural vaporization]->[image search] process and search for [H-90NAW] directly with a shortcut.

At this time, as described above, a corpus or a dictionary of customer words can be applied here as well. In fact, the H-90NAW model has a “vaporizing disk” rather than a “filter”. It is a cylindrical container that allows natural vaporization, but the user thinks of it as a filter. Even at this time, even if the user inputs “filter,” many users recognize “vaporizing disk” as “filter,” and since having a disk in a humidifier is an unfamiliar concept, “filter” is given to the user who recognized it as “filter.” "You can also explain it instead. Of course, we can also tell you that the name of the filter is actually vaporizing disk.

To this end, the input unit 320 collects search history logs, stores the collected raw data (Raw Data) in parallel and distributed, and unstructured data and structured data included in the stored raw data. Big data can be built by refining and semi-structured data, performing pre-processing including classification with meta data, and training the pre-processed data through data mining and machine learning. At this time, machine learning may be any one or a combination of at least one of Supervised Learning, Semi-Supervised Learning, Unsupervised Learning, and Reinforcement Learning. And, data mining is classification, which explores the inherent relationships between preprocessed data and learns a training data set with a known class to predict the class of new data, or clustering, which groups data based on similarity without class information. It may include performing (clustering).

The transmission unit 320 may check whether the application for the guidance service is installed on the user terminal 100 and, if the application is not installed, transmit a URL to install the application to the user terminal 100. At this time, the transmission unit 320 must determine not only whether the application is installed, but also the specifications and environment of the user terminal 100. If the text of the product manual is converted to TTS in real time and streamed, networking resources or computing resources must be checked, and transcoding must be performed to ensure there is no delay. Of course, you can also convert audio data (voice data) based on HTML5 and stream it without installing the application. Here, HTML5 is the latest standard for HTML (Hyoer Text Markup Language), a web document standard, and includes functions for executing pictures, videos, music, etc. in the center of document creation.

HTML5 eliminates the need to install separate programs like plug-in-based Active-X and Flash, and most problems arising from this can be resolved. At this time, the biggest difference between the conventional HTML standard and HTML5 is that it supports semantic markup and provides additional API. Semantic markup is a new element that can semantically structure web documents. For example, if you create a document with elements called <head>, <section>, and <footer> so that the content of the document can be divided into header, body, and footer, you can clearly understand the document's content structurally. Additionally, HTML5 provides various APIs. The API provided by HTML5 allows you to add various functions to web applications, such as supporting two-dimensional graphics or the ability to check the geographical location of the device.

In addition, HTML5 is a markup language used when creating web pages, and can express and provide HTML, which was previously displayed as hyperlinks and text, to various applications such as multimedia. HTML5 can be implemented including HTML, a markup language for web document structure design, CSS for design expression, and JavaScript for expressing interactive actions, and new tags can be added to existing HTML tags. More can be added, and the structure to contain content can be improved to even serve as a platform. In addition, although HTML5 is not a standard for the mobile web, it provides many of the functions necessary for implementing the mobile web among the many functions described above. Two-dimensional graphics, audio, and various web forms can make the UI (User Interface) of the mobile web rich and diverse. In other words, it is possible to provide multimedia information or support dynamic user interaction even in a mobile web environment. In the mobile web environment, there are limitations in transmitting large amounts of data due to the nature of the mobile environment, and environments where the Internet is not connected often occur. However, these limitations can be overcome by implementing offline web applications using the cache of the web client. Alternatively, you can save the DB locally and query it using SQL statements.

When the application is installed on the user terminal 100, the reading unit 330 extracts the unique identification code previously mapped and stored to identify the product, and reads the product user manual previously mapped and stored with the unique identification code. You can. Continuing to refer to the above-described example, for example, if the model name [H-90NAW] is entered in the user terminal 100, and the unique identification code already mapped and stored with H-90NAW is [12345678], the mapped and stored unique identification code is [12345678]. You can read the product user manual.

The conversion unit 340 can convert text in the product manual into voice (Text To Speech) and transmit it to the user terminal 100. In addition, when converting text in the product user manual into voice (Text To Speech) and transmitting it to the user terminal 100, the conversion unit 340 provides a streaming output service for the audio data converted into voice to the user terminal 100. You can. At this time, when the conversion unit 340 converts the text in the product user manual into voice (Text To Speech) and transmits it to the user terminal 100, if there is an artificial intelligence speaker 400 connected to the user terminal 100. , It can be controlled to output voice through the artificial intelligence speaker 400. When converting text in the product user manual into voice (Text To Speech) and transmitting it to the user terminal 100, the conversion unit 340 recognizes the user's voice utterance input from the user terminal 100 and converts it into text ( Speech To Text), extract the product user manual including the converted text, convert it into voice, and transmit it to the user terminal 100.

Before receiving content that can identify a product from the user terminal 100 at the input unit 310, the database unit 350 web crawls or collects product instruction manuals for at least one type of product from at least one manufacturer's server. This can be converted into a database. At this time, a web crawler for web crawling is a computer program that searches the World Wide Web in an organized and automated manner. The work done by a web crawler is called web crawling or spidering and is a type of bot or software agent. Web crawlers are largely divided into general web crawlers and distributed web crawlers.

To explain the basic operation of a web crawler, it first starts by taking a URL from the URL Frontier module and using the HTTP protocol to retrieve the web page for that URL. Then, the Fetch module stores the web page in temporary storage, the Parser module extracts text and links, and the text is sent to the Indexer. In the case of a link, whether it should be added to the URL frontier is determined by going through the Content Seen, URL Filter, and Duplication URL Element modules. At this time, since it is virtually impossible to crawl all web documents with a general web crawler, a distributed web crawler can be used.

Distributed web crawlers are largely divided into two types, one of which is the centralized method and the other is the P2P (or Fully-Distributed) method. A centralized distributed web crawler has a structure in which the URL manager acts as a server and the crawler acts as a client. When a document is downloaded from a crawler, the OutLink URL is extracted and handed over to the URL manager, the URL manager checks whether it is the URL of the downloaded document and removes URL duplication. In other words, the URL manager takes care of URL duplication and URL management in a regular web crawler. Meanwhile, in the P2P method, each crawler has a completely independent structure. In the P2P method, each crawler operates like a regular web crawler. Each crawler operates independently, downloading documents, extracting OutLink URLs, and removing URL duplicates. To do this, the list of downloaded URLs managed by each crawler must be mutually exclusive. Otherwise, the same document will be downloaded from different crawlers. As a way to solve this, each crawler can manage the URL domain to be downloaded by dividing it into an exclusive domain. In other words, it is a method of managing only those belonging to the download domain and passing the remaining URLs to other crawlers. When using this method, each crawler can operate independently.

Next, web content must be extracted, and web content extraction technology provides a function to automatically extract content to be used for information analysis from web documents. A web content extraction system is a device that extracts only content by automatically generating rules for extracting content. It includes a rule generator that automatically generates content extraction rules, and a navigation content eliminator that removes navigation content from a given web document. , Content extraction rules may consist of a content extractor (Core Context Extractor) that extracts content through keyword similarity comparison. Of course, it is also possible to collect real-time data in a variety of ways in addition to the crawling method described above, and is not limited to those listed and is not excluded for reasons not listed.

Before receiving content that can identify a product from the user terminal 100 at the input unit 310, the automatic search unit 360 selects at least one type of IoT device linked to the network to which the user terminal 100 is connected. You can search. In addition, the automatic search unit 360 may extract the product model name of at least one type of IoT device, extract a product user manual previously mapped and stored with the product model name, and store it to be mapped to the user terminal 100. Before the user inputs the product, IoT-connected products map and store the user terminal 100 and unique identification code in advance. In this case, the user does not need to search for the model name, as each product is registered in the product user manual that the user can view, so the user just needs to select it.

After receiving content that can identify the product from the user terminal 100 at the input unit 310, the sharing unit 370 receives a product usage tip (Tip) input from a user terminal 100 other than the user terminal 100. ) Data can be collected and transmitted to the user terminal 100.

The cycle notification unit 380 can be set to receive content identifying the product from the user terminal 100 at the input unit 310 and then deliver a pre-stored notification message for each product usage cycle to the user terminal 100. there is. At this time, when the cycle notification unit 380 is set to deliver a notification message pre-stored for each usage cycle of the product to the user terminal 100, if the pre-stored notification message for each usage cycle is a parts replacement message, the parts are available for purchase. It can be set to provide at least one shopping mall page and discount coupon. For example, assuming that the robot vacuum cleaner's filter replacement cycle is 6 months, a filter can be purchased at the linked shopping mall while transmitting a message to the user terminal 100 that the filter must be replaced 6 months from the current time. By providing a URL and allowing a discount coupon to be applied when purchasing a filter, users can be encouraged to avoid going to other shopping malls when changing filters.

In addition, an embodiment of the present invention may further provide AR content. Each product's instruction manual is written in various languages, and even if it is written in Korean, it is often difficult to know which part it refers to because the terms written in the product manual are unfamiliar. For example, let's say the user wants to clean the filter, but doesn't know which part to remove. The H-90NAW product manual tells you to remove the “water tank unit,” but if you don’t know what the “water tank unit” is or where the “water tank unit” is, display AR content at the location of the “water tank unit” icon (▶), etc. This allows you to identify which part is the “water tank.”

For example, you can create a marker image using the drawMarker() function and obtain the four corner point values and IDs from the markers on the image using the detectMarker() function. Using the four corner point values and IDs of the obtained markers, the 3D position and direction of the camera is found using the Estimate Pose Single Markers Method function provided by the OpenCV library. Before estimating the camera's pose, the camera's unique parameters (Focal Length, Pricipal Point, Skew Coefficient) are obtained and error values are corrected using a camera calibration algorithm to measure precise camera position and direction values. You can. Additionally, using the EstimatePoseSingleMarkers Method function, it receives markers detected from images input from a camera and individually returns pose estimates for the camera.

Therefore, one rotation vector and translation vector are returned for each marker. If the length of one side of the marker is obtained as an input value, the marker coordinate system is located at the center of the marker, and the z-axis is perpendicular to the marker plane. In this way, even if the camera is not a depth camera, the distance between the camera and the subject can be measured simply by photographing an object whose absolute size and position are already known, and camera-specific parameter values can be measured using camera calibration. You can estimate the position and posture of the camera by correcting the camera's error.

At this time, since the user terminal 100 must output video (AR content) in real time, networking and computing resources must be sufficiently supported. For this purpose, it can be designed to achieve high performance even with the same computing resources by using GPU. To prevent real-time speed and stability from deteriorating, parallel processing techniques using multi-GPU are used, and the level of activation for each GPU is identified and the GPU is assigned to a thread to optimally distribute resources. There is more available.

If GPU resources are not allocated efficiently in the process of using multi-GPU, one GPU will be overloaded, causing problems with stable system operation. Therefore, in one embodiment of the present invention, an algorithm can be used to optimally distribute GPU resources using CUDA to N threads. The GPU allocation algorithm first determines the number of activated cores and memory activation level of all GPU devices. Through this process, you can determine how much each GPU is currently being used. Second, the GPU with the lowest occupancy rate is assigned to the thread according to the current resource usage status of each GPU. At this time, the optimal GPU is assigned considering both memory usage and the number of activated cores. For example, GPU #2, which has the lowest resource occupancy, is assigned to thread #1. By performing the first and second processes for thread #N-1 and thread #N, the optimal GPU can be assigned to all threads.

Third, after allocating the optimal GPU to all threads, there is a risk of overloading the GPU if the first and second processes are repeated every frame. Therefore, the first and second processes are performed again only when the GPU share increases above a certain level. If you run the algorithm with a higher threshold, there is a risk that the GPU usage will increase faster than expected and overload the GPU before switching. Conversely, if the standard value is lowered, there is a problem in that GPU switching occurs too frequently and performance deteriorates. Therefore, the GPU share, which is the standard for re-performance, can be set to a preset percentage, for example, 60%, but is not limited to this.

Hereinafter, the operation process according to the configuration of the information service providing server of FIG. 2 described above will be described in detail using FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to FIG. 3, (a) the information service providing server 300 collects product user manuals through web crawling or from at least one manufacturer or seller's server, and provides product names, model names, photos, etc. that can identify each product. A database is built by mapping QR codes, barcodes, etc. and storing them based on unique identification codes. And, (b) the information service providing server 300 converts the product name into text when the user terminal 100 utters the product name, finds the unique identification code mapped to and stored in the text, and reads the product user manual again. Find it and prepare to notify it by voice.

At this time, (c) the guidance service providing server 300 transmits a URL to the user terminal 100 to download the application or transmits an address where access to a web page, etc. can be accessed, and (d) the guidance service providing server 300 Converts the voice spoken by the user terminal 100 into text, extracts the corresponding part of the product manual, converts it back into voice, and streams it to the user terminal 100.

Referring to (a) of FIG. 4, the user terminal 100 can link the artificial intelligence speaker 400, and (b) the information service providing server 300 allows the user to name each part using AR content. Or, you can pinpoint exactly where that part is. And, as in (c), the guidance service providing server 300 may provide guidance to the user terminal 100 through time-series messages, and (d) other user terminals 100(2),...100(N) ) can also share usage tips with the user terminal 100.

Matters that are not explained regarding the method of providing the voice-based product user manual guide service in FIGS. 2 to 4 are the same as, or are derived from, the content described above regarding the method of providing the voice-based product user manual guide service through FIG. 1. Since it can be easily inferred, the description below will be omitted.

FIG. 5 is a diagram illustrating a process in which data is transmitted and received between components included in the voice-based product user manual guide service providing system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of the process of transmitting and receiving data between each component will be described with reference to FIG. 5, but the present application is not limited to this embodiment, and the process shown in FIG. 5 according to the various embodiments described above It is obvious to those skilled in the art that the process of transmitting and receiving data can be changed.

Referring to FIG. 5, the guidance service providing server receives content that can identify the product from the user terminal (S5100), checks whether the application for the guidance service is installed on the user terminal, and, if the application is not installed, installs the user terminal. Send the URL to install the application (S5200).

And, when the application is installed on the user terminal, the information service providing server extracts the unique identification code already mapped and stored with the content that can identify the product, and reads the product user manual previously mapped and stored with the unique identification code (S5300 ).

Finally, the information service providing server converts the text in the product manual into voice (Text To Speech) and transmits it to the user terminal (S5400).

The sequence between the above-described steps (S5100 to S5400) is only an example and is not limited thereto. That is, the order between the above-described steps (S5100 to S5400) may change, and some of the steps may be executed simultaneously or deleted.

Matters that are not explained regarding the method of providing the voice-based product user manual guide service of FIG. 5 are the same as or are derived from the content explained about the method of providing the voice-based product user manual guide service through FIGS. 1 to 4 above. Since it can be easily inferred, the description below will be omitted.

The method of providing a voice-based product user manual guide service according to an embodiment described with reference to FIG. 5 may also be implemented in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. You can. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The method of providing a voice-based product user manual guide service according to an embodiment of the present invention described above includes an application installed by default on a terminal (this may include programs included in a platform or operating system, etc., installed by default on the terminal). It may be executed by an application (i.e., a program) installed by the user directly on the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method of providing a voice-based product user manual guide service according to an embodiment of the present invention described above is implemented as an application (i.e., a program) installed by default in the terminal or directly installed by the user, and is implemented by a computer such as the terminal. It can be recorded on a readable recording medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (10)

In the method of providing a guidance service executed on a guidance service providing server,
Searching for at least one type of IoT device linked to a network to which the user terminal is connected;
Extracting the product model name of the at least one type of IoT device, extracting a product user manual previously mapped and stored with the product model name, and storing the product to be mapped to the user terminal;
Receiving content that can identify a product from the user terminal;
Checking whether an application for a guidance service is installed on the user terminal, and if the application is not installed, transmitting a URL to install the application to the user terminal;
When the application is installed on the user terminal, extracting a unique identification code previously mapped and stored to identify the product, and reading a product user manual previously mapped and stored with the unique identification code; and
Converting the text in the product user manual into voice (Text To Speech) and transmitting it to the user terminal,
The step of receiving content that can identify a product from the user terminal,
It includes the process of selecting one IoT device from at least one type of IoT device linked on a network to which the user terminal is connected,
The step of converting the text in the product user manual into voice (Text To Speech) and transmitting it to the user terminal,
Including the process of providing a streaming output service for the audio data converted into voice to the user terminal,
The user terminal is,
Using AR content synchronized with the audio data, the location of the member or part explained in the current product user manual is displayed through AR content.
The information service providing server is,
When receiving a voice utterance from the user terminal, a corpus or customer word dictionary is built by collecting wordings included in the query and names and keywords incorrectly queried by the user, and the correct keywords or search words are entered into the constructed corpus or customer word dictionary. After mapping and saving, the product user manual mapped to the correct keyword is output even if the user utters the wrong name.
The search term, result image data, and search log searched in the user terminal are stored as history data, the start and end points of the search related to the search term are distinguished, and the time-series change in the search term from the start to the end point is recorded in Big Data. Data is collected, and when a search term corresponding to the starting point is entered as a query, a shortcut search is provided in which results searched with the search term at the end, which is the part corresponding to the response in the product user manual, are output. A method of providing voice-based product user manual guidance service.
delete delete delete delete delete delete delete delete delete

Images