KR20230122394A - Customized feedback lighting system through classification of user emotions based on Wake-Up Words - Google Patents

Abstract

A customized feedback lighting system through activation word-based user emotion classification is disclosed. Classifying the user's emotion by analyzing the tone elements of the trigger word uttered by the user to call a voice assistant, and lighting feedback in a mode mapped to the corresponding emotion based on the user's emotion can provide.

Description

Customized feedback lighting system through classification of user emotions based on Wake-Up Words}

The description below relates to techniques for providing lighting feedback.

Research and commercialization technology for Voice User Interface (VUI), which is a method of controlling devices using human voice, has recently been more actively conducted along with the popularization of voice recognition devices such as artificial intelligence speakers. VUI is easier and more intuitive to control compared to physical buttons (Physical User Interface) or screen (Graphic User Interface) because users use everyday terms. As the biggest technical interest of VUI lies in understanding the speaker's voice accurately and quickly, it is a natural result that voice recognition technology and natural language processing technology are being actively grafted to improve the accuracy of VUI.

This has led to the development of assistants that operate based on voice, and electronic device manufacturers have connected mobile phones to vehicles so that existing assistants can be easily used in vehicles. In addition, since the voice assistant contains the symbolism of the brand, the wake-up word for running the assistant is often produced in association with the brand name or symbol. For example, in the case of Mercedes, "Hi , Mercedes!" I'm using the trigger word. In particular, the reason why the trigger word is attracting attention for controlling in-vehicle devices is that the trigger word is directly related to the driver's safety because it is easy to call the in-vehicle while focusing attention on the front while driving. Therefore, it is common to activate an interface using a startup word to call an in-vehicle voice assistant.

Voice data input into the device is converted into text (Speech-to-Text; hereinafter STT), and the voice assistant is used for the purpose of understanding words or even context, but it can also be used to understand the speaker's emotional state implied in voice. . A number of studies related to Speech-Emotion Recognition (SER) grafted with artificial intelligence analysis technology have been published, and development has been repeated by reflecting the characteristics of each language group or improving the analysis method to advance the technology. .

In vehicles with level 3 or lower autonomous driving, vehicle interfaces consider the driver's safety as the top priority. Therefore, many studies are being conducted on the interaction between emotional satisfaction and the driver's driving ability when mounting an interface or technology in a vehicle.

As artificial intelligence technology develops and SER technology through voice interface becomes increasingly popular, research on technology for distinguishing users from biometric data including voice and facial expressions is being actively conducted.

Existing technologies grasp emotion based on conversation content, so the content of conversation has a great influence on emotion judgment, resulting in a large difference in accuracy between users. Therefore, there is a limit to immediately leading to service.

A user's emotion may be extracted from an activation word used by the user to call a voice assistant, and lighting feedback of a mode suitable for the user's emotion may be provided.

A custom feedback lighting method performed in a computer device, wherein the computer device includes at least one processor configured to execute computer readable instructions contained in a memory, and the customized feedback lighting method is performed by the at least one processor. classifying the user's emotion by analyzing a tone element of a trigger word uttered by the user in order to call a voice assistant; and providing, by the at least one processor, lighting feedback of a mode mapped to the emotion of the user based on the emotion of the user.

According to one aspect, the classifying step includes classifying at least one characteristic of age and gender of the user from tone elements of a trigger word uttered by the user, and the providing step comprises: The lighting feedback may be provided in a mode mapped to emotions and characteristics.

According to another aspect, the classifying step includes the step of classifying at least one characteristic of age and gender of the user from the timbre element of the start word uttered by the user, and the providing step comprises: Selecting a lighting mode according to the emotion, and selecting a lighting mode according to the user's characteristic when the user's emotion corresponds to at least some type of emotion.

According to another aspect, the providing may include providing lighting feedback in which at least one of lighting type, color, saturation, purity, color temperature, and pattern differs according to the user's emotion.

According to another aspect, the customized feedback lighting method further comprises, by the at least one processor, collecting a voice file for the trigger word to build a learning dataset for emotion classification, wherein the classification In the step, the emotion of the user may be classified from the activation word uttered by the user through machine learning using the learning dataset.

According to another aspect, in the step of classifying, among the timbre elements of the start word uttered by the user, the duration at which the voice is recorded, the post-processed pitch, and the roughness through changes in volume ), and the user's emotion may be classified using the main vocal range domain (Dom).

A computer program stored in a computer readable recording medium to execute a customized feedback lighting method on a computer, wherein the customized feedback lighting method analyzes the tone elements of a trigger word uttered by the user to call a voice assistant, so that the user Classifying the emotions of; And providing a computer program comprising the step of providing lighting feedback of a mode mapped to the emotion based on the emotion of the user.

A computer-implemented custom feedback lighting system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor comprising: a trigger word uttered by a user to invoke a voice assistant; classifying the emotion of the user by analyzing the timbre elements of; And it provides a customized feedback lighting system that handles a process of providing lighting feedback of a mode mapped to the corresponding emotion based on the user's emotion.

According to embodiments of the present invention, it is possible to more quickly and accurately grasp emotions by utilizing a trigger word used to call a voice assistant to analyze a user's emotion.

According to the embodiments of the present invention, it is convenient to build a learning dataset by conducting machine learning focusing on the trigger word rather than the entire conversation, and it is possible to judge the emotion more accurately than the sentence without additional analysis of the sentence. .

According to embodiments of the present invention, it is possible to provide a more clear guide to the feedback emotion illumination provided based on the emotion extracted from the trigger word, and through this, it is possible to improve service satisfaction.

1 is a block diagram for explaining an example of an internal configuration of a computer device according to an embodiment of the present invention.
Figure 2 shows a schematic diagram of a system for providing feedback emotional lighting in one embodiment of the present invention.
3 illustrates an example of user style and emotion classification according to an embodiment of the present invention.
4 illustrates an example of user-centered emotion scenario mapping according to an embodiment of the present invention.
5 illustrates an example of a feedback lighting guide for each user emotion mapping type in one embodiment of the present invention.
6 illustrates examples of situations used in trigger word collection for machine learning in one embodiment of the present invention.
7 illustrates an example of a model for predicting a representative emotion for a voice recording of a trigger word according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the invention relate to techniques for providing lighting feedback.

Embodiments including those specifically disclosed herein may extract a user's emotion from an activation word used by the user to call a voice assistant (voice assistant) and provide lighting feedback in a mode suitable for the extracted emotion. .

1 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention. For example, a customized feedback lighting system according to embodiments of the present invention may be implemented by the computer device 100 shown in FIG. 1 .

As shown in FIG. 1, a computer device 100 is a component for executing a customized feedback lighting method according to embodiments of the present invention, and includes a memory 110, a processor 120, a communication interface 130, and input/output. Interface 140 may be included.

The memory 110 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device 100 as a separate permanent storage device distinct from the memory 110. Also, an operating system and at least one program code may be stored in the memory 110 . These software components may be loaded into the memory 110 from a recording medium readable by a separate computer from the memory 110 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 110 through the communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer device 100 based on a computer program installed by files received over network 160 .

The processor 120 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 120 by memory 110 or communication interface 130 . For example, processor 120 may be configured to execute received instructions according to program codes stored in a recording device such as memory 110 .

The communication interface 130 may provide functions for the computer device 100 to communicate with other devices through the network 160 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 110 by the processor 120 of the computer device 100 is transmitted to the network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 100 via the communication interface 130 of the computer device 100 via the network 160 . Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or the memory 110, and files, etc. may be stored as storage media that the computer device 100 may further include (described above). permanent storage).

The communication method is not limited, and may include not only a communication method utilizing a communication network (eg, a mobile communication network, wired Internet, wireless Internet, and broadcasting network) that the network 160 may include, but also short-distance wired/wireless communication between devices. there is. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 160 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

The input/output interface 140 may be a means for interface with the input/output device 150 . For example, the input device may include devices such as a microphone, keyboard, camera, or mouse, and the output device may include devices such as a display and a speaker. As another example, the input/output interface 140 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 150 and the computer device 100 may be configured as one device.

Also, in other embodiments, computer device 100 may include fewer or more elements than those of FIG. 1 . However, there is no need to clearly show most of the prior art components. For example, the computer device 100 may be implemented to include at least a portion of the above-described input/output device 150 or may further include other components such as a transceiver, a camera, various sensors, and a database.

The present embodiments provide a customized feedback lighting system that provides feedback emotion lighting through activation word-based user emotion classification.

The customized feedback lighting system according to the present invention can be applied not only to vehicles but also to all devices that call a voice assistant through a triggering word. Hereinafter, vehicle lighting is described, but is exemplary only and is not limited thereto.

Figure 2 shows a schematic diagram of a system for providing feedback emotional lighting in one embodiment of the present invention.

Referring to FIG. 2 , in step S210, the processor 120 may call the voice assistant of the vehicle using a trigger word uttered by a user (driver or passenger) inside the vehicle.

In step S220, the processor 120 may determine the user's style (age and gender) and emotion from the tone elements of the machine learning-based trigger word.

In step S230, the processor 120 may determine a user-centered emotional scenario based on the user's style and emotion determined in step S220.

In step S240, the processor 120 may provide emotional lighting feedback corresponding to a mode selected as a user-centered emotional scenario as dynamic lighting.

According to the present embodiments, a user's style and emotion may be extracted only by analyzing a trigger word for driving a voice assistant of a vehicle. By analyzing only the trigger word, which is the same sentence for everyone, rather than the dialogue sentences that change every time, it is possible to grasp emotions more quickly. can be used

3 illustrates an example of user style and emotion classification according to an embodiment of the present invention.

Referring to FIG. 3 , the processor 120 may determine the user's emotion and style based on machine learning from the tone elements of the activation word in which the user calls the voice assistant. For example, emotions can be classified into 5 emotions (excited, angry, depressed, calm, everyday), and age, which is one of the characteristics that can identify a user's style, can be classified into adults and children, Gender can be classified as male and female. These classification criteria are illustrative only and are not limited thereto.

4 illustrates an example of user-centered emotion scenario mapping according to an embodiment of the present invention.

Referring to FIG. 4 , the processor 120 may perform primary classification according to emotion by analyzing tone elements of a trigger word uttered by a user, and may determine age and sensitivity for at least some emotions according to the emotion type according to the primary classification. Secondary classification can be performed based on characteristics such as gender. For example, the processor 120 may classify the user's emotion as one of five emotions (excited, angry, depressed, calm, and everyday). At this time, the processor 120 may classify the age of the user as either a child or an adult when the user's emotion is determined to be 'exciting' emotion. Meanwhile, the processor 120 may classify the user's gender as one of female and male when the user's emotion is determined to be 'true' emotion. The processor 120 may determine a customized scenario based on the user's emotion and style (characteristic). For example, user-centered emotion scenarios can be classified into toy, surprise, tension, comfort, romance, and default. Playful mode when classified as 'young child', Surprise mode when classified as 'exciting' emotion and 'adult', Nervous mode when classified as 'angry' emotion, Consolation mode when classified as 'gloomy' emotion, 'calm down' mode It can be mapped to romance mode when classified as 'emotion and 'female', basic mode when classified as 'calm' emotion and 'male', and basic mode when classified as 'everyday' emotion.

5 illustrates an example of a feedback lighting guide for each user emotion mapping type in one embodiment of the present invention.

Referring to FIG. 5 , the processor 120 may provide emotional lighting feedback based on a selected mode by mapping the user's emotion and style (characteristic) to a user-centered emotion scenario. Different types of lighting, color, saturation, purity, color temperature, pattern, etc. may be applied to each mode according to a user-centered emotional scenario. For example, when a user-centered emotional scenario is selected as a playful mode, feedback lighting of a pattern that is converted into various colors of high saturation and purity may be provided. When a user-centered emotional scenario is selected as a surprise mode, a direct light having a high color temperature may provide feedback lighting in a gradually brightening pattern. When the user-centered emotional scenario is selected as the tension mode, feedback lighting using a low color temperature and red-based floor and ceiling indirect lighting can be provided. When the user-centered emotional scenario is selected as the up mode, feedback lighting using a low color temperature and red-type ambient light around the driver can be provided. When the user-centered emotional scenario is selected as the romance mode, feedback lighting using pink or orange vehicle interior lighting having low purity may be provided. When the user-centered emotional scenario is selected as the basic mode, feedback lighting using unique style lighting designed in colors for each vehicle type and brand can be provided. In this way, when the user is in an emotional state that is expected to be angry, indirect lighting with a low color temperature and soft pattern lighting having a calming effect are provided to calm the heightened emotion, while the user's emotional state is predicted to be sad It is possible to respond, such as providing light feedback that seems to be wrapped in a warm embrace.

6 illustrates examples of situations used in trigger word collection for machine learning in one embodiment of the present invention.

The processor 120 may apply a total of five classifications of excitement, anger, depression, calm, and daily life in which characteristics of positive and arousal dimensions are complexly reflected in classification of emotion for each situation.

In order to build a dataset for machine learning, it is possible to collect trigger words of various emotions targeting the experimental group. Before and after situations are set and provided so that the emotion of calling the trigger word can come out naturally, and the sentence following the trigger word is recorded together to make it easier to record a sentence containing emotion during recording. In the recording, the positive and arousal characteristics are evenly considered, and 21 situations considered to be easy to record can be selected (Table of FIG. 6).

Unlike everyday language, mobile language has the same number of phonemes, syllables, and words, so it is easy for artificial intelligence models to recognize with high accuracy.

First, only the activation word voice file can be extracted through a preprocessing process of extracting only the activation word part (eg, Hi, KIA) of the recorded file. Then, for analysis, an open source package that synthesizes non-verbal vocalizations based on meaningful acoustic parameters that can be specified on the command line or in an interactive app is used to identify acoustic properties from various analysis perspectives. 37 acoustic properties can be extracted. 37 acoustic property values are extracted from the extracted start words at every 25 ms interval, and if there is no calculated value among the 37 acoustic property values, the entire extracted value corresponding to that time is excluded from the observation value. As a result, it is possible to obtain extraction values for an average of 18.84 (standard deviation: 4.95) times for one trigger word recording. In other words, as shown in FIG. 7 , emotion classification may be applied every 25 ms to one trigger word voice recording, and at this time, the prediction result may be determined based on the emotion classified with the highest frequency among the classification results.

If the number of extracted values is less than 10, it is judged that the recorded file has damage or is not easy to analyze and is excluded from analysis. As a result, we model a machine learning-based algorithm that learns the acoustic properties of the recorded files collected from the participants in the experimental group, and determine the accuracy of predicting emotion from the trigger word.

The average length of the start words collected in this experiment is 0.58 seconds, which is a very short length compared to the dataset data for machine learning to analyze the emotional characteristics of speech with artificial intelligence technology. That is, the amount of voice attributes that can be collected from a trigger word may be limited in order to classify discriminating voice attributes for each scenario. If so, it is possible to proceed with learning about exciting emotions for the entire recorded sentence and compare the result determined from the start word and its accuracy. Accuracy can be calculated after machine learning by repeating the same process for each sentence in each scenario.

In the machine learning process, after selecting features with high influence using the Boruta function, the emotion classification algorithm can be modeled with only the selected features. Based on the value calculated as mean importance for each operation, the importance of the features used in the algorithm modeling was recorded through duration, post-processed pitch, and roughness through changes in sound volume ( roughness), and the main musical range (Dom) can be selected.

As described above, according to embodiments of the present invention, it is possible to more quickly and accurately grasp emotions by utilizing a trigger word used to call a voice assistant for emotion analysis of a user. In addition, according to the embodiments of the present invention, by conducting machine learning focusing on trigger words rather than entire conversations, it is convenient to build a learning dataset and it is possible to determine more accurate emotion than sentences without additional analysis of sentences. And, according to the embodiments of the present invention, it is possible to provide a clearer guide for the feedback emotion lighting provided based on the emotion extracted from the trigger word, thereby improving service satisfaction.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. there is. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (13)

In the customized feedback lighting method performed on a computer device,
The computer device includes at least one processor configured to execute computer readable instructions contained in a memory;
The customized feedback lighting method,
classifying, by the at least one processor, the emotion of the user by analyzing a tone element of a trigger word uttered by the user to call a voice assistant; and
Providing, by the at least one processor, lighting feedback of a mode mapped to the emotion of the user based on the emotion of the user
Custom feedback lighting method comprising a. According to claim 1,
The classification step is
Classifying at least one characteristic of the age and gender of the user from the tone elements of the starter word uttered by the user
including,
The step of providing,
Providing the lighting feedback in a mode mapped to the user's emotion and characteristics
Characterized by a custom feedback lighting method. According to claim 1,
The classification step is
Classifying at least one characteristic of the age and gender of the user from the tone elements of the starter word uttered by the user
including,
The step of providing,
Selecting a lighting mode according to the user's emotion, but selecting a lighting mode according to the user's characteristic when the user's emotion corresponds to at least some type of emotion
Custom feedback lighting method comprising a. According to claim 1,
The step of providing,
Providing lighting feedback in which at least one of the lighting type, color, saturation, purity, color temperature, and pattern differs according to the user's emotion
Characterized by a custom feedback lighting method. According to claim 1,
The customized feedback lighting method,
Building, by the at least one processor, a learning dataset for emotion classification by collecting voice files for the trigger word
Including more,
The classification step is
Classifying the emotion of the user from the trigger word uttered by the user through machine learning using the learning dataset
Characterized by a custom feedback lighting method. According to claim 1,
The classification step is
Among the timbre elements of the trigger word uttered by the user, the duration of the voice was recorded, the post-processed pitch, the roughness through the change in loudness, and the main range of voice (Dom) Classifying the user's emotion
Characterized by a custom feedback lighting method. A computer program stored in a computer readable recording medium for executing a customized feedback lighting method on a computer,
The customized feedback lighting method,
classifying a user's emotion by analyzing a tone element of a trigger word uttered by the user to call a voice assistant; and
Providing lighting feedback of a mode mapped to the corresponding emotion based on the user's emotion
Including, a computer program. In the customized feedback lighting system implemented by a computer,
at least one processor configured to execute computer readable instructions contained in memory;
including,
The at least one processor,
classifying a user's emotion by analyzing a tone element of a trigger word uttered by the user to call a voice assistant; and
A process of providing lighting feedback of a mode mapped to a corresponding emotion based on the user's emotion
A custom feedback lighting system that handles According to claim 8,
The at least one processor,
classifying at least one characteristic of the user's age and gender from the tone elements of the starter word uttered by the user;
Providing the lighting feedback in a mode mapped to the user's emotion and characteristics
featuring a custom feedback lighting system. According to claim 8,
The at least one processor,
classifying at least one characteristic of the user's age and gender from the tone elements of the starter word uttered by the user;
Selecting a lighting mode according to the user's emotion, but selecting a lighting mode according to the user's characteristic when the user's emotion corresponds to at least some type of emotion
featuring a custom feedback lighting system. According to claim 8,
The at least one processor,
Providing lighting feedback in which at least one of the lighting type, color, saturation, purity, color temperature, and pattern differs according to the user's emotion
featuring a custom feedback lighting system. According to claim 8,
The at least one processor,
Collecting voice files for the trigger word to build a learning dataset for emotion classification,
Classifying the emotion of the user from the trigger word uttered by the user through machine learning using the learning dataset
featuring a custom feedback lighting system. According to claim 8,
The at least one processor,
Classifying the emotion of the user by using the recording time of the voice, the post-processed pitch, the degree of roughness through the change in loudness, and the main vocal range among the timbre elements of the trigger word uttered by the user
featuring a custom feedback lighting system.

Images