CN114783440A - User interaction assisting method and device, electronic equipment and readable storage medium - Google Patents

User interaction assisting method and device, electronic equipment and readable storage medium Download PDF

Info

Publication number
CN114783440A
CN114783440A CN202210396401.8A CN202210396401A CN114783440A CN 114783440 A CN114783440 A CN 114783440A CN 202210396401 A CN202210396401 A CN 202210396401A CN 114783440 A CN114783440 A CN 114783440A
Authority
CN
China
Prior art keywords
sign language
user
information
audio
glasses
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210396401.8A
Other languages
Chinese (zh)
Inventor
王政轩
张方方
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Goertek Techology Co Ltd
Original Assignee
Goertek Techology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Goertek Techology Co Ltd filed Critical Goertek Techology Co Ltd
Priority to CN202210396401.8A priority Critical patent/CN114783440A/en
Publication of CN114783440A publication Critical patent/CN114783440A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/24—Speech recognition using non-acoustical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00—Details of transducers, loudspeakers or microphones
    • H04R1/10—Earpieces; Attachments therefor ; Earphones; Monophonic headphones
    • H04R1/1058—Manufacture or assembly
    • H04R1/1066—Constructional aspects of the interconnection between earpiece and earpiece support
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2460/00—Details of hearing devices, i.e. of ear- or headphones covered by H04R1/10 or H04R5/033 but not provided for in any of their subgroups, or of hearing aids covered by H04R25/00 but not provided for in any of its subgroups
    • H04R2460/13—Hearing devices using bone conduction transducers

Abstract

The application discloses a user interaction assisting method, a user interaction assisting device, electronic equipment and a readable storage medium, which are applied to AR glasses, wherein the AR glasses are provided with bone conduction earphones, and the user interaction assisting method comprises the following steps: picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio; if the picked audio is effective audio, converting the sign language action into corresponding natural language information for playing when the sign language action of the outside is detected so as to assist the wearing user in interacting with the outside; and if the picked audio is not effective audio, displaying each preset answer corresponding to the interactive information in an AR display picture of the AR glasses when the interactive information of the outside is detected so as to assist the wearing user to interact with the outside. The method and the device solve the technical problem that in the prior art, the deaf-mute user and the outside have communication disorder.

Description

User interaction assisting method and device, electronic equipment and readable storage medium
Technical Field
The present application relates to the field of AR (Augmented Reality) technologies, and in particular, to a user interaction assistance method and apparatus, an electronic device, and a readable storage medium.
Background
With the development of the AR technology, the scientific technology provides more improvement for the communication and exchange of people, and the deaf-mute user has defects, and usually needs to communicate with the outside through sign language, but if the communication target of the deaf-mute user does not learn the sign language, the communication target of the deaf-mute user usually cannot understand the will that the deaf-mute user wants to express, and the communication between the deaf-mute user and the outside is obstructed.
Disclosure of Invention
The application mainly aims to provide a user interaction assisting method, a user interaction assisting device, electronic equipment and a readable storage medium, and aims to solve the technical problem that communication obstacles exist between a deaf-mute user and the outside in the prior art.
In order to achieve the above object, the present application provides a user interaction assistance method, which is applied to AR glasses, where the AR glasses are provided with bone conduction earphones, and the user interaction assistance method includes:
picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio;
if the picked audio is effective audio, converting the sign language action into corresponding natural language information for playing when the sign language action of the outside is detected so as to assist the wearing user in interacting with the outside;
and if the picked audio is not effective audio, displaying each preset answer corresponding to the interaction information in an AR display picture of the AR glasses when the interaction information of the outside is detected so as to assist the wearing user in interacting with the outside.
For realizing above-mentioned purpose, this application still provides a user interaction auxiliary device, is applied to AR glasses, AR glasses are provided with the bone conduction earphone, user interaction auxiliary device includes:
the audio pickup module is used for picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain picked-up audio;
the first interaction mode module is used for converting the sign language action into corresponding natural language information to play when the sign language action of the outside is detected if the picked audio is effective audio so as to assist the wearing user in interacting with the outside;
and the second interaction mode module is used for displaying each preset answer corresponding to the interaction information in an AR display picture of the AR glasses when the external interaction information is detected if the picked-up audio is not the effective audio, so as to assist the wearing user in interacting with the outside.
The present application further provides an electronic device, the electronic device is an entity device, the electronic device includes: a memory, a processor and a program of the user interaction assistance method stored on the memory and executable on the processor, the program of the user interaction assistance method realizing the steps of the user interaction assistance method as described above when executed by the processor.
The present application further provides a computer-readable storage medium having stored thereon a program for implementing the user interaction assistance method, which when executed by a processor, implements the steps of the user interaction assistance method as described above.
The present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the user interaction assistance method as described above.
The application provides a user interaction assisting method, a user interaction assisting device, electronic equipment and a readable storage medium, which are applied to AR glasses, wherein the AR glasses are provided with bone conduction earphones, namely, the bone conduction earphones are used for picking up the audio of a user wearing the AR glasses to obtain the picked-up audio; if the picked audio is effective audio, the wearing user is proved to be communicated with the outside through bone conduction, the deaf-mute degree of the wearing user is low, and then when the external sign language action is detected, the external sign language action can be converted into corresponding natural language information to be played, so that the interaction between the wearing user and the outside is assisted; if the picked-up audio is not effective audio, the wearing user is proved not to be communicated with the outside through bone conduction, the deaf-mute degree of the wearing user is high, and then when outside interactive information is detected, preset answer responses corresponding to the interactive information are displayed in an AR display picture of the AR glasses to assist the wearing user to interact with the outside, so that an interactive auxiliary mode which is matched with the wearing user according to the deaf-mute degree of the wearing user is realized, and the obstacle of communication between the deaf-mute user and the outside under different deaf-mute degrees is overcome in the corresponding interactive auxiliary mode.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive labor.
FIG. 1 is a schematic flowchart illustrating a first embodiment of a user interaction assistance method according to the present application;
FIG. 2 is a flowchart illustrating a second embodiment of a user interaction assistance method according to the present application;
fig. 3 is a schematic device structure diagram of a hardware operating environment related to a user interaction assistance method in an embodiment of the present application.
The implementation of the objectives, functional features, and advantages of the present application will be further described with reference to the accompanying drawings.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, embodiments accompanying figures are described in detail below. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without any creative effort belong to the protection scope of the present application.
Example one
The embodiment of the application provides a user interaction assisting method, which is applied to AR glasses in a first embodiment of the user interaction assisting method, wherein the AR glasses are provided with bone conduction earphones, and the user interaction assisting method comprises the following steps:
step S10, picking up the audio of the user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio;
step S20, if the picked-up audio is effective audio, converting the sign language action into corresponding natural language information for playing when the sign language action of the outside is detected so as to assist the wearing user in interacting with the outside;
step S30, if the picked-up audio is not an effective audio, when external interaction information is detected, displaying each preset answer corresponding to the interaction information in an AR display screen of the AR glasses to assist the wearing user in interacting with the outside.
In this embodiment, it should be noted that the user interaction assisting method is applied to AR glasses, and a wearing user of the AR glasses may be a normal person, which does not belong to a deaf-mute group. In the deaf-mute population, a part of users can hear external sound through bone conduction and make corresponding feedback, so that the part of users can interact with the outside through bone conduction earphones. Meanwhile, in the deaf-mute population, a part of users cannot hear external sound and make corresponding feedback even in a bone conduction mode, and the part of users cannot hear the external sound and cannot make sound at all, so that the part of deaf-mute people and the external world have more serious communication barrier, and the degree of deaf-mute is higher, so that the deaf-mute users with different degrees of deaf-mute exist in the deaf-mute population.
As an example, step S10 includes: and picking up the audio frequency sent by the wearing user of the AR glasses through the bone conduction earphone to obtain a picked-up audio frequency, and further judging whether the picked-up audio frequency is an effective audio frequency.
As an example, the step of judging whether the picked-up audio is valid audio comprises:
extracting the audio features of the picked audio, and carrying out secondary classification on the picked audio according to the audio features to obtain two classification labels; and judging whether the picked audio is effective audio or not according to the two classification labels, for example, when the two classification labels are set to be 1, the picked audio is identified as the effective audio, and when the two classification labels are set to be 0, the picked audio is identified as not the effective audio.
As an example, the step of judging whether the picked-up audio is valid audio further comprises:
picking up audio sent by a user wearing the AR glasses through an air conduction microphone to obtain second picked-up audio; judging whether the wearing user can send out effective audio or not according to the picked audio and the second picked audio, and if the wearing user can send out effective audio, executing the following steps: when external sign language actions are detected, converting the sign language actions into corresponding natural language information to be played so as to assist the wearing user in interacting with the outside; if the wearing user can not send out effective audio, executing the following steps: when the external interactive information is detected, displaying each preset answer corresponding to the interactive information in an AR display picture of the AR glasses so as to assist the wearing user in interacting with the outside. Whether the wearing user can send the effective audio frequency or not can be comprehensively judged through 2 ways of the air conduction microphone and the bone conduction microphone in the embodiment of the application, and the reliability and the accuracy of effective audio frequency judgment are improved.
As an example, the determining whether the wearing user can emit valid audio according to the picked-up audio and the second picked-up audio includes:
respectively extracting the audio features of the picked audio and the second picked audio to obtain a first audio feature corresponding to the picked audio and a second audio feature corresponding to the second picked audio; classifying the picked audios according to the first audio characteristics to obtain first and second classification labels, and classifying the second picked audios according to the second audio characteristics to obtain second classification labels; and judging whether the wearing user can send the effective audio or not according to the first and second classified labels, for example, when the second classified label is set to be 1, the picked audio is identified as the effective audio, when the second classified label is set to be 0, the picked audio is not identified as the effective audio, so that when a preset two separate labels of 1 exist in the first and second classified labels, the wearing user can be judged to send the effective audio, and when the preset two separate labels of 1 do not exist in the first and second classified labels, the wearing user can not send the effective audio.
As one example, steps S20 to S30 include: if the picked-up audio is effective audio, the wearing user can communicate with the outside in a bone conduction mode, further, when external sign language actions are detected, the sign language actions are converted into corresponding natural language information, and the natural language information is played to the wearing user through the bone conduction earphone so as to assist the wearing user in interacting with the outside; if the picked audio is not effective audio, it is proved that the wearing user can not communicate with the outside by means of bone conduction, and then when the outside interactive information is detected, preset reply answers corresponding to the interactive information are displayed in an AR display picture of the AR glasses, so that a target reply answer selected by the wearing user at each preset reply answer is determined, and the target reply answer is broadcasted, and therefore the wearing user is assisted to interact with the outside. According to the embodiment of the application, different interaction modes are provided for the deaf-mute users according to different deaf-mute degrees of the deaf-mute users, when the deaf-mute degrees of the deaf-mute users are low, sign language actions are converted into natural language information through a sign language conversion mode, and the natural language information is played to wearing users in a bone conduction mode; when the deaf-mute degree of the deaf-mute user is higher, the preset answer corresponding to the interactive information is directly displayed to the wearing user in a visual mode through an answer fixing mode, so that the wearing user can interact with the outside in a mode of selecting a target to reply the answer, therefore, the user interaction efficiency and the simplicity of the answer fixed mode are obviously lower than those of the sign language conversion mode, but the sign language conversion mode with higher interaction efficiency can be automatically provided for deaf-mute users with lower deaf-mute degrees on the one hand by judging the deaf-mute degrees of the deaf-mute users in advance in the embodiment of the application, the interaction efficiency is ensured while the communication barrier between the deaf-mute user and the outside is overcome, on the other hand, an answer fixed mode with higher interaction accuracy is automatically provided for the deaf-mute user with higher deaf-mute degree, therefore, when the deaf-mute user has a high degree of deaf-mute, the communication barrier between the deaf-mute user and the outside can be overcome.
Wherein, the step of converting the sign language action into corresponding voice information for playing comprises:
step S21, sign language recognition is carried out on the sign language action to obtain a sign language recognition result;
step S22, generating natural language information corresponding to the sign language action according to the sign language identification result;
and step S23, playing the natural language information to the wearing user of the AR glasses.
As an example, the steps S21 to S23 include: shooting the outside through AR glasses to obtain an outside image; identifying whether sign language action exists in the external image, and if so, judging that the sign language action is detected; segmenting a sign language action image from the external image, classifying the sign language action image to identify a sign language action in the sign language action image to obtain a sign language action tag, and using the sign language action tag as the sign language identification result, wherein the sign language action tag is a category tag of the sign language action and is used for identifying the semantic category of the sign language action; according to the mapping relation between the sign language action tag and the natural language information, determining the natural language information corresponding to the sign language action tag; and playing the natural language information to the wearing user of the AR glasses, wherein the playing mode can be a voice broadcasting mode or an AR display picture mode.
The natural language information at least comprises one of broadcast voice and a text display picture, and the step of generating the natural language information corresponding to the sign language action according to the sign language recognition result comprises the following steps:
step A10, converting the sign language action into corresponding voice information according to the sign language recognition result, and generating broadcast voice corresponding to the voice information;
in this embodiment, it should be noted that the number of sign language actions may be one or more. The sign language recognition result can be one or more sign language action labels.
As an example, step a10 includes: determining text word labels corresponding to the sign language action labels in the sign language recognition result according to the corresponding relation between the sign language action labels and the text word labels, wherein the text word labels are semantic labels of the text words and are used for identifying the semantics of the text words; sequencing the text word labels according to the common corresponding sign language word order of each sign language action to obtain a text word label sequence; determining audio sequence data corresponding to the text word tag sequence according to semantic correspondence between the text word tags and audio, wherein the audio sequence data are voice information corresponding to each sign language action, and specifically can be a feature sequence formed by audio features corresponding to each text word; and generating real voice corresponding to the audio sequence data as broadcast voice.
And step B10, converting the sign language action into corresponding text information according to the sign language identification result, and generating a text display picture corresponding to the text information.
As an example, the text information may be a text sentence, and step a20 includes: determining text words corresponding to the sign language action labels in the sign language recognition result according to the corresponding relation between the sign language action labels and the text word labels; sequencing each text word according to the sign language word order corresponding to each sign language action together to obtain a text sentence; and generating an AR display picture with the text sentence as a text display picture.
Wherein, the natural language information includes at least one of broadcast voice and text display picture, to wearing the user broadcast of AR glasses the step of natural language information includes:
step C10, playing the broadcast voice to the wearing user through the bone conduction headset;
step D10, displaying the text display screen to the wearing user via the AR display screen of the AR glasses.
As an example, step D10 includes: and determining a corresponding image area to be replaced in the AR display picture, replacing the image area to be replaced in the AR display picture with the text display picture to obtain a fused AR display picture, and displaying the fused AR display picture. The image area to be replaced may be the entire AR display screen, or a partial image area in the AR display screen, and specifically may be an image area in the AR display screen that does not affect the AR experience of the user, for example, an edge area of the AR display screen.
The step of displaying each preset answer corresponding to the interactive information in an AR display frame of the AR glasses includes:
step S31, carrying out voice recognition on the voice interaction information to obtain a voice recognition result;
step S32, determining each preset answer corresponding to the voice interaction information according to the voice recognition result;
step S33, displaying each preset answer in the AR display screen.
In this embodiment, it should be noted that the voice recognition result may be a semantic feature corresponding to the voice interaction information, and the semantic feature may be an embedded feature that identifies a semantic of the voice interaction information, that is, embedding.
As an example, the steps S31 to S33 include:
semantic feature extraction is carried out on the voice interaction information to obtain semantic features corresponding to the voice interaction information; determining a feature with the highest similarity with the semantic features in all preset semantic features as a target semantic feature; determining each preset answer corresponding to the target semantic features according to the mapping relation between the preset semantic features and the preset answer; displaying each preset answer corresponding to the target semantic feature in an AR display picture of the AR glasses, and detecting feedback information of the wearing user to each preset answer corresponding to the target semantic feature. Further selecting a target answer reply from each preset answer reply corresponding to the target semantic feature according to the feedback information, where the feedback information may be click information of the wearing user on each displayed preset answer reply, for example, there are 3 preset answer replies a, B, and C, and if the wearing user clicks C, the C is the target answer reply; and playing the target answer through a microphone. The purpose of assisting the deaf-mute people to make corresponding feedback on the external information is achieved in the embodiment of the application, even if the deaf-mute people can not make any sound and can not hear any sound completely, namely more serious communication obstacle exists with the outside, the embodiment of the application can assist the deaf-mute users to feed back the external information in time, so that communication with the outside is completed, and further the communication obstacle between the deaf-mute people and the outside is overcome.
The embodiment of the application provides a user interaction assisting method, which is applied to AR glasses, wherein the AR glasses are provided with bone conduction earphones, namely, the bone conduction earphones are used for picking up the audio of a user wearing the AR glasses to obtain the picked-up audio; if the picked audio is effective audio, the wearing user is proved to be communicated with the outside through bone conduction, the deaf-mute degree of the wearing user is low, and then when the external sign language action is detected, the external sign language action can be converted into corresponding natural language information to be played, so that the interaction between the wearing user and the outside is assisted; if the picked-up audio is not effective audio, the wearing user is proved not to be communicated with the outside through bone conduction, the deaf-mute degree of the wearing user is high, and then when outside interactive information is detected, preset answer responses corresponding to the interactive information are displayed in an AR display picture of the AR glasses to assist the wearing user to interact with the outside, so that an interactive auxiliary mode which is matched with the wearing user according to the deaf-mute degree of the wearing user is realized, and the obstacle of communication between the deaf-mute user and the outside under different deaf-mute degrees is overcome in the corresponding interactive auxiliary mode.
Example two
Further, referring to fig. 2, in another embodiment of the present application, the same or similar contents as those in the above embodiment may be referred to the above description, and are not repeated herein. On this basis, the interactive information includes sign language actions, and the step of displaying each preset answer corresponding to the interactive information in the AR display screen of the AR glasses includes:
step E10, performing sign language recognition on the sign language action to obtain a sign language recognition result;
step E20, determining each preset answer corresponding to the sign language action according to the sign language identification result;
step E30, displaying each preset answer in the AR display screen.
In this embodiment, it should be noted that the wearing user is a deaf-mute user, and the interactive target corresponding to the wearing user also belongs to a crowd for deaf-mute, and the interactive target shows sign language to the wearing user, but the wearing user does not have sign language, and a communication obstacle exists between the wearing user and the interactive target.
As an example, the sign language recognition result may be a sign language semantic feature, and steps S31 to S33 include: shooting the outside through AR glasses to obtain an outside image; identifying whether sign language action exists in the external image, and if so, judging that the sign language action is detected; segmenting sign language action images from the external images; extracting features of the sign language action image to extract sign language semantic features corresponding to the sign language actions in the sign language action image, wherein the sign language semantic features can be embedded features of sign language semantics representing the sign language actions; determining a feature with the highest similarity with the semantic features of the sign language in all preset sign language semantic features as a target sign language semantic feature; determining each preset answer corresponding to the target sign language semantic features according to the mapping relation between the preset sign language semantic features and the preset answer; and displaying each preset answer corresponding to the semantic features of the target sign language in an AR display picture of the AR glasses.
Wherein, after the step of displaying each of the preset answer answers in the AR display screen, the user interaction assisting method further includes:
step E40, receiving feedback information of the wearing user to each preset answer;
step E50, selecting a target answer from the preset answers according to the feedback information;
and E60, broadcasting the target answer.
As one example, steps S34 to S36 include: receiving feedback information of the wearing user to each preset answer; selecting a target answer reply from all preset answer replies corresponding to the semantic features of the target sign language according to the feedback information, wherein the feedback information can be click information of a wearing user on all displayed preset answer replies, for example, 3 preset answer replies A, B and C exist, and if the wearing user clicks C, the C is the target answer reply; and playing the target answer through a microphone.
The embodiment of the application provides a user interaction auxiliary method, namely sign language recognition is carried out on the sign language action to obtain a sign language recognition result; determining each preset answer corresponding to the sign language action according to the sign language recognition result; displaying each preset answer in the AR display picture; receiving feedback information of the wearing user to each preset answer; selecting a target answer from the preset answers according to the feedback information; and broadcasting the target answer. Therefore, the external sign language information is transmitted to the deaf-mute user in a visual display mode, so that the deaf-mute user can correctly understand the meaning of the external sign language information to be expressed, the purpose of assisting the wearing user as the deaf-mute to communicate with the outside is achieved, and the communication barrier between the deaf-mute user and other external deaf-mute users can be overcome.
EXAMPLE III
This application still provides a user interaction auxiliary device, is applied to AR glasses, AR glasses are provided with the osteoacusis earphone, user interaction auxiliary device includes:
the audio pickup module is used for picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain picked-up audio;
the first interaction mode module is used for converting the sign language action into corresponding natural language information to play when the sign language action of the outside is detected if the picked audio is effective audio so as to assist the wearing user in interacting with the outside;
and the second interaction mode module is used for displaying each preset answer corresponding to the interaction information in an AR display picture of the AR glasses when the external interaction information is detected if the picked-up audio is not the effective audio, so as to assist the wearing user in interacting with the outside.
Optionally, the first interaction mode module is further configured to:
performing sign language recognition on the sign language action to obtain a sign language recognition result;
generating natural language information corresponding to the sign language action according to the sign language identification result;
and playing the natural language information to the wearing user of the AR glasses.
Optionally, the natural language information at least includes one of broadcast voice and a text display screen, and the first interaction mode module is further configured to:
converting the sign language action into corresponding voice information according to the sign language recognition result, and generating broadcast voice corresponding to the voice information; and/or
And converting the sign language action into corresponding text information according to the sign language identification result, and generating a text display picture corresponding to the text information.
Optionally, the natural language information at least includes one of a broadcast voice and a text display screen, and the first interaction mode module is further configured to:
broadcasting the broadcast voice to the wearing user through the bone conduction earphone; and/or
And displaying the text display picture to the wearing user through the AR display picture of the AR glasses.
Optionally, the second interaction mode module is further configured to:
performing sign language recognition on the sign language action to obtain a sign language recognition result;
determining each preset answer corresponding to the sign language action according to the sign language identification result;
and displaying each preset answer in the AR display picture.
Optionally, the user interaction assisting device is further configured to:
receiving feedback information of the wearing user to each preset answer;
selecting a target answer from the preset answers according to the feedback information;
broadcasting the target answer.
Optionally, the interaction information includes voice interaction information, and the second interaction mode module is further configured to:
performing voice recognition on the voice interaction information to obtain a voice recognition result;
determining each preset answer corresponding to the voice interaction information according to the voice recognition result;
and displaying each preset answer in the AR display picture.
The user interaction assisting device provided by the application adopts the user interaction assisting method in the embodiment, and solves the technical problem that communication obstacles exist between the deaf-mute user and the outside. Compared with the prior art, the beneficial effects of the user interaction assisting device provided by the embodiment of the application are the same as the beneficial effects of the user interaction assisting method provided by the embodiment, and other technical features of the user interaction assisting device are the same as those disclosed by the method of the embodiment, which are not repeated herein.
Example four
The embodiment of the application provides an electronic equipment, electronic equipment can be AR glasses, and electronic equipment includes: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the user interaction assistance method of the first embodiment.
Referring now to FIG. 3, shown is a block diagram of an electronic device suitable for use in implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., car navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 3, the electronic device may include a processing apparatus (e.g., a central processing unit, a graphic processor, etc.) that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) or a program loaded from a storage apparatus into a Random Access Memory (RAM). In the RAM, various programs and data necessary for the operation of the electronic apparatus are also stored. The processing device, the ROM, and the RAM are connected to each other through a bus. An input/output (I/O) interface is also connected to the bus.
Generally, the following systems may be connected to the I/O interface: input devices including, for example, touch screens, touch pads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, and the like; output devices including, for example, Liquid Crystal Displays (LCDs), speakers, vibrators, and the like; storage devices including, for example, magnetic tape, hard disk, etc.; and a communication device. The communication means may allow the electronic device to communicate wirelessly or by wire with other devices to exchange data. While the figures illustrate an electronic device with various systems, it is to be understood that not all illustrated systems are required to be implemented or provided. More or fewer systems may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means, or installed from a storage means, or installed from a ROM. The computer program, when executed by a processing device, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
The electronic equipment provided by the application adopts the user interaction assisting method in the embodiment, and solves the technical problem that the deaf-mute user has communication obstacle with the outside. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present application are the same as the beneficial effects of the user interaction assistance method provided by the above embodiment, and other technical features of the electronic device are the same as those disclosed by the above embodiment method, which are not repeated herein.
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the foregoing description of embodiments, the particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
EXAMPLE five
The present embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon for performing the method of user interaction assistance in the first embodiment.
The computer readable storage medium provided by the embodiments of the present application may be, for example, a usb disk, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or a combination of any of the above. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present embodiment, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer-readable storage medium may be embodied in an electronic device; or may be present alone without being incorporated into the electronic device.
The computer readable storage medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio; if the picked audio is effective audio, converting the sign language action into corresponding natural language information for playing when the sign language action of the outside is detected so as to assist the wearing user in interacting with the outside; and if the picked audio is not effective audio, displaying each preset answer corresponding to the interactive information in an AR display picture of the AR glasses when the interactive information of the outside is detected so as to assist the wearing user to interact with the outside.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. Wherein the names of the modules do not in some cases constitute a limitation of the unit itself.
The computer readable storage medium provided by the application stores computer readable program instructions for executing the user interaction assisting method, and solves the technical problem that communication obstacles exist between the deaf-mute user and the outside. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present application are the same as the beneficial effects of the user interaction assistance method provided by the above embodiment, and are not described herein again.
EXAMPLE six
The present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the user interaction assistance method as described above.
The computer program product solves the technical problem that communication obstacles exist between the deaf-mute user and the outside. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as the beneficial effects of the user interaction assistance method provided by the above embodiment, and are not described herein again.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.

Claims (10)

1. A user interaction assisting method is applied to AR glasses, the AR glasses are provided with bone conduction earphones, and the user interaction assisting method comprises the following steps:
picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain a picked-up audio;
if the picked audio is effective audio, converting the sign language action into corresponding natural language information for playing when the sign language action of the outside is detected so as to assist the wearing user in interacting with the outside;
and if the picked audio is not effective audio, displaying each preset answer corresponding to the interactive information in an AR display picture of the AR glasses when the interactive information of the outside is detected so as to assist the wearing user to interact with the outside.
2. The method as claimed in claim 1, wherein the step of converting the sign language action into corresponding voice information for playing comprises:
performing sign language recognition on the sign language action to obtain a sign language recognition result;
generating natural language information corresponding to the sign language action according to the sign language identification result;
and playing the natural language information to the wearing user of the AR glasses.
3. The user interaction assisting method according to claim 2, wherein the natural language information includes at least one of a broadcast voice and a text display screen, and the step of generating the natural language information corresponding to the sign language action according to the sign language recognition result includes:
converting the sign language action into corresponding voice information according to the sign language recognition result, and generating broadcast voice corresponding to the voice information; and/or
And converting the sign language action into corresponding text information according to the sign language identification result, and generating a text display picture corresponding to the text information.
4. The user interaction assisting method according to claim 2, wherein the natural language information includes at least one of a broadcast voice and a text display screen, and the step of playing the natural language information to the user wearing the AR glasses includes:
broadcasting the broadcast voice to the wearing user through the bone conduction earphone; and/or
And displaying the text display picture to the wearing user through the AR display picture of the AR glasses.
5. The user interaction assisting method according to claim 1, wherein the interaction information includes sign language actions, and the step of displaying each preset answer corresponding to the interaction information on the AR display screen of the AR glasses includes:
performing sign language recognition on the sign language action to obtain a sign language recognition result;
determining each preset answer corresponding to the sign language action according to the sign language identification result;
and displaying each preset answer in the AR display picture.
6. The user interaction assisting method according to claim 5, wherein after the step of displaying each of the preset reply answers in the AR display screen, the user interaction assisting method further comprises:
receiving feedback information of the wearing user to each preset answer;
selecting a target answer from the preset answers according to the feedback information;
and broadcasting the target answer.
7. The user interaction assisting method according to claim 1, wherein the interaction information includes voice interaction information, and the step of displaying each preset answer corresponding to the interaction information in an AR display screen of the AR glasses includes:
performing voice recognition on the voice interaction information to obtain a voice recognition result;
determining each preset answer corresponding to the voice interaction information according to the voice recognition result;
and displaying each preset answer in the AR display picture.
8. A user interaction assisting device applied to AR glasses provided with bone conduction earphones, the user interaction assisting device comprising:
the audio pickup module is used for picking up the audio of a user wearing the AR glasses through the bone conduction headset to obtain picked-up audio;
the first interaction mode module is used for converting the sign language action into corresponding natural language information to be played when the external sign language action is detected if the picked audio is effective audio so as to assist the wearing user in interacting with the outside;
and the second interaction mode module is used for displaying each preset answer corresponding to the interaction information in an AR display picture of the AR glasses when the external interaction information is detected if the picked-up audio is not the effective audio, so as to assist the wearing user in interacting with the outside.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the user interaction assistance method of any one of claims 1 to 7.
10. A computer-readable storage medium, having a program for implementing a user interaction assisting method stored thereon, the program being executable by a processor to implement the steps of the user interaction assisting method according to any one of claims 1 to 7.
CN202210396401.8A 2022-04-15 2022-04-15 User interaction assisting method and device, electronic equipment and readable storage medium Pending CN114783440A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210396401.8A CN114783440A (en) 2022-04-15 2022-04-15 User interaction assisting method and device, electronic equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210396401.8A CN114783440A (en) 2022-04-15 2022-04-15 User interaction assisting method and device, electronic equipment and readable storage medium

Publications (1)

Publication Number Publication Date
CN114783440A true CN114783440A (en) 2022-07-22

Family

ID=82428592

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210396401.8A Pending CN114783440A (en) 2022-04-15 2022-04-15 User interaction assisting method and device, electronic equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN114783440A (en)

Similar Documents

Publication Publication Date Title
CN111381909B (en) Page display method and device, terminal equipment and storage medium
CN108012173B (en) Content identification method, device, equipment and computer storage medium
WO2021008223A1 (en) Information determination method and apparatus, and electronic device
US20170064345A1 (en) Video file processing
JP6906584B2 (en) Methods and equipment for waking up devices
WO2022007565A1 (en) Image processing method and apparatus for augmented reality, electronic device and storage medium
WO2022037419A1 (en) Audio content recognition method and apparatus, and device and computer-readable medium
CN111738041A (en) Video segmentation method, device, equipment and medium
CN110930220A (en) Display method, display device, terminal equipment and medium
WO2023040697A1 (en) Information processing method and apparatus, device, readable storage medium and product
CN110379406B (en) Voice comment conversion method, system, medium and electronic device
CN115174818B (en) Target tracking method based on sound positioning, electronic equipment and readable storage medium
CN114783440A (en) User interaction assisting method and device, electronic equipment and readable storage medium
CN114095747B (en) Live broadcast interaction system and method
US20240096347A1 (en) Method and apparatus for determining speech similarity, and program product
CN113784045B (en) Focusing interaction method, device, medium and electronic equipment
CN114677738A (en) MV recording method, MV recording device, electronic equipment and computer readable storage medium
CN111462548A (en) Paragraph point reading method, device, equipment and readable medium
US20140297285A1 (en) Automatic page content reading-aloud method and device thereof
CN112601170B (en) Sound information processing method and device, computer storage medium and electronic equipment
WO2021170094A1 (en) Method and device for information interaction
CN114708866A (en) Head-mounted display device control method and device, head-mounted display device and medium
CN113655933A (en) Text labeling method and device, storage medium and electronic equipment
CN114690907A (en) User interaction assistance method, electronic device and readable storage medium
CN114120729B (en) Live teaching system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination