KR20230082758A - METHOD AND DEVICE FOR Task-oriented Sounding Guide with Object Detection to Guide Visually Impaired People During Smart Device Usage - Google Patents

Abstract

A task-based sound guide method and apparatus for assisting a blind person to detect an object while using a smart device are disclosed. The sound guide method includes designating an interface object of a function that a user wants to activate as a target button based on a user's voice; detecting the target button and the user's fingertip in a real-time image through a camera; and providing a sound guide according to a result of sensing the target button and the fingertip.

Description

METHOD AND DEVICE FOR Task-oriented Sounding Guide with Object Detection to Guide Visually Impaired People During Smart Device Usage}

The description below relates to techniques for providing acoustic guides for smart devices.

The use of smart devices in our daily lives, such as controlling home appliances, ordering food, and online banking, has recently increased.

However, visually impaired users face some difficulties while using smart devices and applications. For example, you have to find and touch a specific icon or button to use an application on your smartphone or select an option on a smart fan in your home.

Currently, smart devices are usually designed with little or no consideration for assistive functions for the visually impaired. Most smartphones come equipped with screen readers that read the contents of the screen for users, but they often fail to effectively help the visually impaired. For example, when a user taps the URL field of a web browser, the screen reader will say "link", which alone is unlikely to prompt the user to take appropriate action. The problem is even more difficult for devices that are difficult to equip with screen readers, such as smart fans. Since the user cannot recognize the image icon displayed on the button, a visually impaired user cannot distinguish the touch button.

Braille systems and screen readers are some examples of traditional methods of helping blind people. Nevertheless, the various features and options of smart devices that use a touch screen as the main user interface still cause difficulties in manipulating the correct buttons or icons to perform tasks for the visually impaired.

Recently, usability and accessibility evaluations are being conducted so that visually impaired people can use native applications on smart devices. When some mobile applications and web interfaces were tested with visually impaired subjects by asking the applications and interfaces to perform a set of predefined tasks, significant usability issues were discovered, one of which was that users were not able to perform a predefined task. It lacks step-by-step instructions and interactive features to properly utilize the smart device. In other words, it is necessary to provide instructions in the form of interactive messages to help visually impaired people specifically what to do while interacting with smart devices.

In order to improve the usability and accessibility of smart devices for visually impaired users, an acoustic guide with a task-oriented approach that guides users in a step-by-step and interactive manner can be provided.

A sound guide method performed in a computer system, comprising: designating, by at least one processor included in the computer system, an interface object of a function to be activated by a user as a target button based on a user's voice; detecting, by the at least one processor, the target button and the user's fingertip in a real-time image through a camera; and providing, by the at least one processor, a sound guide according to a result of sensing the target button and the fingertip.

According to one aspect, the designating may include converting the user voice into text; and designating an interface object of a keyword corresponding to the text as the target button.

According to another aspect, the detecting may include tracking the location of the target button and the fingertip through a convolution neural network (CNN)-based object recognition model trained with a dataset consisting of a fingertip image and an interface object image. can include

According to another aspect, the detecting may include calculating a time required for the user to touch the target button according to the position of the target button and the fingertip using a CNN-based single-shot multibox detector (SSD). steps may be included.

According to another aspect, the providing of the sound guide may include the movement of the fingertip relative to the target button by comparing coordinates of a bounding box indicating a position of the target button and the fingertip detected from a camera image with each other. It may include guiding directions.

According to another aspect, the providing of the sound guide may include checking whether the target button is captured in a camera image; finding the position of the fingertip in the camera image when the target button is captured in the camera image; comparing positions of the target button and the fingertip in the camera image and providing guide feedback on a moving direction of the fingertip according to a comparison result; and providing feedback as a result of the location of the fingertip reaching the location of the target button.

According to another aspect, the providing of the sound guide may further include providing a voice notification for a corresponding detection result when the target button is not captured in the camera image.

A computer program stored in a computer readable recording medium to execute a sound guide method on a computer, the sound guide method comprising: designating an interface object of a function to be activated by a user as a target button based on a user's voice; detecting the target button and the user's fingertip in a real-time image through a camera; and providing a sound guide according to a result of sensing the target button and the fingertip.

A computer-implemented acoustic guide system comprising at least one processor configured to execute computer readable instructions included in a memory, wherein the at least one processor is configured to interface with a function that a user wants to activate based on a user's voice. The process of designating an object as a target button; detecting the target button and the user's fingertip in a real-time image through a camera; and a sound guide system that processes a process of providing a sound guide according to a result of sensing the target button and the fingertip.

According to the embodiments of the present invention, by providing a task-oriented sound guide with an object detection function for guiding the visually impaired while using the smart device, the visually impaired can easily use various smart devices encountered in various environments. .

1 is a block diagram for explaining an example of an internal configuration of a computer device according to an embodiment of the present invention.
Figure 2 shows an example of an environment using an acoustic guide according to the present invention.
3 is a flowchart illustrating an example of a sound guide method according to an embodiment of the present invention.
4 illustrates an example of an icon used in a smart device according to an embodiment of the present invention.
Figure 5 shows a loss score graph generated by tensorboard for machine learning.
6 illustrates an example algorithm of a process of detecting an interface object and a fingertip according to an embodiment of the present invention.
7 illustrates an example algorithm of a process of generating a voice output for notification in one embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to techniques for providing acoustic guides for smart devices.

Embodiments including those specifically disclosed herein may provide an acoustic guide of a task-oriented approach having an object detection function for guiding a blind person while using a smart device.

1 is a block diagram illustrating an example of a computer device according to one embodiment of the present invention. For example, an acoustic guide system according to embodiments of the present invention may be implemented by the computer device 100 shown in FIG. 1 .

As shown in FIG. 1, the computer device 100 is a component for executing the acoustic guide method according to embodiments of the present invention, and includes a memory 110, a processor 120, a communication interface 130, and an input/output interface. (140).

The memory 110 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device 100 as a separate permanent storage device distinct from the memory 110. Also, an operating system and at least one program code may be stored in the memory 110 . These software components may be loaded into the memory 110 from a recording medium readable by a separate computer from the memory 110 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 110 through the communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer device 100 based on a computer program installed by files received over network 160 .

The processor 120 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 120 by memory 110 or communication interface 130 . For example, processor 120 may be configured to execute received instructions according to program codes stored in a recording device such as memory 110 .

The communication interface 130 may provide functions for the computer device 100 to communicate with other devices through the network 160 . For example, a request, command, data, file, etc. generated according to a program code stored in a recording device such as the memory 110 by the processor 120 of the computer device 100 is transmitted to the network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 100 via the communication interface 130 of the computer device 100 via the network 160 . Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or the memory 110, and files, etc. may be stored as storage media that the computer device 100 may further include (described above). permanent storage).

The communication method is not limited, and may include not only a communication method utilizing a communication network (eg, a mobile communication network, wired Internet, wireless Internet, and broadcasting network) that the network 160 may include, but also short-distance wired/wireless communication between devices. there is. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , one or more arbitrary networks such as the Internet. In addition, the network 160 may include any one or more of network topologies including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, and the like. Not limited.

The input/output interface 140 may be a means for interface with the input/output device 150 . For example, the input device may include devices such as a microphone, keyboard, camera, or mouse, and the output device may include devices such as a display and a speaker. As another example, the input/output interface 140 may be a means for interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 150 and the computer device 100 may be configured as one device.

Also, in other embodiments, computer device 100 may include fewer or more elements than those of FIG. 1 . However, there is no need to clearly show most of the prior art components. For example, the computer device 100 may be implemented to include at least a portion of the above-described input/output device 150 or may further include other components such as a transceiver, a camera, various sensors, and a database.

The present embodiments provide an acoustic guide system, which is a task-oriented approach that guides users in a step-by-step and interactive manner to improve usability and accessibility of smart devices for visually impaired users.

In this embodiment, object recognition technology together with machine learning is used to detect and track two main objects: a user's fingertip and an icon or button, which is a basic element of a smart device. The user can use the system's acoustic guide to instruct the system which type of device to use, such as a smartphone application. Then, using a camera installed in a user device such as a smartphone or smart glasses, the user can show the user interface of the terminal to the sound guide system. The acoustic guide system then recognizes the key functions associated with the buttons displayed on the user interface and provides appropriate instructions for the operation of the device or application that helps the user perform tasks. Acoustic guidance systems provide discrete bits of guiding information, such as voice, in real time so that users can correctly position their fingertips on specific icons or buttons to activate desired functions on the device.

To detect the user's fingertips, object detection techniques combined with machine learning are being used for various tasks. However, it is not focused on providing detailed instructions in real time to enable blind people to interact with smart devices. Some work is being done to enable visually impaired users to navigate web sites and use mobile applications using suggestion-based systems that employ non-visual interfaces. However, this approach only works for website navigation or predefined mobile applications. In addition, existing object detection technologies such as TapTapSee, CamFind, and Talking Goggles cannot be directly used to provide step-by-step instructions for blind people. There are also studies on prototype systems with deep neural network models to detect objects in real time, but these systems only work for detecting physical objects and cannot be used to detect user interface elements of smart devices. One key technique for fingertip detection is the use of body joints and facial expression localization. Improving the effectiveness of these fingertip sensing technologies is important to provide real-time guidance for visually impaired users.

In the present invention, a convolution neural network (CNN) is used to detect and track the position of a user's fingertip in real time in relation to user interface elements of a smart device. CNN is an effective machine learning technique for object detection, and the acoustic guide system uses CNN to process a series of objects that can be processed to recognize multiple objects, including buttons, icons, and the user's fingertips while the user operates the control panel of the smart device. Map image input to data. In addition, a single-shot multibox detector (SSD) widely used for object detection in mobile devices is used. SSD is based on a deep neural network approach using CNNs. The acoustic guide system can use the SSD to create four positions of a bounding box surrounding the detected object. An acoustic guidance system can calculate the exact location of a button or icon based on the object's bounding box and the exact time required for the user to touch the button or icon.

In other words, the acoustic guide system according to the present invention can detect and track interface objects (icons or buttons) on the user's fingertips and the screen of a smart device by using CNN-based object recognition technology, and provide audio-like guidance in real time. Discrete bits of information can be provided to help users activate desired functions on the device by moving their fingertips appropriately on specific objects.

2 illustrates an example of a smart device use environment requiring a sound guide. 2 shows an environment in which a visually impaired user uses a treadmill.

External entities that input information necessary for the acoustic guide system are as follows.

Referring to FIG. 2 , a user 210 interacts with a sound guide system running on a mobile device 220 such as a smart phone.

The acoustic guide system runs on the mobile device 220 of the user 210 .

The target device 230 is a device that the user 210 wants to manipulate, and may mean a device that includes a control panel for user manipulation and is operated with icons or buttons on the control panel.

The sound guide system is executed in the mobile device 220 and captures a manipulation scene of the user 210 with respect to the target device 230 using a camera of the mobile device 220 or smart glasses that can be interlocked with the mobile device 220. can do.

As shown in FIG. 2 , the user 210 may capture the control panel of the target device 230 on the screen of the mobile device 220 after executing the sound guide system in the mobile device 220 .

The internal components of the acoustic guide system are as follows.

The processor 120 may include a text to speech (TTS) module, a speech to text (STT) module, and an object detection module as internal components that compile and estimate the guide output.

The TTS module serves to generate voice by receiving text as an input, and for example, a text-to-speech conversion function can be implemented using Android Studio.

The STT module serves to generate text by receiving voice as an input, and for example, a voice-to-text conversion function can be implemented using Android Studio.

The sound guide system can communicate with the user 210 through the TTS module and the STT module, and at this time, the user 210 can deliver the voice generated by the TTS module and the user 210's intention from the text generated by the STT module. can recognize

The object detection module serves to estimate the location of the user's 210 fingertip and the interface object (icon or button) corresponding to the target object. The object detection module may use a camera and a management tool (TensorFlow) to estimate the position of the target object.

As the camera, a camera of the mobile device 220 or smart glasses that can be linked with the mobile device 220 may be used. The acoustic guide system may establish a connection with a camera for displaying a user interface such as a control panel of the target device 230 .

The management tool is for a guide algorithm, and can be learned using SSD MobileNet Version 2, a CNN architecture for detecting the user's 210 fingertip and interface objects (icons or buttons), and a dataset. In this case, the data set for machine learning may include images of people's fingertips, images of control panels of devices, images of interface objects of devices, and the like.

3 is a flowchart illustrating an example of a sound guide method according to an embodiment of the present invention.

The processor 120 recognizes information on a target button corresponding to an interface object designated by the user based on the user's voice, and then finds the target button in the real-time image while the real-time image is captured through the camera. Then, the processor 120 outputs the location of the target button to the mobile device using the mobile connector. Finally, the processor 120 guides the user to the location of the target button by voice.

Referring to FIG. 3 , in step S1 , the processor 120 may receive a user voice input and designate a target button (ie, an interface object) from the received voice input. The user may make a voice speech for a function to be activated, and the processor 120 may designate an interface object corresponding to a function to be activated as a target button from the user's voice speech.

In step S2, the processor 120 receives the real-time image through the camera and tracks the user's fingertip and the control panel of the target device captured on the image through a machine learning model to locate the user's fingertip on the control panel. Interface objects can be detected. The processor 120 may display an interface object designated as a target button and an interface object currently pointed by the user's fingertip on the screen of the mobile device.

In step S3, the processor 120 may inform the user of a direction in which the user's fingertip should move as guide feedback for the user to touch the interface object designated as the target button through voice.

In step S4, the processor 120 may track the position of the fingertip in real time when the user moves the fingertip, and provide feedback as a result when the current position of the fingertip reaches an interface object designated as a target button.

The processor 120 receives the user's voice as an input. At this time, if the user utters the name of the interface object to be selected as a short and specific keyword, recognition errors can be reduced. In this case, the processor 120 may recognize a keyword related to an interface object to be designated as a target button based on the STT function by utilizing a machine learning-based CNN model. Then, the user may touch a designated button of the target device using a fingertip. The processor 120 may provide feedback to the user through audio output. Voice feedback may be provided when the user's fingertip is positioned on an interface object designated as a target button.

Technical details of each step of the acoustic guide method are as follows.

Target button designation step (S1)

The processor 120 may trigger a sound guide function according to a user's voice input. For example, the processor 120 may recognize a keyword related to a target button from a user's voice through a voice recognizer provided by the Android framework, which is an official integrated environment of the Android operating system. A user-specified keyword is required to access the function type of the target device. For example, as shown in FIG. 4, functions or icons may be classified into 14 types. Each button composed of an interface object of a smart device has an icon for a corresponding function. The processor 120 can convert a user's voice into text through the STT function and designate a button of a keyword corresponding to the text as a target button. The processor 120 may designate a button of a keyword recognized from a user's voice as a target button, and then detect an interface object on a control panel, a user's fingertip, and a target button from a camera image.

Target button and fingertip location (Locating) (S2)

SSD MobileNet Version 2 can be used as a machine learning model for the acoustic guide system to find interface objects on the control panel, user's fingertips, and target buttons. To prevent the required learning parameters from becoming too large when using limited memory, the Rectified Linear Unit (ReLU) function can be used as the activation function of the machine learning model. You can also use standard score converters and use sigmoid functions and L2 regularizers. EgoGesture dataset can be used as a training dataset for fingertip detection and hand gesture recognition. The training dataset may contain images for different hand gestures, along with manually labeled bounding boxes with information about fingertips and joints. For example, a model can be trained using one-hand gesture images with a label 'SingleOne' indicating a gesture using the index finger.

In addition, a button data set consisting of a screen image of a mobile device containing various icons, a panel image of a treadmill or remote control including buttons such as 'Start', 'Stop', 'Add', and 'Minus' is manually created. You can create and label them. The buttons shown in the table of FIG. 4 are classified into 14 classes, and two classes for 'fingertip' and 'SingleOne' gestures can be added.

5 shows a loss score graph generated by tensorboard, a visualization toolkit for machine learning experiments. Figure 5 shows the total loss, which represents the penalty value for incorrect predictions, and the normalized loss, which is the total loss plus an additional loss to generalize better predictions. While optimizing the model, the total loss decreases over time, and the regularization loss is relatively low, resulting in smooth performance.

Guide Feedback (S3)

The trained machine learning model can be converted to TensorFlow Lite (TFLite) to generate guided feedback. TFLite is a lightweight TensorFlow solution for mobile devices that requires low latency and small binaries.

For example, a TTS module required to generate voice feedback in a sound guide system can be implemented using a TTS instance of an Android library. The TTS module can synthesize speech for immediate playback based on input. For example, the feedback keyword synthesized to guide the user may be composed of upper left corner, upper right corner, lower left corner, and lower right corner. These feedback keywords indicate the direction the user should move their finger.

After the location coordinates of the user's fingertip and the target button are detected, a guide to be provided to the user is processed as in Algorithm 1 of FIG. 6 . Algorithm 1 runs continuously during the runtime of the acoustic guide system. Algorithm 1 starts by checking if the target button can be found in the video captured by the camera on line 1.

If the target button is not found, a voice notification is generated in line 2 and notified to the user through TTS conversion using the convert_to_speech() function defined in Algorithm 2 of FIG. 7 . Algorithm 2 can handle the time difference between each function call by using a global variable. Line 1 of Algorithm 2 first retrieves the time the function is called from the new_now variable. Next, compare the time when the convert_to_speec() function was called with the global variable in Algorithm 2, line 6. If this function is called after a response delay (α or β seconds) from the previous call, it converts text to speech. When the user searches to find the target button on the screen, we use 1.5 seconds (α) as the response delay. Inconvenience can be avoided by using 3 seconds (β) as a response delay when there is no target button on the screen. Since guiding the user is the final goal, only the keyword “okay” in line 10 of Algorithm 2 can be converted into speech immediately. A global variable can be updated whenever the Text To Speech function is called.

After detecting the target button in Algorithm 1, find the user's fingertip in line 4 and notify the user. When both the target button and the fingertip are detected, the bounding box coordinates of the target button and the fingertip are compared with each other, and directions are provided to the user according to the results of lines 9, 11, 14, and 16. At this time, the user continues to move the finger in the “SingleOne” gesture in the guide direction.

Result feedback (S4)

If the user's fingertip is within the target button's bounding box, the text-to-speech instance can generate the output "okay" (lines 17 and 18 of Algorithm 1). When the voice “okay” is output, the user can press the target button at the corresponding fingertip position. After the above process is completed, the processor 120 is ready to receive the next voice input. The user may input the next target button and receive guidance for manipulating the next operation of the target device.

The present invention can provide a sound guide that helps visually impaired people to press a button of a desired function while using a device. When a user utters a button to press, the acoustic guide system analyzes the utterance, designates a button the user wants to press as a target, and calculates the location of the target button from an image captured by a camera. The sound guide system may support the user to press a desired button by instructing the direction step by step according to the position of the user's fingertip and the position of the target button detected in the camera image.

As described above, according to the embodiments of the present invention, by providing sound guides for manipulating the device targeting various devices in various environments, it is possible to provide more reliable help to solve the difficulty of manipulating the device to a user with a visual impairment. there is.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. The software and/or data may be embodied in any tangible machine, component, physical device, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. there is. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (14)

In the acoustic guide method performed in a computer system,
designating, by at least one processor included in the computer system, an interface object of a function that the user wants to activate based on the user's voice as a target button;
detecting, by the at least one processor, the target button and the user's fingertip in a real-time image through a camera; and
Providing, by the at least one processor, a sound guide according to a result of sensing the target button and the fingertip
Acoustic guide method comprising a. According to claim 1,
The step of specifying
converting the user's voice into text; and
Designating an interface object of a keyword corresponding to the text as the target button
Acoustic guide method comprising a. According to claim 1,
The detecting step is
Tracking the location of the target button and the fingertip through a convolution neural network (CNN)-based object recognition model trained with a dataset consisting of a fingertip image and an interface object image.
Acoustic guide method comprising a. According to claim 1,
The detecting step is
Calculating a time required for the user to touch the target button according to the position of the target button and the fingertip using a CNN-based single-shot multibox detector (SSD)
Acoustic guide method comprising a. According to claim 1,
Providing the acoustic guide,
guiding a movement direction of the fingertip with respect to the target button by comparing coordinates of the target button detected from the camera image and a bounding box representing the position of the fingertip with each other;
Acoustic guide method comprising a. According to claim 1,
Providing the acoustic guide,
Checking whether the target button is captured in the camera image;
finding the position of the fingertip in the camera image when the target button is captured in the camera image;
comparing positions of the target button and the fingertip in the camera image and providing guide feedback on a moving direction of the fingertip according to a comparison result; and
Providing feedback as a result of the location of the fingertip reaching the location of the target button
Acoustic guide method comprising a. According to claim 6,
Providing the acoustic guide,
Providing a voice notification for a corresponding detection result when the target button is not captured in the camera image
Acoustic guide method further comprising a. A computer program stored in a computer readable recording medium for executing a sound guide method on a computer,
The acoustic guide method,
designating an interface object of a function to be activated by a user as a target button based on a user's voice;
detecting the target button and the user's fingertip in a real-time image through a camera; and
Providing a sound guide according to the detection result of the target button and the fingertip
Including, a computer program. In the acoustic guide system implemented by a computer,
at least one processor configured to execute computer readable instructions contained in memory;
including,
The at least one processor,
designating an interface object of a function that the user wants to activate as a target button based on the user's voice;
detecting the target button and the user's fingertip in a real-time image through a camera; and
A process of providing a sound guide according to the detection result of the target button and the fingertip
Acoustic guide system that handles According to claim 9,
The at least one processor,
converting the user's voice into text;
Designating an interface object of a keyword corresponding to the text as the target button
Acoustic guide system characterized in that. According to claim 9,
The at least one processor,
Tracking the location of the target button and the fingertip through a convolution neural network (CNN)-based object recognition model trained with a dataset consisting of a fingertip image and an interface object image
Acoustic guide system characterized in that. According to claim 9,
The at least one processor,
Calculating a time required for the user to touch the target button according to the location of the target button and the fingertip using a CNN-based single-shot multibox detector (SSD)
Acoustic guide system characterized in that. According to claim 9,
The at least one processor,
Guiding the movement direction of the fingertip with respect to the target button by comparing coordinates of the target button detected from the camera image and a bounding box representing the location of the fingertip with each other
Acoustic guide system characterized in that. According to claim 9,
The at least one processor,
Check whether the target button is captured in the camera image,
If the target button is not captured in the camera image, an audio notification is provided for a corresponding detection result;
When the target button is captured in the camera image, the location of the fingertip is found in the camera image;
comparing the position of the target button and the fingertip in the camera image and providing guide feedback on the moving direction of the fingertip according to the comparison result;
Providing feedback as a result of the location of the fingertip reaching the location of the target button
Acoustic guide system characterized in that.

Images