KR20200000423A - System and method of determining micro spare time using sensor of mobile apparatus - Google Patents

Abstract

According to a method for determining a spare time using a sensor of a mobile device, which comprises the steps of: acquiring user information using at least one of a hardware sensor unit and a software sensor unit; and determining the spare time of a user in real time based on the user information.

Description

System and Method for Determining Stuck Time Using Mobile Devices and Sensors {SYSTEM AND METHOD OF DETERMINING MICRO SPARE TIME USING SENSOR OF MOBILE APPARATUS}

The present invention relates to a system and method for determining the utterance time using a mobile device and a sensor, and more particularly, a system for determining the utterance time based on the physical situation information and the mobile device usage information of a user obtained through the mobile device and the sensor; It is about a method.

Due to the nature of modern people who spend a lot of time at work or at work, it is very difficult to secure enough time for work outside of work (eg, the time parents spend on their children).

This problem can be attributed to the fact that the time spent at work or at work and the time spent outside the work are perceived as mutually exclusive.

It is an object of the present invention to provide a system and method for determining a user's utterance time using a mobile device and a sensor.

According to an embodiment of the present invention, a method for determining a utterance time using a sensor of a mobile device includes: acquiring user information using at least one of a hardware sensor unit and a software sensor unit; Determining the utterance time of the user based on the information of the.

In an embodiment of the present disclosure, the acquiring of the information of the user may acquire physical state information of the user by using the hardware sensor unit.

In an embodiment of the present disclosure, the acquiring of the information of the user may acquire the mobile device usage information of the user by using the software sensor unit.

In an embodiment of the present disclosure, if the user's utterance time is detected, the method may further include immediately notifying the user.

In one embodiment of the present invention, the step of determining the poke time of the user may determine the wearing event walking wear earphone means that the user walks alone wearing the earphone. The earphone wearing walking event may be determined using an earphone event of an accelerometer or a GPS sensor of the hardware sensor unit and an event detector of the software sensor unit.

In one embodiment of the present invention, the step of determining the dolmen time of the user may determine a meeting waiting event that means that the user is waiting before the start of the meeting. The meeting waiting event may be determined using calendar information of the mobile device and GPS location information of a meeting participant.

In one embodiment of the present invention, the step of determining the utterance time of the user is a messenger response waiting event means that the user is talking to the other party using a messenger, the user is waiting for a message of the other party You can judge. The messenger response waiting event may be determined using an app usage monitor of the software sensor unit.

In one embodiment of the present invention, the step of determining the poke time of the user may determine a walking event in the building means that the user is moving in the building. The walking event in the building may be determined using a GPS sensor and a Wi-Fi signal of the hardware sensor unit.

In one embodiment of the present invention, the step of determining the dead time of the user may determine the elevator waiting event means that the user is waiting for the elevator. The elevator waiting event may be determined using a magnetic field sensor and an accelerometer of the hardware sensor unit.

According to the method for determining a utterance time using a mobile device and a sensor according to the present invention, various types of utterance time of a user can be determined in real time using the user's mobile device and sensors. Various application services may be provided to the user by utilizing the user's utterance time.

1 is a block diagram illustrating an audio book production system according to an embodiment of the present invention.
FIG. 2 is a conceptual diagram illustrating a book list page of the first user interface of the mobile device of FIG. 1.
3 is a conceptual diagram illustrating a book recording page of the first user interface of the mobile device of FIG. 1.

With respect to the embodiments of the present invention disclosed in the text, specific structural to functional descriptions are merely illustrated for the purpose of describing the embodiments of the present invention, embodiments of the present invention may be implemented in a variety of forms and It should not be construed as limited to the embodiments described in.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosed form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between. Other expressions describing the relationship between components, such as "between" and "immediately between," or "neighboring to," and "directly neighboring to" should be interpreted as well.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "having" are intended to indicate that the described features, numbers, steps, operations, components, parts or combinations thereof exist, but one or more other features or numbers, It is to be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as meanings consistent with the meanings in the context of the related art, and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. .

On the other hand, when an embodiment is otherwise implemented, a function or operation specified in a specific block may occur out of the order specified in the flowchart. For example, two consecutive blocks may actually be performed substantially simultaneously, and the blocks may be performed upside down depending on the function or operation involved.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. The same reference numerals are used for the same elements in the drawings, and duplicate descriptions of the same elements are omitted.

1 is a block diagram illustrating an audio book production system according to an embodiment of the present invention. FIG. 2 is a conceptual diagram illustrating a book list page of the first user interface of the mobile device of FIG. 1. 3 is a conceptual diagram illustrating a book recording page of the first user interface of the mobile device of FIG. 1.

1 to 3, the audio book production system enables a user (eg, a parent) to make an audio book for a child by using the user's utterance time.

In the present invention, the term “cutting time” refers to a short time unit having a low cognitive load that often occurs in daily life. The cut time may mean unplanned spare time that occurs between the user's planned schedules. For example, the cut time may be a time waiting for an elevator.

The cut time includes the following features.

1) Often occurs in everyday life, but it is easy to overlook.

2) Since the cognitive load is low, additional tasks can be performed.

3) Each one is a short unit of time, but when they come together, it can be a significant large unit of time.

The audio book production system may determine in real time the utterance time at the user's workplace to enable the user to record the audio book in sentence units. In the audio book production system, the child can enjoy the audio book recorded in the voice of the parents.

The design goal of the audio book production system is as follows.

1) It is possible to perform one recording operation during the cut time. The unit of recording task provided by the audio book production system should be short enough to be completed during one narration time session (eg, one sentence recording).

2) Maximize recording opportunities. In order to maximize the opportunity of recording the audio book, the user should be able to know in real time the time at which the recording is possible and actively determine it.

3) Provide recording options appropriate to the situation. At work, the time is often spent in a common space with another person, such as in a break room with a co-worker. Recording audiobooks, such as playing next to others, can be frustrating for the user. Accordingly, the audio book production system can provide a recording option suitable for a situation.

The audio book production system may include a first mobile device 100, a server 200, and a second mobile device 300. The first mobile device 100 may be a mobile device of the user. For example, the first mobile device 100 may be the smart phone of the user. The second mobile device 300 may be a mobile device of a child of the user. For example, the second mobile device 300 may be a tablet of the child of the user.

The first mobile device 100 is a device for the user to record the audio book. The first mobile device 100 continuously monitors the related event of the first mobile device 100 and detects the utterance time of the user.

When the cut time of the user is detected, the first mobile device 100 notifies the user that the cut time is detected. The first mobile device 100 may display a notification bar on the screen together with the vibration.

As shown in FIG. 2, the book list page of the user interface of the first mobile device 100 is registered including a title of a registered book and a cover thumbnail of the registered book so that the user can select a book to record. A list of books can be displayed. In addition, the list of registered books may indicate the progress of recording of the registered book. When the user completes recording of the book, an end mark may be displayed on the list of registered books.

As shown in FIG. 3, the book recording page of the user interface of the first mobile device 100 displays one page of a book on the display unit of the first mobile device 100. In addition, the recording task may be set to one sentence of the book. In the case of an infant picture book, since the length of one sentence is short, it may be an appropriate length for performing a recording task during the utterance session.

If there are several sentences on the page, the book recording page may highlight and display the sentences to be recorded. For example, the highlight may be underlined.

The book recording page may include a record button, a play button, and the like. When swiping the book recording page from side to side, the pages of the book can be turned to previous and next.

The audio book production system may provide two recording modes. The first recording mode is a watch-record mode, the second recording mode is a listen-record mode, and the recording mode may be selected according to the situation of the user.

In the first recording mode, the user may perform recording while watching the display unit of the first mobile device 100. When the user presses the record button of the display unit of the first mobile device 100, the recording may be started, and when the user presses the record button again, the recording may be terminated.

The second recording mode is a recording mode suitable when there is another person around. When the user approaches the first mobile device 100 to the user's ear, the second recording mode is automatically activated. When the user drops the first mobile device 100 from the user's ear, the second recording mode is automatically activated. 2 Recording mode is automatically disabled. The first mobile device 100 may determine activation and deactivation of the second recording mode using a proximity sensor. In the second recording mode, when the user approaches the first mobile device 100 to the user's ear, the sentence to be read through the handset is reproduced as a machine voice. The user reads along the sentence reproduced by the machine voice and proceeds with recording. When the user drops the first mobile device 100 from the user's ear in the second recording mode, the recording may be automatically stored as the second recording mode ends.

The server 200 may store a recording file recorded by the first mobile device 100 and merge the recording file in a sentence unit or in an audio book and output the recorded file to the second mobile device 300.

The second mobile device 300 plays an audio book which is a recording file or a collection of recording files recorded by a user of the first mobile device 100.

For example, the user of the second mobile device 300 may be a child of the user of the first mobile device 100.

The user interface of the second mobile device 300 is almost the same except that there is no record button in the user interface of the first mobile device 100.

In addition, the second mobile device 300 may monitor the interaction of the child of the user, and transmit it to the server 200. The interaction information may be transmitted to the first mobile device 100 via the server 200. For example, the interaction information may include the number of times a child of the user plays the recording file. For example, the interaction information may be an expression of a child of the user while playing the recording file. The expression of the child of the user while playing the recording file may be photographed by the camera of the second mobile device 300. For example, the interaction information may be a voice of the child of the user while playing the recording file. The voice of the child of the user while playing the recording file may be recorded by the microphone of the second mobile device 300. For example, the interaction information may be an emotion of a child of the user while playing the recording file. The emotion of the child of the user while playing the recording file may be determined by the second mobile device 300 through the facial expression of the user of the user or the voice of the child of the user.

Referring back to FIG. 1, the first mobile device 100 includes a first user interface 110, a book manager 120, a narration time determiner 130, a TTS manager 140, a recording manager 150, The notification manager 160, the first web API client 170, the first hardware sensor unit 180, the first software sensor unit 190, and the first database 195 may be included.

For example, the first hardware sensor unit 180 may include an accelerometer 181, a proximity sensor 182, a microphone 183, and a GPS sensor 184. For example, the first software sensor unit 190 may include an app usage monitor 191 and an event detector 192.

The server 200 may include a web API server 210, a book content manager 220, a book database 230, a recording database 240, a family database 250, and an interaction database 260.

The second mobile device 300 may include a second user interface 310, a second web API client 320, an interaction monitor 330, a second hardware sensor unit 340, a second software sensor unit 350, and It may include a second database 360.

As shown in FIGS. 2 and 3, the first user interface 110 provides an interface that allows the first mobile device 100 to select a book to be recorded and proceed with recording.

The book manager 120 receives book information related to the user from the book database 230 of the server 200.

The edge time determining unit 130 determines the edge time of the user using the first hardware sensor unit 180 and the first software sensor unit 190.

The first cut time determined by the cut time determiner 130 may be a 'leisure long-term use event'. The leisure long use event may be determined using the app usage monitor 191 of the first software sensor 190.

For example, when the use of a web browser or a social network service application continues for a specific time or more, the utterance time determining unit 130 may determine the leisure long-term use event. For example, the specific time may be set to 90 seconds.

The second cut time determined by the cut time determiner 130 may be an event immediately after a phone call. An event immediately after the phone call may be determined using the event detector 192 of the first software sensor unit 190. When the user makes a phone call, it may mean that the user is in a situation in which a user can speak loudly. Therefore, the cut time determiner 130 may determine immediately after the phone call as one of the cut time.

For example, when it is recognized through the event detector 192 that the telephone call is ended or the telephone call is cancelled, the utterance time determining unit 130 may determine the event immediately after the telephone call.

The third cut time determined by the cut time determiner 130 may be a 'alone walking event'. The walking alone event may be determined using the accelerometer 181 of the first hardware sensor unit 180 and the event detector 192 of the first software sensor unit 190. Alternatively, the walking alone event may be determined using the GPS sensor 184 of the first hardware sensor unit 180 and the event detector 192 of the first software sensor unit 190. The user may determine that the user moves through the accelerometer 181 or the GPS sensor 184, and may determine whether an earphone event occurs through the event detector 192. As the user walks, connecting the earphone to the earphone jack or operating the earphone may be regarded as a situation in which the user walks alone, and the walking alone may be one of the time periods.

That is, when it is determined that the user has moved through the accelerometer 181 or the GPS sensor 184, and when it is determined that the earphone event has occurred through the event detector 192, the edge time determining unit 130 is determined. May be determined as the walking alone event.

The fifth cut time determined by the cut time determiner 130 may be a 'stop event during driving'. When the vehicle is stopped while driving due to a traffic light, a spare time of several tens of seconds to several minutes may occur, so that the edge time determination unit 130 may stop the driving during other driving or other tasks or activities (for example, read a book). Situation). The edge time determination unit 130 may determine a stop event during the driving by tracking the position change of the first mobile device 100 through the GPS sensor 184.

The sixth drop time determined by the drop time determination unit 130 may be a 'waiting event'. The meeting waiting event may be determined by synchronizing calendar information of the first mobile device 100 to determine whether the meeting is held and a meeting participant, and receiving GPS location information of the meeting participant.

The seventh cut time determined by the cut time determiner 130 may be a 'messenger response wait event'. The messenger response waiting event may be determined using the app usage monitor 191 of the first mobile device 100. If the user is talking to another person through a messenger and the user is waiting for a message of the other party, the utterance time determining unit 130 may determine the messenger response waiting event.

The eighth time determined by the dead time determination unit 130 may be a 'walking event in the building'. When you are moving in a building, it is quieter than the outdoor environment, so it may be suitable for recording books. The edge time determination unit 130 may determine the walking event in the building using the GPS sensor 184 and the Wi-Fi signal. The GPS sensor 184 may determine whether the user is in a building, and the Wi-Fi signal may determine whether the user is moving in a building.

The ninth cut time determined by the cut time determiner 130 may be an 'elevator wait event'. Waiting for the elevator may be a suitable situation for recording a book. The edge time determining unit 130 may determine the elevator waiting event by using a magnetic field sensor and an accelerometer of the hardware sensor unit 180.

The TTS manager 140 is a component for supporting the listen-record mode. The TTS manager 140 supports a text-to-speech function. The TTS manager 140 loads the book manager 120 and reproduces the sentence selected in the first user interface 110 as a machine voice.

The recording manager 150 may perform a recording operation according to the first recording mode and the second recording mode. In the first recording mode, a sentence is displayed on the book recording page, and the user's voice is recorded using the microphone 183. In the second recording mode, the machine voice is output using the TTS manager 140, and the voice of the user is recorded using the microphone 183. The start and end of the second recording mode may be determined by the proximity sensor 182.

The notification manager 160 notifies the user in real time when the dead time determination unit 130 detects the dead time. The notification may use vibration or sound. In addition, the notification manager 160 may display a notification bar on the display unit of the first mobile device 100 when the cut time determiner 130 detects the cut time.

The first web API client 170 communicates with the first web API server 210 of the server 200. The first web API client 170 may receive book information from the book database 230 through the first web API server 210, and the second mobile device 300 from the interaction database 260. Interaction information of the user may be received.

The received interaction information may be displayed on the display unit of the first mobile device 100 through the first user interface 110. The received interaction information may be displayed under a book title of the book recording page.

The first database 195 may store information necessary to perform a function of the first mobile device 100. For example, the first database 195 may store a list of books and sentences of the book received from the book database 230 of the server 200. For example, the first database 195 may store the interaction information received from the interaction database 260 of the server 200. For example, the first database 195 may store a recording file generated by the recording manager 150.

The web API server 210 may communicate with the first mobile device 100 through the first web API client 170. In addition, the web API server 210 may communicate with the second mobile device 300 through the second web API client 320.

The book content manager 220 manages book content stored in the book database 230. For example, the book content manager 220 may determine whether books stored in the book database 230 are books related to the user. For example, the book content manager 220 may divide and manage the contents of the book in sentence units.

In addition, the book content manager 220 may generate an audio book by combining the recording files generated by the first mobile device 100. In addition, the book content manager 220 may separately manage recording files in recording file units.

The book database 230 may store sentences of a book.

The recording database 240 may store the recording files generated by the first mobile device 100.

The family database 250 may include information about a relationship between a user of the first mobile device 100 and a user of the second mobile device 300. For example, information about whether the user of the first mobile device 100 and the user of the second mobile device 300 corresponds to a family may be included.

The interaction database 260 receives the interaction information generated from the interaction monitor 330 of the second mobile device 300 and transmits the interaction information to the first user interface 110 of the first mobile device 100. ) Or to the first database 195.

The second user interface 310 provides an interface that allows the second mobile device 300 to select a book to play and to proceed with playback. As described above, the second user interface 310 is similar to the first user interface 110, and the second user interface 310 does not include a record button.

The second web API client 320 communicates with the web API server 210 of the server 200.

The interaction monitor 330 may obtain the interaction information of the user of the second mobile device 300 and transmit the interaction information to the interaction database 260 of the server 200.

The second hardware sensor unit 340 is a sensor in the second mobile device 300, and may include, for example, a camera and a microphone.

The second software sensor unit 350 is a sensor in the second mobile device 300, and may include, for example, an app usage monitor and an event detector.

The second database 360 may store information necessary to perform a function of the second mobile device 300. For example, the second database 360 may store a recording file received from the recording database 240 of the server 200. For example, the second database 360 may store the interaction information generated by the interaction monitor 330.

According to the present embodiment, the user's mobile device 100 may be used to determine various types of utterance time of the user. When the mobile device 100 determines the cut time, the mobile device 100 may send a notification to the user. The user interface 110 may be provided to enable the user to record voice in units of sentences of a book. Therefore, the user can effectively produce an audio book by using the time spent on the utterance.

The present invention can be applied to various electronic devices capable of performing communication functions, such as a user's smart phone, tablet, notebook, PDA, PMP.

While the above has been described with reference to a preferred embodiment of the present invention, those skilled in the art will be able to variously modify and change the present invention without departing from the spirit and scope of the invention as set forth in the claims below. I will understand.

100: first mobile device 110: first user interface
120: book manager 130: cut time judgment
140: TTS Manager 150: Recording Manager
160: notification manager 170: the first web API client
180: first hardware sensor unit 181: accelerometer
182: proximity sensor 183: microphone
184: GPS sensor 190: first software sensor unit
191: Monitor app usage 192: Event detector
195: first database 200: server
210: web API server 220: book content manager
230: book database 240: recording database
250: family database 260: interaction database
300: second mobile device 310: second user interface
320: second web API client 330: interaction monitor
340: second hardware sensor unit 350: second software sensor unit
360: second database

Claims (15)

Obtaining user information using at least one of a hardware sensor unit and a software sensor unit of the mobile device; And
Detecting a moment of low cognitive load based on the obtained information of the user, and determining the utterance time that can perform ancillary work, the utterance time determination method using a mobile device and a sensor. The method of claim 1, wherein the obtaining of the user information comprises:
Method of determining the utterance time using a mobile device and a sensor, characterized in that to obtain the user's context information or behavior information using the hardware sensor unit. The method of claim 1, wherein the obtaining of the user information comprises:
Method of determining the utterance time using a mobile device and a sensor, characterized in that to obtain the mobile device usage information of the user using the software sensor unit. The method of claim 1, wherein the determining of the utterance time of the user
A method of determining a utterance time using a mobile device and a sensor that detects that the user is in a standby state based on the situation information of the user and determines that the utterance time is. The method of claim 1, wherein the determining of the utterance time of the user
A method of determining a utterance time using a mobile device and a sensor that detects that the user is walking or acting alone based on the user's situation information or behavior information. The method of claim 1, wherein the determining of the utterance time of the user
A method for determining a utterance time using a mobile device and a sensor that detects that the user is using a leisure activity service using the mobile device based on the user's mobile device usage information and determines the utterance time. The method of claim 1, wherein the determining of the time of the user's utterance comprises determining an earphone wearing walking event, which means a case where the user walks with wearing the earphone,
The earphone wearing walking event is determined using the accelerometer or GPS sensor of the hardware sensor unit and the earphone event of the event detector of the software sensor unit. The method of claim 1, wherein the determining of the user's utterance time comprises: determining a meeting waiting event, meaning that the user is waiting before a meeting starts,
The meeting waiting event is determined using the calendar information of the mobile device and the GPS location information of the meeting participant, the edge time determination method using a mobile device and a sensor. The method of claim 1, wherein the determining of the utterance time of the user comprises: determining a messenger response waiting event meaning that the user is talking to the other party using a messenger, and the user is waiting for a message of the other party;
The messenger response waiting event is determined using an app usage monitor of the software sensor unit. The method of claim 1, wherein the determining of the time of the user's utterance comprises determining a walking event in the building, which means that the user is moving in the building.
The walking event in the building is determined using a GPS sensor and a Wi-Fi signal of the hardware sensor unit, the edge time determination method using a mobile device and the sensor. The method of claim 1, wherein the determining of the user's utterance time comprises determining an elevator waiting event, which means that the user is waiting for an elevator.
The elevator wait event is determined using a magnetic field sensor and an accelerometer of the hardware sensor unit, the edge time determination method using a mobile device and a sensor. Obtaining user information using at least one of a hardware sensor unit and a software sensor unit of the mobile device; And
Detecting a moment when there is little cognitive load between the scheduled schedules of the user based on the information of the user, and using the mobile device and the sensor comprising the step of determining the time spent on the schedule, but not ancillary work How to determine the time of detention. Acquiring user information using at least one of a hardware sensor unit and a software sensor unit of the mobile device, and detects a moment when the cognitive load is low based on the obtained user information, and can perform ancillary work. A decay time determination system using a mobile device and a sensor to determine a decay time. The method of claim 1, characterized in that the determination of the rim time using the rim time determination method of the method, and using the rim time for audiobook production. A system for judging a drop time by using the drop time determination system of claim 13, and using the drop time for audiobook production.

Images