KR102078131B1 - Electronic device system including portable terminal and server - Google Patents

Abstract

The present invention relates to an electronic device system including a portable terminal and a server. An object of the present invention is to solve problems related to children′s vision loss by adopting a technique for recognizing a user′s overall facial size so that the distance between a smartphone and an infant or a toddler can be more accurately measured and promote a more positive portable terminal use habit for a user by determining, for example, the distance between the user and the terminal, an angle, and a light amount. In the electronic device system including a portable terminal and a server of the present invention, the portable terminal includes: a camera unit generating an image by imaging a user ahead; a sensor unit configured to generate sensing information by measuring various states of the portable terminal; a first communication unit; a display unit configured to perform screen display on the portable terminal; and a first control unit transmitting, to an external server, a setup image and setup sensing information obtained by imaging and sensing for a predetermined time in a setup mode, setting a reference state based on the reference value received from the external server, and generating the user′s current state by analyzing current sensing information and a current image obtained by periodic imaging and sensing in a daily mode. The first control unit controls the display unit to block the screen display operation when it is determined that the user′s state is a correction-required state as a result of comparison between the current state and the reference state. The server includes: a second communication unit; and a second control unit deriving the reference value by analyzing the set sensing information and the setup image received through the second communication unit from the portable terminal and controlling the second communication unit to transmit the reference value to the portable terminal.

Description

ELECTRONIC DEVICE SYSTEM INCLUDING PORTABLE TERMINAL AND SERVER}

The present invention relates to an electronic device system for preventing visual acuity of infants and toddlers and teaching correct portable terminal usage habits. Specifically, the distance between an infant and a toddler, the angle, the amount of light incident on the portable terminal, and the user's eyes flicker The present invention relates to an electronic device system including a portable terminal and a server capable of automatically controlling a screen output according to a degree and displaying a warning message on a display or blocking the display.

When dining in a restaurant or moving in a passenger car, it is easy to find a parent showing various videos to a toddler using a smartphone. According to the results of the research conducted by the Child Care Policy Research Institute, the initial age of smartphone use in Korean infants was 2.27 years on average, and the dependence of smartphones on infants was 19.1. have.

As described above, the frequent use of smartphones in the early childhood period has a fatal evil effect on infants and toddlers with early vision development, and in fact, the incidence of diseases related to vision loss is gradually increasing due to frequent video viewing through a smartphone. These infants and toddlers' eyesight remains a big concern for parents, but they are inevitably using smartphones to manage their children easily.

In order to solve the above problems a little, a number of smart phone-related applications, products, and the like have been released in the current market to protect the eyesight of infants and toddlers. For example, a product known as a product for protecting eyesight of infants and toddlers is a blue light filter. However, the blue light filter alone has a problem in that it is difficult to provide a great help to children's mobile phone usage habits, and there is a clear limitation in technology as it is not a perfect vision protection technology. In addition, an application widely known as an eye protection technology includes a control technology that measures a distance between an infant and a smartphone through a front camera and blocks a screen when a face is recognized within a set distance.

For example, Patent Publication No. 10-1580918 (Patent Document 1) discloses a technique for controlling an output screen of an application program for protecting eyesight of a user. This prior art is a technology related to an application that blocks the display when an image of a set brightness or more is detected due to a user's display being too close to the user's display. However, according to the related art, since the camera needs to be continuously used, a problem of heat generation of the smartphone and excessive battery consumption occurs, and the illuminance sensor used when measuring the distance from the user has too much error depending on the usage environment. There was a problem that the accuracy was significantly reduced.

In addition, the application does not take into account any information on the angle of the mobile phone, the state of the children, the amount of light generated by the smartphone, etc., so the accuracy is greatly reduced and there are problems in that there are various limitations in use. And, when children use a smartphone avoiding the lens of the camera, there is a fundamental problem that it is difficult to measure the distance itself.

Patent Literature 1: Registered Patent Publication No. 10-1952533 (published on February 20, 2019)

The object of the present invention is to introduce a technology for recognizing the entire size of a user's face so that the distance between a smartphone and an infant can be more accurately measured, the problem related to children's vision can be solved, and the distance between the user and the terminal. It is to provide an electronic device system including a portable terminal and a server capable of positively inducing a user's portable terminal usage habit by determining an angle, an amount of light, and the like.

An electronic device system according to an embodiment of the present invention includes an electronic device system including a portable terminal and a server, wherein the portable terminal includes a camera unit that captures a front user to generate an image, and various states of the portable terminal. A sensor unit configured to measure and generate sensing information, a first communication unit, a display unit configured to display a screen on the portable terminal, and a set image and set sensing information taken and sensed for a predetermined time in a setting mode to an external server Transmits, sets the reference state based on the reference value received from the external server, analyzes the current image and current sensing information periodically photographed and sensed in the daily mode to generate the current state of the user and generates the current state and Compared to the reference state, it is determined that the user's state has become a calibration required state. And a first control unit controlling the display unit to block a screen display operation when the screen display operation is blocked, and the server analyzes the setting image and the setting sensing information received from the second communication unit and the second communication unit from the second communication unit and the portable terminal. And a second control unit controlling the second communication unit to derive the reference value and transmit the reference value to the portable terminal.

According to the present embodiment, in the setting mode, the first control unit provides guide information through the display unit to incline the portable terminal at various angles, and the user can tilt the portable terminal at a predetermined angle according to the guide information. Each time the user is photographed to generate a set image from a plurality of angles, and the first communication unit is controlled to transmit the image generated for each of the plurality of angles to the server, and the second control unit is through the second communication unit. It is configured to set the reference states at various angles by further considering a set image generated for each of the plurality of angles received.

According to the present embodiment, in the everyday mode, the first control unit is configured to calculate a distance, posture, and degree of blinking between the user and the portable terminal based on the current image and the current sensing information.

According to this embodiment, the sensor unit includes at least one of a gyro sensor configured to measure the inclination of the portable terminal and an illuminance sensor configured to measure the amount of light incident on the portable terminal.

According to the present embodiment, the first control unit determines the current state by further considering the amount of light incident on the portable terminal.

According to the present embodiment, the first control unit determines the current posture of the user by further considering the tilt information of the portable terminal, and if it is determined that the determined posture is a posture requiring correction, the screen display operation is blocked. Control the display.

According to the present embodiment, the first control unit controls the display unit to display a warning message when it is determined that the current state is a calibration required state, and blocks the screen display operation when it is determined that the calibration required state lasts for a predetermined time or longer. The display unit is controlled to do so.

According to the proposed invention, since accurate distance calculation can be performed by measuring the entire size of the child's face at a distance determined to be ideal between the smartphone and the child, the reliability of notifications such as warning and blocking of the display can be greatly improved. There is an advantage.

In addition, it is possible to detect the distance between the smart phone and the child, the viewing angle of the child, and the amount of light related to ambient light through a camera, an illuminance sensor, a gyro sensor, etc. built in the smart phone, thereby improving the convenience of use. .

In addition, when viewing a portable terminal with a poor posture such as too close a distance to the smartphone, too dark surroundings, or a neck strain, a warning message is output to the user through the display, and the warning message is ignored. By continuously blocking the screen when viewing the portable terminal, it is possible to tame correct portable terminal habits of infants and toddlers, thereby preventing vision loss and correct posture correction.

1 is a block diagram showing a system for controlling a screen output according to a distance between a user and a portable terminal, a user's viewing angle, a user's blinking, and an amount of light incident on the portable terminal according to an embodiment of the present invention.
2 is a schematic diagram illustrating a method of recognizing a user's face according to an angle at which the portable terminal is placed.
3 is a flowchart illustrating a process of setting a reference state that is a reference point for determining whether a user's current state is within a normal range.
4 is a flowchart illustrating a control method for screen output of a portable terminal according to the system shown in FIG. 1.
5 is a flowchart illustrating a process of providing feedback to a user according to application termination.

The present invention can be applied to various changes and can have various forms, and the embodiments are described in detail in the text. However, this is not intended to limit the present invention to a specific disclosure form, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms.

The terms are used only for the purpose of distinguishing one component from other components. The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, terms such as “comprises” or “consisting of” are intended to indicate the presence of features, numbers, steps, operations, components, parts or combinations thereof described in the specification, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a configuration diagram showing a system for controlling a screen output according to a distance between a user and a portable terminal according to an embodiment of the present invention, a user's viewing angle, a user's eye blinking, and an amount of light incident on the portable terminal, FIG. 2 is a schematic diagram showing a method of recognizing a user's face according to an angle at which the portable terminal is placed.

1 and 2, the terminal device 50 according to the present invention receives data sensed by the terminal device 50, analyzes the data, and then analyzes the data analyzed by the terminal device 50 again. It is configured to be able to communicate with the transmitting server 10 with each other. The terminal device 50 may be representatively a smart phone, or may be various electronic devices such as a personal PDA and a monitor. The terminal device 50 and the server 10 may use various communication methods such as Bluetooth and Wi-Fi and communicate with each other.

The data sensed by the terminal device 50 may be information regarding a user's device usage time, posture, and light quantity, especially when the user is an infant or toddler. In order to collect this data, the terminal device 50 may be equipped with various sensing members. Specifically, the terminal device 50 includes a camera 1 for measuring a comprehensive environment between a portable terminal and a user, an illuminance sensor 2 for recognizing the amount of light entering the terminal device 50, and the It includes a gyro sensor (3) for measuring the degree of inclination of the terminal device (50).

The camera 1, the illuminance sensor 2, and the gyro sensor 3 are configured to check the user's habit of using the terminal device 50 at any time. If a situation in which an image is displayed through a smartphone to an infant or toddler occurs, the user executes the application 100 on the terminal device 50.

The application 100 receives information on data measured by the terminal device 50. That is, the application 100 includes the amount of light flowing into the smartphone measured from the camera unit 101 and the illuminance sensor 2 that receives information about the overall surrounding environment photographed from the camera 1 and the It is configured to include a sensor unit 102 for receiving information about the degree of inclination of the smartphone measured by the gyro sensor (3).

In addition, the information measured by the camera unit 101 and the sensor unit 102 is analyzed by the first control unit 105. Specifically, the first control unit 105 analyzes information about the distance between the user and the smartphone, the user's viewing angle, and the amount of light entering the smartphone. Then, the result analyzed by the first control unit 105 is transmitted to the server 10 through the first communication unit 104.

The server 10 may be configured to determine whether the current user's state needs to be corrected based on the results analyzed by the second communication unit 11 and the first control unit 105 communicating with the first communication unit 104. 2 includes a control unit 12. The result derived from the second control unit 12 is transmitted from the second communication unit 11 to the first communication unit 104 again, and the first control unit 105 sends an alarm signal to the user based on the derivation result. send.

That is, the result analyzed by the first control unit 105 may be notified to the outside through the display unit 103. The display unit 103 may be, for example, a screen of a smartphone. That is, when it is determined that the current user's state is compared to the reference state, and the user's state has become a calibration required state, the first control unit 105 controls the display unit 103 to block the screen display operation.

The reference state may be data calculated by the second control unit 12 and transmitted to the first control unit 105 through the second communication unit 11 and the first communication unit 104. That is, the reference state may be referred to as a standard posture that is set by analyzing and setting the posture, the angle, the amount of light entering the smartphone, and the degree of blinking of the user as a whole by viewing the smartphone. have.

If, when the terminal device 50 is operated in a daily mode, the first control unit 105 monitors the state of the user's viewing environment through the camera 1, the illuminance sensor 2, and the gyro sensor 3. Periodically, the current state of the user is set by analyzing the captured results. In addition, the first control unit 105 compares the current status with the reference status transmitted from the second control unit 12 to determine whether the user's status is a condition requiring calibration.

That is, in the everyday mode, the first control unit 105 may calculate a distance between the user and the terminal device 50, the user's viewing posture, the amount of light, and the degree of blinking based on the current image and sensed information. .

The data and calculation data measured through the above process are converted into numerical values through image processing in the first control unit 105 and the second control unit 12, and the converted numerical values are respectively the first control unit 105. And stored in the second control unit 12 may be utilized as a future big data method.

The first communication unit 104 and the second communication unit 11 are mediums that allow the terminal device 50 and the server 10 to communicate with each other, and may be a mobile communication network that is generally used. As a typical example, there is a wideband code division multiple access (WCDMA) communication network, and other next-generation communication networks such as 3G, 4G (LTE), and 5G networks are also possible.

In FIG. 2, an example of overcoming problems related to a distance between a smartphone and a user and distortion of a user's viewing angle is illustrated by recognizing a user's face as a whole in the terminal device 50. Specifically, in the related art, the operation was normally performed only in the situation 200 in which the portable terminal was placed at a specified angle at a specified position. In other words, the conventional terminal device 50 is used when the user artificially aligns the terminal device 50 with a predetermined set position and set angle, and then shoots the image within a normal distance range regardless of the actual position of the user. There was a problem in that a large loophole was revealed.

However, according to the present invention, since the image processing method using deep learning is used in the user's recognition method, it is possible to recognize the user's face much more clearly than the conventional method. For example, even in a situation in which the terminal device 50 is lying horizontally, it is possible to recognize a face in a forward direction based on the entire face of the user, thereby reducing the occurrence of errors.

Similarly, in the situation where the terminal device 50 is lying on its back (202, 203), that is, in a situation in which the user's smartphone viewing angle is poor, the user's face is recognized to appear smaller and smaller than the forward recognition. Accordingly, the viewing angle of the current user can be checked through comparison with a reference state corresponding to the forward direction recognition.

At this time, the inclination of the terminal device 50 can be checked in more detail through the sensor unit 102 of the application 100 interlocked with the gyro sensor 3, and the user's posture is calculated using the calculated values through the angle and formula. Can find out. As described above, by determining the posture of the user, in particular, the viewing angle of the smartphone, the distance between the user and the terminal device 50 can be effectively measured even when the terminal device 50 is lying down.

Hereinafter, an operation method of the terminal device 50 according to a user's posture or the like will be described in detail. 3 to 5 are flowcharts illustrating a control method for screen output of a portable terminal according to the system shown in FIG. 1.

Referring to the drawings, before the user starts the application 100, first, the application 100 must be set with initial information providing a reference value for determining whether the current state of the user is within a normal category. . As shown in FIG. 3, when the user operates the application 100 (S10), the first control unit 105 determines whether a reference state for correct use of the smartphone is stored (S11). That is, the first control unit 105 determines whether the application 100 is initially used and whether it is ready to set a reference value.

The control signal of the first control unit 105 causes the terminal device 50 to operate in a setting mode. In the setting mode, the first control unit 105 provides guide information to incline the terminal device 50 at various angles, and the user can check the guide information through the display unit 103. In this way, whenever the user tilts the terminal device 50 at a predetermined angle according to the guide information, a photographing is performed to generate a set image from a plurality of angles. The images generated for each of the plurality of angles are transmitted to the server 10 through the communication units 104 and 11 (S12).

The server 10 decomposes and disassembles the video for each frame (scene) using Face Detection (face and pupil detection), Haar Cascade (face feature detection), etc., which are image processing filtering methods implemented in the client. The average value is derived by filtering one image. Thereafter, the derived values are left as basic data for deep learning and communicated with the first communication unit 104 through a mobile communication network and transmitted to the terminal device 50.

The second control unit 12 of the server 10 is configured to analyze a set image generated for each of the plurality of angles received through the second communication unit 11 and set reference states at various angles. The set final information is analyzed by the second control unit 12 and transmitted back to the first control unit 105 of the terminal device 50 (S13). The information transmitted to the first control unit 105 through the step S13 becomes initial information regarding a user's posture and the like, and becomes the reference state. In this way, when it is confirmed that the reference state is stored, an initial setting completion signal is transmitted (S14).

That is, in step S14, a reference value is set with data transmitted from the server 10, and the operation cycle of the camera 1, the illuminance sensor 2, and the gyro sensor 3 is set in the first control unit 105. , Set the tolerance range for the distance to the smartphone, the number of blinks, and the amount of light, and end the initial setting process.

The above steps S10 to S14 may not need to proceed any further if an excellent deep learning method comes out and the data storage system is upgraded and universal system values come out. This is because in the general system, the initial value corresponding to the reference state is stored in advance. However, in the present invention, since it is necessary to set the initial value, it is a principle to proceed with steps S10 to S14.

When such initial setting is completed, the user executes the operation of the application 100 (S21). The operation of the application 100 may be activated through a control signal from the first control unit 105.

Then, the first control unit 105 periodically operates the camera 1, the illuminance sensor 2, and the gyro sensor 3 mounted on the terminal device 50 (S22). The periodic operation is performed according to the operation period set in the initial setting step illustrated in FIG. 3, and a result value is transmitted to the first control unit 105 whenever the corresponding operation is performed. The result value for the periodic copper field becomes the current state of the user.

Then, the first control unit 105 compares the current state of the user with the reference state to determine whether the user's posture is within an allowable range (S23). That is, the first control unit 105, the current state of the user transmitted to the first control unit 105 in the periodic operation of the camera 1, the illuminance sensor 2 and the gyro sensor 3 The data value is compared with the reference state data value that is analyzed by the second control unit 12 and transmitted to the first control unit 105.

If the data value for the current state is out of the allowable range of the reference state data value, the first control unit 105 calls the display unit 103 to output a warning message to the user (S24). However, if the data value for the current state is within the allowable range of the reference state data value, the current screen is maintained (S200) and the process returns to step S22 to periodically photograph the current state of the user.

If the user does not correct the posture despite the warning signal from the display unit 103, the first controller 105 sends the signal that the posture correction of the user is necessary, and the duration of the required correction state is the second. The control unit 12 determines whether it lasted longer than the calculated reference time (S25).

If the duration of the calibration required state is longer than the reference time, step S26 of blocking the screen of the terminal device 50 is called. However, if the duration of the calibration required state has not yet reached the reference time, it means that the user has corrected the posture, so maintain the current screen and return to step S22 to periodically photograph the user's current state. .

In FIG. 5, a process of delivering posture-related feedback to a user according to the termination of the application 100 is illustrated. First, when the step of blocking the screen is performed as in step S26 illustrated in FIG. 4, a process in which the application 100 is automatically terminated is performed. As another example, if the user wants to end viewing the video even if the screen is not blocked, the application 100 may be terminated (S31).

As described above, when the application 100 is terminated at the user's own or intentional basis, the first control unit 105 analyzes feedback information in which information about a user's viewing habits and postures for each hour is recorded (S32). Next, the user determines whether or not to receive the feedback information (S33). If the user wants to receive the feedback information, the first control unit 105 informs the user of the information through the display unit 103, and at the same time transmits the feedback information to the server 10 in the future. It is stored as data that can be used to analyze various information (S34). However, if the user does not want to generate the feedback information, the application 100 is terminated as it is without analyzing and storing the corresponding data (S35).

Thus, according to the present invention, it is possible to detect the distance between the smart phone and the child, the child's viewing angle, and the amount of light related to the ambient light through the camera, illumination sensor, gyro sensor, etc. built into the smart phone, thus preventing visual protection for infants and toddlers. There is an advantage that can be actively achieved.

Although described above with reference to embodiments, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that there is.

1: Camera
2: Illuminance sensor
3: Gyro sensor
10: Server
11: second communication unit
12: second control unit
50: terminal device
100: application
101: camera unit
102: sensor unit
103: display unit
104: first communication unit
105: first control unit

Claims (7)

In the electronic device system including a portable terminal and a server,
The portable terminal
A camera unit that photographs a user on the front side to generate an image;
A sensor unit configured to measure various states of the portable terminal and generate sensing information;
A first communication unit;
A display unit configured to display a screen on the portable terminal; And
In the set mode, the set image and the set sensing information shot and sensed for a predetermined time are transmitted to an external server, the reference state is set based on the reference value received from the external server, and the current image periodically photographed and sensed in the daily mode And analyzing the current sensing information to generate the current state of the user, and comparing the current state with the reference state to control the display unit to block the screen display operation when it is determined that the user's state is in need of calibration. 1 includes a control unit,
The server
A second communication unit; And
A second control unit for controlling the second communication unit to derive the reference value by analyzing the setting image and the setting sensing information received from the portable terminal through the second communication unit, and to transmit the reference value to the portable terminal Including, to set the reference state at various angles by further considering the set image generated for each of a plurality of angles received through the second communication unit, the result derived from the second control unit again the second communication unit To the first communication unit,
In the setting mode, the first control unit provides guide information through the display unit to tilt the portable terminal at various angles, and photographs the user whenever the user tilts the portable terminal at a predetermined angle according to the guide information. To generate a set image from a plurality of angles, and to control the first communication unit to transmit the image generated for each of the plurality of angles to the server.
delete According to claim 1,
In the everyday mode, the first control unit is configured to calculate a distance, posture, and degree of blinking between a user and a portable terminal based on the current image and the current sensing information.
According to claim 1,
The sensor unit includes an electronic device system including at least one of a gyro sensor configured to measure a tilt of a portable terminal and an illuminance sensor configured to measure an amount of light incident on the portable terminal.
According to claim 4,
The first control unit is an electronic device system that determines the current state by further considering the amount of light incident on the portable terminal.
According to claim 4,
The first control unit determines the current posture of the user by further considering the tilt information of the portable terminal, and when it is determined that the determined current posture is a posture requiring correction, an electronic device that controls the display unit to block the screen display operation system.
According to claim 1,
The first control unit controls the display unit to display a warning message when it is determined that the current state is a calibration-needed state, and controls the display unit to block the screen display operation when it is determined that the calibration-needed state continues for a predetermined time or more. Electronic system.

Images