JP7175589B2 - Notification system, terminal device, server - Google Patents

Description

The present invention relates to a notification system, a terminal device, and a server for notifying a user of information from an electronic device.

Conventionally, in air purifiers, it is common to use an LED lamp or the like to indicate when a problem occurs, such as when the filter needs to be cleaned or when the ion generator needs to be replaced. is. The LED lamp is arranged in the operation display section of the air purifier.

In addition, in Patent Document 1, the results of detecting moisture, dust, odors, pollen, etc. contained in an air-conditioned space by an air purifier are captured by a terminal device using AR (Augmented Reality) technology. Techniques have been disclosed for displaying as a virtual image on the image of the purifier.

JP 2014-206291 A

In a configuration in which information from the air purifier to the user is displayed using LED lamps or the like, as the amount of information increases, the number of LED lamps increases or the lighting pattern of the LED lamps becomes complicated, making it difficult to understand. In addition, when the number of LED lamps increases, the operation display section becomes miscellaneous, and the design of the product deteriorates.

Even if the technique of Patent Document 1 is used, it is not possible to reduce the number of LED lamps that indicate problems occurring in the air purifier.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and is intended to allow a user to easily grasp a problem when a problem occurs in the electronic device while simplifying the configuration of the operation display unit in the electronic device. The purpose is to provide a notification system, a terminal device, and a server that can

In order to solve the above problems, a notification system according to an aspect of the present invention includes an electronic device, a terminal device connected to the electronic device via a network, and between the electronic device and the terminal device: an intervening server, wherein the terminal device includes an imaging unit, a display unit that displays an image captured by the imaging unit, and a display control unit that controls display by the display unit and transmitting an image including the electronic device captured by the imaging unit to the server , and the server is combined with the image based on the image including the electronic device captured by the imaging unit. Additional information, which is combined with a portion of the image portion of the electronic device where the problem occurs in the electronic device, and indicates that the problem has occurred in the combined portion of the image portion of the electronic device. an additional information determining unit for determining additional information to be displayed, wherein when an image including the electronic device captured by the photographing unit is received from the terminal device, the additional information determining unit determines based on the image Additional information is transmitted to the terminal device, and when the display control unit receives additional information determined based on the image transmitted to the server, a problem occurs in the electronic device in the image including the electronic device. a composite image obtained by synthesizing the additional information with the image portion of the portion where the image is captured is displayed on the display unit as an image captured by the imaging unit, and the terminal device serves as a terminal device of the user of the electronic device. registered, the electronic device transmits state information indicating its own state including the operating state to the server; the server stores the state information received from the electronic device; A message is sent to the terminal device to prompt the electronic device to take a picture .
In order to solve the above problems, a notification system according to another aspect of the present invention includes an electronic device that blows airflow, a terminal device that is connected to the electronic device via a network, the electronic device and the terminal. a server interposed between a terminal device and a device, wherein the terminal device includes a photographing unit, a display unit for displaying an image photographed by the photographing unit, and a display for controlling display by the display unit. a control unit for transmitting an image including the electronic device captured by the capturing unit to the server, the server based on the image including the electronic device captured by the capturing unit; Additional information that is combined with an image, and is combined with a portion of the image portion of the electronic device where the problem occurs in the electronic device, causing a problem in the combined portion of the image portion of the electronic device. an additional information determining unit for determining additional information indicating that the electronic device is being used, and upon receiving an image including the electronic device photographed by the photographing unit from the terminal device, the additional information determining unit based on the image to the terminal device, and when the display control unit receives the additional information determined based on the image transmitted to the server, the electronic device in the image including the electronic device displaying, on the display unit, a composite image in which the additional information is combined with the image portion of the location where the problem occurs in the device, as the image captured by the imaging unit; When the terminal device requests additional information indicating the airflow blown out from the electronic device, based on the captured image of the electronic device and the installation space of the electronic device, and the information on the air volume obtained from the electronic device. to determine additional information indicating the airflow in the installation space.

According to one aspect of the present invention, there is provided a notification system that allows a user to easily grasp a problem when an electronic device has a problem while the operation display unit of the electronic device has a simple configuration. There is an effect that it can be provided.

1 is a block diagram showing the overall configuration of a notification system according to this embodiment; FIG. 2 is a block diagram showing the hardware configuration of a user terminal included in the notification system shown in FIG. 1; FIG. 2 is a block diagram showing the hardware configuration of a server included in the notification system shown in FIG. 1; FIG. 2 is a block diagram showing the hardware configuration of an air purifier included in the notification system shown in FIG. 1; FIG. FIG. 3 is a block diagram showing a functional configuration of main parts of the user terminal; It is a block diagram which shows the functional structure of the principal part of the said server. It is a block diagram which shows the functional structure of the principal part of the said air cleaner. FIG. 10 is a diagram showing an AR image displayed on the display unit when the user terminal captures an image of the air purifier in which the amount of water remaining in the tank is low and water needs to be replenished. FIG. 10 is a diagram showing an AR image displayed on the display unit when the user terminal captures an image of the air purifier in which the humidifying filter is dirty and needs to be cleaned. It is time to replace the ion generator, and is a diagram showing an AR image displayed on the display unit when an image of the air purifier in a state requiring replacement of the ion generator is captured by the user terminal. FIG. 10 is a diagram showing an AR image in which an image such as an arrow is attached as additional information to guide an imaging region so that a problematic portion is included; FIG. 10 is a diagram showing an AR image displayed on the display unit when the user terminal captures an image of the air purifier whose fan motor is out of order and needs repair. FIG. 10 is a diagram showing an AR image displayed on the display unit when an image of the air purifier in which dust is attached to the filter of the suction port and the external filter needs to be cleaned is captured by the user terminal. FIG. 10 is a diagram showing an AR image displayed on the display unit when the user terminal captures an image of the air purifier with the plug unplugged from the outlet; FIG. 10 is a diagram showing an AR image displayed on the display unit when the user terminal captures an image of the air purifier in a state where the operation button is not pressed and the air purifier is not in use. (a) to (c) are diagrams showing an AR image to which an image of a message indicating a procedure for cleaning a humidifying filter is attached as additional information. 10(a) to 10(c) are diagrams showing AR images continued from FIG. 9. FIG. 11(a) to 11(c) are diagrams showing AR images continued from FIG. 10; FIG. 12(a) to 12(c) are diagrams showing AR images continued from FIG. 11; FIG. 13A and 13B are diagrams showing AR images continued from FIG. 12; FIG. FIG. 11 is a block diagram showing the functional configuration of main parts of a server in a notification system according to another embodiment; It is a figure which shows the AR image displayed on a display part when the instruction|indication method of a driving mode is asked. FIG. 11 is a block diagram showing the functional configuration of main parts of a server in a notification system according to another embodiment; FIG. 10 is a diagram showing an AR image displayed on the display unit when a request is made to visualize the airflow from the air outlet; FIG. 10 is a diagram showing an AR image displayed on the display unit when a request is made to visualize the airflow in the installation space; FIG. 10 is a diagram showing an AR image displayed on the display unit when a request is made to visualize the airflow in the installation space; It is a block diagram which shows the whole structure of the notification system of other embodiment.

[Embodiment 1]
An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a notification system 1 of this embodiment. The notification system 1 includes a user terminal 10 , a server 20 , an air cleaner (electrical device) 30 and a network 40 . The user terminal 10 has already been registered with the air purifier 30 as a user's terminal device.

The user terminal 10 is a communication terminal such as a smart phone or tablet of the user of the air purifier 30 . The user terminal 10 captures an image including the air purifier 30 as a subject with the capturing unit 14 (see FIG. 2), transmits the image data to the server 20, and based on the AR image data received from the server 20 It has a program necessary for performing processing for displaying an image on the display unit 15 (see FIG. 2).

The air purifier 30 is, for example, a device having an air cleaning function of sucking air to remove foreign matter such as dust contained in the air and a humidifying function of humidifying the sucked air. Although not shown, the air purifier 30 includes a fan motor, a humidification filter, a tank, a collection filter, an ion generator, and the like. The fan motor is for generating an airflow, and the humidification filter is for humidifying the air. The tank stores water to be supplied to the humidification filter, and the collection filter collects fine dust from the air. The ion generator generates active species that act to purify the air, or ions that generate the active species, by diffusing them in an air current. Also, the intake is provided with an external filter to remove relatively large dust particles from the air.

The air purifier 30 transmits to the server 20 status information indicating its own status including the operating status. The status information includes information on problematic locations in the air purifier 30 . For example, if there is a failure, the server 20 includes information that allows the location of the failure to be identified. In addition, when the air purifier 30 requires maintenance, the server 20 includes information that allows the server 20 to specify the required maintenance location. Maintenance includes cleaning, cleaning, parts replacement and repair. Further, in the present embodiment, as a more preferable configuration, in addition to the information of the problematic portion, information that allows the server 20 to specify the content of the problem is included.

The server 20 generates additional information to be added to the image of the received image data based on the image data of the air purifier 30 received from the user terminal 10 and the state information received from the air purifier 30. is. In this embodiment, the server 20 generates AR image data by superimposing the generated additional information on the image of the received image data, and transmits the AR image data to the user terminal 10 .

The server 20 stores the status information received from the air cleaner 30 and identifies the status of the air cleaner 30 from the status information. If the air cleaner 30 is out of order, the server 20 identifies the location of the failure, and if maintenance is required, specifies the necessary maintenance. In addition, if the operation button of the air purifier 30 is not pressed, the state is specified, and if the state information cannot be received, the plug is unplugged or there is a failure such that the power cannot be turned on. to be specified.

The network 40 is a communication network for transmitting and receiving data such as audio and video. The air purifier 30 and the network 40 are connected via a WiFi (registered trademark) router, a WiFi (registered trademark) access point, or the like. The user terminal 10 and the network 40 are connected using 4G (4th Generation), LTE (Long Term Evolution), an outdoor or indoor WiFi (registered trademark) AP, or the like.

FIG. 2 is a block diagram showing the hardware configuration of the user terminal 10. As shown in FIG. The user terminal 10 includes a control section 11 , a storage section 12 , a communication section 13 , an imaging section 14 , a display section 15 , an audio input/output section 16 and an operation section 17 .

The control unit 11 controls operations of the user terminal 10 . The control unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit) and a memory, and controls video shooting, data communication, and the like by executing a predetermined program. The storage unit 12 stores data.

The communication unit 13 transmits and receives data via the network 40 . The photographing unit 14 photographs a subject and outputs image data such as moving image data and still image data. The photographing unit 14 is used to photograph an image including the air purifier 30 as a subject.

The display unit 15 displays an image based on the image data captured by the imaging unit 14 . Further, the display unit 15 displays an image based on AR image data (described later) received from the server 20 .

The audio input/output unit 16 inputs and outputs audio, and includes a speaker and a microphone. The operation unit 17 receives user operations, and includes operation buttons and a touch screen provided over the display of the display unit 15 .

FIG. 3 is a block diagram showing the hardware configuration of the server 20. As shown in FIG. The server 20 includes a control section 21 , a storage section 22 and a communication section 23 . The control unit 21 includes an arithmetic processing unit such as a CPU and a memory, and controls the operation of the server 20 by executing a predetermined program. The storage unit 22 stores data, and the communication unit 23 transmits and receives data.

FIG. 4 is a block diagram showing the hardware configuration of the air cleaner 30. As shown in FIG. The air purifier 30 includes a control section 31 , a storage section 32 , a communication section 33 , a device body section 34 and an operation display section 35 . The control unit 31 includes an arithmetic processing unit such as a CPU and a memory, and controls the operation of the air cleaner 30 by executing a predetermined program. The storage unit 32 stores data, and the communication unit 33 transmits and receives data.

The device main body 34 performs an air cleaning operation and a humidifying operation. The operation display unit 35 receives a user's operation of the air purifier 30 and displays information to be notified from the air purifier 30 to the user with an LED lamp or the like.

FIG. 5 is a block diagram showing the functional configuration of main parts of the user terminal 10. As shown in FIG. The user terminal 10 implements the functions shown in FIG. 5 by executing a predetermined program. The control unit 11 of the user terminal 10 includes a shooting control unit 11a, a transmission unit 11b, a reception unit 11c, and a display control unit 11d.

The imaging control section 11 a controls the operation of the imaging section 14 . The transmission unit 11 b transmits image data of an image including the air purifier 30 as a subject, captured by the imaging unit 14 , to the server 20 using the communication unit 13 . In the present embodiment, moving image data captured in display mode is transmitted to server 20 . The display mode is a mode before recording or photographing a still image in the video mode or photo mode in the photographing unit 14 , and is a mode in which the photographed image is simply displayed on the display unit 15 .

The receiving unit 11 c receives AR image data from the server 20 . The AR image data is obtained by adding additional information to the image data transmitted from the transmission unit 11b. Based on the AR image data received from the server 20, the display control unit 11d displays an AR image including the air purifier 30 as a subject and having additional information added (superimposed) on the display unit 15. .

The user terminal 10 receives AR image data from the server 20 while transmitting image data captured in the display mode, and displays an AR image on the display unit 15 based on the AR image data. Since the processing by the server 20 is performed in a relatively short time, the AR image is displayed in real time.

FIG. 6 is a block diagram showing the functional configuration of main parts of the server 20. As shown in FIG. The server 20 implements the functions shown in FIG. 6 by executing a predetermined program. The control unit 21 of the server 20 includes a reception unit 21a, a state information acquisition unit 21b, an image analysis unit 21c, a state analysis unit 21d, an additional information determination unit 21e, an AR image generation unit 21f, and a transmission unit 21g. Prepare.

The receiving unit 21 a receives image data captured by the user terminal 10 from the user terminal 10 . The state information acquisition section 21b acquires state information from the air purifier 30 .

The image analysis unit 21c analyzes the image data received by the reception unit 21a, and identifies the direction, part, etc. of the air cleaner 30 being photographed from the image of the air cleaner 30 included in the image. is.

The state analysis unit 21d identifies the state of the air cleaner 30 by analyzing the state information acquired by the state information acquisition unit 21b. For example, it identifies the state in which the humidification filter needs to be cleaned, the ion generator needs to be replaced, the fan motor is out of order, the external filter needs to be cleaned, and the like.

The additional information determining unit 21e combines the information such as the direction of the air purifier 30 identified by the image analyzing unit 21c and the included parts with the state of the air purifier 30 identified by the state analyzing unit 21d. Based on this, it is information to be notified to the user and determines additional information to be synthesized as an AR image. The additional information is an image such as a circle mark or an arrow that is attached to a problematic location and indicates the problematic location. In the present embodiment, the additional information also includes an image of a message indicating the content of the problem and a coping method for solving the problem.

If there is no information to be notified to the user, the additional information is set to "no additional information" or "there is no information to be notified".

The AR image generation unit 21f uses AR (Augmented Reality) technology to create AR image data by combining the image data received by the reception unit 21a with the additional information determined by the additional information determination unit 21e. If there is no additional information, the received image data is directly sent to the transmission section 21g.

The transmission unit 21 g transmits the AR image data received from the AR image generation unit 21 f to the user terminal 10 using the communication unit 23 .

FIG. 7 is a block diagram showing the functional configuration of main parts of the air purifier 30. As shown in FIG. The air cleaner 30 implements the functions shown in FIG. 7 by executing a predetermined program. The air purifier 30 includes a body controller 31a, a state information generator 31b, and a transmitter 31c.

The body control section 31 a controls the operation of the device body section 34 . The state information generation section 31b generates state information indicating the state of the device body section 34 based on information transmitted from the body control section 31a. The transmission unit 31 c transmits the state information created by the state information generation unit 31 b to the server 20 using the communication unit 33 . Although not shown in FIGS. 6 and 7, the server 20 and the air cleaner 30 communicate with each other.

Next, additional information added to a captured image including the air purifier 30 will be described with a specific example.

FIG. 8A is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 in a state in which the remaining water amount in the tank is low and water needs to be replenished. Note that in the present embodiment, the AR image data is moving image data, so FIG. 8A corresponds to one frame image of the AR image.

In the example of FIG. 8A, a circle mark is attached as additional information to the location that the user accesses when replenishing water in the tank, and a message prompting replenishment of water, such as "Please fill the tank with water" is also included. is displayed. The ○ mark and the image of the message are connected by the image of the leader line extending from the ○ mark. In the present invention, the message is not essential as the additional information, and it is sufficient if at least a circle mark or an arrow is attached so that the user can recognize the problematic part.

The flow of information and data until the AR image in FIG. 8A is displayed will be described with reference to FIG. When the remaining water amount in the tank becomes low, the air purifier 30 transmits status information indicating that the remaining water amount in the tank is low to the server 20 . The server 20 acquires the state information by the state information acquisition unit 21b, and sends the acquired state information to the state analysis unit 21d. The state analysis unit 21d analyzes the state information to specify that the remaining water amount in the tank is low, and sends the specified information to the additional information determination unit 21e.

When the user starts a predetermined program on the user terminal 10 and takes an image of the air purifier 30 in the display mode while the additional information determination unit 21e holds the information indicating that the remaining water in the tank is low, the image is taken. Image data is transmitted from the user terminal 10 to the server 20 . The server 20 receives image data using the receiving unit 21a, and sends the received image data to the image analysis unit 21c and the AR image generation unit 21f.

The image analysis unit 21c analyzes the image data, identifies the direction and part of the air purifier 30 in the image from the air purifier 30 included in the image, and sends the identified information to the additional information determination unit 21e. send to

The additional information determination unit 21e receives the information that the remaining water amount in the tank is low received from the state analysis unit 21d, and the orientation of the air cleaner 30 and the direction of the air cleaner 30 received from the image analysis unit 21c. Additional information to be synthesized with the moving image data received from the air purifier 30 is determined based on the information such as the part. The image of the additional information is, for example, a circle mark, and the added position is, for example, the location where the tank is located or the location accessed by the user when refilling the tank with water. The determined additional information is sent to the AR image generator 21f.

The AR image generation unit 21f uses AR technology to create AR image data by synthesizing the image data received from the user terminal 10 with the image of the additional information determined by the additional information determination unit 21e. The created AR image data is sent to the transmission unit 21g, and the transmission unit 21g uses the communication unit 23 to transmit the AR image data to the user terminal 10. FIG. The user terminal 10 displays the AR image on the display unit 15 based on the AR image data received from the server 20 .

FIG. 8B is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 whose humidifying filter is dirty and needs to be cleaned. In the example of FIG. 8B, a circle mark is attached as additional information to the location that the user accesses when pulling out the humidifying filter, and a message such as "Please clean the humidifying filter" is also displayed.

As additional information, if only circles or arrows are attached to problematic locations, water supply to the tank and cleaning of the humidifying filter are at the same location, and the difference in content of the problem cannot be understood. Therefore, instead of circles and arrows, for water supply to the tank, for example, a face-shaped icon with a thirsty expression is attached, and for cleaning the humidifying filter, dirt is attached. It is preferable to indicate the difference in problematic content by distinguishing between them, for example, by attaching a face-shaped icon that looks like a person.

FIG. 8C is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 in a state in which it is time to replace the ion generator and the ion generator needs to be replaced. be. In the example of FIG. 8C, a circle mark is attached as additional information to the location that the user accesses when replacing the ion exchanger, and a message such as "Please replace the ion exchanger" is also displayed. .

Since FIG. 8C is an image of the rear side of the air purifier 30 photographed obliquely from above, the image includes a portion accessed by the user when replacing the ion exchanger. However, for example, when the air purifier is photographed from the front side as shown in FIGS. 8A and 8B, the access location is not included. Therefore, if the image captured by the user does not include the problematic portion of the air purifier 30, for example, as shown in FIG. image may be added as additional information to guide the imaging part so that the problematic part is included.

FIG. 8E is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 whose fan motor is out of order and requires repair. In the example of FIG. 8E, a circle mark is added as additional information to the location where the fan motor is located, and a message such as "The fan motor is out of order. Please request repair."

FIG. 8F is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 in which dust is attached to the filter of the suction port and the filter needs to be cleaned. . In the example of FIG. 8F, a circle mark is attached to the external filter on the back side of the air purifier 30 as additional information, and a message such as "please clean the external filter" is also displayed.

FIG. 8G is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 with the plug unplugged from the outlet. In the example of FIG. 8G, a circle mark is added as additional information to the location of the pull cord from the back side of the air purifier 30, and the connection (handling) such as "The plug is unplugged. Please plug it into the outlet." method) is also displayed.

FIG. 8H is a diagram showing an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 that is not in use because the operation button is not pressed. In the example of FIG. 8H, a circle mark is added as additional information to the place of the operation button of the operation display unit 35 on the top surface of the air purifier 30, and a message such as "Press the operation button to use" is also displayed. is displayed.

As described above, according to the notification system 1 of the present embodiment, when the user takes an image of the air purifier 30 using the user terminal 10 when there is a problem with the air purifier 30, the problem is displayed on the display unit 15. A certain part is superimposed (synthesized) on the image of the air purifier 30 and displayed.

As a result, the user can know that the air purifier 30 has a problem simply by taking an image of the air purifier 30 using the user terminal 10, and at the same time, can identify the problem location. can. Since the problem location is synthesized with the photographed image and directly displayed, it is possible to grasp it more intuitively than the conventional configuration in which the LED lamp of the operation display unit 35 of the air purifier 30 is turned on and notified. , there is no need to know the relationship between lighting and notification content.

In addition, since the number of LED lamps and the like in the operation display section 35 can be reduced, the design of the air purifier 30 can be improved.

Furthermore, in the present embodiment, the location of the problem, the content of the problem, and the method of coping with it are also displayed in the AR image as additional information, so that the user can know the content of the problem and the coping method thereof at the same time. and can respond quickly.

Moreover, since the notification is received while the user is near the air purifier 30, the user can immediately work on solving the problem on the spot.

A notification from the server 20, for example, may be a trigger for the user to photograph the air purifier 30 using the user terminal 10. FIG. The server 20 notifies the user terminal 10 of a message such as "There is a request from the air purifier. Select [Diagnose the air purifier] and take a picture of the air purifier." Alternatively, a predetermined LED lamp on the operation display unit 35 of the air purifier 30 may blink. Alternatively, notification may be made by moving a louver provided at the outlet of the air purifier 30 in an unusual motion. Further, the mechanism may be such that when the user feels that the movement (condition) of the air purifier 30 is not good, the photograph is taken without notifying the difference 0 and the air purifier 30 .

Although the AR image generator 21f is arranged on the server 20 side in the present embodiment, it may be arranged on the user terminal 10 as well. In this case, the server 20 sends the additional information determined by the additional information determination unit 21e to the user terminal 10. FIG. The user terminal 10 creates AR image data by synthesizing the image data captured by the imaging unit 14 with the additional information received from the server 20 , and displays the AR image data on the display unit 15 .

Further, in the present embodiment, moving image data is used in order to transmit an image shot in the display mode. However, in the case of displaying a problem location and a coping method in the air purifier 30 as a message, it is not always necessary to use moving image data. That is, the user terminal 10 shoots a still image including the air purifier 30 as a subject and transmits the image to the server 20, and the additional information determination unit 21e on the server 20 side determines additional information based on the received still image. It may be a configuration.

[Embodiment 2]
Another embodiment of the present invention will be described. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

Instead of or in addition to the additional information indicating the location of the problem, the notification system 1 of the present embodiment is synthesized in association with the location to be accessed by the user, and instructs how to deal with the problem that has occurred. , the additional information is determined step by step according to the progress of the processing.

The additional information determination unit 21e (see FIG. 6) synthesizes the information to be notified to the user in association with the location to be accessed by the user, and provides an instruction to deal with the problem that has occurred according to the progress of the process. to determine the additional information to be shown step by step. When the user takes an image so that the marked portion is within the screen of the display unit 15 while the problematic portion is marked with a circle, the additional information determination unit 21e indicates a procedure for dealing with the problem. Determine additional information. The additional information determination unit 21e determines the progress of processing by the user based on the image data received from the user terminal 10, and changes the content of the message indicating the procedure in stages based on the progress of the processing.

A specific description will be given with reference to FIGS. 9 to 13. FIG. 9 to 13 are AR images showing the procedures for coping. 9 to 13 correspond to one frame image of an AR image that is a moving image.

While the AR image to which the message (additional information) prompting the cleaning of the humidifying filter is displayed as shown in FIG. , and as shown in FIG. 9(b), an AR image is displayed in which the locations to be accessed when the user pulls out the tank are circled. In the image, a message explaining how to pull out the tank, such as "Push to pull out the tank," is also displayed on a lead line extending from the circle. In addition, a leader line connecting the letters "Tank" and the position of the tank is also displayed so that the user can see which part is the tank.

The additional information determination unit 21e specifies that the state analysis unit 21d is in a state in which the humidification filter needs to be cleaned, and the image of the received image data is the air purifier 30 to be accessed when the humidification filter is cleaned. If the image is taken from the side, the additional information is a message indicating the procedure for cleaning the humidifying filter.

The user can easily grasp the content of the operation by checking the AR image shown in FIG. 9B. Moreover, at this time, the user can operate while photographing his or her hand, so that the user can operate while confirming the AR image (moving image) including his/her own hand, as shown in FIG. I can handle it.

When the removal of the tank is completed, an AR image with a message "Please pull out the tray" is displayed as shown in FIG. 10(a). If the image of the received image data is an image in which the tank is removed, the additional information determination unit 21e determines that the tank has been pulled out, and displays the message for instructing the pulling out of the next tray. switch. The AR image also displays a leader line connecting the letters "tray" and the position of the tray so that the user can see which part is the tray.

The user confirms the AR image shown in (a) of FIG. 10 and performs the next tray pullout operation. As in the case of pulling out the tank, by operating while shooting the hand, as shown in (b) and (c) of FIG. You can deal with it smoothly.

When the tray has been pulled out, an AR image with a message such as "Please remove the humidifying filter" is displayed as shown in FIG. 11(a). If the image of the received image data is an image of the tray pulled out from the main body, the additional information determination unit 21e determines that the tray has been pulled out, and instructs removal of the next humidifying filter. message. The AR image also displays a leader line that connects the letters "humidification filter" and the position of the humidification filter so that it can be seen which part is the humidification filter.

The user confirms the AR image shown in (a) of FIG. 11 and performs the following operation of removing the humidifying filter. Here too, by operating while shooting the hand, as shown in (b) and (c) of FIG. can.

When the removal of the humidifying filter is completed, an AR image with a message "Please wash the humidifying filter with water" is displayed as shown in FIG. 11(c). If the image of the received image data is an image of the humidifying filter removed from the tray, the additional information determining unit 21e determines that the removal of the humidifying filter is completed, and instructs cleaning of the humidifying filter. message.

When the washing of the humidifying filter is completed and the user photographs the washed humidifying filter, an AR image with a message such as "Please attach the humidifying filter to the tray" is displayed as shown in FIG. 12(a). be done. When the image of the received image data is an image obtained by photographing the cleaned humidifying filter, the additional information determining unit 21e determines that the cleaning of the humidifying filter is completed, and switches to a message instructing attachment of the humidifying filter. .

The user confirms the AR image shown in (a) of FIG. 12 and performs the following operation of attaching the humidifying filter. When the humidifying filter is attached, an AR image with a message such as "Please insert the tray into the main body" is displayed as shown in FIG. 12(b). When the image of the received image data is an image in which the humidification filter is attached to the tray, the additional information determination unit 21e determines that the attachment of the humidification filter is completed, and inserts the tray into the main body. switch to a message that instructs

The user confirms the AR image shown in FIG. 12(b) and performs an operation of inserting the next tray into the main body. Again, by operating the user terminal 10 while photographing the user's hand, as shown in (b) and (c) of FIG. You can deal with it smoothly.

When the insertion of the tray is completed, as shown in FIG. 13(a), an AR image with a message such as "Please attach the tank to the main body" is displayed. If the image of the received image data is an image in which the tray is inserted into the main body, the additional information determination unit 21e determines that the tray has been completely inserted, and instructs the tank to be attached to the main body. message.

The user confirms the AR image shown in (a) of FIG. 13 and performs the operation of attaching the next tank to the main body. When the mounting of the tank is completed, as shown in FIG. 13(b), an AR image with a message such as "Press the operation button to restart operation" is displayed. If the image of the received image data is an image in which the tank is attached to the main body, the additional information determination unit 21e determines that the attachment of the tank is completed, and outputs a message instructing restart of operation. switch. The position of the operation button is marked with a circle in the AR image.

As described above, according to the notification system 1 of the present embodiment, when there is a problem with the air purifier 30, the user terminal 10 takes an image of the problem location, thereby displaying a coping method. In addition, the content of the instruction is switched step by step according to the captured image (moving image).

As a result, problems can be easily dealt with by operating according to the contents instructed by the AR image without checking the instruction manual enclosed with the air purifier.

In addition, the user can perform operations while capturing an image of his or her hand while checking an AR image (moving image) including his/her own hand, and can smoothly handle the operation without hesitation. In particular, since the instruction is superimposed on the image taken by the user himself/herself, it is much easier to understand than the explanatory diagrams and photographs in the instruction manual whose orientation is fixed.

In the present embodiment, when the side of the air purifier 30 to be accessed when cleaning the humidifying filter is photographed in a state where the humidifying filter needs to be cleaned, a message indicating the procedure for cleaning the humidifying filter is displayed. It is configured to display additional information. However, as will be described in the third embodiment described below, it is possible to provide a request analysis unit that analyzes a user's request, and to display additional information of a message indicating a procedure for coping when requested by the user. good too.

[Embodiment 3]
Another embodiment of the present invention will be described. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

In the notification system 1 of the present embodiment, instead of or in addition to the above configuration, the server 20 analyzes request information indicating a user's request transmitted from the user terminal 10, uses the requested information as additional information, It is configured to be displayed in conformity with a photographed image (either a moving image or a still image). In this embodiment, when the user terminal 10 requests additional information on how to operate the air purifier 30, the information is synthesized in association with the location to be accessed by the user in order to execute the operation. determine additional information indicating the instruction of

FIG. 14 is a block diagram showing the functional configuration of main parts of the server 20 in the notification system 1 of this embodiment. As shown in FIG. 14, in the server 20, a request analysis section 21h is added. The receiving unit 21 a receives, from the user terminal 10 , request information indicating the content to be displayed as additional information in addition to the image data captured by the user terminal 10 . Request information is input from the operation unit 17 or the voice input/output unit 16 of the user terminal 10 (see FIG. 2).

The request analysis unit 21h analyzes the request information received by the reception unit 21a and specifies the content to be displayed as additional information. For example, if the requested information is a question such as "I want to purify the air as soon as possible. What should I do?" "How to instruct the immediate cleaning mode". The analysis result of the request analysis section 21h is sent to the additional information determination section 21e.

The additional information determination unit 21e determines the user's determines additional information that should be notified to. If the content of the request is "how to instruct the immediate cleaning mode", put a circle on the position of the button such as "ion shower" on the operation display unit 35 of the air purifier 30, and indicate "Please press this button. ” is the additional information.

FIG. 15 shows an AR image displayed on the display unit 15 when the user terminal 10 captures an image of the air purifier 30 and asks, "I want to purify the air as soon as possible. What should I do?" It is a diagram. In the AR image shown in FIG. 15, the position of the ion shower button that can instruct the operation in the immediate cleaning mode is circled, and a message such as "Press this button" is displayed.

Here, the operation mode is illustrated as an example of the operation method, but additional information can be displayed in response to a request for performing the operation method such as the various operation modes described in the enclosed instruction manual. In addition, data is stored in the storage unit 22 (see FIG. 3) of the server 20 .

According to this, even if the user is not familiar with the various operation modes of the air purifier 30, the user terminal 10 requests the operation method for executing the desired operation mode to the server 20, and the operation display unit 35 is displayed on the user terminal 10. By simply taking a picture so as to fit within the screen of the display unit 15, the operation method can be known as additional information.

In this case as well, instructions are superimposed on the image taken by the user himself/herself, so that the instructions are much easier to understand than the explanatory diagrams and photographs in the instruction manual whose orientation is fixed.

Further, in the present embodiment, the method of instructing various operation modes is requested and the method of instruction is added as additional information, but the method of instructing the operation mode is not limited to this. For example, for each part of the air purifier 30, such as the operation button, the air outlet, and the external filter, descriptions such as those described in the enclosed instruction manual are stored in the storage unit 22 as additional information, and the user can A configuration may be adopted in which additional information corresponding to a part of the air purifier 30 that the user wants to know the function or description of is photographed.

[Embodiment 4]
Another embodiment of the present invention will be described. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

In the notification system 1 of the present embodiment, when the additional information determination unit 21e is requested by the user terminal 10 to display the additional information indicating the airflow blown out from the air cleaner 30, the received image of the air cleaner 30 and the , and information on the amount of air acquired from the air purifier 30, additional information indicating the airflow is determined. The user takes an image of the space in which the air purifier 30 is installed with the user terminal 10 and transmits the image to the server 20, and the additional information determination unit 21e determines an image corresponding to the air flow in the space as additional information. do. When showing the movement of the airflow, it is preferable that the image of the airflow moves along the flow.

FIG. 16 is a block diagram showing a functional configuration of main parts of server 20 in notification system 1 of the present embodiment. As shown in FIG. 16, in the server 20, a fluid analysis unit 21i is added. The image analysis unit 21c analyzes the image data to specify the orientation of the air purifier 30 being photographed, the position of the outlet, the orientation of the louvers provided at the outlet, and the like, and sends the information to the fluid analysis unit 21i. In addition, when the space in which the air cleaner 30 is installed is captured, the image analysis unit 21c identifies the position of the air cleaner 30 in the installation space and the positions of furniture and the like arranged in the installation space. and sent to the fluid analysis unit 21i.

Information about the state of the air cleaner 30 is input to the fluid analysis unit 21i from the state analysis unit 21d. The information input to the fluid analysis unit 21i is, for example, information indicating the number of revolutions of the fan motor of the air cleaner 30, information indicating the operating mode being executed, and the like. The fluid analysis unit 21i identifies the airflow (wind) blown out from the outlet of the air purifier 30 based on the information input from the image analysis unit 21c and the information input from the state analysis unit 21d. do. From the position of the air purifier 30 in the installation space and the positions of furniture and the like arranged in the installation space, the flow is specified in consideration of the layout of the installation space. An application that can measure a room and draw a floor plan just by taking a picture with a smartphone is already known, so no further explanation will be given. The analysis result of the fluid analysis unit 21i is sent to the additional information determination unit 21e.

The additional information determining unit 21e obtains information such as the direction of the air purifier 30 specified by the image analyzing unit 21c and the included parts, the request content specified by the request analyzing unit 21h, and the fluid analyzing unit 21i. Additional information to be notified to the user is determined based on the analysis results of the above.

For example, when the content of the request is "visualization of the airflow blown out from the air purifier 30" and the user is photographing the air purifier 30 from oblique information, the display unit 15 of the user terminal 10 displays the image shown in FIG. , an AR image to which an image of an airflow (wind) blown out from the outlet is added is displayed. In FIG. 17A, in addition to the airflow, an image of ions contained in the airflow is also displayed as an image with "- (minus)" and "+ (plus)" written in circles.

Further, when the content of the request is "visualization of the airflow blown out from the air purifier 30" and the user is photographing the installation space of the air purifier 30, the display unit 15 of the user terminal 10 displays the image shown in FIG. Alternatively, as shown in FIG. 17C, an AR image is displayed to which an image of an airflow (wind) blowing from a blowing port and circulating in the installation space is added. FIG. 17B is an AR image when no furniture is installed, and FIG. 17C is an AR image when furniture is installed. In FIGS. 17B and 17C, the airflow is indicated by an image of a white arrow, and the image of ions contained in the airflow is an image in which "- (minus)" and "+ (plus)" are written in circles. is displayed. A place where the air is stagnant can be identified from the image of the airflow and the image of the ions.

According to this, by simply requesting the server 20 to display the airflow blown out from the air purifier 30 and photographing the air purifier 30 within the screen of the display unit 15 with the user terminal 10, An image of the airflow can be visually recognized as additional information.

Since the image of the airflow is superimposed on the image taken by the user himself/herself, it is possible to easily identify the place where the air is stagnant by installing the furniture, etc., and take measures such as devising the layout of the furniture.

[Embodiment 5]
Another embodiment of the present invention will be described. For convenience of description, members having the same functions as those of the members described in the above embodiments are denoted by the same reference numerals, and descriptions thereof are omitted.

The configuration of the notification system 1 of the first to fourth embodiments described above includes the server 20 . On the other hand, in the notification system 1 of the present embodiment, as shown in FIG. A router, a WiFi (registered trademark) access point, or a Bluetooth (registered trademark) function may be used for connection. FIG. 18 is a block diagram showing the overall configuration of the notification system 1 of this embodiment.

In addition, in the notification system 1 of Embodiments 1 to 4 described above, the air cleaner 30 is exemplified as an electronic device. However, if electronic devices such as washing machines, cooking devices, robot cleaners, and air conditioners can be connected to the user terminal 10 and the server 20 via a network 40, a WiFi (registered trademark) access point, etc., what kind of devices can be used? electronic equipment.

[Embodiment 6]
The control unit 11 of the user terminal 10, the control unit 21 of the server 20, and the control unit 31 of the air cleaner 30 described in the first to fifth embodiments are formed on an integrated circuit (IC chip). It may be implemented in hardware using a logic circuit, or may be implemented in software using a CPU (Central Processing Unit).

In the latter case, the user terminal 10, the server 20, and the air purifier 30 each have a CPU that executes instructions of a program, which is software that implements each function, and the computer (or CPU) can read the program and various data. It includes a ROM (Read Only Memory) or a storage device (these are referred to as a “recording medium”) in which it is recorded, a RAM (Random Access Memory) for developing the above program, and the like. The object of the present invention is achieved by a computer (or CPU) reading and executing the program from the recording medium. As the recording medium, a "non-temporary tangible medium" such as a tape, disk, card, semiconductor memory, programmable logic circuit, or the like can be used. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. It should be noted that the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

〔summary〕
A notification system 1 according to aspect 1 of the present invention is a notification system comprising an electronic device (air cleaner 30) and a terminal device (user terminal 10) connected to the electronic device via a network 40, The terminal device includes a photographing unit 14, a display unit 15 for displaying an image photographed by the photographing unit 14, and a display control unit 11d for controlling display by the display unit 15. The notification system 1 includes the An additional information determining unit 21e for determining additional information to be combined with an image including the electronic device captured by the imaging unit 14, the additional information determining unit 21e determining the image of the electronic device. The display control unit 11d determines additional information that is combined with the location where the problem occurs in the electronic device in the part and indicates the location with the problem, and the display control unit 11d selects the electronic device as the image captured by the imaging unit 14. It is characterized in that a synthesized image obtained by synthesizing the additional information with the image including the device is displayed on the display unit 15 .

According to the above configuration, the additional information determination unit 21e determines the additional information that is combined with the problematic location in the electronic device and indicates the problematic location. The display control unit 11d displays a synthesized image obtained by synthesizing additional information as an image shot by the shooting unit on the display unit.

Thereby, the user can know that the electronic device has a problem simply by taking a picture of the electronic device using the terminal device, and at the same time, can specify the problem location. Problem areas are superimposed on the captured image and shown directly, so it is possible to grasp more intuitively than in the conventional configuration where LED lamps, etc. on the operation display of the electronic device are turned on to notify the user. There is no need to know the relationship with the content of the notification.

In addition, since the number of LED lamps in the operation display section can be reduced, the design of the electronic device can be improved.

In the notification system 1 according to the aspect 2 of the present invention, the additional information determination unit 21e further includes: the content of the problem and/or the method of coping with the problem, which is synthesized in association with the additional information indicating the location of the problem. is a configuration for determining additional information indicating

According to the above configuration, the additional information determination unit 21e determines the additional information indicating the content of the problem and/or the countermeasure method thereof, which is synthesized in association with the additional information indicating the problematic portion, and determines the additional information to be added. Information is also displayed on the terminal.

As a result, the user can easily know the content of the problem occurring in the electronic device and how to deal with it from the composite image of the photographed image including the electronic device displayed on the terminal device, so that the user can quickly deal with it. It becomes possible.

In the notification system 1 according to the aspect 3 of the present invention, the additional information determination unit 21e further synthesizes in association with the location on the electronic device to be accessed by the user, and provides an instruction to deal with the problem that has occurred, In this configuration, the additional information to be shown in stages is determined according to the progress of the processing.

According to the above configuration, the additional information determination unit 21e is synthesized in association with the location of the electronic device to be accessed by the user, and provides step-by-step instructions for coping with the problem that has occurred according to the progress of the process. is determined, and the additional information is displayed on the terminal device.

As a result, problems can be easily dealt with by proceeding with processing according to the composite image of the captured image including the electronic device displayed on the terminal device, without having to check the instruction manual that came with the electronic device. can do.

Moreover, since the additional information indicating the content of the instruction is superimposed on the photographed image taken by the user himself/herself, it is much easier to understand than the explanatory diagrams and photographs in the instruction manual whose direction is fixed. It improves workability.

In the notification system 1 according to aspect 4 of the present invention, when the additional information determination unit 21e is requested by the terminal device for additional information about the operation method of the electronic device, the user accesses to execute the operation. The configuration determines additional information that is synthesized in association with a portion to be performed and indicates an instruction for executing the operation.

According to the above configuration, when additional information about the operation method of the electronic device is requested from the terminal device, the additional information determination unit synthesizes the additional information in association with the location to be accessed by the user in order to execute the operation. Additional information indicating an instruction to perform the operation is determined, and the additional information is displayed on the terminal device.

As a result, even if you do not know how to operate the electronic device, you can request the operation method as additional information by taking a picture of the electronic device using the terminal device without checking the instruction manual included with the electronic device. By doing so, it is possible to know the operation method from the composite image of the photographed images including the electronic device displayed on the terminal device.

In the notification system 1 according to aspect 5 of the present invention, the electronic device is an air purifier 30, and the additional information determining unit 21e is requested by the terminal device for additional information indicating the airflow blown out from the air purifier. Then, additional information indicating the airflow is determined based on the image of the air purifier photographed by the photographing unit and the information on the air volume acquired from the air purifier.

According to this, by photographing the air purifier using the terminal device and requesting the image of the airflow as the additional information, the image of the airflow can be visually recognized as the additional information.

A server 20 according to aspect 6 of the present invention is a server 20 interposed between an electronic device and a terminal device connected to the electronic device via a network, wherein a receiving unit 21a for receiving a photographed image including the electronic device; an acquiring unit (information acquiring unit 21b) for acquiring information on a problem occurring in the electronic device from the electronic device; an additional information determination unit 21e for determining additional information to be combined with the image based on the image including the electronic device captured by the electronic device; and transmitting the additional information determined by the additional information determination unit 21e to the terminal device The additional information determination unit 21e determines additional information indicating the problematic portion of the image of the electronic device by combining it with the problematic portion of the electronic device. do.

A terminal device (user terminal 10) according to aspect 7 of the present invention is an electronic device (air purifier 30) and a terminal device connected to the electronic device via network 40. A display unit 15 for displaying a photographed image on the unit 14, and a display control unit 11d for controlling display by the display unit. Additional information to be combined with the image of the electronic device captured by the image capturing unit from the transmission unit 11b that transmits an image including the electronic device to be captured, and the image part of the electronic device from the server 20 and a reception unit 11c for receiving additional information indicating the problematic portion synthesized with the problematic portion of the electronic device in the display control unit 11d. The display unit 15 displays a synthesized image obtained by synthesizing the additional information as an image obtained by combining the additional information.

By using the server 20 according to aspect 6 of the present invention and the terminal device according to aspect 7 of the present invention, the same effects as those of the notification system according to aspect 1 of the present invention can be obtained.

A terminal device (user terminal 10) according to aspect 8 of the present invention is a terminal device that is connected to an electronic device (air purifier 30) via a network, and includes a photographing unit 14 and an image captured by the photographing unit 14. a display control unit 11d for controlling display by the display unit 15; and an additional information determination unit 21e for determining additional information, the additional information determination unit 21e compositing a portion of the image of the electronic device where a problem has occurred in the electronic device to determine the problematic portion. The display control section 11d determines the additional information to be displayed, and displays a composite image obtained by combining the additional information as the image photographed by the photographing section 14 on the display section 15. FIG.

By using the terminal device according to aspect 8 of the present invention, the same effects as those of the notification system according to aspect 1 of the present invention can be obtained.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

1 notification system 10 user terminal (terminal device)
11, 21, 31 control unit 11a imaging control units 11b, 21g, 31c transmission units 11c, 21a reception unit 11d display control units 12, 22, 32 storage units 13, 23, 33 communication unit 14 imaging unit 15 display unit 16 voice input Output unit 17 Operation unit 20 Server 21b Status information acquisition unit (acquisition unit)
21c Image analysis unit 21d State analysis unit 21e Additional information determination unit 21f AR image generation unit 21h Request analysis unit 21i Fluid analysis unit 30 Air purifier (electronic device)
40 network

Claims (5)

A notification system comprising an electronic device, a terminal device connected to the electronic device via a network, and a server interposed between the electronic device and the terminal device,
The terminal device includes a photographing unit, a display unit for displaying an image photographed by the photographing unit, and a display control unit for controlling display by the display unit. sending to said server an image containing
Based on the image including the electronic device photographed by the photographing unit, the server provides additional information to be combined with the image, and the electronic device in the image portion of the electronic device has a problem. an additional information determination unit for determining additional information indicating that a problem has occurred in the combined portion in the image portion of the electronic device; receiving an image including the photographed electronic device, transmitting additional information determined by the additional information determining unit based on the image to the terminal device;
When the display control unit receives the additional information determined based on the image transmitted to the server, the display control unit adds the information to the image portion of the image including the electronic device where the problem occurs in the electronic device. displaying a synthesized image obtained by synthesizing information on the display unit as an image captured by the imaging unit ;
The terminal device has been registered as a terminal device of the user of the electronic device, and the electronic device transmits state information indicating its own state including operating conditions to the server, and the server receives the state information from the electronic device. and the server transmits a message prompting the terminal device to take a picture of the electronic device . The additional information determining unit further determines additional information indicating the content of the problem, the method of coping with the problem, or both, which is synthesized in association with the additional information indicating the location of the problem. Item 1. The notification system according to item 1. The additional information determination unit further includes additional information that is synthesized in association with a location in the electronic device that the user should access, and that indicates, step by step, an instruction to deal with the problem that has occurred according to the progress of processing. 3. A notification system according to claim 1 or 2, characterized in that it determines . When the terminal device requests additional information about the operation method of the electronic device, the additional information determining unit synthesizes the additional information in association with a location on the electronic device to be accessed by the user in order to execute the operation, 4. A notification system according to any one of claims 1 to 3, characterized in that it determines additional information indicating instructions for performing said operation. A notification system comprising: an electronic device that blows airflow; a terminal device connected to the electronic device via a network; and a server interposed between the electronic device and the terminal device,
The terminal device includes a photographing unit, a display unit for displaying an image photographed by the photographing unit, and a display control unit for controlling display by the display unit. sending to said server an image containing
Based on the image including the electronic device photographed by the photographing unit, the server provides additional information to be combined with the image, and the electronic device in the image portion of the electronic device has a problem. an additional information determination unit for determining additional information indicating that a problem has occurred in the combined portion in the image portion of the electronic device; receiving an image including the photographed electronic device, transmitting additional information determined by the additional information determining unit based on the image to the terminal device;
When the display control unit receives the additional information determined based on the image transmitted to the server, the display control unit adds the information to the image portion of the image including the electronic device where the problem occurs in the electronic device. displaying a synthesized image obtained by synthesizing information on the display unit as an image captured by the imaging unit;
When the terminal device requests additional information indicating the airflow emitted from the electronic device , the additional information determination unit acquires an image of the electronic device and the installation space of the electronic device, and an image of the installation space of the electronic device from the electronic device. and determining additional information indicating the airflow in the installation space .

Images