JP7454523B2 - Mirror type display device and information provision system - Google Patents

Description

The present disclosure relates to a mirror-type display device that has both a mirror function and a display function, and information provision technology using the same.

In recent years, mirror-type display devices called smart mirrors have been developed, which have both the function of a mirror that reflects an optical image of objects such as the user's appearance, and the function of a display that displays images of electronic information such as weather forecast information and news reports. is widespread. Usually, a smart mirror functions only as a mirror when the display function is not activated (when no image is displayed), and when the display function is activated (when the image is displayed), it can function as both a mirror and a display. Such a smart mirror is disclosed in, for example, Patent Document 1 (International Publication No. 2018/004615).

The smart mirror disclosed in Patent Document 1 includes a display, a controller that displays an image on the display, a half mirror placed in front of the display, and a device that detects biometric information of the user using the smart mirror. The device includes a plurality of sensors for obtaining information, and a user interface (UI) section. In this smart mirror, the half mirror functions by reflecting part of the light incident from the object in front to project an image of that object, and transmitting the light from the image incident from the display behind it. Further, the controller can determine the focal length of the user from the biological information detected by the sensor.

International Publication No. 2018/004615

Mirror-type display devices such as those described above are used in various fields such as the hairdressing field, but a mirror-type display device that presents image content related to insurance products has not been provided. Insurance companies offer a wide variety of insurance products, and it is not always easy for a typical customer to select an insurance product that is suitable for them from among such a wide variety of insurance products.

In view of the above, an object of the present disclosure is to provide a mirror type display device and an information providing system using the mirror type display device that can present information regarding insurance products suitable for each user.

A mirror-type display device according to a first aspect of the present disclosure includes an image output section that outputs light of a displayed image, transmits the light output from the image output section, and reflects light incident from an object in an external space. a mirror-type display having a mirror portion that forms a mirror image of the object; and at least one sensor that detects sensor information including image information of a user located at a position facing the mirror portion in the external space; The server device includes a communication interface that transmits and receives data to and from an external server device, and a processor configured to transmit the sensor information to the server device via the communication interface, and the server device estimating an attribute of the user based on the information and transmitting user attribute information indicating the estimated attribute to the communication interface, the processor generating an insurance simulation that matches the user attribute information received at the communication interface; The information is configured to be displayed on the mirror type display.

An information providing system according to a second aspect of the present disclosure is characterized by comprising the mirror type display device according to the first aspect and a server device.

According to the first and second aspects of the present disclosure, a user of a mirror type display device matches the user's attributes (for example, age and gender) while viewing his or her own appearance and face reflected on the mirror type display. insurance simulation information can be obtained extremely easily and in a short time. By using such a mirror type display device, a user can significantly reduce the effort required to search for an insurance product suitable for his or her attributes from among a wide variety of insurance products.

1 is a block diagram showing a schematic configuration of an information providing system according to an embodiment of the present disclosure. 1 is a functional block diagram schematically showing a configuration example of a mirror type display device. FIG. FIG. 1 is a schematic cross-sectional view of a mirror-type display. 1 is an external view showing a configuration example of a mirror type display device having a mirror type display. FIG. 2 is a block diagram schematically showing a configuration example of a server device that operates in cooperation with a mirror display device. 2 is a flowchart schematically showing an example of a processing procedure by a processor of a mirror type display device. 2 is a flowchart schematically showing an example of a processing procedure by a processor of a mirror type display device. It is a figure which illustrates a menu image. It is a figure which illustrates a message image. It is a figure which illustrates the image for confirmation and a change. It is a figure which illustrates the image for insurance plan selection. FIG. 3 is a diagram illustrating an example of an image displaying insurance simulation information. It is a figure which illustrates the image for a change. It is a figure which illustrates the image for a change. It is a figure which illustrates the image for a change. FIG. 2 is a block diagram schematically showing an example of a hardware configuration that implements the functions of a processor and a user interface.

Next, various embodiments and modifications thereof according to the present disclosure will be described in detail with reference to the drawings. Note that components designated by the same reference numerals throughout the drawings have the same configuration and function.

FIG. 1 is a block diagram showing a schematic configuration of an information providing system 1 according to an embodiment of the present disclosure. This information providing system 1 includes a server device 10 and mirror type display devices 20 ₁ , 20 ₂ , . . . , 20 _N that are interconnectable with the server device 10 via a communication network 2. Here, N is an integer of 3 or more indicating the number of mirror type display devices 20 ₁ to 20 _N.

Examples of the communication network 2 include, but are not limited to, a wide area network such as the Internet, or a local area network (LAN). Further, in the example of FIG. 1, the number of mirror type display devices 20 ₁ to 20 _N is three or more, but the number is not limited to this. An information providing system including one or two mirror display devices and the server device 10 may also be used.

FIG. 2 is a functional block diagram schematically showing a configuration example of the mirror type display device _20n (n is an integer within the range of 1 to N). The mirror type display device _20n shown in FIG. 2 includes a mirror type display 30, a processor 40, a user interface (UI) 44, an input/output interface (input/output I/F) 42, a communication interface (communication I/F) 43, It also includes a sensor group consisting of an optical sensor 51, an image sensor 52, and an infrared sensor 53. The user interface 44 includes a gesture detection section 44A, a touch operation detection section 44B, and a voice recognition section 44C.

The input/output interface 42 is arranged between the speaker 55, the microphone 56, and the sensor group and the user interface 44, and between the speaker 55, the microphone 56, the sensor group, and the user interface 44. The input/output interface 42 has a function of transmitting the respective outputs of the microphone 56, the optical sensor 51, the image sensor 52, and the infrared sensor 53 to the processor 40 and the user interface 44. It has a function of transmitting a signal to the speaker 55.

The optical sensor 51 includes a depth sensor 51A that generates a depth image related to the distance to the target object, that is, depth, and outputs the depth image information DD, and a depth sensor 51A that generates a two-dimensional image such as an RGB image of the target object and outputs the depth image information DD. It has an image sensor 51B that outputs image information IDa. For example, the depth sensor 51A can be realized by a known stereo camera or a ToF (Time-of-Flight) sensor. The depth image information DD and the image information IDa can be used to estimate the movement (skeleton tracking) of the skeleton of the object (the posture and skeleton of a person). The gesture detection unit 44A can detect a gesture of a specific part of the target person (for example, the head, hand, eyelid, or line of sight) using the depth image information DD and the image information IDa. Furthermore, the processor 40 can estimate the posture of the target person in real time using the depth image information DD and the image information IDa.

The image sensor 52 is an optical sensor that generates a two-dimensional image of the object and outputs the image information IDb. As will be described later, the processor 40 can use the image information IDb to estimate the attributes of the user of the mirror display device _20n (for example, the age and gender of the person). Furthermore, the processor 40 can also use the image information IDb to detect the blood pressure, breathing rate, and heart rate of the target person.

The infrared sensor 53 is an optical sensor that detects light in an infrared wavelength range emitted from an object and outputs infrared detection information TD. The processor 40 can estimate the temperature distribution of the object using the infrared detection information TD.

The microphone 56 detects an acoustic signal arriving from a sound source and generates an electrical signal DS. The voice recognition unit 44C can perform voice recognition processing on the electrical signal DS to generate voice input information such as text information. The speaker 55 converts the electrical signal supplied from the processor 40 into an acoustic signal, and radiates the acoustic signal to an external space.

Next, the mirror type display 30 includes an image output section that outputs light representing the digital display image IS generated by the processor 40, and a mirror section disposed in front of the image output section (light exit surface). has been done. The image output section can be realized by, for example, a liquid crystal display panel (LCD panel) or an organic EL panel, and the mirror section can be realized by, for example, a half mirror. FIG. 3 is a schematic cross-sectional view of the mirror type display 30. As shown in FIG. 3, the mirror-type display 30 includes an image output section 30D that outputs light DL representing a digital display image IS, and transmits light DL incident from the back side, while transmitting light DL incident from an object in external space. and a mirror section 30M that reflects the reflected light IL to form a mirror image of the object.

Further, the mirror display 30 has a touch panel function that senses the position touched by the user on the surface of the mirror section 30M. When the user touches the surface of the mirror section 30M, the touch operation detection section 44B of FIG. 2 can detect the touch position as an operation input by the user.

FIG. 4 is an external view showing a configuration example of a mirror type display device 20 _n having a mirror type display 30. As shown in FIG. FIG. 4 is a diagram of the mirror type display device _20n viewed from the front. The mirror type display device 20 _n shown in FIG. 4 includes a mirror section 30M arranged to face the front of the mirror type display device 20 _n , and an upper bezel of the casing 30C of the mirror type display device 20 _n . The sensor module 50 is provided with an image sensor 52 and a sensor module 50 attached to the upper end of the housing 30C. Sensor groups other than the image sensor 52 can be arranged within the sensor module 50. In the example of FIG. 4, the mirror section 30M projects a mirror image 70I of the user located at a position facing the mirror section 30M, and at the same time displays image contents 70Da and 70Db representing the digital display image IS toward the user. There is. Therefore, the user can visually recognize the state of his or her appearance, and at the same time can visually recognize the image contents 70Da and 70Db.

Next, the configuration of the processor 40 shown in FIG. 2 will be described in detail.

The processor 40 uses the sensor information acquired by the sensor group to execute an insurance simulation process for acquiring insurance simulation information that matches the attributes of the user of the mirror display device _20n and displaying it on the mirror display 30. Executes non-insurance content processing for displaying image content not directly related to insurance (for example, advertising images, or images used for games or fortune-telling) on the mirror display 30 using the function to It has the function of

The processor 40 cooperates with the server device 10 to execute insurance simulation processing. Specifically, the processor 40 acquires sensor information including image information representing the biological characteristics of the user present at a position facing the mirror display 30, and transmits the sensor information to the server device 10 via the communication interface 43. Send to. Here, as the sensor information, for example, image information IDb (particularly face image information) acquired by the image sensor 52 can be used, but it is not limited thereto. Depth image information DD and image information IDa can also be used. The server device 10 estimates the user's attributes (for example, age and gender) based on the sensor information transmitted from the mirror display device _20n , and transmits user attribute information indicating the estimated attributes to the mirror display device. _20n . Furthermore, the server device 10 can also transmit insurance simulation information that matches the user attribute information to the mirror display device _20n . The mirror type display device _20n causes the mirror type display 30 to display the user attribute information and insurance simulation information received through the communication interface 43. As a result, the user can obtain information related to insurance products that match his/her attributes very easily and in a short time.

FIG. 5 is a block diagram schematically showing a configuration example of the server device 10 that operates in cooperation with the mirror type display device _20n . As shown in FIG. 5, the server device 10 includes a processor 11 including an attribute estimation section 12 and an insurance simulation section 13, a database 14 used for insurance simulation, and a communication system for transmitting and receiving data to and from a mirror display device _20n . An interface (communication I/F) 15 is provided.

The attribute estimation unit 12 estimates the user's attributes (eg, age and gender) using the trained model 12M based on the sensor information transmitted from the mirror display device _20n . The trained model 12M is a mathematical model that defines relationships between various types of human image information and human attributes. As the learned model 12M, a neural network model or a statistical model generated by machine learning may be used. The attribute estimation unit 12 transmits user attribute information indicating the estimated attribute to the mirror display device _20n via the communication interface 15.

On the other hand, the insurance simulation unit 13 can generate insurance simulation information that matches the user attribute information generated by the attribute estimation unit 12 by referring to the database 14 that stores information related to a wide variety of insurance plans. can. Furthermore, the insurance simulation unit 13 can refer to the database 14 to generate insurance simulation information that matches the user attribute information transmitted from the mirror display device 20 _n and received by the communication interface 15 . The insurance simulation unit 13 transmits such insurance simulation information to the mirror type display device _20n via the communication interface 15.

Next, an example of the operation of the mirror type display device _20n will be described below with reference to FIGS. 6 and 7. 6 and 7 are flowcharts schematically showing an example of a processing procedure by the processor 40 of the mirror display device _20n .

When the user is located in the proximity area facing the mirror display 30, the processor 40 can sense the presence of the user based on the sensor information and initiate the process of FIG. 6. Upon sensing the presence of the user, the processor 40 causes the mirror display 30 to display a menu image prompting the user to select either insurance simulation or non-insurance content processing (step S11). The processor 40 waits until there is an input by the user (NO in step S12). The user can select either insurance simulation or non-insurance content processing by gesturing his or her own body part, touching the mirror display 30, or inputting voice.

When there is an input by the user (YES in step S12) and if insurance simulation is selected (YES in step S13), the processor 40 executes the insurance simulation process (step S14). On the other hand, if insurance simulation is not selected (NO in step S13) and non-insurance content processing is selected (YES in step S15), processor 40 executes non-insurance content processing (step S16). If neither the insurance simulation nor the non-insurance content processing is selected (NO in step S15), the process ends, and after the insurance simulation process (step S14) or the non-insurance content process (step S16) is executed, the process continues. finish.

FIG. 8 is a diagram illustrating the menu image D1. This menu image D1 includes item B11 for selecting "insurance simulation", item B12 for selecting "entertainment" such as quizzes and fortune telling, item B13 for selecting "temperature measurement", and item B13 for selecting "gymnastics". Item B14 for selecting "Challenge". Item B99 is an item for calling a store clerk when there is a store clerk at the store where the mirror type display device 20 _n is installed. The non-insurance content processing in the example of FIG. 8 is "entertainment,""temperaturemeasurement," and "gymnastics challenge," but is not limited to these. The processor 40 detects a specific gesture (for example, a hand gesture) detected by the gesture detection section 44A or a touch position detected by the touch operation detection section 44B as an operation input, and performs a desired operation according to this operation input. Items can be selected. Alternatively, the processor 40 may select a desired item according to the voice input of a specific speech pattern recognized by the voice recognition unit 44C. When "temperature measurement" is selected, the processor 40 measures the user's body temperature using the infrared detection information TD, and causes the mirror display 30 to display an image showing the measured body temperature value. When "gymnastics challenge" is selected, a game image that allows the user to move his or her body is displayed on the mirror display 30 using a skeleton tracking algorithm.

FIG. 7 is a flowchart schematically showing an example of insurance simulation processing (step S14). Referring to FIG. 7, in response to the user's selection of insurance simulation, the processor 40 causes the mirror display 30 to display a message image indicating that the user's attributes are being estimated (step S21). FIG. 9 is a diagram illustrating such a message image D2.

Next, the processor 40 acquires sensor information including image information representing the user's biological characteristics (step S22), and transmits the sensor information to the server device 10 via the communication interface 43 (step S23). Thereafter, the processor 40 waits until it receives or acquires the user attribute information from the server device 10 (NO in step S24).

When the user attribute information is acquired from the server device 10 (YES in step S24), the processor 40 causes the mirror display 30 to display a confirmation/change image for confirming or changing the user based on the acquired user attribute information ( Step S25). FIG. 10 is a diagram illustrating the confirmation/change image D3. The confirmation/change image D3 displays that the estimated user attribute information is "30 years old" and "male." In addition, the confirmation/change image D3 includes a "confirmation" item B33 for confirming user attribute information without changing it, an "age change" item B31 and a "gender change" item B31 for changing user attribute information. Item B32 is included. If the user does not wish to change the user attribute information, he or she can select item B33, and if he or she wishes to change the user attribute information, he or she can select items B31 and B32.

The processor 40 waits until the user issues an instruction to confirm or change the user attribute information (NO in step S26 and NO in step S31). If confirmation by the user is obtained (YES in step S26), the processor 40 displays the insurance plan selection image on the mirror display 30 (step S27). Thereafter, the processor 40 waits until the user selects an insurance plan (NO in step S28). FIG. 11 is a diagram illustrating the insurance plan selection image D4. This image D4 includes an item B41 for selecting "Insurance Plan X", an item B42 for selecting "Insurance Plan Y", an item B43 for selecting "Insurance Plan Z", and an "Insurance Plan item B44 for selecting "H".

Thereafter, when the user selects an insurance plan (YES in step S28), the processor 40 requests the server device 10 for insurance simulation information that matches the insurance plan and user attribute information, and acquires the insurance simulation information ( Step S29). Then, the processor 40 causes the insurance simulation information to be displayed on the mirror display 30 (step S 30 ). FIG. 12 is a diagram illustrating an image D5 displaying insurance simulation information. This image D5 includes an area B50 for displaying the contents of the insurance plan, an item B51 for returning to the previous image D4 to reselect an insurance plan, an item B52 for calling a store clerk, and an area B50 for displaying the insurance plan contents. It includes an item B53 for scrolling downward and an item B54 for scrolling the displayed contents of the insurance plan upward.

On the other hand, when there is an instruction from the user to change the user attribute information (YES in step S31), the processor 40 causes the mirror display 30 to display an image for changing the user attribute information (step S32). FIGS. 13 to 15 are diagrams illustrating such change images D6, D7, and D8. The image D6 for changing the age shown in FIG. 13 includes an item B61 for selecting an age from 0 to 20 years old, an item B62 for selecting an age from 21 to 40 years old, and an item B62 for selecting an age from 21 to 40 years old. This includes an item B63 for selecting an age group, and an item B64 for selecting an age group from 61 to 85 years old. When the user selects any age item in FIG. 13, the age change image D7 shown in FIG. 14 may be displayed on the mirror display 30. The change image D7 includes an item B71 for lowering the age in stages in response to a selection by the user, an item B72 for increasing the age in stages in response to a selection by the user, and an item B72 for increasing the age in stages in response to a selection by the user, and an item B72 for increasing the age in stages in response to a user's selection, and It includes an item B73 for returning to image D6 and a confirmation item B74 for determining the age. On the other hand, the image D8 for changing gender shown in FIG. 15 includes an item B81 for selecting "male" and an item B82 for selecting "female".

The processor 40 waits until the user inputs user attribute information in response to the display of the change image (NO in step S33). When the user inputs user attribute information (YES in step S33), the processing procedure moves to step S25. In step S25, the processor 40 causes the mirror display 30 to display a confirmation/change image for confirming or changing the user attribute information after the change. After that, step S26 is executed again.

As explained above, the user of the mirror type display device 20 _n can view his or her own appearance and face reflected on the mirror type display 30 and select insurance simulation information that matches the user's attributes (for example, age and gender). It can be easily obtained in a short time. Therefore, by using the mirror type display device _20n , the user can significantly reduce the effort required to search for an insurance product suitable for his or her attributes from among a wide variety of insurance products.

All or part of the functions of the processor 40 and the user interface 44 in the mirror type display device _20n described above are performed by a CPU (Central Processing Unit) or GPU that executes software or firmware program codes read from a non-volatile memory. It may be realized by a single or plural processors including an arithmetic unit such as a Graphics Processing Unit (Graphics Processing Unit). Alternatively, all or part of the functions of the processor 40 and the user interface 44 may be implemented using a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable GaN). Single or multiple semiconductor integrated circuits such as TE Array) This can be realized using a processor of Alternatively, all or part of the functions of the processor 40 and the user interface 44 may be realized by one or more processors including a combination of a semiconductor integrated circuit such as a DSP, ASIC, or FPGA and an arithmetic unit such as a CPU or GPU. It is possible.

FIG. 16 is a block diagram schematically showing the configuration of a signal processing device 90, which is an example of a hardware configuration that implements the functions of the processor 40 and the user interface 44. The signal processing device 90 includes a processor 91 that executes digital signal processing, a random access memory (RAM) 92, a nonvolatile memory 93, a large capacity memory 94, an input/output interface circuit 95, a communication interface circuit 96, and a signal path. 97. The signal path 97 is a bus for interconnecting the processor 91, RAM 92, nonvolatile memory 93, large capacity memory 94, input/output interface circuit 95, and communication interface circuit 96. The RAM 92 is a data storage area used when the processor 91 executes digital signal processing. When the processor 91 includes a built-in arithmetic device such as a CPU, the nonvolatile memory 93 has a data storage area that stores program codes of software or firmware executed by the processor 91. The input/output interface circuit 95 in FIG. 16 corresponds to the input/output interface 42 in FIG. 2, and the communication interface circuit 96 in FIG. 16 corresponds to the communication interface 43 in FIG.

Various embodiments and modifications thereof according to the present disclosure have been described above, but the above embodiments and modifications thereof are merely illustrative and do not limit the scope of the present invention. For example, the mirror type display device 20 _n may be configured as a large display device installed in a store or attached to a wall surface, or may be configured as a portable display device such as a tablet terminal. .

Furthermore, it should be understood that changes, additions, and improvements to the above embodiments can be made as appropriate without departing from the spirit and scope of the present invention. The scope of the present invention should be interpreted based on the claims, and should be understood to include equivalents thereof.

The mirror type display device and information providing system according to the present disclosure can provide insurance simulation information suitable for the attributes of individual users, and therefore are suitable for use in insurance-related business.

1: Information provision system, 2: Communication network, 10: Server device, 11: Processor, 12: Attribute estimation section, 12M: Learned model, 13: Insurance simulation section, 14: Database, 15: Communication interface (communication I/ F), 20 ₁ , 20 ₂ ,..., 20 _N : mirror type display device, 30: mirror type display, 30D: image output unit, 30M: mirror unit, 30C: housing, 40: processor, 42: input/output interface (input/output I/F), 43: communication interface (communication I/F), 44: user interface (UI), 44A: gesture detection unit, 44b: touch operation detection unit, 44c: voice recognition unit, 50: sensor module , 51: Optical sensor, 51A: Depth sensor, 51B: Image sensor, 52: Image sensor, 53: Infrared sensor, 55: Speaker, 56: Microphone, 90: Signal processing device, 91: Processor, 92: RAM , 93: nonvolatile memory, 94: large capacity recording medium (large capacity memory), 95: input/output interface circuit, 96: communication interface circuit, 97: signal path.

Claims (7)

An information providing system comprising a server device and a mirror type display device capable of mutual communication with the server device via a communication network,
The mirror type display device includes:
an image output unit that outputs the light of the display image from the back side to the front side of the mirror type display device; and an image output unit that transmits the light output from the image output unit to display the display image, and simultaneously a mirror-type display having a mirror portion that reflects light incident from a user located on the front side to project a mirror image of the user;
at least one sensor that detects sensor information including image information of the user;
a communication interface that sends and receives data to and from the server device;
a processor configured to transmit the sensor information and the insurance plan selected by the user to the server device via the communication interface;
The server device includes:
Estimating the user's attributes using a trained model that defines a relationship between the person's image information and the person's attributes based on the image information included in the transmitted sensor information,
Generate insurance simulation information that matches the insurance plan selected by the user and user attribute information indicating the estimated attributes by referring to a database storing information related to a wide variety of insurance plans;
configured to send the user attribute information and the insurance simulation information to the communication interface;
The processor is configured to display at least the insurance simulation information, out of the user attribute information and the insurance simulation information received by the communication interface, on the mirror display as the display image.
An information provision system characterized by: 2. The information providing system according to claim 1, wherein the user attribute information includes at least the estimated age and gender of the user. 3. The information providing system according to claim 1, wherein the image information includes at least one of a two-dimensional image and a depth image detected by the at least one sensor. system. The information providing system according to any one of claims 1 to 3,
The processor includes:
A menu image is displayed on the mirror display that prompts the user to select either insurance simulation for displaying the insurance simulation information or non-insurance content processing for displaying image content not directly related to insurance. display,
When there is an input to select the insurance simulation, transmitting the sensor information to the server device and displaying the insurance simulation information on the mirror display;
An information providing system characterized in that, when there is an input to select the non-insurance content processing, the non-insurance content is displayed on the mirror type display. The information providing system according to claim 4,
Further comprising a user interface that accepts input from the user,
The mirror type display further has a touch panel function that senses a touch position by a user on the surface of the mirror part,
The information providing system is characterized in that the user interface includes a touch operation detection unit that receives the sensed touch position as the input. The information providing system according to claim 4,
Further comprising a user interface that accepts input from the user,
The information providing system is characterized in that the user interface includes a gesture detection unit that detects a gesture of a specific part of the user based on a depth image detected by the at least one sensor. The information providing system according to claim 4,
a user interface that accepts input from the user;
further comprising a microphone that detects an acoustic signal arriving from the external space and outputs an electrical signal,
The information providing system is characterized in that the user interface includes a voice recognition unit that performs voice recognition processing on the electrical signal to generate voice input information.

Images