JP2009294740A - Data processor and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate display data in an easy-to-see display format for a user even if the user is busy and cannot perform adjustment work or the like without imposing a burden such as a change in content and program and the adjustment of a character size on the user. <P>SOLUTION: A camera 300 photographs the face of a user who watches display data displayed on a display device 200. A display changing device 100 analyzes image data in the eye region of the face image of the user, determines whether the user is in a normal eye state (a face image A400) or in a state of narrowed eyes (a face image B500). In the latter, it is determined that the user has difficulty in watching the display data, and the device changes the display layout of the display data to a more easy-to-see format, and outputs display data (corresponding layout display B700) along the more easy-to-see display layout on the display device 200. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a context-aware technology that changes the behavior of an application program (hereinafter simply referred to as an application) according to the state of a target user, for example, and changes the character size and layout of a screen to make original data as much as possible. The present invention relates to a user interface technology for making display easy to see without modification.

A method of changing the operation or service of an application according to the surrounding situation is called context-aware.
For example, by using a position sensor such as GPS (Global Positioning System), a specific service that depends on the location, such as explaining a voice of an object placed at that location when a user is located at a specific location. Can be considered.
In addition, by using a camera, a weight sensor, etc., the congestion level of the place is monitored, and a service is provided according to the situation. For example, according to the degree of congestion inside the elevator, there is a service that provides voice guidance indicating that the elevator is busy or over capacity.
In addition, there is a method for providing an optimum data format for each system to be used by converting the content data format according to the set rule.

Patent Document 1 describes a technique for recognizing a person's emotion obtained from a camera, introducing a product that the person is likely to purchase from the emotion result, and a technique for detecting a dangerous act at an early stage.
Patent Document 2 describes a tag conversion method from XML (Extensible Markup Language) to HTML (Hyper Text Markup Language) according to a prepared content conversion rule.
Patent Document 3 describes a technique for inferring emotions and actions from a human facial expression obtained from a camera.
Japanese Patent Laid-Open No. 2004-318587 JP 2005-199403 A JP 2004-086855 A

In the method of changing the operation of the information terminal and the application according to the change of the surrounding situation or expression as described above, the definition of the target situation or expression and the corresponding application action are associated with the application in advance. In order to change the appearance of the display, there is a problem that it is necessary to change the content content to be displayed or to modify the application.
In addition, when it is difficult to see the display on the information terminal, it is necessary to manually adjust the character size of the screen or to change the program so that it is easy to see.
However, it is difficult to adjust the screen in an environment where the hand is blocked, and it is desirable to provide an optimal display for the user without adjusting the character size or changing the program.

  One of the main objects of the present invention is to solve the above-mentioned problems, and it is possible to determine the state of the user without forcing the user to change contents and programs, adjust the character size, etc. The main object is to realize a device that generates display data in a display format that is easy for the user to see.

The data processing apparatus according to the present invention
A data processing device connected to a display device for displaying display data and a photographing device for photographing a user's face facing the display device,
A face image input unit for inputting the face image data of the user, which includes image data of the eye area of the user, which is imaged by the imaging device;
A face image analysis unit that analyzes image data of the user's eye area of the face image data;
A display format determination unit that determines a display format of display data to be displayed on the display device based on an analysis result of the face image analysis unit;
And a display data generation unit that generates display data in accordance with the display format determined by the display format determination unit and outputs the generated display data to the display device.

  According to the present invention, in order to analyze the image data of the user's eye area captured by the imaging device and determine the display format of the display data based on the state of the user's eye area, Display data can be generated in a display format that is easy for the user to view without imposing the burden of adjusting the character size on the user, and even when the user's hand is blocked and adjustment work cannot be performed.

Embodiment 1 FIG.
When a user browses a sentence or an image on a display terminal, the user may try to read the contents of the character or the image by narrowing his eyes when it is difficult to see a display such as a fine character or a small size image.
At this time, in order to make the display easy to see, the user needs to manually change the character size or the like of the display terminal.
The present embodiment solves the above-described problem, and is a technique for photographing facial expressions of a user's face from a camera attached to a display terminal and changing a character size or the like by changing the eyes.
That is, in the present embodiment, the user determines whether the display is “easy to see” or “difficult to see” according to the amount of change in the feature points of the user's eyes and eyebrows acquired from the attached camera image. A device that provides an effective display effect according to the situation in an environment where the user's hand is closed for work without changing the original screen data by changing the character size and layout to suit the situation Will be described.

FIG. 1 is a diagram showing a system configuration according to the present embodiment.
In FIG. 1, the display change device 100 is connected to the display device 200 and the camera 300, determines a display format (character size and layout) suitable for the face image from the face image acquired from the camera 300, and determines the determined display format. Is a means for outputting display data in accordance with the display device 200 to the display device 200.
The display change device 100 is an example of a data processing device.

  The display device 200 is a means for displaying the display data generated by the display change device 100. The display device 200 is an example of a display device.

The camera 300 is a means for acquiring a face image.
The camera 300 captures the face of the user facing the display device 200.
The camera 300 may continue to capture the user's face or may capture the user's face at regular intervals (for example, every 10 seconds).
The camera 300 is an example of a photographing apparatus.

The face image A400 is a face image representing a normal face of the user, and is an example of an image taken by the camera 300.
The face image B500 is a face image when the user narrows his eyes in contrast, and is an example of an image photographed by the camera 300.
Usually, the action of narrowing the eyes is an action of making the display clearer because it is difficult to see the target screen display.

The display change device 100 analyzes the face image acquired from the camera 300 and changes the display format of the display data. For example, the display change device 100 can change the character size.
For example, if the face image acquired by the display change device 100 from the camera 300 is the face image A400, the display change device 100 determines that the user is looking at the screen with a normal face, and displays the screen with a normal character size. Perform (corresponding layout display A600).
On the other hand, when the display change device 100 acquires the face image B500 from the camera 300, the display change device 100 determines that the user is looking at the screen with an expression that is difficult to see, and the character size or character larger than usual is displayed. The display is performed by applying a style such as underlining (corresponding layout display B700).
The above is the basic operation of the display change device 100 according to the present embodiment.

In the first to sixth embodiments, when the number of users is one, that is, according to the amount of change in eyes and eyebrows of one face, the character size, layout, and screen size of the display screen without changing the content content How to change the will be described.
In Embodiments 7 and 8, when there are a plurality of users, that is, according to the amount of change in the eyes and eyebrows of two or more faces, the character size and layout of the display screen without changing the content contents A method for changing the screen size will be described.

  FIG. 2 is a block diagram illustrating an internal configuration example of the display change device 100 according to the present embodiment.

In FIG. 2, a face image acquisition unit 101 is a means for acquiring a face image taken by the camera 300.
Specifically, it refers to a photographing device such as a silver circle camera, a digital camera or a digital video camera, or an image extracting device such as a scanner. The acquired image is stored in the image storage area 111.
Note that the face image acquired by the face image acquisition unit 101 includes image data of the user's eye area.
The face image acquisition unit 101 is an example of a face image input unit.

The face image analysis unit 102 is a means for extracting the feature points of the eyes and eyebrows of the face image, in order to extract the face and eye regions from the image stored in the image storage region 111 and to calculate the feature point widths thereof. Image analysis means.
The extracted feature point extraction results are stored in the face image analysis result storage area 112 as face image analysis results.

The user state determination unit 103 uses the user state determination table stored in the user state determination table storage area 113 according to the change width of the plurality of feature point widths in the face image analysis result, for example, “difficult to see”. It is means for converting into user status information such as “normal” and “easy to see”.
The user status information is stored in the user status information storage area 114.
The style changing unit 104 is a means for selecting a display format corresponding to the analysis result from the user state information stored in the user state information storage area 114.
Here, the display format indicates information relating to the layout such as the font size, width, height, font color, and background color for each element constituting the content.
That is, the style changing unit 104 manages a plurality of types of display formats corresponding to different visibility, and selects one of the plurality of display formats based on the analysis result of the face image. .
Hereinafter, the display format is also referred to as a style.
The relationship between the face image analysis result and the style to be selected is stored as a style change table in the style change table storage area 115. By replacing this style change table, a different layout display can be provided even in the same user state. It is possible.
The user state determination unit 103 and the style change unit 104 are examples of a display format determination unit.

The content analysis unit 105 is a means for lexical analysis of the content information stored in the content storage area 116. The content analysis result 117_1 subjected to the lexical analysis is stored in the content analysis result storage area 117.
The content analysis result here is, for example, DOM (Document Object Model) information created as a result of analyzing such information when the content information is described in an HTML (Hyper Text Markup Language) format. .
The content layout unit 106 is means for generating layout information necessary for drawing the screen display data from the content analysis result stored in the content analysis result storage area 117.
The generated layout information is stored in the layout information storage area 118.
The layout information here is information indicating the display format of the screen display data. For example, the layout information such as the coordinates, width, and height of each element in the content, the background color, the character color, and the like are necessary for drawing. Refers to information.
The layout information drawing unit 107 is means for creating screen display data from the layout information stored in the layout information storage area 118 and displaying it on the display.
The content analysis unit 105, the content layout unit 106, and the layout information drawing unit 107 are examples of a display data generation unit.

In FIG. 2, an image storage area 111 is means for storing a plurality of images 111_1.
The face image analysis result storage area 112 is means for storing a face image analysis result 112_1 in which the feature point widths of eyes and eyebrows extracted from a plurality of images are stored for each image.
The user state determination table storage area 113 is a means for storing a user state determination table 113_1 for associating the change condition of the feature point width with the state information that satisfies the condition condition.
The user state information storage area 114 is a means for storing the user state information 114_1 determined by the user state determination unit 103.
The style change table storage area 115 is a means for storing the style change table 115_1.
The content storage area 116 is means for storing a plurality of content information 116_1 such as HTML information.
The content analysis result storage area 117 is means for storing content analysis result information 117_1 such as a plurality of DOM information.
The layout information storage area 118 is means for storing layout information 118_1 such as a plurality of arrangement information.

FIG. 3 shows an example of a face image acquired by the face image acquisition unit 101. It is only necessary to obtain a part of the face image such as the size and position of the eyes, the position of the eyebrows, and the outline of the face.
Note that image processing such as discoloration processing and color reversal processing may be performed to extract each position.

FIG. 4 is an example of the measurement of eye feature points analyzed by the face image analysis unit 102, and the normal feature points are obtained from the camera in the state of eyes and eyebrows of the user's face staring at the display device 200. An example of the width is shown.
The eye width, eye height, eye-to-eye spacing, and eyebrow height at this time are referred to as eye width 1, eye height 1, eye interval 1, and eyebrow height 1, respectively.

FIG. 5 is a measurement example of eye feature points analyzed by the face image analysis unit 102. When the eyes are narrowed in the state of eyes and eyebrows of the user's face staring at the display device 200 obtained from the camera. An example of the width of the feature point is shown.
The eye width, eye height, eye-to-eye distance, and eyebrow height at this time are referred to as eye width 2, eye height 2, eye interval 2, and eyebrow height 2, respectively.

FIG. 9 is an example of a totaling result of pixel widths of eye width, eye height, eye interval, and eyebrow height extracted from a plurality of face images.
Image1 and Image2 indicate pixel widths of eye width, eye height, eye interval, and eyebrow height extracted from face images taken at different times.
In other words, in the example of FIG. 9, the eye width is enlarged in Image2, the eye height is reduced in Image2, the eye interval is reduced in Image2, and the eyebrow height is reduced in Image2. I understand.

FIG. 10 is an example of a user state determination table showing the relationship between the feature point change width condition and the constant value of the corresponding state.
That is, when the feature point of the eye area of the user's face image photographed by the camera 300 matches the feature point change width condition, the user's current state becomes a corresponding state (difficult to see, approach, etc.). It can be judged that there is.

  FIG. 11 is an example of user status information that stores the current status value of the user analyzed by the user status determination unit 103.

FIG. 12 shows an example of a style change table, which is an example of a table used as a layout interpretation when the content layout unit 106 lays out the content information 116_1 stored in the content storage area 116.
In particular, this table is an example of a table in which the character size of each element is designated for each facial expression (user state).
Note that <H1>, <H6>, <P>, and the like designated as element names indicate elements of the content information 116_1 (display data) illustrated in FIG.
In FIG. 12, when the current state of the user is “difficult to see”, the content (character string or the like) of the element <H1> is displayed at 48 pxl, and when the current state of the user is “normal”, the element < The content of H1> (character string or the like) indicates that it is displayed in 24 pxl.
That is, when the user's state is “difficult to see”, a display format that provides high visibility is selected.

FIG. 13 shows another example of the style change table, which is an example of a table used as a layout interpretation when the content layout unit 106 lays out the content information 116_1 stored in the content storage area 116.
In particular, this table is an example of a table indicating elements after conversion of each element for each facial expression (user state).
The character size is determined for each tag shown in FIG. 13. For example, when the user's state is “difficult to see”, the <H6> tag is replaced with the <H3> tag and converted to a larger character size. Displayed.
The character size of each tag is managed in a separate table.

  FIG. 15 shows an example of content description, which is an example of content described in HTML (Hyper Text Markup Language). In the present invention, the content is not limited to HTML, but may be XML (Extensible Markup Language) or content of a unique format.

  FIG. 16 shows an example of content analysis result information. When the content information is described in the HTML format as shown in FIG. 15, DOM (Document Object Model) information created as a result of analyzing the information is shown.

  FIG. 17 shows an example of layout information, which summarizes arrangement information such as coordinates, width, and height, font size information, and the like for each element of content.

  FIG. 18 shows an example in which the content is analyzed and laid out for the description of FIG. 15 when the facial expression is normal.

  FIG. 19 shows an example in which the contents of FIG. 15 are analyzed and laid out using the style change table of FIG. At this time, the character size is converted from the “normal” character size to the “difficult” character size.

  FIG. 20 shows an example in which the content shown in FIG. 15 is analyzed and laid out using the style change table shown in FIG. At this time, tag conversion corresponding to a conversion tag that is “difficult to see” is performed (<H6> in FIG. 15 is replaced with <H3>, and <P> in FIG. 15 is replaced with <H2>).

The overall processing flow of the display change device 100 in the first embodiment will be described with reference to FIG.
In the display change device 100, the face image acquisition unit 101 performs a camera image acquisition process from the connected camera 300, and stores the acquired image 111_1 in the image storage area 111 (S1).
Next, the face image analysis unit 102 extracts feature points from the face image by the face image analysis process A2 from the acquired plurality of continuous images, calculates the eye width, eye height, eye interval, eyebrow height, etc. The analysis result 112_1 is stored in the face image analysis result storage area 112 (S2).
Next, the user state determination unit 103 compares the face image analysis result 112_1 and the user state determination table 113_1 stored in the user state determination table storage area 113 to determine whether each width value satisfies the condition, and Is stored in the user status information storage area 114 as the user status information 114_1. Here, judgments such as “difficult to see”, “normal”, and “easy to see” are made (S3).
If the user state has changed from the previous state (change from “normal” to “difficult to see”, etc.), the process proceeds to S4, and if the state has not changed, the process returns to S1. .
For example, in the change from the continuously acquired image Image1 to Image2 shown in FIG. 9, the eye height is significantly reduced. Based on the comparison with the user state determination table in FIG. It can be determined that the user is in a “difficult” state, and the user state can be determined to have changed from “normal” to “difficult to see”.

Next, the style change unit 104 refers to the style change table stored in the style change table storage area 115 and acquires the style.
For example, if the style change table is 115_1 in FIG. 12 and the user status is “difficult to see”, the <H6> tag is changed from the normal character size of 8 pxl to the “difficult to see” character size of 16 pxl. In addition, when elements are converted as indicated by 115_2 in FIG. 13, the contents are converted at this timing (S4).
Next, the content analysis unit 105 extracts the content 116_1 stored in the content storage area 116, and analyzes the content.
The analysis result is converted into a data structure that can be interpreted by the content layout unit 106, such as DOM (Document Object Model). The content analysis result 117_1 is stored in the content analysis result storage area 117 (S5).
Next, the content layout unit 106 performs content layout processing using the content analysis result 117_1 and the style change table 115_1 (115_2). The content layout processing is processing for setting the arrangement, width, height, color, and number of elements described in the content 116_1. Through this process, layout information 118_1 is generated and stored in the layout information storage area 118. The layout information is information dependent on the mounting system (S6).
Finally, the layout information drawing unit 107 performs a drawing process based on the layout information stored in the layout information storage area 118, and displays it on the display device 200 (S7).

Thus, in the present embodiment, the camera 300 captures the face of the user who views the display data displayed on the display device 200, and the display change device 100 analyzes the image data of the eye area of the user's face image. It is determined whether the user is in a normal eye state (face image A400) or in a narrowed state (face image B500). It is determined that it is difficult to view, and the display layout of the display data is changed to a more easily viewable format (corresponding layout display B700), and the display data along the more easily viewable display layout is output to the display device 200.
For this reason, the display data is displayed in a display format that is easy for the user to view, without imposing the burden of changing the content or program, adjusting the character size, etc., or when the user's hand is blocked and adjustment work cannot be performed. Can be generated.

In the user state determination table of FIG. 10, the determination criterion for determining the user state is shown by the change width of each of eye width, eye height, eye interval, and eyebrow height. May be.
For example, the user status determination table is provided with a determination criterion that the user's eye width / eye height ratio is “normal” if the ratio is greater than or equal to a certain value, and “difficult to see” if the ratio is less than a certain value. The ratio of the user's eye width and eye height in the image data of the user's eye area is analyzed, and the user's eyes are narrowed according to the determination criteria of the user state determination table, and the state is "difficult to see" It may be determined whether or not.

Further, in the above description, when the user's state is changed from “normal” to “difficult to see”, an example of immediately changing to a display format with high visibility has been described. The number of times it was determined to be in a “difficult to see” state or the time that the user was determined to be in the “difficult to see” state with a narrowed eye (the time during which the user kept narrowing his eyes) was measured. When the threshold value is exceeded, a display format corresponding to high visibility may be selected.
Moreover, you may combine both the frequency | count that the user narrowed eyes, and the time which is narrowing eyes.
In this way, it is possible to increase the character size, etc., only when the user is difficult to see the display screen, and the display screen is not difficult to see, but the user just narrows his eyes for some reason In addition, it is possible to avoid the inconvenience of increasing the character size or the like by mistake.

In the above description, the display format is changed when the user's state changes from “normal” to “difficult to see”. For example, the display width is changed when the display change device 100 is previously set to “normal” by the user. The values of feature points such as heights are stored, and the user's state is judged by comparing the eye width, eye height, etc. obtained from the captured image of the camera with the stored eye width, eye height, etc. Good.
In consideration of the fact that the distance between the user and the camera is not always the same due to changes in the sitting position, posture changes, etc., factors that have little change in length and width in the face (face width, nose width, etc.) Are stored in advance, and these values obtained from the camera image are compared with the stored values, and according to the comparison results, the comparison results such as the eye width and the eye height are adjusted to change the user's state. You may make it judge.

Embodiment 2. FIG.
In the second to fifth embodiments, an example that more specifically describes the first embodiment described above and an example that is derived from the first embodiment will be described.

  In the second embodiment, a facial expression following character size changing method for changing the character size and appearance of content according to a change in the eyes of one user will be described.

For example, assume that the image acquired by the camera has changed from FIG. 4 to FIG.
At this time, it is assumed that a style change table 115_1 as shown in FIG. 12 is stored in the style change table storage area 115 for the content indicated by 116_1 in FIG. 15 stored in the content storage area 116.
In this style change table 115_1, the character size of each element is described, and the character size that is “difficult to see” that is larger than the “normal” character size is registered. At this time, face image analysis is performed by the face image analysis unit 102, and when the user state determination unit 103 corresponds to “difficult to see”, the “unreadable” character size of the style change table 115 </ b> _ 1 is applied.

By changing the character size as described above, the contents of the display shown in FIG. 18 are displayed as shown in FIG.
Thereby, when the user narrows his eyes, an image with narrow eyes is obtained. By analyzing the image, it is determined that the display is difficult to see, and the character size can be automatically increased.

As a result, it is possible to provide a display with a character size that is easy for the user to view without performing an operation on the display terminal, by showing the user a gesture that makes it difficult to see the display on the display terminal (an operation that narrows the eyes).
Note that the character size may be changed in accordance with the presence or absence of an eye accessory instead of the action of narrowing the eye, as in the sixth embodiment described later. In addition, the character size may be changed according to the number of times the user approaches instead of the action of narrowing the eyes as in the fifth embodiment described later.

Embodiment 3 FIG.
In the present embodiment, as an example of the first embodiment, a facial expression following content conversion method in which an element in content is replaced by a change in the eyes of one user and the content is rewritten will be described.

For example, it is assumed that the style change table 115_2 shown in FIG. 13 is stored in the style change table storage area 115 for the content shown in FIG. 15 stored in the content storage area 116.
At this time, when the face image analysis is performed by the face image analysis unit 102 and the user state determination unit 103 corresponds to “difficult to see”, the <H6> element in the content is set to <H3> according to the style change table of FIG. The element is converted into an element, and the <P> element is converted into an <H2> element.
The character size of each element is managed in a separate table. If the user's status is “difficult to see,” the element is replaced based on the table to change it to a larger character size and make it more visible. The display format can be changed to a highly specific one.
However, if it is not desired to modify the original content, it is desirable to store a copy of the original content in the content storage area 116 in advance (referred to as copy content) and perform the above element conversion on the copy content.

  By performing the content conversion as described above, the contents of the display display in FIG. 18 are displayed as shown in FIG.

As a result, it is possible to provide a content display that is easy for the user to view without performing an operation on the display terminal by showing the user that the display on the display terminal is difficult to see (the action of narrowing his eyes).
Note that the tag conversion may be performed in accordance with the presence or absence of an eye accessory instead of the operation of narrowing the eye, as in the sixth embodiment described later. Further, as in the fifth embodiment described later, the tag conversion may be performed according to the number of times the user approaches instead of the operation of narrowing the eyes.

Embodiment 4 FIG.
In the present embodiment, as an example in the first embodiment, a facial expression following CSS changing method for changing a CSS (Cascading Style Sheet) to be applied according to a change in eyes of one user will be described.

  The present embodiment solves the same problems as those of the first to third embodiments, and is a technique for performing character size and layout without changing existing content by changing CSS applied as a solution. . CSS is a definition file or description that describes layout interpretation for each element described in content such as HTML.

FIG. 14 shows an example of a style change table according to the present embodiment, which is an example of a table used as a layout interpretation when the content layout unit 106 lays out the content information 116_1 stored in the content storage area 116.
In particular, this table is an example of a style sheet list table applied to each facial expression.
Here, the style sheet is template information that summarizes the definition of the appearance of the document of layout information, represented by CSS.

It is assumed that a style change table 115_3 as shown in FIG. 14 is stored in the style change table storage area 113 as the style change table.
This CSS change table describes the file name of the CSS file to be referred to according to the facial expression of each face.
At this time, face image analysis is performed by the face image analysis unit 102, and if it is determined as “difficult to see” by the user state determination unit 103, BigSize. Apply css and reflect in layout.

  By changing the CSS to be applied as described above, the contents of the display on the display in FIG. 18 are displayed larger as shown in FIG.

As a result, it is possible to provide a content display that is easy for the user to view without performing an operation on the display terminal by showing the user that the display on the display terminal is difficult to see (the action of narrowing his eyes).
Note that the CSS file may be changed according to the presence / absence of an eye accessory instead of the operation of narrowing the eye as in the sixth embodiment described later. Further, as in the fifth embodiment to be described later, the CSS file may be changed in accordance with the number of times the user approaches instead of the operation of narrowing the eyes.

Embodiment 5 FIG.
In the present embodiment, as a derivative example of the first embodiment, a face approaching and following character size changing method for changing the character size of content according to the approach of one user's face will be described.

When the user keeps his eyes close to the display terminal, it is considered difficult to see the display, and the same problem as in the first embodiment occurs to make the display easy to see.
The present embodiment solves the above-described problem. The user's face is photographed from a camera attached to the display terminal, and the user moves his eyes closer to the display terminal from the change in the size of the eyes. If it is approaching or there are many approaches, it is determined that the display is difficult to see, and the character size can be automatically increased.

FIG. 8 is a measurement example of eye feature points analyzed by the face image analysis unit 102, approaching the display device 200 in the state of eyes and eyebrows of the user's face looking at the information terminal obtained from the camera. An example of the width of the feature point when the
The eye width, eye height, eye-to-eye distance, and eyebrow height at this time are referred to as eye width 3, eye height 3, eye interval 3, and eyebrow height 3, respectively. Here, all the values are larger than those in FIG.
As described above, the state in which the user is approaching the display device 200 is considered to be a state in which display data is difficult to see.

When the camera image changes from FIG. 4 to FIG. 8 as to whether or not the user is approaching the display device 200, the user state determination unit 103 follows the determination criterion of FIG. , Eye height 1 <eye height 3, eye interval 1 <eye interval 3, eyebrow height 1 <eyebrow height 3, all of which are determined to be “approaching”.
When the number of times determined as “approaching” is a predetermined number or more, the style changing unit 104, the content analyzing unit 105, the content layout unit 106, and the layout information drawing unit 107 are “difficult to see” in the first to fourth embodiments. The display format of the display data is changed to a display format with high visibility (such as character size expansion) in the same procedure as the state, and the display data with high visibility is output to the display device 200.

  Accordingly, it is possible to estimate the current viewability of the display from the number of times the user approaches, and to provide a display with a character size that is easy for the user to view without performing an operation on the display terminal.

Further, in the above description, the example in which the display format is changed to a highly visible display format when the number of times the user has approached the display device is greater than or equal to the predetermined number has been described. The display format may be changed to a high-visibility display format when the time (the time of being kept approaching) is equal to or longer than a predetermined time.
Moreover, you may use combining the frequency | count of approach of a user's face, and approach time.

Embodiment 6 FIG.
In the present embodiment, as a derivative example of the first embodiment, a face accessory that changes the character size of the content in accordance with the presence or absence of an accessory (attachment) attached to the eye area of one user's face. The corresponding character size changing method will be described.

FIG. 6 shows an example of eye feature points analyzed by the face image analysis unit 102. The contact lens is worn in the state of eyes and eyebrows of the user's face looking at the information terminal obtained from the camera. It is an example.
At this time, a circle drawn thinly around the pupil of the eye can be extracted from the captured image.

FIG. 7 is an example of eye eye feature points analyzed by the face image analysis unit 102. The eyewear and eyebrows of the user's face staring at the information terminal obtained from the camera are worn. This is an example.
At this time, a square outline around the eyes can be extracted from the captured image.

When a user uses contact lenses, glasses, or the like, it is considered that the user's visual acuity is poor, and the same problem as in the first to fifth embodiments occurs in order to make the display easy to see.
The present embodiment solves the above-described problem. The user's face is photographed from the camera attached to the display terminal, and the presence or absence of glasses or contact lenses attached to the eyes as shown in FIGS. To extract. If these incidental objects exist, it is determined that the display is difficult to see, and the character size can be automatically increased.

  Thus, it is possible to provide a display with a character size that is easy for the user to view without performing an operation on the display terminal depending on the presence or absence of the user's glasses.

Further, for example, the storage area of the display change device 100 stores that the user normally wears glasses or contact lenses, and the face image analysis unit 10 captures the user's face image captured by the camera 300. As a result of analyzing the above, when it is detected that there are no incidental objects in the eye area (there is no square outline around the eye and no circle outline around the pupil of the eye), the user is assumed to be naked Therefore, the user state determination unit 103 may determine that the user state is “difficult to see”.
The subsequent operations are the same as those in the first to fourth embodiments, and the style changing unit 104, the content analyzing unit 105, the content layout unit 106, and the layout information drawing unit 107 are in the “difficult to see” state in the first to fourth embodiments. The display format of the display data is changed to a display format with high visibility (such as character size expansion) in the same procedure as described above, and the display data with high visibility is output to the display device 200.

Embodiment 7 FIG.
In a projector in which a plurality of users can view display data at the same time, when it is difficult to see the displayed screen, the lens is often manually adjusted to adjust the size of the displayed canvas.
In the present embodiment, the method shown in the first embodiment is extended, and the character size, layout, and screen of the display screen are changed without changing the content according to the amount of change in the eyes and eyebrows of a plurality of faces. A method for changing the size will be described.
In the present embodiment, as will be described later, a plurality of face images are collected by using a camera, and changes in the eyes are analyzed. At this time, assuming that the size of the canvas is small if you are narrowing your eyes, compare the number of faces with narrowed eyes with the number that does not, and if the number of faces with narrowed eyes is large, the canvas Try to increase the size.
The inside of the display change device is not changed from that of the first embodiment. Hereinafter, differences from the first embodiment will be described.

In the present embodiment, the face image analysis unit 102 has means for extracting eye and eyebrow feature points from a plurality of faces shown in one image and judging their facial expressions.
FIG. 22 is a diagram illustrating an example of an image in which a plurality of faces are included in a single image. That is, in the present embodiment, a case where a plurality of users are facing the display device 200 (projector) and a plurality of users are viewing display data displayed on the display device 200 is targeted. And the camera 300 is image | photographing the face part containing the eye area | region of these multiple users.

The face image acquisition unit 101 according to the present embodiment inputs imaging data of a plurality of users as shown in FIG.
The face image analysis unit 102 according to the present embodiment determines the number of users from the outline of the face and calculates the change value of the feature point from the eyes and eyebrows of a plurality of faces in the shooting data as shown in FIG. To do.
In addition, the user state determination unit 103 according to the present embodiment determines “difficult to see”, “normal”, and “easy to see” for each face according to the change value of the feature point for each face. Count the numbers. The method for determining the state of each user is as described in the first embodiment, and “difficult to see”, “normal”, and “easy to see” are determined for each of a plurality of users.
For example, as a result of analyzing and totaling FIG. 22, a totaling result as shown in FIG. 23A is obtained.
At this time, it is shown that there are three faces in the “difficult to see” state, two faces in the “normal” state, and one face in the “sleeping” state. “It is difficult to see” with many results is evaluated as a facial expression judgment as a whole (FIG. 23B).
The means for changing the font size and layout in accordance with the facial expression determination is the same as in the first to fourth embodiments. That is, the style changing unit 104, the content analyzing unit 105, the content layout unit 106, and the layout information drawing unit 107 change the display format of the display data in the same manner as in the “difficult to see” state in the first to fourth embodiments. The display format is changed to a display format (such as character size expansion), and display data with high visibility is output to the display device 200.

  In this way, by analyzing the change and number of eyes of multiple users, it is possible to provide a projector screen that is easy to see for multiple users who are present automatically without the user having to manually adjust the lens of the projector. Is possible.

  In the above description, when the number of users in the “difficult to see” state is the largest, the display format of the display data is changed to a format with high visibility. However, the number of users in the “difficult to see” state is predetermined. The display format of the display data is changed to a highly visible format when the ratio of the number of users in the “difficult to see” state to the total number of users exceeds a predetermined threshold. You may make it do.

In the above description, it is determined whether each user is in a “difficult to see” state by analyzing whether the user narrows his eyes. It may be determined whether or not the face is approaching, and the display format of the display data may be changed to a format with high visibility when the number of users approaching the face is the largest.
In addition, when the number of times the user is narrowing his eyes, the time, or the number of times the user is approaching his face, the time is measured, and the measurement result is equal to or greater than a predetermined threshold, the display format of the display data is You may make it change into a high form.

Embodiment 8 FIG.
In the present embodiment, as an example derived from the seventh embodiment, a method for changing the display speed (moving speed of a display object) of a display object on an electronic bulletin board according to the rate of change of eyes of a plurality of users will be described. To do.

In order to display a large amount of information in a limited space, an electric bulletin board on the premises of a station is used to scroll a character string from right to left at a certain speed. The scrolling speed is determined for each electronic bulletin board. To change the scrolling speed, it is necessary to modify the program written in the memory in the electronic bulletin board apparatus or switch the hard switch.
In the present embodiment, a plurality of face images are collected by using a camera, and changes in the eyes are analyzed. At this time, the left and right movements of the plurality of users are analyzed for the left and right movements of the pupil of the eye and the face, and the number of users is analyzed for each movement amount and used as a style change table. At this time, this changes the style according to the amount of movement and reflects it in the display.
Also, if there are many people who are narrowing their eyes, it may be determined that the display speed is too fast and the display speed may be slowed down.

  As a result, by analyzing the changes and number of eyes of multiple users, users can analyze the facial expressions of the places where they are installed and the surrounding people without manually changing the program on the electronic bulletin board. By doing so, it is possible to automatically provide a projector screen that is easy to see for a plurality of users who are present, without manually adjusting the lens of the projector.

As described above, in Embodiments 1 to 8, only the feature points of the eyes and the eyebrows are used as the feature points of the face, and the target user can be changed by changing the character size and layout of the target screen according to the amount of change. A method for providing a suitable display effect has been described.
In addition, the use of eyes and eyebrows and the presence or absence of an accessory around the eyes (contact lenses, glasses, etc.) as facial feature points has been described.
Furthermore, it explained that the following was used as a change of the feature point of eyes and eyebrows.
(A) Change in size of part of eyes (whether eyes are narrowed)
(A) Presence or absence of eye accessories (presence of eyeglasses, presence of contact lenses)
(C) Changes in the size of the eyes and eyebrows (whether they are approaching)
Moreover, it demonstrated using the following display effects as a display effect effective for a user.
(A) Change of character size (b) Change of content element (c) Change of style sheet (CSS etc.) (d) Change of display speed (e) Change of display screen size

Finally, a hardware configuration example of the display change device 100 shown in the first to eighth embodiments will be described.
FIG. 24 is a diagram illustrating an example of hardware resources of the display change device 100 described in the first to eighth embodiments.
Note that the configuration in FIG. 24 is merely an example of the hardware configuration of the display change device 100, and the hardware configuration of the display change device 100 is not limited to the configuration described in FIG. Also good.

In FIG. 24, the display change device 100 includes a CPU 911 (also referred to as a central processing unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, and a processor) that executes a program.
The CPU 911 is connected to, for example, a ROM (Read Only Memory) 913, a RAM (Random Access Memory) 914, a communication board 915, a display device 200, a keyboard 902, a mouse 903, a magnetic disk device 920, and a camera 300 via a bus 912. And control these hardware devices.
Further, the CPU 911 may be connected to an FDD 904 (Flexible Disk Drive), a compact disk device 905 (CDD), a printer device 906, and a scanner device 907. Further, instead of the magnetic disk device 920, a storage device such as an optical disk device or a memory card (registered trademark) read / write device may be used.
The RAM 914 is an example of a volatile memory. The storage media of the ROM 913, the FDD 904, the CDD 905, and the magnetic disk device 920 are an example of a nonvolatile memory. These are examples of the storage device.
A communication board 915, a keyboard 902, a mouse 903, a camera 300, a scanner device 907, an FDD 904, and the like are examples of input devices.
The communication board 915, the display device 200, the printer device 906, and the like are examples of output devices.

  For example, the communication board 915 may be connected to a LAN (local area network), the Internet, a WAN (wide area network), or the like.

The magnetic disk device 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924.
The programs in the program group 923 are executed by the CPU 911 using the operating system 921 and the window system 922.

The RAM 914 temporarily stores at least part of the operating system 921 program and application programs to be executed by the CPU 911.
The RAM 914 stores various data necessary for processing by the CPU 911.

The ROM 913 stores a BIOS (Basic Input Output System) program, and the magnetic disk device 920 stores a boot program.
When the display change device 100 is activated, the BIOS program in the ROM 913 and the boot program in the magnetic disk device 920 are executed, and the operating system 921 is activated by the BIOS program and the boot program.

  The program group 923 stores a program for executing the function described as “˜unit” in the description of the first to eighth embodiments. The program is read and executed by the CPU 911.

In the file group 924, in the description of the first to eighth embodiments, “determination of”, “calculation of”, “calculation of”, “comparison of”, “analysis of”, “extraction of” ”,“ Update of ”,“ setting of ”,“ registration of ”,“ selection of ”, etc. It is stored as each item of "~ file" and "~ database".
The “˜file” and “˜database” are stored in a recording medium such as a disk or a memory. Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 911 via a read / write circuit, and extracted, searched, referenced, compared, and calculated. Used for CPU operations such as calculation, processing, editing, output, printing, and display.
Information, data, signal values, variable values, and parameters are stored in the main memory, registers, cache memory, and buffers during the CPU operations of extraction, search, reference, comparison, calculation, processing, editing, output, printing, and display. It is temporarily stored in a memory or the like.
The arrows in the flowcharts described in the first to eighth embodiments mainly indicate input / output of data and signals. The data and signal values are the RAM 914 memory, the FDD 904 flexible disk, the CDD 905 compact disk, and the magnetic field. Recording is performed on a recording medium such as a magnetic disk of the disk device 920, other optical disks, mini disks, DVDs, and the like. Data and signals are transmitted online via a bus 912, signal lines, cables, or other transmission media.

  In addition, what is described as “to part” in the description of the first to eighth embodiments may be “to circuit”, “to device”, “to device”, and “to step”, It may be “˜procedure” or “˜processing”. That is, what is described as “˜unit” may be realized by firmware stored in the ROM 913. Alternatively, it may be implemented only by software, only hardware such as elements, devices, substrates, wirings, etc., or a combination of software and hardware, and further a combination of firmware. Firmware and software are stored as programs in a recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD. The program is read by the CPU 911 and executed by the CPU 911. That is, the program causes the computer to function as “to part” in the first to eighth embodiments. Alternatively, the computer executes the procedure and method of “to unit” in the first to eighth embodiments.

  As described above, the display change device 100 according to the first to eighth embodiments includes a CPU as a processing device, a memory as a storage device, a magnetic disk, a keyboard as an input device, a mouse, a communication board, and a display device as an output device, a communication device, and the like. A computer including a board or the like, and implements the functions indicated as “to part” as described above using these processing devices, storage devices, input devices, and output devices.

FIG. 3 is a diagram illustrating an example of a system configuration according to the first embodiment. FIG. 3 shows a configuration example of a display change device according to the first embodiment. FIG. 4 is a diagram illustrating an example of a face image acquired by a face image acquisition unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of measuring eye feature points analyzed by a face image analysis unit according to the first embodiment; FIG. 6 is a diagram illustrating an example of measuring eye feature points analyzed by a face image analysis unit according to the first embodiment; FIG. 10 is a diagram illustrating an example of measuring eye feature points analyzed by a face image analysis unit according to Embodiment 6; FIG. 10 is a diagram illustrating an example of measuring eye feature points analyzed by a face image analysis unit according to Embodiment 6; FIG. 10 is a diagram illustrating a measurement example of feature points of eyes analyzed by a face image analysis unit according to the fifth embodiment. The figure which shows an example of the total result of the pixel width of the eye width extracted from the face image which concerns on Embodiment 1, eye height, eye space | interval, and eyebrow height. The figure which shows an example of the user state judgment table which shows the relationship between the condition of the feature point change width which concerns on Embodiment 1, and the state corresponding to it. The figure which shows an example of the user state information which stored the user's present state value analyzed by the user state judgment part which concerns on Embodiment 1. FIG. The figure which shows the example of the style change table which concerns on Embodiment 1,2. FIG. 6 shows an example of a style change table according to Embodiments 1 and 3. FIG. 10 shows an example of a style change table according to the fourth embodiment. FIG. 6 is a diagram showing an example of content description according to the first embodiment. FIG. 6 is a diagram showing an example of content analysis result information according to the first embodiment. FIG. 5 is a diagram showing an example of layout information according to the first embodiment. FIG. 6 is a diagram showing a display example of the display device when the user state according to Embodiment 1 is “normal”. The figure which shows the example of a display of a display apparatus when the user state which concerns on Embodiment 1, 2 is "it is hard to see." The figure which shows the example of a display of a display apparatus when the user state which concerns on Embodiment 1, 3 is "it is hard to see." FIG. 3 is a flowchart showing an operation example of the display change device according to the first embodiment. FIG. 18 shows an example of a plurality of face images according to the seventh embodiment. FIG. 18 shows an example of a user state analysis result according to the seventh embodiment. The figure which shows the hardware structural example of the display change apparatus which concerns on Embodiment 1-8.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 Display change apparatus, 101 Face image acquisition part, 102 Face image analysis part, 103 User state determination part, 104 Style change part, 105 Content analysis part, 106 Content layout part, 107 Layout information drawing part, 111 Image storage area, 112 Face image analysis result storage area, 113 User state determination table storage area, 114 User state information storage area, 115 Style change table storage area, 116 Content storage area, 117 Content analysis result storage area, 118 Layout information storage area, 200 Display device , 300 camera, 400 face image A, 500 face image B, 600 corresponding layout display A, 700 corresponding layout display B.

Claims (19)

A data processing device connected to a display device for displaying display data and a photographing device for photographing a user's face facing the display device,
A face image input unit for inputting the face image data of the user, which includes image data of the eye area of the user, which is imaged by the imaging device;
A face image analysis unit that analyzes image data of the user's eye area of the face image data;
A display format determination unit that determines a display format of display data to be displayed on the display device based on an analysis result of the face image analysis unit;
A data processing apparatus, comprising: a display data generation unit that generates display data in accordance with the display format determined by the display format determination unit and outputs the generated display data to the display device. The display format determination unit
As the display format of the display data, a plurality of types of display formats corresponding to different visibility are managed, and one of the display formats is selected based on the analysis result of the face image analysis unit, The data processing apparatus according to claim 1. The face image analysis unit
Analyzing the image data of the user's eye area to determine whether the user is in a state of narrowing his eyes,
The display format determination unit
As the display format of the display data, at least a display format corresponding to a normal level visibility and a display format corresponding to a higher visibility than the normal level are managed, and the user is controlled by the face image analysis unit. The data processing apparatus according to claim 2, wherein a display format corresponding to a visibility higher than a normal level is selected when it is determined that is in a state where the eye is narrowed. The face image analysis unit
Analyzing the image data of the user's eye area to determine whether the user is in a state of narrowing his eyes,
The display format determination unit
When display data generated in accordance with any display format is displayed on the display device, when the face image analysis unit determines that the user is in a state of narrowing his eyes, The data processing device according to claim 2 or 3, wherein a display format corresponding to a higher visibility than a display format of display data displayed on the display device is selected. The face image analysis unit
Analyzing the ratio between the user's eye width and eye height in the image data of the user's eye area to determine whether or not the user is in a state of narrowing his eyes. The data processing apparatus according to claim 3 or 4. The display format determination unit
At least one of the number of times that the face image analysis unit determines that the user is in a state of narrowing his eyes and the time when the face image analysis unit determines that the user is in a state of narrowing his eyes The data processing apparatus according to claim 3, wherein a display format corresponding to high visibility is selected when the measurement result exceeds a predetermined threshold value. The face image analysis unit
Analyzing the image data of the user's eye area to determine whether the user is in a state of approaching the display device,
The display format determination unit
As the display format of the display data, at least a display format corresponding to a normal level visibility and a display format corresponding to a higher visibility than the normal level are managed, and the user is controlled by the face image analysis unit. The display format corresponding to the visibility higher than the normal level is selected when it is determined that the face is close to the display device. The data processing apparatus described. The face image analysis unit
Analyzing the image data of the user's eye area to determine whether the user is in a state of approaching the display device,
The display format determination unit
When the display data generated in accordance with any display format is displayed on the display device, the face image analysis unit determines that the user is approaching the display device. 8. A data processing apparatus according to claim 2, wherein a display format corresponding to a higher visibility than a display format of display data displayed on the display device is selected when the display data is displayed. . The face image analysis unit
Analyzing the size of the user's eyes in the image data of the user's eye area to determine whether or not the user is in a state of approaching the display device; The data processing apparatus according to claim 7 or 8. The display format determination unit
The number of times that the user has determined that the face is approaching the display device by the face image analysis unit and the state in which the user is approaching the display device by the face image analysis unit 10. The method according to claim 7, wherein at least one of the times determined to be measured is measured, and when the measurement result exceeds a predetermined threshold, a display format corresponding to high visibility is selected. A data processing device according to any one of the above. The face image analysis unit
Analyzing the image data of the user's eye area to determine whether there is an attachment in the user's eye area,
The display format determination unit
The data processing according to any one of claims 1 to 10, wherein a display format of display data is determined based on the presence or absence of an attachment in the user's eye area analyzed by the face image analysis unit. apparatus. The face image input unit
The face image data of the plurality of users including the image data of the eye areas of the plurality of users photographed by the photographing device is input,
The face image analysis unit
Analyze the eye area image data of the face image data for each user,
The display format determination unit
The data processing apparatus according to claim 1, wherein a display format of display data to be displayed on the display device is determined based on an analysis result for each user of the face image analysis unit. The face image analysis unit
Analyze the image data of the eye area for each user, determine whether each user has a narrowed eye,
The display format determination unit
As the display format of the display data, a plurality of types of display formats having different visibility are managed, and at least a display format corresponding to a normal level visibility and a display format corresponding to a higher visibility than the normal level When the number of users determined to be in a state of narrowing eyes by the face image analysis unit is greater than a threshold value, a display format corresponding to visibility higher than the normal level is selected. The data processing apparatus according to claim 12. The face image analysis unit
Analyze the image data of the eye area for each user, determine whether each user has a narrowed eye,
The display format determination unit
As the display format of the display data, a plurality of types of display formats having different visibility are managed, and the display data generated in accordance with any display format is displayed on the display device, the face A display format corresponding to a higher visibility than the display format of the display data displayed on the display device when the number of users determined to be narrowed by the image analysis unit is greater than the threshold. 14. The data processing apparatus according to claim 12, wherein the data processing apparatus is selected. The face image analysis unit
Analyzing the image data of the eye area for each user, determining whether the user is in a state of approaching the face to the display device for each user,
The display format determination unit
As the display format of the display data, a plurality of types of display formats having different visibility are managed, and at least a display format corresponding to a normal level visibility and a display format corresponding to a higher visibility than the normal level And the display corresponding to the visibility higher than the normal level when there are more users than the threshold that the face image analysis unit determines that the face is approaching the display device. The data processing apparatus according to claim 12, wherein a format is selected. The face image analysis unit
Analyzing the image data of the eye area for each user, determining whether the user is in a state of approaching the face to the display device for each user,
The display format determination unit
As the display format of the display data, a plurality of types of display formats having different visibility are managed, and the display data generated in accordance with any display format is displayed on the display device, the face When the number of users determined by the image analysis unit to have their faces approaching the display device is greater than a threshold value, the visibility is higher than the display format of the display data displayed on the display device 16. The data processing apparatus according to claim 12, wherein a corresponding display format is selected. The display format determination unit
The display format of the display data is determined by at least one of a size of a display object displayed in the display data, a type of the display object, a number of the display objects, and a display size of the display data. Item 17. The data processing device according to any one of Items 1 to 16. The display format determination unit
The data processing according to any one of claims 1 to 17, wherein when a display object displayed in the display data moves, a moving speed of the display object is determined as a display format of the display data. apparatus. A computer connected to a display device for displaying display data and a photographing device for photographing a user's face facing the display device;
A face image input process for inputting the user's face image data, which includes image data of the user's eye area, imaged by the imaging device;
A face image analysis process for analyzing image data of the user's eye area of the face image data;
A display format determination process for determining a display format of display data to be displayed on the display device based on an analysis result in the face image analysis process;
A program for generating display data in accordance with the display format determined by the display format determination processing and executing display data generation processing for outputting the generated display data to the display device.

Images