WO2013108438A1

WO2013108438A1 - Display device, display method and display program

Info

Publication number: WO2013108438A1
Application number: PCT/JP2012/073958
Authority: WO
Inventors: 新開　誠
Original assignee: シャープ株式会社
Priority date: 2012-01-16
Filing date: 2012-09-19
Publication date: 2013-07-25
Also published as: JP2013145515A; JP5911168B2

Abstract

A display device according to an embodiment of the present invention has a control unit that sets the image layout after a change in orientation so as to display the whole of at least one video contained in contents when the orientation of the device changes.

Description

Display device, display method, and display program

The present invention relates to a display device, a display method, and a display program.
This application claims priority based on Japanese Patent Application No. 2012-006263 filed in Japan on January 16, 2012, the contents of which are incorporated herein by reference.

In recent years, portable terminals called mobile phones, smartphones, and tablets have become widespread. Along with this, so-called electronic book services that allow users to subscribe to books on a display on a terminal are spreading. Compared to conventional paper books, electronic books have the advantages of being able to carry a large number of books at once and playing multimedia data such as video and audio, and are expected to become more popular in the future.

In particular, content that incorporates multimedia data such as video and audio in an e-book can be expressed in a way that could not be realized with conventional paper. Therefore, such content can be said to be content that is beneficial to both content creators and users. For example, interactive content is assumed in which when a user selects a specific character or image, a moving image corresponding to the selected character or image is reproduced. Further, for example, a content is assumed that a moving image is automatically played when a specific page is displayed.

In the content of an electronic book incorporating such moving image data, for example, as a method of displaying a moving image, a method of displaying inline (embedded on the spot) in a book or a method of displaying on a full screen is conceivable. . For example, Patent Document 1 discloses an information display device that does not affect the layout of other characters on a screen by displaying a moving image as it is on a still image display area when displaying a moving image. .

JP 2006-53606 A

Some terminals such as smartphones and tablets use a built-in acceleration sensor to switch the display according to the orientation (vertical / horizontal) of the terminal. Therefore, even when the terminal displays an electronic book, it is required to display the content in a form that matches the orientation of the terminal. That is, when the orientation of the terminal is changed to the landscape orientation (rotation) while displaying a certain page in the portrait orientation, it is necessary to display the same page on the landscape orientation screen.

However, when the content includes a video, if the orientation of the terminal changes, depending on the content, the video may be located outside the display area after the orientation changes, and the video may not be displayed on the screen after the orientation has changed. . In particular, when the orientation of the terminal changes during playback of a moving image, depending on the content, the moving image may be located outside the display area after the orientation change, and the moving image may not be displayed on the screen after the orientation change. As a result, the user cannot comfortably view the content including the moving image.

Therefore, the present invention provides a display device, a display method, and a display program that enable comfortable viewing of content including moving images.

(1) The display device according to one aspect of the present invention determines the screen layout after the orientation change so that the entire at least one moving image included in the content is displayed on the screen when the orientation of the device changes. A control unit is provided.

(2) The display device may include a display unit that displays content including a moving image on a screen, and a direction detection unit that detects a change in the direction of the device itself on a plane including the screen. The control unit is configured to determine a screen layout after the change of direction so that at least one moving image included in the content is displayed on the screen when the direction detection unit detects a change in the direction. It may be.

(3) In the display device, when the control unit detects a change in orientation, the video viewing determination unit that determines whether or not the user is viewing the video displayed on the display unit, and the direction detection unit And when the moving image viewing determination unit determines that the moving image is being viewed, a layout unit that determines a screen layout after changing the orientation based on a position on the screen on which the moving image is displayed is provided. Also good.

(4) In the display device, the control unit may include a page position holding unit that holds a displayed page position and a change direction of the direction when the direction detected by the direction detection unit is changed. Good. The layout unit detects a page held by the page position holding unit when the change in direction detected by the direction detection unit is opposite to the direction change of the direction held by the page position holding unit. The screen layout after the change may be determined based on the position.

(5) In the display device, the moving image viewing determination unit may be configured to determine that the moving image is being watched at least when the moving image is being played back.

(6) In the display device, the control unit may include a line-of-sight detection unit that detects a user's line of sight. The moving image viewing determination unit may be configured to determine whether or not the user is viewing the moving image based on a reproduction state of the moving image and a line-of-sight direction detected by the line-of-sight detection unit.

(7) In the display device, the control unit may include a video selection unit that selects one video from among a plurality of videos included in the content as a video being viewed based on a viewing index. .

(8) In the display device, the viewing index may be a priority for each moving picture described in the document data. The moving image selection unit may be configured to select one moving image from among a plurality of moving images included in the content based on a priority for each moving image as a moving image being viewed.

(9) In the display device, the viewing index may be a display size of the moving image. The moving image selection unit may be configured to select one moving image from among a plurality of moving images included in the content as a moving image being viewed based on the display size of the moving image.

(10) In the display device, the viewing index may be a user's line of sight. The said control part may be provided with the 2nd gaze detection part which detects the said user's gaze direction. The moving image selection unit may be configured to select one moving image from among a plurality of moving images included in the content based on the line-of-sight direction detected by the line-of-sight detection unit.

(11) A display method according to another aspect of the present invention is a display method executed by the display device, and displays the entire at least one moving image included in the content on the screen when the orientation of the device changes. As described above, the method includes the step of determining the screen layout after the change of direction.

(12) A display program according to another aspect of the present invention is arranged so that when the orientation of the device changes, the computer of the display device displays at least one entire moving image included in the content on the screen. A step of determining the screen layout after the orientation change is executed.

According to the aspect of the present invention, it is possible to comfortably view content including moving images.

It is a schematic block diagram of the electronic book apparatus in one Embodiment of this invention. It is a schematic block diagram which shows the logical structure of the control part in this embodiment. It is a figure which shows an example of the page layout before rotating horizontally and after rotating horizontally. It is a figure which shows the example which specified the priority with respect to moving image data. It is a figure which shows the example which designated display size with respect to moving image data. It is a schematic block diagram of the gaze detection part in this embodiment. It is a schematic block diagram of the moving image viewing determination part in this embodiment. It is a figure for demonstrating the eye position and gaze direction which make the center position of a lens the origin. It is a flowchart which shows an example of the process of the basic part of the electronic book apparatus of this embodiment. In the electronic book apparatus of this embodiment, it is a flowchart which shows an example of a process when the direction of an own apparatus changes. 10 is a flowchart illustrating an example of a processing flow of the electronic book device when the orientation of the device is returned to the original state immediately after the orientation of the device is changed. It is a figure for demonstrating the 1st comparative example of the system which produces | generates a page dynamically according to the display size of an electronic book apparatus. It is a figure for demonstrating the 2nd comparative example which displays two pages side by side when an electronic book apparatus is turned sideways.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, an electronic book device that displays an electronic book will be described as an example of a display device for displaying an electronic book.

First, a comparative example to be compared with the electronic book device of the present embodiment will be described with reference to FIGS. There are several ways to display electronic books depending on the type of content. For example, a method of dynamically generating pages according to the display size of the electronic book device (FIG. 12), or the electronic book device in landscape orientation. In such a case, a method of displaying with two pages (for two pages) can be used.

FIG. 12 is a diagram for explaining a first comparative example of a method of dynamically generating pages according to the display size of the display device. In the figure, an example of the display screen 71 in the case where the display device is oriented vertically is shown. In the display screen 71, it is shown that a character area 71a and a moving image 71b are displayed.

Further, the display screen 72 in the figure is an example of the display screen when the display device displaying the display screen 71 is turned sideways. In the display screen 72, it is shown that the character area 72a corresponding to the character area 71a is displayed. Also, in the figure, the moving image corresponding to the moving image 71b is not displayed on the display screen 72 displayed when the display device is in the landscape orientation, and the moving image 73b corresponding to the moving image 71b is displayed on the display screen 72. Is displayed on the display screen 73 of the next page.

In the case of the display method shown in the figure, the page break position is not determined, and the page break position is dynamically determined according to the result of arranging characters, images, or moving images. When the display device is rotated, the display content after rotation is basically determined based on the first character of the displayed page. For this reason, when the display device is in landscape orientation, such as when the line breaks of characters continue as shown in FIG. 12, the moving image arranged at the bottom of the page may be arranged on the next page.

As a result, by rotating the display device, the reproduced moving image is not displayed, so that the user who has viewed the moving image is prevented from viewing, which may lead to great dissatisfaction.
Of course, the movie itself remains, and it is possible to play the movie if the next page is displayed, but the user's operation is required and the movie will be played again from the beginning. It will be burdensome for this.

FIG. 13 is a diagram for explaining a second comparative example in which two pages are displayed side by side when the display device is turned sideways. In the figure, an example of a display screen 81 when the display device is in landscape orientation is shown. In the display screen 81 shown in the figure, it is shown that a text area 81a is displayed on the left page. On the display screen 81, the page on the right side shows that a text area 81b and a moving image 81c are displayed.

Moreover, the display screen 82 in the figure is an example of the display screen when the display device displaying the display screen 81 is oriented vertically. In the display screen 82, it is shown that the text area 82a is displayed.
It is shown that the moving image corresponding to the moving image 81c is not displayed on the display screen 82, and the moving image 83b corresponding to the moving image 81c is displayed on the display screen 83 of the next page.

In the display method shown in FIG. 13, the layout of each page is fixed, and two pages are displayed side by side when in landscape orientation. In this case, when the young page is displayed out of the two displayed pages when rotated from landscape to portrait, the video displayed on the other page is not displayed. The same problem as that occurs.

The electronic book device 100 according to the present embodiment includes a display for displaying a book, and a touch panel for inputting a user operation is disposed on the display. The user can use the touch panel to give instructions such as page movement, display enlargement / reduction, and content selection. In addition, instead of the touch panel or in addition to the touch panel, a configuration including physical buttons for operation may be used.

The electronic book device 100 includes an acceleration sensor inside the device, and has a function of detecting the orientation of the device and outputting a screen according to the orientation of the device. That is, if the orientation of the electronic book apparatus 100 changes during display of the content, the electronic book apparatus 100 changes the orientation of the content according to the orientation.

In addition, the content in the present embodiment is not a book with only characters and images but a content in which at least one image is included in the content. Here, in the present embodiment, as an example, content including a moving image will be described. Here, the content including the moving image only needs to include the moving image in at least a part of the content, and the entire content may be the moving image.

In this embodiment, the moving image is displayed inline in the content as an example. Thereby, the user can see a character and an image | video simultaneously. As the contents, for example, contents such as pictorial books that explain not only characters but also images and sports instructional books are assumed.

Note that the electronic book device 100 may have a function of switching the display method so that the moving image can be viewed on the full screen, assuming that the moving image is mainly viewed. Further, the electronic book apparatus 100 does not need to actually include the moving image data in the book data, and may acquire it from the outside via the network at the timing of reproduction.

FIG. 1 is a schematic block diagram of an electronic book device 100 according to this embodiment. The electronic book device 100 includes a user input unit 101, an orientation detection unit 102, an imaging unit 103, a storage unit 104, a communication unit 105, a display unit 106, and a control unit 110.

The user input unit 101 receives a user input input by the user and outputs the received user input to the control unit 110. The user input includes operations for moving images such as moving image reproduction start, pause, and skip in addition to operations for books such as page movement and display enlargement or reduction. The user input unit 101 is an input device, for example, a touch panel.

The orientation detection unit 102 includes, for example, an acceleration sensor and acquires the orientation of the own device and detects that the orientation of the own device has changed. That is, the orientation detection unit 102 detects a change in the orientation of the device itself on a plane including the screen of the display unit 106.
The orientation detection unit 102 outputs the acquired orientation information indicating the orientation of the own device and the orientation change information indicating the detected change in the orientation of the own device to the control unit 110.
Under the control of the control unit 110, the imaging unit 103 captures an image of the user's face of the user apparatus while the moving image is being reproduced on the display unit 106, and outputs image data obtained by the imaging to the control unit 110. The imaging unit 103 is provided, for example, at a location on the front surface of the device itself (location on the surface where the display unit is provided).
The memory | storage part 104 hold | maintains the book data which shows the text contained in a book, and the moving image data contained in a book.

The communication unit 105 wirelessly communicates with other devices on the network via the wireless network interface included in the communication unit 105 and transmits / receives data under the control of the control unit 110. Here, assuming that the electronic book device is carried, wireless communication is used. The wireless communication in the communication unit 105 is, for example, mobile communication such as 3G (3rd Generation) and LTE (Long Term Evolution), or a wireless LAN.
Note that the communication in the communication unit 105 may be wired.

When the orientation of the electronic book device 100 changes, the control unit 110 determines the screen layout after the orientation change so that at least one moving image included in the content is displayed on the screen. More specifically, when the orientation detection unit 102 detects a change in orientation, the control unit 110 determines the screen layout after the orientation change so that at least one entire moving image included in the content is displayed on the screen. To do. Specific processing of the control unit 110 will be described later.
In addition, the control unit 110 performs the following processing. For example, when the orientation of the electronic book apparatus 100 changes, the control unit 110 sets the moving image size after the change in direction to be the same as the moving image size before the change in direction. Here, the moving image size is the size of the moving image displayed on the screen.
In addition, the control unit 110 performs processing for a book or processing for a moving image in accordance with a user input input from the user input unit 101. Here, the processing for the book is, for example, page movement, display enlargement or reduction, and the like. Moreover, the process with respect to a moving image is the reproduction start of a moving image, a pause, a skip, etc., for example.
In addition, when direction change information is input from the direction detection unit 102, the control unit 110 determines the text and video layout according to the direction of the own device input from the direction detection unit 102. Then, the control unit 110 reads the book data or moving image data from the storage unit 104 and causes the display unit 106 to display the read book data and moving image data with the determined text and moving image layout.
In addition, the control unit 110 causes the image capturing unit 103 to capture an image of the user of the own apparatus while a moving image is being reproduced on the display unit 106. Then, the control unit 110 determines whether or not the user is watching the moving image being played based on the captured image. The control unit 110 also controls the communication unit 105 to acquire document data or moving image data stored in another device on the network.

Note that when the orientation of the electronic book device 100 changes, the control unit 110 may change the size of the at least one image included in the content on the screen after the orientation change so as to fit on the screen. In addition, when the orientation of the electronic book device 100 changes, the control unit 110 may make the movie size after the orientation change the same as the movie size before the orientation change for at least one movie included in the content. .

The display unit 106 displays the moving image or book information input from the control unit 110. Here, the book information is a sentence included in the electronic book. That is, the display unit 106 displays content including moving images on the screen.

FIG. 2 is a schematic block diagram showing a logical configuration of the control unit 110 in the present embodiment. The control unit 110 includes a video playback unit 111, a layout unit 112, a page position holding unit 113, a video selection unit 114, a video display position calculation unit 115, a data management unit 116, a line-of-sight detection unit 120, and a video A viewing determination unit 130.

The moving image reproducing unit 111 decodes moving image data incorporated in the book and causes the display unit 106 to display the decoded data. The video playback unit 111 manages the playback state of the video (playing / paused / stopped), and processes such as playback start, pause, skip, etc. according to user input from the user input unit 101 I do.

Here, as the format of the video data, for example, the video codec is H.264. H.264, the container is in the mp4 format, but other formats such as MPEG (Moving Picture Experts Group) -2 and WMV (Windows (registered trademark) Media Video) may be used. Also, audio data may be included. There are two timings for starting playback of the video included in the content: automatically start playback when displayed, or start playback when the user touches the video area. Can be set. Depending on the capabilities of the device, a plurality of moving images may be decoded simultaneously and the decoded data may be displayed. In this case, the electronic book device 100 can also reproduce content in which a plurality of moving images are embedded in one page of the book.

The layout unit 112 acquires the book data via the data management unit 116, lays out the book according to the orientation of the device and the size of the display, and causes the display unit 106 to display the data after layout. The book data format is, for example, XMDF (ever-eXtending Mobile Document Format), but other formats such as EPUB (Electronic PUBlication) may be used. Furthermore, a format capable of expressing a document such as HTML (HyperText Markup Language) or XML (Extensible Markup Language) may be used.

The book display method is specified for each content. Here, a display method for dynamically generating a page as shown in FIG. 12 is assumed. In this case, the layout unit 112 does not determine the page break position, and dynamically determines the page break position according to the result of arranging characters, images, and moving images. When the user's own device is rotated by the user, the display content after the rotation is determined based on the first character of the displayed page. However, as shown in FIG. 12, since the displayed range is different between the portrait screen and the landscape screen, the page separation position also changes. Therefore, the character at the top of the next page differs between when the page is advanced vertically and when the page is advanced horizontally.
In addition, when the casing of the own apparatus as shown in FIG. 13 is in landscape orientation, the content is displayed on two pages, and when the casing of the own apparatus is in portrait orientation, the content is displayed on one page. Also good.

The layout unit 112 determines the page layout after rotation based on the relative position of the moving image area calculated by the moving image display position calculating unit 115 described later. Specifically, in order to make it appear as if the rotation is performed with reference to the position where the moving image area is displayed, the layout unit 112 arranges the layout so that the relative position of the moving image area is as close as possible to the relative position before the rotation. decide. That is, when the orientation detection unit 102 detects a change in orientation and the video viewing determination unit 130 determines that the video is being viewed, the layout unit 112 is based on the position on the screen on which the video is displayed. Thus, the screen layout corresponding to the direction after the change detected by the direction detection unit 102 is determined. Details of the processing of the layout unit 112 will be described later.

When the orientation detection unit 102 detects that the orientation of the device itself has changed, the page position holding unit 113 receives orientation change information from the orientation detection unit 102. When the page position holding unit 113 receives the direction change information, the page position holding unit 113 holds the page position and the direction change direction displayed before the rotation. That is, the page position holding unit 113 holds the displayed page position and direction change direction when the orientation detected by the orientation detection unit changes.
As a result, when the user returns the own apparatus to the original orientation before the rotation immediately after the own apparatus is rotated, the layout unit 112 holds the change in the direction detected by the orientation detection unit 102 in the page position holding unit 113. When the direction is the direction opposite to the change direction, the screen layout after the change is determined based on the page position held by the page position holding unit 113.

As an example, when the top of the page is a character, the page position holding unit 113 holds an offset value on the data of the top character of the page. On the other hand, as an example, when the top of the page is an element other than a character (image, video, etc.), the page position holding unit 113 holds an offset value on the data of the element. As a result, the page position holding unit 113 holds the offset value on the data of the element, so that the layout unit 11 can specify the element.
The layout unit 112 specifies the first character of the page or the first element of the page by using this offset value when returning to the state before the rotation after the rotation. A detailed description of the processing of the layout unit 112 will be described later.

The line-of-sight detection unit 120 detects the user's line-of-sight direction SL based on the image captured by the imaging unit 103 during video reproduction, and outputs the line-of-sight direction information indicating the detected line-of-sight direction SL to the video viewing determination unit 130. . Specific processing will be described later.

The moving image viewing determination unit 130 determines whether or not the user is viewing the moving image displayed on the display unit 106. More specifically, the moving image viewing determination unit 130 determines that the user who has detected the visual line direction SL based on the visual line direction SL indicated by the visual line direction information input from the visual line detection unit 120 when the moving image is being reproduced. It is determined whether or not the moving image is being viewed. That is, the moving image viewing determination unit 130 determines whether or not the user is viewing the moving image based on the reproduction state of the moving image and the visual line direction SL detected by the visual line detection unit 120.

At that time, the moving image viewing determination unit 130 views the moving image when, for example, the probability that the moving image has been viewed for a predetermined time (for example, the last few seconds) is lower than a predetermined threshold. Judge that there is no. On the other hand, the moving image viewing determination unit 130 determines that the moving image is being viewed when, for example, the probability that the moving image has been viewed for a predetermined time is equal to or greater than a predetermined threshold. Specific processing of the moving image viewing determination unit 130 will be described later.
The moving image viewing determination unit 130 outputs the determination result to the moving image selection unit 114.

Note that the video viewing determination unit 130 may determine whether or not the user is viewing the video based on the playback state of the video. That is, the video viewing determination unit 130 determines that the user is watching the video if the video is being played back, and determines that the user is not watching the video if the video is not being played (stopped, paused, etc.). To do. That is, the moving image viewing determination unit 130 may determine that the moving image is being watched at least when the moving image is being played back.

The video selection unit 114 is a case where the determination result input from the video viewing determination unit 130 indicates that a video is being viewed, and the video playback unit 111 is playing back a plurality of videos simultaneously. Choose which videos are most likely to be watched. Even if a plurality of moving images are arranged on the same page, if there is only one moving image being reproduced, the moving image selecting unit 114 selects the moving image being reproduced. Specifically, the moving image selection unit 114 selects one moving image from among a plurality of moving images included in the content based on the viewing index.

As an example, the moving image selection unit 114 selects a moving image based on the priority included in the content data. The content creator embeds priority information in which priority is specified for each video at the time of content data creation in the content data, and the video selection unit 114 selects the video with the highest priority indicated by the priority information. In other words, the viewing index is a priority for each moving image described in the document data as an example, and the moving image selection unit 114 selects one of the plurality of moving images included in the content based on the priority for each moving image. Video as the currently viewed video. Here, if there is a difference in meaning such as main content, supplemental content, and mere presentation for each video, the content creator sets the priority accordingly.

The moving image display position calculation unit 115 calculates the relative position of the moving image area on the screen. Specifically, when the character is written horizontally, the moving image display position calculation unit 115 acquires the vertical relative position of the moving image area. For example, the moving image display position calculation unit 115 calculates the relative position using the number of lines of characters before and after the moving image area. Specifically, for example, if the number of lines of characters before the moving image area is L1 and the number of lines of characters after the moving image area is L2, the moving image display position calculation unit 115 sets the relative position of the moving image area to L1. / (L1 + L2). Then, the moving image display position calculation unit 115 outputs the calculated relative position of the moving image area to the layout unit 112.

The data management unit 116 reads the data recorded in the storage unit 104 in response to a request from the layout unit 112 or the moving image reproduction unit 111, and outputs the read data to the requested layout unit or moving image reproduction unit 111. In addition, when data on the network is designated as a data reference destination, the data management unit 116 acquires the data via the communication unit 105.

Next, details of the processing of the moving image display position calculation unit 115 and the layout unit 112 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a page layout before being rotated horizontally and after being rotated horizontally. For example, in the vertically oriented screen 31 of FIG. 3, it is shown that there are seven lines before the moving image area and there are no characters after the moving area 22 (that is, zero lines). For this reason, the moving image display position calculation unit 115 obtains a value of 7 / (7 + 0) = 1 as the relative position in the vertical direction. This vertical relative position takes a value of 0 if the moving image area is arranged at the top of the screen, 1 if it is arranged at the bottom of the screen, and 0-1 depending on the position otherwise. Therefore, the moving image display position calculation unit 115 calculates the vertical relative position as the relative position of the moving image area on the screen.

In addition, in the figure, a first horizontal screen 23, a second horizontal screen 25, and a third horizontal screen 27 are shown as page layout candidates after being rotated horizontally.
In these three horizontally oriented screens, the moving image display position of the first horizontally oriented screen 23 is the upper part of the screen. In the third landscape screen 27, the moving image area 28 is at the bottom of the screen. In the second horizontal screen 25, the moving image area 26 is located between the sentences (that is, in the middle).

In this example, only one of the second horizontal screens 25 exists as a position that is neither the upper part nor the lower part. However, when there are more lines that can be displayed, a plurality of candidates can be considered for the intermediate position. As a result, there may be four or more position candidates. The layout unit 112 calculates the relative position of the moving image region for each candidate, and selects the one closest to the relative position before rotation.

For example, in FIG. 3, the moving image display position calculation unit 115 calculates the relative position of the moving image area as 0 / (0 + 2) = 0 on the first horizontally oriented screen 23. In addition, the moving image display position calculation unit 115 calculates the relative position of the moving image area as 1 / (1 + 1) = 0.5 on the second horizontally oriented screen 25. In addition, the moving image display position calculation unit 115 calculates the relative position of the moving image area as 2 / (2 + 0) = 1 on the third horizontal screen 27. The layout unit 112 adopts the third horizontal screen 27 closest to 1 which is the relative position before rotation as the layout after rotation.

<Details of processing of moving image selection unit 114>
Next, details of the processing of the moving image selection unit 114 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example in which priority is specified for moving image data. In the figure, a tag 41, a tag 42, and a tag 43 are shown. Here, moving image data is described by a <movie> tag, and each priority is specified by an attribute value called priority.
Specifically, the tag 41 has the moving image data “movie1.mp4” and the priority thereof is 1. The tag 42 has moving image data “movie2.mp4” and a priority of 10. The tag 43 has moving image data “movie3.mp4” and a priority of 50.

For example, if the priority is higher when the value of priority is lower, when these three videos are being played simultaneously, the video selection unit 114 determines that movie1. Select mp4. For example, movie2. mp4 and movie3. When the mp4 is played back at the same time, the moving image selection unit 114 determines that movie2. Select mp4.

Note that the moving image selection unit 114 may select one moving image from among a plurality of moving images that are simultaneously reproduced based on the user's line of sight. Specifically, for example, when the moving image viewing determination unit 130 determines the viewing state using the user's line of sight, the moving image selection unit 114 determines the probability of viewing the moving image, which is the immediately preceding information to be determined. You may select the animation with the highest probability by using it as it is. That is, the viewing index is the user's line of sight, and the moving image selection unit 114 is configured to display a plurality of moving images included in the content based on the user's line-of-sight direction detected by the line-of-sight detection unit (second line-of-sight detection unit) 120. One of the videos may be selected as the currently viewed video.

Also, the moving image selection unit 114 may select one moving image from a plurality of moving images that are simultaneously played based on the display size of the moving image. Specifically, for example, the video selection unit 114 has the highest display size from among a plurality of videos that are simultaneously played in view of the high possibility that a large video is being viewed. You may select a large video. That is, the viewing index is, for example, the size of the video that is displayed, and the video selection unit 114 is viewing one video from among the plurality of videos included in the content based on the size of the video that is displayed. May be selected as the video.

The process in which the moving image selection unit 114 selects a moving image based on the display size of the moving image will be described with reference to FIG. FIG. 5 is a diagram illustrating an example in which a display size is specified for moving image data. In the figure, a tag 51, a tag 52, and a tag 53 are shown. In the figure, the horizontal width of the display size is specified by the attribute value “width”, and the vertical width of the display size is specified by the attribute value “height”.

The size designation method includes a method of designating by the number of pixels of a pixel and a method of designating by a relative value with respect to the screen size. When the relative value is specified for the screen size, the moving image selection unit 114 calculates the display size that is actually displayed on the screen. When only one of the vertical width and the horizontal width is specified, the moving image selection unit 114 calculates the other value according to the aspect ratio of the moving image content. For example, if the number of pixels of the screen is 600 pixels wide and 1024 pixels long, and the aspect ratio of the moving image content is all 4: 3, movie1. mp4 is 300 × 225 pixels, movie3. mp4 is displayed with 273 × 204 pixels. Therefore, when these two moving images are being played back simultaneously, the moving image selection unit 114 assumes that the movie 1. Select mp4.

FIG. 6 is a schematic block diagram of the line-of-sight detection unit 120 in the present embodiment. The line-of-sight detection unit 120 includes a feature amount extraction unit 121, a face region specification unit 122, a parts specification unit 123, a black eye region specification unit 124, an eyeball center radius calculation unit 125, a black eye center position calculation unit 127, and a line of sight A direction calculation unit 128.

Based on the image data input from the imaging unit 103, the feature amount extraction unit 121 extracts a feature amount such as a color (skin color) or a boundary (contour) in the image indicated by the image data. Then, the feature amount extraction unit 121 outputs the extracted feature amount to the face area specifying unit 122.
The face area specifying unit 122 specifies the face area based on the feature amount input from the feature amount extracting unit 121, and sends information indicating the position of the face image area obtained by the specification to the part specifying unit 123. Output.

The parts specifying unit 123 specifies the positions of facial parts such as eyes, nose and mouth. The part specifying unit 123 outputs the face area position information R indicating the position of the face image area specifying the position of the face part and the eye position information P indicating the eye position to the video viewing determination unit 130. In addition, the part specifying unit 123 outputs eye position information indicating the position of the eyes to the black eye region specifying unit 124 and the eyeball center radius calculating unit 125.
As a result, even when there are a plurality of face area candidates specified by the face area specifying unit 122, the parts specifying unit 123 checks whether there are face parts such as eyes, nose, and mouth. It is possible to more accurately determine whether or not the face image area specified by the area specifying unit 122 corresponds to a face.

The black eye area specifying unit 124 specifies a black eye area among the eye areas based on the eye position information input from the parts specifying unit 123. The black eye area specifying unit 124 outputs black eye position information indicating the position of the specified black eye area to the black eye center position calculating unit 127.
The eyeball center radius calculation unit 125 extracts the positions of the eyes and the corners of the eyes based on the eye position information input from the part specifying unit 123, and based on the extracted positions of the eyes and the corners of the eyes, the center of the eyeball Calculate the position and radius of the eyeball. The eyeball center radius calculation unit 125 outputs information indicating the calculated center position of the eyeball and the radius of the eyeball to the gaze direction calculation unit 128.

The black eye center position calculating unit 127 specifies the center position of the black eye based on the black eye position information input from the black eye region specifying unit 124. Specifically, the black eye center position calculation unit 127 calculates, for example, the center of the ellipse as a black eye center position when the black eye is approximated as an ellipse from the position of the black eye region indicated by the black eye position information. Then, the black eye center position calculation unit 127 outputs black eye center position information indicating the specified black eye center position to the line-of-sight direction calculation unit 128.

The line-of-sight direction calculation unit 128 includes the eyeball center position and the eyeball radius indicated by the information input from the eyeball center radius calculation unit 125 and the black eye center position indicated by the black eye center position information input from the black eye center position calculation unit 127. Based on the above, the direction from the eyeball center to the black eye center is detected as the line-of-sight direction SL. The gaze direction calculation unit 128 outputs gaze direction information SL indicating the detected gaze direction to the moving image viewing determination unit 130.

FIG. 7 is a schematic block diagram of the moving image viewing determination unit 130 in the present embodiment. The video viewing determination unit 130 includes a face area size calculation unit 131, a distance calculation unit 132, a horizontal pixel interval calculation unit 133, an eye position calculation unit 134, a gaze determination unit 135, a probability calculation unit 136, A viewing determination unit 137.

The face area size calculation unit 131 calculates the size of the face image area based on the position of the face image area indicated by the face area position information R input from the line-of-sight detection unit 120. Specifically, for example, when the outline of the face image area is approximated as an ellipse, the face area size calculation unit 131 calculates the major axis of the face image area as the size of the face image area. Then, the face area size calculation unit 131 outputs face size information indicating the calculated size of the face image area to the distance calculation unit 132.
Note that the face area size calculation unit 131 may calculate the minor axis of the face image area as the size of the face image area.

The distance calculation unit 132 calculates the distance d from the lens included in the imaging unit 103 to the face, assuming that the actual face size of the user is the average face size. Here, the distance calculation unit 132 holds the focal length f of the imaging unit 103 and the average face major axis (actual major axis) y. Based on the focal length f of the imaging unit 103, the average face major axis (actual major axis) L, and the face region major axis (major axis on the image) l indicated by the face size information, from the lens included in the imaging unit 103 The distance d (= L × f / l) to the face is calculated. Then, the distance calculation unit 132 outputs distance information indicating the calculated distance d from the lens included in the imaging unit 103 to the face to the eye position calculation unit 134.

The distance calculation unit 132 captures images based on the focal length f of the imaging unit 103, the average minor axis of the face (actual minor axis), and the calculated minor axis of the face region (minor axis on the image). The distance d from the lens included in the unit 103 to the face may be calculated.

The horizontal pixel interval calculation unit 133 is a horizontal line between the eye position on the image (for example, the center position of the black eye on the image) indicated by the eye position information P input from the line-of-sight detection unit 120 and the center position of the image. The pixel interval a in the direction is calculated. The horizontal pixel interval calculation unit 133 outputs horizontal pixel interval information indicating the calculated horizontal pixel interval a to the eye position calculation unit 134.

The eye position calculation unit 134 holds information indicating the focal length f of the imaging unit 103 in advance. The eye position calculation unit 134 includes the horizontal pixel interval a indicated by the horizontal pixel interval information input from the horizontal pixel interval calculation unit 133, the focal length f of the imaging unit 103, and the distance information input from the distance calculation unit 132. Based on the indicated distance d, the eye position with the center position of the lens as the origin is calculated. Then, the eye position calculation unit 134 outputs eye position information b indicating the calculated eye position to the line-of-sight determination unit 135.

Based on the eye position indicated by the eye position information b input from the eye position calculation unit 134 and the line-of-sight direction indicated by the line-of-sight direction information SL input from the line-of-sight detection unit 120, the line-of-sight determination unit 135 It is determined whether or not it is within the display area. The line-of-sight determination unit 135 outputs the determination result to the probability calculation unit 136 at a predetermined interval (for example, an interval of 0.1 seconds).

The probability calculation unit 136 calculates the probability that the line of sight is within the display area of the moving image. For example, when the line-of-sight determination unit 135 determines whether or not the line of sight is within the moving image display area at 0.1 second intervals (10 fps) for 5 seconds (that is, 50 times), the line of sight moves to the moving image 35 times. The probability calculation unit 136 calculates the probability of watching a moving image as 70%. The probability calculation unit 136 outputs probability information indicating the calculated probability to the viewing determination unit 137.

If the line of sight is directed to the center of the moving image display area, the probability calculating unit 136 may weight the probability so that the probability of watching the moving image is high.
Further, the line-of-sight determination unit 135 determines whether or not the line of sight is within the moving image display area, but is not limited thereto, and determines whether or not the line of sight is within the display area of the display unit 106. Also good. In that case, the probability calculation unit 136 may calculate the probability that the line of sight is within the display area of the moving image.

The viewing determination unit 137 determines whether or not the user is watching a video based on the probability indicated by the probability information input from the probability calculation unit 136. Specifically, the viewing determination unit 137 determines that the moving image is not viewed when the probability is lower than a predetermined threshold. On the other hand, the viewing determination unit 137 determines that the video is being viewed when the probability is equal to or greater than a predetermined threshold. The viewing determination unit 137 outputs the determination result obtained by the determination to the moving image selection unit 114.

Next, details of the processing of the line-of-sight determination unit 135 will be described with reference to FIG.
FIG. 8 is a diagram for explaining the eye position and the line-of-sight direction with the center position of the lens as the origin. The electronic book apparatus 100 including the lens 61 and the display unit 106 included in the imaging unit 103 and the user's left eye 62 are illustrated in FIG. Here, the distance between the display area of the display unit 106 and the user's left eye 62 is d1. In the xyz coordinate system, the position of the user's left eye is indicated by coordinates (x1, y1, d1). As shown in the figure, as an example, the lens 61 is disposed on the surface on which the display unit 106 is located. In the example shown in the figure, the positions of the lens 61 and the imaging unit 103 are merely examples, and may be arranged at positions where the user's eyes can be imaged.

In the example of the figure, the line-of-sight determination unit 135 moves from, for example, the position of the contour in the moving image display unit 106 displayed on the display unit 106 to a position in the xyz coordinate system (hereinafter referred to as a moving image contour absolute position). Convert coordinates. Then, the line-of-sight determination unit 135 calculates the intersection of a straight line from the position (x1, y1, d1) of the user's left eye to the line-of-sight direction SL with a plane including the display surface of the display unit 106, that is, a plane with z = 0. .

Then, the line-of-sight determination unit 135 determines whether the intersection is within the moving image display area based on the calculated intersection and the moving image absolute position. When the calculated intersection is within the moving image display area, the line-of-sight determination unit 135 determines that the line of sight is within the moving image display area. On the other hand, if the intersection is outside the moving image display area, the line-of-sight determination unit 135 determines that the line of sight is not within the moving image display area.

FIG. 9 is a flowchart illustrating an example of processing of a basic part of the electronic book device 100 of the present embodiment. First, the user input unit 101 receives a user input indicating a book selected by the user, and outputs book information indicating the book to the layout unit 112 (step S101). The layout unit 112 acquires the book data stored in the storage unit 104 via the data management unit 116 for the selected book (step S102).

Next, the layout unit 112 determines the layout of the document according to the orientation of the own device and the size of the display included in the display unit 106, and displays the book data on the display unit 106 with the determined layout (step S103). .
Next, when a page including moving image data is displayed, for example, when the user touches a position corresponding to the moving image area displayed on the display unit 106 on the touch panel, the user input unit 101 displays the moving image data. The user input indicating the start of playback is accepted (step S104). Then, the user input unit 101 outputs the user input to the moving image playback unit 111.

Next, the moving image reproduction unit 111 acquires a moving image before decoding from the storage unit 104, and decodes the acquired moving image. Then, the moving image reproducing unit 111 causes the display unit 106 to reproduce the decoded moving image (step S105).
Next, when the user performs a page turning operation, the user input unit 101 receives a user input indicating page turning and outputs the received user input to the layout unit 112 (step S106).

Next, the layout unit 112 acquires the next page data from the storage unit 104 via the data management unit 116 (step S107). Then, the layout unit 112 determines the layout in the same manner as described above, and causes the display unit 106 to display the next page data with the determined layout (step S108). At that time, the display unit 106 performs screen transition by displaying, for example, an animation of turning a page. Above, the process of this flowchart is complete | finished.

FIG. 10 is a flowchart illustrating an example of processing when the orientation of the device itself changes in the electronic book device 100 of the present embodiment. When the orientation detection unit 102 detects that the orientation of the own device has changed, the following processing is executed.
First, the moving image viewing determination unit 130 determines whether or not the user is viewing a moving image (S201). If the currently displayed page includes a plurality of moving images, the moving image viewing determination unit 130 determines whether or not each of the moving images is being viewed individually. When there is one or more videos determined to be viewed and the probability that the user's line of sight is within the video display area is equal to or greater than a predetermined threshold, the video viewing determination unit 130 allows the user to view the videos. It is determined that it is in the middle.

Next, when the moving image viewing determination unit 130 determines that the moving image is being viewed (step S201: YES), the rotation is performed with reference to the position where the moving image is displayed on the screen so that the moving image is displayed even after the rotation. I do. First, considering the case where there are a plurality of moving images being viewed, the moving image selection unit 114 selects one moving image with priority (step S202). Even when there are a plurality of moving images being viewed, one moving image is selected in this process, and thereafter, the same processing as in the case where there is only one moving image is performed.

Subsequently, the relative position of the moving image area on the screen is acquired for the moving image selected by the layout unit 112 (step S203).
Next, the layout unit 112 determines the layout of the rotated page based on the acquired relative position of the moving image area (step S204). Specifically, the layout unit 112 determines the layout so that the position of the moving image area is as close as possible to the position before the rotation in order to make it appear as if it has been rotated based on the position where the moving image area is displayed. .

Since the display content after the rotation is determined by the above processing, the page position holding unit 113 stores the position of the page before the rotation (step S205). For example, in FIG. 2, the page position holding unit 113 stores information for specifying “K” that is the first character of the page before rotation. This information is used when returning to the state before the rotation after the direction is changed, and specific processing will be described later.

The layout unit 112 displays the rotated page on the screen based on the determined layout (step S207). Thus, if a moving image is being reproduced, the moving image reproducing unit 111 continuously reproduces the moving image, so that the user can seamlessly view the moving image.

On the other hand, in step S201, when the video viewing determination unit 130 determines that the video is not being viewed (step S201: NO), the rotation processing may be performed as a normal book, and therefore the rotation is based on the first character of the page. Do. Accordingly, the position of the page after rotation is set as the same position as the current position (step S206). That is, even in the rotated page, the page layout is such that the first character of the current page is “K”, and the layout unit 112 displays the rotated page on the screen (step S207). In this case, for example, the landscape display screen 72 shown in FIG. 12 is displayed. Above, the process of this flowchart is complete | finished.

Through the above processing, the electronic book device 100 determines the layout so that the relative position of the moving image area after rotation is as close as possible to the relative position before rotation. As a result, the electronic book device 100 can maintain the relative position of the moving image area even when the device itself is rotated while watching the moving image, and thus can continuously display the moving image. As a result, even if the electronic book apparatus 100 is rotated, a moving image can be viewed seamlessly, and user convenience can be improved. Therefore, the user can comfortably view the electronic book in which the moving image is incorporated.

In addition, since the electronic book device 100 holds the relative positions before and after the rotation of the position on the screen on which the moving image is displayed, the user can easily associate the moving image before and after the rotation. Can be done. As a result, the user can comfortably view the electronic book in which the moving image is incorporated.

Subsequently, when the electronic book apparatus 100 is rotated when the moving image is being viewed, the electronic book apparatus 100 rotates the display screen based on the position where the moving image is displayed. change. Therefore, immediately after the electronic book apparatus 100 is rotated, when the electronic book apparatus 100 is returned to the direction before the rotation, the position may be shifted from the original page. Therefore, processing of the electronic book device 100 when the orientation of the device itself is restored immediately after the orientation of the device is changed will be described with reference to FIG.

FIG. 11 is a flowchart illustrating an example of a processing flow of the electronic book device 100 when the orientation of the device itself is restored immediately after the orientation of the device itself is changed. The following processing is executed when the orientation of the terminal is returned immediately after execution of the rotation processing shown in FIG. For example, the following processing is executed when the image is rotated from the vertical orientation (0 °) to the horizontal orientation (90 °) and then returned from the horizontal orientation (90 °) to the vertical orientation (0 °).

First, the layout unit 112 determines whether or not the page position before rotation is stored in the page position holding unit 113 (step S301). When the page position before the rotation is stored in the page position holding unit 113 (step S301: YES), it means that the page position has moved during the previous rotation, so the layout unit 112 stores the page position in the page position holding unit 113. The stored page position is acquired (step S302).

Next, the layout unit 112 sets this page position as the position of the page after rotation (step S303). That is, the layout unit 112 performs a page layout with a designated element (character, image, moving image, etc.) as the head as a rotated page. The page generated as a result is the same page as before the rotation. The layout unit 112 displays the rotated page on the display unit 106 (step S304). Thereby, the layout part 112 can return to the same display state as before rotation. Next, since the page position stored in the page position holding unit 113 becomes unnecessary, the page position holding unit 113 deletes the information (step S305).

In step S301, if the page start position is not stored in the page position holding unit 113 (step S301: NO), it means that the page position is not moved during the previous rotation. Normal rotation processing is performed. Specifically, similarly to the processing shown in FIG. 10, the layout unit 112 sets the page position after rotation to the same position as the current position (step S306). The layout unit 112 displays the rotated page on the display unit 106 in this layout (step S307). Thereby, the layout unit 112 can display a page rotated with reference to the first character of the page.

As described above, the electronic book device 100 executes the process of FIG. 11 only when the orientation of the terminal is returned to the original orientation immediately after the rotation of the device itself. On the other hand, when the device itself is rotated once and then the page is moved and then returned to the original orientation, the electronic book device 100 executes the rotation processing (FIG. 10) without executing the processing of FIG. To do.

In addition, when the device is rotated from the portrait orientation (0 °) to the landscape orientation (90 °) and then changed from the landscape orientation (90 °) to the opposite portrait orientation (180 °), the electronic book device 100 can be The processing of FIG. 10 is executed without executing the processing. That is, the electronic book apparatus 100 basically executes the process of FIG. 10 when the own apparatus is rotated, but executes the special process shown in FIG. 11 only when the direction of the own apparatus is returned. To do.

Thereby, the electronic book device 100 can give the meaning of returning to the original state (before rotation) to the operation of returning the orientation of the device itself. Therefore, even when there are a plurality of moving images being played back, even if the moving image preferentially displayed by the device is different from the user's intention, the user can simply change the orientation of the terminal to restore the state. Since it is guaranteed to return to the original, the user can use the electronic book apparatus 100 without a sense of incongruity.

In addition, in this embodiment, although the electronic book apparatus 100 was demonstrated as an example, it is not restricted to this, What is necessary is just a display apparatus provided with display screens, such as a mobile telephone apparatus, a smart phone, an information terminal device, and a tablet PC.

Further, by recording a program for executing each process of the electronic book apparatus 100 of the present embodiment on a computer-readable recording medium, and causing the computer system to read and execute the program recorded on the recording medium. The above-described various processes related to the electronic book device 100 may be performed.

Note that the “computer system” referred to here may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” refers to a volatile memory (for example, DRAM (Dynamic) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc. that hold a program for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design and the like within a scope not departing from the gist of the present invention.

The present invention can be applied to terminals such as mobile phones, smartphones, and tablets that can display electronic books.

DESCRIPTION OF SYMBOLS 100 Electronic book apparatus 101 User input part 102 Direction detection part 103 Image pick-up part 104 Storage part 105 Communication part 110 Control part 111 Movie reproduction part 112 Layout part 113 Page position holding part 114 Movie selection part 115 Movie display position calculation part 116 Data management part DESCRIPTION OF SYMBOLS 120 Eye-gaze detection part 121 Feature-value extraction part 122 Face area specific | specification part 123 Parts specific part 124 Black eye area | region specific part 125 Eyeball center radius calculation part 126 Black eye center position calculation part 128 Gaze direction calculation part 130 Movie viewing determination part 131 Face area size Calculation unit 132 Distance calculation unit 133 Horizontal pixel interval calculation unit 134 Eye position calculation unit 135 Gaze determination unit 136 Probability calculation unit 137 Viewing determination unit

Claims

A display device including a control unit that determines a screen layout after the orientation change so that at least one moving image included in the content is displayed on the screen when the orientation of the device changes.
A display unit for displaying content including a video on the screen;
An orientation detection unit for detecting a change in orientation of the device itself on a plane including the screen;
With
2. The control unit according to claim 1, wherein, when the orientation detection unit detects a change in orientation, the control unit determines a screen layout after the orientation change so that at least one moving image included in the content is displayed on the screen. The display device described.
The controller is
A video viewing determination unit that determines whether the user is viewing the video displayed on the display unit;
When the orientation detection unit detects a change in orientation and the video viewing determination unit determines that the video is being viewed, the screen after the orientation change is based on the position on the screen on which the video is displayed. A layout part for determining the layout;
A display device according to claim 2.
The control unit includes a page position holding unit that holds the displayed page position and the change direction of the orientation when the orientation detected by the orientation detection unit is changed,
The layout unit detects a page held by the page position holding unit when the change in direction detected by the direction detection unit is opposite to the direction change of the direction held by the page position holding unit. The display device according to claim 3, wherein the screen layout after the change is determined based on the position.
The display device according to claim 3 or 4, wherein the moving image viewing determination unit determines that the moving image is being watched at least when the moving image is being played back.
The control unit includes a gaze detection unit that detects a gaze direction of the user,
The display device according to claim 5, wherein the moving image viewing determination unit determines whether or not the user is viewing the moving image based on a reproduction state of the moving image and a line-of-sight direction detected by the line-of-sight detection unit.
7. The control unit according to claim 1, further comprising: a video selection unit that selects one video as a currently viewed video from among a plurality of videos included in the content based on a viewing index. The display device described in 1.
The viewing index is a priority for each video included in the content,
The display device according to claim 7, wherein the moving image selection unit selects one moving image from among a plurality of moving images included in the content based on a priority for each moving image.
The viewing index is a display size of the video,
The display device according to claim 7, wherein the moving image selection unit selects one moving image as a currently viewed moving image from a plurality of moving images included in the content based on the display size of the moving image.
The viewing index is a user's line of sight,
The control unit includes a second gaze detection unit that detects a gaze direction of the user,
The display device according to claim 7, wherein the moving image selection unit selects one moving image from among a plurality of moving images included in the content based on the line-of-sight direction detected by the line-of-sight detection unit.
A display method executed by a display device,
A display method comprising a step of determining a screen layout after the orientation change so that at least one moving image included in the content is displayed on the screen when the orientation of the device changes.
In the computer of the display device,
A display program for executing a step of determining a screen layout after a change of orientation so that at least one moving image included in the content is displayed on the screen when the orientation of the own device changes.