JP5849815B2 - Content display device - Google Patents

Description

  The present invention relates to a content display device, and more particularly, to a content display device having a function of displaying content of a content desired by a user on a display screen and displaying auxiliary information related to the content.

  Over the past few years, electronic devices such as smartphones and tablet-type electronic terminals have become more widespread as they surpassed the number of PCs shipped, and devices with the function of presenting digital content have rapidly penetrated the real world. In particular, in the publishing industry, the importance of the business of providing content in the form of electronic books has increased, and users can obtain many books in the form of electronic books. In the future, it is expected that browsing electronic books on electronic terminals will be generalized in the same way as browsing Web pages on the Internet.

  If a digital content such as an electronic book or a Web page is displayed on the display screen of the electronic device and read by the user, an additional service using functions unique to the electronic device can be provided. For example, an electronic device incorporating a dedicated browsing application program can additionally present auxiliary information related to content being displayed as a browsing target.

  As an example of a technique for presenting such auxiliary information, the following Patent Document 1 includes a part of content related to the content displayed on the display unit and information such as the author as auxiliary information. A technique for displaying is disclosed, and Patent Document 2 discloses a technique for inserting information related to the contents of the electronic book as additional information into a blank area generated when the electronic book is displayed on the screen. It is disclosed. In addition, when the user designates a specific word in the displayed character string, auxiliary technology related to the word (for example, content obtained by drawing the word with a dictionary) is displayed on the screen.

  In general, various types of methods for retrieving information related to a character string from a reference database have been known for a long time. For example, if a keyword is added to each piece of information in the reference database, information with a keyword that matches the target character string can be extracted as a search result. If such a search method is used, information related to a specific character string in the content being displayed can be searched from the reference database, and the searched information can be presented as auxiliary information.

  On the other hand, recently, techniques for searching for images have also been proposed. For example, Patent Document 3 below discloses a technique for dividing an keyword into a noun and a sensitivity word, and performing an image search by associating the combination of the sensitivity word and the noun with a feature amount indicating the feature of the image. In Patent Document 4, in order to predict a keyword to be assigned to an arbitrary image from the image feature amount, a database storing a correspondence relationship between the image feature amount and the keyword predicted value is created, and the database is used. A method for performing an image search is disclosed. Further, in Patent Document 5, the input language sentence is analyzed by a language analysis means, a search is performed, and it is identified whether the searched target data represents a background or a part, and is divided into a background and a part. A technique for searching for an image is disclosed, and Patent Document 6 discloses a technique for searching for a graphic corresponding to an ambiguous word input by a user among electronic documents composed of text and graphics. Yes.

JP 2010-198084 A JP 2011-123786 A JP 2011-065499 A Japanese Patent No. 3996723 JP-A-6-119405 Japanese Patent Laid-Open No. 7-85092

  Many digital contents include images as well as character strings. Therefore, in practice, it is preferable to present auxiliary information not only for character strings in content but also for images. However, while the technology for searching for information related to a character string has been established for a long time, the technology for searching for information related to an image as a target is shallow. In this method, it is difficult to select and present appropriate auxiliary information related to the image.

  Of course, as disclosed in the above-mentioned publications, attempts have been made to quantitatively capture the characteristics of the image itself. However, many of these techniques analyze the pattern on the image and Therefore, it is necessary to perform a process of extracting various feature amounts from the image and to grasp the characteristics of the patterns constituting the image. Such a process is a relatively heavy process, and is not suitable for use in an electronic device having a relatively low processing capability such as a tablet electronic terminal, and can be applied to a commercial content display device for general users. Not efficient.

  Accordingly, an object of the present invention is to provide a content display device that can efficiently select and present more appropriate auxiliary information related to an image included in the content being displayed.

(1) In the first aspect of the present invention, the contents of the main content including the image are displayed, auxiliary information related to the image in the main content being displayed is searched from the sub content, and the searched auxiliary information is displayed. In a content display device having a function to present,
A main content data storage unit for storing main content data including image data to which a search character string is attached;
A sub content data storage unit for storing a plurality of sub content data belonging to different categories,
An instruction input unit for inputting user instructions;
A display that displays the content, and
A main content display processing unit that displays a predetermined portion of the main content data stored in the main content data storage unit as main content on a display screen based on a user instruction;
Based on a user instruction, a search process for searching auxiliary information related to a specific target image in the main content displayed on the display screen from a plurality of sub-content data is executed and obtained as a result of the search process. A subsidiary content presentation processing unit for presenting the auxiliary information as subsidiary content;
Provided,
In the secondary content presentation processing unit,
A search character string recognition unit for recognizing a search character string attached to the target image;
A search processing unit that executes a search process for a plurality of sub-content data using the recognized search character string as a search key, and extracts auxiliary information that has been hit;
A relevance information storage unit that stores relevance information indicating the relevance between individual categories and colors;
A color recognition unit for recognizing the color of the target image;
A priority assigning unit that refers to the relevance information based on the color recognized by the color recognizing unit, and gives higher priority to the auxiliary information extracted from the sub-content data belonging to the more relevant category;
An auxiliary information presentation processing unit for presenting the extracted auxiliary information based on the priority order according to the given priority;
Is provided.

(2) According to a second aspect of the present invention, in the content display device according to the first aspect described above,
The relevance information storage unit stores relevance information that associates the distribution range of any one of hue value, saturation value, and lightness value or the combination range of these values for each category. And
The color recognition unit recognizes the representative color for the target image as one representative value of a hue value, a saturation value, and a lightness value or a combination thereof,
The priority assigning unit has a hue range, saturation value, brightness value distribution range associated with each category, or a combination range of these values, and a hue value, saturation value, A representative value of lightness values or a combination thereof is compared, and a priority according to the degree of approximation of both is given.

(3) According to a third aspect of the present invention, in the content display device according to the second aspect described above,
The relevance information storage unit includes relevance information that associates the distribution range of combinations of saturation value, lightness value, and hue value for each category,
The color recognition unit recognizes the representative value of saturation, the representative value of brightness, and the representative value of hue for the target image,
The priority assigning unit has a distribution range of combinations of saturation values, lightness values, and hue values associated with individual categories, and the representative value of saturation, the representative value of lightness, and the representative of hue recognized by the color recognition unit The combination of values is compared, and a priority is given according to the degree of approximation between the two.

(4) According to a fourth aspect of the present invention, in the content display device according to the third aspect described above,
Closed area corresponding to each category in a three-dimensional color space where the relevance information storage unit has a lightness value for the position on the reference coordinate axis, a saturation value for the distance from the reference coordinate axis, and a hue value for the angle around the reference coordinate axis The mapping information that maps is stored as relevance information,
The priority assigning unit plots the color recognized by the color recognition unit as a recognition point in the three-dimensional color space, with the representative value of saturation, the representative value of lightness, and the representative value of hue as coordinate values. The priority is given based on the positional relationship between the recognized recognition points and the mapped individual closed regions.

(5) According to a fifth aspect of the present invention, in the content display device according to the second aspect described above,
The relevance information storage unit includes relevance information that associates distribution ranges of combinations of saturation values and lightness values for each category,
The color recognition unit recognizes a representative value of saturation and a representative value of brightness for the target image,
The priority assigning unit determines the distribution range of the combination of the saturation value and the lightness value associated with each category, and the combination of the representative value of the saturation and the representative value of the lightness recognized by the color recognition unit. In comparison, priority is given according to the degree of approximation between the two.

(6) According to a sixth aspect of the present invention, in the content display device according to the fifth aspect described above,
Relevance information storage unit uses mapping information obtained by mapping closed areas corresponding to individual categories on a two-dimensional color plane having a saturation value on the first coordinate axis and a lightness value on the second coordinate axis. Store and
The priority assigning unit plots the color recognized by the color recognition unit on the two-dimensional color plane as a recognition point with the representative value of saturation and the representative value of lightness as coordinate values, and the plotted recognition points The priority is given based on the positional relationship with each of the mapped closed regions.

(7) According to a seventh aspect of the present invention, in the content display device according to the second to sixth aspects described above,
The color recognition unit recognizes the representative color of the target image as a weighted average value considering the area of the hue value, saturation value, or lightness value of each part constituting the target image, or a combination thereof. It is.

(8) According to an eighth aspect of the present invention, in the content display device according to the first aspect described above,
The secondary content presentation processing unit further includes a complexity recognition unit that recognizes the complexity of the target image,
The relevance information storage unit stores relevance information indicating the relevance between individual categories and color as well as the relevance between individual categories and colors.
Sub-content data belonging to a category with higher relevance by the relevance information referring to the relevance information based on both the color recognized by the color recognizing unit and the complexity recognized by the complexity recognizing unit A higher priority is given to the auxiliary information extracted from.

(9) According to a ninth aspect of the present invention, in the content display device according to the eighth aspect described above,
The relevance information storage unit stores relevance information that associates the distribution range of the combination of saturation value, lightness value, and complexity for each category,
The color recognition unit recognizes a representative value of saturation and a representative value of brightness for the target image,
The priority assigning unit has a distribution range of combinations of saturation value, lightness value, and complexity associated with each category, a representative value of saturation recognized by the color recognition unit, and a lightness value recognized by the color recognition unit. The combination of the representative value and the complexity recognized by the complexity recognition unit is compared, and a priority according to the degree of approximation of both is given.

(10) According to a tenth aspect of the present invention, in the content display device according to the ninth aspect described above,
The relevance information storage unit maps closed regions corresponding to individual categories on a three-dimensional space having a saturation value on the first coordinate axis, a lightness value on the second coordinate axis, and a complexity on the third coordinate axis. Store mapping information as relevance information,
The priority assigning unit uses the color recognized by the color recognizing unit and the complexity recognized by the complexity recognizing unit as coordinate values for the representative value of saturation, the representative value of lightness, and the complexity on a three-dimensional space. Plotting is performed as recognition points, and priority is given based on the positional relationship between the plotted recognition points and the mapped closed regions.

(11) An eleventh aspect of the present invention is the content display device according to the fourth, sixth, and tenth aspects described above,
The priority assigning unit determines whether or not the closed region corresponding to each category includes a recognition point, and the auxiliary information extracted from the sub-content data belonging to the category corresponding to the closed region including the recognition point , Higher priority is given.

(12) According to a twelfth aspect of the present invention, in the content display device according to the fourth, sixth, tenth, and eleventh aspects described above,
A sub-content belonging to a category corresponding to a closed region with a small mutual distance by the priority assigning unit calculating a Euclidean distance between a predetermined reference point defined for the closed region corresponding to each category and a recognition point. A higher priority is given to the auxiliary information extracted from the data.

(13) According to a thirteenth aspect of the present invention, in the content display device according to the twelfth aspect described above,
The priority assigning unit defines the center of gravity of each closed region as a reference point.

(14) According to a fourteenth aspect of the present invention, in the content display device according to the first aspect described above,
The secondary content presentation processing unit further includes a complexity recognition unit that recognizes the complexity of the target image,
Each sub-content data stored in the sub-content data storage unit is set with a predetermined complexity,
The priority assigning unit gives higher priority to the auxiliary information extracted from the sub-content data in which the complexity approximated by the complexity recognized by the complexity recognizing unit is set.

(15) According to a fifteenth aspect of the present invention, in the content display device according to the eighth to tenth and fourteenth aspects described above,
The complexity recognition unit has a function of detecting the spatial frequency of the target image, and recognizes a value corresponding to the detected spatial frequency value as the complexity.

(16) According to a sixteenth aspect of the present invention, in the content display device according to the eighth to tenth and fourteenth aspects described above,
The complexity recognition unit has the function of counting points defined in advance in association with individual geometric figures for the target image composed of a collection of geometric figures. It is intended to be recognized.

(17) According to a seventeenth aspect of the present invention, in the content display device according to the eighth to tenth and fourteenth to sixteenth aspects described above,
The instruction input unit has a function of inputting a user instruction to set a desired difficulty level for the auxiliary information to be presented,
As the priority assignment unit indicates difficulty, the auxiliary information extracted from the sub-content data belonging to the category with higher complexity or the sub-content data set with higher complexity is more A high priority is given.

(18) According to an eighteenth aspect of the present invention, in the content display device according to the first to seventeenth aspects described above,
The search processing unit executes a search process for extracting auxiliary information whose degree of match with the search key is equal to or higher than a predetermined reference as hit auxiliary information,
The priority assigning unit gives higher priority to auxiliary information having a high degree of match.

(19) According to a nineteenth aspect of the present invention, in the content display device according to the first to eighteenth aspects described above,
The auxiliary information presentation processing unit presents only auxiliary information within the upper nth (n is a predetermined natural number) based on the priority among the auxiliary information extracted by the search processing unit. is there.

(20) According to a twentieth aspect of the present invention, in the content display device according to the nineteenth aspect described above,
The search processing unit executes the search process for the sub-content data in the order from the highly relevant category to the less relevant category, and stops the subsequent search process when the top nth auxiliary information is extracted. It is a thing.

(21) According to a twenty-first aspect of the present invention, in the content display device according to the first to twentieth aspects described above,
When there are a plurality of pieces of auxiliary information to be presented, the auxiliary information presentation processing unit presents the plurality of pieces of auxiliary information according to the arrangement order or the time order based on the priority order.

(22) According to a twenty-second aspect of the present invention, in the content display device according to the first to twenty-first aspects described above,
The instruction input unit has a function of inputting a user instruction to specify an instruction point on the display screen,
The sub-content presentation processing unit recognizes an image displayed near the designated point as a target image and presents auxiliary information related to the target image.

  (23) In a twenty-third aspect of the present invention, the content display device according to the first to twenty-second aspects described above is configured by incorporating a program into a computer.

(24) According to a twenty-fourth aspect of the present invention, the contents of the main content including the image are displayed, auxiliary information related to the image in the main content being displayed is searched from the sub-content, and the searched auxiliary information is displayed. In the content display method to be presented,
A main content display stage in which a computer displays a predetermined portion of the main content on a display screen based on a user instruction;
Target recognition for recognizing a specific image in the main content displayed on the display screen as a target image and a search character string attached to the target image based on a user instruction Stages,
A search processing stage in which the computer executes a search process for a plurality of sub-content data belonging to different categories, using the recognized search character string as a search key, and extracts the hit auxiliary information;
A color recognition stage in which the computer recognizes the color of the target image;
The computer refers to the relevance information that indicates the relevance between individual categories and colors, and the auxiliary information extracted from the sub-content data belonging to the more relevant category has a higher priority. A priority granting stage for granting
An auxiliary information presentation stage in which the computer presents the extracted auxiliary information based on the priority order according to the given priority;
Is to do.

(25) According to a twenty-fifth aspect of the present invention, in the content display method according to the twenty-fourth aspect described above,
The computer further comprises a complexity recognition stage for recognizing the complexity of the target image;
In the prioritization stage, the computer looks at the relevance information that shows the association between each predefined category and color, and the association between each category and complexity, and makes the category more relevant A higher priority is given to the auxiliary information extracted from the sub-content data to which it belongs.

(26) According to a twenty-sixth aspect of the present invention, in the content display method according to the twenty-fourth aspect described above,
The computer further comprises a complexity recognition stage for recognizing the complexity of the target image;
In the priority assignment stage, the computer refers to the complexity of each sub-content data set in advance, and is extracted from the sub-content data in which the complexity approximated by the complexity recognized in the complexity recognition stage is set A higher priority is given to the auxiliary information.

  In the content display device according to the present invention, sub-content data including auxiliary information is prepared for each category, and further, relevance information indicating the relevance between individual categories and colors is prepared. Then, based on the color recognized for the target image included in the main content being displayed, the relevance of each category is evaluated, and the auxiliary information extracted from the sub content data belonging to the more relevant category, A higher priority is given, and auxiliary information is presented based on a priority order corresponding to the priority. For this reason, it is possible to efficiently select and present more appropriate auxiliary information related to the image included in the displayed content.

  In the present invention, it is only necessary to perform a task of attaching a search character string in advance to the target image, and it is not necessary to extract various feature amounts of the image by complicated image analysis processing. Further, when searching for auxiliary information, the color of the target image is recognized, and the recognized color is compared with the color associated with the category, so the work load during the search process is reduced. Therefore, appropriate auxiliary information can be efficiently presented with a relatively simple work load and processing load.

  In the embodiment in which information indicating the relationship with the complexity is prepared, more appropriate auxiliary information can be selected and presented by considering the complexity recognized for the target image.

It is a block diagram which shows the structure of the content display apparatus 100 which concerns on the 1st Embodiment of this invention. It is a top view which shows an example of the display form of the content display apparatus 100 shown in FIG. It is a block diagram which shows the specific structural example of the main content data relevant to the content currently displayed on the main content display area 150M of FIG. FIG. 5 is a plan view showing a state in which a user issues an auxiliary information request instruction using an image 152 as a target image in the display state shown in FIG. 2. It is a top view which shows another example of the state in which the auxiliary information relevant to the target image in main content was displayed in the content display apparatus 100 shown in FIG. FIG. 6 is a block diagram showing sub-content data stored in the sub-content data storage unit 160 shown in FIG. 1 for each category (only three categories C1 to C3 are shown in FIG. 1; In this example, five categories C1 to C5 are defined). It is a perspective view which shows a general HSV three-dimensional color space. It is a perspective view which shows the example using the information which defines the closed area | region on HSV three-dimensional color space as the relevance information stored in the relevance information storage part 145 shown in FIG. It is a top view which shows the example using the information which defines the closed area | region on SV two-dimensional color plane as the relevance information stored in the relevance information storage part 145 shown in FIG. It is a top view which shows the method of defining the priority based on the relevance of the recognition point of the color recognized from the target image, and each category using the relevance information shown in FIG. FIG. 10 is a plan view showing a state in which the center of gravity point for each closed region is defined as a reference point in the relevance information shown in FIG. 9. It is a top view which shows the method of defining the priority based on the relevance of the recognition point of the color recognized from the target image, and each category using the relevance information shown in FIG. It is a top view which shows the example using the information which defines the area on a one-dimensional chroma-axis as the relevance information stored in the relevance information storage part 145 shown in FIG. It is a top view which shows an example of the color recognition method by the color recognition part 143 shown in FIG. It is a block diagram which shows the structure of 100 A of content display apparatuses which concern on the 2nd Embodiment of this invention. It is a top view which shows an example of the complexity recognition method by the complexity recognition part 147 shown in FIG. It is a perspective view which shows the example using the information which defines the distribution range in SVC three-dimensional space as the relevance information stored in relevance information storage part 145A shown in FIG. It is a perspective view which shows the method of defining the priority about each category using the relevance information shown in FIG. It is a block diagram which shows the structure of the content display apparatus 100B which concerns on the 3rd Embodiment of this invention. It is a perspective view which shows an example of the priority determination method by the priority provision part 144B shown in FIG. It is a figure which shows an example of the complexity set about the subcontent data which belongs to the category C4 shown in FIG. It is a flowchart which shows the basic procedure of the content display method which concerns on this invention.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. First embodiment of the present invention >>>
FIG. 1 is a block diagram showing a configuration of a content display device 100 according to the first embodiment of the present invention. As illustrated, the content display device 100 includes a main content data storage unit 110, an instruction input unit 120, a main content display processing unit 130, a sub content presentation processing unit 140, a display 150, and a sub content data storage unit 160. Yes.

  A feature of the content display device according to the present invention is that it has a function of displaying content that is directly viewed by the user and presenting auxiliary information related to images in the content. Here, the content that is directly browsed by the user is called main content, and the content that constitutes auxiliary information presented in association with the image is called sub-content.

  The main content is stored as main content data in the main content data storage unit 110. When the user gives an instruction to the instruction input unit 120, the main content display processing unit 130 displays the content desired by the user on the display. 150 screens are displayed. On the other hand, in the sub-content data storage unit 160, contents to be presented as auxiliary information are stored as sub-content data, and when the main content being displayed includes an image, the sub-content presentation processing unit 140 Auxiliary information related to the image is retrieved from the sub-content data, and the retrieved auxiliary information is presented.

  Note that the content display device 100 can actually be configured by incorporating a dedicated program into an electronic device with a built-in computer, such as a tablet electronic terminal, a smartphone, or a personal computer. In this case, the main content data storage unit 110 and the sub content data storage unit 160 are configured by a memory or a storage medium built in the electronic device, and the instruction input unit 120 is configured by an input device such as a touch panel or a keyboard. The display 150 is configured by a display for the electronic device, and the main content display processing unit 130 and the sub content presentation processing unit 140 are electronic components such as a built-in dedicated program (software) and a CPU built in the electronic device. This is realized as an information processing function executed in cooperation with (hardware resources).

  In the case of the example shown in FIG. 1, the main content data storage unit 110 shows a state in which three main content data M1, M2, and M3 are stored. Each content can be anything, such as a novel, story, essay, article, or movie. In the figure, an example is shown in which the main content data M1 is composed of files of a plurality of text data T, a plurality of image data P, and a plurality of audio data A.

  In general, various formats are used for the configuration of content data. For example, HTML-format content data is widely used as Web page display data, and EPUB-format content data is widely used as electronic book display data. When these formats are adopted, the text data T includes not only the text information included in the content but also information indicating its arrangement, color, size, font, etc., and information for pasting the specific image data P at a specific position. Information for displaying a playback button for reproducing specific audio data A is inserted as tagged data, and image data P and audio data A are referred to by the tagged data and presented as necessary. Will be.

  Of course, in carrying out the present invention, there is no particular limitation on the format of the content data to be prepared, and it is possible to use content data described in various formats. However, when the present invention is implemented, the main content data M needs to be content that includes at least the image data P, and the search character string K needs to be attached to the image data P. is there. This is because in the present invention, it is necessary to search auxiliary information related to an image in the main content being displayed using a search character string attached to the image.

  Therefore, in the present invention, the main content data M prepared in the main content data storage unit 110 is content (such as a picture book or a photo book) that does not include text data T and audio data A but includes only image data P. It doesn't matter. In the present invention, the image data P includes moving image data, and moving image content may be used as the main content data M. In the case of a moving image, the search character string K may be prepared for each frame image that constitutes the moving image. However, in practice, for each scene or every certain period (for example, every second). ) Is sufficient.

  On the other hand, the sub content data storage unit 160 stores a plurality of sub content data S belonging to different categories. In FIG. 1, three categories C1, C2, and C3 are defined, respectively, as enclosed by a one-dot chain line. The category C1 includes sub-content data S1, S2, and S3, and the category C2 includes sub-content data S4. , S5, and category C3 show the states to which the sub-content data S6, S7, and S8 belong. Actually, if a belonging list or the like indicating to which category each sub-content data belongs is prepared in the storage unit 160, the belonging category of each sub-content data is referred to by referring to the belonging list. Can be recognized.

  In the present invention, the sub-content is for constructing auxiliary information related to a specific image in the main content being displayed, and as described above, the search character string attached to the image is used. It is a search target. Therefore, in practice, it is preferable to use content (for example, contents such as a dictionary, an encyclopedia, an ilustration, and bean knowledge), which is a collection of information for each unit suitable for presentation as auxiliary information, as sub-contents.

  The instruction input unit 120 is a component that inputs a user instruction. The input instruction is given to the main content display processing unit 130 and the sub content presentation processing unit 140. As a result, the content of the content desired by the user is determined. It is displayed on the display 150. That is, the main content display processing unit 130 displays a predetermined portion of the main content data M stored in the main content data storage unit 110 as main content on the screen of the display 150 based on a given user instruction. I do. On the other hand, the sub-content presentation processing unit 140 obtains auxiliary information related to a specific target image in the main content displayed on the screen of the display 150 from the plurality of sub-content data S based on a given user instruction. A search process for searching is executed, and a process of presenting auxiliary information obtained as a result of the search process as sub-contents is performed.

  FIG. 2 is a plan view showing an example of a display form of the content display apparatus 100 shown in FIG. Here, an example in which the content display apparatus 100 is configured by incorporating a dedicated program into a tablet electronic terminal will be illustrated. The example shown in FIG. 2 shows a usage mode when the tablet-type electronic terminal is held in landscape orientation, with the left half of the display screen constituting the main content display area 150M and the right half constituting the sub content display area 150S. Yes. FIG. 2 shows a state immediately after the main content is displayed in the main content display area 150M, and the sub-content display area 150S is not displayed.

  As described above, the user can input an instruction to display a predetermined portion of the desired main content from the instruction input unit 120, and the main content display processing unit 130 responds to the given instruction in the main content display area 150M. Display accordingly. In a general tablet electronic terminal, the touch panel on the display screen functions as the instruction input unit 120, and thus the user can input various instructions to the touch panel using a finger. .

  For example, when the user performs a slide operation using one finger on the main content display area 150M shown in FIG. 2, an instruction for page turning can be input, and two fingers are in contact with the touch panel. When an operation for widening or narrowing the interval between fingers is performed, an instruction to enlarge or reduce can be input. Since the function itself of the main content display processing unit 130 is a known technique that is already used in various content display apparatuses, detailed description thereof is omitted here.

  In the example shown in FIG. 2, a text body 151, an image 152, and an audio presentation button 153 are displayed in the main content display area 150M. The information presentation is performed based on the main content data stored in the main content data storage unit 110 shown in FIG. FIG. 3 is a block diagram showing a specific configuration example of main content data related to the content displayed in the main content display area 150M of FIG. As shown in the figure, the main content data is composed of text data T, image data P, and audio data A, and five search character strings K are assigned to the image data P. The text body 151 is displayed based on the character information in the text data T, and the image 152 is displayed based on the image data P. When the user taps the voice presentation button 153 to give a playback instruction, the voice data A is played back.

  In the case of the example shown in FIG. 3, five sets of search character strings K such as “wolf”, “full moon”, “moon”, “cloud”, and “grassland” are attached to the image data P. However, the number of search character strings K is not particularly limited, and it is sufficient that at least one set of character strings is attached. However, in order to perform a more appropriate search, it is preferable to attach a plurality of character strings. The search character string K may be given in advance when the main content data M is created. Specifically, the author or editor of the content appropriately determines a character string that seems to be appropriate for each image, describes this in a predetermined format as a search character string K for each image, and each image An operation to be added to the data P may be performed. The five sets of search character strings K attached to the image data P shown in FIG. 3 are character strings determined in this way.

  The function that characterizes the present invention is a function that presents auxiliary information related to an image in the main content displayed on the display 150 in response to a user request. Here, as shown in FIG. 2, a case where the user inputs an instruction to request auxiliary information related to the image 152 in a state where the main content including the image 152 is displayed in the main content display area 150M is considered. let's see. If the instruction input unit 120 is provided with a function of inputting a user instruction for designating an arbitrary instruction point on the screen of the display 150, the sub-content presentation processing unit 140 displays the image displayed in the vicinity of the instruction point. Can be recognized as a target image, and auxiliary information related to the target image can be presented.

  FIG. 4 is a plan view showing a state in which the user issues an auxiliary information request instruction using the image 152 as a target image in the display state shown in FIG. In this example, since the touch panel on the display screen functions as the instruction input unit 120, when the user performs an operation of touching the image with the finger F, the relevant image is requested to be related auxiliary information. It is handled as if the instructions were given. Therefore, as shown in the figure, when the user's finger F touches the display area of the image 152, the instruction input unit 120 determines that the auxiliary information request instruction has been made with the image 152 as the target image, and the instruction is sub-addressed. This is transmitted to the content presentation processing unit 140. Of course, instead of touching, an auxiliary information request instruction may be issued by an operation such as a double tap.

  The sub-content presentation processing unit 140 searches the contents of the plurality of sub-content data S1 to S8 stored in the sub-content data storage unit 160 based on such a request instruction, and assists related to the target image 152. Information is extracted and displayed in the sub-content display area 150S. The example shown in FIG. 4 is an example in which the searched three auxiliary information 154, 155, and 156 are displayed in the sub-content display area 150S. Specifically, auxiliary information 154 with the heading “Wolf”, auxiliary information 155 with the heading “Obamitsu custom”, and auxiliary information 156 with the heading “Wolf man?” Are displayed as search results. ing. When there are a lot of searched auxiliary information, for example, the sub-content display area 150S may be scrolled so that it can be sequentially displayed.

  These auxiliary information 154 to 156 are all auxiliary information related to the target image 152, and the user can obtain more detailed knowledge about the target image 152. Such auxiliary information is all extracted by a search using the search character string K attached to the image data P as a search key. In the case of the example shown in FIG. 3, the image data P corresponding to the target image 152 includes five sets of search character strings K of “wolf”, “full moon”, “moon”, “cloud”, and “meadow”. The sub-content presentation processing unit 140 searches the sub-content data S1 to S8 using these character strings as search keys, and displays the extracted auxiliary information in the sub-content display area 150S. Do.

  If the user gives an auxiliary information request instruction for another image as a target image, auxiliary information related to the newly specified target image is displayed in the sub-content display area 150S. Of course, if necessary, it is possible to weight each search character string K and perform a search in consideration of the weight.

  FIG. 5 is a plan view showing another example of a state in which auxiliary information related to the target image 152 in the main content is displayed on the content display device 100 shown in FIG. This example is a display example when the tablet electronic terminal is held vertically, and in a state where auxiliary information is not displayed, the full screen area constitutes the main content display area 150M, and the content of the main content is the full screen. Is displayed. In this state, when the user touches the image 152 with the finger F, a search is performed by the sub-content presentation processing unit 140, and auxiliary information related to the image 152 is displayed in the sub-content display area 150S.

  In the example shown in FIG. 5, the sub-content display area 150S is displayed so as to be superimposed on the main content display area 150M as necessary. The auxiliary information 154 is a balloon-shaped sub-content display area 150S. It is displayed in. In the example shown in FIG. 5, since the area of the sub-content display area 150S is smaller than that in the example shown in FIG. 4, only the auxiliary information 154 with the heading “Wolf” is displayed. Each time the sub content display area 150S is tapped, the auxiliary information 155, the auxiliary information 156, and the remaining auxiliary information that has been searched may be sequentially displayed. When the user finishes reading the auxiliary information, the sub-content display area 150S may be erased from the screen by touching the target image 152 again with the finger F, for example.

  As described above, the example shown in FIG. 4 and the example shown in FIG. 5 have been shown as the forms for displaying the main content display area 150M and the sub-content display area 150S on the screen of the display 150. The display form is not limited. For example, only when it is necessary to present auxiliary information, the entire screen is switched from the main content display area 150M to the sub-content display area 150S, and when the presentation of auxiliary information is completed, the entire screen is returned to the main content display area 150M. It doesn't matter if you do.

  An important feature of the present invention resides in the function of the subsidiary content presentation processing unit 140. As described above, the sub-content presentation processing unit 140 executes a search process for searching auxiliary information related to a specific target image from a plurality of sub-content data based on a user instruction, and is obtained as a result of the search process. However, when there are a plurality of pieces of auxiliary information hit in the search, it is necessary to set a priority order for them.

  For example, when the sub-content display area 150S has only a space for displaying auxiliary information for three cases as in the example shown in FIG. 4, or displays auxiliary information for one case as in the example shown in FIG. If there is only space, only three or one auxiliary information can be displayed unless scroll display or time-division display (display using a method such as page switching) is performed. In this case, it is necessary to set a priority order for a plurality of auxiliary information hit by the search and display only the top 3 or top 1 auxiliary information.

  Of course, if scroll display or time-division display is performed, it is theoretically possible to present any number of auxiliary information. However, as long as multiple pieces of auxiliary information are presented, the order is determined mutually and the order is followed. I have to make a presentation. For example, in the example shown in FIG. 4, since there is a space for displaying auxiliary information for three items in the sub-content display area 150S, if the number of hit auxiliary information is three, all items are displayed. It is possible to display simultaneously. Nevertheless, since the sub-content display area 150S is divided into three spaces of an upper stage, a middle stage, and a lower stage, it is necessary to determine and display the order by some method.

  In general, when a search is performed using a character string as a search key, a method is used in which the degree of matching of each hit item with the search key is calculated and the priority is defined in descending order of the degree of matching. Therefore, also in the present invention, when a plurality of pieces of auxiliary information are hit as a result of performing a search using the search character string K as a search key, a degree of match is given to each piece of auxiliary information (for example, one character string matches) In this case, one point may be added or a unique weight value may be added to the character string), or presentation may be performed in descending order of the degree of match.

  However, in the present invention, since the auxiliary information to be presented is information related to the target image, a unique method is adopted in which priority is determined in consideration of the color of the target image. As described above, the sub content data storage unit 160 stores the sub content data S classified for each category. Therefore, a color is associated with each category, and higher priority is given to auxiliary information extracted from sub-content data belonging to a category that matches the color of the target image. Thus, the important feature of the present invention is that the priority of auxiliary information to be presented is determined according to the color of the target image. By adopting such a method, it becomes possible to select and present more appropriate auxiliary information related to the image included in the content being displayed. The specific method will be described in detail below.

<<< §2. Relationship between color and category as the basic principle of the present invention >>>
As described in §1, the sub-content presentation processing unit 140 searches auxiliary information related to a specific target image from a plurality of sub-content data, and defines a priority for the search result based on the color of the target image. The auxiliary information is presented in consideration of this priority. Here, the element directly associated with the color is a category of sub-content data. Therefore, a specific example will be given below to describe the relationship between colors and categories in the present invention and explain the basic principle of the present invention. The “color” in the present invention does not mean a general color such as red / blue / yellow, that is, a hue, but a broad color including saturation and lightness. Therefore, the following explanation also considers hue, saturation, and lightness as three elements of color.

  The block diagram of FIG. 1 shows an example in which eight sets of content data S1 to S8 belonging to any of the three categories C1 to C3 are stored in the sub content data storage unit 160. Here, as shown in FIG. 6, let us consider a case where 13 sets of content data S1 to S13 belonging to any of the five categories C1 to C5 are stored. Here, as shown in the figure, the five categories are “Category C1: Dictionary”, “Category C2: Specialized Book”, “Category C3: Ilustration / Bean Knowledge”, “Category C4: Life Information”, “Category C5: Business” "

  The subcategory “S1: Japanese dictionary”, “S2: Kanji large dictionary”, “S3: Historian dictionary” belongs to “Category C1: Dictionary”. The sub-contents “S4: World Animal and Plant System” and “S5: Introduction to Polymer Materials” belong to “Category C3: Ilustration / Bean Knowledge”, “S6: World Joke Collection”, “S7: Seasonal Beans”. Sub-contents such as “Knowledge” and “S8: Monsters in Fairy Tale” belong to “Category C4: Life Information”, “S9: Child Life Encyclopedia”, “S10: Walk in Kyoto”, “S11: Cleansing Book” "Category C5: Business Document" belongs to "S12: Basic Knowledge of Securities Transactions" and "S13: Sales Points" (the title of each content is And an illustration of a fictitious title in order to facilitate understanding of the bright, not related to the content of the real).

  Actually, in the case of the example shown in FIG. 4, for example, the auxiliary information 154 with the heading “Wolf” is information extracted from “Sub-content S1: Japanese dictionary” belonging to “Category C1: Dictionary”. Auxiliary information 155 with the headline “Otsukimi no Kanto” is information extracted from “Sub-content S7: Seasonal Bean Knowledge” belonging to “Category C3: Ilustration / Bean Knowledge”, headline “Wolf Man” The supplementary information 156 to which “is it?” Includes various categories such as information extracted from “Sub-contents S8: Monsters in the fairy tale” belonging to “Category C3: Ilustration / Bean Knowledge”. A plurality of auxiliary information is extracted from various sub-content data belonging to.

  In the present invention, the sub-content data is classified into a plurality of categories as described above, and a unique color is associated with each category. In general, colors can be used as a means of communication, and each color can read some message. Therefore, when considering what kind of message is transmitted for each category, a unique color can be associated with each category.

  From the viewpoint of color, the following words are associated with each color, and the following images are associated.

(1) “Color with high brightness and low saturation”
Words such as light, light, delicate, and soft are associated with light and romantic images.

(2) "Color with lightness above and saturation above middle"
Words such as bright, clear, and clear are associated, and a flashy image is associated with maintaining dignity.

(3) “Color with low brightness and low saturation inside”
Words such as plain, jealous, and subtle are associated, and jealous and fantastic images are associated.

(4) “Medium lightness and high saturation”
Words such as clear, strong and stimulating are associated with flashy and lively images.

(5) “Color with low brightness and low saturation”
Words such as dull, dark, and elder are associated, and formal images and images lacking individuality and speed are associated.

(6) “Medium lightness and high saturation”
Words such as dark, strong, skilled, profound, and responsive are associated with the image of skilled veterans and old-fashioned images.

(7) "Color with medium brightness and low saturation"
Words such as dull, sober, dull, astringent, and sultry are associated, and an old-fashioned image, an astringent image, and an image with a personality are associated.

  As shown in the above example, the inventor of the present application associates each color with an appropriate word or image, and conversely, for each category as illustrated in FIG. I thought it would be matched. For example, for “Category C1: Dictionary”, “low-lightness and low-saturation color” is appropriate, and for “Category C2: Special Book”, “low-lightness and medium-saturation color” is appropriate. As for “Category C3: Ilustration / Bean Knowledge”, “medium color, high chroma” is appropriate.

  In this way, if a specific color is associated with the category to which the sub-content data belongs, it recognizes the color of the target image for which the auxiliary information is to be presented, and then belongs to the category highly related to the recognized color. It is possible to give higher priority to the auxiliary information extracted from the sub-content data and to present the extracted auxiliary information based on the priority order according to the given priority. . If the priority order is determined and the auxiliary information is presented by such a method, more appropriate auxiliary information related to the target image can be efficiently selected and presented.

  For example, in the example shown in FIG. 4, if the color of the target image 152 is “a color with low brightness and low saturation”, the content of the text body 151 constituting the main content also has “low brightness and saturation”. It is highly possible that the content corresponds to an image of “low color” (formal image), and the user is likely to be browsing content corresponding to such an image. Therefore, when there are a plurality of pieces of auxiliary information to be presented for the target image 152, it belongs to a category (“Category C1: Dictionary” in this example) that is highly relevant to “color with low brightness and low saturation”. It can be said that it is preferable to preferentially present auxiliary information extracted from sub-content data.

  In the case of the above example, the auxiliary information (category C1) extracted from the “sub-content S1: Japanese dictionary” is the auxiliary information hit with the same search key “wolf / full moon / moon / cloud / prair”. It is presented with priority over auxiliary information (category C3) extracted from “sub-content S7: seasonal knowledge” and “sub-content S8: monsters in a fairy tale”. The display result in the sub content display area 150S of FIG. 4 is a result based on such priorities. This is a mode of presentation of auxiliary information that is more appropriate for a user who is viewing the main content estimated to be “formal image content”.

  On the other hand, in the example shown in FIG. 4, if the color of the target image 152 is “a color with medium brightness and high saturation”, the content of the text body 151 constituting the main content is also “medium brightness, It is highly possible that the content corresponds to the image of “color with high saturation” (flashy and lively image), and the user is likely to be browsing content corresponding to such an image. Therefore, when there are a plurality of pieces of auxiliary information to be presented for the target image 152, a category that is highly relevant to “a color with medium brightness and high saturation” (in this example, “category C3: ilustration / bean knowledge”). It can be said that it is preferable to preferentially present auxiliary information extracted from sub-content data belonging to “)”.

  In the case of the above example, even if the auxiliary information is hit with the same search key “wolf / full moon / moon / cloud / grass”, “sub content S7: seasonal knowledge” and “sub content S8: in fairy tales” The auxiliary information (category C3) extracted from “monsters of” is presented in preference to the auxiliary information (category C1) extracted from “sub-content S1: Japanese dictionary”. That is, unlike the example shown in FIG. 4, the auxiliary information 154 with the heading “Wolf” is displayed in the lower row, and the auxiliary information 155 and 156 are moved up and displayed in the upper and middle rows, respectively. This is a more appropriate mode of presentation of auxiliary information for a user who is browsing the main content estimated to be “flashy and lively image content”.

<<< §3. Configuration and operation mode of sub-content presentation processing unit >>
<3-1: Basic Configuration of Sub-Content Presentation Processing Unit>
Here, the basic configuration of the subsidiary content presentation processing unit 140 shown in FIG. 1 will be described in detail. The sub-content presentation processing unit 140 is a component having a function of presenting the content of the searched sub-content as auxiliary information based on the basic principle described in §2, and as shown in FIG. Part 141, search processing part 142, color recognition part 143, priority assignment part 144, relevance information storage part 145, and auxiliary information presentation processing part 146.

  The search character string recognition unit 141 is a component that recognizes the search character string K attached to the target image. For example, as in the example illustrated in FIG. 4, when the user uses the finger F to request presentation of auxiliary information using the image 152 as the target image, the instruction input unit 120 requests the search character string recognition unit 141. The presentation request is transmitted. On the other hand, the main content display processing unit 130 reads text data T for displaying the text body 151 constituting the main content and image data P for displaying the target image 152. Therefore, the search character string recognizing unit 141 may perform a process of extracting the search character string K attached to the image data P and recognizing it as a character string necessary for the search.

  The search processing unit 142 uses the search character string K recognized by the search character string recognition unit 141 as a search key to perform a search process on the plurality of sub content data S1 to S8 stored in the sub content data storage unit 160. Are executed, the auxiliary information that has been hit is extracted, and this is processed as a search result and reported to the auxiliary information presentation processing unit 146 (if necessary, the auxiliary information extracted from which sub-content data in which category) Also report information indicating whether there is.) When there are a plurality of hit auxiliary information, all of them are extracted and reported to the auxiliary information presentation processing unit 146. As described above, a technique for performing a search on content data using a character string as a search key and extracting a hit content is a well-known technique used in various fields, and thus detailed description thereof is omitted here.

  On the other hand, the color recognition unit 143 performs processing for recognizing the color of the target image 152. Here, as described above, the “color” is composed of three elements of hue, saturation, and brightness, and the color recognition unit 143 has a hue value, saturation value, and brightness of representative colors that represent the entire target image 152. Performs processing to obtain a value. Specific methods for recognizing colors will be described with some examples in §3-5 below.

  The relevance information storage unit 145 is a component that stores relevance information indicating the relevance between each category and color to which the sub content data stored in the sub content data storage unit 160 belongs. For example, in the case of the simple example described in §2, “Category C1: Dictionary” is highly related to “Category C1: Dictionary”. On the other hand, it is only necessary to store relevance information such as “relevance to“ medium color, high chroma ”. More specific definition methods of relevance will be shown as some examples in the following §3-2 to §3-4.

  The priority assigning unit 144 refers to the relevance information in the relevance information storage unit 145 based on the color recognized by the color recognizing unit 143, and among the auxiliary information retrieved by the search processing unit 142, the relevance information is more relevant. A process of giving higher priority to auxiliary information extracted from sub-content data belonging to a higher category is performed. Specific methods of assigning priorities will be described as some examples in the following §3-2 to §3-4.

  The auxiliary information presentation processing unit 146 presents the auxiliary information extracted by the search processing unit 142 based on the priority order corresponding to the priority given by the priority assigning unit 144. In the case of the embodiment shown in FIGS. 4 and 5, the auxiliary information 154 to 156 is composed of text data, and the auxiliary information presentation processing unit 146 displays auxiliary information consisting of text in the sub-content display area 150 </ b> S of the display 150. Will be displayed.

  When there are a plurality of pieces of auxiliary information extracted by the search processing unit 142, the auxiliary information presentation processing unit 146 selects the n-th subordinate information (n is a predetermined number) based on the priority among the plurality of pieces of auxiliary information. By presenting only the auxiliary information of (natural number), presentation of the auxiliary information can be simplified. For example, as in the example shown in FIG. 4, if there is only a space for displaying auxiliary information for three cases in the sub-content display area 150 </ b> S, n = 3 is set, and among the extracted auxiliary information, If only the upper third auxiliary information is presented, the presented auxiliary information will fit in the sub-content display area 150S. Of course, as shown in FIG. 5, when displaying one piece of auxiliary information in a balloon shape, n = 1 is set, and only the auxiliary information with the highest priority is extracted from the extracted auxiliary information. It should be presented.

  In the case where only the top n-th auxiliary information is presented as in the above example, the search processing unit 142 searches the sub-content data in the order from the highly relevant category to the less relevant category. The process may be executed, and the subsequent search process may be stopped when the top nth auxiliary information is extracted. That is, the order of search is performed in the order of the category having high relevance to the color recognized by the color recognizing unit 143 and the category having low relevance, and the priority assigning unit 144 prioritizes only based on the relevance to the color. If the degree is given, since the auxiliary information with the highest priority is searched in the order of the low auxiliary information, the subsequent search processing becomes unnecessary when the top n-th auxiliary information is extracted. . Therefore, in this case, useless search time can be saved by canceling the subsequent search processing.

  If necessary, the auxiliary information presentation processing unit 146 can display the number of auxiliary information items that cannot be displayed at one time in the display space of the sub-content display area 150S by scroll display or time-division display. For example, in the example shown in FIG. 4, if the display content in the sub-content display area 150S can be scrolled, four or more pieces of auxiliary information can be displayed. Alternatively, the pages in the sub-content display area 150S can be switched, and the first page includes the first to third auxiliary information, the second page includes the fourth to sixth auxiliary information, and so on. You may make it display in a time division format, switching. In the case of the example shown in FIG. 5 as well, the second and subsequent auxiliary information can be displayed by scrolling within the balloon or switching pages within the balloon.

  Of course, when there are a plurality of pieces of auxiliary information to be presented, the auxiliary information presentation processing unit 146 performs presentation based on the priority order. For example, when there is a space for displaying auxiliary information for three items at a time as in the sub-content display area 150S shown in FIG. 4, the priority order is defined in the order of the upper level, the middle level, and the lower level. What is necessary is just to determine an arrangement | positioning order so that it may arrange | position to the position which precedes the thing with a high order | rank. In addition, when performing time-division display by a method such as page switching, the time order may be determined so that the higher the order, the higher the order is presented.

<3-2: Example of mapping closed region on three-dimensional color space>
Subsequently, a specific example of the relevance information stored in the relevance information storage unit 145 will be described. First, an embodiment in which relevance information is configured by mapping closed regions on a three-dimensional color space will be described.

  In the embodiment shown here, “color” is defined by three elements of hue, saturation, and brightness. In general, when a color is defined by such three elements, each color is represented by one point on the HSV three-dimensional color space as shown in FIG. That is, three elements of a hue value H (Hue), a saturation value S (Saturation), and a lightness value V (Value) can be defined at an arbitrary point in this color space.

  Here, the lightness value V is defined as a coordinate value related to a direction along the lightness axis V extending in the vertical direction in the drawing. For example, the point P1 on the brightness axis V is a point having a brightness value of V = 0, the point P2 is V = 0.5, and the point P3 is a point having a brightness value of V = 1.0 (numbers enclosed in a rectangle in the figure). Indicates a specific brightness value V).

  The saturation value S is defined as the distance from the brightness axis V shown in the figure. More specifically, as shown, if a circumference C having a radius r orthogonal to the lightness axis V is defined, all points on the circumference C are points having a saturation value r. . Then, if the saturation axis S is defined in the radial direction, the point P2 on the saturation axis S is S = 0, the point P4 is S = 0.5, and the point P5 is S = 1.0. (The numbers surrounded by ellipses in the figure indicate specific saturation values S).

  On the other hand, the hue value H is defined as an angle around the brightness axis V shown in the figure. More specifically, in the case of the illustrated example, the azimuth angle of the saturation axis S is defined as 0 °, and 0 ° to 360 ° is defined in the clockwise direction when the circumference C is viewed from above. Then, the point P5 on the circumference C is H = 0 °, the point P6 is H = 120 °, the point P7 is H = 180 °, and the point P8 is a point having a hue value of H = 240 °. (Numbers enclosed in triangles in the figure indicate specific hue values H). Specifically, a pair of colors corresponding to 0 ° corresponding to red, 120 ° corresponding to green, and 240 ° corresponding to blue and having an angle of 180 ° are complementary to each other.

  Here, the range of the lightness value V is set to 0 to 1.0, the range of the saturation value S is set to 0 to 1.0, and the range of the hue value H is set to 0 ° to 360 °. . With this setting, the color defined by these three elements is represented as one point in the coordinate space in a cylinder having a height of 1.0 and a radius of 1.0. For example, the illustrated point P6 corresponds to a color having a hue value H = 120 °, a saturation value S = 1.0, and a lightness value V = 0.5.

  Now, the relevance information stored in the relevance information storage unit 145 is information indicating the relevance between individual categories and colors to which the sub content data S stored in the sub content data storage unit 160 belongs. It can be shown as a closed region mapped on a three-dimensional color space. FIG. 8 is a perspective view showing an example in which a closed region is mapped on the HSV three-dimensional color space as such relevance information. In the illustrated example, an example is shown in which three closed regions C1, C2, and C3 are mapped in the HSV color space shown in FIG. The closed regions C1, C2, and C3 correspond to the categories C1, C2, and C3 in the sub-content data storage unit 160 of FIG. 1, and indicate the color distribution ranges that are related to the respective categories. Yes.

  The information indicating the closed regions C1, C2, and C3 on the HSV three-dimensional color space illustrated in FIG. 8 is used as the relationship information indicating the relationship between the individual categories C1, C2, and C3 and the color. If stored in the storage unit 145, the priority assigning unit 144 refers to this relevance information based on the color recognized by the color recognizing unit 143, and from the sub-content data belonging to the category having higher relevance. A process of assigning a higher priority to the extracted auxiliary information can be performed. For example, by plotting the color recognized by the color recognition unit 143 (in the example shown in FIG. 4, the representative color for the target image 152) in the HSV color space, the point Q shown in FIG. 8 (hereinafter referred to as a recognition point). ) Is obtained, priority can be given based on the positional relationship between the plotted recognition points Q and the individual closed regions C1, C2, C3 mapped.

  The details of the color recognition method performed by the color recognition unit 143 will be described in §3-5. However, the color recognition unit 143 recognizes the representative value of saturation, the representative value of brightness, and the representative value of hue for the target image. By doing so, the priority assigning unit 144 can plot the recognition points Q on the HSV color space as shown in FIG. 8 as colors having the recognized representative values as coordinate values. On the other hand, in the relevance information storage unit 145, for each category C1, C2, and C3, a distribution range of combinations of saturation values, lightness values, and hue values (ie, closed regions C1, C2, and C3) is associated. Since the relevance information is stored, the priority assigning unit 144 has a distribution range of combinations of saturation values, lightness values, and hue values associated with the individual categories C1, C2, and C3 (that is, the closed region C1). , C2, C3) and the combination of the representative value of saturation, the representative value of lightness, and the representative value of hue (that is, the position of the recognition point Q) recognized by the color recognition unit 143. A priority according to the degree of approximation can be given.

  In the HSV color space shown in FIG. 8, if the brightness axis V is called a reference coordinate axis, the position on the reference coordinate axis is the brightness value V, the distance from the reference coordinate axis is the saturation value S, and the angle around the reference coordinate axis It can be said that mapping information is obtained by mapping the closed regions C1, C2, and C3 corresponding to the individual categories C1, C2, and C3 on the three-dimensional color space in which the hue value is H. In the relevance information storage unit 145, such mapping information is stored as relevance information. Then, the priority assigning unit 144 recognizes the color recognized by the color recognition unit 143 in this three-dimensional color space with the representative value of saturation, the representative value of lightness, and the representative value of hue as coordinate values, respectively. Plotting is performed as Q, and a priority is given based on the positional relationship between the plotted recognition point Q and each of the mapped closed regions C1, C2, and C3.

  The simplest method of assigning priority based on the positional relationship between the two is a method of assigning an alternative priority based on the criterion whether the corresponding closed region includes the recognition point Q or not. It is. For example, if the priority level 1 is given when the recognition point Q is included, and the priority level 0 is not given, the auxiliary information extracted from the sub-content data belonging to the category C3 in the example shown in FIG. Is given priority 1, and priority 0 is given to auxiliary information extracted from sub-content data belonging to categories C1 and C2. In other words, binary information is given as a priority.

  Of course, the priority to be given does not necessarily need to be binary information, and a stepwise priority according to the degree of approximation regarding the positions of the closed regions C1, C2, C3 and the recognition point Q can be given. is there. For example, when the recognition point Q is located near the boundary in the closed region, the priority is 1, the priority is 2 when it is located inside the closed region, the priority is 0.5 when located near the boundary outside the closed region, Other than that, it is possible to adopt a method of setting the priority to 0. A method for assigning such stepwise priorities will be described in more detail in the example of §3-3. In short, it is only necessary to compare the color distribution range associated with each category with the color recognized by the color recognition unit 143 and to give priority according to the degree of approximation between the two.

<3-3: Example of mapping a closed region on a two-dimensional color plane>
In the above §3-2, the distribution range of the color associated with each category is shown by the closed region mapped on the HSV three-dimensional color space in which the color is defined by the three elements of hue, saturation, and lightness. An example was given. Since all colors can be defined by such three elements, it makes sense to define relevance information by mapping closed regions on a three-dimensional color space.

  However, when practicing the present invention, from a practical point of view, it is sufficient to indicate the color distribution range associated with each category by the closed region mapped on the SV two-dimensional color plane. The reason is that, in practice, no significant correlation is found between the category of the sub-content data and the hue.

  In other words, whether it is content in the category of “living information”, content in the category of “story / bean knowledge”, content of the category of “specialized books”, Is normally “red”, “blue” is normally used for the object “sea”, and “green” is normally used for the object “tree”. Of course, even if it is the same "red", "highly light red" is used for "living information", "highly saturated red" is used for "especial knowledge / beans", and "specialized books" There may be a tendency that “red with low brightness” is used, but there is no change in using the hue “red” for the object “strawberry”.

  Therefore, the inventor of the present application should prepare information relating to the saturation value S and the lightness value V as relevance information indicating the relevance between individual categories and colors in practice in implementing the present invention. I think that is enough. In this case, it is sufficient to prepare relevance information in which the closed region is mapped on the SV two-dimensional color plane. That is, the relevance information storage unit 145 stores relevance information in which the distribution range (closed region on the two-dimensional plane) of the combination of the saturation value S and the lightness value V is associated with each category. The color recognizing unit 143 recognizes the representative value of saturation and the representative value of lightness for the target image, and the priority assigning unit 144 determines the saturation value associated with each category. The distribution range of the combination with the lightness value (the position of each closed region) is compared with the combination of the representative value of saturation and the representative value of lightness recognized by the color recognition unit 143, and according to the degree of approximation of both What is necessary is just to give a priority.

  FIG. 9 is a plan view showing an example in which information defining a closed region on the SV two-dimensional color plane is used as the relevance information stored in the relevance information storage unit 145 shown in FIG. In the example shown in FIG. 1, only three categories C1 to C3 are defined as sub-content data categories. However, in FIG. 9, five categories C1 to C5 are provided as in the example shown in FIG. An example of defining

  That is, in FIG. 9, an SV color plane is defined with the coordinate axis of saturation S (coordinate value 0 to 1.0) as the horizontal axis and the coordinate axis of lightness V (coordinate value 0 to 1.0) as the vertical axis. Corresponding to “Category C1: Dictionary”, “Category C2: Special Book”, “Category C3: Ilustration / Bean Knowledge”, “Category C4: Life Information”, “Category C5: Business Book” An example in which closed regions C1 to C5 are defined is shown. Since some of the closed areas overlap each other, in FIG. 9, the outline of the closed area C2 corresponding to “Category C2: Specialized Book” is indicated by a broken line, and “Category C3: Ilustration / That Knowledge” The closed region C3 corresponding to is indicated by a one-dot chain line so that the closed regions can be distinguished from each other.

  Of course, the relevance information shown in FIG. 9 is shown as an example, and the setting of each category and the position and shape of each closed region can be freely determined by the designer of the content display device according to the present invention. However, in order to obtain the effect of the present invention of selecting and presenting more appropriate auxiliary information, it is necessary to set the position and shape of each closed region in consideration of the characteristics of each category.

  In the case of the example shown in FIG. 9, the closed region C1 for “Category C1: Dictionary” occupies a low region where the saturation S and the lightness V are both low or medium, and the closed region C2 for “Category C2: Technical Book” is , Saturation S is moderate and occupies an area where brightness V is low, and closed area C3 for “Category C3: Ilustration / Bean Knowledge” occupies an area where saturation S is medium and above and brightness V is excluding both high and low ends. The closed area C4 for “Category C4: Life information” occupies an area where the saturation S is medium to low and the brightness V is relatively high, and the closed area C5 for “Category C5: Business book” has the saturation S Medium to high and lightness V occupies a medium region. Such a region distribution takes into consideration the characteristics regarding the color of each category.

  As described above, in the example shown in FIG. 9, the closed regions C1 to C1 corresponding to the individual categories are displayed on the two-dimensional SV color plane having the saturation value S on the first coordinate axis and the lightness value V on the second coordinate axis. Relevance information is configured by mapping information obtained by mapping C5. If such relevance information is stored in the relevance information storage unit 145, the priority assigning unit 144 displays the color recognized by the color recognition unit 143 on the two-dimensional SV color plane as a representative of saturation. A value and a representative value of brightness are plotted as recognition points Q each having a coordinate value, and priorities are given based on the positional relationship between the plotted recognition points Q and the individual closed regions C1 to C5 mapped. Can do.

  FIG. 10 shows a specific method for defining priorities based on the positional relationship between the color recognition points recognized from the target image and the closed regions corresponding to the respective categories using the relevance information shown in FIG. FIG. The simplest method for assigning priority based on the positional relationship between the two includes whether or not a recognition point is included, as in the embodiment for mapping a closed region on the three-dimensional color space described in §3-2. This is a method of giving an alternative priority based on the standard. In this case, the priority assigning unit 144 determines whether or not the closed regions C1 to C5 corresponding to each category include a recognition point, and extracts from the sub-content data belonging to the category corresponding to the closed region including the recognition point. A higher priority is given to the auxiliary information.

  For example, if binary information of priority 1 is included when a recognition point is included and priority 0 is not included when the recognition point is not included, in the case of the example shown in FIG. 10, when the recognition point Q1 is obtained, Priority 1 is given to the auxiliary information extracted from the sub-content data belonging to “Category C4: Life information”, and priority 0 is given to the other auxiliary information. Similarly, when the recognition point Q2 is obtained, priority 1 is given to the auxiliary information extracted from the sub-content data belonging to “Category C5: Business Book” and “Category C3: Ilustration / Bean Knowledge”. The other auxiliary information is given priority 0. However, when the recognition point Q3 is obtained, since the recognition point Q3 is not included in any closed region, priority level 0 is given to any auxiliary information. Therefore, no priority is given.

  After all, when the recognition point acquired by the color recognition unit 143 is not included in any closed region as in the recognition point Q3 shown in FIG. 10, no priority is given and the sub-contents belonging to all categories. The auxiliary information extracted from the data is handled equally. Of course, there is no problem in performing such handling, but if fine priority is given, stepwise priority according to the degree of approximation regarding the positions of the closed regions C1 to C5 and the recognition point Q. The degree should be given. For example, as described in §3-2, priority 1 when the recognition point Q is located near the boundary in the closed region, priority 2 when the recognition point Q is located inside the closed region, and near the boundary outside the closed region It is also possible to adopt a method in which priority is set to 0.5 when positioned, and priority is set to 0 otherwise. In short, in the present invention, the color distribution range associated with each category and the color recognized by the color recognition unit 143 may be compared to give priority according to the degree of approximation between them.

  Here, as an example of a method for assigning finer priorities, the Euclidean distance between a predetermined reference point defined for the closed region corresponding to each category and the recognition point is calculated, and the distance between them is small. A method for giving higher priority to auxiliary information extracted from sub-content data belonging to a category corresponding to a closed region will be described. As a reference point for each closed region, a point at a position representing the closed region may be employed. Specifically, for example, the geometric center of gravity of each closed region may be defined as the reference point.

  FIG. 11 is a plan view showing a state in which the geometric barycentric points for the closed regions C1 to C5 are defined as reference points G1 to G5 in the relevance information shown in FIG. The positions of the closed regions C1 to C5 are represented by the positions of the reference points G1 to G5, respectively. When a specific recognition point is determined, the priority of each category with respect to the recognition point is It is determined based on the Euclidean distance (the length of the line segment connecting both points) between each of the reference points G1 to G5. Specifically, a higher priority is given as the distance is shorter.

  FIG. 12 is a plan view showing a method of defining priorities based on the relevance between the color recognition points Q4 and Q5 recognized from the target image and each category using the relevance information shown in FIG. In this example, since the recognition point Q4 is plotted at a position included in both of the closed regions C3 and C4, when the above-described method of adding binary information is adopted, the subordinates belonging to the categories C3 and C4 are used. Priority 1 is assigned to the auxiliary information extracted from the content data, and priority 0 is assigned to the other auxiliary information.

  On the other hand, when the method of assigning priority based on the distance to the reference point is adopted, the priority based on the distance L3 between the recognition point Q4 and the reference point G3 is assigned to the category C3, and the category C4 is assigned. Is given priority based on the distance L4 between the recognition point Q4 and the reference point G4. In the case of the illustrated example, since L3> L4, if the higher priority is given as the distance is shorter, the auxiliary information (life information) extracted from the sub-content data belonging to the category C4 However, a higher priority is given to the auxiliary information (ilustration / bean knowledge) extracted from the sub-content data belonging to the category C3. Of course, if the distance between the recognition point Q4 and the reference points G1, G2, and G5 is obtained, priority can be given to the categories C1, C2, and C5.

  On the other hand, since the recognition point Q5 is plotted at a position that is not included in any closed region, the sub-content data belonging to all the categories C1 to C5 is adopted when the above-described method of adding binary information is adopted. Priority 0 is assigned to the auxiliary information extracted from the above, and priority is not substantially assigned. On the other hand, when the method of assigning priority based on the distance to the reference point is adopted, priority is assigned to each category C1 to C5 based on the distance between the recognition point Q5 and each of the reference points G1 to G5. can do. For example, the priority for the category C4 is determined based on the distance L5 between the recognition point Q5 and the reference point G4 as illustrated.

  In order to determine the priority based on the distance, for example, if a priority that is inversely proportional to the distance is defined in advance, the priority that becomes higher as the distance is shorter can be determined (distance approaches 0). Since the priority increases rapidly, in practice, an upper limit may be set for the priority).

  In addition, since the priority does not necessarily need to take a continuous value, it is also possible to define a stepwise priority based on the distance. In the case of the illustrated example, the saturation value S is defined in the range of 0 to 1.0 and the lightness value V is defined in the range of 0 to 1.0. Therefore, the SV color plane is a square having a size of 1.0 both vertically and horizontally. It will be defined in the area. Therefore, for example, let L be the distance between the recognition point Q and an arbitrary reference point G, priority 3 if L <0.1, priority 2 if 0.1 ≦ L <0.2, If a stepwise definition is made such that the priority is 1 when 0.2 ≦ L <0.3, and the priority is 0 when 0.3 ≦ L, 4 within the range of 0 to 3 Stage priorities can be defined.

  As described above, when the priority is determined based only on the distance L between the recognition point Q and the reference point G, it is not necessary to define a closed region for each category, and only the reference point is defined. It will be enough. However, in reality, it is difficult to express the characteristics regarding the color of each category by only one point, which is the reference point G, and the closed region having a certain extent, that is, the saturation value S and the lightness value V It is preferable to express as a distribution range of the combination. Therefore, in practice, the recognition point Q is adopted while adopting the basic policy of defining the priority based on binary information whether the recognition point Q is included in the closed region defined for each category or not. It is preferable to perform fine priority definition in consideration of the distance L between Q and the reference point G.

  Specifically, for example, when the recognition point Q is included in the closed region defined for each category, the point 5 is added for the category, and further, according to the distance between the recognition point Q and the reference point G, As in the example described above, four levels of points in the range of 0 to 3 are determined and added, and the total of these points may be handled as a priority. Then, in the case of the recognition point Q4 shown in FIG. 12, the point 5 is added to the categories C3 and C4, and further, for all the categories C1 to C5, depending on the distance between the recognition point Q and the reference point G. 0 to 3 points will be added. Further, since the recognition point Q5 shown in FIG. 12 is not included in any closed region, the addition of the point 5 is not performed, but the recognition point Q, the reference point G, and all of the categories C1 to C5. 0 to 3 points are added according to the distance.

  As described above, a method for defining priority based on binary information indicating whether the recognition point Q is included or not included in the closed region defined for each category, the recognition point Q, and the reference point G The method of using the priority definition method that also takes into account the distance L to the above is also effective in the embodiment in which the closed region is mapped on the three-dimensional HSV color space described in §3-2.

<3-4: Example of mapping closed section on one-dimensional color coordinate axis>
Up to now, as an example of the relevance information stored in the relevance information storage unit 145, an example in which a closed region is mapped on the three-dimensional HSV color space in §3-2 is shown, and a two-dimensional SV in §3-3. An example of mapping a closed region on a color plane has been shown. Therefore, here, an embodiment in which the closed section is mapped on the one-dimensional color coordinate axis will be shown.

  FIG. 13 is a plan view showing an example in which information defining a section on the one-dimensional saturation axis is used as the relevance information stored in the relevance information storage unit 145 shown in FIG. Since this example is an example using a one-dimensional color coordinate axis, only the coordinate axis of saturation S (coordinate values 0 to 1.0) is defined as the horizontal axis, and the vertical axis is not defined (in the figure). For convenience, the categories are shown side by side in the vertical direction, but the vertical axis is not defined.)

  In the example shown in FIG. 13, similar to the example shown in FIG. 6, the corresponding closed sections for the five categories C1 to C5 are mapped on the saturation axis. That is, the closed section C1 for “Category C1: Dictionary” occupies a low area where the saturation S is medium or lower, and the closed section C2 for “Category C2: Specialized book” is an area where the saturation S is medium. The closed section C3 for "Category C3: Ilustration / Bean Knowledge" occupies an area where the saturation S is medium or higher, and the closed section C4 for "Category C4: Living information" has a medium to low saturation S The closed section C5 for “Category C5: Business book” occupies an area where the saturation S is medium to high. Such a region distribution takes into consideration the characteristics regarding the color of each category.

  FIG. 13 shows an example in which the saturation axis S is taken as the one-dimensional color coordinate axis. Of course, the lightness axis V or the hue axis H can be taken as the one-dimensional color coordinate axis. it can. If such relevance information is stored in the relevance information storage unit 145, the priority assigning unit 144 determines the color recognized by the color recognition unit 143 as the position Q (recognition point Q on the one-dimensional color coordinate axis). ) And a priority can be given based on the positional relationship between the plotted recognition points Q and the individual closed sections C1 to C5 mapped.

  In this case as well, a method for defining priority based on binary information indicating whether or not the recognition point Q is included in the closed section defined for each category, and the recognition point Q and the reference point G (for example, , A method of defining priority taking into account the distance L to the center point of each closed section). If the priority is defined based on the binary information, the category C1 and C4 are given priority 1 and the categories C2, C3 and C5 are given priority 0 for the recognition point Q shown in FIG.

  After all, based on the examples shown in §3-2 to §3-4, the relevance information storage unit 145 has one of hue value H, saturation value S, and brightness value V for each category. Relevance information in which a distribution range of one value or a combination range of these values is associated may be stored. Then, if the color recognition unit 143 recognizes the representative color for the target image as one of the representative values of the hue value H, the saturation value S, and the lightness value V, or a combination thereof, the priority The assigning unit 144 includes a distribution range of hue values H, saturation values S, lightness values V or combinations of these values associated with individual categories, and a hue value H recognized by the color recognition unit 143, It is possible to compare the saturation value S, the representative value of the lightness value V, or a combination thereof, and give priority according to the degree of approximation of both.

<3-5: Color recognition method by color recognition unit>
Here, a specific color recognition method by the color recognition unit 143 shown in FIG. 1 will be briefly described. For example, when the auxiliary information request instruction for the target image 152 is given by the method shown in FIG. 4, the color recognition unit 143 displays the hue value H and saturation for the representative color representing the entire target image 152. A process for obtaining the value S and the lightness value V may be performed. Specifically, the color recognition unit 143 sets the representative color for the target image 152 as a weighted average value in consideration of the area of the hue value H, the saturation value S, or the lightness value H of each unit constituting the target image 152. (In the case of the embodiment in which the closed section is mapped on the one-dimensional color coordinate axis described in §3-4), or a combination thereof (in the three-dimensional color space described in §3-2 and §3-3 or two-dimensional color) In the case of the embodiment in which the closed region is mapped on the plane, it may be recognized.

  For example, in the case where the target image 152 is a raster image composed of a collection of pixel values for a large number of pixels arranged in a vertical and horizontal matrix, if the average value of pixel values of individual pixels is obtained, the average value is , The weighted average value considering the area (that is, the number of pixels having the same pixel value), and the color indicated by the average value is a representative color representing the entire target image 152.

  In addition, when the target image 152 is a vector image expressed as a collection of graphics, if a weighted average value considering the area of the graphic is obtained for each color in the graphic, the average value is indicated. The color is a representative color that represents the entire target image 152.

  FIG. 14 is a plan view showing an example in which the pattern in the target image 152 is prepared as a vector image. In the case of this example, for example, the pattern partial area E1 (the pattern portion of the moon) is represented by a circle, and is constituted by data indicating the center point coordinates of the circle, the radius of the circle, the internal color, and the color of the contour line. can do. On the other hand, the partial areas E2, E3, and E4 (clouds, wolves, and rock pattern parts) are configured by data indicating contours of arbitrary shapes (for example, Bezier curve data), internal colors, and data indicating contour colors. can do. The partial area E5 is a background, is defined as an area excluding the partial areas E1 to E4 from the formation area of the entire image, and can be expressed only by data indicating color.

  In the example shown in FIG. 14, the figure constituting each partial area includes a value indicating the color painted inside the figure and a value indicating the color constituting the contour line. A value indicating a representative color representing the entire target image 152 can be obtained by performing a weighted average in consideration of the area of the value indicating the color.

  Usually, the color of the image displayed on the display screen is expressed by a combination of the component values of the three primary colors R, G, and B. For example, when one color is expressed by 8 bits, the component values of the three primary colors R, G, and B take values of 0 to 255, respectively. When the target image 152 is a raster image, each pixel has pixel values of three primary colors R, G, and B (for example, values of 0 to 255). If the average Rav, Gav, and Bav of the pixel values are obtained, the color indicated by the obtained average value (Rav, Gav, Bav) is the representative color of the image.

  When the target image 152 is a vector image, the color of the inside of each figure or the outline has pixel values (for example, values of 0 to 255) of the three primary colors R, G, and B. If the average Rav, Gav, and Bav of the pixel values are obtained for each of the three primary colors R, G, and B in consideration of the area of each figure and outline, the colors indicated by the obtained average values (Rav, Gav, and Bav) Is the representative color of the image.

In the embodiment described so far, the HSV color system is used which defines colors by three elements of hue, saturation, and brightness. However, the RGB color system and the HSV color system are different from each other. There is a one-to-one correspondence between them, and the color represented by the component values (R, G, B) can be replaced with the component values (H, S, V) by the following conversion formula.
H = 60 × (GB) / Max−Min) +0 (when Max = R)
H = 60 × (BR) / Max−Min) +120 (when Max = G)
H = 60 × (RG) / Max−Min) +240 (when Max = B)
S = (Max-Min) / Max
V = Max
However, Max refers to the largest one of R, G, and B, and Min refers to the smallest one of R, G, and B. When H <0, H = H + 360 °. Further, when S = 0, H is not defined.

  In consideration of such a correspondence, even a color expressed in the RGB color system can be handled by replacing it with the HSV color system. Therefore, even if the representative color recognized by the color recognition unit 143 is expressed by the value (R, G, B) using the RGB color system, this is expressed by the value (H, S, V). The color recognized on the three-dimensional HSV color space or the two-dimensional SV color plane can be plotted as the recognition point Q by replacing with the HSV color system.

<<< §4. Second embodiment of the present invention >>>
Subsequently, here, a second embodiment will be described in which some modifications are added to the first embodiment described so far. In the first embodiment, the color recognition unit 143 recognizes the color of the target image, and the process of determining the priority for each category is performed based on the recognized color. In the second embodiment described here, the complexity of the image (here, Complexity is represented by C by taking the acronym for Complexity) is recognized together with the color, and based on the recognized color and the complexity C. A process for determining the priority for each category is performed.

  FIG. 15 is a block diagram showing a configuration of a content display device 100A according to the second embodiment. The only difference from the content display device 100 according to the first embodiment shown in FIG. 1 is the configuration of the sub-content presentation processing unit 140A. First, in the sub-content presentation processing unit 140A shown in FIG. 15, a complexity recognition unit 147 is added as a new component. Further, the priority assignment unit 144A and the related information storage unit 145A shown in FIG. 15 have slightly different functions from the priority assignment unit 144 and the related information storage unit 145 shown in FIG. Hereinafter, only these differences will be described, and description of the remaining components will be omitted.

  First, the complexity recognition unit 147 is a component that performs processing for recognizing the complexity C for the target image. For example, as illustrated in FIG. 4, when the user designates an image 152 as a target image and requests auxiliary information regarding the target image 152, the color recognition unit 143 recognizes the color of the target image 152. At the same time, the complexity recognition unit 147 recognizes the complexity C of the target image 152. Then, the priority assigning unit 144A gives priority to the auxiliary information searched by the search processing unit 142 based on both the color recognized for the target image 152 and the complexity C. .

  For this purpose, the relevance information storage unit 145A stores relevance information indicating the relevance between the individual categories and the complexity C as well as the relevance between the individual categories and the colors. Based on both the color recognized by the color recognizing unit 143 and the complexity C recognized by the complexity recognizing unit 147, the relevance information is referred to and extracted from the sub-content data belonging to the category having higher relevance A process for assigning a higher priority to the auxiliary information is performed.

  Here, the complexity C recognized by the complexity recognition unit 147 means the complexity of the pattern expressed on the image, the frequency of occurrence of shades and colors. In order to obtain such a complexity C as a quantitative value, the complexity recognition unit 147 has a function of detecting the spatial frequency of the target image, and a value corresponding to the detected spatial frequency value is set as the complexity. It can be recognized as C.

  In general, for a raster image composed of an aggregate of a large number of pixels, the spatial frequency can be obtained by a known method. Specifically, if the fluctuation of the pixel value is extracted as a waveform in a predetermined direction and the frequency component of the waveform is recognized by a general signal analysis method, the spatial frequency of the image in the direction can be obtained. If one representative spatial frequency is obtained for one image, the average of the spatial frequency in the vertical direction and the spatial frequency in the horizontal direction may be taken. In the case of a color image, the spatial frequency may be obtained for the pixel values of the respective primary color components and averaged. Of course, the higher the spatial frequency, the higher the complexity C.

  Further, it is possible to define the complexity C by a simpler method for a vector image constituted by a collection of geometric figures. Specifically, the complexity recognizing unit 147 performs a process of counting points defined in advance in association with individual geometric figures for a target image configured by a collection of geometric figures, and totaling the points. The value may be recognized as complexity C.

  FIG. 16 is a plan view showing an example of a method for recognizing the complexity C based on such a principle. The example shown here is an example in which a vector image is composed of a set of figures such as a rectangle, a circle, an ellipse, a straight line, a polygonal line, and a polygon. Each figure is shown in the descriptor column of the figure. Is represented by a symbol. In the parameter column, the content of the parameter used as an argument is shown in parentheses of each descriptor.

  For example, a rectangle is represented by a descriptor “Rect (x, y, w, h)”, where the coordinates of the center point are (x, y), the width is w, and the height is h, and the circle is the center point The coordinates of (x, y) and the radius of r are represented by a descriptor “Circle (x, y, r)”. Here, the parameters indicating the color inside the figure and the color of the outline are omitted, but actually, the parameters indicating the colors (for example, the components of the three primary colors R, G, and B) for each figure. Value). A general vector image is composed of data in which descriptors as exemplified in FIG. 16 are listed.

  On the other hand, the points shown in the rightmost column in FIG. 16 are points defined for obtaining the complexity C. In the case of this example, quadrilateral, circle, and ellipse are each associated with 4 points, straight lines are associated with 2 points, and broken lines and polygons are associated with points corresponding to the number of constituent points. Therefore, if a table for associating such individual descriptors with points is prepared in the complexity recognition unit 147, descriptors included in the data constituting the corresponding image (vector image) are one by one. The complexity C can be determined as the total value of the points by performing a process of adding and totaling the points corresponding to the descriptor while checking the two.

  As described above, the complexity C in the present invention can be determined based on the spatial frequency when an image is captured as a raster image, or the number of points of a figure that is a constituent element when the image is captured as a vector image. It can also be determined based on the value. In practice, an upper limit reference value is set for the spatial frequency and the point total value, and this upper limit reference value is set to the maximum value of the complexity C, and the spatial frequency and the point total value are set to the upper limit reference value. When exceeding, it is preferable to always associate the maximum value of the complexity C. That is, the maximum complexity C is uniformly given to an image that is more than a certain degree.

  As described above, the relevance information storage unit 145A used in the second embodiment stores relevance information indicating the relevance between the individual categories and the complexity C as well as the relevance between the individual categories and the colors. Has been. FIG. 17 is a perspective view showing an example in which information defining a distribution range in the SVC three-dimensional space is used as the relevance information stored in the relevance information storage unit 145A. This SVC three-dimensional space is a space defined by taking the saturation value S, the lightness value V, and the complexity C on mutually orthogonal coordinate axes. An arbitrary point on the SVC space is a specific chromatic space. This is a point having a degree value S, a lightness value V, and a complexity C.

  In the relevance information storage unit 145A, mapping information obtained by mapping closed regions corresponding to individual categories may be stored as relevance information on the SVC space. FIG. 17 shows relations formed by mapping closed regions C1, C2, and C3 corresponding to the three categories C1, C2, and C3 set in the sub-content data storage unit 160 of FIG. An example of sex information is shown.

  After all, it can be said that this relevance information is obtained by associating distribution ranges of combinations of the saturation value S, the lightness value V, and the complexity C with respect to the individual categories C1, C2, and C3. Such relevance information is stored in the relevance information storage unit 145A, the color recognition unit 143 recognizes the representative value of saturation and the representative value of lightness for the target image, and the complexity recognition unit 147 If the complexity of the image is recognized, the priority assigning unit 144A can determine the distribution range of the combination of the saturation value S, the lightness value V, and the complexity C associated with each category C1, C2, and C3, The combination of the representative value of saturation S recognized by the recognition unit 143, the representative value of lightness V recognized by the color recognition unit 143, and the complexity C recognized by the complexity recognition unit 147 (S, V, C). And priority according to the degree of approximation of both can be given.

  More specifically, the relevance information storage unit 145A has a saturation value S on the first coordinate axis, a lightness value V on the second coordinate axis, and a brightness value V on the third coordinate axis, as in the example shown in FIG. Since mapping information obtained by mapping the closed areas C1, C2, and C3 corresponding to individual categories is stored as relevance information on the SVC three-dimensional space having the complexity C, the priority assigning unit 144A performs color recognition. The color recognized by the unit 143 and the complexity C recognized by the complexity recognizing unit 147 are represented in this SVC three-dimensional space with the representative value of saturation S, the representative value of lightness V, and the complexity C as coordinate values, respectively. A process of plotting as recognition points Q and assigning priority based on the positional relationship between the plotted recognition points Q and the individual closed regions C1, C2, and C3 mapped may be performed.

  FIG. 18 is a perspective view showing a method of defining priorities for each category using the relevance information shown in FIG. Also in the case of the second embodiment, similarly to the first embodiment described above, binary information indicating whether the recognition point Q is included in the closed regions C1 to C3 defined for each category is included in the binary information. A method of defining priority based on the above and a method of defining priority taking into account the distance L between the recognition point Q and the reference points G1 to G3 (for example, the center point of each closed region). Of course, both can be used together.

  For example, if the priority is defined based on the binary information, the recognition point Q6 shown in FIG. 18 is given priority 1 to the category C1, priority 0 to the categories C2 and C3, and recognition point Q7. With respect to, all of the categories C1 to C3 are given priority 0. On the other hand, when defining the priority taking into account the distance L, for example, the priority of the category C3 for the recognition point Q7 is determined based on the Euclidean distance L7 between the recognition point Q7 and the reference point G3. It will be.

  As described in §3-3, when practicing the present invention, with regard to color, in practical terms, focusing on the saturation S and the lightness V, by the closed region mapped on the SV two-dimensional color plane, It is sufficient to show the color distribution range associated with each category. In the second embodiment, a priority C for a category can be more appropriately evaluated by adding a complexity C to the two elements S and V of this color.

  In other words, the basic concept of the second embodiment is that, as the correlation between the color and the category is seen, the correlation between the complexity and the category will be seen. It is based on an idea. For example, information extracted from content belonging to a category such as “specialized books” or “business books” is more appropriate as auxiliary information to be presented for images with a high degree of complexity C, but conversely, images with a low degree of complexity C As supplementary information to be presented, information extracted from content belonging to categories such as “living information” and “ilustration / bean knowledge” is considered more appropriate. As described above, the advantage of the second embodiment is that it is possible to select and present more appropriate auxiliary information by considering not only the color but also the complexity.

<<< §5. Third embodiment of the present invention >>>
In §4 mentioned above, the form which determines the priority with respect to each category based on the color and complexity of a target image as 2nd Embodiment was described. Also in the third embodiment described here, the color and complexity of the target image are recognized. However, in the third embodiment, the priority for each category is determined based only on the color, and the complexity is used to set different priorities between sub-contents in the same category.

  FIG. 19 is a block diagram showing a configuration of a content display device 100B according to the third embodiment. The only difference from the content display device 100 according to the first embodiment shown in FIG. 1 is the configuration of the sub-content presentation processing unit 140B and the configuration of the sub-content data storage unit 160B.

  The sub-content data storage unit 160B is basically a component that stores a plurality of sub-content data S1 to S8 belonging to different categories C1 to C3, as in the above-described embodiments. A predetermined complexity C is set in each of the sub-content data S1 to S8 stored here. In FIG. 19, the circled “duplicate” marks shown in the blocks of the sub-content data S1 to S8 indicate the complexity C set for each.

  On the other hand, a complexity recognition unit 147 is added to the sub-content presentation processing unit 140B shown in FIG. The complexity recognition unit 147 is the same component as the complexity recognition unit 147 shown in FIG. 15, and has a function of performing processing for recognizing the complexity C for the target image. The specific method for recognizing the complexity C is as already described in §4, and thus the description thereof is omitted here.

  The relevance information storage unit 145 in the sub-content presentation processing unit 140B shown in FIG. 19 is the same component as the relevance information storage unit 145 shown in FIG. 1, and the relationship indicating the relationship between individual categories and colors. It has a function to store sex information. The relevance information storage unit 145A shown in FIG. 15 stores relevance information indicating the relevance between individual categories, colors and complexity as shown in FIG. 17, but it is shown in FIG. The relevance information storage unit 145 does not store information indicating the relevance related to the complexity C (the information related to the complexity C is stored in each sub content data in the sub content data storage unit 160B as described above. Is set.)

  On the other hand, the priority assigning unit 144B unique to the third embodiment basically refers to the relevance information in the relevance information storage unit 145 based on the color recognized by the color recognizing unit 143, and more A process of giving higher priority to auxiliary information extracted from sub-content data belonging to a highly relevant category is performed, but the complexity approximated by the complexity C recognized by the complexity recognition unit 147 is further increased. The auxiliary information extracted from the set sub-content data also has a function of performing a process of assigning a higher priority.

  In short, in the third embodiment, a predetermined priority is assigned to each category based on the color recognized by the color recognition unit 143, and the complexity C recognized by the complexity recognition unit 147 is added. Based on this, a predetermined priority is directly given to each sub-content data. For this reason, even if the sub-content data belongs to the same category, different priorities are given based on the set complexity.

  Hereinafter, as a more specific example, five categories C1 to C5 as shown in FIG. 6 are defined, and closed regions C1 to C5 corresponding to these categories are arranged on a two-dimensional SV color plane as shown in FIG. The third embodiment will be described with respect to an example in which the relevance information is configured by arranging and stored in the relevance information storage unit 145. FIG. 20 is a perspective view showing a priority determination method by the priority assigning unit 144B in such a specific example. The space shown in FIG. 20 is a three-dimensional coordinate space in which the saturation S, the lightness V, and the complexity C are taken as coordinate axes, as in FIG. 17, but is stored in the relevance information storage unit 145. The information is only information on the two-dimensional SV color plane (that is, information shown in FIG. 9). As described above, the relevance information does not include information on the complexity C.

  As shown in FIG. 19, information on the color recognized from the color recognition unit 143 (in this example, a representative value of saturation S and a representative value of lightness V) is given to the priority assigning unit 144B, and the complexity The complexity C recognized from the recognition unit is given. When these pieces of information (S, V, C) are plotted in the coordinate space of FIG. 20, a recognition point Q8 as shown is obtained. However, since the relevance information is only defined as closed regions C1 to C5 on the two-dimensional SV color plane, the priority given to each category is eventually the two-dimensional SV color described in §3-3. Similar to the embodiment in which the closed region is mapped on the plane, the determination is made based on the information of the closed regions C1 to C5 on the SV color plane.

  That is, the priority for each of the categories C1 to C5 is determined by the method described in §3-3 based on the recognition point Q9 obtained by orthographic projection of the recognition point Q8 on the SV color plane. For example, if the priority is defined on the basis of binary information indicating whether it is included or not included in the closed region, for the recognition point Q9 shown in FIG. 20, the category C4 has a priority 1, and the categories C1 to C3. , C5 is given priority 0. Of course, the priority can be determined in consideration of the distance between the recognition point Q9 and each of the reference points G1 to G5.

  In any case, since the priority determined based on the recognition point Q9 on the SV color plane is a priority for each category, the same priority is given to sub-content data belonging to the same category. A degree will be given. On the other hand, the priority assigning unit 144B is also provided with information on the complexity C, and the recognition point Q8 taking into account the complexity C is a distance L8 from the recognition point Q9 on the SV color plane as shown in the figure. (Ie, the value of complexity C). In the case of the third embodiment described here, this complexity C is used to give priority according to the set complexity to each sub-content data belonging to the same category. Processing is performed.

  Here, for convenience of explanation, it is assumed that the sub content data storage unit 160B stores a total of 13 sets of sub content data S1 to S13 belonging to five categories C1 to C5 as shown in FIG. Assume that the complexity shown in FIG. 21 is set for the sub-content data S9 to S11 belonging to the category C4 (in FIG. 19, as indicated by a circle with a “double” character, A predetermined complexity C is set for each sub-content data).

  Here, the degree of complexity C for each sub-content data may be appropriately set by the designer of this apparatus according to the complexity of the contents of each content. For example, in the case of the example shown in FIG. 21, a low complexity of C = 0.1 is set for information for children “S9: Encyclopedia of children's life”, and information for adults “S10: stroll in Kyoto” is set. Has a relatively high complexity of C = 0.8, and a medium complexity of C = 0.5 is set for general information “S11: Clean reading”.

  Therefore, even if the auxiliary information is extracted from the sub-content data S9 to S11 belonging to the same category “C4: life information”, different priorities are assigned based on the complexity C recognized for the target image. Will be. That is, higher priority is given to the auxiliary information extracted from the sub-content data in which the complexity approximated by the recognized complexity C is set. For example, when the target image is a very simple image, the priority of auxiliary information extracted from “S9: Child Life Encyclopedia” is high, and when the target image is a complex image, The priority of the auxiliary information extracted from “S10: Kyoto walk” becomes high.

  Specifically, for example, based on the difference between the complexity set in advance in each sub-content data and the complexity C recognized by the complexity recognition unit 147, a higher priority is given as the difference is smaller. You can do it. In this case, practically, the setting is made so that the complexity distribution range preset for each sub-content data and the distribution range of the complexity C recognized by the complexity recognition unit 147 are substantially the same. It is preferable to do this. For example, the distribution range is set to be in the range of 0 to 1.0, and priority 1 is assigned to sub-contents whose difference is within 0.1 and sub-contents whose difference is within 0.3. If it is decided that a priority of 0.5 is assigned to content, and the complexity C recognized by the complexity recognition unit 147 is 0.2, “S9: Child” shown in FIG. “Encyclopedia of Life” has a priority of 1 (difference is 0.1), “S10: Walk in Kyoto” has a priority of 0 (difference is 0.6), and “S11: Cleanup Reader” has a priority of 0.5 (difference 0.3) is given.

  Of course, since each category is given priority based on the color recognized by the color recognition unit 143, eventually, the individual auxiliary information searched by the search processing unit 142 includes color. Both the priority based on the priority and the priority based on the complexity are given. For example, in the above example, since the recognition point Q9 is included in the closed region C4, if priority 1 based on color is given to the category “C4: life information”, “S9: Child Life Encyclopedia” The extracted auxiliary information is given a total priority of 2, and the auxiliary information extracted from “S10: Kyoto walk” is given a total priority of 1, and the extracted auxiliary information is extracted from “S11: clean reading”. The information is given a total priority of 1.5.

  As described above, also in the case of the third embodiment, since priority is given in consideration of not only color but also complexity, it is possible to select and present more appropriate auxiliary information.

<<< §6. Content display method according to the present invention >>
Here, a procedure of the method when the present invention is grasped as a content display method will be described. FIG. 22 is a flowchart showing the basic procedure of the content display method according to the present invention. This method is a content display method for displaying the content of the main content including the image, searching auxiliary information related to the image in the main content being displayed from the sub content, and presenting the searched auxiliary information. As shown in the figure, each stage of Steps N1 to N7 is configured. Each of these steps is executed by a computer.

  First, in step N1, a main content display stage is performed in which the computer displays a predetermined portion of the main content on the screen of the display based on a user instruction. Specifically, for example, as illustrated in FIG. 2, the main content is displayed in the main content display area 150M.

  In the subsequent step N2, the computer recognizes a specific image in the main content displayed on the screen of the display as a target image based on a user's instruction, and a search character string attached to the target image. An object recognition stage for recognizing is performed. For example, as illustrated in FIG. 4, when the user performs an input specifying a specific image 152 in the main content with the finger F, the image 152 is recognized as a target image, and as illustrated in FIG. 3. The search character string K (the search character string attached to the image 152) is recognized.

  In the next step N3, the computer executes a search process for a plurality of sub-content data belonging to different categories using the recognized search character string K as a search key, and extracts the hit auxiliary information. Stages are performed. In step N4, a color recognition stage for recognizing the color of the target image 152 is performed. Further, in step N5, a complexity recognition stage for recognizing the complexity C for the target image 152 is performed. Note that this step N5 is a stage necessary when the second embodiment described in §4 and the third embodiment described in §5 are implemented, and the first embodiment described in §1 is the same as step N5. No need to do.

  In the subsequent step N6, the computer refers to the relevance information indicating the relevance between the individual categories and colors defined in advance, and adds the auxiliary information extracted from the sub-content data belonging to the more relevant category. A priority giving step for giving higher priority is executed.

  However, in the case where the second embodiment described in §4 is implemented, in this step N6, the computer associates the predefined categories with the colors and the relationships between the individual categories and the complexity. A process of giving higher priority to auxiliary information extracted from sub-content data belonging to a more relevant category is performed with reference to relevance information indicating sex.

  When implementing the third embodiment described in §5, in this step N6, the computer refers to the complexity of each sub-content data set in advance, and the complexity recognition stage in step N5. A process of assigning a higher priority to the auxiliary information extracted from the sub-content data in which the complexity approximated by the complexity recognized in (2) is set is performed.

  In the final step N7, an auxiliary information presentation stage is performed in which the computer presents the extracted auxiliary information based on the priority order corresponding to the assigned priority.

<<< §7. Other variations >>
As mentioned above, although this invention was demonstrated based on some embodiment, finally the some modification of this invention is described.

<7-1: Form of auxiliary information>
In the embodiment shown in FIG. 4 and FIG. 5, an example in which information configured by text data is used as the auxiliary information 154 to 156 is shown, but the auxiliary information does not necessarily need to be information configured by text data. May be information composed of image data, moving image data, and audio data (in this case, the sub-content data S is composed of image data, moving image data, and audio data, and some sort of search index) To be added.) When the auxiliary information is composed only of audio data, it is not displayed on the screen of the display 150, but is presented as sound from a speaker or the like, so it is necessary to display the sub-content display area 150S. There is no.

<7-2: Giving priority based on search match>
In the present invention, first, the search processing unit 142 performs a search using the search character string K as a search key, and extracts auxiliary information from the sub-content data. Normally, in such a search using the search character string K as a search key, auxiliary information whose degree of match with the search key is a predetermined standard or more is extracted as hit auxiliary information.

  For example, when five words are set as the search character string K as in the example shown in FIG. 3, a search is performed using these five words as a search key, and therefore only one word is matched. Some auxiliary information is matched, and some auxiliary information is matched for a plurality of words. Then, it can be extracted as auxiliary information that is hit when even one word is matched, or it can be extracted as auxiliary information that is hit only when two or more words are matched. In this case, the number of matched words can be grasped as the degree of matching.

  As described above, when the search is performed in such a manner that the degree of match can be grasped, the priority assigning unit 144 can also give a higher priority to auxiliary information having a high degree of match at the time of search processing. . That is, in addition to the priority based on the color and the complexity, a priority based on the matching degree at the time of search is added. Of course, when each search character string K is weighted, the search processing unit 142 can perform a search in consideration of the weighting, and can give priority to the hit auxiliary information based on the weighting. is there.

<7-3: Difficulty level setting by user>
In the second embodiment described in §4 and the third embodiment described in §5, the priority is given in consideration of the complexity C, and thus the operation in which the complexity C is taken into consideration is performed. In the case of the form, the instruction input unit 120 is provided with a function of inputting a user instruction for setting a desired difficulty level for the auxiliary information to be presented, and auxiliary information corresponding to the difficulty level according to the user's desire is provided. It is also possible to present it with priority.

  For example, when the user is an adult with knowledge, the difficulty level setting indicating “difficult” is set in advance in the instruction input unit 120. When the user is a child, the instruction input is performed. The difficulty level setting indicating “easy” is set in advance for the unit 120. These difficulty level settings are information indicating the difficulty level desired by the user with respect to the presented auxiliary information, and the difficulty level settings indicating “difficulty” are difficult contents that require advanced knowledge by the user. The difficulty level setting indicating that the auxiliary information is desired and indicating “easy” indicates that the user desires auxiliary information with easy contents that does not require advanced knowledge.

  And in the case of 2nd Embodiment, 144 A of priority provision parts are added to the auxiliary information extracted from the subcontent data which belongs to the category with higher complexity, so that the set difficulty level has shown difficulty. To give higher priority. Further, in the case of the third embodiment, the priority assigning unit 144B displays the auxiliary information extracted from the sub-content data set with a higher complexity as the set difficulty level indicates difficulty. To give higher priority. As a result, auxiliary information with higher complexity is preferentially presented to the adult user of the above example, and auxiliary information with lower complexity is provided to the user of the upper example. Information is presented preferentially. That is, appropriate auxiliary information according to the degree of difficulty desired by the user can be preferentially presented.

100: Content display device 100A according to the first embodiment of the present invention: Content display device 100B according to the second embodiment of the present invention: Content display device 110 according to the third embodiment of the present invention: Main content data Storage unit 120: Instruction input unit 130: Main content display processing unit 140: Sub content presentation processing unit 141: Search character string recognition unit 142: Search processing unit 143: Color recognition units 144, 144A, 144B: Priority giving unit 145 145A: Relevance information storage unit 146: Auxiliary information presentation processing unit 147: Complexity recognition unit 150: Display 150M: Main content display area 150S: Sub content display area 151 1: Text body constituting main content 152: Main content Constituent image (target image)
153: Voice presentation buttons 154 to 156: Auxiliary information (sub-contents)
160: Sub-content data storage unit A: Audio data C: Circumference C1-C5: Closed area / closed section E1-E5 indicating the category / color distribution range of the sub-content data F: User's finger G1 -G5: Reference point (centroid point) of each closed region
H: Hue value K: Search character string L3 to L8: Euclidean distance between two points M1 to M3: Main content data N1 to N7: Steps in flowchart P: Image data P1 to P8: Three-dimensional color space Points Q, Q1 to Q9: Recognition point r: Circumference radius S: Saturation value S1 to S13: Sub-content data T: Text data V: Lightness value

Claims (26)

A content display device having a function of displaying contents of main content including an image, searching auxiliary information related to an image in the main content being displayed from sub-contents, and presenting the searched auxiliary information ,
A main content data storage unit for storing main content data including image data to which a search character string is attached;
A sub content data storage unit for storing a plurality of sub content data belonging to different categories,
An instruction input unit for inputting user instructions;
A display that displays the content, and
A main content display processing unit that displays a predetermined portion of main content data stored in the main content data storage unit as main content on the screen of the display based on an instruction from the user;
Based on the user's instruction, a search process for searching auxiliary information related to a specific target image in the main content displayed on the display screen from the plurality of sub-content data is executed, and the result of the search process A sub-contents presentation processing unit that presents the auxiliary information obtained as sub-contents,
With
The sub-content presentation processing unit
A search character string recognition unit for recognizing the search character string attached to the target image;
A search processing unit that performs a search process on the plurality of sub-content data using the recognized search character string as a search key, and extracts auxiliary information that has been hit;
A relevance information storage unit that stores relevance information indicating the relevance between individual categories and colors;
A color recognition unit for recognizing a color of the target image;
A priority assigning unit that refers to the relevance information based on the color recognized by the color recognizing unit and gives higher priority to auxiliary information extracted from sub-content data belonging to a more relevant category When,
An auxiliary information presentation processing unit for presenting the extracted auxiliary information based on the priority order according to the given priority;
A content display device comprising: The content display device according to claim 1,
The relevance information storage unit stores relevance information that associates the distribution range of any one of hue value, saturation value, and lightness value or the combination range of these values for each category. And
The color recognition unit recognizes the representative color for the target image as one representative value of a hue value, a saturation value, and a lightness value or a combination thereof,
The priority assigning unit has a hue range, saturation value, brightness value distribution range associated with each category, or a combination range of these values, and a hue value, saturation value, A content display device characterized by comparing a representative value of brightness values or a combination thereof and assigning a priority according to the degree of approximation between the two. The content display device according to claim 2,
The relevance information storage unit includes relevance information that associates the distribution range of combinations of saturation value, lightness value, and hue value for each category,
The color recognition unit recognizes the representative value of saturation, the representative value of brightness, and the representative value of hue for the target image,
The priority assigning unit has a distribution range of combinations of saturation values, lightness values, and hue values associated with individual categories, and the representative value of saturation, the representative value of lightness, and the representative of hue recognized by the color recognition unit A content display device that compares a combination of values and gives a priority according to the degree of approximation between the two. The content display device according to claim 3,
The relevance information storage unit corresponds to each category in a three-dimensional color space in which a position on a reference coordinate axis is a brightness value, a distance from the reference coordinate axis is a saturation value, and an angle around the reference coordinate axis is a hue value. Store the mapping information mapping the closed area as relevance information,
The priority assigning unit plots the color recognized by the color recognition unit on the three-dimensional color space as a recognition point having a representative value of saturation, a representative value of brightness, and a representative value of hue as coordinate values, A content display device, wherein a priority is given based on a positional relationship between a plotted recognition point and each mapped closed region. The content display device according to claim 2,
The relevance information storage unit includes relevance information that associates distribution ranges of combinations of saturation values and lightness values for each category,
The color recognition unit recognizes a representative value of saturation and a representative value of brightness for the target image,
The priority assigning unit determines the distribution range of the combination of the saturation value and the lightness value associated with each category, and the combination of the representative value of the saturation and the representative value of the lightness recognized by the color recognition unit. A content display device that compares and assigns a priority according to the degree of approximation between the two. The content display device according to claim 5,
Relevance information storage unit uses mapping information obtained by mapping closed areas corresponding to individual categories on a two-dimensional color plane having a saturation value on the first coordinate axis and a lightness value on the second coordinate axis. Store and
The priority assigning unit plots the color recognized by the color recognition unit on the two-dimensional color plane as a recognition point having a representative value of saturation and a representative value of brightness as coordinate values, respectively, and plotted recognition A content display device characterized in that a priority is given based on a positional relationship between a point and each mapped closed region. In the content display apparatus in any one of Claims 2-6,
The color recognition unit recognizes the representative color of the target image as a weighted average value considering the hue value, saturation value, or brightness value area of each unit constituting the target image, or a combination thereof. Content display device. The content display device according to claim 1,
The secondary content presentation processing unit further includes a complexity recognition unit that recognizes the complexity of the target image,
The relevance information storage unit stores relevance information indicating the relevance between individual categories and color as well as the relevance between individual categories and colors.
A sub-content belonging to a category with higher relevance by the priority giving unit referring to the relevance information based on both the color recognized by the color recognizing unit and the complexity recognized by the complexity recognizing unit A content display device, wherein higher priority is given to auxiliary information extracted from data. The content display device according to claim 8,
The relevance information storage unit stores relevance information that associates the distribution range of the combination of saturation value, lightness value, and complexity for each category,
The color recognition unit recognizes a representative value of saturation and a representative value of brightness for the target image,
The priority assigning unit has a distribution range of combinations of saturation value, lightness value, and complexity associated with each category, a representative value of saturation recognized by the color recognition unit, and a lightness value recognized by the color recognition unit. A content display device that compares a representative value and a combination of complexity recognized by a complexity recognition unit and assigns a priority according to the degree of approximation of both. The content display device according to claim 9,
The relevance information storage unit maps closed regions corresponding to individual categories on a three-dimensional space having a saturation value on the first coordinate axis, a lightness value on the second coordinate axis, and a complexity on the third coordinate axis. Store mapping information as relevance information,
The priority assigning unit displays the color recognized by the color recognizing unit and the complexity recognized by the complexity recognizing unit on the three-dimensional space as a representative value of saturation, a representative value of lightness, and a complexity as coordinate values, respectively. A content display device characterized in that a priority is given based on the positional relationship between the plotted recognition points and the mapped individual closed regions. The content display device according to claim 4,
The priority assigning unit determines whether or not the closed region corresponding to each category includes a recognition point, and the auxiliary information extracted from the sub-content data belonging to the category corresponding to the closed region including the recognition point A content display device characterized by giving higher priority. The content display device according to any one of claims 4, 6, 10, and 11,
A sub-content belonging to a category corresponding to a closed region with a small mutual distance by the priority assigning unit calculating a Euclidean distance between a predetermined reference point defined for the closed region corresponding to each category and a recognition point. A content display device, wherein higher priority is given to auxiliary information extracted from data. The content display device according to claim 12,
A content display apparatus, wherein the priority assigning unit defines the center of gravity of each closed region as a reference point. The content display device according to claim 1,
The secondary content presentation processing unit further includes a complexity recognition unit that recognizes the complexity of the target image,
Each sub-content data stored in the sub-content data storage unit is set with a predetermined complexity,
A content display in which the priority assigning unit assigns a higher priority to the auxiliary information extracted from the sub-content data in which the complexity approximated by the complexity recognized by the complexity recognizing unit is set. apparatus. In the content display apparatus in any one of Claims 8-10, 14,
A content display device, wherein the complexity recognition unit has a function of detecting a spatial frequency of a target image, and recognizes a value corresponding to the detected spatial frequency value as a complexity. In the content display apparatus in any one of Claims 8-10, 14,
The complexity recognition unit has the function of counting points defined in advance in association with individual geometric figures for the target image composed of a collection of geometric figures. A content display device characterized by recognition. In the content display apparatus in any one of Claims 8-10 and 14-16,
The instruction input unit has a function of inputting a user instruction to set a desired difficulty level for the auxiliary information to be presented,
In the auxiliary information extracted from the sub-content data belonging to the category with higher complexity or the sub-content data set with higher complexity, the priority assigning unit indicates the difficulty level, A content display device characterized by giving a higher priority. The content display device according to any one of claims 1 to 17,
The search processing unit executes a search process for extracting auxiliary information whose degree of match with the search key is equal to or higher than a predetermined reference as hit auxiliary information,
A content display device, wherein a priority assigning unit assigns a higher priority to the auxiliary information having a high degree of match. The content display device according to any one of claims 1 to 18,
The auxiliary information presentation processing unit presents only auxiliary information within the top nth (n is a predetermined natural number) based on the priority among the auxiliary information extracted by the search processing unit. Content display device. The content display device according to claim 19,
The search processing unit executes the search processing for the sub-content data in the order from the highly relevant category to the less relevant category, and stops the subsequent search processing when the top nth auxiliary information is extracted. A content display device characterized by the above. The content display device according to any one of claims 1 to 20,
When the auxiliary information presentation processing unit has a plurality of auxiliary information to be presented, the content display device presents the plurality of auxiliary information according to an arrangement order or a time order based on the priority order. . The content display device according to any one of claims 1 to 21,
The instruction input unit has a function of inputting a user instruction to specify an instruction point on the display screen,
A content display apparatus, wherein the sub-content presentation processing unit recognizes an image displayed in the vicinity of the designated point as a target image and presents auxiliary information related to the target image.   A program causing a computer to function as the content display device according to any one of claims 1 to 22. A content display method for displaying contents of main content including an image, searching auxiliary information related to an image in the main content being displayed from sub-contents, and presenting the searched auxiliary information,
A main content display stage in which a computer displays a predetermined portion of the main content on a display screen based on a user instruction;
The computer recognizes a specific image in the main content displayed on the screen of the display as a target image and recognizes a search character string attached to the target image based on a user instruction The recognition stage;
A search processing stage in which the computer executes a search process for a plurality of sub-content data belonging to different categories, using the recognized search character string as a search key, and extracts the hit auxiliary information;
A color recognition stage in which a computer recognizes a color of the target image;
The computer refers to the relevance information that indicates the relevance between individual categories and colors, and the auxiliary information extracted from the sub-content data belonging to the more relevant category has a higher priority. A priority granting stage for granting
An auxiliary information presentation stage in which the computer presents the extracted auxiliary information based on the priority order according to the given priority;
A content display method characterized by comprising: The content display method according to claim 24, wherein
The computer further comprises a complexity recognition stage for recognizing the complexity of the target image;
In the prioritization stage, the computer looks at the relevance information that shows the association between each predefined category and color, and the association between each category and complexity, and makes the category more relevant A content display method, wherein higher priority is given to auxiliary information extracted from sub-content data to which the user belongs. The content display method according to claim 24, wherein
The computer further comprises a complexity recognition stage for recognizing the complexity of the target image;
In the priority assignment stage, the computer refers to the complexity of each sub-content data set in advance, and from the sub-content data in which the complexity approximated by the complexity recognized in the complexity recognition stage is set A content display method, wherein higher priority is given to extracted auxiliary information.

Images