JP4061379B2 - Information processing apparatus, portable terminal, information processing method, information processing program, and computer-readable recording medium - Google Patents

Description

  The present invention relates to augmented reality, and is an information processing apparatus, a portable terminal, an information processing method, an information processing program, and a computer-readable recording capable of adaptively presenting annotation information of an object of interest to a user It relates to the medium.

  Augmented Reality (Augmented Reality) is a technology that adds information to the real world by overlaying or presenting annotation information such as text or CG on the real world object. As a result, the user can obtain more information than the information of the appearance of the object. Since the purpose of the AR system is to present information on an object that the user is interested in, it is necessary to recognize the object that the user is paying attention to. In existing systems, object recognition methods can be broadly classified into methods for tagging for object recognition and methods for not performing object recognition, and research is actively conducted by each method.

A method of attaching a tag to an object, recognizing the object by reading the tag, and presenting associated information (see Non-Patent Documents 1 to 4) attaches a two-dimensional barcode, RFID tag, or infrared LED to the object. There have been proposed a method (see Non-Patent Documents 1 to 3) and a method of projecting a tag with infrared rays instead of attaching a physical tag (see Non-Patent Document 4). In these methods, in order to present annotation information, a user needs an explicit operation of reading a tag with a device such as a tag reader or a camera. On the other hand, in the method of recognizing an object without tagging and presenting the associated information (see Non-Patent Documents 5 and 6), the user is prompted based on the object image acquired in advance. Recognizes objects that are reflected in the attached camera. In these methods, the recognition accuracy when the entire object is reflected in the image is 100%.
Motoki Kobayashi, Hideki Koike, “Desktop Real World Interface“ EnhancedDesk ”for Displaying and Manipulating Electronic Information”, Interactive System and Software V: Japan Software Science Society WISS1997. Ichiro Shiio, Toshiyuki Masui, Kentaro Fukuchi, “Real World Interaction with FieldMouse”, Interactive System and Software VII: Japan Software Science Society WISS1999. Tsuyoshi Aoki, “Infrared tags read by cameras and their applications”, Interactive System and Software VIII: Japan Software Science Society WISS2000. Yoshinari Shirai, Mitsunori Matsushita, Satoshi Oguro, “Secret Projector: Expansion of Real Environment by Invisible Information”, Interactive System and Software XI: Japan Software Science Society WISS2003. T. Kurata, T. Okuma, M. Kourogi, T. Kato, and K. Sakaue, “VizWear: Toward HumanCentered Interaction through Wearable Vision and Visualization”, The Second IEEE PacificRim Conference on Multimedia, 2001. T. Jebara, B. Schiele, N. Oliver, A. Pentland, “DyPERS: Dynamic Personal Enhanced Reality System”, MIT Media Lab. Perceptual Computing Section Technical Report, No. 468, 1998. Mitsuo Ikeda, “What does the eye see?”, Heibonsha, 1998.

  However, in the prior art, when presenting information related to an object, since it is not determined whether or not the user is interested in the object, information unnecessary for the user may be presented. Further, since only object recognition is performed, it is impossible to detect which part of the object is focused.

  The present invention has been made in view of the above-described conventional problems, and is an information processing apparatus, a portable terminal, an information processing method, and an information processing program capable of accurately presenting annotation information regarding an object that the user is paying attention to. And a computer-readable recording medium.

In order to solve the above problem, an information processing apparatus according to the present invention includes an eyeball motion detection unit that detects a state in which a user is paying attention to an object, based on an eyeball image obtained by capturing the user's eyeball movement, and the eyeball motion When the detection means detects that the user is in the state of interest, the object recognition means for recognizing the object being noticed by the user in the view image obtained by photographing the user's view, and the user is paying attention to the object. a region of interest determining unit that determines a region of interest, that have an annotation information presenting means for presenting the annotation information about the determined region of interest by the target region determining means to the user.

  According to the above configuration, since the object that the user is interested in is recognized by the object recognizing means after the eye movement detecting means detects that the user is in the attention state, the user is truly interested. Can recognize the existing object.

  In addition, the attention area discriminating means discriminates the attention area that the user is paying attention to in the object, so that it is possible to grasp which part of the object the user is paying attention to. And since the annotation information presentation means presents information on the region of interest to the user, it is possible to accurately present annotation information about the region in which the user is truly interested to the user, unnecessary information. Can be prevented from being presented to the user.

  As described above, according to the information processing apparatus of the present invention, since the annotation information regarding the area in which the user is interested is accurately presented to the user, a comfortable use environment of the information processing apparatus of the present invention is given to the user. be able to.

In the information processing apparatus of the present invention, in order to solve the above-described problem, the eye movement detecting means includes a line-of-sight detecting means for extracting a pupil region from the eyeball image, and a gaze point from a center coordinate in the pupil region. The gazing point calculation means for calculating the moving distance and the viewing angle of the user with respect to the object are calculated from the moving distance of the gazing point, and the fixation state and the jumping state of the eyeball are detected based on the size of the viewing angle. user based on the state and the number of jumping state viewing that are provided by an attention state detecting means for detecting a state of focusing on an object.

  It is known that when a human is paying attention to something, the fixation state and the jumping state are frequently repeated. Therefore, if this eye movement feature is used, it is possible to accurately detect that the user is in the focused state.

  Therefore, in the above configuration, the eye movement detection means includes a gaze detection means, a gaze point calculation means, and an attention state detection means.

  That is, the fixation state and the jumping state can be detected by detecting the viewing angle of the user, and the viewing angle of the user can be obtained by calculating the moving distance of the gazing point.

  The information processing apparatus configured as described above extracts the pupil region using, for example, binarization processing by the line-of-sight detection unit, and calculates the movement distance of the gazing point by the gazing point calculation unit. Then, the attention state detection means detects the frequency of the fixation state and the jump state by calculating the user's viewing angle from the movement distance of the gazing point obtained by the gazing point calculation means, and thus accurately detects the user's attention state. can do.

  In this way, according to the above configuration, the user's attention state can be accurately grasped, so that the object that the user is interested in is accurately recognized by the object recognition means, and the annotation information presentation means is more accurate. Annotation information can be presented to the user.

  Furthermore, in order to solve the above-described problem, the information processing apparatus of the present invention connects the plurality of gazing points so that the attention state detection unit includes a distribution of gazing points while the user is in the attention state. And the minimum polygonal area is defined in the field-of-view image, and the attention area discriminating means has a maximum overlap with the minimum polygonal area among a plurality of areas in the registered image stored in the database. A region in the field-of-view image corresponding to the region is determined as the region of interest.

  According to the above configuration, the minimum polygonal area is defined by the attention state detection unit. Since the minimum polygonal area is obtained by connecting a plurality of gazing points so as to include the distribution of the gazing points, it can be said that the minimum polygonal area accurately represents an area that the user is paying attention to in the view field image.

  Then, the attention area determination means determines the area where the overlap with the minimum polygon is maximum for the registered image in the database, and determines the area in the view image corresponding to the area as the attention area. Can be accurately determined as a region of interest.

Furthermore, in the information processing apparatus of the present invention, in order to solve the above-described problem, the object recognition unit converts the hue in the field-of-view image and the hue of a registered image that can be a candidate for the field-of-view image stored in the database. Based on this, it is preferable to recognize an object that the user is paying attention to .

  If the visual field image is an image having a color such as, for example, a painting image, the object included in the visual field image can be characterized by the hue. Therefore, in the above configuration, the object being noticed by the user is recognized based on the hue of the field-of-view image and the hue of the registered image, so that the object recognition accuracy can be improved.

Furthermore, in the information processing apparatus of the present invention, in order to solve the above-described problem, it is preferable that the annotation information presenting unit further presents the user with annotation information regarding a region having a hierarchical structure with the region of interest. .

  According to the above configuration, the annotation information related to the region that forms a hierarchical structure with the region of interest is also presented to the user, so that the user can obtain more information about the region of interest and deepen their understanding of the region of interest.

  In addition, by mounting one or both of the first camera for capturing the eyeball image and the second camera for capturing the field image on the mobile terminal, the eyeball image and field image can be captured even when the user moves. Therefore, the convenience of the information processing apparatus of the present invention can be further enhanced.

  Furthermore, since the annotation information can be presented to the moving user by reproducing the annotation information using the portable terminal, the convenience of the information processing apparatus of the present invention can be further improved.

In order to solve the above problem, the information processing method of the present invention includes a first step of detecting a state in which the user is paying attention to the object based on an eyeball photographed image of the user's eye movement; When it is detected that the user is in an attention state in one step, the second step of recognizing the object that the user is paying attention in the view image obtained by capturing the user's view, and the user pays attention to the object. a third step of determining a region of interest, and a fourth step of presenting the user with annotation information about the determined region of interest in the third step, the first step, a pupil region from the eyeball captured image A gaze detection step for extracting, a gaze point calculating step for calculating a moving distance of the gaze point from center coordinates in the pupil region, and the gaze The viewing angle of the user with respect to the object is calculated from the moving distance of the object, and the fixation state and the jumping state of the eyeball are detected based on the size of the viewing angle, and the user pays attention to the object based on the number of the fixation state and the jumping state. An attention state detection step for detecting a state of being connected, and the attention state detection step further includes connecting a plurality of attention points so as to include a distribution of attention points while the user is in the attention state, A step of demarcating a minimum polygonal area in the view image, wherein the third step overlaps the minimum polygonal area among a plurality of areas in a registered image stored in the database as a candidate for the viewable image. Is a step of determining a region in which the maximum value of the field of view is maximized, and determining a region in the view field image corresponding to the region as the region of interest It is characterized by a door.

  According to the information processing method having the above configuration, it is possible to obtain the same operational effects as those of the information processing apparatus of the present invention.

  In addition, the effect similar to the information processing method of this invention can be obtained using a computer with the information processing program which makes a computer perform each step in the said information processing method. Furthermore, the information processing program can be executed on any computer by storing the information processing program in a computer-readable recording medium.

  According to the information processing apparatus of the present invention, if attention is focused on the object as a whole, the annotation information regarding the entire object is presented, and if attention is focused on the object, the annotation information regarding the portion is presented. It is possible to adaptively present the annotation information corresponding to the location that the user has focused on to the user.

[1. Configuration of information processing apparatus]
An information processing apparatus according to an embodiment of the present invention will be described with reference to FIGS. As shown in FIG. 1, the information processing apparatus 1 includes an eyeball movement detection unit 2, an object region extraction unit 3, an object recognition unit 4, a region of interest determination unit 5, an annotation information presentation unit 6, a database 7, It has.

  The eyeball motion detection unit 2 detects the movement of the user's line of sight from the image of the camera 17 that captures the eyeball (hereinafter referred to as an eyeball image) in order to detect that the user is paying attention to something. Is. More specifically, the eye movement detection unit 2 includes a line-of-sight detection unit 8, a gaze point calculation unit 9, and an attention state detection unit 10. The processing content in each block provided in these eyeball motion detection units 2 will be described later.

  The object area extraction unit 3 extracts an object area from an image of a camera 18 that captures a field of view (hereinafter referred to as a field of view image) when the user is paying attention to something. Various methods can be employed for the object region extraction method. Here, as an example, a horizontal direction edge detection unit 11, a vertical direction edge detection unit 12, and a hue division unit provided in the object region extraction unit 3 are used. 13 and a procedure for extracting an object region by the histogram calculation unit 14 will be described. Details of processing contents in each block provided in the object area extraction unit 3 will be described later.

  The object recognizing unit 4 performs a matching process between the image shown in the region extracted from the view field image by the object region extracting unit 3 and the registered image stored in the database 7, and selects the image that the user is paying attention to. Recognize. Note that a registered image is an image that is stored in advance in the data database and that is a candidate for a view field image.

  With respect to this matching processing method, various methods can be employed as in the object region extraction method. Here, as an example of the matching process, the hue average calculation unit 15 and the color difference calculation unit 16 provided in the object recognition unit 4 are used to compare the image extracted by the object region extraction unit 3 and the registered image in the database 7. A method of paying attention to the average value and the color difference of each hue will be described (details will be described later).

  The attention area discriminating unit 5 discriminates in more detail which area the user is paying attention to in the image recognized by the object recognition unit 4 as being noticed by the user. This attention area discriminating unit 5 is an area in the image registered in the database 7 and has an area where the overlap with the minimum convex polygon area (details will be described later) determined based on the gazing point coordinates is maximum. , It is determined that the user is paying attention. Details of the attention area determination processing will be described later.

  The annotation information presenting unit 6 reads annotation information (for example, text data, audio data, still image data, moving image data) related to the region determined to be noticed by the user by the attention region determining unit 5 from the database 7, and information This is for outputting to an output device (for example, a notebook personal computer, a head mounted display, headphones, a speaker) provided outside the processing device 1.

  The database 7 stores a registered image in which the annotation information is provided to the user, the color of the object, the coordinates of the region to which the annotation information is added, and the annotation information as data related to the object in the registered image. is doing.

  The camera 17 is a CCD camera, for example, and is installed below the eyeball. The camera 18 is a CCD camera, for example, and is installed between the eyebrows. These two cameras 17 and 18 are arranged in such a positional relationship that the eyeball when the user is looking at the front and the center of the visual field at that time are reflected in the center of each camera image.

  With the above configuration, the information processing apparatus 1 can present the annotation information of the object focused on by the user by detecting the state where the user is focused on the object from the user's eye movement. A series of processes until the annotation information is presented will be described with reference to the flowchart of FIG.

  First, as shown in FIG. 2, a state in which the user is paying attention to an object is detected by the eyeball motion detection unit 2 (step 1; hereinafter, steps are simply referred to as S). Thereafter, a view field image is acquired by the object region extraction unit 3 (S2), and an object region is extracted from the view field image (S3). Further, the image recognizing the user's attention is recognized by the object recognizing unit 4 (S4), and in the image recognized as the user's attention, which region the user is paying attention to is the attention region determining unit 5 is determined in more detail (S5). Then, the annotation information presenting unit 6 presents the annotation information regarding the area determined to be noticed by the user to the user (S6), and the presentation of the annotation information is finished after a predetermined time (S7).

  Finally, it is determined whether or not attention detection is to be resumed (S8). If it is resumed, the process returns to S1, and if it is not resumed, the series of processes is terminated. Note that the determination in S8 can be made by determining whether or not the user has input “to resume attention detection” from an input device such as a switch, keyboard, or mouse.

  The information processing apparatus 1 according to the present embodiment can accurately present the annotation information related to the area that the user is paying attention to the user with the above-described configuration and processing flow. Details of processing performed by each block in the information processing apparatus 1 will be described below.

[2. Processing in Eyeball Motion Detection Unit 2]
As described above, the eyeball motion detection unit 2 detects the movement of the user's line of sight from the eyeball photographed image in order to detect that the user is paying attention to something. In the following description, the section from the start of attention to the end of attention when the user is detected to be in the attention state is defined as “attention section”.

  Further, as described above, the eyeball motion detection unit 2 includes the line-of-sight detection unit 8, the gaze point calculation unit 9, and the attention state detection unit 10 (see FIG. 1). And a gaze point coordinate corresponding to the line of sight is calculated by the gaze point calculation unit 9. Note that the gaze point coordinate is a point (hereinafter referred to as a gaze point) at which the user has matched his / her line of sight in the real world, and is indicated by coordinates in the view field image. Further, the spread of the gazing point in the attention section is defined as “gaze point distribution”.

[2-1. Processing in the line-of-sight detection unit 8]
Detection of the user's line of sight by the line-of-sight detection unit 8 is performed by detecting the position of the pupil. That is, in order to obtain a more accurate line-of-sight position, infrared rays are emitted from the camera 17 to the eyeball, and the contrast between the pupil and the iris is enhanced. Then, the line-of-sight detection unit 8 extracts a pupil region by distinguishing between a binarization process of an eyeball image, that is, a pupil and a portion that is not.

  Further, the line-of-sight detection unit 8 calculates the center coordinates (x (t), y (t)) of the pupil region as the position of the pupil at time t (frame). The horizontal direction of the eyeball image is set as the x axis, the vertical direction is set as the y axis, and the origin is set as the lower left of the image. Note that the various setting values in the following description are examples of the case where the image size is 160 pixels in the horizontal direction and 120 pixels in the vertical direction and acquired at 10 frames per second. When it is assumed that a camera having a characteristic different from that of the object being used is used, an optimal value can be appropriately adopted.

[2-2. Processing in the gaze point calculation unit 9]
Next, the gaze point calculation unit 9 obtains the gaze point coordinates (G _u (t), G _v (t)) by the following equation (1). In the view image, the horizontal direction is the u axis, the vertical direction is the v axis, and the origin is the lower left of the image.

Here, a partial area (40 ≦ x ≦ 120, 40 ≦ y ≦ 80) of the view field image and the eyeball image is divided into 16 blocks of 4 × 4. Also, (D (i, j) _x , D (i, j) _y ) (i, j = 1, ..., 4) are the center coordinates of the block (i, j) of the eyeball image, and (Cv (i , J) _u and Cv (i, j) _v ) are the center coordinates of the block (i, j) of the view image.

  Note that the block (i, j) is the i-th block in the horizontal direction and the j-th block in the vertical direction, counting from the lower left of the image. Further, α (i, j) and β (i, j) are values representing the ratio of the moving distance of the gazing point to the moving distance of the center of the pupil in each block. Further, the moving distance d of the center of the pupil is calculated by the gaze point calculation unit 9 in units of pixels based on the following formula (2).

[2-3. Processing in attention state detection unit 10]
Eye movements represent the attention and interest of human beings, and when humans are paying attention to stationary objects, they often repeat a fixation state of about 300 milliseconds and a jumping movement that occurs in about 30 milliseconds. It is known (see Non-Patent Document 7). The attention state detection unit 10 detects a state in which the user is paying attention to the object by using the feature of the eye movement.

  That is, the attention state detection unit 10 detects a state in which the position of the pupil does not move for 3 frames (about 300 milliseconds) or moves slightly, and the position of the pupil jumps in the 4th frame. The state is detected as a fixation / jumping motion. Then, the attention state detection unit 10 determines that the user is in the attention state when the fixation / jumping motion is detected three or more times continuously.

  However, if the movement of the pupil position is less than 2.1 ° in terms of viewing angle between three frames, the attention state detection unit 10 regards as fixation, and the movement of the pupil position from the fixation state is 2.1 ° or more in terms of viewing angle. If so, it is considered a jump. Further, when the next jump does not occur in 3 seconds after the jump occurs, the attention state detection unit 10 determines that the fixation / jump has ended. Note that the value of 2.1 ° is an example of a threshold value for distinguishing between fixation and jumping, and other values may be used.

  Here, the viewing angle θ is calculated based on the following equation (3).

  The numerical value “0.024” in the above formula (3) is the distance (cm) per pixel in the eyeball image. Of course, depending on the definition of the image, the value 0.024 may be another value. Further, the numerical value “2.0” in the equation (3) is the distance (cm) between the eyeball and the camera 17 for photographing the eyeball. Of course, the value of 2.0 may be changed to another value depending on the imaging condition of the eyeball.

  In addition, since each of the gaze point coordinates generated in the attention section is calculated from the position of the pupil when the user is in the attention state, it appears on or near the object that the user is paying attention to. For this reason, the attention state detection unit 10 connects a plurality of gazing point coordinates so that the gazing point distribution from one attention interval, that is, from the start of attention to the end of attention, is included, thereby reducing the minimum convexity. A square area (minimum polygon area) is defined. Accordingly, the minimum convex polygonal region includes the entire object of interest or a part thereof.

[2-4. Processing flow in eyeball motion detection unit 2]
Next, the process for detecting that the user is paying attention to something, which is executed in the eyeball movement detection unit 2, will be described with reference to the flowchart of FIG.

  First, the attention state detection unit 10 resets the number of consecutive times to 0 (S10), and also resets the number of fixations to 0 (S11). The number of continuous times indicates the number of times that the fixation / jumping motion is continued. Hereinafter, the term “number of consecutive times” is used in the same meaning.

  Thereafter, an eyeball image is acquired from the camera 17 by the line-of-sight detection unit 8 (S12). Further, the line-of-sight detection unit 8 extracts a pupil region from the eyeball image by performing binarization processing on the eyeball image acquired in S12 (S13).

Thereafter, the gazing point calculation unit 9 calculates the movement distance of the center of the pupil by the above equation (2) (S14), and the center coordinates (x (t), y (t) of the pupil region obtained by the gaze detection unit 8 )), The gaze point coordinates (G _u (t), G _v (t)) are calculated based on the above equation (1) (S15).

  Then, the attention state detection unit 10 calculates the viewing angle θ based on the above equation (3) (S16), and determines whether or not the value is 2.1 ° or more (S17). If the viewing angle θ is not 2.1 ° or more, the attention state detection unit 10 determines that the user is in a fixation state, and increases the number of fixations by one (S18). Then, when the process of S18 is completed, the process proceeds to S12 again.

  On the other hand, when it is determined in S17 that the viewing angle θ is 2.1 ° or more, the attention state detection unit 10 determines that the user's line of sight has jumped (S19), and whether or not the number of fixations so far is three or more. Is determined (S20).

  If it is determined in S20 that the number of fixations is not 3 or more, the attention state detection unit 10 resets the number of fixations to 0 (S21), and then the process proceeds to S12. If it is determined in S20 that the number of fixations is 3 or more, the attention state detection unit 10 determines that the user is in a fixation / jumping state after resetting the number of fixations to 0 (S22) ( S23).

  After S23, the attention state detection unit 10 determines whether or not the next jump occurs within 3 seconds after the last jump occurs (S24). If it is determined in S24 that the next jump has occurred within 3 seconds, the attention state detection unit 10 determines that the user's fixation / jumping motion is continuous, and increases the number of consecutive times by one (S25). ). Then, when the process of S25 is completed, the process proceeds to S12 again.

  If it is determined in S24 that the next jump has not occurred within 3 seconds, the attention state detection unit 10 determines whether the number of consecutive times is 3 or more (S26). If the number of continuous times is not 3 or more in S26, the attention state detection unit 10 resets the number of continuous times to 0 (S27). After S27, the process proceeds to S12. If the number of consecutive times is 3 or more in S26, the attention state detection unit 10 determines that the user is in the attention state (S28).

  In this way, the steps of S10 to S28 are performed, and the eye state detection processing by the eye movement detection unit 2 ends.

[3. Processing in Object Area Extraction Unit 3]
As described above, the object area extraction unit 3 extracts an object area from the view field image. In the present embodiment, a method for extracting a painting area shown in a visual field image at the time of attention will be described using paintings displayed in a museum as an example of an object.

  How the object area extraction unit 3 extracts a picture area will be briefly described. In a field of view image showing a painting, straight edges appear in the horizontal and vertical directions at the boundary between the wall and the picture frame. The object area extraction unit 3 detects the horizontal edge by the horizontal edge detection unit 11 and the vertical edge by the vertical edge detection unit 12.

  In the hue of the HSV color system, the hue changes at the boundary between the wall and the frame. The object region extraction unit 3 detects the change in hue by the hue division unit 13 and the histogram calculation unit 14 and extracts a painting region from the visual field image based on the straight edge and the boundary of the hue.

  Next, a series of processes in the object area extraction unit 3 will be described with reference to the flowchart of FIG. As shown in FIG. 4, first, one or both of the horizontal edge detection unit 11 and the vertical edge detection unit 12 generates an edge image from the grayscale image of the field-of-view image using, for example, a roberts filter. (S30). After S30, the horizontal edge detection unit 11 calculates the length of the straight edge appearing in the horizontal direction (S31). Further, after S31, the length of the straight edge appearing in the vertical direction is calculated by the vertical edge detection unit 12 (S32).

  In addition, S32 does not necessarily need to be performed after S31, S32 may be performed before S31, and the process of S31 and the process of S32 may be performed in parallel. Details of the processes in S31 and S32 will be described later.

  Further, the horizontal direction edge detection unit 11 detects a longer edge from adjacent edges in the horizontal direction and sets it as a frame edge candidate. Further, the vertical edge detection unit 12 detects a longer edge from adjacent edges in the vertical direction and sets it as a frame edge candidate. In the flowchart of FIG. 4, the edge candidate detection processing by the horizontal edge detection unit 11 and the vertical edge detection unit 12 is collectively described as S <b> 33.

  Thereafter, the hue division unit 13 divides the hue of the edge image into seven grades (S34). Then, the histogram calculator 14 calculates a hue histogram from the frame edge candidates (S35). Details of the processes in S34 and S35 will be described later.

Then, for every frame of edge candidates, the histogram calculating section 14, with respect to both sides of the region of the edge candidates, a difference CS _e histogram of the hue, it is calculated by the following equation (4) (S36).

Note that hist _ln is a hue histogram in a region shifted by one pixel to the left (or top) with respect to the frame edge candidate, and hist _rn is right (or down) with the frame edge candidate as a boundary. It is a hue histogram in a region shifted by one pixel. Note that n is an integer from 1 to 7, indicating the hue grade. It has been confirmed by experiments that the hue is roughly divided at the boundary of the wall, the frame, and the picture shown in the view image when the hue grade number is 7.

When the calculation process of CS _e according to Equation (4) is completed, the horizontal edge detection unit 11 detects two edges in the horizontal direction that have a higher CS _e magnitude (S37). Further, the vertical edge detection unit 12 detects two edges in the vertical direction whose CS _e is higher in size (S37). Then, the object area extraction unit 3 extracts the area surrounded by the four edges detected in S37 as a painting area (S38).

  By following the series of steps S30 to S38 described above, the object area extraction unit 3 extracts a picture area as an object area from the view field image.

[3-1. (Calculation process of horizontal edge length)
Next, details of the process of S31 in the flow of FIG. 4 described above will be described. For example, assume that a field-of-view image as shown in FIG. In this case, as shown in FIG. 6, the horizontal direction edge detection unit 11 performs the following processing on a straight line indicated by y = a (1 ≦ a ≦ 118, a is an integer) in the view field image.

  First, the horizontal edge detection unit 11 resets the number of continuous pixels on the straight line y = a (the number of continuous pixels) to 0 (S40). And the horizontal direction edge detection part 11 changes the value of b which shows the value of x coordinate as an integer value from 1 to 159 in the state which fixed the value of a, and processes of the following S41-S45 I do.

  First, in S <b> 41, the horizontal edge detection unit 11 determines whether any of the pixels at the coordinates (b, a−1), (b, a), and (b, a + 1) is an edge. Determine whether.

  If the determination in S41 is YES, that is, if any of the pixels at coordinates (b, a-1), (b, a), and (b, a + 1) is an edge, the horizontal edge detection unit 11 regards a pixel at coordinates (b, a) as an edge (S42). Thereafter, the horizontal edge detection unit 11 determines whether or not the pixels at the coordinates (b-1, a) and (b, a) are both edges (S43).

  The reason why the pixels (b, a-1) and (b, a + 1) are edges as described above is that the edge to be obtained is an edge that is roughly a straight line. Also, if the determination in S41 is NO, that is, if it is determined that none of the pixels at coordinates (b, a-1), (b, a), and (b, a + 1) is an edge pixel. The horizontal edge detection unit 11 performs the process of S43 without performing the process of S42.

  If it is determined in S43 that the pixels at the coordinates (b-1, a) and (b, a) are both edges, the horizontal edge detection unit 11 increases the number of continuous pixels by 1 (S44). On the other hand, when it is determined in S43 that the pixels at the coordinates (b-1, a) and (b, a) are not edges, the horizontal edge detection unit 11 resets the number of continuous pixels to zero. (S45).

  In this way, the processing of S41 to S45 is performed while changing the value of b as an integer value included between 1 and 159. And the horizontal direction edge detection part 11 calculates the largest continuous pixel number among the continuous pixel numbers calculated | required about each integer value as the length of the straight line edge in y = a (S46).

  The processes in S40 to S46 described above are performed while changing the value of a as an integer value included between 1 and 118. The length of the edge in the horizontal direction thus obtained is shown in the graph on the right side of FIG.

[3-2. (Calculation process of vertical edge length)
Next, details of the process of S32 in the flow of FIG. 4 described above will be described. For example, assume that a field-of-view image as shown in FIG. In this case, as shown in FIG. 7, the vertical edge detection unit 12 performs the following processing on a straight line indicated by x = b (1 ≦ b ≦ 158, b is an integer) in the view field image.

  First, the vertical edge detection unit 12 resets the number of continuous pixels on the straight line x = b (the number of continuous pixels) to 0 (S50). Then, the vertical edge detection unit 12 changes the value of a indicating the value of the y coordinate as an integer value from 1 to 119 in a state where the value of b is fixed, thereby performing the following processing of S51 to S55. I do.

  First, in S51, the vertical edge detection unit 12 determines whether any one of the pixels at the coordinates (b-1, a), (b, a1), and (b + 1, a) is an edge. Determine whether.

  If the determination in S51 is YES, that is, if any of the pixels at coordinates (b-1, a), (b, a), and (b + 1, a) is an edge, the vertical edge detection unit 12 considers a pixel at coordinates (b, a) as an edge (S52). Thereafter, the vertical edge detection unit 12 determines whether or not the pixels at the coordinates (b, a-1) and (b, a) are both edges (S53).

  If the determination in S51 is NO, that is, if it is determined that none of the pixels at coordinates (b-1, a), (b, a), and (b + 1, a) is an edge pixel. The vertical edge detection unit 12 performs the process of S53 without performing the process of S52.

  If it is determined in S53 that the pixels at the coordinates (b, a-1) and (b, a) are both edges, the vertical edge detection unit 12 increases the number of continuous pixels by 1 (S54). On the other hand, if it is determined in S53 that the pixels at the coordinates (b, a-1) and (b, a) are not both edges, the vertical edge detection unit 12 resets the number of continuous pixels to zero. (S55).

  In this way, the processing of S51 to S55 is performed while changing the value of a as an integer value included between 1 and 119. The vertical edge detection unit 12 calculates the maximum number of continuous pixels among the number of continuous pixels obtained for each integer value as the length of the straight edge at x = b (S56).

  The processing in S50 to S56 described above is performed while changing the value of b as an integer value included between 1 and 158.

[3-3. Hue division processing and histogram calculation processing)
Next, details of the processing of S34 and S35 in the flow of FIG. 4 described above will be described. First, as shown in FIG. 8, the hue division unit 13 calculates a 360-degree hue histogram from the entire view image, and detects the hue having the maximum frequency (S60). Then, the hue dividing unit 13 sets a range of ± 25 ° centered on the hue detected in S60 as a first grade range (S61).

  Thereafter, the hue dividing unit 13 divides the hue into seven grades by dividing the range other than the range set in S61 into six equal parts (S62).

  When the processing in S62 ends, the histogram calculation unit 14 sets a range of 7 for the regions on both sides of the edge candidate as shown in FIG. 9 for all the edge candidates set in S33 (see FIG. 4). A histogram of hues divided into grades is calculated (S63). In this way, the processes of S34 and S35 in FIG. 4 are completed.

[4. Processing in the object recognition unit]
Next, details of processing in the object recognition unit 4 will be described. The object recognizing unit 4 performs a matching process between the image shown in the region extracted from the view field image by the object region extracting unit 3 and the registered image stored in the database 7, and selects the image that the user is paying attention to. Recognize. Here, an example will be described in which the image focused on by the user is a painting, and the painting image as a registered image stored in the database 7 is matched with the image focused on by the user. .

  First, as shown in FIG. 10, the object recognition unit 4 divides the picture area extracted by the object area extraction unit 3 from the field-of-view image into 8 and 6 equal parts in the horizontal and vertical directions, and divides it into 48 blocks ( S70). Further, the hue average calculation unit 15 in the object recognition unit 4 calculates the average value of the hue of each block (S71).

  As shown in FIG. 11, the calculation process of the average value of hues in S71 is performed by performing the process of making the average value the value obtained by dividing the sum of the hues of the pixels in each block by the number of pixels for all the blocks. Executed.

Then, as shown in FIG. 10, the color difference calculation unit 16 calculates the color difference CS _pi in all blocks from the color difference in each block for all the painting images in the database 7 by the following equation (5). calculate.

  In the above formula (5),

  Indicates the average value of the hues of the blocks (j, k) (j = 1,..., 8), (k = 1,..., 6) in the painting region extracted from the view field image.

  Also,

  Indicates the average value of the hue of the block (j, k) in the i-th painting image in the database 7.

  Note that object recognition and region-of-interest discrimination may be performed by attaching a tag such as a two-dimensional barcode to an object or region.

[5. Presentation processing of annotation information based on gaze point distribution)
In the information processing apparatus 1 according to the present embodiment, the attention area determination unit 5 determines the attention area of the object, and the annotation information presentation section 6 presents annotation information and hierarchy information corresponding to the attention area. Each registered image in the database 7 defines color information, vertex coordinates of a convex polygonal region of the region to which annotation information is added, annotation information thereof, and hierarchical information of regions constituting the image. Yes.

  FIG. 12A is a diagram illustrating a region to which annotation information is added when the registered image is, for example, a painting image, and FIG. 12B is a diagram illustrating a configuration of hierarchical information of the painting image. It is.

  When the painting shown in FIG. 12A is divided into two roughly divided areas, it can be divided into an area 1 (house) and an area 2 (forest). Further, when the region 1 (house) is further subdivided, it can be divided into a region 11 (roof) and a region 12 (veranda). Therefore, as shown in FIG. 12B, the hierarchical information has a tree structure in which the area 11 and the area 12 are included in the lower level of the area 1.

  Hereinafter, the attention area determination processing performed by the attention area determination unit 5 and the annotation information presentation processing performed by the annotation information presentation unit 6 will be described more specifically with reference to a case where the registered image is a picture image.

[5-1. Processing in the attention area determination unit 5]
The attention area discriminating section 5 adds annotation information to the minimum convex polygon area including the gazing point distribution of the attention section and the painting in the database after the processing in the object area extraction section 3 and the object recognition section 4. The attention area is determined based on the position information of the existing area.

  Here, the size of the picture area extracted from the view field image and the picture area in the database 7 are different. Therefore, the attention area discriminating unit 5 normalizes the size of the picture area based on the reduction / enlargement ratio of the extracted picture area with respect to the picture area in the database 7. Thereafter, the attention area determination unit 5 determines the attention area included in the minimum convex polygon area according to the flowchart shown in FIG.

  First, the attention area discriminating unit 5 calculates the horizontal and vertical distances to the vertices of the minimum convex polygon area with reference to the lower left of the painting area, and calculates the position of the minimum convex polygon area on the painting area. (S80).

  The attention area discriminating unit 5 also calculates the horizontal and vertical distances to the vertices of the convex polygon area to which the annotation information is added with reference to the lower left of the painting area for the painting area in the database 7. (S81).

  Note that the process of S81 is performed on a region in which annotation information is defined in advance for a painting image in the database 7. Moreover, the process of S81 does not necessarily need to be performed after S80, and may be performed in advance. Further, S81 may be performed by a processing unit other than the attention area determination unit 5.

  Then, the attention area determination unit 5 normalizes the size of the painting area extracted from the view field image with reference to the size of the painting area in the database 7 (S82). Further, a normalized position is calculated for each vertex of the minimum convex polygonal area on the painting area whose distance has been calculated in S80 (S83).

  Then, the attention area determination unit 5 determines how much the overlap between the minimum convex polygonal area in the painting area of the view field image and each area to which the annotation information is added in the database 7 is, The region where the overlap is maximum is determined as the region noted by the user (S84).

[5-2. Processing in the annotation information presentation unit 6]
The annotation information presentation unit 6 presents the annotation information related to the attention region after the attention region is determined by the attention region determination unit 5 from the gazing point distribution. For example, if the attention area discriminating unit 5 determines that the user is paying attention to the mansion part of the painting image, the annotation information presenting unit 6 sends the information about the mansion to the outside of the information processing apparatus 1 as shown in FIG. The image is displayed by a provided output device (notebook PC, head mounted display, etc.). In addition, as shown in FIG. 14, you may display the information (author, author name) regarding the painting itself as annotation information.

  Further, the annotation information presentation unit 6 may present not only annotation information related to the attention area but also hierarchical information of the attention area. In other words, in the hierarchical structure indicated by the hierarchical information, the annotation information of the region that becomes the parent or the child of the region of interest is presented to the user using the annotation information presentation unit 6, so that other information related to the region of interest is displayed. The user can be provided with a clue to obtain In this case, the closer the hierarchy between the region to which the annotation information is displayed and the region of interest is closer, the more information can be obtained that accurately supplements the information about the region of interest.

  Further, while the annotation information is presented on the screen of the display device, the eye movement may not be analyzed by the information processing apparatus 1. And it is good also as a trigger which detects that a user closes eyes for a definite period of time and restarts analysis of eyeball movement. Thereby, since the user can receive the presentation of the annotation information again only by closing his eyes, the convenience of the information processing apparatus 1 is improved.

  In addition, when the user pays attention to the entire painting, the annotation information about the painting itself and a region that is a child of the painting may be displayed superimposed on the painting. In addition, when the user is paying attention to a certain area, annotation information related to the attention area and a parent or child area of the attention area may be presented to the user.

[6. Experiment and evaluation)
Since an experiment was performed to evaluate whether the attention area determined by the information processing apparatus 1 of the present embodiment is an area of actual attention, the following description will be given. Note that, as a preprocessing for discriminating the region of interest, a painting region is extracted from the field-of-view image and the extracted painting region is recognized. Therefore, an evaluation experiment on the extraction of the painting region and the recognition of the painting was also performed.

  The PC used as the information processing apparatus 1 is a notebook PC equipped with a Pentium (registered trademark) III having a frequency of 500 MHz as a CPU. In addition, the cameras 17 and 18 have a frame size of 160 × 120 [pixel] and a processing speed of 10 [fps]. Note that the image taken by the eyeball is 256 gray scale, and the view image is 24 bit color.

  In addition, 20 A3-sized paintings were placed in a picture frame and hung on a single-colored wall. The breakdown of the 20 types of paintings consists of 10 people and animal paintings and 10 landscape paintings.

  Further, FIG. 15 shows a state when the prototype of the information processing apparatus 1 is used, and FIG. 16 shows a state of picture extraction. Of the upper five images in FIG. 16, the lower right image represents the point of interest.

[6-1. (Picture area extraction accuracy and picture recognition accuracy)
[6-1-1. (Accuracy to extract picture area from view image)
In order to evaluate whether the region extracted by the object region extraction unit 3 by the above-described method is a painting region, the ratio (Pr) that is actually a painting region among the regions extracted by the object region extraction unit 3; Evaluation was performed based on the ratio (Re) of the area extracted by the object area extraction unit 3 among the picture areas shown in the view field image.

  For the 20 types of paintings, the system acquired 20 view images each showing the paintings when viewed almost from the front, and extracted the painting areas from a total of 400 view images. The results are shown in Table 1. Table 1 shows the average values of Pr and Re obtained by extracting a picture from each view image, and the average value of the area occupancy ratio of the picture. In addition, the area occupation rate of a painting is a ratio of the area which a painting occupies in a view field image.

[6-1-2. (Picture recognition accuracy)
After the painting region shown in the view field image is extracted by the object region extraction unit 3 by the above-described method, the object recognition unit 4 recognizes the painting by the above-described method. About 20 types of paintings, 20 images of the field of view, each of which shows the paintings when viewed from the front, were acquired by the information processing apparatus 1 and the paintings were recognized for a total of 400 fields of view images. If the picture discriminated in each view image is the same as the picture shown in the view image, the recognition is successful. The results are shown in Table 2.

  From Table 2, even if the distance between the user and the painting is increased, the painting can be recognized with a certain degree of accuracy.

[6-2. (Identification accuracy of attention area)
It was evaluated whether or not the attention area determined by the attention area determination unit 5 by the above-described method was the area that was actually noticed. For 20 types of paintings, five locations such as people and buildings that are constituent elements are defined as attention areas.

  These areas may be adjacent, separated, or contained (done). Each subject paid attention to a total of 100 attention areas once, and if the attention area determined by the attention area determination unit 5 was the same as the actual attention area, the determination was successful. The average of the results of 4 subjects is shown in Table 3. When the distance between the user and the painting is 1.0 m, the viewing angle with respect to the painting is 17.8 ° vertically and 22.6 ° horizontally. When the distance is 1.5 m, the viewing angle with respect to the painting is 11.4 ° up and down and 15.2 ° left and right.

  Comparing the accuracy when the distance between the user and the painting is 1.0 m and when the distance is 1.5 m, the discrimination accuracy of the attention area is better when the distance is shorter. This is because if the distance between the user and the painting is close, the accuracy of preprocessing will be good, and the influence on the extraction of the gazing point when normalizing the painting area extracted from the view field image will be small. Conceivable.

  In addition, when the distance between the user and the painting is short, the change in the size of the region is small when the extracted painting region is normalized. For this reason, even if the position of the minimum convex polygon after normalization described in S83 of FIG. 13 is calculated, the minimum convex polygon is determined due to the error in extraction of the gazing point and the influence of the distance between the user and the painting. There was no deformation.

[7. Supplement)
The processing procedure of the information processing apparatus 1 according to the present embodiment is such that a calculation unit such as a CPU executes a program stored in a storage unit such as a ROM (Read Only Memory) or a RAM, and an input unit such as a keyboard or a display It can be realized by controlling output means or communication means such as an interface circuit.

  Therefore, various processes of the information processing apparatus 1 of the present embodiment can be realized simply by a computer having these means reading the recording medium storing the program and executing the program. In addition, by recording the program on a removable recording medium, the various functions and various processes described above can be realized on an arbitrary computer.

  As the recording medium, a program such as a memory such as a ROM may be used as a recording medium (not shown in the figure for processing by a microcomputer). It may be a program medium provided with a reading device and readable by inserting a recording medium therein.

  In any case, the stored program is preferably configured to be accessed and executed by the microprocessor. Furthermore, it is preferable that the program is read out, and the read program is downloaded to a program storage area of the microcomputer and the program is executed. It is assumed that this download program is stored in advance in the main unit.

  The program medium is a recording medium configured to be separable from the main body, such as a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a flexible disk or a hard disk, or a disk such as a CD / MO / MD / DVD. Fixed disk, IC card (including memory card), etc., or semiconductor ROM such as mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM, etc. In particular, there are recording media that carry programs.

  In addition, if the system configuration is capable of connecting to a communication network including the Internet, the recording medium is preferably a recording medium that fluidly carries the program so as to download the program from the communication network.

  Further, when the program is downloaded from the communication network as described above, it is preferable that the download program is stored in the main device in advance or installed from another recording medium.

  As described above, the information processing apparatus 1 according to the present embodiment includes an eyeball motion detection unit 2 that detects a state in which the user is paying attention to an object, and an eyeball motion detection unit based on an eyeball image obtained by capturing the user's eyeball motion 2, the object recognizing unit 4 for recognizing the object of interest of the user in the field-of-view image obtained by photographing the user's field of view, and the user is paying attention to the object. An attention area discriminating section 5 that discriminates an attention area, and an annotation information presentation section 6 that presents annotation information related to the attention area discriminated by the attention area discrimination section 5 to the user.

  According to the above configuration, since the eye movement detection unit 2 detects that the user is in the attention state, the object recognition unit 4 recognizes the object that the user is paying attention to. The object shown can be recognized.

  In addition, since the attention area discriminating unit 5 discriminates the attention area in which the user is paying attention, it is possible to grasp which part of the object the user is paying attention to. And since the annotation information presentation part 6 presents the information regarding the said attention area | region to a user, it is possible to present to the user exactly the annotation information regarding the area | region where the user is really interested, and is unnecessary. Information can be prevented from being presented to the user.

  As described above, according to the information processing apparatus 1, the annotation information regarding the area in which the user is interested is accurately presented to the user, so that a comfortable use environment of the information processing apparatus 1 can be given to the user.

  Further, the eyeball motion detection unit 2 includes a line-of-sight detection unit 8 that extracts a pupil region from an eyeball photographed image, a gaze point calculation unit 9 that calculates a movement distance of a gaze point from center coordinates in the pupil region, and the gaze point The viewing angle of the user with respect to the object is calculated from the moving distance, the fixation state and the jumping state of the eyeball are detected based on the size of the viewing angle, and the user pays attention to the object based on the number of the fixation state and the jumping state. And an attention state detection unit 10 that detects the state of

  That is, the fixation state and the jumping state can be detected by detecting the viewing angle of the user, and the viewing angle of the user can be obtained by calculating the moving distance of the gazing point.

  In the information processing apparatus 1 configured as described above, the eye gaze detection unit 8 extracts the pupil region using the binarization process, and the gaze point calculation unit 9 calculates the movement distance of the gaze point. Then, the attention state detection unit 10 calculates the user's viewing angle from the movement distance of the gazing point obtained by the gazing point calculation unit 9, and detects the number of fixation states and jumping states. Can be detected.

  In this way, according to the above configuration, the user's attention state can be accurately grasped, so that the object that the user is interested in is accurately recognized by the object recognition unit 4, and the annotation information presentation unit 6 further includes Accurate annotation information can be presented to the user.

  Furthermore, the attention state detection unit 10 defines a minimum polygonal region in the view image by connecting a plurality of attention points so as to include a distribution of attention points while the user is in the attention state. At the same time, the attention area determination unit 5 determines an area where the overlap with the minimum polygonal area is maximum among a plurality of areas in the registered image stored in the database 7, and the view image corresponding to the area The inner area is discriminated as the attention area.

  According to the above configuration, the attention state detection unit 10 defines the minimum polygonal region. Since the minimum polygonal area is obtained by connecting a plurality of gazing points so as to include the distribution of the gazing points, it can be said that the minimum polygonal area accurately represents an area that the user is paying attention to in the view field image.

  Then, the attention area determination unit 5 determines the area where the overlap with the minimum polygon is maximum for the registered image in the database 7, and determines the area in the view field image corresponding to the area as the attention area. Thus, it is possible to accurately determine the region that the user is paying attention to as the attention region.

  Furthermore, the object recognizing unit 4 recognizes an object that the user is paying attention to based on the hue in the view image and the hue of a registered image that can be a candidate for the view image stored in the database 7.

  If the visual field image is an image having a color such as, for example, a painting image, the object included in the visual field image can be characterized by the hue. Therefore, in the above configuration, the object being noticed by the user is recognized based on the hue of the field-of-view image and the hue of the registered image, so that the object recognition accuracy can be improved.

  The annotation information presentation unit 6 presents the user with annotation information regarding a region having a hierarchical structure with the region of interest.

  According to the above configuration, the annotation information related to the region that forms a hierarchical structure with the region of interest is also presented to the user, so that the user can obtain more information about the region of interest and deepen their understanding of the region of interest.

  Further, by mounting one or both of the camera 17 that captures the eyeball image and the camera 18 that captures the field image on the mobile terminal, the eyeball image and field image can be captured even if the user moves. Therefore, the convenience of the information processing apparatus 1 can be further enhanced.

  Furthermore, since the annotation information can be presented to the moving user by reproducing the annotation information using the portable terminal, the convenience of the information processing apparatus 1 can be further improved.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

  According to the information processing apparatus of the present invention, in an environment where a large number of paintings are exhibited, for example, in an art museum, information on the entire work or information on a part of the work is adapted to the attention area of the user as annotation information. It can be presented. As a result, the user can deepen their understanding of the paintings on display, and can also omit labor costs for arranging personnel to explain the paintings.

  In addition, the use environment of the information processing apparatus of the present invention is not limited to an environment where pictures are displayed. For example, it is good to store the image which image | photographed each part of the machine tool as a registration image in a database, and to show a user text data and voice data which explain the operation method of a machine tool as annotation information. Thereby, the user can know the operation method of the machine tool only by paying attention to each part of the machine tool, so that even a beginner can operate the machine tool.

It is a block diagram which shows the structure of the information processing apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of a series of processes until annotation information is shown by the information processing apparatus of FIG. It is a flowchart which shows the flow of the process which the eyeball movement detection part in the information processing apparatus of FIG. 1 performs. It is a flowchart which shows the flow of the process which the object area extraction part in the information processing apparatus of FIG. 1 performs. FIG. 5A is a diagram showing an example of a field-of-view image, and FIG. 5B is a graph showing the edge image obtained from the field-of-view image of FIG. 5A and the length of the edge in the horizontal direction. is there. It is a flowchart which shows in detail the calculation process of the length concerning the edge of a horizontal direction. It is a flowchart which shows in detail the calculation process of the length concerning the edge of a perpendicular direction. It is a flowchart which shows the division | segmentation process of a hue in detail. It is a flowchart which shows the histogram calculation process of a hue in detail. It is a flowchart which shows in detail the process which calculates a color difference about the block which divided | segmented the object. It is a flowchart which shows the calculation process of the average value of a hue in detail. FIG. 12A is a diagram illustrating a region to which annotation information is added when the registered image is, for example, a painting image, and FIG. 12B is a diagram illustrating a configuration of hierarchical information of the painting image. It is. It is a figure which shows the detail of an attention area determination process. It is a figure which shows an example of the screen which presents annotation information. It is a figure which shows the state which uses one Embodiment of the information processing apparatus of this invention. It is a figure which shows a mode that the pictorial image is extracted by one Embodiment of the information processing apparatus of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Eye movement detection part (eye movement detection means)
4 Object recognition unit (object recognition means)
5 Attention area discrimination unit (attention area discrimination means)
6 Annotation information presentation part (annotation information presentation means)
7 Database 8 Gaze detection unit (Gaze detection means)
9 Gaze point calculation unit (gaze point calculation means)
10 Attention state detection unit (attention state detection means)
17 Camera (first camera)
18 cameras (second camera)

Claims (10)

Eye movement detecting means for detecting a state in which the user is paying attention to the object, based on an eyeball photographed image of the user's eye movement;
An object recognizing unit that recognizes an object focused on by the user in a view field image of the user's view when the eye movement detecting unit detects that the user is in the focused state;
Attention area discrimination means for discriminating the attention area that the user is paying attention to in the object;
Annotation information presenting means for presenting annotation information related to the attention area determined by the attention area determination means to the user ,
The eye movement detecting means is
Eye-gaze detection means for extracting a pupil region from the eyeball image,
Gazing point calculation means for calculating a moving distance of the gazing point from the center coordinates in the pupil region;
The viewing angle of the user with respect to the object is calculated from the moving distance of the gazing point, the fixation state and the jumping state of the eyeball are detected based on the magnitude of the viewing angle, and the user is based on the number of the fixation state and the jumping state. Attention state detection means for detecting a state of attention to the object,
The attention state detection means further defines a minimum polygonal region in the view image by connecting a plurality of attention points so as to include a distribution of attention points while the user is in the attention state. ,
The attention area determination means determines an area where the overlap with the minimum polygonal area is maximum among a plurality of areas in the registered image stored in the database as a candidate for a view field image, and corresponds to the area An information processing apparatus for discriminating an area in a view field image as the attention area. The object recognizing means recognizes an object focused on by a user based on a hue in the view image and a hue of a registered image that can be a candidate for the view image stored in a database. The information processing apparatus according to claim 1. The information processing apparatus according to claim 1, wherein the annotation information presenting unit further presents annotation information relating to a region having a hierarchical structure with the region of interest to the user. A mobile terminal used in the information processing apparatus according to claim 1,
A portable terminal comprising a first camera for photographing the eyeball photographed image. A mobile terminal used in the information processing apparatus according to claim 1,
A portable terminal comprising a second camera that captures the view image. A mobile terminal used in the information processing apparatus according to claim 1,
  A portable terminal comprising: a first camera that captures the eyeball image; and a second camera that captures the field-of-view image. A mobile terminal used in the information processing apparatus according to claim 1,
  A portable terminal characterized in that the annotation information is fetched from the annotation information presenting means and the annotation information is reproduced. A first step of detecting a state in which the user is paying attention to the object based on an eyeball image obtained by photographing the user's eye movement;
A second step of recognizing an object that the user is paying attention to in the view image obtained by photographing the user's view when it is detected that the user is in the attention state by the first step;
A third step of determining a region of interest in which the user is paying attention in the object;
A fourth step of presenting annotation information regarding the attention area determined in the third step to the user ,
The first step is
A line-of-sight detection step of extracting a pupil region from the eyeball image,
A gazing point calculation step of calculating a moving distance of the gazing point from the center coordinates in the pupil region;
The viewing angle of the user with respect to the object is calculated from the moving distance of the gazing point, the fixation state and the jumping state of the eyeball are detected based on the magnitude of the viewing angle, and the user is based on the number of the fixation state and the jumping state. And a state of interest detection step for detecting a state of attention to the object,
The attention state detection step is a step of defining a minimum polygonal area in the view field image by connecting a plurality of attention points so as to include a distribution of attention points while the user is in the attention state. ,
The third step determines a region having the largest overlap with the minimum polygonal region from among a plurality of regions in the registered image stored in the database as a candidate for the view image, and the view corresponding to the region An information processing method comprising the step of discriminating an area in an image as the attention area. An information processing program for causing a computer to execute each step of the information processing method according to claim 8. A computer-readable recording medium on which the information processing program according to claim 9 is recorded.

Images