JP2004062883A - Information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device which efficiently acquires a part which is important for a user and a level of an importance degree by an easy operation and the sorting, processing and presentation of information based on them when multimedia information is treated in the information processor. <P>SOLUTION: Visual lines of the user at the time of information viewing are recorded and the level of the importance degree to original information is presumed based on the recorded visual level. In the presumption of the importance degree, a type of eye movement is first determined and a time considered as a visual system acquires information at high efficiency is discriminated. The activeness/inactiveness of the activity of the visual system are digitized based on information collection efficiency and the importance degree to information is presumed by indexing activeness at a cut, and scene. Thus, an important part for the user in the original information and the importance degree are presumed without a special operation of the user. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
[Industrial applications]
The present invention relates to a multimedia information processing apparatus that integrally displays, records, reproduces, and edits video, audio, documents, and the like.
[0002]
[Prior art]
It is well known that the performance of microprocessors mounted on personal computers and the like has been remarkably improved, and an environment in which a large amount of information can be processed at high speed by a popular personal computer is being prepared. Recent developments in optical communication technology, communication control technology, magneto-optical disks, and memory elements based on high-density integrated circuits have also led to the increase in the capacity of information transmission paths and the increase in capacity of information storage devices. A world in which large amounts of information can be obtained electronically at low cost in homes and offices is being realized. In addition, since information can be transmitted with simple operations, the privilege of transmitting information, which was previously limited to broadcast stations and newspaper companies, has been expanded to general users, and more and more information has been exchanged. is there. This can be seen from the current spread of personal computer communications, e-mail, and electronic news. Terminals capable of exchanging information in this manner are not limited to personal computers, but are in the form of various devices such as electronic organizers, mobile phones, and facsimile machines. The types of information that they can handle have expanded from conventional character information (text data) to voice / music information, still image information, and moving image information, and their quality has been dramatically improved. I have.
[0003]
However, in many cases, a conventional method is used for the information classification method. Taking the world of electronic news as an example, there are hundreds of newsgroups (= “cuts” for each topic), and dozens of posts are made to each newsgroup per day. Users have to spend a considerable amount of time searching for news that they need / interest in. This is because each news is dealt with in a very simple, newsgroup-to-news layer structure that users cannot usually define on their own. Even on music tapes and video tapes, the start position of each song / program can be determined by detecting a silent portion or an index signal for starting recording, but the content of the song / program, or a scene or portion of particular interest, is determined. Since the information to be pointed cannot be recorded, the user has no other way than to write on those labels and the like.
[0004]
Development for automating the task of classifying the information or extracting a characteristic portion from the information is still being actively performed. However, many of them aim to substitute human judgment mechanisms, require advanced artificial intelligence, and are said to be costly in terms of development time and cost at the moment. I have no choice. Also, such an automatic classification system requires the user to indicate what he or she wants, and is difficult for a user with an unclear vision of what to see and what kind of information to obtain. It can be assumed that such unfamiliar users will increase dramatically in an era when information is full.
[0005]
Rather, what is needed is information that states that "the place I feel is important" is not necessarily required to describe its contents. For example, if an attractive actor appears only in one scene in a movie, it is always necessary to know who the actor is, whether it is a man or a woman, and whether the object (subject) is a human. Rather, a device that simply points to the scene often suffices. Nevertheless, in the current flow of information classification, the procedure is as follows: "Information is scrutinized by the device → Extract a characteristic part as a candidate → Compare with the request information entered by the user → Classify and present" The information content must be carefully analyzed.
[0006]
On the other hand, now that general users are becoming information transmission sources, demands for information processing tools are increasing. Conventional processing tools include a word processor for document information, a CAD (Computer Aided Design) for graphic information, and drawing software. However, there are very few general-purpose tools that are easy to operate, especially for video and audio. The user connects two decks, searches for a target location on one deck, plays it back, and records it on the other deck to perform "cutting" editing. At least two decks are required to perform "cutting" editing, and the operation is complicated, so that ordinary users tend to avoid editing work. At present, some of the proposed editing methods temporarily store video / audio information in a storage medium such as a computer memory or a hard disk and process the information in a computer editing environment. A large-capacity storage medium is required and the cost is high. Since the editing is basically "cutting", no particular improvement in operability can be expected.
[0007]
The lack of a simple and efficient information automatic classification mechanism and information processing means as described above may greatly hinder the spread of a multimedia integrated environment in the future.
[0008]
[Problems to be solved by the invention]
As described above, in the conventional information processing apparatus, there is no means for classifying and organizing various kinds of information efficiently across types of information and reflecting the intention of the user. For this reason, the user must spend time and effort on the processing of the information, and there is a drawback that even when the information is automatically processed as a device, the processing cannot always be adapted to the individual user. Conventionally, the information processing method is complicated, and there is a disadvantage that the cost is likely to be high for realizing the method.
[0009]
The present invention has been made to solve the above-described problem, and provides an information processing apparatus capable of efficiently processing and presenting information required by a user from a large amount of various information with a simple operation.
[0010]
[Means for Solving the Problems]
In order to solve the problem of estimating the temporal change of the importance level without discriminating the subject boundary from the image, the information processing apparatus according to the present invention includes an image, and information of an image and a sound. Means for observing eye movements of a user who is viewing or viewing the information, means for sequentially generating a velocity component of the eye movements of the user observed by the means, And means for estimating the degree of activity of the eye movement from the velocity component generated in step (a), and estimating the importance of the information based on the degree of activity of the eye movement obtained by the means for obtaining the eye movement.
[0011]
[Action]
In the information processing apparatus according to the present invention, it is possible to express numerically how important the user recognizes the presented information by using the line of sight. Since a part of the user's intention is obtained from the natural movement of the gaze, the user becomes less proficient in the operation, and the user can input his / her intention without being aware that the measurement is being performed. Is In this way, the parts that the user recognizes as important in the original information and the degree thereof can be obtained by a simple input method, so that processing such as presentation and sorting biased toward “important parts” can be performed. It is possible to improve the efficiency of information access.
[0012]
【Example】
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an information processing apparatus according to one embodiment of the present invention. 12 to 14 are flowcharts illustrating the following processing flow.
[0013]
The multimedia information stored in the recording medium 116 is read out by the readout control unit 114, and is sent by the image information presenting unit 110, the audio information presenting unit 111, the music information presenting unit 112 via the presentation form creating unit 113, and the like. It is presented to the user 101. The presentation form creation unit 113 specifically refers to a video RAM, a video processor, a sound manager, a speaker driver, a MIDI interface, and the like. In the above process, there is no difference from the method of presenting information of a conventional multimedia device such as a combination of a television and a video tape recorder (VTR) and a personal computer equipped with a CD-ROM drive.
[0014]
On the other hand, the line of sight of the user 101 viewing the information presented in this way is measured by the line of sight detector 102 and sent to the viewpoint calculation unit 103. The eye-gaze detector may be of any type, but the following description is based on the assumption that the eye-gaze detector outputs two-channel voltage values corresponding to the two-dimensional coordinates of the viewpoint. I do. The data sent from the eye gaze detector 102 to the viewpoint calculation unit 103 is, for example, as shown in FIG. 4A (in FIG. 12, the data as in FIG. 4A flows at the location of a in FIG. 12; the same applies hereinafter). . The viewpoint calculation unit 103 converts this voltage value into a position in the screen and stores it in the temporary storage unit 108 (S1201 in FIG. 12). The data output from the viewpoint calculation unit 103 is as shown in FIG. 4B. Based on the coordinate data, the eye movement type determining unit 104 determines the type of eye movement at the time and stores it again in the temporary storage unit 108 (S1203 to S1209 in FIG. 12).
[0015]
As shown in FIG. 2, human eye movements are roughly divided into fixation fine movements and tracking movements, and jumping movements can be distinguished from fixation fine movements and follow-up movements by observing the velocity components. FIG. 3 schematically shows an actual line of sight when viewing a moving image. In the screen, two main objects (subjects), a person 301 and a television stand 302, are shown. A circle (○) in FIG. 3 is a viewpoint recorded every 1/60 second. As shown here, it can be seen that the slow motion continues while looking at the object (303), and the fast motion is performed (304) when moving between objects.
[0016]
In FIG. 5, similarly, the data of the viewpoint actually recorded are shown with the lapse of time on the horizontal axis and coordinate values on the vertical axis. In the figure, a solid line and a broken line represent an X coordinate and a Y coordinate, respectively. Although three seconds are shown in the figure, the state in which the coordinates rapidly change in an extremely short time is seen five times (501). This corresponds to a jumping movement. On the other hand, a pulse-like motion much larger than the jumping motion is also seen three times (502), which is a blink. Such movement is caused by the characteristics of the eye-gaze detector. However, since the coordinate value hardly changes before and after the pulse-like movement, it can be distinguished from the jumping movement.
[0017]
As described above, it is the eye movement type determination unit 104 in FIG. 1 that distinguishes the eye movement into three types of “blink”, “jump”, and other “gaze”. The data determined by the eye movement type determination unit 104 has a form as shown in FIG. 4C. It should be noted that an object whose viewpoint has been moved during blinking should be regarded as a “jump” because of the movement of the object, and it is assumed that the eye-movement-type determining unit 104 makes such determination. The eye movement type at the time and time is again stored in the temporary storage unit 108 by the eye movement type determination unit 104.
[0018]
As for human visual characteristics, it is known that the resolution of an image viewed within about 200 milliseconds after a jumping motion is reduced. This is described in, for example, "Experimental Psychology of Eye Movement" (Nagoya University Press, 1993). Therefore, it can be said that the user can actually gaze at the object at the normal resolution 200 ms after the jumping motion. Therefore, the time for viewing each object is as shown by 503 in FIG. Reference numeral 504 denotes a time during which the resolution is reduced due to the jumping motion. The "time when each object was viewed" per unit time in the past, that is, the time excluding the jumping motion and 200 milliseconds thereafter is defined as the "information acquisition rate" (S1302 to S1305 in FIG. 13). Here, the jumping movement includes a case where the object is moved during blinking. Conversely, when the viewpoint before and after the blink does not change significantly (502 in FIG. 5), the object is not moved, and the following process is handled in the same manner as the gaze. The information acquisition rate calculation unit 105 calculates the information acquisition rate from the time transition of the eye movement type in this way, and stores the information acquisition rate in the temporary storage unit 108 again. This data takes a form as shown in FIG. 4D, for example.
[0019]
When the information acquisition rate decreases, it can be said that the visual system attempts to see many objects even at the expense of resolution reduction due to jumping motion. Conversely, when there are few objects to be seen in the screen, it is considered that the time during which the information acquisition rate is high is long. Therefore, it is considered that the visual system is active in collecting information while the information acquisition rate is decreasing, and a time function for evaluating the activity is defined as a “visual activity level”. The visual activity level is the time during which the information acquisition rate has been decreasing in the past unit time. The “time when the information acquisition rate is decreasing” is a time section surrounded by two consecutive moments at the moment when the second derivative of the information acquisition rate with time becomes 0, and includes the minimum value of the information acquisition rate. That is, it is the time section 601 in FIG. In FIG. 6, 602 is the information acquisition rate, 603 is the time at which the second derivative of the information acquisition rate 602 becomes 0, and 604 is the minimum value of the information acquisition rate 602. Since the actual calculation inside the computer takes a discrete value in the time direction, the above “derivative” may be replaced with “difference” to perform the calculation. The visual activity level calculator 106 performs such calculation. The visual activity level calculation unit 106 determines whether the eye movement is “active” or “inactive” at the time through the above-described calculation process based on the information acquisition rate calculated so far, and outputs the result. This output is stored again in the temporary storage 108 (S1306 to S1308). The output data is in a form as shown in FIG. 4e, for example.
[0020]
The time resolution of the above process is defined by the capability of the processing system of the present invention, and is divided into time units that are easy for the user to understand. The time resolution specified by the system is, for example, 1/30 second, 1/60 second, 1/100 second, etc., and the time unit that is easy for the user to understand is, for example, “cut”, “scene” or 30 seconds, 3 minutes, and so on. The importance information generation unit 107 calculates the information acquisition rate in a time unit that is easy for the user to understand, and outputs the calculation result and the time unit as the importance information. That is, data such as "the cut number 14 is from 3:23:15 field to 3:29:27 field. Its importance is 0.73". In the case where a time unit such as a cut here is recorded in advance together with the original information, the information on the start time and the end time can be omitted. In addition, in what time unit the importance information is added may be selected by the user from among the above “cut”, “scene”, and how many seconds.
[0021]
This time unit is determined by the information unit detection unit 109 and sent to the importance information generation unit 107. In the case where a break in information is not recorded in the original information, the information unit detection unit 109 detects a scene change or the like by detecting a moment such as a sudden change in color tone or luminance in the image information. The definition method of these information units may be any method. When the user particularly wants to correct the break of the information automatically detected by the information unit detection unit 109, the break of the information newly defined by the user may be set as the time unit for the importance information. Further, when such a time unit is recorded in the original information in advance, the information unit detection unit 109 may use the recorded time unit.
[0022]
Among the importance information in the time unit, the method of calculating the importance is, for example, as follows. As described above, since the information acquisition rate “active” or “inactive” is known for each time, the ratio of the “active” time to the time unit is adopted as the importance (FIG. 14). If the time that was "active" for a cut of 10 seconds in length is 4 seconds in total, the importance is 0.40, and the time of "active" in a scene of 3 minutes in length is 2 in total For example, 0.70 for a minute of 6 seconds, and 0.33 for a total of 40 minutes of "active" time for a movie having a length of 2 hours. The importance information is again stored in the temporary storage unit 108, and is recorded on the recording medium in association with the original information at an appropriate time. The appropriate time depends on the operating environment of the system. When the recording medium 116 and its peripheral mechanism are capable of recording at the same time as information reproduction, the above process is performed simultaneously with information presentation, and the importance information is recorded in parallel with reproduction / presentation. Is also good. Further, the information may be stored in the temporary storage unit 108 until the information presentation is completed, and the recording of the importance information may be started collectively after the presentation is completed. On the other hand, in FIG. 1, the recording medium 116 is illustrated assuming an optical disk or the like, but the form of the recording medium 116 is not particularly limited to this, and the importance information is recorded on a medium different from the original information. You may. As the recording medium 116 for recording the original information and the importance information, a magnetic tape such as an optical disk, a magneto-optical disk (MO), a video tape, a magnetic disk such as a floppy (R) disk or a hard disk, an IC memory, or a communication cable Alternatively, a remote computer or the like may be used.
[0023]
The importance recorded in this manner is effective for comparison with other importance information for which a time unit is set in the same manner. For example, when the importance is recorded in a video of a family trip photographed by a home video camera by the above-described means, the playback apparatus performs a cut with an importance of 0.35 and a cut with an importance of 0.64. Then, the system presumes that the latter is more likely to be searched after a lapse of time, and stores in advance the audio and video information of the first few seconds of the cut in a memory element. Since the information stored in the memory element in advance can be accessed at high speed, it can be expected that the user can reach the target image more smoothly. Alternatively, when there is an optical disk of a movie of importance 0.20 and an optical disk of a movie of importance 0.77, a disc changer (a reproducing device capable of storing a plurality of discs) stores the latter at a position where it can be easily taken out, and The effect can be expected.
[0024]
Next, FIG. 7 shows an experimental result in which the importance is given by the above-described method and the matching with the importance actually considered by the user is compared. FIG. 7 shows a series of moving images composed of five scenes to the subject, and when the scenes are rearranged in the order of “interesting / important”, the first, second, third,. 7 is a graph showing what importance is given to the system using the method of the present invention. Note that the image is a movie, commercial, landscape, music video, and soccer, each of which is continuous for 30 seconds and a solid screen of 15 seconds. If the method of the invention is valid, the graph should show a monotonically decreasing trend. FIG. 7 shows that, in the subject KNT, the order of importance declared by the subject matches the order of importance assigned by the present system. Also, in the subject NZS, the order is consistent except for the image of the third place. Further, even with data such as the subject KTS, it is possible to discriminate the 1st, 2nd and 3rd to 5th places.
[0025]
In an image generated by CG (computer graphics) or the like, it may be known which position in the screen belongs to which object. In such a case, there is a possibility that the accuracy of the above method can be improved by utilizing the relationship between the viewpoint position and the object, and the method will be described.
[0026]
When an image as shown in FIG. 8A is present, it is assumed that the object area is known, such as the person 801 and the paper 802 in FIG. 8B, and the respective boundaries are defined. At this time, assuming that the viewpoint is like “O” or “X” in FIG. 8B, the viewpoint that is “X” among those viewpoints is in a place that does not belong to either of the two objects. Among the jumping motions seen in FIG. 8b, in the importance estimation procedure described so far, the movement within the defined object (803) or the movement between the defined objects (804) and the movement to the non-object area are performed. The movement (805) and the movement in the non-object area (806) were treated exactly the same. For this reason, if there are many jumping movements in spite of movement relating to the non-object area, the movement is considered to be “active” and a high degree of importance may be given. If the object area is defined in advance, it is regarded as a jumping movement (movement between objects) only when the end of the movement is a defined object, and the following information acquisition rate, visual activity level, importance May be calculated. In this case, the object definition information recorded on the recording medium 116 is read by the read control unit 114, and is passed to the eye movement type determination unit 104 as object information. Further, even when the object area is not defined in advance, in a system having a slight image analysis function, the system itself may analyze an image, define an object area, and use the same. Regarding this analysis, any method may be used as long as the object can be cut out from the image.
[0027]
On the other hand, when the object information further includes information on the movement, “active” or “inactive” may be determined based on the similarity between the movement of the object and the movement of the viewpoint. It is assumed that there is a moving image as shown in FIG. 9A and the soccer ball 901 is flying toward the player 902. At this time, if the motion vector of the soccer ball is like 903, the jumping motion 904 may occur by following the ball. When the inner product of the vector 903 and the vector 904 is calculated, a large numerical value is obtained when the two vectors are “similar”. On the other hand, when the viewpoint moves like the vector 905, this is irrelevant to the movement 903 of the object, and the inner product of the vector 905 and the vector 903 becomes a small (or large negative) numerical value. If the motion is defined in addition to the area information for all objects in the screen, and there is no object near the viewpoint that has a large inner product with the viewpoint vector, Movement ". In such a case, it is not regarded as a jumping motion for calculating the importance as in the case described above.
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 10 is a block diagram showing the configuration of the information processing apparatus according to one embodiment of the present invention. FIG. 15 is a flowchart illustrating the following processing flow.
[0028]
The process until the image recorded on the recording medium 1013 is presented by the image information presentation unit 1009 is the same as that described at the beginning of the first embodiment, and a description thereof will be omitted. The line of sight of the user 1001 who is previewing the image is also detected by the line of sight detector 1002, and sent to the temporary storage unit 1006 as a position on the screen by the viewpoint calculation unit 1003 as in the first embodiment (see FIG. 15). S1501). On the other hand, from the image information recorded on the recording medium 1013, the motion detection unit 1008 estimates the object area and detects the direction of the motion using the luminance, color, and the like as clues (S1502). This motion detection means may be of any type, including currently used methods. The motion detection unit 1008 sends the area of the object at each time and its motion to the vector comparison unit 1004. The vector comparison unit 1004 compares this movement with the movement of the line of sight stored in the temporary storage unit 1006 (S1503). FIG. 11 schematically shows the state of the operation performed by the vector comparison unit 1004. When the image changes over time as shown in a, b, and c in FIG. 11, the motion detection unit outputs an object area 1101 and a motion vector 1102 of the object (a and d, b, e, and c and f correspond to each other). On the other hand, assuming that the state of the line of sight at that time is as shown by “○” in FIGS. 11D to 11F, the tendency 1103 of the line of sight movement can be obtained by observing the average of past number data from the respective moments of d to f. When the inner product of the motion vector 1102 of the object and the motion vector 1103 of the line of sight is obtained, a large numerical value is obtained when moving in the same direction. When the viewpoint is on or near the object and the inner product of the motion vector of the viewpoint and the object is a large number (about 70 to 80% or more of the product of the lengths of these two vectors), the line of sight follows the object. Therefore, it can be said that the object was of interest to the user. Thus, when the viewpoint is on or near one object, the inner product of the viewpoint motion and the object motion vector (or the inner product may be divided by the length of both vectors. Is the cosine of the angle formed by the two vectors). The vector comparison unit 1004 stores the result of the inner product in the temporary storage unit 1006. The importance information generation unit 1005 determines the inner product thus accumulated, and the position and movement of the object area defined by the motion detection unit 1008 by dividing the information defined by the information unit detection unit 1007 (by operating the user. This is described in the first embodiment), and the data is returned to the temporary storage unit 1006 in the form of writing back to the recording medium together with the result calculated in the unit (S1504). This importance information is recorded on the recording medium 1013 via the write control unit 1012 at an appropriate time. The “appropriate timing” has been described in the first embodiment. In addition, the original information and the importance information may be different recording media, which have also been described in the first embodiment, and will not be described here. Further, when information relating to the object area and its movement is recorded in the original information in advance, the movement detecting unit 1008 may not function.
[0029]
As described above, when the importance of an object is recorded based on the movement, the following utilization method is expected. For example, in FIG. 11, it is assumed that an object other than the person 1104 exists and its importance is lower than that of the person 1104. Then, it is determined that the person 1104 is the most main object in this scene (or cut), and an image in which the main object is at the center, that is, FIG. 11B is selected as a key image for scene search.
[0030]
【The invention's effect】
According to the present invention, it is possible to generate additional information of importance to multimedia information, and by using the importance information, a user can easily and quickly reach necessary information for himself / herself. Can be expected. Further, by using the line of sight, the input operation of the importance information can be performed without particularly consciousness of the operation. This provides an information retrieval and access environment that is easy to use even for users unfamiliar with operations in an environment where a large amount of multimedia information is available to anyone.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an information processing apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating classification based on the speed of eye movement according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating an example of a movement of an actual viewpoint of a person according to an embodiment of the present invention;
FIG. 4 is a diagram showing a calculation progress of the information processing apparatus according to one embodiment of the present invention.
FIG. 5 is a diagram showing an example of actual eye movement of a person according to an embodiment of the present invention.
FIG. 6 is a graph showing a processing method of the information processing apparatus according to one embodiment of the present invention.
FIG. 7 is a graph showing the results of an experiment performed by the method shown in one example of the present invention.
FIG. 8 is a conceptual diagram showing movement of a human viewpoint according to an embodiment of the present invention.
FIG. 9 is a conceptual diagram showing movement of a human viewpoint according to an embodiment of the present invention.
FIG. 10 is a block diagram illustrating a configuration of an information processing apparatus according to an embodiment of the present invention.
FIG. 11 is a conceptual diagram showing a processing method of the information processing apparatus according to one embodiment of the present invention.
FIG. 12 is a flowchart illustrating a processing method of the information processing apparatus according to an embodiment of the present invention.
FIG. 13 is a flowchart illustrating a processing method of the information processing apparatus according to an embodiment of the present invention.
FIG. 14 is a flowchart illustrating a processing method of the information processing apparatus according to an embodiment of the present invention.
FIG. 15 is a flowchart illustrating a processing method of the information processing apparatus according to an embodiment of the present invention.
[Explanation of symbols]
303: Viewpoint when gazing
304: Movement of the viewpoint when the eyeball jumps
501: Changes in viewpoint coordinates when the eyeball is jumping
502: Data disorder when blinking
504: Time during which the resolution of the visual system is reduced
601: Time during which the eye movements are active
603: Time when the second order difference of the information acquisition rate becomes 0
604: Minimum point of information acquisition rate
803: Jumping motion in the object
804: Jumping motion between objects
805: Jumping movement outside the object
806: Jumping movement outside the object
903: Ball motion vector
904: motion vector of eye movement following the ball
905: motion vector of eye movement unrelated to ball movement
1102: Object motion vector
1103 ... Motion vector of viewpoint

Claims (1)

Means for presenting image information to the user;
Viewpoint detection means for sequentially detecting a viewpoint position on the image information of a user who is viewing the image information,
Viewpoint motion detecting means for obtaining a viewpoint motion vector from the detected viewpoint,
Object motion detecting means for detecting a motion vector of the object in the image information,
Means for calculating an inner product of the motion vector of the object and the motion vector of the viewpoint,
Importance estimation means for estimating the importance of an object in an image based on the obtained inner product;
Information processing device having

Images