JPH07200632A - Information processor - Google Patents

Abstract

PURPOSE:To provide an information processor which can produce an index for retrieval of the image information by the also effective processing and also can properly retrieve a desired image without applying any burden to a user. CONSTITUTION:A position detector 417 detects the eye positions of a user set to the image information when an access is performed to a storage medium 410 which records the image information. Then the detected eye positions are stored in relation to the image information. When the correspondence is analyzed between an image and the eyes of the user, a marked point decider 419 calculates the coordinates of the marked point. Then an edge recognizing part 404 and an image part extractor 405 extract a partial image. Based on this extracted image, an index is generated for retrieval of images. Then the index image is shown when the image information is reproduced so that a list is generated for selection of a desired image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information processing device,
In particular, the present invention relates to an image information access device suitable for searching image information, a remote operation device for realizing simple remote operation, and an information recording device having a function of displaying a file link structure as a map on a display device.

[0002]

2. Description of the Related Art An apparatus for recording an image directly converts optical information of a scene that a user wants to take into a chemical change of particles on a film, or converts the information into an electromagnetic signal to be left on a medium. Photo cameras and movie cameras are directly used for recording on film, and electronic still cameras and video recorders are used for recording on media as signals, and magnetic media such as tapes and disks, and optical characteristics are used for media. An optical disk that partially changes, an integrated circuit having a function of holding a memory, and the like are used. With the improvement of the performance of the storage medium and the improvement of the recording method, the amount of image information that can be recorded on a single medium has increased, and further improvement is expected in the future.

In such an image recording apparatus, it is desired to separately provide a mark from the accumulated image information so that the desired image can be searched efficiently. This is a function to copy the shooting date and time on a film with a photo camera or a video camera, and a VTR (video tape recorder).
Has been put into practical use as an index function for recording a special signal at the position where the recording starts. However, in these cases, information management regarding the image contents is left to humans, and in order to search for the image portion containing the desired contents, it is necessary to follow the user's memory of the recording location or Need to scrutinize the image of. Alternatively, the description about the content and the corresponding recording location must be manually recorded and saved in advance.

In this case, in order to describe the contents corresponding to the search from an unspecified user, it is necessary to objectively describe more objects that can be searched in the image. , In order to allow retrieval from the state of objects and mutual relationships, it is necessary to depict the contents in writing. This not only requires a large amount of work for the user, but also needs to prepare an extremely large storage area for the index information, which increases the amount of medium used.

If all attempts are made to automate the understanding and description of the contents, all objects are separated from each other, the recognition of the objects, and the state of the objects are displayed on the screen in which various objects are represented only by the difference in brightness and color. It is necessary to follow a process such as the description of, and a system with advanced intelligent processing is required. Further, the above-mentioned writing description requires a larger-scale knowledge database, and for this reason, no one has been completed yet.

Therefore, conventionally, when retrieving a specific image, a target portion is searched while viewing the image, or an index in which contents are described in detail is referred to, or an advanced process is performed. It was required to search from the objective and subdivided expressions automatically given by the system, which required a great burden on the user, the index maker, and the system designer, respectively. In addition, in order for the device to understand the intended image, it is necessary to perform complicated work with an input device such as a keyboard and a mouse, and an index prepared by a person other than the user or the system is not always suitable for a specific user. There was also the problem that it was not an index.

On the other hand, the remote operation device that has been realized conventionally has been operated by an operator holding it in his hand. The most popular remote control device for home use is a so-called infrared remote controller. In this, an operator holds the infrared ray emitting section toward the machine and pushes a button to project infrared rays on the machine and give a predetermined instruction.

[0008] Further, studies have also been conducted to recognize hand gestures and gestures by utilizing image recognition technology and apply this to remote control. However, in order to recognize these by image processing, the background must be monotonous, how large the hand must be in the center of the screen, and there are severe constraints, so it is still in practical use. Not in.

[0009] In addition, there is also a study to grasp the three-dimensional shape of an object by a stereo method using the principle of triangulation and to utilize it for recognition.

On the other hand, the function of confirming what kind of file is recorded on a certain recording medium in an information device such as a computer has been conventionally used in most systems. This is because the information recorded on the recording medium cannot be seen directly by the user, so that the user can make the system read the information and display the information about the file such as the file name on the display device. .

When music or video is recorded on a cassette tape or a video tape, the user directly attaches a sticker describing the recorded contents to the tape so that the contents of the information file on the recording medium can be understood. There is. Such a method is possible only when the number of files that can be recorded on the recording medium is small. If the number of files is large, structure the files and record them,
It becomes necessary to display it easily.

The simplest file structure is that the files are one-dimensionally arranged in order. When displaying such a file, a method of displaying the file name in order, or expressing the file by an icon and arranging them in order can be adopted. At times, the display order may be rearranged and displayed according to some attribute of the file, for example, alphabetical order of file names or recording time.

A tree structure is a slightly more complicated file structure. Files belong to a unit called a directory, and this directory has a parent directory and a child directory. A directory may have multiple child directories, but generally there is only one parent directory. The tree structure of the files makes it easy for the user to handle many files. That is, the one-dimensional file structure described above means that all files exist in one directory, and when this is displayed, it is difficult to find the target file when the number of files is large. Met. Therefore, by collecting related files in the same directory, the user can search for a target file from the display of relatively few files without looking at unrelated files. As a method of displaying such a file structure, there are a method of displaying only files in one directory, a method of displaying a tree structure of a directory as a map, and the like.

On the other hand, recently, it has been attempted to create a structure that can represent the relationship of files that cannot be expressed by the tree structure of files. In the conventional tree structure, a file always belongs to one directory, and the relation between this and a file belonging to another directory cannot be freely defined. On the other hand, a structure has been introduced in which related files are linked with each other to show that they are related. This is a so-called hypertext structure, and with such a structure, the relationship between any two files can be freely expressed. You can also enter not only files but also files. For example, when there is an image of a cat in a landscape photograph as an image file, the cat portion of this image and the voice file of the bark of the cat can be linked by a link. Considering such things, hypertext has a structure that can express even a very complicated relationship.

On the other hand, however, there has been a problem that the structure becomes too complicated and it is difficult for the user to grasp what the structure is. One way to solve this is to display the hypertext structure of the file as a map. This represents a file by an icon, a link drawn in the file by a line, and a structure by connecting the files. By displaying the hypertext structure of the file directly through the map, the user can
You will be able to know the relationship between the file you are interested in and other files.

However, when the hypertext structure of a file is displayed in a map in this way, if the file icons are arranged in alphabetical order of file name, recording time order, etc., the file icons are drawn as lines according to the links provided between the files. When they are connected, the link line becomes complicated and the structure is not well understood. Also, it seems that files that are near each other seem to be related, but they are not linked because they are not linked, and there is a problem that files that are far away are actually linked by a link. This is because when the map is displayed, the links between the files are not taken into consideration, even though the positions of the files and the links have a close relationship.

[0017]

As described above, in the conventional image recording / reproducing apparatus, in order to use the index for retrieving the image intended by the user in a form that is easy to retrieve,
It is required that the video producer prepares a large-scale index or makes a complex and sophisticated image recognition system objectively describe the content. The index is redundant, not always efficient for the user, and the processing time increases. However, there are problems that the processing system is complicated and expensive, and that the burden on the user is heavy.

The conventional remote control device must be operated by holding it in the hand, and cannot be operated unless the device is at hand. There is also a problem that it is not preferable to operate with dirty hands. Even in techniques such as hand gesture recognition by image processing, it is difficult to accurately recognize the operator's hand due to the influence of noise due to immaturity of the recognition technique. For this reason, it has been extremely desirable to provide an environment in which it is possible to perform remote control only with bare hands, without holding or wearing any device such as a remote control device in the home, especially, even at home. .

Further, in the conventional information recording apparatus, when the hypertext structure of the recorded file is displayed as a map, the file structure is complicated and difficult to understand because the relationship between the position of the file icon and the link is not taken into consideration. There was a problem.

The present invention provides an image information access apparatus which can create an index for image information retrieval by an efficient process and appropriately retrieve a target image without imposing a burden on the user. The first purpose is to do so.

A second object of the present invention is to provide a remote control device capable of operating the system with only the bare hands of the operator without holding any device in the hand.

Further, a third object of the present invention is to provide an information recording device in which the file link structure can be easily understood.

[0023]

An image information access device according to the present invention includes an image information recording means for recording image information and a position of a line of sight of a user with respect to the image information when accessing the image information recording means. Based on the position of the line of sight stored in the line-of-sight position information storage unit, and a line-of-sight position information storage unit that stores the position of the line-of-sight detected by the line-of-sight detection unit in association with the image information. Index information generating means for extracting a target portion in the image information recorded in the image information recording means and generating index information for searching the image information.

The remote control device according to the present invention includes a distance image input means for inputting a distance image, a change amount detecting means for detecting a change amount of the distance image input to the distance image input means, and the obtained distance. A moving object detection unit that calculates the size, direction, and moving distance of a moving object from an image and its change amount, and recognizes that the moving object is the operator's hand, and the moving object. In the moving object tracking means, a moving object tracking means for tracking the motion of the hand detected by the detection means in the screen, a cursor display means for displaying a cursor on the screen according to the tracking result by the moving object tracking means, and the moving object tracking means. When it detects that the hand being tracked has been stationary for a predetermined time or when it has detected a predetermined movement, it executes a predetermined command registered in advance based on the cursor position. Characterized by comprising a that command execution unit.

The information recording apparatus according to the present invention includes a recording means for storing a plurality of files in association with each other by linking the related files, a plurality of files recorded in the recording means, and a link between them. Optimizing means for evaluating the parameters that determine the visibility when displaying the structure and obtaining the optimum parameters, and the link structure between the plurality of files specified by the optimum parameters obtained by the optimizing means Is provided with a display means for displaying the file icon and information associating the file icon with the file icon.

[0026]

According to the image information access device of the present invention, the position of the line of sight of the user at the time of recording or reproducing the image is detected and recorded in association with the image information, and the position of the line of sight within the image information is recorded from the recorded content. , A motion, a dwell time, and the like are used to selectively extract a portion of the image information that the user is paying attention to, and generate an index for searching the image information. This allows the user to select an object or part that is of particular interest from all the images and use the selected part as an index.

According to the present invention, during normal observation, recording, or reproduction of an image, only a portion of high interest is specified and an index is generated without requiring a special information input operation, so that the burden on the user is increased. Can be significantly reduced. Further, since the analysis is performed focusing on the points of interest, the recognition and understanding of all the images are not required, and the processing is simplified.

Further, according to the remote control device of the present invention, it is not judged that the hand is a hand by analyzing the shape, but the hand is judged from the information of the general shape and the movement thereof. Specifically, for example, when the operator extends his / her hand forward (with the intention of performing an operation), it can be determined that it is a hand, so that the size of the hand is similar to a certain distance. It detects that it has moved forward (for example, about half the length of one arm) and determines that it is a hand. After that, the movement of the hand can be input by tracing the portion, so that it can be used as a pointing device on the screen. Therefore, for example, when stopped for a certain period of time or when a specific movement is performed, a specific command can be executed based on the cursor position.

According to the present invention, it is possible to realize an environment in which the system can be operated only with bare hands without holding or mounting any device such as a remote controller on the hand.

Further, according to the information recording apparatus of the present invention, as the parameter of the visibility of the map, for example, the number of intersections of the lines representing the links when displayed, the distribution of the lines representing the links,
Use one or some of the total lengths of the lines that represent links to combine and use the combination optimizing means to find and display the combination of file icon display positions that optimizes the value of this parameter. Therefore, when the hypertext structure of the file is displayed as a map on the display device, the file icon is displayed at the position where a certain parameter is optimized. Since this parameter is selected to represent the legibility, the display position that optimizes this value should be excellent in terms of legibility. By displaying the map in this way, it is the original role of the map display. The display is excellent in the sense that the user understands the file structure.

[0031]

Embodiments of the present invention will be described below with reference to the drawings.

First, an image information access apparatus according to an embodiment of the present invention will be described with reference to FIGS.

FIG. 1 is a block diagram showing the arrangement of an image recording apparatus, showing an example of a digital video apparatus using an optical disc. This device is an example in which the photographer views the photographer's line of sight at the time of shooting a moving image and selects an object that can be an index in an image when the device himself or herself points the device at the subject.

A video signal to be recorded is input through an input terminal 101, quantized by an A / D converter 102, converted into a signal suitable for recording by an encoder 103, and a recording signal generator 104. Through storage medium 1
Recorded in 05.

At the same time, the movement of the photographer's eyeball is supplied to the input terminal 106 as an electric signal. This eye movement signal is quantized by the A / D converter 107, and then calculated by the position detector 108 as viewpoint position information in the image. The calculated viewpoint position information is used as the encoding device 109.
Is converted into a signal suitable for recording by the storage medium 10 by the output of the recording signal generator 104 together with the video signal.
Recorded in 5.

Here, the viewpoint position information is obtained by a recording signal generator different from the recording signal generator 104 for the video signal.
You may store in the area | region different from the video signal of the storage medium 105. Further, both the storage signal generator and the storage medium may be recorded by using different ones from the video signal.

A signal for transmitting a photographer's instruction to the system controller 111 is supplied to the input terminal 110 through a switch, a button and the like. With this signal, the image recording apparatus controls the start and end of recording, zoom-in, zoom-out, and the like. The device control information 112 is the encoding device 113.
Then, the signal is also converted into a signal suitable for recording, and is recorded in the storage medium 105 by the output of the recording signal generator 104 together with the video signal and the line-of-sight position information.

As the instruction method by the photographer, an input signal for notifying the device of the object to be indexed is prepared, and the device control information 112 from the input terminal 110 via the above path is prepared. May be included in.

Here, the device control information is the coding device 113.
After that, it may be stored in an area of the storage medium 105 different from that of the video signal by a recording signal generator different from the video signal recording signal generator 104. Further, both the storage signal generator and the storage medium may be recorded by using different ones from the video signal.

FIG. 2 is a block diagram showing the structure of the index creating device. This apparatus creates an index on the basis of the video signal recorded by the image recording apparatus shown in FIG. 1, the line-of-sight position information, and the apparatus control information.

The video signal, line-of-sight position information and device control information stored in the storage medium 201 are stored in the storage signal reading device 20.
2 and is divided into a video signal 203, line-of-sight position information 217, and device control information 212. Here, the storage medium is, as described above, a video signal, line-of-sight position information,
Different device control information may be used. In addition, the same storage medium may be used, or different reading devices may be used to retrieve from different prerecorded areas. The video signal 203 is converted from a recording signal into a digital image signal by the decoder 204, and an image for a fixed time is stored in the image storage unit 205.

On the other hand, the device control information 212 is similarly converted from the recording signal to the digital control signal by the decoder 213 and supplied to the system controller 214. The system controller 214 receives the user's instruction input from the input terminal 215, manages the entire apparatus, detects a portion (cut) at which continuous recording is interrupted from the recorded apparatus control information, and cuts this. Information 21
6 is supplied to the image storage unit 205. The image storage unit 205 receives this and stores the image in units of cuts.

Similarly, the line-of-sight position information 217 is also converted from a recording signal into a digital signal by the decoder 218. The signal at this stage merely represents the position of the line of sight in each frame of shooting. Eye movements are roughly divided into three categories: fixational eye movements that cause subtle vibrations even when gazing at one point, following movements when tracking a moving object, and jumping movements when moving between visual objects. To be done. The type determination machine 219
From the line-of-sight position information digitized by the decoder 218, the nature of such movement is determined, and information regarding the type is added to the line-of-sight position information. The line-of-sight position information added with the information regarding the type of eye movement is the gazing point determiner 22.
Supplied to zero. The gazing point determiner 220 supplies the coordinates on the screen that can be determined to be the gazing point to the edge recognition device 206 based on the motion type information from the type determiner 219.

On the other hand, the image in the series of cuts is supplied from the image storage unit 205 to the edge recognition device 206 as a scanning unit (one frame) of the image or as a group of continuous frames, and the image information and the gazing point are determined. Based on the gazing point coordinate information supplied from the device 220, a boundary line (edge) between the gazing target object and its background and foreground is determined. Here, the edge recognition device 206 may be instructed by the user through the system controller 214 so that the edge is recognized in accordance with the instruction.

The image information is supplied to the image partial extractor 207 with information about the contour of the object including the gazing point added to the frame where the gazing point exists. Image part extractor 20
At 7, the object including the gazing point (target object) is extracted as a partial image for all the frames in which the gazing point exists, and is supplied to the duplication checker 208. Here, a plurality of target objects may exist in the cut or in the same frame.

The duplication inspector 208 inspects that the same object has a different posture in different frames or has a different size due to zooming, and that they are the same, and the supplied image of the object of interest is supplied. At the same time, it is supplied to the image selector 209 as overlapping information. Here, the duplication inspector 208 may inspect not only the cut being inspected but also the presence or absence of duplication across the cut. Further, when the image recording apparatus has previously recorded an instruction for notifying the user of an object to be used as an index, this may be directly added to the object of the index candidate.

The image selector 209 selects from among a plurality of supplied target objects, based on secondary information obtained from the target object image such as the number of appearances, gaze time, and size, in addition to the overlapping information with the target object image. An object suitable for indexing is selected, and an image of the same object duplicated is selected in a state suitable for indexing. The index selected here uses a method such as making all candidates for which the ratio of the gaze time to the object with respect to the cut duration exceeds a certain value as a candidate,
There may be more than one in the cut. Further, a plurality of same objects may exist in the index in different states. Further, the selection work here may be performed by giving a user's instruction through the system controller 214 and selecting an image as an index according to the instruction.

The object determined to be the index as described above is output as the extracted partial image itself, or information indicating the frame of the indexed image and information indicating the image area to be indexed. Information of other frames (related frame information) including the object is also created and output based on the duplicated information.

The index image or the area information of the index image and the related frame information output from the image selector 209 are converted into a signal suitable for recording by the encoding device 210,
It is re-recorded in the storage medium 201 by the output of the recording signal generator 211. Here, the storage medium used for recording the index may be different from the storage medium 201 in which the original image and the line-of-sight position information are recorded.

FIG. 3 is a block diagram showing the structure of the image reproduction / retrieval device. This device presents an index based on the video signal, index image signal and related frame information recorded by the index creating device of FIG. 2 to enable the user to search for the target image.

The video signal, index image signal and related frame information stored in the storage medium 301 are stored in the storage signal reading device 3.
02, and is divided into a video signal 303, an index image signal 314, and related frame information 308. Here, the storage medium is a video signal, an index image signal,
Different pieces of related frame information may be used. Also,
Even if the same storage medium is used and different reading devices are used,
It may be taken out from a different area recorded in advance.

For normal playback, which requires index lookup,
The video signal 303 is converted from a recording signal into a digital image signal by a decoder 304, and colors, display ranges, captions, etc. are displayed by a screen creator 305 based on an instruction from a system controller 312 which receives an instruction from a user through an input terminal 311. Is processed, converted into a signal to be output to the monitor television by the D / A converter 306, and then output.
Here, the D / A converter 306 does not have to operate when the monitor television is one that allows digital data.

On the other hand, the index image signal 314 is converted into a digital image signal by the decoder 315 and stored in the index image storage unit 316 in preparation for the search by the user. Here, as described above, in the case of the method of recording the position (address) of the image to be the index image and the display area in the frame instead of the index image, the image is displayed based on this position and display area information. The data is referred to and similarly stored as an image in the index image storage unit 316.

The related frame information 308 in which the frame in which the object to be referred to by the index exists is recorded is converted into a digital signal by the decoder 309 and stored in the index related information storage unit 310. In the index related information storage unit 310, the storage location in the index image storage unit 316 of the image designated as the index and the address of the frame in which the object corresponding to the index exists are stored, and this information is stored in the system controller 312. Can be supplied, or this correspondence can be changed or updated by an instruction from the system controller 312 which receives an instruction from the user through the input terminal 311.

The system controller 312 causes the list creation device 317 to create an index list according to the user's search request instruction. The list creation device 317 receives the index image in the range desired by the user from the index image storage unit 316 and sends it to the screen creator 305. Screen creator 305
Then, the screen to be presented to the user through the instruction of the system controller 313 is created by combining the original video signal and the index list, and is supplied to the D / A converter 306. D /
The A converter 306 converts the signal to be output to the monitor TV and outputs the signal to the output terminal 307. Here, the D / A converter 306 does not have to operate when the monitor television is one that allows digital data.

The user selects the target object from the instructed index and transmits the instruction to the input terminal 311 through the input means such as the switch and the button. A plurality of target objects may be used here. Alternatively, a method may be used in which the line of sight is used again as a means for selecting an index, and the fact that the user gazes at the image to be selected is recognized as the intention of selection.

The system controller 312 inquires of the index related information storage unit 310 about the frame associated with the selected index, and supplies the target image to the screen in the same manner as in normal reproduction.

In the above-described embodiment, the line-of-sight position detection, the index image creation, and the search / playback are performed by different devices, but any two or all three of these are performed by the same device. Good.

FIG. 4 is a block diagram showing the structure of the image recording device / index creating device, and is a diagram showing an embodiment in which the line-of-sight position detection and index image creation are performed by the same device.

The video signal to be recorded is supplied to the input terminal 401, quantized by the A / D converter 402, converted into a signal suitable for recording by the encoder 411, and stored by the recording signal generator 409. Medium 410
Recorded in. At the same time, images for a certain period of time are stored in the image storage unit 403.

On the other hand, at the same time, the movement of the photographer's eyeball is supplied to the input terminal 415 as an electric signal. The eye movement signal is quantized by the A / D conversion device 416 and then calculated by the position detector 417 as viewpoint position information in the image. The calculated viewpoint position information is judged by the type judging device 418 as to the nature of the motion. The line-of-sight position information added with the information regarding the type of eye movement is supplied to the gazing point determiner 419. The gazing point determiner 419 supplies the coordinates on the screen that can be determined to be the gazing point to the edge recognition device 404 based on the motion type information from the type determiner 418.

A signal for transmitting a photographer's instruction to the system controller 413 is supplied to the input terminal 412 through a switch, a button, or the like. With this signal, the image recording apparatus controls the start and end of recording, zoom-in, zoom-out, and the like. The device control information 420 includes an encoding device 421.
Is converted into a signal suitable for recording, and is recorded in the storage medium 410 by the output of the recording signal generator 409 together with the video signal and index image information. In addition, the system controller 413 is a unit (cut) of continuous recording.
Is detected and used as cut information 414 in the image storage unit 4
Supply to 03. The image storage unit 403 receives this and stores images in cut units.

On the other hand, the image in the series of cuts is supplied from the image storage unit 403 to the edge recognition device 404 as an image scanning unit (one frame) or as a group of continuous frames, and the image information and the gazing point determiner are supplied. Based on the gazing point coordinate information supplied from 419, the boundary line (edge) between the gazing target object and its background and foreground is determined. Here, the edge recognition device 404 may be instructed by the user through the system controller 413 to recognize the edge in accordance with the instruction.

The image information is supplied to the image part extractor 405 with information about the contour of the object including the gazing point added to the frame in which the gazing point exists. Image part extractor 40
In 5, the object including the gazing point (target object) is extracted as a partial image for all the frames in which the gazing point exists, and is supplied to the duplication checker 406. Here, a plurality of target objects may exist in the cut or in the same frame.

The duplication inspector 406 inspects that the same objects having different postures in different frames or different sizes due to zooming are the same, and the supplied image of the object of interest. Along with this, it is supplied to the image selector 407 as overlapping information. Here, the duplication inspector 406 may inspect not only the cut being inspected but also the presence or absence of duplication across the cut.

The image selector 407 selects from among a plurality of supplied objects of interest, based on side information obtained from the object of interest image such as the number of appearances, gaze time, and size in addition to the overlapping information of the object of interest image. An object suitable for indexing is selected, and an image of the same object duplicated is selected in a state suitable for indexing. The index selected here uses a method such as making all candidates for which the ratio of the gaze time to the object with respect to the cut duration exceeds a certain value as a candidate,
There may be more than one in the cut. Further, a plurality of same objects may exist in the index in different states. Further, the selection work here may be performed by giving a user's instruction through the system controller 413 and selecting an image as an index in accordance with the instruction.

The object determined to be the index as described above is output as the extracted partial image itself, or the information indicating the frame of the indexed image and the information indicating the image area to be indexed. Information on other frames including the object is also created and output based on the duplicate information.

The index image or the area information of the index image and the related frame information output from the image selector 407 are converted into a signal suitable for recording by the encoder 408,
It is recorded in the storage medium 410 by the output of the recording signal generator 409. Here, the storage medium used to record the index is
Storage medium 41 in which the original image and device control information are recorded
A value different from 0 may be used.

As the photographer's instruction, an input signal for notifying the device of the object to be indexed while photographing is prepared, and input from the input terminal 412 to the device control information 420 passing through the above path. May be included.

In the above, the video signal, the index picture signal, the related frame information, and the device control information are stored in the storage medium 41 by the different recording signal generators after the encoding device 421.
It may be stored in different areas of 0. Further, different recording signal generators and different storage media may be used for recording.

FIG. 5 is a diagram showing the concept of an image recording device such as a digital video device using another optical disk. While observing the line of sight of the photographer at the time of shooting a moving image, the device is automatically oriented in the direction of the line of sight. , An example of simultaneously recording an object that can be an index in an image is shown. This embodiment is different from the previous embodiments in that the photographer himself does not change the shooting direction of the recording apparatus. Further, as an example, a case will be described in which a medium having a dedicated optical system for recording an index image and a medium for recording a photographed image intended by a photographer and a medium for recording an index image are different.

The image recording apparatus main body 501 has two optical systems, one for taking a photograph intended by the photographer and the other for taking an index image, as described above. Photographer 50
A line-of-sight detector 502 is attached to the fourth eye portion and is connected to the main body 501 by wire or wirelessly. Body 501
Alternatively, the line-of-sight detector 502 has a function of sensing the positional relationship between each other, and the positional relationship is grasped by the main body 501. The photographer 504 holds the terminal 503,
This controls the apparatus main body 501. Terminal 503
Is wired or wirelessly connected to the main body 501. 5
Reference numeral 05 denotes a subject that the photographer 504 is about to photograph.

Image 5 taken by the main body 501
Of the images 06 and 508, one image 506 is the photographer 50.
What is zoomed by the instruction of 4 (user image)
Is. When the photographer 504 gazes at the subject 505, the main body 501 automatically turns to the direction of the subject 505, and photographs with the enlargement ratio instructed by the photographer 504 by the terminal 503. The symbol 507 in the user image 506 indicates that the viewpoint of the photographer 504 is displayed. The other image 508 is an index image, and is captured around the gazing point using a different optical system from the user image 506 and at a magnification rate that is predetermined by the device side.

FIG. 6 is a block diagram showing the structure of this image recording apparatus, and corresponds to the processing apparatus inside the main body 501 in the conceptual diagram of FIG.

Video signals input from the two optical systems are supplied to the apparatus as a user image signal 601 and an index image signal 606. The user image signal is converted into a digital signal by the A / D converter 602, and then encoded by the encoder 60.
And the recording signal generator 60
It is recorded in the storage medium 605 via the output of No. 4. Here, a digital zoom, which is an enlargement process on the electric signal, may be performed before the encoding.

On the other hand, the index image signal supplied from the input terminal 606 is similarly converted into a digital signal by the A / D converter 607 and supplied to the digital zoom 608. Since the camera faces the gazing point of the photographer, the index image signal input as it is includes the target object.
The digital zoom 608 outputs an image signal, which is subjected to a predetermined zoom centering on the gazing point, to the image storage unit 60.
Supply to 9.

Simultaneously with photographing, an eye movement signal is supplied from the input terminal 620, and a signal indicating the positional relationship between the line-of-sight detector and the main body (imaging device) is supplied from the input terminal 624. These two signals are respectively A / D converter 621 and A / D converter 62.
5 is converted into a digital signal. After this, the eye movement signal is converted into the position of the line of sight within the visual field by the position detector 622. In parallel with this, the detector position signal is converted by the position detector 626 into information such as the orientation of the main body, the orientation of the detector and the orientation and distance of the main body / detector, and the eye gaze information to the gazing point determiner 623. Supplied.

The gazing point determiner 623 determines the absolute position of the gazing point of the photographer from the positional relationship between the main body and the detector, and the line-of-sight directions of the photographer's eyes, and informs the system controller 615 as three-dimensional coordinates. . The system controller 615 calculates the direction to which the camera should be directed and the distance to which the optical system should be focused from the obtained gazing point coordinates, and outputs it to the output terminal 619 as a camera direction displacement signal.

On the other hand, the photographer gives an instruction to the apparatus from the control terminal, and the signal is provided from the input terminal 618 to the system controller 615. System controller 6
In accordance with the instruction, 15 transmits a zoom instruction to the output terminal 614 as a user image zoom focus signal, and zooms the optical system. Here, when there is no need for optical zooming and focusing by providing a digital zoom in front of the encoding device 603, this output may be omitted. In addition, a zoom control signal for shooting a zoom image around the gazing point at a predetermined enlargement ratio is output from the output terminal 617 based on the information on the distance between the camera and the subject.

The cut detector 616 takes out from the photographer's instructions input from the input terminal 618 to the system controller 615, a cut signal relating to the start and end of recording and supplies it as a cut signal to the image storage unit 609. To do. The image storage unit 609 uses the image selector 61 as an image signal as one frame or a group of consecutive frames.
Send to 0. In the image selector 610, a target suitable for indexing is supplied from a plurality of supplied target objects based on side information obtained from the target object image such as the number of appearances and gaze time from the image in which the target object is moved. While selecting the item,
Among the duplicated images of the same object, the one suitable for indexing is selected. A plurality of indexes selected here may be present in the cut by using a method in which all the indexes in which the ratio of the gaze time to the object with respect to the cut duration exceeds a certain value are set as candidates. Further, a plurality of same objects may exist in the index in different states. The selection work here may be performed by giving a user's instruction through the system controller 618 and selecting an image as an index in accordance with the instruction.

The index image thus determined is the encoding device 6.
The signal is converted into a signal suitable for recording by 11, and is recorded in the storage medium 613 by the output of the recording signal generator 612. Here, the same storage medium as the video signal recording medium 605 may be used as the storage medium and may be stored in an area different from the video signal of the medium 605. Further, the index image signal encoded by the encoding device 611 is mixed with the video signal encoded by the encoding device 603, and the recording signal generating device 6
The output of 04 may be stored in the storage medium 605.

Further, in the present embodiment, only the enlargement of the image around the gazing point is used as an index, but as in the first embodiment,
You may recognize the edge of a target object and take it out as a partial image. At this time, the image and line-of-sight recording device and the index creating device may be integrated or may be different devices.

As another embodiment, an example of observing the line of sight of a viewer (listener) during reproduction of a moving image and recording an object that can serve as an index in an image will be described. This embodiment is different from the previous embodiment in that the user's line of sight is observed during reproduction, not during recording.

Since the block diagram showing the configuration of the present embodiment is similar to that of FIG. 4, it will be described with reference to FIG. 4 while showing the differences from the previous embodiment. Note that this embodiment is a digital video device that records a video signal and an index signal on the same medium.

The video signal is supplied to the viewer (listener) as a reproduction screen and is also supplied to the input terminal 401. The video signal is converted into a digital signal by the A / D converter 402, and an image for a fixed time is stored in the image storage unit 403. The image storage unit 403 detects a cut from the provided images and uses it as a unit of storage. In this embodiment, the channel connecting the encoding device 411 from the middle of the A / D conversion device 402 and the image storage unit 403, and the encoding device 4
11 is not needed.

On the other hand, the eye movement of the viewer (listener) is supplied from the input terminal 415 to the A / D conversion device 416 including the information on the positional relationship between the eye movement detector and the reproduction screen and converted into a digital signal. It The converted digital signal is calculated by the position detector 417 as the position of the line of sight on the screen. The calculation result is supplied to the type determiner 418, and the type of eye movement is analyzed. The analysis result is the gazing point determiner 4.
19 and gives the point of interest information to the edge recognition device 404 as the point of interest on the screen.

The image in the series of cuts is supplied to the edge recognition device 404 from the image storage unit 403 as one frame or a group of continuous frames, and the image information and the gazing point supplied from the gazing point determiner 419. From the coordinate information, the edge of the gazed object is determined. Here, the edge recognition device 404 includes a system controller 4
A user's instruction may be given through 13 and the edge may be recognized in accordance with the instruction.

The image information is supplied to the image part extractor 405 together with the information about the contour of the object including the gazing point in the frame where the gazing point exists. The image portion extractor 405 extracts an object (object of interest) including the gazing point as a partial image for all frames in which the gazing point exists, and supplies it to the duplication checker 406. Here, a plurality of target objects may exist in the cut or in the same frame.

The overlap inspector 406 inspects that the same object has a different posture in different frames or has a different size due to zooming, and that they are the same, and the supplied image of the object of interest is supplied. Along with this, it is supplied to the image selector 407 as overlapping information. Here, the duplication inspector 406 may inspect not only the cut being inspected but also the presence or absence of duplication across the cut.

The image selector 407 selects from among a plurality of supplied objects of interest, based on secondary information obtained from the object of interest image such as the number of appearances, gaze time, and size in addition to the overlapping information of the object of interest image. An object suitable for indexing is selected, and an image of the same object duplicated is selected in a state suitable for indexing. The index selected here uses a method such as making all candidates for which the ratio of the gaze time to the object with respect to the cut duration exceeds a certain value as a candidate,
There may be more than one in the cut. Further, a plurality of same objects may exist in the index in different states. Further, the selection work here may be performed by giving a user's instruction through the system controller 413 and selecting an image as an index in accordance with the instruction.

The object determined to be the index as described above is output as the information indicating the extracted partial image or the frame of the indexed image and the information indicating the image area to be indexed, and together with that information. The information of the frame including the object is also created and output based on the duplicate information.

The index image or the area information of the index image and the related frame information output from the image selector 407 are converted into a signal suitable for recording by the encoder 408,
It is recorded on the recording medium 410 by the output of the recording signal generator 409. Here, the storage medium used to record the index is
Storage medium 41 in which the original image and device control information are recorded
A value different from 0 may be used.

As an instruction of a viewer (listener), an input signal for notifying the device when an object to be used as an index on the screen is focused is prepared, and device control from the input terminal 412 through the above path is performed. It may be included in the information 420.

In the above, the index image signal and the index related information,
After the encoding device 421, the device control information may be stored in different areas of the storage medium 410 by different recording signal generation devices. Further, different recording signal generators and different storage media may be used for recording.

In this embodiment, the line-of-sight detection and the index creation are performed by the same device, but the line-of-sight detection device and the index creation device may be different. In addition, a plurality of eye movement detectors are prepared, the viewpoints of a plurality of persons are analyzed at the same time, and the image selector 4
You may use it for the judgment criterion of 07. Further, when the same image is repeatedly reproduced, the index up to the previous time and the new viewpoint information may be input to the image selector 407 as a criterion to update the index.

In this embodiment, the same image reproduction / retrieval device as that in the embodiment of FIG. 3 is used. Here, the line-of-sight detection / cumulative index generation device and the image reproduction / retrieval device may be integrated.

In the above embodiments, the description has been made assuming that the recording system is the digital system, but the recording system may be used for the analog system video. In addition to the optical disc, a magnetic disc, a magnetic tape, a memory element, or the like may be used as the recording medium. Further, although the description has been made regarding the moving image, the present invention may be applied to an apparatus for recording a still image. In addition, the example in which the position of the line of sight in the image is calculated in the viewpoint detection device has been described, but only the direction of the viewpoint may be recorded and the position of the line of sight in the image may be calculated by the index creation device. In the previous example, the target object was extracted using the edge of the target object, but at the time of shooting, the distances from the camera to all the objects in the image were measured and recorded, and the information was extracted from the information. The object may be extracted as a set of similar distances including the viewpoint.
Further, regarding the embodiment of FIG. 5, although an optical system different from that for user image photographing is used for photographing the index image, it may be photographed using the same optical system. In this case, one input image may be branched into two and the same processing as that in the above-described embodiment may be performed, or the user image shooting and the index image shooting may be performed alternately by temporally dividing.

According to the above-mentioned image information access device, the eye movement of a human being when an image to be indexed is presented is observed, the gazing point is judged from the movement state, and the image portion centered on the gazing point is displayed. Since the characteristic image element is obtained, the index can be created without recognizing all the objects in the image. In addition, since the object that the photographer is interested in is extracted from the index thus created, an efficient index can be created.

Further, since the area to be handled by the image recognition is reduced, the index creation process can be simplified and the processing speed can be improved. Further, since the recording amount is reduced, the recording medium can be saved.

Next, a remote control device according to an embodiment of the present invention will be described with reference to FIGS.

FIG. 7 is a functional block diagram of this remote control device.

First, the area where the user's hand is present is input by the distance image input means 701, and the change amount detection means 702 obtains the time difference of the distance image. The moving object detection means 703
From the distance image and its time difference, the size of the object moved in which direction is calculated. By this series of processing, when the user extends his / her hand forward, the movement can be detected.

Each of the above means will be described in more detail.

Steps S1 to S4 in FIG. 9 show the flow of processing in this apparatus.

First, the moving area is extracted from the distance image (S1). Various types of distance image input means 701 have been proposed. Non-contact distance measurement methods include a stereo method that uses the principle of triangulation, a method that detects the focal plane of the lens, and a method that measures the phase difference or time difference between the projected modulated light and the reflected light. Change amount detecting means 70
In 2, the background image having no change is removed by taking the difference between two consecutive frames on the time axis. The moving object detection unit 703 detects a moving object from the distance image and the change amount of the distance image.

Next, pixels having a short distance in the motion area are separated and it is determined that they are the same object (S2). Then, it is determined from the distance and size of the object whether it is a hand (S3). For the purpose of detecting the hand when it is extended, it is only necessary to be able to detect an object having the same size as the hand. At this time, it is conceivable that the operator happens to hold something in his hand, so it is not necessary to strictly determine the shape of the hand. Rather, in order not to mistakenly judge an object as large as a hand to be a hand, it is necessary to optimally decide what movement of this object should be judged as a hand. Here, it is detected when the hand is extended forward. Therefore, it is detected that an object about the same size as the hand has advanced toward the detector by a certain range of distance (S4).

If it is desired to reduce the noise, it may be determined that the object is a hand when the object of the same size as the hand approaches and then remains stationary for a certain period of time. Also, if there is a spatial resolution smaller than the thickness of a finger, there is an object that is about the size of a hand,
Further, there is also a method of determining that it is a hand even when a part of it locally protrudes toward the detector. In this case, it is detected when the hand is extended with the finger directed toward the detector. Moreover, the same effect can be obtained by extending a hand with a rod-shaped object such as a pen toward the detector, which is a natural shape in a work pointed by a person.

FIG. 8 is a functional block diagram showing another embodiment of the remote control device of the present invention.

The distance image input means 801, the change amount detecting means 802 and the moving object detecting means 803 are the same as those in FIG. 1, and the processing after the moving object detecting means 803 detects the hand is performed in the lower part. There is. FIG. 9 shows the processing of this device.

When the moving object tracking means 803 detects that an object having the same size as the hand approaches the front (S
4), the subsequent processing is entrusted to the moving object tracking means 804. When the processing moves to the moving object detecting means 803, the cursor display means 805 displays the cursor on the portion pointed by the hand according to the tracking result of the moving object tracking means 804 (S).
5). The motion vector detected by the moving object tracking unit 804 is sequentially stored in the motion vector storage unit 806 (S6). The movement command determination unit 807 detects when the hand makes a special movement given in advance (S7). When a special movement is detected, the command execution unit 809 executes a command registered in advance (S8). The command to be executed when performing any movement is registered in advance in the motion command registration unit 808. The hand movements registered here and the commands corresponding thereto include, for example, when the user has been stationary for a certain period of time, the menu at that point is selected, and after moving for a certain period of time, it moves downward only a short distance, and When you come back to (that is, when you do something like "push" a button), select the menu at that point, hold your open hand at some point, move it and open it again There are various possibilities such as a drag operation (open / closed captures a change in the size of an object detected as a hand).

Next, with reference to FIGS. 10 to 12, an information recording apparatus according to an embodiment of the present invention will be described.

FIG. 10 is a functional block diagram of the information reproducing apparatus according to the embodiment of the present invention.

In this embodiment, one moving image, still image, voice, text, etc. information file is represented by one icon on the map, and the link between the information files is an icon corresponding to each information file. Express by connecting with a line.

The input interface 901 is a remote control button, a handwriting input device, a keyboard, or a pointing device such as a mouse or a trackball. The command to display the structure of the information file and the link is as follows:
The user does this through the input interface 901. The information file / link recording device 902 is a large-capacity recording medium such as a hard disk or an optical disk. In addition to moving image and still image data, voice data and text data can be recorded here, and information on links between them can also be recorded. The display device 903 displays images, messages, etc. to the user. The combination optimizing unit 904 calculates and outputs a combination that maximizes or minimizes a given parameter from the arrangement of icons in the information file. The simplest combination optimizing means 904 performs a full search, calculates parameter values for all combinations, and outputs the combinations that maximize or minimize this value. The control device 905 controls the entire system.

When the user wants to know the structure of the information file and the link provided between the files, the user inputs a map display command to the input interface 901. When the map display command is input, the control device 905 reads the information file and the link information set between the information files from the information file / link recording device 902. If the information file / link recording device 902 is a hard disk, an optical disk, or a floppy disk, the file name, information about the information file such as what icon to display, and the link information are read into the buffer once and the map is created. When there is a display command, information about the information file and link information may be read from the buffer. The information / link information about the read information file is the combination optimizing means 9.
04, and the display position of the information file link on the display device 903 is determined.

By the determination of the display position, for example, the map of the information fal link which is normally displayed as shown in FIG. 11 is displayed more easily as shown in FIG. However, (1) and (2) represent icons of video files, and the lines connecting these represent the links provided between the information files.

The display position is determined, for example, as follows. First, the display screen of the display device 903 is divided into some sections. An information file icon representing an information file is displayed in this section. Maximum of 1 in one section
Two information file icons are displayed. Files with links are displayed so that information file icons representing them are connected by a line representing a link. The combination of the display positions of the information file icons is M, where N is the number of information file icons and M is the number of sections.
!! / (M-N)! There is a street. From this combination,
The combination optimizing means 904 selects one combination, which is determined by the value of the parameter representing the legibility specified by the combination. That is, when the number of intersections of lines that represent links is adopted as a representative parameter of visibility, select a combination that minimizes this value, and if there are multiple combinations that minimize the parameter, select which one. You may choose. In addition to the number of intersecting link lines, the distribution of the lengths of the link lines, the total sum of the lengths of the link lines, and the like can be used as the representative parameter of the visibility. It is possible to use one of these to select the optimum combination, or to combine several parameters to create a new parameter and use that to select the optimum combination. In addition, it may be possible to consider whether the information can be displayed on one screen, or whether the information file icons are displayed in one direction.

The combination optimizing means 904 selects a combination that optimizes these parameters. The simplest method is to calculate parameters for all combinations and select the optimum one. Generally, this method requires a great deal of calculation as N and M increase. Therefore, there is also a method in which the parameters are calculated for some combinations instead of calculating all, and the optimum one is selected and approximated to the actual optimum solution. In general, the optimum combination cannot be obtained unless calculation is performed for all combinations. However, if it is not always the optimum combination and some optimization is sufficient, use the approximation method of the optimum solution. Many. Also in this embodiment, such an approximation may be used.

When the display position of the information file icon / link is determined by the combination optimizing means 904, the map is displayed on the display device 903 accordingly. At this time, it is also possible to partially zoom or pan the map by inputting a command via the input interface 901.

[0120]

As described above, according to the image information access apparatus of the present invention, the line of sight of the user at the time of recording or reproducing an image is detected, and the line of sight of the user is recorded together with the image information, which is characteristic for the user. Since various objects are automatically and selectively extracted, the burden on the user is small, and it is possible to efficiently create an index with a simple processing system and reduce processing time and recording medium consumption.

Further, according to the remote control device of the present invention, without holding any device such as a remote control device in the hand,
In addition, it is possible to realize an environment in which the system can be operated only with bare hands without wearing it. As a result, it is possible not only to replace the current remote controller in the home, but also to meet the demands such as wanting to operate while holding something in the hand and operating the machine with dirty hands.

Further, according to the information recording apparatus of the present invention, the display position of the file icon is determined in consideration of the display position of the file icon and the display of the link between the files. Even in the map display of a file having a hypertext structure, which is a complicated structure, the user can easily view the map. As a result, the user can quickly find the file he is looking for, grasp the position of the file he is looking at, and easily reconstruct the hypertext structure. The present invention is applicable to map displays of anything with a hypertext structure.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an image recording apparatus according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of an index creating device according to an embodiment of the present invention.

FIG. 3 is a block diagram showing a configuration of an image reproduction / retrieval device according to an embodiment of the present invention.

FIG. 4 is a block diagram showing the configuration of an image recording device / index device according to an embodiment of the present invention.

FIG. 5 is a conceptual diagram showing the configuration of another image recording apparatus according to the embodiment of the invention.

6 is a block diagram showing the configuration of the image recording apparatus of FIG.

FIG. 7 is a functional block diagram of the remote control device according to the embodiment of the present invention.

FIG. 8 is a functional block diagram of another remote control device according to the embodiment of the present invention.

FIG. 9 is a flowchart showing processing of the apparatus.

FIG. 10 is a functional block diagram of an information reproducing device according to an embodiment of the present invention.

FIG. 11 is a diagram showing a map before optimization in the apparatus.

FIG. 12 is a diagram showing a map after optimization in the apparatus.

[Explanation of symbols]

101 ... Video signal, 104 ... Recording signal generator, 105
Storage medium, 106 ... Eye movement signal, 108 ... Eye position detector, 110 ... Operation signal, 202 ... Storage signal reading device, 206 ... Edge recognition device, 207 ... Image part extractor, 208 ... Duplication checker, 209 ... Image selector 219
... Type judging device, 220 ... Gaze point deciding device, 303 ... Video signal, 305 ... Screen creating device, 307 ... Monitor output terminal,
308 ... Related frame information, 317 ... List creation device,
401 ... Video signal, 412 ... Operation signal, 415 ... Eye movement signal, 501 ... Image recording device main body, 502 ... Line-of-sight detector, 504 ... Photographer, 505 ... Subject, 508 ... Index image, 601 ... User image signal, 606 … Index image signal,
608 ... Digital zoom, 614 ... User image zoom focus signal, 616 ... Cut detector, 617 ... Zoom control signal, 618 ... Operation signal, 619 ... Camera direction displacement signal, 620 ... Eye movement signal, 624 ... Detector position signal, 701, 801 ... Distance image input means, 70
2, 802 ... Change amount detecting means, 703, 803 ... Moving object detecting means, 804 ... Moving object tracking means, 807 ... Motion command determining section, 901 ... Input interface, 902
... information false link recording device, 903 ... display device,
904 ... Combination optimization means, 905 ... Control device.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location G06T 1/00 7/00 G06F 15/62 380 415

Claims (4)

[Claims] 1. An image information recording means for recording image information, a visual axis detecting means for detecting a position of a visual axis of a user with respect to the image information when accessing the image information recording means, and the visual axis detecting means for detecting the visual axis position. Line-of-sight position information storage means for storing the position of the selected line-of-sight in association with the image information, and in the image information recorded in the image information recording means based on the position of the line-of-sight stored in the line-of-sight position information storage means. An image information access device, comprising: index information generating means for extracting a portion of interest and generating index information for searching the image information. 2. A range image input means for inputting a range image, a change amount detecting means for detecting a change amount of the range image input to the range image input means, and a range image obtained and a movement based on the change amount. A moving object detecting means for recognizing that the moving object is the operator's hand by calculating the size, direction and moving distance of the moving object, and the hand detected by the moving object detecting means. A moving object tracking means for tracking the movement of the moving object on the screen, a cursor display means for displaying a cursor on the screen according to the tracking result by the moving object tracking means, and a hand tracked by the moving object tracking means is predetermined. A remote control device comprising: a command execution means for executing a predetermined command registered in advance based on the cursor position when it is detected that the user has stopped for a certain time. 3. A range image input means for inputting a range image, a change amount detecting means for detecting a change amount of the range image input to the range image input means, and a range image obtained and a movement based on the change amount. A moving object detecting means for recognizing that the moving object is the operator's hand by calculating the size, direction and moving distance of the moving object, and the hand detected by the moving object detecting means. A moving object tracking means for tracking the movement in the screen, a cursor display means for displaying a cursor on the screen according to the tracking result by the moving object tracking means, and a hand that has been tracked by the moving object tracking means in advance. A remote control device comprising: a command executing means for executing a predetermined command registered in advance based on the position of the cursor when detecting that a predetermined movement is made. 4. A recording means for storing a plurality of files in association with each other by linking them with each other and a structure for displaying a plurality of files recorded in the recording means and a structure of links between them, which is easy to see. And a linking structure between the plurality of files specified by the optimal parameter obtained by this optimizing means and the optimizing means for evaluating the parameter that determines An information recording apparatus comprising: a display unit for displaying information.