JP3372096B2 - Image information access device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION , Painting Suitable for searching image information
Image information access equipment In place Related. 2. Description of the Related Art An apparatus for recording an image is used for photographing by a user.
Chemical information of particles on film directly
Or convert the information into an electromagnetic signal
To remain. Photographs to be recorded directly on film
Cameras and cinema cameras, those that record on the medium as signals
Electronic still cameras and video recorders
Media, such as magnetic tapes and disks, optical
Maintains optical disks and memories that partially change characteristics
An integrated circuit having a function is used. Storage media
Recording on a single medium by improving the recording performance and recording method
The amount of image information that can be recorded has increased, and further improvements are expected in the future.
I have been waiting. In such an image recording apparatus, the stored
Efficiently searches for the desired image separately from image information
It is desirable to have a mark that can be searched. this is
Filming date and time taken with a photo camera or video camera
Function to copy to video, VTR (video tape recorder)
To record a special signal at the recording start position with
It has been put to practical use as a function. But in these cases,
Information management on image content is left to humans,
To search for the part of the image that contains the content
Follow the user's memory of
All images need to be scrutinized. Or,
Manual description of the content and the corresponding recording location.
It must be recorded and stored. [0004] In this case, search from an unspecified user is not possible.
In order to make a content description tailored to the
Objective description of more possible objects
While it must be
In order to allow search, the content must be written
It is needed. This requires a lot of work from the user
In addition, prepare an extremely large storage area for index information
And the amount of media used increases. [0005] Attempting to automate all understanding and description of contents
Then, various objects are represented only by differences in brightness and color
From among the objects on the screen
Follow processes such as separation, object recognition, and state description
Must have a system with sophisticated intelligent processing.
It becomes important. In addition, the descriptive description as described above further includes
A large knowledge database is needed and for this reason
Nothing has been completed yet. Therefore, conventionally, when searching for a specific image,
Search for the desired part while browsing the images, or
Please refer to the index which is described in detail
Search or automatically attached by advanced processing system.
Search from given objective and fragmented expressions
Required, user, indexer, system design respectively
Required a heavy burden on the people. Also, the intended image
Input, such as a keyboard or mouse,
Complicated work by the device is required, and moreover,
The index prepared by the person or the system is not necessarily specified
There is also a problem that the index is not suitable for the user of the. SUMMARY OF THE INVENTION Problems to be Solved by the Invention As described above, the prior art
The image recording / reproducing apparatus searches for an image intended by the user.
In order to use the index for search in a form that is easy to search,
A large index must be prepared by the video producer or complex and advanced
A simple image recognition system to describe objective content
Required, redundant index, not necessarily efficient for users
Undesired, increases processing time, complicated and expensive processing system
There was a problem that the burden on the user was large. According to the present invention, the efficiency is reduced without burdening the user.
Creates an index for image information search with a simple process,
Image information access that can search for
Providing equipment Eyes Target. According to the present invention, there is provided an image information processing apparatus comprising:
Access device for storing image information
Recording the image information in the image information recording means
Or the position of the line of sight of the user with respect to the image information at the time of reproduction
Gaze detecting means for detecting
A visual point that stores the position of the visual line of sight associated with the image information.
Line position information storage means;
Recorded in the image information recording means based on the position of the line of sight
Extracting means for extracting a partial image in the extracted image information,
Represented by the partial image extracted by the extracting means,
Objects that appear repeatedly in one cut unit are
Inspection means for inspecting that the object is an object;
For retrieving the image information according to the inspection result by the step
Index information generating means for generating index information.
Features. According to the image information access apparatus of the present invention, the image
Detects the position of the user's line of sight during recording or
Recorded in association with the information, and
Based on the gaze position, movement, and
Selectively extract the part of the image information that the user is paying attention to
Then, an index for searching image information is generated. This
Objects and parts of particular interest to the user
Can be selected from the list and the selected part can be used as an index.
Wear. In the present invention, normal observation, recording, or
Interest in the process of playback without requiring special information input operations
Index is generated by identifying only the
The burden on the user can be greatly reduced. Also attention
Recognition and understanding of all images because analysis is performed around points
And the process is simplified. Embodiments of the present invention will be described below with reference to the drawings.
Will be explained. First, referring to FIG. 1 to FIG.
An image information access device according to the embodiment will be described. FIG. 1 is a block diagram showing the configuration of an image recording apparatus.
In the diagram, an example of a digital video device using an optical disc
Is shown. This device allows the photographer to
Observe the photographer's line of sight when shooting movies
In this example, an object that can be an index in an image is selected. A video signal to be recorded is input through an input terminal 101.
Input and quantized by the A / D converter 102.
After that, the encoding device 103 converts the signal into a signal suitable for recording.
The storage medium 1 is converted and transmitted through the recording signal generator 104.
05 is recorded. At the same time, the input terminal 106
Eye movements are provided by electrical signals. This eye movement signal
After the signal is quantized by the A / D converter 107,
The position detector 108 calculates the viewpoint position information in the image.
Is calculated. The calculated viewpoint position information is stored in the encoding device 109.
Is converted to a signal suitable for recording by the
The output of the recording signal generator 104 is used for the storage medium 10.
5 is recorded. Here, the viewpoint position information is the recording signal of the video signal.
With a recording signal generator different from the signal generator 104,
It may be stored in an area of the storage medium 105 different from the video signal.
No. In addition, both the storage signal generator and storage medium
You may record using a different thing. The input terminal 110 has a switch, a button, etc.
To the system controller 111
A signaling signal is provided. The image recording device
Starts, ends, zooms in, zooms out, etc.
Control. The device control information 112 includes an encoding device 113
Is also converted to a signal suitable for recording
Along with the line-of-sight position information,
It is recorded on the storage medium 105 by the output. The method of instruction by the photographer is to use an index.
When shooting while paying attention to the object you want to know
Prepare an input signal to be
It may be included in the device control information 112 via the route. Here, the device control information is transmitted to the encoding device 113
After that, recording different from the recording signal generator 104 of the video signal
The signal generator differs from the video signal of the storage medium 105.
May be stored in an area. In addition, a memory signal generator,
Even if the storage medium is recorded using something different from the video signal
Good. FIG. 2 is a block diagram showing the configuration of the index creation device.
FIG. This apparatus is recorded by the image recording apparatus shown in FIG.
Recorded video signal, gaze position information and device control information
Based on this, an index is created. The video signal stored in the storage medium 201,
The line position information and the device control information are stored in the storage signal reading device 20.
2, the video signal 203, the line-of-sight position information
217 and device control information 212. here
In, the storage medium is a video signal, line-of-sight position information,
Different device control information may be used. Also the same
Even if the storage medium is used, a different reading device can be used.
It may be taken out from a different area recorded in advance. Movie
The image signal 203 is decoded by the decoder 204 from the recording signal.
Digital image signal and stored in the image storage unit 205
Images for the interval are stored. On the other hand, the device control information 212
Conversion of recording signal to digital control signal by 213
Then, it is supplied to the system controller 214. Cis
In the system controller 214, an input from the input terminal 215
To manage the entire device
Continuous recording from the recorded device control information
A cut (cut) is detected, and this is cut information 21.
The number 6 is supplied to the image storage unit 205. Image storage unit
In response to this, the image 205 stores an image in cut units. Further, the line-of-sight position information 217 is similarly decoded by the decoder.
218, the recorded signal is converted into a digital signal.
You. The signal at this stage is the position of the line of sight in each frame of shooting
It just represents. Eye movements stare at one point
Fixation tremor that makes subtle vibrations when moving
Follow-up movements when tracking objects and when moving between viewing objects
Are generally divided into three types of jumping movements. The type determination device 219
Eye position information digitized by the decoder 218
To determine the nature of these movements and
Information is added to gaze position information. Regarding types of eye movement
The gaze position information to which the information is added is
0 is supplied. In the fixation point determiner 220, a type determiner
Can be determined as the point of gaze based on the exercise type information from 219
The coordinates on the screen are supplied to the edge recognition device 206. On the other hand, the edge recognition device 206
The images in the series of cuts are stored in the
Position (one frame) or a group of consecutive frames
Is supplied as the image information and the fixation point determiner 220
From the gaze point coordinate information supplied from
Determine the boundary (edge) between an object and its background and foreground
You. Here, the system control is provided to the edge recognition device 206.
The user's instruction is given through the
Edges may be recognized. Image information is stored in the frame where the point of interest exists.
Is added with information on the outline of the object including the point of regard.
It is supplied to the image part extractor 207. Image partial extractor 20
At 7, the object including the point of interest (the object of interest) is determined by
Extracted as partial images for all frames
It is supplied to the double inspection device 208. Here the attention object is in the cut
Alternatively, there may be more than one in the same frame. In the duplicate checker 208, different frames are used.
Different postures and different sizes due to zoom
Objects that are the same
Is inspected, along with the supplied image of the target object
It is supplied to the image selector 209 as overlapping information. Heavy here
The double inspection device 208 not only cuts during inspection but also cuts
The presence or absence of overlap may be checked. Also the above image
When viewing an object that you want to index in advance on the recording device
If instructions have been recorded to inform
It may be added to the object of the indexing candidate. The image selector 209 overlaps the target object image.
Attention object such as appearance frequency, gaze time, size in addition to information
Based on the side information obtained from the image,
To select an object that is suitable for indexing
In the case of duplicate images of the same object,
Choose the one that is in the state. The index selected here is cut
The ratio of the gaze time to the object to the duration is constant
Using a method such as making all candidates that exceed
There may be more than one in the cut. In the index, the same object
May exist in different states. Further selection here
The work is performed by the user's finger through the system controller 214.
And select the image to be indexed according to the instruction.
No. The object determined as the index as described above
Is the extracted partial image itself or an index
Information indicating the frame of the image and the image area to be used as an index
Is output as information indicating
Other frame information (related frame information)
It is created and output based on the information. Index image output from image selector 209
Alternatively, the index area existence area information and related frame information
The signal is converted into a signal suitable for recording through the encoding device 210,
To the storage medium 201 by the output of the recording signal generator 211
Will be re-recorded. Here, the storage medium used for recording the index
Is a storage medium on which the original image and the line-of-sight position information are recorded
A different one from 201 may be used. FIG. 3 is a block diagram showing the structure of the image reproduction / search apparatus.
FIG. This device is operated by the indexing device shown in FIG.
Recorded video signal, index picture signal and related frame information
Based on the information, search the target image of the user by presenting the index
Is what makes it possible. The video signal and the video signal stored in the storage medium 301
The drawing signal and the related frame information are stored in the storage signal reading device 3.
02, the video signal 303, the index picture signal
314 and related frame information 308. This
Here, the storage medium is a video signal, an index picture signal,
Different related frame information may be used. Also,
Even if the same storage medium is used, different reading devices are used,
May be taken from different pre-recorded areas
No. In the case of normal reproduction requiring index reference,
The video signal 303 is decoded by the decoder 304 from the recording signal.
Is converted into a digital image signal,
The finger of the system controller 312 receiving the user's instruction
Based on the color, display range, subtitles
And the like, and monitor the monitor with the D / A converter 306.
After being converted into a signal to be output to the TV, it is output.
Here, the monitor television allows digital data
If so, the D / A converter 306 may not work
Good. On the other hand, the index image
The signal 314 is converted by the decoder 315 into a digital image signal.
The image data is stored in the index image storage unit 316. Where
There is an image that should be an index image instead of an index image like
Position (address) and display area in the frame
In the case of the method of recording
Image data based on the
It is stored in the storage unit 316. Frame in which object to be referred to by index exists
Is stored in the decoder 309.
Therefore, it is converted into a digital signal, and the index related information storage
310. In this index related information storage unit 310
Is the index of the image designated as the index in the index image storage unit 316.
The storage location and the frame where the object corresponding to the index exists
System addresses are stored in the system controller.
This information is supplied to 312 or used from input terminal 311.
Instruction of system controller 312 which received user's instruction
This correspondence can be changed and updated by the. The system controller 312 is provided for the user.
Index list in list generating device 317 according to search request instruction
Make a table. The list creation device 317 is an index image storage unit.
From 316, supply of the index image in the range desired by the user is received.
And sends it to the screen creator 305. Screen creator 305
Now, the user is given an instruction from the system controller 313.
Screens to be presented in the original video signal and index list
It is also created and supplied to the D / A converter 306. D /
A converter 306 is a signal to be output to the monitor television
And outputs it to the output terminal 307. Where the monitor
-If the television allows digital data
In this case, the D / A converter 306 may not operate. The user selects the target from the designated index.
Select an object and use input means such as switches and buttons
To the input terminal 311. Here is the purpose
There may be a plurality of objects. Also as a means to select an index
Using your gaze again, select that you gaze at the image you want to select.
A method of indicating the intention of selection may be used. The system controller 312 has selected
The frame related to the index is stored in the index related information storage unit 310.
Inquiry, the desired image is displayed on the screen in the same
Supply the image. In the above embodiment, the detection of the line of sight and the index image
Creation and search / playback are performed on different devices.
But any two or all three of these are identical
May be performed by the above-described apparatus. FIG. 4 shows the structure of the image recording device and the index creating device.
Is a block diagram showing how to detect the gaze position and create an index image.
FIG. 3 is a diagram showing an embodiment in the case where the same apparatus is used. A video signal to be recorded is supplied to an input terminal 401.
And quantized by the A / D converter 402
After that, it is converted into a signal suitable for recording by the encoding device 411.
And the recording medium 410 through the recording signal generator 409.
Will be recorded. At the same time, the image
An image is stored. On the other hand, at the same time, the input terminal 415
Eye movements are provided by electrical signals. This eye movement signal
Is quantized by the A / D converter 416 and the position
Calculated as viewpoint position information in the image by detector 417
Is done. The calculated viewpoint position information is sent to the type determiner 418.
Therefore, the character of the exercise is determined. To the type of eye movement
The gaze position information to which information about
419. In the fixation point determiner 419, the type
At the point of gaze and judgment based on the exercise type information from the fixed unit 418
The coordinates on the screen are supplied to the edge recognition device 404. The input terminal 412 has switches, buttons, etc.
To the system controller 413
A signaling signal is provided. The image recording device
Starts, ends, zooms in, zooms out, etc.
Control. The device control information 420 includes an encoding device 421
Is also converted to a signal suitable for recording
Output of the recording signal generator 409 along with the signal and index picture information.
Recorded on the storage medium 410 by force. Also the system
The controller 413 is a unit (cut) of continuous recording.
Is detected, and this is used as cut information 414 in the image storage unit 4.
03. The image storage unit 403 receives this, and
Store images on a per-unit basis. On the other hand, the edge recognition device 404 stores an image.
The image in the series of cuts from the unit 403 is the scanning unit of the image.
(1 frame) or a group of consecutive frames
And the image information and the point of interest determiner 419
From the supplied gaze point coordinate information, the gaze target is
And the border (edge) between the background and the foreground
You. Here, the edge recognition device 404 has a system control.
The user's instruction is given through the
Edges may be recognized. The image information is stored in the frame where the point of interest exists.
Is added with information on the outline of the object including the point of regard.
It is supplied to the image part extractor 405. Image part extractor 40
In the case of 5, the object including the point of interest (object of interest), the presence of the point of interest
Extracted as partial images for all frames
It is supplied to the double inspection device 406. Here the attention object is in the cut
Alternatively, there may be more than one in the same frame. In the duplication checker 406, different frames are used.
Different postures and different sizes due to zoom
Objects that are the same
Is inspected, along with the supplied image of the target object
It is supplied to the image selector 407 as overlapping information. Heavy here
The double inspection device 406 performs not only the cut under inspection but also the cut.
The presence or absence of overlap may be checked. The image selector 407 overlaps the target object image.
Attention object such as appearance frequency, gaze time, size in addition to information
Based on the side information obtained from the image,
To select an object that is suitable for indexing
In the case of duplicate images of the same object,
Choose the one that is in the state. The index selected here is cut
The ratio of the gaze time to the object to the duration is constant
Using a method such as making all candidates that exceed
There may be more than one in the cut. In the index, the same object
May exist in different states. Further selection here
The work is performed by the user's finger through the system controller 413.
And select the image to be indexed according to the instruction.
No. The object determined as the index as described above
Is the extracted partial image itself or an index
Information indicating the frame of the image and the image area to be used as an index
Is output as information indicating
Information of other frames to be created is also created based on the duplicated information.
Is output. Index image output from image selector 407
Alternatively, the index area existence area information and related frame information
The signal is converted into a signal suitable for recording through the encoding device 408,
The output of the recording signal generator 409 is used to store data in the storage medium 410.
Be recorded. Here, the storage medium used for recording the index is:
Storage medium 41 on which original image and device control information are recorded
A different value from 0 may be used. As the instruction of the photographer, the object to be indexed
To inform the device when shooting while paying attention to
A force signal is prepared and the input terminal 412 passes through the above path.
May be included in the device control information 420. In the above description, the video signal, the index picture signal and the related
The frame information and device control information are transmitted after the encoding device 421.
The storage medium 41 is provided by different recording signal generators.
0 may be stored in different areas. Also, a memory signal generator
Recording using different storage and storage media.
You may. FIG. 5 is a diagram showing a digital image using another optical disk.
Is a diagram showing the concept of an image recording device such as a video device.
Observes the photographer's line of sight during shooting and automatically
While pointing the device, simultaneously record objects that can serve as indexes in the image
An example of recording is shown. In this embodiment, the photographer himself records
It differs from the previous embodiment in that the shooting direction of the device is not changed.
You. In addition, here, as an example, an index image is recorded.
It has a special optical system and records the image intended by the photographer
When the media to be indexed and the media to record index images are different
I do. The image recording apparatus main body 501 is, as described above,
Optical system for taking the photographer's intended
It has two optical systems for shooting. Photographer 50
The eye-gaze detector 502 is attached to the eye part 4 and
501 is wired or wirelessly connected. Body 501
Alternatively, the line-of-sight detectors 502 detect the mutual positional relationship
And the positional relationship depends on the main body 501.
Is understood. The photographer 504 holds the terminal 503 in his hand,
Thus, the apparatus main body 501 is controlled. Terminal 503
Is connected to the main body 501 by wire or wirelessly. 5
05 is a subject that the photographer 504 is going to photograph.
It is. Image 5 photographed by main body 501
06, 508, one image 506 is the photographer 50
One that performs zooming according to the instruction of 4 (user image)
It is. When the photographer 504 gazes at the subject 505, the book
The body 501 automatically turns in the direction of the subject 505,
Shooting at the magnification specified by the photographer 504 by the end 503
Do the shadows. The symbol 507 in the user image 506 is temporarily
The viewpoint of the shadow person 504 is displayed. The other
Image 508 is an indexed image that is separate from user image
Gaze at a predetermined magnification on the device side using a system
The image is shot around a point. FIG. 6 is a block diagram showing the configuration of this image recording apparatus.
FIG. 5 is a conceptual diagram of FIG.
Corresponds to the processing device. The video signals input from the two optical systems are
To the device as a user image signal 601 and an index image signal 606.
Supplied. The user image signal is sent to the A / D converter 602.
Therefore, after being converted into a digital signal, the encoding device 60
3, the recording signal generator 60
4 is recorded on the storage medium 605 via the output of the storage medium 605. here
Digital encoding, which is an expansion process on electrical signals before encoding
A zoom may be applied. On the other hand, the index supplied from the input terminal 606
The image signal is similarly digitized by the A / D converter 607.
Digital zoom signal and supplied to the digital zoom 608.
It is. Because the camera is looking at the photographer's point of interest,
The index image signal input as it is contains the object of interest.
Digital zoom 608 is pre-
The image signal subjected to the predetermined zoom is stored in the image storage unit 60.
9. At the same time as shooting, the eyeball movement is
A motion signal is input from the input terminal 624 to the line-of-sight
A signal indicating the positional relationship with the shadow device is supplied. These two
The signals are supplied to the A / D converter 621 and the A / D converter 62, respectively.
5 is converted into a digital signal. After this, the eyeball
The motion signal is detected by the position detector 622 to detect the line of sight in the visual field.
Converted to position. At the same time, the detector position signal is
The orientation of the body, the orientation of the detector and the book
Converted to information such as body and detector orientation and distance,
The gaze information is supplied to the fixation point determiner 623 together with the gaze information. At the gazing point determiner 623, the main body and the detector
From the positional relationship and the line of sight of the photographer's eyes, the photographer
The absolute position of the gazing point of the
System controller 615. System controller
In 615, the camera should be pointed at the obtained fixation point coordinates
Calculate the direction, the distance to focus the optical system, and
It is output to the output terminal 619 as a displacement signal in the direction of the axis. On the other hand, the photographer instructs the device from the control terminal.
The signal is input from the input terminal 618 to the system
Provided to the Troller 615. System controller 6
15 is a user image zoom focus signal according to the instruction
The zoom instruction is transmitted to the output terminal 614 as
Perform the zoom. Here, before the encoding device 603,
Optical zoom and digital zoom
This output is not available when there is no need for focusing.
Is also good. Also, based on information on the distance between the camera and the subject,
Zoom around the point of interest at a predetermined magnification
Output terminal for zoom control signal for taking pictures
617. The cut detector 616 is connected to the input terminal 618
Instructions given by the photographer to the system controller 615
Cut signal for start and end of recording from among
And stored in the image storage unit 609 as a unit of storage.
Supply. The image storage unit 609 stores the image signal in one frame.
Or the image selector 61 as a set of continuous frames.
Send to 0. The object of interest has moved in the image selector 610
From the images, select the image of the object of interest, such as the number of appearances and gaze time
Attention provided multiple times based on secondary information obtained
In addition to selecting objects suitable for indexing from among objects,
Among images of the same duplicated object, some are suitable for indexing.
Is selected. The index selected here is when cutting is continued
The ratio of the gaze time for the object to the interval exceeds a certain value
Using a method such as making everything a candidate
There may be more than one. Also, the same object is different in the index
There may be more than one in the state. The selection work here is system
The instruction of the user is given through the
The image to be indexed may be selected according to the indication. The index image determined in this way is
11 converts the signal into a signal suitable for recording and generates a recording signal.
The output of the raw device 612 is recorded on the storage medium 613.
You. Here, the storage medium is the same as the recording medium 605 for the video signal.
Using one, the area different from the video signal of the medium 605
May be stored. Also, the encoding device 611
The encoded index image signal is encoded by the encoding device 603.
Recording signal generation device 6
04 may be stored in the storage medium 605. Further, in this embodiment, the enlargement of the image around the fixation point is performed.
Only large was used as an index, but as in the first example,
Even if you recognize the edge of the object and extract it as a partial image
Good. At this time, the video and gaze recording device and the index creation device
The devices may be integrated or different devices. As another embodiment, a method for viewing a moving image
(Listening) Observing the line of sight of the person, an object that can be an index in the image
An example of recording is described. This embodiment is not for recording
It differs from the previous embodiment in that it observes the user's gaze during playback.
You. FIG. 4 is a block diagram showing the configuration of this embodiment.
Since they are similar, with reference to FIG.
The description will be given while showing the differences. In this embodiment, the video signal and the
Digital video device that records the reference signal on the same medium.
is there. The video signal is presented to the viewer (listener) as a playback screen.
At the same time, it is also supplied to the input terminal 401. Video
The signal is converted into a digital signal by the A / D converter 402.
The converted image is stored in the image storage unit 403 for a certain period of time.
Can be The image storage unit 403 selects from among the provided images
Detect cuts and use them as the unit of accumulation. This implementation
In the example, an intermediate point between the A / D converter 402 and the image storage unit 403
And a channel connecting the encoding device 411 and the encoding device 4
11 is not required. On the other hand, the eye movements of the viewer (listener) are the eye movements.
Input terminal including information on the positional relationship between the detector and the playback screen
415 to the A / D conversion device 416,
Is converted to a video signal. The converted digital signal is
Calculated as the position of the line of sight in the screen by the detector 417
You. The result of the calculation is supplied to the type determiner 418,
The type of motion is analyzed. The analysis result is the fixation point determiner 4
19, and the edge recognition device
Attention information is given to the location 404. The edge recognition device 404 has the image storage unit
The image in a series of cuts from 403 is one frame, or
The image is supplied as a set of consecutive frames
Information and gazing point coordinate information supplied from the gazing point determiner 419
From the information, the edge of the object being watched is determined. This
Here, the edge recognition device 404 has a system controller 4
13 to give the user's instruction,
May be recognized. The image information is stored in the frame where the point of interest exists.
Is the image section with information on the contour of the object including the point of regard.
It is supplied to the fraction extractor 405. In the image part extractor 405
Indicates the object containing the point of interest (the object of interest)
All frames are extracted as partial images and duplicated
To the detector 406. Here, the object of interest is
A plurality may exist in one frame. In the duplicate checker 406, different frames are used.
Different postures and different sizes due to zoom
Objects that are the same
Is inspected, along with the supplied image of the target object
It is supplied to the image selector 407 as overlapping information. Heavy here
The double inspection device 406 performs not only the cut under inspection but also the cut.
The presence or absence of overlap may be checked. The image selector 407 overlaps the object image of interest.
Attention object such as appearance frequency, gaze time, size in addition to information
Based on the side information obtained from the image,
To select an object that is suitable for indexing
In the case of duplicate images of the same object,
Choose the one that is in the state. The index selected here is cut
The ratio of the gaze time to the object to the duration is constant
Using a method such as making all candidates that exceed
There may be more than one in the cut. In the index, the same object
May exist in different states. Further selection here
The work is performed by the user's finger through the system controller 413.
And select the image to be indexed according to the instruction.
No. The object determined as the index as described above
Is the extracted partial image or indexed image
Information representing a frame and information representing an image area to be an index
Report that includes the object.
Information is also created and output based on the duplicated information.
You. Index image output from image selector 407
Alternatively, the index area existence area information and related frame information
The signal is converted into a signal suitable for recording through the encoding device 408,
The output of the recording signal generator 409 causes the recording medium 410
Be recorded. Here, the storage medium used for recording the index is:
Storage medium 41 on which original image and device control information are recorded
A different value from 0 may be used. As an instruction of the viewer (listener), an index in the screen
Notify the device when you are looking at the object you want
Prepare an input signal and follow the above path from the input terminal 412.
May be included in the passed device control information 420. In the above, the index picture signal and the index related information,
The device control information is different after the coding device 421.
Depending on the recording signal generator, different areas of the storage medium 410 may be used.
It may be stored in the area. Storage signal generator, storage medium
Both may be recorded using different ones. In this embodiment, the detection of the line of sight and the creation of the index are the same.
Gaze detection device and index creation device are different.
It may be. Also, prepare multiple eye movement detectors
And simultaneously analyze the viewpoints of multiple people,
07 may be used as a criterion. In addition, the same image
When replayed, the previous index and new viewpoint information
Information and input it to the image selector 407 as a criterion.
Then, the index may be updated. In this embodiment, the image reproduction and retrieval apparatus is
The same one as in the embodiment of FIG. 3 is used. Here, the line-of-sight
The output device / index generation device and the image reproduction / search device are integrated
May be. In the above embodiment, the recording method is digital.
Although the explanation was made assuming that the
May be used for video. As a recording medium, an optical disc
Disk, magnetic disk, magnetic tape, memory device, etc.
May be used. I explained about video,
The present invention may be applied to an apparatus for recording still images. Also a viewpoint
Explanation of the example of calculating the position of the line of sight in the image in the detection device
Was performed, but only the direction of the viewpoint was recorded, and the index creation device
The position of the line of sight in the image may be calculated. Further implementation
In the example, the target object is extracted using the edge of the target object.
However, at the time of shooting, the camera
Measure and record the distance from
The target object may be extracted as a set of similar distances.
In addition, with respect to the embodiment of FIG.
Although an optical system different from that used for imaging was used, this was
You may take pictures using academic systems. In this case one input
Divide the image into two and perform the same processing as in the previous embodiment
May be divided into user images and index images
Shooting may be performed alternately. According to the above image information access device,
Observe human eye movements when presenting an image to be attracted
Measurement, determine the gazing point from the movement state, and center on the gazing point
Image elements that are characteristic of the image part
Index without recognizing every object in the image
Can be created. Also, the index created in that way
Are the objects that the photographer is interested in.
A good index can be created. Further, the area to be handled in image recognition is reduced.
Simplifies the index creation process and increases the processing speed
Can be expected. It also saves recording media because the recording volume is reduced.
Becomes possible. Next, referring to FIG. 7 to FIG.
A remote control device according to the embodiment will be described. FIG. 7 is a functional block diagram of the remote control device.
FIG. First, the area where the user's hand is located is represented by a distance image.
Input by the input means 701 and distance by the change amount detecting means 702
Find the time difference between images. The moving object detection means 703
From the distance image and its time difference, what size object
Calculate in which direction has moved. This series of processing
Rather, when the user extends his hand forward,
Can be detected. Each of the above means will be described in more detail.
This will be described in detail. Steps S1 to S4 in FIG.
3 shows the flow of processing. First, a motion area is extracted from a distance image.
(S1). Various types of distance image input means 701
Has been proposed. The distance can be measured without contact
Stereo method using the principle of triangulation, lens focus
Surface detection method, the phase difference between the projection of modulated light and reflected light
Or time difference measurement. Change amount detecting means 70
In 2, calculate the difference between two consecutive frames on the time axis
Thus, the background image that does not change is removed. Moving object
The detecting unit 703 detects the distance image and the amount of change in the distance image.
Detect moving objects. Next, a pixel having a short distance in the moving area is separated.
Then, it is determined that the objects are the same (S2). Soshi
The distance and size of the object
Is determined (S3). Detecting when reaching
If the purpose is to do something similar to the size of the hand
It just needs to be able to detect the body. I happen to have something in my hand
It may be possible to operate the product while holding it,
There is no need to judge the condition. Rather, an object similar to a hand
What is this object so that you do not mistakenly judge it as a hand?
When you make such a movement,
You have to decide appropriately. Here, I reached my hand forward
To be detected when. Therefore, as good as a hand
Object of size is within a certain distance toward the detector
It is detected that the vehicle has advanced (S4). If you want to reduce noise,
Judgment as a hand when stationary for a certain time after an object approaches
You can make it happen. Also, the sky finer than the thickness of the finger
If there is an inter-resolution, there is an object of the same size as the hand,
In addition, some of them are protruding locally in the direction of the detector.
There is also a method of determining that the hand is held when the hand is pressed. in this case
With the finger pointing toward the detector and reaching when reaching
Will be issued. Also, point a rod-shaped object such as a pen at the detector.
The same effect can be obtained by reaching
Is a natural form of work. FIG. 8 shows another embodiment of the remote controller according to the present invention.
FIG. 3 is a functional block diagram showing Distance image input means 801, change amount detecting means
802 and the moving object detecting means 803 are the same as those in FIG.
The processing after the moving object detection means 803 detects the hand.
Is done in the lower part. FIG. 9 shows the processing of this device.
Is shown. When the moving object tracking means 803 is as large as the hand,
When it is detected that the object of the size is approaching (S
4) The subsequent processing is left to the moving object tracking means 804
You. When the processing moves to the moving object detection means 803, the cursor
The display means 805 displays the tracking result of the moving object tracking means 804.
Therefore, the cursor is displayed at the position indicated by the hand (S
5). The motion vector detected by the moving object tracking unit 804
Are sequentially stored in the motion vector storage unit 806.
(S6). The movement command determination unit 807
Detect when a given special movement is made
(S7). If a special movement is detected, the command
Execution unit 809 executes a command registered in advance
(S8). What kind of movement
Whether to execute a command is to register a motion command in advance.
Registered in the section 808. Hand movement registered here
And the corresponding command are, for example,
When the camera is stationary for a while, select the menu at that point.
Move for a short distance after standing still for
When you come back again (that is, press the button
Action), the menu at that point
Select a menu, hold an open hand at a location, and
Then drag the mouse by opening it again
(Open and closed means changes in the size of the hand and the detected object
Capture), and so on. Next, the present invention will be described with reference to FIGS.
An information recording apparatus according to the embodiment will be described. FIG. 10 shows information reproduction according to the embodiment of the present invention.
It is a functional block diagram of a device. In this embodiment, one moving image, still image, sound
Information files such as voice, text, etc.
Expressed as icons, the links between the information files are
Connect the icon corresponding to each information file with a line
Expressed by An input interface 901 is provided with a remote control button.
Button, handwriting input device, keyboard or mouse,
It is a pointing device such as a rack ball. Affection
The command to display the report file and link structure on the map
This is performed by the user through the input interface 901. information
The file link recording device 902 includes a hard disk,
It is a large-capacity recording medium such as an optical disk. Here are the videos
Image and still image data, as well as audio and text data
Data can also be recorded, and
Link information is also recorded. The display device 903 displays
Display images, messages, etc. to the user. Combination
Optimization means 904 is for arranging icons of information files.
From among, set the given parameter to the maximum or minimum
Is calculated and output. The simplest set
The matching optimization means 904 performs a full search.
Calculate the value of the parameter for the combination of
Output the combination that maximizes or minimizes. control
The device 905 controls the entire system. [0115] The user can select between the information file and the file.
If you want to know the structure of the link attached to
A command for displaying a map is input to the interface 901. Ma
When a command for displaying a loop is input, the control device 905 causes the information file to be displayed.
File and information from the file / link recording device 902.
Read the link information between the information files. Information
When the file link recording device 902 is a hard disk or an optical
Disk or floppy disk.
File name, what icon to display, etc.
Once the information about the report file and link information has been
The map display command.
Information about the information file from the buffer
May be read. Read information file
Information and link information relating to the combination optimization means 9
04 on the display device 903.
The display position of the link is determined. By determining the display position, for example,
Then, the information faal link displayed as shown in Fig. 11
Map is displayed more easily as shown in FIG.
It is. However, (1) and (2) indicate the icon of the video file.
The line connecting them is placed between the information files
Represents a link. The display position is determined, for example, as follows.
You. First, the display screen of the display device 903 is divided into several sections.
Divide into images. This section contains an information file representing the information file.
A file icon is displayed. At most one per block
Two information file icons are displayed. Link
Files that have the same information file icon
Are displayed as if they were connected by a line representing a link. Affection
The combination of information file icon display positions depends on the information file
If the number of file icons is N and the number of sections is M, M
! / (M-N)! There is a street. From this combination,
Combination optimization means 904 selects one combination
However, it is the legibility specified by the combination
Is determined by the value of the parameter representing Sand
A line representing a link as a representative parameter of readability
If the number of intersections of is adopted, minimize this value
Choose a combination, a combination that minimizes parameters
If there are a plurality of, any one may be selected. This look
As a representative parameter of SUSU, the intersecting link line
In addition to the number, the variance of the link line length, the link line length
Summation can also be used. Using one of these
You can choose the right combination, and some
Create a new parameter by combining parameters
And use it to select the best combination. This
In addition to the information file icon,
It is also possible to consider whether or not the display of the buttons is not offset. The combination optimizing means 904 uses these parameters.
Choose the combination that optimizes the meter,
The simplest method is to pass on all combinations.
Calculate parameters and select the best one.
You. Generally, this method requires a large calculation when N and M are large.
Cost. So don't calculate for everything
Parameters for some combinations and calculate
Is to choose the best one and approximate it to the actual optimal solution.
There is also a method. In general, the total for all combinations
If you don't calculate the best combination,
It is good if some optimization is done even if it is not optimal
In this case, an approximation method of the optimal solution is often used. Real truth
In the embodiment, such approximation may be used. An information file is obtained by the combination optimizing means 904.
Once the display position of the file icon / link is determined,
Accordingly, a map is displayed on the display device 903. This and
Interface for zooming and panning the map
Can also be performed by inputting an instruction via the
is there. As described above, the image information storage device of the present invention has the following advantages.
According to the access device, the user can
Detect gaze, record it with image information and use
And selective extraction of objects that are characteristic for the elderly
So that the burden on the user is small and the processing is simple.
Efficient and efficient indexing, processing time and storage media
Costs can be reduced. [0122]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram illustrating a configuration of an image recording apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of an index creation device according to an embodiment of the present invention. FIG. 3 is a block diagram showing a configuration of an image reproduction / search apparatus according to an embodiment of the present invention. FIG. 4 is a block diagram showing a configuration of an image recording apparatus and an index apparatus according to the embodiment of the present invention. FIG. 5 is a conceptual diagram showing a configuration of another image recording apparatus according to the embodiment of the present invention. FIG. 6 is a block diagram illustrating a configuration of the image recording apparatus of FIG. 5; FIG. 7 is a functional block diagram of the remote control device according to the embodiment of the present invention. FIG. 8 is a functional block diagram of another remote control device according to the embodiment of the present invention. FIG. 9 is a flowchart showing processing of the apparatus. FIG. 10 is a functional block diagram of the information reproducing apparatus according to the embodiment of the present invention. FIG. 11 is a diagram showing a map before optimization in the same device. FIG. 12 is a diagram showing a map after optimization in the same device. [Description of Signs] 101: video signal, 104: recording signal generator, 105
... storage medium, 106 ... eye movement signal, 108 ... eye position detector, 110 ... operation signal, 202 ... storage signal reading device, 206 ... edge recognition device, 207 ... image partial extractor, 208 ... duplication checker, 209 ... Image selector, 219
... Type judging device, 220 ... Point of interest determining device, 303 ... Video signal, 305 ... Screen creator, 307 ... Monitor output terminal,
308 ... related frame information, 317 ... list creation device,
Reference numeral 401: video signal, 412: operation signal, 415: eye movement signal, 501: image recording apparatus main body, 502: gaze detector, 504: photographer, 505: subject, 508: index image, 601: user image signal, 606 … Index image signal,
608: Digital zoom, 614: User image zoom focus signal, 616: Cut detector, 617: Zoom control signal, 618: Operation signal, 619: Camera direction displacement signal, 620: Eye movement signal, 624: Detector position signal, 701, 801... Distance image input means, 70
2, 802: change amount detecting means, 703, 803: moving object detecting means, 804: moving object tracking means, 807: motion command determining section, 901: input interface, 902
... information faal link recording device, 903 ... display device,
904: combination optimization means; 905: control device.

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-5-282380 (JP, A) JP-A-5-233726 (JP, A) JP-A-3-292571 (JP, A) 224875 (JP, A) JP-A-61-21523 (JP, A) JP-A-5-108795 (JP, A) JP-A-1-289388 (JP, A) (58) Fields investigated (Int. Cl. ^7, DB name) G06F 17/30 G06F 3/00 G06F 12/00 G06T 7/20 JICST file (JOIS)

Claims (1)

(57) and the image information recording means for recording the Patent Claims: 1. A picture information, recording of the image information for the image information recording means or
A visual axis detecting means for detecting the position of the line of sight with respect to the image information of the user during playback, the line-of-sight position information storage means for storing the detected position of the line of sight in the sight line detecting means in association with said image information, said A part in the image information recorded in the image information recording unit based on the position of the line of sight stored in the line of sight information storage unit
Extraction means for extracting an image is represented by the partial image extracted by said extraction means,
Objects that appear repeatedly in the same cut unit,
Inspection means for inspecting the same object, said inspection
Image information access apparatus characterized by comprising a index information generating means for generating index information for retrieval of the image information based on the detection result by the means.