KR20040007615A - Information storing apparatus and method therefor - Google Patents

Abstract

In storing main information such as moving image data and metadata accompanying the main information, a priority table in which the priority of each description format is preset for each item is maintained for metadata items having a plurality of description formats. do. Under the constraints regarding the storage of metadata due to the data collection time of metadata or the calculation time of values corresponding to the description format, the metadata items and description formats are adopted in the order of high priority maintained in the priority table. (S122) Metadata is formed and stored in association with the main information (S126). With such a configuration, it is possible to reliably make use of incidental information in a minimum available format with a wide range of devices and control methods, and also use incidental information that can make use of the characteristics of individual devices and control methods.

Description

Information storage device and its method {INFORMATION STORING APPARATUS AND METHOD THEREFOR}

Conventionally, Japanese Patent Application Laid-Open No. 06-318373, 07-303239, etc. propose a technique for storing accompanying data indicating the state of the device and the like at the same time as photographing data, and a method of encoding and encrypting the accompanying information. It is becoming. In these proposals, in order to assign and store the unit information, the unit information is integrated in units such as types, and an information structure for assigning and storing the unit information in the unit is created.

The case of storing incident information on various devices or methods is contemplated. Even in the case of the accompanying information of the same item, it is common that various representation formats and encoding / encryption methods actually exist for each device or control method as appropriate. However, the above-described storage method of incidental information can only correspond to one representation format or an encoding / encoding method. In this manner, in a method in which only one format is allowed, the associated information only needs to be associated with the single structure or expression format. However, there are cases in which the features of each device or control method are lost.

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information storage device and method for storing main information and associated information corresponding thereto, and in particular, suitable for storing a moving image as main information and information indicating a state of the image pickup apparatus as auxiliary information in a recording medium. An information storage device and method are disclosed.

The accompanying drawings constitute a part in conjunction with the present specification, together with the description show embodiments of the invention, and illustrate the principles of the invention.

1 is a block diagram showing the configuration of an imaging device in this embodiment.

2 is a block diagram showing the configuration of the system control unit 21.

Fig. 3 is a flowchart showing the outline of control by the system control unit 21 when recording moving pictures and metadata.

4 is a diagram illustrating a metadata storage method.

5 illustrates a data structure of metadata.

6 is a table showing priority setting of metadata having a plurality of description formats.

Fig. 7 is a table showing an example of group separation of metadata items in this embodiment.

Fig. 8 is a table showing an example of group separation of metadata items in this embodiment.

FIG. 9 is a flowchart for explaining an outline of metadata storage processing executed in step S37 of FIG.

FIG. 10 is a flowchart for explaining the metadata acquisition process in step S90 of FIG.

FIG. 11 is a flowchart for describing a conversion process of a description format of metadata in step S91 of FIG. 9;

FIG. 12 is a flowchart for explaining the storing process of metadata in step S92 of FIG.

13 is a table explaining correspondence of metadata formats in a plurality of imaging devices.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and it is possible to reliably make use of incidental information in a form that can be at least available in a wide range of devices and control methods, and to utilize the characteristics of individual devices and control methods. It also aims to make it available.

According to the present invention, the above object is achieved by providing a device for storing main information and accompanying information items accompanying the same, wherein the device includes the plurality of descriptions for each of the accompanying information items including a plurality of description formats. Determination means for predetermining a priority in each format, selection means for selecting a technique format to be used in recording according to the priority among the technique formats available in the apparatus, and accompanying information items in the selected technique format. And recording means for recording in correspondence with the information.

Further, according to another aspect of the present invention, the above object is achieved by providing a method of storing main information and accompanying information items accompanying the same, wherein the method includes a plurality of description formats in the accompanying information items. In this case, a decision process of predetermining a priority for each of the plurality of technology formats for each of the accompanying information items and a technology format to be used for recording in accordance with the priority are selected from the technology formats available in the apparatus. And a recording step of recording the incident information item corresponding to the main information in the selected technical format.

Other features and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the present embodiment described below, an example of a moving image pickup apparatus that simultaneously stores information of various sensors and user operation information as incidental information as in a moving image will be described.

In the present embodiment, when the moving picture image pickup device encodes / encodes accompanying information such as the state of the device at the time of shooting and simultaneously stores the image information, the apparatus stores and stores the accompanying information by a plurality of expression formats or encoding / encoding methods. Allow to save. Here, when setting priorities for a plurality of representation formats or encoding / encryption schemes, it is guaranteed to store the side information using the minimum necessary representation format or encoding / encryption scheme. Incidental information may be stored together in an expression format that can be characterized by each device or control method, or by an encoding / encryption method. In addition, when storing and storing the accompanying information, the storage location or the like can be changed in correspondence with the supporting information, and the accompanying information can be stored in a storage location advantageous for each use mode or control method.

1 is a block diagram showing the configuration of an image pickup apparatus according to the present embodiment. In Fig. 1, the lens unit 11 includes a focus lens for adjusting the subject distance and a zoom lens for adjusting the focal length, and forms an image of the subject on the imaging device. The aperture 12 adjusts the amount of light. The imaging element 13 is formed from the CCD to convert the input light into an electric signal. The sample and hold / AGC circuit 14 performs sample and hold and gain adjustment.

The A / D conversion unit 15 performs analog / digital conversion (A / D conversion) on the analog signal in the sample and hold / AGC circuit 14. In addition, the image signal processor 16 processes the signal to generate an image signal. The noise reduction unit (NR) 18 includes a field memory and removes noise of a video signal. The electronic zoom unit 17 has a field memory and performs image extraction / enlargement and interpolation processing.

The lens position detector 19 detects the position of the lens. The lens driver 20 drives the lens. The system control unit 21 controls the entire imaging device. 2 is a block diagram showing the configuration of the system control unit 21. The system control unit 21 has a CPU 31, a ROM 32, a RAM 33, an I / O interface 34, and a bus 35. In the ROM 32, a control program for realizing the processing described by the flowchart described later by the CPU 31, and various table values (a priority table described later in FIG. 6 or a group described later in FIGS. 7 and 8). Table) and the like.

Returning to FIG. 1, the user operation part 22 is equipped with the user interface for performing various control of the imaging device including the initial setting of AEMode (automatic exposure mode), and a zoom operation, various settings, and various operations. The other sensor 23 includes a sensor necessary for determining the state of the imaging device, and a sensor for detecting an environmental state in which the imaging device such as a GPS and a thermometer is placed.

The other device 24 is an accessory of an imaging device such as an electronic flashlight or an interchangeable lens, an external computer (PC) connected via a communication means 25 such as the IEEE 1394 or USB, another moving picture imaging device, Devices on a network, and the like.

The MPEG-CODEC 26 encodes video data in the format of MPEG. The recording unit 27 is constituted by a drive or the like for recording information on the recording medium 28. The recording medium 28 is formed of, for example, an optical disk, a magnetic disk, a magneto-optical disk, a magnetic tape, a hard disk, a nonvolatile semiconductor memory (flash memory or the like), or the like.

The operation of the imaging device of this embodiment having the above configuration will be described. The light amount from the subject received from the lens unit 11 is adjusted by the aperture 12 to form an image on the surface of the imaging element 13. This light is converted into an electrical signal by the image pickup device 13, A / D converted by the A / D converter 15 through the sample-and-hold / AGC unit 14 to be a digital signal, and the image signal processor It is input to (16). In the video signal processing unit 16, processing such as aperture correction, gamma correction, and white balance correction is performed for each of the luminance and each color component of the input signal to generate a video signal. This video signal is output to the noise reduction unit 18.

The noise reduction unit 18 is controlled by a control signal from the system control unit 21 and removes noise of the received video signal. The video signal from which the noise is removed is output to the electronic zoom unit 17. The electronic zoom unit 17 extracts an image from the image developed in the field memory so that the output image becomes a magnification based on a control signal from the system control unit 21 with respect to the input image, and performs magnification and interpolation processing to perform frame image data. As a result, the video is output to the MPEG-CODEC 26. The MPEG-CODEC 26 encodes the received frame image data. At this time, the MPEG-CODEC 26 generates data so that metadata (described later) input from the system control unit 21 is stored together with the frame image data, and outputs the data to the recording unit 27. In such an imaging device, the application order of the noise reduction unit 18 and the electron zoom unit 17 may be reversed. That is, the electronic zoom unit 17 processes the processing result of the video signal processing unit 16, processes the processing result of the electronic zoom unit 17 in the noise reduction unit 18, and processes the noise reduction unit 18. The result may be processed in the MPEG-CODEC 26. This method and this embodiment are both general methods. Other configurations are also included in the present invention regardless of the procedure of the processing.

The storage method of the metadata will be described later. The metadata is related to aperture correction, gamma correction, and white balance correction, which are used when the system control unit 21 controls the video signal processing unit 16, the NR unit 18, the electron zoom unit 17, and the lens driver 20. Information, information obtained from sensors such as the lens position detecting unit 19 and other sensors 23, information about initial setting by the user obtained from the user operating unit 22 (such as AEMode), and information obtained from the user operating unit 22. And various auxiliary information such as information relating to the user's operation and information obtained from the other device 24 through the communication means 25.

<1. Metadata Storage Processing>

3 is a flowchart showing the outline of the control by the system control unit 21 at the time of recording the moving picture and the metadata. First, in step S31, it is determined whether or not the user sets the imaging device via the user operation unit 22. The setting of the imaging device includes, for example, setting of AEMode. If YES in step S31, the setting information of the imaging device is changed in step S32 and the setting information is stored in the RAM 33 or the like. The changed setting information of the imaging device is transmitted to the MPEG-CODEC 26 as metadata in step S37 (to be described later). In step S33, it is checked whether the recording button is in the ON state. If NO in step S33, the standby state is set, and the flow returns to step S31. If YES in step S33, the flow advances to step S34 to set the imaging device to the recording state and start imaging. The recording start time at this time is transmitted to the MPEG-C0DEC 26 as metadata in step S37 (to be described later). In addition, a change in the imaging device under imaging, for example, a change in focus during autofocus, a change in the environment in which the imaging device is placed, for example, GPS information, is collected by the system control unit 21 in step S37 (to be described later). And are sequentially transmitted to the MPEG-CODEC 26.

In step S35, it is checked whether the operation of the imaging device has been performed by the user. Operation of the imaging device includes, for example, ON / OFF of a zoom button, panning, and the like. If YES in step S35, the flow advances to step S36, and the system controller 21 drives the lens driver 20 or the like based on the information of the lens position detector 19 and the other sensors 23 to control the imaging device. . For example, when the zoom is instructed, the zoom lens of the lens 11 is moved by the lens driving unit 20 only during the period in which the zoom button is pressed to zoom. The sensor information such as the ON / OFF state and the focal length of the zoom button at this time is collected as metadata in the next step S37 and transmitted to the MPEG-CODEC 26.

In step S37, as will be described later with reference to FIG. 9, information is collected for metadata generation, metadata is generated, and metadata is transmitted to the MPEG-CODEC 26. In the MPEG-CODEC 26, the video data transmitted from the electronic zoom unit 17 and the metadata transmitted from the system control unit 21 are combined by a method (described later) (for example, a method described later in FIG. 4). And output to the recording unit 27. The recording unit 27 records the data transmitted from the MPEG-CODEC 26 to the storage medium 28. In step S38, it is checked whether the recording button is OFF. If NO in step S38, the photographing state is maintained. Therefore, the flow returns to step S34 to repeat the above-described operation. If YES in step S38, the flow returns to step S31 to be in a standby state.

4 is a diagram illustrating a metadata storage method. In Fig. 4, reference numeral 401 denotes a frame sequence constituting a moving image. Reference numeral 402 denotes a data structure of one frame. One frame 402 includes a management information area 403 in which the size of image data, information on encoding, a time code of a frame, and the like are stored, and an image data area 404 in which the image data itself is contained. An oblique portion of the management information area 403 represents an unused portion. In this embodiment, the metadata storage area 405 is formed in this unused portion. Reference numeral 501 also denotes metadata associated with the frame 402. The metadata 501 is stored in the metadata storage area 405. More specifically, for example, the MPEG-2 stream may be stored in a user data area of a picture header or a user data area of a GOP (group of pictures) header.

5 is a diagram illustrating an example of a data structure of metadata. The metadata 501 includes a plurality of types of data that differ in ease of acquisition, frequency of use, etc. by metadata items. Therefore, the more easily the acquisition and the higher the frequency of use, the more easily the data is stored and read. On the contrary, it is only necessary to store only a relatively rare amount of information that can be earned or a relatively rare amount of information. The data structure shown in Fig. 5 satisfies this condition.

As shown in FIG. 5, the metadata 501 includes metadata management information 502, a storage area 503 of metadata belonging to group 1, and a storage area 504 of metadata belonging to group 2. . The metadata management information 502 consists of the total size of the metadata, offset into group two. The storage area 503 of the metadata belonging to the group 1 enumerates and records all items of the metadata classified into the group 1 by a method described later in a predetermined order. The storage area 504 of the metadata belonging to the group 2 includes management information storing metadata items classified into the group 2 by the method described later, such as the size and type of the metadata, metadata and management information. Enumerate and store as many sets as necessary. In some cases, for some reason on the device side, the predetermined metadata cannot be obtained. In that case, for example, measures are taken such as storing information indicating no value. However, which treatment is appropriate may often vary depending on the nature of the metadata. This point does not specifically define this.

Since the storage area 503 of the metadata belonging to the group 1 has a fixed data size and order, it is possible to collectively write and read data on the storage medium. On the other hand, in the storage area 504 of the metadata belonging to the group 2, it is necessary to create and write management information for each metadata item, and to read and analyze management information for each metadata item. Therefore, compared with the storage area 504 of the metadata belonging to the group 2, the storage area 503 of the metadata belonging to the group 1 has a structure in which data can be easily written and read.

Next, the description format of the metadata will be described. There may be a plurality of description formats for a single information item. In Tables 1 to 3 below, as an example of metadata having a plurality of description formats, description formats of metadata of "focus", "zoom", and "fan" are shown.

There are different types of values, etc., depending on the difference between the sensors. For example, in Table 1, " focus-reactive " is mainly a technical format used when the lens position detection unit 19 detects the relative position of the lens unit 11 with respect to the moving range of the focus lens. "Focus-Absolute" is a technical form mainly used when a sensor that directly measures a subject distance is mounted.

In some cases, the value can be converted to a separate format. For example, in Table 2, "zoom-relative" is a technical format mainly used when the lens position detection unit 19 detects the relative position of the lens unit 11 with respect to the movable range of the zoom lens, -Pushroot "is a technical format mainly used when the lens position detection part 19 detects a focal length. "Zoom-normalize" is mainly a format used by the lens position detection unit 19 to describe a value obtained by detecting a focal length and converting the value into a camera of a 35 mm film.

In addition, the accuracy of the value which can be acquired may differ according to the performance of a sensor, the timing which acquires a value, etc. may differ. For example, in Table 3, "Pan-1" is a form in which only the direction of a fan can be acquired. "Pan-2" is a form in which the speed of a fan can be acquired approximately. "Pan-3" is a form in which the speed of a fan can be acquired correctly.

Next, priority is determined for each single metadata. 6 is a table showing priority setting of metadata having a plurality of description formats. For example, priority 1 indicates a higher priority than priority 2. The priority is set for each data item (for example, "focus", "zoom", "pan"). Various criteria can be considered in determining the priority. For example, a format closer to the widely used sensor output format has a higher priority, and a format that needs to convert values obtained by the sensor has a lower priority, or a format having a higher precision has a lower priority. Therefore, the priority may be determined according to the ease of information acquisition. In addition, there is a method of increasing the priority of the format with a high frequency of use.

Then, to determine the storage area of the metadata, grouping of metadata items is performed. There are various methods of group separation. As a first method, there is a method of classifying a single metadata item according to the ease of information acquisition or the frequency of use. The metadata items that are easy to acquire or have a high frequency of use are referred to as the first group. The second item is a metadata item which is relatively difficult to acquire or has a relatively low use frequency. In the example of FIG. 7, "Pan" includes "Direction" and "Speed" as a set of information. "Pan-1" and "Pan-2" are group 1 because they are frequently used. The exact "Speed" information, "Pan-3", is not always needed and is therefore Group 2.

Alternatively, the group separation of metadata items may be determined based on the priority. 8 shows an example. In other words, the metadata item having priority 1 is referred to as group 1, and the other metadata items are referred to as group 2.

The programs created according to the rules of the tables shown in FIGS. 6 to 8 are stored in the ROM 32 or the RAM 33 of the system control unit 21. Alternatively, these tables are stored in the ROM 32 of the system control unit 21, and the system control unit 21 performs processing with reference to this.

Next, the metadata storage process will be described.

First, with reference to FIG. 9, the outline | summary of the metadata storage process performed by step S37 is demonstrated. In step S90, the system control unit 21 queries various sensors, for example, to acquire metadata. In step S91, if there is metadata that can convert the format using the conversion formula, the format is converted. However, it is not always necessary to perform this process (conversion process). This process may be omitted. In step S92, the metadata is transferred to the MPEG-CODEC 26, and the metadata is stored in the storage medium 28 together with the image data. The details of the processing operations in each of steps S90 to S92 will be described below.

First, the details of the acquisition process of the metadata in step S90 will be described with reference to FIG. In step S100, 1 is set to N. In step S101, it is determined whether or not the system control unit 21 can continue to acquire the sensor value. As this determination condition, the following two constraint conditions are mentioned, for example. In other words,

(1) Is there time left to acquire?

(2) Is there enough capacity in memory to hold data temporarily?

to be.

The constraint condition of (1) is demonstrated. In order to store metadata in real time in each frame unit at the time of moving picture shooting, there is a limitation in the time from acquisition of the metadata to storage. For NTSC, the frame rate is about 30 frames / second, so the processing time should be less than about 1/30 second. The limitation in (2) is when there is a limit on the amount of memory that holds metadata at one time.

In the present embodiment, when the above constraints (1) and (2) are satisfied, another sensor value can be obtained, and the process proceeds to step S102. In step S102, the system control unit 21 inquires a sensor corresponding to the format of priority N to obtain information.

In step S103, it is determined whether other sensors corresponding to priority N exist. If YES in step S103, the processing of steps S101 and S102 is repeated. If NO in step S103, the flow advances to step S104, where N is incremented to obtain information from the sensor for the next priority. In step S105, it is determined whether or not a metadata item having a format of priority N exists. If YES in step S105, the processing from step S101 to step S104 is repeated.

In the case where it is determined in step S101 that any one of the constraints (1) and (2) is not satisfied, or in the case where it is determined in step S105 that a metadata item having a format of priority N does not exist, this processing Ends. Incidentally, only the metadata obtained from the sensor will be described here for the sake of simplicity. However, the same processing is performed even when metadata is obtained from the user operating unit 22 or the other device 24.

By the above processing, even when a constraint condition exists, information is preferentially obtained from the metadata item with high priority.

Next, details of the conversion processing of the description format of the metadata in step S91 will be described with reference to FIG. 11. In this process, the acquisition description format of the data (output value from the sensor) acquired by the metadata acquisition process is converted. First, in step S110, 1 is set to N. In step S111, it is determined whether or not conversion of the data format (descriptive format) can be continued. As the determination condition, for example, the following two constraints can be used as in the case of the metadata acquisition process of step S90 (step S101).

(1) Is there time left to convert?

(2) Is there enough capacity in the memory to hold data temporarily?

Of course, other conditions may be used.

If the above constraints (1) and (2) are satisfied, the process proceeds to the next step S112. In step S112, it is determined whether the metadata currently being noted is a metadata item that can be converted into a priority N format. In the judgment condition in this case,

(3) Does the system control unit 21 have the computing power necessary for the conversion?

(4) Is the value of the format on which the conversion is based?

For example.

The condition (3) corresponds to the case where the CPU 31 of the system control unit 21 has a limitation on the computing power. The condition (4) corresponds to the case where a value is not set in some cases because information cannot be detected in accordance with the timing at which the sensor inquires.

In the present embodiment, when the constraints of (3) and (4) are satisfied, it is determined that the conversion to the format of priority N is possible, and the flow proceeds to step S113. If the condition is not satisfied, step S113 is skipped and step S114 is reached. In step S113, the data is converted into a priority N format (descriptive format). In step S114, it is determined whether or not other metadata items that can be converted to the description format of priority N are obtained. If YES in step S114, the process of step S113 is repeated from step S111. If NO in step S114, the flow advances to step S115 to increment N.

In step S116, it is determined whether or not a metadata item having a format of priority N exists. If YES in step S116, the processing from step S111 to step S115 is repeated. If the constraints (1) and (2) are not satisfied in step S111, or if there are no metadata items having the format of priority N in step S116, this processing ends. For simplicity, only the metadata obtained from the sensor is described here. However, the same processing is also performed when metadata is obtained from the user operation unit 22 or another device 24.

By the above processing, even when a constraint exists, format conversion is preferentially performed from the metadata item with high priority.

The details of the storing process of the metadata in step S92 will be described with reference to FIG. In step S120, 1 is set to N. In step S121, it is determined whether or not the metadata item of the description format of priority N exists as a result of the metadata acquisition process of step S90 and the metadata conversion process of step S91. If YES in step S121, the flow advances to step S122, where it is determined to employ metadata items according to the description format.

In step S123, it is determined whether further metadata can be stored. In this judgment condition,

(5) Is there a time remaining for storing metadata with respect to the time required for recording previously obtained from the MPEG CODEC 26 or the like?

(6) Is there any amount of metadata that can be stored at one time with respect to the amount of recordable metadata previously obtained from the MPEG CODEC 26 or the like?

For example.

The constraint (5) corresponds to the case where a predetermined time is required for storing data in the storage medium. Further, the constraint (6) corresponds to the case where there is a limit on the amount of metadata that can be stored at one time. The metadata acquisition step (step S90) may be configured such that the value from the sensor is not acquired when it is determined that the amount of metadata exceeds the limit.

In the present embodiment, when the above described constraints (5) and (6) are satisfied, it is determined that another metadata can be stored, and the flow proceeds to step S124. In step S124, as a result of the metadata acquisition process of step S90 and the metadata conversion process of step S91, it is determined whether or not metadata items having a format of priority N to be employed in the metadata remain. If YES in step S124, step S123 is repeated from step S122. If NO in step S124, N is incremented in step S125 to repeat step S124 from step S121.

If NO in step S121 or if the constraints of (5) and (6) are not satisfied in step S123, the flow proceeds to step S126. In step S126, the data item adopted as the metadata in step S122 is transmitted from the system control unit 21 to the MPEG-CODEC 26. The MPEG-CODEC 26 loads the metadata received from the system control unit 21 into the management information area 403 according to the format of FIG. 5 according to the group separation set as shown in FIG. 7 or 8. The data is output to the recording unit 27. The recording unit 27 records the data received from the MPEG-C0DEC 26 on the storage medium 28.

By the above processing, even when a constraint condition exists, data is preferentially stored from a metadata item having a high priority.

<2. Metadata Utilization Processing>

Next, the utilization of the stored metadata as described above will be described. For simplicity of explanation, the case where there are three imaging devices A to C shown in FIG. 13 will be described. In the following description, " Pan " is taken as an example of the metadata item. It is assumed that the three imaging devices A to C comply with the rules of the table in FIG. As shown in FIG. 13, the imaging device A can store only the description format of "Pan-1" of priority 1 with respect to "Pan". Regarding "Pan", the imaging device B can store the description format up to priority 2, that is, the formats of "Pan-1" and "Pan-2". Regarding " Pan " The imaging device C can store a description format up to priority 3, that is, description formats " Pan-1 ", " Pan-2 ", " Pan-3 ".

In the apparatus which can use the data of each imaging apparatus, the following operation is performed in the apparatus which classifies and outputs frames according to "Pan" information. When classifying all the data of imaging apparatus A-C, it carries out by the description format of "Pan-1" of priority 1 (1301 of FIG. 13). By doing so, the classification result in the direction of "Pan" is output. Therefore, the user can always obtain the classification result in the form of priority 1 with respect to the data of all the imaging apparatuses. In this manner, data of a plurality of different imaging apparatuses can be uniformly handled by a single video editing apparatus or the like.

When classifying all the data of the imaging devices B and C, it classifies into the description format of "Pan-1" of priority 1 and "Pan-2" of priority 2 (1302 of FIG. 13). By doing so, the classification results in the direction of "Pan" and the approximate classification results by the speed of "Pan" are output.

When classifying data of only the imaging device C, all formats can be handled (1303 in Fig. 13). Therefore, it is classified into the format of "Pan-1" of priority 1 and the format of "Pan-3" of priority 3. Thereby, the classification result by the direction of "Pan" and the detailed classification result by the speed of "Pan" are output.

As described above, the user can always obtain the classification result in the format with the highest precision, corresponding to the capability of each imaging device.

In the above embodiment, all the metadata is via the system control unit 21. However, the present invention is not necessarily limited to this.

In the above embodiment, the metadata is received by the MPEG-CODEC 26 and loaded into the management information area of the image frame data. However, the present invention is not limited to this. For example, the recording unit 27 may receive the metadata, and may record the metadata together with the information indicating the relationship with the frame after recording the image data.

In the above embodiment, metadata is stored in each frame unit. However, this is only one example. For example, the metadata may be stored together with the information indicating the relationship with the frame in the video data and stored in the footer area of the moving image file.

In the above embodiment, metadata belonging to each group is stored in the format of FIG. 5. However, metadata is stored divided into places where access is relatively easy and places that are relatively difficult. For example, when metadata is stored in the footer area of the above-described moving image file and recorded on the storage medium, the metadata is collectively stored for each group in the footer area, and only metadata belonging to group 1 is stored on the disk. You can also save it to a housewife.

In the above embodiment, the group separation of the metadata is two. However, it may of course be divided into three or more groups depending on the ease of storage, the frequency of use, or the characteristics of the storage area.

As described above, according to the present embodiment, metadata items of a high priority technology format are preferentially stored irrespective of data captured by a moving image photographing apparatus, so that a search or classification in a high priority format can be assured. It is possible to do so.

In addition, when a search or classification apparatus is provided, it is possible to perform a search or classification with higher accuracy or better functionality by making use of a description format of metadata items having a lower priority but with higher accuracy. Even in the case described above, it goes without saying that a description format of metadata having a high priority can be used to perform a search or a classification.

Further, only by creating a search or classification device corresponding to a single data structure, it is possible to search or classify data stored in a low priority format using a format having a low priority.

An object of the present invention is to supply a storage medium storing program codes of software for realizing the functions of the above embodiments to a system or an apparatus, and the program code stored in the storage medium by a computer (or CPU or MPU) of the system or apparatus. Can be achieved by reading and executing.

In this case, the program code read out from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, nonvolatile memory card, ROM, or the like can be used.

The functions of the above-described embodiments are realized not only when the computer executes the read program code but also when the operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. It includes being.

The program code read out from the storage medium is written into the memory of the function expansion board inserted into the computer or the function expansion unit connected to the computer, and based on the instruction of the program code, the CPU of the function expansion board or the function expansion unit is This includes the realization of the functions of the above-described embodiments when performing some or all of the actual processing.

As described above, according to the present invention, it is possible to reliably make use of incidental information in a minimum available format with a wide range of apparatuses and control methods, and also use incidental information that can utilize the characteristics of individual apparatuses and control methods. You can do that.

Apparently, widely different embodiments of the present invention can be made in various ways without departing from the spirit and scope of the invention, and it is obvious that the invention is not limited to the specific embodiments thereof except as defined in the appended claims. Do.

Claims (56)

An apparatus for storing main information and accompanying information items accompanying the main information, Determination means for predetermine a priority for each of the plurality of description formats for each of the accompanying information items including a plurality of description formats; Selecting means for selecting a technical format to be used in recording according to the priority, from among the technical formats available in the apparatus; Recording means for recording incident information items corresponding to the state information in the selected technical format Information storage device comprising a. The method of claim 1, And the selecting means selects a plurality of description formats to be used within the available range in consideration of the constraints of the devices, in order from the highest priorities among the description formats available in the apparatus. The method of claim 1, The selecting means selects a plurality of techniques or expression formats to be used within the range available in consideration of the constraints of the apparatus, in order from the highest priorities among the technical formats available in the apparatus, and the selected technique. An information storage device that selects all of the technology formats that have a higher priority than the lowest priority among the formats. The method of claim 1, If there are technical formats that cannot be used simultaneously due to the limitations of the apparatus among the technical formats available in the apparatus, the selecting means selects one technical format having a high priority for the technical format and the remaining techniques. An information storage apparatus for selecting a plurality of available description formats to be used in order from the highest priority to the format. The method of claim 1, Among the technical formats available in the apparatus, if the number or combination of technical formats available is limited by the constraints of the apparatus, the selection means is a technical format with high priority as much as possible within the limitation. Select the information storage device. The method of claim 1, The selecting means selects the technical format to be used within the available range in consideration of the constraints of the apparatus, in order from the highest priority among the technical formats available in the apparatus, and correspondingly to the priority. An information storage device for varying a storage location or a storage method on a storage medium of an accompanying information item. The method of claim 6, In the case where the storage location or the storage method of the auxiliary information item is changed on the storage medium, the selecting means stores the auxiliary information item in a technical format having a high priority in a storage location that is easy to store or extract, or by a storage method. Information storage device. The method of claim 6, In the case where the storage location or the storage method of the supplementary information item is changed in a different manner, the selection means is stored in a storage location where the supplementary information item can be easily stored or extracted according to the priority order of the description format or the expression format. Or an information storage device for storing the unit information in a storage manner. The method of claim 6, In the case where the storage location or storage method of the auxiliary information item is changed on the storage medium, the selecting means pre-rules a rule for allocating a description format of a specific priority to each of the plurality of storage locations or storage methods on the storage medium. And an information storage device that determines a storage location or a storage method according to the rule. The method of claim 6, In the case where the storage location or storage method of the auxiliary information item is changed on the storage medium, the selection means, for each of the plurality of storage locations or storage methods on the storage medium, describe a description format of a specific priority, for each auxiliary information to be recorded. An information storage device which determines a storage location or a storage method according to the rule, by defining a rule to be allocated for each item in advance. The method of claim 1, The device is a moving image pickup device, And said main information is a moving picture, and said incident information item includes at least one of information of an imaging device, status information of an optical device, information on a user's operation, and information on a shooting environment at the time of shooting. The method of claim 11, And an information storage device for determining which sensor or device information is selected and stored when there are a plurality of sensors or devices related to the accompanying information item item. The method of claim 11, An information storage device that determines, in order of high priority, which sensor or other device information is selected and stored when there is a limitation in obtaining information from a sensor or other device. The method of claim 13, The limit is an information storage device which is a limit regarding a time for acquiring information from a sensor or another device. The method of claim 13, The limit is an information storage device that is a limit regarding a temporary storage amount of information acquired from a sensor or another device. The method of claim 11, An information storage device for determining which unit system to select in order of priority when there is a limitation in arithmetic operations for simultaneously converting information from a sensor or a device into a plurality of unit systems or precisions. The method of claim 16, The limitation is an information storage apparatus which is a limitation regarding time for simultaneously recalculating information from a sensor or a device with a plurality of unit systems or precisions. The method of claim 16, And the restriction is a restriction on the temporary storage amount of the recalculated information. The method of claim 16, The limitation is an information storage apparatus which is a limitation regarding arithmetic ability to recalculate information from a sensor or another device simultaneously with a plurality of unit systems or precisions. The method of claim 16, The restriction is an information storage device in which unacquired information is included in information from a sensor or another device. The method of claim 11, Information storage device that can describe information from sensors and other devices at the same time in a plurality of formats, and determines which sensor or other device information is selected and stored in order of high priority when there is a limitation on the technology. The method of claim 21, Wherein said limitation is a limitation on the time available to the technology. The method of claim 21, And said limit is a limit on the amount of information that can be described. The method of claim 11, An information storage device that defines a priority in accordance with the ease of acquisition of information or the frequency of use of the information. The method of claim 11, An information storage device for dividing a storage area into a plurality of predefined priorities for each piece of information when information from a sensor or another device can be simultaneously described in a plurality of formats. The method of claim 11, An information storage device for selectively describing each information according to a predefined storage area when information from a sensor or another device can be simultaneously described in a plurality of formats. The method of claim 26, An information storage device that defines a storage area for each information in response to the ease of acquisition of the information or the frequency of use of the information. A method of storing state information and accompanying information items accompanying the same, A determination step of prioritizing each of the plurality of description formats in advance for each incident information item including a plurality of description formats; A selection step of selecting a technology format to be used in recording according to the priority, from among the technology formats available in the apparatus; Recording process to record incident information items corresponding to state information in selected technology format Information storage method comprising a. The method of claim 28, And the selection step selects a plurality of description formats to be used within the ranges available in consideration of the constraints of the devices, in order from the highest priorities among the description formats available in the apparatus. The method of claim 28, The selection process selects a plurality of technical forms or expression formats to be used within the range available in consideration of the constraints of the apparatus, in order from the highest priorities among the technical formats available in the apparatus, An information storage method that selects all the technical formats having a higher priority than the lowest priority among technical formats. The method of claim 28, If there are technical formats that cannot be used simultaneously due to the constraints of the apparatus among the technical formats available in the apparatus, the selection step selects one technical format having a high priority for the technical format and the remaining techniques. An information storage method for selecting a plurality of available description formats to be used in order from the highest priority to the format. The method of claim 28, Among the technical formats available in the apparatus, when the number or combination of technical formats available is limited by the constraints of the apparatus, the selection step is a technical format with high priority within the limitation as much as possible. Choose how to save the information. The method of claim 28, The selecting step selects the technical format to be used within the available range in consideration of the constraints of the apparatus, in order from the highest priorities among the technical formats available in the apparatus, and correspondingly to the priority. An information storage method in which a storage location or a storage method of an auxiliary information item is changed on a storage medium. The method of claim 33, wherein In the case where the storage location or the storage method of the auxiliary information item on the storage medium is changed differently, the selection process stores the auxiliary information item in a high priority technology format in a storage location that is easy to store or extract, or by a storage method. How to store information. The method of claim 33, wherein In the case where the storage location or the storage method of the supplementary information item is changed differently, the selection process is performed at a storage location where the supplementary information item can be easily stored or extracted according to the priority order of the description format or the expression format. Information storage method for storing unit information in a storage method. The method of claim 33, wherein In the case where the storage location or storage method of the auxiliary information item is changed on the storage medium, the selection step pre-sets a rule for allocating a description format of a specific priority to each of the plurality of storage locations or storage methods on the storage medium. And a storage location or a storage method according to the rule. The method of claim 33, wherein In the case where the storage location or the storage method of the auxiliary information item is changed on the storage medium, the selection process is performed for each of the additional information items to be recorded in a specific priority description format with respect to the plurality of storage locations or storage methods on the storage medium. The information storage method which determines a storage location or a storage method according to the said rule in advance which rules are assigned every time. The method of claim 28, The device is a moving image pickup device, And said main information is a moving picture, and said incident information item includes at least one of information of an imaging device, status information of an optical device, information on a user's operation, and information on a shooting environment at the time of shooting. The method of claim 38, And determining, in order of high priority, which sensor or device information is selected and stored when there are a plurality of sensors or devices related to the accompanying information item item. The method of claim 38, When the acquisition of information from a sensor or another device is limited, an information storage method for determining which sensor or other device information is selected and stored in the order of high priority. The method of claim 40, And the limit is a time limit for acquiring information from a sensor or another device. The method of claim 40, And the restriction is a restriction on a temporary storage amount of information acquired from a sensor or another device. The method of claim 38, An information storage method for deciding which unit system to select in order of priority when there is a limitation in arithmetic operations for simultaneously converting information from a sensor or a device into a plurality of unit systems or precisions. The method of claim 43, The restriction is an information storage method which is a limitation regarding time for recalculating information from a sensor or a device simultaneously in a plurality of unit systems or precisions. The method of claim 43, And the restriction is a restriction on a temporary storage amount of information to be recalculated. The method of claim 43, The limitation is an information storage method which is a limitation on arithmetic ability to recalculate information from a sensor or another device simultaneously with a plurality of unit systems or precisions. The method of claim 43, The restriction is an information storage method in which unacquired information is included in information from a sensor or another device. The method of claim 38, An information storage method in which information from a sensor or another device can be simultaneously described in a plurality of formats, and when there is a limitation on the technology, which sensor or other device information is selected and stored in order of high priority. The method of claim 48, Wherein said limitation is a limitation on the time available for the description. The method of claim 48, Wherein said limit is a limit on the amount of information that can be described. The method of claim 38, An information storage method which defines a priority in accordance with the ease of acquisition of information or the frequency of use of the information. The method of claim 38, An information storage method, in which information from a sensor or another device can be described simultaneously in a plurality of formats, by dividing the storage area for each information at a predetermined priority level. The method of claim 38, An information storage method for selectively describing each information according to a predefined storage area when information from a sensor or another device can be simultaneously described in a plurality of formats. The method of claim 53, An information storage method for defining a storage area of each information in response to the ease of acquisition of the information or the frequency of use of the information. A control program for executing the information storage method according to any one of claims 28 to 54 by a computer. A storage medium for storing a control program for executing the information storage method according to any one of claims 28 to 54 by a computer.