JP4341503B2 - Information signal processing method, information signal processing apparatus, and program recording medium - Google Patents

Description

  The present invention performs predetermined band compression processing such as MPEG on video and audio information signals such as video signals and audio signals in broadcast programs, for example, and a predetermined recording medium such as a magneto-optical disk, hard disk (HDD), and semiconductor memory The present invention relates to an information signal processing method, an information signal processing apparatus, and a program recording medium when a special reproduction operation such as a predetermined digest reproduction process is performed in a recording / reproducing apparatus that records and reproduces information on a recording medium.

  In a conventional VTR or disc recording / reproducing apparatus, when the recorded contents recorded for a long time are reproduced in a short time and the contents are grasped, the speed of understanding the voice information is taken into consideration at about 1.5 to 2 times speed. A playback process is being performed.

  If playback is further performed in a short time and summary playback (digest playback) is to be performed, it is difficult to understand the content with high-speed audio output.

  Therefore, predetermined feature data is extracted from the video / audio data of the broadcast program to be recorded, and a key frame section that is considered to be a key frame (important frame) is detected using the predetermined feature data, and a predetermined predetermined data is detected. It is considered to perform digest reproduction within a predetermined time shorter than the recording time of the originally recorded broadcast program by sequentially selecting and reproducing predetermined key flake sections according to the rules.

  In addition, position information data indicating a playback position is automatically generated at a predetermined time interval, for example, every 3 minutes, 5 minutes, 10 minutes, or the like, or a user manually sets a desired position in a recorded predetermined section. It is considered to generate position information data, so-called chapter data generation, and to perform skip reproduction, editing operation, and thumbnail image display using the position information data (chapter data).

JP 2002-344872A

  The feature data can detect a plurality of feature data for each image signal and sound signal, and each feature data is extracted together with the image sound recording data, for example, when the image sound signal is recorded. It is conceivable to record data on a recording medium.

  The recorded feature data is read out and signal processing is performed to determine the digest playback section by a predetermined rule process. However, if a plurality of feature data are separately recorded on the recording medium as files, The number of files increases and handling of files during signal processing becomes complicated and is not efficient. Conventionally, there has been no effective technique for solving these problems.

  Therefore, in view of the conventional situation as described above, an object of the present invention is to efficiently process feature extraction data and perform effective digest reproduction in order to perform effective digest reproduction operation or chapter processing using feature extraction data. Another object of the present invention is to provide an information signal processing method, an information signal processing apparatus, and a program recording medium for efficiently performing various operations using chapter data.

  Other objects of the present invention and specific advantages obtained by the present invention will become more apparent from the description of embodiments described below.

The present invention provides an information signal processing method for recording or reproducing a predetermined image / audio information signal by using a predetermined band compression signal process using a predetermined recording medium, in the case of recording an image / audio information signal . The image / audio information signal is automatically extracted at the time of recording a predetermined characteristic signal in the image / audio signal . The image / audio information signal of the digital input system is the image / audio signal after the recording of the image / audio information signal is completed. The characteristic data extraction process for extracting a predetermined characteristic signal at the time is automatically performed, or the characteristic data extraction process is automatically performed by a predetermined selection operation after the recording of the video / audio information signal is completed, or manually when desired. It is characterized in that information signal processing performed by selecting whether or not to perform is performed .

An information signal processing apparatus according to the present invention comprises a recording means for recording a predetermined image / audio information signal on a predetermined recording medium by predetermined band compression signal processing, and predetermined characteristic data for each predetermined section of the image / audio information signal. a characteristic data extraction unit for performing characteristic data extraction process for extracting, when recording image audio information signal from the recording unit, image and sound information signals of the analog input system, the characteristic data extraction process by the characteristic data extraction unit Is automatically performed at the time of recording the audio / video information signal, and the audio / video information signal of the digital input system is automatically subjected to the characteristic data extraction processing by the characteristic data extracting means after the recording of the audio / video information signal. Or after the recording of the video / audio information signal is completed, the characteristic data extraction process is automatically performed by a predetermined selection operation means or manually when desired. Characterized by comprising an information processing means for performing either the select performed.

The present invention is a program recording medium in which a control program for recording or reproducing a predetermined video / audio information signal by a predetermined band compression signal process using a predetermined recording medium is recorded so as to be readable and executable by a computer. When recording an audio information signal, an analog input system image audio information signal is automatically extracted at the time of recording a predetermined characteristic signal in the image or audio signal, and a digital input system image audio information signal is The characteristic data extraction process for extracting a predetermined characteristic signal in the image or audio signal is automatically performed after the recording of the image / audio information signal is completed, or the characteristic data extraction process is performed after the recording of the image / audio information signal is completed. A control program characterized by selecting whether to perform automatically by a selection operation or manually when desired is stored in a computer Ri is characterized by comprising a read executable recorded.

According to the present invention , predetermined feature extraction processing can be efficiently performed in video / audio information signal recording / reproduction, thereby enabling effective digest reproduction and chapter data (predetermined recording position information data) setting using feature extraction data. Can be efficiently realized.

  In addition, according to the present invention, even after a user purchases a recording / reproducing apparatus not equipped with a certain function as described above, if the function is desired, the function can be easily operated on the purchased apparatus itself. State.

  In addition, recording / playback devices equipped with only basic functions can be sold at an early stage, and after that, according to the demands of various users, they can easily be equipped later on the purchased devices themselves. Therefore, the user can purchase the device efficiently.

  According to the present invention, it is possible to efficiently combine a plurality of different types of image feature data and audio feature data as each feature data file or one feature data file. For example, a plurality of feature data such as a camera feature, a telop feature, a scene feature, a color feature, and an audio feature of an image feature can be efficiently processed as a data file in a predetermined format. File management, file processing during signal processing, etc. can be efficiently performed.

  In addition, since no file is provided for each feature data, the recording capacity occupied by the file can be reduced as compared with the case where a file is provided for each feature data in the recording medium.

Hereinafter, embodiments of the present invention will be described in detail in the following order with reference to the drawings. The present invention is not limited to the following examples, and it is needless to say that the present invention can be arbitrarily changed without departing from the gist of the present invention.
(1) Outline of System to which Present Invention is Applied 1.1 Summary Reproduction and Predetermined Time Setting Process Using Feature Extraction Data Here, an outline of operation process of the present invention will be described.

  The signal processing related to the following operation outline will be described in detail in the following items in addition to the items here.

  In addition, in the following description, in addition to specially describing the play list (play list data) generation, play list data generation and predetermined chapter data (predetermined recording position information data; the same applies hereinafter) even when not specifically described. May be generated and processed together.

  Explanatory diagrams of summary reproduction (digest reproduction) and chapter processing using feature extraction data are shown in FIGS.

First, a summary reproduction operation using feature extraction data will be described.
(Digest playback process using feature extraction data)
Here, it is assumed that there is a video / audio data sequence as shown in FIG.

  The video / audio data series may be broadcast programs, movie software, etc., using a predetermined recording medium such as a hard disk (HDD), a magneto-optical disk, a large-capacity semiconductor memory, and a predetermined band compression signal processing such as MPEG. It is assumed that recording and playback processing is performed.

  FIG. 1B is a conceptual diagram of a predetermined section in which predetermined semantics are set in the video / audio data series and divided into predetermined video structures (semantic video structures) according to scene changes, audio segments, and the like. Show.

  The predetermined meaning setting process, the predetermined section setting process, the video structure, and the like will be described later.

  Here, as shown in FIG. 1C, for each section obtained by dividing the above meaning into predetermined sections, all sections recorded within a predetermined time, predetermined program sections, etc. A predetermined evaluation value for each of the above sections is set.

  Here, it is assumed that a predetermined higher evaluation value (evaluation data) is set as the predetermined evaluation value becomes a predetermined key frame (important frame, important (video / audio) interval) in all the predetermined intervals.

  That is, if a predetermined evaluation data section is reproduced, the section includes a key frame section, so that the outline can be grasped without reproducing all the sections.

(C) in FIG. 1 shows an outline of the predetermined evaluation value section. In the video / audio data series shown in (A) in FIG. 1, each section of f1 to f2, f4 to f5, and f7 to f8 is an evaluation value. As shown in FIG. 1D, the A1, A2, and A3 sections are skip-played in a predetermined summary playback mode to perform a predetermined digest playback in a section equal to or greater than the threshold value Th set in FIG. Become.
(Automatic chapter processing using feature extraction data)
FIG. 1E is a conceptual diagram in the case of setting a chapter point. As described above, the beginning of the predetermined key frame (important frame, important evaluation value section) or its vicinity, and the key frame. It is conceivable to set a chapter point at or near the beginning of a section that is not the key frame section that follows (connects to the end) at the end of the section.

  Here, for example, in a conventional DVD recording / reproducing apparatus, by setting a breakpoint of a predetermined section called an automatic chapter function, that point can be used as a guideline for editing operation, fast forward playback (FF playback), fast forward reverse It can be used in the case of reproduction (rewind reproduction, REW reproduction) or the like.

  In the prior art, as the above-described automatic chapter function, for example, a process in which time intervals are processed at equal intervals such as 5 minutes, 10 minutes, 15 minutes, and the like is known. In such a chapter process, as shown in FIG. 1G, there may be a case where a chapter point cannot be set as a start point at a time point that seems to be a key frame.

  In addition, in the prior art, there is a function called manual chapter processing that allows a user to set a chapter point at an arbitrary time as desired. However, a program (program) recorded or recorded by the user himself / herself is actually viewed. This is a troublesome operation for the user and is not efficient.

  However, as described above, in the chapter point setting process using the feature extraction data of the present invention, as shown in FIG. 1E, the head of the key frame section or its vicinity and the key frame are appropriately displayed. Since chapter points can be automatically set at or near the beginning of a section that is not connected to the end of the section (or continues to the last) key frame section, it is more effective than chapter processing according to the prior art. It is possible to set chapter points, and to perform effective editing operations (editing processing) using this chapter processing, FF playback, and REW playback.

  Here, FIG. 2 shows a conceptual diagram in a case where the automatically set chapter points shown in FIG. 1F are displayed on a predetermined image monitor as thumbnail images of a predetermined size.

  As shown in FIG. 1 (F), f1, f4, and f7 are the heads of the predetermined key frame sections A1, A2, and A3 (or in the vicinity thereof), and f3, f6, and f9 are A1, A2, and A3, respectively. The head (or the vicinity thereof) of the sections B1, B2, and B3 that are not the key frame sections after the section, and the user sees the display screen as shown in FIG. 2, for example, the processing recording medium of the recording / reproducing apparatus. Processing such as cutting out the key frame sections A1, A2, and A3 shown in FIG. 1D in the broadcast program recorded on the hard disk and recording them on a disk recording medium such as a DVD by an editing operation, and f1, f4, and f7 It is conceivable to assume an operation such as skip playback at the point of time.

  As shown in FIG. 1 (G) as an example of a predetermined time point set point (chapter point) according to the prior art, a set point (at a predetermined constant interval such as a 5-minute interval, a 10-minute interval, or a substantially constant interval). The chapter points are set, but as shown in FIGS. 1C and 1G, they are not necessarily set to key frames (important frames).

  As described above, by performing the predetermined chapter point (predetermined set point or predetermined breakpoint) or segment processing automatically using the feature extraction data in the present invention, more effective editing operation and skip reproduction can be performed. it can.

1.2 Example of Processing Process of the Present Invention An example of the processing process of the present invention is shown in FIG.

  The example of the processing process shown in FIG. 3 includes a feature extraction process (2) for extracting image and audio feature data from MPEG video and audio stream data.

  Here, for simplicity, the MPEG data (1) is assumed to be recorded on a predetermined recording medium or data recorded on the predetermined recording medium. For example, a predetermined transmission system (wired system or wireless system) is assumed. The present invention can be similarly applied to the image and sound data transmitted in (1).

  The feature extraction process (2) can be performed at the same time as the recording process. However, if image / audio data has already been recorded on a predetermined recording medium, the characteristic extraction process (2) is performed by reproducing from the recording medium. It is also possible.

  Here, rule processing (rule processing) will be described.

  The rule processing is performed using a file or data in which the rules are described in a predetermined format.

  In the rule file and rule data, for example, rules based on feature data corresponding to the program genre are described, and this rule file and rule feature data and PU feature data file describing each feature data of a predetermined section (reproduction unitization) A predetermined play unit file is generated by calculation with the feature data.

  Here, for the sake of convenience, for convenience, it is assumed that a rule file for a predetermined program genre n is Rf (n), a PU file (feature data file) is Pu, a playlist file is Df, and a desired summary time is t. , And can be expressed by the following equation (1).

Df = Pu (*) Rf (n) (*) t (1) where (*) is assumed to be a predetermined operator using data of a predetermined file.

  As will be described below, the rule file and the rule data Rf (n) are described in a predetermined format, for example, a predetermined time correction function, a description of meaning, a weighting coefficient (evaluation value, importance) of the meaning, etc. It is composed of data of predetermined parameters.

(Playback unit processing)
After the feature extraction process (2), a PU (playback unit (play unit)) process (3), which is one of the features of the present invention, is performed.

In the PU process (3), each feature data is recorded (stored) in a predetermined recording medium or buffer memory as predetermined data or as a file at a segment (segment) (4) called PU.
(Rule 1 processing)
The PU data is subjected to PU meaning processing by predetermined rule 1 processing (5). As will be described later, the outline of the rule 1 process (5) is as follows.

(Processing 1) Extraction of each feature (Processing 2) From the combination of features, select the one that satisfies the most conditions among the meanings expressed by the first rule. (Processing 3) Adopt the selected meaning as the meaning of the PU. In the rule 1 process (5), parameters such as program genre, program genre, program genre watched by the user in the past, time zone, number of playbacks, playback time, playback date and time, etc. according to EPG (electronic program guide) and the like, side When information or the like can be used, predetermined processing may be performed in consideration of these parameters.

  The process of the time correction function performed in association with this process will be described later.

(Rule 2 processing)
The given PU (6) is subjected to a predetermined evaluation value process in a predetermined rule 2 process (7).

  In the rule 2 process (7), the evaluation value process for the importance of the following (process 1) and (process 2) is performed.

(Processing 1) Importance of meaning (Processing 2) Importance based on appearance pattern of meaning In PU (8) for which a predetermined evaluation value process has been performed, a PU alone or a PU group in which several PUs are connected is specified. The evaluation value is attached.

  Here, in the rule 1 process (5) and the rule 2 process (7), as shown in FIG. 4, the rule switching process system 900 performs genre A rule data, genre as rule process data corresponding to a plurality of program genres. B rule data, genre C rule data,... And some rule processing data (rule processing data), and according to the program genre information data input to the system controller system 20, rule 1 processing (5), It is also conceivable to switch the rule 2 process (7) or one of the rule processes.

  In addition, as shown in FIG. 4, it is conceivable to switch by providing some rule processing data for each individual.

  In this case, in the predetermined operation mode, any one of the personal 1 rule processing data, the personal 2 rule processing data, the personal 3 rule processing data, etc., is set by a predetermined user input to the system controller. However, selection processing is performed via the system controller system 20, and predetermined rule processing is performed based on the selected rule processing data.

  By providing individual rule processing data as shown in FIG. 4, for example, a predetermined reproduction operation such as normal reproduction or special reproduction is performed for each individual, and the reproduction state, operation information such as reproduction position, operation position information, etc. Operations such as storing data in predetermined memory means so that it can be reflected in predetermined individual rule processing, and updating the data as required by individual learning processing as individual rule processing data at a predetermined timing It is considered that this is an effective signal processing method for such individual learning processing.

  As shown in FIG. 4, even when switching each individual rule process (rule process) by the rule switching processing system 901, the rule 1 process (5), the rule 2 process (7), or one of the rule processes is performed. It is possible to switch.

(Rule processing format)
(Rule 1 processing)
Here, for example, assuming a certain broadcast program, the PU that has been subjected to meaning processing is described in association with predetermined audio / video feature data by setting the following English characters and meanings.

  For the meaning of characters, select a scene that is assumed to be a key frame (important frame, important scene) in the broadcast program, or a predetermined recording and playback section that is assumed to be effective for summary playback, chapter setting, etc. Describe.

  In some cases, it may be possible to describe a scene desired by the user.

  In this case, it may be possible to describe a rule desired by the user in a predetermined adjustment mode.

  Here, as shown in the upper part of FIG. 5, an example of a news program and a sumo program is shown in Table 1 below.

  Here, for example, in this example, a rule for extracting an announcer's scene with a is described, but it is not possible to extract all the assumed a scenes (announcer appearing scenes) in one rule process. Because it seems, it can be considered to be divided into several rules.

  Similarly, other cases such as b, c, d, and e can be divided into a plurality of rules.

  In the case of a sumo program, it is as shown in Table 2 below.

  Also in this case, it is conceivable to set a plurality of rules for each character.

  Depending on the case, the description is made by dividing the assumed rules of the scene (key frame) to be extracted.

  Also, since it is possible to assume scenes that cannot be uniquely defined in a broadcast program, for example, it is possible to consider @ as a setting character and set as shown in Table 3 below.

  The rule 1 process for the set characters (meaning characters) set in this way will be specifically described taking a news program as an example.

  As shown in FIG. 18, when each predetermined feature data is detected, it is assumed that each scene corresponding to the definition characters a, b, c, d, and e in the case of the news program described above corresponds.

  Here, it is assumed that a predetermined process of logical product is performed in the case of ◯, and logical sum in the case of △. For example, in the announcer scene of the definition character a, the speech feature attribute is speaker speech, color feature detection region 2 or A predetermined color is detected in the detection area 3, the first or second frequency of the similar image information is detected, detected in the detection area 1 or the detection area 2 or the detection area 5 of the human feature, and the camera feature is stationary. Can be assumed.

  Other b, c, d, e, and the like can be associated with each predetermined characteristic data and can be associated with the meaning character and the characteristic data in the same manner as in the case of a, according to the marks ◯ and Δ in FIG. .

  As described above, each semantic character and each feature data can be described according to a predetermined format in order to perform predetermined processing, that is, rule 1 processing and rule 2 processing.

  FIG. 6 (A) is an example, and is described assuming a vector component.

  That is, each feature data shown in FIG. 5 is, for example, an attribute of a voice feature, the attribute is A1 for speaker voice, A2 for music, and A3 for other attributes.

  In the color feature of the video feature, region 1 is B1, region 2 is B2, and so on.

  Similarly, B1 to B4, C1 to C2, D1 to D5, E1 to E4, F1 to F4, and G1 can be set for each feature.

  In FIG. 6A, for example, the meaning character a can be described as the following equation (2).

a = 1.0 (A1) 100 * (1.0 (B2) 100 + 1.0 (B3) 100) * (1.0 (C1) 100 + 1.0 (C2) 100) * (1.0 (D1) 100 + 1 0.0 (D2) 100 + 1.0 (D5) 100) * 1.0 (F1) 100 (2) Formula Other meaning characters can also be described as shown in FIG.

  Here, it is assumed that “*” represents a predetermined operation similar to a logical product (AND) and “+” represents a logical sum (OR).

  Here, for example, a description of 1.0 (A1) 100 is considered.

  As described above, (A1) represents the case where the attribute is speaker voice and the voice feature.

(Weighting factor)
1.0 of 1.0 (A1) 100 is a weighting coefficient for (A1), and here, for convenience, a range of 0 to 1.0 is assumed.

  Since the weighting coefficient is a convenient coefficient for performing a predetermined calculation, the weighting coefficient may be set (described) in the range of 0 to 100, or 0 to 10.

(Detection rate coefficient)
100 of 1.0 (A1) 100 is a detection ratio coefficient for (A1), and when 100% is detected in the playback unit section, 1.0 (A1) 100 satisfies the condition. .

  For example, in the case of 1.0 (A1) 50, when 50% is detected in the playback unit section, 1.0 (A1) 100 satisfies the condition.

  This detection ratio will be described with the following equation (3).

  Here, the detection ratio coefficient assumes the range of 0-100 for convenience.

  Since the detection ratio coefficient is a convenient coefficient for performing a predetermined calculation, it may be set in the range of 0 to 1 or set (described) in the range of 0 to 10.

  Here, the detection ratio coefficient can be considered as a ratio at which the characteristic can be detected in the reproduction unit section.

  For example, in the above 1.0 (A1) 100, it can be considered that if the speaker voice is not detected 100%, it is not determined that the characteristic (A1) is detected.

  For example, in 1.0 (A1) 50, when 50% is detected, it is determined that the characteristic is detected.

  That is, it can be considered that the ratio of the predetermined characteristic detected in the predetermined section can be expressed by the coefficient.

(Characteristic (extraction) data detection rate)
Therefore, the ratio of characteristic detection is considered.

  The processing method will be described later with reference to FIGS. 34 to 35. In the present invention, a predetermined section called a playback unit (or play unit) (PU) that is set according to the audio segment feature and the scene change feature is used. The processing concept to be set is introduced.

  Therefore, for example, it is conceivable to calculate the ratio of each predetermined characteristic mentioned above at the ratio at which the predetermined feature data is detected with respect to the entire PU section.

  For example, in FIG. 7, assuming that the section length (frame length, time length, etc.) of a certain playback unit is fa and the detection sections of certain feature data P are f0 and f1, the detection rate F of feature data P in this case is The arithmetic processing can be performed by the following equation (3).

F = Σfi / fa
= (F0 + f1) / fa (3) Expression The calculation value according to the expression (3) is used in the evaluation value processing described later.

(Example of evaluation value calculation method) (Example 1)
An example of an evaluation value (importance) calculation method is shown.

For each feature data, the following processing is performed considering the ideal value and the detection result.
For example, the following (Process 1) to (Process 5) are performed considering p = m (M) n.

  (Process 1) The detection ratio s of each predetermined feature data is calculated using the above equation (3).

(Process 2) Compared with the detection ratio coefficient n,
When s <n, p = m × s (4)
When s ≧ n, p = m × 100 (5).

  (Process 3) In the above process, in each feature M, in the case of features having the same attribute such as M1, M2,.

  In the case of logical product (*) processing, for the sake of convenience, a processing concept such as a logical product coefficient r is introduced, and a value multiplied by the result of the average processing is considered.

  (Process 4) The above-described process is performed for each feature data M, the calculation values are added, and the processing result is used as the evaluation value.

  (Process 5) The calculated evaluation value is compared, and the meaning when the evaluation value is the largest is the meaning of the reproduction unit a.

  The above evaluation value processing is an example of a processing method, and the processing method in which the correspondence between the detected feature data or the ratio detected in the playback unit section and the set “meaning” has a predetermined validity. If so, other processing methods may be used.

  For example, in the case of the logical product process in the above (Process 3), there may be a case where only the process of adding feature data having the same attribute is performed without applying the averaging process or the logical product coefficient.

  In the case of the above processing (processing 3), in the case of logical product processing with the same feature data, the detection condition becomes stricter than in the case of logical sum processing, so the detection value is taken larger than in the case of logical sum processing. It can also be considered to perform the process.

  Here, consider the case of the above equation (2).

  For example, the detection ratio of each feature is as shown in Table 4 below, and the detection ratio coefficient and the weighting coefficient are shown together.

  Here, in the case of logical sum processing (+) when the same feature type, such as B2, B3, C1, C2, and the like, or when the detection attribute is different or when the detection area is different, the average is used for convenience. Considering the processing, the evaluation value h is expressed by the following equation (6) from the equation (2).

h = 100 + (80 + 80) / 2 + (100 + 100) / 2 + (80 + 80 + 80) / 3 + 80
= 100 + 80 + 100 + 80 + 80
= 440 (6) Alternatively, it is also possible to use the value averaged by the type of feature data as an evaluation value. In that case, since the feature data is five types of A to F, the following equation (7) is used. It can also be considered as an evaluation value as shown.

h = 440/5
= 88 Equation (7)

(When the relationship between feature data with the same attribute is logical product processing)
Here, in the processing of equation (2), when feature data having the same attribute, for example, B2 and B3 are logical product processing, that is, (1.0 (B2) 100 * 1.0 (B3) 100) Think about the case.

  The concept of logical product processing coefficient r is introduced from (process 3) of the evaluation value process, and a process such as r (80 + 80) / 2 is considered.

For example, if r = 1.5,
h = 100 + 1.5 × (80 + 80) / 2 + (100 + 100) / 2 + (80 + 80 + 80) / 3 + 80
= 100 + 120 + 100 + 80 + 80
= 480 Equation (8) Also, averaging processing is performed using the feature data type 5 h = 480/5
= 96 An evaluation value of equation (9) can be considered.

  This is a case where the condition of the logical product process is stricter than that of the logical sum process, so that it is better to set the evaluation value of the detected “meaning” larger.

In some cases, r = 0.8,
h = 100 + 0.8 × (80 + 80) / 2 + (100 + 100) / 2 + (80 + 80 + 80) / 3 + 80
= 100 + 64 + 100 + 80 + 80
= 424 (10) Also, averaging processing is performed with the feature data type 5 h = 424/5
= 84.5 An evaluation value of equation (11) can also be considered.

  This is a case opposite to the above case, where the condition for the logical product process is stricter than that for the logical sum process, and it is considered that it is better to set the evaluation value smaller.

(When the relationship between feature data with different attributes is ethical processing)
Here, for example, as shown in the equation (2), feature data having different attributes is expressed by a logical product operator (*), but a case of a logical sum operator (+) is also conceivable.
For simplicity, consider only the first item A1 and the second item B2 in equation (2),
a = 1.0 (A1) 100 + 1.0 (B2) 100 (12) Formula For convenience, processing is performed in consideration of a concept such as the logical sum coefficient w as described in the evaluation value calculation method (3). Can be considered.
In this case, from the equation (12), the evaluation value h is
h = (100 + 80) w (13) here,
w = 1
In the case of AND operation,
a = 1.0 (A1) 100 * 1.0 (B2) 100 (14) Formula h = 100 + 80
= 180 Equation (15).

For example, in the case of the logical sum processing of equation (8),
w = 1.5 (16)
h = (100 + 80) × 1.5
= 270 It is also possible to consider processing that gives an evaluation value higher than that of the formula (17) and the logical product processing.

Also,
w = 0.8 (18)
h = (100 + 80) × 0.8
= 144 As shown in the equation (19), a process with an evaluation value smaller than the logical product process is also conceivable.

  Since the evaluation value processing is a concept introduced for the sake of evaluation of the value of an expression that combines the set meaning and each feature data, various coefficients, etc., the range, value, etc. of each coefficient considered in the above evaluation expression are: Not only the case described above but also a small or large setting is conceivable.

  By calculating the evaluation value as described below, the evaluation value of each section of the playback unit described in the rule is determined by the rule file. For example, in the summary playback mode, the evaluation value is determined according to the summary playback time. A PU section with a small evaluation value is selected gradually so that a large PU section is selected and is as close as possible to the summary time.

  Then, by reproducing each selected PU section, predetermined summary reproduction can be realized.

(Other processing methods for evaluation value processing)
Considering w (M) * k from one term of each feature data n described above and a predetermined operator *, the detection rate det and detection rate coefficient k of each predetermined feature data is a feature of each term of the evaluation formula. Consider a calculation function P and an operator *, where w (n) is a weighting factor of data n.

Consider P (* k (n), det (n))
d (n) = P (* k (n), det (n)) Equation (20).

Here, as one of the following operators *
d (n) is
(1) When * = (||>), that is, P ((||>) k (n), det (n)),
if (k (n) ≦ det (n)) then d (n) = 0 (21) formula else d (n) = 100 (22) formula (2) In the case of * = (|| <), that is, P ( (|| <) k (n), det (n))
if (k (n)> det (n)) then d (n) = 0 Expression (23) else d (n) = 100 Expression (24).

  In the case of the processes (1) and (2) above, the midway processing value d (n) is processed to 100 or 0 according to the detection det (n) and the set detection ratio k (n). Compared with the case where the midway processing value described in (3) or (4) is a difference value, it is considered effective when it is desired to characterize feature data significantly.

In addition,
(3) When * = (|>), that is, P ((|>) k (n), det (n)),
if (k (n) <det (n)) then d (n) = 0 (25) equation else d (n) = | k (n) −det (n | (26) equation (4) * = (| <), That is, P ((| <) k (n), det (n)),
if (k (n)> det (n)) then d (n) = 0 (27) expression else d (n) = | k (n) −det (n) | Becomes the following equation (29).

  By introducing the above operator, for example, when there is feature data of A1 and B2, it can be described as follows.

a = 1.0 (A1) (|| <) 100 + 1.0 (B2) (| <) 100 (30) In this case, for example, the detection ratio (actual detection value) of the A1 feature is 100, and the B2 feature Assuming that the detection ratio (actual detection value) is 80, from (1) and (4) above, the evaluation value h is
h = (1.0 * (100-0) +1.0 (100-80)) / (1.0 + 1.0)
= (100 + 20) / 2
= 60 An evaluation value of equation (31) can be considered.

  As described above, several methods can be considered as the evaluation value processing method, and the method is not limited to the method described here.

  Here, in the description of Rule 1 shown in FIG. 6A, an example of a method for expressing the appearance pattern (meaning) of the data to be described is assumed to have a, b, c. It is also conceivable to use A, B, C,...

(Rule 2 processing)
In the rule 2 process, the process is performed in consideration of the connection of the meanings of the reproduction units that are the predetermined sections given in the rule 1 process.

  In addition, temporal correction, that is, temporal weighting processing is performed using a time correction function.

For example, in the rule 1 process, assuming that the evaluation value of meaning a is 70 and the evaluation value of meaning b is 80, the evaluation value g of (ab) is
g = 70 + 80
= 150
Or, considering the average number of meanings, here it is two of ab,
g = 510/2
= 75
Or, considering the product of each evaluation value,
g = 70 × 80
= 5600
For example, for convenience, assume that the maximum value is 100 and normalize with the maximum value,
g = 5600/100
= 56
It is also possible to do so.

  The weighting of the time correction function is, for example, that (ab) can be detected at a certain time t, the evaluation value is g, and the time correction coefficient (weighting coefficient) at t is w. Value.

  The time correction function describes the change point (information data of the change point coordinate system) at the predetermined description location of rule 2 in the rule file according to the predetermined description rule.

An example of the rule 2 process is shown in FIG.
(Time correction function)
First, the time correction function will be described.

  This time correction function can be used to correct the summary time in a predetermined program genre in the rule file.

  For some users, it may be assumed that the user wants to play the first half or second half of the broadcast time intensively depending on a predetermined broadcast program.

  Therefore, in consideration of various parameters depending on the broadcast program such as the genre of the program to be recorded, the broadcast time, etc., the time (in some cases, the time in some cases) for a predetermined playback section in which digest playback (summary playback) is performed. ) Weighting can be considered.

  In other words, the interval for performing the weighting is processed so that the importance of reproduction when performing digest reproduction is larger than the interval for which no other weighting is performed in time.

  8A to 8I show an example of a time correction function for performing the time weighting described above.

  (A) of FIG. 8 is a case where the time is not weighted for a predetermined summary reproduction section with a flat characteristic.

  FIG. 8B shows a case where weighting for increasing the weight of reproduction as importance in summary reproduction is performed by comparing the first half with the second half within a predetermined section.

  FIG. 8C shows a case where weighting is performed to increase the weight of reproduction as the importance in summary reproduction by comparing the latter half with the first half within a predetermined section.

  FIG. 8D shows a case where weighting is performed to increase the weight of reproduction as the importance in summary reproduction by comparing the first half and the second half with the middle in a predetermined section.

  FIG. 8 (E) shows a case where weighting is performed to increase the weight of reproduction as the importance level in the summary reproduction by comparing the middle part with the first half part and the second half part within a predetermined section.

  (F) in FIG. 8 is like connecting two correction functions having different shapes from those in FIG. 8 (D), and between the first half, the first half and the middle, the middle, and between the middle and the second half. The weights are assigned to the latter half, and the weights are made different.

  (G) in FIG. 8 is such that two correction functions having different shapes from those in (E) in FIG. 8 are connected, and between the first half, the first half and the middle, the middle, and between the middle and the second half. The weights are assigned to the latter half, and the weights are made different.

  (H) in FIG. 8 is a combination function of (C) and (D) in FIG. 8, and (I) in FIG. 8 is a combination function in (D) and (B) of FIG.

  FIG. 9 shows a general time correction function. The coordinates of the start point, the change point, and the end point are P0 (ts, s3), P1 (t1, s3),..., Pe (te , S0).

  Here, since the y component of the coordinates represents weighting, here, for convenience, the maximum value is 100, the minimum value is 0, and the value is between 0 and 100, and the x coordinate is position information, which will be described later. The same dimension value as the “start / end position information” data shown in FIG. 41 to FIG. 43, or the ratio from the start point based on the section between the start and end points, set between 0 and 100 to indicate the position It is also possible.

(Reproduction unit meaning, connection relationship, judgment processing)
As described above, the meaning of the reproduction unit (PU) can be set from feature data obtained by a predetermined feature extraction process.

  Here, the structure of video data as shown in FIG. 10 will be considered.

  Assuming one program (program) k, it can be considered to classify into several scenes m, m + 1,..., And the scene can be classified into several shots.

  Each shot is considered to be a single frame.

  A scene change is a scene change.

  A segment (or a shot or an image segment, the same applies hereinafter) is considered to be a group of similar images or a group of similar image (video) characteristics according to the scene.

  A segment, a scene, or the like is considered to have a unique meaning concept in the program (program).

  Then, a video structure can be considered in which several segments and scenes having respective meanings are grouped to constitute the program.

  For example, if a baseball program is assumed and the batter's screen continues, it is considered that a similar image of the batter is detected and classified into the similar characteristic segments.

  The segment has the meaning (concept of meaning) of “batter's image”.

  Further, if the pitcher throwing screen continues, a similar pitcher screen is detected, and it can be considered that the screen can be classified into segments according to the similar characteristics.

  Then, the segment has a meaning (concept of meaning) of “pitcher image”.

  Here, for example, when considering a case where a pitcher throws, a batter hits and the batter strikes, “pitcher image scene”, “batter image scene”, “batter strikes” It is possible to consider the connection of image scenes each having meaning.

  Consider that image feature data and audio feature data are processed for each PU described above in a predetermined program (program), and the meaning of the PU is set according to the feature data.

  For example, when a news program is considered, if a caster (announcer) first reads a news item (news program headline) and assumes a scene (image headline), the scene (image) has one person feature. ~ Two people, telop (Tlp feature), voice feature attribute is speaker voice, etc. In addition, when considering a news program, it is considered that there are several scenes that read the news in the news program, There are some scenes similar to the news reading scene, and it is considered that the appearance frequency of the similar image feature, that is, the specific scene ID increases.

  In this way, as described in the rule 1 process, it is conceivable to set the meaning of the PU according to the person feature, voice feature, telop feature, similar image feature, and other predetermined feature data.

  For example, as in the case of the above baseball program, it is possible to consider PU connection relationships having a predetermined meaning.

  That is, a predetermined connection between PUs having predetermined feature data or characteristic data can be considered.

  FIG. 11 shows a connection relationship of PUs having the above-described predetermined meaning, that is, a predetermined meaning set.

  In FIG. 11, when a predetermined meaning a to meaning d is set in the program (program) currently considered, and connection relation is considered in a certain section PU (n) to PU (n + 2), PU (n) Meaning a, PU (n + 1) meaning b, and PU (n + 2) meaning c are considered to be the most natural connections.

  That is, the example shown in FIG. 11 assumes the case of sumo, and it is most reasonable and reasonable that the meaning a “approach introduction scene” is followed by the meaning b “witness scene”. The meaning b “attendance scene” is followed by the meaning c “working scene”.

  Then, as a connection relationship, a character sequence defining the meaning of abc can be considered. If this abc sequence is a key frame, the abc is searched for in the currently considered program (program), and the predetermined section searched. It is conceivable that the setting process is performed with the first and last of or the vicinity thereof as a predetermined set point.

  As another example, for example, when the program genre is baseball, if the playback unit is “throw”, “hit”, “no meaning”, “score” in a certain section, Except for “None”, PUs determined to have three meanings, “throw”, “hit” and “score” are combined into one, and a set of predetermined PUs “throw, hit, score” is collected. Can be assumed.

  Here, since it is determined that there is no meaning for PU of “no meaning”, it is considered that there is no problem even if it is included, and the above four PUs are combined into one and called “throw, hit, meaningless, score” A set of predetermined PUs can also be considered.

  Here, “no meaning” is considered because a predetermined meaning process is performed from a plurality of set meanings in a predetermined evaluation process from a predetermined feature data in the above rule 1 process. This is because it can be assumed that a certain meaning cannot be given based on the predetermined signal processing.

  Instead of “no meaning”, it is possible to set “any meaning”. This is the same processing as @ described above.

  In the case of a certain news program, a connection of aabb, that is, a connection of “announcer scene”, “announcer scene”, “site scene”, “site scene” is considered to be reasonable and reasonable in FIG. Shown in (A).

  Moreover, the case of the sumo program demonstrated previously is shown to (B) of FIG.

  FIG. 13 shows a case where the above-mentioned program genre is a news program. As shown in FIG. 13A, the reference pattern (reference character sequence) is “aabb” described above, and FIG. As shown, the section “aabb” is searched for in the predetermined program recording section under consideration, and the sections A1 and A2 match “aabb”, indicating that the search has been completed.

  Then, as shown in FIG. 13B, for example, the first positions p1 and p3 and the last positions p2 and p4 of the searched “aabb” section are set as predetermined setting positions, which will be described later. A predetermined process is performed as position information data of the playlist.

  For example, in the summary playback mode, playback control processing is performed so that the set positions p1 to p2 and p3 to p4 are played back.

  In the case of predetermined time point setting (predetermined position setting) processing such as chapter setting, predetermined processing is performed with each time point of p1, p2, p3, and p4 or a position in the vicinity of each of these points as the set position. It is possible.

  In this way, it is determined that the meaning of the predetermined PU is determined from the predetermined feature data, and if it has the predetermined meaning, it is set to the PU, and the meaning connection relation is assumed from the PU set by determining the meaning. Thus, processing can be performed assuming a connection of a predetermined number of PUs according to a predetermined meaning or a set of a predetermined number of PUs.

  In the example of the description method of the rule 2 process shown in FIG. 6B, the key frame (important frame) is assumed, the character sequence to be searched is set as (abb), and then 100 is set as the weighting coefficient. is doing. Subsequent Ps (ts, s4), P1 (t1, s4), and Pe (te, s3) are the time correction functions described above. In this example, as shown in FIG. It is a function whose importance gradually decreases in the second half.

  In the case of the time correction function as shown in FIG. 14, it is considered that the time correction function is suitable for the case where it is desired to watch the first half of the program with priority.

  Here, the description of the rule 2 process shown in FIG. 6B is an example of a method for expressing the appearance pattern (meaning) of the data to be described, and the meaning is a, b, c,. However, A, B, C,... Can be used as the negation, and * can be used as a wild card.

  In the description of the rule 2 process shown in FIG. 6B, when the news program genre is one, for example, (Abb), A is other than “announcer scene”, b Will be referred to as “on-site scene”, and a case where two “on-site scenes” continue in addition to “announcer scene” will be detected.

Here, an example of the evaluation value calculation method is the following processing.
For example, let us consider a case where the reproduction unit group currently considered is (abc) and the detection ratios (values) of a, b, and c and the weighting coefficients are as shown in Table 5 below according to the above equation (1).

Here, 100 is multiplied for the sake of convenience because the ratio (%) is taken into consideration. However, as described above, the scale of the evaluation value can be subjected to predetermined evaluation processing, and there is no problem in predetermined arithmetic processing. Since there is no problem in processing such as overflow, there may be a case where the ratio need not be considered.
(Modification of playback unit group in rule 2 processing)
Here, a case is considered in which a reproduction unit group in which a plurality of “meaning” reproduction units are connected in rule processing 2 is defined as one semantic group unit, and a plurality of semantic group units are connected.

  In the rule 1 process, one playback unit was considered. It was to find out from the feature data a playback unit that would be more likely to detect "meaning".

  Further developing this idea, it is also possible to detect a section in which the units are connected by using the unit of playback units, that is, the units having the meaning connected in the rule 2 process as one unit.

  For example, assuming that (aabb) is Ga1, a connection such as (Ga1Ga1) can be considered. In this case, considering the evaluation value of Ga1, it is conceivable to perform processing similar to the above rule 1.

  As an evaluation value calculation method in that case, for example, an average of sums of evaluation values of the reproduction units of the respective meanings, an average of products of the evaluation values of the reproduction units of the respective meanings, or the like can be considered.

For example, when the evaluation value of a is 80 and the evaluation value of b is 60, the evaluation value of Ga1 is
(80 + 80 + 60 + 60) / 4 = 70
Thus, 70 can be considered as the evaluation value.

In the case of rule 3 processing Normally, as shown in FIG. 15A, it is considered that the processing up to the above rule 2 processing is sufficient. However, when feature data is provided for a plurality of programs, for example, time is set for each program. In the case where it is desired to perform a weighting process, a rule 3 process may be provided as a rule process as shown in FIG.

  As an example, FIG. 6C shows an example in which weighting and time correction are performed on news programs (news) and sports programs (sports).

  In the example shown in FIG. 6C, the news program is weighted 100%, and the start point Ps (ts, s4), the change point P1 (t1, s4), and the end point Pe (te, s3) are used as time correction functions. ), Weighting 70% for sports programs, and starting points Ps (ts, s4), change points P1 (t1, s4), end points Pe (te, s3) as time correction functions To correct.

  The processing content described in FIG. 3 will be further described with reference to FIG.

  As shown in FIG. 16A, according to the rule 1 process, each scene is subjected to some meaning process based on various predetermined feature data.

  Here, an evaluation value is set to each scene given by rule 2 by a predetermined process as shown in FIG.

  For example, in the summary playback mode, when playback is performed at a time t1 desired by the user, a scene (image) having the highest evaluation value is selected from the above scenes, and a scene with a high evaluation value is as close to t1 as possible. The position information is set so that the selected section is reproduced.

  The set position information is stored in a predetermined data memory, and when performing reproduction control, the position information is read and reproduction of a predetermined section is performed.

  Then, predetermined digest reproduction (digest reproduction) is performed by sequentially reproducing each section (skip reproduction).

  In the example shown in FIG. 16C, assuming that the total recording time is 60 minutes, for example, and that summary playback is to be performed in 15 minutes, a PU with an evaluation value of 70 or more is selected, and is slightly less than 15 minutes. If not, the section of PUn + 8 with an evaluation value of 60 is selected, and processing is performed as close as possible to the desired reproduction time of 15 minutes.

  In this way, the predetermined PU section having a large evaluation value is selected, and the PU section is selected so as to be as close as possible to the predetermined reproduction time.

In order that the reproduction time T is within a predetermined allowable range tc with respect to the desired reproduction time Tm,
Tm-tc <T <Tm + tc
Based on the evaluation value, a predetermined PU section is selected.

  Further, as shown in FIG. 16D, for example, a predetermined position (chapter) is set at the beginning (or the vicinity thereof) of the above-signified high evaluation value section or at the end (or the vicinity thereof) of the high evaluation value section. ) Can be used to perform predetermined operations such as editing processing of the section, skip playback pause processing, repeated playback processing, and the like.

(2) Block Configuration Example Here, for the sake of simplicity, it is assumed that the recorded audio / video information signal is broadcast program data and is subjected to predetermined band compression processing by MPEG (Moving Picture Export Group). In addition, the case where wavelet transformation, fractal analysis signal processing, etc. are used as other band compression signal processing is also considered. For example, in the following description, the DCT coefficient of image data corresponds to an analysis coefficient or the like in multi-resolution analysis in the case of wavelet transform, and the same signal processing may be performed.

2.1 Block configuration example 1
FIG. 17 shows an example of the overall block configuration of a recording / reproducing apparatus 30 to which the present invention is applied.

  Here, for the sake of simplicity, it is assumed that a television broadcast is received and the received broadcast program is recorded.

2.1.1 Recording Signal Processing System In this recording / reproducing apparatus 30, a predetermined broadcast program is received by the receiving antenna system 1 and the receiving system 2, and an audio signal is received by the audio A / D conversion processing system 3 at a predetermined sampling frequency, Predetermined A / D conversion signal processing is performed with a predetermined number of quantization bits, and then input to the speech encoder processing system 4.

  In the audio encoder processing system 4, signal processing is performed by a predetermined band compression method such as MPEG audio or AC3 audio (Dolby AC3 or Audio Code number 3).

  Similarly, the video signal is subjected to predetermined A / D conversion signal processing at a predetermined sampling frequency and a predetermined number of quantization bits in the video A / D conversion processing system 8, and then input to the image encoder processing system 9. .

  The image encoder processing system 9 performs signal processing using a predetermined band compression method such as MPEG video or wavelet transform.

  Audio data and image data processed by the audio encoder processing system 4 and the image encoder processing system 9 are input to the recording processing system 6 via the multiplexing processing system 5.

  In order to perform feature extraction of the audio signal, a part of the signal input to the audio encoder processing system 4 or a part of the signal in the process of the signal processing in the predetermined encoder signal processing is input to the feature extraction system 10.

  Here, in the recording / reproducing apparatus 30 shown in FIG. 17, a signal is input from the audio encoder processing system 4 to the feature extraction system 10 as a part of a signal input to the audio encoder processing system 4. It may be inputted to the feature extraction system 10 as well as being inputted to the encoder processing system 4.

  Similarly, in order to perform the feature extraction of the video (image) signal, a part of the signal input to the video encoder processing system 9 or a part of the signal in the signal processing process in the predetermined encoder signal processing is transferred to the feature extraction system 10. Entered.

  In the recording / reproducing apparatus 30 shown in FIG. 17, a signal is input from the video encoder processing system 9 to the feature extraction system 10 as part of a signal input to the video encoder processing system 9. Both may be input to the encoder processing system 9 and input to the feature extraction system 10.

  In the recording mode, the feature data is sequentially detected for each predetermined section, and is recorded in a predetermined recording area of the predetermined recording medium 7 together with the image / sound data subjected to the predetermined encoder processing.

  Playlist data generation (9) or chapter data generation (11) for performing predetermined summary playback (digest playback) from the feature extraction data is performed by the playlist / chapter generation system 19 with predetermined signal processing.

  Here, it is conceivable that playlist data and chapter data are generated by the following signal processing process (processing a or processing b).

  (Processing a) After the feature extraction data is accumulated in a predetermined memory area of a predetermined memory system or system controller system, a predetermined playlist data generation process and a predetermined chapter data generation process are performed.

  (Process b) Each time the predetermined feature extraction process is performed on the recording medium 7 for recording the image / audio data, the characteristic data is sequentially recorded, and after the predetermined amount of data is recorded, the data is reproduced to obtain the predetermined playlist data, the predetermined Generate chapter data.

  In the case of the above (Processing a), for example, when recording a broadcast program having a predetermined time length t, when the recording of the time length t is completed, all the predetermined feature extraction data in the broadcast program are accumulated. At this point in time, it is possible to perform playlist data generation processing that determines where the key frame corresponding to the predetermined summary playback time td is within the time length t. That is, the feature data processed for this time length t is accumulated (stored or recorded) in a predetermined memory area of the memory system or system controller system.

  In the case of (Processing b), as in the case of (Processing a), after recording for a predetermined time length t, it is detected (detected) that the recording for a predetermined time t has been completed, and predetermined feature extraction data is reproduced and stored in a predetermined manner. The playlist data generation process corresponding to the summary playback time td is started.

  When the playlist data generation processing is completed, it is ready to perform a predetermined summary playback operation, and a predetermined summary playback (digest playback) can be performed using this playlist data.

  Since the playlist data has already been generated for the predetermined feature extraction data, it may be possible to perform signal processing to delete the playlist data if it is no longer generated. For example, the data generation may be performed again, so that the feature extraction data may be recorded and left as it is.

  As for the feature extraction data, after the feature data is accumulated in a predetermined section via the system controller system 20, the playlist / chapter generation system 19 generates predetermined playlist data for digest playback (summary playback).

  The generated playlist data is recorded in a predetermined recording area of the recording medium 7 after being subjected to a predetermined recording process in the recording processing system 6.

  Here, the playlist data is composed of data that is a pair of playback start point information and playback end point information for each predetermined playback section for skip playback of a predetermined recorded section. It consists of a data pair of a playback start frame number and a playback end frame number.

  The playlist data is used for the process of performing digest playback (summary playback) by skip-playing a predetermined required section in the recorded program. Therefore, in addition to the frame data as described above, Code data or time stamp data such as PTS (Presentation Time Stamp) or DTS (Decode Time Stamp) in MPEG may be used.

  In the recording mode for recording image / audio information data such as a broadcast program as described above, the playlist data is subjected to a feature generation process in a playback mode, which will be described later, in addition to performing a predetermined generation process after recording a predetermined program. Predetermined processing may be performed using data.

  In FIG. 17, for example, assuming that an image and audio data that have been subjected to a predetermined encoding process such as MPEG are recorded, it is not necessary to perform the encoding process in the audio encoding processing system 4 and the image encoding processing system 9; It is conceivable that the data is input to the multiplexing processing system 5, recorded by the recording processing system 6 and recorded on a recording medium.

  Here, it can be detected by the system control system 20 whether the digital image and audio data are directly inputted and recorded, or whether the analog signal is inputted by the receiving system 2 and recorded after a predetermined encoding process, In this way, depending on the input system difference, it is determined whether the predetermined image and audio feature data extraction processing is automatically performed in the recording mode or after the recording is completed, or digital image and audio data are input. In this case, it is conceivable that the predetermined data structure analysis process is not performed because the data does not pass through the predetermined encoding processing system, and that the recording is performed after the recording is completed.

  In the recording mode, the analog input system or the digital input system can be set by a user's predetermined operation via the user input I / F system 21.

  In FIG. 17, a signal from the audio encoder processing system 4 or the audio A / D conversion processing system 3, the video encoder processing system 9 or the image A / D conversion processing system 8, and a digital image and audio data subjected to predetermined encoding processing. Can be automatically detected by directly detecting by the system controller system 20.

  If predetermined encoded digital data is detected and no data is detected by the audio encoder system 4 or the audio A / D conversion processing system 3, the video encoder system 9 or the image A / D conversion processing system 8, the predetermined encoding process is performed. It can be determined that digital image and audio data are input.

  Data from the audio encoder system 4 or the audio A / D conversion processing system 3, the video encoder system 9 or the image A / D conversion processing system 8 is detected by the system controller system 20 without detecting the predetermined encoded digital data. Can be determined as analog input.

  When both analog input and encoded digital data are detected, for example, it is conceivable to perform a predetermined recording process with an analog input signal from the receiving system 2 as an initial setting (default setting).

  Since the feature extraction process uses, for example, DCT data of an image or the like, when a predetermined encoding process is performed, the DCT process performed for a normal recording process can be used as the feature extraction process. In the case of audio, considering the use of subband processing data in the predetermined encoding process, when the predetermined encoding process is performed, the subband process performed for the normal recording process is also used as the feature extraction process. be able to.

  However, as described above, when the encoded digital data is directly input, it is not necessary to perform the encoding process. Therefore, it is necessary to analyze the data and extract data such as DCT, which increases the processing load. Will occur.

  Therefore, it may be considered to perform the feature extraction process after the recording is completed as necessary. In addition, as a case where the feature extraction processing is performed after the recording is finished, even in the case of the analog input, it may be automatically performed after the predetermined recording is finished depending on the load of the signal processing system.

  For example, as shown in FIG. 20, feature extraction processing can also be performed by software processing. Depending on the performance of the system controller system, it cannot be performed simultaneously with each predetermined signal processing in the recording mode, and after the predetermined recording processing ends. It is possible to do it. The system controller system 20 may be composed of a CPU, a DSP (digital signal processor), and other various processors. However, the higher the performance, the higher the cost, so the feature extraction processing is recorded according to the processing capability as described above. It is also possible to decide whether to perform it simultaneously with processing or after processing.

  After completion of the predetermined recording mode in which the feature extraction processing is performed, for example, the predetermined feature extraction processing is performed after the end of a predetermined timer recording operation or at night when it can be assumed that the user is not normally operating the device. And so on. In that case, for example, it is also conceivable that the time at which the apparatus is operating is stored in a predetermined memory means in the system controller system 20, and the time for feature extraction processing is automatically set appropriately by a predetermined learning process. It is done.

  In addition, when the system controller system 20 detects a time during which normal operation such as recording / reproduction is not performed, the predetermined feature extraction process may be performed while the system controller 20 is not operating. In this case, it is assumed that not all of the predetermined data is processed, but the location in the middle of processing is stored in a predetermined memory means in the system controller system 20 to detect that the apparatus is not operating normally such as recording and reproduction. Then, if it is determined that there is a time that can be processed, it may be possible to perform predetermined signal processing from the middle of the process.

2.1.2 Playback side processing (normal playback mode operation)
Next, playback signal processing in the recording / playback apparatus 30 shown in FIG. 17 will be described.

  First, the operation in the normal playback mode will be described.

  When the normal reproduction mode is set by the user input I / F system 21, predetermined recording data such as predetermined image / audio information data and feature extraction data is reproduced from the recording medium 7, and predetermined reproduction processing is performed in the reproduction processing system 12. Is made.

  The reproduced predetermined data is separated into predetermined data in the reproduction data separation processing system 13, and the audio data is input to the audio decoding processing system 14 and corresponds to a signal processing method in which band compression signal processing is performed at the time of recording. A predetermined decoding process is performed, and thereafter, the signal is input to the audio D / A processing system 15 and subjected to a D / A conversion process, and then output as an audio signal.

  Further, the image (video) data subjected to the predetermined classification processing is subjected to a predetermined decoding process corresponding to a signal processing method in which a band compression signal process is performed at the time of recording in the video decoding processing system 16, and then the video D / A processing system 17 is subjected to D / A conversion processing and output as a video signal.

(Summary playback (digest playback) mode)
When considering the summary reproduction mode, the signal processing method differs depending on whether the feature extraction data and the playlist data are recorded on the recording medium together with the image and sound data.

  It can be arranged as shown in FIG. 18 whether or not the collection data and the playlist data are recorded on the recording medium.

  First, in the case where playlist data (playlist data file) and chapter data corresponding to the cases (a) and (b) shown in FIG. 18 can be reproduced, that is, the playlist data and chapter data are stored in a predetermined recording medium (predetermined data). A case will be described in which recording is performed on the recording medium) and playback in the summary playback mode or thumbnail display of a predetermined chapter image in the chapter display mode will be described.

  That is, a case where the user selects the summary playback mode or the predetermined chapter mode will be described.

  When a command for performing a predetermined summary playback (digest playback) mode operation is input to the system controller system 20 via the user input I / F system 21 by user input, the playback data separation processing system 13 performs predetermined data separation. When feature extraction data, parameter data, playlist data, chapter data, and the like are recorded, predetermined separated feature extraction data, predetermined parameter data, predetermined playlist data, chapter data, etc. Is input to the system controller system 20.

  When the reproduction data separation processing system 13 cannot separate the feature extraction data, parameter data, playlist data, and chapter data, each of the above data is not input to the system controller system 20. The system controller system 20 has a function of determining whether or not the predetermined feature extraction data, playlist data, predetermined chapter data, parameter data, and the like are recorded on the predetermined recording medium 7.

  The playlist data is composed of playback start information data and playback end information data of a predetermined number of playback sections in order to perform predetermined summary playback.

  The chapter data includes the beginning or the vicinity of the predetermined feature section, the end or the vicinity of the predetermined feature section, the beginning or the vicinity of the section other than the feature section connected to the feature section, or the section other than the feature section. It consists of position information at the end or its vicinity.

  The system controller system 20 performs a digest playback operation by performing skip playback according to the skip playback start data information and skip playback end data information of the playlist data detected for playback.

  In addition, a predetermined display process is performed by the display processing system 27 using the predetermined chapter data as an image at a chapter point or in the vicinity thereof as a predetermined thumbnail image, and a predetermined image display is performed.

  Next, when play list data (play list data file) and chapter data corresponding to the cases (c) and (d) shown in FIG. 18 cannot be reproduced, that is, play list data and chapter data are stored in a recording medium or a storage medium. A case in which a series of chapter related processing such as displaying a predetermined thumbnail at a predetermined thumbnail mode and chapter playback cannot be performed in the predetermined chapter mode when it is not recorded (stored) and cannot be reproduced in the summary playback mode will be described.

  Reproduce image / audio data from other recording media, for example, when the recording medium 25 is DVD software and reproduced by the recording medium processing system 26 and the reproduction processing system 12 instead of the image / audio data received from the broadcast program described above. The case described above, or the case where image / audio data that has not undergone feature extraction are reproduced corresponds to the processing described here.

  If playlist data or chapter data has not been generated and playback detection cannot be performed, or if it is desired to regenerate the playlist data and chapter data that have been detected for playback, from the predetermined feature extraction data and parameter data that have been detected for playback. Summary play list data and chapter data for a predetermined chapter related mode can be generated.

  In the case shown in (c) of FIG. 26, that is, when feature extraction processing is performed at the time of recording and feature data can be reproduced, the reproduction processing system 12 or reproduction data separation in the recording / reproducing apparatus 30 shown in FIG. The playlist data or the predetermined feature extraction data is input from the processing system 13 to the playlist / chapter generation processing system 19 to generate predetermined playlist data or predetermined chapter data.

  In the case of the operation described here, when the user performs a command for summary playback mode, a predetermined display indicating that there is no playlist data as shown in FIG. 19 may be performed on the display system 27. .

  The generated playlist data is input to the system controller system 20.

  The system controller system 20 controls the reproduction control system 18 so as to sequentially reproduce (skip reproduction) a predetermined reproduction section based on the playlist data in accordance with a predetermined summary reproduction time input by the user, and recording is performed by the control. The reproduction of the medium 7 is controlled.

  The generated chapter data is input to the system controller system 20.

  The system controller system 20 displays an image thumbnail at a predetermined chapter point based on the chapter data, editing processing such as chapter point cut and connection, and the user-selected chapter point in accordance with a predetermined chapter-related operation mode by user input. The playback control system 18 is controlled so that a predetermined chapter-related operation such as skip playback can be performed, and the playback control of the recording medium 7 is performed by the control, and the display processing system 27 is controlled via the system controller system 20. .

  As described above, for example, when an external recording medium such as a DVD is summarized and reproduced as the recording medium 25, it can be performed by the same signal processing as described above, and the recording medium processing system 26 is controlled by the reproduction control system 18, The predetermined summary reproduction process as described above is performed.

  In addition, a series of operations related to a predetermined chapter such as editing processing (editing operation) using the chapter data, skip reproduction between predetermined chapter points (or the vicinity thereof), and thumbnail image display of the chapter points (or the vicinity thereof) are described above. The recording signal processing system 26 is controlled by the reproduction control system 18 to perform the predetermined signal processing as described above.

  Furthermore, (d) shown in FIG. 16, that is, a case where the feature extraction data cannot be reproduced will be described.

  In the above example, the case where playlist data and chapter data are generated from the feature extraction data has been described. For example, when considering a case where an external recording medium 25 recorded by another user is copied to the recording medium A26, the feature is considered. It can be assumed that the extracted data cannot be reproduced.

  Consider a case where video and audio data such as a broadcast program is recorded on the recording medium 7 but feature extraction data is not recorded and cannot be reproduced.

  In the case of the operation described here, when the user performs a command of the summary playback mode or the above-mentioned predetermined chapter related operation mode, a predetermined display indicating that there is no feature data as shown in FIG. You may make it do.

  In the case of reproducing the audio / video data from the recording medium A7 in the summary reproduction mode in this case, the data reproduced by the reproduction processing system 12 is input to the reproduction data separation processing system 13, and a predetermined band compression method is used at the time of the separated recording. The image data and audio data processed in the above are input to the feature extraction processing system 10, and DCT DC coefficients, AC coefficients, motion vectors (motion vectors), which are image characteristic data, audio power detection, etc., which are audio characteristic data, etc. Various predetermined characteristic data detection processes are performed.

  In the feature extraction processing system 10, predetermined telop feature data (telop section determination data), person feature data, and other image feature data (image feature section determination data) are further determined based on the above-described various image / audio characteristic data and predetermined parameter data. ) And speaker voice feature data (speaker voice determination data), applause cheer feature data (applause cheer determination data) and other voice feature data (speech feature section determination data).

  The various image feature extraction data (image feature data) and audio feature extraction data (audio feature data) are input to the system controller system 20, and the predetermined feature extraction processing is performed for all of the predetermined program or the predetermined image / audio section. When finished, it is determined that the feature extraction process is finished.

  Here, when the feature extraction processing is completed, a signal indicating that the predetermined signal processing is completed is input from the system controller system 20 to the display processing system 27, and a predetermined display as shown in FIG. You may make it do.

  Next, processing for generating predetermined playlist data and chapter data from the feature extraction data will be described.

  The various feature extraction data are stored in the memory system 11 for each predetermined feature extraction section. When all the predetermined feature data have been processed, they are input to the playlist / chapter generation processing system 19 and predetermined playlist data Alternatively, chapter data is generated.

  Here, the feature extraction processing data of a predetermined section may be sequentially input directly from the feature extraction processing system 10 to the playlist / chapter generation system 19, and as described above, all predetermined sections or predetermined sections may be input. When the broadcast program feature extraction processing is completed, the playlist / chapter generation system 19 may perform the predetermined playlist data or chapter data generation processing as described above by a predetermined signal from the system controller system 20.

  It is also conceivable to perform signal processing so that the processed feature data from the feature extraction processing system is input to the playlist / chapter generation processing system 19 via the system controller system 20.

  When predetermined playlist data or chapter data is generated in the playlist / chapter generation processing system 19, a signal indicating that the predetermined processing has been completed is input to the system controller system 20, and according to a desired summary time It is possible to perform summary playback or a predetermined chapter related operation using predetermined chapter data.

  In this case, as shown in FIG. 19, a predetermined display indicating that playlist data or chapter data has been generated is performed, a summary playback mode, a chapter-related predetermined operation mode, or the like is displayed on the display system. 27 may be performed.

  When a user performs summary playback, for example, if the recorded broadcast program is 1 hour, and the user wants to play it in 30 minutes or 20 minutes, it does not know the summary playback time desired by the user. It can be considered that playlist data corresponding to several summarization times is generated in advance according to the total time length of all sections in which the features of image and sound data such as recorded broadcast programs are extracted.

  For example, if the recording time of a broadcast program for feature extraction is 1 hour, each playlist data for summary playback for 40 minutes, 30 minutes, and 20 minutes is generated. By generating playlist data in this way, when a summary time is selected by user input from the remote controller 22 or the like, a summary reproduction operation corresponding to a predetermined summary time can be immediately performed.

  The case of reproducing the recording medium 25 is the same as the case of reproducing the recording medium A7, and the recording medium processing system 26 detects the recording medium 25, the reproduction processing system 12 performs reproduction signal processing, and the reproduction data separation processing system. In step 13, predetermined image / audio data is separated. Subsequent signal processing is the same as in the case of the recording medium 7 described above, and is therefore omitted.

  Here, the control program for executing the above-described series of processing is capable of executing various functions by installing a computer incorporated in dedicated hardware or various programs, for example, a general-purpose personal computer. Installed from a recording medium on a computer or the like.

  This recording medium is not only a hard disk in which a control program is recorded, but also a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor in which the program is recorded, which is distributed to provide a program to a user separately from a computer Consists of package media consisting of memory.

2.2 Block configuration example 2
A modification of the recording / reproducing apparatus 30 shown in FIG. 17 will be described with reference to FIG.

  Since the process of signal processing for recording / reproducing broadcast programs is the same as that of the recording / reproducing apparatus 30 shown in FIG. 17, a description will be given of parts different in signal processing from the recording / reproducing apparatus 30 shown in FIG. 17.

2.2.1 Recording-side signal processing The recording / reproducing apparatus 30A shown in FIG. 20 differs from the above-described recording / reproducing apparatus 30 in that a series of signal processing for performing feature extraction processing in the recording mode is performed by software in the system controller 20 system. To do.

  Further, in the recording / reproducing apparatus 30A, predetermined software is downloaded by the network system 24, and feature extraction processing, playlist processing (chapter generation processing (reproduction section, reproduction time position information generation) by software processing as described below. It is also possible to perform processing)).

  In the case of downloading by software, for example, when there is an apparatus that is not equipped with the processing of the present invention at the beginning, there is an advantage that the present invention can be applied in software later, such as time for manufacturing and sales. If it is not in time, the design and manufacturing side can provide the user with both a system with a simple configuration to which the present invention is not applied and a system to which the present invention is applied later.

  On the user side, the present invention can be applied by software processing after purchasing a system having a simple configuration to which the present invention is not applied. Therefore, there is an advantage that functions can be added later.

  In addition, there is an advantage that can be handled by downloading and upgrading the software when the processing system is modified or improved.

  When the present invention is installed by software download, a user connects to a predetermined Internet site via a network system 24 with a predetermined operation system (such as the remote control 22), and operates the software of the present invention by operation using the predetermined operation system. to download.

  The downloaded software of the present invention is subjected to predetermined decompression processing, installation processing, and the like in the system controller system 20, and will be described later, including feature extraction processing, playlist processing, chapter processing, and other predetermined processing of the present invention. Equipped with functions.

  By using a microprocessor (MPU or CPU) having a predetermined performance as the system controller system 20, it is conceivable to perform the above-described predetermined feature extraction process simultaneously with the predetermined recording process.

  It is conceivable that the memory system 11 also uses a predetermined data storage memory provided in the system controller system 20.

  Here, as described above, as predetermined recording processing, when performing band compression of predetermined video and audio, an MPU or CPU or DSP (digital signal processor) having the predetermined performance as described above is used. It is also possible to perform the predetermined feature extraction processing, playlist generation processing, and the like with the same MPU or CPU or DSP that performs this band compression processing.

2.2.2 Playback-side signal processing The recording / reproducing apparatus 30A shown in FIG. 20 differs from the recording / reproducing apparatus 30 described above in that the feature data cannot be detected in the reproduction mode as described above, and the feature extraction process is performed. When performing, a series of signal processing is performed by software in the system controller 20 system.

  By using a microprocessor (MPU or CPU) having a predetermined performance as the system controller system 20, it is conceivable to perform the above-described predetermined feature extraction process simultaneously with the predetermined recording process.

  It is conceivable that the memory system 11 also uses a predetermined data storage memory provided in the system controller system 20.

(3) Feature Extraction Processing Next, each signal processing of audio system feature extraction processing and video (image) system feature extraction processing will be described.
3.1 Audio Feature Extraction Processing In the audio feature extraction processing system, MPEG video / audio stream data is input to the stream separation system 100 and the separated audio data is input to the audio data decoding system 101 as shown in FIG. The inputted decoding process is performed.

  The decoded audio data (audio signal) is input to the level processing system 102, the data counter system 103, and the data buffer system 104. The level processing system 102 calculates the average power (or average level) Pav in a predetermined section of the audio data. In order to perform the calculation, the data is converted into an absolute value, and the voice data integration process 105 performs the integration process until the data counter system 103 measures a predetermined number of sample data.

  Here, the average power Pav can be obtained by the calculation of the following equation (32), where the value (level) of the audio data is Ad (n).

  As the predetermined interval for calculating the average level, for example, about 0.01 sec (10 msec) to 1 sec can be considered. For example, if the sampling frequency Fs is Fs = 48 KHz, the integration calculation of 480 to 48000 samples is performed, and the average is obtained with the number of samples Sm. Processing is performed to obtain an average level (average power) Pav.

  The data Pav output from the sound data integration processing system 105 is input to the determination processing system 106, and is compared with a predetermined threshold value Ath set by the threshold setting system 107, and a silence determination process is performed.

  Here, in setting the predetermined threshold value Ath in the threshold setting system 107, it is conceivable that Ath is set as a fixed value Ath0, but in addition to the fixed value Ath0, the fluctuation according to the average level of the predetermined voice section It is also conceivable to set the threshold value Athm.

  As the variation threshold value Athm, for example, an interval in which processing is considered is n, and an average level Pav (nk) in an interval (nk) before that is considered, and the following equation (33) is given. Can be considered.

For example, if t = 2
Athm = (Pav (n−1) + Pav (n−2)) / m (34) For example, it is conceivable that m is set from a range of approximately 2 to 20.

(Other audio feature extraction processing)
The predetermined audio data accumulated in the data buffer system 104 is input to the frequency analysis processing system 108, and a predetermined frequency analysis process is performed.

  Here, FFT (Fast Fourier Transform) or the like can be considered as the frequency analysis processing, and the number of predetermined analysis sample data of data from the data buffer system 104 is, for example, a predetermined power of 2 such as 512, 1024, 2048, etc. A predetermined analysis process is performed with a number.

  A signal (data) from the frequency analysis processing system 108 is input to the determination processing system 109, and a predetermined determination process is performed.

  Music (musical sound) discrimination processing can be performed from the continuity of a spectrum peak in a predetermined frequency band.

  For example, Japanese Unexamined Patent Application Publication No. 2002-116784 discloses such techniques.

  In the determination of the speaker voice, since there is a breathing interval in the human conversation voice waveform, a predetermined steep rising or falling interval is seen in the waveform, and by detecting the predetermined rising or falling interval, Predetermined signal processing can be performed.

  In this case, in the case of the music (musical sound) signal waveform, it is generally considered that the probability that the rising or falling section of the waveform appears is smaller than that in the case of the speaker voice. In consideration of the above characteristics (features), the attribute determination of the audio signal is comprehensively performed.

  When performing speech signal attribute determination based on the difference between the waveform characteristics (waveform characteristics) of the speaker voice signal and the waveform characteristics (waveform characteristics) of the music (musical sound) signal, the temporal physical characteristics of the waveform In addition to a method of performing predetermined determination signal processing after performing frequency analysis as described above (signal analysis and determination processing in the frequency domain), predetermined determination processing in the baseband region is performed. A method of performing (signal analysis and determination processing in the time domain) is also conceivable.

  Here, FIG. 22 shows a configuration example of an audio system feature extraction processing system in the case where signal attribute analysis is performed in the compression band without decoding audio signals (audio data).

  In the audio system feature extraction processing system shown in FIG. 22, a data stream that has undergone predetermined band compression signal processing, for example, image audio data such as MPEG, is input to the stream separation system 100 and separated into image data and audio data. The audio data is input to the stream data analysis system 110, signal analysis processing such as a predetermined sampling frequency, the number of quantization bits, and the like is performed, and the predetermined audio data is input to the subband analysis processing system 111.

  Predetermined subband analysis processing is performed in the subband analysis processing system 111, and predetermined signal processing similar to that described in the above equations (32) to (34) is performed on data in the predetermined subband band.

  That is, a predetermined integration process is performed until a predetermined number of sampling data is detected by the data count system 103 after being input to the audio data integration processing system 105, and then a predetermined threshold set by the threshold setting system 107. Based on the value, a predetermined silence determination process is performed in the determination processing system 106.

  In this silence determination process, it is conceivable to use a predetermined data band of approximately 3 KHz or less as a subband band in a band where a lot of energy is collected in consideration of the spectrum of audio data.

  In addition, although it has been described that determination processing of musical sound (music) and speaker voice can be performed by the above frequency analysis, the processing by the subband analysis system 111 is the same as when predetermined frequency analysis is performed by this signal processing system. Therefore, it is conceivable to perform signal processing for attribute determination by performing continuity determination processing for a predetermined spectrum peak as described above.

  In this case, the spectrum peak can be considered as the maximum data band in each predetermined subband band, and it is considered that the same signal processing as in the FFT analysis processing can be performed.

3.2 Image System Feature Next, video (image) system feature (feature) extraction processing will be described.

  In the video system feature extraction processing system, as shown in FIG. 23, the image data that has undergone the predetermined separation processing in the stream separation system is input to the stream data analysis system 200, and is subjected to predetermined detection such as rate detection, pixel number detection, and the like. Data analysis is performed, and DCT coefficient processing system 201 performs DCT calculation processing (inverse DCT calculation processing) such as DCT DC coefficient detection and AC coefficient detection. Based on the output of DCT coefficient processing 201, scene change is performed. Various processes are performed in the detection system 202, the color feature detection processing system 203, the similar image detection processing system 204, the person detection processing system 205, and the telop detection determination processing system 206. In the motion vector system 208, a predetermined motion vector detection process is performed. Done.

3.2.1 Scene Change Feature In the scene change detection system 202, for example, an average value of Y (luminance data), Cb, and Cr (color difference data) of DC coefficient data of DCT is divided into predetermined screen areas. It is conceivable to calculate a difference between frames or calculate a difference between fields for each region and compare a predetermined threshold value to detect a predetermined scene change.

  When there is no scene change, the difference data between frames (or fields) in each region is smaller than a predetermined threshold value, and when there is a scene change, a case where the difference data becomes larger than the threshold value can be detected.

  Here, for example, an area for dividing the effective screen into 16 as shown in FIG.

  The screen division method to be calculated is not limited to the case of FIG. 24, and it is conceivable to increase or decrease the number of divisions. However, if the number is too small, the accuracy of scene change detection becomes insensitive. Therefore, it is conceivable to set an appropriate predetermined number of divisions within a range of about 256 (16 × 16) or less.

3.2.2 Color Feature The color feature detection processing system 203 can detect a color feature from the average value of Y, Cb, and Cr data of a DCT DC coefficient in a predetermined region.

  As the predetermined area, for example, an area as shown in FIG. 25 can be considered.

  In FIG. 25, the effective screen is divided into four in the horizontal direction, and detection areas 1 to 4 are provided, and detection areas 5 to 8 are divided into four in the vertical direction. Each detection area is given an area ID, and the data of each detection area is identified by the area ID.

  Here, depending on the case, it is conceivable to provide detection regions 1 to 4 only in the horizontal direction or detection regions 5 to 8 only in the vertical direction.

  In addition to the area division as shown in FIG. 25, a grid-like division method such as 5 × 5 or 6 × 6 is also conceivable.

  For example, assuming the case of “sumo” as the program genre in a broadcast program, it can be assumed that the probability of “soil scene” is high when brown can be detected in the detection region 3 of FIG.

  When this color feature is combined with, for example, a voice attribute feature, it can be assumed that the probability of a “scene where the effort starts” is high from “soil scene” + “voice attribute other (or speaker voice)”. A scene section can be set as a key frame section.

  In this case, in the approach start scene, the sound level may increase due to the cheering of the audience, or data in the sound frequency band different from the normal state may be detected. The frequency domain data can also be considered as feature data.

3.2.3 Similar Scene (Similar Image) Features In the similar image detection processing system 204, a predetermined ID (identification number or identification symbol) is assigned to the image (scene) for each similar scene (similar image, similar video). In the process of giving (adding) (or assigning), a similar image (scene) is given (assigned) by the same ID. For example, Japanese Patent Laid-Open No. 2002-344872 discloses the technique.

  In the process of adding (applying), the ID is recorded in the memory means (data recording means) corresponding to the image (scene) or the position information (frame number, PTS, recording time, etc.) of the image (scene). Therefore, when performing an operation such as displaying or reproducing the image, the position information and ID of the image (scene) have a one-to-one correspondence, and the image (scene) itself and the position information are also Needless to say, since it corresponds one-to-one, for example, various predetermined operations using IDs can be performed, such as displaying images with the same ID, skipping playback of similar image classifications and image scenes with the same ID, etc. it can.

  As the feature data, the detection appearance ranks such as the first and second detection frequencies can be considered as mentioned for the scene ID.

  Further, as shown in FIG. 7, the ratio of the detection length of the detected ID such as the first rank and the second rank in the appearance order with respect to the PU section length can be considered.

  In this feature extraction process, for example, the screen is divided into a plurality of parts (for example, 25 parts), the DCT average DC coefficient of the area corresponding to each divided screen area is calculated, and the calculated average DC coefficient is a vector. As a component, an image (scene) corresponding to a place where a predetermined vector distance is smaller than a predetermined threshold is set as a similar image (similar scene), and the same predetermined ID (scene ID) is assigned to the similar image (similar scene). It is.

  For example, if an image (scene) having an initial value of 1 and less than the predetermined threshold is not detected as an ID, a value obtained by adding 1 to the maximum value of the ID is set as a new ID and the image (scene) is added. assign.

  In the present invention, as described above with reference to FIG. 5, as a method of using the feature data, a process of calculating the appearance frequency of the ID in a predetermined section and detecting the first to second frequencies is performed. A method is conceivable.

  For example, when a news program is assumed, an announcer scene frequently appears, or in a program genre where many similar scenes such as sumo and baseball can be expected, it can be used for effective processing. It is done.

  That is, it is considered that the probability that an announcer scene that can be assumed to have a high appearance frequency can be detected at the first and second appearance frequencies in a news program.

  FIG. 26 shows an outline for explaining a method for calculating the appearance frequency of IDs. For example, ID1 having the same ID is detected in four sections of sections f1 to f2, f3 to f4, f5 to f6, and f7 to f8. is made of.

  That is, it is considered that a similar scene appears in this section.

  As shown in FIG. 26, a section having the same ID in a predetermined section is counted as one, and the number of such sections is calculated.

  When similar scenes do not appear, they do not have the same ID. Therefore, it may be possible to calculate a predetermined frequency by calculating the number of ID continuity and discontinuity.

3.2.4 Person Features In the person detection processing system 205, as shown in FIG. 27, the screen area is divided, and a predetermined specific color in each area is detected to determine whether a person appears on the screen. It is possible.

  In the example shown in FIG. 27, four areas of areas 1 to 4 obtained by dividing the effective screen into 2 × 2 and five areas of the area 5 near the center of the screen are assumed.

  For example, in a news program, the probability that an announcer's face appears in the area 5 may be high.

  Further, assuming that a flip or telop and an announcer scene appear, the announcer's face may appear in the region 1 or the region 2. In that case, it can be assumed that a flip or a telop appears in the region 2 or the region 1.

  For example, when a white skin color is assumed as the specific color, it is known from experiments that the specific color is detected by the following conditional expression.

0.6 <Cb / Cr <0.9 to 0.97 (35) (0 ≦ Cb ≦ 255, 0 ≦ Cr ≦ 255) (36) As described below, in the region shown in FIG. A method different from the method by detecting a specific color is also conceivable.

  Here, for simplicity, the screen size is assumed to be 720 × 480.

(Process 1)
Detection conditions from color signals (Cb, Cr) (white skin color conditions) DCT coefficients Cb, in the DC component of Cr, the macroblock is 16 × 16, the x direction is 720/16 = 45, (0 to 44), The y direction is 480/16 = 30, and data points are detected every (0 to 29) under the determination condition shown in the following formula (3.2.3). In some cases, each of the x and y directions is compressed to ½ and processed as x direction 0 to 22 and y direction 0 to 14. Here, 0 ≦ Cb ≦ 255 and 0 ≦ Cr ≦ 255.

0.6 <Cb / Cr <0.9 to 0.97 (37) Here, for example, an 8-bit shift (128 times) and a determination condition such as the following (38) is also conceivable.

      77 << (Cb << 8 / Cr) <115 to 124 (38)

(Process 2)
Detection condition from luminance signal AC coefficient (contour detection condition of person, face, etc.) In the determination condition shown by the above-mentioned formulas (37) and (38), data larger than a predetermined threshold Ath is x, y Detect for each direction.

xh (x)> Ath (39) equation yh (y)> Ath (40) equation In some cases, covariance processing is performed from detected data.

  For example, as shown in FIG. 28, the black circles are detection points, for example, as follows.

xh (0) = 0 yh (0) = 0
xh (1) = 2 yh (1) = 0
xh (2) = 2 yh (2) = 3
...
...
...

(Process 3)
Considering the detection condition of the correctness of the size of the detected object, the detection data point that is larger than the predetermined threshold value Hth in the detection of xh (x), yh (y)
xh (x)> Hth (41)
yh (y)> Hth Data larger than a predetermined threshold number Lth in the x and y directions in equation (42)
xl (n)> Lth (43)
yl (m)> Lth The continuation length of the equation (44) is detected.

For example, in the case shown in FIG. 28, when Hth = 2
xh (x)> 2
yh (y)> 2
The continuation length of the part where the detection point of
xl (0) = 6
xl (1) = 1
yl (0) = 7
yl (1) = 2
For example, if Lth = 3, xl (0) and yl (0) are detected data in this processing.

(Process 4)
Consider the detection conditions for the correctness of the shape of a person's face. For each of the detected xl (n) and yl (m), data whose difference or ratio is within a predetermined range (0 to Dth or eth1 to eth2) is detected.

| Xl (n) −yl (m) | <Dth (45) or eth1 <xl (n) / yl (m) <eth2 (46) In the example of FIG. 28, xl (0), yl (0) Perform an operation on.

  Here, considering the shape of the human face, it is assumed that the face is approximated by a quadrangle, and the aspect ratio is calculated.

For example,
0.8 <xl (n) / yl (m) <1.5 Assuming the equation (47) as a detection condition,
yl (0) / xl (0) = 1.2 It can be determined that the object in the region of xl (0) and yl (0) in FIG.

  Here, it is conceivable to perform the bit shift processing as described in (3.2.4) above.

  In addition to the detection conditions of the above (Process 1) to (Process 4), it is also conceivable to perform detection data continuity determination as in the following (Process 5).

(Process 5)
Consider the condition of time continuity of the detected shape.
(Continuity determination method 5.1)
In some cases, it may be possible to determine the detection time continuity (detection stability) in the above (Process 1) to (Process 4).

For example, from the above equation (48), the detected value S (N) in the picture N is represented by the following equation: S (N) = yl (0) / xl (0) (49), and S (N + 1), S (N + 2), etc. It is also conceivable to detect and determine continuity.

For example,
0.8 <S (N) <1.5 (50) Equation 0.8 <S (N + 1) <1.5 (51) Equation 0.8 <S (N + 2) <1.5 Equation (52) and 3 It is also conceivable to determine that detection has been made when the picture is continued.

  Here, an I picture can be considered as a picture to be detected.

(Continuity determination method 5.2)
As another method, it is also possible to determine whether any one of the detection values in the above (Process 1) to (Process 3) can be continuously detected as N + 1, N + 2, N + 3 as some detection data in the picture N. Conceivable.

For example, consider the detection value Col (N) = (Cb << 8) / Cr (53) in frame N,
77 <Col (N) <115 (54) Equation 77 <Col (N + 1) <115 (55) Equation 77 <Col (N + 2) <115 As shown in Equation (56), three I pictures can be detected continuously. It may be possible to determine whether or not to proceed to the next detection process.

  It is also conceivable to determine the condition by calculating the average value of the detected N to (N + 2) picture data.

That is, the average value of the detected three picture data is AvCol,
AvCol = (Col (N) + Col (N + 1) + Col (N + 2)) / 3
(57) Formula 77 <AvCol <115 It is also conceivable to perform determination processing of formula (58).

(Continuity determination method 5.3)
According to the above formulas (39) and (40), detection values xh (N) (x) and yh (N) (y) are detected in the picture N, and detection continuity in pictures such as N + 1 and N + 2 is continued. It is also possible to see.

That is,
xh (N) (x)> Ath Formula (59)
xh (N + 1) (x)> Ath Formula (60)
xh (N + 2) (x)> Ath Formula (61)
yh (N) (y)> Ath Formula (62)
yh (N + 1) (y)> Ath Formula (63)
yh (N + 2) (y)> Ath It can be considered that it is determined whether or not three I pictures have been detected continuously, and the process proceeds to the next detection process.

  It is also conceivable to determine the condition by calculating the average value of the detected N to (N + 2) picture data.

That is, the average value of the detected three picture data is Avxh and Avyh,
Avxh = (xh (N) (x) + xh (N + 1) (x) + xh (N + 2) (x)) / 3
(65) Formula Avyh = (yh (N) (y) + yh (N + 1) (y) + yh (N + 2) (y)) / 3
(66) Formula
Avxh> Ath Formula (67)
Abyh> Ath (68) It is also conceivable to perform the determination processing.

(Continuity determination method 5.4)
According to the above equations (43) and (44), the detection values xl (N) (x) and yl (N) (y) in the picture N are used as detection values continuity of detection determination in pictures such as N + 1 and N + 2. It is also possible to see.

That is,
xl (N) (x)> Lth (69) Formula
xl (N + 1) (x)> Lth (70) Formula
xl (N + 2) (x)> Lth (71) Formula
yl (N) (y)> Lth (72) Formula
yl (N + 1) (y)> Lth Formula (73)
yl (N + 2) (y)> Lth It can be considered that it is determined whether or not 3I picture can be detected continuously, and the process proceeds to the next detection process, as in the equation (74).

  It is also conceivable to determine the condition by calculating the average value of the detected N to (N + 2) picture data.

That is, the average value of the detected three picture data is Avxl and Avyl,
Avxl = (xl (N) (x) + xl (N + 1) (x) + xl (N + 2) (x)) / 3
(75) Formula Avyl = (yl (N) (y) + yl (N + 1) (y) + yl (N + 2) (y)) / 3
(76) Formula
Avxl> Lth Formula (77)
It is also conceivable to perform determination processing of Avyl> Lth (78).

(Outline of basic processing method for detecting the number of people)
Here, the detection determination of the number of persons is considered.
(Number of people judgment method 5.1B)
For example, in the case of FIG. 29, it is assumed that two pieces of data xl (0) and xl (1) exceeding a predetermined threshold value in the x direction are detected and one piece of yl (0) is detected in the y direction. To do.

  Here, consider the data density of region 1 specified by xl (0) and yl (0) and region 2 specified by xl (1) and yl (0).

For region 1, all data points S1 of the region are
S1 = xl (0) × yl (0)
= 20 (79) Formula The number of data larger than the predetermined threshold is
Σxh (x) = 17 (80) Data density Δ1, that is, the number of data Δ1 per unit data point is
Δ1 = 0.85 (81) Here, when all the data in the region 1 is detected to be larger than the threshold value, the data density is Δ1 = 1. Therefore, a predetermined threshold value Mth is set,
Δ1> Mth (82) is determined.

Similarly, for region 2, all data points S2 of the region are
S2 = xl (1) × yl (0)
= 25 (83). The number of data larger than the predetermined threshold is
Σxh (x) = 21 (84) The data density Δ2 is
Δ2 = 0.84 (85).

Here, for example, the threshold value Mth is set to
Assuming that Mth = 0.80 (86), it is determined from Expressions (81) and (85) that the regions 1 and 2 satisfy the condition and the probability that a person is detected is high.

Here, in the x direction, considering the region St specified by xl (0) + Xl (1) and yl (0), the total number of data points is
(Xl (0) + xl (1)) × yl (0) = 45 (87). The number of detected data is Σxh (x) = 17 + 21
= 38 Equation (88) is obtained. Data density is Δ
Δ = 8.4 Equation (89).

Here, also for the region St
Since Δ> Mth (90), it is determined that the person is detected at the same position in the y direction in the area 1 and the area 2.

(Other people detection example 1 (when areas overlap))
Considering FIG. 30, one xl (0) is detected in the x direction and one yl (0) is detected in the y direction.

For the region R specified by xl (0) and yl (0), all data points Sr are
Sr = xl (0) × hl (0)
= 90 (91). The number of detected data is
Σxh (x) = 44 (92) The data density Δr is
Δr = 0.49 (93)

here,
Δr <Mth
Therefore, it cannot be determined that one person is detected in the region R.

Given the reciprocal of data density,
1 / Δr = 2.0
It is considered that there is a possibility that there are two objects, but the data density may be the same when the data exists in a sparse state as shown in FIG.

  For FIG. 30, consider the variance σ in the y direction.

The average value of yh (y) is yhav and the number of data m
m = yl (0) (94) Formula
σy = (Σ (yh (y) −yhav) ^ 2) / m
= 2.32 (95). For the x direction, the average value is xhav data number n.
n = xl (0) (96) Formula
σx = (Σ (xh (x) −xhav) ^ 2) / n
= 1.04 (97).

Next, regarding FIG. 37, similarly, the dispersion in the y direction and the x direction is
σy = 0.99 Equation (98)
σx = 0.64 (99).

  From the above results, it can be seen that the variance value of data is larger in FIG.

  Therefore, it is conceivable to set the predetermined threshold value Bth and the threshold values d1 and d2 corresponding to the number of detected objects for the variance value, determine the following conditions, and detect the number of detected objects.

σy> Bth (100) formula
σx> Bth (101)
d1 <1 / Δ <d2 (102) For example, in the example of FIG.
Bth = 2.0 Formula (103)
d1 = 1.8 Equation (104)
d2 = 2.3 It is also conceivable to determine by setting a threshold value as in the equation (105).

(Other detection example 2 (when regions are diagonally separated))
When considering FIG. 32, two of xl (0) and xl (1) are detected in the x direction, and two of yl (0) and yl (1) are detected in the y direction.

For the region R00 specified by xl (0) and yl (0), all data points S00 are
S00 = xl (0) × hl (0)
= 20 (106) Expression The number of detected data is
Σxh (x) = 17 (107) The data density Δr is
Δ00 = 0.85 (108) From the above (3.2.52)
Mth = 0.80
Because
Δ00> Mth (109) Formula, and it is determined that the probability that one person is detected in the region R00 is high.

Next, for the region Ra specified by xl (0) and (yl (0) + yl (1)), all data points Sa are Sa = xl (0) × (yl (0) + yl (1))
= 40 (110). The total number of detected data is from the equation (107), Σxh (x) = 17 (111), and the data density Δa is
Δa = 17/40
= 0.43 (112). This does not satisfy the threshold condition.

  That is, since xl (0) and (yl (0) + yl (1)) are now considered, if Δa is larger than a predetermined threshold, it is determined that the probability that two persons are detected is high. Is done.

  However, since Δa is equal to or smaller than a predetermined threshold value from the equation (112), it is determined that two persons are detected in the region specified by xl (0) and (yl (0) + yl (1)). It seems that it is more sure to determine that one person is detected.

  That is, it is determined that the probability that a person is detected is low in the areas specified as xl (0) and yl (1).

Similarly, for the region Rb specified by xl (1) and (yl (0) + yl (1)), the total number of data is expressed by Σxh (x) = 17 (113). The total number of data points Sb is
Sb = xl (1) (yl (0) + yl (1))
= 40 (114). The data density Δb is
Δb = 17/40
= 0.43 (115).

  From this equation (115), the probability that two persons are detected in the region Rb is low.

Here, the data density Δ10 of the region specified by xl (1) and yl (0) is the number of detected data.
Σxh (x) = 17
And the total number of data points is
xl (1) × yl (0) = 20
So,
Δ10 = 17/20
= 0.85 (116) Similarly for the data density Δ11 of the region specified by the expressions xl (1) and yl (1)
Δ11 = 0.85 (117).

  From the above formulas (115) to (117), the probability that a person is detected is low in either the area 10 or the area 11.

Next, consider the data density of the region Rc specified by (xl (0) + xl (1)) and yl (0). The number of detected data is
Σyh (y) = 17
The total number of data points is (xl (0) + xl (1)) × yl (0) = 40
Therefore, the data density Δc is
Δc = 17/40
= 0.43 (118) Since this is not more than the above-mentioned predetermined threshold value Mth, the probability that two persons are detected in the region Rc is low, and the above equations (109) and (115) to (117) As a result, a person is eventually detected in two areas: an area specified by xl (0) and yl (0) and an area specified by xl (1) and yl (1).

  The possibility of detecting the number of persons is also conceivable by the determination process as described above.

(Other human detection processing methods (method 2))
As another method, a method of sequentially determining whether a predetermined threshold condition is satisfied in the x direction (0 to 44) and the y direction (0 to 29) is also conceivable.

Assuming that the data series is d (x) (y), the data series that satisfies the conditions of the above formulas (37), (41), and (42) is detected.
d (x1) (y1), d (x2) (y1)
d (x1) (y2), d (x2) (y2)
As described above, if the detection is continuously performed in the x direction and the y direction, the size and the position of the detected object can be known at the same time.

  However, in this method, all the data is detected one by one and the continuity of the series data is determined, so that it seems that the calculation time is longer than that in the above (Method 1).

  In the case of performing this method, for example, it is conceivable that the data is compressed by half in the x direction and the y direction to reduce the number of data processing.

(Other human detection processing methods (method 3))
As another method similar to the above (Method 2), a person is approximated by a quadrangular shape, the size of the quadrangular shape is sequentially changed, and it is determined whether the data of the quadrangular region satisfies a predetermined condition. It is also possible.

  For example, consider (2 × 2), (3 × 3), and (4 × 4) quadrangles as shown in FIG.

  The quadrangular regions having different sizes as described above are sequentially moved one by one from the smaller quadrangles, and it is determined whether the data in the region satisfies the conditions. The same process is performed for the quadrangular size.

  Although the detection area and the number of detections are known at the time when the processing is completed for all the quadrangles, it is considered that the processing time is long as in the above (Method 2).

3.2.5 Telop Feature The telop detection determination processing system 206 detects an average value of DCT AC coefficients in a screen area as shown in FIG.

  A telop including character information of a predetermined size within a screen in a predetermined area has a relatively clear outline, and when a telop image appears in any area of FIG. It is considered that the AC coefficient can be detected, and thus the telop can be detected.

  In addition to the method of detecting the AC coefficient of DCT as described above, a method of detecting an edge in the baseband region (time domain signal) is also conceivable. For example, an edge is detected by a difference between frames of luminance data of an image. Is also possible.

  In addition, multi-resolution analysis is performed by wavelet transform, and the AC coefficient is calculated by calculating the average value of the region corresponding to FIG. 25 using data in a predetermined multi-resolution region including predetermined high-frequency component data. It is also conceivable to perform a signal similar to that used.

  A telop is not limited to a light-colored area for flipping, but is, for example, text information that appears at the bottom of a news video. The appearance area depends on the bottom, top, or left side of the screen, depending on the program genre. In the case of right side, etc., the possibility is great.

  It is also possible to combine the telop feature and the flip feature into a character feature.

3.2.6 Camera Features The camera feature determination processing system 209 is a feature related to camera operations such as zooming, panning, and the like. For example, as disclosed in Japanese Patent Application Laid-Open No. 2003-289881, It is conceivable to make a determination using a motion vector (motion vector) of a P picture.

  In addition, for example, Japanese Unexamined Patent Publication No. 2002-535894 discloses a technique related to camera characteristics.

(4) Playback unit (play unit) processing Here, digest playback (summary playback) is a predetermined signal processing using each feature data (feature extraction data) of audio system feature data and video system feature data by predetermined signal processing. Thus, it is considered that several important playback sections (key frame sections) are selected (selected) within a predetermined section and each section is sequentially subjected to skip playback operation.

  When performing skip playback, for example, when skipping in the middle of the speaker's voice section, even if there is not much discomfort when looking on the screen, some users may experience discomfort in the sense of hearing when the sound is interrupted in the middle Therefore, it is conceivable that a section below a predetermined sound level (volume) is set as a silent section, and a predetermined time point in the section is set as a skip time candidate.

  In addition, a video scene change can be considered as a topic break point in broadcast programs, movies, and other video reproductions. Therefore, a scene change point or its vicinity can be considered as a skip point candidate.

  From the above, it is possible to consider skip reproduction time points and skip reproduction intervals according to a predetermined silent period in a predetermined audio signal and a scene change time point of the predetermined video signal or a predetermined time point in the vicinity thereof.

  Here, from the above viewpoint, (for the sake of convenience, a predetermined playback unit (hereinafter referred to as playback unit or play unit Play Unit (or PU)) is set between ((skip playback time points)). Think about it.

  Predetermined image system feature data and predetermined audio system feature data in the playback unit (PU) thus set are subjected to predetermined processing, and predetermined summary playback (digest playback) is performed according to the video, audio feature data and summary playback time. ) A section is set, and predetermined summary playback is executed by performing skip playback in a predetermined summary playback mode.

  In addition to performing summary playback as described above, a chapter (or edit point or playback breakpoint) at the first (or nearby) or last (or nearby) time point of a PU set by predetermined signal processing. ) Can also be considered.

  That is, by setting the chapters as described above, the chapter points can be displayed as thumbnails by predetermined signal processing, and the user can perform operations such as editing while viewing the thumbnail display. .

  Next, an example of processing of a playback unit (play unit) (PU) will be described with reference to FIGS.

(In the case of a sound section (when the audio signal is above a certain level))
As in processing method 1 shown in FIG. 34 (A), when the voice segment is in the range of 10 to 20 seconds in a voiced segment of a predetermined average level or higher, the voice segment is 15 regardless of the scene change. The break closest to the second (predetermined silence detection time) is defined as a playback unit break.

  As in processing method 2 shown in FIG. 34 (B), when the voice section is continuously longer than 20 seconds and the scene change section is 20 seconds or less, the scene change break is closest to 15 seconds. The detection point is defined as a playback unit break.

  When the voice is continuously longer than 20 seconds and the scene change section is longer than 20 seconds as in the processing method 3 shown in FIG. 34C, the playback unit is 20 seconds regardless of the voice segment and the scene change. When it becomes, it is set as a delimiter at that time.

  When the attribute of the voice feature changes in the range of 10 to 20 seconds as in the processing method 4 shown in FIG. 34D, the attribute change point is set as the break point of the playback unit.

  When the CM (commercial) is detected as in the processing method 5 shown in FIG. 34E, the CM detection point is set as a break point of the playback unit.

  Here, a CM detection method will be described with reference to FIG.

  In general, the CM section length of a broadcast program is a predetermined time length (usually generally 15 seconds, 30 seconds or 60 seconds), and there is a scene change at the breakpoint (start and end time) of the CM. By detecting the predetermined time length and detecting a scene change, it is possible to detect a CM as shown in FIG.

(In case of silent section (when the average level of audio is below a certain level))
As in the processing method 6 shown in FIG. 35A, when the silent section (the section where the sound average level is not more than a predetermined value) is longer than 20 seconds and the scene change detection section length is 20 seconds or less, it is set to 15 seconds. The closest scene change detection point is set as the break point of the playback unit.

  When the silence interval is longer than 20 seconds and the scene change detection interval is longer than 20 seconds as in the processing method 7 shown in FIG. 35B, the start point of the playback unit is used regardless of the scene change detection point. A breakpoint is set at 20 seconds.

  In any of the above description of the playback unit processing, the initial value of the start point of the playback unit is the start time when the program (broadcast program) is recorded.

  Through the playback unit processing as described above, a predetermined playback unit corresponding to a predetermined audio feature and a predetermined video feature (scene change feature) can be played back.

(Example of playback unit generation processing system block configuration)
FIG. 37 shows a block configuration example of a processing system generated by the playback unit described above and a unitized feature data processing system which will be described later and in which feature data is inserted into the playback unit.

  Predetermined time point setting processing such as summary playback and chapter point setting is performed at the start point and end point of the playback unit, and thus processing is performed in association with the feature extraction data for each playback unit described above.

  That is, processing is performed to reflect each predetermined feature extraction data, audio feature extraction data, and video feature extraction data extracted for each predetermined section based on the section of the playback unit.

  Here, a block configuration example of FIG. 37 will be described.

  In the block configuration example shown in FIG. 37, silence determination information data is input to the time measurement system 301, a predetermined interval (time length) based on the playback unit processing described above is measured, and the processing output is the playback unit processing system. 302 is input.

  The playback unit processing system 302 also receives scene change determination information data and CM detection determination information data, and performs signal processing as described in the description of each processing method of the playback unit processing to generate a predetermined playback unit.

  Here, the CM detection system 304 receives silence feature detection information data, scene change feature information data, and channel information for determining whether the channel of the program on which the CM is broadcast is input. CM detection processing is performed by a signal processing method.

  The playback unit feature data processing system 303 receives voice feature data such as voice attribute information and silence information, and feature data such as scene change features, color features, similar image features, person features, telop features, and person features. Then, as will be described later, the process of inserting each feature extraction data into the reproduction unit is performed.

(5) Playback Unitized Feature Data Processing Next, feature data file processing will be described.

  The feature extraction data includes audio feature extraction data and video (image) feature extraction data.

  This feature data processing is data (data file) obtained as a result of performing the process of inserting the audio system and video system feature data extracted in the playback unit described above. Various feature data is predetermined for each playback unit. Recorded on the recording medium.

  Here, it is considered that feature data is recorded for each reproduction unit. However, each feature data obtained by detecting each feature data according to a predetermined detection section is recorded on a predetermined recording medium. It is also conceivable to perform processing on feature data according to the above.

  Feature extraction data is obtained by extracting a predetermined characteristic signal (characteristic data) from an audio signal (audio data) and an image (video) signal (image (video) data), and subjecting the extracted signal (data) to predetermined processing. By doing this, it can be considered as feature data indicating the characteristics of audio and images, but here, unless otherwise noted, a signal indicating the characteristics by performing predetermined processing from the characteristic data (characteristic signal) (Data) is also described as feature data (feature signal) or feature extraction data (feature extraction signal).

  The video (image) signal includes, as characteristic data from the MPEG stream, a luminance signal (Y signal) in an I picture, a DC signal of a DCT of a color signal (color difference signal) (Cb, Cr signal), a motion vector (motion of a B or P picture) Vector) data and AC coefficients of DCT are extracted, and from the extracted screen position information, predetermined threshold value, correlation calculation, etc., scene change feature (scn feature), camera operation feature (camera feature) (cam feature), Similar image features (similar scene features or scene ID features) (sid features), telop features (tlp features), color features (color features) (col features), person features (Person features), and the like can be considered.

  For example, an average level of an audio signal is calculated every about 20 ms as characteristic data processing, and the attribute (type) of the audio signal in a predetermined section, average power (average level), etc. are calculated from the calculated data and a predetermined threshold value. Voice features (seg features) can be considered.

  Here, speaker voice, music (musical sound), and other voices (cheers in sports programs can be assumed) can be considered as voice attributes.

5.1 Configuration of Feature Extraction Data File Configuration example 1 of the feature extraction data file shown in FIG. 38 is a video of the above-described audio system feature data, scn feature, cam feature, sid feature, tlp feature, col feature, Person feature, etc. This is an example in which system feature data is made into separate feature data files.

  Each feature data file is written in text format data or binary format data.

  These characteristic data are temporarily stored (recorded) in a predetermined recording medium (such as a semiconductor memory) as normal data in addition to file data to be recorded in a predetermined recording medium, and will be described later. It is also conceivable to read and use for a predetermined process such as summary list data generation or predetermined set time point generation (chapter point). The same applies to FIGS. 39 and 40 described below.

  In Example 2 shown in FIG. 39, all the above-mentioned audio system feature data is collected as one file in text format or binary format, and all the above-mentioned video system feature data is stored as one file in text format or binary format. It is an example in the case of putting it together.

  Example 3 shown in FIG. 40 is an example in the case where all the audio system feature data and all the video system feature data described above are collected as one file in a text format or a binary format.

  In this way, as a single file, the number of files is only one compared to the case of Example 1 in FIG. 38, so that it is easy to handle as a file, and further in binary format, the data size (file Size and file capacity) will be smaller and more efficient.

  Here, a case will be described in which feature data is written in a binary format when the feature data file is as shown in Example 3 of FIG.

  In addition, Example 3 in FIG. 40 is the same as Example 2 in FIG. 39, in which all audio system feature data is described in binary format and all video system feature data is described in binary format. Can think.

  From the above, the audio system feature data processing method (description method) in the following description of the feature data file can be applied to the audio system feature data in Example 2 of FIG. The (description method) can be applied to the video system feature data in Example 2 of FIG.

5.2 Hierarchical Structure of Feature Data FIG. 41 shows a hierarchical structure of feature data in units of playback units.

  In the following description, predetermined feature data processing in a predetermined predetermined processing unit (reproduction unit) in the present invention will be considered.

  As shown in FIG. 41, it is composed of feature data header information, program 1 feature data, program 2 feature data, and the like.

  As shown in FIG. 42, the characteristic data header information is composed of predetermined data such as the total recording time of the whole program such as the program 1 and the program 2, the recording start time, the recording end time, the number of programs (number of programs), and other information. ing.

  Next, program (program) feature data will be described using program 1 feature data as an example.

  As shown in FIG. 41, the program 1 feature data includes program 1 information, playback unit 1 information, playback unit 2 information, and the like.

  As shown in FIG. 42, the program 1 information includes predetermined data such as program recording time, program start time, end time, program genre (program genre), and other information.

  Next, the data structure of the playback unit will be described using the playback unit 1 information as an example.

  As shown in FIG. 41, the playback unit 1 information is composed of audio feature data and video feature data.

(Structure of voice system feature data)
As shown in FIG. 41, the audio system feature data is composed of sequence number information, start / end position information, audio attribute information, feature data, other information data, and the like.

(Configuration of video feature data)
As shown in FIG. 41, the video system feature data is composed of predetermined feature information data such as a scene change feature, a color feature, a similar image feature, a person feature, a telop feature, and a camera feature.

  In the following explanation of each feature data such as scene change feature, color feature, similar image feature, person feature, telop feature, camera feature, etc., the feature data of each item is recorded on a predetermined recording medium in every predetermined section (writing process) In addition to the processing to be performed, for example, a method of performing predetermined data processing so as to record (write processing) as predetermined data on a predetermined recording medium only when feature data of a predetermined threshold value or more is detected is also conceivable.

  As described above, when the predetermined data processing is performed only when feature data equal to or greater than the predetermined threshold is detected, the predetermined feature data is not written when the threshold data is smaller than the threshold value. When the above feature data is detected, a predetermined recording (writing) process is performed, and the number of feature data detected from the beginning can be known from the sequence number information described below.

(Scene change feature)
As shown in FIG. 43, it consists of sequence number information, start / end position information, feature data, and other data.

  Here, the sequence number information is information indicating the order in which scene changes have occurred from the beginning of 0, 1, 2, 3,... And the program (method program).

  The start / end position information is information data indicating the start / end positions of the scene changes in each order, and information data such as a frame (field) number, PTS, DTS, and time may be used.

(Color features)
As shown in FIG. 43, it includes sequence number information, information data for identifying a detection area, start / end position information data, feature data, and other data.

  Here, the sequence number information is information indicating the order of color feature detection from the beginning of 0, 1, 2, 3,... And its program (method program).

  The start / end position information is information data indicating the start / end position where the feature detection of each area is performed in the color feature detection in each order, and information data such as a frame (field) number, PTS, DTS, and time is used. Conceivable.

  For example, data such as RGB, Y, Cb, and Cr can be considered as the feature data.

(Similar image features)
As shown in FIG. 43, it consists of sequence number information, frequency information start / end position information, feature data, and other data.

  Here, the sequence number information is information indicating 0, 1, 2, 3,... And the order of similar image feature detection from the beginning of the program (method program).

  As the feature data, the DCT average DC coefficient of each divided area obtained by dividing the effective screen as described above into a predetermined number of areas (for example, 25 divisions) can be considered.

(Characteristics)
As shown in FIG. 43, it includes sequence number information, information data for identifying a detection area, start / end position information data, feature data, and other data.

Here, the sequence number information is information indicating 0, 1, 2, 3,... And the order of similar image feature detection from the beginning of the program (method program).
(Telop features)
As shown in FIG. 43, it includes sequence number information, information data for identifying a detection area, start / end position information data, feature data, and other data.

  Here, the sequence number information is information indicating the order of telop feature detection from the beginning of 0, 1, 2, 3,... And its program (method program).

(Camera features)
As shown in FIG. 43, it includes sequence number information, information data for identifying a detection area, start / end position information data, feature data, and other data.

  Here, the sequence number information is information indicating the order of camera feature detection from the beginning of 0, 1, 2, 3,... And its program (method program).

  Here, when a broadcast program is recorded, it can be considered to perform a feature extraction process and a feature data writing process (recording process) described here simultaneously with a predetermined recording process of the broadcast program. It can be considered that a predetermined feature extraction process is performed on a recorded broadcast program, a movie, a drama, and other image / audio software to generate a feature data file.

  As described above, PU and feature data can be considered for program 1, and when recording other programs 2, 3 and the like, PU and feature data are considered in the same manner as in program 1 described above. Can do.

(6) Playlist processing (summary playlist generation processing)
Next, the summary data processing for performing summary playback (digest playback) from the PU file (PU feature data file) generated by the above feature extraction processing will be described.

6.1 Summary Rule Processing In summary playback (digest playback) using feature data described in the present application, desired digest playback is performed by performing skip playback processing on the above-described predetermined playback section in units of PUs.

6.2 Predetermined Time Setting Process (Playlist File) Process Next, the playlist file will be described.

  This file contains predetermined data indicating which one of the PUs or PU joints (PU aggregates or PU concatenations) that have been given meaning according to the above-described feature data is to be selected for playback processing. It is data in which information is described according to a predetermined format.

  Here, in addition to the case where the data is recorded on a predetermined recording medium on which the image / sound data on which the feature extraction is based (recording process) is recorded, the data may be temporarily stored in a predetermined memory unit. .

  An example of the playlist file is shown in FIGS. 44 (A) and 44 (B).

  The vertical data series of (a) in Example 1 shown in (A) of FIG. 44 is data of the start position information of the playback section, and includes the frame number, time (time), and stream (compressed video and audio data). Predetermined information data such as PTS (presentation time stamp) or DTS (decode time stamp) is also conceivable.

  The vertical data series (b) in Example 1 shown in (A) of FIG. 44 is the data of the end position information of the playback section, and corresponds to the data (a) in Example 1 above, and corresponds to the frame number, time (time). ), Predetermined information data such as PTS (Presentation Time Stamp) or DTS (Decode Time Stamp) from a stream (compressed video / audio data) is also conceivable.

  The vertical data series of (c) in Example 1 shown in FIG. 44A is the importance of the playback unit (PU) or playback unit group (PU group).

  The vertical data series (d) in Example 1 shown in FIG. 44A is character data having the meaning defined or set by the summary rule.

  Example 2 shown in FIG. 44B describes semantic characters and evaluation values (importance) for all PU sections, and “1” and “0” are used to indicate predetermined points in time such as playback sections and chapter settings. This is an example when the identification data is provided.

  It can be seen that the start point and end point shown in (a) and (b) of Example 2 shown in (B) of FIG. 44 are continuous with the data of the next stage.

  For example, in Example 2 shown in FIG. 44B, the first start point 0 and the end point 229 are continuously connected to the next start point 230.

  The vertical data series (e) in Example 2 shown in (B) of FIG. 44 is flag information data indicating whether or not summary playback is performed. When “1”, playback is performed and when “0” is performed. Is a case where no reproduction is performed.

It is also conceivable that the first time point of “1” and the first time point of “0” are regarded as a predetermined time point set point (chapter point).
(7) Operation Flowchart FIG. 45 is an example of an operation flowchart of the present invention, which will be described.

  When the process is started, it is first determined in step S1 whether the recording mode or the reproduction mode is selected. If the recording mode is selected, the process proceeds to the recording process (R), and if the reproduction mode is selected, the process proceeds to step S2.

7.1 Reproduction Process Related Operation Flowchart (Example of Reproduction Process Operation Flowchart)
In the case of the reproduction mode, it is determined in step S2 whether the digest reproduction (digest reproduction) mode or the normal reproduction mode is selected, and in the case of the normal reproduction mode, the process proceeds to the normal reproduction process (P).

  In the case of the digest playback mode, a determination process is performed in step S3 to determine whether or not the predetermined feature extraction data is recorded on the predetermined recording medium or whether or not the predetermined feature extraction data is recorded in the predetermined recording area of the recording medium as the predetermined file data. Is done.

  If predetermined feature extraction data is detected in step S3, it is detected in step S4 whether predetermined playlist data (data file) is recorded in a predetermined recording area of a predetermined recording medium. If a (list file) is detected, predetermined playlist data is read and processed in step S5.

  If it is determined in step S3 that the predetermined feature extraction data is not detected, in step S8, the image / audio data (program, broadcast program) to be summarized and reproduced is read and predetermined feature extraction processing is performed, and step S9 is performed. In step S8, it is determined whether or not the process has been completed.

  If it is determined in step S9 that the predetermined feature extraction process has been completed, the process proceeds to step S6, and a predetermined playlist data generation process is performed.

  If it is determined in step S4 that the predetermined playlist data (file) is not detected, the predetermined feature extraction data recorded (or stored) in the predetermined recording area of the predetermined recording medium in step S6. Whether read processing is performed to generate predetermined playlist data (file) and data is written sequentially to a predetermined area of a predetermined recording medium or after the processing is completed, and all playlist generation processing is completed in step S7 If the process is not completed, the process returns to step S6 to repeat the process. If it is determined in S7 that all predetermined play list data has been generated, the play list data written in step S5 is read and processed.

  Here, in step S6, the playlist data generated sequentially may be sequentially recorded in a predetermined recording area on the same recording medium in which the image / audio information data such as the broadcast program is recorded, or Information may be written on a recording medium other than the one on which the image / audio data is recorded, for example, a predetermined memory means that can be attached and detached.

  Also in this case, the predetermined playlist data may be sequentially generated and written (stored) sequentially, or all the predetermined playlist data may be generated and the playlist processing is completed. Then, all the generated playlist data may be recorded (stored) together.

  In addition, as will be described later with reference to FIGS. 46 and 47, the playlist data is stored in a plurality of play modes in accordance with the recording time so that the user can select a plurality of summary playback times. List data may be generated.

  Here, as described above, since the predetermined evaluation value is also set for each predetermined PU section or each predetermined section obtained by joining a plurality of PU sections, the summary playback time can be manipulated according to the evaluation value. .

  In step S10, the playback time selection mode is set. In step S11, the user selects the playback time immediately, or after selecting the summary playback mode, the user sets the playback time within a predetermined time tmod after the completion of the playlist data detection process. It is determined whether or not the selection process has been performed. If the selection process has not been selected, it is determined whether or not the reproduction stop has been selected by the user in S12.

  If the reproduction stop is selected by the user in step S12, the process ends. If not, the process returns to step S10 to repeat the predetermined process.

  If it is determined in step S11 that the user has immediately selected a playback time, or if no playback time is selected within the predetermined time tmod, the process proceeds to a summary playback operation process in step S13.

  Here, when the user selects the playback time, the summary playback time is set. When the predetermined time tmod has elapsed without selecting the playback time, a predetermined default set playback time (preset playback time) tpb0 is set. The

  Here, the digest playback time may be arbitrarily selected by the user, or may be selected from a preset playback time based on the recorded program recording time and playlist data.

  In this case, for example, a time of 5 minutes, 10 minutes, 15 minutes, 20 minutes, 30 minutes, etc. can be considered, and the default summary playback time may be set as shown in FIG. 46 according to the recording time, for example. It is done.

  In the example of FIG. 46, the summary playback mode can be set only when the recording time is equal to or longer than the predetermined recording time (Trecmin). As the predetermined recording time Trecmin, when the recording time Trec is less than 10 minutes, the time is short. Summary playback is not set and normal playback only.

  As an example, for example, when the recording time Trec is 60 minutes from FIG. 46, the summary playback time that can be selected by the user is 10, 15, 30, 40 minutes, and the default setting time is 30 minutes. .

  In the example shown in FIG. 46, as the recording time Trec increases, the number of summary playback times that can be selected by the user increases. As described above, when the recording time is short, summary playback by skip playback processing is performed. If the total number of skipped sections increases, information will be lost, and it may be impossible to grasp the playback content, so the number of selections can be reduced, and an appropriate summary time can be selected and compared. When the recording time is long, the amount of information is large, so the number of selections is increased so that the user can perform an effective and effective operation.

  Information such as a list of summary playback times that can be selected by the user, default playback time, and the like is stored in the recording / playback apparatus to which the present invention is applied, or the predetermined display means or apparatus connected to the apparatus. Display on a predetermined display screen such as a liquid crystal on the remote control of the display.

  Here, the chapter setting process can also be performed simultaneously with the playlist generation process. As shown in FIG. 44 according to the recording time, a predetermined chapter setting process is automatically performed according to the number of settable chapters. .

  For example, from FIG. 44, when the recording time is 1 hour, predetermined signal processing is performed so that 5 to 40 chapters are set.

  In step S13, the summary playback operation is performed. As described above, since the predetermined evaluation value is set for each of the predetermined PU sections or the joint sections of the plurality of PU sections, the skip playback processing is performed according to the set time and the evaluation value. Thus, summary reproduction is performed.

  That is, the PU section with the highest evaluation value is sequentially selected with the highest priority, and the section with the smaller evaluation value is sequentially selected and compared with the highest priority evaluation value so as to be as close as possible to the selected summary playback time. Go.

  In step S14, it is determined whether or not the reproduction operation is to be terminated. If it is terminated, the process is terminated. If not, it is determined whether or not the predetermined program (program) being reproduced is terminated in step S15. If it is finished but not finished, the process proceeds to step S16 to determine whether to change the playback time.

  If the playback time is changed in step S16, the process returns to step S10 and the above processing is repeated. If not changed, the process returns to step S13 and the summary playback operation is repeated.

7.2 Recording process-related operation flowchart (an example of a recording process operation flowchart)
FIG. 48 shows an example of an operation flowchart in the case of the recording mode.

  When the recording mode is selected in step S1 of the flowchart shown in FIG. 45, it is determined in step R1 of the flowchart shown in FIG. 48 whether the timer recording mode or the normal recording mode is selected. Transition to normal recording operation.

  The routine proceeds to predetermined recording signal processing in the normal recording operation in step R9, and predetermined feature extraction processing is performed from the image / audio data subjected to predetermined encoding processing such as MPEG or the encoded audio / video data in step R10.

  Here, the recording signal processing and the feature extraction signal processing can be performed simultaneously.

  The image / audio data to be subjected to the predetermined encoding process is to perform a predetermined feature extraction process using the image / audio data in the middle of the predetermined encoding process. For example, from the DCT processing system of the image, the DC coefficient data of the DCT signal processing, AC coefficient data and the like can be taken out, and predetermined signal processing is performed using the predetermined data to detect each scene change (feature) detection (cut point (feature) detection), telop (feature) detection, etc. Performs feature extraction signal processing.

  The audio data may be subjected to signal processing such as speaker voice and music (musical tone) determination detection by using data in a predetermined subband band in predetermined subband signal processing in predetermined band compression signal processing.

  As for the musical tone determination signal processing, for example, it is conceivable to perform the determination processing by determining continuity of data in a predetermined subband band.

  It is also conceivable to use video / audio data in the baseband band. For example, by using the baseband signal of the image, scene change detection processing by frame (or field) differential signal processing, or edge detection by the differential signal. It is conceivable to perform other predetermined feature extraction signal processing such as telop feature signal processing.

  Here, each image and the feature data (feature extraction data) subjected to the audio feature extraction signal processing are the same predetermined recording medium on which the image audio data is recorded, or predetermined data storage means (data recording means) such as a predetermined buffer memory. To record.

  In step R11, it is determined whether or not the normal recording mode is ended. If not, the process returns to step R9, and the above operation is repeated. If it is ended, the process proceeds to step R12 and shifts to playlist data generation processing (or chapter data generation processing). To do.

  If the timer recording mode is set at step R1, the recording start time and the recording end time are set at step R2, and it is determined at step R3 whether the predetermined operation time is reached. If it is not the predetermined time, the operation waits at step R7, and step R8 In step S3, it is determined whether the timer operation cancellation interrupt process has been performed by the user. If the timer operation is to be continued, the process returns to step R3 to repeat the above operation.

  If the timer operation is canceled in step R8, the process returns to step S1 in FIG. 45 and the first operation mode selection process is performed.

  If it is determined in step R3 that the predetermined recording operation time has come, the recording operation is started, and the same operations as in steps R9 to R11 described above are performed in steps R4 to R6.

  As described above, the feature data is each image, and the feature data processed by the audio feature extraction signal (feature extraction data) is the same predetermined recording medium on which the image / audio data is recorded, or predetermined data storage means such as a predetermined buffer memory. (Data recording means) If it is determined in step R6 that the recording end time is reached, the process proceeds to step R12 to perform playlist data generation processing or chapter data generation processing.

  In step R12, the above-described various feature data subjected to the predetermined feature extraction processing (the predetermined feature data subjected to the feature extraction processing is subjected to predetermined processing, predetermined signal processing, and predetermined determination processing is performed using these data. Data (including data) is read out from a predetermined recording medium, and predetermined playlist data (file) generation processing and chapter data generation processing are performed.

  The generated playlist data and chapter data are recorded on a predetermined recording medium, and it is determined whether or not the generation process is completed in step R13. If not, the process returns to step R12 and repeats the above processing operations, and the process ends in step R13. If it is determined that the operation has been performed, the operation is terminated.

Here, in addition to the case where the playlist data and chapter data are sequentially recorded on a predetermined recording medium simultaneously with the data generation processing, the predetermined broadcast program, program, or predetermined recording section to be processed is predetermined. After all the generation processing of the play list data and the chapter data is completed, they may be recorded together on a predetermined recording medium.
(When performing playlist data (chapter) processing in parallel with feature extraction processing)

  Here, in the above description, predetermined feature extraction processing is performed simultaneously with recording processing of image / audio information data such as predetermined broadcast programs and programs, and various feature data (feature extraction data or feature extraction data) subjected to feature extraction processing are used. (Including a signal subjected to predetermined processing and predetermined signal processing) on a predetermined recording medium, and after the predetermined broadcast program or program ends, the recorded feature data is read out and playlist data ( Although the case of generating a file), chapter data, etc. has been described, it is also conceivable to perform playlist data (file), chapter data generation processing simultaneously with the feature extraction processing or in parallel with the feature extraction processing.

7.3 Operational flowchart related to playback unit processing (operational flowchart when performing audio segment processing and scene change processing for each predetermined data section in PU processing)
FIG. 49 shows an example of an operation flowchart for performing predetermined signal processing from the audio segment detection point and the scene change detection point in the case of the PU signal processing described above.

  When the processing is started, in step P1, audio data and a predetermined number of sample data of image data are read out from a predetermined recording medium on which image / audio information data is recorded, and a scene change detection process described later, and step P2 The data read in (1) is subjected to storage processing (writing processing, recording processing) in a data buffer which is a predetermined recording means such as a memory.

  If it is determined in step P3 that data of a predetermined number of samples has been recorded in the buffer, the process proceeds to step P4. If it is determined that predetermined sample data has not yet been recorded, the process returns to step P2 and the operation is repeated.

  Here, in step P2 to step P7, since predetermined processing of voice signal sound / silence determination processing is considered for PU processing, the number of predetermined sample data in step P2 is about 0.1 second to about 1 second. The buffer processing corresponding to the number of data corresponding to the predetermined interval is performed.

  For example, when the sampling frequency is 48 KHz, the data is 48000 samples per second, and when 0.1 seconds, the data of 4800 samples is recorded in the buffer.

  In step P4, the audio data is read out from the buffer. In step P5, the audio level calculation process for the predetermined section as described above is performed. In step P6, a comparison process is performed with the predetermined level. A determination process of whether the level is lower or lower is performed, and a silence detection (silence determination) process is performed.

  If it is determined in step P6 that the section is a silent section, the information is stored (recorded) in a predetermined memory (buffer) in step P7. If it is determined that the section is not silent but there is a voice, the process proceeds to step P8. It is determined whether or not the audio buffer processing of the buffer data read in step P1 has been completed. If not, the processing returns to step P2, and the above processing is repeated. If the processing has ended, the processing moves to step P9.

  In step P9, the voice segment information data processed in step P8 is read, and in the step P10, the segment processing of the short silent section, the voiced section, the long silent section, and the voiced section described above is performed.

  In step P11, DCT processing data of image data of a predetermined number of data samples is recorded in a predetermined buffer memory (predetermined data recording means). In step P12, it is determined whether recording of a predetermined amount of data has been completed. If it is not the amount, the process returns to Step P11 to repeat the writing process to the buffer memory system. If it is determined in Step P12 that the writing process of the predetermined data amount has been completed, the process proceeds to Step P13.

  In step P13, predetermined DCT data recorded (written) from the predetermined buffer memory system is read out. In step P14, predetermined signal processing such as interframe difference is performed, and predetermined scene change detection processing is performed.

  In step P15, it is determined whether or not a predetermined scene change has occurred. If it is determined that a scene change has occurred, the position at the time when the predetermined memory means (data recording means, data buffer means, etc.) has changed in step P16 The information data is stored (write process), and the process proceeds to step P17. If it is determined in step P15 that there is no scene change, the process proceeds to step P17.

  In step P17, it is determined whether or not the scene change detection process for a predetermined amount of data in the predetermined data buffer has been completed. If not, the process returns to step P11 to repeat the signal processing, and it is determined that the process has been completed in step P17. Shifts to Step P18.

  In step P18, the scene change position information recorded (stored) in the predetermined buffer memory means is read, and in step P19, a section that is too short, such as shorter than the predetermined section length, is joined to the preceding and following sections. Perform correction processing.

  In step P20, the audio segment position information data and scene change position information data generated and processed in the predetermined section are read. In step P21, predetermined information data such as an audio segment position, an audio segment section length, a scene change position, and a scene change section length are read. Then, predetermined PU information data such as predetermined PU position information and section information is generated.

  In step P22, from the PU information processed in step P21, feature data (or feature extraction data or a signal obtained by performing predetermined signal processing on the feature extraction data) corresponding to the PU section is stored in a predetermined recording medium, or Write processing to a predetermined data buffer.

  As described above, these recording media are different from the predetermined recording area on the same predetermined recording medium on which the image / audio information data of the predetermined section such as the broadcast program or program to be processed is recorded. It is also conceivable to perform recording (storage and writing processing) on a predetermined recording medium.

  In step P23, it is determined whether or not a series of signal processing such as audio segment processing, scene change processing, and PU processing for a predetermined amount of data has been completed. If it is determined that the processing has ended, the processing ends, and it is determined that the processing has not ended. If so, the process returns to step P1 to repeat the above process.

(Operation flowchart when scene change processing is performed after all audio segment processing is performed in PU processing)
Here, in the above, the segment processing of the audio data is sequentially performed for each predetermined section of the image / audio data, such as the predetermined broadcast program and program that are currently considered, and then the scene change detection process of the image is performed. However, instead of processing for each predetermined section as described above, all the scene change detection processes are performed after the audio segment processing for all the predetermined sections of the broadcast program or program to be processed is completed, and all scene changes are performed. It is also conceivable to perform the predetermined PU process after the detection process is completed.

  FIG. 50 shows another example of an operation flowchart for performing predetermined signal processing from the audio segment detection point and the scene change detection point in the case of the PU signal processing described above.

  When the process is started, first, in a first step T1, predetermined voice segment processing as described in steps P1 to P9 in the flowchart of FIG. 49 is performed.

  Here, the audio data is obtained by sequentially reading data of a predetermined data sample amount into a predetermined buffer memory.

  The segment position information data subjected to the audio segment processing in step T2 is recorded in predetermined memory means (data storage means, data recording means), and in step T3, the broadcast program, program, etc. that are currently processed are recorded. It is determined whether or not the predetermined segment processing has been completed for all the audio data in the predetermined section. If it is determined that the predetermined segment processing has not ended, the process returns to step T1 to repeat the above-described processing.

  In step T4, a predetermined scene change process as described in steps P11 to P18 in the flowchart of FIG. 49 is performed. Here, DCT data of an image is obtained by sequentially reading a predetermined data sample amount of data into a predetermined buffer memory.

  The data of the scene change position information that has undergone the predetermined scene change process in step T5 is recorded in predetermined memory means (data storage means, data recording means), and in step T6, the broadcast program or program that is currently processed is recorded. It is determined whether or not the predetermined scene change process has been completed for the DCT data of all the images in the predetermined section. If it is determined that the predetermined scene change process has not been completed, the process returns to step T4 and the above process is repeated. Migrate to

  In step T7, predetermined audio segment position information data and predetermined scene change position information data are read from the predetermined memory means, predetermined PU processing is performed in step T8, and in step T9, the broadcast program currently being processed, It is determined whether or not the predetermined PU process has been completed over all the predetermined sections such as a program. If it is determined that the predetermined PU process has ended, the process ends. If it is determined that the predetermined PU process has not ended, the process returns to T7 and the above operation is repeated.

It is operation | movement explanatory drawing of the summary reproduction | regeneration and the chapter process in the recording / reproducing apparatus to which this invention is applied. It is a figure which shows an example of the display by the said chapter process typically. It is a figure which shows typically an example of the process in the said recording / reproducing apparatus. It is explanatory drawing of the rule process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the relationship between the meaning assignment | providing process and characteristic data in the said recording / reproducing apparatus. It is explanatory drawing of an example of the rule file format in the said recording / reproducing apparatus. It is explanatory drawing of an example of the calculation processing method of the evaluation value in the said recording / reproducing apparatus. It is explanatory drawing of an example of the time correction function in the said recording / reproducing apparatus. It is explanatory drawing of an example of the general type of the time correction function in the said recording / reproducing apparatus. It is explanatory drawing of an example of the structure of the video data in the said recording / reproducing apparatus. It is explanatory drawing of an example of the connection relation between the reproduction | regeneration units in the said recording / reproducing apparatus. It is explanatory drawing of an example of the meaning addition process between the reproduction | regeneration units in the said recording / reproducing apparatus. It is explanatory drawing of an example of the rule 2 process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the time correction function in the said recording / reproducing apparatus. It is explanatory drawing of an example of a structure of the rule file in the said recording / reproducing apparatus. It is explanatory drawing of an example of the process of this invention in the said recording / reproducing apparatus. It is a block diagram which shows the structural example of the recording / reproducing apparatus to which this invention is applied. It is explanatory drawing of an example of the various predetermined data recording states in the said recording / reproducing apparatus. It is a figure which shows an example of the display in the said recording / reproducing apparatus. FIG. 10 is a block diagram illustrating another configuration example of a recording / reproducing apparatus to which the present invention is applied. It is a block diagram which shows an example of a structure of the audio | voice system characteristic extraction processing system in the said recording / reproducing apparatus. It is a block diagram which shows the other example of a structure of the audio system characteristic extraction processing system in the said recording / reproducing apparatus. It is a block diagram which shows an example of a structure of the video-type feature extraction processing system in the said recording / reproducing apparatus. It is explanatory drawing of the scene change process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the telop and color feature detection area | region in the said recording / reproducing apparatus. It is explanatory drawing of an example of the similar image characteristic in the said recording / reproducing apparatus. It is explanatory drawing of an example of the person characteristic detection area | region in the said recording / reproducing apparatus. It is explanatory drawing of an example of the person detection process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the person detection (number determination) process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the number of persons detection process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the number of persons detection process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the number of persons detection process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the number of persons detection process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the reproduction | regeneration unit process in the said recording / reproducing apparatus. It is explanatory drawing of an example of the reproduction | regeneration unit process in the said recording / reproducing apparatus. It is explanatory drawing of an example of CM (commercial) detection process in the said recording / reproducing apparatus. It is a block diagram which shows the structural example of the reproduction | regeneration unit processing system in the said recording / reproducing apparatus. It is explanatory drawing of an example of a structure of the characteristic data file in the said recording / reproducing apparatus. It is explanatory drawing of an example of a structure of the characteristic data file in the said recording / reproducing apparatus. It is explanatory drawing of an example of a structure of the characteristic data file in the said recording / reproducing apparatus. It is explanatory drawing of an example of the hierarchical structure of the reproduction | regeneration unit data in the said recording / reproducing apparatus. It is explanatory drawing of an example of the hierarchical structure of the reproduction | regeneration unit data in the said recording / reproducing apparatus. It is explanatory drawing of an example of a structure of the reproduction | regeneration unit video characteristic data in the said recording / reproducing apparatus. It is explanatory drawing of an example of the play list (summary) data in the said recording / reproducing apparatus. It is a flowchart which shows the operation example of the said recording / reproducing apparatus. It is explanatory drawing of an example of the relationship between the recording time in the said recording / reproducing apparatus, and the selectable summary reproduction | regeneration time. It is explanatory drawing of an example of the recording time in the said recording / reproducing apparatus, and the number of automatic setting chapters. It is a flowchart which shows the example of recording operation | movement of the said recording / reproducing apparatus. It is a flowchart which shows the example of reproduction | regeneration operation | movement of the said recording / reproducing apparatus. It is a flowchart which shows the other example of reproducing | regenerating operation | movement of the said recording / reproducing apparatus.

Explanation of symbols

  1 receiving antenna system, 2 broadcast reception signal processing system, 3 audio A / D conversion processing system, 4 audio encoder processing system, 5 multiplexing processing system, 6 recording processing system, 7 recording medium system, 8 video A / D conversion processing System, 9 video encoder processing system, 10 feature extraction processing system, 11 memory system, 12 playback processing system, 13 playback data separation processing system, 14 audio decoding processing system, 15 audio D / A processing system, 16 video decoding processing system, 17 Video D / A processing system, 18 Playback control system, 19 Playlist data generation processing system, 20 System controller system, 21 User input I / F system, 22 Remote control system, 23 Network I / F system, 24 Network system, 25 Recording medium system, 26 recording medium processing system, 27 display processing system, 30, 30A recording / reproducing apparatus, 100 stream separation system, 10 1 audio data decoding system, 102 level processing system, 103 data count system, 104 data buffer system, 105 audio data integration processing system, 106 judgment processing system, 107 threshold setting system, 108 frequency analysis processing system, 109 judgment processing system 110 Stream data analysis system, 111 Subband analysis processing system, 112 Data buffer system, 200 Stream data analysis system, 201 DCT coefficient processing system, 202 Scene change detection processing system, 203 Color feature detection processing system, 204 Similar image detection processing System, 205 human detection determination processing system, 206 telop detection determination processing system, 207 flip detection processing system, 208 motion vector processing system, 209 camera feature determination processing system, 210 object detection processing system, 301 section length measurement system, 302 playback unit Processing system, 03 playback unit, wherein the data processing system, 900 rule switching processing system, 901 rule switching processing system

Claims (5)

In an information signal processing method for recording or reproducing a predetermined video / audio information signal by a predetermined band compression signal processing using a predetermined recording medium,
When recording image and audio information signals,
The image / audio information signal of the analog input system automatically extracts a predetermined characteristic signal in the image or the audio signal when the image / audio information signal is recorded,
For the audio / video information signal of the digital input system, characteristic data extraction processing for extracting a predetermined characteristic signal in the image or audio signal is automatically performed after the recording of the audio / video information signal is completed, or the recording of the audio / video information signal is performed. An information signal processing method comprising: performing information signal processing by selecting whether the characteristic data extraction processing is automatically performed by a predetermined selection operation or manually when desired after completion. Capture software for executing the information signal processing by a predetermined operation by a predetermined data input system,
Set the above information signal processing to an executable state,
2. The information signal processing method according to claim 1, wherein the information signal processing is executed when the predetermined operation mode is set by a predetermined operation system. Recording means for recording a predetermined video / audio information signal on a predetermined recording medium by predetermined band compression signal processing ;
Characteristic data extraction means for performing characteristic data extraction processing for extracting predetermined characteristic data for each predetermined section of the video and audio information signal;
When recording the video / audio information signal from the recording means, the video / audio information signal of the analog input system is automatically subjected to the characteristic data extraction processing by the characteristic data extraction means when the video / audio information signal is recorded. The video / audio information signal of the digital input system automatically performs the characteristic data extraction processing by the characteristic data extraction means after the recording of the video / audio information signal is completed, or the characteristic after the recording of the video / audio information signal is completed. An information signal processing apparatus comprising: information processing means for selecting whether the characteristic data extraction processing by the data extraction means is automatically performed by a predetermined selection operation means or manually when desired. . A data input system for capturing software for executing predetermined information signal processing by a predetermined operation;
Signal processing setting means for setting to a state in which predetermined information signal processing can be executed by software fetched by the data input system,
The information processing unit automatically performs the characteristic data extraction process on the analog input system image / audio information signal when the image / audio information signal is recorded by the characteristic data extraction unit , and the digital input system image For the audio information signal, the characteristic data extraction means automatically performs the characteristic data extraction process after the recording of the video / audio information signal or the predetermined characteristic data extraction process is performed after the recording of the video / audio information signal is completed. 4. The information signal processing apparatus according to claim 3, wherein the information signal processing apparatus is configured to select whether to perform automatically or manually when desired. A program recording medium in which a control program for recording or reproducing a predetermined video / audio information signal by predetermined band compression signal processing using a predetermined recording medium is recorded so as to be read and executed by a computer,
When recording image and audio information signals,
The image / audio information signal of the analog input system automatically extracts a predetermined characteristic signal in the image or the audio signal when the image / audio information signal is recorded ,
For the audio / video information signal of the digital input system, a characteristic data extraction process for extracting a predetermined characteristic signal in the image or audio signal is automatically performed after the recording of the audio / video information signal is completed, or the recording of the audio / video information signal is performed. A control program is recorded so as to be readable and executable by a computer, characterized by selecting whether the characteristic data extraction process is automatically performed by a predetermined selection operation or manually when desired after completion. A program recording medium characterized by the above.

Images