JP2014164644A - Signal process device, display device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a signal process device that allows a form of a subject to be easily compared.SOLUTION: The signal process device comprises: a first detection section that detects a displacement among frames of a moving image; a first extraction section that extracts a characteristic point of the displacement detected by the first detection section; a collation section that collates the characteristic point in a first moving image extracted by the first extraction section with the characteristic point in a second moving image thereby, and associates a first characteristic point in the first moving image with a second characteristic point corresponding to the first characteristic point in the second moving image.

Description

  The present invention relates to a signal processing device, a display device, and a program.

  A form practice device for comparison practice with a model form has been disclosed (for example, see Patent Document 1).

JP-A-7-124281

  In the apparatus described in Patent Document 1, there is a case where the timing of each form cannot be synchronized. As described above, there has been a problem in the prior art that it may not be possible to easily compare the trainee's form with the model form.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a signal processing device, a display device, and a program capable of easily comparing forms of different subjects.

  The present invention has been made to solve the above-described problems, and one aspect of the present invention provides a first detection unit that detects a displacement between frames of a moving image, and the displacement detected by the first detection unit. A first extraction unit that extracts a feature point of the first moving image, a feature point in the first moving image extracted by the first extracting unit is matched with a feature point in the second moving image, and the first feature in the first moving image And a collating unit that associates a point with the second feature point corresponding to the first feature point in the second moving image.

  Another embodiment of the present invention is a display device including the above-described signal processing device.

  According to another aspect of the present invention, a computer includes a first step of detecting a displacement between frames of a moving image, a second step of extracting a feature point of the displacement detected in the first step, and the first step. The feature points in the first video extracted in step 2 are matched with the feature points in the second video, and the first feature points in the first video and the second video correspond to the first feature points. A third step of associating the second feature point with the program.

  The forms of the subject can be easily compared.

It is a schematic block diagram which shows an example of a structure of the display apparatus by 1st Embodiment of this invention. It is a figure which shows an example of the detailed structure of the timing process part of 1st Embodiment, and a memory | storage part. It is a figure which shows an example of the data with which the moving image, the position, and the time point were matched. It is a figure which shows an example of the result of collating and matching moving image data. It is a figure which shows another example of the result of collating and matching moving image data. It is a figure explaining an example of the process in which a display apparatus reproduces | regenerates a moving image. It is a figure explaining the specific example (1st example) of a reproduction | regeneration process. It is a figure explaining the specific example (2nd example) of a reproduction | regeneration process. It is a figure explaining the specific example (3rd example) of a reproduction | regeneration process. It is a schematic block diagram which shows an example of a structure of the timing process part of 2nd Embodiment. It is a schematic block diagram which shows an example of a structure of the timing process part of 3rd Embodiment. It is a schematic block diagram which shows an example of a structure of the process part of 4th Embodiment. It is a schematic block diagram which shows an example of a structure of the imaging device of 5th Embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
[First Embodiment]
FIG. 1 is a schematic block diagram showing an example of the configuration of the display device 20 according to the first embodiment of the present invention. The display device 20 includes a display unit 4, a storage medium unit 5, a signal processing unit 10 (signal processing device), and a bus 11.

  The display unit 4 is a display that displays images or character information. For example, the display unit 4 includes a liquid crystal display, an organic EL (Electro Luminescence) display, and the like, and is based on an image based on an image signal input by the control of the signal processing unit 10 or an input character signal. Display text information and so on. In addition, the display unit 4 displays two types of moving images (hereinafter referred to as “moving images”) input under the control of the signal processing unit 10 side by side. In addition, the display part 4 may be comprised including the touch panel which detects the touch operation with respect to a display surface. In addition, the display unit 4 may be a display device as an external device that displays images or character information input via a wired or wireless connection with the display device 20.

  The storage medium unit 5 is connected to a removable storage medium 50 (such as a card memory), and reads an image or information stored (recorded) in the storage medium 50. For example, the storage medium unit 5 reads a moving image stored (recorded) in the storage medium 50 under the control of the signal processing unit 10. Note that the storage medium unit 5 may perform writing or erasing of an image or information on the storage medium 50.

  The storage medium 50 is a memory that is detachably connected to the display device 20 and stores, for example, a captured moving image. Instead of the storage medium unit 5, a storage medium or a storage device included in another device connected via wire or wireless may be used.

  The signal processing unit 10 includes a processing unit 1 and a storage unit 2. The processing unit 1 includes a timing processing unit 110 and a reproduction processing unit 120. The timing processing unit 110 performs signal processing on the moving image. The reproduction processing unit 120 reproduces a moving image based on the signal processing result of the timing processing unit 110. For example, the reproduction processing unit 120 reproduces the moving image read from the storage medium unit 5 based on the signal processing result of the timing processing unit 110 and causes the display unit 4 to display the moving image. The storage unit 2 stores data necessary when the timing processing unit 110 performs signal processing on a moving image or data generated as a result of signal processing.

  The bus 11 is connected to the display unit 4, the storage medium unit 5, and the signal processing unit 10, and transfers image signals, control signals, and the like output from each unit.

  Next, the configuration of the timing processing unit 110 and the storage unit 2 will be described in detail with reference to FIG. FIG. 2 is a diagram illustrating an example of a detailed configuration of the timing processing unit 110 and the storage unit 2. First, a detailed configuration of the timing processing unit 110 will be described. The timing processing unit 110 includes a selection unit 112, a template registration unit 113, a displacement detection unit 114, a change feature point extraction unit 115, and a collation unit 116.

  The selection unit 112 selects an image area including a predetermined subject from the moving image. For example, the selection unit 112 selects an image area including a predetermined subject from among the image areas of the moving image in accordance with a user operation. Here, the predetermined subject is, for example, a sports or dance practitioner recorded in a moving image or a sample (teacher). For example, the selection unit 112 selects an image region including a subject whose form is to be compared from a moving image according to a user operation. The user operation may be an operation on an operation unit (not shown), or may be a touch operation on the display surface when the display unit 4 includes a touch panel.

  The template registration unit 113 (registration unit) registers the image of the image area selected by the selection unit 112 as a template image. For example, the template registration unit 113 (registration unit) registers and stores the template image in the storage unit 2.

  The displacement detector 114 (first detector) detects a displacement between frames of the moving image. For example, the displacement detection unit 114 performs template matching on each frame image of the moving image using the template image registered by the template registration unit 113, thereby detecting the displacement between frames of the image region including the predetermined subject. To detect.

  Specifically, the displacement detection unit 114 detects the coordinate position of an image area including a predetermined subject in the image area of the frame image for each frame image of the moving image, and based on the detected coordinate position of each frame image. To detect the displacement of the coordinate position between frames. Further, the displacement detection unit 114 generates “position arrangement data” in which the coordinate positions of image areas including a predetermined subject for each frame are arranged in the order of frames. In this position array data, the displacement detection unit 114 associates, for example, the coordinate position of an image area including a predetermined subject for each frame with a frame number. Further, the displacement detection unit 114 supplies information indicating the displacement between frames of the detected coordinate position to the change feature point extraction unit 115.

  The change feature point extraction unit 115 (first extraction unit) extracts a feature point of displacement between frames detected by the displacement detection unit 114. For example, the change feature point extraction unit 115 extracts a timing at which the change is large among the displacements between frames as a feature point of the displacement. Here, the timing at which the change is large is, for example, “timing at which the direction of displacement is reversed” or “timing at which the displacement speed change is large (for example, timing at which movement starts from a stationary state or movement from a stationary state) Etc.) ”. Information indicating the timing of the large change extracted as the displacement feature point is referred to as a “time point” in the following description.

  Further, the change feature point extraction unit 115 generates “time point array data” in which the extracted “time points” are associated with frame numbers and arranged in the order of frames. For example, the change feature point extraction unit 115 includes flag information indicating that the displacement feature point is “present” among the flag information indicating the presence / absence of the displacement feature point (for example, presence: “1”, absence: “0”). (For example, “Yes”: “1”) is associated with the frame number as “time point”. Note that the change feature point extraction unit 115 may associate different flag information as “time point” with the frame number according to the type of displacement feature point.

  The collation unit 116 collates the displacement feature points in the moving image with the displacement feature points in the other moving images and performs association. For example, the two moving images to be compared are “A moving image” (first moving image) and “B moving image” (second moving image). In this case, the change feature point extraction unit 115 extracts the displacement feature point in the “A movie” and the displacement feature point in the “B movie”, respectively, Time point array data of “B moving image” is generated. The collation unit 116 refers to the time point array data of “A movie” and the time point sequence data of “B movie”, and collates the time point of “A movie” and the time point of “B movie”. To do. That is, the collation unit 116 collates the feature points in the “A moving image” extracted by the change feature point extraction unit 115 and the feature points in the “B moving image”.

  Then, the collation unit 116 includes the first feature point in “A movie” (“time point” in “A movie”) and the second feature point corresponding to the first feature point in “B movie” (“B movie”). To “time point” corresponding to “time point” in “A moving image”). For example, the collation unit 116 includes a first feature point (“time point”) in “A movie” and a second feature point (“time point”) indicating the same displacement feature as the first feature point in “B movie”. ). Next, a specific example of processing in which the collation unit 116 collates and associates will be described below with reference to FIGS. 3 and 4.

(Specific example of verification processing)
In FIG. 3, a moving image (image and sound (including sound other than voice)), “position” of an image region including a predetermined subject, and “time point” of the timing of a displacement feature point are associated with each other. It is a figure which shows an example of data. The horizontal axis represents the time axis of the moving image. In this figure, for example, “A video” is a video where a teacher who is a model of dance is dancing, and “B video” is a video where a practitioner who is practicing the dance is dancing Suppose that

  In addition, the “time point” shown in this figure is associated with a frame at a timing at which a feature point of displacement in the dance movement is extracted (a timing when the movement change is large). For example, “time point” is associated with flag information (for example, “1”) indicating that the displacement feature point is “present”. That is, the change feature point extraction unit 115 includes flag information indicating “existence” of the displacement feature point in the frame with the large change timing extracted as the displacement feature point in “A movie” and “B movie” (for example, , Existence: “1”) is associated as “time point”.

  In this figure, in “A moving image”, flag information “1” is “time point (0)” and “time point (1)” in each of the frames fa (12), fa (18), and fa (23). , “Time point (2)”, respectively. In the “B moving image”, the flag information “1” is set to “time point (0)”, “time point (1)”, “figure” in each of the frames fb (12), fb (17), and fb (20). Time points (2) ”are associated with each other.

  FIG. 4 is a diagram illustrating an example of a result of collating and matching the moving image data illustrated in FIG. The collation unit 116 sets each of “time point (0)”, “time point (1)”, and “time point (2)” in “A moving image” in “B moving image” in the order of the passage of time of the moving image. Corresponding to “time point (0)”, “time point (1)”, and “time point (2)”. That is, the collation unit 116 associates each of the plurality of first feature points in “A movie” with the second feature points corresponding to each of the plurality of first feature points in “B movie”.

  For example, the collation unit 116 determines that each of “time point (0)”, “time point (1)”, and “time point (2)” in “B moving image” is “time point (0)” in “A moving image”. ”,“ Time point (1) ”, and“ time point (2) ”so as to be reproduced at the same timing.

  Specifically, the collation unit 116 has the same frame fb (12) of “time point (0)” in “B moving image” as frame fa (12) of “time point (0)” in “A moving image”. Match to be played at the timing. In addition, the collating unit 116 reproduces the frame fb (17) of “time point (1)” in “B moving image” at the same timing as the frame fa (18) of “time point (1)” in “A moving image”. Correlate as Further, the collating unit 116 reproduces the frame fb (20) of “time point (2)” in “B moving image” at the same timing as the frame fa (23) of “time point (2)” in “A moving image”. Correlate as

  In this way, the collating unit 116 associates each of the displacement feature points in the dance movement of the trainee so as to be reproduced at the same timing as each of the displacement feature points in the teacher's dance movement as a sample.

  Note that the matching unit 116 determines the first feature point in “A movie” (time point in “A movie”) and the second feature point corresponding to the first feature point in “B movie” (the time in “B movie”). Point). Then, the collation unit 116 performs frame image of “A moving image” corresponding to the first feature point and frame image of “B moving image” corresponding to the second feature point with respect to the integrated feature point (time point). May be associated.

  FIG. 5 shows an example in which “time points” are integrated by collating the moving image data shown in FIG. In this figure, the collation unit 116 adds “A point” (“0”) in “A movie” and “Time point (0)” in “B movie” to “A point (0)”. The frame “fa” (12) of “Movie” and the frame fb (12) of “B Movie” are associated with each other. In addition, the collation unit 116 adds “time point (1)” in “A moving image” and “time point (1)” in “B moving image” to “time point (1)” and “A moving image”. The frame fa (18) is associated with the frame fb (17) of “B moving image”. In addition, the collation unit 116 adds “time point (2)” in “A moving image” and “time point (2)” in “B moving image” to “time point (2)” by combining “time point (2)”. The frame fa (23) and the frame fb (20) of “B moving image” are associated with each other.

  Returning to FIG. 2, the detailed configuration of the storage unit 2 will be described next. The storage unit 2 includes a template image storage unit 21, a position array data storage unit 22, a time point array data storage unit 23, and an image processing buffer memory 24.

  The template image storage unit 21 stores a template image registered by the template registration unit 113. For example, the template image storage unit 21 stores an image of an image area including a predetermined subject selected by the selection unit 112 as a template image under the control of the template registration unit 113. The template image storage unit 21 supplies the stored template image in response to a request from the displacement detection unit 114 when the displacement detection unit 114 performs template matching.

  The position array data storage unit 22 stores the position array data generated by the displacement detection unit 114. For example, the position array data storage unit 22 stores the coordinate position of the image area including a predetermined subject for each frame in association with the frame number.

  The time point array data storage unit 23 stores the time point array data generated by the change feature point extraction unit 115. For example, the time point array data storage unit 23 stores “time point”, which is a timing with a large change extracted as a feature point of displacement, in association with a frame number.

  The image processing buffer memory 24 is used as a work area when the signal processing unit 10 performs signal processing on a moving image or a work area when executing a process of playing back a moving image. Remembered. For example, the image processing buffer memory 24 includes an A buffer memory used when processing “A moving image” and a B buffer memory used when executing processing “B moving image”. And have.

  The signal processing unit 10 configured as described above, based on the result of collation by the collation unit 116, the timing of the movement of the predetermined subject (displacement feature point) in each of the “A moving image” and the “B moving image”. Play each video so that That is, the signal processing unit 10 reproduces each moving image so that the timing of the movement of the predetermined subject (displacement feature point) in each of the two moving images matches. In addition, the display device 20 displays the moving images “A moving image” and “B moving image” reproduced by the signal processing unit 10 side by side on the display unit 4.

  Therefore, the display device 20 can display the moving images side by side on the display unit 4 so that the timing of the movement of the predetermined subject (displacement feature point) in each of the two moving images matches.

  For example, the display device 20 (the signal processing unit 10) includes “time point (0)”, “time point (1)”, “time point (2)” in “A moving image” illustrated in FIG. “A movie” and “B movie” are reproduced so that “time point (0)”, “time point (1)”, and “time point (2)” in “B movie” are reproduced at the same timing. Can be displayed side by side on the display unit 4 (reproduced). Next, the reproduction process in the display device 20 will be described in detail.

(Description of playback processing)
FIG. 6 is a diagram illustrating an example of a process in which the display device 20 reproduces a moving image. In this figure, the signal processing unit 10 generates time point array data of “A moving image” and time point array data of “B moving image”, and the generated time point array data is stored in the time point array data. It is assumed that it is stored in the unit 23.

  The collating unit 116 refers to the time point array data of “A moving image” and the time point array data of “B moving image” stored in the time point array data storage unit 23, and the “time point” of “A moving image” ”And“ time point ”in“ B moving image ”(see S101).

  The reproduction processing unit 120 reads the “A moving image” from the storage medium unit 5, and arranges the frame image array (A image array), the sound array (A sound array), and the position array (A position) of “A moving image”. The data for a predetermined number of frames of the respective data are temporarily stored in the A buffer memory of the image processing buffer memory 24 according to the frame order (see S102). Further, the reproduction processing unit 120 reads “B moving image” from the storage medium unit 5, and arranges a frame image array (B image array), a sound array (B sound array), and a position array (“B moving image”). (B position array) and a predetermined number of frames of the data are temporarily stored in the B buffer memory of the image processing buffer memory 24 according to the frame order (see S103).

Next, the reproduction processing unit 120 reproduces the “time point” corresponding to each of “A moving image” and “B moving image” based on the result (corresponding time point) collated by the collating unit 116. The “A moving image” and the “B moving image” are read from the image processing buffer memory 24 so as to have the same timing, reproduced (see reference S104), and displayed side by side on the display unit 4 (refer to reference S105).
In addition, in response to reading “A moving image” and “B moving image” from the image processing buffer memory 24, the reproduction processing unit 120 sequentially and temporarily stores data of the next frame in the image processing buffer memory 24. Remember me.

  For example, as illustrated in FIG. 4, the reproduction processing unit 120 sets “time point (0)”, “time point (1)”, and “time point (2)” in “B moving image” to “A moving image”. "A movie" and "B movie" start to be played at the same timing as "time point (0)", "time point (1)", and "time point (2)" The frame and the frame rate of “B moving image” are adjusted.

  Specifically, the reproduction processing unit 120 performs the “fa movie” frame fa (12), the “B movie” frame fb (12), the “A movie” frame fa (18), and the “B movie” frame. Playback is performed so that the frame fb (17), the frame A (moving picture A) fa (23), and the frame B (moving picture B) fb (20) have the same timing. That is, the playback processing unit 120 converts the frame fb (12) from the frame fb (12) to the frame fb (17) of the “B moving image” into the “A moving image” during the period of reproducing the frame fa (12) from the frame fa (12) of the “A moving image”. "Is played back at a frame rate lower than the frame rate. Further, the playback processing unit 120 converts the frame fb (12) from the frame fb (12) to the frame fb (17) of the “B moving image” into the “A moving image” during the period of reproducing the frame fa (12) from the frame fa (12) of “A moving image”. The frame rate is lower than the frame rate of “

  As described above, the reproduction processing unit 120 reproduces the frame image from which the first feature point is extracted in “A moving image” and the frame image from which the second feature point is extracted in “B moving image” at the same timing. In this way, “A moving image” and “B moving image” are reproduced. In addition, the reproduction processing unit 120 performs “A movie” or “B movie” so that the frame image from which the first feature point is extracted and the frame image from which the second feature point is extracted are reproduced at the same timing. Convert the frame rate. In addition, the reproduction processing unit 120 displays the reproduced “A moving image” and “B moving image” side by side on the display unit 4.

  That is, the display device 20 reproduces each moving image and displays it side by side on the display unit 4 so that the timing of the movement of the predetermined subject (displacement feature point) in each of the two moving images matches. Therefore, the display device 20 can display the respective moving images so that the forms of the subjects recorded in the different moving images can be easily compared.

  Note that the playback processing unit 120 changes the playback time for each frame of “B moving image” to be longer when changing “B moving image” to a frame rate lower than the frame rate of “A moving image”. Alternatively, the frame (image) of “B moving image” within the period is changed so that the number of frames of “B moving image” and the number of frames of “A moving image” are the same within the period for changing the frame rate. May be added. For example, in FIG. 4, the image added during the period from frame fb (12) to frame fb (17) of “B moving image” is the image of frames “b moving image” from frame fb (12) to frame fb (17). It may be an image of at least one frame, or may be an image generated based on an image of at least one frame. Similarly, the image added in the period from the frame fb (17) to the frame fb (20) of the “B moving image” is at least one of the images of the frame fb (17) to the frame fb (20) of the “B moving image”. It may be a frame image or an image generated based on at least one frame image.

  In the example described with reference to FIG. 4 and FIG. 6, the example in which “B moving image” is changed to a frame rate lower than the frame rate of “A moving image” is shown. If “B video” is larger than “A video”, “B video” may be changed to a frame rate higher than the frame rate of “A video” and played back. In this case, the playback time for each frame of “B moving image” may be changed, or the number of frames of “B moving image” and the number of frames of “A moving image” within the period for changing the frame rate may be changed. So that the frames of “B moving image” within the period may be thinned out.

  The display device 20 is information indicating “position” corresponding to the frame image to be reproduced and displayed based on the position array data (stored in the position array data storage unit 22) generated by the displacement detector 114. May be displayed on the display unit 4 so as to overlap the video. Further, when the moving image is reproduced and displayed on the display unit 4, the display device 20 may display information indicating that for a frame image including an image region including a predetermined subject.

  Further, the displacement detector 114 does not have to generate position array data. For example, the displacement detection unit 114 does not have to execute the process of generating the position array data by executing the process of detecting the displacement between the frames at the coordinate position based on the coordinate position of each frame image.

  The example of the moving image described with reference to FIGS. 3 to 6 is a moving image including an image and sound. However, the moving image is not limited to this, and the moving image to which the processing according to the present embodiment is applied is sound (sound). May not be included.

(Specific example of processing for matching the playback timings of two videos)
According to the present embodiment, even if there is a partial omission or a different movement in the movement of a predetermined subject in each of the two moving images, the two moving images can be associated with each other by associating feature points of other movement displacements. Each of the predetermined subjects can be played back at the same timing. Next, with reference to FIGS. 7, 8, and 9, a specific example of a reproduction process for reproducing in accordance with the timing of movement of a predetermined subject of each of the two moving images will be described.

  3 and 4. As in the process described with reference to FIG. 5, among the two videos of “A movie” and “B movie”, a teacher whose “A movie” serves as a model for dance is dancing. It is assumed that the video is a video (example form video), and the “B video” is a video (practice form video) in which a practitioner who practices the dance is dancing. Further, a description will be given of a process of matching the playback timing of “B moving image” with the playback timing of “A moving image”.

  7, 8, and 9, FIGS. 3, 4. Also, “image”, “sound”, and “position” of each frame shown in FIG. 5 are omitted, and “time point” and “frame number” are shown.

Further, here, feature points of displacement between frames of a predetermined subject (teacher or practitioner) are extracted for each type of feature points of displacement as shown below. Further, flag information as shown below is predetermined for each type of displacement feature point.
・ "Displacement direction is reversed": Flag information "2"
・ "Jump": Flag information "3"
・ "Move from stationary state": Flag information "4"
・ "Still from moving": Flag information "5"
Then, it is assumed that each flag information is associated as a “time point” with respect to the frame in which the feature points of these displacements are extracted.

<First example>
FIG. 7 is a diagram for explaining a specific example (first example) of the reproduction process. The example shown in this figure shows a case where the “time point” included in “A movie” (example form movie) is missing in “B movie” (practice form movie).

  In this figure, “time point” is associated with frames in the order of flag information “2”, “3”, “3”, “2” in “A movie”, whereas “B movie” “Time point” is associated with the frame in the order of flag information “3”, “3”, and “2”. That is, the flag information “2” that is the first “time point” in “A moving image” does not exist at the beginning of “B moving image”. This is because, for example, the practitioner does not perform the movement of “the direction of displacement is reversed” (flag information “2”), which is the first movement (swing), but the next “jump” (flag information “3”). This is the case where the movement has been performed.

  FIG. 7A shows an array of “time points” and an array of “frame numbers” for “A moving image” and “B moving image”, respectively. As the “time point” of “A movie”, flag information “2” in frame fa (12), flag information “3” in frame fa (14), flag information “3” in frame fa (17), and frame The flag information “2” is associated with fa (22). On the other hand, as “time point” of “B moving image”, flag information “3” in the frame fb (17), flag information “3” in the frame fb (21), and flag information “2” in the frame fb (25). Are associated with each other.

  FIG. 7B is a diagram illustrating a result of the collation unit 116 collating the “time points” of “A moving image” and “B moving image” illustrated in FIG. . The collation unit 116 associates the “time point” of “A moving image” and the “time point” of “B moving image” with the “time point” of the same flag information in the order according to the time flow of the moving image.

  For example, the collation unit 116 associates the “time point” of the frame fa (14) of “A moving image” with the “time point” of the frame fb (17) of “B moving image”. Further, the collating unit 116 associates the “time point” of the frame fa (17) of “A moving image” with the “time point” of the frame fb (21) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (22) of “A moving image” with the “time point” of the frame fb (25) of “B moving image”. Note that the collation unit 116 does not have the “time point” corresponding to the “time point” of the frame fa (12) of the “A moving image” in the “B moving image”, and therefore the frame fa (12) of the “A moving image”. Do not associate “time point”.

  Based on the collation result of the collation unit 116, the reproduction processing unit 120 reproduces the “time point” frames corresponding to “A moving image” and “B moving image” at the same timing.

  For example, the reproduction processing unit 120 includes the frame fa (14) of “A moving image”, the frame fb (17) of “B moving image”, the frame fa (17) of “A moving image”, and the frame fb (21 of “B moving image”). ), And “A movie” and “B movie” are reproduced and displayed so that the frame fa (22) of “A movie” and the frame fb (25) of “B movie” have the same timing. Part 4 is displayed side by side. Further, the reproduction processing unit 120 matches the reproduction timing of “B moving image” with the timing of “A moving image”, so that frames “b moving image” fb (17) to fb (21) and frames fb (21) to Each frame rate in the period of fb (25) is converted.

  FIG. 7B shows an example in which the length of time of each frame in the period of the frames fb (17) to fb (21) is shortened, but the frames fb (17) to fb ( Any frame within the period 21) may be thinned out. FIG. 7B shows an example in which the length of time of each frame in the period of the frames fb (21) to fb (25) is increased, but the frames fb (21) to fb ( A frame may be added within the period of 25).

  As described above, the display device 20 (signal processing unit 10) may have a partial omission in the movement of the practitioner in the “B moving image” with respect to the movement of the teacher in the “A moving image” (for example, Even in the case shown in FIG. 7A, the timing of the teacher's movement in the “A moving image” and the practice person's movement in the “B moving image” are matched (for example, as shown in FIG. 7B). Can play).

<Second example>
FIG. 8 is a diagram for explaining a specific example (second example) of the reproduction process. The example shown in this figure shows a case where “time point” that is not included in “A moving image” (exemplary form moving image) is included in “B moving image” (practice form moving image).

  In this figure, “time point” is associated with frames in the order of flag information “2”, “3”, “3”, “2” in “A movie”, whereas “B movie” “Time point” is associated with the frame in the order of flag information “2”, “3”, “3”, “3”, “2”. That is, “time point” of flag information “3” is associated with two frames in “A movie”, whereas “time point” of flag information “3” is 3 in “B movie”. Is associated with one frame. This is the case, for example, when the practitioner makes a mistake in the number of “jumps” (flag information “3”) with respect to the movement (swing) of the model (when jumping once many times). It is.

  FIG. 8A shows an array of “time points” and an array of “frame numbers” for “A moving image” and “B moving image”, respectively. As the “time point” of “A movie”, flag information “2” in frame fa (12), flag information “3” in frame fa (14), flag information “3” in frame fa (17), and frame The flag information “2” is associated with fa (22). On the other hand, as “time point” of “B moving image”, flag information “2” in frame fb (15), flag information “3” in frame fb (17), flag information “3” in frame fb (19), frame Flag information “3” is associated with fb (21), and flag information “2” is associated with frame fb (25).

  FIG. 8B is a diagram showing a result of the collation unit 116 collating the “time points” of “A moving image” and “B moving image” shown in FIG. . The collation unit 116 associates the “time point” of “A moving image” and the “time point” of “B moving image” with the “time point” of the same flag information in the order according to the time flow of the moving image.

  For example, the collation unit 116 associates the “time point” of the frame fa (12) of “A moving image” with the “time point” of the frame fb (15) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (14) of “A moving image” with the “time point” of the frame fb (17) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (17) of “A moving image” with the “time point” of the frame fb (19) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (22) of “A moving image” with the “time point” of the frame fb (25) of “B moving image”. Note that the collation unit 116 does not have the “time point” corresponding to the “time point” in the frame fb (21) of the “B moving image” in the “A moving image”, and therefore the frame fb (21) of the “B moving image”. Do not associate “time point”.

  Based on the collation result of the collation unit 116, the reproduction processing unit 120 reproduces the “time point” frames corresponding to “A moving image” and “B moving image” at the same timing.

  For example, the reproduction processing unit 120 includes the frame fa (12) of “A moving image”, the frame fb (15) of “B moving image”, the frame fa (14) of “A moving image”, and the frame fb (17 of “B moving image”). ), A frame fa (17) of “A movie” and a frame fb (19) of “B movie”, and a frame fa (22) of “A movie” and a frame fb (25) of “B movie”, respectively. "A moving image" and "B moving image" are reproduced and displayed side by side on the display unit 4 so as to be at the same timing. In addition, the playback processing unit 120 matches the playback timing of “B moving image” with “A moving image”, so that the frames “b moving image” fb (17) to fb (19) and frames fb (19) to fb ( Each frame rate in the period 25) is converted.

  8B shows an example in which the length of time of each frame in the period of the frames fb (17) to fb (19) is increased, the frames fb (17) to fb ( A frame may be added within the period of 19). 8B shows an example in which the time length of each frame in the period of the frames fb (19) to fb (25) is shortened, the frames fb (19) to fb ( Any frame within the period of 25) may be thinned out.

  As described above, the display device 20 (signal processing unit 10) has a movement that is not included in the movement of the “A moving image” in the movement of the practitioner of “B moving image” (for example, FIG. 8 (a), the timing of the movement of the model in “A video” and the movement of the practitioner in “B video” are matched (for example, as shown in FIG. 8B). Can play).

<Third example>
FIG. 9 is a diagram for explaining a specific example (third example) of reproduction processing. In the example shown in this figure, “time point” included in “A movie” (example form movie) and “time point” in “B movie” (practice form movie) are partially different ( This shows the case where the characteristics of the movement displacement are different.

  In this figure, “time points” are associated with frames in the order of flag information “4”, “2”, “5”, and “3” in “A movie”, whereas in “B movie” “Time point” is associated with the frame in the order of flag information “4”, “2”, “2”, “3”. That is, the flag information of “time point” is partially different between “A moving image” and “B moving image”. This is because, for example, the practitioner should perform the movement from “moving to stationary” (flag information “5”) after the movement of “displacement direction is reversed” (flag information “2”). By moving the direction of “displacement direction reversed” (flag information “2”) and returning to the position where it should have been stationary and performing “jump” (flag information “ 3)).

  FIG. 9A shows an array of “time points” and an array of “frame numbers” for “A moving image” and “B moving image”, respectively. As the “time point” of “A movie”, flag information “4” in frame fa (13), flag information “2” in frame fa (16), flag information “5” in frame fa (20), and frame The flag information “3” is associated with fa (25). On the other hand, as “time point” of “B moving image”, flag information “4” in the frame fb (12), flag information “2” in the frame fb (15), flag information “2” in the frame fb (22), and The flag information “3” is associated with the frame fb (26).

  FIG. 9B is a diagram showing a result of collation unit 116 collating and matching each “time point” of “A video” and “B video” shown in FIG. 9A. . The collation unit 116 associates the “time point” of “A moving image” and the “time point” of “B moving image” with the “time point” of the same flag information in the order according to the time flow of the moving image.

  For example, the collating unit 116 associates the “time point” of the frame fa (13) of “A moving image” with the “time point” of the frame fb (12) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (16) of “A moving image” with the “time point” of the frame fb (15) of “B moving image”. The collating unit 116 associates the “time point” of the frame fa (25) of “A moving image” with the “time point” of the frame fb (26) of “B moving image”. The collation unit 116 does not have the “time point” corresponding to the “time point” of the frame “f” (22) of “B moving image” in the “A moving image”, and therefore the frame fb (22) of “B moving image”. Do not associate “time point”. In addition, since the “time point” corresponding to the “time point” of the frame fb (20) of the “A moving image” is not included in the “B moving image”, the collating unit 116 determines the frame fb (20) of the “A moving image”. Do not associate “time point”.

  Based on the collation result of the collation unit 116, the reproduction processing unit 120 reproduces the “time point” frames corresponding to “A moving image” and “B moving image” at the same timing.

  For example, the reproduction processing unit 120 includes the frame fa (13) of “A moving image”, the frame fb (12) of “B moving image”, the frame fa (16) of “A moving image”, and the frame fb (15) of “B moving image”. ) And “A movie” and “B movie” are reproduced and displayed so that the frame fa (25) of “A movie” and the frame fb (26) of “B movie” are in the same timing. Part 4 is displayed side by side. In addition, the playback processing unit 120 converts the frame rate of the period “b moving image” from frames fb (15) to fb (26) in order to match the playback timing of “B moving image” with “A moving image”. FIG. 9B shows an example in which the time length of each frame in the period of the frames fb (15) to fb (26) is shortened, but the frames fb (15) to fb ( Any frame within the period of 26) may be thinned out.

  As described above, the display device 20 (signal processing unit 10) has a movement that is not included in the movement of the “A moving image” in the movement of the practitioner of “B moving image” (for example, FIG. 9 (a), the timing of the movement of the model in “A movie” and the exerciser's movement in “B movie” are matched (for example, as shown in FIG. 9B). Can play).

  As described above, as described in the first to third examples, the signal processing unit 10 may have other motions that are partially lost or different in the motion of a predetermined subject in each of the two moving images. Association is performed using feature points of movement displacement. As a result, the signal processing unit 10 determines the timing of the movement of each predetermined subject of the two moving images even if the movement of the predetermined subject of each of the two moving images partially lacks or is different. You can play them together. Therefore, the signal processing unit 10 can reproduce the moving images so that the forms of the subjects recorded in the different moving images can be easily compared.

  In addition, the display device 20 adjusts the timing of the movement of each predetermined subject of the two moving images even if the movement of the predetermined subject of each of the two moving images partially lacks or is different. It is possible to reproduce and display two moving images side by side on the display unit 4. Therefore, the display device 20 can reproduce and display the respective moving images so that the forms of the subjects recorded in the different moving images can be easily compared.

  In addition, although the example which compares the dance form in the said embodiment was demonstrated, the form to compare is not restricted to a dance form. For example, the form to be compared may be a form other than dance such as tennis, golf, gymnastics, baseball, etc., karate type, cheerleading swing, etc.

  The relationship between the time point interval and the number of frames shown in FIG. 3 to FIG. 5 and FIG. 7 to FIG. 9 is merely schematically shown and does not represent the actual speed of movement. Further, the “time point” may be associated with one frame or may be associated with a plurality of continuous frame groups.

[Second Embodiment]
Next, a display device 20 according to a second embodiment of the present invention will be described with reference to FIG. FIG. 10 is a schematic block diagram illustrating an example of the configuration of the timing processing unit 110A included in the display device 20 according to the second embodiment. The timing processing unit 110A shown in this figure is different from the timing processing unit 110 of the first embodiment shown in FIG. 2 in that it has a configuration including an image feature point extraction unit 111. In FIG. 10, components corresponding to those in FIG. 2 are assigned the same reference numerals, and descriptions thereof are omitted.

  The image feature point extraction unit 111 (second extraction unit) extracts feature points of a predetermined subject from the moving image. In the configuration of the second embodiment, the selection unit 112 selects an image region including the feature point of the predetermined subject extracted by the image feature point extraction unit 111. Then, the template registration unit 113 registers the image area selected by the selection unit 112 as a template image.

  That is, the timing processing unit 110A of the second embodiment selects an image region including a predetermined subject, for example, an image region including a subject whose form is to be compared, based on a feature point of the predetermined subject.

  Here, the feature point of a predetermined subject is, for example, a feature point based on the subject's face, hands, feet, or an object (eg, baton, tennis racket, golf club, etc.) that the subject has. is there.

  Thereby, the signal processing unit 10 can appropriately detect a notable displacement (displacement for comparing forms) in a predetermined subject. Therefore, the signal processing unit 10 can reproduce the movement timings of the predetermined subjects of the two moving images in accordance with the feature points of the displacement of the predetermined subjects of the two moving images. Further, the display device 20 can reproduce the two moving images side by side on the display unit 4 by reproducing the two moving images in accordance with the movement timing of a predetermined subject.

[Third Embodiment]
Next, with reference to FIG. 11, the display apparatus 20 by 3rd Embodiment of this invention is demonstrated. FIG. 11 is a schematic block diagram illustrating an example of a configuration of the timing processing unit 110B included in the display device 20 according to the third embodiment. The timing processing unit 110B shown in this figure is different from the timing processing unit 110 of the first embodiment shown in FIG. 2 in that the configuration includes a sound change detection unit 117. In FIG. 10, components corresponding to those in FIG. 2 are assigned the same reference numerals, and descriptions thereof are omitted.

  The sound change detection unit 117 (second detection unit) detects a change in sound between frames in a moving image. For example, the sound change detecting unit 117 detects a change in sound volume (volume), a change in sound type, or the presence or absence of sound. In the configuration of the third embodiment, the change feature point extraction unit 115 (third extraction unit) extracts the feature point of the sound change detected by the sound change detection unit 117. Here, the characteristic points of the sound change are, for example, a point that the change in sound volume (volume) is large, a point that the type of sound changes greatly, a point that the presence or absence of sound changes, and the like. Further, the change feature point extraction unit 115 generates “time point array data” in which the information indicating the feature points of the sound change is “time point” and is arranged in the order of frames in association with the frame number.

  Then, the matching unit 116, for example, the feature point (sound change feature point) in “A movie” extracted by the change feature point extraction unit 115 and the feature point (sound change feature point) in “B movie”. ) To match the first feature point in “A moving image” and the second feature point corresponding to the first feature point in “B moving image”.

  Thereby, the signal processing unit 10 can reproduce the timing of the feature points of the sound changes of the two moving images in accordance with the feature points of the sound changes of the two moving images. In addition, the display device 20 can reproduce and display the two moving images side by side on the display unit 4 by reproducing the timings of the characteristic points of the sound changes of the two moving images. For example, when practicing dance according to music, the signal processing unit 10 adjusts the timing of the feature points of the sound changes of the two moving images to thereby match the predetermined subjects of the two moving images. Can be played at the same timing. Further, the display device 20 can reproduce the two moving images side by side on the display unit 4 by reproducing the two moving images in accordance with the movement timing of a predetermined subject.

  Note that the change feature point extraction unit 115 determines the “time point” based on the feature point of the displacement between frames detected by the displacement detection unit 114 and the feature point of the sound change detected by the sound change detection unit 117. Sequence data "may be generated. Further, the signal processing unit 10 (display device 20) synchronizes the timings of the two moving images based on the feature points of the displacement of the predetermined subjects of the two moving images and the feature points of the sound change. You may replay it.

  Note that the timing processing unit 110B of FIG. 11 may include an image feature point extraction unit 111, similar to the configuration of the timing processing unit 110A of FIG.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described with reference to FIG. FIG. 12 is a schematic block diagram illustrating an example of the configuration of the processing unit 1C included in the signal processing unit 10 according to the fourth embodiment. The processing unit 1C shown in this figure is different from the processing unit 1 of the first embodiment shown in FIG. 1 in that a data generation unit 130 is provided instead of the reproduction processing unit 120. In FIG. 12, the components corresponding to those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

  The data generation unit 130 (generation unit) reproduces the frame image from which the first feature point has been extracted in “A movie” and the frame image from which the second feature point has been extracted from “B movie” at the same timing. The converted moving image (moving image data) is generated.

  For example, the data generation unit 130 sets a frame for “A video” or “B video” so that the number of frames between “time points” corresponding to “A video” and “B video” are the same. Generate video data that has been inserted or thinned out.

  In this way, the signal processing unit 10 (display device 20) generates a moving image in which the reproduction timing of the movement (displacement feature point) of a predetermined subject in each of the “A moving image” and the “B moving image” matches. . The generated video may be at least one of “A video” and “B video”, or a video in which both “A video” and “B video” are collected. There may be.

  As a result, the signal processing unit 10 (signal processing device) reproduces the generated moving image without performing the reproduction processing based on the “time point” and the reproduction processing for converting the frame rate. Reproduction can be performed so that the timing of the movement of the predetermined subject (displacement feature point) matches. In addition, the moving image generated by the signal processing unit 10 (signal processing device) described above is also used in other display devices and playback devices that do not have a function of performing playback processing based on “time points” and playback processing for converting the frame rate. By reproducing the video, it is possible to reproduce (display) the respective moving images so that the timing of the movement of the predetermined subject of each moving image (displacement feature point) matches.

  Note that the processing unit 1C in FIG. 12 may further include the reproduction processing unit 120 in FIG. Further, instead of the timing processing unit 110 in FIG. 12, the timing processing unit 110A in FIG. 10 or the timing processing unit 110B in FIG. 11 may be used.

[Fifth Embodiment]
Next, a fifth embodiment of the present invention will be described with reference to FIG. The fifth embodiment is an aspect of the imaging device 30 including the signal processing unit 10 of any of the first to fourth embodiments. FIG. 13 is a schematic block diagram illustrating an example of the configuration of the imaging device 30. The imaging device 30 includes a display unit 4, a storage medium unit 5, an imaging unit 6, an operation input unit 7, an image processing unit 8, a control unit 9, a signal processing unit 10, and a bus 12. ing. In FIG. 13, the components corresponding to those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

  The imaging unit 6 includes a lens unit 61, an imaging element 62, and an A / D conversion unit 63, and takes a moving image (moving image) or a still image (still image). The imaging unit 6 is controlled by the control unit 9 based on the set imaging conditions (for example, aperture value, exposure, etc.), and bundles the light beam input through the lens unit 61 on the imaging surface of the imaging element 62. Let me image. The image sensor 62 outputs an analog signal obtained by photoelectrically converting the light beam formed on the imaging surface to the A / D converter 63. The A / D converter 63 converts the analog signal input from the image sensor 62 into a digital signal, and outputs the converted digital signal. The lens unit 61 described above may be integrated with the imaging device 30 or may be detachably attached to the imaging device 30.

  For example, the imaging unit 6 captures a moving image in response to a moving image capturing operation on the operation input unit 7 and outputs an image signal of the captured moving image. The imaging unit 6 captures a still image according to a still image capturing operation on the operation input unit 7 and outputs an image signal of the captured still image. Then, the image signal of the moving image and the image signal of the still image captured by the imaging unit 6 are recorded in the storage medium 50 as image data after being subjected to image processing by the image processing unit 8. In addition, the imaging unit 6 displays a display image (for example, displayed on the display unit 4 for confirmation of an angle of view) obtained by continuously shooting at a predetermined interval during a shooting standby state for performing still image shooting or moving image shooting. The image signal of the through image (live view image) that is the image to be output) is output.

  The operation input unit 7 includes, for example, a shutter button, a power switch, a mode dial for selecting an operation mode, a playback button for reproducing a recorded moving image or still image, an enlargement / reduction button, an up / down / left / right selection button, and the like. . The operation mode includes a moving image shooting mode for shooting a moving image, a still image shooting mode for shooting a still image, and the like. The operation input unit 7 detects an operation input operated by the user, and supplies an operation signal indicating the detected operation content to the control unit 9.

  The image processing unit 8 performs image processing such as white balance adjustment, sharpness adjustment, gamma correction, and gradation correction on the image (moving image or still image) captured by the imaging unit 6. The image processing unit 8 executes image compression processing when recording a captured image (moving image or still image) (stored in the storage medium 50).

  The control unit 9 controls each unit included in the imaging device 30. For example, the control unit 9 controls each unit included in the imaging device 30 based on an operation signal received from the operation input unit 7 or a control condition stored in the storage unit 2.

  In addition to the information stored in the template image storage unit 21, the position array data storage unit 22, and the time point array data storage unit 23 illustrated in FIG. For example, predetermined shooting control conditions, image processing conditions, reproduction control conditions, recording control conditions, and the like for controlling 30 are stored and referred to by the imaging unit 6, the image processing unit 8, the control unit 9, and the like. It is good also as a structure. The storage unit 2 may be configured to be used as a work area when the control unit 9 controls the imaging device 30. For example, an image taken by the imaging unit 6 or an image read from the storage medium 50 may be temporarily stored in the storage unit 2 in the course of image processing executed by the image processing unit 8.

  The bus 12 is connected to the display unit 4, the storage medium unit 5, the imaging unit 6, the operation input unit 7, the image processing unit 8, the control unit 9, and the signal processing unit 10, and the image signal output from each unit and the control Transfer signals, etc. Although not shown, the imaging device 30 may be configured to include a microphone that picks up sound (sound) at the time of moving image shooting, and a moving image including the collected sound is stored in the storage medium 50. Also good.

  In the imaging apparatus 30 configured as described above, the signal processing unit 10 can execute a process of reproducing the timings of the two moving images described with reference to the first to third embodiments. Note that the signal processing unit 10 may execute processing for generating moving image data in which the reproduction timings of two moving images described with reference to the fourth embodiment are combined.

  In addition, before the moving image shot by the imaging unit 6 is stored in the storage medium 50, the signal processing unit 10 includes the image feature point extraction unit 111, the selection unit 112, the template registration unit 113, and the displacement detection of the timing processing unit 110. The processing by the unit 114 or the change feature point extraction unit 115 may be performed on the captured moving image. Thereby, the imaging device 30 can generate position arrangement data or time point arrangement data in real time from the moving image taken by the imaging unit 6.

  Note that the imaging device 30 may store the generated position arrangement data or time point arrangement data in the storage unit 2 or the storage medium 50 in association with the moving image.

  In addition, the imaging device 30 may perform the process of the collation unit 116 of the timing processing unit 110 on the captured moving image before storing the moving image captured by the imaging unit 6 in the storage medium 50. For example, the collation unit 116 collates the “time point” extracted in real time from the captured video (for example, “B video”) with the time point of the video (for example, “A video”) stored in the storage medium 50. Thus, the association may be performed.

  In addition, the reproduction processing unit 120 uses a moving image (for example, “A moving image”) stored in the storage medium 50 for a captured moving image (for example, “B moving image”) based on the result of the collation by the matching unit 116 in real time. The frame rate conversion in accordance with the timing of “)” (the timing of movement of the subject) may be performed in real time for reproduction. Further, the data generation unit 130, based on the result of the collation performed in real time by the collation unit 116, the moving image (for example, “A moving image”) stored in the storage medium 50 with respect to the captured moving image (for example, “B moving image”). )) May be generated in real time so as to match the playback timing.

  Thus, the imaging device 30 stores the captured moving image in the storage medium 50 so that the timing of the movement of the predetermined subject (displacement feature point) in each of the moving image stored in the storage medium 50 matches. It is possible to reproduce and display the recorded video and the captured video in real time. In addition, the imaging device 30 performs real-time moving image data obtained by converting the captured moving image and the moving image stored in the storage medium 50 so that the reproduction timing of the movement of the predetermined subject (displacement feature point) matches. Can be generated.

  As described above, the signal processing unit 10, the display device 20, or the imaging device 30 according to the above-described embodiment is configured so that the subject forms (for example, the form of the practice person and the form of the practice person) recorded in different moving images. And) can be easily compared. That is, the signal processing unit 10, the display device 20, or the imaging device 30 can easily compare the forms of the subject.

  In the above-described embodiment, the example of the process of reproducing the “time point” in “B video” so as to be the same timing as “time point” in “A video” has been described. The reproduction may be performed so that the “point” has the same timing as the “time point” in the “B moving image”.

  In the above embodiment, the process of matching the playback timing of “B video” with the playback timing of “A video” has been described. However, the present invention is not limited to the process of matching the playback timing for two videos. It is good also as a process which matches reproduction timing for the above moving image. For example, the signal processing unit 10, the display device 20, or the imaging device 30 sets the playback timings of two moving images (for example, “B moving image” and “C moving image”) to one other moving image (for example, “A moving image”). )) May be performed in accordance with the reproduction timing.

  In the above embodiment, the example in which the template registration unit 113 registers the template image in the template image storage unit 21 has been described. However, the present invention is not limited to this. For example, the template image is stored in the template image storage unit 21 in advance. It may be stored. Further, a template image acquired from an external server via a communication unit (not shown) may be stored in the template image storage unit 21. Further, the template image may be an image included in a face image database used for face detection or information based on the image, for example.

  In the fourth embodiment, the configuration in which the signal processing unit 10 is provided in the display device 20 or the imaging device 30 has been described, but the signal processing unit 10 may be, for example, a mobile phone, a personal computer, a tablet terminal, Alternatively, it may be provided in another device such as an electronic toy.

  The timing processing unit 110 or the reproduction processing unit 120 in FIG. 1, the timing processing unit 110 or the data generation unit 130 in FIG. 12, or the timing processing unit 110 (110A, 110B) in FIGS. Each unit may be realized by dedicated hardware, or may be realized by a memory and a microprocessor.

  The timing processing unit 110 or the reproduction processing unit 120 in FIG. 1, the timing processing unit 110 or the data generation unit 130 in FIG. 12, or the timing processing unit 110 (110A, 110B) in FIGS. Each unit may be realized by dedicated hardware, and each unit described above is configured by a memory and a CPU (central processing unit), and a program for realizing the function of each unit described above is stored in the memory. The function may be realized by loading and executing.

  Further, the timing processing unit 110 or the reproduction processing unit 120 in FIG. 1, the timing processing unit 110 or the data generation unit 130 in FIG. 12, or the timing processing unit 110 (110A, 110B) in FIGS. A program for realizing the function of each unit is recorded on a computer-readable recording medium, and the program recorded on this recording medium is read into a computer system and executed, so that the processing by each unit described above can be performed. Good. The “computer system” here includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a hard disk built in the computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design within a range not departing from the gist of the present invention.

  2 storage unit, 4 display unit, 10 signal processing unit (signal processing device), 20 display device, 30 imaging device, 111 image feature point extraction unit (second extraction unit), 112 selection unit, 113 template registration unit (registration unit) ), 114 displacement detection unit (first detection unit), 115 change feature point extraction unit (first extraction unit, third extraction unit), 116 collation unit, 117 sound change detection unit (second detection unit), 120 reproduction processing , 130 Data generator (generator)

Claims (10)

A first detector for detecting displacement between frames of the moving image;
A first extraction unit for extracting feature points of the displacement detected by the first detection unit;
The feature point in the first moving image extracted by the first extraction unit is collated with the feature point in the second moving image, and the first feature point in the first moving image and the second feature in the second moving image are compared. A matching unit that associates a second feature point corresponding to the point;
A signal processing apparatus comprising: A reproduction process for reproducing the first moving image and the second moving image so that the frame image from which the first feature point is extracted and the frame image from which the second feature point is extracted are reproduced at the same timing. Part,
The signal processing apparatus according to claim 1, comprising: The reproduction processing unit
The frame rate of the first moving image or the second moving image is converted so that the frame image from which the first feature point is extracted and the frame image from which the second feature point is extracted are reproduced at the same timing. The signal processing device according to claim 2. The reproduction processing unit
The signal processing apparatus according to claim 2, wherein the reproduced first moving image and the second moving image are displayed side by side on a display unit. A moving image obtained by converting the first moving image or the second moving image so that the frame image from which the first feature point is extracted and the frame image from which the second feature point is extracted are reproduced at the same timing. The signal processing apparatus according to claim 1, further comprising: a generation unit that generates the signal processing unit. A selection unit for selecting an image area including a predetermined subject from the moving image;
A registration unit for registering an image of the image area selected by the selection unit as a template image;
With
The first detection unit includes:
The template image registered by the registration unit is used to perform template matching on each frame image of the moving image, thereby detecting a displacement between frames of the image area including the predetermined subject. The signal processing device according to any one of claims 1 to 5. The collation unit
Integrating the first feature point and the second feature point and associating the integrated feature point with a frame image corresponding to the first feature point and a frame image corresponding to the second feature point. The signal processing apparatus according to claim 1, wherein the signal processing apparatus is characterized in that: A second detector for detecting a change in sound between frames in the moving image;
A third extraction unit for extracting feature points of the change in sound detected by the second detection unit;
With
The collation unit
The feature point in the first moving image extracted by the third extraction unit is collated with the feature point in the second moving image, and the first feature point in the first moving image and the second moving image The signal processing device according to claim 1, wherein the second feature point corresponding to the first feature point is associated. The signal processing device according to any one of claims 1 to 8,
A display device comprising: On the computer,
A first step of detecting displacement between frames of a moving image;
A second step of extracting feature points of the displacement detected in the first step;
The feature points in the first moving image extracted in the second step are matched with the feature points in the second moving image, and the first feature points in the first moving image and the second moving image are compared. A third step of associating a second feature point corresponding to
A program for running

Images