JP2008262594A - Image association apparatus, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image association apparatus, method and program for automatically and synchronously associating images. <P>SOLUTION: In OHP files and moving image obtained by imaging a lecture using the OHP file with a camera, similarity between each OHP file and an OHP image area included in a frame image constituting a moving image is calculated. The each OHP file and frame image is associated on the basis of the calculated similarity. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention includes a plurality of first images (for example, an OHP (Overhead Projector) file used in a lecture) and a plurality of second images (for example, moving images taken of a lecture composed of a plurality of frame images). The present invention relates to an image association apparatus, an image association method, and an image association program.

  E-learning is being introduced in in-house education and general education. E-learning is a remote education by a computer using the Internet or the like, and is often performed using a WEB screen. There are various types of E-learning, but the lecture that is usually performed at the educational site is best reproduced on a computer as a reproduction of the moving image of the lecture and the OHP image used in the lecture ( Hereinafter, there is content that displays an image displayed by OHP in time synchronization with an enlarged display of an OHP file or an OHP image. By using this content, the student can easily determine on the computer screen which part of the OHP image the instructor is pointing to, and by viewing the enlarged display of the OHP image, Clearly understandable. In other words, such content enables the location indicated by the instructor on the OHP image and details of the content to be displayed at the same time, and provides almost the same environment as actually taking the lecture.

  In order to realize such a system, it is necessary to synchronize temporally the display of the moving image described using the OHP and the enlarged display of the OHP image displayed there. Currently, various E-learning content authoring systems (authoring software) for creating such content have been commercialized.

  However, in such authoring software, an operator manually performs processing for associating the display of the moving image and the teaching material document so that they are synchronized in time. Therefore, there is a problem that it takes a lot of time to create content.

  The present invention has been made in view of the above-described circumstances, and an object thereof is to provide an image association apparatus, an image association method, and an image association program capable of automating synchronous association of images. Yes.

  In order to solve the above-described problem, the present invention is an image association device, which is a plurality of first images and a plurality of second images, at least one of the plurality of second images. Is based on any one of the first images and a second image having a predetermined relevance, and each of the second images has the predetermined relevance with any image of the first image. Relevance determining means for determining whether to have, association means for associating each of the first image and the second image determined to have a predetermined relevance by the relevance determining means, and a second image A frame image thinning means for extracting a frame image at a predetermined frame interval from a plurality of consecutive frame images when the frame image corresponds to each of a plurality of frames constituting a series of moving images; and the frame image Thinning Thinning interval changing means for reducing the frame interval thinned out by the frame image thinning means when two first images extracted by the means and associated with two frame images having a predetermined frame interval are not the same; Points when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, and the first image corresponding to successive frame images is different between the preceding and succeeding frames. Detecting a point, and the relevance determining means determines the relevance of the frame image thinned out by the frame image thinning means to the first image, and the associating means includes: The same first image is associated with each frame image between points detected by the point detecting means. A.

  In addition, the present invention is an image association program, which is a plurality of first images and a plurality of second images, and at least one of the plurality of second images is the first image. A relationship for determining which of the first images has the predetermined relationship with each of the first images based on any one of the images and the second image having the predetermined relationship A determination step, an association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step, and the second image constitute a series of moving images A frame image thinning step for extracting a frame image at a predetermined frame interval from a plurality of consecutive frame images when the frame image corresponds to each of the plurality of frames to be framed; A thinning interval changing step for reducing the frame interval thinned out by the frame image thinning step when two first images extracted by the step and associated with two frame images having a predetermined frame interval are not the same; Points when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, and the first image corresponding to successive frame images is different between the preceding and succeeding frames. A point detection step of detecting the frame image, and the relevancy determination step determines the relevance of the frame image thinned out by the frame image thinning step to the first image, and the association step Each frame image between the points detected by the point detection step It is characterized in that associating the same first image.

  Furthermore, the present invention is an image association method, wherein a plurality of first images and a plurality of second images, wherein at least one of the plurality of second images is the first image. A relationship for determining which of the first images has the predetermined relationship with each of the first images based on any one of the images and the second image having the predetermined relationship A determination step, an association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step, and the second image constitute a series of moving images A frame image thinning step for extracting a frame image at a predetermined frame interval from a plurality of consecutive frame images when the frame image corresponds to each of the plurality of frames to be performed; A thinning interval changing step for reducing the frame interval thinned out by the frame image thinning step when the two first images extracted by the step and associated with the two frame images having a predetermined frame interval are not the same; When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, the first image corresponding to successive frame images is different between the preceding and following frames. A point detecting step for detecting points; and the relevancy determining step determines relevance of the frame image thinned out by the frame image thinning step to the first image, and the associating step includes The same first image is added to each frame image between the points detected by the point detection step. It is characterized in that the attached response.

  In the present invention, the computer-readable medium is a portable storage medium such as a CD-ROM, a flexible disk, a DVD disk, a magneto-optical disk, an IC card, a database holding a computer program, or other It includes computers and their databases, as well as transmission media on the line.

  Conventionally, in order to create content for E-learning, the content creation authoring system can synchronize and display moving images such as lectures using OHP and the OHP images displayed there. The image is manually associated by the operator. However, according to the present invention, since this association can be automated, there is an effect that the time required for the association can be greatly shortened without taking time and effort. Play.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a conceptual diagram of an embodiment of the present invention. First, when the lecture 2 using OHP is used as an E-learning teaching material, the video of the lecture is photographed 3 as a moving image and stored in the image association / content creation apparatus 1. Further, the OHP file 4 used for the lecture is stored in the same device (image association / content creation device 1).

  The image association / content creation apparatus 1 uses various image processing techniques to determine the replacement timing of OHP in a moving image, and indicates which frame of the moving image the original OHP image corresponds to Create a table. Further, the image association / content creation apparatus 1 creates E-learning content 5 for reproducing a moving image and an OHP image in synchronization with each other based on the correspondence table. As a result, it is possible to automate the association between a moving image and each page of OHP, which has been performed manually, and content can be easily created. In the image association / content creation apparatus 1, the part for creating the “correspondence table” is the part for explaining the present invention, but here, the apparatus also has a function for creating content from the created correspondence table. This will be described in detail by taking as an example.

Embodiment 1 FIG.
FIG. 2 is a diagram illustrating an example of a functional configuration of the image association / content creation apparatus 1 described above. In the figure, an image association / content creation apparatus 1 is composed of an image association apparatus 1a and a content creation apparatus 1b. The image association apparatus 1a includes a moving image input unit 101, a moving image storage unit 102, an OHP file input unit 103, an OHP file storage unit 104, a frame acquisition unit 105, an OHP region extraction unit 106, and an OHP acquisition. Unit 107, similarity calculation unit 108, association processing unit 109, and association table 110.

  The moving image input unit 101 inputs a moving image such as a lecture taken by the camera to the image association / content creation apparatus 1. The moving image storage unit 102 stores the moving image input by the moving image input unit 101 in the AVI format or the like. The OHP file input unit 103 inputs an OHP file to the image association / content creation apparatus 1 from another computer system or the like. The OHP file storage unit 104 stores an image file for each page of the OHP file input by the OHP file input unit 103 and character information corresponding to the image file.

  The frame acquisition unit 105 acquires frame images constituting the moving image from the moving image storage unit 102. The frame acquisition unit 105 includes an OHP region extraction unit 106. The OHP region extraction unit 106 searches and extracts a region in which an OHP image appears in the frame image. The OHP acquisition unit 107 acquires an OHP image for each page from the OHP file storage unit 104. The similarity calculation unit 108 calculates the similarity between the frame image of the moving image acquired by the frame acquisition unit 105 and the OHP image acquired by the OHP acquisition unit 107. The association processor 109 determines the association between the frame image and the OHP image from the similarity calculated by the similarity calculator 108, and the association table 110 displays the association determined by the association processor 109. Remember.

  In addition, the content creation device 1b includes a content creation unit 111 and a content storage unit 112. The content creation unit 111 refers to the association between the frame image and the OHP image from the association table 110 and creates content for E-learning. The content storage unit 112 stores the content created by the content creation unit 111.

  FIG. 3 is a diagram illustrating an example of a hardware configuration of the image association / content creation apparatus 1. In the figure, an image association / content creation device 1 includes a video input control device 11, a CPU (Central Processing Unit) 12, a memory 13, an image HDD (Hard Disk Drive) 15, and a file input / output device 14. The program HDD 16, the content HDD 17, the association table HDD 18, the OHP file HDD 19, a LAN (Local Aria Network) interface 20, and a CD-ROM drive 21.

  In the present embodiment, each functional block shown in the functional configuration diagram of FIG. 2 is a program group stored in the program HDD 16 of FIG. 3 expanded on the memory 13 when the image association / content creation apparatus 1 is activated. The state is shown and executed by the CPU 12. However, the moving image input unit 101 or the OHP file input unit 103 in FIG. 2 cooperates with the program group developed on the memory 13 by the video input control device 11 or the file input / output control device 14 in FIG. Function. The moving image storage unit 102 includes an image HDD 15, the OHP file storage unit 104 includes an OHP file HDD 19, the association table 110 includes an association table HDD 18, and the content storage unit 112 includes a content HDD 17. Shall be.

  Note that the program group is not stored in the program HDD 16 but is stored in a computer-readable medium such as a portable storage medium such as a CD-ROM, FD, magnetic disk, DVD disk, magneto-optical disk, or IC card. It may also be downloaded from a database holding a computer program, or another computer and its database, or a transmission medium on a line, etc., expanded on the memory 13, and executed. The installation form of the program group is not limited. As described above, in this embodiment, the CD-ROM drive 21 is provided, and a portable storage medium can be read. Further, the LAN interface 20 and the file input / output device 14 can easily download a program group from another computer or database via the LAN.

  Hereinafter, the flow of the association process of the image association apparatus 1a will be described in detail using a flowchart. FIG. 4 is a flowchart showing a processing flow of the entire association processing. First, the frame acquisition unit 105 extracts the first frame image of the moving image from the moving image storage unit 102 (S1000). The OHP area extraction unit 106 searches and extracts an area in which the OHP image appears in the extracted frame image (hereinafter referred to as an OHP area image) (S1001). Details of the OHP area extraction process will be described below.

  FIG. 5 is a flowchart showing the flow of the OHP region image extraction process. First, frame image data is input to the OHP region extraction unit 106 (S1100). Next, the frame image is binarized, and edge extraction from the frame image is performed (S1101). Subsequently, a Hough transform is performed on the extracted edge to extract a straight line (S1102). The extracted linear components are integrated to create a rectangle (S1103). Only rectangles larger than a predetermined size are extracted from the rectangles (S1104). If a rectangle is not extracted by this extraction process (S1105, NO), it is determined that the entire frame image is an OHP image area, and the entire image is extracted (S1107). When a rectangle is extracted (S1105, YES), the largest extracted rectangle is extracted as an OHP area image (S1106).

  Usually, in a moving image obtained by taking a lecture using OHP, the OHP image is generally the largest rectangle, and the region is extracted in the processing of S1106.

  After extracting the OHP area image, 0 is set as the initial value for the maximum similarity (FIG. 4, S1002). Subsequently, the OHP acquisition unit 107 extracts the first page of the OHP image from the OHP file storage unit 104 (S1003). Here, the similarity calculation unit 108 collates the first page of the OHP image extracted in S1003 with the image of the OHP area of the frame extracted in S1001, and calculates the similarity (S1004). Hereinafter, the similarity calculation will be described in detail with reference to the drawings.

  FIG. 6 is a diagram illustrating an example of a functional configuration of the similarity calculation unit 108. In the figure, the similarity calculation unit 108 includes an image normalization unit 120, a frame pixel extraction unit 121, an OHP pixel extraction unit 122, a pixel matching unit 123, and a matching result addition unit 124. The image normalization unit 120 corrects the inclination by executing a rotation process so that the OHP region image extracted from the frame image matches the direction of the OHP image. If the size is different from that of the OHP image, an enlargement process is executed to correct the size to be the same as the OHP image. Such normalization processing may be executed by the OHP region extraction unit 106 at the time of extraction in S1106 of FIG. 5, and is not particularly limited.

  The frame pixel extraction unit 121 extracts pixels from the OHP region image extracted in S1106 of FIG. The OHP pixel extraction unit 122 extracts the pixels of the page acquired by the OHP acquisition unit 107. The pixel collation unit 123 collates the pixels extracted from the frame pixel extraction unit 121 and the OHP pixel extraction unit 122. The collation result adding unit 124 converts the results collated by the pixel collation unit 123 into numerical values and adds them. The result of addition is the similarity.

  FIG. 7 is a flowchart showing the flow of similarity calculation processing executed by the similarity calculation unit 108. First, an OHP region image is acquired from the frame acquisition unit 105 (S1200), and the acquired OHPb region image is normalized by the image normalization unit 120 expanding it to the original document size, that is, the size of the OHP image (S1201). . At the same time, an OHP image is acquired from the OHP acquisition unit 107 (S1201 '). Subsequently, the pixel collating unit 123 collates the pixel of the OHP region image extracted by the frame pixel extracting unit 121 and the pixel of the OHP image extracted by the OHP pixel extracting unit 122 to obtain a pixel difference. This difference is added by the matching result adding unit 124 to calculate the similarity (S1202). An example of the similarity calculation formula is shown below.

Similarity = 1 / V
(V = sum of absolute values of corresponding pixel differences)
The smaller V is, that is, the smaller the pixel difference is, the higher the similarity is. However, when V is 0, it is determined that the similarity has reached the maximum value unconditionally.

The similarity calculation is not limited to the above method. Hereinafter, another method will be described.
FIG. 8 is a functional configuration diagram of the similarity calculation unit 108a for realizing another method. In this method, similarity is calculated using character information. Here, the similarity is calculated by collating whether or not two consecutive characters are the same. In the figure, the similarity calculation unit 108a includes an image normalization unit 130, a frame character extraction unit 131, an OHP character extraction unit 132, a character collation unit 133, and a collation result addition unit 134. The image normalization unit 130 has the same function as the image normalization unit 120 of FIG.

  The frame character extraction unit 131 recognizes and extracts characters from the OHP area image extracted in S1106 of FIG. The OHP character extraction unit 132 extracts the characters of the page acquired by the OHP acquisition unit 107. In the present embodiment, the character information of the OHP image is already stored separately in the OHP file storage unit 104 for each page and the corresponding character information, so there is no need to perform character recognition processing. Characters can be extracted simply by referring to the OHP file storage unit 104. The character collating unit 133 collates the characters extracted from the frame character extracting unit 131 and the OHP character extracting unit 132. The collation result adding unit 134 digitizes and adds the results collated by the character collating unit 133. The result of addition is the similarity.

  FIG. 9 is a flowchart showing the flow of similarity calculation processing executed by the similarity calculation unit 108a. First, an OHP area image is acquired from the frame acquisition unit 105 (S1300), and the frame character extraction unit 131 performs a character recognition process (S1301). This character recognition process is performed using a known technique. FIG. 10 is a flowchart showing details of the character recognition process. First, the obtained OHP region image is binarized (S1310), and region identification is performed (S1311). Subsequently, character recognition is performed (S1312), and the recognition result is output as text (S1313).

After the character recognition process, the initial value of similarity is set to 0 (FIG. 9, S1302). Next, the first character of the recognition result by the frame character extraction unit 131 is set as a reference character. In addition, the OHP character extraction unit 132 extracts a character at a position corresponding to the set reference character and an adjacent character in the OHP image (S1303). Here, the character collation unit 133 compares the character string recognized by the frame character extraction unit 131 with the character string extracted by the OHP character extraction unit 132, and collates whether or not the two consecutive characters are the same. To do. If they are the same (S1304, YES), the collation result adding unit 134 increments the similarity (S1305). If they are not the same (S1304, NO), the similarity is not incremented. Subsequently, the adjacent character is set as the reference character and collation is performed in the same manner (S1306). The processes from S1304 to S1306 are repeated until the character as a result of the character recognition process ends (S1307, YES). Finally, the matching result adding unit 134 corrects the similarity as follows (S1308).
Similarity = similarity / number of OHP characters

  Note that the overall similarity may be calculated using both the similarity calculation processing using the pixels shown in FIG. 6 and the similarity calculation processing using the characters shown in FIG. FIG. 11 is a functional configuration diagram of the similarity calculation unit 108b for realizing such a method. In the figure, the similarity calculation unit 108b includes a pixel matching result calculation unit 140 having the similarity calculation mechanism of FIG. 6 and a character matching result calculation unit 141 having the similarity calculation mechanism of FIG. An overall similarity calculation unit 142 that calculates the overall similarity by adding the similarities is provided.

  In addition to the above-described method, a method for calculating similarity using layout information will be described. FIG. 12 is a diagram illustrating a functional configuration of the similarity calculation unit 108c for realizing such a method. In the figure, the similarity calculation unit 108c includes an image normalization unit 150, a frame layout acquisition unit 151, an OHP layout acquisition unit 152, a layout verification unit 153, and a verification result addition unit 154. The image normalization unit 150 has the same function as the image normalization unit 120 of FIG.

  The frame layout acquisition unit 151 acquires layout information from the OHP area image extracted in S1106 of FIG. The OHP layout acquisition unit 152 acquires page layout information acquired by the OHP acquisition unit 107. Here, region attributes such as diagrams, tables, and character string blocks and their coordinate information are referred to as layout information. The layout collation unit 153 collates the layout information acquired by the frame layout acquisition unit 151 and the OHP layout acquisition unit 152. The collation result adding unit 154 digitizes and adds the results collated by the layout collating unit 153. The result of addition is the similarity.

  FIG. 13 is a diagram showing an example of layout information. FIG. 13A shows an OHP image, and the region attribute and its coordinate information shown in the image are acquired by the OHP layout acquisition unit 152. FIG. 13B shows an image captured from a frame image, and the region layout and coordinate information shown in the image are acquired by the frame layout acquisition unit 151.

  In this method, the layout collation unit 153 collates both pieces of layout information, and as a result, the collation result adding unit 154 calculates the similarity. Specifically, if the region attribute is the same and the absolute value of the difference between the region coordinates is smaller than a predetermined value, the layout matching unit 153 determines that the region is a corresponding region. A predetermined value is output to the unit 154 and added as a similarity.

  For example, the region coordinates of a certain character string region extracted from the OHP image as shown in FIG. 14A are (xmin, ymin)-(xmax, ymax), and the moving image as shown in FIG. 14B. If the region coordinates of the character string region of the image captured from (xmin ', ymin')-(xmax ', ymax') and the predetermined value is α, the similarity is satisfied if the following conditions are satisfied: Is incremented.

| Xmin-xmin '| <α and | ymin-ymin' | <α and | xmax-xmax '| <α and | ymax-ymax' | <α

  The layout collation unit 153 performs such collation for all regions and checks the region attributes and coordinate differences. For example, 1 is output to the collation result adding unit 154 when the condition is satisfied, and 0 is output as the similarity when the condition is not satisfied. In other words, in this method, the number of regions with correspondence becomes the similarity.

  In addition to the method described above, a method for calculating the similarity by adding a character string feature to the layout information and performing collation will be described. The character string characteristics include coordinate information of the rectangle when the character string block is further divided into character strings of a certain ratio or more, the character size of the character string, the shape of the character string, the color of the character string, and the like. For example, the character string area data such as the character string area data extracted from the OHP image shown in FIG. 15A and the character string area data of the image captured from the moving image shown in FIG. Then, a character string area having a certain area or more is divided and extracted as lines 1 to N and lines 1 ′ to N ′, and coordinate information for each line is acquired as character string information and collated. Also good.

Here, the layout matching unit 153 checks for each row whether the slip difference is equal to or less than a predetermined value (α).
For example, in row 1 and row 1 ′
| Xmin1-xmin1 '| <α and | ymin1-ymin1' | <α and | xmax1-xmax1 '| <α and | ymax1-ymax1' | <α
It is checked whether the condition is satisfied. Furthermore, collation is performed for all rows, and 1 is output to the collation result adding unit 154 when the same condition is satisfied for all the rows, and 0 is added to the collation result adding unit 154 otherwise.

  FIG. 16 shows an example in which the layout information includes color information of the character string. FIG. 16A shows character string region data extracted from an OHP image, and FIG. 16B shows character string region data of an image captured from a moving image. As shown in the figure, representative colors for each row are acquired as layout information, and these are also collated. When there are a plurality of character colors in one line, the representative color is represented by a three-dimensional vector representation of (R, G, B) in the line layout in the frame layout acquisition unit 151 or the OHP layout acquisition unit 152 for clustering. A plurality of clusters are obtained so that the distance between the clusters is equal to or greater than a predetermined value, an average color is obtained, and the obtained average color is used as a representative color.

  Here, the layout collation unit 153 checks for each row whether the slip difference is a predetermined value (α) or less and whether the color difference of the row is a predetermined value (β) or less.

For example, in row 1 and row 1 ′,
| Xmin1-xmin1 '| <α and | ymin1-ymin1' | <α and | xmax1-xmax1 '| <α and | ymax1-ymax1' | <α
Or,
| R-R '| <β and
| G-G '| <β and
| B-B '| <β
Is checked. Similarly, all rows are collated.

  If the same condition is satisfied in all the rows, 1 is output to the collation result adding unit 154 otherwise, and is added as the similarity.

  In addition to the method described above, a method may be used in which only the title characters in the image are extracted and collated to calculate the similarity. FIG. 17 shows a functional configuration diagram of the similarity calculation unit 108d for performing the method. In the figure, the similarity calculation unit 108d includes an image normalization unit 160, a frame title extraction unit 161, an OHP title extraction unit 162, a title verification unit 163, and a verification result addition unit 164.

  The frame title extraction unit 161 extracts a title from the OHP area image. The OHP title extraction unit 162 extracts the title information of the page acquired by the OHP acquisition unit 107. The title collating unit 163 collates the titles acquired by the frame title extracting unit 161 and the OHP title extracting unit 162. The collation result calculation unit 164 digitizes the result collated by the title collation unit 163 and calculates the similarity.

  FIG. 18 is a flowchart clearly showing the title extraction process and the title collation process. First, the OHP title extraction unit 162 and the frame title extraction unit 161 first extract a character string region from an image 190a directly created from an OHP or an image 190b captured from a moving image, and in a plurality of extracted character string regions, respectively. The character string area having the maximum size is extracted as a title (S1400). In the present embodiment, it is assumed that the extracted title information is as shown in S1401. Next, collation processing is performed by the title collation unit 163 (S1402). At that time, if the absolute value of the coordinate difference is within a predetermined value, the colors match, and further, a certain percentage or more of the character contents match, OK is indicated, otherwise NG is indicated, and the collation result calculation unit To 164. The collation result calculation unit 164 digitizes the input value and sets the similarity to 1000 if OK and 0 to NG.

  Alternatively, a method of calculating similarity by integrating a method of acquiring and collating layout information and a method of collating pixels may be executed. FIG. 19 is a functional configuration diagram of the similarity calculation unit 108e for performing the method. The OHP layout acquisition unit 171 acquires an image of a certain region (for example, region I) extracted from the OHP image. Further, the frame layout acquisition unit 170 acquires an image of a region (for example, region J) in the image extracted from the moving image. After normalizing the region J to the size of the region I by the image normalization unit 172, the pixel collation unit 175 collates the pixels extracted by the frame region pixel extraction unit 173 and the OHP region pixel extraction unit. The pixel difference is output to the matching result calculation unit 176, and the matching result calculation unit 176 calculates the sum d of the absolute values of the differences between them, and further calculates the similarity 1 / d.

  As described above, the degree of similarity is calculated by various methods. However, the present invention is not particularly limited to any method, and any method may be used, or may be calculated using a plurality of the above methods. Good.

  Note that the similarity calculation units 108 to 108e described above (hereinafter referred to as the similarity calculation unit 108) calculate the similarity between all OHP images for an OHP region image of a certain frame. Specifically, in FIG. 4, after the similarity of the first page of the OHP image is calculated (S1004), if the similarity is greater than or equal to the maximum similarity (0 in the first stage) (YES in S1005). The similarity is set as the maximum similarity, and the page number of the OHP image is recorded (S1006). If collation of all the pages of the OHP image has not been completed at this time (S1007, NO), the next page of the OHP image is extracted, and the similarity calculation process of S1004 is repeated. When the collation of all pages of the OHP image is thus completed (S1007, YES), the similarity calculation unit 108 associates the frame number that is the processing target at that time with the page number of the OHP image having the maximum similarity. It outputs to 109 (S1008). Here, if the processing has not been completed for all the frames in the moving image (S1009, NO), the frame acquisition unit 105 extracts the next frame image and repeats the processing from S1001 to S1009.

  When the collation processing for all the frames is completed, the correspondence processing unit 109 creates a frame number / OHP number correspondence table (hereinafter referred to as a correspondence table) as shown in FIG. As is clear from the figure, the correspondence between each frame number and the OHP image having the highest similarity is shown. The association processing unit 109 uses this correspondence table to create a table (hereinafter referred to as a section correspondence table) indicating the frame section corresponding to the OHP image as shown in FIG. As described above, if the frame section of the moving image is associated with the OHP image, the efficiency in creating the content for E-Learning increases. The entry numbers in FIG. 20 and FIG. 21 mean indexes assigned to the records in the table.

  FIG. 22 is a flowchart showing details of the section correspondence table creation processing. In this flowchart, the entry numbers in FIG. 20 are counted using I and J. First, the correspondence table of FIG. 20 is read (S1500). 0 is set to I (S1501). Subsequently, a record corresponding to entry I of the correspondence table is extracted. The frame number extracted at this time is recorded as the start frame number (S1502). Subsequently, I + 1 is set to J (S1503). Then, it is determined whether J has reached the number of entries, that is, whether it has reached N in FIG. 20 (S1504). If J has not reached N (S1504, YES), a record corresponding to J is extracted (S1505). It is determined whether or not the OHP number extracted at this time matches the I OHP number (S1506). If they match (S1506, YES), J is incremented (S1507), and the processing from S1504 to S1506 is repeated. By this process, the records are searched in the order of the entry numbers, and when the records having different OHP numbers are reached (S1506, NO), it is determined that the frame number of the previous record corresponds to the same OHP image, The frame number of J-1 is recorded as the end frame number of the I OHP number (S1508).

  In the case of FIG. 20, since frame number 0 and frame number 1 correspond to OHP number 0, the start frame number of OHP number 0 is 0 and the end frame number is 1 as shown in FIG.

  Next, J is set to I (S1509), and if I has not reached N (S1510, YES), the frame section corresponding to the number of the next OHP image is subsequently searched. In other words, at this time, since 2 is set in J, 2 is set in I. Subsequently, the processing after S1502 is similarly performed. Since the frame number of entry number 2 is 2, 2 is recorded as the start frame number. Similarly, the end frame number is searched in order while incrementing J. In this way, the section correspondence table as shown in FIG. 21 is created.

  In the section correspondence table created in this way, when an extremely short frame section exists, and when the OHP image numbers of the frames before and after such a frame section are the same, the association in the frame section is performed. Is likely not appropriate. For example, when a failure occurs temporarily in a moving image and an OHP region image is not clearly captured, appropriate similarity calculation is not performed in similarity calculation, and only that part is incorrectly associated. It is because there are things. In particular, when a moving image is a lecture, OHP images are often used in the order of page numbers, and therefore the above-described association between frame sections is determined to be inappropriate. Assuming such a case, in the present embodiment, after creating the table in FIG. 21, the association processing unit 109 deletes the association (entry) of the short frame section and moves the frame section to the front and back To be integrated into the frame interval.

FIG. 23 is a flowchart showing the flow of such association change processing. In this flowchart, the entry number in FIG. First, the frame section table of the OHP number in FIG. 21 is read (S1700). Subsequently, 0 is set to I (S1701). Subsequently, it is checked whether or not I is an entry number that has already been deleted in this association changing process (S1702). If it has not been deleted (S1702, NO), a record whose entry number corresponds to I is extracted. Further, the frame length of I is calculated by the following equation (S1703).
Frame length of I = end frame−start frame + 1

  If the calculated frame length is less than the predetermined value (S1704, YES), it is verified whether the OHP numbers of the undeleted entries before and after I match (S1705). If they match (S1705, YES), the I entry is marked for deletion, and the end frame number of the entry immediately before I is changed to the end frame number of the entry immediately after I. Further, the entry immediately after I is also marked for deletion (S1706). By this deletion mark, the short frame section is finally integrated into the preceding and subsequent frame sections, and the record of the corresponding section and the entry number immediately thereafter is deleted in the section correspondence table of FIG. Subsequently, the process proceeds to S1707.

  If the OHP numbers do not match in S1705 (S1705, NO), the process immediately proceeds to S1707 and subsequent steps. In addition, the case where it does not correspond shows the case where matching is appropriate even if it is a case where a frame area is shorter than a predetermined value.

  After incrementing I in S1707, it is checked whether I exceeds the total number of entries (M). If I is equal to or less than the total number of entries (S1708, NO), the processing from S1702 onward is repeated to check the frame sections of all the entries and their correspondences. If I exceeds the total number of entries (S1708, YES), it is determined that all entries have been confirmed, and the entry marked for deletion is deleted (S1709).

  The section correspondence table created in this way is stored in the association table 110 by the association processing unit 109. The content creation device 1b refers to the association table, creates content for E-learning using existing technology, and stores the content in the content storage unit 112.

  As described above, according to the present embodiment, the association between the OHP image and the moving image is automatically and easily performed, and thereby content creation that has been performed manually in the past can be performed quickly and easily. It becomes.

Embodiment 2. FIG.
With reference to the functional configuration diagram of FIG. 2 shown in the first embodiment and the hardware configuration diagram of FIG. 3, an embodiment in which a device is further added to the association processing will be described below. In the present embodiment, a threshold is set for the similarity, and when the similarity is equal to or higher than a certain similarity, the same processing as in FIG. If the similarity is low, the OHP image is associated with the frame image immediately before or after the frame image.

Details of the above processing will be described in detail with reference to the drawings.
FIG. 24 is a flowchart showing in detail the flow of association processing of the image association apparatus 1a of the present embodiment.

  Since S1800 to S1804 are processed in the same manner as in the first embodiment, description thereof is omitted. In S1805, when the similarity is equal to or greater than a predetermined value (S1805, NO), the similarity calculation unit 108 performs the same processing as in the first embodiment (S1806 to S1808). When the similarity is less than a predetermined value (S1805, YES), the maximum similarity value is not changed and collation with the next OHP image is performed. The processing from S1804 to S1808 is performed between all OHP images.

  When the collation processing is completed for one frame image (S1808, YES), the similarity calculation unit 108 associates the frame number that is the processing target at that time with the page number of the OHP image that has the highest degree of similarity. At this time, if the maximum similarity is 0, it is assumed that no correspondence is found with any OHP image, and the OHP number is output as −1 (S1809). As a result, an entry with a frame number of -1 appears in FIG.

  FIG. 25 is a flowchart showing a flow of processing performed before the association processing unit 109 performs the section correspondence table creation processing of FIG. 22 in Embodiment 1 in the present embodiment. First, a correspondence table as shown in FIG. 20 is read (S1900), and 0 is set in entry I. Next, I is incremented (S1903) and checked in order, and an entry I whose OHP number is not -1 is searched (S1902). For an entry I whose OHP number is not -1, I is set to the variable K, the OHP number of the entry K is set to the variable OHP1, and the OHP numbers of all the entries from 0 to K in the correspondence table are set to OHP1. Change (S1904).

  Next, K + 1 is set to I (S1905), and then an entry I having an OHP number of -1 is searched (S1906). If the OHP number of the I entry is not −1 (S1906, NO), the OHP number of the entry I is set in OHP1 (S1907). Further, OHP1 is set to the OHP number of entry I, and I is incremented (S1908). If the OHP number of the I entry is −1 (S1906, YES), the value of OHP1 that has already been set is set to the OHP number of entry I, and I is incremented (S1908).

  That is, the OHP number of the frame number that is -1 in the correspondence table by this processing is replaced with an OHP number other than -1 immediately before the entry (S1905 to S1908). When −1 appears in an entry with I = 0, it is replaced with an OHP number other than −1 immediately after the entry (S1902 to S1904). This process loops until I reaches the total number of entries.

  After deleting the OHP number −1 from the correspondence table in this way, the section correspondence table is completed by performing the processing of FIG.

Embodiment 3 FIG.
Another embodiment in which a device is further added to the association process will be described using the functional configuration diagram of FIG. 2 shown in the first embodiment and the hardware configuration diagram of FIG. 3. In the present embodiment, instead of collating all frame images and OHP images, frame images are extracted at predetermined intervals, the extracted frame images are collated with OHP images, and associated with the same OHP image. All the frame images existing between two frames are associated with the same OHP image. As a result, processing efficiency is improved.

  FIG. 26 is a flowchart showing in detail the flow of association processing of the image association apparatus 1a of the present embodiment. FIG. 27 shows an example of a specific collation process performed along the flowchart of FIG. The upper row shows the frame number, and the lower row shows the OHP number having the maximum similarity obtained as a result of matching with the frame number. Here, for convenience, the corresponding OHP number is already specified, but the number is obtained by actually performing the collation process, and is unknown before the start of the process shown in FIG. To do.

  The frame extraction process in FIG. 26 is performed by the frame acquisition unit 105, and the collation process (similarity calculation process) is performed by the similarity calculation unit 108, but all other processes are performed by the association processing unit 109. To do.

  First, in FIG. 26, the first frame number (0) of the moving image is extracted. Also, 0 is set in the variable “OHP number 1”. (S1600). Next, the frame number (0 at the first time point) is set to the variable “Start frame of OHP number 1” and the variable “Start frame”, and the variable “End frame” has a predetermined interval predetermined in the Start frame. A value obtained by adding (INTERVAL) is set (S1601). As a result, a frame is extracted for each interval, and collation processing is performed.

  In FIG. 27, INTERVAL is set to 5, and a frame number extracted by a downward arrow is shown. The Start frame in S1601 is indicated by the frame number in (1) column, and the End frame is indicated by the frame number in (2) column.

  Subsequently, the end frame (extracted frame) is collated with the OHP image (S1602). Since this collation processing is the same as the processing from S1001 to S1007 in FIG. Further, the OHP number having the maximum similarity obtained by this collation processing is set in the variable “OHP number 2” (S1602).

  Next, it is verified whether OHP number 2 and OHP number 1 are equal. In the case of FIG. 27, since OHP number 2 and OHP number 1 are equal to 0 (S1603, YES), an End frame is subsequently set in “End frame of OHP number 1” and “Start frame”. A value obtained by adding INTERVAL to the current End frame is set in “Frame” (S1604). At this time, the Start frame has the frame number at the position indicated by (2), and the End frame has the frame number at the position indicated by (3).

  Subsequently, the process returns to S1602, and the matching process is performed again. While the OHP number 1 and the OHP number 2 are the same, the processing from S1602 to S1604 is repeated. In the example ((2) and (3)) shown in FIG. 27, since the OHP number 1 is 0 and the OHP number 2 is 1, they are not the same (S1603, NO), and the processing of S1605 is not performed. Transition. In S1605, the End frame at that time is set in the variable “End frame 2”, and Start frame + (End frame 2−Start frame + 1) / 2 is set in “End frame”. That is, the frame at the position obtained by dividing INTERVAL by 2 becomes the next End frame. Note that (End frame 2−Start frame + 1) / 2 is rounded down. In the example of FIG. 27, the End frame has the frame number indicated at the position (4).

  Next, it is checked whether or not the End frame 2 is the same as the End frame (S1606). If they are not the same, next, a matching process between the current End frame and the OHP image is performed (S1607). The OHP number obtained by this collation process is set to “OHP number 2”. Here, it is verified whether or not the OHP number 2 and the OHP number 1 are equal (S1608). In the example shown in FIG. 27, since the OHP number 1 is 0 and the OHP number 2 is 1, they are not the same (S1608, NO), and the process proceeds to S1610. In S1610, an End frame is set in “End frame 2”, and Start frame + (End frame 2−Start frame + 1) / 2 is set in “End frame”. At this time, the End frame is the frame number at the position indicated by (5).

  Next, returning to S1606, it is checked again whether End frame 2 and End frame are the same. In the example of FIG. 27, since the End frame 2 is 8 and the End frame is 7 at this time (S1606, NO), the collation process of S1607 is performed. As a result, since OHP number 2 and OHP number 1 are equal (S1608, YES), the process proceeds to S1609. In S1609, the End frame is set in “End frame of OHP number 1” and “Start frame”, and Start frame + (End frame 2−Start frame + 1) / 2 is set in “End frame”.

At this time, the start frame number is set to 0 and the end frame number is set to 1, and this value is output to the entry whose OHP number is 0 in the section correspondence table. Further, since End frame 2 and End frame are 7 (YES in S1606), the process proceeds to S1611 in S1606, End frame 2 is set in the frame number, OHP number 1 is incremented, and the process returns to S1601. .
By repeating the above processing, the section correspondence table of FIG. 21 can be obtained without matching all the frame images, and can be associated with the OHP image.

  While various embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can of course be applied without departing from the scope of the present invention.

(Supplementary Note 1) A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined association with any one of the first images Relevance determining means for determining which of the first images each of the second images has the predetermined relevance on the basis of the second image having sex;
Association means for associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination means;
An image association apparatus comprising:
(Supplementary Note 2) In the image association apparatus according to Supplementary Note 1,
When the predetermined relationship is that one of the second images includes one of the images corresponding to the first image,
The relevance determining means includes
Predetermined area extraction means for extracting a predetermined area from the second image;
An image association apparatus comprising: a corresponding image determination unit that determines which image of the first image the image of the predetermined region extracted by the predetermined region extraction unit corresponds to.
(Supplementary Note 3) In the image association apparatus according to Supplementary Note 2,
The corresponding image determining unit includes a similarity calculating unit that calculates a similarity between the image of the predetermined region and the first image, and the corresponding image determining unit stores the predetermined region based on the similarity calculated by the similarity calculating unit. An image association apparatus for determining which image of the first image corresponds to the included image.
(Supplementary Note 4) In the image association apparatus according to Supplementary Note 3,
The similarity calculation means calculates a similarity between all the images of the first image with respect to the image of the predetermined area, and associates the first image with the image of the predetermined area based on the similarity An image association apparatus characterized by obtaining a candidate.
(Supplementary note 5) In the image association apparatus according to any one of supplementary notes 2 to 4,
The first image has a rectangular shape and is included in a part of the second image,
The predetermined area extracting unit detects a linear component in a second image and detects the rectangular shape as the predetermined area based on the detected linear component.
(Supplementary Note 6) In the image association apparatus according to Supplementary Note 5,
The predetermined area extraction unit performs edge extraction processing on the second image when detecting the straight line component, creates an edge binary image, and performs Hough transform on the obtained edge binary image An image association apparatus characterized by that.
(Supplementary Note 7) In the image association apparatus according to Supplementary Note 5 or Supplementary Note 6,
The predetermined area extracting means detects the rectangular shape by integrating the detected linear components, and extracts the largest one of the obtained rectangular shapes as the predetermined area. .
(Supplementary note 8) In the image association apparatus according to supplementary note 3,
The similarity calculating means collates a plurality of pixels constituting the image in the predetermined area extracted by the predetermined area extracting means with a plurality of pixels constituting the first image, and calculates the similarity based on the collation result. An image association apparatus characterized by calculating.
(Supplementary note 9) In the image association device according to supplementary note 8,
The similarity calculation means includes image size changing means for changing the size of at least one of the image of the predetermined area and the first image,
An image associating device that performs the collation for a changed image size.
(Supplementary Note 10) In the image association apparatus according to Supplementary Note 3,
The similarity calculation means includes character recognition means for recognizing characters included in an image,
An image association characterized in that a similarity is calculated based on a character recognized in an image of a predetermined region of the second image extracted by the predetermined region extraction means and a character recognized in the first image apparatus.
(Supplementary Note 11) In the image association apparatus according to Supplementary Note 10,
The similarity calculation means performs matching for all characters recognized at positions corresponding to each other between the image of the predetermined region and the first image, and calculates the similarity based on the matching result. An image association apparatus.
(Supplementary note 12) In the image association apparatus according to supplementary note 10,
The similarity calculation means performs collation on a predetermined number of consecutive characters recognized at positions corresponding to each other in the image of the predetermined area and the first image, and calculates the similarity based on the collation result An image association apparatus characterized by the above.
(Supplementary note 13) In the image association device according to supplementary note 3,
The similarity calculation means includes layout information detection means for detecting layout information included in the image,
The similarity is calculated based on layout information detected in an image of a predetermined area of the second image extracted by the predetermined area extraction means and layout information detected in the first image. Image association device.
(Supplementary note 14) In the image association apparatus according to supplementary note 13,
The image association apparatus, wherein the layout information includes at least one of attribute information of the predetermined area, information about an arrangement state of the predetermined area, and a feature amount of the predetermined area.
(Supplementary Note 15) In the image association apparatus according to Supplementary Note 14,
The image association apparatus characterized in that the attribute information of the predetermined area includes at least one attribute value of each character string in a figure, a table, a character string block, and a character string block.
(Supplementary Note 16) In the image association apparatus according to any one of Supplementary Notes 13 to 15,
The feature amount of the predetermined region includes at least one of information on a position of a rectangle surrounding the character string, a character string size, a character string shape, and a character string color. .
(Supplementary note 17) In the image association apparatus according to supplementary note 16,
The feature amount of the color information existing for each character string detected by the layout information detecting means is information on a predetermined representative color obtained by clustering the color information existing for each character string. An image association apparatus characterized by that.
(Supplementary note 18) In the image association device according to any one of supplementary notes 1 to 17,
When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images,
When the first images corresponding to two frame images separated by a certain number of frames are the same, the associating means adds a frame image existing between the two frame images to the two frame images. An image association apparatus that associates the same image with a corresponding first image.
(Supplementary note 19) In the image association apparatus according to any one of supplementary note 1 to supplementary note 17,
When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images,
Frame image thinning means for extracting frame images at predetermined frame intervals from a plurality of consecutive frame images,
The relevance determining means determines the relevance of the frame image thinned out by the frame image thinning means to the first image.
(Supplementary note 20) In the image association device according to supplementary note 19,
A thinning interval for reducing the frame interval thinned out by the frame image thinning unit when the two first images extracted by the frame image thinning unit and associated with two frame images having a predetermined frame interval are not the same. An image association apparatus comprising a changing means.
(Supplementary note 21) In the image association device according to any one of supplementary notes 1 to 17,
When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images,
Point detecting means for detecting a point when a first image corresponding to a continuous frame image is different between preceding and following frames;
The image association apparatus, wherein the association unit associates the same first image with each frame image between the points detected by the point detection unit.
(Supplementary note 22) In the image association device according to any one of supplementary notes 1 to 17,
When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images,
The associating means may include a case in which frame images that are not associated with the same first image exist between a plurality of frame images associated with the same first image over a predetermined number of frames. An image association apparatus that associates the same first image with these frame images.
(Supplementary note 23) In the image association device according to any one of supplementary notes 1 to 17,
When the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images,
For a frame image whose relevance to the first image is not determined by the relevance determination means, the association means is the same as the first image associated with the frame image immediately before or immediately after the relevance can be found. An image association apparatus characterized by associating images.
(Supplementary Note 24) In the image association apparatus according to any one of Supplementary Notes 1 to 25,
The second image is a moving image obtained by being imaged by the imaging means,
The image association apparatus, wherein the first image is an image captured by the imaging unit and included in the predetermined area in the moving image.
(Supplementary Note 25) In the image association apparatus according to any one of Supplementary Notes 1 to 26,
The first image is an image projected by OHP, and the second image is an image in which an image projected by the OHP is captured in a predetermined area.
(Supplementary Note 26) A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined association with any one of the first images A relevance determination step for determining which of the first images each of the second images has the predetermined relevance based on the second image having a sex;
An association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step;
An image association method comprising:
(Supplementary note 27) An image association program stored in a computer-readable medium and causing the computer to execute an image association process,
A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined relationship with any one of the first images. A relevance determining step for determining, based on the second image, which of the first images each of the second images has the predetermined relevance;
An association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step;
An image association program that causes a computer to execute.

It is a conceptual diagram of embodiment of this invention. It is a function block diagram of an image matching and content creation apparatus. It is a hardware block diagram of an image matching and content creation apparatus. It is the flowchart which showed the flow of the matching process of an image matching apparatus. It is the flowchart which showed the flow of the OHP area | region image extraction process. It is a functional block diagram of the similarity calculation part which calculates similarity by a pixel. It is the flowchart which showed the flow of the similarity calculation process performed by the similarity calculation part of FIG. It is a function block diagram of the similarity calculation part which calculates similarity using character information. It is the flowchart which showed the flow of the similarity calculation process performed by the similarity calculation part of FIG. It is the flowchart which showed the detail of the character recognition process. It is a functional block diagram of the similarity calculation part for calculating the whole similarity using both the similarity calculation process using a pixel, and the similarity calculation process using a character. It is a functional block diagram of the similarity calculation part which calculates similarity using layout information. It is the figure which showed an example of layout information. It is the figure which showed an example of the collation process using layout information. It is the figure which showed an example of the collation process using layout information. It is the figure which showed an example of the collation process using layout information. It is a functional block diagram of the similarity calculation part for extracting the title character in an image and calculating similarity. It is the flowchart shown about the title extraction process and the title collation process. It is a functional block diagram of the similarity calculation part for calculating the similarity by integrating the method of acquiring and collating layout information and the method of collating pixels. It is an example of a frame number / OHP number correspondence table. It is an example of the table which shows the frame area corresponding to an OHP image. It is a flowchart which shows the detail of a section corresponding table creation process. It is the flowchart which showed the process for deleting the correspondence of the short frame area in an area correlation table, and integrating this frame area in the frame area before and behind that. 10 is a flowchart illustrating in detail a flow of association processing of the image association apparatus according to the second embodiment. It is the flowchart which showed the process for performing a matching process about the frame number by which matching with the OHP image was not made | formed by the calculated similarity. 10 is a flowchart illustrating in detail a flow of association processing of the image association apparatus according to the third embodiment. An example of the specific collation process implemented along the flowchart of FIG. 26 is shown.

Explanation of symbols

  1 Image association / content creation device, 1a Image association device, 1b Content creation device, 2 Lecture using OHP, 3 Camera shooting, 4 OHP file, 5 E-learning content, 11 Video input control device, 12 CPU, 13 memory, 14 file input / output device, 15 image HDD, 16 program HDD, 17 content HDD, 18 association table HDD, 19 OHP file HDD, 20 LAN interface, 21 CD-ROM drive, 101 moving image input unit, 102 Moving image storage unit, 103 OHP file input unit, 104 OHP file storage unit, 105 frame acquisition unit, 106 OHP region extraction unit, 107 OHP acquisition unit, 108, 108a, 108b, 108c, 108d, 108e Similarity calculation unit, 1 9 association processing unit, 110 association table, 111 content creation unit, 112 content storage unit, 120 image normalization unit, 121 frame pixel extraction unit, 122 OHP pixel extraction unit, 123 pixel collation unit, 124 collation result addition unit, 130 image normalization unit, 131 frame character extraction unit, 132 OHP character extraction unit, 133 character verification unit, 134 verification result addition unit, 140 pixel verification result calculation unit, 141 character verification result calculation unit, 142 total similarity calculation unit, 150 image normalization unit, 151 frame layout acquisition unit, 152 OHP layout acquisition unit, 153 layout verification unit, 154 verification result addition unit, 160 image normalization unit, 161 frame title extraction unit, 162 OHP title extraction unit, 163 title verification Part, 164 verification result total Arithmetic unit, 170 frame layout acquisition unit, 171 OHP layout acquisition unit, 172 image normalization unit, 173 frame region pixel extraction unit, 174 OHP region pixel extraction unit, 175 pixel verification unit, 176 verification result calculation unit.

Claims (3)

A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined relationship with any one of the first images. Relevance determining means for determining which of the first images each of the second images has the predetermined relevance based on the two images;
Association means for associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination means;
Frame image thinning means for extracting a frame image at a predetermined frame interval from a plurality of consecutive frame images when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images. When,
A thinning interval for reducing the frame interval thinned out by the frame image thinning unit when the two first images extracted by the frame image thinning unit and associated with two frame images having a predetermined frame interval are not the same. Change means,
Points when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, and the first image corresponding to successive frame images is different between the preceding and succeeding frames. And point detecting means for detecting
The relevance determining means determines the relevance of the frame image thinned out by the frame image thinning means to the first image;
The image association apparatus, wherein the association unit associates the same first image with each frame image between the points detected by the point detection unit. A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined relationship with any one of the first images. A relevance determining step for determining, based on the second image, which of the first images each of the second images has the predetermined relevance;
An association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step;
A frame image thinning step for extracting frame images at predetermined frame intervals from a plurality of consecutive frame images when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images. When,
A thinning interval that reduces the frame interval thinned out by the frame image thinning step when two first images extracted by the frame image thinning step and associated with two frame images having a predetermined frame interval are not the same. Change steps,
Points when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, and the first image corresponding to successive frame images is different between the preceding and succeeding frames. A point detecting step for detecting
The relevance determining step determines the relevance of the frame image thinned out by the frame image thinning step to the first image,
In the image association program, the association step associates the same first image with each frame image between the points detected by the point detection step. A plurality of first images and a plurality of second images, wherein at least one of the plurality of second images has a predetermined relationship with any one of the first images. A relevance determining step for determining, based on the second image, which of the first images each of the second images has the predetermined relevance;
An association step of associating each of the first image and the second image determined to have a predetermined relevance by the relevance determination step;
A frame image thinning step for extracting frame images at predetermined frame intervals from a plurality of consecutive frame images when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images. When,
A thinning interval that reduces the frame interval thinned out by the frame image thinning step when two first images extracted by the frame image thinning step and associated with two frame images having a predetermined frame interval are not the same. Change steps,
Points when the second image is a frame image corresponding to each of a plurality of frames constituting a series of moving images, and the first image corresponding to successive frame images is different between the preceding and succeeding frames. And a point detecting step for detecting,
The relevance determining step determines the relevance of the frame image thinned out by the frame image thinning step to the first image,
The image associating method, wherein the associating step associates the same first image with each frame image between the points detected by the point detecting step.

Images