WO2015146237A1

WO2015146237A1 - Still image related information detection device, detection method, detection program, and recording medium on which that program has been stored

Info

Publication number: WO2015146237A1
Application number: PCT/JP2015/051294
Authority: WO
Inventors: 哲弥高森; 與那覇　誠; 学斌胡
Original assignee: 富士フイルム株式会社
Priority date: 2014-03-28
Filing date: 2015-01-20
Publication date: 2015-10-01
Also published as: JP2015192258A

Abstract

Provided are a related information detection device, a related information detection method and a related information detection program for finding information related to still images in an electronic album. Also provided is a recording medium on which that program has been stored. The image of a still image (I2) in a photobook (30) is captured as a first area and the feature of the still image (I2) is detected as a first feature. The image of a second area different from the first area is captured and the feature of the second area is detected as a second feature. From the two features, which are the detected first and second features, a moving image related to the still image can be found. Unless not only the still image (I2) but also the information of the second area can be acquired, the moving image related to the still image cannot be acquired. Therefore, only the owner of the photobook (30) can view the moving image related to the still image.

Description

Still image related information detection device, detection method, detection program, and recording medium storing the program

The present invention relates to a still image related information detection apparatus, a detection method, a detection program, and a recording medium storing the program.

In recent years, it has become easier to shoot videos with digital cameras, smartphones, etc., and the opportunities for shooting have increased. Recorded videos are stored on recording media, but are difficult to find when needed.

Also, it has become possible to select impressive frames from videos and output them to photos and albums. When looking at an album created in this way, there is a need to view the video that is the source of the photo. For this purpose, by taking a picture included in the album on a device with a shooting function and a display function, such as a digital camera, a mobile phone, a smartphone, a tablet terminal, etc., the video related to the picture is reproduced. (Patent Document 1). In addition, there is a technique that displays information related to a captured image displayed by the camera function of the mobile terminal on the screen of the mobile terminal (Patent Document 2). In addition, there is a technique in which a print authentication code is printed on paper from a print image printed on paper so that the contents of the print printed on the paper can be verified (Patent Document 3).

JP 2013-92903 JP JP 2013-222406 A JP 2006-340215 A

The one described in Patent Document 1 reads a digital watermark embedded in a still image and reproduces a moving image. However, since the digital watermark code is artificially generated, browsing cannot be restricted only from the code. . In the device described in Patent Document 2, an index for specifying a captured image is automatically extracted from the captured image, but related information is displayed only by obtaining the captured image. In the devices described in

Patent Documents

1 and 2, the security is low so that only a specific user can obtain information associated with a still image. Moreover, in the thing of patent document 3, it is only detecting the authenticity of the print content, and it is not considered to find out the information associated with the print image, but such information is associated with the print image. Not even thought about it.

An object of the present invention is to enable a user having a still image display paper medium to find information associated with a still image included in the still image display paper medium.

According to the still image related information detecting apparatus of the present invention, the first area including one still image and the first of the still image display paper media among the plurality of still images represented on the still image display paper medium. Imaging means for imaging a second area different from the first area, first feature detection means for detecting a first feature that identifies one still image from the first area imaged by the imaging means, and imaging by the imaging means Second feature detection means for detecting a second feature that identifies the second region from the second region thus obtained, and the first feature and the second feature detection detected by the first feature detection means Command means for outputting a command for finding information associated with one still image from the second feature detected by the means is provided.

The present invention also provides a method for detecting related information of still images. That is, in this method, the imaging unit has a first area including one still image among a plurality of still images represented on the still image display paper medium and a first area of the still image display paper medium. A second region different from the first region, and the first feature detection unit detects a first feature that identifies one still image from the first region captured by the imaging unit, and the second feature detection unit The second feature that identifies the second region is detected from the second region imaged by the imaging unit, and the command unit detects the first feature detected by the first feature detection unit and the second feature. A command for finding information associated with one still image from the second feature detected by the feature detection means is output.

The present invention also provides a program for detecting related information of a still image and a recording medium storing the program.

For example, the image pickup unit picks up another still image different from the one still image, and the second feature detection unit detects, for example, the second feature from the other still image picked up by the image pickup unit. To do.

The imaging means may capture an area outside the still image excluding a plurality of still images represented on the still image display paper medium. In this case, the second feature detection unit will detect the second feature from the outside of the still image captured by the imaging unit.

The area outside the still image is, for example, a blank area.

When the image pickup means picks up an area outside the still image, the second feature detection means is represented by the width, area, color and shape of the area outside the still image picked up by the image pickup means and the area outside the still image. The second feature can be detected based on at least one of texture, character, and figure.

When the digital watermark is embedded in the second area, the second feature can be detected from the digital watermark.

For example, the command unit may include a moving image and a moving image associated with one still image from the first feature detected by the first feature detecting unit and the second feature detected by the second feature detecting unit. A command for finding at least one of the chapter start times is output.

The first area is, for example, a partial area of the imaging range in the imaging means, and the second area is, for example, an area corresponding to the imaging range. In this case, it is preferable that the imaging unit further includes a display device that simultaneously images the first area and the second area and displays a part of the area on the display screen.

A display device that displays an image of an area imaged by an imaging unit on a display screen of a still image display paper medium, and a designation unit that specifies a part of the image displayed on the display screen of the display device May be further provided. In this case, the first feature detection means will determine the area designated by the designation means as the first area and detect the first feature.

Of the still image display paper medium, a display device that displays an image of an area captured by the imaging unit on a display screen, and a display that displays a mark that identifies the first area on the image displayed on the display screen Control means may be further provided.

The still image display paper medium is, for example, a photo book composed of a plurality of pages in which one or more still images are represented on one sheet or one page on which a plurality of still images are represented. .

According to the present invention, among the plurality of still images displayed on the still image display paper medium, the first region including one still image and the second region different from the first region are imaged. A first feature that identifies one still image is detected from the first region, and a second feature that identifies the second region is detected from the second region. A command for finding information associated with one still image from the detected first feature and second feature is output, for example, to a server accessible to a database storing the information. Information associated with one still image is found from the first feature and the second feature, and the found information is given to the related information detection device. In the related information detection device, information associated with one still image is obtained. If the first region and the second region are not imaged, information associated with one still image cannot be obtained. Therefore, even if only one still image is obtained, it is possible to obtain one still image. The associated information cannot be obtained. A user having a still image display paper medium can find information associated with one still image.

It is the outline | summary of the related information detection system of a still image. It is a block diagram which shows the electric constitution of a smart phone. It is an example of a photo book. It is an example of a photo book. It is an example of a still image feature table. It is an example of the feature table outside a still image. It is an example of a feature index table. It is a flowchart which shows the process sequence of a smart phone. It is a flowchart which shows the process sequence of a server. It is an example of the display screen of a smart phone. It is an example of the display screen of a smart phone. It is an example of a photo book. It is an example of a photo book. It is an example of the display screen of a smart phone. It is an example of the display screen of a smart phone.

FIG. 1 shows an embodiment of the present invention and is an outline of a related information detection system for still images.

The related information detection system for still images includes a smartphone (related information detection device) 1 and a server 20 that can communicate with each other via the Internet (may be another network). As will be described later, the server 20 stores a plurality of moving images (not limited to moving images but any information associated with still images) associated with still images included in the photo book.

FIG. 2 is a block diagram showing an electrical configuration of the smartphone 1.

The overall operation of the smartphone is controlled by CPU2.

The smartphone 1 includes a camera 8 and a display device 6. A touch panel 7 is provided on the display screen of the display device 6. The smartphone 1 also includes a memory 3 for storing data, a memory card interface 4 for accessing the memory card 5, a communication device 9 for connecting to the Internet, an interface for connecting to a personal computer, and the like. 10 is also included. Furthermore, the smartphone 1 also includes a microphone 12, a speaker 13, and a voice control device 11 for calling.

The memory card 5 stores a program for controlling operations described later. Such a memory card 5 is connected to the memory card interface 4, and the program stored in the memory card 5 is read by the smartphone 1. When the read program is installed in the smartphone 1, the operation described later is performed. Even if the program is not stored in a recording medium such as the memory card 5, it may be downloaded by the communication device 9 via the Internet.

3 and 4 show an example of a photobook (paper album, still image display paper medium) 30. FIG.

FIG. 3 shows the first and second pages of the photo book 30, and FIG. 4 shows the third and fourth pages of the photo book 30. The photobook 30 also includes the fifth and subsequent pages, but is omitted. However, a photo book up to the fourth page may be used.

Referring to FIG. 3, still images (photographs) I1 to I5 are shown on the first and second pages. A region between these still images I1 to I5 is referred to as a non-still image region. From the first page to the second page, still image outside areas a1 to a4 are shown. The still image outside areas a1, a3, and a4 are margins (margin areas), and the letter “Spring Day” and a heart mark are displayed in the still image outside area a2. The still image outside area a1 is an area surrounded by still images I1, I2 and I3 as indicated by chain lines. Other areas outside the still image are defined in the same manner. Frames (marks) 31 are displayed at the upper left and lower right of the still images I2 and I5, respectively. This frame 31 indicates that the moving image relating to the enclosed still image (I2 and I5 in the case of FIG. 3) is stored in the information database 21. However, the frame 31 is not always necessary. The still images I2 and I5 are one frame still images constituting the related moving images, but the one frame still images constituting the moving images are not necessarily represented in the photo book 30. The moving image may be stored in the information database 21 in association with a still image that is not a still image constituting the moving image.

Referring to FIG. 4, still images I6 to I9 are represented on the third and fourth pages. Between the still images I6 to I9, there are still image outside areas a5 to a8. The above-described frame 31 is provided on the upper left and lower right of the still images I7 and I8. It can be seen that there are moving images related to the still images I7 and I8.

FIG. 5 is an example of a still image feature table.

The still image feature table stores still image features included in the photo book 30 for each still image. The still image feature table is stored in the information database 21.

The features of the still image I1 are the position of the dark spot (the coordinate position when the upper left of the still image I1 is the origin), the position of the bright spot, the position of the edge, the gradation value, the color histogram, the spatial frequency, the gradient histogram, and the electronic Watermarks and the like are stored in the still image feature table. Similarly, still picture features other than the still picture I1 are also stored in the still picture feature table.

FIG. 6 is an example of a still image extra feature table.

The still image outside feature table stores the features of the still image outside region included in the photo book for each region outside the still image. A still image outside feature table is also stored in the information database 21.

As the features of the still image outside area a1, the area width, area area, color, shape, digital watermark, and the like are stored in the still image outside feature table. The width of the area indicates the distance from the left end of the photo book 30 to the left end of the still image I2 in FIG. 3, and the area and shape of the area are the area of the portion surrounded by the chain line as shown in FIG. Shape. When a digital watermark is embedded in the still image outside area a1, the data is stored in the still image outside feature table.

As shown in FIG. 3 described above, since the still image outside area a2 includes “Spring Day” and a heart mark, those characters, character color, figure, figure color, etc. are also stationary. It is stored in the off-screen feature table.

FIG. 7 is an example of a feature index table.

The feature index stored in the feature index table specifies a still image and manages a moving image associated with the still image. The feature index is associated with features of different areas stored in FIGS. 5 and 6. The different areas include at least one still picture area, and may be two or more still picture areas, or one still picture area and one or more still picture outside areas. A feature index is found from the features of the different areas, and a still image and a moving image associated with the still image are found from the found feature index. For example, given two features, the position of the dark spot of the still image I1 and the width of the still image outside area a1, a feature index Ix11 that identifies the still image I1 can be found by that feature. A combination of these two features can also be referred to as a feature index Ix11. A still image I1 and a moving image M1 are found from the found feature index Ix11. The same applies to other still images and moving images. The same movie may be associated with different still images, or a part of the movie may be associated with the still image as a movie.

FIG. 8 is a flowchart showing the processing procedure of the smartphone 1, and FIG. 9 is a flowchart showing the processing procedure of the server 20.

In these processing procedures, a moving image associated with a still image included in the photo book 30 shown in FIGS. 3 and 4 is found out and reproduced on the smartphone 1.

In this embodiment, it is assumed that a moving image associated with the still image I2 is reproduced.

The user of the smartphone 1 images the first area including one still image I2 in the photo book 30 with the camera (imaging means) 8 using the smartphone 1 (step 41 in FIG. 8). Then, an image representing the first area is temporarily stored in the memory 3 of the smartphone 1. FIG. 10 shows a state in which the smartphone 1 captures an image of the first area including one still image I2. The user of the smartphone 1 determines and captures the camera angle so that the entire one still image I2 appears on the entire display screen 14 of the smartphone 1. However, it is not always necessary to determine the camera angle so that the entire still image I2 at the position is reflected on the entire display screen 14.

Subsequently, the user of the smartphone 1 can select a second area (a page different from the first area or the same page as the first area) of the photobook 30. The first area must be completely coincident with the first area. Is captured by the camera (imaging means) 8 (step 41 in FIG. 8), and the shutter is pressed. Then, an image representing the second area is temporarily stored in the memory 3 of the smartphone 1. FIG. 11 shows a state in which the second area is imaged by the smartphone 1. In this example, a region including a still image I4 (another still image different from one still image) is imaged as the second region. The still image I4 is displayed on the display screen 14 of the smartphone 1.

Subsequently, a first feature that identifies one still image I2 is detected by the CPU 2 from the first imaged area (first feature detection means) (step 42 in FIG. 8). For example, the CPU 2 detects the position of the dark spot and the position of the bright spot of one still image I2. Further, the second feature that identifies the second region is detected by the CPU from the captured still image I4 of the second region (second feature detecting means) (step 43 in FIG. 8). For example, the edge position and gradation value of the still image I4 included in the second area are detected. Similarly to the one still image I2, the position of the dark spot and the position of the bright spot of the still image I4 may be detected.

The detected first feature and second feature and related information detection command (command for finding information associated with one still image) is output (transmitted) from the smartphone 1 to the server 20 (step 44 in FIG. 8). ) (Command means). In this example, since the moving image is associated with the still image I2 and the moving image is not associated with the still image I4, the related information detection command for the still image I2 is transmitted from the smartphone 1 to the server 20.

When the detected first feature and the second feature and the related information detection command transmitted from the smartphone 1 are received by the server 20 (YES in step 51 in FIG. 9), the first feature and the second feature are used. A feature index specifying one still image represented by the first region is detected (step 52 in FIG. 9). As described above, the position of the dark spot and the position of the bright spot of one still image I2 are detected as the first feature, and the edge position and gradation value of the still image I4 are detected as the second feature. It can be seen from the still image feature table that the region having the dark spot position and the starting point position of the first feature is the still image I2, and the region having the edge position and the gradation value of the second feature is It can be seen from the still picture feature table that it is a still picture I4. A feature index corresponding to the combination having the position of the dark point and the start point of the first feature and the position of the edge of the second feature and the gradation value is detected from the feature index table (FIG. 9 step 52). In this example, still images I2 and I4 are captured. However, since the still image I2 is associated with the moving image, a feature index specifying the still image I4 is not found (stored in the feature index table). A feature index specifying the still image I2 is found.

A still image (in this case, still image I2) corresponding to the found feature index is found, and a moving image (in this case, moving image M1) associated with the still image is detected from the information database 21 (step 53 in FIG. 9). . The detected moving image is transmitted from the server 20 to the smartphone 1 (step 54 in FIG. 9).

When the moving image transmitted from the server 20 is received by the smartphone 1 (YES in step 45 in FIG. 8), the moving image is reproduced on the smartphone 1 (step 46 in FIG. 8).

The second region different from the first region is not captured by capturing the first region including the one still image I4 and reproducing the moving image associated with the one still image from the feature of the one still image. Since an area is imaged and a moving image associated with one still picture is found from the characteristics of one still picture and the characteristics of the second area, one still picture can be obtained even if only one still picture is obtained. The movie associated with the image cannot be played. Only when both areas of one still image and the second area can be imaged, a moving image associated with one still image can be reproduced. If the photo book 30 itself is not owned, the moving picture associated with the still picture of the photo book cannot be reproduced, so that it proves that the photo book 30 itself has the authority to reproduce the moving picture.

FIG. 12 and FIG. 13 are examples of the photo book 30, and correspond to FIG. 3 and FIG.

In the above-described embodiment, a still image different from one still image is captured as the second region. However, even if one still image is included, a second region different from the first region is captured. Just do it.

For example, as shown by a chain line 61 in FIG. 12, although one still image I2 is included, an area larger than the first area may be imaged as the second area. In this case, the feature amount of the second region is detected from the still image outside regions a1, a2, a3, etc. around the one still image I2. For example, the width of the area outside the still image a2, the width of the area a3, and the like are detected as the feature quantities of the second area. The feature index corresponding to the combination of the feature amount detected as the feature amount of the second area and the feature amount of one still image is found from the feature index table as described above, and still images and moving images are found. It will be.

Further, as indicated by a chain line 62 in FIG. 12, a still image outside area a2 including characters, figures, and the like may be imaged as the second area. In this case, characters, figures, and the like included in the still image outside area a2 are detected as the feature quantities of the second area. Also in this case, the feature index corresponding to the combination of the feature quantity detected as the feature quantity of the second region and the feature quantity of one still image is found from the feature index table as described above, and The video will be found.

Further, as indicated by a chain line 63 in FIG. 13, an area of a page (a page not in the spread page) different from the one still image I2 may be imaged as the second area.

As described above, the second feature amount is detected based on at least one of the width, area, color, shape, texture, character, and figure represented in the non-still image area. Alternatively, the second feature value may be detected from the digital watermark by imaging the area in which the digital watermark is embedded. In addition, when an area outside the still image is imaged as the second area, the blank area is not limited to the area represented by characters such as the area a2 and the blank area may be imaged as the second area. Needless to say, it is good.

Furthermore, when the area indicated by the chain line 61 in FIG. 12 is an area corresponding to the imaging range, a part of the imaging range is displayed on the display screen 14 of the smartphone 1 as shown in FIG. Also good. In this case, the area displayed on the display screen 14 is the first area including one image, and the imaging range is the second area. In addition, the first area and the second area are imaged simultaneously. The user captures a second area wider than one still image (first area) at the same time as the first area while capturing one still image I2 as the first area and displaying it on the display screen 14. It will be possible. Needless to say, the first feature value and the second feature value are detected from the first region and the second region thus obtained, and the moving image can be found as described above.

FIG. 14 shows a modification and is an example of the display screen 14 of the smartphone 1.

The first and second pages of the photo book are captured by the smartphone 1. On the display screen 14 of the display device 6, images of the first page and the second page are displayed. The user of the smartphone 1 uses the finger 70 to touch the touch panel (designating means) on the still image on which the associated moving image is to be viewed (designate some area). Then, the designated still image is determined as the first area, and the first feature amount is detected by the CPU 2. In this case, a region around the designated region (a region indicated by a chain line 64) may be automatically determined as the second region. A second feature amount is detected from the determined second region.

FIG. 15 shows a display screen 14 of the smartphone 1 and shows a modification.

The display screen 14 displays four designated frames 15. A still image associated with a moving image is captured so as to match the designated frame 15. Then, the first feature amount is detected from the first region with the matched still image as one still image. As described above, the CPU 15 (display control means) may display the frame 15 (mark) for specifying the first area on the image displayed on the display screen. The second area may be imaged again, or may be an area around the frame 15 as described above.

In the above-described embodiment, the photo book has a plurality of pages on which one or a plurality of still images are represented. However, a plurality of still images are displayed on one sheet like a post card. It goes without saying that the same processing can be carried out for those that are provided.

In addition, if the above-described still image feature table, feature outside the still image feature table, feature index table, and moving image are stored in the smartphone 1, the moving image associated with the still image can be displayed on the smartphone 1 without accessing the server 20. Can be played. In addition, the above-described processing can be performed not only on smartphones but also on other mobile phones, tablet terminals, and the like.

1 Smartphone (related information detection device)
2 CPU (first feature quantity detection means, second feature quantity detection means, display control means)
8 Camera (imaging means)
9 Communication device (command means)

Claims

Of the plurality of still images represented on the still image display paper medium, a first region including one still image and a second region different from the first region of the still image display paper medium are imaged. Imaging means to perform,
First feature detection means for detecting a first feature that identifies the one still image from the first region imaged by the imaging means;
Second feature detection means for detecting a second feature that identifies the second region from the second region imaged by the imaging means, and the first feature detected by the first feature detection means. Command means for outputting a command for finding information associated with the one still image from one feature and the second feature detected by the second feature detection means;
An apparatus for detecting related information of still images.
The imaging means is
It captures other still images different from the one still image above,
The second feature detection means includes:
Detecting the second feature from the other still image imaged by the imaging means;
The related information detection apparatus for still images according to claim 1.
The imaging means is
An area outside the still image excluding a plurality of still images represented on the still image display paper medium is imaged.
The second feature detection means includes:
The second feature is detected from a still image outside area imaged by the imaging means.
The related information detection apparatus for still images according to claim 1 or 2.
The area outside the still image is a margin area.
The related information detection apparatus for still images according to claim 3.
The imaging means is
To capture the area outside the still image,
The second feature detection means includes:
The second feature based on at least one of the width, area, color and shape of the region outside the still image imaged by the imaging means and the texture, character and figure represented in the region outside the still image Is to detect
The related information detection apparatus for still images according to claim 3.
A digital watermark is embedded in the second area, and the second feature is detected from the digital watermark.
The related information detection apparatus for still images according to any one of claims 1 to 5.
The command means is
A moving image and a moving image chapter associated with the one still image from the first feature detected by the first feature detecting unit and the second feature detected by the second feature detecting unit Outputs a command to find at least one of the start times of
The related information detection apparatus for still images according to any one of claims 1 to 6.
The first area is a partial area of the imaging range in the imaging means,
The second area is an area corresponding to the imaging range,
The imaging means is for imaging the first area and the second area simultaneously,
A display device for displaying the partial area on the display screen;
The related information detection device for still images according to any one of claims 1 to 7, further comprising:
Of the still image display paper medium, a display device that displays an image of an area captured by the imaging unit on a display screen, and a part of the image displayed on the display screen of the display device Further comprising a designation means for designating,
The first feature detection means includes:
The area designated by the designation means is determined as the first area, and the first feature is detected.
The related information detection apparatus for still images according to any one of claims 1 to 7.
Of the still image display paper medium, a display device for displaying an image of an area imaged by the imaging means on a display screen, and a mark for identifying the first area on the image displayed on the display screen Display control means for displaying
10. The related information detection device for still images according to any one of claims 1 to 9, further comprising:
The still image display paper medium is
A photo book composed of a plurality of pages in which one or more still images are represented on one sheet or one page on which a plurality of still images are represented.
11. The related information detection device for a still image according to claim 1, wherein the related information detection device is a still image.
The imaging means includes a first region including one still image among a plurality of still images represented on the still image display paper medium, and a second different from the first region in the still image display paper medium. Image the area of
A first feature detection unit that detects a first feature that identifies the one still image from the first region imaged by the imaging unit;
A second feature detecting unit detects a second feature specifying the second region from the second region imaged by the imaging unit;
The command means is
A command for finding information associated with the one still image from the first feature detected by the first feature detection unit and the second feature detected by the second feature detection unit. Output,
A method for detecting related information of still images.
A program for controlling a computer of an apparatus for detecting related information of still images,
Of the plurality of still images represented on the still image display paper medium, a first region including one still image and a second region different from the first region of the still image display paper medium are imaged. Let
Detecting a first feature specifying the one still image from the imaged first region;
A second feature that identifies the second region is detected from the captured second region;
A computer that controls the computer of the still image related information detection device that outputs a command to find information associated with the one still image from the detected first feature and the second feature is readable by a computer Program.
A recording medium storing the program according to claim 13.