JP2014134898A - Image search apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quickly search a desired still image from a moving image.SOLUTION: A user sets a search condition for searching a still image form a moving image in a system control part (112). The system control part (112) reproduces the moving image stored in a recording medium (111) and allows the user to select a subject. The system control part (112) calculates movement quantity between frames in which the same subject as the subject selected by the user exists on the basis of a frame selected by the user. A search range determination part (115) determines an image search range for extracting the still image from the moving image in accordance with the movement quantity. An image search part (116) searches a frame including the same subject as the subject selected by the user and matched with the search condition within the image search range determined by the search range determination part (115).

Description

  The present invention relates to an image retrieval apparatus that retrieves a desired still image from a moving image.

  A digital camera equipped with a function capable of saving a frame arbitrarily selected by a user from captured moving images as a still image has appeared.

  In general, there are a plurality of frames similar to the frame selected by the user before and after the frame selected by the user of the moving image. Therefore, there is a possibility that a frame with better conditions (subject size, focus, hue, etc.) than the frame selected by the user may exist. In order to find such a frame with better conditions, it is necessary to reproduce and check the moving image one frame at a time. In such visual confirmation, the best shot is selected from a plurality of frames. It is difficult.

  In order to avoid this problem, it is conceivable to use an image search technique. Patent Document 1 describes a technique for specifying one search source image, setting a feature portion of the search source image and accompanying information of the image as a search condition, and searching for a still image that matches the search condition. Yes.

JP 20081826262 A

  The prior art is required to improve the search efficiency, that is, to shorten the search time, although the search conditions can be easily set.

  An object of the present invention is to present an image search device that can search a desired still image from a moving image more efficiently.

  The image search apparatus according to the present invention includes a setting unit that sets a search condition for searching for a still image from a moving image including a plurality of frames, a selection unit that selects a subject included in the moving image, and the selection unit that A search range determining unit that determines an image search range of the moving image according to the state of the subject around the frame in which the subject is selected, and a subject that is the same subject as the subject selected by the selection unit and that satisfies the search condition Search means for searching for a frame including a matching subject within the image search range.

  According to the present invention, the image search range is determined from the moving image according to the state of the subject selected by the user, and a frame that satisfies the search condition is searched within the image search range. Can be extracted.

It is a schematic block diagram of one Example of this invention. It is an operation | movement flowchart of a present Example. 6 is a schematic diagram of an explanatory example 1. FIG. 10 is a schematic diagram of an explanatory example 2. FIG. It is a schematic diagram of the explanatory example 3. 10 is a schematic diagram of an explanatory example 4. FIG. 10 is a schematic diagram of an explanatory example 5. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic block diagram of a digital camera to which an embodiment of an image search apparatus according to the present invention is applied. In this embodiment, the frame search range is changed according to the amount of movement of the subject.

  In FIG. 1, reference numeral 101 denotes a photographing lens including a focus lens. Reference numeral 102 denotes a shutter having an aperture function. An imaging unit 103 includes a CCD image sensor, a CMOS image sensor, or the like that converts an optical image into an electrical signal. Reference numeral 104 denotes an A / D converter that converts an analog image signal into a digital signal. The A / D converter 104 is used to convert an analog signal output from the imaging unit 103 into a digital signal.

  An image processing unit 108 performs resizing processing and color conversion processing such as predetermined pixel interpolation and reduction on the data from the A / D converter 104 or the data from the memory control unit 107. The image processing unit 108 performs predetermined calculation processing using the captured image data, and the system control unit 112 performs exposure control and distance measurement control based on the obtained calculation result. Thereby, TTL (through the lens) AF (autofocus) processing and AE (automatic exposure) processing are performed. The image processing unit 108 further performs predetermined calculation processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained calculation result.

  Output data from the A / D converter 104 is written into the memory 109 via the image processing unit 108 and the memory control unit 107 or via the memory control unit 107. The memory 109 stores image data obtained by the imaging unit 103 and converted to digital data by the A / D converter 104 and image data to be displayed on the display unit 106. The memory 109 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time.

  The memory 109 also serves as an image display memory (video memory). The D / A converter 105 converts the display image data stored in the memory 109 into an analog signal and supplies the analog signal to the display unit 106. Thus, the display image data written in the memory 109 is supplied to the display unit 106 via the D / A converter 105 and displayed as an image by the display unit 106. The display unit 106 performs display in accordance with an analog signal from the D / A converter 105 on a display device such as an LCD.

  The nonvolatile memory 118 is an electrically erasable / recordable memory, and for example, an EEPROM or the like is used. The nonvolatile memory 118 stores constants and programs for operating the system control unit 112. Here, the program is a program for executing various flowcharts described later.

  A system control unit 112 controls the entire digital camera. By executing the program recorded in the non-volatile memory 118, each process described later is realized. A system memory 119 uses a RAM. Constants and variables for the operation of the system control unit 112 are temporarily written in the system memory 119, and a program read from the nonvolatile memory 118 is developed. The system control unit 112 also performs display control by controlling the memory 109, the D / A converter 105, the display unit 106, and the like.

  Reference numeral 113 denotes an operation unit, which is an operation unit for inputting various operation instructions to the system control unit 112. The operation unit 113 includes a shutter button, a power button, a mode switching button (a button for switching an operation mode to any one of a still image recording mode, a moving image recording mode, a reproduction mode, and the like), a touch panel, and the like. The operation unit 113 is used by a user to operate a digital camera.

  Reference numeral 117 denotes a compression / decompression unit, which is a means for compressing / decompressing moving image data for recording / reproduction, and uses, for example, an MPEG (Moving Picture Experts Group) system. When playing back a moving image, the compressed moving image data stored in the memory 109 or the recording medium 111 is decompressed to the uncompressed moving image data by the compression / decompression unit 117 and displayed as an image on the display unit 106. The moving image data consists of a plurality of frames at a constant rate.

  The subject recognition unit 114, the search range determination unit 115, and the image search unit 116 are used while a moving image is being reproduced or paused. The subject recognition unit 114 recognizes the shape of the subject selected by the user using the operation unit 113. As a recognition method, for example, four coordinate points whose color information changes vertically and horizontally from the coordinate point selected by the user are inspected, and the shape of the subject selected by the user is recognized from the detected coordinate points. The search range determination unit 115 refers to the characteristics of the subject recognized by the subject recognition unit 114 and determines a range for searching for an image. The image search unit 116 searches for a frame in which the subject recognized by the subject recognition unit 114 satisfies the search condition within the search range determined by the search range determination unit 115.

  An interface (I / F) 110 is an interface with a recording medium 111 such as a memory card or a hard disk. The recording medium 111 is a recording medium such as a memory card, and includes a semiconductor memory or a magnetic disk.

  FIG. 2 is a flowchart showing the overall operation of this embodiment. With reference to FIG. 2, the processing procedure of the present embodiment will be described. Note that the search range changing process performed in step S205 can be changed across scenes. A scene is one scene from the start of recording to the end of recording.

  In step S <b> 201, the user inputs frame search conditions to the system control unit 112 through the operation unit 113. The system control unit 112 holds the frame search condition set by the user in the memory 109. Search conditions include, for example, the degree of focus (the degree of focus), the color, the size of the subject, the position of the subject, and the like. Here, the frame search condition is that the subject is most focused.

  In step S202, the captured moving image is reproduced. Specifically, the user selects captured moving image data using the operation unit 113 and instructs the system control unit 112 to perform reproduction. The system control unit 112 reads the compressed moving image data of the moving image selected by the user from the recording medium 111 to the compression / decompression unit 117, expands it, and reproduces and displays it on the display unit 106.

  In step S203, the system control unit 112 waits for the user to select a subject in the moving image. When the user selects any subject in the moving image, the system control unit 112 proceeds to step S204.

  FIG. 3A is a schematic diagram showing how the user selects a subject of moving image data. Frames 301, 302, 303, 304, and 305 constitute a moving image. Assume that time flows from the frame 301 toward the frame 305. When the user finds a frame 303 he / she likes during moving image reproduction, the user selects a subject to be searched from a plurality of subjects in the frame 303.

  In step S204, the system control unit 112 instructs the subject recognition unit 114 to recognize the shape of the subject selected by the user. The subject recognition unit 114 notifies the system control unit 112 of the characteristics of the recognized subject.

  In step S205, the system control unit 112 causes the search range determination unit 115 to determine the image search range according to the characteristics of the selected subject. In the present embodiment, the search range determination unit 115 changes the image search range according to the amount of movement of the subject recognized in step S204. For example, as shown in FIG. 3B, the image search range is designated by temporally forward n1 frames and backward n2 frames centering on a frame 303 selected by the user. The amount of movement of the subject means the amount of movement of the subject between frames. For example, the system control unit 112 calculates a motion vector between frames of the subject recognized by the subject recognition unit 114, and determines a subject movement amount between frames. The moving speed of the subject can be calculated from the moving distance between frames and the time.

  Based on the frame 303 selected by the user, the system control unit 112 calculates a motion vector between the subject 311 selected by the user and a frame in which the same subject exists. In the example illustrated in FIG. 3B, the system control unit 112 calculates the motion vectors of the subjects 310, 312, and 313 shown in the frames 302, 304, and 305 with respect to the subject 311 of the frame 303.

  The system control unit 112 calculates the moving speed of the subject from the motion vectors of the subjects 310, 311, 312, and 313, and notifies the search range determination unit 115 of it. The search range determination unit 115 determines the image search range according to the moving speed information from the system control unit 112. Specifically, the search range determination unit 115 determines the image search range narrower as the subject moving speed is faster, and determines the image search range wider as the subject moving speed is slower. This is because it is conceivable that when the amount of movement of the subject is large, the composition changes greatly between the frames, and when the amount of movement of the subject is small, the composition changes between the frames is small.

  FIG. 3B shows an example in which the amount of movement of the subject selected by the user is large. In contrast to the example shown in FIG. 3B, in this embodiment, when the amount of motion of the subject 311 selected by the user is large, n1 and n2 that define the image search range are reduced. As an extreme example, both n1 and n2 are determined to be 1. As a result, the possibility that the frame 305 greatly different from the composition of the subject 311 selected by the user is included in the image search range is reduced.

  FIG. 4 shows an example in which the amount of movement of the subject selected by the user is small. In the example illustrated in FIG. 4, the amount of motion of the subjects 410, 411, 413, and 414 is small with respect to the subject 412 selected by the user. Therefore, in this example, the search range determination unit 115 largely determines the frame numbers n1 and n2 indicating the image search range. Accordingly, there is a high possibility that a frame similar to the composition of the subject 412 selected by the user is included in the image search range.

  In step S206, the image search unit 116 searches the image search range determined in step S205 for a frame in which the subject recognized in step S204 meets the search condition set in step S201. That is, the system control unit 112 controls the image search unit 116 based on the information set or determined in steps S201, S204, and S205 to search for a frame that matches the search condition.

  In the example shown in FIG. 3B, the image search unit 116 selects the search condition (subject focus best) set in step S201 from the frames 302, 303, and 304 that are the image search range determined in step S205. Frame). For example, when the subject 310 is most focused, the frame 302 is detected as a search result. The image search unit 116 notifies the system control unit 112 of a frame that matches the search condition or a frame that is closest to the search condition.

  In step S207, the system control unit 112 displays an image of the frame that is the search result in step S206 on the display unit 106. That is, the system control unit 112 reads the image data of the frame notified as the search result from the image search unit 116 from the recording medium 111 and displays the image on the display unit 106. In FIG. 3B, the frame 302 is displayed on the display unit 106.

  In this embodiment, since the image search range is changed according to the amount of movement of the subject selected by the user, an efficient search can be realized while increasing the probability of finding a frame preferable to the frame selected by the user. The user can easily select a subject to be used as a search index or reference, which facilitates the search operation.

  Although the subject recognition unit 114, the search range determination unit 115, and the image search unit 116 are illustrated as separate components from the system control unit 112, these means 114, 115, and 116 are implemented as functions of the system control unit 112. You may do it.

  A modified embodiment in which the image search range is changed in accordance with the change in the size of the subject selected by the user will be described.

  For this purpose, step S205 in FIG. 2 is changed as follows. That is, in the case of a moving image in which the size of the subject selected by the user increases rearward in time series, the search range determination unit 115 determines the rear search range wider. Conversely, in the case of a moving image that increases toward the front, the search range determination unit 115 determines a wider search range in the front.

  FIG. 5 shows a schematic diagram of an example in which the subject selected by the user becomes larger backward in time. Frames 501, 502, 503, 504, and 505 form a moving image. Assume that time flows from the frame 501 toward the frame 505.

  In step S205, the system control unit 112 causes the subject recognition unit 114 to recognize the shape of the subject 512 selected by the user. In the example shown in FIG. 5, the subject recognition unit 114 recognizes a subject similar to the subject 512 and its shape within a predetermined range before and after the frame 503. In FIG. 5, the subject recognition unit 114 detects subjects 510, 511, 513, and 514 similar to the subject 512 in the frames 501, 502, 504, and 505.

  Next, the system control unit 112 compares the shapes and sizes of the subjects 510, 511, 513, and 514 with each other based on the shape of the subject 512 selected by the user. Then, the search range determination unit 115 determines the image search range according to the result of comparison of the shape and size by the system control unit 112.

  For example, the search range determination unit 115 calculates the relative sizes of the subjects 513 and 514 with respect to the subject 512 in the backward direction with respect to the frame 503. Similarly, the search range determination unit 115 calculates the relative sizes of the subjects 511 and 510 with respect to the subject 512 in terms of time in the forward direction with reference to the frame 503. Then, the search range determination unit 115 determines whether a large subject larger than the subject 512 is included in front of or behind the frame 503 selected by the user.

  In the example illustrated in FIG. 5, the subject is larger in the backward direction than the frame 503, so the search range determination unit 115 determines the number of frames n2 behind the image search range to be larger. As a result, more frames in which the subject 512 selected by the user is larger can be included in the image search range.

  FIG. 6 shows a schematic diagram of an example in which the subject selected by the user becomes smaller backward in time. In the example illustrated in FIG. 6, the size of the subject decreases as the subject 612 selected by the user moves backward with respect to the subjects 613 and 614 in time. On the contrary, the subject becomes larger as it moves forward in time with the subjects 611 and 610. In such a case, the search range determination unit 115 determines the number of frames n1 ahead of the image search range to be larger. Thereby, more frames in which the subject 612 selected by the user is shown larger can be included in the search range.

  In this embodiment, since the image search range is changed according to the size of the subject selected by the user, the same frame including the subject selected by the user in a larger size can be found, and an efficient search can be realized. The user can easily select a subject to be used as a search index or reference, which facilitates the search operation.

  A description will be given of a modified embodiment in which the image search range is set while the subject selected by the user is continuously detected. For this purpose, step S205 in FIG. 2 is changed so that the period during which the subject selected by the user is continuously detected in the frame is determined as the image search range.

  FIG. 7 is a schematic diagram illustrating an example of the present embodiment. Frames 701, 702, 703, and 704 form a moving image. Assume that time flows from the frame 701 toward the frame 704.

  In step S <b> 205, the system control unit 112 causes the subject recognition unit 114 to recognize the same subject as the subject 712 selected by the user until the subject is no longer recognized in time and forward. In the example illustrated in FIG. 7, the same subject 711 as the subject 712 selected by the user in the frame 702 that is temporally ahead of the frame 703 is detected. However, the same subject as the subject 712 is not detected in the front frame 701 and the rear frame 704.

  The system control unit 112 transfers the recognition result of the subject recognition unit 114 to the search range determination unit 115, and the search range determination unit 115 sets a range in which the same subject as the subject 712 selected by the user is continuously detected as an image search range. decide. In the example shown in FIG. 7, the number n1 of frames indicating the image search range ahead of the frame 703 is 1, and only the frame 702 is included in the image search range after all. On the other hand, the frame number n2 indicating the image search range that is temporally behind the frame 703 is zero. This is because the same subject as the subject 712 selected by the user was not detected in the frame 704 immediately after the frame 703.

  In this embodiment, since a range including the same subject as the subject selected by the user is determined as the image search range, a preferable frame is searched from the frames including the subject selected by the user, and an efficient search is realized. it can. The user can easily select a subject to be used as a search index or reference, which facilitates the search operation.

  In each of the first to third embodiments described above, the image search range is determined beyond the scene, but may be limited to one scene. For example, one file edited with a playlist or the like may be set as one scene. In the first to third embodiments, the temporal image search range is limited to the scene start frame, and the temporal image search range is limited to the scene end frame.

  By limiting the image search range to the same scene in this way, the search can be made efficient.

Claims (7)

Setting means for setting a search condition for searching for a still image from a moving image consisting of a plurality of frames;
Selecting means for selecting a subject included in the moving image;
Search range determining means for determining an image search range of the moving image according to the state of the subject around the frame where the subject is selected by the selection means;
An image search apparatus comprising: search means for searching within the image search range for a frame that includes the same subject as the subject selected by the selection means and includes a subject that satisfies the search condition.   The image search apparatus according to claim 1, wherein the state of the subject is a movement amount between frames of the subject.   The image search apparatus according to claim 2, wherein the search range determination unit determines the image search range to be narrow when the amount of motion is large.   The image search apparatus according to claim 2, wherein the search range determination unit determines the image search range widely when the amount of motion is small.   The image search apparatus according to claim 1, wherein the state of the subject is a change in size of the subject within a frame.   2. The image search apparatus according to claim 1, wherein the search range determination unit determines, as the image search range, a period during which the same subject as the subject selected by the selection unit is continuously detected in a frame. .   The image search apparatus according to claim 1, wherein the search range determination unit determines the image search range within the same scene.

Images