JP5640377B2 - Image processing apparatus, camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, a camera, and an image processing program.

  The following digital cameras are known. This digital camera crops images in a plurality of areas from within the captured image at the same time as capturing (for example, Patent Document 1).

JP 2005-175683 A

  However, with conventional digital cameras, when image recognition is performed on a cropped image, if image recognition is performed only within the cropped range, information necessary for image recognition is insufficient and the accuracy of image recognition is reduced. Could be reduced.

An image processing apparatus according to the present invention includes a partial image extraction unit that extracts a partial image from a predetermined range of a captured image, an image recognition unit that recognizes a subject included in the captured image, and a partial image extraction. Specifying means for identifying a subject included in the partial image extracted by the means based on the recognition result by the image recognition means, and generating an image file based on the partial image extracted by the partial image extraction means And the image file generation means associates information relating to the subject included in the partial image specified by the specifying means and information relating to the subject included in the captured image recognized by the image recognition means to the image file. Record .
The image processing apparatus according to the present invention further includes scene discrimination means for discriminating a shooting scene of the partial image for the partial image .
In the present invention, the scene discrimination means may discriminate the photographic scene of the partial image in consideration of the photographic scene discrimination result for the entire photographic image.
In the present invention, the image file generation means further records information relating to the photographing scene of the partial image specified by the specifying means in association with the image file .
The camera according to the invention, obtain Bei any one of the above image processing apparatus.
An image processing program according to the present invention includes a computer for extracting a partial image from a predetermined range of a captured image, an image recognition procedure for recognizing a subject included in the captured image, A specific procedure for identifying a subject included in a partial image extracted by the partial image extraction procedure based on a recognition result by the image recognition procedure, and an image for generating an image file based on the partial image extracted by the partial image extraction procedure an image processing program for executing the file generation procedure, an image file generation step, information about the subject included in the partial image specified by a particular procedure, the photographed image recognized by the image recognition procedure Information related to the included subject is recorded in association with the image file.
The image processing program according to the present invention is also a computer, as a target partial image, to perform another scene determination procedure to determine the shooting scene of the partial image.

  According to the present invention, it is possible to perform image recognition on an extracted partial image with high accuracy.

It is a block diagram which shows the structure of one Embodiment of a camera. It is a 1st figure which shows the specific example of a crop image. It is a 1st figure which shows the specific example of tag information. It is a 2nd figure which shows the specific example of a crop image. It is a 2nd figure which shows the specific example of tag information. It is the figure which showed typically the position and magnitude | size of the crop range with respect to the whole picked-up image.

  FIG. 1 is a block diagram illustrating a configuration of an embodiment of a camera according to the present embodiment. The camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106. The operation member 101 includes various input members operated by the user, such as a power button, a release button, a zoom button, a cross key, an enter button, a play button, and a delete button.

  The lens 102 is composed of a plurality of optical lenses, but is representatively represented by one lens in FIG. The image sensor 103 is an image sensor such as a CCD or a CMOS, for example, and captures a subject image formed by the lens 102. Then, an image signal obtained by imaging is output to the control device 104.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and controls the camera 100. Note that the memory constituting the control device 104 includes SDRAM and flash memory. The SDRAM is a volatile memory, and is used as a work memory for the CPU to develop a program when the program is executed or as a buffer memory for temporarily recording data. The flash memory is a non-volatile memory in which data of a program executed by the control device 104, various parameters read during program execution, and the like are recorded.

  The control device 104 generates image data in a predetermined image format, for example, JPEG format (hereinafter referred to as “main image data”) based on the image signal input from the image sensor 103. Further, the control device 104 generates display image data, for example, thumbnail image data, based on the generated main image data. The control device 104 generates an image file that includes the generated main image data and thumbnail image data, and further includes header information, and outputs the image file to the memory card slot 105.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium, and the image file output from the control device 104 is written and recorded on the memory card. The memory card slot 105 reads an image file stored in the memory card based on an instruction from the control device 104.

  The monitor 106 is a liquid crystal monitor (rear monitor) mounted on the back surface of the camera 100, and the monitor 106 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. . Further, when the user sets the mode of the camera 100 to the shooting mode, the control device 104 outputs the display image data generated based on the image signal acquired in time series from the image sensor 103 to the monitor 106. As a result, a through image is displayed on the monitor 106.

  In the camera 100 according to the present embodiment, the control device 104 has a function of extracting an image of a predetermined range in a captured image and generating an individual image file based on an instruction from the user. Yes. That is, the user can operate the operation member 101 to specify at least one crop range in the captured image, and the control device 104 extracts a partial image within the crop range specified by the user. A separate image file can be generated based on the extracted crop image and recorded on the memory card.

  In addition, the camera according to the present embodiment performs scene discrimination processing for discriminating a photographic scene and image recognition for recognizing a subject included in the cropped image for a cropped image cropped from the photographic image as described above. Processing can be performed. Hereinafter, a process for performing scene discrimination and image recognition for a cropped image with high accuracy will be described.

  First, a process when image recognition of a crop image is performed with high accuracy will be described. When performing image recognition on the cropped image, the control device 104 uses the image recognition result performed on the entire captured image that is the cropping source. Specifically, as shown in FIG. 2, the control device 104 recognizes the subject included in the cropped image extracted from the cropped range 2a so that the subject recognized by the image recognition on the entire captured image 2b. Only the subject included in the crop range 2a is extracted from the inside, and this is used as the image recognition result of the cropped image extracted from the crop range 2a.

  As a result, when image recognition is performed only on a crop image extracted from within the crop range 2a, only cars that are entirely included in the crop range can be recognized, and only a part is included. Although the building cannot be recognized, if the image recognition is performed on the entire captured image 2b and the result is used for the image recognition of the crop image, the vehicle and the building can be recognized as subjects included in the crop image. it can. That is, there is a possibility that the information of the cropped image 2b may be deteriorated in the image recognition of the cropped image 2b as described above, although the information of the cropped image within the cropped range 2a is insufficient and the accuracy of the image recognition in the cropped image may be lowered. By using the recognition result, the image recognition accuracy of the cropped image can be improved.

  As another example, the image recognition accuracy of the cropped image can be improved by associating the subject in the cropped image with the subject outside the cropped image. For example, if a goalkeeper, goalpost, and soccer ball are shown in the entire shot image, and only the goalkeeper and goalpost are shown in the cropped image, the image recognition result of the entire shot image is used. By adding the soccer ball recognized outside the cropping range, it is possible to recognize with high accuracy that the subjects in the cropped image are the goalkeeper and the goal post.

  Alternatively, even when only one ingredient is shown in the crop image, if other dishes or dishes are shown outside the crop range, the image recognition result of the entire photographed image is used, It is also possible to recognize that the subject is a dish.

  Next, processing when performing scene determination of a cropped image with high accuracy will be described. In the case where the scene determination is performed on the cropped image, the control device 104, in principle, targets the cropped image. However, if the control device 104 determines that the accuracy of scene determination can be improved by taking the cropping range into consideration. Then, scene discrimination is performed for the entire captured image. Various conditions can be considered as conditions for determining whether scene discrimination is performed on the cropped image or on the entire captured image. In the present embodiment, for example, the color of the entire captured image is determined as the determination condition. The case of using as will be described.

  Specifically, the control device 104 specifies the color of the entire captured image from the result of white balance processing at the time of shooting. In other words, in the white balance processing, the type of illumination reflected in the photographed image, for example, street light, fluorescent light, sun, etc., is identified, and the color of the subject is illuminated by the illumination light. Adjust the white balance. In the present embodiment, the color of the entire captured image is specified from the result of the white balance process, and the specified result is used as the determination result.

  For example, in the present embodiment, when the captured image is entirely reddish, the control device 104 performs scene determination on the entire captured image, and otherwise, the cropped image is displayed. Scene discrimination is performed as an object. As a result, scene discrimination can be performed with high accuracy as follows. The fact that scene discrimination can be performed with high accuracy by switching the scene discrimination method based on the color of the entire captured image will be described with a specific example.

  First, an example in which the captured image is entirely reddish, that is, an example in which scene determination is performed on the entire captured image will be described. Here, for example, a case will be described in which the captured image is a human image that has been subjected to flash photography at sunset. At this time, if the cropped image is obtained by cropping so that the face portion of the person in the photographed image becomes large, when scene discrimination is performed for this cropped image, the face portion of the person who is the main subject is flash light. Since it is irradiated, it is difficult to discriminate it from a sunset scene, and there is a high possibility that a cropped image shooting scene is determined to be a portrait image. On the other hand, if scene discrimination is performed for the entire captured image, the entire image is reddish and the face of the person is reflected, so the cropped image capture scene is correctly a sunset portrait scene. It becomes possible to discriminate.

  Next, an example in which the captured image is not entirely reddish, that is, an example in which scene determination is performed on a crop will be described. Here, a case will be described in which, for example, the photographed image is a person image photographed with the landscape in the background when the weather is fine. At this time, when the cropped image is obtained by cropping so that the face portion of the person in the photographed image becomes large, if scene discrimination is performed on the entire photographed image as described above, the crop image has a large person. Although it is shown, the shooting scene is determined to be a landscape portrait scene. On the other hand, if scene discrimination is performed for a cropped image, a person's face is reflected in the cropped image and the background is hardly reflected, so the scene where the cropped image is taken is a portrait scene. It is possible to correctly determine that there is.

  Alternatively, when a cropped image is obtained by cropping a landscape portion in a captured image, if scene determination is performed on the entire captured image, the captured scene is displayed even though only the landscape is captured in the cropped image. Is determined to be a landscape portrait scene. On the other hand, if scene discrimination is performed for a crop image, since a human face is not captured in the crop image, it is possible to correctly determine that the shooting scene of the crop image is a landscape scene. .

  In the camera 100 according to the present embodiment, when an image file (crop image file) of a crop image is generated, the above-described image recognition result and scene determination result for the crop image are recorded in association with the crop image file. The control device 104 records, for example, by embedding an image recognition result or a scene determination result as tag information in the header information of the cropped image file.

  Specifically, as shown in FIG. 3, the control device 104 uses the tag information of each crop image file as information on the captured image (original image) that is the crop source for each of the crop range and the entire captured image. 3a, the number of detected faces 3b, the number of faces recognized as a specific person 3c, the image recognized object type 3d, and the determined scene type 3e are recorded in the tag information of the cropped image file. Each data to be recorded in the tag information is 4-bit data as shown in FIG. 3, and these data may be recorded in the fixed address of the tag, or start data and end data, that is, an indicator is provided. Alternatively, an unnecessary area may be omitted by setting the variable length.

  Hereinafter, as shown in FIG. 4, when cropped image files are generated by cropping images in the cropped ranges 2a, 4a, and 4b from the captured image 2b, specific examples of tag information recorded in the respective crop image files Will be described.

(A) Tag information of a crop image file obtained by cropping an image in the crop range 2a Tag information recorded in a crop image file generated by cropping an image in the crop range 2a shown in FIG. As shown in a). First, as the information 3a regarding the captured image 2b regarding the cropped range 2a, the position of the cropped range 2a in the entire captured image 2b and the size of the cropped range 2a are recorded.

  Here, the position of the crop range 2a is represented by the ratio of the position of the crop range of the crop range to the height H and width W of the captured image 2b shown in FIG. 6, and in the example shown in FIG. The position is a / H and the horizontal position is c / W. Further, the size of the crop range 2a is represented by the ratio of the height b and width d of the crop range 2a to the height H and width W of the captured image 2b shown in FIG. 6, and in the example shown in FIG. The height of 2a is b / H and the width is d / W. Further, as the information 3a regarding the captured image 2b about the entire captured image 2b, the file name of the captured image file, for example, IMG — 0001. jpg is recorded.

  As the number 3b of detected faces in the cropping range 2a, the number of human faces detected from the cropped image by the image recognition process described above is recorded. As shown in FIG. 4, since no human face is detected from within the cropping range 2a, 0 is recorded in the number of detected faces 3b within the cropping range 2a as shown in FIG. 5A. On the other hand, the number of faces detected from the photographed image 2b, that is, four, is recorded in the number of faces 3b detected for the entire photographed image 2b.

  As the number of recognized faces 3c in the cropping range 2a, a specific person recognized based on face data of a person registered in advance among the faces of persons detected from the cropped image by the image recognition processing The number of faces is recorded. In the present embodiment, the faces of Mr. A and Mr. B are registered in advance, and when the control device 104 detects a person's face by image recognition processing, Mr. A and Mr. B registered therein are detected. If there is a person whose facial features match, the person can be recognized as Mr. A or Mr. B.

  As shown in FIG. 4, since the face of a specific person is not detected in the crop range 2a, 0 is recorded in the number 3c of recognized faces in the crop range 2a as shown in FIG. 5A. The On the other hand, the number 3c of recognized faces for the entire captured image 2b records the number of faces of Mr. A and Mr. B detected from the captured image 2b, that is, 2.

  As the recognized object type 3d in the cropping range 2a, the type of the object recognized in the cropped image in the cropping range 2a by the image recognition processing is recorded. As described above with reference to FIG. 2, since the car and the building are recognized in the crop image in the crop range 2a, as shown in FIG. 5A, the recognized object type 3d in the crop range 2a. These objects are recorded in. On the other hand, in the recognized object type 3d for the entire captured image 2b, all objects recognized in the captured image 2b, that is, cars, buildings, Mr. A, Mr. B, and trees are recorded.

  In the determined scene type 3e in the crop range 2a, the scene type of the cropped image in the crop range 2a determined by the above-described scene determination method, for example, a landscape is recorded. On the other hand, the determined scene type 3e for the entire photographed image 2b records the scene type of the photographed image 2b, for example, a landscape portrait.

(B) Tag information of a crop image file obtained by cropping an image in the crop range 4a Tag information recorded in a crop image file generated by cropping an image in the crop range 4a shown in FIG. As shown in b). In FIG. 5B, the information about the entire captured image 2b is the same as that in FIG.

  First, in the information 3a related to the captured image 2b for the crop range 4a, for example, the vertical position a1 / H and the horizontal position c1 / W of the crop range 4a are recorded as the positions, and the height b1 / H, The width d1 / W is recorded. In the number of detected faces 3b in the cropping range 4a, two faces are detected from the cropping range 4a as shown in FIG. 4, so 2 is shown in FIG. 5 (b). To be recorded.

  In the number 3c of recognized faces in the cropping range 4a, as shown in FIG. 4, since Mr. A and Mr. B registered in advance are detected from within the cropping range 4a, FIG. 2 is recorded as shown in FIG. In the recognized object type 3d in the cropping range 4a, as shown in FIG. 4, since Mr. A, Mr. B, and the tree are recognized in the cropped image in the cropping range 4a, these objects are recorded. The

  In the determined scene type 3e in the crop range 4a, the scene type of the cropped image in the crop range 4a determined by the above-described scene determination method, for example, a portrait is recorded.

(C) Tag information of a crop image file obtained by cropping an image in the crop range 4b Tag information recorded in a crop image file generated by cropping an image in the crop range 4b shown in FIG. As shown in c). In FIG. 5C, the information about the entire photographed image 2b is the same as that in FIG.

  First, in the information 3a regarding the captured image 2b regarding the cropping range 4b, for example, the vertical position a2 / H and the horizontal position c2 / W of the cropping range 4b are recorded as the positions, and the height b2 / H, The width d2 / W is recorded. In the number of detected faces 3b in the cropping range 4b, two faces are detected from the cropping range 4b as shown in FIG. 4, so 2 is shown in FIG. 5C. To be recorded.

  In the number 3c of recognized faces in the cropping range 4b, as shown in FIG. 4, there are only unknown persons in the cropping range 4b, and A and B who are registered in advance are not detected. As shown in 5 (c), 0 is recorded. In the recognized object type 3d in the cropping range 4b, as shown in FIG. 4, since the building is recognized by the cropped image in the cropping range 4b, this object is recorded.

  In the determined scene type 3e in the crop range 4b, the scene type of the cropped image in the crop range 4b determined by the above-described scene determination method, for example, a landscape portrait is recorded.

  As described above, by recording the image recognition result and the scene determination result as the tag information in the header information of the cropped image file, it becomes possible to search for the image file later using such information. For example, when a crop image in which Mr. A is shown is detected, an image file in which Mr. A is recorded in the recognized object type 3d may be extracted.

  Also, not only information within the cropping range but also information outside the cropping range, that is, information about the entire captured image is recorded in the tag information of the cropped image file, so that information outside the cropping range is taken into account when searching for images. It is also possible to perform a search. For example, when Mr. B is designated as a search condition, an image file in which Mr. B is recorded in the recognized object type 3d within the cropping range is extracted, and the recognized object of the entire captured image is extracted. By extracting the image file in which Mr. B is recorded in type 3d, even if Mr. B does not appear in the cropping range, the image taken at the event where Mr. B participated is extracted. Is possible.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 104 extracts a cropped image from the cropped range of the captured image, performs image recognition on the entire captured image, recognizes a subject included in the captured image, and crops based on the recognition result. The subject included in the image was specified. As a result, the image recognition accuracy of the cropped image can be improved even when the information is insufficient for only the cropped image within the cropped range.

(2) The control device 104 extracts a cropped image from the cropped range of the captured image, and determines the captured scene of the cropped image for the extracted cropped image. Thereby, it is possible to discriminate the shooting scene of the cropped image with high accuracy.

(3) The control device 104 discriminates the shooting scene of the cropped image in consideration of the scene discrimination result for the entire shot image. Thereby, it is possible to discriminate the shooting scene of the cropped image with higher accuracy.

(4) The control device 104 records the image recognition result and the scene discrimination result of the crop image as tag information in the header information of the crop image file. As a result, the image recognition result and the scene discrimination result of the cropped image can be used in subsequent processing such as image file search.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the control device 104 has described an example in which tag information related to the crop range and tag information related to the entire captured image are recorded in the header information of the crop image file. However, only the tag information related to the crop range may be recorded in the header information of the crop image file.

(2) In the above-described embodiment, the case where the present invention is applied to the camera 100 has been described. However, the present invention can also be applied to other devices having a function of cropping an image within a crop range specified in a captured image to generate a crop image file.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

100 Camera, 101 Operation member, 102 Lens, 103 Image sensor, 104 Control device, 105 Memory card slot, 106 Monitor

Claims (7)

Partial image extraction means for extracting a partial image from a predetermined range of the photographed image;
Image recognition means for recognizing a subject included in the captured image for the captured image;
Specifying means for specifying a subject included in the partial image extracted by the partial image extraction means based on a recognition result by the image recognition means;
Image file generation means for generating an image file based on the partial image extracted by the partial image extraction means ,
The image file generation means includes information on the subject included in the partial image specified by the specifying means and information on the subject included in the captured image recognized by the image recognition means in the image file. An image processing apparatus for recording in association with each other . The image processing apparatus according to claim 1.
An image processing apparatus further comprising a scene discriminating unit for discriminating a photographing scene of the partial image for the partial image. The image processing apparatus according to claim 2 ,
The scene discrimination unit, in consideration also the discrimination result of the shooting scene that targets the captured whole image, an image processing apparatus for determining a photographic scene of the partial image. The image processing apparatus according to claim 2 or 3 ,
The image file generation means, the image processing apparatus further recorded in association with information related to a shooting scene of the partial image specified by the specifying means in the image file. The camera provided with the image processing apparatus as described in any one of Claims 1-4 . On the computer,
A partial image extraction procedure for extracting a partial image from a predetermined range of a captured image;
An image recognition procedure for recognizing a subject included in the captured image for the captured image;
A specific procedure for identifying a subject included in the partial image extracted by the partial image extraction procedure based on a recognition result by the image recognition procedure;
An image processing program for executing an image file generation procedure for generating an image file based on the partial image extracted by the partial image extraction procedure ,
In the image file generation procedure, information on the subject included in the partial image specified by the specifying procedure and information on the subject included in the captured image recognized by the image recognition procedure are stored in the image file. An image processing program for recording in association. The image processing program according to claim 6,
On the computer,
Examples target partial image, the image processing program for further executing the scene determination procedure to determine the shooting scene of the partial image.

Images