WO2013131536A1

WO2013131536A1 - Image recording method and corresponding camera device

Info

Publication number: WO2013131536A1
Application number: PCT/EP2012/001073
Authority: WO
Inventors: Anders WIKANDER
Original assignee: Sony Mobile Communications Ab
Priority date: 2012-03-09
Filing date: 2012-03-09
Publication date: 2013-09-12

Abstract

Methods and camera devices are disclosed where image objects (35, 36, 37, 38, 39, 310) from a plurality of images (30, 31, 32, 33) may be selected and combined to form a new image (34).

Description

TITLE OF THE INVENTION

Image recording method and corresponding camera device FIELD OF THE INVENTION

The present application relates to methods for recording images and to corresponding camera devices. In particular, the present invention relates to possibilities for improving a quality of an image finally obtained. The term camera device, as used herein, does not only refer to dedicated camera devices, but also to camera devices incorporated in other electronic devices, for example mobile phones, smartphones, laptop computers or mobile gaming devices.

BACKGROUND OF THE INVENTION

With the arrival of digital cameras and their replacing of conventional analog cameras to a large extent, image processing and image modification have become increasingly popular and easier to perform. In the recent years, camera devices increasingly themselves have offered possibilities for processing images recorded, for example color filtering and the like. Furthermore, by analyzing images recorded by a corresponding sensors the quality of a final image could be improved to match certain criteria. For example, in portrait photography methods are known which ensure that the final image is only recorded if the subject person of the portrait is smiling and/or has his or her eyes open. Also, some cameras offer the possibility to automatically select a "best picture" out of a plurality of images recorded from the same scene according to a predefined criterion.

However, in some situations a problem may occur that even if a plurality of images is recorded of the same scene, none of the images is really satisfying. For example, when recording images of a group of persons, even if a plurality of images is recorded, in one image a person may have his or her eyes closed, in another image another person may be moving this causing the image of that person to be blurred, etc., such that in the end none of the images recorded may be completely satisfying. Therefore, there is a continuing need for methods and camera devices which aid in obtaining images which have a better chance of meeting predetermined quality criteria.

SUMMARY OF THE INVENTION

According to an embodiment, a method as defined in claim 1 is provided. According to another embodiment, a camera device as defined in claim 12 is provided. The dependent claims define preferred embodiments. According to an embodiment, a method is provided, comprising: recording a plurality of images of a scene, automatically selecting one image object from each of at least two sets of image ob- jects, each set comprising corresponding image objects of at least two of the plurality of images, and automatically generating a new image based on the selected image objects. By selecting one image object from each of the at least two sets and generating a new image based on the selected image object, image objects from different ones of the plurality of images may be combined, thus in some cases a desired final image may be obtained. In general, corresponding image objects refer to the same item in the scene or the same portion of an item in the scene in different ones of the images, for example eyes, mouth, and/or face of the same person as recorded in the different ones of the plurality of images. The selecting may be performed based on a predetermined criterion, like open eyes, smiling/laughing mouth, focus, absence of movement, non-smiling mouth, specific movement of a person on the image etc. or a combination of two or more criteria. In such a way, in an embodiment for example when the scene comprises a plurality of persons, it may be ensured that all persons have their eyes open or are smiling in the final image provided that each person has its eyes open and smiles in at least one of the images.

The method may further comprise identifying the at least two sets of corresponding image objects in the plurality of images recorded.

The predetermined criterion may be user selectable to satisfy certain needs.

In a case the scene comprises a plurality of faces, in an embodiment different sets of image objects may comprise image objects for different faces.

The image objects may for example be eyes, mouth, faces or limbs of a person.

The method may further comprise transmitting the plurality of images to an external device to have at least part of the automatically selecting and the automatically generating (i.e. part or all of the automatically selecting and/or part or all of the generating) performed at the external device.

In this case, the method may further comprise receiving the new image from the ex- temal device.

According to another embodiment, a camera device is provided, comprising: an image sensor, and a processor arrangement, that processor arrangement being configured to control the image processor to record a plurality of images of a scene, to select one image object from each of at least two sets of image objects, each set comprising correspond- ing image objects of at least two of the plurality of images, and to generate a new image based on the selected image object.

The camera device may further comprise conventional camera device elements like a lens arrangement, a memory, or a display. The camera device may be a dedicated camera device or may be part of another electronic device like the mobile phone, a smartphone, a laptop computer or a gaming device.

The processor arrangement may be configured to communicate with an external de- vice to have at least part of the selecting and the generating performed at the external device. In other words, it is not required that all of the processing is performed within the processor arrangement.

With such a camera device, in some embodiments one or more of the methods men- tioned above may be implemented.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting embodiments of the invention will be described with reference to the at- tached drawings.

Fig. 1 is a block diagram of a camera device according to an embodiment.

Fig. 2 is a flow chart illustrating a method according to an embodiment.

Fig. 3 shows example images for illustrating some features of some embodiments. DETAILED DESCRIPTION In the following embodiments will be described in detail with reference to the attached drawings. However, it has to be emphasized that the present invention is not limited to the embodiments described. Moreover, features from different embodiments may be combined with each other unless specifically noted otherwise. On the other hand, describing an embodiment with a plurality of features is not to be construed as indi- eating that all these features are necessary for practicing the invention, as other embodiments may comprise less features and/or alternative features.

In Fig. 1 , a camera device 10 according to an embodiment is schematically shown. Camera device 10 may be a dedicated camera device, i.e. a camera device essen- tially used only for image recording, e.g. recording of still images and/or recording of video images, but may also be a device having other functions besides the camera functions described in more detail in the following. For example, camera device 10 may be a mobile phone, a smartphone, a laptop computer or a mobile gaming device incorporating camera functionality as described in the following. Elements for providing such additional functions, for example telecommunication capabilities in the case of smartphones or mobile phones, may be implemented in any conventional manner and are not discussed here for simplicity's sake. Camera device 10 as shown in Fig. 1 comprises a lens arrangement 11 , for simplicity's sake depicted as a single lens, for guiding light to an image sensor 12. Lens arrangement 11 may for example be a prime lens arrangement having a fix focal length or a zoom lens arrangement having a variable focal length and may be implemented in any conventional manner. Image sensor 12 may for example by a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semi conductor) sensor.

Image sensor 12 in the embodiment of Fig. 1 is coupled with an image processor arrangement 13 which may comprise one or more processors like a conventional multi- purpose processor, a digital signal processor or a dedicated image processor. Image processor arrangement 13 in some embodiments may solely be used for image recording and processing. However, in other embodiments image processor 13 may also be used to control and/or execute other functions of camera device 10 besides image recording and processing functions.

Image processor 13 is coupled to a memory 14 where images recorded via sensor 2 may be stored. Memory 14 may comprise one or more of a solid state memory, a flash memory or also a removable memory card, to give some examples for suitable memories. Furthermore, image processor 13 is coupled with a display 15, for exam- pie for displaying images recorded or also for displaying menus to a user of device 10. Finally, image processor 13 in the embodiment of Fig. 1 is coupled to a user input 16 allowing a user to influence the behavior of camera device 10. Input 16 may for example comprise one or more input elements like a joystick, buttons or the like. In some embodiments, display 15 may be a touch sensitive display and therefore also be used as input device 16 or part thereof.

In an embodiment, as will be described in the following in more detail, image proces- sor 13 is configured, for example programmed, to control camera device 10, for example sensor 12, to record a plurality of images of a scene. Image processor 13 then processes the recorded images by selecting image objects from different of the plurality of images and combining them to a new image. In some embodiments, image processor 13 may be configured to communicate with one or more external device(s) 17 via a communication channel 18. The communication may be a communication via a network like the internet or a direct connection and may be performed by a wireless connection (e.g. WLAN or a cellular network), a wire-based connection (e.g. LAN or USB) or a combination of wireless and wire- based connections. In some embodiments, the plurality of images may be transmitted to external device(s) 17 for processing, e.g. for performing the above-mentioned selecting and combining or part thereof. The new image may then be received back from the external device(s) 17 by image processor 13, and/or external device(s) may publish the new image in some manner, e.g. on the internet (e.g. in a gallery).

A corresponding method will now be described with reference to Fig. 2. The method of Fig. 2 may be implemented in camera device 10 of Fig. 1 , for example by programming image processor arrangement 13 accordingly, but may also be implemented independently therefrom.

Furthermore, examples for illustrating some of the actions described with reference to Fig. 2 will be explained with reference to Fig. 3.

The example images presented with reference to Fig. 3, however, are merely given as illustration and are not to be construed as limiting, as the embodiments of the present invention may be used to record virtually any kind of images of any kind of scen- e desired. At 20 in Fig. 2, a plurality of images of a scene is recorded. For example, upon an appropriate user input like the pressing of a shutter button or the touching of a certain element on a touch screen, a predetermined, possibly user configurable number of images may be recorded, for example three images, four images or five images. In some embodiments, also images recorded prior to the above-mentioned user action may be taken into account. For example, some camera devices continually record images or continually record images when being in a certain mode of operation, and images prior to the user action, for example prior to pressing a shutter button, may be taken as at least part of the plurality of recorded images. The recorded images may be stored in a memory like memory 14 of Fig. 1.

As an example, in Fig. 3 four images 30, 31 , 32 and 33 are recorded of a scene where three persons, in the following referred to as father (in the middle), mother (at the right) and child (at the left) are sitting on a sofa.

As can be seen, in the example of Fig. 3 none of the images 30-33 corresponds to an ideal family portrait. For example, in image 30, one eye of the mother is closed, the child is crying and the child has its eyes closed. Also in image 31 , for example the child has closed eyes. In the image 32, for example both father and mother have closed eyes, whereas in image 33 for example the father is yawning and the child has one eye closed.

At 21 , sets of corresponding image objects are identified, each set comprising one image object from at least one of the images, for example from all of the images, the image objects of one set corresponding with each other, for example corresponds to the same portion of the scene. For example, in the images shown in Fig. 3 a first set may comprise four image objects corresponding to the eyes of the father in each image, a second set may comprise four image objects corresponding to the mouth of the father in each image, a third set may comprise four image objects corresponding to the eyes of the mother in each image, a fourth set may comprise four image objects corresponding to the mouth of the mother in each image, a fifth set may comprise four image objects corresponding to the eyes of the child in each image and a sixth set may comprise four image objects corresponding to the mouth of the child in each image. At 22, one image object from each set is selected according to a (i.e. one or more) predefined criterion. The criterion may be set by the manufacturer of a corresponding camera device or may be user-defined. For example, for the images of Fig. 3 the cri- terion for the sets containing image objects corresponding to eyes (first, third and fifth set in the example above) may be that both eyes are open, and for the set comprising image objects corresponding to a mouth the criteria might be that the mouth is smiling or laughing. In the example of Fig. 3, this may lead to the selection indicated by boxes. For example, for the first set (eyes of the father) an image object 35 may be selected as both eyes are open, for the second set (mouth of father) an image object 36 may be selected as the father is smiling, for the third set (eyes of the mother) an image object 37 may be selected (both eyes open), for the fourth set (mouth of mother) an image object 39 may be selected (mouth laughing), for the fifth set (eyes of the child) an image object 38 may be selected (both eyes open), and for the sixth set (mouth of the child) an image object 310 (mouth laughing) may be selected.

In case more than one of the image objects in a set matches the predetermined crite- rion, a selection may be made according to a default rule, for example from the first image recorded where the image object matches the criteria. Likewise, in case none of the image objects matches the predetermined criteria, also such a default selection may be made. In some cases also then a "second best" criterion may be used, for example to select a closed mouth even if it is not smiling or laughing rather than a yawn (for example mouth of the father of image 31 or 32 rather than mouth of the father in image 33).

At 23, a new image is generated based on the selected objects. For example, one of the images may be taken, for example image 30 in Fig. 3, and non-selected image objects (for example in image 30 eyes and mouths of mother and child) may be replaced by the corresponding selected image objects from other images. An example resulting image is indicated by reference numeral 34 in Fig. 3 and in the example shown image 34 comprises all the selected image objects. In this way, in the exam- pie of Fig. 3 an image has been generated where all three persons, i.e. father, mother and child, have their eyes open and are smiling or laughing.

As mentioned above with respect to Fig. 2, the operations at 21 , 22 and/or 23 may be performed by one or more external device(s). In this case, the necessary data, e.g. the recorded images, may be transmitted to the one or more external device(s) via a suitable communication connection.

It should be noted that the embodiment of Fig. 2 serves merely as an example, and certain steps may be omitted or performed in a different order. For example, after identifying each set, immediately a corresponding image object may be selected, such that the actions described with reference to 21 or 22 are executed in an interwoven manner. In still other embodiments, the sets may not be expressly identified. For example, in an embodiment the image objects may be identified in a first image (or any other image) of the plurality of images, for example image 30 in Fig. 3, and only for those image objects which do not match the predefined criterion (in case of Fig. 3 eyes open and mouth smiling or laughing) corresponding image objects are identified in further images. This may be done in a consecutive manner until for all image objects an image object matching the predefined criterion has been found, or until the last image has been processed.

In case one or more of the sets does not contain an image object matching the corresponding criterion, a warning may be issued to a user of the corresponding camera device.

It should be noted that in other embodiments also other kinds of image objects and other criteria than the one presented as example with reference to Fig. 3 may be used. For example, also the complete face may form an image object, and suitable criteria then may comprise eyes open, smiling, laughing, etc. As a matter of course, also the opposite states may be used as criteria, for example eyes closed, non- smiling, etc., in case a photographer would like to obtain a corresponding image. Still another criterion may include focus (if it is desired to have all image objects focused). Along the same lines, a further criterion may be absence of motion blur. In particular when photographing children or animals in poor light conditions, the subject (for ex- ample child or animal) may not be holding still during the exposure time, which may lead to unwanted motion blur in the image.

A further criterion may be a general posture of bodies or body limbs. For example, a photographer may want to obtain an image where all persons in the image jump in the air simultaneously, and for such cases a jumping posture may be the corresponding criteria, and the image object may comprise certain limbs like legs or also the complete body. In another embodiment, the criterion may be that persons in the image have to hold their hands up.

As already mentioned above with respect to animals, embodiments of the present invention are not limited to recording images of persons, but may generally apply to cases where image condition may change for example through movement or also rapid change of light conditions and a photographer may desire to have a plurality of image objects be matching certain criteria.

It should be noted that in some embodiments also the type of image objects may be user configurable. For example, in some instances a photographer may want the subjects to have their eyes open, but may not care about the mouth, i.e. whether they smile or look serious. In still other embodiments, a user may select to which parts of the image the process described above should be applied. For example, for portraits of a group of persons on a crowded square, some persons not belonging to the group may also be part of the image, and such persons may be excluded from consideration.

It should be noted that in some instances slight motions may occur between the images recorded. In this case, conventional algorithms like face detection algorithms may be used to compensate for such movement. Also, it should be noted that embodiments of the present invention are not only applicable to groups of persons, but also to single persons where for example an image of a person generated should match more than one criteria, for example eyes open on the one hand and mouth smiling on the other hand. In this case, for example through the method described above the eyes may be taken from one image of the plurality of images, and the mouth may be taken from another image of the plurality of images.

It should be noted that the above methods may for example be carried out in a par- ticular operation mode of a corresponding camera device and not carried out in other operation modes of the camera device.

As can be clearly seen from the above, numerous modifications and alterations are possible, and therefore the scope of the invention is not to be seen as restricted to the embodiments described.

Claims

1. A method, comprising:

recording a plurality of images of a scene,

automatically selecting one image object from each of at least two sets of image objects, each set of image objects comprising corresponding image objects in at least two of the plurality of images, and

automatically generating a new image based on the selected image objects. 2. The method of claim 1 , wherein the new image is generated to comprise the selected image objects.

3. The method of claim 1 or 2, wherein generating the new image comprises replacing at least one non-selected image object in one of the plurality of images with a re- spective selected image object.

4. The method of any one of claims 1-3, wherein the selecting of one image object comprises selecting the one image object based on a predetermined criterion. 5. The method of claim 4, wherein the predetermined criterion comprises at least one of the group consisting of eyes open, mouth smiling, mouth laughing, mouth non- smiling, absence of motion blur, person jumping, face focused, person focused, hand of person raised. 6. The method of claim 4 or 5, wherein the predetermined criterion is user selectable.

7. The method of any one of claims 1-6, wherein the image objects comprise at least one image object selected from the group consisting of eyes, a mouth, a face, a limb, and a person.

8. The method of any one of claims 1-7, further comprising identifying the at least two sets of corresponding image objects.

9. The method of any one of claims 1 -8, wherein recording the plurality of images comprises recording at least one of the plurality of image prior to a user action instructing the recording of the plurality of images. 0. The method of any one of claims 1-9, further comprising transmitting the plurality of images to an external device to have at least part of the automatically selecting and the automatically generating performed at the external device.

1 1 . The method of claim 10, further comprising receiving the new image from the ex- ternal device.

12. A camera device, comprising:

an image sensor, and

a processor arrangement, wherein the processor arrangement is configured to con- trol the image sensor to record a plurality of images and is further configured to select one image object from each of at least two sets of image objects, each set comprising corresponding image objects in at least two of the plurality of images, and to generate a new image based on the selected image object. 13. The camera device of claim 12, wherein the processor arrangement is configured to communicate with an external device to have at least part of the selecting and the generating performed at the external device.

14. The camera device of claim 12 or 13, wherein the camera device is one of a dedi- cated camera device, a mobile phone, a smartphone, a portable gaming device and a laptop computer.

15. The camera device of any one of claims 12 or 14, wherein the device is configured to perform the method of any one of claims 1 -9.