CN102207950B - Electronic installation and image processing method - Google Patents

Abstract

The invention discloses electronic installation, image processing method and program. This electronic installation includes memorizer, controller and output unit. Memorizer storage is classified as multiple images of multiple groups, indicate multiple affair character information projects of the feature of object specific to each event, and instruction is for for each group selection multiple Rule Information projects by the rule of the presentation graphics of the event of image expression, the plurality of Rule Information project is different for each event and everyone relevant from event. Controller extracts multiple metamessage project based on multiple affair character information projects from multiple images of each group, senior metamessage is analyzed from the metamessage project extracted, with derive have expressed what and in multiple images event relevant to whom, and the presentation graphics based on the event selecting representative to derive with the corresponding Rule Information project of event of derivation from multiple images. Output unit is for the thumbnail image of each group of output presentation graphics.

Description

Electronic installation and image processing method

Technical field

The present invention relates to the image processing method and program that can determine the electronic installation of the image representing this event and this electronic installation from the motion image data project relevant to certain event (event) or Still image data project.

Background technology

Currently exist the technology moving image being made up of multiple scenes or rest image being grouped and being extracted the presentation graphics representing each group.

Such as, Japanese Patent Application Laid-Open No.2010-9608 is (following, it is called patent documentation 1) disclose following content: based on the instruction of user, multiple images are grouped, and the image that the image desired by user is included from this group extracts, as the presentation graphics of each group.

Additionally, Japanese Patent Application Laid-Open No.2003-203090 is (following, it is called patent documentation 2) disclose a kind of image space display packing, wherein based on the characteristic quantity extracted from image, by similar image sets together into group, and extract image one by one from each group to show.

Summary of the invention

But, in the technology disclosed in patent documentation 1, user manually determines presentation graphics, the time and efforts of its cost user.

It addition, in the technology disclosed in patent documentation 2, the similarity of image is by utilizing the distance between characteristic quantity (signal intensity, for instance histogram feature, edge feature and textural characteristics) as a reference to determine. But, in the situation using the characteristic quantity being only made up of signal intensity, when namely box lunch image self does not have the similarity of characteristic quantity, user is likely to hope and these images is categorized as same group. Technology disclosed in patent documentation 2 is difficult to support this situation.

It addition, by utilizing the subordinate semantic information detected by technology such as face detection/face recognition, smile's identifications, compared with the situation of the characteristic quantity being only made up of signal intensity, significant classification can be performed and processes. But, as the presentation graphics of the scene of serious event, it is suitable to be not considered as with smile or corresponding image of laughing. Additionally, there are scenario described below: even if in cheerful and light-hearted event, it is also possible to the smile of the complete stranger of user detected, and, it is inappropriate that this scene is extracted representatively property image.

It addition, in detect the situation of scene of multiple candidate that can be presentation graphics from certain image sets, when namely box lunch uses subordinate semantic information, it is also difficult to judge which scene will be set to presentation graphics.

In view of above-mentioned environment, it is desirable to provide a kind of electronic installation, image processing method and program, it can select reflect the details of event and be adapted as the image of presentation graphics from multiple images relevant to certain event.

According to embodiments of the invention, it is provided that a kind of electronic installation, it includes memorizer, controller and output unit. This memorizer storage is classified as multiple images of multiple groups; Multiple affair character information projects, the plurality of affair character information project indicates the feature of object specific to each event; And multiple Rule Information project, the instruction of the plurality of Rule Information project is for representing by the rule of presentation graphics of the event of multiple image expressions for each group selection, and the plurality of Rule Information project is different for each event and everyone relevant from event. This controller is based on multiple affair character information projects, multiple metamessage project is extracted from multiple images of each group, senior metamessage is analyzed from the multiple metamessage projects extracted, to derive what relevant with whom of the plurality of image expression, and based on the Rule Information project corresponding with the event derived, from multiple images, select to represent the presentation graphics of the event derived. The thumbnail image of this output unit presentation graphics selected by each group of output.

Utilize this structure, this electronic installation takes out multiple metamessage project, and derive the event expressed by the multiple images in each group, and it is subsequently based on the Rule Information project alternatives presentation graphics corresponding to this event, as a result of which it is, the details of the event that reflects can be selected and is adapted as the image of presentation graphics. Further, since above-mentioned Rule Information project is different for everyone relevant from event, therefore such as depends on the degree of depth of relation between the people relevant to event and user, be also different by selected presentation graphics. Therefore, this electronic installation can select the presentation graphics of the best for the user of electronic installation. Herein, image not only includes the rest image caught at first by stationary cameras, also includes the rest image (frame) extracted from moving image.

Memorizer can store personal characteristic information, and the instruction of this personal characteristic information has the feature of the people of predetermined relationship with user. In this case, controller extracts metamessage project based on personal characteristic information and multiple affair character information project.

Correspondingly, by identifying specific people, this electronic installation can derive the event relevant to specific people, and correspondingly selects presentation graphics.

For each event, multiple Rule Information projects can include the multiple metamessage projects being included in presentation graphics and multiple score information project, and each of the plurality of score information project indicates the mark corresponding with the importance degree of metamessage project. In this case, the mark corresponding with the respective metamessage project of multiple images can be added by controller based on multiple score information projects, and the image with highest score is elected to be presentation graphics.

Correspondingly, by arranging the mark corresponding with the importance degree of each metamessage project for each event, electronic installation can reliably select to have expressed best the presentation graphics of each event.

Output unit can together with thumbnail image output character information, what and event the instruction of this character information have expressed and who is correlated with.

Correspondingly, electronic installation can present the thumbnail image of presentation graphics, and also makes user easily understand " whose " event indicated by the event expressed by presentation graphics and " what " event.

Controller can be chosen with the presentation graphics of the predetermined number of balloon score, and exports the thumbnail image of the presentation graphics of predetermined number so that the presentation graphics with higher fractional has bigger visibility region.

Correspondingly, by exporting presentation graphics according to its mark, compared with the situation that a presentation graphics is output, electronic installation enables to user and the details of event is more easily understood. Herein, phrase " output thumbnail image makes the presentation graphics with higher fractional have bigger visibility region " such as includes showing multiple thumbnail image, carry out lap image with the order of mark simultaneously, and change the size of thumbnail image with the order of mark.

According to another embodiment of the present invention, it is provided that a kind of image processing method, including storage herein below: be classified as multiple images of multiple groups; Multiple affair character information projects, the plurality of affair character information project indicates the feature of object specific to each event; And multiple Rule Information project, the instruction of the plurality of Rule Information project is for representing by the rule of presentation graphics of the event of multiple image expressions for each group selection, and the plurality of Rule Information project is different for each event and everyone relevant from event. Based on multiple affair character information projects, from multiple images of each group, extract multiple metamessage project. From the metamessage project extracted, analyze senior metamessage, with derive have expressed what and in multiple images event relevant to whom. Based on the Rule Information project corresponding with the event derived, from multiple images, select to represent the presentation graphics of the event derived. The thumbnail image of the presentation graphics selected by each group of output.

According to still another embodiment of the invention, it is provided that a kind of program, this program makes electronic installation perform storing step, extraction step, derivation step, selects step and output step. In storing step, store herein below: be classified as multiple images of multiple groups; Multiple affair character information projects, the plurality of affair character information project indicates the feature of object specific to each event; And multiple Rule Information project, the instruction of the plurality of Rule Information project is for representing by the rule of presentation graphics of the event of multiple image expressions for each group selection, and the plurality of Rule Information project is different for each event and everyone relevant from event. In extraction step, based on multiple affair character information projects, from multiple images of each group, extract multiple metamessage project. Deriving in step, by analyzing senior metamessage from the metamessage project extracted, with derive have expressed what and in multiple images event relevant to whom. In selecting step, based on the Rule Information project corresponding with the event derived, from multiple images, select to represent the presentation graphics of the event derived. In output step, the thumbnail image of the presentation graphics selected by each group of output.

As it has been described above, according to embodiments of the invention, it is possible to select reflect the details of event and be adapted as the image of presentation graphics from multiple images relevant to certain event.

In view of the following detailed description to its optimal mode embodiment (as shown in the drawing), these and other objects of the present invention, feature and advantage will be apparent from.

Accompanying drawing explanation

Fig. 1 shows the diagram of the hardware configuration of PC according to an embodiment of the invention;

Fig. 2 shows for being shown that application selects the diagram of the functional block of presentation graphics by the image of PC according to an embodiment of the invention;

Fig. 3 shows the diagram that the presentation graphics in Fig. 2 selects the details of unit;

Fig. 4 shows the flow chart of the flow process processed by the presentation graphics selection of PC according to an embodiment of the invention;

Fig. 5 is the diagram conceptually illustrating process, and wherein, PC derives highest metamessage from subordinate metamessage according to an embodiment of the invention;

Fig. 6 is the diagram conceptually illustrating the state selecting the selection of presentation graphics to process from motion image data in an embodiment of the present invention;

Fig. 7 shows the diagram of the display example of the thumbnail of presentation graphics in an embodiment of the present invention;

Fig. 8 shows the diagram of the display example of the thumbnail of presentation graphics in another embodiment of the invention;

Fig. 9 shows the diagram of the display example of the thumbnail of the presentation graphics in another embodiment of the present invention; And

Figure 10 shows the flow chart of the flow process of the presentation graphics selection process of PC according to another embodiment of the present invention.

Detailed description of the invention

Hereinafter, will be described with reference to the accompanying drawings embodiments of the invention.

(hardware configuration of PC)

Fig. 1 shows the diagram of the hardware configuration of PC (personal computer) according to an embodiment of the invention. As it is shown in figure 1, PC100 is provided with the bus 14 of CPU (CPU) 11, ROM (read only memory) 12, RAM (random access storage device) 13, input and output interface 15 and the those above assembly that is connected to each other.

As necessary, CPU11 accesses RAM13 etc., and while performing various types of computings, performs the overall control of the whole frame of PC100. ROM12 is nonvolatile memory, and wherein fixed storage is by the firmware of the OS performed by CPU11 and such as program and many kinds of parameters. RAM13 is used as the working region etc. of CPU11, and, store OS, executory various application or various data items being processed temporarily.

Display 16, input block 17, memorizer 18, communication unit 19, driver element 20 etc. are connected to input and output interface 15.

Display 16 is the display device using liquid crystal, EL (electroluminescent), CRT (cathode ray tube) etc. Display 16 can be embedded within PC100, or can be coupled externally to PC100.

Input block 17 is such as the location equipment of such as mouse, keyboard, touch panel or another operation device etc. Including in the situation of touch panel at input block 17, touch panel can be integrated with display 16.

Memorizer 18 is the nonvolatile memory of such as HDD (hard disk drive), flash memory and another solid-state memory. In memorizer 18, store OS, various application and various data item. Specifically, in this embodiment, it is also stored on memorizer 18 from the data of the moving image of record medium 5 loading, rest image etc. and the image display application for showing the list of the thumbnail of moving image or rest image.

Multiple moving images or rest image can be categorized as multiple groups by image display application, for each group of event derived expressed by moving image or rest image, and select to represent the presentation graphics of this event. Memorizer 18 also stores personal characteristic information and affair character information, derivation event is the feature that necessary and instruction and PC100 user has the people (father and mother, spouse, child, brother, friend etc.) of predetermined relationship by this personal characteristic information, and this affair character information indicates the feature of object specific to certain event.

Driver element 20 drives the removable record medium 5 of such as storage card, optical recording media, floppy disk (registered trade mark) and magnetic recording disk etc, reads record data on record medium 5, and writes data into record medium 5. Generally, record medium 5 is inserted into the storage card in digital camera, and PC100 reads the data of rest image or moving image from the storage card taken out from digital camera and be inserted in driver element 20. Digital camera and PC100 can pass through USB (USB (universal serial bus)) cable etc. and connect, and so that rest image or moving image are loaded into PC100 from storage card, this storage card is inserted in digital camera.

Communication unit 19 is the NIC (NIC) etc. that can be connected with LAN (LAN), WAN (wide area network) etc. and be used for communicate with another device. Communication unit 19 can perform wired or wireless communication.

(software configuration of PC)

As it has been described above, rest image or moving image can be categorized as multiple groups by PC100, and select and display presentation graphics (best shooting) by image display application for each group. Herein, in the situation of moving image, this group refers to the once shooting that is made up of multiple frames or a scene, or, in the situation of rest image, this group such as refers at identical date and time or in the image sets of identical time period IT. Fig. 2 shows for being shown that application selects the diagram of the functional block of presentation graphics by the image of PC100.

As in figure 2 it is shown, PC100 includes reading unit 21, moving image decoder 22, audio decoder 23, rest image decoder 24, movement image analysis unit 25, audio analysis unit 26, static image analysis unit 27, senior (superordinate) semantic information analytic unit 28 and presentation graphics selects unit 29.

Read unit 21 and read dynamic image content or Still image data from record medium 5. Still image data is to be read for each group that such as corresponds to date or time section. The data read be dynamic image content situation in, read unit 21 dynamic image content is divided into motion image data and voice data. Then, read unit 21 and motion image data is exported moving image decoder 22, voice data is exported audio decoder 23, and Still image data is exported rest image decoder 24.

Motion image data is decoded by moving image decoder 22, and outputs data to movement image analysis unit 25. Voice data is decoded by audio decoder 23, and outputs data to audio analysis unit 26. Still image data is decoded by rest image decoder 24, and outputs data to static image analysis unit 27.

Movement image analysis unit 25 extracts objective (objective) characteristic information and feature based information retrieval subordinate (subordinate) metamessage (semantic information) from motion image data. In the same way, audio analysis unit 26 and static image analysis unit 27 extract objective characteristics information from voice data and Still image data respectively, and based on this feature information extraction subordinate metamessage. In order to extract subordinate metamessage, employ personal characteristic information or affair character information. Additionally, in order to extract subordinate metamessage, also use the technology described in such as Documents: UnderstandingVideoEvents:ASurveyofMethodsforAutomaticint erpretationofSemanticOccurencesinVideo, GalLavee, EhudRivlin, and MichaelRudzsky, IEEETRANSACTIONSONSYATEMS, MAN, ANDCYBERNETICS-PARTC:APPLICATIONSANDREVIEWS, VOL.39, NO.5,2009 year JIUYUE.

When characteristic information extraction, movement image analysis unit 25 performs such as color and texture feature extraction, gradient (gradient) calculates and the process based on pixel of edge extracting etc, or performs the object-based process of the detection of such as human body or face and identification, the identification to object, the motion detection and velocity measuring etc to human body, face or object. In human detection, movement image analysis unit 25 uses the characteristic filter device of assignor's shape etc., thus detection indicates the region of human body from moving image. In face detection, movement image analysis unit 25 such as uses the position relationship of instruction eyes, nose, eyelash, hair, cheek etc. or the characteristic filter device of the feature of skin color information, thus detection indicates the region of face from moving image.

It addition, movement image analysis unit 25 not only identifies the presence or absence of human body or face, and, also by the concrete individual utilizing personal characteristic information identification to have the predetermined relationship with user. Such as, as personal characteristic information, employ edge strength characteristics of image, frequency intensity image feature, the automatic correlated characteristic of higher order, color conversion characteristics of image etc. Such as, in the situation using edge strength image, gray scale (grayscale) image and edge strength image are stored as the characteristic by identified individual (such as the individual of father and mother, child, spouse and friend) by movement image analysis unit 25, the face image of the individual being detected from its face in the same way extracts gray level image and edge strength image, and gray level image and the edge strength image of the two to the two perform pattern match, thus identify the face of concrete individual.

In Object identifying, movement image analysis unit 25 uses the model of cognition being taken as the storage of affair character information, thus determining whether to include the object by identified. Model of cognition is that the machine learning by such as SVM (support vector machine) builds in advance from the image for learning.

It addition, except the individual in moving image with except object, movement image analysis unit 25 can also identify background. Such as, movement image analysis unit 25 uses the model that the machine learning by such as SVM builds in advance from the image for learning, be scape in such as small town, indoor, outdoor, seashore, water by the background class of moving image, night scene, sunset, snow scenes and congested etc scene.

The feature in the extraction of characteristic information of the audio analysis unit 26 sound from the voice of audio data detection people, environment except people and such as power and tone. In order to distinguish the sound in the voice of people and environment, for instance use the persistent period etc. of the audio frequency of predetermined power.

When characteristic information extraction, static image analysis unit 27 performs can static treatment outside the analyzing and processing performed by passive movement image analyzing unit 25, for instance color and texture feature extraction, gradient calculation, edge extracting, the detection to human body, face or object and the identification to background.

It addition, labelling (label) information at such as text etc is comprised in the situation in each data items, label information is extracted as characteristic information by analytic unit 25 to 27. Such as, indicate the information of the details of event or the information of the date and time of shooting image, and the information of the position of shooting image is used as label information.

Based on by each characteristic information extracted in analytic unit 25 to 27, analytic unit 25 to 27 extracts the subordinate metamessage (semantic information) that with the addition of meaning more specifically.

Such as, based on the characteristics of human body extracted or face feature, individual, the sex of people, age, countenance, attitude, clothing, number, formation etc. are identified as subordinate metamessage by movement image analysis unit 25. It addition, based on motion feature, movement image analysis unit 25 identification activity or inactive motion, quickly or motion slowly or such as stand, sit down, walk and the physical activity of running etc, or identify the gesture etc. that employment wrist-watch reaches.

Audio analysis unit 26 extract from the audio frequency characteristics such as extracted from the applause of spokesman, cheer, sound, corresponding to the sensation of voice, laugh, sob, the details of talk, degree of particularity of obtaining based on echo etc. as subordinate metamessage.

Static image analysis unit 27 is from identifying not relevant to motion feature metamessage the metamessage that identifies of passive movement image analyzing unit 25.

Such as, for the extraction to subordinate metamessage as above, can use a kind of based on the state space representative of such as Bayesian network, finite state machine, condition random field (CRF), and the method for hidden Markov model (HMM), a kind of discrete event system based on the meaning model of such as logical method, such as petri net (Petrinet), and the traditional mode identification/method of sorting technique of limited satisfaction model, such as SVM, nearest-neighbors method, and neutral net or multiple additive method.

Senior semantic information analytic unit 28 analyzes senior metamessage based on the subordinate metamessage extracted by each in analytic unit 25 to 27, and derive highest metamessage, it can illustrate once shooting or one group of rest image whole of moving image, i.e. an event. In order to derive event, also use the technology disclosed in such as Documents: EventMininginMultimediaStreams:Researchonidentifyinganda nalyzingeventsandactivitiesinmediacollectionshadledtonew technologiesandsystems, LexingXie, HariSundaran, MurrayCampbell, ProceedingsoftheIEEEVol.96, No.4,2008 year April.

Specifically, based on subordinate metamessage project, senior semantic information analytic unit 28 is analyzed and whose (Who), what (What), when (When), where (Where), why (Why) and how (How) is (following, it is called 5W1H) corresponding multiple metamessage projects, step up abstract grade, and the most once shooting of moving image or multiple rest image are categorized as an event.

Such as, from moving image or rest image, extract such as " a large amount of child ", " a large amount of father and mother and child ", and the metamessage relevant to people of " sportswear " etc, the metamessage that such as " movable motion " is relevant with the action to people of " running form " etc, and the metamessage relevant to general object of such as " School Buildings thing " etc. From sound, extract such as the metamessage of " voice by the people of speaker ", " applause " and " laugh " etc. Additionally, in season (date and time) information obtaining such as the positional information of " primary school " etc, such as " autumn " etc etc. as in the situation of other metamessages, senior semantic information analytic unit 28 derives an appreciable event, " athletic meeting in primary school " by these information projects integrated.

It addition, such as, about the element " who " in the element of 5W1H, senior semantic information analytic unit 28 can by utilizing the word of the concrete individual of instruction to express event. In other words, in be extracted as the situation of information of instruction " who " to relevant subordinate metamessages such as shooting the people (user) of image, kinsfolk, senior semantic information analytic unit 28 utilizes information self as subordinate metamessage, to judge the event of " athletic meeting in the primary school of boy A ".

After event (highest metamessage) is derived by senior semantic information analytic unit 28, presentation graphics selects to select the unit 29 once shooting from moving image or multiple rest image to express best the image (being frame in the situation of moving image) of (representative) event. Fig. 3 shows the figure that the presentation graphics in Fig. 2 selects the details of unit 29.

As it is shown on figure 3, presentation graphics selects unit 29 to include rules selection unit 31, score calculating unit 32, presentation graphics output unit 33 and Rule Information memorizer 34.

Rule Information is stored as the reference being used for for the best representational image of each abstract EventSelect by Rule Information memorizer 34. In other words, for image display each event of being capable of identify that of application and everyone relevant with this event, Rule Information memorizer 34 is preserved for the importance degree of the metamessage (subordinate semantic information or objective characteristics information) of extraction event. Herein, importance degree is the order of priority when presentation graphics is selected by the reference of use.

Such as, in the situation that the event of above-mentioned " athletic meeting of the primary school of boy A " is exported, following items is included as preference.

(1) " boy A occurs in the picture " (face is focused and is not blurred)

(2) " boy A has movable posture " (preferably during movement)

(3) " boy A smile "

On the other hand, only have expressed in the situation of " athletic meeting in primary school " in the event derived, adopt following preference.

(1) " face of primary school student occurs as much as possible in the picture "

(2) " there is the posture of activity "

(3) " people of many smiles "

But, in this case, namely when box lunch item " specific people occur in the picture " is included in Rule Information (with and the rule of above-mentioned event of " athletic meeting in the primary school of boy A " similar), there will not be any problem, and as a result, the image including " boy A " is selected as presentation graphics.

By this way, by being arranged to the rule for each EventSelect presentation graphics derived by senior semantic information analytic unit 28, it is possible to select to reflect better the more suitably presentation graphics of the details of event.

Then, Rule Information memorizer 34 stores score information, the mark that the instruction of this score information is corresponding with the importance degree of each in the preference being included as Rule Information.

Rules selection unit 31 reads the Rule Information for each event from Rule Information memorizer 34.

According to the score information included in above-mentioned Rule Information, score calculating unit 32 calculate for each image (rest image or frame) extract senior/mark of subordinate metamessage. Such as, in the example of above-mentioned athletic meeting, necessary condition is " photo that boy A occurs ". Score calculating unit 32 adds predetermined score for each metamessage project, such as, + 100 are added when in photo " boy A occurs and it is not by out of focus and fuzzy frame ", when boy A has " movable posture "+50, or when boy A is with " smile "+50, and calculate the total score of each image.

Presentation graphics output unit 33 by the multiple rest images from the once shooting or a group of moving image by the calculated image with highest score of score calculating unit 32 selected as presentation graphics, and export this image.

(operation of PC)

It follows that the description that the presentation graphics to PC100 configured as above selects to operate will be provided. In the following description, the CPU11 of PC100 is operating main body. But, below operate and also perform with the software collaboration of another hardware or such as image display application etc. Fig. 4 shows the flow chart of the flow process of the presentation graphics selection process of PC100.

As shown in Figure 4, CPU11 is extracted subordinate metamessage (step 41) by analytic unit 25 to 27 first as mentioned above, is then derived highest metamessage by senior semantic information analytic unit 28, i.e. an event (step 42). Fig. 5 is the diagram conceptually illustrating the process deriving highest metamessage from subordinate metamessage.

As it is shown in figure 5, first CPU11 extracts the subordinate metamessage project corresponding with " who " and " what " from multiple photos 10 of certain group. Such as, such as the metamessage of " child (includes the child of user) " or " kinsfolk with smiling " etc is extracted as the subordinate metamessage corresponding with " who ", further, such as the metamessage of " sportswear ", " running ", " exercise attitudes " or " culinary art " etc is extracted as the subordinate metamessage corresponding with " what ".

Then, CPU11 extracts the senior metamessage of " child " from the above-mentioned subordinate metamessage corresponding with " who ", and extracts the senior metamessage of " sport event " from the above-mentioned subordinate metamessage corresponding with " what ".

Then, CPU11 extracts the higher level metamessage of " sport event of the child that the child of user participates " from the metamessage of the metamessage of " child " and " sport event ".

Additionally, as the metamessage except the metamessage corresponding with " who " and " what ", CPU11 is using the metamessage from the extraction of photo 10 as " primary school " of GPS information (positional information), by the metamessage on " playground " that analysis background scene is extracted, and it is mutually integrated with the metamessage of " sport event of the child that the child of user participates " to extract the metamessage as " autumn " of calendar information (date and time information), thus finally deriving the highest metamessage (event) of " athletic meeting in the primary school of the child of user ".

Referring back to Fig. 4, subsequently, CPU11, according to the event derived, is selected the rules selection unit 31 of unit 29 to determine for selecting Rule Information (step 43) necessary to presentation graphics by presentation graphics.

Then, CPU11 is based on above-mentioned Rule Information, calculate the mark of each metamessage project for each in multiple rest images of certain target group or the multiple frames once shot of component movement image, and be added those marks (step 44 to 48).

Then, CPU11 determines the rest image or frame with calculated highest score element from the frame of multiple rest images or moving image, it can be used as presentation graphics (step 49).

Herein, will be given for selecting the description of the details of presentation graphics from motion image data. Fig. 6 is the diagram of the state that the selection conceptually illustrating and selecting presentation graphics from motion image data processes.

Under the supposition that all frames of moving image are all rest image, selecting the selection of presentation graphics to process from motion image data can by performing with the identical method of rest image. But, in a practical situation, when locating reason diverse ways and performing, efficiency is improved.

As shown in Figure 6, the once shooting of original moving image 60 such as based on the objective characteristics information by such as the process of the detection (camera work) of motion vector or extraction theme etc being extracted, is divided into multiple scene 65 by CPU11. For hereafter performed process, it is considered to two kinds of methods.

As shown in the bottom left section of Fig. 6, in first method, such as, it is based in the situation indicated by label information or other metamessages in the event expressed by whole moving image 60, for each scene 65, first CPU11 selects the best scene 65 expressing part of getting over, feature specific to the moving image of motion simultaneously considering such as theme etc. After this, CPU11, from the frame of selected scene 65, selects representative frame in the framework identical with above-mentioned rest image group.

As shown in the lower right-most portion of Fig. 6, in the second approach, CPU11 is primarily based on objective characteristics constriction representative frame from the frame of scene 65. After this, CPU11 selects representative frame from the frame of constriction in the framework identical with above-mentioned rest image. In this case, equally in the process of the constriction representative frame of respective scene 65, under the supposition that a scene is an event, CPU11 can select each representative frame by the process identical with selecting final representative frame.

Referring back to Fig. 4, when have selected presentation graphics, CPU11 creates the thumbnail (step 50) of presentation graphics, and shows thumbnail (step 51) on the display 16.

Fig. 7 shows the diagram of the display example of the thumbnail of presentation graphics. As shown in the top of Fig. 7, for instance, before presentation graphics is selected, the thumbnail 10a of photo 10 is displayed as the list in matrix. Thumbnail 10a can be shown for each group (file) based on the date etc. On the top of Fig. 7, the thumbnail 10a of the photo 10 belonging to multiple groups is displayed as list.

When above-mentioned presentation graphics selection processes and is performed from this state in predetermined timing, as shown in the lower part of Figure 7, breviary Figure 70 of the presentation graphics of this group is shown, and the thumbnail 10a of non-photograph 10. Each in breviary Figure 70 is shown so that indicate multiple rectangles of the photo 10 in this group to be stacked on together, and breviary Figure 70 is positioned at the top of rectangle, in order to user is it will be appreciated that breviary Figure 70 have expressed the presentation graphics of photo 10.

(summary)

As mentioned above, according to this embodiment, PC100 extracts subordinate metamessage project from multiple images (rest image/moving image), and the result of the senior metamessage this subordinate metamessage project and PC100 derived is (namely, event) mutually integrated, and select presentation graphics according to the Rule Information set by each event subsequently. Therefore, PC100 can present the details of the event of reflecting to user and be adapted as the image of presentation graphics. Correspondingly, user can easily understand event and organization charts's picture from great amount of images. It addition, PC100 derives " what " and whose event (" who ") of event, and selecting presentation graphics based on the result derived, by this result, user can be more easily understood event.

(modification)

The present invention is not limited to above embodiments, and, under the premise not necessarily departing from idea of the invention, it is possible to changed in many ways.

In the embodiment above, as it is shown in fig. 7, breviary Figure 70 of PC100 each presentation graphics of being shown on the top rectangle in stacking rectangle, but the display pattern of presentation graphics is not limited to this. Fig. 8 and Fig. 9 shows the diagram of other display patterns of breviary Figure 70 of presentation graphics.

In first example, as shown in Figure 8, the thumbnail 10a of multiple photos can be divided in groups (bunch) by PC100 based on the date etc., display thumbnail 10a so as in each bunch random overlapping each other, and each group bunch adjacent place show breviary Figure 70 of presentation graphics of each group.

In this case, as bunch, it is possible to select the photo of predetermined quantity with the metamessage of above-mentioned more balloon score, but not belong to the thumbnail of all photos of this group, and can show that the photo with more balloon score is to be located at front end. Furthermore it is possible to display has the photo of more balloon score, so that it has bigger visibility region. Herein, for instance, the operation being categorized as multiple groups can be unit but not the date is unit performs by similar image. It addition, such as, the title of the event derived is displayed at the adjacent place of each bunch, but not shows the date in fig. 8. " what " and whose (" who ") of the title instruction event of event.

In second example, as it is shown in figure 9, for each event, PC100 is possible not only to show by different level breviary Figure 70 of presentation graphics, it is also possible to breviary Figure 75 of the sub-presentation graphics of the subevent in Explicit Expression event by different level. In this case, it is also possible to display event name 71 and subevent name 72.

In the example of figure 9, about the event of " athletic meeting of girl A ", breviary Figure 70 and the event name 71 of presentation graphics are displayed in the top layer of level. In the second layer, it is shown that subevent name 72, this subevent name have expressed the first subevent of the time course corresponding to " family "-> " actual motion meeting "-> " family ". In third layer, for each in the first subevent, show breviary Figure 75 of the sub-presentation graphics of subevent name 72 and subevent name 72, this subevent name have expressed " breakfast ", " admission ", " the second subevent of pitching (wherein, ball is put into basketry), " race ", " dinner " and " going to bed ".

In order to perform this display packing by different level, PC100 needs to understand the more details of event than the method shown in above-mentioned Fig. 5. In other words, PC100 needs degree can derive subevent name to identify in detail and classify subordinate metamessage. Example as the method, for instance, PC100 can derive subevent for each in the subordinate metamessage project corresponding with " who " and " what ", and selects presentation graphics for each subevent in method shown in Figure 5. The Rule Information used in this case not necessarily as prepared (because would be likely to occur not relevant to people subevent) for each concrete individual in the situation of the Rule Information of above-described embodiment, and thus, it is only necessary to prepare the specifying information of each subevent.

In the above-described embodiment, subordinate metamessage and senior metamessage are extracted by PC100, but, can be extracted by another equipment at least partially in those information projects, and can be transfused to together with image when image is imported into PC100. Such as, the subordinate metamessage project of photo can be extracted when photograph taking by digital camera, and is imported into PC100 together with photo, and then, PC100 can from those subordinate senior metamessages of metamessage item extraction. It addition, the subordinate metamessage (it can be extracted with relatively small amount of calculation by digital camera) in face detection, night scene detection etc. can be extracted by digital camera. Metamessage (wherein, extract necessary amount of calculation and become relatively large) in motion detection, general object identification etc. can be extracted by PC100. It addition, metamessage can be extracted by the server substituting PC100 in a network, and it is imported into PC100 via communication unit 19.

It addition, the process performed by PC100 can also be performed by following equipment in the embodiment above: television equipment, Digital Still Camera, digital video camera, mobile phone, smart phone, record and transcriber, game machine, PDA (personal digital assistant), e-book terminal, electronic dictionary, portable AV equipment and any other electronic installations.

In the embodiment above, as shown in Figure 4, after event is exported, the correspondingly mark of computation of meta-information project. But, mark can be calculated at the synchronization being performed when the process extracting subordinate metamessage from image. Figure 10 is the flow chart of the process illustrating presentation graphics selection process in this case.

As shown in Figure 10, CPU11 is extracted subordinate metamessage to calculate the mark of each metamessage project by analytic unit 25 to 27, and stores this mark (step 81) explicitly with image. Then, after event is exported, CPU11 is loaded into the stored mark (step 85) of each image, and stored mark is added (step 86), thus selects representational image (step 88).

In the above-described embodiments, the subordinate metamessage of analytic unit 25 to 27 and senior semantic information analytic unit 28 and senior metamessage extraction process are not limited to above-mentioned process. In other words, any process can be performed, as long as extracting with the subordinate metamessage acting on some objective characteristics describing respective image and the senior metamessage derived from subordinate metamessage project. Such as, each metadata item can be added the information project as label information by individual.

Selecting in the rules selection unit 31 of unit 29 at presentation graphics, although not necessarily, but it is desirable in advance metamessage project can be ranked up for all types of events, this event can be identified by image display application. Such as, PC100 can previously generate Rule Information clearly only for the event group with high use frequency (derivation frequency) especially, and this Rule Information general rule is substituted about other events. General rule refers to such as the subordinate metamessage project of the degree (derived actually by study or obtain) of " composition quality " or " fluctuation/fuzzy " or the order of priority of objective characteristics amount. It addition, in the situation that the Rule Information with the high event group using frequency is generated, the respective metamessage project implementation can subjectively be weighed by user, or certain can be adopted to use the machine learning method of type.

In the above-described embodiment, score calculating unit 32 calculates total score based on " presence or absence " of metamessage, but, this mark can be the assessed value of (stepped) continuously, such as movable degree or the degree of smile, but not " existence " and " being absent from " two values. Those metamessage projects can be calculated by score calculating unit 32, or can be calculated by the analytic unit 25 to 27 of Fig. 2. In other words, it is possible in analytic unit 25 to 27, perform process, it not only includes the metamessage directly related with the derivation of event, also includes the information for selecting presentation graphics after a while.

It addition, in the combination of rules selection unit 31 in the above-described embodiments and score calculating unit 32, it is possible to the mark of each event is calculated by machine learning. Determining mark by machine learning, and come compared with the subjective situation arranging mark for each event in advance, it is contemplated that many metamessage projects, its result is that event can be derived more accurately.

In the above-described embodiments, based on a scene of once shooting or moving image, presentation graphics is chosen and is shown. But, for instance, presentation graphics can be used for moving image editing and processing. In other words, although the thumbnail of frame is shown at the in-edit specified by user in the prior art, in order to indicate the scene conversion in once shooting, but, the thumbnail of presentation graphics can be shown. It addition, such as, when performing scene search, the presentation graphics of each scene can be shown, but not as in the prior art be shown in the frame that predetermined frame interval place extracts. Correspondingly, the accessibility of scene is improved by user.

The application comprises the theme relevant to the content disclosed in the JP2010-084557 Japanese Priority Patent Application that on March 31st, 2010 submits to Japan Office, and the full content of this application is incorporated herein by reference.

It will be appreciated by those skilled in the art that and depend on designing requirement and other factors, it is possible to various modification, combination example, sub-portfolio example and change case occur, as long as they are in the scope of claim or its equivalent.

Claims (6)

1. an electronic installation, including:

Memorizer, described memorizer is arranged to storage

It is classified as multiple images of multiple groups,

Multiple affair character information projects, the plurality of affair character information project indicates the feature of object specific to each event, and

Multiple Rule Information projects, the instruction of the plurality of Rule Information project is for representing by the rule of presentation graphics of the event of the plurality of image expression for each group selection, and the plurality of Rule Information project is different for each event and everyone relevant from described event;

Controller, it is shown that controller is arranged to

From multiple images of each group, multiple metamessage project is extracted based on the plurality of affair character information project,

Senior metamessage is analyzed from the multiple metamessage projects extracted, relevant with whom to derive the plurality of image expression what and this event, and

Based on the Rule Information project corresponding with the event derived, from the plurality of image, select to represent the described presentation graphics of the event derived; And

Output unit, described output unit is arranged to the thumbnail image of the presentation graphics selected by each group of output.

2. electronic installation as claimed in claim 1, wherein,

Described memorizer storage personal characteristic information, the instruction of this personal characteristic information has the feature of the people of predetermined relationship with user, and

Described controller extracts the plurality of metamessage project based on described personal characteristic information and the plurality of affair character information project.

3. electronic installation as claimed in claim 2, wherein,

For each event, the plurality of Rule Information project includes the multiple metamessage projects being included in described presentation graphics and multiple score information project, each of the plurality of score information project indicates the mark corresponding with the importance degree of each metamessage project, and

The mark corresponding with the respective metamessage project of the plurality of image, based on the plurality of score information project, is added, and the image with highest score is elected to be described presentation graphics by described controller.

4. electronic installation as claimed in claim 3, wherein,

Described output unit is output character information together with described thumbnail image, and this character information indicates described event representation what and described event to be correlated with whom.

5. electronic installation as claimed in claim 3, wherein,

Described controller selects the presentation graphics with the predetermined number of balloon score, and exports the thumbnail image of the presentation graphics of described predetermined number so that the described presentation graphics with higher fractional has bigger visibility region.

6. an image processing method, including:

Storage herein below:

It is classified as multiple images of multiple groups,

Multiple affair character information projects, the plurality of affair character information project indicates the feature of object specific to each event, and

Multiple Rule Information projects, the instruction of the plurality of Rule Information project is for representing by the rule of presentation graphics of the event of the plurality of image expression for each group selection, and the plurality of Rule Information project is different for each event and everyone relevant from described event;

Based on the plurality of affair character information project, from multiple images of each group, extract multiple metamessage project;

Senior metamessage is analyzed from the multiple metamessage projects extracted, relevant with whom to derive the plurality of image expression what and this event;

Based on the Rule Information project corresponding with the event derived, from the plurality of image, select to represent the described presentation graphics of the event derived; And

The thumbnail image of the presentation graphics selected by each group of output.