JP5762075B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to a technique for retrieving image data desired by a user from an image data group.

  In recent years, miniaturization and increase in capacity of built-in memories used in digital cameras and the like have been promoted, and a large amount of image data has been accumulated. As a result, the accumulated number of image data captured in the past increases, and there is a problem that it takes a long time for the user to find desired image data from among a large number of image data.

  Patent Document 1 discloses a technique for preferentially displaying image data having a feature that matches or is similar to the feature of the part of interest by detecting the line-of-sight direction of the operator. Patent Document 2 discloses a technique for displaying the characteristics of image data in a specific area different from the image data on the screen, and changing the display of the image data according to the area and direction traced by the user. .

JP-A-9-114846 JP 2010-109788 A

  In the technique disclosed in Patent Document 1, it is possible to preferentially display image data having a feature that matches or is similar to the feature of the portion of interest. However, when it is desired to specify a plurality of features, it is difficult to specify by line of sight.

  In the technique disclosed in Patent Document 2, image data can be easily browsed along the axis of features such as time and events. However, the technique disclosed in Patent Document 2 has a problem that the place to touch is not intuitive because an area different from the image data is provided separately.

  SUMMARY OF THE INVENTION An object of the present invention is to preferentially search image data that a user wants to see without performing complicated operations.

An information processing apparatus of the present invention is an information processing apparatus that displays on a screen at least a part of a plurality of image data according to a predetermined display order, and is an object included in image data displayed on the screen and identifying means for identifying the object selected by the user, based on the features of the identified said object by said identifying means, among the plurality of image data, having characteristics similar to the image data including the object Determining means for determining a display order of the plurality of image data giving priority to image data; and displaying at least a part of the plurality of image data rearranged in the display order determined by the determining means on the screen. it is characterized by having a display control means for changing the display contents of the screen.

According to the present invention, without causing I lines a complicated operation on the user, the user is able to display an image group obtained by arranging image data in priority manner that desired.

It is a figure which shows the structure of the information processing system which concerns on embodiment of this invention. It is a figure which shows a part of hardware constitutions of the information processing system which concerns on embodiment of this invention. It is a figure which shows an example of an image data group. It is a figure which shows the display state of an image data group. It is a figure which shows the display state of an image data group. It is a figure which shows the display state of an image data group. It is a flowchart which shows the process of a selection operation determination part. It is a flowchart which shows the determination process whether multiple places (N places) were touched simultaneously. It is a flowchart which shows the process in the case of image browsing. It is a figure which shows the display state etc. of an image data group. It is a figure which shows the display state etc. of an image data group. It is a figure which shows the display state etc. of an image data group. It is a figure which shows the arrangement | positioning pattern of 1 to 5 fingers of a typical right hand and left hand. It is a flowchart which shows the process in the 9th modification of a selection operation determination part. It is a figure which shows the display state of an image data group. It is a flowchart which shows the process in the case of image browsing.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  First, a first embodiment of the present invention will be described. FIG. 1 is a diagram showing a configuration of an information processing system according to the first embodiment of the present invention. As illustrated in FIG. 1A, the information processing system according to the present embodiment includes an input device 101, a display device 102, a control device 103, and a storage device 104. In the example of FIG. 1, 101 to 104 are configured as independent devices, but a configuration in which the functions of these devices are provided in a single device may be employed.

  The multiple object designation unit 105 accepts a touch input by the user. In addition to a single touch or multiple touches, it detects a drag (operation to move with a finger while touching) or a flick (operation to lightly touch the touched finger). The multi-object designating unit 105 also detects pressure during touch, drag, and flick. Note that an input unit such as a mouse or a voice may be used instead of the touch panel, and a click operation or a voice input may be used instead of the touch input.

  The display unit 106 displays image data, text data, and the like. The selection operation determination unit 107 determines whether an object in the image data has been selected. The object specifying unit 108 specifies the object determined to be selected by the selection operation determining unit 107.

  The feature extraction unit 109 extracts the feature of the object specified by the object specification unit 108. There are various ways to grasp the characteristics of an object, for example, people, things, colors, shapes, sizes, and the like. In this embodiment, the feature of the object is extracted from the touched location and feature data 112 described later, but an existing image recognition technique may be used.

  The image retrieval unit 110 retrieves image data from the image data group 111 stored in the storage device 104 based on the feature of the object extracted by the feature extraction unit 109, or the image data based on the feature similarity. A process of rearranging the group 111 is executed. The image search unit 110 selects image data to be displayed and causes the display unit 106 to display the selected image data.

  As shown in FIG. 1B, the feature data 112 is data in which the ID of the image data, the area information of the object of the image data, and the feature information of the object are associated with each other. As described above, the object features include, for example, people, things, colors, shapes, sizes, and the like. The feature data 122 can be composed of text, list, XML, or the like. The history information 113 is information indicating the history operated by the user and the result (determination result and extraction result of each component). The history reference unit 114 refers to the history information 113.

  FIG. 2 is a diagram illustrating a part of the hardware configuration of the information processing system according to the present embodiment. As shown in FIG. 2, the information processing system according to the present embodiment includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, a touch panel 204, and an LCD (Liquid Crystal Display). 205 is connected by a system bus 206.

  The multi-object designating unit 105 in FIG. 1 includes the touch panel 204 in FIG. Further, the display unit 106 in FIG. 1 includes the LCD 205 in FIG. The selection operation determination unit 107, the object specification unit 108, the feature extraction unit 109, the image search unit 110, and the history reference unit 114 have a configuration obtained by the CPU 201 reading the program and data stored in the ROM 203 into the RAM 202 and executing them. is there. The storage device 104 includes a RAM 202 and the like.

  FIG. 3 is a diagram illustrating an example of the image data group 111. In FIG. 3, ten pieces of image data of image data 1 to image data 10 are shown as the image data group 111. As illustrated in FIG. 4, the image search unit 110 causes the display unit 106 to display the image data group 111 based on the display order. In this embodiment, the display order of the image data group 111 shown in FIG. 3 is as follows: image data 1, image data 2,..., Image data 9, image data 10, image data 1, image data 2,. It has established. Therefore, as shown in FIG. 4, the display unit 106 displays five pieces of image data in the image data group 111, but the five pieces of image data are displayed according to the display order. Specifically, as shown in FIG. 4A, the image data 9, image data 10, image data 1, image data 2, and image data 3 are displayed in this order from the left side of the display unit 106. Alternatively, as shown in FIG. 4B, the image data 1, image data 2, image data 3, image data 4, and image data 5 are displayed in this order from the left side of the display unit 106. Note that the display order of the image data when the application is started is determined based on rules defined by the application, such as date order or name order. In the example of FIG. 4A, the image data 1 with the highest display order is positioned in the middle. However, as shown in FIG. 4B, the image data 1 with the highest display order is one. It may be located on the far left. Note that the display position of the image data with the highest display order is determined by the application. Further, the display order of the image data at the time of starting the application and the display position of the image data with the highest display order may be set by the user through the application.

The display unit 106 can be touched via the multiple object designating unit 105, and can be touched at one place or touched at multiple places. Also, drag (operation to move with a finger while touching) and flick (operation to lightly touch the touched finger) are possible. In the present embodiment, the image data is dragged or flick (hereinafter, a description will be given only Flick) during image browsing Once interprets the display switching of the image data is instructed, display switching of the image data is performed .

  Next, an example of an operation when flicking during image browsing will be described with reference to FIGS. 5 and 6. 5 and 6, the touched portions are indicated by black circles 501 and 601, and the direction of dragging or flicking is indicated by arrows 502 and 602. As shown in FIG. 5A, when image data 1 is touched and flicked to the left, as shown in FIG. 5B, the image data group 111 moves to the left so that the image data group 111 moves to the left. The display is switched. Further, as shown in FIG. 6A, when the image data 1 is touched and flicked in the right direction, the image data group 111 is moved in the right direction as shown in FIG. 6B. The data display is switched.

  At this time, when the display switching of the image data is instructed without selecting an object in the image data, the display switching of the image data is performed while maintaining the display order of the image data. On the other hand, when image data display switching is instructed in a state where one or more objects in the image data are selected, the display order of the image data is changed based on the characteristics of the selected objects, and the display of the image data is performed. Switching takes place. That is, in this embodiment, whether the display switching of the image data is instructed in a state where the object in the image data is not selected, or whether the display switching of the image data is instructed in the state where the object in the image data is selected. Judgment is required. Therefore, in the present embodiment, the above determination is performed by the selection operation determination unit 107.

  Next, processing of the selection operation determination unit 107 will be described with reference to FIG. FIG. 7 is a flowchart showing processing of the selection operation determination unit 107. Here, the time when the image data is touched is Tt, and the time when the image data is flicked is Tf. These times are information acquired by the multiple object specifying unit 105 and are stored in the storage device 104 as history information 113. In step S <b> 901, the selection operation determination unit 107 uses the history reference unit 114 as the history information 113, the time when the image data is touched (touch time) Tt and the time when the image data is flicked (flick time) Tf. Via the storage device 104.

  In step S702, the selection operation determination unit 107 determines whether the operation is an object selection operation based on the touch time Tt and the flick time Tf. Specifically, if the time interval from the touch time Tt to the flick time Tf is short, that is, if (Tf−Tt) is equal to or shorter than a predetermined time interval (for example, 0.1 second), the selection operation determination unit 107 Then, it is determined that the display switching of the image data is instructed in a state where no object in the image data is selected. On the other hand, if the time interval from the touch time Tt to the flick time Tf is somewhat longer, that is, if (Tf−Tt) is longer than a predetermined time interval (for example, 0.1 second), the selection operation determination unit 107 It is determined that the display switching of the image data is instructed with the object in the image data selected. Further, when a plurality of locations (N locations) are touched, if these are touched at the same time and the time interval from the touch time Tt to the flick time Tf is short as described above, the selection operation determination unit 107 displays the image data It is determined that image data display switching has been instructed in a state in which no object is selected.

  Note that whether or not a plurality of locations (N locations) are touched at the same time is determined based on whether or not there is almost no difference in all touch times. Next, a process for determining whether or not a plurality of locations (N locations) have been touched simultaneously will be described with reference to FIG. FIG. 8 is a flowchart showing a process for determining whether or not a plurality of locations (N locations) are touched simultaneously. In step S801, the selection operation determination unit 107 sets each touch time as Tt1, Tt2, Tt3,..., TtN. In step S802, the selection operation determination unit 107 substitutes 1 for a variable n. In step S803, the selection operation determination unit 107 substitutes n + 1 for the variable m. In step S804, the selection operation determination unit 107 determines whether or not a condition of N <n or N <m is satisfied. When the condition of N <n or N <m is satisfied, the process proceeds to step S808. On the other hand, if neither N <n nor N <m is satisfied, the process proceeds to step S805. In step S808, the selection operation determination unit 107 determines that a plurality of locations have been touched simultaneously.

  In step S805, the selection operation determination unit 107 determines whether or not the difference between the time Ttm and the time Ttn is sufficiently small, that is, the absolute value of the difference between the time Ttm and the time Ttn is a predetermined value (for example, 0.01 seconds). ) It is determined whether it is smaller. When the difference between time Ttm and time Ttn is sufficiently small, the process proceeds to step S806. On the other hand, if the difference between time Ttm and time Ttn is not sufficiently small, the process proceeds to step S807. In step S806, the selection operation determination unit 107 adds 1 to n. Then, the process returns to step S803. In step S807, the selection operation determination unit 107 determines that a plurality of locations are not touched simultaneously.

  As described above, it is determined whether or not a plurality of locations have been touched at the same time. If it is determined that touch has been made at the same time, one of time Ttm and time Ttn or an average value of time Ttm and time Ttn is calculated. Let Tt. If the time at which the image data is flicked is Tf, if (Tf−Tt) is equal to or shorter than a predetermined time interval (for example, 0.1 second), the selection operation determination unit 107 determines that the object in the image data is It is determined that image data display switching has been instructed without being selected. On the other hand, if (Tf−Tt) is longer than a predetermined time interval (for example, 0.1 second), the selection operation determination unit 107 is instructed to switch display of image data while an object in the image data is selected. It is determined that

  As described above, when the user does not select an object, determination based on a natural operation that the time interval from the touch time to the flick time is short is possible.

  Next, processing when performing image browsing will be described with reference to FIGS. FIG. 9 is a flowchart showing a process at the time of image browsing. 10 to 12 are diagrams showing a display state of the image data group 111 and the like. That is, FIG. 10A, FIG. 11A, and FIG. 12A all show the image data group 111 including an object having a triangular or circular feature, and FIG. 10B and FIG. FIGS. 12B and 12B show the display state of the image data on the display unit 106. FIG. 10C shows an example of division of image data. In the example of FIG. 10C, the image data is divided into two parts in the vertical direction and two parts in the horizontal direction, and four areas A, B, C, and D are defined for the image data. FIG. 10D shows an example of the feature data 112 when four areas are defined for the image data as described above. In the “ID” column, the ID of the image data is defined, in the “object region information” column, the region to which the object belongs is defined, and in the “feature information” column Defines the characteristics of the object. Black circles 1101 and 1201 in FIGS. 11B and 12B indicate locations touched by the user. FIG.11 (c) and FIG.12 (c) have shown the characteristic information of the object corresponding to the touched location.

  In step S901, the image search unit 110 displays the image data group 111 on the display unit 106 based on the display order. The display order is determined based on rules determined by the application, such as date order or name order. The initial order may be set by the user through an application. In the example of FIG. 10A, the image data group 111 is arranged in order of date. As shown in FIG. 10B, the last two pieces of image data (image data 9, image data 9) are displayed from the left side of the display unit 106. Data 10) and the first three pieces of image data (image data 1, image data 2, image data 3) are displayed.

  In step S <b> 902, the selection operation determination unit 107 determines whether one or more touch inputs are notified from the multiple object specification unit 105. When one or more touch inputs are notified, the process proceeds to step S903. On the other hand, if one or more touch inputs are not notified, the process waits until a touch input is notified from the multiple object designating unit 105.

In step S903, the object identification unit 108, when it is determined that the object is selected by the selection operation determination unit 107, for identifying the object. The feature extraction unit 109 extracts the feature of the object. Specifically, the object identification unit 108 calculates whether or touched point is located at which position in the image data, further with reference to the feature data 112, which objects in the touched point is image data It is judged whether it belongs to. Since the feature data 112 includes object area information, it can be easily determined which object belongs to an arbitrary position in the image data. If a plurality of locations are touched and the positions belong to different objects, a plurality of objects are selected. In addition, when a plurality of locations are touched and the positions belong to the same object, one object is selected. When the selected object is specified, the feature extraction unit 109 extracts the feature of the selected object. Since the feature data 112 is associated with the object region information and the object feature information, the feature of the selected object can be extracted. When a plurality of objects are selected, features are extracted for all the objects. If the feature data 112 is not given, an object (person, thing, etc.) and its feature (name, color, shape, etc.) at an arbitrary position of the image data are utilized by using an existing image recognition technique. Can be recognized and acquired. In the example of FIG. 11, the feature of a circle is extracted as shown in FIG. In the example of FIG. 12, a feature of a triangle is extracted as shown in FIG.

  In the present embodiment, since the area information on the image data of the specified object can be acquired, the object on the image data is made conspicuous, such as coloring the area or edging the object. May be. Further, the feature (character string) of the object may be displayed superimposed on the image data.

  In step S <b> 904, the image search unit 110 determines whether the object is specified by the object specifying unit 108 and the feature is extracted by the feature extracting unit 109. If the touched position does not belong to any object, and thus no object is specified and no feature is extracted, the process proceeds to step S906. On the other hand, when the touched position belongs to one of the objects, and the object is specified and the feature is extracted, the process proceeds to step S905.

  In step S905, the image search unit 110 calculates the feature similarity for the image data group 111, searches the image data group 111 based on the similarity, and changes the display order of the image data according to the search result. . Here, the image data group 111 is searched and rearranged in descending order of similarity. The image data group 111 may actually be rearranged, or the value indicating the display order may be updated. The feature similarity is a value indicating how many objects having the feature are included in the image data. For example, recognition likelihood or the like may be used in addition to the number of objects having the characteristics and the area of the objects. In the example of FIG. 11, the image search unit 110 increases the similarity of the image data 1, 4, and 8 including the object having the feature of a circle by referring to the feature data 112 shown in FIG. The image data is rearranged as shown in FIG. In the example of FIG. 11A, the priority in the image data including the object having the feature of the circle and the priority in the image data not including the object having the feature of the circle are the order before the rearrangement. However, it is not limited to this. For example, the image data including objects having the same characteristics may be arranged in order of date.

  When there are a plurality of features, the image data group 111 may be rearranged according to the similarity based on the AND (logical product) of the features. Furthermore, the image data group 111 may be rearranged according to the similarity based on OR (logical sum) instead of AND of a plurality of features. When there are a plurality of features, the user may be able to set through an application whether the feature is based on AND or OR.

  In step S906, the selection operation determination unit 107 determines whether or not the touch input has been changed. That is, the selection operation determination unit 107 compares the touch input state detected in step S902 described above, and if the touch input state is different, the process proceeds to step S907. On the other hand, if the touch input state detected in step S902 described above is the same, the process proceeds to step S908. Note that the state of touch input is different from the increase or decrease in the number of inputs such as “from one to two touch inputs” and “from two to one touch inputs”, as well as the touch position. The change etc. are mentioned. Here, changing the touch position means that, unlike dragging and flicking, the touch at a certain position is released and another position is touched again.

  In step S907, the image search unit 110 restores the display order of the image data when the display order of the image data is changed in the previous step S905. On the other hand, when the display order of the image data is not changed in the previous step S905, the image search unit 110 performs no processing. Then, the process returns to step S902.

  In step S908, the image search unit 110 determines whether there is an instruction to switch display of image data. As described above, in this embodiment, when flicking on image data during image browsing, it is interpreted that there is an instruction to switch display of image data. For this reason, when it is notified from the multiple object designation | designated part 105 that the flick was performed, a process transfers to step S909. On the other hand, if it is not notified that the flick has been performed, the process proceeds to step S906. In step S909, the image search unit 110 causes the display unit 106 to display the image data group 111 based on the display order.

  When there is an instruction for termination from the application, the application is terminated by interrupt processing at any step. Moreover, the process of the selection operation determination part 107 is not restricted to the process mentioned above. For example, the following processes are mentioned. The process of the selection operation determination unit 107 described below is performed in parallel with the process of FIG.

  First, a first modification of the selection operation determination unit 107 will be described. As described above, when the time interval from the touch time to the flick time is used, whether or not the time interval is short may vary depending on the user. In the first modified example, in step S701 in FIG. 7, the selection operation determination unit 107 refers to the current touch time and flick time, and the time interval from the past touch time to the flick time, which are the history information 113. To do. In step S702, the selection operation determination unit 107 calculates the time interval from the current touch time to the flick time, and the time interval is the same as the time interval from the past touch time to the flick time, or the time interval. If the difference from the time interval from the past touch time to the flick time is equal to or less than a predetermined value (for example, 0.1 second), it is determined that the object has been selected. On the other hand, if the above condition is not satisfied, the selection operation determination unit 107 determines that no object is selected. Further, the selection operation determination unit 107 stores the determination result in the storage device 104 as the history information 113 in association with the time interval from the current touch time to the flick time, and uses it for the subsequent determination. Further, the time interval from the past touch time to the flick time used for the determination may be limited to the history information 113 within a few minutes, for example. Thereby, the time interval from the touch time to the flick time can be customized for each individual user.

  Next, a second modification of the selection operation determination unit 107 will be described. It takes some time for the user to select an object. For this reason, when the time interval from the previous flick time to the current flick time is sufficiently short, it can be considered that no object is selected. In this modified example, in step S701 in FIG. 7, the selection operation determination unit 107 refers to the current flick time and the previous flick time as the history information 113. In step S702, the selection operation determination unit 107 calculates a time interval between the current flick time and the previous flick time, and when the time interval is a predetermined value (for example, 0.5 seconds) or less, It is determined that the object is not selected. On the other hand, if the time interval is longer than a predetermined value (for example, 0.5 seconds), the selection operation determination unit 107 determines that an object has been selected. Thereby, when the user does not select an object, the determination based on a natural operation of quickly switching the image data can be performed.

  Next, a third modification of the selection operation determination unit 107 will be described. When a user browses image data, the display of the image data is often switched at a certain time interval. On the other hand, the time interval may not be constant when gazing at something or performing a specific operation. Object selection is one of them. In consideration of such circumstances, in this modification, it is determined that an object has been selected when the time interval of the flick time is not constant. That is, in step S701, the selection operation determination unit 107 refers to the current flick time and the previous flick time as the history information 113. In step S702, the selection operation determination unit 107 calculates a time interval between the current flick time and the previous flick time. The time interval is the same as the average value of the time intervals of the past flick times, or the difference between the time interval and the average value of the time intervals of the past flick times is a predetermined value (for example, 0.5 seconds) or less. If there is, the selection operation determination unit 107 determines that no object is selected. On the other hand, when the difference between the time interval and the average value of the time intervals of past flick times is longer than a predetermined value (for example, 0.5 seconds), the selection operation determination unit 107 determines that the object has been selected. Moreover, the average value of the time intervals of the past flick times used in the determination may be limited to the average value of the time intervals of the following flick times within a few minutes, for example. Thereby, when the user selects an object, it is possible to make a determination based on a natural operation in which display switching of image data is not performed at a certain time interval.

  Next, a fourth modification of the selection operation determination unit 107 will be described. In the case of the third modification described above, it is difficult to determine at the first flick after selecting an object. Therefore, in the present modification, the time interval from the past touch time to the flick time and the determination result of the selection operation determination unit 107 are associated with each other and stored in the storage device 104 as history information 113. In step S701, the selection operation determination unit 107 refers to the history information 113, the time interval from the past touch time to the flick time and the determination result of the selection operation determination unit 107. In step S702, the selection operation determination unit 107 determines that the time interval from the current touch time to the flick time is the same as the average value of the time intervals from the past touch time to the flick time when it is determined that the object has been selected. Or, if the difference between the time interval from the current touch time to the flick time and the average value of the time interval from the past touch time to the flick time is equal to or less than a predetermined value (for example, 0.1 second), the object It is determined that is selected. On the other hand, if the above condition is not satisfied, the selection operation determination unit 107 determines that no object is selected. Moreover, the time interval of the past flick time used in the determination may be limited to the history information 113 within a few minutes, for example. Thereby, when the user selects an object, it is possible to make a determination based on a natural operation such that display switching of image data is not performed at regular intervals.

  Next, a fifth modification of the selection operation determination unit 107 will be described. Generally, when a user touches to select something, particularly when multiple selections are made by touch, pressure is applied rather than simply flicking. Therefore, the selection operation determination unit 107 in this modification determines that an object has been selected when the touch or flick pressure is greater than or equal to a predetermined value, and if the pressure is less than the predetermined value, the object is selected. Judge that it is not. Accordingly, it is possible to make a determination based on a user's natural operation that pressure is applied when an object is selected.

  Next, a sixth modification of the selection operation determination unit 107 will be described. When pressure is used as in the fifth modification, whether the pressure is large may vary depending on the user. Therefore, in this modification, the past touch or flick pressure and the determination result of the selection operation determination unit 107 are associated with each other and stored in the storage device 104 as history information 113. In step S <b> 701, the selection operation determination unit 107 refers to the current touch or flick pressure and the past touch or flick pressure, which are the history information 113. In step S <b> 702, the selection operation determination unit 107 determines that the current touch or flick pressure is the same as the average value of the past touch or flick pressure when it is determined that the object is selected, or the current touch or flick pressure. If the difference between the pressure and the average value of the past touch or flick pressure is equal to or less than a predetermined value, it is determined that the object has been selected. On the other hand, if the above condition is not satisfied, the selection operation determination unit 107 determines that no object is selected. Note that the current determination result and the touch or flick pressure are stored as history information 113 and used in subsequent determinations. The past touch or flick pressure used for the determination may be limited to the history information 113 within a few minutes, for example. Thereby, the magnitude | size of a pressure can be customized for every user.

  Next, a seventh modification of the selection operation determination unit 107 will be described. When the user browses image data without selecting an object, the total number of locations touched in succession (for example, the number of fingers to touch) is often constant. On the other hand, since a plurality of objects may be specified, the total number of touched or flicked portions may change. In consideration of such circumstances, in this modification, in step S901, the selection operation determination unit 107 refers to the total number of past touched or flicked portions, which is the history information 113. In step S902, the selection operation determination unit 107 determines that the total number of locations touched or flicked this time is the same as the total number of locations touched or flicked in the previous several inputs (for example, the previous five inputs). , It is determined that the object is not selected. On the other hand, if the total number of locations touched or flicked this time is different from the total number of locations touched or flicked in the last several inputs (for example, the last five inputs), the selection operation determination unit 107 selects the object. It is determined that In this case, since it can be determined that the display switching of the image data has been instructed in a state where the object in the image data is not selected at the time of step S902 in FIG. 9 described above, there is no need to move to step S903. Thereby, when the user does not select an object, the determination based on the user's natural operation that the total number of touched portions does not change is possible.

  Next, an eighth modification of the selection operation determination unit 107 will be described. When the user browses image data without selecting an object, it is often possible to switch image data by continuously touching or flicking a specific area. On the other hand, when selecting an object, it is rare to touch a specific area continuously. In this case, in step S <b> 701, the selection operation determination unit 107 refers to information (for example, coordinate information) of a past touched or flicked portion that is the history information 113. In step S <b> 702, the selection operation determination unit 107 determines that an object has not been selected when the location touched or flicked this time is within a predetermined range from the location touched or flicked in the past. On the other hand, if the above condition is not satisfied, the selection operation determination unit 107 determines that an object has been selected. In addition, the information of the place touched or flicked in the past used for the determination may be limited to the history information 113 within a few minutes, for example. The term “within the predetermined range” means not the coordinates on the individual image data but the coordinates on the entire screen (display unit 106) or the entire touch panel on which the image data is displayed. Thereby, when the user does not select an object, it is possible to make a determination based on a user's natural operation of continuously touching and flicking a specific area.

  Next, a ninth modification of the selection operation determination unit 107 will be described. When the user browses image data without selecting an object, the finger placement pattern is natural. On the other hand, since the arrangement of objects often does not match the natural arrangement of fingers, the arrangement of fingers when selecting an object is often unnatural. Considering such circumstances, the similarity between the touched part placement pattern and the object finger placement pattern is higher than the similarity between the touched part placement pattern and the typical finger placement pattern. The selection operation determination unit 107 determines that an object has been selected.

  Hereinafter, a ninth modification of the selection operation determination unit 107 will be specifically described. A typical arrangement pattern of one to five fingers of the right hand and the left hand as shown in FIG. 13 is stored in the storage device 104 in advance. The finger arrangement pattern may be a numerical value indicating the arrangement instead of image data. Hereinafter, the processing in the ninth modification of the selection operation determination unit 107 will be described with reference to FIG. FIG. 14 is a flowchart showing processing in the ninth modification of the selection operation determination unit 107.

  In step S <b> 1401, the selection operation determination unit 107 acquires, from the multiple object specification unit 105, the total number N of touched locations and the arrangement pattern of the touched locations. In step S1402, the selection operation determination unit 107 acquires the total number M of objects on the image data and the arrangement pattern of the objects. If there are a plurality of touched image data as shown in FIG. 15, the total number of objects included in the plurality of image data (image data 10 and image data 1) is obtained. Note that black circles 1501 and 1502 in FIG. 15 indicate touched locations. In addition, the arrangement pattern of the object needs to consider the arrangement information of the plurality of image data. Taking image data 10 and image data 1 as an example, arrangement information of image data 10 and image data 1 (in the example of FIG. 15, image data 10 is arranged on the left and image data 1 is arranged on the right). The arrangement pattern of the object changes depending on. Further, the target range for acquiring the total number M of objects may be the entire image data, but the minimum rectangle including all fingers may be set as the target range.

  In step S1403, the selection operation determination unit 107 compares the total number N of touched locations with the total number M of objects. If the total number N of touched locations is larger than the total number M of objects, the process proceeds to step S1404. On the other hand, if the total number M of objects is equal to or greater than the total number N of touched locations, the process proceeds to step S1407.

  In step S <b> 1404, the selection operation determination unit 107 calculates the similarity between the touched part placement pattern and the object placement pattern, and the touched part placement pattern and the typical finger placement pattern. Calculate each. The pattern similarity can be easily calculated by using an existing technique.

  In step S1405, the selection operation determination unit 107 compares the two similarities calculated in step S1404. If the similarity between the touched location placement pattern and the typical finger placement pattern is higher than the similarity between the touched location placement pattern and the object placement pattern, the process moves to step S1407. . On the other hand, if the degree of similarity between the arrangement pattern of the touched location and the arrangement pattern of the object is higher than the similarity between the arrangement pattern of the touched location and the typical finger arrangement pattern, the process proceeds to step S1406. Migrate to In step S1407, the selection operation determination unit 107 determines that an instruction to switch display of image data has been given in a state where no object is selected. In step S1406, the selection operation determination unit 107 determines that an instruction to switch display of image data is given in a state where an object is selected.

  Each of the first to ninth modifications described above may be applied, or any of a plurality of modifications of the first to ninth modifications may be applied in combination.

  As described above, according to the present embodiment, the user can search and display image data including the object with priority over other image data through an intuitive operation in which the user touches the object to be viewed with priority. Can do. In addition, since it is possible to determine whether or not an object has been selected based on the user's natural operation, the burden on the user is not increased.

  Next, a second embodiment of the present invention will be described. In the first embodiment, the method of selecting an object having a plurality of features at the same time when specifying a plurality of features at the same time has been described. On the other hand, in the second embodiment, a method for selecting an object in multiple times when a plurality of features are simultaneously specified will be described. The configuration of the information processing system according to the second embodiment is the same as the configuration illustrated in FIG. 1, and therefore the reference numerals in FIG. 1 are used in the following description.

  According to the second embodiment, for example, as shown by 1101 in FIG. 11, after an object having a feature of a circle is designated and the display of image data is switched, subsequently, as shown by 1201 in FIG. When an object having a triangular feature is designated, an operation of giving priority to image data having a plurality of features such as a circle and a triangle becomes possible.

  Next, processing when performing image browsing will be described with reference to FIG. FIG. 16 is a flowchart showing a process at the time of image browsing. In FIG. 16, the same reference numerals as those in FIG. 9 are assigned to the processes common to those in FIG. Hereinafter, steps S1601, S1602, and S1603, which are processes different from those in FIG. 9, will be described.

  In step S1601, the image search unit 110 basically performs the same processing as in step S905. However, the feature is that a previously designated feature is held in the storage device 104 until it is cleared in step S1603, which will be described later, and a newly designated feature is added to the feature designated in the past. As shown in FIG. 11, when an object having a feature of a circle is designated and image data display is switched, and then an object having a feature of a triangle is designated, the feature held in the storage device 104 is a circle. It becomes a triangle. Whether multiple (AND) or OR (logical sum) is selected when a plurality of selections are made is the same as in the first embodiment.

  In step S1602, the input device 103 determines whether the user has issued an instruction to clear the feature information through the application. For example, a clear button is arranged on the display unit 106, and the input device 101 determines whether or not the button is pressed. If an instruction to clear the feature information is issued, the process proceeds to step S1603. On the other hand, if an instruction to clear the feature information has not been issued, the process returns to step S902. In step S1603, the control device 103 clears the feature information. Specifically, the control device 103 deletes the feature information held in the storage device 104. Note that the user may select one or more objects at a time.

  As described above, according to the present embodiment, it is possible to display the image with priority over other image data by an intuitive operation in which the user touches the objects that the user wants to see with priority in order. Similarly to the first embodiment, the image can be displayed with priority over other image data by an intuitive operation in which the user touches an object that the user wants to see with priority. In addition, since it is possible to determine whether an object has been selected based on a user's natural operation, the burden on the user is not increased.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  101: input device, 102: display device, 103: control device, 104: storage device, 105: multiple object designation unit, 106: display unit, 107: selection operation determination unit, 108: object specification unit, 109: feature extraction unit 110: Image retrieval unit 111: Image data group 112: Feature data 113: History information 114: History reference unit

Claims (19)

An information processing apparatus for displaying on a screen at least a part of a plurality of image data according to a predetermined display order,
A specifying means for specifying an object selected by a user among objects included in the image data displayed on the screen ;
Based on the features of the object specified by the specifying means, among the plurality of image data, the display order of the plurality of image data of the image data with priority having characteristics similar to the image data including the object A decision means to decide ;
Display control means for changing the image data displayed on the screen by causing at least a part of the plurality of image data rearranged in the display order determined by the determining means to be displayed on the screen ;
An information processing apparatus comprising: The display control unit is determined based on a display order determined by the determination unit and a direction instructed by the specific operation in response to a specific operation performed on the screen. The information processing apparatus according to claim 1 , wherein the displayed image data is changed to image data . The display control means, from the position of the image data including the object specified by the specifying means, in the display order determined by the determining means in response to a specific operation being performed on the screen, The information processing apparatus according to claim 2 , wherein the displayed image data is changed to a part of image data existing on a direction side instructed by a specific operation . The display control unit is configured to display at least a part of the plurality of image data on the screen without performing the rearrangement when the specific operation is input in a state where the object is not selected by the user. When the specific operation is input in a state where the object is selected by the user, at least a part of the rearranged image data is displayed on the screen. The information processing apparatus according to claim 2, wherein the image data displayed on the screen is changed . The display control means has the predetermined display order based on a direction instructed by the specific operation when the specific operation is input in a state where the object is not selected by the user. 5. The information processing apparatus according to claim 4 , wherein image data to be displayed on the screen among the plurality of image data is changed by moving a portion to be displayed on the screen among the plurality of image data. . A determination unit for determining whether or not the object is selected by a user;
The information processing apparatus according to claim 2, wherein the specifying unit specifies the object when the determination unit determines that the object is selected . The specific operation is a flick operation,
Before Symbol judging means, based on a time interval from the time it is touched by pairs on the touch panel until the time of the flick, according to claim 6, wherein the object is to determine whether the selected Information processing device. The information processing apparatus according to claim 7, wherein the determination unit determines that the object is selected when the time interval is longer than a predetermined time interval . The specific operation is a flick operation,
Before Symbol judging means, based on the time interval of the flick operation against the touch panel, the information processing apparatus according to claim 6, wherein the object and judging whether the selected. The specific operation is a flick operation,
The information processing apparatus according to claim 6, wherein the determination unit determines whether the object is selected based on a pressure of a touch operation or a flick operation on the touch panel. The information processing apparatus according to claim 10, wherein the determination unit determines that the object is selected when the pressure is greater than a predetermined pressure. The specific operation is a flick operation,
The information processing apparatus according to claim 6, wherein the determination unit determines whether or not the object has been selected based on the number of places touched or flicked on the touch panel. The specific operation is a flick operation,
The information processing apparatus according to claim 6, wherein the determination unit determines whether or not the object is selected based on a position of a place touched or flicked on the touch panel. The specific operation is a flick operation,
The information processing apparatus according to claim 6, wherein the determination unit determines whether or not the object has been selected based on an arrangement pattern of a place touched or flicked on the touch panel. The specific operation is a flick operation,
When the arrangement pattern of the place touched or flicked on the touch panel is not a predetermined arrangement pattern representing a natural arrangement derived from the structure of a human hand, the object is not selected. The information processing apparatus according to claim 14, wherein the information processing apparatus determines. The determining means is based on characteristics of the object specified by the specifying means and characteristics of objects included in other image data that is not image data including the specified object among the plurality of image data. The information processing apparatus according to claim 1, wherein a display order in which the image data having the similar characteristics is prioritized is determined. Based on the characteristics of the object specified by the specifying means and the characteristics of an object included in other image data that is not the image data including the specified object among the plurality of image data. A calculation means for calculating the similarity between the image data including the object and the plurality of other image data;
The information according to any one of claims 1 to 16, wherein the determining unit determines an order of high similarity calculated by the calculating unit as a display order in which the plurality of images are rearranged. Processing equipment. An information processing method executed by an information processing apparatus that displays on a screen at least a part of a plurality of image data according to a predetermined display order,
A specifying step of specifying an object selected by the user among the objects included in the image data displayed on the screen;
Based on the characteristics of the object specified by the specifying step, the display order of the plurality of image data giving priority to image data having characteristics similar to the image data including the object is determined among the plurality of image data. A decision step to
A display control step of changing the image data displayed on the screen by displaying at least a part of the plurality of image data rearranged in the display order determined by the determining step on the screen;
An information processing method comprising: A computer that displays at least a part of a plurality of image data according to a predetermined display order on a screen,
A specifying step of specifying an object selected by the user among the objects included in the image data displayed on the screen;
Based on the characteristics of the object specified by the specifying step, the display order of the plurality of image data giving priority to image data having characteristics similar to the image data including the object is determined among the plurality of image data. A decision step to
A display control step of changing the image data displayed on the screen by displaying at least a part of the plurality of image data rearranged in the display order determined by the determining step on the screen;
A program for running

Images