JP6450070B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to a technique for utilizing captured images.

  There is a technology for utilizing the captured image. For example, there is a technique of posting a photographed image to SNS (Social Networking Service) and sharing it with other users. In addition, Non-Patent Document 1 describes a technique for searching for the name of an object and related information from the captured image when the object is imaged.

APPBANK, “CamFind: An application that allows you to search for images from photos taken with a camera.”, [Online], [October 30, 2013 search], Internet, <URL: http://www.appbank.net/2013 /08/31/iphone-application/659436.php>

  In the above-described technology for sharing an image, for example, a browser or the like is activated to access the SNS, and an operation for selecting an image to be posted from a stored image group is performed, or a captured image is displayed. It is necessary to start a program for posting the image to the SNS and perform an operation of posting the image. Further, in the technique of Non-Patent Document 1, it is necessary to perform an operation of starting a specific program before taking an image, and take an image using the function of the program.

As described above, when there are a plurality of techniques for utilizing images, it is cumbersome for the user to operate different procedures depending on the technique used, and these techniques are difficult to use due to the complexity. It can happen.
Therefore, an object of the present invention is to make it easy to use a plurality of techniques for utilizing images.

In order to achieve the above object, the present invention provides display means for displaying an image, an image pickup device for picking up an image including a shooting target, an image picked up by the image pickup device, and image data indicating the image. One of the display control means for causing the display means to display a plurality of operators corresponding to each of a plurality of first processes that can be executed using the display means, and the plurality of operators displayed on the display means When one operator is operated, a second process of storing image data indicating an image captured by the image sensor in the storage unit is executed, and the one operator is used using the stored image data. and a execution means for executing the first process corresponding to the display control means, thumbnail images of the image data is image showing stored in the storage means, wherein by using the image data A processing image representing the first process executed by the line unit is superimposed on the display unit and displayed on the display unit, and the display control unit displays a change operation element that receives an operation for changing the first process indicated by the process image as the thumbnail. An image or an original image of the thumbnail image is displayed on the display unit, and the execution unit displays image data associated with the processed image when an operation is performed on the change operator. An imaging apparatus is provided that executes the first process after the change .

In addition, for each of the plurality of first processes, a determination unit that determines a degree of relevance that indicates the degree of relevance with a shooting target included in an image captured by the image sensor in three or more stages ,
The display control means may display the plurality of operators at positions or modes corresponding to the result determined by the determination means.
Furthermore, the determination unit may determine the degree of association based on a type of the shooting target included in an image captured by the image sensor.

In addition, a positioning unit that measures the position of the device itself may be provided, and the determination unit may determine the degree of association based on the position measured by the positioning unit.
Furthermore, the determination unit may determine the degree of association based on a tendency of the first process executed in the past by the execution unit.

Further, when a plurality of the operators are operated, the execution means performs a plurality of first processes corresponding to the plurality of operators in the order in which the operations are started or the order in which the operations are completed. In accordance with the order, the execution of the first process may be started when a predetermined time has elapsed since the execution of the first process was started .
Further, the image captured by the imaging element includes a plurality of shooting targets, and the display control unit moves the operation element when the user performs a moving operation to move the operation element. The execution means displays the outline of the object to be imaged that is displayed so as to overlap the operation element that is moved when the movement operation is completed, and displays the outline. The first process may be executed using image data indicating an image obtained by extracting a portion to be included.

Further, according to the present invention, an imaging device including a display unit that displays an image and an imaging element that captures an image including an imaging target uses an image captured by the imaging element and image data indicating the image. A display control step for causing the display means to display a plurality of operators corresponding to each of a plurality of first processes that can be executed, and the photographing apparatus includes a plurality of operators displayed on the display means. When any one of the operating elements is operated, a second process for storing image data indicating an image captured by the image sensor in a storage unit is executed, and the stored image data is used to perform the second process. and a execution step of executing the first process corresponding to a single operator, the field of the imaging device, in the display control step, said indicating the image data stored in the storage means A processing image representing the first processing executed in the execution step using the image data is superimposed on the thumbnail image and displayed on the display means, and the photographing device performs the processing image in the display control step. A change operator that accepts an operation to change the first process indicated by is displayed on the display unit so as to be superimposed on the thumbnail image or the original image of the thumbnail image, and the photographing apparatus performs the change operator in the execution step. When the operation is performed, there is provided an imaging method for executing the first process after the change using image data associated with the processed image .

  According to the present invention, it is possible to easily use a plurality of techniques for utilizing an image.

The figure which shows an example of the whole structure of the information processing system of 1st Embodiment The figure which shows an example of the hardware constitutions of an imaging device The figure which shows an example of a function structure of an imaging device The figure which shows an example of a plurality of displayed controls The figure which shows an example of the image displayed after a photography post-process is performed Flow chart showing an example of the operation of the photographing apparatus in the photographing process The figure which shows an example of the hardware constitutions of the imaging device of 2nd Embodiment. The figure which shows an example of a function structure of an imaging device The figure which shows an example of a 1st process relevance table The figure which shows an example of a 2nd process relevance degree table The figure which shows an example of a 3rd process relevance table The figure which shows an example of a plurality of displayed controls The figure which shows an example of the several operation element displayed in 3rd Embodiment The figure which shows an example of the displayed process result image The figure which shows an example of the image containing several imaging | photography object The figure which shows an example of the operation element of a modification

[1] First Embodiment [1-1] Overall Configuration FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to a first embodiment. In this example, an information processing system 1 including an imaging device 10, an SNS server device 21, a related search server device 22, a peripheral search server device 23, and a network 2 to which these devices are connected is shown. Yes. The network 2 is a system that performs communication including the Internet and a mobile communication network, and mediates exchange of data between devices connected to the system.

  The imaging device 10 is an apparatus that captures an object and generates image data (hereinafter referred to as “captured image data”) indicating an image including the captured object. The imaging device 10 is, for example, a digital camera or a smartphone, and is a smartphone in this embodiment. The imaging device 10 executes a plurality of processes that can be executed on the generated captured image data. These processes are performed after an image is captured, and are hereinafter referred to as “post-imaging processing”.

  The multiple post-shooting processes include, for example, a posting process for posting shot image data to the SNS, a related search process for searching for information related to a shooting target (this is referred to as “related search”), This is processing such as peripheral search processing for searching for information related to the vicinity of the location (this is referred to as “perform peripheral search”). In this embodiment, the imaging device 10 does not perform operations such as posting, related search, and surrounding search, but transmits captured image data to a server device that performs these operations, and uses the transmitted captured image data. Then, the processing that requests to perform these operations is executed as post-shooting processing.

  The SNS server device 21 is a computer that provides an SNS to a user, and posts, for example, photographed image data transmitted from the photographing device 10 to the SNS. The related search server device 22 is a computer that performs the above-described related search. For example, the related search server device 22 performs a related search using the captured image data transmitted from the image capturing device 10. The peripheral search server device 23 is a computer that performs the peripheral search described above. For example, the peripheral search server device 23 performs a peripheral search using photographed image data transmitted from the photographing device 10. These server devices may perform operations such as posting, related search, and peripheral search in the form of cloud computing.

[1-2] Hardware Configuration FIG. 2 is a diagram illustrating an example of a hardware configuration of the imaging apparatus 10. The imaging device 10 is a computer that includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and an image sensor 16. The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and a real time clock. The CPU is stored in the ROM or the storage unit 12 using the RAM as a work area. The operation of each part is controlled by executing a program. The real time clock calculates the current date and time and notifies the CPU.

  The storage unit 12 includes a flash memory and the like, and functions as a storage unit that stores data and programs used for control by the control unit 11 and stores image data of an image captured by the own device. The communication unit 13 includes a communication circuit for performing communication, and is a communication unit that exchanges data via the network 2. The display unit 14 includes a liquid crystal display, for example, and is a display unit that displays an image based on control from the control unit 11. The display unit 14 displays an operator for operating the own device under the control of the control unit 11.

  The operation unit 15 includes a transmissive touch screen provided on the liquid crystal display of the display unit 14, and supplies position data indicating the position at which the touch screen detects contact or proximity of an object to the control unit 11. Depending on how the position indicated by the position data changes, operations such as tap, double tap, long tap (long press), and drag and drop are represented. When position data indicating that an operation such as a tap has been performed on the position where the operation element is displayed is supplied, the control unit 11 performs processing according to the operation element.

  The imaging element 16 is an imaging unit that captures an image including a shooting target based on light from the shooting target that reaches through the lens. The image sensor 16 generates image data indicating the image being captured and supplies the image data to the control unit 11. This image data is photographed image data indicating an image including a subject to be photographed. By displaying the captured image data supplied by the control unit 11 on the display unit 14, the user performs imaging while confirming the imaging target. When an operation for capturing an image (hereinafter referred to as “imaging operation”) is performed by the user, the control unit 11 stores captured image data indicating the displayed image in a storage unit (storage unit 12 in the present embodiment). Remember.

  For example, if the own apparatus has an interface with an external storage medium such as an SD card (registered trademark), the control unit 11 stores the captured image data in a storage unit such as an SD card connected via the interface. It may be memorized. Hereinafter, the captured image data that is displayed on the display unit and is not yet stored in the storage unit is referred to as “pre-photographing image data”, and the captured image after the shooting operation is performed and stored in the storage unit. The data is called “post-shoot image data”. When either data is acceptable, it is simply referred to as photographed image data.

[1-3] Functional Configuration When the control unit 11 of the photographing apparatus 10 executes a program stored in the storage unit 12 and controls each unit, the functions described below are realized.
FIG. 3 is a diagram illustrating an example of a functional configuration of the imaging apparatus 10. The photographing apparatus 10 includes a display control unit 101, an execution unit 102, an imaging unit (imaging element 16), a display unit (display unit 14), and a storage unit (storage unit 12).

[1-3-1] Display control means 101
The display control unit 101 is an example of a unit that causes the display unit (display unit 14) to display an image captured by the image sensor 16 and a plurality of controls. Specifically, the display control unit 101 displays a plurality of operators corresponding to each of a plurality of processes that can be executed on captured image data indicating a captured image. The plurality of processes referred to here are post-shooting processes described above, and in the present embodiment, the above-described post process, related search process, and peripheral search process.

  FIG. 4 is a diagram illustrating an example of a plurality of displayed controls. In the example of FIG. 4, the display control unit 101 displays the operators A1, A2, and A3 on the display unit 14. The operators A1, A2, and A3 are operators associated with the posting process, the related search process, and the peripheral search process, respectively. Moreover, the display control means 101 displays the image C1 containing the imaging | photography object B1 of a building. The image C1 is an image captured by the image sensor 16. The controls A1, A2, and A3 are displayed superimposed on the image C1. When the display control unit 101 displays the captured image and a plurality of operation elements as shown in FIG. 4, the position of each operation element displayed on the display unit 14 (hereinafter referred to as “operation element position”). Is supplied to the execution means 102.

[1-3-2] Execution means 102
When the operation element displayed on the display means (display unit 14) is operated, the execution unit 102 performs post-shooting corresponding to the operation element on the captured image data indicating the image captured by the image sensor 16. It is an example of a means for executing processing. The captured image data for which the execution unit 102 executes post-shooting processing may be pre-shooting image data or post-shooting image data. When executing post-shooting processing on pre-shooting image data, the execution unit 102 reads out pre-shooting image data stored in, for example, a RAM, executes post-shooting processing, and performs post-shooting image data. When the post-shooting process is executed, the post-shooting image data stored in the storage unit 12 (or external storage device) is read and the post-shooting process is executed.

  The execution unit 102 also executes processing for storing the pre-photographing image data in the storage unit (hereinafter referred to as “storage processing”). When the execution unit 102 performs the storage process, an image including the imaging target is captured. The execution unit 102 executes the storage process when the shooting operation is performed, regardless of which of the operators A1, A2, and A3 the shooting operation is performed. In the present embodiment, an operation of tapping the operation element is performed as a shooting operation. For example, when the position indicated by the operator position data supplied from the display control unit 101 is tapped when the display shown in FIG. 4 is being performed, the execution unit 102 determines that the shooting operation has been performed and displays the image C1. The pre-photographing image data shown is stored in the storage unit 12. The execution means 102 performs the storage process in this way.

The execution means 102 transmits a request data requesting to post the photographed image data with the user account in the SNS to the SNS server device 21 shown in FIG. Execute as post processing. Upon receiving this request data, the SNS server device 21 posts post-photographing image data included in the received request data with the user's account, and transmits page data indicating the post-posting SNS page to the photographing device 10. The execution unit 102 supplies the received page data to, for example, the display control unit 101 and causes the display unit 14 to display the page data.
FIG. 5 is a diagram illustrating an example of an image displayed after the post-shooting process is executed. In FIG. 5A, the SNS page indicated by the page data transmitted from the SNS server device 21 is displayed. An image C1 is posted on this page.

  Further, if the operator that has performed the shooting operation is A2, the execution unit 102 transmits request data for requesting the related search using the post-shooting image data to the related search server device 22 illustrated in FIG. The process is executed as a related search process. When the related search server device 22 receives the request data, the related search server device 22 specifies a shooting target included in the image indicated by the shooting image data included in the received request data, and a search result indicating a result of searching for information regarding the specified shooting target. Data is transmitted to the photographing apparatus 10. The execution unit 102 supplies the received search result data to, for example, the display control unit 101 and displays it on the display unit 14. In FIG. 5B, the search result of the Sagrada Familia indicated by the search result data transmitted from the related search server device 22 is displayed. The execution means 102 executes the peripheral search process even when a shooting operation is performed on the operator A3, as in the case where a shooting operation is performed on the operator A2.

[1-4] Operation The imaging device 10 performs imaging processing including the above-described post-imaging processing based on the above configuration. The operation of the photographing apparatus 10 in the photographing process will be described with reference to FIG.
FIG. 6 is a flowchart showing an example of the operation of the imaging apparatus 10 in the imaging process. The shooting process is started, for example, when a program for performing the shooting process is started. First, the imaging device 10 captures an image including an imaging target (step S11), and displays the captured image and an operator (step S12). In this example, the photographing apparatus 10 displays the operators A1, A2, and A3 shown in FIG. Step S11 is an operation performed by the imaging device 16, and step S12 is an operation performed by the display control means 101 and the display unit 14 in cooperation.

  Next, when the photographing apparatus 10 receives a photographing operation for any of the displayed operators (step S13), the photographing apparatus 10 performs an operation according to the manipulation member on which the photographing operation has been performed (step S14). When the operator who has performed the shooting operation is A1 (step S14; A1), the shooting apparatus 10 stores the shot image data indicating the image being shot (step S21), and performs the above-described post processing. This is performed (step S22). Then, the photographing apparatus 10 displays the result of the post-shooting process (Step S15) and ends the shooting process. When the operation of step S22 is performed, the photographing apparatus 10 displays the result of posting the photographed image data as shown in FIG. 5A to the SNS, for example.

  In addition, when the operator who performed the shooting operation is A2 (step S14; A2), the shooting device 10 stores the shot image data indicating the image being shot (step S31), and the related search described above. Processing is performed (step S32), and the operation of step S15 is performed. When the operation of step S32 is performed, the photographing apparatus 10 displays a result of searching for information related to the photographing target, for example, as illustrated in FIG. Further, when the operator who performed the shooting operation is A3 (step S14; A3), the shooting apparatus 10 stores the shot image data indicating the image being picked up (step S41), and the peripheral search described above. Processing is performed (step S42), and the operation of step S15 is performed. Each operation from step S13 to S42 is an operation performed by the execution means 102. Note that the processing in step S15 may be included in the post-shooting processing, and the execution unit 102 may perform the operation in step S15.

[1-5] Effects According to First Embodiment Posting captured image data to the SNS, searching for information about a shooting target indicated by the shooting image data, and searching for information about the vicinity of the location of the shooting target are as follows: Both are techniques for utilizing captured images. When there are a plurality of such techniques, it is cumbersome and inconvenient for the user to perform operations with different procedures depending on the technique used. In the present embodiment, by using a plurality of operators corresponding to post-shooting processing, each of the plurality of post-shooting processings is displayed while displaying the image being picked up. What is necessary is just to follow the procedure of performing a photographing operation on the operator. According to the present embodiment as described above, it is possible to make it easier to use a plurality of techniques for utilizing an image, compared to a case where a plurality of operators corresponding to post-shooting processes are not used.

[2] Second Embodiment Hereinafter, a second embodiment of the present invention will be described focusing on differences from the first embodiment. In the first embodiment, there is no significant difference in the sizes and display positions of a plurality of operation elements. In the second embodiment, the user can easily operate an operation element that is likely to be photographed by the user. Each control is displayed.

[2-1] Hardware Configuration FIG. 7 is a diagram illustrating an example of a hardware configuration of the imaging apparatus according to the second embodiment. In the example of FIG. 7, an imaging device 10 a including a positioning unit 17 in addition to each unit illustrated in FIG. 2 is illustrated. The positioning unit 17 is a positioning unit that measures the position of its own device using, for example, GPS (Global Positioning System) technology. The positioning unit 17 supplies positioning data indicating the measured position of the own device to the control unit 11. In the present embodiment, the positioning data indicates the position of the own device by latitude and longitude.

[2-2] Functional Configuration FIG. 8 is a diagram illustrating an example of a functional configuration of the photographing apparatus 10a. The imaging device 10a includes a determination unit 103 and a positioning unit (positioning unit 17) in addition to the units illustrated in FIG.
The determination unit 103 determines a degree of relevance (hereinafter referred to as a “process relevance degree”) that indicates the degree of relevance to the photographing target included in the image captured by the image sensor 16 for each of the plurality of post-shooting processes. It is an example of the means to do.

  For example, if a posting process is often executed when a person is photographed as a photographing target, the photographing target of the person and the posting process are highly related. On the other hand, if the surrounding search process is hardly executed when a person is shot as the shooting target, the relationship between the shooting target called a person and the surrounding search process is low. As described above, the higher the degree of processing relevance between the photographing target and the post-shooting process, the higher the possibility that the post-shooting process is performed on the image data indicating the image including the photographing target. In this embodiment, the determination unit 103 determines the processing relevance by any one or a plurality of determination methods among the first, second, and third determination methods.

[2-2-1] First Determination Method The determination unit 103 determines the degree of processing relevance based on the type of shooting target included in the image captured by the image sensor 16. The determination unit 103 stores, for example, a first process relevance level table in which types of shooting targets, post-shooting processes, and process relevance levels are associated with each other.
FIG. 9 is a diagram illustrating an example of the first process relevance degree table. In this example, for the type of subject to be photographed “person”, post-photographing processing “post processing”, “related search processing”, and “surrounding search processing” and processing “high”, “low”, and “low” Relevance levels are associated with each other.

  These process relevance levels represent the relative relationship of the process relevance levels of each post-shooting process. In other words, when the subject to be photographed is a person, the degree of processing relevance with the posting process is the highest (that is, the possibility that the posting process is executed is the highest). Is low (that is, the possibility that the related search process and the surrounding search process are executed is low). Similarly, the processing relevance level of each post-shooting process for the types of shooting targets “food” and “building” is “medium”, “low”, “high” and “medium”, “high”, and “medium”. It is associated. For example, the processing relevance with the surrounding search process is the highest when the shooting target is food, and the processing relevance with the related search process is highest when the shooting target is a building.

  The determination unit 103 performs image recognition on the captured image, and determines the type of shooting target included in the image. The determination unit 103 performs this determination, for example, by extracting the contour of the photographing target and matching it with a pattern image corresponding to the type of the photographing target. In addition, the determination unit 103 may use various other well-known image recognition techniques for determining the type of imaging target. When determining the type of the imaging target, the determination unit 103 determines the processing relevance level of each post-shooting process based on the post-shooting process and the process relevance level associated with the type in the first process relevance level table. The determination unit 103 supplies the display control unit 101 with determination result data that associates the post-shooting process and the processing relevance used for the determination.

[2-2-2] Second Determination Method The determination unit 103 determines the processing relevance based on the position measured by the positioning unit (positioning unit 17). The determination unit 103 stores, for example, a second process relevance level table in which a position where the user is assumed to perform shooting (hereinafter referred to as “imaging shooting position”), a post-shooting process, and a process relevance level are associated. ing.
FIG. 10 is a diagram illustrating an example of the second process relevance degree table. In this example, a post-shooting process called “posting process”, “related search process”, and “surrounding search process” and a process called “high”, “medium”, and “high” for an assumed shooting position of “sightseeing spot” Relevance levels are associated with each other.

  This means that when a user takes a picture at a sightseeing spot, whatever the shooting target is, the degree of processing relevance with the posting process and the surrounding search process is as high as that, and the degree of processing relevance with the related search process is lower than those. Represents that. Similarly, the processing relevance of each post-shooting process for the assumed shooting positions of “Shopping Street” and “Movement Route” is “Low”, “High”, “Medium” and “Low”, “Medium”, “High”. Are associated. For example, when a user shoots in a shopping street, the relevance to the related search process is the highest regardless of the shooting target, and when the user shoots while moving, the shooting target is It shows that the degree of processing relevance with search processing is the highest.

  The determination unit 103 determines whether or not the position measured by the positioning unit 17 is included in each assumed shooting position. Specifically, the determination unit 103 stores range data representing a range on a map of a sightseeing spot, a shopping street, and a movement route (a road or a track site) as a set of latitude and longitude, and the positioning unit 17 It is determined whether or not the position measured by is included in any of these ranges. For example, if the measured position is included in the sightseeing spot, the determination unit 103 determines whether each post-shooting process is based on the post-shooting process and the processing relevance associated with the sightseeing spot in the second execution determination table. The degree of processing relevance is determined, and the determination result data is supplied to the display control means 101.

[2-2-3] Third determination method The determination unit 103 determines the degree of processing relevance based on the tendency of the post-shooting processing executed in the past by the execution unit 102. When executing the post-shooting process, the executing unit 102 notifies the determining unit 103 of the post-shooting process that has been executed. The determination unit 103 counts the number of executions for each notified post-shooting process, and stores a third execution determination table indicating that the higher the number of executions counted, the higher the degree of processing relevance.

  FIG. 11 is a diagram illustrating an example of the third process relevance degree table. In this example, the number of executions of “post processing”, “related search processing”, and “surrounding search processing” is “50”, “30”, and “20”, respectively, and “high”, “medium”, and “low”. Are associated with each other. The determination unit 103 updates the number of executions in the third execution determination table every time the post-shooting process is executed by the execution unit 102. If the magnitude relation of the number of executions changes, the determination unit 103 performs processing according to the change. Also change the relevance.

  For example, in the example of FIG. 11, when the number of executions of the surrounding search process is greater than the number of executions of the related search process, the degree of process relevance of “related search process” and “ambient search process” is set to “low” and “medium”. change. When the number of executions is the same, the degree of processing relevance of both may be “medium” or “low”. In short, it is only necessary to be able to understand the relative magnitude relationship of the processing relevance level of each post-shooting process. The determination unit 103 determines the processing relevance level of each post-shooting process based on the stored third execution determination table, and supplies the determination result data to the display control unit 101.

[2-3] Display Control Unit The display control unit 101 displays the operation element in a position and manner according to the result determined by the determination unit 103. In the photographing processing shown in FIG. 6, the determination by the determination unit 103 is performed before step S12, and when the display control unit 101 displays a plurality of controls in step S12, those operations are performed according to the determination result. Display children.

  FIG. 12 is a diagram illustrating an example of a plurality of displayed controls. In the example of FIG. 12, the display control unit 101 causes the display unit 14 to display the image C2 including the food photographing target B2 and the operators A1, A2, and A3 illustrated in FIG. In this example, for the sake of brevity, the position of the photographing apparatus 10 is not included in any assumed photographing position, and the number of executions is zero, so the second and third determinations It is assumed that there is no difference in processing relevance between methods. In this case, the determination unit 103 performs the determination by the first determination method, determines that the type of the imaging target is “food”, and processes related to “post processing”, “related search processing”, and “surrounding search processing” The degrees are determined as “medium”, “low”, and “high”, respectively.

  The display control unit 101 is configured so that the operator A3 corresponding to the “surrounding search process” having the highest degree of processing relevance indicated by the determination result data supplied from the determination unit 103 is the leftmost position and the size is maximized. It is displayed. Further, the display control unit 101 displays the operation element A1 corresponding to the “post process” having the next highest processing relevance level in a mode in which the operation element A1 has the next largest size at the middle position and the operation element A3. The operator A2 corresponding to the “related search process” with the smallest is displayed in the rightmost position and in the form with the smallest size.

  When operating the imaging device 10 with the left hand, the operator is easier to operate as the operator is closer to the left side. Also, the larger the size of the operator, the easier the user can operate. That is, in the example of FIG. 12, the display control unit 101 displays an operation element having a higher degree of processing relevance in a position and manner that the user can easily operate. As a result, compared to the case where the operation element is not displayed in the position and manner according to the determination result, the operation element of the post-shooting process that the user wants to execute is easier to operate. Can improve convenience.

  When the processing relevance levels are determined by the first, second, and third determination methods, for example, the display control unit 101 indicates that the processing relevance levels indicated by the determination result data are “high” and “medium”, for example. , “3”, “2”, and “1” are assigned to “low”, respectively, and the total of the points assigned to the results determined by each determination method is related to each post-shooting process. Compare the points as degrees. The larger the score, the higher the processing relevance. For example, when the type of shooting target is “person”, the assumed shooting position is “shopping street”, and the number of executions is as shown in FIG. 11, the processing relevance of the posting process is “high” and “low”. Since it is “high”, it is represented by a score of 3 + 1 + 3 = 7. Similarly, the related search processing is 1 + 3 + 2 = 6 for “low”, “high”, and “medium”, the surrounding search processing is 1 + 2 + 1 = 4 for “low”, “medium”, and “low”, and the processing relevance is post processing. Is the highest and the surrounding search processing is the lowest.

  In the present embodiment, the first determination method and the second determination method are used, so that the degree of processing relevance varies depending on the type of shooting target and the shooting location, that is, post-shooting processing that is highly likely to be executed. Even if they are different, the convenience of the user can be improved. In addition, by using the third determination method, it is possible to easily operate the post-shooting processing operation element that is often executed by the user, and to improve the convenience for the user.

  Note that how to express the processing relevance is not limited to that described above. The process relevance may be represented by a character or character string different from the above, such as “A”, “B” and “C”, “Lv1”, “Lv2” and “Lv3”, or from the beginning. It may be expressed by a numerical value such as a score. In the above example, the degree of processing relevance is expressed in three stages, but it may be two stages or four or more stages. In short, any representation may be used as long as the processing relevance of each post-shooting process can be compared.

  In the above example, the display control unit 101 displays the operation element in the position and manner according to the order of the processing relevance level, but displays the operation element in the position and manner according to the process relevance level itself. Also good. For example, the display control unit 101 displays the operation element with a size corresponding to the above-described score. Specifically, if the score is 7, the control is displayed with a size of 7 × k pixels (k is a constant), and if the score is 4, the control is displayed with a size of 4 × k pixels. Condition. In this case, the user can be informed of the difference in the degree of processing relevance determined for each post-shooting process.

[3] Third Embodiment Hereinafter, a third embodiment of the present invention will be described focusing on differences from the first and second embodiments. In the third embodiment, when an image indicated by the captured image data stored in the storage unit, that is, when a list of captured images is displayed, the post-shooting process executed on the captured image data indicating the images Is shown to the user.

[3-1] Display Control Unit In the present embodiment, the display control unit 101 is executed by the execution unit 102 on the captured image data stored in the storage unit (for example, the storage unit 12) and the captured image data. In addition, an image representing the post-shooting process (hereinafter referred to as “processed image”) is displayed in association with the display unit.
FIG. 13 is a diagram illustrating an example of a plurality of operators displayed in the third embodiment. In the example of FIG. 13, the display control unit 101 displays the thumbnail image C11 of the image C1 shown in FIG. 4 and the thumbnail image C12 of the image C2 shown in FIG.

  In addition, the display control unit 101 displays a processed image D1 representing post-shooting processing (post processing in this example) performed on the shot image data indicating the image C1 so as to overlap the thumbnail image C11, and shows the image C2. A processed image D2 representing post-shooting processing (periphery search processing in this example) performed on the shot image data is displayed superimposed on the thumbnail image C12. As a result, the user can easily confirm which post-shooting process has been executed for the shot image data corresponding to the displayed thumbnail image C11 as compared to the case where the processed image is not displayed.

For example, when the user performs an operation of tapping the thumbnail image C11 or the processed image D1, the display control unit 101 displays the original image C1 and the enlarged processed image D1 of the thumbnail image C11.
FIG. 14 is a diagram illustrating an example of a state in which the processed image is enlarged and displayed. In the example of FIG. 13, the display control unit 101 displays the image C <b> 1 and the processed image D <b> 1 enlarged on the upper side of the display unit 14.

  In addition, the display control unit 101 displays operating elements A21 and A31 corresponding to post-shooting processing that has not been executed on the lower side of the display unit 14. The operators A21 and A31 are operators for accepting an operation for changing the post-shooting process indicated by the processed image displayed on the display unit 14 (hereinafter referred to as “change operation”), and hereinafter referred to as “change operator”. . The display control unit 101 may display the change operator together with the thumbnail image instead of displaying it together with the processing result image. The post-shooting process is changed by the execution means 102 by operating the change operator.

[3-2] Execution Unit The execution unit 102 associates the processing image when the above-described change operation, that is, an operation to change the post-shooting process indicated by the processing image displayed on the display unit 14 is performed. The post-shooting process after the change is executed on the captured image data. For example, if the operation of tapping the operation element A2 in the example of FIG. 14 is performed as a change operation, the execution unit 102 shows the image C1 of FIG. 4 in which the post-shooting process represented by the processed image D1 is executed. The related retrieval process corresponding to the operation element A2 is executed as the post-shooting process after the change for the shot image data. Accordingly, the user can easily execute post-shooting processing that he / she wants to execute again when he / she looks back at an image shot in the past as compared with a case where the post-shooting process is not changed.

  As described above, when the execution unit 102 executes the post-shooting process after the change, the display control unit 101 displays an image representing the post-shooting process after the change as a processed image. In addition, for example, when an operation is performed on the processed image D1 in FIG. 14, the display control unit 101 displays an image displayed when post-shooting processing has been executed in the past on the shot image data indicating the image C1 (FIG. 5). Image) is displayed. For example, when displaying an image indicating the result of post-shooting processing (hereinafter referred to as “processing result image”) in the past, the display control unit 101 stores the processing result image in association with the shot image data. When the aforementioned operation is performed, the processing result image is read and displayed.

Note that the display control unit 101 may display a result of execution of the post-shooting processing represented by the processed image again as the processing result image without storing the processing result image.
Further, when displaying the processed image as shown in FIG. 13, the display control unit 101 may display the thumbnail images rearranged according to the type of the processed image. As a result, the user can collectively display, for example, captured image data that has been subjected to specific post-shooting processing.

[4] Modifications Each of the above-described embodiments is merely an example of an embodiment of the present invention, and may be modified as follows. Moreover, you may implement each embodiment mentioned above and each modification shown below, combining each as needed.

[4-1] Method of Displaying Operators In the embodiment, the display control unit 101 displays a plurality of operators arranged in the left-right direction. However, the display control unit 101 is not limited thereto, and may be displayed in an up-down direction, for example. However, they may be displayed at the four corners. In addition, the display control unit 101 may display a portion of the captured image where the operation element overlaps so as to be lightly transmitted through the operation element. In addition, the display control unit 101 may display the operators in a superimposed manner, or display a part of the operators and scroll the displayed operators to display the remaining operators. You may be made to do. In the embodiment, the operation element is displayed so as to be superimposed on the captured image. However, the display control unit 101 may display them at different positions without overlapping them. In any case, it is desirable that the user can easily check the shooting target included in the captured image even when a plurality of controls are displayed.

[4-2] Method of Shooting Operation In the embodiment, the operation of tapping the operation element is performed as the shooting operation, but the shooting operation is not limited to this. For example, a double tap or a long tap (long press) may be performed as a shooting operation. In addition, when the captured image and the operation element are displayed without overlapping, an operation of dragging the operation element and dropping it in a state of being superimposed on the captured image may be performed as a photographing operation. Note that when drag and drop is a shooting operation, the display control unit 101 may display an image representing a position where the operation element is dropped, and the operation element may be dropped at the position of the image.

  In addition, an operation for selecting and determining one of a plurality of displayed operators by operating an operator having a mechanical structure such as a button or a wheel may be performed as a shooting operation. . Further, when the photographing apparatus is provided with a microphone, an operation for designating an operation element by voice (for example, if the voice is “Sosasei”), the operation element A is operated as a photographing operation. May be. In short, any operation may be performed as a photographing operation as long as it can be understood which of the plurality of operators has been operated.

[4-3] Shooting operation for two or more operators In the embodiment, a shooting operation is performed for any one of a plurality of operators. However, a shooting operation is performed for two or more operators. It may be broken. For example, a shooting operation such as a tap is simultaneously performed on the controls A1 and A2 shown in FIG. As described above, when the shooting operation is performed on the plurality of operators, the execution unit 102 executes a plurality of post-shooting processes corresponding to the plurality of operators. If the photographing operation is performed on the operators A1 and A2, the execution unit 102 executes both the posting process and the related search process.

  When executing a plurality of post-shooting processes, the execution unit 102 may execute the processes in parallel or sequentially. For example, the execution unit 102 executes the plurality of post-shooting processes in the order corresponding to the order in which the shooting operations are started or the order in which the shooting operations are completed. That is, if the operator A2 is operated first and the operator A1 is operated later, the execution unit 102 first executes the related search process corresponding to the operator A2, and performs the posting process corresponding to the operator A1. Run later. The execution unit 102 may start the subsequent post-shooting process when a predetermined time (for example, 30 seconds or 1 minute) has elapsed since the start of the previous post-shooting process. The post-shooting process may be executed after the post-shooting process is finished.

According to this modification, when the user wants to execute a plurality of post-shooting processes, the user can easily execute them as compared with a case where only a shooting operation for one operator is accepted.
When a plurality of post-shooting processes are executed in parallel, the processing capacity and communication bandwidth of the CPU used in these processes are divided, and each post-shooting process is compared with the case where one post-shooting process is executed. The time required to complete the execution will be extended. In this modification, since the post-shooting process is executed in order, for example, if there is a post-shooting process that the user wants to execute first, the post-shooting process is performed in order by performing the shooting operation so that it is executed first. The post-shooting process can be completed earlier than when not executed.

[4-4] Post-shooting Process for Individual Shooting Target In the embodiment, the post-shooting process is performed on the shot image data indicating the entire image picked up by the image pickup device 16. Post-shooting processing may be performed on the shot image data indicating a part of the shot image data. The part of the captured image is, for example, an image obtained by cutting out a shooting target included in the image. A method of shooting operation and a method of executing post-shooting processing in this case will be described below.

  FIG. 15 is a diagram illustrating an example of an image including a plurality of shooting targets. In this example, as shown in FIG. 15A, an image C3 imaged by the image sensor 16 is displayed on the display unit 14, and the image C3 includes an imaging object B3 (teapot) and an imaging object B4. A plurality of subjects to be photographed (steamers and coasters) are included. In addition, the display unit 14 displays the operators A1, A2, and A3. When the user performs an operation for moving the operation element (hereinafter referred to as “movement operation”), the display control unit 101 displays the operation element while moving the operation element in accordance with the movement operation. The movement operation is, for example, a drag operation. FIG. 15B shows the dragged operation element A2.

  The execution unit 102 executes post-shooting processing on image data indicating an image including a shooting target displayed overlapping the operation element moved when the moving operation is completed. For example, the execution unit 102 performs image recognition on the image C3 to acquire the positions of the shooting targets B3 and B4. When the user drags the operation element A2 to move it to the position of the object to be imaged B3 and performs a drop operation (releases the finger from the operation element A2), the execution means 102 determines that the movement operation has ended, and the movement operation It is determined whether or not the position of the operation element A2 when the operation is completed, that is, the position where the operation element A2 is dropped is included in the obtained position of the photographing target.

  In the example of FIG. 15B, the execution unit 102 determines that the image is included in the position of the imaging target B3, extracts an outline of the imaging target B3, and extracts an image obtained by extracting a portion including the outline from the image C3. Captured image data (hereinafter referred to as “partial image data”) is generated. The execution unit 102 performs post-shooting processing corresponding to the operated operator on the partial image data generated in this way. For example, when related search processing and surrounding search processing are executed, if captured image data indicating the image C3 is used, information regarding the imaging target B4 (bathcup and coaster) is also obtained as a search result. When the partial image data indicating B3 is used, information limited to the photographing target B3 (teapot) is easily obtained as a search result. According to such a modification, information that the user wants to obtain can be obtained by the post-shooting process as compared with the case where the post-shooting process based on the moving operation is not executed.

[4-5] Operators corresponding to a plurality of post-shooting processes In the embodiment, one operator corresponds to a single post-shooting process. However, one operator corresponds to a plurality of post-shooting processes. May be.
FIG. 16 is a diagram illustrating an example of an operation element according to this modification. FIG. 16A shows an operator A4 corresponding to the posting process and the related search process. When the operation element A4 is operated, the execution unit 102 performs a plurality of post-shooting processes corresponding to the operation element A4 on the captured image data, that is, a posting process and a related search process. Thereby, a plurality of post-shooting processes can be executed with a simpler operation than operating a plurality of operators.

  Operators corresponding to a plurality of post-shooting processes may be generated from the plurality of operators. For example, when the display control means 101 displays the controls A1 and A2 as shown in FIG. 16B and is dropped when the control A1 is dragged and overlaps the control A2, the control A1 is displayed. And the operation element A4 is displayed instead of A2. The display control means 101 may store the operator generated in this way and display it on and after the next time. As a result, when there are a plurality of combinations of post-shooting processes that the user often performs, the user can create an operator corresponding to the combination and use it after that.

[4-6] Position and Mode According to Processing Relevance The position and mode in which the display control unit 101 displays the operation element in the second embodiment are not limited to those described above. For example, the display control unit 101 may increase the saturation of the color of the operation element so that the post-shooting process having a higher degree of processing relevance may be noticeable, or may change the color or shape of the operation element depending on the degree of processing relevance. Also good. Further, if the display control unit 101 displays the controls in the vertical direction or near the four corners as described above, the post-shooting processing controls having a higher processing relevance are displayed at positions that are easier for the user to operate. Good.

  For example, if the user operates with the left hand only while holding the photographing device, the post-photographing processing operator having a higher processing relevance is displayed at a position closer to the lower left where the thumb is located. Note that the position at which the user can easily operate may vary depending on how the user holds the bag or the bag, so that the user can determine the position at which the user can easily operate. In that case, the display control means 101 displays the post-shooting processing operator having a higher processing relevance at a position closer to the position determined by the user that is easy to operate. Further, the display control unit 101 may display the operation element by causing only one of the position and the mode (that is, the position or the mode) to be determined according to the result determined by the determination unit 103.

[4-7] Post-shooting processing The post-shooting processing is not limited to that described above. For example, a process for searching for information on a store that sells a shooting target or a process for searching for information on a coupon that can be used for purchasing a shooting target may be executed as the post-shooting process. Further, image processing for performing edge extraction, color conversion, and the like on the image indicated by the captured image data may be executed as post-shooting processing, or processing for adding information called a tag to the captured image data is executed as post-shooting processing. May be.

  It should be noted that the post-shooting process may require a user operation. For example, in the process of adding a tag, the user may be made to perform an operation of inputting the content of the tag to be added after the operation of the operator is performed. In the embodiment, the imaging device performs post-imaging processing including processing for transmitting captured image data to a server device such as the SNS server device 21, but performs post-imaging processing that does not include processing for transmission to the server device. May be. For example, the imaging device may execute a related search process and a peripheral search process as a post-imaging process instead of the server apparatus.

[4-8] Shooting of Moving Image In the embodiment, the captured image data is data indicating a still image such as a photograph, but is not limited thereto, and may be data indicating a moving image. Also in this case, as in the case of a still image, a post process to SNS and a process for adding a tag can be executed as post-shooting processes. In addition, processing that can be performed on image data indicating a moving image, such as compression, division, and conversion of the moving image, may be performed as post-shooting processing. In addition, if a service for performing a related search or a peripheral search based on a moving image is provided, the related search process or the peripheral search process may be performed on captured image data indicating a moving image. In short, any of the post-shooting processes described above may be executed as long as it can be executed.

[4-9] Storage Processing The storage processing executed by the execution unit 102 is not limited to that described in the embodiment. For example, the execution unit 102 may execute, as a storage process, a process of transmitting captured image data to a server device that provides a service for synchronizing captured image data with a plurality of information processing apparatuses. In this case, the execution unit 102 performs storage processing using the server device as a storage unit. In short, the execution unit 102 may execute a process of storing the captured image data in some storage unit as a storage process.

[4-10] Category of the Invention The present invention is an information processing system including an external device (such as an SNS server device in the embodiment) that performs the photographing device and post-photographing processing in cooperation with the photographing device in addition to the photographing device. Can also be caught. The present invention can also be understood as a photographing method for realizing processing performed by the photographing apparatus. The processing here is, for example, the photographing processing shown in FIG. The present invention can also be understood as a program for causing a computer such as a photographing apparatus to function as each unit shown in FIG. This program may be provided in the form of a recording medium such as an optical disk storing the program, or may be provided in the form of being downloaded to a computer via a network such as the Internet, and making it available for installation. You may do.

DESCRIPTION OF SYMBOLS 1 ... Information processing system, 2 ... Network, 10 ... Imaging device, 21 ... SNS server device, 22 ... Related search server device, 23 ... Surrounding search server device, 11 ... Control part, 12 ... Memory | storage part, 13 ... Communication part, DESCRIPTION OF SYMBOLS 14 ... Display part, 15 ... Operation part, 16 ... Image sensor, 17 ... Positioning part, 101 ... Display control means, 102 ... Execution means, 103 ... Determination means.

Claims (8)

Display means for displaying an image;
An image sensor that captures an image including an imaging target;
Display control means for causing the display means to display an image captured by the image sensor and a plurality of operators corresponding to each of a plurality of first processes executable using image data indicating the image;
A second process for storing, in a storage unit, image data indicating an image picked up by the image sensor when any one of the plurality of operators displayed on the display unit is operated; And executing means for executing the first process corresponding to the one operator using the stored image data ,
The display control unit superimposes a processing image representing the first process executed by the execution unit using the image data on a thumbnail image of the image indicated by the image data stored in the storage unit. Display on the display means,
The display control unit causes the display unit to display a change operation element that receives an operation of changing the first process indicated by the processed image on the thumbnail image or the original image of the thumbnail image,
The execution device executes the first process after change using image data associated with the processed image when an operation is performed on the change operator . For each of the plurality of first processes, a determination unit is provided that determines a degree of relevance that indicates a degree of relevance in three or more stages with respect to a photographing target included in an image captured by the image sensor,
The imaging apparatus according to claim 1, wherein the display control unit displays the plurality of operators at positions or modes corresponding to the result determined by the determination unit. The imaging apparatus according to claim 2, wherein the determination unit determines the relevance level based on a type of the imaging target included in an image captured by the imaging element. It has positioning means to measure the position of its own device,
The imaging device according to claim 2, wherein the determination unit determines the degree of association based on a position measured by the positioning unit. The imaging device according to any one of claims 2 to 4, wherein the determination unit determines the degree of association based on a tendency of the first process executed in the past by the execution unit. Before SL execution means, when a plurality of the operators is operated, depending on the order in which the plurality of first process corresponding to the plurality of operators, the operation starts the order or the operation is completed The imaging according to any one of claims 1 to 5 , wherein, in a predetermined order, the execution of the first process is started when a predetermined time has elapsed since the execution of the first process was started. apparatus. Includes a plurality of imaging target in the image being captured by the previous SL imaging device,
When the user performs a moving operation for moving the operating element, the display control means displays the operating element while moving the operating element according to the moving operation,
The execution means extracts image data indicating an image obtained by extracting a contour of the subject to be imaged that is displayed so as to overlap with the operator moved when the moving operation is completed and extracting a portion including the contour. The imaging device according to any one of claims 1 to 6 , wherein the first process is executed using the imaging device. An imaging device including a display unit that displays an image and an imaging device that captures an image including an imaging target can be executed using an image captured by the imaging device and image data indicating the image. A display control step of causing the display means to display a plurality of operators corresponding to each of the first processes;
When the photographing device operates any one of the plurality of operators displayed on the display unit, image data indicating an image captured by the imaging element is stored in the storage unit. Performing a second process to be stored, and executing the first process corresponding to the one operator using the stored image data,
Processing that represents the first process executed in the execution step by using the image data for the thumbnail image of the image indicated by the image data stored in the storage unit in the display control step in the imaging device Overlapping images to display on the display means,
In the display control step, the photographing apparatus causes the display unit to display a change operation element that receives an operation to change the first process indicated by the processed image on the thumbnail image or the original image of the thumbnail image,
When the operation is performed on the change operator in the execution step, the imaging apparatus executes the first process after the change using image data associated with the processed image.
Shooting method.

Images