JP7533011B2 - Information processing device, program, and information processing method - Google Patents

Description

The present invention relates to an information processing device, a program, and an information processing method.

Patent document 1 discloses a technology that determines the size of a layout frame according to pre-set rules based on the width, height, and resolution of an image, and automatically adjusts the display size of the image by fitting the image into the determined layout frame.

Patent No. 3870834

The technology disclosed in Patent Document 1 requires that rules for determining the size of the layout frame be set in advance, which places a large processing burden on the user when setting the rules. In addition, if the aspect ratio of the image and the aspect ratio of the layout frame differ significantly, the image fitted into the layout frame may appear unnatural.

The present invention has been made in consideration of these circumstances, and its purpose is to provide an information processing device etc. that can properly display a subject (object) at a specified display size without increasing the processing load.

An information processing device according to one aspect of the present invention includes an image acquisition unit that acquires an image including an object, an aspect ratio acquisition unit that acquires the aspect ratio of a display area in which the object is displayed, a comparison unit that detects the object from the image and compares the aspect ratio of the detected object area with the aspect ratio of the display area, a switching unit that sequentially switches the object area having the aspect ratio of the display area when the aspect ratio of the object area differs from the aspect ratio of the display area, and an identification unit that identifies the area to be switched based on the relevance between the object and an image cut out based on the switched area.

In one aspect of the present invention, a subject (object) can be appropriately displayed at a specified display size without increasing the processing load.

1 is a block diagram showing an example of the configuration of an information processing device. FIG. 11 is a schematic diagram for explaining an image editing process. 13 is a flowchart illustrating an example of an image editing process. 13 is a flowchart illustrating an example of an image editing process. FIG. 11 is a schematic diagram for explaining an image editing process. FIG. 11 is a schematic diagram for explaining an image editing process. FIG. 11 is a schematic diagram for explaining an image editing process. 13 is a flowchart showing another example of the image editing processing procedure. FIG. 13 is a schematic diagram showing an example of a screen. 13 is a flowchart illustrating an example of a score calculation process procedure according to the second embodiment. FIG. 11 is a schematic diagram for explaining a score calculation process. FIG. 11 is a schematic diagram for explaining a layout process. 13 is a flowchart illustrating an example of a layout processing procedure. 13 is a flowchart illustrating an example of a layout processing procedure. 13 is a flowchart illustrating an example of a layout processing procedure. FIG. 11 is a schematic diagram for explaining a layout process. FIG. 11 is a schematic diagram for explaining a layout process. FIG. 11 is a schematic diagram for explaining a layout process. FIG. 11 is a schematic diagram for explaining a layout process.

The information processing device, program, and information processing method disclosed herein are described in detail below with reference to drawings showing embodiments thereof.

(Embodiment 1)
An information processing device that generates an image for posting by extracting (clipping) an area with high visibility to a subject from an image of a photographed subject (object) will be described. FIG. 1 is a block diagram showing an example of the configuration of an information processing device. The information processing device 10 is a device capable of various information processing and sending and receiving information, such as a smartphone, a tablet terminal, a personal computer, a server computer, etc. The information processing device 10 may be configured by a virtual machine running on a large-scale computer, a cloud computing system, a quantum computer, etc., or may be configured by a dedicated terminal. When posting an image (photographed image) to a social networking service (SNS), for example, the information processing device 10 of this embodiment generates an image for posting by clipping an area with high visibility to a subject from the image. Note that the image to be processed is not limited to an image for posting to an SNS, and may be an image for any medium such as a book, magazine, weekly magazine, pamphlet, catalog, newspaper, menu, flyer, etc., and the medium may be a paper medium or a digital medium such as an electronic book. The image to be processed may be a photograph or an illustration.

The information processing device 10 includes a control unit 11, a memory unit 12, a communication unit 13, an input unit 14, a display unit 15, a camera 16, a reading unit 17, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 appropriately executes a control program 12P stored in the memory unit 12, thereby performing various information processing and control processing to be performed by the information processing device of the present disclosure.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 pre-stores a control program 12P executed by the control unit 11 and various data required for executing the control program 12P. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the control program 12P. The storage unit 12 further stores an image editing application program 12AP (hereinafter referred to as image editing app 12AP) for generating an image to post from an image.

The communication unit 13 has an interface for connecting to a network such as the Internet by wired or wireless communication, and transmits and receives information to and from other devices via the network. The input unit 14 accepts operation input by a user who operates the information processing device 10, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

The camera 16 is an imaging device having a lens and an imaging element, etc., and acquires image data of a subject image through the lens. The camera 16 captures images according to instructions from the control unit 11, acquires image data (still image) of, for example, one frame, and stores the acquired image data in the storage unit 12. The camera 16 may be configured to be built into the information processing device 10, or may be configured to be externally attached to the information processing device 10. In this case, the information processing device 10 has a connection unit that can connect an external camera or a camera communication unit for performing wired or wireless communication with the external camera, and acquires image data captured by the external camera through the connection unit or camera communication unit. The information processing device 10 of this embodiment does not need to have the camera 16, and may be configured to acquire image data captured by another information processing device or camera via a network or a portable storage medium 1a.

The reading unit 17 reads information stored in a portable storage medium 1a, which may include a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The control program 12P, the image editing application 12AP, and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 1a via the reading unit 17 and stored in the storage unit 12. The control program 12P, the image editing application 12AP, and various data stored in the storage unit 12 may also be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

The process in which the information processing device 10 of this embodiment generates an image for posting from an image (photographed image) is described below. FIG. 2 is a schematic diagram for explaining image editing processing. The information processing device 10 of this embodiment receives as input data the photographed image to be edited and the aspect ratio (horizontal length of the image: vertical length of the image) of the image for posting after editing, and generates an image for posting (output data) by extracting an area from the photographed image where the subject has good visibility and has the input (set) aspect ratio. Thus, an image for posting in which the subject is displayed in an easy-to-see state is generated from an image photographed by the user.

Figures 3 and 4 are flowcharts showing an example of an image editing process procedure, and Figures 5 to 7 are schematic diagrams for explaining the image editing process. The score calculation process shown in Figure 4 is the "score calculation process" during the image editing process shown in Figure 3. The following process is executed by the control unit 11 in accordance with the control program 12P and image editing application 12AP stored in the storage unit 12 of the information processing device 10. Part of the following process may be realized by a dedicated hardware circuit.

In the information processing device 10 of this embodiment, when a user posts an image of a subject (target object) on an SNS or the like, the user performs a predetermined operation via the input unit 14, specifies the aspect ratio of the captured image and the image to be posted, and issues an instruction to execute the process of generating the image to be posted. Note that, for example, if the aspect ratio of the image to be posted is set in advance according to the SNS to which the image is to be posted, the user only needs to specify the captured image. When the control unit 11 (image acquisition unit) of the information processing device 10 receives an instruction to execute the process of generating the image to be posted via the input unit 14, it acquires the specified captured image (S11). For example, if the captured image is stored in the storage unit 12, the control unit 11 reads out the captured image from the storage unit 12. The captured image is not limited to an image captured by the camera 16, but may be an image captured by a camera of another device and stored in the storage unit 12 via a network or a portable storage medium 1a. FIG. 5A shows an example of a captured image. When the aspect ratio of the image to be posted (display area of the image) is specified via the input unit 14, the control unit 11 (aspect ratio acquisition unit) acquires the aspect ratio of the specified image to be posted, and when the aspect ratio of the image to be posted is set for the SNS to which the image is to be posted, it acquires the aspect ratio of the SNS to which the image is to be posted. The following describes an example in which the aspect ratio of the image to be posted is 1:1.

Next, the control unit 11 performs object detection processing on the acquired captured image to detect a subject (target object) in the image (S12). Object detection in the image can be performed using a learning model such as R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, Mask R-CNN, YOLO (You only Look Once), or SSD (Single Shot Multibox Detector). Specifically, the control unit 11 inputs the captured image into the learning model described above, and detects the target object (subject) in the captured image based on the output information from the learning model. Note that since object detection processing using a learning model is publicly known, detailed explanations are omitted. The learning model described above may be incorporated in the image editing application 12AP or may be stored in the storage unit 12. In FIG. 5B, a bounding box indicating a subject (here, a dog) detected by object detection using a learning model is superimposed on the captured image shown in FIG. 5A with a dashed line. Object detection in the image may also be performed using a template matching technique. In this case, templates indicating the image features of the object to be detected are stored in advance in the storage unit 12, and the control unit 11 can detect whether the object is present depending on whether an area that matches any of the templates is present in the captured image.

The control unit 11 sets a clipping range to be extracted from the captured image based on the subject detected from the captured image (S13). For example, the control unit 11 sets the area of a bounding box (circumscribing rectangle) surrounding the subject detected from the captured image using the learning model as the clipping range, as shown by the dashed rectangle in FIG. 5B. This sets the area including the subject (target object) in the captured image as the clipping range. Note that the control unit 11 may set the area obtained by expanding the bounding box by a predetermined area (predetermined number of pixels) in the vertical and horizontal directions as the clipping range.

When users take images to post on social media, they often take the image so that one subject is in the center of the image. In this case, the control unit 11 detects one subject from the captured image, and sets a clipping range that includes the one subject. In addition, if the captured image includes multiple subjects, the control unit 11 detects multiple subjects from the captured image. In this case, the control unit 11 may set a circumscribing rectangular area that includes the multiple subjects as the clipping range.

The control unit 11 (comparison unit) compares the aspect ratio of the clipping range set in step S13 with the aspect ratio of the specified image for posting, and determines whether the two aspect ratios match (S14). Specifically, the control unit 11 counts the number of pixels in the left-right direction (horizontal direction) and the number of pixels in the up-down direction (vertical direction) of the image within the clipping range, and calculates the aspect ratio of the clipping range (number of pixels in the left-right direction: number of pixels in the up-down direction). The control unit 11 then determines whether the calculated aspect ratio matches the aspect ratio of the image for posting. If it is determined that the aspect ratios match (S14: YES), the control unit 11 (generation unit) proceeds to the process of step S29, and extracts the image (pixels) within the clipping range set in step S13 from the captured image acquired in step S11 to generate the image for posting (display image) (S29).

If it is determined that the aspect ratios do not match (S14: NO), the control unit 11 specifies the adjustment direction of the clipping range set in step S13 (S15). The control unit 11 specifies the adjustment direction in which the clipping range should be adjusted to match the aspect ratio of the clipping range to the aspect ratio of the image for posting. Figures 6 and 7 are explanatory diagrams of the adjustment direction of the clipping range, where Figure 6 shows the adjustment direction for a horizontally long clipping range and Figure 7 shows the adjustment direction for a vertically long clipping range. Figures 6A and 7A show the clipping range set in step S13 with a dashed rectangle. The clipping range shown in Figure 6A has an image size of 330 pixels in the horizontal direction and 270 pixels in the vertical direction (330 x 270 pixels), and the aspect ratio of this clipping range is 11:9. In order to match such a clipping range to an aspect ratio of 1:1, it is possible to expand it in the vertical direction or shrink it in the horizontal direction. Therefore, in this embodiment, as shown in FIG. 6B, the image size is expanded to 330×330 pixels by expanding 60 pixels upward, 30 pixels upward and downward, and 60 pixels downward, and the clipping range is adjusted (expanded) so that the aspect ratio is 1:1. When performing such expansion, if an area in the captured image in which no pixels exist is generated, black pixels may be added to the area in which no pixels exist, and an image of the expanded clipping range may be generated. Also, as shown in FIG. 6C, the left end side is reduced by 60 pixels, the left and right ends are reduced by 30 pixels each, and the right end side is reduced by 60 pixels, and the clipping range is adjusted (reduced) so that the image size is reduced to 270×270 pixels, and the aspect ratio is 1:1.

The clipping range shown in FIG. 7A has an image size of 270 pixels horizontally and 330 pixels vertically (270×330 pixels), and the aspect ratio of this clipping range is 9:11. In order to make such a clipping range match the aspect ratio of 1:1, it is possible to expand it left and right or shrink it up and down. Therefore, in this embodiment, as shown in FIG. 7B, the clipping range is expanded to an image size of 330×330 pixels by expanding 60 pixels leftward, 30 pixels leftward, and 60 pixels rightward, and adjusted (expanded) so that the aspect ratio is 1:1. As shown in FIG. 7C, the clipping range is reduced to an image size of 270×270 pixels by reducing the top end by 60 pixels, the top and bottom ends by 30 pixels, and the bottom end by 60 pixels, and adjusted (reduced) so that the aspect ratio is 1:1.

When the control unit 11 identifies the adjustment direction of the clipping range, it expands or reduces the clipping range according to the identified adjustment direction. This allows the control unit 11 (switching unit) to sequentially switch between areas that include the subject and have the aspect ratio of the image for posting (adjusted clipping range). The control unit 11 expands the clipping range, for example, upward or leftward (S16). Here, if the clipping range is horizontal, the control unit 11 expands it upward as shown on the left side of FIG. 6B, and if the clipping range is vertical, the control unit 11 expands it leftward as shown on the left side of FIG. 7B. Then, the control unit 11 performs a calculation process of a score relating to the relevance of the expanded clipping range to the subject based on an image cut out from the captured image according to the expanded clipping range (S17).

In the score calculation process shown in FIG. 4, the control unit 11 (proportion score calculation unit) calculates a score according to the proportion of the subject area to the image cut out from the captured image based on the expanded (adjusted) clipping range. Specifically, the control unit 11 (object detection unit) identifies the subject area within the adjusted clipping range based on the subject area detected from the captured image in step S12 (S41). In the clipping range shown on the left side of FIG. 5C, the area of the dog (subject area) shown in white on the right side of FIG. 5C is identified. The control unit 11 then calculates the number of pixels within the identified subject area (S42), and calculates the ratio of the number of pixels within the subject area to the number of pixels within the adjusted clipping range, thereby calculating a score related to the area of the subject area relative to the adjusted clipping range (S43). In the example shown in FIG. 5C, a subject area of 55,806 pixels is detected for a clipping range of 330×330 pixels, and 0.51 is calculated as the score related to the area. As a result, the larger the subject area within the clipping range, the higher the score related to the area is calculated.

Next, the control unit 11 (position score calculation unit) calculates a score according to the position of the subject area relative to the adjusted clipping range. Specifically, the control unit 11 calculates the coordinate value of the center (image center) of the adjusted clipping range and the coordinate value of the center (center of gravity) of the subject area within the clipping range identified in step S41 (S44). The coordinate value of each pixel within the clipping range is expressed, for example, by the number of pixels to the right from the origin (0, 0) and the number of pixels to the bottom from the origin, with the upper left corner of the clipping range being the origin. In the example shown in FIG. 5D, for a clipping range of 330 x 330 pixels, the coordinate value of the center (image center) of the clipping range is calculated as (165, 165), and the coordinate value of the center of the subject area is calculated as (207, 179). The center of the subject area may be represented by, for example, the average value of the coordinate values of all pixels in the subject area, the coordinate values of the horizontal center positions of the left and right pixels of the subject area, and the coordinate values of the vertical center positions of the top and bottom pixels of the subject area, or the average value of the coordinate values of each pixel on the contour of the subject area. Next, the control unit 11 calculates the length of a half diagonal in the clipping range after adjustment (S45). The half diagonal is a line segment between the image center of the clipping range and one of the four corners of the clipping range, and in the example shown in FIG. 5D, 233.3 is calculated as the length of the half angle line. In addition, the control unit 11 calculates the distance (center-to-center distance) between the image center of the clipping range and the center of the subject area based on the coordinate values of the image center of the clipping range calculated in step S44 and the coordinate values of the center of the subject area (S46). In the example shown in FIG. 5D, 44.3 is calculated as the center-to-center distance. The control unit 11 calculates a score for the position of the subject region relative to the adjusted clipping range based on the length of the semi-diagonal and the center-to-center distance (S47). For example, the control unit 11 calculates the position score by 1.0-(center-to-center distance)/(length of the semi-diagonal), and in the example shown in FIG. 5D, 0.81 is calculated as the position score. As a result, the closer the subject region is to the center of the clipping range, the higher the position score calculated.

The control unit 11 stores the area score calculated in step S43 and the position score calculated in step S47 in association with the adjusted clipping range (S48). For example, the control unit 11 stores the area score and the position score in association with information indicating the position of the adjusted clipping range for the captured image. Note that the position of the clipping range is represented, for example, by the coordinate values of the pixels at the four corners of the clipping range in the captured image, and the coordinate values of the pixels at the four corners are represented, for example, by the number of pixels to the right and the number of pixels to the bottom from the origin, with the top left corner of the captured image being the origin (0,0). The position of the clipping range may also be represented by the coordinate values of the top left pixel of the clipping range in the captured image and the image size of the clipping range.

The control unit 11 returns to the image editing process shown in FIG. 3 and performs the processes of steps S18 to S19 on the clipping range set in step S13. Specifically, the control unit 11 expands the clipping range set in step S13 in the vertical or horizontal direction (S18). Here, if the clipping range is horizontal, the control unit 11 expands it in the upward and downward directions as shown in the center of FIG. 6B, and if the clipping range is vertical, the control unit 11 expands it in the left and right directions as shown in the center of FIG. 7B. Then, the control unit 11 performs a score calculation process for the expanded clipping range based on an image cut out from the captured image according to the expanded clipping range (S19). The score calculation process here is the same as step S17 and is the process shown in FIG. 4. As a result, the area score and position score are calculated and stored for the clipping range expanded in step S18.

Then, the control unit 11 performs the processes of steps S20 to S21 on the clipping range set in step S13. Specifically, the control unit 11 expands the clipping range set in step S13 in a downward or rightward direction (S20). Here, if the clipping range is horizontal, the control unit 11 expands it in a downward direction as shown on the right side of FIG. 6B, and if the clipping range is vertical, the control unit 11 expands it in a rightward direction as shown on the right side of FIG. 7B. Then, the control unit 11 performs a score calculation process for the expanded clipping range based on the image within the expanded clipping range (S21). As a result, area-related scores and position-related scores are calculated and stored for the clipping range expanded in step S20 as well.

Similarly, the control unit 11 performs the processes of steps S22 to S27 on the clipping range set in step S13. In step S22, the control unit 11 reduces the left side or the top side of the clipping range. Here, if the clipping range is horizontal, the control unit 11 reduces the left side of the clipping range as shown in the left side of FIG. 6C, and if the clipping range is vertical, the control unit 11 reduces the top side of the clipping range as shown in the left side of FIG. 7C. In addition, in step S24, the control unit 11 reduces the left and right sides, or the top and bottom sides of the clipping range. Here, if the clipping range is horizontal, the control unit 11 reduces the left and right sides of the clipping range as shown in the center of FIG. 6C, and if the clipping range is vertical, the control unit 11 reduces the top and bottom sides of the clipping range as shown in the center of FIG. 7C. Furthermore, in step S26, the control unit 11 reduces the right or bottom side of the clipping range. Here, if the clipping range is horizontal, the control unit 11 reduces the right side of the clipping range as shown on the right side of Fig. 6C, and if the clipping range is vertical, the control unit 11 reduces the bottom side of the clipping range as shown on the right side of Fig. 7C. The control unit 11 performs a process of calculating scores for the reduced clipping ranges based on the images within the respective reduced clipping ranges (S23, S25, S27). As a result, area scores and position scores are calculated and stored for the clipping ranges reduced in steps S22, S24, and S26, respectively.

By the above-mentioned process, the clipping range set in step S13 is expanded or contracted as shown in FIG. 6B and FIG. 6C, or FIG. 7B and FIG. 7C, to generate clipping range candidates for generating an image to be posted. Then, for each clipping range candidate, a score for the size and position of the subject area is calculated. The control unit 11 (identification unit) identifies an optimal (appropriate) clipping range based on the score for each clipping range candidate calculated by the above-mentioned process (S28). For example, the control unit 11 may identify the clipping range with the highest score for area, the clipping range with the highest score for position, or the clipping range with the highest score for area and the highest score for position as the optimal clipping range. The control unit 11 may also weight each of the score for area and the score for position, calculate a total score taking both scores into account, and identify the clipping range with the highest total score as the optimal clipping range. The rules for identifying the optimal clipping range are set in advance and stored in the storage unit 12.

The control unit 11 extracts images (pixels) within the optimal clipping range identified in step S28 from the captured image acquired in step S11, generates an image to post (S29), and ends the process. Through the above-described process, an image to post that has the specified aspect ratio and in which the captured area of the subject is located closer to the center of the image and is larger in size can be generated from the captured image. This makes it possible to post an image that displays the subject in an easy-to-view position and size on SNS, etc. Furthermore, in this embodiment, an image to post in which the subject is easy to view is automatically generated from the captured image, reducing the workload of the user who edits the image.

In this embodiment, the clipping range candidate is generated by expanding and reducing the clipping range set based on the subject detected from the captured image as shown in FIG. 6B and FIG. 6C or FIG. 7B and FIG. 7C, but the present invention is not limited to this configuration. For example, the clipping range candidate shown in FIG. 6A may be expanded by 30 pixels upward and reduced by 30 pixels on the right side to generate a clipping range candidate with an image size of 300×300 pixels (aspect ratio of 1:1). The clipping range shown in FIG. 6A may be expanded by 15 pixels upward and downward and reduced by 15 pixels on the left and right sides to generate a clipping range candidate with an image size of 300×300 pixels (aspect ratio of 1:1). In this way, clipping range candidates can be generated by various methods. In addition, it is sufficient to generate an appropriate number of candidates, calculate the score for each candidate, and identify the optimal clipping range, taking into account the processing load due to the score calculation process performed for each candidate.

In this embodiment, an example has been described in which an optimal clipping range is automatically identified according to a predetermined rule based on the score for each candidate clipping range, and an image to be posted is generated, but this is not a limitation. For example, a configuration is possible in which multiple clipping range candidates with high scores are identified and presented to the user, and the user selects the optimal clipping range from the multiple candidates. Figure 8 is a flowchart showing another example of the image editing process procedure, and Figure 9 is a schematic diagram showing an example screen. The process shown in Figure 8 is obtained by adding steps S51 to S53 between steps S27 and S28 in the process shown in Figure 3. Explanation of the same steps as in Figure 3 will be omitted. Note that steps S11 to S25 in Figure 3 are not shown in Figure 8.

In the image editing process shown in FIG. 8, the control unit 11 selects multiple clipping range candidates with high scores based on the score calculated for each clipping range candidate after the process of step S27 (S51). For example, the control unit 11 may select a predetermined number of clipping ranges in descending order of area-related scores, a predetermined number of clipping ranges in descending order of position-related scores, or a predetermined number of clipping ranges in descending order of area-related scores and position-related scores. Here, too, the control unit 11 may weight each of the area-related scores and the position-related scores, calculate a total score taking both scores into account, and select a predetermined number of clipping ranges in descending order of total score. Rules for selecting clipping range candidates with high scores are also set in advance and stored in the storage unit 12.

The control unit 11 displays the selected multiple clipping range candidates, generates a selection screen for accepting the selection of the optimal (appropriate) clipping range from these candidates, and displays it on the display unit 15 (S52). FIG. 9 shows an example of the selection screen, and the screen shown in FIG. 9 displays the posting images generated based on the clipping ranges for the three clipping range candidates. The selection screen may display the calculated scores (area score and position score) in association with the posting images corresponding to each clipping range candidate. In this case, the score for each clipping range candidate can be presented to the user, and the user can evaluate each posting image based on the score. The selection screen is configured to accept the selection of one of the displayed posting images, and has a posting button for instructing posting with the selected posting image. The screen shown in FIG. 9 shows a state in which the posting image (clipping range) at the bottom left is selected. The user instructs posting with the selected posting image by selecting one of the posting images displayed on the selection screen and operating the posting button.

The control unit 11 determines whether a selection of any clipping range (image to post) has been accepted on the selection screen via the input unit 14 (S53), and if it determines that a selection has not been accepted (S53: NO), it continues to display the selection screen and waits. If it determines that a selection of any clipping range has been accepted (S53: YES), the control unit 11 identifies the selected clipping range as an optimal (appropriate) clipping range (S28) and generates an image to post based on the identified clipping range (S29).

In the above-described process, multiple appropriate candidates for the clipping range for generating an image to post from a captured image can be selected based on the subject's shooting position and shooting size, and presented to the user. The user can select any clipping range from multiple clipping range (image to post) candidates. This makes it possible for the user to select a preferred image to post from candidates for images to post that have a specified aspect ratio, have the subject's shooting area located in the center of the image, and are large in size, and post the image to post on SNS, etc.

(Embodiment 2)
An information processing device that considers the area of each part of a subject included in a clipping range when setting a clipping range for generating an image for posting from a photographed image of the subject (object) will be described. The information processing device of this embodiment has a configuration similar to that of the information processing device 10 of the first embodiment, so detailed description of the configuration will be omitted. In addition to the configuration of the first embodiment shown in FIG. 1, the information processing device 10 of this embodiment stores a dictionary DB (database) in the storage unit 12 in which information on each part of the object that is the subject is registered. Although not shown in the figure, the dictionary DB has terms such as dog's paws, dog's eyes, dog's mouth, dog's nose, etc. registered in advance for a dog.

In the information processing device 10 of this embodiment, the control unit 11 executes a process similar to the process shown in FIG. 3. Note that in the image editing process shown in FIG. 3, the score calculation process is slightly different from the process shown in FIG. 4. FIG. 10 is a flowchart showing an example of the score calculation process procedure of the second embodiment, and FIG. 11 is a schematic diagram for explaining the score calculation process. The process shown in FIG. 10 adds steps S61 to S62 before step S41 in the process shown in FIG. 4. Explanations of the same steps as in FIG. 4 will be omitted.

In the score calculation process of this embodiment, the control unit 11 calculates a score based on each part of the subject included in the clipping range (adjusted clipping range) after expansion or reduction in steps S16, S18, S20, S22, S24, and S26 in FIG. 3. Specifically, the control unit 11 (part detection unit) performs segmentation on the image within the clipping range and classifies (classifies) the area by part of the subject (S61). For example, in step S12 in FIG. 3, the control unit 11 detects the subject area from the captured image using Mask R-CNN and performs segmentation on the detected subject area to classify the area by part of the subject, and can identify the area of each part in the image within the clipping range based on the class classification result. In the clipping range shown on the left side of FIG. 11A, the area (part area) of each part of the dog shown on the right side of FIG. 11A other than black (background) is identified. In Figure 11A, the dog's paws, eyes, mouth, nose, neck, face, and torso are detected within the clipping range.

The control unit 11 calculates a score for the subject's body parts based on each body part area in the image within the clipping range (S62). For example, the control unit 11 determines whether each body part area included in the clipping range is a body part stored in the dictionary DB in association with the subject (object), adds 1 to the body parts stored in the dictionary DB, and subtracts 1 from the body parts not stored in the dictionary DB to calculate the score. In the example shown in FIG. 11A, the dog's paws, eyes, mouth, and nose are included in the clipping range, so a score of 4.0 is calculated for the body parts of the subject. In the example shown in FIG. 11B, only a portion of the dog's paws, mouth, and nose are included, and only the dog's eyes are included in the clipping range, so a score of 1.0 is calculated for the body parts of the subject. Note that the determination of whether each body part of the subject is only partially or completely included in the clipping range is made based on, for example, the images inside and outside the clipping range. For example, if the contour of the clipping range is on each body part area of the subject, it can be determined that this body part is only partially included in the clipping range. In the example shown in Figure 11C, in addition to the dog's paws, eyes, mouth, and nose, the lower right area of the image (the area surrounded by a closed curve in the image on the right side of Figure 11C) contains something other than the subject (the dog) (here, a cat), so a score of 3.0 is calculated for the subject's body parts. As a result, the more body parts of the subject that are included within the clipping range, and the fewer things other than the subject are included, the higher the score calculated for the subject's body parts.

Then, the control unit 11 performs the same processes as steps S41 to S48 shown in FIG. 4. As a result, for each candidate clipping range, a score for the part of the subject, a score for the area of the subject region, and a score for the position of the subject region are calculated. Therefore, in this embodiment, the control unit 11 stores the score for the part of the subject calculated in step S62, the score for the area calculated in step S43, and the score for the position calculated in step S47 in association with the adjusted clipping range (S48).

In this embodiment, in step S28 in FIG. 3, the control unit 11 identifies an optimal (appropriate) clipping range based on the score for the subject's part, the score for the area of the subject region, and the score for the position of the subject region, which are calculated for each candidate clipping range (S28). Here, the control unit 11 may identify the clipping range with the highest score for the subject's part, the clipping range with the highest score for the area, the clipping range with the highest score for the position, or the clipping range with the highest score for all three scores as the optimal clipping range. The control unit 11 may also weight each of the three scores, calculate an overall score taking into account the three scores, and identify the clipping range with the highest overall score as the optimal clipping range.

By the above-mentioned processing, the information processing device 10 of this embodiment can generate an image for posting from a captured image that has a specified aspect ratio, includes many parts of the subject, has the captured area of the subject located closer to the center of the image, and is larger in size. This makes it possible to post an image on SNS or the like in this embodiment, in which the subject can be displayed in an easy-to-see state. Also, in this embodiment, the modified examples shown in Figures 8 and 9 can be applied, and similar effects can be obtained when applied.

In this embodiment, the same effects as those of the first embodiment described above can be obtained. Furthermore, in this embodiment, it is possible to generate an image for posting that has a specified aspect ratio, includes more of the body parts set on the subject, has the subject's shooting area positioned closer to the center of the image, and is larger in size. This makes it possible to post an image on SNS or the like in which each body part of the subject is more easily visible. In this embodiment as well, it is possible to apply the modified examples described in each of the above-mentioned embodiments as appropriate.

(Embodiment 3)
The following describes an information processing device that extracts (clips) an area from an image that has high visibility of an object (subject) described in a text based on the image (image data) and the text (text data) associated with the image. The information processing device of this embodiment has a similar configuration to the information processing device 10 of the first embodiment, and therefore a detailed description of the configuration will be omitted.

The following describes the process in which the information processing device 10 of this embodiment lays out images and text in their respective layout frames to generate a page layout. FIG. 12 is a schematic diagram for explaining the layout process. The information processing device 10 of this embodiment receives as input data the image and text to be laid out, and layout data in which the layout frames in which the images and text should be placed are set. The text is text in which information on the contents of the image is described. In this embodiment, the layout objects are two images and one piece of text, but the number of images and text is not limited to these. The information processing device 10 of this embodiment extracts areas from each of the images to be laid out, in which the visibility of the subject according to the content of the text is good and which have the same aspect ratio as the aspect ratio of the assigned layout frame, to generate a layout image. As a result, a layout image is generated in which the subject according to the content described in the text is displayed in an easy-to-see state. The information processing device 10 also generates a page layout (output data) by arranging the generated layout images and text in their respective layout frames. Thus, a page layout is generated in which the images and text are arranged in a state in which the user can easily read them.

Figures 13 to 15 are flow charts showing an example of a layout processing procedure, and Figures 16 to 19 are schematic diagrams for explaining the layout processing. The score calculation processing shown in Figure 15 is the "score calculation processing" during the layout processing shown in Figure 14. The processing shown in Figures 13 to 14 is obtained by adding steps S71 to S75 before step S11 during the image editing processing shown in Figure 3, and adding steps S76 to S78 instead of step S29. The processing shown in Figure 15 is obtained by adding steps S81 to S82 before step S41 during the processing shown in Figure 4. In the processing shown in Figures 13 to 15, explanations of the same steps as in Figures 3 to 4 will be omitted.

In the information processing device 10 of this embodiment, when a user generates a page layout including images and text, the user performs a predetermined operation via the input unit 14 to specify the images and text to be laid out and the layout data, and issues an instruction to execute a page layout generation process. When the control unit 11 (text acquisition unit) of the information processing device 10 receives an instruction to execute a page layout generation process via the input unit 14, it acquires the specified text (S71). For example, if the text is stored in the storage unit 12, the control unit 11 reads the text from the storage unit 12. The text is not limited to text generated by a user's operation via the input unit 14, but may be text stored in the storage unit 12 from another device via a network or via a portable storage medium 1a.

The control unit 11 (occurrence frequency calculation unit) calculates the occurrence frequency of each word that appears in the acquired text (S72). For example, the control unit 11 extracts each word from the text using a method such as morphological analysis, and counts the number of occurrences of each word. The control unit 11 then calculates the occurrence frequency for each word using, for example, the following formula (1). The upper part of FIG. 16A shows an example of text, and the lower part of FIG. 16A shows a graph showing the occurrence frequency of each word included in the text. The horizontal axis of the graph at the lower part of FIG. 16A shows the words included in the text, and the vertical axis shows the occurrence frequency of each word. In the example shown in FIG. 16A, the occurrence frequency of "dog" is calculated as 0.7, and the occurrence frequencies of "owner", "grassland", and "water" are calculated as 0.1. In the example shown in FIG. 16B, the occurrence frequency of "dog" is calculated as 0.5, the occurrence frequency of "cat" is calculated as 0.3, and the occurrence frequency of "owner" and "water" is calculated as 0.1.

Frequency of occurrence of a word = number of times a word occurs / total number of times all words occur … (1)

The control unit 11 (object identification unit) identifies the object (subject) in the image indicated by the text based on the frequency of occurrence of each word in the text (S73). For example, the control unit 11 identifies words whose frequency of occurrence is equal to or greater than a preset threshold (e.g., 0.28), and identifies the object indicated by the identified word. In the example shown in FIG. 16A, the only word (object) whose frequency of occurrence is equal to or greater than the threshold is "dog," and the control unit 11 identifies "dog." In the example shown in FIG. 16B, the words (objects) whose frequency of occurrence is equal to or greater than the threshold are "dog" and "cat," and the control unit 11 identifies "dog" and "cat."

The control unit 11 also generates terms related to the image indicated by the text from each word appearing in the text (S74). For example, the control unit 11 generates terms by combining each word appearing in the text while taking into account the content of each word using a technique such as dependency analysis. In the example text shown in FIG. 16A, terms such as "dog's paws," "dog's eyes," "dog's mouth," "dog's nose," "owner," "grassland," and "water" are generated as shown in FIG. 17. Note that the terms may be generated using, for example, the dictionary DB used by the information processing device 10 of embodiment 2.

The control unit 11 then associates each of the generated terms with a degree of relevance based on the frequency of occurrence of each word (S75). In the example shown in FIG. 16A, the frequency of occurrence of the word "dog" is 0.7, so a degree of relevance of 0.7 is associated with dog-related terms such as "dog paws," "dog eyes," "dog mouth," and "dog nose." In addition, the frequency of occurrence of the words "owner," "grassland," and "water" is each 0.1, so a degree of relevance of 0.1 is associated with these terms.

After that, the control unit 11 performs the same processes as steps S11 to S28 shown in FIG. 3. In this embodiment, in step S11, the control unit 11 acquires an image of the specified layout target. In step S12, the control unit 11 (detection unit) performs object detection processing on the acquired image to detect the object (subject) identified in step S73. In step S13, the control unit 11 sets a clipping range including the object detected in step S12 for the image of the layout target. As a result, in this embodiment, the area of the object related to the content of the text can be set as the clipping range. In this embodiment, too, the area including the object in the image is set as the clipping range, as shown by the dashed rectangle in FIG. 5B. In addition, as shown in FIG. 18A, if multiple objects are identified in step S73, the control unit 11 sets the area of a circumscribed rectangle including multiple objects as the clipping range, as shown in FIG. 18B. In this case, the area including multiple objects described in the text can be set as the clipping range. Also, in step S14, the control unit 11 compares the aspect ratio of the clipping range set in step S13 with the aspect ratio of the layout frame specified for the image to be laid out here.

In the score calculation process of this embodiment shown in FIG. 15, the control unit 11 calculates a score related to the relevance of the clipping range after it has been expanded or reduced (adjusted) in steps S16, S18, S20, S22, S24, and S26 in FIG. 14 to the text to be laid out. Specifically, the control unit 11 performs segmentation on the image within the clipping range, and classifies (classifies) the regions by part of the object (S81). The process of step S81 is the same as the process of step S61 in FIG. 10 described in embodiment 2.

Then, the control unit 11 (relevance calculation unit) calculates a score for the relevance between the image within the clipping range and the text based on each part area in the image within the clipping range (S82). In the example shown in FIG. 19A, the dog's paws, eyes, mouth, and nose are included in the clipping range, and a relevance of 0.7 is associated with the terms for these parts, so a score of 2.8 is calculated for the relevance between the image within the clipping range and the text shown in FIG. 19A. In the example shown in FIG. 19B, only a portion of the dog's paws, mouth, and nose are included in the clipping range, and only the dog's eyes are included in the clipping range, so a relevance of 0.7 is associated with the term "dog's eyes", so a score of 0.7 is calculated for the relevance between the image within the clipping range and the text shown in FIG. 19B. In the example shown in FIG. 19C, in addition to the dog's paws, eyes, mouth, and nose, the lower right area of the image (the area surrounded by a closed curve in the image on the right side of FIG. 19C) contains things other than the target object (dog) (here, a cat), and since a relevance score of -1.0 is associated with terms other than the target object, a score of 1.8 is calculated for the relevance between the image and text within the clipping range shown in FIG. 19C. As a result, the more terms generated from the text that are included within the clipping range, and the fewer things other than the target object are included, the higher the score calculated for the relevance between the image and text within the clipping range.

Then, the control unit 11 performs the same processes as steps S41 to S48 shown in FIG. 4. As a result, for each candidate clipping range, a score related to the relevance between the image and text within the clipping range, a score related to the area of the subject region (object region), and a score related to the position of the subject region are calculated. Therefore, in this embodiment, the control unit 11 stores the score related to the relevance with the text calculated in step S82, the score related to the area calculated in step S43, and the score related to the position calculated in step S47 in association with the adjusted clipping range (S48).

In this embodiment, in step S28 in FIG. 14, the control unit 11 identifies an optimal (appropriate) clipping range based on the score for relevance to the text, the score for the area of the subject region, and the score for the position of the subject region, which are calculated for each candidate clipping range (S28). Here, the control unit 11 may identify the clipping range with the highest score for relevance to the text, the clipping range with the highest score for area, the clipping range with the highest score for position, or the clipping range with the highest score for all three scores as the optimal clipping range. The control unit 11 may also weight each of the three scores, calculate an overall score taking into account the three scores, and identify the clipping range with the highest overall score as the optimal clipping range.

In this embodiment, after the processing of step S28, an image (pixels) within the optimal clipping range identified in step S28 is extracted from the image to be laid out obtained in step S11 to generate an image for layout (S76). The control unit 11 stores the generated image for layout in the storage unit 12. The control unit 11 then determines whether or not there is an image that has not been edited (S77), and if it determines that there is an image that has not been edited (S77: YES), the process returns to step S11, and the unprocessed image for layout is obtained (S11), and the processing of steps S12 to S28 and S76 is performed on the obtained image. This allows a layout image to be generated from each of the images to be laid out.

If it is determined that there are no unprocessed images (S77: NO), the control unit 11 generates a page layout by arranging the text to be laid out and the layout images generated in step S76 based on the specified layout data (S78). Specifically, the control unit 11 generates a page layout by fitting each of the layout images into the corresponding layout frame and fitting the text to be laid out into the corresponding layout frame. When fitting the text into a specified layout frame, the control unit 11 may change the character size and insert line breaks as appropriate.

The above-described process makes it possible to generate a layout image from the image to be laid out, which has the aspect ratio specified in the layout data, and in which the area of the object (subject) related to the content of the text is located closer to the center of the image and is larger in size. This makes it possible to generate an image in which the object mentioned in the text is displayed in a position and size that makes it easy to see, and by using such an image in various media, it is possible to generate page layouts that are highly visible and easy to read.

In this embodiment, the same effects as those of the above-mentioned embodiments can be obtained. In addition, in this embodiment, when generating a page layout including images and text, a layout image with a clipping range that matches the content of the text can be generated from the image. This makes it possible to use images in which objects suitable for the content of the text are displayed more easily in various media. In addition, in this embodiment, a layout image in which the objects are easily visible is automatically generated from the image to be laid out, thereby reducing the workload of the user who edits the images. In this embodiment, the modified examples described as appropriate in each of the above-mentioned embodiments can also be applied.

In each of the above-described embodiments, the information processing device 10 may be configured to perform the process of generating an image for posting or an image for layout from an image on a specified server connected to a network. In this case, the control unit 11 of the information processing device 10 may transmit the image to be processed, or the image and text to be laid out, to the specified server via the network, and obtain the image for posting or the image for layout generated by the specified server. The server in such a configuration may be realized using a server computer or a personal computer, may be realized using multiple virtual machines provided in a single server, or may be realized using a cloud server.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the claims.

REFERENCE SIGNS LIST 10 Information processing device 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Display unit 16 Camera

Claims (9)

an image acquisition unit for acquiring an image including an object;
an aspect ratio acquisition unit that acquires an aspect ratio of a display area that displays the object;
a comparison unit that detects the object from the image and compares an aspect ratio of a region of the detected object with an aspect ratio of the display region;
a switching unit that, when an aspect ratio of the object region and an aspect ratio of the display region are different, sequentially switches between the object regions having the aspect ratio of the display region;
and a specifying unit that specifies the area to be switched based on a degree of association between an image cut out based on the switched area and the object. an object detection unit that detects an area of the object from an image cut out based on the switched area;
a ratio score calculation unit that calculates a score according to a ratio of an area of the detected object to the cut-out image,
The information processing device according to claim 1 , wherein the specifying unit specifies the area to be switched based on a score corresponding to the calculated ratio. an object detection unit that detects an area of the object from an image cut out based on the switched area;
a position score calculation unit that calculates a score according to a position of the detected object region for the cut-out image,
The information processing device according to claim 1 , wherein the specifying unit specifies the area to be switched based on the calculated score according to the position. a part detection unit that detects the area of each part of the object from the image cut out based on the switched area;
The information processing device according to claim 1 , wherein the specifying unit specifies the area to be switched based on an area of each part of the detected object. a text acquisition unit that acquires text data corresponding to an image including an object;
an object identification unit that identifies an object indicated by a word appearing in the acquired text data;
a detection unit that detects an object appearing in the text data from the acquired image,
The information processing device according to claim 1 , wherein the comparison unit compares an aspect ratio of the region of the object detected by the detection unit with an aspect ratio of the display region. an occurrence frequency calculation unit that calculates an occurrence frequency of an object that appears in the text data;
a relevance calculation unit that calculates a relevance between an object appearing in the text data and an image cut out based on the switched region based on an appearance frequency of the object and the object included in the image cut out based on the switched region,
The information processing device according to claim 5 , wherein the specifying unit specifies the area to be switched based on the calculated relevance. The information processing device according to claim 1 , further comprising: a generation unit configured to generate a display image by cutting out the region of the object from the acquired image when the aspect ratio of the region of the object and the aspect ratio of the display region are the same. Acquire an image including the object;
Obtaining an aspect ratio of a display area for displaying the object;
Detecting the object from the image, and comparing an aspect ratio of the region of the detected object with an aspect ratio of the display region;
when the aspect ratio of the object region and the aspect ratio of the display region are different, sequentially switching the object region having the aspect ratio of the display region;
A program for causing a computer to execute a process of identifying a region to be switched based on a degree of association between an image cut out based on the switched region and the object. Acquire an image including the object;
Obtaining an aspect ratio of a display area for displaying the object;
Detecting the object from the image, and comparing an aspect ratio of the region of the detected object with an aspect ratio of the display region;
when the aspect ratio of the object region and the aspect ratio of the display region are different, sequentially switching the object region having the aspect ratio of the display region;
The information processing method includes a process of identifying an area to be switched based on a degree of association between the object and an image cut out based on the switched area.

Images