JP2019148649A - Image processing apparatus, control method, and program - Google Patents

Abstract

To provide an image processing apparatus, a control method, and a program capable of surely displaying correlation between objects contained in electronic data independently of the displayed state of targeted electronic data.SOLUTION: A material display terminal 102, when displaying a part or whole of a page image containing a plurality of subregions on a scenario edit screen 1301, takes displayed subregions among the plurality of subregions as targeted subregions (main body objects) and determines whether or not to display subregions related to the main body objects (related objects). When the related objects are displayed, the main body objects and the related objects are displayed so as to be associated with each other. When the related objects are not displayed, alternative objects which serve as alternatives to the related objects are created and displayed, and the main body objects and the alternative objects are displayed so as to be associated with each other.SELECTED DRAWING: Figure 12A

Description

  The present invention relates to an image processing device, a control method, and a program, and in particular, when displaying data having a plurality of objects on a screen, an image processing device, a control method, and an information processing device that display information representing the relationship between objects. Regarding the program.

  Conventionally, in the case of business negotiations for products that do not have a physical entity (insurance products, financial products, travel products, etc.) or products that do not have the actual thing at hand, the product sales representatives face the customer who is the buyer. In the scene of explaining the above, a paper medium such as a pamphlet has been used. In recent years, with the spread of portable electronic data display terminals such as tablet PCs, which have excellent portability but large screens and excellent visibility, it is possible to use pamphlets that have been converted into electronic data even in face-to-face business meetings as described above. is increasing. However, in a portable electronic data display terminal, if portability is emphasized, the screen size cannot be as large as a paper medium pamphlet. For this reason, the visibility when displaying 100% of a pamphlet converted into electronic data is inferior to that of a paper-based pamphlet. Therefore, when a portable electronic data display terminal is used, a partially enlarged display of a pamphlet converted into electronic data is usually performed.

  Patent Document 1 discloses an image processing apparatus that recognizes an object included in an image, divides the image, and performs display while enlarging each divided partial area. If this image processing apparatus is used, an image of a pamphlet that has been output to a conventional paper medium is used as it is, and the product is explained on the portable electronic data display terminal while being enlarged for each object in the image. It is possible.

  Patent Document 2 discloses a display technique for enhancing the convenience of document browsing in a portable electronic data display terminal. Specifically, partial areas in the material are analyzed, and the relationship between the partial areas is acquired. Then, when one of the partial areas is displayed on the screen, information on the partial area that is not displayed on the screen is displayed on the screen. Thereby, it becomes possible for the user to grasp information about a partial area that is not displayed on the screen.

Patent No. 5984439 Japanese Patent No. 4765831

  However, in Patent Document 1, when one of the partial areas is displayed in an enlarged manner, information on the other partial areas is not displayed on the screen, so it is difficult to grasp the relationship between the entire image and the enlarged partial area.

  On the other hand, in Patent Document 2, the relationship between partial areas is presented only when only one partial area is displayed and no other partial areas are displayed on the screen. However, there are cases where the relevance between the partial areas is necessary when the material is displayed in a bird's-eye view. For example, when taking a display form in which partial areas are displayed in order, it is preferable that the order in which partial areas are displayed can be confirmed in a bird's-eye view. In addition, the instructor may enlarge and display an arbitrary area including all or part of two or more partial areas. In such a case, it is desirable to be able to confirm the relationship between the partial area being displayed and other partial areas not displayed on the screen.

  An object of the present invention is to provide an image processing apparatus, a control method, and a program that can always display the relationship between objects included in electronic data without depending on the display state of the target electronic data.

  An image processing apparatus according to claim 1 of the present invention is a display that displays a content display unit that displays part or all of a content including a plurality of display units on a screen and a display unit associated with the content. An image processing apparatus comprising: a processing unit, wherein the display processing unit includes: a first determination unit that determines whether each of the plurality of display units is displayed on the screen by the content display unit; Second determination means for determining whether or not a display unit related to a target display unit among the display units determined to be displayed on the screen by the first determination means is displayed on the screen; Display that associates the target display unit with the related display unit when the second determining means determines that the related display unit is displayed on the screen. When the first display means to perform and the second determination means determine that the relevant display unit is not displayed on the screen, a substitute object is generated that substitutes the relevant display unit. And a second display means for displaying the target display unit and the alternative object in association with each other.

  According to the present invention, it is possible to always display the relationship between objects included in electronic data without depending on the display state of the target electronic data.

1 is a block diagram illustrating a configuration of an information processing system including a material display terminal as an image processing apparatus according to Embodiment 1. FIG. It is a block diagram which shows the hardware constitutions of the control apparatus which a document display terminal has. FIG. 2 is a functional block diagram of each of a material creation terminal, a material display terminal, a material data management server, and a page analysis server in FIG. 1. It is a figure for demonstrating the content transmitted to the page analysis server from a material creation terminal, and the page image contained in it. It is a flowchart of the structure analysis process of the page image which the page structure analysis part of a page analysis server performs. FIG. 6 is a diagram for explaining a case where the structure analysis process of FIG. 5 is performed on the page image of the first page of the content of FIG. 4. It is a figure for demonstrating the partial area management table produced in the partial area management part of a page analysis server. It is a figure for demonstrating the management method of the material data by the material data management part of a material data management server when scenario data exist. It is an external view of the touchscreen display of a document display terminal. It is a flowchart which shows the procedure of the document data display process which the document data display process part which comprises the display part of a document display terminal performs. It is a figure which shows the example of a screen display at the time of the document data display process in step S1004 of FIG. It is a flowchart which shows the procedure of the related information display process of step S1006 of FIG. FIG. 12D is a diagram showing a processing target list to which information is sequentially added in step S1224 of FIG. 12A. It is a figure which shows an example of the scenario edit screen displayed on a touchscreen display in step S1006-S1009 of FIG. It is a continuation of FIG. 13A. It is a figure which shows the example of the process target list | wrist obtained by the object display determination process in FIG.12A step S1202.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the present invention according to the claims, and all combinations of features described in the following embodiments are not necessarily essential to the solution means of the present invention. Absent.

(Configuration of information processing system)
FIG. 1 is a block diagram illustrating a configuration of an information processing system 100 including a material display terminal as an image processing apparatus according to the first embodiment.

  In FIG. 1, a material creation terminal 101, a material display terminal 102, a material data management server 103, and a page analysis server 104 are connected to a network 120 as system components. The form of the network 120 is not particularly limited as long as it is a communication network capable of transmitting and receiving data between the system components 101 to 104. Examples of the network 120 include the Internet, a wired LAN, a wireless LAN, and combinations thereof. Each system component 101 to 104 is connected to the network 120 directly or via a connection device such as a router, a gateway, or a proxy server (not shown). Of course, the configuration of the network 120 and the connected devices of each element are not limited to these. In the present embodiment, each of the material creation terminal 101 and the material display terminal 102 is connected to the network 120 one by one, but a plurality of each may be connected to the network 120. The material creation terminal 101 is operated by a material creator 141, and the material display terminal 102 is operated by a material explainer 142. The document creator 141 and the document explainer 142 may be different users or the same user.

  FIG. 2 is a block diagram illustrating a hardware configuration of the control device 200 included in the material display terminal 102. Note that the material creation terminal 101, the material data management server 103, and the page analysis server 104 also have the same control device, but since they are redundant, the same components are denoted by the same reference numerals and description thereof is omitted.

  The control device 200 includes a CPU 201, a ROM 202, a RAM 203, a storage device 204, a network I / F 205, a display I / F 206, an operation input I / F 207, and an external I / O 208.

  FIG. 3 is a functional block diagram showing functional configurations of the material creation terminal 101, the material display terminal 102, the material data management server 103, and the page analysis server 104 as system components in FIG. The function of each block shown in FIG. 3 is executed by executing a program by the CPU 201 of the control device 200 included in each of the system components 101 to 104. A program corresponding to each block is stored in the storage device 204 included in each of the system components 101 to 104.

  FIG. 3A is a functional block diagram of the material creation terminal 101. The control unit 301 performs control of the control device 200 included in the material creation terminal 101 and devices connected to the control device 200, and control of processing execution of each functional block. The communication unit 302 transmits / receives data to / from other devices on the network 120 via the network I / F 205 included in the material creation terminal 101. An external display (not shown) such as a liquid crystal display is connected to the display I / F 206 of the control device 200 included in the document creation terminal 101. A display unit 303 displays a user interface on an external display connected via the display I / F 206. Operation input devices such as a keyboard, a mouse, and a touch panel are connected to the operation input I / F 207 of the control device 200 included in the document creation terminal 101. The operation unit 304 receives an input from an operation input device connected via the operation input I / F 207 and operates a user interface displayed on the display unit 303. The document data generation unit 305 transmits the document data designated via the user interface displayed on the display unit 303 to the page analysis server 104. Thereby, the information processing system 100 starts generating the material data regarding the document data. The partial area management unit 306 performs management of partial area data, which will be described later, in cooperation with the page analysis server 104. The scenario editing unit 307 displays a user interface for editing a scenario on the external display via the display unit 303, and edits the scenario according to the operation received via the operation unit 304. The user interface displayed by the scenario editing unit 307 will be described later.

  FIG. 3B is a functional block diagram of the material display terminal 102. The control unit 311 performs control of the control device 200 included in the material display terminal 102 and devices connected to the control device 200, and control of processing execution of each functional block. The communication unit 312 transmits / receives data to / from other devices on the network 120 via the network I / F 205 included in the material display terminal 102. A liquid crystal display (not shown) is connected to the display I / F 206 of the control device 200 included in the document display terminal 102. The display unit 313 displays a user interface on a liquid crystal display connected via the display I / F 206. A touch panel is connected to the operation input I / F 207 of the control device 200 included in the document display terminal 102. The operation unit 314 receives an input from a touch panel connected via the operation input I / F 207 and operates a user interface displayed on the display unit 313. The appearance and functions of a touch panel display including a liquid crystal display and a touch panel connected to the document display terminal 102 will be described later with reference to FIG.

  The document data display processing unit 315 (content display means) performs display processing of document data, and the related information display processing unit 316 displays related information according to the display state of the document data, and according to the user's operation, Confirm and edit related information. The processing of the document data display processing unit 315 and the related information display processing unit 316 will be described later.

  FIG. 3C is a functional block diagram of the material data management server 103. The control unit 331 performs control of the control device 200 included in the document data management server 103 and devices connected to the control device 200, and control of processing execution of each functional block. The communication unit 332 transmits / receives data to / from other devices on the network 120 via the network I / F 205 included in the material data management server 103. The storage device 204 of the control device 200 included in the document data management server 103 has a large capacity, and the document data management unit 333 manages a large amount of document data using the storage device 204. Details of the material data managed by the material data management unit 333 will be described later.

  FIG. 3D is a functional block diagram of the page analysis server 104. The control unit 321 performs control of the control device 200 included in the page analysis server 104 and devices connected to the control device 200, and control of processing execution of each functional block. The communication unit 322 transmits / receives data to / from other devices on the network 120 via the network I / F 205 included in the page analysis server 104. The page structure analysis unit 324 analyzes the structure of the input document data and generates partial area data described later. The processing of the page structure analysis unit 324 will be described later with reference to FIGS. The partial area management unit 325 manages partial area data described later.

(Content and page image)
FIG. 4 is a diagram for explaining contents transmitted from the material creation terminal 101 to the page analysis server 104 and page images included in the contents.

  As shown in FIG. 4, the content 400 composed of document data is composed of page images 401 to 40N composed of N pages.

  The content 400 is a document file handled by a general information processing apparatus, and the page images 401 to 40N are each page of the document file. For example, the page image format is a raster image such as a JPEG or TIFF file.

(Page structure analysis process)
(Page image and structural analysis example)
FIG. 5 is a flowchart showing the procedure of the page image structure analysis process executed by the page structure analysis unit 324 of the page analysis server 104. This page image structure analysis process is a process of automatically analyzing a plurality of partial areas (display units) in a page image in accordance with the document component of the page image.

  The page images to be processed are the page images 401 to 40N included in the content 400 of FIG.

  Hereinafter, the case where the structure analysis process of FIG. 5 is performed on the page image 401 of the first page of the content 400 will be described with reference to FIG.

  As shown in FIG. 6A, the page image 401 includes a background 600, texts 601 to 604, 606, 607, 609, 615, 616, figures 605, 608, 611 to 613, and a table 610 as document components. , 614 are laid out.

  The page structure analysis unit 324 sequentially executes the page image structure analysis process on all pages of the content 400. That is, the process is started from the page image 401 of the first page, and the process for the page image 402 of the next page is executed after the first page is completed. Although not shown in FIG. 6, this is repeated up to the page image 40N of the final page.

  Each process of FIG. 5 is implement | achieved by running a program by CPU201 in the control apparatus 200 which the page analysis server 104 has.

  In step S501, the page structure analysis unit 324 sets a page number to be processed as a variable i. The variable i is an increment index for executing processing on all page images of the content 400 to be processed. Initially, the first page number 1 is set.

  In step S <b> 502, the page structure analysis unit 324 reads the page image of the i-th page of the content 400 stored in the storage device 204 included in the page analysis server 104.

  In step S503, the page structure analysis unit 324 recognizes a partial area for the read page image according to the document component.

  For example, FIG. 6B shows a result of the page structure analysis unit 324 executing step S503 on the page image 401 shown in FIG. 6A and automatically analyzing the partial areas according to the document components. Each automatically analyzed partial area is indicated by a rectangular dotted line in FIG.

  In FIGS. 6A and 6B, the background 600 is automatically analyzed as a partial area 620. Similarly, the texts 601 to 604, 606, 607, 609, 615 and 616 are automatically analyzed as partial areas 621 to 624, 626, 627, 629, 625 and 626. Further, the figures 605, 608, 611 to 613 are automatically analyzed as partial areas 625, 628, 631 to 633, and the tables 610 and 614 are automatically analyzed as partial areas 630 and 634.

  In step S504, the page structure analysis unit 324 recognizes the coordinates and the height of each partial area.

  For example, FIG. 6C shows a result of the page structure analysis unit 324 recognizing the coordinates and the height of the partial region 631 among the plurality of partial regions 620 to 636 automatically recognized in FIG.

  The coordinates recognized in step S504 are the upper left coordinates of each partial area, and are expressed in a coordinate system having the origin at the upper left of the partial area 620 of the background area attribute (the area in the same range as the entire page image 401). Therefore, as the coordinates of the partial region 631, the distance X31 in the X-axis direction from the origin to the upper left coordinate of the partial region 631, and the distance Y31 in the Y-axis direction from the origin to the upper left coordinate of the partial region 631 are recognized. .

  Further, as the height of the partial region 631, the length W31 in the X-axis direction and the length H31 in the Y-axis direction of the partial region 631 are recognized.

  In step S505, the page structure analysis unit 324 performs OCR processing, and determines whether there is a character in each partial area recognized in step S503. If there is a character (YES in step S505), the process proceeds to step S506. If there is no character (NO in step S505), the process proceeds to step S508.

  In step S506, the page structure analysis unit 324 acquires a character size from the OCR processing result.

  In step S507, the page structure analysis unit 324 recognizes the character direction from the OCR processing result. The character direction is the writing direction and indicates whether the character is written horizontally or vertically.

  In step S508, the page structure analysis unit 324 recognizes the region attribute of each partial region. The area attribute indicates an attribute corresponding to a document component, and is an attribute such as a title, text, header, footer, table, figure, and background.

  In step S509, the page structure analysis unit 324 determines the display order of the partial areas. The display order is determined according to the character direction, coordinates, and height of each partial area recognized in steps S504 and S507. Basically, when the character direction is horizontal writing, the display order is determined from the partial area located at the upper left in the page image to the partial area located at the lower right. When the character direction is vertical writing, the display order is determined from the partial area located at the upper right to the partial area located at the lower left among the partial areas having characters in the page image.

  In the example of FIG. 6B, the character direction is horizontal. Therefore, the part located in the upper left of the partial areas having characters in the page image 401, such as the display order of the partial area 621 is 1, the display order of the partial area 622 is 2, and the display order of the partial area 623 is 3. The display order is determined from the area to the partial area located at the lower right. Hereinafter, the display order in the partial areas 624 to 636 is determined to be 4 to 16, respectively. The partial area 620 as the background includes the coordinates of the origin, so the display order is determined to be zero. That is, in the display process of the material data display processing unit 315 described later, the partial area whose display order is determined to be 0 in step S509 is not enlarged for each partial area.

  Note that the display order determined in step S509 is the display order when there is no scenario data described later. It can also be used as an initial value of the display order on the scenario data.

  In step S510, the page structure analysis unit 324 stores information regarding each partial area obtained by executing steps S503 to S509 as partial area data. In practice, when the page structure analysis unit 324 transmits the partial region data to the partial region management unit 325, the partial region management unit 325 stores the transmitted partial region data and also manages the partial region data. Create an area management table. This partial area management table will be described in detail with reference to FIG.

  In step S511, the page structure analysis unit 324 increments the variable i by one to process the next page.

  In step S512, the page structure analysis unit 324 determines whether the variable i has exceeded the number of pages of the content 400 being processed. If the variable i does not exceed the number of pages, the process returns to step S502, and the processing of steps S502 to S510 is repeated for the page image of the next page, and then steps S511 and S512 are executed.

  If the variable i exceeds the number of pages of the content 400 being processed, the processing for all the page images of the content to be processed has been completed, and thus this processing ends.

(Management of partial area data)
FIG. 7 is a diagram for explaining a partial area management table created by the partial area management unit 325 of the page analysis server 104.

  The partial area management unit 325 creates a partial area management table so as to hold and manage the partial area data transmitted from the page structure analysis unit 324 in step S510 of FIG. Thereafter, the partial area management unit 325 stores the partial area data and the partial area management table in the RAM 203 or the storage device 204 of the control device 200 included in the page analysis server 104. The created partial area management table 700 is also transmitted to the material data management server 103.

  As shown in FIG. 7, partial area data items 701 to 709 are arranged in the row direction of the partial area management table 700. Note that the partial area 721 in FIG. 7 is not actually an item managed in the partial area management table, but is described in order to clarify the association with each partial area in FIG.

  First, the row direction of the partial area management table 700 will be described.

  The items of the partial area data are page number 701, partial area number 702, coordinates 703, width 704, area attribute 705, display setting 706, character direction 707, character size 708, and display order 709.

  A page number 701 is a page number of a page image including a partial area.

  The partial area number 702 is a number for identifying the partial area. As shown in FIG. 7, the display order of each partial area in the page image of one page determined in step S509 of FIG. 5 is displayed as this number.

  Each partial area can be uniquely identified in the content by the page number 701 and the partial area number 702.

  A coordinate 703 is the upper left coordinate of each partial area recognized in step S504 in FIG. The width height 704 is the width and height of each partial area recognized in step S504. As described above, for example, in the partial region 631 shown in FIG. 6C, (X31, Y31) is managed as information on the coordinates 703, and (W31, H31) is managed as information on the width height 704.

  The area attribute 705 is an area attribute of each partial area recognized in step S508 in FIG. Specifically, as the information of the region attribute 705, one of attributes such as a title, text, header, footer, table, figure, and background is managed.

  A display setting 706 indicates whether to display or hide the partial area in the presentation process. The partial area displayed in the presentation process is set to “display”. The partial area that is hidden in the presentation process is set as “hidden”.

  The character direction 707 is the character direction (writing direction) in the partial area including the character, recognized in step S507 of FIG. If the text direction is horizontal writing, set it to “Horizontal”. If the text direction is vertical writing, set “vertical”.

  The character size 708 is a character size in a partial area including characters.

  The display order 709 is the display order of the partial areas in the page image of one page determined in step S509 in FIG. When scenario data to be described later is created, the display order 709 shown in FIG. 7 is changed according to the scenario data.

  Next, the column direction of the partial area management table 700 will be described.

  One row of the partial area management table 700 corresponds to one partial area data. For example, in the page image 401 of FIG. 6B, since there are 17 partial areas 620 to 636, the partial area data with the page number 701 of 1 is managed in 17 columns in the partial area management table 700 of FIG. Is done.

  Similarly, the partial area management table 700 manages all partial area data included in all page images 401 to 40N of the content 400.

  The partial area 721 indicates each partial area extracted by the page analysis in FIG.

(Management of content and partial area management table)
The content 400 and the partial area management table 700 described above are transmitted from the page analysis server 104 to the material data management server 103, and stored and managed on the storage device 204 of the material data management server 103 by the material data management unit 333.

  Hereinafter, a method for managing material data by the material data management unit 333 of the material data management server 103 when scenario data edited by the scenario editing unit 307 of the material creation terminal 101 exists will be described with reference to FIG.

  FIG. 8A is a schematic diagram of a material data management method by the material data management unit 333. The material data management unit 333 has a data ID management area 801 on the storage device 204 of the material data management server 103, and uses an unused ID on the data ID management region 801 when storing data on the storage device 204. Assign a unique ID to the saved data. For example, the material data management unit 333 assigns a content ID 802 when saving the content 400, and a partial area management table ID 803 when saving the partial area management table 700. Further, when saving the partial area management table 700, the material data management unit 333 saves the content ID 805 of the content 400 transmitted from the page structure analysis unit 324 at the time of saving as a header of the partial area management table 700.

  The material data management unit 333 has a data name management area 806 on the storage device 204 of the material data management server 103. A data name management area 806 is an area in which a data name consisting of a character string is stored for an ID in the data ID management area 801. The material data management unit 333 associates the data name with the ID and manages the data having the data name so that the data ID can be acquired from the data name management area 806 when the data is designated by the data name. .

  FIG. 8B is a diagram showing scenario data 810 representing the display order and display time of partial areas in the content 400.

  The scenario data 810 is data in a table format edited by the scenario editing unit 307 of the material creation terminal 101, and display unit information indicating a display method of one partial area is shown for each line. The display unit information has a content ID 811, a partial area management table ID 812, a page number 813, and a partial area number 814, and can specify a partial area to be displayed and its partial area data. Further, the display unit information has a display time 815. When the content 400 is actually displayed on the material display terminal 102, when the time indicated by the display time 815 of the line after the start of the display of the partial area specified in the line of the scenario data 810 has elapsed automatically The partial area specified in the next line may be displayed. The display time 815 is a reference time, and the display of the partial area specified in the next line of the scenario data 810 may be performed after the document explainer 142 is operated. A value indicating “not set” as the display time 815 (0 in this embodiment) as in the partial region (partial region 626 in this embodiment) specified in the second line of the scenario data 810 in FIG. Can also be set. However, in this case, after the partial area (partial area 626 in the present embodiment) is displayed on the material display terminal 102, it is always specified in the next line of the scenario data 810 after waiting for the operation of the material explainer 142. Display partial area.

  Also, as shown in FIG. 8A, when the scenario data 810 is transmitted from the material creation terminal 101 to the material data management server 103, it is stored on the storage device 204 of the material data management server 103. At the time of saving, the material data management unit 333 assigns the scenario ID 804 of the scenario data 810 to the data ID management area 801.

  The scenario data 810, the content 400 and the partial area management table 700 referred to in the scenario data 810 are collectively referred to as a set of material data. The material data management server 103 can store and manage a plurality of material data by the material data management method shown in FIG.

(Details of document display terminal)
FIG. 9 is an external view of the touch panel display 901 constituting the display unit 313 of the material display terminal 102. Although not shown in FIG. 9, the touch panel display 901 is connected to the display I / F 206 and the operation input I / F 207 (FIG. 2) of the material display terminal 102.

  The touch panel display 901 has a width of W00 and a height of H00, and information on the width and height is stored in advance in the storage device 204 of the material display terminal 102. The liquid crystal display portion of the touch panel display 901 is connected to the display I / F 206 of the material display terminal 102 and can display an image. The display unit 313 of the document display terminal 102 controls the display I / F 206 to display the content 400 on the touch panel display 901 and a user interface that receives operations on the content 400. Further, the touch panel portion of the touch panel display 901 is connected to the operation input I / F 207 of the material display terminal 102, and when the material explainer 142 touches the touch panel display 901, the coordinates (touch coordinates) can be received. The operation unit 314 of the document display terminal 102 receives the touch coordinates by controlling the operation input I / F 207, and operates the document explainer 142 (tap, swipe, etc.) based on the touch coordinates, the time change, and the number of touches simultaneously. Is determined. The document data display processing unit 315 displays the document data and responds to the operation of the document explainer 142 via the display unit 313 and the operation unit 314.

  By the description so far, the configuration of the material data handled in this embodiment and the overall configuration of the information processing system have been described. From here, the display mode when the material display terminal 102 displays the material data will be described with reference to a flowchart and a diagram showing three display examples.

(Document data display processing)
The material data display process executed by the material data display processing unit 315 of the material display terminal 102 will be described with reference to the flowcharts of FIGS. The document data display process is performed on a set of document data including one scenario data 810, content 400 referenced from the scenario data 810, and partial area management table 700. In the present embodiment, a display form of a scene where the material explainer 142 confirms / edits the configuration of the material data on the material display terminal 102 will be described.

  When the document data display processing is started, the processing from step S1001 to S1010 sequentially performs display unit information included in the scenario data 810 for each row from the first row.

  First, in step S1001, a processing target row is set from the scenario data 810.

  In step S1002, the content ID 811 information is acquired from the display unit information of the processing target row, and the content 400 is acquired based on the information. Further, partial area data of the partial area specified from the partial area management table ID 812, page number 813, and partial area number 814 is acquired.

  In step S1003, it is determined whether the region attribute 705 included in the partial region data acquired in step S1002 is the background. If the background is determined in step S1003, the process advances to step S1004 to display the entire page as shown in FIG. Note that the display order of the background partial areas is always set to 0 in step S509 in FIG. 5, so the determination in step S1003 is not whether or not the area attribute 705 is background but whether or not the partial area number 702 is 0. You may make it perform based on. If it is determined in step S1003 that the background is other than the background, the process proceeds to step S1005, and the partial area enlargement display process specified in step S1002 is performed as shown in FIG. Details of the partial area enlargement display process will be described later.

  After either the whole page display in step S1004 or the partial area enlarged display in step S1005 is performed, the process proceeds to step S1006 to perform related information display processing. Details of the related information display processing will be described later.

  When the related information display process ends in step S1006, the process proceeds to step S1007 and waits for the operation of the material explainer 142. When the operation of the document explainer 142 is detected, the process advances to step S1008 to determine whether the detected operation is a scenario transition operation or another operation. The scenario transition operation is an operation for advancing display according to the scenario or returning to the previous display. For example, in this scenario transition operation, a gesture operation such as advancing a scenario when a tap operation in the right half area on the screen is detected, and returning a scenario when the left half area on the screen is tapped, etc. It is also possible to press a button displayed on the screen.

  If it is determined in step S1008 that the operation is a scenario transition operation, the process advances to step S1010. On the other hand, if it is determined in step S1008 that the operation is other than the scenario transition operation, the process advances to step S1009 to update the screen display corresponding to the detected operation.

  For example, when the operation detected in step S1007 is a pinch-out operation, the screen display is updated in step S1009 so that an image displayed on a part of the screen is enlarged. If the operation detected in step S1007 is a drag operation, in step S1009, the screen display range in the page image including the partial area specified in step S1002 is updated. If the background is determined in step S1003, the dragging operation is performed to update the screen display range to another page image.

  As described above, after the display state of the document data is updated in step S1009 according to the operation detected in step S1007, the process returns to step S1006 to perform related information display processing.

  In steps S1006 to S1009, a scenario editing screen shown in FIGS. 13A and 13B described later is displayed.

  In step S1010, the processing target line in the loop processing in steps S1001 to S1010 is reset according to the contents of the scenario transition operation determined in step S1008, and the process proceeds to step S1002. For example, if the content of the scenario transition operation is an instruction to display the previous scenario, the line immediately above the current processing target line is reset to the processing target line. On the other hand, when the current processing target line is the last line of the scenario data 810 and the content of the scenario transition operation is an instruction to display the next scenario, the loop processing is exited and the document data display is performed. The processing unit 315 ends this process.

(Enlarged display processing of partial area)
With reference to the flowchart of FIG. 10B, the partial area enlargement display process in step S1005 of FIG. 10A executed by the material data display processing unit 315 will be described.

  In this process, an appropriate display enlargement ratio and display coordinates are set for each partial area, and the entire area is enlarged and displayed so that the entire area is displayed on the touch panel display 901 (display unit 313) of the document display terminal 102 for each partial area. It is.

  In step S1021, the display enlargement ratio is determined based on the information on the width height 704 of the partial area data of the partial area specified in step S1002 and the information on the width height of the touch panel display 901 held in the storage device 204 of the material display terminal 102. To do.

  For the display enlargement ratio, the enlargement ratio in the width direction and the height direction is once calculated, and the lower one of the calculated enlargement ratio values is set as the display enlargement ratio. For example, in the case of the partial area data (page number = 1, partial area number = 1) in the second row of the partial area management table 700 in FIG. 7, the expansion ratio W00 / W21 in the width direction and the expansion ratio H00 in the height direction. / H21 is compared. As a result of this comparison, the lower value is set as the display magnification of the corresponding partial area.

  In step S1022, the center coordinates of the touch panel display 901 are set as the center coordinates of the partial region based on the information of the partial region data coordinates 703 and the width height 704 of the partial region specified in step S1002. As a result, the partial area is displayed at the center of the touch panel display 901. For example, in the case of the partial area data (page number = 1, partial area number = 1) in the second row of the partial area management table 700 of FIG. 7, first, the central coordinates (X21 + (W21 / 2), Y21 +) of the partial area. (H21 / 2)) is calculated. Thereafter, the central coordinates (W00 / 2, H00 / 2) of the touch panel display 901 are set as the calculated central coordinates.

  In step S1023, the partial area is displayed on the touch panel display 901 in accordance with the display magnification set in step S1021 and the center coordinates set in step S1022, and then the present process is terminated.

(Related information display processing)
The related information display process in step S1006 of FIG. 10 will be described using the flowcharts of FIGS. 12A (a) and (b) and the process target list of FIG. 12B. Thereafter, three examples suitable for explaining the present invention will be described with reference to FIGS. 13A (a) to (f) and FIGS. 13B (a) to (e).

  First, when the related information display process of FIG. 12A (a) is started, the displayed related information is initialized in step S1201. That is, as the display state of the material data is changed in any one of steps S1004 and S1005, it is necessary to change the display form of the related information. Therefore, if the related information is already displayed after the start of the process of FIG. 10, the related information is initialized. Thereby, the related information is not displayed in the display area of the touch panel display 901.

  In step S1202, an object display determination process is performed to determine whether an object for displaying related information is displayed in the display area. The object display determination process will be described with reference to the flowchart of FIG.

  The process from step S1221 to S1225 in FIG. 12A (b) is a loop process that ends after sequentially processing the partial area data of each partial area included in the currently displayed page image. More specifically, first, partial area data of each partial area included in the currently displayed page image is acquired from the partial area management table 700. After that, the obtained partial area data is sequentially processed one line at a time in ascending order of the partial area number 702, and then the loop process ends. In other words, the page number 701 of the partial area data to be looped matches the page number of the currently displayed page image.

  In step S1222, the area ratio in which the partial area (object) to be processed is displayed on the touch panel display 901 is calculated. Specifically, this area ratio is calculated from the coordinates 703 and the width height 704 included in the partial area data of the processing target row, and the width height of the material display area 1302 of FIG. 13A (a) described later of the touch panel display 901. The If partial area enlargement display processing has been performed in step S1005, the display enlargement ratio is also calculated.

  In step S1223, it is determined whether or not the area ratio calculated in step S1222 is greater than or equal to a threshold value, that is, whether or not display on the touch panel display 901 is present. The threshold value may be set to a fixed value in advance, or may be set to a different value according to the display magnification ratio of the object, the region attribute 705 included in the partial region data, and the like. If it is determined in step S1223 that the value is equal to or greater than the threshold value, it is determined that the object is displayed, and the process proceeds to step S1224. If it is determined in step S1223 that it is less than the threshold value, it is determined that no object is displayed, and the process advances to step S1225.

  In step S1224, the object information is added as the main object information of the processing target list 1230.

  FIG. 12B is a diagram showing a processing target list 1230 to which information is sequentially added in step S1224 of FIG. 12A. The processing target list 1230 has a table shape, and one row of the table corresponds to one related information. Here, the related information includes information on one partial area (main body object) and information on a partial area (related object) associated with the main body object. Specifically, the processing target list 1230 has four items of a page number 1231 of the main object, a partial area number 1232, a page number 1233 of the related object, and a partial area number 1234, and a combination of related partial areas is displayed. It can be uniquely identified. Further, the processing target list 1230 includes an item of relevance information 1235 indicating what kind of relation the main body object and the related object have. The relevance information 1235 is used to determine the display form at the time of display, for example, when the direction of the relation such as the display order has directionality, such as storing the master-slave relationship. In addition to the master-slave relationship, numerical information such as the degree of association may be stored as the relationship information 1235.

  There is a case where the relevance between the partial areas is indirectly expressed in the form of setting the display order (scenario) as in the present embodiment. Even in such a case, information is converted in step S1224 and described in the processing target list 1230. For example, in the present embodiment, the page number 813 and the partial area number 814 in the scenario data 810 indicate the relationship between the partial areas. Therefore, in step 1224, in the scenario data 810, it is determined that the partial areas existing before and after the partial area that is the main object of the processing target list 1230 are related objects having a main object subordinate relationship with the main body object. Further, when the related object information matches one of the main body object information in the processing target list 1230, the related object information is added to the processing target list 1230. After the addition process is completed for all the body objects in the processing target list 1230, the process proceeds to step S1225.

  When the loop processing in steps S1221 to S1225 is finished, this processing is finished, and the process proceeds to step S1203 in FIG.

  In this embodiment, in step S1223, whether or not an object is displayed is determined based on the area ratio calculated in step S1222, but other determination methods may be used. For example, when the display is performed in units of pages, display / non-display of the objects may be determined depending on whether the related object is included in the currently displayed page image.

  The process from step S1203 to S1207 is a loop process in which the related information in the processing target list 1230 is sequentially processed for each line from the first line, and is finished when the process is completed up to the last related information.

  In step S1204, if the processing target related information in the processing target list 1230 acquired in step S1202 includes not only the information on the main body object but also information on the related object, it is determined that the target related object exists on the screen. To do. On the other hand, if the related object information is not included, it is determined that the target related object does not exist on the screen. If it is determined in step S1204 that the target related object exists on the screen, the process proceeds to step S1206. If it is determined in step S1204 that the target related object does not exist on the screen, the process proceeds to step S1205.

  In step S1205, information on partial areas existing before and after the main body object (partial area) specified from the relevant information to be processed is acquired from the scenario data 810 as information on related objects. After that, based on the acquired information, a substitute object that is a substitute is generated and displayed. An alternative object is an object that is displayed on the screen to indicate the presence of a related object that is not displayed, such as a thumbnail of the related object or an icon representing area information. The substitute object may have any display form as long as it represents the target related object. The substitute object may be displayed not only on the display area of the page image but also outside the display area of the page image. The display position of the substitute object may be in any form as long as the relationship between the target related object and the main body object can be visually recognized. For example, the substitute object may be displayed near the area where the main body object is displayed. The display position of the substitute object may be determined based on the positional relationship between the main body object and the target related object. The size of the substitute object is preferably adjusted in consideration of the size ratio between the width and height of the target related object and the area in which the substitute object is displayed. After the generation / display of the substitute object is completed, the process proceeds to step S1206.

  In step S1206, association display between objects is performed. If it is determined in step S1204 that the target related object exists on the screen, a display indicating the relationship between the main body object and the target related object is performed. On the other hand, if it is determined in step S1204 that the target related object does not exist on the screen, a display indicating the relationship between the main body object and the substitute object generated in step S1205 is performed. The display indicating relevance is, for example, linking or display by connecting objects with lines, or a display that expresses an order relationship by connecting arrows. The display form is not unique, and the form may be changed according to the relationship between objects. For example, when there is a numerical magnitude in the relevance, the color and shape may be changed and a display may be made so that the magnitude relationship can be grasped by display. When the association display is completed, the process proceeds to step S1207.

  In step S1207, the related information to be subjected to the loop processing in steps S1203 to S1207 is changed. When the execution of the loop processing in steps S1203 to S1207 is completed for all the related information existing in the processing target list 1230, the loop processing is exited, and the related information display processing unit 316 ends this processing.

  Next, a specific example of related information display processing will be described.

  13A and 13B are diagrams illustrating an example of a scenario editing screen displayed on the touch panel display 901 in steps S1006 to S1009 in FIG. Here, the scenario editing screen is a screen for the material explainer 142 who operates the material display terminal 102 to confirm or edit the material data.

  As shown in FIGS. 13A and 13B, on the scenario editing screen 1301, the whole or part of one page image of the content 400 is displayed in the material display area 1302. Further, an area 1303 for displaying an alternative object is secured around the material display area 1302. The material explainer 142 operates the display material using operation buttons 1306 to 1308 displayed on the scenario editing screen 1301, gestures obtained from the touch panel, and the like.

  First, the case where the related information display process of FIG. 12A is started in the state of FIG. 13A in which the entire page image 401 of the content 400 is displayed in the material display area 1302 will be described. It is assumed that scenario data 810 is set for the content 400 to be displayed.

  FIG. 13A (b) illustrates a state when the object display determination process (FIG. 12A (b)) in step S1202 of FIG. 12A (a) is executed with respect to the display state of FIG. 13A (a). FIG. In FIG. 13A (b), the partial areas added as information on the main body object to the processing target list 1410 shown in FIG. 14 (a) are represented as shaded parts 1311-1320. In the case of FIG. 13A (a), it is determined that the shaded parts 1311 to 1320 of FIG. 13A (b), that is, all partial areas in the page image 401 are displayed. Therefore, information on all partial areas in the page image 401 is added to the processing target list 1410 as information on the main body object. Note that the shaded portions 1311 to 1320 in the partial area shown in FIG. 13A (b) are displayed for convenience of explanation of the present embodiment, and the shaded display is not performed during actual processing.

  FIG. 13A (c) shows a state in which related information display processing is executed for the display state of FIG. 13A (a), and linked display between objects is performed. In FIG. 13A (c), since the partial areas of the shaded portions 1312, 1316, 1317, 1319, and 1320 that are the related objects to be targeted are all present on the screen, the direct relationship is expressed between these partial areas. Displays 1321 to 1324 are made. That is, a display that directly links these partial areas is performed. In the present embodiment, the arrow direction of the display indicating the relevance is determined according to the master-slave relationship shown in the relevance 1405 of the processing target list 1410. As shown in this example, if all the partial areas that are the related objects of interest are displayed, the relevance between the partial areas is clearly shown to the document explainer 142 by displaying the direct relation between the partial areas. Can be presented.

  Next, an example in which a screen enlargement operation is performed on the display state of FIG. 13A (c) will be described with reference to FIGS. 13A (d) to 13 (f) and 13B (a).

  FIG. 13A (d) is a diagram showing that the document explainer 142 performs a screen enlargement operation to display the range 1325 in step S1008 in the display state of FIG. 13A (c). In the document display terminal 102, when a screen enlargement operation from the document explainer 142 is detected, the display is updated in step S1009, and an image 1331 of the page image 401 in which the range 1325 is enlarged as shown in FIG. 13A (e). Is displayed in the material display area 1302. A case where the related information display process is executed for this display state will be described.

  FIG. 13A (f) is a diagram for explaining a state when the object display determination process is executed for the display state of FIG. 13A (e). In FIG. 13A (f), partial areas added as information on the main body object to the processing target list 1420 shown in FIG. 14B are represented as shaded portions 1316 to 1318. Thus, when the partial area displayed on the screen by the screen enlargement operation becomes a part of the partial area in the page image 401, only the partial areas of the shaded portions 1316 to 1318 in FIG. 13A (f) are displayed. It is determined that In this case, information on the partial areas of the shaded parts 1316 to 1318 is added to the processing target list 1420 as information on the main body object. Thereafter, related information display processing is sequentially performed on the related information in each row of the processing target list 1420.

  FIG. 13B (a) shows a state in which related information display processing is executed for the display state of FIG. 13A (e) and the association display between objects is performed. In FIG. 13A (e), since the partial areas of the shaded portions 1316 and 1317, which are the related objects to be targeted, are both present on the screen, a display 1344 indicating a direct relationship between these partial areas. On the other hand, in FIG. 13A (e), the partial areas of the shaded portions 1312 and 1319 in FIG. 13A (c), which are target related objects, do not exist on the screen. Therefore, a display 1321 representing the relationship between the partial areas of the shaded parts 1312 and 1317 and a display 1323 representing the relation between the partial areas of the shaded parts 1316 and 1319 shown in FIG. Direct linking is not possible. In this case, alternative objects 1341 and 1342 are generated for the partial areas of the shaded parts 1312 and 1319 that do not exist on the screen, and are displayed in the area 1303. In this embodiment, as substitute objects 1341 and 1342, images obtained by converting thumbnail images of partial areas 1312 and 1319 (related objects) into thumbnails are used. Thereafter, for each of the alternative objects 1341 and 1342 displayed in the area 1303, displays 1343 and 1345 indicating the relationship with the partial area being displayed are displayed. As in this example, when a part of the partial area that becomes the target related object is not displayed in the material display area 1302 by the screen operation, the alternative object is displayed in the area 1303, and the alternative object and the displayed part are displayed. Display the relationship with the area. Therefore, related information can be presented to the material explainer 142.

  Finally, an example in which a screen movement operation is performed on the display state of FIG. 13B (a) will be described with reference to FIGS. 13B (b) to (e).

  FIG. 13B (b) is a diagram showing that the document explainer 142 has performed a screen movement operation so as to display the range 1351 to the range 1351 in step S1008 in the display state of FIG. 13B (a). In the document display terminal 102, when a screen movement operation from the document explainer 142 is detected, the display is updated in step S1009, and an image 1361 of the page image 401 obtained by enlarging the range 1351 as shown in FIG. 13B (c). Is displayed in the material display area 1302. A case where the related information display process is executed for this display state will be described.

  FIG. 13B (d) is a diagram for explaining a state when the object display determination process is executed for the display state of FIG. 13B (c). In FIG. 13B (d), the partial areas added as the information of the main body object to the processing target list 1430 shown in FIG. 14C are represented as shaded parts 1312, 1313, 1316, 1317. As described above, when the partial area displayed on the screen is changed by the screen moving operation, it is determined that only the partial areas of the shaded portions 1312, 1313, 1316, and 1317 in FIG. 13B (d) are displayed. The In this case, information on the partial areas of the shaded parts 1312, 1313, 1316, and 1317 is added to the processing target list 1420 as information on the main body object. Thereafter, related information display processing is sequentially performed on the related information in each row of the processing target list 1420.

  FIG. 13B (e) shows a state where the related information display process is executed for the display state of FIG. 13B (c) and the association display between objects is performed. In FIG. 13B (c), since the partial areas of the shaded portions 1316 and 1317 which are the related objects to be targeted exist on the screen, a display 1383 representing the direct relationship between these partial areas is displayed. In addition, regarding the partial areas of the shaded portions 1316 and 1319 in FIG. 13A (c) that are related, the partial area of the shaded portion 1316 exists on the screen even after the screen moving operation in FIG. 13B (b). The state that the partial area 1319 does not exist on the screen is not changed. Therefore, a substitute object 1381 for the partial area of the shaded portion 1319 is generated, and a display 1384 indicating the relevance with the partial area of the shaded portion 1316 is made. On the other hand, the partial area of the shaded portion 1312 in FIG. 13A (c) that is not displayed in FIG. 13B (a) comes to exist on the screen by the screen moving operation. That is, as shown in FIG. 13B (d), the partial areas of the shaded portions 1312 and 1317 that are related to each other exist on the screen. Therefore, the relevance of the partial areas of the shaded portions 1312 and 1317 in FIG. 13A (c) shown using the display 1343 representing the relevance with the alternative object 1341 in FIG. 13B (a) is shown in FIG. 13B (e). A display 1382 showing a direct relationship between these partial areas is shown.

  Up to this point, an example has been shown in which the display form of the target related object is changed according to the display state of the content.

  As described above, according to the present invention, the relationship between objects included in content is displayed without depending on the display state of the content to be displayed. Therefore, the material explainer 142 can always grasp the relationship between objects included in the content. In particular, as shown in FIGS. 13B (a) and 13 (e), two or more partial areas of the page image are enlarged and displayed on the screen by the operation of the touch panel display 901 of the material explainer 142. Even in such a case, the material explainer 142 can grasp the relationship between the objects included in the content.

[Other Examples]
It goes without saying that the object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to the apparatus. At this time, the computer (or CPU or MPU) including the control unit of the supplied apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the OS (basic system or operating system) running on the apparatus performs part or all of the processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. Needless to say, cases are also included.

  Further, the program code read from the storage medium may be written into a memory provided in a function expansion board inserted into the apparatus or a function expansion unit connected to the computer, and the functions of the above-described embodiments may be realized. Needless to say, it is included. At this time, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

DESCRIPTION OF SYMBOLS 100 Information processing system 101 Material preparation terminal 102 Material display terminal 103 Material data management server 104 Page analysis server 120 Network 200 Control apparatus 311 Control part 312 Communication part 313 Display part 314 Operation part 315 Material data display process part 316 Related information display process part

Claims (6)

An image processing apparatus comprising: content display means for displaying part or all of content including a plurality of display units on a screen; and display processing means for performing display in association with display units associated with the content. ,
The display processing means includes
First determination means for determining whether each of the plurality of display units is displayed on the screen by the content display means;
Second determination means for determining whether or not a display unit related to a target display unit among the display units determined to be displayed on the screen by the first determination means is displayed on the screen;
When the second determination means determines that the relevant display unit is displayed on the screen, a first display that associates the target display unit with the relevant display unit is performed. Display means;
When the second determining means determines that the related display unit is not displayed on the screen, it generates a substitute object to replace the related display unit and displays it on the screen. An image processing apparatus comprising: a second display unit configured to display the target display unit and the substitute object. The display processing means further comprises calculation means for calculating an area ratio displayed on the screen for each of the plurality of display units,
The image processing apparatus according to claim 1, wherein the first determination unit determines whether or not each of the plurality of display units is displayed on the screen based on the calculated area ratio.   The determination by the second determination unit is performed based on information indicating which of the other display units each of the plurality of display units is related to and a result of the determination by the first determination unit. The image processing apparatus according to claim 1 or 2.   The image processing apparatus according to claim 1, wherein the content display unit displays two or more display units on the screen. A control method for an image processing apparatus, comprising: a content display step for displaying a part or all of content including a plurality of display units on a screen; and a display processing step for performing a display in which display units associated with the content are linked. And
The display processing step includes
A first determination step of determining whether or not each of the plurality of display units is displayed on the screen by the content display step;
A second determination step of determining whether or not a display unit related to a target display unit among the display units determined to be displayed on the screen in the first determination step is displayed on the screen;
When it is determined in the second determination step that the display unit having the relation is displayed on the screen, a first display that associates the target display unit with the display unit having the relation is performed. A display step;
If it is determined in the second determination step that the related display unit is not displayed on the screen, a substitute object that substitutes for the related display unit is generated on the screen and the screen is displayed. And a second display step for displaying the target display unit and the substitute object in association with each other. A program for executing the control method according to claim 5.

Images