JP2015049656A - Information processing device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a scrap function that is suitable for information processing terminals only with comparatively small touch panels, and does not require cumbersome user operations.SOLUTION: A device has display means that includes a scrap function of managing an area designated within a page image as a scrap area and enabling browsing of the scrap area, and the device comprises: display range determination means for determining a display magnification in accordance with a size of the display means for each object designated by a user out of objects included in the page image; scrap area setting means for setting an area composed of one or more objects having the display magnification determined as the scrap area; management means for managing the scrap area set by the scrap area setting means; and display control means for, upon displaying the scrap area to be managed by the management means in the display means, controlling the one or more objects consisting of the scrap area so as to be displayed at the display magnification determined by the display range determination means.

Description

  The present invention relates to an image display technique for making structured image data suitable for browsing on a small screen.

  Conventionally, several prior arts for cutting out a certain part (partial region) of a page image have been proposed. For example, in Patent Document 1, a bookmark designation button is provided for a partial area of a page image, and a bookmark designation button is selected one by one, whereby a partial area corresponding to the button is cut and bookmarked. Techniques that can be used are disclosed. Further, in Patent Document 2, a cutout range of a scanned image is determined based on document information input by a user when scanning a document, and the range to be cut out extends over a plurality of image pages. A method for easily cutting out an area is disclosed.

JP 2010-224927 A JP 2008-0778757 A

  In the use of newspapers and paper documents, a necessary portion on the paper surface is cut out by using scissors and a cutter knife, pasted and stored in a scrapbook, and used for browsing at a later date.

  The applicant of the present application considers the realization of a function similar to scrap in such a paper document (scrap function performed on a rasterized page image) by an information processing terminal such as a smartphone. In other words, it is considered to provide a function (hereinafter referred to as a scrap function) that allows a region of interest in a page image to be specified and scrapped, and that a partial region of the scraped page image can be browsed as a scrapbook later. ing.

  Here, if it is going to implement | achieve a scrap function using the area | region extraction method as disclosed by patent document 1 and 2, the whole page image will be displayed on a screen, and a desired partial area | region will be selected on it. . However, in such a method, it is difficult to select a partial area in an information processing terminal having a small screen size such as a smartphone. For example, when an entire page image of A4 size or the like is displayed on a small screen such as a smartphone, characters in the page image become small, and it is very difficult to select a scrap area while checking the document content. In addition, the selection operation of the partial area is usually performed by an enlargement / reduction operation such as pinch-in / pinch-out on the touch panel. Since the displayed characters are small, the enlargement / reduction operation and the area selection operation must be repeated, and the user operation is not performed. It becomes complicated.

  An information processing apparatus according to the present invention is an information processing apparatus having a display unit having a scrap function that manages a designated area in a page image as a scrap area and enables browsing of the scrap area, For each object specified by the user among the objects included in the page image, a display range determining unit that determines a display magnification according to the size of the display unit, and one or a plurality of objects for which the display magnification is determined Scrap area setting means for setting the area to be set as the scrap area, management means for managing the scrap area set by the scrap area setting means, and displaying the scrap area managed by the management means on the display means When performing the display range determination, the one or more objects constituting the scrap area are selected. Characterized by comprising a display control means for controlling to display by in the determined display magnification.

  ADVANTAGE OF THE INVENTION According to this invention, the scrap function suitable for the information processing terminal which has only a comparatively small touch panel and which does not require complicated user operation can be provided.

1 is a block diagram illustrating a configuration of an image processing system according to Embodiment 1. FIG. 2 is a block diagram illustrating a functional configuration of the MFP. FIG. It is a flowchart which shows the flow of the process which produces | generates page image data from bitmap image data. It is a figure which shows an example of the result of having divided bitmap image data into a plurality of objects. It is a figure which shows an example of block information and input file information. It is a figure which shows an example of a data structure of a vectorization process result (intermediate data). It is a flowchart which shows the detail of a page image data generation process. It is a flowchart which shows the detail of a tree structure production | generation process. It is a figure which shows an example of the document used as the object of a tree structure production | generation process. It is a figure which shows an example of a tree structure. It is a figure which shows an example of page image data. It is a block diagram which shows an example of a structure of a portable information terminal. It is a block diagram which shows the structure of the software module performed by CPU of a portable information terminal. It is a figure which shows an example of the gesture event which generate | occur | produces in a gesture event generation module. It is a flowchart which shows the flow of the process performed when a portable information terminal receives page image data. It is a figure which shows an example of an object information management table. It is an example of the display screen of the touch UI in which the display state is updated. It is a block diagram which shows the internal module structure regarding the operation control of page image data in a gesture event processing module. It is a flowchart which shows the detail of a mode switching process. It is a figure which shows an example of the screen display of touch UI at the time of partial area display mode. It is a flowchart which shows the detail of a display range determination process. It is a flowchart which shows the detail of a next selection process. It is a flowchart which shows the detail of a pre-selection process. It is a flowchart which shows the flow of a scrap area | region setting process. 6 is a screen display example when designating a scrap area in the touch UI according to the first embodiment. It is a flowchart which shows the detail of the process which displays the frame which shows a scrap area | region on touch UI. It is a flowchart which shows the detail of a scrap area | region expansion process. It is a figure which shows the scrap area | region management table as an example of the format which preserve | saves a scrap area | region. It is a figure which shows an example of the menu screen in a portable information terminal, a document list screen, and a scrapbook screen. It is a flowchart which shows the flow of a scrap display process. 10 is a flowchart illustrating a flow of scrap area expansion processing according to the second embodiment. 10 is a screen display example when designating a scrap area in the touch UI according to the second embodiment. It is a figure which shows an example of the state which displayed the scrap area | region in the scrapbook based on Example 2. FIG.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an image processing system according to the present embodiment.

  In FIG. 1, a LAN 102 constructed in an office is connected to a multifunction peripheral (hereinafter, MFP) 100 that realizes a plurality of types of functions such as copying, printing, and transmission, and a client PC 101. In addition, a proxy server 103, a document management server 106, a database 105 for the document management server 106, and a portable information processing device (hereinafter referred to as a portable information terminal) 107 are connected to the LAN 102 wirelessly or by wire. The LAN 102 is connected to the network 104 via the proxy server 103.

  Each of the client PC 101 and the proxy server 103 has standard components (for example, a CPU, a RAM, a ROM, a hard disk, a network I / F, a display, a keyboard, and a mouse) mounted on a general-purpose computer. For example, the client PC 101 can print prints based on the print data by the MFP 100 by transmitting the print data to the MFP 100. 1 is merely an example, and a plurality of offices having similar components may be connected on the network 104.

  The network 104 is typically realized by any of the Internet, LAN, WAN, telephone line, dedicated digital line, ATM, frame relay line, communication satellite line, cable TV line, data broadcasting wireless line, and the like. Needless to say, a so-called communication network realized by a combination of these may be used as long as data can be transmitted and received.

  FIG. 2 is a block diagram showing a functional configuration of MFP 100. 2, the MFP 100 includes a scanner unit 110, an HDD (hereinafter referred to as BOX) 111 as a storage device, a printer unit 112, a data processing unit 115, an input unit 113, a display unit 116, and a network I / F 114.

  The scanner unit 110 has an ADF (Auto Document Feeder) (not shown), irradiates a bundled image or a single document image with a light source, and forms a reflected image on a solid-state image sensor with a lens. The solid-state imaging device generates an image reading signal having a predetermined resolution (for example, 600 dpi) and a predetermined luminance level (for example, 8 bits), and image data including raster data is configured from the image reading signal.

  When a normal copying function is executed, the bitmap image data obtained by the scanner unit 110 is converted into a recording signal by performing a scan image processing described later by the data processing unit 115, and image formation (printing output) is performed by the printer unit 112. ) In the case of copying a plurality of sheets, a recording signal for one page is temporarily stored in the BOX 111 and then sequentially output to the printer unit 112 to form an image on recording paper.

  The MFP 100 receives PDL data output from the local PC 102 (or other general-purpose PC) using a driver via the LAN 102 and the network I / F 114, and outputs an image based on the received PDL data to the printer unit 112. Can be recorded by. That is, PDL data output from the local PC 102 via the driver is input from the LAN 107 to the data processing unit 115 via the network I / F 114. Therefore, the PDL is converted into a recordable recording signal by interpreting and processing and then recorded as an image on a recording sheet in the MFP 100.

  The BOX 111 stores data obtained by rendering data from the scanner unit 110 and PDL data output from the local PC 102 via a driver. Further, MFP 100 can be operated through a key operation unit (input unit 113) provided in MFP 100, and the state of operation input can be displayed on display unit 116.

  Next, how image data (hereinafter, page image data) in a predetermined format for display on the portable information terminal 107 in the present embodiment is generated from bitmap image data will be described. FIG. 3 is a flowchart showing a flow of processing for generating page image data from bitmap image data. The bitmap image data is acquired by the scanner unit 110 of the MFP 100, or is generated by rendering a document created by application software on the local PC 102 in the MFP 100.

  In step 301, the data processing unit 115 performs an object division process for dividing the bitmap image data into objects for each attribute. In the present embodiment, four types of attributes, specifically, characters (horizontal writing or vertical writing), photographs, graphics (drawings, line drawings, tables, lines), and background objects are divided. FIG. 4 is an example of a result obtained by dividing the bitmap image data into a plurality of objects, and shows a state where the input image 401 is divided into rectangular blocks for each attribute.

  Examples of the object division processing method include the following methods. First, image data stored in a RAM (not shown) in the MFP 100 is binarized to black and white, and a pixel block surrounded by a black pixel outline is extracted. Further, the size of the extracted black pixel block is evaluated, and the contour tracking is performed for the white pixel block inside the black pixel block whose size is a predetermined value or more. As long as the internal pixel block is equal to or greater than a predetermined value, such as size evaluation for the white pixel block and tracking of the internal black pixel block, the internal pixel block is extracted recursively and the contour is traced. The size of the pixel block is evaluated by, for example, the area of the pixel block. Then, a rectangular block circumscribing the obtained pixel block is generated, and attributes are determined based on the size and shape of the rectangular block. For example, a rectangular block having an aspect ratio close to 1 and having a constant size is assumed to be a character equivalent block that may be a character area rectangular block. Further, when adjacent character equivalent blocks are regularly arranged, a new rectangular block in which these character equivalent blocks are collected is generated, and the new rectangular block is set as a character area rectangular block. In addition, a flat pixel block or a black pixel block that is larger than a certain size and includes square white pixel blocks in a well-aligned manner is defined as a graphic area rectangular block, and other irregular pixel blocks are defined as a photographic area rectangular block.

  In the object division processing, block information such as attributes and input file information are generated for each of the rectangular blocks generated in this way. FIG. 5 shows an example of block information and input file information, respectively. The block information includes attributes of each block, position coordinates X, coordinates Y, width W, and height H. The attribute is given by a numerical value of 0 to 3, where 0 is a horizontally written character area rectangular block, 1 is a vertically written character area rectangular block, 2 is a photo area rectangular block, and 3 is a graphic area rectangular block. The coordinates X and Y are the X and Y coordinates (the coordinates of the upper left corner) of the starting point of each rectangular block in the input image. The width W and the height H are the width in the X coordinate direction and the height in the Y coordinate direction of the rectangular block. Furthermore, the total number N of blocks indicating the number of rectangular blocks is also stored as input file information. The block information for each rectangular block is used for vectorization in a specific area. In addition, the relative positional relationship when combining specific areas with other areas can be specified by block information, and vectorized areas and bitmap areas can be combined without impairing the layout of the input image. Become.

  Returning to the flowchart of FIG.

  In step 302, the data processing unit 115 determines the type of object (character, photograph, graphic, background) for each divided object. For an object determined to be a photo or a background, in step 303, the bitmap image of the object is JPEG compressed. On the other hand, an object determined to be a character or a graphic is converted into pass data (vector data) by performing vectorization processing in step 304. FIG. 6 is a diagram illustrating an example of a data configuration of a vectorization processing result (intermediate data). In the present embodiment, it is assumed that the intermediate data is stored in a format called a document analysis output format (hereinafter referred to as DAOF). As shown in FIG. 6, the DAOF includes a header 601, a layout description data portion 602, a character recognition description data portion 603, a table description data portion 604, and an image description data portion 605. The header 601 holds information related to the input image to be processed. The layout description data portion 602 includes information such as characters, graphics (line drawings, drawings, tables, lines), photographs, and the like, which are attributes of the rectangular block in the input image, and position information of each rectangular block in which these attributes are recognized. Retained. The character recognition description data portion 603 holds a character recognition result obtained by character recognition in the character area rectangular block. The table description data portion 604 stores details of the table structure of the graphic area rectangular block having table attributes. The image description data portion 605 holds a set of data representing the internal structure of the block, the shape of the image, the character code, and the like obtained by the vectorization process in the specific area where the vectorization process is instructed. On the other hand, in a rectangular block other than the specific area that is not subject to vectorization processing, the bitmap image data itself divided into objects is held.

  In step 305, the data processing unit 115 determines whether the object converted into vector data is a character. If the object is a character, the data processing unit 115 further performs OCR processing in step 308, and character-coded data (the OCR result) Character code data) is also acquired.

  In step 306, the data of each object (JPEG data, vector data) obtained in steps 303 and 304 and the character code data obtained in step 308 are combined into one file. Then, metadata such as attribute information is assigned to each object collected in one file.

  In step 307, the data processing unit 115 generates page image data that can be displayed by the portable information terminal 107, using a file in which metadata is assigned to each object. Details of this page image data generation processing will be described later.

  Finally, in step 309, the data processing unit 115 transmits the generated page image data to the portable information terminal 107.

(Page image data generation processing)
FIG. 7 is a flowchart showing details of the page image data generation process.

  In step 701, the data processing unit 115 acquires DAOF format intermediate data generated by vectorization in step 304.

  In step 702, the data processing unit 115 performs a tree structure generation process. FIG. 8 is a flowchart showing details of the tree structure generation process. As a basic rule of overall control in the tree structure generation process shown in FIG. 8, the flow of processing shifts from a micro block (single rectangular block) to a macro block (aggregate of rectangular blocks). Hereinafter, “rectangular block” means both a micro block and a macro block.

  In step 801, the data processing unit 115 regroups the rectangular blocks (grouping processing) in units of rectangular blocks based on the vertical relevance. Here, the relevance is defined by features such as a short distance and substantially the same block width (height in the horizontal direction). Information such as distance, width, and height is extracted with reference to DAOF. Although the tree structure generation process may be repeatedly executed, the determination is made in units of micro blocks immediately after the start of the process. FIG. 9 shows an example of a document to be subjected to the tree structure generation process. For example, in the bitmap image data V1 of the first page shown in FIG. 9A, a rectangular block T1 exists at the top, and a horizontal separator U1 exists below it. Rectangular blocks T2 to T8 exist in the lower left of the separator U1. Similarly, in the bitmap image data V2 of the second page shown in FIG. 9B, the horizontal separator U11 exists at the top and rectangular blocks T11 to T18 exist below the separator U11. By the grouping process based on the vertical relationship in this step, in the bitmap image data V1 of the first page, the rectangular blocks T2 and T3 are combined into one group (rectangular block) S1, and the rectangular blocks T4 and T5 are combined. These are grouped into one group (rectangular block) S2. Groups S1 and S2 are on the same level. Similarly, in the bitmap image data V2 of the second page, the rectangular blocks T11 to T14 are grouped into one group (rectangular block) S11, and the rectangular blocks T16 and T17 are grouped into one group (rectangular block) S12. It is done.

  Next, in step 802, the data processing unit 115 checks whether or not there is a vertical separator. This separator is an object having a line attribute in DAOF and has a function of explicitly dividing a block. When the separator is detected, the area of the input image is divided into left and right with the separator as a boundary in the processing target hierarchy. In the example shown in FIG. 9, there is no vertical separator.

  In step 803, the data processing unit 115 determines whether the total group height in the vertical direction is equal to the height of the input image. That is, when performing grouping in the horizontal direction while moving the region to be processed in the vertical direction (for example, from top to bottom), when the processing of the entire input image is completed, the total group height becomes the height of the input image. Is used to determine the end of the process. When the grouping is finished, the process is finished as it is, and when the grouping is not finished, the process proceeds to Step 804.

  In step 804, the data processing unit 115 executes a grouping process based on the relevance in the horizontal direction. Accordingly, for example, in the case of the first page shown in FIG. 9A, the rectangular blocks S1 and S2 are combined into one group (rectangular block) W1, and the rectangular blocks T6 and T7 are combined into one group (rectangular block). ) W2 Groups W1 and W2 are on the same level. Again, immediately after the start of processing, the determination is made in units of micro blocks.

  In step 805, the data processing unit 115 checks whether or not there is a horizontal separator. When the horizontal separator is detected, the area of the input image is divided vertically in the processing target hierarchy with the separator as a boundary. For example, in the case of the first page shown in FIG. 9A, since the horizontal separator U1 exists, the area of the input image is divided vertically with this separator U1 as a boundary. The above processing results are registered as a tree. FIG. 10 is a diagram illustrating an example of a tree structure, where V1 represents a page, and objects in the lower hierarchy of V1 represent objects. In FIG. 10, the input bitmap image data V1 for the first page has a rectangular block T1, a separator U1, a group W1, a group W2, and a rectangular block T8 in the highest hierarchy. The group W1 includes the second layer groups S1 and S2, the group S1 includes the third layer rectangular blocks T2 and T3, and the group S2 includes the third layer rectangular blocks T4 and T5. The group W2 includes rectangular blocks T6 and T7 in the second hierarchy.

  Finally, in step 806, it is determined whether the total group length in the horizontal direction is equal to the width of the input image. In this way, the end determination regarding the grouping in the horizontal direction is performed. If the horizontal group length is the page width, the tree structure generation process is terminated. If the group length in the horizontal direction is not equal to the page width, the process returns to step 801, and the relevance check in the vertical direction is repeated again at the next higher level.

  The above is the content of the tree structure generation process.

  Returning to the flowchart of FIG.

  In step 703, the data processing unit 115 acquires actual data in the DAOF based on the generated tree structure, and generates page image data. FIG. 11 is a diagram illustrating an example of page image data. In the present embodiment, description will be made using SGV (Scalable Vector Graphics) format as the format of page image data. In the example of FIG. 11, the notation of each object is surrounded by frames 1101, 1102, and 1104 for explanation. Each object has area information indicating the area of the object and a drawing element acquired from actual data in the DAOF. It is also possible to have an object that has only drawing information and no area. A frame 1101 corresponds to a photo attribute, in which region information indicating a region of a photo object and bitmap information as a drawing element are shown. A frame 1102 corresponds to a character attribute, and is obtained by vector data of a character object, character code data (character code data obtained by the OCR processing in step 308) shown in a dashed frame 1103, and vectorization processing in step 304. The character size is indicated. It is also possible to describe the text area combination direction (vertical writing or horizontal writing) obtained by vectorization processing. A frame 1104 corresponds to a graphic object such as a line drawing.

  In this embodiment, the page image data is expressed in the SVG format, but any image format that can describe and hold the meaning and structure of the document is acceptable, and the present invention is not limited to this.

  FIG. 12 is a block diagram illustrating an example of the configuration of the portable information terminal 107.

  The portable information terminal 107 includes a main board 1200, an LCD 1201, a touch panel 1202, and a button device 1203. The LCD 1201 and the touch panel 1202 are collectively referred to as a touch UI 1204.

  The main components of the main board 1200 are as follows. CPU 1205, wireless LAN module 1206, power supply controller 1207, display controller (DISPC) 1208, panel controller (PANELC) 1209, ROM 1210, RAM 1211, secondary battery 1212, timer 1213. These components 1205 to 1213 are connected by a bus (not shown).

  The CPU 1205 is a processor that controls each device connected to the bus, and develops and executes a later-described software module stored in the ROM 1210 on the RAM 1211. The RAM 1211 functions as a main memory of the CPU 1205, a work area, an area for video images displayed on the LCD 1201, and a storage area for page image data transmitted from the MFP 100.

  A display controller (DISPC) 1208 switches the video image output developed in the RAM 1211 at high speed in response to a request from the CPU 1205 and outputs a synchronization signal to the LCD 1201. As a result, the video image in the RAM 1211 is output to the LCD 1201 in synchronization with the synchronization signal from the DISPC 1208, and the image is displayed on the LCD 1201.

  A panel controller (PANELC) 1209 controls the touch panel 1202 and the button device 1203 in response to a request from the CPU 1205. With this control, the CPU 1205 is notified of the pressing position of an indicator such as a finger or a stylus pen on the touch panel 1202, the pressed key code on the button device 1203, and the like. The pressed position information includes a coordinate value (hereinafter referred to as X coordinate) indicating the absolute position in the horizontal direction of the touch panel 1202 and a coordinate value (hereinafter referred to as Y coordinate) indicating the absolute position in the vertical direction. The touch panel 1202 can detect pressing of a plurality of points, and in this case, the CPU 1205 is notified of pressing position information for the number of pressing points.

  The power controller 1207 is connected to an external power source (not shown) and receives power. Thus, power is supplied to the entire portable information terminal 107 while charging the secondary battery 1212 connected to the power controller 1207. When power is not supplied from the external power source, power from the secondary battery 1212 is supplied to the entire portable information terminal 107.

  Based on the control of the CPU 1205, the wireless LAN module 1206 establishes wireless communication with a wireless LAN module on a wireless access point (not shown) connected to the LAN 102 and mediates communication with the portable information terminal 107. The wireless LAN module 1206 includes, for example, IEEE 802.11b.

  The timer 1213 generates a timer interrupt to a gesture event generation module 1301 described later based on the control of the CPU 1205. In addition, a geomagnetic sensor (not shown) and an acceleration sensor (not shown) are connected to the portable information terminal 107 to a bus (not shown). The timer 1213 detects the inclination of the portable information terminal 107 based on the control of the CPU 1205, and when the portable information terminal 107 obtains a predetermined inclination or more, changes the orientation of the portable information terminal 107 and causes the drawing module 1303 to be described later. A drawing instruction is sent to the LCD 1201. When the orientation of the portable information terminal 107 is changed, the CPU 1205 switches the width and height of the LCD 1201 and performs the subsequent processing.

  FIG. 13 is a block diagram showing a configuration of a software module that is executed by the CPU 1205 of the portable information terminal 107. The software module 1300 includes a gesture event generation module 1301, a gesture event processing module 1302, and a drawing module 1303. Hereinafter, each module will be described.

  The gesture event generation module 1301 receives various user touch inputs and generates various gesture events described later. The generated gesture event is sent to the gesture event processing module 1302. 14A to 14M show examples of gesture events that occur in the gesture event generation module 1301. FIG.

  FIG. 14A shows a touch pressing event, in which the latest coordinate value of touch coordinates and the number of touch coordinates are transmitted. The touch coordinates are the coordinates of one point where the user's finger touches the touch panel 1202 and have a set of coordinate values represented by the X coordinate and the Y coordinate. The number of touch coordinates indicates the number of touch coordinates where the user's finger touches the touch panel 1202. The touch coordinates are updated when the user's finger touches the touch panel 1202, when the finger moves, when the finger is released, or when an interrupt from the timer 1213 occurs.

  (B) is a swipe event, in which the movement distance calculated from the coordinate value of the latest touch coordinates and the difference between the latest and previous coordinate values is transmitted. Here, “swipe” refers to an operation of moving (sliding) in one direction while keeping the fingertip in contact with the touch panel 1202.

  (C) is a pinch-in event, which transmits the center coordinate value of the latest two touch coordinates and the pinch-in reduction rate calculated from the reduction distance of the straight line connecting the two touch coordinates. Here, pinch-in refers to an operation in which two fingertips are brought close to each other (pinch) while being in contact with the touch panel 1202.

  (D) is a pinch-out event, and the latest center coordinate value of the two touch coordinates and the expansion ratio of the pinch out calculated from the expansion distance of the straight line connecting the two touch coordinates are transmitted. Here, “pinch out” refers to an operation in which two fingertips are kept away from each other while touching the touch panel 1202 (like spreading fingers).

  (E) is a two-point swipe event, in which the coordinate values of the latest two touch coordinates and the movement distance calculated from the difference between the latest and previous coordinate values of the two touch coordinates are transmitted. A two-point swipe event occurs when the touch coordinates of two points are moving in the same direction.

  (F) is a rotation event, and the rotation center coordinate value calculated from the coordinate values of the latest two touch coordinates, and the rotation angle calculated from the latest and previous coordinate values of the two touch coordinates. Sent. Here, “rotate” refers to an operation of rotating two fingertips with respect to the touch panel 1202 while keeping the two fingertips in contact with the touch panel 1202.

  The same (g) is a flick event, and the coordinate value of the latest touch coordinates and the moving speed of the finger calculated from the latest and previous coordinate values are transmitted. Here, “flick” refers to an operation of releasing a finger during swiping (like repelling a finger).

  The same (h) is a touch release event, and the coordinate value and the number of coordinates of the latest touch coordinates when the user's finger is released from the touch panel 1202 are transmitted.

  (I) is a double tap event, and the coordinate value of the latest touch coordinate is transmitted. Here, the double tap means that a single tap event described later has occurred within a predetermined time.

  (J) is a single tap event, and the coordinate value of the latest touch coordinate is transmitted. Here, the single tap means that a touch release event has occurred within a predetermined time after the aforementioned touch pressing event.

  (K) is a long tap event, and the coordinate value of the latest touch coordinate is transmitted. Here, the long tap means that a touch release event has occurred after a predetermined time or more has elapsed after the touch pressing event described above.

  The same (l) is a touch and hold event, and the coordinate value of the latest touch coordinate is transmitted. Here, the touch and hold event means that a predetermined time or more has passed without moving even once the user's finger touches the touch panel 1202.

  The same (m) is a drag event, and the movement distance calculated from the coordinate value of the latest touch coordinate and the difference between the latest and previous coordinate values is transmitted. Here, “drag” refers to an operation of moving while keeping the fingertip in contact with the touch panel 1202. Swipe and drag are similar operations, but an operation of grabbing an object such as an icon or handle and moving it to another place is a drag, and a swipe is not specifying a touched part such as a screen scroll.

  Here, a case where a finger is used is shown as an example of the user's touch input, but the touch input may be input with a stylus pen or the like.

  Returning to the description of FIG.

  The gesture event processing module 1302 receives the gesture event, and executes a process according to each gesture event and the document structure described in the page image data.

  The drawing module 1303 draws page image data transmitted from the MFP 100 on the LCD 1201 in accordance with the execution result of the gesture event processing module 1302. A method of displaying page image data will be described later.

  Next, processing when page image data is received in the portable information terminal 107 will be described. FIG. 15 is a flowchart showing a flow of processing performed when the portable information terminal 107 receives page image data.

  In step 1501, portable information terminal 107 receives page image data from MFP 100 via wireless LAN module 1206. The received page image data is stored in the RAM 1211 in the object information management table. FIG. 16 is a diagram illustrating an example of the object information management table. The object information management table includes information (hereinafter referred to as object information) such as attribute, position coordinate X, coordinate Y, width W, height H, and storage location of a rectangular block (vector data, JPEG data) for each object. It is. In this embodiment, the attribute is given by a numerical value from 0 to 3, with 0 being a horizontally written character area rectangular block, 1 being a vertically written character area rectangular block, 2 being a photograph area rectangular block, and 3 being a graphic area rectangular block. Show. The attribute, position coordinate X, coordinate Y, width W, and height H are the same as the values managed by the block information shown in FIG. 5 described above, and these values are stored in the RAM 1211 when page image data is received. Saved.

  In step 1502, the CPU 1205 analyzes the syntax of the page image data stored in the RAM 1211 and reads the first page and objects included in the page.

  In step 1503, the CPU 1205 causes the drawing module 1303 to render all the background, character, photo, and graphic objects included in the read first page according to the coordinates, width, and height of the start point of the object. As a result, the display state of the touch UI 1204 is updated. FIG. 17 is an example of a display screen of the touch UI 1204 whose display state is updated. In FIG. 17, the display magnification of the first page 1700 is controlled in accordance with the width of the touch UI 1204. Further, when the height of the page when it is reduced in accordance with the width of the touch UI 1204 is smaller than the touch UI 1204, the start point of the first page 1700 is displayed in the coordinates on the touch UI 1204 so as to be displayed at the center of the touch UI 1204. Be controlled. If the height of the first page 1700 when it is reduced to the width of the touch UI 1204 is larger than the touch UI 1204, the start point of the first page 1700 is the start point of the touch UI 1204 (for example, the upper left corner of the screen) in the coordinates on the touch UI 1204. ). In this way, the display control method for displaying the entire page on the touch UI 1204 is referred to as “page display mode” in the present embodiment. In the page image data according to the present embodiment, the character object holds character code data together with vector data (see frame 1102 and frame 1103 in FIG. 11). Note that the drawing module 1303 in this embodiment uses only vector data for page rendering when there is a character object in the page.

  Next, operation control of page image data in the portable information terminal 107 will be described. FIG. 18 is a block diagram showing an internal module configuration related to operation control of page image data in the gesture event processing module 1302.

  The display change event processing module 1800 performs processing for a single tap event ((j) in FIG. 14) among the received gesture events. Specifically, when a single tap event is received, the coordinate values of the touch coordinates of the single tap event are set to the “mode switch” button 1701, the “next” button 1702, and the “previous” button 1703 (see FIG. 17). Determine if it is above any. When the touch coordinates of the single tap event are on the “mode switch” button 1701, a mode switch process described later is performed. If the touch coordinates are on the “next” button 1702, the next selection process (next button selection process) described later is performed. If the touch coordinates are on the “previous” button 1703, the process is performed before. Perform selection processing (previous button selection processing). The “next selection process” and the “previous selection process” are performed in the display order control module 1801 and the display range control module 1802 in the display change event processing module 1800.

  The swipe event processing module 1810 performs processing for the above-described swipe event ((b) in FIG. 14). Specifically, when the swipe event is received, the starting point of the page is moved at the coordinates on the touch UI 1204 according to the moving distance. Then, the display state of the touch UI 1204 is updated. Further, the movement range restriction module 1811 in the swipe event processing module 1810 performs a movement range restriction process described later to restrict the movement range of the page displayed on the touch UI 1204.

  The enlargement / reduction event processing module 1820 performs processing for the above-described pinch-in event (FIG. 14C) and pinch-out event (FIG. 14D). Specifically, when a pinch-in event or a pinch-out event is received, the character size at the time of rendering the displayed page is changed according to the reduction rate or enlargement rate of the two events, and the display state of the touch UI 1204 is then changed. Update. Further, the reduction-time display mode switching module 1821 in the enlargement / reduction event processing module 1820 performs a reduction-time display mode switching process, which will be described later, and an enlargement-time display mode switching module 1822, which will be described later.

  The object selection processing module 1830 performs processing for the above-described double tap event ((i) in FIG. 14). Specifically, when a double tap event is received, an object selection process described later is performed using the coordinate value of the touch coordinates of the double tap event. The object selection processing is controlled by the object selection processing module 1830 so as to operate only in the above-described page display mode.

  The scrap control module 1840 performs processing for the above-described long tap event ((k) in FIG. 14) and the drag event ((m) in FIG. 14). For example, when a long tap event is received in a partial area display mode described later, a “scrap” button described later is displayed on the enlarged display object. When a drag event is received after accepting a tap for the “scrap” button, a scrap area expansion process, which will be described later, is executed.

(Mode switching process)
Next, a mode switching process executed by the display change event processing module 1800 when the above-described “mode switching” button 1701 is tapped (designated) by the user will be described. FIG. 19 is a flowchart showing details of the mode switching process, which is executed in response to a mode switching button 1701 being tapped by the user.

In step 1901, the display change event processing module 1800 acquires the display mode being set in the portable information terminal 107 when the mode switching button 1701 is tapped. Here, the display mode is a mode for displaying page image data on the touch UI 1204, and in the present embodiment, has the following three display modes.
-Page display mode for displaying the entire page image for one page-Partial area display mode for enlarging and displaying a partial area (each object) in the page image for one page-Specified by the user Scrap Display Mode for Displaying Scrap Area As described above, the page display mode is set immediately after the mobile image terminal 107 receives the page image data.

  In the partial area display mode, the display magnification and start point of the page are controlled so that each object in the page is displayed in a large size. FIG. 20 is a diagram illustrating an example of a screen display of the touch UI 1204 in the partial area display mode. A screen displayed when the character object 2000 in the page image shown in FIG. 20A is selected as an object to be enlarged is shown in FIG. The broken line surrounding the object 2000 is drawn for easy understanding and does not exist on the actual page. Further, in this embodiment, a translucent mask 2001 in which the area of the object to be enlarged (character object 2000 in the example of FIG. 20) is transparent and the other areas are displayed in translucent gray is displayed on the page image. It is displayed overlaid on. By displaying such a semi-transparent mask in an overlapping manner, objects other than the object to be enlarged are displayed darkly, so that the object of interest is emphasized and the user specifies the area of the object to be enlarged. It becomes easy.

  The display change event processing module 1800 proceeds to step 1902 if the display mode being set when the mode switching button 1701 is tapped is the partial area display mode. On the other hand, if the display mode being set is the page display mode, the process proceeds to step 1904.

  In step 1902, the display change event processing module 1800 sets the above-described semi-transparent mask 2001 to the non-display setting (semi-transparent mask OFF) and switches to the page display mode.

  In step 1903, the display change event processing module 1800 controls the display magnification of the page according to the width of the touch UI 1204, and controls the start point of the page to determine the display range of the page.

  In step 1904, the display change event processing module 1800 switches the display mode to the partial area display mode and changes the setting to display the semi-transparent mask 2001 (semi-transparent mask ON).

  In step 1905, the display change event processing module 1800 reads the top object in the page, and acquires the start point, width, and height of the top object. Here, the head object is an object that is first read in the tree structure of page image data.

  In step 1906, the display range control module 1802 in the display change event processing module 1800 executes processing for determining the display range of the partial area (hereinafter, display range determination processing). FIG. 21 is a flowchart showing details of the display range determination process.

  In step 2101, the display range control module 1802 determines the attribute of the read object. If the attribute is a character, the process proceeds to step 2102; if the attribute is a table, the process proceeds to step 2112; otherwise, the process proceeds to step 2113.

  In step 2102, the display range control module 1802 determines whether or not the object to be displayed whose attribute is a character is a bullet. The bulleted object here refers to an object having a bullet character such as a dot or a number at the beginning of each character string or line. The bullet can be acquired from the OCR result. If the character object is not a bullet but a normal character string, the process proceeds to step 2103, and if it is a bullet, the process proceeds to step 2113.

  In step 2103, the display range control module 1802 obtains information on the text composition direction in the text object. Information on the direction of character combination is obtained in the vectorization process in step 304 described above.

  In step 2104, the display range control module 1802 determines whether the acquired character combination direction is vertical writing or horizontal writing. If the writing direction is vertical writing, the process proceeds to step 2105. If the writing direction is horizontal writing, the process proceeds to step 2106.

  In step 2105, the display range control module 1802 sets the display magnification of the page so that the height of the object falls within the height of the touch UI 1204.

  In step 2106, the display range control module 1802 sets the display magnification of the page so that the width of the object falls within the width of the touch UI 1204.

  In step 2107, the display range control module 1802 determines whether the entire object scaled according to the display magnification set in step 2105 or 2106 can be displayed on the touch UI 1204. If the entire object is larger than the touch UI 1204 and cannot be displayed, the process proceeds to step 2108. On the other hand, if the entire object is smaller than the touch UI 1204 and the entire object can be displayed, the process proceeds to step 2111.

  In step 2108, the display range control module 1802 determines the text composition direction of the object. If it is determined that the writing is vertical writing, the process proceeds to step 2109, and if it is determined that writing is horizontal writing, the process proceeds to step 2110.

  In step 2109, the display range control module 1802 determines that the entire object displays a vertically written character area that cannot be displayed on the touch UI 1204, and therefore sets the display position so that the first line of vertically written text is displayed. Specifically, the start point position of the page is set so that the upper right end of the object matches the upper right end of the touch UI 1204.

  In step 2110, the display range control module 1802 determines that the entire object displays a horizontally written character area that cannot be displayed on the touch UI 1204. Therefore, the display range control module 1802 sets the display position so that the horizontally written first line is displayed. Specifically, the starting point of the page is set so that the upper left corner of the object matches the upper left corner of the touch UI 1204.

  In step 2111, the display range control module 1802 determines that the entire object fits within the screen of the touch UI 1204. Therefore, the display range control module 1802 sets the start point of the page so that the center of the object matches the center of the touch UI 1204.

  If it is determined in step 2101 that the attribute of the object is a table, in step 2112, the display range control module 1802 detects the header position of the table. The position of the header is, for example, whether the font type of the characters in the first row (top row) and the first column (leftmost column) is bold, or the approximate curve of the vector data when vectorized It can be determined by the thickness, the thickness of the table ruled line, the background color of each cell of the table, and the like. If the header position of the table detected in step 2112 is the first row, the process proceeds to step 2106 via step 2104. If the header position of the table is the first column, the process proceeds to step 2105 via step 2104. Also, since the table header is generally present in the top row or the leftmost column, after step 2108, the page start point is set so that the top left corner of the object matches the top left corner of the touch UI. It is set (step 2110). In this way, the display position is set so that the header of the table is displayed.

  Further, when it is determined in step 2101 that the attribute of the object is an attribute other than a character / table, and in the case where it is determined in step 2102 that it is a bulleted item, it is as follows. That is, in step 2113, the display range control module 1802 sets the display magnification of the page so that the entire object can be accommodated in the touch UI 1204. In step 2111, the starting point of the page is set so that the center of the object matches the center of the touch UI 1204.

  As described above, in the display range determination process, the display magnification and start point of the page are controlled according to the attribute of the object read in step 1905, and the display range of the partial area displayed on the touch UI 1204 is determined. At this time, since the gray semi-transparent mask is applied to the region other than the partial region of the object to be displayed, the user can easily identify the object to be displayed.

  Returning to the flowchart of FIG.

  In step 1907, the display change event processing module 1800 updates the display state of the touch UI 1204 based on the page display range determined in step 1903 or step 1906.

  The above is the content of the mode switching process.

(Next selection process)
Next, the “next selection process (next button selection process)” executed by the display change event processing module 1800 when the “next” button 1702 is tapped (designated) by the user will be described. FIG. 22 is a flowchart showing details of the next selection process.

  In Step 2201, the display change event processing module 1800 acquires the display mode set in the portable information terminal 107 when the “Next” button 1702 is tapped. If the acquired display mode is the partial area display mode, the process proceeds to step 2202, and if the acquired display mode is the page display mode, the process proceeds to step 2206.

  In step 2202, the display order control module 1801 selects an object to be displayed next based on a tree structure (see FIG. 10) from all objects on the currently read page, and reads the selected object. In the present embodiment, the display order on the document tree structure when the “next” button 1702 is designated is the order of the first upper layer object and the objects belonging to the lower layer of the object in the document tree structure. . Furthermore, after all the display of the objects in the lower hierarchy is finished, it is assumed that the order is the next object in the upper hierarchy, the object in the lower hierarchy. For example, in FIG. 10 described above, in the case of page V1, the object that is read first is T1. When the “next selection process” is performed in a state where the object of T1 is displayed on the touch UI 1204 in the partial area display mode, T1 has no lower layer, and therefore W1 of the next higher layer is called. Since W1 has a lower layer, the object of S1 is read, and T2 of the lower layer is further read. If the “next selection process” is performed while the object of T2 is displayed, T3 is read because T3 exists in the same hierarchy. When the “next selection process” is performed in a state where the object of T3 is displayed, T3 has no lower layer and the next object does not exist in the same layer, so that S2 of the next higher layer is read. In this embodiment, W1 and W2 having no drawing elements are selected and read. However, only T1 and T2 having drawing elements may be selected. Further, for example, it is possible to select only an object having a specific attribute such as only a character attribute object, or to exclude and select only a specific attribute. Furthermore, it is also possible to select only objects having drawing elements in lower layers, such as W1 and V1, which do not have drawing elements alone.

  Note that the order determined by the above tree structure may be a configuration in which an object to be read next is determined while referring to the object information management table (see FIG. 16) once recorded. good.

  In step 2203, the display order control module 1801 determines whether or not the next object has been successfully read in step 2202. If the next object can be read without any problem (if there is a next object that can be selected), the process proceeds to step 2204 with the read object as a processing target. On the other hand, if loading of the next object fails (when there is no next object that can be selected), the last object in the page has already been loaded, and display processing of all objects in the page is completed. The process proceeds to step 2205.

  In step 2204, the display range control module 1802 executes the above-described display range determination process for the read object, and determines the display range of the partial area displayed on the touch UI 1204.

  In step 2205, the display range control module 1802 hides the semi-transparent mask and switches the display mode from the partial area display mode to the page display mode.

  In step 2206, the display order control module 1801 analyzes the syntax of the page image data stored in the RAM 1211, and reads the next page and objects included in the page.

  In step 2207, the display order control module 1801 determines whether the next page has been successfully read. If the next page has been successfully read, the process proceeds to step 2209. On the other hand, when the next page could not be read, the last page of the page image data already stored in the RAM 1211 has been read. In this case, the process proceeds to Step 2208.

  In step 2208, the display order control module 1801 analyzes the syntax of the page image data stored in the RAM 1211 and reads the first page and the objects included in the page.

  In step 2209, as described above, the display range control module 1802 controls the display magnification of the page according to the width of the touch UI 1204, and controls the start point of the page to determine the display range of the page.

  In step 2210, the display range control module 1802 updates the display state of the touch UI 1204 based on the partial region display range determined in step 2204 or the page display range determined in step 2209.

  The above is the content of the next selection process.

(Pre-selection process)
Next, the “previous selection process (previous button selection process)” executed by the display change event processing module 1800 when the “previous” button 1703 is tapped (designated) by the user will be described. FIG. 23 is a flowchart showing details of the pre-selection process. Since the configuration is almost the same as that of the above-described next selection process, different points will be mainly described here.

  In step 2301, the display change event processing module 1800 acquires the display mode set in the portable information terminal 107 when the “Previous” button 1703 is tapped. If the acquired display mode is the partial area display mode, the process proceeds to step 2302, and if the acquired display mode is the page display mode, the process proceeds to step 2311.

  In step 2302, the display order control module 1801 selects an object to be displayed next from all objects on the currently read page based on the tree structure, and reads the selected object. In this embodiment, when the “Previous” button 1703 is designated, the display order on the document tree structure is such that the lower hierarchy object at the end of the document tree structure and the higher hierarchy object of the object are lower hierarchy. It is assumed that the order is from to the upper hierarchy. In other words, after all the display of the objects in the lower layer is finished, the process proceeds to display of other objects in the upper layer. For example, in FIG. 10 described above, in the case of page V1, the first read object is T8 which is the last lower layer object. When the “previous selection process” is performed in a state where the T8 object is displayed on the touch UI 1204 in the partial area display mode, the T8 does not have a lower hierarchy and the W2 exists in the same hierarchy. Is read. Since W2 has T7 in the lower layer, T7 is read. Further, when the “pre-selection process” is performed with the T7 object being displayed, T6 is read because T6 exists in the same hierarchy. When the “pre-selection process” is performed while the object of T6 is displayed, since no other object exists in the same hierarchy, W1 that is the upper hierarchy is read. When the “pre-selection process” is performed while the object of W1 is displayed, since W1 has S2 in the lower hierarchy and S2 has the object in the lower hierarchy, T5 at the end of the lower hierarchy of S2 The object is loaded. In the “previous selection process”, as in the “next selection process”, only an object having a drawing element may be selected, or only an object having a specific attribute may be selected. It is also possible to select only the attribute. Furthermore, it is also possible to select only objects having drawing elements in lower layers, such as W1 and V1, which do not have drawing elements alone.

  Note that the order determined by the above tree structure may be a configuration in which an object to be read next is determined while referring to the object information management table (see FIG. 16) once recorded. good.

  In step 2303, the display order control module 1801 determines whether the previous object has been successfully read in step 2302. If the previous object can be read without any problem (if there is a selectable previous object), the process proceeds to step 2304 with the read object as a processing target. On the other hand, when reading of the previous object fails (when there is no previous object that can be selected), the first object in the page has already been read, and display processing of all objects in the page is completed. The process proceeds to step 2305.

  In step 2304, the display range control module 1802 performs the above-described display range determination process for the read object, and determines the display range of the partial area displayed on the touch UI 1204.

  In step 2305, the display range control module 1802 hides the semi-transparent mask and switches the display mode from the partial area display mode to the page display mode.

  In step 2306, the display order control module 1801 analyzes the syntax of the page image data stored in the RAM 1211 and reads the previous page and the objects included in the page.

  In step 2307, the display order control module 1801 determines whether the previous page has been successfully read. If the previous page has been successfully read, the process proceeds to step 2309. On the other hand, if the previous page could not be read, the first page of the page image data already stored in the RAM 1211 has been read. In this case, the process proceeds to Step 2308.

  In step 2308, the display order control module 1801 analyzes the syntax of the page image data stored in the RAM 1211 and reads the last page and the objects included in the page.

  In step 2309, as described above, the display range control module 1802 controls the display magnification of the page in accordance with the width of the touch UI 1204, and controls the start point of the page to determine the display range of the page.

  In step 2210, the display range control module 1802 updates the display state of the touch UI 1204 based on the partial region display range determined in step 2304 or the page display range determined in step 2309.

  The above is the content of the pre-selection process.

(Scrap function)
Next, the scrap function will be described. As described above, the scrap function is a function that manages a designated area in the page image as a scrap area and enables browsing of the scrap area. This scrap function is roughly divided into a process for selecting a scrap area and a process for displaying the selected scrap area (scrapbook).

(Scrap area setting process)
First, the scrap area setting process will be described. Scrap area setting processing is easy on a mobile information terminal that has only a small touch panel, displaying one or more objects as a scrap area while displaying the partial areas of the page image one by one at an appropriate display magnification. This is a process for setting with a simple operation.

  FIG. 24 is a flowchart showing the flow of the scrap area setting process.

  In step 2401, the display change event processing module 1800 acquires the display mode set when the “next” button 1702 or the “previous” button 1703 is tapped. If the display mode being set is the partial area display mode, the process proceeds to step 2402. On the other hand, when the display mode being set is the page display mode, the present process is exited.

  In step 2402, the display order control module 1801 reads an object to be displayed from among all objects currently read. Here, it is assumed that the object T5 in the page shown in FIG. 9A is read as an object to be displayed.

  In step 2403, the display range control module 1802 performs the above-described display range determination processing (step 1906) for the read object, and determines the display range of the partial area displayed on the touch UI 1204.

  In step 2404, the display change event processing module 1800 updates the display state of the touch UI 1204 based on the determined display range.

  In step 2405, the display change event processing module 1800 determines a tap event to be made during the enlarged display of the object. When a tap event for the “next” button 1702 or the “previous” button ”1703 is received, the process returns to step 2402 to perform the above-described display range determination process for the corresponding object. On the other hand, if a long tap event has been received, the process proceeds to step 2406.

  In step 2406, the scrap control module 1840 displays a “scrap” button 2501. FIG. 25A shows a state in which the “scrap” button 2501 is displayed when the object T5 in FIG. 9 is selected as the object 2500 to be displayed.

  When the display change event processing module 1800 receives a tap on the “scrap” button 2501, in step 2406, the scrap control module 1840 displays a mark for designating the range of the scrap area so as to overlap the object 2500 to be enlarged and displayed. To do. In addition, the display change event processing module 1800 switches the “mode switching” button 1701 being displayed to a “decision button” 2510 for designating a scrap area. Hereinafter, a state in which the “scrap” button is displayed on the object is referred to as a “scrap selection state”. A scrap area indicated by a broken line frame 2502 in FIG. 25B is an area stored as an area in which the user is interested in the page image data (vector data, JPEG data) displayed on the touch UI 1204. is there. In the present embodiment, a frame indicating the extension of the scrap area is used as the mark. However, a mark other than the frame, such as square brackets indicating four corners, may be used. A frame 2502 indicating a scrap area has handles 2503 and 2504 for changing the range. The positions of the handles 2503 and 2504 can be switched depending on the character direction. FIG. 26 is a flowchart showing details of processing for displaying a frame indicating a scrap area on the touch UI 1204 (hereinafter, scrap area display processing).

  When a tap on the “scrap” button 2501 is detected, in step 2601, the scrap control module 1840 determines the attribute of the object being displayed. If the attribute of the displayed object is a character, the process proceeds to step 2602. On the other hand, if the attribute of the object being displayed is an image or a graphic, the process proceeds to step 2605.

  In step 2602, the scrap control module 1840 determines whether the displayed character object is horizontal writing or vertical writing. If the direction of the character object is horizontal writing, the process proceeds to step 2603. On the other hand, if it is vertical writing, the process proceeds to step 2604.

  In step 2603, as shown in FIG. 25B, the scrap control module 1840 displays a frame 2502 indicating a scrap area provided with handles 2503 and 2504 above and below the object 2500 being displayed. . The user can change the vertical width 2505 of the scrap area by dragging the handle 2503/2504 in the directions of arrows 2507 and 2508 (see FIG. 25C). That is, the handle 2503/2504 can be dragged only in the vertical direction, and the width 2506 in the horizontal direction is fixed at this time. The variable width 2505 in the height direction is referred to as a “selection width”.

  In step 2604, the scrap control module 1840 displays a frame (not shown) indicating a scrap selection area with handles attached to the left and right of the frame on the displayed object. Similarly to the case where the character object is written horizontally, the horizontal selection width of the scrap area can be changed by dragging the left and right handles. In this case, the left and right handles can be dragged only in the horizontal direction, and the vertical width 2505 is fixed.

  In step 2605, the scrap control module 1840 determines the character direction of the photographic object / graphic object being displayed. In this case, since the character direction cannot be determined like a character object, an object having a character attribute with the closest display order is acquired from the objects before and after the character object, and the character direction of the character object is determined. If the character direction of the character object is horizontal writing, the process proceeds to step 2606. If the character direction of the character object is vertical writing, the process proceeds to step 2607. If there is no character object before or after the object being displayed (if all objects in the page image data are images or graphics), the process advances to step 2608.

  In step 2606, the scrap control module 1840 displays a frame indicating a scrap area with handles attached vertically on the object being displayed.

  In step 2607, the scrap control module 1840 displays a frame indicating a scrap area with handles on the left and right in an overlapping manner with the object being displayed.

  In step 2608, the scrap control module 1840 displays a frame indicating a scrap area in which a handle is given at a predetermined position (for example, up and down) in a superimposed manner on the displayed object.

  In this way, when the “scrap” button 2501 is tapped, a frame indicating a scrap area having a handle vertically or horizontally is displayed so as to overlap the object being displayed.

  Returning to the flowchart of FIG.

  In step 2407, the display change event processing module 1800 determines the next content of the user operation. If the accepted operation is an instruction to change the scrap area (drag event for the handle 2503/2504), the process proceeds to step 2408. On the other hand, if the accepted operation is an instruction to determine a scrap area (a tap event to the “OK” button 1711), the process proceeds to step 2411.

  In step 2408, the scrap control module 1840 determines whether or not the handle has moved beyond the boundary of the object being magnified by a drag operation. If it is determined that the movement destination (positional coordinates) of the handle exceeds the boundary of the object being magnified, the process proceeds to step 2409. For example, in FIG. 25C, it is assumed that the handle 2504 is moved to the position 2509 by being dragged in the direction of the arrow 2507. As described above, when the handle by the drag operation moves beyond the boundary of the displayed object, the process proceeds to step 2409. On the other hand, if it is determined that the handle has not moved beyond the boundary of the enlarged object, the process proceeds to step 2410 to update the display state. In this case, the frame 2502 indicating the scrap area is only changed within the range of the object being magnified. After the update process, the process returns to step 2407 to wait for the next user operation.

  In step 2409, the scrap control module 1840 performs a process of extending the scrap area based on the direction in which the handle of the frame indicating the scrap area exceeds the boundary of the object being enlarged. The scrap area expansion process is a partial area display mode in which partial areas (objects) of a page image are displayed one by one at an appropriate display magnification. In order to select a scrap area while expanding it into a plurality of partial areas, a necessary object is selected. It is a process of reading. With this process, the scrap area can be expanded with a simple operation. FIG. 27 is a flowchart showing details of the scrap area expansion processing.

  In step 2701, the display change event processing module 1800 determines which handle has moved. If the handle related to the received drag event is one of 2503/2504 given above and below the frame 2502 indicating the scrap region, the process proceeds to step 2702. On the other hand, if the handle related to the received drag event is one of the handles (not shown) attached to the left and right of the frame 2502 indicating the scrap area, the process proceeds to step 2705.

  In step 2702, the scrap control module 1840 determines which border of the displayed object the dragged handle has moved. If the destination of the dragged handle has exceeded the lower boundary of the object 2500 being magnified, the process proceeds to step 2703. On the other hand, if the destination of the dragged handle exceeds the upper boundary of the object 2500 that is being magnified, the process proceeds to step 2704.

  In step 2703, the scrap control module 1840 reads the next object in the display order according to the tree structure, with the object being enlarged displayed as a reference. For example, if the object being magnified is the object T5 in FIG. 9, the object T6 whose display order is next to T5 is read according to the tree structure in FIG. The method for determining the display order according to the tree structure is as described in the flowchart of FIG. When the next object in the display order is read, the process proceeds to step 2708.

  In step 2704, the scrap control module 1840 reads the object in the previous display order based on the object being enlarged and displayed according to the tree structure. For example, if the object being magnified is the T7 object in FIG. 9, the T6 object whose display order is before T7 is read according to the tree structure of FIG. The method of determining the display order from the tree structure is as described in the flowchart of FIG. When the object with the previous display order is read, the process proceeds to step 2708.

  In step 2705, the scrap control module 1840 determines which boundary of the displayed object the dragged handle has moved. If the destination of the dragged handle has exceeded the left boundary of the object 2500 that is being magnified, the process advances to step 2706. On the other hand, if the destination of the dragged handle exceeds the right boundary of the enlarged object 2500, the process proceeds to step 2707.

  In step 2706, the scrap control module 1840 reads the next object in the display order according to the tree structure, based on the object being enlarged. When the next object in the display order is read, the process proceeds to step 2708.

  In step 2707, the scrap control module 1840 reads the object in the previous display order based on the object being enlarged and displayed according to the tree structure. When the object with the previous display order is read, the process proceeds to step 2708.

  In step 2708, the scrap control module 1840 performs display range determination processing on the read object, and determines the display range of the partial area displayed on the touch UI 1204. When the display range of the partial area is determined, the process proceeds to step 2709.

  In step 2709, the scrap control module 1840 sets a new frame in which the expanded area including the object newly read in accordance with the movement destination of the handle is used as the scrap area in the display range of the partial area determined in step 2708. Display overlaid on top. FIG. 25D shows a display screen on the touch UI 1204 when the handle 2504 is dragged beyond the lower boundary of the object 2500 in FIG. 25D, the object located at the lower left of the object 2500 in FIG. 25C is highlighted as an object to be expanded, and a new frame 2511 with the lower limit of the movement destination of the handle 2504 is displayed in an overlapping manner. Has been. In FIG. 25 (d), the reason why the handle does not exist on the upper side of the new frame 2511 is that the scrap area continues from the object 2500 existing on it (that is, the scrap area is expanded from the object 2500). It represents. In FIG. 25D, the scrap area is continued depending on the presence or absence of the handle, and the object 2500 is displayed in a semi-transparent mask, but it can be seen that it is included in the scrap area. If it is. For example, the object 2500 may be displayed as it is without mask display, or may be displayed in a different mode (for example, a hatched line) so that the object 2500 can be recognized as the original scrap area.

  The above is the content of the scrap area expansion process.

  Returning to the flowchart of FIG.

  After completion of the scrap area expansion processing, in step 2410, the scrap control module 1840 updates the display state. By repeating steps 2407 to 2410 in this way, the width and / or height of the scrap area is changed (expanded). Thus, the user can easily select an area including a plurality of objects having different sizes (widths) as a scrap area.

  In step 2411, the scrap control module 1840 stores one or more objects included in the range designated as the scrap area.

  The above is the content of the scrap area setting process.

(Scrap area storage / management)
FIG. 28 shows a scrap area management table as an example of a format for storing scrap areas. The scrap area management table shown in FIG. 28 includes an identifier for specifying a scrap area, link information, an offset, and a selection width. These pieces of information are used to display a scrap image in a scrapbook, which will be described later, at the same display magnification and range as when the scrap area was selected. Hereinafter, each piece of information included in the scrap area management table will be described.

  The identifier is information for uniquely identifying the scrap area to be stored. Now, “scrap 2” is assigned as an identifier for the scrap area composed of the areas surrounded by the frames 2502 and 2511 shown in FIGS. 25C and 25D.

  The link information is link information to the original object selected as the scrap area, and indicates each object managed in the object information management table of FIG. With this link information, it is possible to refer to the object information targeted for the scrap area from the scrap area management table of FIG. In the present embodiment, object data (JPEG data, vector data) is not copied as a scrap area, but is referenced using link information to display a scrap image in a scrapbook described later. That is, in addition to the object data, the width W and the height H managed by the object information management table are also referred to and used for the display magnification in the scrapbook described later. Thus, there is an effect of suppressing consumption of the RAM 1911 of the portable information terminal 107 by not copying the object data for the scrap image. In FIG. 28, the identifier of “scrap 2” is a scrap area including the objects T4 and T5 in FIG. 9, and the identifiers T4 and T5 of each object are stored as link information.

  The offset indicates a position coordinate at which selection of an object that is a target of the scrap area is started. For example, in the case of a scrap area specified by the identifier of “scrap 2” described above, coordinates indicating start positions 2510 and 2512 when objects T4 and T5 are selected are stored.

  The selection width indicates the selected width of the object that is the target of the scrap area. For example, in the case of the scrap area specified by the identifier of “scrap 2”, the values of the vertical widths 2505 and 2513 of the objects T4 and T5 are stored.

  The object data traced with reference to the link information as described above is displayed at the display magnification determined by the display range determination process described above, and by specifying the offset and selection width for it, a scrap image can be selected. Can be displayed with the same display magnification and range.

  When the storage of the object in the range designated as the scrap area is finished, the display change event processing module 1800 ends this processing, and switches the button to be displayed from the “OK” button 1711 to the “mode switching” button 1701.

  The above is the content of the scrap area selection process. Thereby, in a portable information terminal having only a small touch panel, a plurality of objects can be selected as a scrap area by a simple operation while displaying a part of a page image one by one at an appropriate display magnification and checking the contents. it can. A plurality of objects in which the display order is continuous can be selected as a scrap area with a simple operation even if their sizes are different. Further, the selected scrap area is stored and managed together with information for displaying an appropriate enlargement ratio for each scrap area in the scrapbook described below.

(scrapbook)
Next, display processing by a scrapbook (hereinafter, scrap display processing) will be described. In the scrap display process, it is possible to display at an appropriate enlargement ratio for each scrap area selected in the above-described scrap area selection process.

  FIG. 29A is a diagram showing an example of a menu screen in the portable information terminal 107, and a “document list” button 2901 and a “scrapbook” button 2902 are displayed on the menu screen 2900. When the “document list” button 2901 is tapped, a document list screen 2910 shown in FIG. 29B is displayed. A document list screen 2910 displays a list of document names (file names) of page image data received by the portable information terminal 107. When a document name in the list is selected and tapped, the corresponding page image data is displayed on the touch UI 1204 as shown in FIG. When the “scrapbook” button 2902 is tapped on the menu screen 2900, a scrapbook screen as shown in FIG. 29C is displayed on the touch UI 1204.

  FIG. 30 is a flowchart showing the flow of the scrap display process.

  In step 3001, when the display change event processing module 1800 receives a tap operation on the “scrapbook” button 2902, the display change event processing module 1800 sets the display mode to the scrap display mode.

  In step 3002, the scrap control module 1840 accesses the above-described scrap area management table and reads information on each scrap area (hereinafter, scrap area information). Here, the scrap area information refers to information such as link information, offset, and selection width to object data that is the target of each scrap area.

  When the information of each scrap area is read, in step 3003, the scrap control module 1840 executes display range determination processing for each scrap area, and determines the display magnification of the objects constituting each scrap area. Specifically, first, using the read link information, information such as attributes relating to objects constituting the scrap area is read from the object information management table (FIG. 16). Then, using the read object data, attribute, width W, and height H information, the display range determination process described above is executed to determine the display magnification of the object.

  When the display magnification of the objects constituting the scrap area is determined, in step 3004, the scrap control module 1840 executes clip processing for each scrap area. Specifically, the range specified by the offset and selection width read in step 3002 is clipped on the objects constituting each scrap area read in step 3003. Thereby, the same range as the range selected in the above-described scrap area setting process is displayed on the scrapbook. Looking at (c) of FIG. 29, it can be seen that the regions 2921 and 2922 correspond to the regions surrounded by the frames 2502 and 2511 in FIGS. 25 (b) and 25 (c), respectively.

  In step 3005, the scrap control module 1840 updates the display state of the touch UI 1204.

  The above is the content of the scrap display process. As a result, the scrap area selected in the scrap area selection process can be displayed at the same appropriate enlargement ratio as at the time of selection.

  As described above, according to the present embodiment, even a portable information terminal having only a relatively small touch panel device can select a plurality of objects having different sizes as one scrap area by a simple operation. Further, the selected scrap area can be displayed at the same appropriate enlargement ratio as that at the time of selection. In addition, although this invention represented by the Example mentioned above is suitable for the portable information terminal of a comparatively small touchscreen device, the application range of this invention is not limited to this. The present invention can be applied regardless of the size of the touch panel.

  Next, an embodiment in which a scrap area can be selected more flexibly so as to cope with a case where an area different from the user's intention is included in the scrap area will be described as a second embodiment.

  Since only a part of the scrap area expansion processing is different from the first embodiment, the following description will focus on the differences.

  In the present embodiment, an example will be described in which the objects T7 and T11 whose display order according to the tree structure (FIG. 10) is discontinuous in the page image data (document) shown in FIG. 9 are selected as one scrap area. It shall be.

  In the tree structure shown in FIG. 10, the display order of the objects is T7, T8, and T11. In this embodiment, first, these three consecutive objects are once selected as a scrap area. After that, by removing the T8 object having different attributes from the scrap area, a scrap area composed of T7 and T11 is finally designated. FIG. 31 is a flowchart showing the flow of scrap area expansion processing according to the present embodiment for realizing such processing.

  In step 3101, the scrap control module 1840 acquires attribute information of the object being enlarged and displayed. FIG. 32A shows a state in which the above-described “scrap” button is pressed in a situation where the object T7 in FIG. 9 is selected as the display target object 3200, and a frame 3201 indicating a scrap area is displayed in an overlapping manner. Show. In this case, the above-described object information management table is referred to, and “horizontal writing character” information is acquired as attribute information of the object T7.

  In step 3102, the display change event processing module 1800 determines the type of gesture event generated by the user's touch operation, and distributes subsequent processing. In this embodiment, if the content of the touch operation is a drag event, the process proceeds to step 3103. On the other hand, if the content of the touch operation is a swipe event, the process proceeds to step 3116.

  In step 3103, the display change event processing module 1800 determines which handle is the handle related to the drag event. If the handle related to the drag event is one of the handles provided above and below the frame indicating the scrap region, the process proceeds to step 3104. On the other hand, if the handle related to the drag event is one of the handles (not shown) attached to the left and right of the frame 3201 indicating the scrap area, the process proceeds to step 3107.

  In step 3104, the scrap control module 1840 determines which boundary of the displayed object the dragged handle has moved. If the destination of the dragged handle exceeds the lower boundary of the enlarged object, the process proceeds to step 3105. On the other hand, if the destination of the dragged handle exceeds the upper boundary of the object being enlarged, the process proceeds to step 3106. In FIG. 32A, since the handle relating to the drag event is the handle 3204 provided below the frame 3201 indicating the slap region, and has moved beyond the lower boundary of the object, the process proceeds to step 3105. It will be.

  In step 3105, the scrap control module 1840 obtains the object of the next object in the display order and the attribute information thereof based on the object being enlarged according to the tree structure. In the case of FIG. 32A, the object next to the object T7 is T8 according to the tree structure shown in FIG. Therefore, the object information of the object T8 in the object information management table is referred to, and “photograph” is acquired as the attribute information together with the object T8.

  In step 3106, the scrap control module 1840 obtains the object of the object whose display order is the previous and the attribute information thereof based on the object being enlarged according to the tree structure.

  In step 3107, the scrap control module 1840 determines which boundary of the displayed object the dragged handle has moved. If the destination of the dragged handle exceeds the left boundary of the object being magnified, the process proceeds to step 3108. On the other hand, if the destination of the dragged handle exceeds the right boundary of the object being enlarged, the process proceeds to step 3109.

  In step 3108, the scrap control module 1840 obtains the object of the next object in the display order and the attribute information thereof based on the object being magnified according to the tree structure.

  In step 3109, the scrap control module 1840 obtains the object of the object whose display order is the previous and its attribute information based on the object being enlarged and displayed according to the tree structure.

  In step 3110, the scrap control module 1840 obtains the expansion condition of the scrap selection area. Here, the expansion condition is a standard that defines what kind of object is allowed to be expanded with respect to the object being displayed. For example, it is assumed that the default extension condition is such that a plurality of objects having the same display order according to the tree structure have the same attribute when the attributes are the same.

  In step 3111, the scrap control module 1840 compares the attribute of the object being displayed with the attribute of the object acquired in any of steps 3105, 3106, 3108, and 3109, and determines whether the extension condition is met. To do. By this determination processing, it is possible to block the extension of the scrap area that does not meet a predetermined expansion condition (in this case, whether the attributes are the same between the objects whose display order is continuous). As a result of the determination, if the extension condition is met, the process proceeds to step 3112. On the other hand, if the extension condition is not met, the process proceeds to step 3114. In FIG. 32A, if the extended condition is the default described above, the object T7 being displayed has the character attribute, while the next object T8 read in step 3105 has the photograph attribute. Therefore, the process proceeds to step 3114.

  In step 3112, the scrap control module 1840 performs display range determination processing on the new object acquired in any of steps 3105, 3106, 3108, and 3109 to determine a display range to be displayed on the touch UI 1204.

  In step 3113, the scrap control module 1840 displays a frame indicating that the scrap area has been extended to the determined display range, overlaid on the newly acquired object.

  In step 3114, the display change event processing module 1800 determines whether a predetermined user operation (here, a long tap) has been performed at the position (coordinates) where the handle is dragged. This is a process for allowing an exceptional expansion process of the scrap area even when the predetermined expansion condition is not met. Thereby, it becomes possible to respond flexibly to a user's request. If a long tap has been made, the process proceeds to step 3115. On the other hand, if a long tap has not been made, the process is exited. In FIG. 32A, when a long tap for extending the scrap area to the subsequent object T8 is made at the dragged position (coordinates) 3205, the process proceeds to step 3115.

  In step 3115, the scrap control module 1840 temporarily invalidates the expansion condition so that the new object acquired in any of steps 3105, 3106, 3108, and 3109 can be expanded. Specifically, from the content of allowing expansion when the attributes of multiple objects in the display order are the same, to the content of allowing expansion if there are multiple objects that are continuous regardless of the attribute difference. The extension condition is temporarily changed. For example, when a long tap is performed in the state of FIG. 32A, the expansion condition is temporarily changed from the default state to a state in which expansion is allowed for a plurality of continuous objects regardless of differences in attributes. Will be. As a result, the scrap area is extended to T8 objects having different attributes. Instead of temporarily disabling the extended condition, the extended condition may be updated to the changed content, and the extended condition of the changed content may be applied to the subsequent processing.

  By repeating the processing so far, three objects T7, T8, and T11 in which the display order is continuous are once selected as a scrap area. FIG. 32B is a diagram showing a state where the scrap area is extended to the object T8, and FIG. 32C is a diagram showing a state where the scrap area is further extended to the object T11. Thereafter, an object of T8 having a different attribute is excluded from the scrap area by each processing of steps 3116 to 3118 described later.

  A user who wants to exclude a specific object from the selected scrap area first displays the object to be excluded on the touch UI 1204 and performs a swipe operation. As a result, it is determined in step 3102 that a swipe event touch operation has been performed, and the process proceeds to step 3116.

  In step 3116, the display change event processing module 1800 displays a “release” button that is a button for excluding the object being displayed from the scrap area. FIG. 32D shows a state in which a “release” button 3206 is displayed on the T8 object to be excluded.

  In step 3117, the display change event processing module 1800 determines whether a tap operation on the “release” button displayed on the displayed object has been performed within a predetermined time. If a tap operation on the “release” button is received within a predetermined time, the process proceeds to step 3118. On the other hand, if the tap operation on the “release” button is not performed within a predetermined time, the process proceeds to step 3119.

  In step 3118, the scrap control module 1840 cancels the setting of the scrap area designated for the object T8, and exits this process.

  In step 3118, the scrap control module 1840 deletes the “release” button displayed on the displayed object, and exits the present process.

  Through the processing described above, a specific object can be deleted from an area once selected as a scrap area, and a new scrap area can be easily selected. FIG. 33 shows a state in which the scrap area is displayed in the scrapbook when only the objects T7 and T11 whose display order is finally discontinuous are selected and stored as one scrap area by the above processing. FIG. It can be seen that the scrap area 3300 includes only character objects T7 and T11, and the photo object of T8 is excluded.

  As described above, according to the present embodiment, even a portable information terminal having only a relatively small touch panel device can more flexibly specify and display a scrap area in consideration of object attributes.

(Other examples)
The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (14)

An information processing apparatus having a display unit that has a scrap function that manages a designated area in a page image as a scrap area and enables browsing of the scrap area,
Display range determining means for determining a display magnification according to the size of the display means for each object specified by the user among the objects included in the page image;
Scrap area setting means for setting, as the scrap area, an area composed of one or more objects for which the display magnification is determined;
Management means for managing the scrap area set by the scrap area setting means;
When displaying the scrap area managed by the management means on the display means, control is performed so that one or a plurality of objects constituting the scrap area are displayed at a display magnification determined by the display range determination means. An information processing apparatus comprising display control means.   The information processing apparatus according to claim 1, wherein a method of determining the display magnification in the display range determining unit is different depending on an attribute of the object.   The scrap area setting means displays a mark for designating a range of the scrap area so as to be superimposed on an object being displayed on the display means, and the display magnification is determined based on a user operation on the mark. The information processing apparatus according to claim 1, wherein a plurality of objects are set as the scrap area.   The said mark is a frame which shows the extension of the said scrap area | region, The user operation with respect to the said mark is operation which moves to the up-down or left-right direction with respect to the handle attached | subjected to the said frame. Information processing device.   The information processing apparatus according to claim 4, wherein a position of the handle is different depending on an attribute of an object displayed on the display unit.   The scrap area setting means, when a user operation on the mark is performed beyond the boundary of the object being displayed on the display means, displays another object continuous with the object being displayed on the display means, The information processing apparatus according to claim 3, wherein the area is expanded and set so that a scrap area is configured by a plurality of objects including the displayed other objects. The page image data has a tree structure,
The information processing apparatus according to claim 6, wherein another object continuous with the object being displayed is determined by a display order according to the tree structure.   The management means manages the scrap area set by the scrap area setting means by a management table including at least an identifier for specifying the scrap area and link information to objects constituting the scrap area. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized.   The scrap area setting means displays another object that is continuous with the object being displayed on the display means when the expansion condition that specifies whether or not to allow the expansion is satisfied, and displays the displayed different The information processing apparatus according to claim 6, wherein the scrap area is extended and set so that the scrap area includes a plurality of objects including the object.   10. The information processing apparatus according to claim 9, wherein the extended condition is that an attribute of the continuous object is the same as an attribute of the object being displayed.   The scrap area setting means extends the scrap area by invalidating the expansion condition when a predetermined user operation is performed even when an expansion condition for specifying whether or not the expansion is permitted is not satisfied. The information processing apparatus according to claim 9 or 10.   12. The apparatus according to claim 1, further comprising means for excluding a specific object from a plurality of objects constituting the scrap area for the scrap area set by the scrap area setting means. The information processing apparatus according to claim 1. A method in an information processing apparatus having a display unit that includes a scrap function that manages a designated area in a page image as a scrap area and enables browsing of the scrap area,
A display range determination step for determining a display magnification according to the size of the display means for each object specified by the user among the objects included in the page image;
A scrap area setting step for setting, as the scrap area, an area composed of one or a plurality of objects for which the display magnification is determined;
A management step for managing the scrap area set in the scrap area setting step;
When displaying the scrap area managed in the management step on the display means, control is performed so that one or more objects constituting the scrap area are displayed at the display magnification determined by the display range determining means. A display control step.   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 12.