JP7841830B2 - Modifying digital scripts using generative adversarial networks - Google Patents

Description

This invention generally relates to methods for modifying digital scripts, and more particularly to methods and related systems for improving software techniques related to generating and modifying image sequences associated with the text content of a digital story, and dynamically changing the associated digital text content.

A first aspect of the present invention is a generative adversarial network (GAN) hardware device including a processor coupled to a computer-readable memory unit, wherein the memory unit includes instructions for performing a digital script modification method that enables natural language processing (NLP) when executed by the processor, the method comprising: generating an image sequence related to the text content of a digital story by the processor; identifying multiple contextual dimensions within the text content by the processor through the execution of NLP code; selecting a group of dimensions of the multiple contextual dimensions by the processor in response to user input; scaling the image sequence in combination with the group of dimensions by the processor; and processing the dimensions by the processor The present invention provides a generative adversarial network (GAN) hardware device that includes: modifying an image sequence based on detected interactions with groups; the processor extracting dimensions from groups of dimensions during the presentation of a digital story and image sequences; the processor enabling a scriptwriter associated with the text content of the digital story to modify dimensions; the processor modifying the image sequence based on the dimension modifications that occur in response to the activation; the processor enabling a hardware interface device to interact with various image sequences and modify multiple contextual dimensions; and the processor dynamically modifying the text content of the digital story in response to the activation.

A second aspect of the present invention provides a digital script modification method that enables natural language processing (NLP), comprising: generating an image sequence related to the text content of a digital story by a processor in a generative adversarial network (GAN) hardware device; identifying multiple contextual dimensions within the text content by the processor executing NLP code; selecting a group of dimensions of the multiple contextual dimensions by the processor in response to user input; scaling the image sequence in combination with the group of dimensions by the processor; modifying the image sequence based on detected interactions with the group of dimensions by the processor; extracting dimensions from the group of dimensions during the presentation of the digital story and the image sequence by the processor; activating a script writer related to the text content of the digital story to modify the dimensions by the processor; modifying the image sequence based on the dimension modifications that occur in response to the activation by the processor; activating a hardware interface device to interact with various image sequences of the image sequence and modify multiple contextual dimensions by the processor; and dynamically changing the text content of the digital story in response to the activation by the processor.

A third aspect of the present invention comprises a computer-readable hardware storage device for storing computer-readable program code, the computer-readable program code including an algorithm that implements a digital script modification method that enables natural language processing (NLP) when executed by a server processor, the method comprising: generating an image sequence related to the text content of a digital story by the processor; identifying multiple contextual dimensions within the text content by the processor through the execution of NLP code by the processor; selecting a group of dimensions of the multiple contextual dimensions by the processor in response to user input; and scaling the image sequence in combination with the group of dimensions by the processor. This provides a computer program product that includes: modifying an image sequence based on detected interactions with groups of dimensions; the processor extracting dimensions from groups of dimensions during the presentation of a digital story and image sequences; the processor enabling a script writer associated with the text content of the digital story to modify dimensions; the processor modifying the image sequence based on the dimension modifications that occur in response to the activation; the processor enabling hardware interface devices to interact with various image sequences and modify multiple contextual dimensions; and the processor dynamically changing the text content of the digital story in response to the activation.

This invention advantageously provides a simple method and related system that can automate the modification of digital scripts.

This invention presents a system for improving software techniques related to generating and modifying image sequences associated with text content of a digital story, and dynamically changing the associated digital text content, according to embodiments of the present invention. This document details an algorithm that illustrates the process flow enabled by the system in Figure 1 for improving software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content, according to embodiments of the present invention. This is an internal structure diagram of the software/hardware according to an embodiment of the present invention. This invention illustrates a system, comprising a GAN module and an NLP module, for modifying the digital script of digital story content, according to an embodiment of the present invention. This document illustrates a process for modifying a digital script and generating a corresponding image sequence, according to an embodiment of the present invention. This document illustrates a process for modifying a digital script and generating a corresponding image sequence, according to an embodiment of the present invention. This document illustrates a process for modifying a digital script and generating a corresponding image sequence, according to an embodiment of the present invention. This document illustrates a process for modifying a digital script and generating a corresponding image sequence, according to an embodiment of the present invention. This is a detailed diagram of the text-to-image GAN network component shown in Figure 5, according to an embodiment of the present invention. This is a detailed diagram of the GAN network component for converting images to text according to an embodiment of the present invention, as shown in Figure 5. To improve software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content, according to embodiments of the present invention, a computer system used by the system in Figure 1 is shown. This document illustrates a cloud computing environment according to an embodiment of the present invention. This describes a set of functional abstraction layers provided by a cloud computing environment according to embodiments of the present invention.

Figure 1 shows a system 100 for improving software techniques related to generating and modifying image sequences associated with text content of a digital story and dynamically changing the associated digital text content, according to an embodiment of the present invention. A typical application script writing system may require a script writer entity to visualize a video presentation that requires text content analysis for image sequence creation. Similarly, during the aforementioned process, the script writer entity may generate requests to expand the generated image sequence in terms of various dimensions. Furthermore, the requests may include specifications for displaying a summary of the associated content, along with commands to modify the associated images, in relation to updating the text content related to image modification. Therefore, the system is configured to enable the script writer entity to expand the generated image content in various contextual dimensions for performing image modification. Similarly, system 100 enables automatic updating of text content in relation to image modification.

System 100 includes a system that analyzes digital text story content, identifies various relevant contextual dimensions, and enables natural language processing (NLP) to automatically generate image sequences based on the text story content through the execution of a generative adversarial network (GAN). Similarly, System 100 is configured to allow a scriptwriter entity to scale, modify, or bother with the generated images in order to dynamically update the digital text story content. System 100 enables the following functions:

System 100 enables the process of identifying multiple possible contextual dimensions from the digital script content (during the process of generating an image sequence from the digital script content using GANs), and ensures that the generated image sequence is scaled up or down with respect to the selected dimensions. Similarly, System 100 enables the process of modifying the image sequence (using GANs) based on interaction with the identified dimensions.

System 100 further enables a process of extracting contextual dimensions from the digital text story content presented with the generated image sequence, thereby enabling scriptwriter entities to modify the dimensions based on determined needs. The modified dimensions enable modification of the image sequence. System 100 can be configured to enable scriptwriter entities to add additional dimensions to the generated image sequence. The newly added dimensions are modified with respect to the generated image sequence, thereby modifying the image sequence and (through the execution of an inverse GAN model) the associated text script content. System 100 may further be configured to enable scriptwriter entities to selectively modify, delete, add, or combine one or more objects to the generated image sequence, thereby dynamically modifying the written text story content.

The scriptwriter entity can be enabled to split or join multiple image sequences (while interacting with the image sequence), thereby enabling an automated process of splitting or merging text story contexts to create new story content. A virtual reality (VR) user interface can allow users to interact with various image sequences, change the dimensions of the context, and dynamically modify the text story content.

The system 100 in Figure 1 includes GAN hardware 139, a text/digital story input component 140, a hardware interface 115, and a network interface controller 153 , all interconnected via a network 117. The GAN hardware 139 includes a sensor 112, a circuit 127, and software/hardware 121. The hardware interface may include any type of hardware-based interface, including, in particular, a virtual reality interface. The GAN hardware 139, the text/digital story input component 140, and the hardware interface 115 may each include an embedded device. In this specification, an embedded device is defined as a dedicated device or computer that includes a combination of computer hardware and software (fixed function or programmable) specifically designed to perform a particular function. A programmable embedded computer or device may include a special programming interface. In one embodiment, the GAN hardware 139, the text/digital story input component 140, and the hardware interface 115 may each include special hardware devices, including special (non-common) hardware and circuitry (i.e., special discrete non-common analog, digital, and logic-based circuits) for performing the processes described with respect to Figures 1 to 6 (independently or in combination). The special discrete non-common analog, digital, and logic-based circuits (e.g., sensor 112, circuitry/logic 127, software/hardware 121, etc.) may include uniquely designed components (e.g., special integrated circuits, such as application-specific integrated circuits (ASICs), designed solely to perform automated processes for generating and modifying image sequences related to the text content of a digital story and improving software techniques related to dynamically changing the associated digital text content). Sensor 112 may include any type of internal or external sensor, including, in particular, GPS sensors, Bluetooth beaconing sensors, mobile phone detection sensors, Wi-Fi positioning detection sensors, triangulation detection sensors, activity tracking sensors, temperature sensors, ultrasonic sensors, optical sensors, video search devices, humidity sensors, voltage sensors, and network traffic sensors. Network 117 may include any type of network, including, in particular, local area networks (LANs), wide area networks (WANs), the Internet, and wireless networks.

System 100 is capable of performing a process (through the execution of a natural language processing model) to perform text analysis of the digital content of the story script. Based on the analysis, System 100 is configured to identify various contextual dimensions within the story content. Similarly, System 100 analyzes a knowledge corpus with respect to various dimensions, particularly those related to weather, events, location, time, physical X, Y, Z position, and velocity. System 100 is further configured to correct for varying degrees of contextual dimensions, particularly between mild and severe weather. Furthermore, System 100 is configured to generate an image sequence from the digital story script content. The image sequence is useful for identifying various possible dimensions from the text story script content.

Figure 2 shows an algorithm detailing a process flow enabled by the system 100 of Figure 1 to improve software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content, according to embodiments of the present invention. Each step in the algorithm of Figure 2 can be enabled and executed in any order by a computer processor executing computer code. Furthermore, each step in the algorithm of Figure 2 may be enabled and executed in combination by the GAN hardware 139, the text/digital story input component 140 , and the hardware interface 115. In step 200, an image sequence associated with the text content of the digital story is generated by the GAN hardware device. In step 202, multiple context dimensions within the text content are identified (via the execution of NLP code). Context dimensions may include, in particular, dimensions such as weather dimensions, event dimensions, location dimensions, time dimensions, physical X, Y, Z location dimensions, and velocity dimensions.

In step 204, a group of dimensions representing multiple contextual dimensions is selected in response to user input. In step 208, the image sequence is scaled up or down in combination with the group of dimensions. In step 210, the image sequence is modified based on the detected interaction with the group of dimensions. In step 212, dimensions are extracted from the group of dimensions during the presentation of the digital story and image sequence. In step 214, a scriptwriter entity (related to the text content of the digital story) is activated to modify the dimensions. In step 216, the image sequence is modified based on the dimensional modifications that occur in response to the results of step 214. In step 218, a hardware interface device is activated to interact with various image sequences of the image sequence and modify multiple contextual dimensions. The hardware interface device may include a virtual reality (VR) interface device. In step 220, the text content of the digital story is dynamically modified.

In step 224, the script writer entity functionality can be enabled (via GAN hardware) as described in the following implementation scenario.

In the first scenario, a script writer entity is enabled to add an additional contextual dimension to the image sequence (via a hardware interface device) so that the image sequence is modified. The text content is then modified by running an inverse GAN model on the result of the image sequence modification.

The second scenario involves enabling the scriptwriter entity (via a hardware interface device) to selectively modify at least one visual object in an image sequence so that the text content is modified via the execution of an inverse GAN model with respect to the results of enabling the scriptwriter entity.

The third scenario involves enabling the scriptwriter entity (via a hardware interface device) to selectively remove at least one visual object from an image sequence so that the text content is modified via the execution of an inverse GAN model with respect to the results of enabling the scriptwriter entity.

The fourth scenario involves enabling the scriptwriter entity (via a hardware interface device) to selectively add at least one visual object to an image sequence so that the text content is modified via the execution of an inverse GAN model with respect to the results of enabling the scriptwriter entity.

The fifth scenario involves enabling a script writer entity to split multiple image sequences (via a hardware interface device during interaction with the various image sequences). In response, the text content is split, and new text content is generated for the digital story.

The sixth scenario involves enabling a script writer entity to combine multiple image sequences (via a hardware interface device during interaction with the various image sequences). In response, text content is merged, generating new text content for the digital story.

Figure 3 is an internal structure diagram of the software/hardware 121 (i.e., 121) of Figure 1, according to an embodiment of the present invention. The software/hardware 121 includes an identification module 304, a modification module 305, an extraction module 308, a modification/activation module 314, and a communication controller 312. The identification module 304 includes dedicated hardware and software for controlling all functions related to the identification step in Figure 2. The modification module 305 includes dedicated hardware and software for controlling all functions related to the modification step described with respect to the algorithm in Figure 2. The extraction module 308 includes dedicated hardware and software for controlling all functions related to the extraction step in Figure 2. The modification/activation module 314 includes dedicated hardware and software for controlling all functions related to the modification and activation steps of the algorithm in Figure 2. The communication controller 312 is activated to control all communication between the identification module 304, the modification module 305, the extraction module 308, and the modification/activation module 314.

Figure 4 shows a system 400, according to an embodiment of the present invention, which includes a GAN module 402a and an NLP module 404 for modifying a digital script 405 of digital story content. The system 400 is configured to present multiple possible dimensions 412 associated with an image sequence 408 generated from the digital script 405, thereby enabling a scriptwriter entity to modify the dimensions 412. Similarly, the GAN module 402a is configured to modify the images in the generated image sequence 408 so that the digital story content is dynamically updated. Modification of the images in the generated image sequence 408 results in the generation of a modified image sequence 410. The GAN module 402a may be enabled to generate an image sequence corresponding to input text, and the resulting generated images may be viewed by the user.

The NLP module 404 may be configured to retrieve input text and a knowledge corpus containing various dimensions 412 of the digital script 405 (e.g., weather dimension, color dimension, etc.). In response, the system 400 analyzes the input text to identify the dimensions available within the digital script 405. Various dimensions 412 of the input text and their relative degrees are identified. For example, the "weather" dimension may include various degrees associated with it, such as sunny, cloudy, windy, and rainy. The identified dimensions 412 and their relative degrees may be displayed for the user. Similarly, the user may modify (e.g., add, update, delete, etc.) the dimensions 412 and their degrees with respect to selection. The image generated (from the GAN module 402a), and the modified dimensions and associated dimensional degrees selected by the user, are sent to a second GAN module 402b as conditional input. System 400 is further configured to execute a conditional text-to-image translation code through the use of a cycle-consistent adversarial network that searches for input (i.e., user-selected images and modified dimensions). The text-to-image module (GAN module 402b) is enabled to generate a modified version of the input image with respect to user-selected dimensions and degree. The inverse GAN module 402c (i.e., the image-to-text translation module) takes the modified image sequence 410 as input and generates the associated text (i.e., the modified script 415). The modified image sequence 410 and the corresponding modified script 415 are used by the user to finalize the digital script.

Figures 5A to 5D show a process 500 for modifying a digital script 502 and generating corresponding image sequences 504a and 504b according to an embodiment of the present invention. Process 500 is initiated when text content (i.e., the digital script 502) is provided as input to the text-to-image GAN module 506 and NLP module 507 for performing text analysis of the digital script 502 (with respect to the knowledge corpus 509). In response to the text analysis of the digital script 502, the system 500 identifies various context (and degree dimensions) 511 from the story content of the digital script 502. Context dimensions 511 may include, in particular, weather dimensions, event dimensions, location dimensions, time dimensions, physical X, Y, Z location dimensions, velocity dimensions, etc. The system 500 further enables various degrees of the context dimensions 511. Subsequently, the GAN module 506 generates an image sequence 504a from the text content of the digital script 502 (through the execution of the GAN module 506) and identifies various possible dimensions (of the context dimension 511) from the text content. The image sequence 504a may be displayed via a hardware/software interface 514 (e.g., a 2D display, a VR device, etc.). The system 500 then presents one or more context dimensions in combination with the image sequence 504a so that the story script writer entity 517 can be enabled (through the text-to-image GAN network component 522 and the image-to-text GAN network component 524) to change the degree of the various dimensions presented with the image sequence 504a. In response to the selection of various modified context dimensions 519, the system 500 receives relevant user input and identifies the modified context dimension 519. The modified context dimension 519 is used to analyze the current image sequence (i.e., image sequence 504a) in order to modify the images within the image sequence 504a. All modifications to the context dimension 511 are considered for updating the images in image sequence 504a. Similarly, system 500 activates the scriptwriter entity 517 to add additional dimensions and selected dimensions to image sequence 504a, and the images in image sequence 504a are updated accordingly. The scriptwriter entity 517 may be activated to selectively modify/add/remove one or more image objects from image sequence 504a so that the images in image sequence 504a are changed. The scriptwriter entity 517 can split or join different images, resulting in the generation of an updated image sequence 504b, which can be displayed via the hardware-software interface 527. Once the modification process is complete, system 400 performs the process of updating the digital script 502 with the modified images of image sequence 504b, resulting in the generation of a modified digital script 528.

Figure 6 is a detailed view of the text-to-image GAN network component 522 of Figure 5, according to an embodiment of the present invention. The GAN network component 522 includes a first stage 602 (stage 1) and a second stage 604 (stage 2). The first stage 602 includes a pair of generator G1 and discriminator D1. Similarly, the second stage 604 includes a pair of generator G2 and discriminator D2. Generator G1 is configured to generate a low-resolution image 607 (e.g., 64 x 64 ppi), and generator G2 is configured to generate a high-resolution image 609 (128 x 128 ppi). Relevant text embedding data 605 (i.e., script) and associated noise can be used as input to the first stage 602. Furthermore, the image and user-modified dimensions and degree may also be used as input to the first stage 602. Generator G1 may be configured to skip thought text embedding and generate a composite image (i.e., the low-resolution image 607). Similarly, the first stage 602 classifier D1 is conditioned on the same text embeddings and trained to classify between real and composite images with a resolution of 64x64 ppi. The generator G1 includes a series of upsampling blocks 611. The upsampling blocks 611 include enabling a nearest-neighbor upsampling process followed by a 3x3 stride 1 convolution process to project the input onto a 3x64x64 image (i.e., the low-resolution image 607) containing the low-resolution image (64x64 ppi image). The classifier D1 includes a series of downsampling blocks 612 that project the input onto a 512x4x4 dimension. The aforementioned 512x4x4 dimension is concatenated with a 128-dimensional compressed embedding, and a sigmoid layer 615 is used to generate an output between 0 (fake) and 1 (real) to distinguish the low-resolution image. The stage 2 generator G1 takes I1 along with the embeddings as input and generates a higher-resolution 128x128 image.

The generator G2 includes a series of downsampling blocks 622 that project the 3x64x64 input image 624 to a 512x16x16 dimension. A 128-dimensional embedding is then concatenated. The input image 624 is transmitted as a series of residual blocks 626, followed by a series of upsampling blocks 628, to generate a (higher resolution) image 609 (i.e., a 128x128 image). The discriminator D2 receives image 609 (as input). The discriminator D2 includes a series of downsampling blocks 629, allowing the sigmoid layer 632 to generate an output between 0 (fake) and 1 (real) to distinguish the high resolution image (128x128). Therefore, the output of the first stage 602 is used as input to the second stage 604 to generate a higher resolution image (i.e., high resolution image 609), as an output including the user-modified dimension.

Figure 7 shows a detailed diagram of the image-to-text GAN network component 524 (i.e., the caption GAN network component) of Figure 5, according to an embodiment of the present invention. The GAN network component 524 includes components 702 and 704. Component 702 forms a caption generator that includes long-term short-term memory (LSTM) components 707a...707n for retrieving convolutional neural network (CNN) features 708 and noise Z709 as input for outputting a caption 711. The input to component 702 includes the output high-resolution image 715 (from the GAN network component 522 in Figure 6), including user-modified dimensions and degree.

Component 704 forms a classifier that performs a dot product with respect to the CNN features 712 of the high-resolution modified image 715a and the outputs from the LSTM components 707a...707n. The high-resolution modified image 715a, containing the user's preferred dimensions and degrees, is sent as input to perform a regular sequence modeling process with respect to (LSTM components 717a...717n) to generate the corresponding. The resulting output text script 720 contains the user-updated dimensions and degrees, which are available for the user to make a final decision.

Figure 8 shows a computer system 90 (e.g., the GAN hardware 139, text/digital story input component 140, and hardware interface 115 in Figure 1) used by or configured by the system 100 in Figure 1 to improve software techniques related to generating and modifying image sequences associated with text content of a digital story and dynamically changing the associated digital text content, according to embodiments of the present invention.

The embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware embodiments, all of which are generally referred to herein as “circuits,” “modules,” or “systems.”

The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium storing computer-readable program instructions for causing a processor to execute aspects of the present invention.

A computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. Examples of computer-readable storage media include, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or appropriate combinations thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM or flash memory, SRAM, CD-ROM, DVD, memory stick, floppy disk, punch cards or grooved raised structures, and mechanically encoded devices on which instructions are recorded, and appropriate combinations thereof. The computer-readable storage media used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing unit, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network consists of copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing/processing unit receives computer-readable program instructions from the network and transfers them for storage on the computer-readable storage medium within each computing/processing unit.

The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, Spark, and R, and procedural programming languages such as the C programming language or similar programming languages. The computer-readable program instructions are executable as a standalone software package, either entirely on the user's computer or partially on the user's computer. Alternatively, they may be executable partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, electronic circuits including programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute computer-readable program instructions by personalizing them using state information of computer-readable program instructions in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowcharts or block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks in a flowchart or block diagram, or both, can be implemented using computer-readable program instructions.

These computer-readable program instructions can be provided to a general-purpose computer, a dedicated computer processor, or other programmable data processing device to generate a machine, such that instructions executed via the processor of a computer or other programmable data processing device generate means for implementing functions/operations specified in one or more blocks of a flowchart or block diagram, or both. These computer-readable program instructions can also be stored in a computer-readable storage medium that can be connected to a computer, a programmable data processing device, or other device or combination of devices that function in a particular way, such that the computer-readable storage medium on which the instructions are stored constitutes one of the outputs containing instructions that implement the modes of function/operations specified in one or more blocks of a flowchart or block diagram, or both.

Computer-readable program instructions, like instructions that perform functions/actions specified in one or more blocks of a flowchart or block diagram, or both, on a computer, other programmable device, or other device, can also be loaded into a computer, other programmable data processing device, or other device and perform a series of operational steps on that device, generating a computer-implemented process.

The flowcharts and block diagrams in the figures illustrate the configuration, function, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or part of an instruction, which constitutes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions shown in the blocks may differ from the order shown in the figures. For example, two consecutively shown blocks may actually be achieved as a single step, executed simultaneously, substantially simultaneously, partially or entirely in overlapping time, or the blocks may be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, can be implemented by a special-purpose hardware-based system that performs a specified function or operation, or a combination of special-purpose hardware and computer instructions.

The computer system 90 shown in Figure 8 includes a processor 91, an input device 92 coupled to the processor 91, an output device 93 coupled to the processor 91, and storage devices 94 and 95, respectively, coupled to the processor 91. The input device 92 may be, in particular, a keyboard, mouse, camera, touchscreen, etc. The output device 93 may be, in particular, a printer, plotter, computer screen, magnetic tape, removable hard disk, floppy disk, etc. The storage devices 94 and 95 may be, in particular, optical storage devices such as hard disks, floppy disks, magnetic tape, compact discs (CDs) or digital video discs (DVDs), dynamic random access memory (DRAM), read-only memory (ROM), etc. The storage device 95 includes computer code 97. The computer code 97 includes algorithms (e.g., the algorithm in Figure 2) for improving software techniques related to generating and modifying image sequences related to the text content of a digital story and dynamically changing the associated digital text content. The processor 91 executes the computer code 97. The storage device 94 includes input data 96. Input data 96 includes inputs requested by computer code 97. Output device 93 displays the output from computer code 97. Either or both of storage devices 94 and 95 (or one or more additional storage devices such as read-only storage device 85) can be used as computer-readable media (or computer-readable media or program storage device) containing an algorithm (e.g., the algorithm in Figure 2), having computer-readable program code implemented therein, or having other data stored therein, or both, and the computer-readable program code includes computer code 97. Generally, a computer program product (or, alternatively, a manufactured product) of a computer system 90 may include computer-readable media (or program storage devices).

In some embodiments, instead of being stored and accessed from a hard drive, optical disc, or other writable, rewritable, or removable hardware storage device 95, the stored computer program code 84 (e.g., including algorithms) may be stored in a static, non-removable, read-only storage medium such as a read-only storage (ROM) device 85, or may be directly accessed by the processor 91 from such a static, non-removable, read-only medium. Similarly, in some embodiments, the stored computer program code 97 may be stored as computer-readable firmware 85, or directly accessed by the processor 91 from such firmware 85, rather than from a more dynamic or removable hardware data storage device 95 such as a hard drive or optical disc.

Nevertheless, any component of the present invention may be created, integrated, hosted, maintained, deployed, managed, serviced, etc. by a service supplier that provides services to improve software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content. Accordingly, the present invention discloses a process for deploying, creating, integrating, hosting, maintaining, or integrating, or a combination thereof, a computing infrastructure, which includes integrating computer-readable code into a computer system 90, and the code combined with the computer system 90 can perform methods to enable the process of improving software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content. In another embodiment, the present invention provides a business method for performing the process steps of the present invention on a subscription, advertising, or fee basis, or a combination thereof. That is, a service supplier, such as a solution integrator, may provide services to enable the process of improving software techniques related to generating and modifying image sequences associated with the text content of a digital story and dynamically changing the associated digital text content. In this case, the service supplier may create, maintain, support, etc., a computer infrastructure that performs the process steps of the present invention for one or more customers. In return, the service supplier may receive payments from customers based on a subscription, a fee agreement, or both, or from the sale of advertising content to one or more third parties, or both.

Figure 8 shows computer system 90 as a specific hardware and software configuration; however, any hardware and software configuration known to those skilled in the art can be used in combination with the specific computer system 90 in Figure 8 for the purposes described above. For example, storage devices 94 and 95 may not be separate storage devices, but rather part of a single storage device.

<Cloud Computing Environment>
This disclosure includes a detailed description of cloud computing, but the implementations of the teachings described herein are not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in any other type of computer environment that is currently known or may be developed in the future.

Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

The characteristics are as follows:

On-demand self-service: Cloud consumers can unilaterally prepare computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

Broad network access: Computing power is available over the network and accessible through standard mechanisms. This facilitates utilization by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, PDAs).

Resource Pooling: A provider's computing resources are pooled and delivered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated as needed. Generally, consumers have a sense of location independence because they do not manage or know the exact location of the resources provided. However, consumers may be able to identify locations at higher levels of abstraction (e.g., country, state, data center).

Rapid Flexibility: Computing power can be prepared quickly and flexibly, allowing it to scale out automatically and immediately, and to be quickly released and scale in immediately. For consumers, the computing power available for preparation often appears unlimited and can be purchased in any quantity at any time.

Measured Services: Cloud systems leverage metric capabilities at a certain level of abstraction, appropriate for the type of service (e.g., storage, processing, bandwidth, active user accounts), to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

The service model is as follows:

Software as a Service (SaaS): The functionality offered to consumers is the ability to use the provider's applications running on a cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., webmail). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functions. However, this does not apply to configuring a limited number of user-specific application configurations.

Platform as a Service (PaaS): The functionality offered to consumers is the ability to deploy applications they have created or acquired to cloud infrastructure using programming languages and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, and storage, but they can control the deployed applications and, in some cases, the configuration of their hosting environment.

Infrastructure as a Service (IaaS): The functionality provided to consumers is the provision of processors, storage, networking, and other fundamental computing resources that enable consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they can control the operating system, storage, and deployed applications, and in some cases, partially control certain network components (e.g., host firewalls).

The deployment model is as follows:

Private Cloud: This cloud infrastructure is operated exclusively for a specific organization. This cloud infrastructure can be managed by that organization or a third party and can reside on-premises or off-premises.

Community Cloud: This cloud infrastructure is shared by multiple organizations to support specific communities with common interests (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by the organization or a third party and can reside on-premises or off-premises.

Public Cloud: This cloud infrastructure is provided to a large number of people or large industry groups and is owned by organizations that sell cloud services.

Hybrid Cloud: This cloud infrastructure combines two or more cloud models (private, community, or public). While maintaining the unique entities of each model, they are bound together by standards or individual technologies to achieve data and application portability (e.g., cloud bursting for load balancing across clouds).

Cloud computing environments are service-oriented environments that emphasize statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is the infrastructure, including a network of interconnected nodes.

Referring to Figure 9, an exemplary cloud computing environment 50 is shown. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10. Local computer devices used by cloud consumers (e.g., personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or a combination thereof) can communicate with these nodes. The nodes 10 can communicate with each other. The nodes 10 can be grouped physically or virtually (not shown) in one or more networks, such as the private, community, public, or hybrid clouds or a combination thereof. This allows the cloud computing environment 50 to provide infrastructure, platforms, or software as a service, or a combination thereof, without requiring cloud consumers to maintain resources on their local computer devices. Note that the types of computer devices 54A, 54B, 54C, and 54N shown in Figure 9 are illustrative only, and it should be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network addressable connection (e.g., using a web browser) or both.

Referring to Figure 10, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 9) is shown. It should be understood that the components, layers, and functions shown in Figure 10 are illustrative only, and the embodiments of the present invention are not limited to these. As illustrated, the following layers and corresponding functions are provided:

The hardware and software layer 60 includes hardware and software components. Examples of hardware components include a mainframe 61, a reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage 65, and a network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

The virtualization layer 70 provides an abstraction layer. From this layer, for example, the following virtual entities can be provided: a virtual server 71, virtual storage 72, a virtual network 73 including a virtual private network, a virtual application and operating system 74, and a virtual client 75.

As an example, the management layer 80 can provide the following functions: Resource preparation 81 enables the dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking when resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. For example, these resources may include application software licenses. Security enables not only protection of data and other resources, but also identification and verification of cloud consumers and tasks. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 87 enables the allocation and management of cloud computing resources to ensure that requested service levels are met. Service Level Assurance (SLA) planning and execution 88 enables the pre-arrangement and procurement of cloud computing resources expected to be needed in the future in accordance with the SLA.

Workload layer 101 provides examples of the capabilities available in a cloud computing environment. Examples of workloads and capabilities available from this layer include mapping and navigation 102, software development and lifecycle management 103, virtual classroom education delivery 133, data analysis processing 134, transaction processing 106, and improvements to software technologies related to the generation and modification of image sequences associated with the text content of digital stories, and the dynamic modification of associated digital text content 107.

While embodiments of the present invention have been described herein for illustrative purposes, many modifications and changes will become apparent to those skilled in the art. Therefore, the appended claims are intended to encompass all such modifications and changes that fall within the scope of the present invention.

Claims (20)

A generative adversarial network (GAN) hardware device including a processor coupled to a computer-readable memory unit, wherein the memory unit includes instructions that, when executed by the processor, perform a digital script modification method that enables natural language processing (NLP), and the method is
The aforementioned processor generates an image sequence related to the text content of the digital story,
The processor identifies multiple contextual dimensions within the text content by executing NLP code,
The processor selects a group of dimensions of the plurality of context dimensions in response to user input,
The processor enlarges or reduces the image sequence by combining it with the group of dimensions,
The processor modifies the image sequence based on the detected interaction with the group of dimensions,
The processor extracts dimensions from the group of dimensions during the presentation of the digital story and the image sequence,
The processor enables the scriptwriter associated with the text content of the digital story in order to correct the dimension,
The processor modifies the image sequence based on the dimensional correction that occurs in response to the activation.
The processor enables a hardware interface device to interact with various image sequences of the image sequence and to change the multiple context dimensions.
The processor dynamically changes the text content of the digital story in response to the activation,
Generative Adversarial Network (GAN) hardware devices, including those mentioned above. The GAN hardware device according to claim 1, wherein the plurality of context dimensions include dimensions selected from the group consisting of weather dimensions, event dimensions, location dimensions, time dimensions, physical X, Y, Z location dimensions, and velocity dimensions. The above method further,
The processor enables the script writer via the hardware interface device to add an additional contextual dimension to the image sequence,
The processor performs a first modification to the image sequence with respect to the additional context dimension,
The processor, which executes an inverse GAN model with respect to the result of the first modification, performs a second modification on the text content.
A GANG hardware device according to claim 1, including the above. The above method further,
The processor enables the script writer via the hardware interface device to selectively modify at least one visual object in the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
A GANG hardware device according to claim 1, including the above. The above method further,
The processor enables the script writer via the hardware interface device to selectively remove at least one visual object from the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
A GANG hardware device according to claim 1, including the above. The above method further,
The processor enables the script writer via the hardware interface device to selectively add at least one visual object to the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
A GANG hardware device according to claim 1, including the above. The above method further,
The processor enables the script writer to divide the image sequence into multiple image sequences via the hardware interface device during interaction with the various image sequences,
The processor divides the text content in response to the result of enabling the script writer,
The processor generates new text content for the digital story in response to the division,
A GANG hardware device according to claim 1, including the above. The above method further,
The processor enables the script writer via the hardware interface device to concatenate multiple image sequences of the image sequence during interaction with the various image sequences,
The processor, in response to the activation result, merges the text content,
The processor generates new text content for the digital story in response to the merge,
A GANG hardware device according to claim 1, including the above. The GAN hardware device according to claim 1, wherein the hardware interface device comprises a virtual reality (VR) interface device. A digital script modification method that enables natural language processing (NLP),
The processor of a Generative Adversarial Network (GAN) hardware device generates image sequences related to the text content of a digital story,
The processor identifies multiple contextual dimensions within the text content by executing NLP code,
The processor selects a group of dimensions of the plurality of context dimensions in response to user input,
The processor enlarges or reduces the image sequence by combining it with the group of dimensions,
The processor modifies the image sequence based on the detected interaction with the group of dimensions,
The processor extracts dimensions from the group of dimensions during the presentation of the digital story and the image sequence,
The processor enables the scriptwriter associated with the text content of the digital story in order to correct the dimension,
The processor modifies the image sequence based on the dimensional correction that occurs in response to the activation.
The processor enables a hardware interface device to interact with various image sequences of the image sequence and to change the multiple context dimensions.
The processor dynamically changes the text content of the digital story in response to the activation,
Methods that include... The method according to claim 10, wherein the plurality of context dimensions include dimensions selected from the group consisting of weather dimensions, event dimensions, location dimensions, time dimensions, physical X, Y, Z location dimensions, and velocity dimensions. The above method further,
The processor enables the script writer via the hardware interface device to add an additional contextual dimension to the image sequence,
The processor performs a first modification to the image sequence with respect to the additional context dimension,
The processor, which executes an inverse GAN model with respect to the result of the first modification, performs a second modification on the text content.
The method according to claim 10, including the method described in claim 10. The processor enables the script writer via the hardware interface device to selectively modify at least one visual object in the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
The method according to claim 10, further comprising: The processor enables the script writer via the hardware interface device to selectively remove at least one visual object from the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
The method according to claim 10, further comprising: The processor enables the script writer via the hardware interface device to selectively add at least one visual object to the image sequence,
The text content is modified by the processor that executes an inverse GAN model with respect to the result of the script writer being activated,
The method according to claim 10, further comprising: The processor enables the script writer to divide the image sequence into multiple image sequences via the hardware interface device during interaction with the various image sequences,
The processor divides the text content in response to the result of enabling the script writer,
The processor generates new text content for the digital story in response to the division,
The method according to claim 10, further comprising: The processor enables the script writer via the hardware interface device to concatenate multiple image sequences of the image sequence during interaction with the various image sequences,
The processor, in response to the activation result, merges the text content,
The processor generates new text content for the digital story in response to the merge,
The method according to claim 10, further comprising: The method according to claim 10, wherein the hardware interface device comprises a virtual reality (VR) interface device. The method according to claim 10, further comprising providing a computer system that provides at least one support service for at least one of the creation, integration, hosting, maintenance, and deployment of computer-readable code, wherein the computer-readable code is executed by the processor and provides the processor to perform the generation, the identification, the selection, the expansion or reduction, the modification, the extraction, the activation of the script writer, the correction, the activation of the hardware interface device, and the dynamic modification. The computer-readable program code includes an algorithm that performs a digital script modification method that enables natural language processing (NLP) when executed by the server's processor, and the method is
The aforementioned processor generates an image sequence related to the text content of the digital story,
The processor identifies multiple contextual dimensions within the text content by executing NLP code,
The processor selects a group of dimensions of the plurality of context dimensions in response to user input,
The processor enlarges or reduces the image sequence by combining it with the group of dimensions,
The processor modifies the image sequence based on the detected interaction with the group of dimensions,
The processor extracts dimensions from the group of dimensions during the presentation of the digital story and the image sequence,
The processor enables the scriptwriter associated with the text content of the digital story in order to correct the dimension,
The processor modifies the image sequence based on the dimensional correction that occurs in response to the activation.
The processor enables a hardware interface device to interact with various image sequences of the image sequence and to change the multiple context dimensions.
The processor dynamically changes the text content of the digital story in response to the activation,
A computer program that includes [this].