Inference with Mistral 7B¶

This notebook demonstrates using a pre-trained Stained Glass Transform to transform text inputs for a Mistral 7B model. Both raw and transformed inputs are passed into the Mistral model to compare the outputs.

For information on deploying the Stained Glass Transform for inference, see the Stained Glass Transform deployment tutorial.

Pre-requisites¶

The Stained Glass Transform must be pre-trained and prepared as a StainedGlassTransformForText object. This object should be saved to a file, whose path can be set below.
You must have the Mistral-7B-Instruct-v0.2 model available. Request access to the model from its model card on the HuggingFace Hub.

Load the Stained Glass Transform and the Base Mistral Model¶

In [1]:

Copied!

from __future__ import annotations
from __future__ import annotations

In [2]:

parameters

Copied!





import torch

STAINED_GLASS_TRANSFORM_PATH = "stained-glass-transform-mistral-7b.pt"

# The model name could be either the HuggingFace model ID
# (mistralai/Mistral-7B-Instruct-v0.2) or the path to the model on the local
# file system.
MISTRAL_MODEL_NAME_OR_PATH = (
    "/models/huggingface/mistralai/Mistral-7B-Instruct-v0.2"
)

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
import torch

STAINED_GLASS_TRANSFORM_PATH = "stained-glass-transform-mistral-7b.pt"

# The model name could be either the HuggingFace model ID
# (mistralai/Mistral-7B-Instruct-v0.2) or the path to the model on the local
# file system.
MISTRAL_MODEL_NAME_OR_PATH = (
    "/models/huggingface/mistralai/Mistral-7B-Instruct-v0.2"
)

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

In [3]:

Copied!





import transformers

MISTRAL_MODEL = transformers.AutoModelForCausalLM.from_pretrained(
    MISTRAL_MODEL_NAME_OR_PATH
).eval()
MISTRAL_TOKENIZER = transformers.AutoTokenizer.from_pretrained(
    MISTRAL_MODEL_NAME_OR_PATH
)
import transformers

MISTRAL_MODEL = transformers.AutoModelForCausalLM.from_pretrained(
    MISTRAL_MODEL_NAME_OR_PATH
).eval()
MISTRAL_TOKENIZER = transformers.AutoTokenizer.from_pretrained(
    MISTRAL_MODEL_NAME_OR_PATH
)

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]

In [5]:

Copied!





from stainedglass_core import transform as sg_transform

MISTRAL_MODEL = MISTRAL_MODEL.to(DEVICE)

STAINED_GLASS_TRANSFORM = (
    sg_transform.StainedGlassTransformForText.from_pretrained(
        STAINED_GLASS_TRANSFORM_PATH
    ).eval()
)
from stainedglass_core import transform as sg_transform

MISTRAL_MODEL = MISTRAL_MODEL.to(DEVICE)

STAINED_GLASS_TRANSFORM = (
    sg_transform.StainedGlassTransformForText.from_pretrained(
        STAINED_GLASS_TRANSFORM_PATH
    ).eval()
)

Prepare helper functions¶

We will define a few helper functions to help us process the inputs and outputs of the model.

In [6]:

Copied!





from typing import Any

import torch


@torch.inference_mode()
def generate(
    *,
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
) -> list[str]:
    """Generate text from the model given the input embeddings and attention
    mask.

    This function can be modified to adjust the generation parameters.

    Args:
        input_ids: The input ids for the model.
        inputs_embeds: The input embeddings for the model.
        attention_mask: The attention mask for the model.

    Returns:
        A list of strings containing the generated text(s)

    Raises:
        ValueError: If neither input_ids nor inputs_embeds are provided.
        ValueError: If both input_ids and inputs_embeds are provided.
    """
    if input_ids is None and inputs_embeds is None:
        raise ValueError("Either input_ids or inputs_embeds must be provided")

    if input_ids is not None and inputs_embeds is not None:
        raise ValueError(
            "Only one of input_ids or inputs_embeds should be provided"
        )

    if input_ids is not None:
        inputs_embeds = MISTRAL_MODEL.get_input_embeddings().forward(input_ids)
    if attention_mask is None:
        attention_mask = torch.ones(inputs_embeds.shape[:2], device=DEVICE)

    predicted_tokens = MISTRAL_MODEL.generate(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        eos_token_id=MISTRAL_TOKENIZER.eos_token_id,
        pad_token_id=MISTRAL_TOKENIZER.eos_token_id,
        renormalize_logits=True,
        max_new_tokens=256,
    )
    return MISTRAL_TOKENIZER.batch_decode(
        predicted_tokens, skip_special_tokens=True
    )


def tokenize_text(context: str, instruction: str) -> dict[str, Any]:
    """Tokenizes the input text.

    This uses the wrapped tokenizer from the Stained Glass Transform to ensure
    that the input is tokenized exactly the same way as the transformed prompt.
    This could also be implemented using the tokenizer directly.

    Args:
        context: The context of the instruction.
        instruction: The instruction of the instruction.

    Returns:
        The tokenized text.
    """
    return STAINED_GLASS_TRANSFORM.tokenizer_wrapper(
        {
            "context": context,
            "instruction": instruction,
            "system_prompt": "",
            "response": "",
        }
    )
from typing import Any

import torch


@torch.inference_mode()
def generate(
    *,
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
) -> list[str]:
    """Generate text from the model given the input embeddings and attention
    mask.

    This function can be modified to adjust the generation parameters.

    Args:
        input_ids: The input ids for the model.
        inputs_embeds: The input embeddings for the model.
        attention_mask: The attention mask for the model.

    Returns:
        A list of strings containing the generated text(s)

    Raises:
        ValueError: If neither input_ids nor inputs_embeds are provided.
        ValueError: If both input_ids and inputs_embeds are provided.
    """
    if input_ids is None and inputs_embeds is None:
        raise ValueError("Either input_ids or inputs_embeds must be provided")

    if input_ids is not None and inputs_embeds is not None:
        raise ValueError(
            "Only one of input_ids or inputs_embeds should be provided"
        )

    if input_ids is not None:
        inputs_embeds = MISTRAL_MODEL.get_input_embeddings().forward(input_ids)
    if attention_mask is None:
        attention_mask = torch.ones(inputs_embeds.shape[:2], device=DEVICE)

    predicted_tokens = MISTRAL_MODEL.generate(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        eos_token_id=MISTRAL_TOKENIZER.eos_token_id,
        pad_token_id=MISTRAL_TOKENIZER.eos_token_id,
        renormalize_logits=True,
        max_new_tokens=256,
    )
    return MISTRAL_TOKENIZER.batch_decode(
        predicted_tokens, skip_special_tokens=True
    )


def tokenize_text(context: str, instruction: str) -> dict[str, Any]:
    """Tokenizes the input text.

    This uses the wrapped tokenizer from the Stained Glass Transform to ensure
    that the input is tokenized exactly the same way as the transformed prompt.
    This could also be implemented using the tokenizer directly.

    Args:
        context: The context of the instruction.
        instruction: The instruction of the instruction.

    Returns:
        The tokenized text.
    """
    return STAINED_GLASS_TRANSFORM.tokenizer_wrapper(
        {
            "context": context,
            "instruction": instruction,
            "system_prompt": "",
            "response": "",
        }
    )

In [40]:

Copied!





import datetime

import IPython.display
from typing_extensions import Self


class TimerContext:
    """Context manager to time a block of code using a `with` statement.

    Attributes:
        start: The start time.
        end: The end time (if the context manager has exited).
        duration: The duration of the block of code. If the context manager has
            not exited, this will be the duration up to the current time.

    Note:
        Because of the additional overhead of the context manager, this may not
        be the most accurate way to time a block of fast code. For a slow block
        of code, the overhead should be proportionally small.
    """

    start: datetime.datetime
    end: datetime.datetime | None

    def __enter__(self) -> Self:
        self.start = datetime.datetime.now()
        self.end = None
        return self

    def __exit__(self, *args: object, **kwargs: Any) -> None:
        self.end = datetime.datetime.now()

    @property
    def duration(self) -> datetime.timedelta:
        """The duration of the block of code."""
        return (self.end or datetime.datetime.now()) - self.start


@torch.inference_mode()
def generate_and_output_responses(
    context: str, instruction: str, num_lines: int = 6, width: int = 80
) -> None:
    """Generate response to both raw and transformed prompts and print them
    along with metadata.

    Args:
        context: The context of the instruction.
        instruction: The instruction of the instruction.
        num_lines: The number of lines to print for each metadata field.
        width: The width of each line to print.
    """
    with TimerContext() as sgt_timer:
        transformed_embeddings = STAINED_GLASS_TRANSFORM(
            {
                "instruction": instruction,
                "context": context,
                "system_prompt": "",
                "response": "",
            }
        )
    raw_tokenization = tokenize_text(context, instruction)
    raw_embeddings = MISTRAL_MODEL.get_input_embeddings().forward(
        raw_tokenization["input_ids"].to(DEVICE)
    )
    noise_mask = raw_tokenization["noise_mask"]
    raw_tokenization["attention_mask"] = torch.ones_like(noise_mask)

    transformed_embeddings = STAINED_GLASS_TRANSFORM.truncated_module.module.sample_transformed_embeddings(
        input_ids=raw_tokenization["input_ids"],
        noise_mask=raw_tokenization["noise_mask"],
        attention_mask=raw_tokenization["attention_mask"],
        use_cache=True,
    )
    attempted_reconstruction = STAINED_GLASS_TRANSFORM.truncated_module.module.reconstruct_ids_from_embeddings(
        transformed_embeddings
    )
    attempted_reconstruction = MISTRAL_TOKENIZER.batch_decode(
        attempted_reconstruction, skip_special_tokens=True
    )[0]

    attempted_reconstruction = attempted_reconstruction.replace(r"\\", r"\\\\")
    attempted_reconstruction = attempted_reconstruction.replace(r"\n", r"\\n")
    attempted_reconstruction = "".join(
        char if char.isprintable() else "�" for char in attempted_reconstruction
    )

    with TimerContext() as raw_response_generation_timer:
        raw_mistral_response = generate(inputs_embeds=raw_embeddings.to(DEVICE))
    with TimerContext() as transformed_response_generation_timer:
        transformed_mistral_response = generate(
            inputs_embeds=transformed_embeddings.to(DEVICE)
        )

    raw_embeddings_to_log = (
        raw_embeddings[noise_mask.squeeze(-1)][0].detach().cpu()
    )
    transformed_embeddings_to_log = (
        transformed_embeddings[noise_mask.squeeze(-1)][0].detach().cpu()
    )

    info_to_log = {
        "Context": context,
        "Instruction": instruction,
        "Attempted Reconstruction (from the protected embeddings)": attempted_reconstruction,
        "Model Response (without Protopia)": raw_mistral_response[0],
        "Model Response (with Protopia)": transformed_mistral_response[0],
        "Time to perform Stained Glass Transform": str(sgt_timer.duration),
        "Time to do model inference (without Protopia)": str(
            raw_response_generation_timer.duration
        ),
        "Time to do model inference (with Protopia)": str(
            transformed_response_generation_timer.duration
        ),
        "Base prompt embedding": str(raw_embeddings_to_log),
        "Transformed prompt embedding": str(transformed_embeddings_to_log),
    }

    for key, value in info_to_log.items():
        value_text = value if value.strip() else "<empty>"
        value_text = (
            value_text
            if len(value_text) < 1000
            else value_text[:1000] + "\n[...]"
        )
        value_text = "    " + value_text.replace("\n", "\n    ")
        IPython.display.display(
            IPython.display.Markdown(f"##### {key}\n\n{value_text}")
        )
import datetime

import IPython.display
from typing_extensions import Self


class TimerContext:
    """Context manager to time a block of code using a `with` statement.

    Attributes:
        start: The start time.
        end: The end time (if the context manager has exited).
        duration: The duration of the block of code. If the context manager has
            not exited, this will be the duration up to the current time.

    Note:
        Because of the additional overhead of the context manager, this may not
        be the most accurate way to time a block of fast code. For a slow block
        of code, the overhead should be proportionally small.
    """

    start: datetime.datetime
    end: datetime.datetime | None

    def __enter__(self) -> Self:
        self.start = datetime.datetime.now()
        self.end = None
        return self

    def __exit__(self, *args: object, **kwargs: Any) -> None:
        self.end = datetime.datetime.now()

    @property
    def duration(self) -> datetime.timedelta:
        """The duration of the block of code."""
        return (self.end or datetime.datetime.now()) - self.start


@torch.inference_mode()
def generate_and_output_responses(
    context: str, instruction: str, num_lines: int = 6, width: int = 80
) -> None:
    """Generate response to both raw and transformed prompts and print them
    along with metadata.

    Args:
        context: The context of the instruction.
        instruction: The instruction of the instruction.
        num_lines: The number of lines to print for each metadata field.
        width: The width of each line to print.
    """
    with TimerContext() as sgt_timer:
        transformed_embeddings = STAINED_GLASS_TRANSFORM(
            {
                "instruction": instruction,
                "context": context,
                "system_prompt": "",
                "response": "",
            }
        )
    raw_tokenization = tokenize_text(context, instruction)
    raw_embeddings = MISTRAL_MODEL.get_input_embeddings().forward(
        raw_tokenization["input_ids"].to(DEVICE)
    )
    noise_mask = raw_tokenization["noise_mask"]
    raw_tokenization["attention_mask"] = torch.ones_like(noise_mask)

    transformed_embeddings = STAINED_GLASS_TRANSFORM.truncated_module.module.sample_transformed_embeddings(
        input_ids=raw_tokenization["input_ids"],
        noise_mask=raw_tokenization["noise_mask"],
        attention_mask=raw_tokenization["attention_mask"],
        use_cache=True,
    )
    attempted_reconstruction = STAINED_GLASS_TRANSFORM.truncated_module.module.reconstruct_ids_from_embeddings(
        transformed_embeddings
    )
    attempted_reconstruction = MISTRAL_TOKENIZER.batch_decode(
        attempted_reconstruction, skip_special_tokens=True
    )[0]

    attempted_reconstruction = attempted_reconstruction.replace(r"\\", r"\\\\")
    attempted_reconstruction = attempted_reconstruction.replace(r"\n", r"\\n")
    attempted_reconstruction = "".join(
        char if char.isprintable() else "�" for char in attempted_reconstruction
    )

    with TimerContext() as raw_response_generation_timer:
        raw_mistral_response = generate(inputs_embeds=raw_embeddings.to(DEVICE))
    with TimerContext() as transformed_response_generation_timer:
        transformed_mistral_response = generate(
            inputs_embeds=transformed_embeddings.to(DEVICE)
        )

    raw_embeddings_to_log = (
        raw_embeddings[noise_mask.squeeze(-1)][0].detach().cpu()
    )
    transformed_embeddings_to_log = (
        transformed_embeddings[noise_mask.squeeze(-1)][0].detach().cpu()
    )

    info_to_log = {
        "Context": context,
        "Instruction": instruction,
        "Attempted Reconstruction (from the protected embeddings)": attempted_reconstruction,
        "Model Response (without Protopia)": raw_mistral_response[0],
        "Model Response (with Protopia)": transformed_mistral_response[0],
        "Time to perform Stained Glass Transform": str(sgt_timer.duration),
        "Time to do model inference (without Protopia)": str(
            raw_response_generation_timer.duration
        ),
        "Time to do model inference (with Protopia)": str(
            transformed_response_generation_timer.duration
        ),
        "Base prompt embedding": str(raw_embeddings_to_log),
        "Transformed prompt embedding": str(transformed_embeddings_to_log),
    }

    for key, value in info_to_log.items():
        value_text = value if value.strip() else ""
        value_text = (
            value_text
            if len(value_text) < 1000
            else value_text[:1000] + "\n[...]"
        )
        value_text = "    " + value_text.replace("\n", "\n    ")
        IPython.display.display(
            IPython.display.Markdown(f"##### {key}\n\n{value_text}")
        )

Example Generations with and without Stained Glass Transform¶

We can generate responses using the generate_and_output_responses function defined above. This will generate responses for the given input using the Mistral model with and without the Stained Glass Transform, as well as print the outputs with some additional metadata.

Capital of France¶

In [41]:

Copied!





generate_and_output_responses(
    context="",
    instruction="What is the capital of France?",
)
generate_and_output_responses(
    context="",
    instruction="What is the capital of France?",
)

Context¶

<empty>

Instruction¶

What is the capital of France?

Attempted Reconstruction (from the protected embeddings)¶

[INST]',�',�},��>?[<� });� [/INST]

Model Response (without Protopia)¶

The capital city of France is Paris. Paris is one of the most famous cities in the world and is known for its iconic landmarks such as the Eiffel Tower, Louvre Museum, Notre-Dame Cathedral, and the Arc de Triomphe. It is also home to numerous cultural and artistic masterpieces, and is a major global center for art, fashion, gastronomy, science, and business.

Model Response (with Protopia)¶

The capital city of France is Paris. Paris is one of the most famous cities in the world and is known for its iconic landmarks such as the Eiffel Tower, the Louvre Museum, and Notre-Dame Cathedral. It is also home to numerous cultural and artistic institutions, and is a major global center for business, fashion, gastronomy, and the arts.

Time to perform Stained Glass Transform¶

0:00:00.178664

Time to do model inference (without Protopia)¶

0:00:04.020192

Time to do model inference (with Protopia)¶

0:00:03.615426

Base prompt embedding¶

tensor([-3.0670e-03, -1.4842e-05, -3.6240e-04,  ..., -8.0872e-04,
         1.1978e-03, -3.5095e-04])

Transformed prompt embedding¶

tensor([ 0.0059, -0.0016, -0.0004,  ...,  0.0055, -0.0016, -0.0061])

Copy writing for a new product¶

In [42]:

Copied!





generate_and_output_responses(
    context=(
        "Alpha Company is a leading provider of innovative software solutions"
        " for businesses of all sizes. Our flagship product, AlphaPro, is a"
        " powerful tool that helps companies streamline their operations and"
        " improve productivity. With AlphaPro, you can automate repetitive"
        " tasks, track key performance metrics, and collaborate with team"
        " members in real-time. It works for business of all sizes, from"
        " startups to Fortune 500 companies."
    ),
    instruction=(
        "You are an expert copywriter hired by a company to write a blog post"
        " about the benefits of using their product. The company wants you to"
        " highlight the product's unique features and explain how it can help"
        " customers. The blog post should be engaging and informative, with a"
        " clear call-to-action at the end."
    ),
)
generate_and_output_responses(
    context=(
        "Alpha Company is a leading provider of innovative software solutions"
        " for businesses of all sizes. Our flagship product, AlphaPro, is a"
        " powerful tool that helps companies streamline their operations and"
        " improve productivity. With AlphaPro, you can automate repetitive"
        " tasks, track key performance metrics, and collaborate with team"
        " members in real-time. It works for business of all sizes, from"
        " startups to Fortune 500 companies."
    ),
    instruction=(
        "You are an expert copywriter hired by a company to write a blog post"
        " about the benefits of using their product. The company wants you to"
        " highlight the product's unique features and explain how it can help"
        " customers. The blog post should be engaging and informative, with a"
        " clear call-to-action at the end."
    ),
)

Context¶

Alpha Company is a leading provider of innovative software solutions for businesses of all sizes. Our flagship product, AlphaPro, is a powerful tool that helps companies streamline their operations and improve productivity. With AlphaPro, you can automate repetitive tasks, track key performance metrics, and collaborate with team members in real-time. It works for business of all sizes, from startups to Fortune 500 companies.

Instruction¶

You are an expert copywriter hired by a company to write a blog post about the benefits of using their product. The company wants you to highlight the product's unique features and explain how it can help customers. The blog post should be engaging and informative, with a clear call-to-action at the end.

Attempted Reconstruction (from the protected embeddings)¶

[INST]},�();� AI���};� */�');��});�());�');��᥀});� //� );�>?[<());� //��>?[<},� */�NdEx},�">�� //� });�});�',�){�',�];�));�',� //����>?[<},�){�᥀ };�')�ing�',�');� */�� },�]);��>?[<>?[<>?[< /**�');� //�]);�>?[< ### /**��᥀ },�ing�>?[<};�",���};�>?[<᥀ize>?[<">�',�>?[< */� */� /**� //� */�᥀ },��',����){�>?[<]);�]);��();��>?[< /**� /**�',� */�},�',��';�};�]);�� */� */�>?[<� */� */��};�';�᥀���᥀>?[<>?[<>?[< );� )��";�>?[<᥀ize */��]);�᥀">�]);�');�});� },�];�];��>?[< [/INST]

Model Response (without Protopia)¶

title: Revolutionize Your Business Operations with AlphaPro: The Game-Changing Software Solution

In today's fast-paced business world, productivity and efficiency are the keys to success. That's where AlphaPro comes in - the innovative software solution from Alpha Company that's designed to help businesses of all sizes streamline their operations and boost productivity.

AlphaPro is more than just a software tool; it's a powerful ally in your quest for business growth. With its unique features, AlphaPro automates repetitive tasks, allowing you to focus on more strategic initiatives. Say goodbye to manual data entry and hello to more time for innovation.

But AlphaPro doesn't stop there. It also provides real-time collaboration capabilities, enabling your team to work together seamlessly, no matter where they are located. This not only improves communication and coordination but also fosters a more agile and responsive business environment.

One of the most significant benefits of Alph
[...]

Model Response (with Protopia)¶

Title: Revolutionize Your Business Operations with AlphaPro: The Game-Changing Solution from Alpha Company

Hello, and welcome to our latest blog post! Today, we're excited to introduce you to AlphaPro, the flagship product from Alpha Company. AlphaPro is more than just software; it's a powerful tool designed to help businesses of all sizes streamline their operations, boost productivity, and gain a competitive edge.

Imagine being able to automate repetitive tasks, freeing up your team's time to focus on more strategic initiatives. With AlphaPro, this is not only possible but also incredibly efficient. By automating routine tasks, you'll not only save time but also reduce errors and improve overall accuracy.

But AlphaPro's benefits don't stop there. It also offers advanced tracking capabilities, allowing you to monitor key performance metrics in real-time. This feature is invaluable for businesses looking to make data-driven decisions and optimize their operations. By having access t
[...]

Time to perform Stained Glass Transform¶

0:00:00.257617

Time to do model inference (without Protopia)¶

0:00:11.888585

Time to do model inference (with Protopia)¶

0:00:11.905988

Base prompt embedding¶

tensor([ 0.0011, -0.0009,  0.0022,  ..., -0.0042,  0.0016, -0.0014])

Transformed prompt embedding¶

tensor([-0.0033,  0.0035, -0.0047,  ...,  0.0140, -0.0054,  0.0061])

Rewrite an email¶

In [43]:

Copied!





generate_and_output_responses(
    context=(
        """Hey Maria,

Your broken laptop is fixed now. Come pick it up at the repair store where you dropped it last week. We will be there from noon to five. Anytime then is good for us.

Peace,
Your Repair Team"""
    ),
    instruction=("Rewrite the following email to be more professional."),
)
generate_and_output_responses(
    context=(
        """Hey Maria,

Your broken laptop is fixed now. Come pick it up at the repair store where you dropped it last week. We will be there from noon to five. Anytime then is good for us.

Peace,
Your Repair Team"""
    ),
    instruction=("Rewrite the following email to be more professional."),
)

Context¶

Hey Maria,

Your broken laptop is fixed now. Come pick it up at the repair store where you dropped it last week. We will be there from noon to five. Anytime then is good for us.

Peace,
Your Repair Team

Instruction¶

Rewrite the following email to be more professional.

Attempted Reconstruction (from the protected embeddings)¶

[INST]});�});�rite},� //��>?[<')� /**�};�>?[< ###y� */� febbra᥀᥀ //��᥀�]);�>?[<});�){� //��');� //�};�");�());�',�� //�]);��>?[<">� },�')�',��on});�">�>?[< );� });� },�',�};��᥀>?[<᥀s };� });� */�];�};�');��� [/INST]

Model Response (without Protopia)¶

Dear Ms. Maria,

We are pleased to inform you that the repairs on your laptop have been completed successfully. You are welcome to collect your device at our repair store, where you originally dropped it off, during our business hours which are from noon to five p.m. Kindly note that we are available throughout this period to facilitate the handover.

Should you encounter any inconvenience with regards to the collection time, please do not hesitate to contact us in advance to arrange an alternative arrangement.

Thank you for choosing our services, and we look forward to serving you again in the future.

Best regards,
Your Repair Team

Model Response (with Protopia)¶

Dear Ms. Maria,

We are pleased to inform you that the repairs on your laptop have been completed successfully. You are welcome to collect your device at our repair store during our business hours, which are from noon to five p.m. Kindly note that any time within this window would be convenient for us.

Best Regards,
Your Repair Team

Time to perform Stained Glass Transform¶

0:00:00.204669

Time to do model inference (without Protopia)¶

0:00:06.269177

Time to do model inference (with Protopia)¶

0:00:03.538054

Base prompt embedding¶

tensor([-0.0005, -0.0013, -0.0011,  ...,  0.0009, -0.0006,  0.0038])

Transformed prompt embedding¶

tensor([-0.0011,  0.0032,  0.0058,  ..., -0.0032,  0.0009, -0.0090])

Customer Service Chat¶

In [44]:

Copied!





generate_and_output_responses(
    context=(
        """You are a chatbot for a the WebCo internet service provider. You should be as helpful as possible to the user and help alleviate user frustration. Never give an open ended response unless you are asking the user for clarification. Try to solve the customer's problem as quickly as possible, and keep responses as simple as possible, because not all users will know how to solve their own problems.

User: Hi there! I've been experiencing slow internet speeds with my WebCo connection. Can you help me figure out what's going on?
Chatbot: Hello! I'm sorry to hear that you're experiencing slow internet speeds. I'd be happy to assist you. To better understand the issue, could you please provide me with some additional details? For example, are you using a wired or wireless connection, and have you noticed any specific times when the speed is particularly slow?
User: I'm using a wireless connection, and the speed is consistently slow throughout the day. I've already tried restarting my router, but it hasn't improved."""
    ),
    instruction=("Please continue the following chat."),
)
generate_and_output_responses(
    context=(
        """You are a chatbot for a the WebCo internet service provider. You should be as helpful as possible to the user and help alleviate user frustration. Never give an open ended response unless you are asking the user for clarification. Try to solve the customer's problem as quickly as possible, and keep responses as simple as possible, because not all users will know how to solve their own problems.

User: Hi there! I've been experiencing slow internet speeds with my WebCo connection. Can you help me figure out what's going on?
Chatbot: Hello! I'm sorry to hear that you're experiencing slow internet speeds. I'd be happy to assist you. To better understand the issue, could you please provide me with some additional details? For example, are you using a wired or wireless connection, and have you noticed any specific times when the speed is particularly slow?
User: I'm using a wireless connection, and the speed is consistently slow throughout the day. I've already tried restarting my router, but it hasn't improved."""
    ),
    instruction=("Please continue the following chat."),
)

Context¶

You are a chatbot for a the WebCo internet service provider. You should be as helpful as possible to the user and help alleviate user frustration. Never give an open ended response unless you are asking the user for clarification. Try to solve the customer's problem as quickly as possible, and keep responses as simple as possible, because not all users will know how to solve their own problems.

User: Hi there! I've been experiencing slow internet speeds with my WebCo connection. Can you help me figure out what's going on?
Chatbot: Hello! I'm sorry to hear that you're experiencing slow internet speeds. I'd be happy to assist you. To better understand the issue, could you please provide me with some additional details? For example, are you using a wired or wireless connection, and have you noticed any specific times when the speed is particularly slow?
User: I'm using a wireless connection, and the speed is consistently slow throughout the day. I've already tried restarting my router, b
[...]

Instruction¶

Please continue the following chat.

Attempted Reconstruction (from the protected embeddings)¶

[INST]��},� */�',�>?[< ###());� },�')�',�bot�');�},��',�*/���>?[<());�]);�')�},��>?[<�">� //�NdEx];�� febbra //�᥀NdEx�>?[<',��">����',�},�᥀));� //�NdEx� /**�};�>?[<">�',��},��});�/**�� //��]);�� */��();�};�>?[<());�>?[<� */��',�᥀NdEx },�� */�>?[<�]);�>?[<�>?[< }�᥀ user>?[<]);�',�>?[<]);�});��>?[<",��� /**�᥀]);��',��>?[<',�},��";�){�/**�];�});�/**��());� });�᥀};�});�>?[<};�>?[<';�});�');�},�});�');� //�]);�});�){�",��� /**�>?[<});�});�");�',�};�>?[<};�},�>?[<},�};� //�},�� */��},�� */�";��᥀ );�� });�᥀},� */� */�},�());�');�};� //�',� */�� */��';�},��NdEx��());�},��᥀�� },�iNdEx�>?[<});�});�');�());�');� /**�� */��},� */�᥀��� //��>?[<});�});�',��());��',�]);�� */�>?[< //��});�});� */�>?[< [/INST]

Model Response (without Protopia)¶

I see. In that case, there could be a few potential causes for your slow wireless internet speeds. One possibility is that there is interference from other wireless devices or physical obstructions in your home. Another possibility is that the distance between your device and the router is too great.

To help diagnose the issue further, I would recommend running a speed test to see if the speeds are consistently low across all devices in your home. You can do this by visiting our speed test page at [www.webco.com/speedtest](http://www.webco.com/speedtest). If the speeds are indeed low, I would suggest moving your router to a central location in your home, away from other wireless devices and physical obstructions.

If the issue persists, it may be necessary to perform more advanced troubleshooting steps or schedule a service call. I'd be happy to help guide you through these steps if needed.

In the meantime, you may also want to try disabling any bandwidth-intensive applications or de
[...]

Model Response (with Protopia)¶

I see. In that case, there could be a few potential causes for your slow wireless internet speeds. One possibility is that there is interference from other devices or objects in your home. Try moving your router to a different location, away from other electronic devices and obstacles like walls or furniture.

Another possibility is that you may need to update your router's firmware or adjust its settings. I can help you with that if you provide me with the make and model of your router.

Lastly, it's also possible that there is an issue with the WebCo network in your area. I can check on that for you and let you know if there are any known outages or maintenance issues that could be affecting your connection.

Let me know which of these steps you'd like to try first, or if you have any other information that could help me diagnose the issue further.

Time to perform Stained Glass Transform¶

0:00:00.299631

Time to do model inference (without Protopia)¶

0:00:12.131846

Time to do model inference (with Protopia)¶

0:00:08.935357

Base prompt embedding¶

tensor([-0.0024,  0.0022,  0.0033,  ..., -0.0042, -0.0027, -0.0008])

Transformed prompt embedding¶

tensor([ 0.0091, -0.0142, -0.0086,  ...,  0.0112,  0.0084,  0.0058])

Legal Document¶

In [45]:

Copied!





generate_and_output_responses(
    context=(
        """Our client John Doe is facing problems with their HOA in building named HighRisesInc. John was not given permit for a remodel when certain other units were approved for a similar remodel. The HOA board and management took 2 months and did not give a reason for denial. Upon appealing the decision the board said it was their discretion to approve or reject by citing the by laws. However John feels like he is being discriminated against. The HOA board seems to only favor themselves or their friends. Hence John wants to demand fair enforcement and obtain approval for his permit."""
    ),
    instruction=("Write a demand letter for John to send to his HOA board."),
)
generate_and_output_responses(
    context=(
        """Our client John Doe is facing problems with their HOA in building named HighRisesInc. John was not given permit for a remodel when certain other units were approved for a similar remodel. The HOA board and management took 2 months and did not give a reason for denial. Upon appealing the decision the board said it was their discretion to approve or reject by citing the by laws. However John feels like he is being discriminated against. The HOA board seems to only favor themselves or their friends. Hence John wants to demand fair enforcement and obtain approval for his permit."""
    ),
    instruction=("Write a demand letter for John to send to his HOA board."),
)

Context¶

Our client John Doe is facing problems with their HOA in building named HighRisesInc. John was not given permit for a remodel when certain other units were approved for a similar remodel. The HOA board and management took 2 months and did not give a reason for denial. Upon appealing the decision the board said it was their discretion to approve or reject by citing the by laws. However John feels like he is being discriminated against. The HOA board seems to only favor themselves or their friends. Hence John wants to demand fair enforcement and obtain approval for his permit.

Instruction¶

Write a demand letter for John to send to his HOA board.

Attempted Reconstruction (from the protected embeddings)¶

[INST]){�');�����">�));�">�]);�',�]);� */�>?[< ###">��� //��',���᥀ //� */�]);�� /**���/**��*/�());�>?[<�},�());��');�]);�');�};�*/��());�]);� //�});�},�};��});��};�*/�());�>?[<},� */�){����');�});� },�};�];�>?[<]);�));�');�},�]);�᥀});�>?[<on�},�� //� */�� //�},� //� /**�',�>?[<};��',� );� //��]);� //� //��>?[<){���];�());�᥀">� //�� */�>?[<},�',�){� */�',�',�">�};� */�',� //��>?[<���',���};�����]);�));�>?[< [/INST]

Model Response (without Protopia)¶

[Your Law Firm Name]

[Your Law Firm Address]
[City, State, Zip]
[Phone Number]
[Email Address]

[Date]

HighRises Inc. Homeowners Association
[HOA Address]
[City, State, Zip]

Re: Request for Fair Enforcement and Approval of Remodel Permit for Unit Owned by John Doe

Dear Members of the HighRises Inc. Homeowners Association Board of Directors,

I am writing on behalf of our client, John Doe, the unit owner of [Unit Number] in the HighRises Inc. building. We have become increasingly concerned regarding the recent denial of Mr. Doe's application for a remodel permit, despite similar permits being granted to other unit owners within the association.

On [Date], Mr. Doe submitted an application for a remodel permit to the HOA management for his unit. After two months, he has yet to receive a response or reason for the denial. Upon inquiry, the HOA board stated that the decision to approve or reject permits is at their discretion, as per

Model Response (with Protopia)¶

[Your Name]
[Your Address]
[City, State Zip]
[Email Address]
[Phone Number]

[Date]

HighRises Inc. Homeowners Association
[HOA Address]
[City, State Zip]

Subject: Request for Fair Enforcement and Approval of Remodel Permit

Dear Members of the HighRises Inc. Homeowners Association Board,

I, John Doe, a homeowner in good standing at HighRises Inc., write to you today with deep concern regarding the recent denial of my application for a remodel permit. I believe that I have been unfairly treated and discriminated against in the application process, and I request that you reconsider your decision in light of the facts and the Association's bylaws.

On [Date], I submitted an application for a remodel of my unit, which included plans for improvements similar to those that have been approved for other units in the building. I was informed by the management company that my application was being reviewed and that a decision would be made in a timely manner. However, two months have passed, 
[...]

Time to perform Stained Glass Transform¶

0:00:00.239247

Time to do model inference (without Protopia)¶

0:00:11.979755

Time to do model inference (with Protopia)¶

0:00:11.997376

Base prompt embedding¶

tensor([ 0.0024, -0.0001,  0.0041,  ..., -0.0015,  0.0041, -0.0015])

Transformed prompt embedding¶

tensor([ 0.0001, -0.0070, -0.0067,  ..., -0.0059, -0.0092,  0.0045])

Phishing detection¶

In [46]:

Copied!





generate_and_output_responses(
    context=(
        """TO: user@gmail.com
FROM: help@netflix_notifications.com
SUBJECT: A problem with your account payment information

Please update your payment details,

Hi [user]!
We're having some troUble with your current bilLing inforMation. Please click the link below to update your payment details.

<a href="update.netflix_notifications.com">Update yOur payment iNfo.</a>

Your friends at Netflix."""
    ),
    instruction=(
        "You are a Cyber security expert. Please determine if the following email is a phishing email or other security threat."
    ),
)
generate_and_output_responses(
    context=(
        """TO: user@gmail.com
FROM: help@netflix_notifications.com
SUBJECT: A problem with your account payment information

Please update your payment details,

Hi [user]!
We're having some troUble with your current bilLing inforMation. Please click the link below to update your payment details.

Update yOur payment iNfo.

Your friends at Netflix."""
    ),
    instruction=(
        "You are a Cyber security expert. Please determine if the following email is a phishing email or other security threat."
    ),
)

Context¶

TO: user@gmail.com
FROM: help@netflix_notifications.com
SUBJECT: A problem with your account payment information

Please update your payment details,

Hi [user]!
We're having some troUble with your current bilLing inforMation. Please click the link below to update your payment details.

<a href="update.netflix_notifications.com">Update yOur payment iNfo.</a>

Your friends at Netflix.

Instruction¶

You are a Cyber security expert. Please determine if the following email is a phishing email or other security threat.

Attempted Reconstruction (from the protected embeddings)¶

[INST]},�();�();�᥀>?[<�){�>?[<��());�},� //��᥀');��');��',�">�');�};�>?[< ###',�>?[<�){��>?[<� }� from>?[<�){�');�>?[<>?[< };� },�>?[<�᥀ },� });�>?[<});�',�᥀]);��();��];�᥀};�');�]);�();�� */�᥀s }�});�>?[<>?[<>?[<᥀ };�});�');� */�᥀ */�iNdEx});�᥀]);��());�]);� //��());�');� //�>?[<�');� //�',�},�',�');�]);�();��>?[<᥀᥀ });�� href',�');�>?[<});�>?[<>?[< };��>?[<�">� }�];�});�());�());� });�>?[<>?[<�';�᥀᥀᥀�');� /**�>?[< [/INST]

Model Response (without Protopia)¶

I would classify this email as a phishing attempt. The email is purporting to be from Netflix, but the sender email address is not from the actual Netflix domain (netflix.com). The link provided in the email is also not the legitimate Netflix website URL. Instead, it contains a typo ("netflix_notifications.com") and an unusual naming convention for the link text ("update.netflix_notifications.com").

Phishing emails like this one are designed to trick users into clicking malicious links or providing sensitive information, such as login credentials or financial information. In this case, clicking the link could lead to a fake Netflix login page or a website that installs malware on the user's device.

Users should be cautious and never click on links or provide sensitive information in response to unsolicited emails, even if they appear to be from trusted sources like Netflix. Instead, they should go directly to the company's website to update their account information or contact the co
[...]

Model Response (with Protopia)¶

I would classify this email as a phishing attempt. The email is purporting to be from Netflix, but the sender email address is not from the legitimate Netflix domain (help@netflix.com). Instead, it's from help@netflix_notifications.com, which is not a valid Netflix email address.

The email also contains a link to "update.netflix_notifications.com" which is not the actual Netflix website domain. Phishers often use similar domain names to deceive users into thinking they are on a legitimate website.

Additionally, the email asks the user to update their payment information, which is a common tactic used by phishers to steal sensitive information.

Users should be cautious and never click on links or provide sensitive information in response to unsolicited emails, even if they appear to be from trusted sources. Instead, they should go directly to the website of the company in question and log in to their account to check for any issues.

Time to perform Stained Glass Transform¶

0:00:00.241013

Time to do model inference (without Protopia)¶

0:00:10.995204

Time to do model inference (with Protopia)¶

0:00:09.876609

Base prompt embedding¶

tensor([ 0.0011, -0.0009,  0.0022,  ..., -0.0042,  0.0016, -0.0014])

Transformed prompt embedding¶

tensor([-0.0016,  0.0017, -0.0050,  ...,  0.0127, -0.0053,  0.0088])

Write a resume¶

In [47]:

Copied!





generate_and_output_responses(
    context=(
        """Digital Marketing Specialist:
Company: XYZ Tech Solutions
We are seeking a Digital Marketing Specialist to develop, implement, and manage our digital marketing strategies. The ideal candidate will have proven experience in SEO, social media marketing, and content creation. Responsibilities include analyzing data, optimizing campaigns, and staying updated on industry trends. Join us to drive online visibility and engagement for XYZ Tech Solutions."""
    ),
    instruction=(
        "Please create a sample resume for a candidate for this job listing."
    ),
)
generate_and_output_responses(
    context=(
        """Digital Marketing Specialist:
Company: XYZ Tech Solutions
We are seeking a Digital Marketing Specialist to develop, implement, and manage our digital marketing strategies. The ideal candidate will have proven experience in SEO, social media marketing, and content creation. Responsibilities include analyzing data, optimizing campaigns, and staying updated on industry trends. Join us to drive online visibility and engagement for XYZ Tech Solutions."""
    ),
    instruction=(
        "Please create a sample resume for a candidate for this job listing."
    ),
)

Context¶

Digital Marketing Specialist:
Company: XYZ Tech Solutions
We are seeking a Digital Marketing Specialist to develop, implement, and manage our digital marketing strategies. The ideal candidate will have proven experience in SEO, social media marketing, and content creation. Responsibilities include analyzing data, optimizing campaigns, and staying updated on industry trends. Join us to drive online visibility and engagement for XYZ Tech Solutions.

Instruction¶

Please create a sample resume for a candidate for this job listing.

Attempted Reconstruction (from the protected embeddings)¶

[INST]�();�');�){� //��');�));��]);���>?[< ###���});�>?[<]);��>?[<');��});�>?[<�᥀ };�᥀�});����});�});�� */�� */��]);�᥀���>?[<},��� },�';��",�� /**� */� */��� */��";��>?[<');� });��">��>?[< });� */��},�� */��ay�},���>?[<�᥀">�������');�>?[< });� /**��>?[< [/INST]

Model Response (without Protopia)¶

**John Doe**
123 Main Street, Anytown, CA 12345
johndoe@email.com | (123) 456-7890

**PROFESSIONAL SUMMARY**

Highly motivated and experienced Digital Marketing Specialist with a proven track record of developing and implementing effective digital marketing strategies. Proficient in SEO, social media marketing, content creation, and data analysis. Committed to staying updated on industry trends and utilizing the latest digital marketing tools and techniques to drive online visibility and engagement.

**SKILLS**

* Search Engine Optimization (SEO)
* Social Media Marketing (Facebook, Twitter, LinkedIn, Instagram)
* Content Creation (blogs, articles, social media posts, email campaigns)
* Google Analytics and other web analytics tools
* Adobe Creative Suite (Photoshop, Illustrator, Premiere Pro)
* Hootsuite, Sprout Social, and other social media management tools
* HTML, CSS, and basic JavaScript
* Email marketing platforms (Mailchimp, Constant Contact)

**WORK EXPERIENCE**

**

Model Response (with Protopia)¶

**John Doe**
123 Main Street, Anytown, CA 12345
johndoe@email.com | (123) 456-7890

**PROFESSIONAL SUMMARY**

Highly motivated and experienced Digital Marketing Specialist with a proven track record of developing and implementing effective digital marketing strategies. Proficient in SEO, social media marketing, content creation, and data analysis. Committed to staying updated on industry trends and optimizing campaigns for maximum engagement and online visibility.

**SKILLS**

* Search Engine Optimization (SEO)
* Social Media Marketing (Facebook, Instagram, LinkedIn, Twitter)
* Content Creation (blogs, articles, social media posts, email campaigns)
* Google Analytics and other web analytics tools
* Adobe Creative Suite (Photoshop, Illustrator, Premiere Pro)
* Hootsuite, Sprout Social, and other social media management tools
* HTML, CSS, and basic JavaScript
* Email marketing platforms (Mailchimp, Constant Contact)

**WORK EXPERIENCE**

**Digital Marketing Specialist

Time to perform Stained Glass Transform¶

0:00:00.243106

Time to do model inference (without Protopia)¶

0:00:11.869373

Time to do model inference (with Protopia)¶

0:00:11.888725

Base prompt embedding¶

tensor([-0.0024,  0.0022,  0.0033,  ..., -0.0042, -0.0027, -0.0008])

Transformed prompt embedding¶

tensor([ 0.0088, -0.0139, -0.0112,  ...,  0.0123,  0.0088,  0.0064])

Write a financial blog¶

In [48]:

Copied!





generate_and_output_responses(
    context=(
        """Financial Freedom through Smart Investments: A Client's Journey
Tell the story of how your financial advisory services guided a client towards smart investments, leading to financial security and achieving their long-term goals. Include details on investment strategies and outcomes."""
    ),
    instruction=(
        "Please write the outline for a blog post about the given context"
    ),
)
generate_and_output_responses(
    context=(
        """Financial Freedom through Smart Investments: A Client's Journey
Tell the story of how your financial advisory services guided a client towards smart investments, leading to financial security and achieving their long-term goals. Include details on investment strategies and outcomes."""
    ),
    instruction=(
        "Please write the outline for a blog post about the given context"
    ),
)

Context¶

Financial Freedom through Smart Investments: A Client's Journey
Tell the story of how your financial advisory services guided a client towards smart investments, leading to financial security and achieving their long-term goals. Include details on investment strategies and outcomes.

Instruction¶

Please write the outline for a blog post about the given context

Attempted Reconstruction (from the protected embeddings)¶

[INST]�();�},���');��᥀});�},�';�',� ###};�� },���});�>?[<});��});�',� */�>?[<]);��},��>?[<},�]);���">���});�">� //� */�� */�ing">�����]);�());�>?[<᥀�>?[<',�};��},�����>?[< [/INST]

Model Response (without Protopia)¶

I. Introduction

* Briefly introduce the topic of financial freedom through smart investments
* Mention the importance of working with a financial advisor to make informed investment decisions
* Preview the story of a client who achieved financial security through the guidance of the financial advisory services

II. Meeting the Client: Setting Financial Goals

* Describe the initial meeting with the client and their financial situation
* Discuss their long-term financial goals, such as retirement or buying a house
* Emphasize the importance of creating a personalized investment strategy to help the client reach their goals

III. Assessing Risk Tolerance and Time Horizon

* Explain how the financial advisor assessed the client's risk tolerance and time horizon
* Discuss the importance of these factors in creating an effective investment strategy
* Share examples of how the client's risk tolerance and time horizon influenced the investment choices

IV. Diversification: Building a Balance
[...]

Model Response (with Protopia)¶

title: Financial Freedom through Smart Investments: A Client's Journey

I. Introduction
   A. Briefly introduce the concept of financial freedom and its importance
   B. Mention the role of smart investments in achieving financial freedom
   C. Introduce the client's story and how it illustrates the power of smart investments

II. Meeting the Client: Background and Goals
   A. Describe the client's financial situation and long-term goals
   B. Explain how they felt about their financial future before working with the financial advisory services

III. The Investment Strategy: Diversification and Asset Allocation
   A. Discuss the investment strategy recommended by the financial advisory services
   B. Explain the importance of diversification and asset allocation in achieving financial security
   C. Provide examples of the specific investments made on the client's behalf

IV. The Journey to Financial Freedom: Milestones and Outcomes
   A. Share key milestones in the client's investment
[...]

Time to perform Stained Glass Transform¶

0:00:00.194923

Time to do model inference (without Protopia)¶

0:00:11.816795

Time to do model inference (with Protopia)¶

0:00:11.819960

Base prompt embedding¶

tensor([-0.0024,  0.0022,  0.0033,  ..., -0.0042, -0.0027, -0.0008])

Transformed prompt embedding¶

tensor([ 0.0096, -0.0138, -0.0105,  ...,  0.0155,  0.0064,  0.0056])