Emotion Translating Using Transformers
Harshit Shah, Charlie Floeder, Milad Farizian, Rizq Khateeb, Yash Sharma
Emotional Translating is the task of changing the emotion conveyed in a piece of text while preserving its original meaning.
Motivation
- Emotionally charged text can many negative consequences. Cyberbullying and online attacks can have negative impacts on mental and physical health.
- Emotional paraphrasing could be used to moderate excessively negative or hateful messages on social media.
- There are numerous applications of emotionally-aware natural language understanding systems: healthcare, customer service, and more.
Problem Definition
Evaluate and compare the ability of different transformer models to be trained for emotion paraphrasing while ensuring emotional context is preserved in the altered text
Methodology
Sample Result: Disapproval → Admiration
Input: Paraphrase disapproval to admiration: Nietzsche admired Wagner's ability to express his own suffering and misery in large creations. He criticized Wagner's attempt to produce short works. of music.
Output: Nietzsche admired Wagner's ability to express his own suffering and misery in large creations.
Dataset Creation
- Used three publicly available datasets: Google PAWS-Wiki (PAWS), Microsoft Research Paraphrase Corpus (MRPC), and Quora Questions Pairs
- Each data sample contains text, an emotion, and an emotion confidence score
- Emotions and confidence scores were generated using an open-source package: GoEmotions
- Filtered out paraphrases that didn’t change emotions, and ordered text pairs in decreasing emotion intensity
- Sample
- Text: “The accusation that I misappropriated money from Jennifer Lopez is both untrue and offensive .”
- Emotion: Anger
- Confidence: 0.896
Fine-tuning Methods
- Fully Supervised Training:
- Train a pre-trained model on a large labeled dataset specific to the target task
- Use the entire dataset and provide all inputs and targets to the model
- Few-Shot Training:
- Adapt a pre-trained model to a new task using a very small amount of labeled data
- Define a few_shot parameter that defines how much data to use, essentially trying to rely on model’s inherent capability
- Zero Shot Training:
- Leverage a pre-trained model to make predictions on tasks it wasn’t explicitly trained for, without any additional task-specific training data
- Use the base model to generate paraphrasing
Evaluation
Metrics
Paraphrasing Metrics:
- BLEU:
- Measures the similarity of the inputted text to the outputted text. Largely based on precision
- ROUGE-L:
- Measures summarization quality of outputted text to reference text. Uses longest common subsequence instead of comparing n-grams
- METEOR:
- Measures the similarity of the inputted text to the outputted text and accounts for semantic similarity. Weights recall higher than precision
Emotion Transition Metrics:
- Sentiment Accuracy:
- The percentage of predicted sentiment ranges that match the target sentiment ranges
- e.g. high positive, neutral, low negative
- Emotion Accuracy:
- The percentage of predicted emotions that match the target emotion
- e.g. sadness, anger, love, approval
Initial Results on Paraphrasing
With use of the PAWS dataset and T5:
- BLEU score:
............0.40314368800802153
- ROUGE-L score:
............0.7428520650606598
- METEOR score:
............0.6614509549262314
Results, Limitations, and Future Works
- Results:
- Model fine-tuning and evaluation is still in progress, and full results will be available in final report
- Initial results for T5 model show promising metrics
- Limitations:
- Few training samples
- Lack of large public paraphrase datasets
- Many paraphrases have the same emotions in the original and paraphrased text
- Future Ideas:
- Investigate newer LLMs (e.g. GPT-4) performance on emotion transition task
- Research applications of emotion neutralization
Google Colab