Introducing Intel’s new NLP model: Polite Guard

Ramya Ravi, AI Software Marketing Engineer, Intel | LinkedIn
Ehssan Khan, Senior AI Software Solutions Engineer, Intel

Intel has developed Polite Guard, an open-source Natural Language Processing model, for text classification tasks. This model is fine-tuned from BERT for classifying text into four different categories: polite, somewhat polite, neutral, and impolite. In addition to the model, explore the accompanying dataset on Hugging Face and the source code on GitHub.

Benefits

1. Scalable model development pipeline: Makes it easier for developers to generate their own synthetic data and fine-tune their models.
2. Improved Robustness: Enhances the resilience of systems by providing a defense mechanism against adversarial attacks.
3. Benchmarking and Evaluation: Allows developers to evaluate and compare the performance of their models in terms of politeness classification.
4. Enhanced Customer Experience: Improves customer satisfaction and loyalty by ensuring respectful and polite interactions on various platforms.

Synthetic Data Generation and Fine-tuning Process

Here, we implement a synthetic data generator in Python and run it on Intel® Xeon® Processors to produce labeled samples detailing customer service interactions across various sectors, including finance, travel, food, retail, sports, culture, and professional development. Then we fine-tune the base model, BERT, on this dataset.

To ensure data regularization, we randomly select the labels and categories and instruct a language model to generate synthetic data based on the specified categories and labels. To ensure data diversity, we use a variety of prompts and three large language models, namely, Llama 3.1-8B-Instruct, Mixtral 8x7B-Instruct-v0.1, and Gemma 2-9B-It, during the generation process.

Then, we tune the learning rate and weight decay hyperparameters by using the Tree-structured Parzen Estimator (TPE) algorithm via Optuna, aiming to maximize the validation F1-score. Also, we use Optuna’s pruning callback to terminate underperforming hyperparameter trials. For more information on the hyperparameter search space, and the best performing hyperparameters, refer to the model card on Hugging Face.

Figure 1. Parallel coordinate plot visualizing the relationship between hyperparameters (learning rate and weight decay) and the model's F1 score (objective value) during hyperparameter optimization. Each line represents a trial, showing how different hyperparameter combinations influence the performance.

Figure 2. Timeline plot of Optuna trials (0-34) showing trial progression over time, with completed and pruned trials distinguished.

Access and Usage

• Datasets:  The Polite Guard dataset on Hugging Face has three components:
  • 50,000 labeled samples generated using Few-Shot prompting.
  • 50,000 labeled samples generated using Chain-of-Thought (CoT) prompting.
  • 200 annotated samples from corporate trainings with masked personal identifiers.

The synthetic data is split into training (80%), validation (10%), and test (10%) sets, with each set balanced according to the label. Polite Guard is trained solely on synthetic data but evaluated on the test split of synthetic and real annotated data achieving 92.4% in accuracy and F1-score.

• Source Code: The source code for the Synthetic Data Generator and the Fine-tuner utilizing Al accelerators such as Intel® Gaudi® is available on Polite Guard’s GitHub repository.
• Execution: Developers are encouraged to clone the repository and run the code on the Intel® Tiber™ AI Cloud.

What’s Next

You can build more robust, respectful, and customer-friendly NLP applications by leveraging Polite Guard. Feel free to contribute to this open-source project and benefit from ongoing advancements in generative AI.

We also encourage you to check out and incorporate Intel’s other AI/ML Framework optimizations and tools into your AI workflow and, learn about the unified, open, standards-based oneAPI programming model that forms the foundation of Intel’s AI Software Portfolio.