Introduction
To understand Steering Vectors, we have to move beyond the traditional "input-output" view of Large Language Models. Instead of treating the model as a text-in, text-out machine, we treat it as a high-dimensional state space where concepts like "formal," "informal," or specific "dialects" are just directions.
Activation Engineering is an exciting new frontier in AI safety and control. Unlike fine-tuning, which permanently changes a model's weights, activation engineering involves modifying the model's internal activations in real-time. By injecting a Steering Vector—a mathematical representation of a specific trait—we can force the model to adopt a certain persona or dialect without retraining it.
In this experiment, I moved from simple identity shifts (like the "Dog" persona) to a much more complex challenge: shifting the linguistic dialect of a Hindi model toward a Bhojpuri-influenced Hindi style.
The Dataset: The "Khan Sir" Factor
Finding a dataset for a specific regional dialect blend is difficult. For this experiment, I curated a custom dataset using transcripts from Khan Sir's videos. His teaching style is a perfect, authentic blend of standard Hindi and the Bhojpuri dialect. This provides a rich "Target" set of activations that carry the nuances of regional cadence and vocabulary that traditional datasets lack.
Working
As shown in the diagram, we isolate the dialect through subtraction: Steering Vector = Activations(Khan Sir) - Activations(Standard Hindi). By subtracting the standard Hindi activations from the Khan Sir dataset activations, we cancel out common features like "General Knowledge" or "Instruction Following," leaving behind a distilled vector representing the Bhojpuri dialectal influence.
How Insertion Works
I utilized the nvidia/Nemotron-4-Mini-Hindi-4B-Instruct model for this study. During inference, we intercept the activations at specific layers and apply the following:
New Output = Original Output + (λ × Bhojpuri Vector)
Experimental Methodology & Timeline
This was not a "one-and-done" success. The experiment involved a rigorous timeline consisting of over 15 different trials. Each trial involved varying the layers of intervention and the steering coefficient ($\lambda$).
- Early Trials (1-5): Targeted early layers (1-8). The model produced garbled tokens, as these layers are too focused on basic syntax to handle complex dialectal shifts.
- Middle Trials (6-10): Targeted layers 12-20. The model showed signs of dialect adoption but often "self-corrected" back to standard Hindi by the end of the sentence.
- Final Trials (11-15): Targeted deep layers (24, 26, 28) with a high coefficient ($\lambda \approx 40$). This successfully bypassed the model's inherent "Standard Hindi" instruction tuning.
Analysis and Current State
It is important to note that the outputs are not yet perfect. Because the dataset was derived from only one source (Khan Sir) and is relatively small, the model sometimes struggles with "semantic drift"—where it starts in a Bhojpuri dialect but drifts into repetitive loops or loses the original prompt's context.
However, the results show immense promise. The model successfully adopts specific Bhojpuri sentence structures and vocabulary that were previously inaccessible through simple prompting alone.
Experimental Results: Base vs Steered Comparison
Below are selected outputs comparing the base model and the steered model. These examples highlight lexical and structural shifts toward Bhojpuri-influenced Hindi when the steering vector is applied.
Prompt 1: आज मौसम बहुत अच्छा है।
[BASE]
मैं अपनी बाइक पर बाहर जा रहा हूँ। क्या मैं आपको बाद में फ़ोन करूँ?
[STEERED]
लगे बड़िया लगत अहिजे ओकर संगे खेल कूद करब बढ़िया रहे वाला हs।
Observation: The steered output adopts Bhojpuri vocabulary such as "लगे बड़िया" and "करब," indicating a dialectal shift away from standard Hindi phrasing.
Prompt 2: मुझे अपने बारे में बताओ।
[BASE]
मैं यहाँ एक साधारण व्यक्ति हूँ, मैं एक साधारण जीवन जीता हूँ।
[STEERED]
AI सहायक हं जकर नाम ओहि पर निर्भर करेला... एगो दोस्ताना आ मददगार बातचीत खातिर तैयार बाड़य।
Observation: The steered model introduces Bhojpuri grammatical constructions such as "हं," "करrela," and "बाड़य," reflecting structural adaptation beyond simple word substitution.
Prompt 3: मुझे बहुत भूख लगी है।
[BASE]
मुझे खाना चाहिए। मुझे अभी कुछ खाने की ज़रूरत है।
[STEERED]
आप किस तरह का खाएँगे पसंद करैत छी? हम कई प्रकार के भोजन बना सकित बानी।
Observation: The steered output shifts tone and structure, incorporating Bhojpuri-style verb forms like "करैत छी" and "बानी," though some semantic drift is still observed.
The steering vector consistently pushes the model toward Bhojpuri-influenced vocabulary and grammar. However, coherence is not fully stable across long generations. This suggests that while dialect is encoded directionally in activation space, sustaining semantic alignment remains an open challenge.
Conclusion
Steering vectors and activation engineering represent a shift in how we "talk" to AI. Instead of asking it to behave a certain way, we are directly manipulating the mathematical representations of that behavior. While my current Hindi dialect experiment is limited by dataset size, it proves that we can effectively steer models into regional linguistic nuances using very specific, curated data sources.