Measuring LLM Susceptibility to Stylistic Perturbations in Social Media Text

Large Language Models (LLMs) are becoming increasingly integrated into important domains such as social media and politics. This makes it essential that we understand their susceptibility to superficial text variations. In this thesis, I investigated whether small stylistic perturbations in social media posts influence how an LLM perceives a post's political leanings. To do this, I systematically manipulated punctuation and capitalization across both political and apolitical social media posts. I then used VADER sentiment analysis to measure the changes in semantic intensity to quantify the text manipulation. Finally, I prompted each LLM to provide its perceived political leanings of each given post. I used linear regression to assess the VADER score correlation with the LLMs' political alignment ratings. While there was no statistically significant relation between perceived political extremity and VADER scores, category-level analysis revealed that the stylistic perturbations cause significant inconsistencies in model outputs. Furthermore, in posts with neutral semantic content, the stylistic perturbations introduce a right-leaning bias. These findings add to the growing body of literature on LLM instability under prompt perturbation and highlight risks associated with automated content analysis.