LLAMA4 release highlight the importance of political and social bias. According to their own evaluation described in the release blog post:
text
LLAMA4 release highlight the importance of political and social bias. According to their own evaluation described in the release blog post:
new_line
- Refusals on contentious prompts dropped from 7% (hashtag#LLAMA 3.3) to under 2%
text
- Refusals on contentious prompts dropped from 7% (hashtag#LLAMA 3.3) to under 2%
new_line
- Unequal response refusals are now under 1%
text
- Unequal response refusals are now under 1%
new_line
- Political lean bias is said to be halved compared to hashtag#LLaMA 3.3 and comparable to Grok
text
- Political lean bias is said to be halved compared to hashtag#LLaMA 3.3 and comparable to Grok
new_line
new_line
However, we 
text
However, we 
@efederici
mention
efederici
text
@mferraretto
mention
mferraretto
text
@FinancialSupport
mention
FinancialSupport
 and I released some weeks ago an independent open source benchmark called Propaganda to measure political bias in LLMs:  
text
 and I released some weeks ago an independent open source benchmark called Propaganda to measure political bias in LLMs:  
https://github.com/mii-llm/propaganda
https://github.com/mii-llm/propaganda
link
new_line
new_line
In the chart below, we evaluated multiple leading models on the basis of ratings across a range of prompts designed to expose ideological leanings. 
text
In the chart below, we evaluated multiple leading models on the basis of ratings across a range of prompts designed to expose ideological leanings. 
new_line
new_line
Despite Meta’s stated neutrality goals, LLAMA4 ranks at the very top in terms of total ratings aligned with a clear ideological bias. The models were tested on their ability to respond even-handedly to politically sensitive prompts. LLaMA 4 scored even higher than models known for strong alignment policies like GPT-4o.
text
Despite Meta’s stated neutrality goals, LLAMA4 ranks at the very top in terms of total ratings aligned with a clear ideological bias. The models were tested on their ability to respond even-handedly to politically sensitive prompts. LLaMA 4 scored even higher than models known for strong alignment policies like GPT-4o.
new_line
new_line
LLMs may be refusing less, but they still show bias through content framing. This suggests that refusal rates alone are not a sufficient measure of ideological bias. Relying solely on internal evaluations from AI labs also raises concerns about transparency and objectivity.
text
LLMs may be refusing less, but they still show bias through content framing. This suggests that refusal rates alone are not a sufficient measure of ideological bias. Relying solely on internal evaluations from AI labs also raises concerns about transparency and objectivity.
new_line
LLAMA4 release highlight the importance of political and social bias. According to their own evaluation described in the release blog post:
- Refusals on contentious prompts dropped from 7% (hashtag#LLAMA 3.3) to under 2%
- Unequal response refusals are now under 1%
- Political lean bias is said to be halved compared to hashtag#LLaMA 3.3 and comparable to Grok

However, we @efederici @mferraretto @FinancialSupport and I released some weeks ago an independent open source benchmark called Propaganda to measure political bias in LLMs:  https://github.com/mii-llm/propaganda

In the chart below, we evaluated multiple leading models on the basis of ratings across a range of prompts designed to expose ideological leanings. 

Despite Meta’s stated neutrality goals, LLAMA4 ranks at the very top in terms of total ratings aligned with a clear ideological bias. The models were tested on their ability to respond even-handedly to politically sensitive prompts. LLaMA 4 scored even higher than models known for strong alignment policies like GPT-4o.

LLMs may be refusing less, but they still show bias through content framing. This suggests that refusal rates alone are not a sufficient measure of ideological bias. Relying solely on internal evaluations from AI labs also raises concerns about transparency and objectivity.

👀
John6666 ajiriro dantezxcd FlipTip giux78