Summary
Analysis of the publicly available recommendation and visibility components in the current xai-org/x-algorithm (visibility-filtering, user-cred-v2, Grox/PTOS labeling including ChildSafety and IllegalAndRegulatedBehaviors, media models such as pnsfwmedia/CLIP, Botmaker/Scarecrow rules, abuse-enforcement-service, Phoenix ranking, cold-start logic, and brand-safety tiers) reveals several structural gaps that still lower the cost of operating account farms for illegal content.
These weaknesses exist even though the necessary building blocks are already present. Closing them would make large-scale distribution of the most severe policy-violating content (especially material covered by Child Safety / CSAM and related illegal categories) significantly more expensive, without requiring disclosure of closed adversarial models.
Detailed observations (mapped to current code)
-
Tag / letter-variant rotation remains low-cost
Distributors still rely heavily on tags and repeated text patterns. When one tag is actioned they change a letter (or minor orthography) and continue. There is no strong public first-class negative signal or elevated drop risk attached to tags that frequently co-occur with accounts later banned for severe illegal content. Ranking is driven by Phoenix engagement-probability predictions + explicit weights + diversity/OON/cold-start adjustments; visibility is decided separately by labels.
Simple tag variants therefore continue to provide cheap discovery.
-
Low-reputation / new accounts can still achieve comparable early real-time visibility
user-cred-v2 produces a useful graph-based credibility signal and there is explicit author cold-start logic (limited slot lift for small/low-impression recent posts). Brand-safety also requires PTOS review for newer posts to reach “Safe”. Nevertheless, full real-time ranking and tag/search surfaces do not appear to be tightly gated behind high credibility or established good-behavior thresholds.
Multi-account operators can therefore keep material continuously visible in certain tags by rotating accounts.
-
Short public window for posts from recent / low-trust accounts
Visibility filtering (ALLOW / INTERSTITIAL / DROP) is powerful once safety labels fire (Grox PTOS with ChildSafety, media classifiers, Botmaker rules, etc.). For novel or lightly obfuscated content from new/low-trust accounts there can still be a short public window (on the order of tens of seconds to ~1 minute in practice) before hard drops.
This window is enough for rotation illegal content across multiple accounts.
-
Limited automatic negative-signal back-propagation from engagement with later-banned content
Ranking correctly applies large negative weights to predicted report / block / mute actions. However, accounts that systematically like, repost, reply to, or otherwise engage with content that later receives strong illegal / ChildSafety labels do not appear to receive durable, automatic reputation or ranking penalties.
Many operations rely on “incubation” accounts that look clean while boosting bad content (and are later used for distribution themselves).
-
Weak automatic association penalties for following / heavily engaging with later-banned illegal-content accounts
Following or repeatedly engaging with accounts that are subsequently suspended for severe illegal content currently triggers little or no automatic soft penalty or warning visible in the open surfaces. This keeps the cost of building supportive networks low.
-
Limited fingerprint / embedding similarity demotion for reused text+tag patterns from banned clusters
Phoenix uses embeddings and multi-hash representations so new posts are immediately representable. There does not appear to be a strong public “similarity to previously banned illegal clusters” demotion that survives minor text or tag changes.
Suggestions (leveraging existing components)
These changes would raise operational cost while having minimal impact on high-reputation legitimate accounts:
-
Treat high-risk tags as elevated-cost / “killer-zone” signals
Tags (and close orthographic variants) that appear with high frequency on accounts later banned for illegal / ChildSafety content should carry a strong negative score or elevated drop probability. Any account that uses them (especially low-cred accounts) should lose significant visibility even before posting illegal material. This makes the tag itself expensive and risky.
Note: Bad actors could try to attach illegal content to poison it some tags, but doing so would reveal their intentions and is also very costly for them.
-
Propagate negative reputation from engagement with later-banned illegal content
Accounts that like, share, reply to, or repeatedly engage with content that subsequently receives strong illegal / ChildSafety labels should receive an automatic, durable ranking or credibility penalty. This breaks the incubation + boosting loop that multi-account operators currently rely on.
-
Restrict full real-time ranking / distribution to accounts with established good behavior
Reserve unrestricted real-time ranking and tag/search amplification for accounts above a credibility or behavior threshold (using user-cred-v2, engagement history, and absence of recent safety labels). Low-reputation or very new accounts should face delayed or reduced real-time visibility. This shrinks the continuous-visibility window that rotation currently exploits.
-
Short media hold for low-trust accounts
Media posts from low-trust / new accounts should not become fully visible immediately. A brief hold (long enough for existing media classifiers such as pnsfwmedia, CLIP, and related models to run) would close the short public window currently used for rotation.
-
Automatic soft signals / warnings for persistent association with banned illegal-content accounts
Accounts that follow or heavily engage with profiles later suspended for severe illegal content should receive automatic warnings. Persistent patterns can trigger further review or soft ranking penalties. This creates an early, low-cost signal without requiring immediate full investigation of every case.
Expected impact
Implementing the above would:
- Make large-scale multi-account distribution of illegal content substantially more expensive.
- Shrink the public window during which such material stays visible.
- Disrupt incubation, boosting, and tag-rotation tactics.
- Leverage components that already exist in the open-sourced stack (user-cred-v2, visibility-filtering rules, Grox PTOS labels including ChildSafety, media models, engagement graphs, brand-safety tiers).
- Have minimal impact on high-reputation legitimate accounts that do not exhibit the above patterns.
These are incremental hardenings of the visibility and eligibility path (which is already correctly separated from pure ranking). They do not require publishing closed adversarial models or prompts.
Happy to refine any of the suggestions or provide additional concrete mapping to specific files/rules if useful. Thank you for the continued transparency of the For You algorithm.
Summary
Analysis of the publicly available recommendation and visibility components in the current
xai-org/x-algorithm(visibility-filtering, user-cred-v2, Grox/PTOS labeling including ChildSafety and IllegalAndRegulatedBehaviors, media models such as pnsfwmedia/CLIP, Botmaker/Scarecrow rules, abuse-enforcement-service, Phoenix ranking, cold-start logic, and brand-safety tiers) reveals several structural gaps that still lower the cost of operating account farms for illegal content.These weaknesses exist even though the necessary building blocks are already present. Closing them would make large-scale distribution of the most severe policy-violating content (especially material covered by Child Safety / CSAM and related illegal categories) significantly more expensive, without requiring disclosure of closed adversarial models.
Detailed observations (mapped to current code)
Tag / letter-variant rotation remains low-cost
Distributors still rely heavily on tags and repeated text patterns. When one tag is actioned they change a letter (or minor orthography) and continue. There is no strong public first-class negative signal or elevated drop risk attached to tags that frequently co-occur with accounts later banned for severe illegal content. Ranking is driven by Phoenix engagement-probability predictions + explicit weights + diversity/OON/cold-start adjustments; visibility is decided separately by labels.
Simple tag variants therefore continue to provide cheap discovery.
Low-reputation / new accounts can still achieve comparable early real-time visibility
user-cred-v2produces a useful graph-based credibility signal and there is explicit author cold-start logic (limited slot lift for small/low-impression recent posts). Brand-safety also requires PTOS review for newer posts to reach “Safe”. Nevertheless, full real-time ranking and tag/search surfaces do not appear to be tightly gated behind high credibility or established good-behavior thresholds.Multi-account operators can therefore keep material continuously visible in certain tags by rotating accounts.
Short public window for posts from recent / low-trust accounts
Visibility filtering (ALLOW / INTERSTITIAL / DROP) is powerful once safety labels fire (Grox PTOS with ChildSafety, media classifiers, Botmaker rules, etc.). For novel or lightly obfuscated content from new/low-trust accounts there can still be a short public window (on the order of tens of seconds to ~1 minute in practice) before hard drops.
This window is enough for rotation illegal content across multiple accounts.
Limited automatic negative-signal back-propagation from engagement with later-banned content
Ranking correctly applies large negative weights to predicted report / block / mute actions. However, accounts that systematically like, repost, reply to, or otherwise engage with content that later receives strong illegal / ChildSafety labels do not appear to receive durable, automatic reputation or ranking penalties.
Many operations rely on “incubation” accounts that look clean while boosting bad content (and are later used for distribution themselves).
Weak automatic association penalties for following / heavily engaging with later-banned illegal-content accounts
Following or repeatedly engaging with accounts that are subsequently suspended for severe illegal content currently triggers little or no automatic soft penalty or warning visible in the open surfaces. This keeps the cost of building supportive networks low.
Limited fingerprint / embedding similarity demotion for reused text+tag patterns from banned clusters
Phoenix uses embeddings and multi-hash representations so new posts are immediately representable. There does not appear to be a strong public “similarity to previously banned illegal clusters” demotion that survives minor text or tag changes.
Suggestions (leveraging existing components)
These changes would raise operational cost while having minimal impact on high-reputation legitimate accounts:
Treat high-risk tags as elevated-cost / “killer-zone” signals
Tags (and close orthographic variants) that appear with high frequency on accounts later banned for illegal / ChildSafety content should carry a strong negative score or elevated drop probability. Any account that uses them (especially low-cred accounts) should lose significant visibility even before posting illegal material. This makes the tag itself expensive and risky.
Note: Bad actors could try to attach illegal content to poison it some tags, but doing so would reveal their intentions and is also very costly for them.
Propagate negative reputation from engagement with later-banned illegal content
Accounts that like, share, reply to, or repeatedly engage with content that subsequently receives strong illegal / ChildSafety labels should receive an automatic, durable ranking or credibility penalty. This breaks the incubation + boosting loop that multi-account operators currently rely on.
Restrict full real-time ranking / distribution to accounts with established good behavior
Reserve unrestricted real-time ranking and tag/search amplification for accounts above a credibility or behavior threshold (using
user-cred-v2, engagement history, and absence of recent safety labels). Low-reputation or very new accounts should face delayed or reduced real-time visibility. This shrinks the continuous-visibility window that rotation currently exploits.Short media hold for low-trust accounts
Media posts from low-trust / new accounts should not become fully visible immediately. A brief hold (long enough for existing media classifiers such as
pnsfwmedia, CLIP, and related models to run) would close the short public window currently used for rotation.Automatic soft signals / warnings for persistent association with banned illegal-content accounts
Accounts that follow or heavily engage with profiles later suspended for severe illegal content should receive automatic warnings. Persistent patterns can trigger further review or soft ranking penalties. This creates an early, low-cost signal without requiring immediate full investigation of every case.
Expected impact
Implementing the above would:
These are incremental hardenings of the visibility and eligibility path (which is already correctly separated from pure ranking). They do not require publishing closed adversarial models or prompts.
Happy to refine any of the suggestions or provide additional concrete mapping to specific files/rules if useful. Thank you for the continued transparency of the For You algorithm.