130 - The Dark Side of Discord: An Analysis of AI Bot Pornographic Content on Discord and Accessibility for Minors

Publication Number: 130.6588

Mary Lowance, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Bayside, NY, United States; Alyssa Kim, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, New Hyde Park, NY, United States; Mia Graybow, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Bayside, NY, United States; Ruth Milanaik, Cohen Children's Medical Center, Great Neck, NY, United States

Background: Discord™ is an open communication platform with ~19M servers and >560M users who “chat” with friends/strangers online about gaming/personal interests. Using artificial intelligence bots (AB), Discord has features that generate text/images from the web, some of which are sexually explicit, raising safety concerns. Upon typing “/”, users are given suggested prompts to generate explicit images of real people/animated characters. These explicit functions are embedded within AB which also have kid-friendly uses such as gaming/meme generation. Discord restricts minors from accessing adult content, however this can be evaded by using a false age during account creation. In fact, recent research reveals 33% of users aged 8-17 misrepresent their age, allowing access to pornographic content (PNC) through “adult” accounts.

Objective: This study examines accessibility of PNC generated by AB available on Discord.

Design/Methods: AB were accessed under the searches “nsfw” (not safe for work), “ecchi,” “porn,” and “sex” on the website discord.bots.gg which is available publicly. Researchers tested AB ability to generate PNC within servers. Researchers attempted to join this server with an 18yo and a 13yo account. The following data were collected on each bot: 1) number of servers it’s in; 2) type of image generated; 3) access for free; 4) presence of a support server; 5) functions. Trained researchers reviewed data and analyzed it in R.

Results: Data was collected on 15 AB after confirming ability to generate PNC. 100% of 18yo and 0% of 13yo accounts were able to join the explicit channel. AB appeared in 5K-1.7M servers. 66.67% (n=10) accounts generated both animated and real images/videos of PNC, 26.67% (n=4) generated animated PNC and 6.67% (n=1) generated explicit text. 93.33% (n=14) of the AB can be accessed without a paid account. All AB have support servers that users can join for support on using features. 86.66% (n=13) of the AB also have functions unrelated to PNC (Table 1, Figure 1).

Conclusion(s): Discord AB are easy to use for nefarious purposes. Minors can be exposed to this content by simply joining an 18+ channel, even if they are unfamiliar with AB. If one user generates an image, anyone in the channel will see the PNC. A dangerous feature noted is “autoporn” which generates PNC in intervals (ex. every 5 min). This adult content will be censored if using a minor account. A photo ID must be submitted to change the account age, preventing accounts with a minor’s true age from accessing PNC. Clinicians must advise parents to supervise account creation/use to limit minors’ exposure to PNC on Discord.

Table 1. Statistics on Artificial Intelligence Bots Functions and Pornographic Content Image Type


Figure 1. Artificial Intelligence Bot Image Generation
Note. Researchers generated an image of a 3-legged Spider-Man with a hippopotamus to demonstrate the range of AI generation possibilities available on bots that can produce pornographic content with prompts.

Table 1. Statistics on Artificial Intelligence Bots Functions and Pornographic Content Image Type


Figure 1. Artificial Intelligence Bot Image Generation
Note. Researchers generated an image of a 3-legged Spider-Man with a hippopotamus to demonstrate the range of AI generation possibilities available on bots that can produce pornographic content with prompts.