Performance Issue(s)

[punkdis]

6d04149

AWS EC2, Ubuntu 14.04.5 LTS on a m4.xlarge

When we have over 150 participants over 20 mins the CPU slowly goes to 80%. Once at 80% performance goes to the crapper and the UI is slow and participants are seeing the impact in game play.

[juliannagler]

Thanks for filling the issue @rfrillman - we'll investigate asap.

[justinwray]

Do you happen to know if you were running Memcached? Was the database server on the same system as the webserver (a single system)?

[punkdis]

Yes Memcached was running.

memcache 1359 0.0 0.2 327440 2580 ? Sl 04:11 0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
ubuntu 1886 0.0 0.0 11744 932 pts/0 S+ 04:19 0:00 grep --color=auto memcached

Yes the database server was on the same system as the webserver.

[justinwray]

Interesting, nearly all of the data is cached now (even sessions). So you shouldn't see a high load in relation to the database. In the past we've seen this issue when Memcached wasn't running/working.

Did you have the binlog enabled by chance? Or the general query log? (Note: You probably shouldn't for performance reasons - but it would be useful for some investigative information in this case).

[punkdis]

binlog is/was disabled. I can see if there are any query logs

[punkdis]

After reviewing the logs there is nothing that stands out. Maybe the resources for m4.xlarge is not enough.

I have another event in a month or so and I am going to move to the C4.4xlarge

[shargon]

We would like to expose our experience using the Facebook CTF platform.
First of all, the platform was deployed on a Virtual Private Server (OVH SSD3 VPS Server), whose specifications are:

• 8 GB RAM.
• 2v cores.
• 2.4 GHz.
• SSD 40 GB.
• KVM OpenStack.
• Local RAID 10.

After deploying the platform for our CTF, was tested with 15 players and we did not experience any problem of flooding or lagging in the platform.

Once the competition was opened, 340 participants were registered and the platform remained open from 02/16/2017 to 02/19/2017.
At this point the platform began to flood, being almost impossible to interact with the interface, even when there were only 30 players ingame.

The memory system (RAM) was not a problem, however, the CPU load increased considerably.
To correct these problems, we made the following modifications:

• Modifying Mysql stack threads from 10 to 25.
• Restart HHVM service every 5 minutes.
• Deletion of active sessions of the BBDD every hour, to remove the inactive users (cleaning memcache included).
• Refresh callbacks changed from 5000 to 30000 and 10000 to 120000

The main problem is given by HHVM, and although the modifications managed to fix the lagging problems at times, we did not manage to correct the problems completely and all users started complaining about the flooding of the system until we delete the user sessions and restart HHVM service again.

[punkdis]

Any other thoughts?

[wisco24]

I had this same issue awhile back, and still do at times for large events. HHVM is tying up the CPU waiting for the database queries. I just ran an event with 30 teams last week 50-60 sessions, and used a c4.4xl (16 cores) but moved the DB to AWS Aurora db.r3.2xl. (working on the code so that I can do a pull request for it) Late in in the afternoon when the game log grew due to all the guesses/answers, the CPU was sitting around 60%. The biggest help from my testing was having the DB separate. so the DB and HHVM were not both beating up on the same CPU.

[wisco24]

From the queries I looked at, the issue is with how the leader board scores are done. They are recalculated every time a team scores by querying/adding the full game logs table. A much more efficient way would be to store the score in a separate DB table so it doesn't have to go through the all the guesses each time.

[wisco24]

Just looked at pull request #459 and that might be a mitigation method as well as it will create separate DB's, etc. and then possibly put a Load Balancer in front of it with sticky sessions so a team stays stuck to one system.

[punkdis]

Has anyone used the Live Sync API?