I wanted to wait a while to see the result of all of the tweaks before I made any additional comment regarding server stability and performance.
Bottom line (tldr) is that we're sitting absolutely fantastic in terms of server performance.Micro hangs that were being mostly caused by disk I/O issues in a configured array were the cause for various zone issues and desyncs. After various configurations and benchmarks and tweaks we have an optimum configuration for the server resources that need to be
highly available at the lowest latency possible.I've also optimized the virtual processor configuration which should also help in the latency of actions performed by processes. In a setup such as this, every optimization has a force multiplier in which makes a huge difference overall.
I've also chopped off about 150 excess zone processes that Hunter had configured several years ago that are unnecessary. I believe he put them up because he could and he had RAM, but in reality even when most of the processes are 'sleeping' they are still chewing up processor cycles and have the potential to interrupt other threads in the need to process data in a moment where a burst of activity is occurring.
The extreme increase in disk IOPs/low latency has also freed up CPU hangs in waiting for data which has overall left the server humming at
20-30% CPU even with 400-500 players as opposed to what we were seeing which was 50-60% with 100% bursts periodically.
There has also been a change in how the server automatically brings up zone processes as they crash or reboot, I have adjusted the timers so after reboots we can prevent from seeing
'rolling crashes' which have appeared to
have gone away completely.There you have it! More information than most of you wanted to know
Thanks,
-Akka
