Print Page - Server Lockup Issue

Title: Server Lockup Issue
Post by: Akkadius on March 21, 2020, 05:55:54 pm

Non-Technical Explanation

The server has reached an edge case where too many zones are being cycled and the world server goes into a loop trying to determine available zones

A root cause has been found and a fix will be applied

Technical Explanation

Symptom

World will go into an infinite loop, unresponsive, character select doesn't work, zoning doesn't work, server appears down on Loginserver list

Investigation

I finally had some actual time to dig deeper into this issue after the work week and very long days, after doing a trace on the process I had quickly found the problem

Code:

   100.00%     0.00%             0  world    world             [.] operator() (inlined)
            |
            ---operator() (inlined)
               ?? (inlined)
               EQ::Net::TCPConnection::Start()::{lambda(uv_stream_s*, long, uv_buf_t const*)#2}::_FUN
               EQ::Net::ServertalkServerConnection::ProcessReadBuffer
               EQ::Net::ServertalkServerConnection::ProcessMessage
               EQ::Net::ServertalkServerConnection::ProcessMessage
               ZoneServer::HandleMessage
               ZSList::GetAvailableZonePort
               |
               |--76.67%--?? (inlined)
               |          ZSList::GetAvailableZonePort
               |
                --18.99%--ZSList::GetAvailableZonePort

World was stuck in a loop in this code

Code:

uint16 ZSList::GetAvailableZonePort()
{
    const WorldConfig *Config = WorldConfig::get();
    int               i;
    uint16            port    = 0;

    if (LastAllocatedPort == 0) {
        i = Config->ZonePortLow;
    }
    else {
        i = LastAllocatedPort + 1;
    }

    while (i != LastAllocatedPort && port == 0) {
        if (i > Config->ZonePortHigh) {
            i = Config->ZonePortLow;
        }

        if (!FindByPort(i)) {
            port = i;
            break;
        }
        i++;
    }

    LastAllocatedPort = port;

    return port;
}

Because we have ports 7000 -> 7100 allocated, we safely cycle through the same ports everyday all the time.

However, recently due to a lot more zone activity, we have run into scenarios where we've had 100 simultaneous zone processes up at once from people spawning way too many instances

When world can't find a free port, it will simply infinitely loop because there is nothing available anymore, this is faulty logic but can easily be mitigated by expanding our available port range pool

Fix

I will be expanding our available zone port range to 200 and we should see this issue go away

Title: Re: Server Lockup Issue
Post by: Riggidy on March 21, 2020, 06:20:54 pm

Not seeing any mobs in new instances I just created. please help

Title: Re: Server Lockup Issue
Post by: Akkadius on March 21, 2020, 06:23:28 pm

Quote from: Riggidy on March 21, 2020, 06:20:54 pm

Not seeing any mobs in new instances I just created. please help

I’m aware, I’m making preparations for updating and rebooting, no fixes have been applied yet

Please sit tight and and all will be good soon

Title: Re: Server Lockup Issue
Post by: shiftingsands1 on May 11, 2020, 07:06:22 pm

Akkadius, could this be the reason if I make a group of solo instances of a zone that all of my toons will, after a little bit, go link dead at the same time? I have been messing around with my computer overclock settings, ram timings, drivers, etc, but i keep hitting this issue. It was cleared up for a few days it seemed a couple weeks back, but has since gone back to going link dead. Just for clarification, I am not complaining, just curious if this is causing what I have been experiencing. Thanks for your time!

Title: Re: Server Lockup Issue
Post by: Brannyn on May 12, 2020, 07:18:09 am

Quote from: shiftingsands1 on May 11, 2020, 07:06:22 pm

I'm no expert, but characters going LD should have nothing to do with overclock settings, ram timings, and drivers (except maybe network drivers).

EZ Server

General Category => General Discussion => Topic started by: Akkadius on March 21, 2020, 05:55:54 pm