EZ Server

General Category => Server Status => Topic started by: Fugitive on August 20, 2012, 03:56:55 pm



Title: LAG SPIKES FROM HELL
Post by: Fugitive on August 20, 2012, 03:56:55 pm
8/20/12 1600 CST

Every mob we pull stutters... then multiple party members are down =(

What are ya querying? !!! Or changing !!

almost unplayable...  have to rez 1 or 2 every pull which is way abnormal..  

Also the deaths have cause the zone to crash x2....which is way abnormal.. 


Title: Re: LAG SPIKES FROM HELL
Post by: Mattnaik on August 21, 2012, 10:28:57 am
I noticed this too when trying to loot anything it would take up to 15 seconds for the loot window to come up yesterday.

Any chance this is caused by the new race changer? I know literally nothing about the code so this is just a shot in the dark but just throwing it out there.


Title: Re: LAG SPIKES FROM HELL
Post by: Hunter on August 21, 2012, 10:49:26 am
Actually yeah, its probably the race changer. There is a huge 'blob' field in the character table that is probably causing lag from being accessed.


Title: Re: LAG SPIKES FROM HELL
Post by: Camric on August 21, 2012, 11:23:44 am
For the race changer it might be best to setup a table to store all the daily requests and then have it execute after reboot and before server is made available. Or worst case, it occurs as soon as the server comes up.





Title: Re: LAG SPIKES FROM HELL
Post by: Mattnaik on August 21, 2012, 12:16:50 pm
Actually yeah, its probably the race changer. There is a huge 'blob' file in the character table that is probably causing lag from being accessed.

I'm a software developer by trade and when I read this I started twitching uncontrollably.   :o


Title: Re: LAG SPIKES FROM HELL
Post by: hateborne on August 21, 2012, 12:37:26 pm
Actually yeah, its probably the race changer. There is a huge 'blob' file in the character table that is probably causing lag from being accessed.

I'm a software developer by trade and when I read this I started twitching uncontrollably.   :o

Could you join the EQEMU team and rewrite the SQL queries through quest scripts to be considerably more efficient and/or allow "out of band" queries that may pull in extra CPU cycles (instead of riding on the same ones and causing massive server lag when a special query is run)? :-P

-Hate


Title: Re: LAG SPIKES FROM HELL
Post by: Mattnaik on August 21, 2012, 01:15:05 pm
This reminds of one job I worked at where these people stored an XML configuration into a single table column. They would run "LIKE" queries against it. Thankfully they let me redesign that mess.

I never understood the concept of storing multiple pieces of data into a single column...just add another column. It makes your indexes actually do something other than take up hard drive space.


Title: Re: LAG SPIKES FROM HELL
Post by: Kenshou on August 21, 2012, 02:08:01 pm
I love the race changer, but if its going to cause problems like this to continue, id still like to suggest the permanent  illusion AA the enchanter gets be used instead. I spent some time in t5 yesterday and it was horrible swimming with the lag spikes


Title: Re: LAG SPIKES FROM HELL
Post by: hateborne on August 21, 2012, 02:11:16 pm
This reminds of one job I worked at where these people stored an XML configuration into a single table column. They would run "LIKE" queries against it. Thankfully they let me redesign that mess.

I never understood the concept of storing multiple pieces of data into a single column...just add another column. It makes your indexes actually do something other than take up hard drive space.

So is that a yes or a no?

If you want to redesign the entire character system, redesign how gear is handled, and massively increase the character database size...go for it. As long as it is more efficient, it works, and you don't mind completely rebuilding the bulk of how the EQEMU server source handles gear.

http://code.google.com/p/projecteqemu/

Have at it.

-Hate


Title: Re: LAG SPIKES FROM HELL
Post by: Hunter on August 21, 2012, 02:23:37 pm
I asked dev's before and umm yeah its a big task.


Title: Re: LAG SPIKES FROM HELL
Post by: hateborne on August 21, 2012, 02:46:34 pm
I asked dev's before and umm yeah its a big task.

Is it actually a server issue or simply a connection issue? No point in reinventing the wheel (through code) for something that is simply a logistics issue.

-Hate


Title: Re: LAG SPIKES FROM HELL
Post by: Kenshou on August 21, 2012, 02:55:24 pm
GIMMIE THE AA, problem solved


Title: Re: LAG SPIKES FROM HELL
Post by: Mattnaik on August 21, 2012, 03:12:46 pm
This reminds of one job I worked at where these people stored an XML configuration into a single table column. They would run "LIKE" queries against it. Thankfully they let me redesign that mess.

I never understood the concept of storing multiple pieces of data into a single column...just add another column. It makes your indexes actually do something other than take up hard drive space.


So is that a yes or a no?

If you want to redesign the entire character system, redesign how gear is handled, and massively increase the character database size...go for it. As long as it is more efficient, it works, and you don't mind completely rebuilding the bulk of how the EQEMU server source handles gear.

http://code.google.com/p/projecteqemu/

Have at it.

-Hate

Normalizing the data properly (3NF) should reduce the size on disk, not increase it. What type of data access layer is in place? Does it use stored procedures for data access or is the SQL hardcoded in compiled code?

If I had more free time I would definitely consider taking a look at it. I work 60 hours a week so most of my free time is split between spending time with the wife and doing shit around the house.


Title: Re: LAG SPIKES FROM HELL
Post by: hateborne on August 21, 2012, 03:42:37 pm
All the relevant info is in the source on the link I sent. :-)

-Hate


Title: Re: LAG SPIKES FROM HELL
Post by: lerxst2112 on August 21, 2012, 04:21:03 pm
I asked dev's before and umm yeah its a big task.

Yeah, changing the character info to be stored as actual data fields instead of the blob is a huge task.  I think Trevius was chipping away at it a bit for a while, but until the whole thing is done the problem would still exist since it needs to save/load the whole thing until everything is out of it since the blob needs to stay the same size even if some of the data isn't used.


Title: Re: LAG SPIKES FROM HELL
Post by: Camric on August 22, 2012, 09:58:31 am
Not only would it reduce the drag on the database but it would also increase the efficiency of the client.  Instead of front loading the data on start-up, query on demand when required.

It would be good to speak with Tervius to understand why the decision was made to use a blob versus loading the data directly to tables.





Title: Re: LAG SPIKES FROM HELL
Post by: lerxst2112 on August 22, 2012, 11:37:12 am

The decision to store as a blob was made a long time ago, and not by Trevius as far as I know.  It is a convenient way to serialize a large struct all at once occasionally, but not a good solution when the data changes frequently as we appear to be seeing.


Title: Re: LAG SPIKES FROM HELL
Post by: Mattnaik on August 22, 2012, 12:01:00 pm

The decision to store as a blob was made a long time ago, and not by Trevius as far as I know.  It is a convenient way to serialize a large struct all at once occasionally, but not a good solution when the data changes frequently as we appear to be seeing.


...or to query against it, or to gain performance benefits of indexing, or to maintain referential integrity, or to normalize the data to prevent repetition, or basically anything you use a database for.

I don't mean to sound so critical I would just really like to know what benefit is gained by doing it this way. Cause I can't think of a single one. I'm not saying there isn't one, and I have been known to overlook things before so I'm genuinely curious.


Title: Re: LAG SPIKES FROM HELL
Post by: lerxst2112 on August 22, 2012, 12:36:21 pm

Feel free to ask on the emu boards.  From my perspective it was just a convenient way to store a C++ struct in the easiest manner while reverse engineering what each byte in the struct was for.