EZ Server

Source code was updated for loot lag.

Disabled logging who discovered items for the first time, see if that helps with loot lag.

Got the tip from Secrets.

Character:EnableDiscoveredItems   false   

If enabled, it enables EVENT_DISCOVER_ITEM and also saves character names and timestamps

Actually might need to update source for it to work. Will update later.

Yeah there's a source fix that goes along with it that solves a majority of the lag. The actual adding of items was not an issue, just the database query every time an item is looted is ><

Thank you Secrets for your continued expertise! =)

I'm honestly not sure if that was the cause but it didn't happen till this recent update... so yeah, assuming it's the cause.

If only someone mentioned that 4 days ago...

http://ezserver.online/forums/index.php?topic=2639.msg34387#msg34387

;)

Yeah, I looked into it because of that post. It's been on the back of my head to look at the code, never got around to it...

Also, update SVN again, I committed two more fixes lerxst2112 pointed out.

They should be gone in r2102.

I'm recompiling again.

Good news, hopefully it will fix the issue. Does this go active next server reboot?

You guys rock!!!

Still getting loot lag with SoD client

Yea Ive noticed no change using UF, but others have said looting on Titanium is now very fast.

This may seem weird but using the same computer with no macroquest running.

6 toons all older are looting with a 3 to 5 sec lag before it opens up the corpse.

6 new toons, are looting instantly.

Any idea what I can do to get the first 6 faster?

Awesome info Shawo! If Hunter and Secrets can find the update just before the loot lag started it should help to see the difference between new and old toons looting code. Is there a way to look at a single toons code for looting?

very interesting find!

Deleted my INI and UI files for every toon and it's still slower on the older toons.

I wonder if Secrets or lerxst2112 has any ideas as to whats making the older toons loot slower than newly created toons

All toons use the same looting code.  Presumably toons that are older may have more junk in the bank, so things like lore checks might take longer, but I'd imagine it's still a very small fraction of a second. and the same check is performed on all clients.  Besides, if it's slow all the time, like when just looting gems, that wouldn't apply since looting a non-lore item skips 99% of the check.

The packets sent to Titanium versus Underfoot are different, and the data sent to Underfoot looks more complex at a glance, but in practice I'd imagine it's not a drastic difference.  If you're using auto loot it's possible the client throttles the requests being sent to the server where on Titanium it's might be as fast as you can click.

I haven't played enough recently to really comment on how slow it is, and I haven't used Titanium in more than a year, so I wouldn't know the difference.

I am not yet really convinced that discovered items were the cause of loot lag.  My server didn't start reporting the loot lag until I switched from Linux to Windows on November 11th and discovered items were added on Nov 1st on my server when it was still running Linux.  For EZ, it looks like the last big update prior to the latest ones was on Nov 5th and I don't see the loot lag reports from posts I could find around that time, even though Discovered Items would have been in at that time already.  The loot lag reports here seem to have started after the last update at the end of January.  Though if the SQL updates for the discovered_items table weren't applied and the rule was still enabled, I could definitely see that causing a problem (just as any missing required SQL for updates would cause a problem).

I still have more testing and research to do, but the more information we have, the better it is for tracking down these types of issues.  All players can help in the testing if they wish.

Here is a list of loot lag questions that I think might help isolate the issue.  I haven't been able to experience the loot lag myself, so it is hard for me to troubleshoot:

1. Which part of looting actually shows lag?  Is it when right clicking the body, it takes a couple of seconds before the loot window shows up?  Is it when clicking an item, it takes a couple of seconds for it to be looted and move to inventory?

2. Does loot lag always happen, or is it sporadic?

3. Does it get worse if more players are online?

4. Does the amount of lag time vary?

5. If looting a corpse with multiple items, do all items lag the same amount, or just the first one lags and the others are fine?

6. Has anyone confirmed for sure that it only happens on certain clients, such as Titanium, SoF, SoD, or UF?  Or, does it happen on all clients?

7. Do all characters experience the loot lag, or only certain ones?  Based on a post in this thread, it sounds like older characters experience it the most.

8. Does it happen on all corpses?

9. For anyone who normally runs MQ2; Have you tried shutting off MQ2 completely and playing to see if you still experience the problem without it?

10. Has anyone tested to see if the UI being used might be related to the issue?  Do you experience the issue if you use a completely default UI vs running a custom UI?

11. Does anyone know exactly when the loot lag issue started to happen?  Searching posts, I am getting mixed results and it makes it hard to track down an exact date, so the we know which updates in the source to look closer at.

12. Has the loot lag issue changed at all since EZ updated the source again and disabled Discovered Items?  Is it better, completely gone, or about the same?

If all of these questions can be answered with good details and info, I think it will help a lot to isolate the loot lag issues.

My girlfriend and I created our toons the exact same time and have been playing nearly the same amount. We play off the same router. Her box is running Vista and mine is on 7. My graphics card and processor are slightly better. I can loot at times 3 corpses to her one. It slows down for both of us the more people that are online, and lately have been lagged out just pulling mobs in Ldon. Neither of us have tried shutting off MQ2, and our UIs are the same standard ones. Lag happens from both clicking on the mob, as well as clicking on the item. And I dont know if it is related, but looting mobs in HoH will sometimes leave only 1 gem on a corpse when you hit loot all.

This bug was HUGE a long time ago when the server would crash a lot. I could tell when the zone was going to the crash if a corpse did that.. then in 5~ minutes the zone would crash. But now it still does that leave 1 gem on corpse randomly.. however the zone does not crash now.. its just an anomaly. I did 11 clears of HoH in one day and hat it only happen twice... and looting was just fine every clear no lag at all.. just those 2 weird occurrences.

Wonder if this would cause any lag? It might have been put in to prevent people from accidently deleting their items by accident.

People who are still having lag must have something wrong on their end... I've tried Titanium // SOD // UF... with basic MQ2 installed and never see any loot lag.. but did notice Titanium to be the fastest looting.. but by a tiny bit.

I ran two days of ROA and today I just didn't bother loading MQ2.  There was a huge difference in loot speed in CoM.  Without MQ2 there was zero loot lag.  With MQ2 on yesterday the lag was bad enough that I didn't bother looting gems.  I don't know if it was mainly due to server pop or unplugging MQ2, so more to test but I like the results so far.

I get very little loot lag with the social i made for loot all, it's a social via everquest and yes I have mq2 loaded.  anyhow- tbh i think it's the input interface or whatever (mouse clicks/movements)  my social reads as follows:

/target corpse
/shiftkey /click right target


this works perfect for me. that is if the corpse is within loot range (obviously)

hope that helps

Well if you have MQ loaded lex, add in a line that does /stick after the /target corpse and it will target the nearest corpse and walk to it, might have to press it twice if it is too far as it will execute the /loot immediately after you stick not leaving time to run to the corpse

Can also put a pause 20 after you put the loot command and and then add another line with the loot command again.

you are making it to difficult all you have to do is

/target corpse
/loot

thats what i use, though you are looting all so i guess yours would work better, the one i do doesnt loot all cause i dont like looting the trash also

Here you go...

/target corpse
/pause 1
/lootall

yes... lootall.. it works with SoD+

the words i use might appear a difficult route but as far lag goes it works wonders for me.

I am using the same computer and same Install and even fresh UI's and toons are still looting slower than the new ones.  I'll try to delete the stuff in the bank to see if it helps at all.

Haven't read any posts above this one, but coming in to report that on HoT (UF client), I have no loot lag except on the named mobs in LDoN. Any other mob I loot pops up instantly with whatever it has on the corpse and I have no problems looting.

Don't have problems looting in any other zone, either.

Thanks for the information so far.  I recently moved back to Linux on another host for my server and all reports say that loot lag is now gone.  I don't know yet if it was due to the better hard drive speeds on the host we are on now, or the difference from Windows to Linux.

Anyway, I did find something just now that could definitely cause a noticeable amount of extra unnecessary work on the server during looting.  Basically, in the Corpse::MakeLootRequestPackets(Client* client, const EQApplicationPacket* app) function, it does a client->Save() after it gives out any cash from the corpse.  The problem is that client->Save() is a very heavy database query.  Within that same function, it also uses the AddMoneyToPP() function to add money to the player profile, which also saves the character with save().  This means it is doing a very heavy query 2 times in a row when it only needs to be doing it once.  I think we can just remove the extra client->Save() from the MakeLootRequestPackets() function to make a noticeable improvement on loot performance.

Related to that find, I have another question:

Does anyone notice a difference in looting response times depending on if they have /autosplit turned on or off?

As mentioned above, the Save() function causes some pretty heavy work on the server.  For anyone with autosplit enabled, it does a Save() for each member of the group when they get money from the split, so for a full group using autosplit, that would cause 7 Saves() (including the 1 extra one it does that isn't needed).  I would guess that if you disable autosplit, on your character that is looting, the looting performance would increase considerably.   Furthermore, I would guess that Shawo has /autosplit enabled on the older characters that are taking longer to loot than the newer character (which I also guess do NOT have /autosplit enabled).  Right now, that is the only explanation I can find that would explain why certain characters would experience loot lag while others do not.

Please let me know what your testing finds.  I will keep looking and will probably update the source to remove the extra Save() that is not needed unless Secrets or someone else gets to it first.


*** EDIT ***

Well, from looking through the source, I think there are probably a few more places where Save() may be able to be removed to improve performance.  For one; in client_packet.cpp in Handle_OP_ShopPlayerSell it also uses AddMoneyToPP() which does Save() as well as another Save(1) at the end of the function.  So, if people are experiencing loot lag, they may also see lag while selling to merchants due to the same double Save() reason as looting.

Also, while looking closer at the Save() function, I noticed that  SavePetInfo() was added by Leere on the Rev2069 Commit on Nov 26 2011.
http://code.google.com/p/projecteqemu/source/detail?r=2069

If you take a look at the SavePetInfo() function, it does quite a few extra DB queries.
http://code.google.com/p/projecteqemu/source/search?q=SavePetInfo&origq=SavePetInfo&btnG=Search+Trunk

For anyone who ran a server a few years ago, they may remember the issue with lag caused by server-wide character Save()s that happened every 60 seconds by default.  With all of the Save()s in the code now, a big server could probably suffer from similar performance issues like were seen at that time, but more on a player by player basis I think depending on what they are doing.

I think to really fix this, it might require a few complex changes.  The easiest fix to improve performance would probably be to move the SavePetInfo() function out of the Save() function, but then we would need to find the best spots to put it, which would take some research and testing.  Leere may know offhand which times are essential to make sure that info is saved.  I would guess anytime you zone, when you summon a pet, when a pet dies, when a pet is given items or buffed, and when you log out might be enough, but there could be others.

Next, the best way to improve performance would be to get rid of the damn player profile blob in the database.  Then, instead of having to do a whole Save() (which saves a ton of extra crap), we could just have it save money value changes when selling something, or inventory changes, etc all separately as needed.  Though, that would require individual functions for each and also all of those functions to be distributed out in the source to the correct functions as needed.  I think this could boost overall performance considerably and allow a much higher number of players on a server before lag is seen (assuming the internet connection can handle it).

I do still think that Discovered Items can be written to not cause nearly as many DB hits as it does by just loading Discovered Items into memory so we only need to hit the database if something isn't already loaded into memory as being discovered.  But, I don't think it is the cause of the loot lag that has been seen on several servers recently.

Awesome find Trev. Will test autosplit asap.

I dont have autosplit turned on for our box toons, so it is not causing our lag at least.

Good info as always.

FYI our character_ table is up to ID of 141,000+ and I hate those huge char "blobs" and been wanting for a long long time for that stuff to be split up into fields rather than 1 big blob. I can see how that would reduce lag when the char is saved as well. I realize it would take a lot of work, but would be well worth it and appreciated if anyone was able to accomplish this.

On a side now, I'm still looking for a way to purge old characters by time played, last played, etc. There was a tool made already, but took FOREVER for the program to read the character_ table even when ran locally, that I gave up trying to use it.

Keep up the good work Trevius and thanks for stopping in again! Hopefully this can get figured out.

:)

my idea is maybe a little easier,

implement bots so that you could have 5 bots insted of 6 chars logged in, the only problem with this is bots cant use clicky effects, and there would have to be quest wrote for things like bot epics and the such.

that would meen a little work writing a few quests and maybe even making a few epics to be bot specific like changing the click effect of the epic to maybe a proc effect or something cause things like epic mage pets and other pet classes wouldnt get to use thier epic pets as bots.

this would cut our char log in down drastically like maybe to say 1/2 cause people would only box the chars they absolutly had to use click effects for like pet classes.

as far as tier spells are conserned it wouldnt be hard to make a quest you do and turn in with the bot up and in group and do a check sum to check if the class bot is in your group to teach the bot the spell

i dont know maybe it is harder than i am thinking to do as far as that goes, but it would cut down on char lag if even 1 bot per char was activated.

With one group of my own toons I can do what 3 groups of bots can do. Bots do not work very well at all when compared to boxed toons.

"/autosplit off" has no effect on both Hateborne and Verlorenen.

Both Verlorenen and Hateborne have been here over a year. My new berserker is CRAZY fast looting. Even when mobs where bug-pathing across the zone in random directions, zerker said *loot* *loot* *loot*.

If there is anything I can do to help with this, lemme know.

-Hate

+100

And most of the people who box already would not give up their boxed toons to make crappy bots.... So it would be counter-intuitive, really... The newer players to the server or the very few who do not box may like this idea, but I doubt the majority of the server would. (not that I can speak for all of them.)

Spot on there.  Boxing with alt toons takes effort and there is a steeper learning curve compared to #spawn bot.  I'll stick with mashing buttons to get my heals rather than hoping the AI works.

Been about a year since I played on the server, im glad my toons did not get deleted. But yea, there is a noticeable difference in the loot time. I wanted to do Qvic and get gems to get my plat amount up, but having to wait 2-3 seconds for the loot window to come up most of the time made me lose interest pretty quick.

Going to test something tonight, I have a hunch.

Yesterday I was running 4 and 5 with minimal loot lag during the day. My girlfriend got home and we ran 4 together. Lag was taking around 2-3 seconds per mob again. We noticed it was getting so bad even pulls were lagging us out, and eventually it got so bad that we ended up dying because the mobs were hitting us from no where. I ran the coh mage back to rez our corpses and when we loaded back in, we could loot 1-2 mobs per second. With that short of a time frame I dont imagine 100 people logged off or anything. So I guess for the people that know, what happens to character data after dying? Does it maybe do something that would speed up looting?