PvPlog too big

Forum Dedicated to PvPLog, (The WoW PvP data collector addon), and PvPLog related topics

Moderator: bsmorgan

PvPlog too big

Postby gomi » Tue Jan 29, 2008 4:21 am

One of my new guildmates is trying to upload his pvplog and it's too big for my server to handle. (2.5 MB)
It's not the actual physical file size but when it is being parsed. I can't edit the php.ini so I was wondering how I could break the file apart and do it in increments.

Thanks
User avatar
gomi
WR.net Apprentice
WR.net Apprentice
 
Posts: 4
Joined: Thu Jul 06, 2006 8:31 am

PvPlog too big

Postby bsmorgan » Tue Jan 29, 2008 8:47 am

I'm not sure I understand the problem yet.

Is the pvplog.lua file 2.5mb in size?

What error do you get that leads you to say its too big?

How many characters are in the pvplog.lua file? You might be able to figure this out by grepping for "PurgeCounter" and displaying a couple of lines on either side. On Windows, look at the file with a text editor and search for and count all the occurances of PurgeCounter. This counter starts at 5000 and counts up by one for each record added.

I'm just guessing but you'd either break the file into pieces separated by character and if that isn't small enough, you'd break each character into pieces separating the battles across the pieces. I'm not familiar enough with the parsing process to know for sure how either of these methods will work.

Regards,

Brad
User avatar
bsmorgan
WoW Interface Developer
WoW Interface Developer
 
Posts: 160
Joined: Tue Nov 28, 2006 10:17 pm
Location: Colorado Springs, Colorado

PvPlog too big

Postby gomi » Tue Jan 29, 2008 10:07 am

The actual pvplog file is 2.5 MB.
There are 7 occurrences of PurgeCounter
I get a 500 internal server error.
I removed all references of his main character and was able to load the file with no problem. Of course when I did this the file size went to 200k. This lead me to believe the problem was in the parsing of the file.

I'll work on separating the battles out and see if that works.
User avatar
gomi
WR.net Apprentice
WR.net Apprentice
 
Posts: 4
Joined: Thu Jul 06, 2006 8:31 am

PvPlog too big

Postby PleegWat » Tue Jan 29, 2008 12:33 pm

After parsing, the size of a typical LUA file increases by a factor of 5. This means a 2.5 MB pvplog.lua will take around 12.5 MB of system memory after it has been parsed. The default php memory limit is much lower than that, at 8 MB.

I think the only way to reduce the file size here is to clean up old data that is already in the roster. Easiest way to do that is deleting the file after each upload. But I'm not too familiar with pvplog, either clientside or serverside, so don't take my word on it.
I <3 /bin/bash
User avatar
PleegWat
WoWRoster.net Dev Team
WoWRoster.net Dev Team
 
Posts: 1636
Joined: Tue Jul 04, 2006 1:43 pm

PvPlog too big

Postby bsmorgan » Wed Jan 30, 2008 8:09 am

While deleting the pvplog.lua file after each upload will solve the parsing problem, it eliminates a lot of the in-game functionality provided by PvPLog.

I believe the PurgeLogData is the only data parsed by WoWRoster and it is the data that could probably be deleted from saved variables file after being uploaded. I'll have to look at the code for a bit to see what would be lost if that data were removed. Some intelligence might need to be implemented to remove old data while preserving recent data for the maximum functionality.

Regards,

Brad
User avatar
bsmorgan
WoW Interface Developer
WoW Interface Developer
 
Posts: 160
Joined: Tue Nov 28, 2006 10:17 pm
Location: Colorado Springs, Colorado

PvPlog too big

Postby zanix » Wed Jan 30, 2008 11:29 am

Yes, only PurgeLogData is parsed by Roster
It even skips old entries as well
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5543
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

PvPlog too big

Postby bsmorgan » Wed Jan 30, 2008 2:55 pm

I've modified the code to copy some data from PurgeLogData into PvPLogData so that old PurgeLogData can be removed with minimal adverse effect.

I've added a command (/pl keep N) that will delete old records in PurgeLogData so that only the most recent N remain.

I'll post a new version after I've done a bit more testing.
User avatar
bsmorgan
WoW Interface Developer
WoW Interface Developer
 
Posts: 160
Joined: Tue Nov 28, 2006 10:17 pm
Location: Colorado Springs, Colorado

PvPlog too big

Postby gomi » Fri Feb 01, 2008 9:40 pm

Thanks.
User avatar
gomi
WR.net Apprentice
WR.net Apprentice
 
Posts: 4
Joined: Thu Jul 06, 2006 8:31 am


Return to PvPLog

Who is online

Users browsing this forum: No registered users and 0 guests

cron