Approach to DDoS Filter

Sync Blizzards' Armory data with WoWRoster (addon depreciated no longer works see ApiSync)

Moderators: Ulminia, poetter

Approach to DDoS Filter

Postby poetter » Sat Apr 05, 2008 7:44 am

Well, no i didn't make any progress and no i will not discontinue. I'm just not having any idea how to get this problem fixed.

AS is using SimpleXML and cURL. I read the links you posted and can't see any difference. All I see is some standard examples on how to use curl. SimpleXML has nothing to do with it in first place. It takes place when data is already fetched and converts XML to what ever you want.

So if you have a fix for that post some code! Not only I will be thankful.
Last edited by poetter on Sat Apr 05, 2008 7:46 am, edited 1 time in total.
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: Approach to DDoS Filter

Postby jharvey74 » Tue Apr 15, 2008 10:42 pm

I don't know if Blizzard made some changes to the Armory API or if I just got lucky tweaking my ArmorySynch settings, but it seems to be working much better now. It is able to synchronize with the Armory with no errors. The settings I use:

Synch Cutoff Time - 2
Use Ajax - On
Reload Wait Time - 5
Fetch Method - Be Smart
Fetch Timeout - 4
Fetch Retrys - 2
Update Incomplete Data - Off

Turning the incomplete data update off was the last thing I did. It goes through no errors now, but several of the characters don't synch (from incomplete data). But in successive trys, it will eventually synch them all (usually takes me 3 passes for a guild of about 150 members).
jharvey74
WR.net Apprentice
WR.net Apprentice
 
Posts: 8
Joined: Tue Mar 04, 2008 2:48 am

Re: Approach to DDoS Filter

Postby jerricho » Mon Apr 28, 2008 9:52 pm

No it's still not working.

After 5 people get updated it no longer will update anyone whether or not I have incomplete on or not.

I would imagine that until Blizzard allows people to do an XML export from their site, it's just not going to work because they are limiting connections due to so much data mining.
jerricho
WR.net Apprentice
WR.net Apprentice
 
Posts: 4
Joined: Thu Jul 06, 2006 10:52 pm

Re: Approach to DDoS Filter

Postby Rhoy » Tue Apr 29, 2008 1:22 am

I've been having pretty good luck lately. I generally try to sync a few times throughout the day. When the sync starts to pull data then everything comes through fine. It's an all or nothing type of situation.

We have 52 members and I get all their info in 1 pass. With the 52 members it takes about 2 hours to get everything updated.

I still get a good chunk of failed first tries but this timing appears to be the sweet spot. Any faster and it starts to get ugly.

Sync cutoff time - 1
Use AJAX - On
Reload wait time - 3
Armory Fetch method - per Page
Armory Fetch timeout - 3
Armory Fetch Retrys - 5
Update incomplete data - On
Skip start page - On

hope this helps!
Rhoy
WR.net Apprentice
WR.net Apprentice
 
Posts: 1
Joined: Sat Feb 16, 2008 9:34 am

Re: Approach to DDoS Filter

Postby jerricho » Tue Apr 29, 2008 1:57 am

Still not working. I've tried several times today. :(
jerricho
WR.net Apprentice
WR.net Apprentice
 
Posts: 4
Joined: Thu Jul 06, 2006 10:52 pm

Re: Approach to DDoS Filter

Postby jharvey74 » Thu May 01, 2008 10:23 pm

I'm still getting the same results from my end. I never get a complete failure anymore (where it would synch first few then fail OR fail on all of them). There are always several who don't update on each pass, and usually takes 3 or 4 passes to get them all. There is one guy who will not update ever (for incomplete data) and I'm not sure why, but oh well. I would suggest playing with all of the settings until you find a config that works for you (if there is one that will). I think the problem is more involved than just Blizzard's throttling and armorysych's core update process. Multiple people are getting different results and different configurations are working for different people, which indicates that conditions between armorysynch and the armory are heavily affecting it's performance.

Just my $0.02.
jharvey74
WR.net Apprentice
WR.net Apprentice
 
Posts: 8
Joined: Tue Mar 04, 2008 2:48 am

Approach to DDoS Filter

Postby Samadhi » Thu Jun 12, 2008 7:50 pm

Am I the only one who is being totally locked out by this filter? I can't get even a single one of my members to update. curl simply refuses to connect.

Roster Warning line 1108: UrlGrabber Error [CURL]: connect() timed out!

Has their been any progress on any front regarding how to get this working again?
Image
User avatar
Samadhi
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 114
Joined: Mon Jul 17, 2006 10:38 pm
Location: NJ, USA

Re: Approach to DDoS Filter

Postby tuigii » Thu Jun 12, 2008 8:46 pm

Samadhi wrote:Am I the only one who is being totally locked ....


Common mistake.

DO NOT believe that your side of the Internet communication is perfect - and that Blizzard is filtering.

What about your host (or your ISP) : it has a firewall and it is programmed.
CURL is considered as a "lean and mean" tool, and many hosts just block it "to protect their clients" (yeah, I lov'em....Thanks, Mum...).

Dono how to check it, but you get my point.

PS : Idea : load a Apache/PHP/SQL free package on your PC.
Read how it works : be carefull : one A4 to read ^^
Install Wowroster in it.
Check again.
You'll see.
Last edited by tuigii on Thu Jun 12, 2008 8:49 pm, edited 1 time in total.
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

Approach to DDoS Filter

Postby Samadhi » Thu Jun 12, 2008 9:03 pm

Ah, but I can use the same sort of code with a WoW tooltip mod on my forums if I hit the EU armory instead of the US one. If my host were blocking CURL, that wouldn't be true, right?
Image
User avatar
Samadhi
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 114
Joined: Mon Jul 17, 2006 10:38 pm
Location: NJ, USA

Approach to DDoS Filter

Postby zanix » Thu Jun 12, 2008 10:54 pm

They may be blocking the US site (www.wowarmory.com) and not the EU one (eu.wowarmory.com)
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5543
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

Approach to DDoS Filter

Postby Samadhi » Thu Jun 12, 2008 11:10 pm

Ok, fair enough. Thanks.
Image
User avatar
Samadhi
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 114
Joined: Mon Jul 17, 2006 10:38 pm
Location: NJ, USA

Approach to DDoS Filter

Postby petrakid » Sat Jun 14, 2008 9:51 pm

Sync cutoff time - 1
Use AJAX - On
Reload wait time - 3
Armory Fetch method - per Page
Armory Fetch timeout - 3
Armory Fetch Retrys - 5
Update incomplete data - On
Skip start page - On

These settings ARE working for me!! The process takes a LONG time, especially when updating gear info, but I haven't gotten a red circle yet, and it has not timed out. I think, until blizzard does something to help us out and improve at their end, we are going to have to expect a lengthy update process.
Image
User avatar
petrakid
WR.net Apprentice
WR.net Apprentice
 
Posts: 53
Joined: Sat Jul 07, 2007 7:34 am

Re: Approach to DDoS Filter

Postby MattM » Wed Jun 18, 2008 4:12 pm

poetter wrote:Hi,

in the last svn commit 378 I implemented a global pause and a inter char pause. The defaults are 0.25 secs for global pause and 10 secs for inter char pause. Global pause is the time between two single fetches and inter char pause is the time between two character updates.

With these values i was able to sync my own and one US guild completely without any errors. As the conditions vary from host to host you may need to play a little with these values.

Let me know how it is doing and witch values you tried. Make sure to do a break if you run into the filter.

If you still have problems let me know your values and what kind of host you are using(dedicated, vps, own ip, what so ever).

As I already made some changes for the latest template system, so this build is available from svn only and requires WoWRoster > svn[1722]. With the next nightly build i will pack another archive for non svn users.


Changing armorysync_global_pause to 10 from .25 and armorysync_inter_char_pause to 20 from 10 has no noticeable difference and speed, and the DDOS prevention kicks in as usual.

I don't think the while loop/usleep code is working to pause anything.
MattM
UA/UU Developer
UA/UU Developer
Gimpy Developer
Gimpy Developer
 
Posts: 886
Joined: Tue Jul 04, 2006 9:53 pm
Location: USA

Re: Approach to DDoS Filter

Postby MattM » Wed Jun 18, 2008 4:21 pm

petrakid wrote:Sync cutoff time - 1
Use AJAX - On
Reload wait time - 3
Armory Fetch method - per Page
Armory Fetch timeout - 3
Armory Fetch Retrys - 5
Update incomplete data - On
Skip start page - On

These settings ARE working for me!! The process takes a LONG time, especially when updating gear info, but I haven't gotten a red circle yet, and it has not timed out. I think, until blizzard does something to help us out and improve at their end, we are going to have to expect a lengthy update process.


AJAX is broken and without it on, I get crapped on:
Attachments
MattM
UA/UU Developer
UA/UU Developer
Gimpy Developer
Gimpy Developer
 
Posts: 886
Joined: Tue Jul 04, 2006 9:53 pm
Location: USA

Approach to DDoS Filter

Postby MattM » Wed Jun 18, 2008 4:30 pm

I may need to write another tool like UU for this.
MattM
UA/UU Developer
UA/UU Developer
Gimpy Developer
Gimpy Developer
 
Posts: 886
Joined: Tue Jul 04, 2006 9:53 pm
Location: USA

PreviousNext

Return to ArmorySync - Depreciated

Who is online

Users browsing this forum: No registered users and 0 guests

cron