Help needed with curl

Sync Blizzards' Armory data with WoWRoster (addon depreciated no longer works see ApiSync)

Moderators: Ulminia, poetter

Help needed with curl

Postby poetter » Mon Nov 26, 2007 12:30 am

Hi all,

i modified the urlgrabber method in functions.lib.php, but only the fsockopen part. My problem is, that i don't have a host with curl support to test and debug this. So i need your help.

What is happening on a guild update?

Roster fetches the following url (eu example):
Code: Select all
http://eu.wowarmory.com/guild-info.xml?r=[realm_name]&n=[guild_name]&p=1&locale=[locale]

The armory returns a 302 Moved temporarily. So with fsockopen it returns a header with a location line in it. I parse the answers header for a line containing Localtion: http://.... and call the urlgrabber method again with the url provided in the Location header.

Here is the code doing it. See the preg_match line for parsing and the if ( isset($redirect[1]) ) line for calling urlgrabber once again:

Code: Select all
      $file = fsockopen($host, $port, $errno, $errstr, $timeout);
      if( !$file )
      {
         trigger_error("UrlGrabber Error [fsock]: $errstr ($errno)", E_USER_WARNING);
         return false;
      }
      else
      {
         $header = "GET $page$page_params HTTP/1.0\r\n"
               . "Host: $host\r\n"
               . "User-Agent: $user_agent\r\n"
               . "Connection: Close\r\n\r\n";
         fwrite($file, $header);
         stream_set_timeout($file, $timeout);
         $inHeader = true;
         $redirect = false;
         while( !feof($file) )
         {
            $chunk = fgets($file, 256);
            if( $inHeader )
            {
               $pos = strpos($chunk, '<');
               if( $pos !== false )
               {
                  $contents .= substr( $chunk, $pos, strlen($chunk) );
                  $inHeader = false;
               }
               if( preg_match ( '/^(?:Location:\s)(.+)/', $chunk, $redirect ) )
               {
                  break;
               }
               continue;
            }
            $contents .= $chunk;
         }
         fclose($file);
         if ( isset($redirect[1]) )
         {
            return urlgrabber( $redirect[1], $timeout, $user_agent );
         }
         else
         {
            return $contents;
         }
              }


So what needs to be done with the curl part?

I don't know. Lets have a look at the code:

Code: Select all
   if( function_exists('curl_init') )
   {
      $ch = curl_init($url);

      curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
      if( $user_agent )
      {
         curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
      }
      $contents = curl_exec($ch);

      // If there were errors
      if( curl_errno($ch) )
      {
         trigger_error('UrlGrabber Error [CURL]: ' . curl_error($ch), E_USER_WARNING);
         return false;
      }

      curl_close($ch);

      return $contents;
   }


I would expect that the error curl_errno($ch) should contain something useful the recognize that the site is redirected. If not, there must at least something in the content.

Can anyone debug this function and post the content of curl_errno($ch), curl_error($ch) and $contents? There should be a way to get into this.

Thx for your help.
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: Help needed with curl

Postby tuigii » Mon Nov 26, 2007 1:16 am

Yeah,

.... curl_errno($ch), curl_error($ch) and $contents? ....

That where I'm am right now.
I know curl works on my host(see PM for example).

So, what I think is that wowarmory introduced/prepared some auth. scheme stuff, for the vault acces, and thus f*ck*ng up the acces with a redirect.

When I use the obtained $url myself in IE :
Code: Select all
http://eu.wowarmory.com/guild-info.xml?r=Khaz+Modan&n=Papy+team&p=1&locale=fr


it will replace it with a
Code: Select all
http://eu.wowarmory.com/guild-info.xml;jsessionid=968D53B2C40BFE70A6A2CEF8BB74772C.app06_03?r=Khaz+Modan&n=Papy+team&p=1

right away.....
Dono if that's related.

Note that the &locale=fr is missing.
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

Re: Help needed with curl

Postby tuigii » Mon Nov 26, 2007 2:07 am

For what it's worth :

armory/clas.php : line 904 - blow away the param locale like this
Code: Select all
$url = $base_url . $mode; // . '&locale=' . $locale;


It looks like (when looking in the cache folder) that info is getting in fast .... :D

Then, a blowup will happen here (and only here) :
armorysync.class.php - line (env. I patched) : 740 :
Code: Select all
require_once ($addon['dir'] . 'inc/talenticons_'. $class. '.php');

$class is not defined ..... :shock:

So, I modified this block (same neighbourhood - a couple of line sup) :
Code: Select all
            $armoryTalents = $content->characterInfo->talentTab->talentTree->value;
            $talentArray = preg_split('//', $armoryTalents, -1, PREG_SPLIT_NO_EMPTY);
            echo "--->".$this->data["Class"]."<--- ";
            //$dl_class = $roster->locale->act['class_to_en'][$this->data["Class"]];
            $dl_class = $this->data["Class"];
            echo "--->".$dl_class."<--- ";
            //$dl_class = $this->_getDelocalisedClass($this->data["Class"]);
            $class = strtolower($dl_class);
            $locale = $roster->config['locale'];

In the XML array that came in I only found "Warrior" for a Class, instead of a "Geurrier" (French) because, well, look above :wink:

Have also a look at the other "act['class_to_en'][$this->data["Class"]" part in the code --->
It is stated just above :
// This is an ugly workaround for an encoding error in the armory


Related ?
Last edited by tuigii on Mon Nov 26, 2007 4:09 am, edited 1 time in total.
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

Help needed with curl

Postby poetter » Mon Nov 26, 2007 7:56 am

All right,

thx to tuigii i was able to get redirects work with curl to.

For the impatient ones among us add this patch to functions.lib.php. All others wait till the patch is committed to the trunk.

Code: Select all
--- C:/Dokumente und Einstellungen/Daniel/Lokale Einstellungen/Temp/functions.lib.php-revBASE.svn000.tmp.php   Sun Nov 25 23:49:04 2007
+++ D:/xampp/roster.com/htdocs/lib/functions.lib.php   Sun Nov 25 23:48:52 2007
@@ -1138,6 +1138,7 @@
 
       curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
+      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
       if( $user_agent )
       {
          curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: Help needed with curl

Postby tuigii » Mon Nov 26, 2007 8:18 am

tuigii wrote:.....
Note that the &locale=fr is missing.


Me being stupid: lost precious time because I didn't interpreted this as I should: Bliz is redirecting ^^^^^^

Poetter :: GG, man.
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

Help needed with curl

Postby zanix » Mon Nov 26, 2007 10:57 am

functions.lib.php fixed svn[1478]
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5546
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

Re: Help needed with curl

Postby Dooks » Mon Nov 26, 2007 6:54 pm

poetter wrote:All right,

thx to tuigii i was able to get redirects work with curl to.

For the impatient ones among us add this patch to functions.lib.php. All others wait till the patch is committed to the trunk.

Code: Select all
--- C:/Dokumente und Einstellungen/Daniel/Lokale Einstellungen/Temp/functions.lib.php-revBASE.svn000.tmp.php   Sun Nov 25 23:49:04 2007
+++ D:/xampp/roster.com/htdocs/lib/functions.lib.php   Sun Nov 25 23:48:52 2007
@@ -1138,6 +1138,7 @@
 
       curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
+      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
       if( $user_agent )
       {
          curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);


Thanks, works like a charm!
Dooks
WR.net Apprentice
WR.net Apprentice
 
Posts: 3
Joined: Mon Nov 26, 2007 6:52 pm

Re: Help needed with curl

Postby wallfw » Mon Nov 26, 2007 10:20 pm

zanix wrote:functions.lib.php fixed svn[1478]


not in roster diag yet?
wallfw
WR.net Apprentice
WR.net Apprentice
 
Posts: 15
Joined: Mon Apr 30, 2007 7:52 pm

Help needed with curl

Postby zanix » Tue Nov 27, 2007 2:22 am

Not yet, but if you have a svn client, you can download the current svn revision yourself
http://www.wowroster.net/SVNergy.html
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5546
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

Help needed with curl

Postby fewyn » Wed Nov 28, 2007 11:38 am

Hmm... running the SVN version and when I run it goes "No Information for Character" and doesn't pick up anyone. Seems to be happening with all my guilds. All the guilds are on Norgannon (A Series of Tubes and Argent Dusk) ideas?
User avatar
fewyn
WR.net Apprentice
WR.net Apprentice
 
Posts: 25
Joined: Wed Jul 05, 2006 8:38 am

Help needed with curl

Postby benzo » Thu Nov 29, 2007 3:03 am

same here - i got some down but now it wont get the last 30 persons it just keeps giving me that theres no data
benzo
WR.net Apprentice
WR.net Apprentice
 
Posts: 13
Joined: Thu Nov 09, 2006 2:50 am

Re: Help needed with curl

Postby poetter » Sun Dec 09, 2007 3:14 am

Hi there,

after analyzing the armory i found that blizz i throttling the request if it does not use the session ids. I was able to get this working with fsock, but as always i need someone top test some curl code.

Please apply this patch to functions.lib.php and let me know if its working:

Hmm, had to attach the diff, cause the code tag was leaving some stuff out. Any problems known in the forum code with the tag :jsessionid ?
Attachments
functions.lib.php.diff.zip
(1.21 KiB) Downloaded 291 times
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Help needed with curl

Postby zanix » Sun Dec 09, 2007 6:01 am

Changed svn[1513]
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5546
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

Re: Help needed with curl

Postby poetter » Sun Dec 09, 2007 6:06 am

zanix wrote:Changed svn[1513]


Did it work?
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: Help needed with curl

Postby poetter » Sun Dec 09, 2007 7:38 am

There was a bug in the patch. Here is a patch for the patch:

Code: Select all
--- C:/Dokumente und Einstellungen/Daniel/Lokale Einstellungen/Temp/functions.lib.php-revBASE.svn002.tmp.php   Sat Dec 08 23:36:46 2007
+++ D:/xampp/roster.com/htdocs/lib/functions.lib.php   Sat Dec 08 23:35:58 2007
@@ -1225,7 +1225,6 @@
                }
                if( preg_match('/^(?:Location:\s)(.+)/', $chunk, $tmp) )
                {
-                  break;
                   $redirect = $tmp[1];
                }
                continue;
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Next

Return to ArmorySync - Depreciated

Who is online

Users browsing this forum: No registered users and 1 guest

cron