New approach to functions.lib.php

Sync Blizzards' Armory data with WoWRoster (addon depreciated no longer works see ApiSync)

Moderators: Ulminia, poetter

New approach to functions.lib.php

Postby poetter » Fri Dec 14, 2007 3:22 am

Hi,

i found a way to fetch infos from the armory without getting redirects. Anyway i made the function capable of handling redirects and cookies. This is not needed now, but may help in the future.

I was able to get my windows system using curl, so the code is tested with fsock and curl. Hope this is the last approach and we can go one fixing data related stuff in AS. Try out the patches an let me know how they are doing on your system. If you encounter any problems please give me as much infos as possible.

The changes need two patches, so here they are:

Code: Select all
--- C:/Dokumente und Einstellungen/Daniel/Lokale Einstellungen/Temp/functions.lib.php-revBASE.svn001.tmp.php   Thu Dec 13 19:13:12 2007
+++ D:/xampp/roster.com/htdocs/lib/functions.lib.php   Thu Dec 13 19:12:14 2007
@@ -1132,22 +1132,15 @@
 {
    global $roster;
 
-   $cache_tag = 'JSESSIONID';
-
-   $jSessionId = '';
-   if( $roster->cache->check($cache_tag) && ! preg_match('/jsessionid/', $url) )
-   {
-      $jSessionId = $roster->cache->get($cache_tag);
-      list ($mainUrl, $getVars) = split ("\?", $url);
-      $url = $mainUrl. ';jsessionid=' . $jSessionId . '?' . $getVars;
-   }
+   $pUrl = parse_url($url);
+   $cache_tag = $pUrl['host'].'_cookie';
 
    $loopcount++;
    $contents = '';
 
    if( $loopcount > 2 )
    {
-      trigger_error("UrlGrabber Error: No many loops. Unable to grab URL ($url)", E_USER_WARNING);
+      trigger_error("UrlGrabber Error: To many loops. Unable to grab URL ($url)", E_USER_WARNING);
       return $contents;
    }
 
@@ -1155,9 +1148,15 @@
    {
       $ch = curl_init($url);
 
+      $httpHeader = array( 'Accept-Language: '. substr($roster->config['locale'], 0, 2) );
+      if( $roster->cache->check($cache_tag) )
+      {
+         $httpHeader[] = 'Cookie: '. $roster->cache->get($cache_tag);
+      }
       curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
-      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
+      curl_setopt($ch, CURLOPT_HEADER, true);
+      curl_setopt($ch, CURLOPT_HTTPHEADER, $httpHeader);
       if( $user_agent )
       {
          curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
@@ -1170,17 +1169,36 @@
          trigger_error('UrlGrabber Error [CURL]: ' . curl_error($ch), E_USER_WARNING);
          return false;
       }
-      $lastUrl = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
+
+      if ( preg_match ('/\r/', $contents, $tmp ) ) {
+         list($resHeader, $data) = explode("\r\n\r\n", $contents, 2);
+      } else {
+         list($resHeader, $data) = explode("\n\n", $contents, 2);
+      }
+        $http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
+      curl_close($ch);
+
       $tmp;
-      if( preg_match('/(?:jsessionid=)(.+)?\?/', $lastUrl, $tmp) )
+      if( preg_match('/(?:Set-Cookie: )(.+)/', $resHeader, $tmp) )
       {
          $roster->cache->put($tmp[1], $cache_tag);
       }
 
+        if ($http_code == 301 || $http_code == 302)
+        {
+            $matches = array();
+            preg_match('/Location:(.*?)\n/', $resHeader, $matches);
+            $redirect = trim(array_pop($matches));
+            if (!$redirect)
+            {
+                //couldn't process the url to redirect to
+                return $data;
+            }
 
-      curl_close($ch);
-
-      return $contents;
+            return urlgrabber( $redirect, $timeout, $user_agent, $loopcount );
+        } else {
+            return $data;
+        }
    }
    elseif( preg_match('/\bhttps?:\/\/([-A-Z0-9.]+):?(\d+)?(\/[-A-Z0-9+&@#\/%=~_|!:,.;]*)?(\?[-A-Z0-9+&@#\/%=~_|!:,.;]*)?/i', $url, $matches) )
    {
@@ -1201,16 +1219,22 @@
          $header = "GET $page$page_params HTTP/1.0\r\n"
                . "Host: $host\r\n"
                . "User-Agent: $user_agent\r\n"
-               . "Connection: Close\r\n\r\n";
+               . "Accept-Language: ". substr($roster->config['locale'], 0, 2). "\r\n"
+               . "Connection: Close\r\n";
+         if( $roster->cache->check($cache_tag) ) {
+            $header .= "Cookie: ". $roster->cache->get($cache_tag). "\r\n";
+         }
+         $header .= "\r\n";
          fwrite($file, $header);
          stream_set_timeout($file, $timeout);
          $inHeader = true;
          $redirect = false;
+         $resHeader = '';
          $tmp = '';
          while( !feof($file) )
          {
             $chunk = fgets($file, 256);
-            //print htmlspecialchars($chunk). "<br />";
+            1;
             if( $inHeader )
             {
                $pos = strpos($chunk, '<');
@@ -1218,10 +1242,8 @@
                {
                   $contents .= substr( $chunk, $pos, strlen($chunk) );
                   $inHeader = false;
-               }
-               if( preg_match('/^(?:Set-Cookie: JSESSIONID=)(.+)?\;/', $chunk, $tmp) )
-               {
-                  $roster->cache->put($tmp[1], $cache_tag);
+               } else {
+                  $resHeader .= $chunk;
                }
                if( preg_match('/^(?:Location:\s)(.+)/', $chunk, $tmp) )
                {
@@ -1232,6 +1254,10 @@
             $contents .= $chunk;
          }
          fclose($file);
+         if( preg_match('/(?:Set-Cookie: )(.+)/', $resHeader, $tmp) )
+         {
+            $roster->cache->put($tmp[1], $cache_tag);
+         }
          if( $redirect != false )
          {
             return urlgrabber( $redirect, $timeout, $user_agent, $loopcount );

Code: Select all
--- C:/Dokumente und Einstellungen/Daniel/Lokale Einstellungen/Temp/armory.class.php-revBASE.svn000.tmp.php   Thu Dec 13 19:19:11 2007
+++ D:/xampp/roster.com/htdocs/lib/armory.class.php   Thu Dec 13 19:19:03 2007
@@ -901,7 +901,7 @@
             break;
       }
 
-      $url = $base_url . $mode . '&locale=' . $locale;
+      $url = $base_url . $mode;
 
       if( $this->debug_url )
       {
Last edited by poetter on Fri Dec 14, 2007 3:29 am, edited 2 times in total.
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

New approach to functions.lib.php

Postby zanix » Fri Dec 14, 2007 4:18 am

What svn revision of functions.lib.php is this?

Plus, can I get the whole file, I can't find a decent diff patch program for windows
Read the Forum Rules, the WiKi, and Search before posting!
WoWRoster v2.1 - SigGen v0.3.3.523 - WoWRosterDF
User avatar
zanix
Admin
Admin
WoWRoster.net Dev Team
WoWRoster.net Dev Team
UA/UU Developer
UA/UU Developer
 
Posts: 5543
Joined: Mon Jul 03, 2006 8:29 am
Location: Idaho Falls, Idaho
Realm: Doomhammer (PvE) - US

Re: New approach to functions.lib.php

Postby poetter » Fri Dec 14, 2007 4:26 am

zanix wrote:What svn revision of functions.lib.php is this?

Plus, can I get the whole file, I can't find a decent diff patch program for windows


That was svn[1516]. File's attached. Just unzip.

Patch for Windows
Attachments
functions.lib.php.zip
(11.07 KiB) Downloaded 84 times
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: New approach to functions.lib.php

Postby tuigii » Fri Dec 14, 2007 4:41 am

Well, what can I say ?

Let's do it with an image :wink:

Image

But I saw also some

Image

Although I think these 'errors' are not to blame to your software.


PS : I did a manual merge.
What kind of tools do you use ?
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

New approach to functions.lib.php

Postby poetter » Fri Dec 14, 2007 4:51 am

Well actually, i never used patch on Windows. Always just made the diffs. I use WinMerge to create the diffs.

I tried Patch for Windows. Just go to your lib directory. Copy patch.exe somewhere in your PATH. Copy the diff to lib dir and run

patch --verbose < functions.lib.php.diff
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

Re: New approach to functions.lib.php

Postby tuigii » Fri Dec 14, 2007 5:46 am

:D

While testing (the Armory is sloooow again this evening on EU realms) en image capturing, my post was burend by Zanix, and you - with answers on the question that I even didn't typed yet ^^

Thanks for the tools.

Still syncing well :wink:
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

New approach to functions.lib.php

Postby PleegWat » Fri Dec 14, 2007 5:56 am

I rarely need to merge either, but I frequently view diffs generated by SVN or CVS.

$ svn diff -r HEAD:WORKING some/path/or/url

Will generate a (unified, if a directory is specified) diff between your working copy version and the current SVN head revision of a file. This command should work on windows as well, if you've got the command-line svn client.
I <3 /bin/bash
User avatar
PleegWat
WoWRoster.net Dev Team
WoWRoster.net Dev Team
 
Posts: 1636
Joined: Tue Jul 04, 2006 1:43 pm

New approach to functions.lib.php

Postby Ulminia » Fri Dec 14, 2007 6:20 am

ok i work with 2 simple rules dont ask and readling 11000 lines of code sucks... i got it worken on my server as updates a character :)
Ulminia of Zangarmarsh
Zonous of Zangarmarsh
Author of Roster Gallery
WoWRoster-Profiler Redesigner
User avatar
Ulminia
WoWRoster.net Dev Team
WoWRoster.net Dev Team
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 1223
Joined: Tue Jul 04, 2006 4:41 pm
Location: New Brunswick, Canada
Realm: Zangarmarsh (PvE) - US
gmail/gtalk: ulminia@gmail.com

New approach to functions.lib.php

Postby Equisetum » Fri Dec 14, 2007 12:00 pm

Hello all,

I installed the lib above, and am still getting the following error:

lib/functions.lib.php
Roster Warning line 1169: UrlGrabber Error [CURL]:
lib/armory.class.php
Roster Notice line 249: RosterArmory:: Failed to fetch http://www.wowarmory.com/character-shee ... &locale=en
addons/armorysync/inc/armorysync.class.php
Warning line 850: array_keys() [function.array-keys]: The first argument should be an array
Warning line 851: Invalid argument supplied for foreach()
Notice line 844: Undefined offset: 0


This is after I let it go through a few users thinking maybe one would kick in, but they all had the "No infos" in the log at the right.

php 4.4.6
mysql 5.044
Safe mode is off
Curl is enabled
open_basedir is still on, though (hence this post)

Let me know if you need more information.

Great work so far on everything! Really top notch. Thanks so much for the help and your patience getting this rolling!
Image
User avatar
Equisetum
WR.net Apprentice
WR.net Apprentice
 
Posts: 65
Joined: Mon Jul 24, 2006 6:44 am

New approach to functions.lib.php

Postby poetter » Fri Dec 14, 2007 12:36 pm

You need to apply the patch to armory.class.php, too
Image
User avatar
poetter
Roster AddOn Dev
Roster AddOn Dev
 
Posts: 462
Joined: Sat Jun 30, 2007 9:41 pm
Location: Germany/Hamburg

New approach to functions.lib.php

Postby Equisetum » Fri Dec 14, 2007 1:59 pm

Ok, that was pretty stupid of me. Thanks!

It still didn't work though, but I did notice that the locale parameter is dropped from the makeurl now...

lib/functions.lib.php
Roster Warning line 1169: UrlGrabber Error [CURL]: connect() timed out!
lib/armory.class.php
Roster Notice line 249: RosterArmory:: Failed to fetch http://www.wowarmory.com/character-shee ... lexstrasza
addons/armorysync/inc/armorysync.class.php
Warning line 850: array_keys() [function.array-keys]: The first argument should be an array
Warning line 851: Invalid argument supplied for foreach()
Notice line 844: Undefined offset: 0


I tried patching the file three times and looked at it to see if it worked alright, and so far so good (this coming from someone who forgot to patch the file in the first place). Is there something else I can check?

Is it possible that armory's just taking so long to respond that it times out?

Thanks :)

EDIT: /sigh Perhaps this is why it's not working.... :?
Last edited by Equisetum on Fri Dec 14, 2007 2:12 pm, edited 2 times in total.
Image
User avatar
Equisetum
WR.net Apprentice
WR.net Apprentice
 
Posts: 65
Joined: Mon Jul 24, 2006 6:44 am

New approach to functions.lib.php

Postby Knevil » Fri Dec 14, 2007 10:19 pm

i tried to patch the files with this winpatch thing but it dont work, maybe i'm to stupid ;)

i copied the stuff out of the first post an created a file named functions.lib.php.diff, copied this one and the svn 1516 file in the bin folder of patch an enter in patch command line: patch --verbose < functions.lib.php.diff
but nothing happened.

sorry for my bad english, maybe poetter can give me the instructions in german :)
User avatar
Knevil
WR.net Apprentice
WR.net Apprentice
 
Posts: 29
Joined: Thu May 31, 2007 7:27 pm
Location: Germany

New approach to functions.lib.php

Postby tuigii » Fri Dec 14, 2007 11:20 pm

Poetter has posted the entire 'test' functions.lib.php file also :-)

Remember to change this file : armory.class.php also (just one line)
User avatar
tuigii
WR.net Master
WR.net Master
 
Posts: 891
Joined: Wed Dec 27, 2006 12:57 pm
Location: Somewhere in the South Ouest of France

New approach to functions.lib.php

Postby Knevil » Fri Dec 14, 2007 11:50 pm

ok now it works, thank you tuigii,
char sync works a little bit, guild update not
think you will find a way to fix it :)

*edit*
Guild update works but char update gives a white screen after some seconds, like timeout"
*/edit*
Last edited by Knevil on Sat Dec 15, 2007 1:41 am, edited 1 time in total.
User avatar
Knevil
WR.net Apprentice
WR.net Apprentice
 
Posts: 29
Joined: Thu May 31, 2007 7:27 pm
Location: Germany

New approach to functions.lib.php

Postby Equisetum » Sat Dec 15, 2007 4:11 am

I'm glad you're able to run it ok, Knevil!! What am I doing different?

Here's what I've done so far:
- Reinstalled
- Applied both patches x3
- Tried to update the roster in multiple ways (various armorysync buttons)
- Changed options for AS addon including AJAX, waittime and fetch timeouts
- Tried updating using both US and EU realms to see if there was a different way armory was being done by blizz (You EU ppl seem to have it working ok, I thought I could find a difference and troubleshoot myself)
- I even tried a diff computer and browsers (I know this wouldnt make a diff, but had nothing to lose)

I get the same or a variation on the errors I posted earlier.

lib/functions.lib.php
Roster Warning line 1169: UrlGrabber Error [CURL]:
lib/armory.class.php
Roster Notice line 249: RosterArmory:: Failed to fetch http://www.wowarmory.com/character-shee ... lexstrasza
addons/armorysync/inc/armorysync.class.php
Warning line 850: array_keys() [function.array-keys]: The first argument should be an array
Warning line 851: Invalid argument supplied for foreach()
Notice line 844: Undefined offset: 0


Safe mode Off
Curl On
open basedir is on

Any suggestions? Need more info?

Thanks!
Image
User avatar
Equisetum
WR.net Apprentice
WR.net Apprentice
 
Posts: 65
Joined: Mon Jul 24, 2006 6:44 am

Next

Return to ArmorySync - Depreciated

Who is online

Users browsing this forum: No registered users and 1 guest