Another curl update script

Miscellaneous code snippets and examples that members have posted

Another curl update script

Postby lukes » Tue Sep 09, 2008 5:30 pm

Hi,

I guess I should've checked the forum first, but I too made a upload batch file with curl. Perhaps you guys could link to one in the download section? Thats where I looked. :)

What can mine do:
- Upload data from CharacterProfiler, GroupCalendar and PvPLog
- compress with gzip
- update the saved signatures
- nice progress bar from curl
What it can't do:
- I don't have passwords, so I didn't implement them

Edit: Fixed gzip issue.

So, if anyone is interested, here it comes:
Code: Select all
ECHO OFF
ECHO.
ECHO Author: Jakob Lenfers (jakob@drss.de)
ECHO LICENCE: Creative Commons: Attribution-Share Alike 3.0 Unported
ECHO http://creativecommons.org/licenses/by-sa/3.0/
ECHO.
REM Please set the variables according to your environment.
REM You'll need curl from http://curl.haxx.se/ in and set
REM the path accordingly.
REM If you want your uploads to be faster, download gzip
REM from http://gnuwin32.sourceforge.net/packages/gzip.htm and set
REM the path. But thats optional. (only gzip.exe is needed from above
REM package.)

REM ###
REM ### General settings
REM ###

REM url of the roster without http:// and the trailing slash
SET WOWROSTERURL=roster.example.de

REM WoW directory
SET WOW=d:\world of warcraft

REM your WoW username
SET ACCOUNT=WOWUSERNAME

REM directory with curl in it
SET CURL=%ProgramFiles%\curl

REM directory with gzip in it. Optional, leave empty if
REM you don't want compression.
SET GZIP=%ProgramFiles%\gzip

REM do you want the result of your upload in your %TEMP% dir?
REM ("%TEMP%.\%WOWROSTERURL%-output.html")
SET SAVETEMP=FALSE

REM ###
REM ### Updating saved images settings
REM ###

REM Set to TRUE to update saved images. Needs the chars and sigs set up below.
SET SAVEDIMG=FALSE

REM charnames seperated by space, commas or semicolons. The code
REM might need more work for sites with multiple guilds...
SET CHARS=Char1;Char2;Char3

REM The available signatures from your roster, seperated just
REM like the chars with comma, semicolon or space.
SET SIGS=avatar;signature

REM ###
REM ### Other settings, shouldn't need no tweaking
REM ###

REM check curl doku for more info on some settings.
SET SAVEDVARS=%WOW%\wtf\account\%ACCOUNT%\savedvariables
SET STATS=-w "\nTime:       %%{time_total}s\nDownloaded: %%{size_download}b (%%{speed_download}b/s)\nUploaded:   %%{size_upload}b (%%{speed_upload}b/s)\n\n"
IF X%SAVETEMP%==XTRUE (
   SET OUTPUT=-o "%TEMP%.\%WOWROSTERURL%-output.html"
) ELSE (
   SET OUTPUT=-o "nul"
)

REM ###
REM ### Code
REM ###

REM check for curl, without it we can't do jack.
IF NOT EXIST "%CURL%\curl.exe" (
   ECHO Couldn't find curl.
   ECHO Please set up this script by editing it and filling
   ECHO out the variables at the top.
   pause
   exit
)

REM check what extensions are used and can be uploaded
IF EXIST "%SAVEDVARS%\CharacterProfiler.lua" (
   ECHO characterProfiler.lua found, uploading character data.
   IF NOT "X%GZIP%"==X IF EXIST "%GZIP%" (
      "%GZIP%\gzip" -c "%SAVEDVARS%\characterProfiler.lua" > "%SAVEDVARS%\CharacterProfiler.lua.gz" 2> nul
      SET FORMS=%FORMS% -F "characterprofiler=@%SAVEDVARS%\characterProfiler.lua.gz"
   ) ELSE (
      SET FORMS=%FORMS% -F "characterprofiler=@%SAVEDVARS%\characterProfiler.lua"
   )
)
IF EXIST "%SAVEDVARS%\GroupCalendar.lua" (
   ECHO GroupCalendar.lua found, uploading calendar data.
   IF NOT "X%GZIP%"==X IF EXIST "%GZIP%" (
      "%GZIP%\gzip" -c "%SAVEDVARS%\GroupCalendar.lua" > "%SAVEDVARS%\GroupCalendar.lua.gz" 2> nul
      SET FORMS=%FORMS% -F "groupcalendar=@%SAVEDVARS%\GroupCalendar.lua.gz"
   ) ELSE (
      SET FORMS=%FORMS% -F "groupcalendar=@%SAVEDVARS%\GroupCalendar.lua"
   )
)
IF EXIST "%SAVEDVARS%\PvPLog.lua" (
   ECHO PvPLog.lua found, uploading PvP data.
   IF NOT "X%GZIP%"==X IF EXIST "%GZIP%" (
      "%GZIP%\gzip" -c "%SAVEDVARS%\PvPLog.lua" > "%SAVEDVARS%\PvPLog.lua.gz" 2> nul
      SET FORMS=%FORMS% -F "pvplog=@%SAVEDVARS%\PvPLog.lua.gz"
   ) ELSE (
      SET FORMS=%FORMS% -F "pvplog=@%SAVEDVARS%\PvPLog.lua"
   )
)

ECHO.
ECHO Uploading...
"%CURL%\curl.exe" %FORMS% -F "process=process" --progress-bar %OUTPUT% %STATS% "http://%WOWROSTERURL%/index.php?p=update"


ECHO Updating images
IF X%SAVEDIMG%==XTRUE (
   FOR %%I IN (%CHARS%) DO (
      ECHO Updating saved images for %%I.
      FOR %%J IN (%SIGS%) DO (
         "%CURL%\curl.exe" -s -o "nul" "http://%WOWROSTERURL%/index.php?p=util-siggen-%%J&etag=0&member=%%I&saveonly=1"
      )
   )
)

ECHO.
ECHO Cleaning up.
FOR %%I IN (CharacterProfiler;GroupCalendar;PvPLog) DO (
   IF EXIST "%SAVEDVARS%\%%I.lua.gz" (
      DEL "%SAVEDVARS%\%%I.lua.gz"
   )
)
ECHO.

pause


Jakob
Last edited by lukes on Mon Sep 22, 2008 4:07 pm, edited 1 time in total.
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Re: Another curl update script

Postby lukes » Wed Sep 10, 2008 11:39 am

I found the idea to compress the uploaded data in the forum and tried that with other tools (with curl, zlib and --compress) for some time too. It looked like it worked fine, but I didn't notice that roster didn't like the data. :-)
I use gzip file.lua -c file.lua.gz and the just upload that, just like in the other solution in http://www.wowroster.net/Forums/viewtopic/t=2780.html

Code: Select all
# Fehler während Analyse von characterProfiler nach 0 Sekunden
# LUA parsing error. Are you sure this is a SavedVariables file?


Anyone an idea, why it isn't working? Need the compression, the files are over 1mb, it's annoying :-)

Jakob
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Another curl update script

Postby PleegWat » Wed Sep 10, 2008 1:21 pm

The error doesn't tell much. Try having roster print the posted file entirely.
I <3 /bin/bash
User avatar
PleegWat
WoWRoster.net Dev Team
WoWRoster.net Dev Team
 
Posts: 1636
Joined: Tue Jul 04, 2006 1:43 pm

Re: Another curl update script

Postby lukes » Wed Sep 10, 2008 3:50 pm

Strange, I just tried the curl line on my linux box at work and it worked... I'll check my win32 gzip version at home, maybe it doesn't get compressed right...

Jakob
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Re: Another curl update script

Postby lukes » Thu Sep 11, 2008 12:08 am

All right, I found out why it doesn't work, but it's strange: The foo.lua.gz file doesn't contain the foo.lua file, but instead gzip.exe!?

Strange. When I activate echoing the commands (ECHO ON) and execute the exact same command for compression the script uses, it works as it should. I find a foo.lua in the .gz. But as soon as I run it with my script, it contains gzip.exe again.

The command I'm executing is in my environment
Code: Select all
L:\Programme\gzip\gzip.exe "d:\world of warcraft\wtf\account\LENFI\savedvariables\characterProfiler.lua" -c  1>"d:\world of warcraft\wtf\account\LENFI\savedvariables\CharacterProfiler.lua.gz"

There seems nothing wrong with it, but why doesn't it work in my script?

Jakob
Last edited by lukes on Thu Sep 11, 2008 12:21 am, edited 1 time in total.
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Another curl update script

Postby ScratchMonkey » Thu Sep 11, 2008 5:15 am

You're probably getting burned by spaces in paths and insufficient quotes. Edit your script to put echo in front of the gzip command line so that it echoes the parsed line instead of executing it.

I recommend moving gzip.exe to c:\bin (or another drive) and putting that in your path. Don't put command line and programs or programs used by scripts in paths with spaces.

Similarly, remove spaces from the name of your WoW directory. I use c:\games\WoW, as it's easy to type. (I have other games installed to the same tree, such as \games\BF2 and \games\ETQW.)
User avatar
ScratchMonkey
WR.net Expert
WR.net Expert
 
Posts: 212
Joined: Wed Jul 05, 2006 4:32 pm
Location: San Pablo, CA

Re: Another curl update script

Postby lukes » Thu Sep 11, 2008 9:29 pm

Tried what you said, but it made no difference. :-( And with ECHO ON I can check what gets executed. I'ts properly quoted and works, when I execute it in a cmd.com seperatly.

Jakob
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Re: Another curl update script

Postby lukes » Thu Sep 11, 2008 11:00 pm

I just don't get it...

When I don't use the -c parameter (only gzip file.lua), gzip removes the .lua file and replaces it with a lua.gz. And that works. And that would work for me too, but it also creates a %GZIP-PATH%\gzip.exe.gz!? Why? Is my windows somehow broken? I tried two different gzip versions and can't find reports of this on google.

Going crazy here :bash:
Jakob
Last edited by lukes on Fri Sep 12, 2008 12:07 am, edited 1 time in total.
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm

Another curl update script

Postby MattM » Fri Sep 12, 2008 1:36 am

sounds like a bug in the executable

try a different gzipper
MattM
UA/UU Developer
UA/UU Developer
Gimpy Developer
Gimpy Developer
 
Posts: 886
Joined: Tue Jul 04, 2006 9:53 pm
Location: USA

Another curl update script

Postby ScratchMonkey » Fri Sep 12, 2008 2:03 am

So a 1-line batch file with no variable expansion exhibits the same problem?

I'm remembering that cmd.exe has some kind of options file under \Windows but I don't recall its name. Perhaps something in there is affecting how commands in batch files are executed. It sounds like gzip is seeing its own name as if it were an argument, as if the argv vector is getting generated incorrectly.
User avatar
ScratchMonkey
WR.net Expert
WR.net Expert
 
Posts: 212
Joined: Wed Jul 05, 2006 4:32 pm
Location: San Pablo, CA

Re: Another curl update script

Postby MattM » Fri Sep 12, 2008 2:31 am

ScratchMonkey wrote:So a 1-line batch file with no variable expansion exhibits the same problem?

I'm remembering that cmd.exe has some kind of options file under \Windows but I don't recall its name. Perhaps something in there is affecting how commands in batch files are executed. It sounds like gzip is seeing its own name as if it were an argument, as if the argv vector is getting generated incorrectly.


You should take some sanity pills before diving into "windows logic". Good luck!!!! :grin:
MattM
UA/UU Developer
UA/UU Developer
Gimpy Developer
Gimpy Developer
 
Posts: 886
Joined: Tue Jul 04, 2006 9:53 pm
Location: USA

Re: Another curl update script

Postby lukes » Mon Sep 22, 2008 4:10 pm

I've tried a lot of things, but I think I've got it now:
This didn't work:
Code: Select all
SET GZIP=%ProgramFiles%\gzip\gzip.exe
"%GZIP%" -c stuff > stuff.gz

But when you only save the path in the var, it works. Strange...
Code: Select all
SET GZIP=%ProgramFiles%\gzip
"%GZIP%\gzip" -c stuff > stuff.gz


Well, don't quite get why, but it works for me now. :)
I've edited the first post accordingly.

Jakob
Last edited by lukes on Mon Sep 22, 2008 7:22 pm, edited 1 time in total.
User avatar
lukes
WR.net Apprentice
WR.net Apprentice
 
Posts: 35
Joined: Tue Jan 22, 2008 4:46 pm


Return to Code Cache

Who is online

Users browsing this forum: No registered users and 0 guests

cron