Subversion fix-up script

This is for irrisor, or another developer who does Subversion admin on UNIX or Cygwin.  It fixes and optimizes several Subversion file properties in your 2.0 trunk baseline.  I assume you know how to set up a script file with the interpreter line intact, etc.  Save it anywhere, and run like "path/to/svncleanup.bash -h" to see syntax message.  The script won't commit anything, so after running it, verify your workspace mods with "svn status", then "svn commit".



2008-06-11  Very minor update to script below (2 character change).  Simplified svn:keyword settings due to a Subversion parsing bug being fixed recently.



#!/bin/bash -p
PROGNAME="${0##*/}"

shopt -s xpg_echo || {
    echo 'Your Bash version is ancient.  Please upgrade.'
    exit 5
}
set +u

TMPFILE=/tmp/${PROGNAME%.*}.$$

[ $# -gt 0 ] && case "$1" in -*)
    case "$1" in *o*) OVERRIDE=1;; esac
    case "$1" in *v*) VERBOSE=1;; esac
    case "$1" in *n*) NORUN=1;; esac
    case "$1" in *h*) JUSTHELP=1;; esac
    shift
esac

[ $# -ne 0 ] && JUSTHELP=1

[ -n "$JUSTHELP" ] && {
    echo "SYNTAX:  $PROGNAME [-ovnh]
    -o: Override startup validation that no work space mods are present.
    -v: Verbose
    -h: display this Help message
    -n: echo instead of execute subversion command which will effect workspace." 1>&2
    exit 3
}

Failout() {
    echo "Aborting $PROGNAME:  $*
You can easily back out any effects by running 'svn revert -R .'" 1>&2
    exit 1
}

[ -d src/jmetest/TutorialGuide/.svn ] ||
Failout "You must invoke this script from the root directory of a jme 2.0 trunk work area"

[ -d build/com ] &&
Failout "You must clean your work area before running this script.  Run 'ant clean'."

[ -n "$OVERRIDE" ] || {
    svn status | grep -v '^?' > /dev/null &&
    Failout 'Looks like you have uncommitted modifications to your work area.
You can run with the -o switch to override, but be aware that this script
will commit your work area.'
}

[ -d build/.svn ] ||
Failout "Check out directory 'build' from the repository.
A bug in the 'clean' target of build.xml file wipes out this directory."

[ -w /tmp ] ||
Failout "This script must be run by a user with write privileges on '/tmp'"

###########   1  FIX svn:executable   ###################
[ -n "$VERBOSE" ] && echo "svn propdel -qR svn:executable"
[ -n "$NORUN" ] || {
    svn propdel -qR svn:executable ||
    Failout "'svn propdel -R svn:executable' failed"
}
find . -type f '(' -name '*.so' -o -name '*.dll' -o -name '*.jnlp' ')' -print0 |
xargs -0 ${NORUN:+/bin/echo} svn propset -q svn:executable '*' ||
Failout "Command failed: xargs -0 ${NORUN:+/bin/echo} svn propset -q svn:executable '*'..."
[ -n "$VERBOSE" ] && echo "svn propset -q svn:executable '*' jnlp/jmedemo.php"
# Don't have time to check, but in case jmedemo.php may be executed directly.
[ -n "$NORUN" ] || {
    svn propset -q svn:executable '*' jnlp/jmedemo.php ||
    Failout "'svn propset -q svn:executable * jnlp/jmedemo.php' failed"
}
# Subversion is smart enough to automatically revert the propset + propdel


############  2  SET BEST mime-types  ##################
# These file types were mistakenly diagnosed as binary during import.
find . -type f '(' -name '*.frag' -o -name '*.dae' -o -name '*.mtl' -o -name '*.obj' -o -name '*.[vf]p' -o -name '*.jnlp' -o -name '*.php' ')' -print0 |
xargs -0 ${NORUN:+/bin/echo} svn propdel -q svn:mime-type ||
Failout "Command failed: xargs -0 ${NORUN:+/bin/echo} svn propdel -q svn:mime-type..."

# Would be more efficient to use xargs for all, but we could exceed the
# command line size max, due to the large number of target files.

function setTypeForPatterns() {
    local prop="$1"; shift
    [ -n "$prop" ] || Failout 'SYNTAX: setTypeForPattern prop val pattern1...'
    local val="$1"; shift
    [ -n "$val" ] || Failout 'SYNTAX: setTypeForPattern prop val pattern1...'
    [ $# -lt 1 ] && Failout 'SYNTAX: setTypeForPattern prop val pattern1...'
    local -a nameArgs
    for pattern in "$@"; do
        [ ${#nameArgs[@]} -gt 0 ] && nameArgs=("${nameArgs[@]}" -o)
        nameArgs=("${nameArgs[@]}" -name "$pattern")
    done
    find . -type f '(' "${nameArgs[@]}" ')' | grep -v '.svn/' > "$TMPFILE"
    [ -n "$VERBOSE" ] && echo "Setting property '$prop' to '$val'..."
    [ ! -s "$TMPFILE" ] && [ -n "$VERBOSE" ] &&
    echo "No file match for following patterns: ${@}"
    [ -s "$TMPFILE" ] && while read file; do
        ${NORUN:+/bin/echo} svn propset -q "$prop" "$val" "$file" ||
        Failout "Command failed: ${NORUN:+/bin/echo} svn propset -q '$prop' '$val' '$file'"
    done < "$TMPFILE"
}
# Clean up after ourselves
trap "rm -f $TMPFILE" EXIT

setTypeForPatterns svn:mime-type image/png '*.[Pp][Nn][Gg]'
setTypeForPatterns svn:mime-type image/jpeg '*.jpg'
setTypeForPatterns svn:mime-type image/gif '*.gif'
setTypeForPatterns svn:mime-type image/bmp '*.bmp'
setTypeForPatterns svn:mime-type image/x-icon '*.ico'
setTypeForPatterns svn:mime-type application/pdf '*.pdf'
setTypeForPatterns svn:mime-type application/zip '*.zip'
setTypeForPatterns svn:mime-type application/jar '*.jar'
setTypeForPatterns svn:mime-type application/x-sharedlib '*.so'
setTypeForPatterns svn:mime-type image/x-3ds '*.3[Dd][Ss]'
setTypeForPatterns svn:mime-type application/x-java '*.class'
setTypeForPatterns svn:mime-type image/x-tga '*.tga'
setTypeForPatterns svn:mime-type image/x-xcf '*.xcf'
setTypeForPatterns svn:mime-type image/x-compressed-xcf '*.xcf.gz'
setTypeForPatterns svn:mime-type image/x-compressed-xcf '*.xcf.bz2'
setTypeForPatterns svn:mime-type application/ogg '*.ogg'
setTypeForPatterns svn:mime-type audio/mpeg '*.mp3'
setTypeForPatterns svn:mime-type audio/x-wav '*.wav'

setTypeForPatterns svn:mime-type text/plain '*.txt'
setTypeForPatterns svn:mime-type text/html '*.html'
setTypeForPatterns svn:mime-type text/x-java '*.java'
setTypeForPatterns svn:mime-type text/css '*.css'
setTypeForPatterns svn:mime-type text/csrc '*.vert'
setTypeForPatterns svn:mime-type text/xml '*.xml'


############  3  SET portable EOL style for text files ##################
setTypeForPatterns svn:eol-style native 
    '*.txt' '*.html' '*.java' '*.ase' '*.cfg' .cvsignore '*.css' '*.dae' 
    '*.[vf]p' '*.frag' '*.jnlp' '*.mtl' '*.obj' '*.php' '*.properties' 
    '*.vert' '*.x3d' '*.xml'


############  4  SET RCS keywords to work just like CVS for text files #######
# .cvsignore is the only text file style where we won't expand RCS keywords
setTypeForPatterns svn:keywords 'Id Date Author URL Rev' 
    '*.txt' '*.html' '*.java' '*.ase' '*.cfg' '*.css' '*.dae' 
    '*.[vf]p' '*.frag' '*.jnlp' '*.mtl' '*.obj' '*.php' '*.properties' 
    '*.vert' '*.x3d' '*.xml'


[ -n "$VERBOSE" ] && echo "Completed successfully.  Run 'svn commit' now."

Thanks for the script. I deleted the mkdir part, as the TMPDIR variable is not assigned and not used. After that it works to some extend, but:



No idea why, but I get this error message:

svn: Can't move 'src/com/jmex/model/collada/schema/.svn/tmp/entries' to 'src/com/jmex/model/collada/schema/.svn/entries': Permission denied
Aborting jmesvncleanup.sh:  'svn propdel -R svn:executable' failed


I tried to do a fresh checkout, but that wouldn't help. Probably a cygwin problem - I'll try on Linux when I get home.
irrisor said:

Thanks for the script. I deleted the mkdir part, as the TMPDIR variable is not assigned and not used.


Oops.  TMPDIR became unnecessary when I refined the script to use one temp file instead of 2.  (Works for me regardless since I happen to have TMPDIR set).  I fixed it below.  Also just removed a debug echo line that I accidentally left in.

After that it works to some extend, but:

No idea why, but I get this error message:

svn: Can't move 'src/com/jmex/model/collada/schema/.svn/tmp/entries' to 'src/com/jmex/model/collada/schema/.svn/entries': Permission denied
Aborting jmesvncleanup.sh:  'svn propdel -R svn:executable' failed


I tried to do a fresh checkout, but that wouldn't help. Probably a cygwin problem - I'll try on Linux when I get home.


That looks like a typical Windows file locking problem (which can usually be worked around by rebooting), except that it happened again after a new checkout. 

After updating tortoise svn it worked well on windows o_O



But before committing it, I have a questions regarding the rcs keywords like $Id$ etc.: When showing differences between local and remote versions in other svn repos (where someone enabled keywords), I get an expanded keyword on the one side and a plain one on the other side. This results in patches which cause conflict, while there should not be any… so, are those svn repos misconfigured or is this a general issue with expanding keywords with svn? In the latter case I would rather switch off keywords.

irrisor said:

After updating tortoise svn it worked well on windows o_O

But before committing it, I have a questions regarding the rcs keywords like $Id$ etc.: When showing differences between local and remote versions in other svn repos (where someone enabled keywords), I get an expanded keyword on the one side and a plain one on the other side. This results in patches which cause conflict, while there should not be any... so, are those svn repos misconfigured or is this a general issue with expanding keywords with svn? In the latter case I would rather switch off keywords.


Sorry that I've been unavailable.  Was tied up for 2 solid days with hardware problems.

I can't really answer your question as given, because you can't use Subversion to compare artifacts from two different repositories.  Maybe you mean different projects in the same repository, or maybe you mean you are using some other tool to compare work areas which have been checked out of Subversion repositories.  The latter case would require scripting to zero (or equate) all keyword values from the branches being compared.

For comparing subdirectories within a single repository, Subversion automatically filters out keyword contents from differences whenever compared artifacts rrequire it (wherease CVS requires -kk switch for this purpose with some rdiff commands).  It's a matter of opinion whether to use RCS keywords, but if RCS keywords appear in code, then the keywords property should definitely be set.  As soon as that is done for your "other" branch, the false positives will no longer occur.  If you decide against using RCS keywords in Subversion, you'll need a script to remove all uses of keywords throughout your code (otherwise your baseline will be filled with incorrect revision/author/filepath/date information).

Uh, I didn't mean comparing different repositories. I meant comparing two revisions of the same file. I once had problems with this in another repository than jME.

But you still answered my actual question - I understood: keywords will not arise in diffs if configured correctly. That's good :slight_smile:



I'll go ahead and commit the changes on Monday, then.

irrisor said:

Uh, I didn't mean comparing different repositories. I meant comparing two revisions of the same file. I once had problems with this in another repository than jME.
But you still answered my actual question - I understood: keywords will not arise in diffs if configured correctly. That's good :)

I'll go ahead and commit the changes on Monday, then.


It's Friday now and nothing has been committed.  Did you forget?

It's thursday over here :wink: - but I didn't forget it. I was sick for a couple of days and try to catch up with my work. Thus I need to delay jME stuff a bit…

irrisor said:

It's thursday over here ;) - but I didn't forget it. I was sick for a couple of days and try to catch up with my work. Thus I need to delay jME stuff a bit...


Understood.

committed

irrisor said:

committed


No commit has been applied to the repository.  The repository is still at revision 3857, which Josh made yesterday.  Maybe run some local and remote log and status commands.

Maybe your work area is for some other branch?  On UNIX you can "grep http .svn/entries" to see the target URL.

it's actually 3858, but that's probably because I just checked in some particle updates.  Hope that didn't cause issues for your checkin.

renanse said:

it's actually 3858, but that's probably because I just checked in some particle updates.  Hope that didn't cause issues for your checkin.


You committed after I wrote that (7 minutes before you committed 5858).  It's impossible for your commit to have effected irrisor, since he attempted his commit well before you committed.

Uh, I didn't check the result. Because everything should have been fine, but:

Error: Commit would put repository over quota limit.


:-o

uh, what?

blaine said:

Seems counter-intuitive that it would need much disk space just to make property changes.  Make sure to "update" your work area before applying the commit, in case it makes server-side copies to resolve synchronization issues.

updating does not help :(
irrisor said:

blaine said:

Seems counter-intuitive that it would need much disk space just to make property changes.  Make sure to "update" your work area before applying the commit, in case it makes server-side copies to resolve synchronization issues.

updating does not help :(


Hey Irissor.  I see that the Subversion properties have still not been fixed.  Could you give this a try again?

Thanks for granting access.



FYI, I'm just waiting for final decision from the filename convention topics before fixing up all the properties, to have all of the fixes applied in single commit (to facilitate review, possible rollback, etc.)

It is taking hours to get these big deltas to commit.  The JME code base has a serious weight problem.  The quantity of meaningful code does not justify the time taken for checkouts and for maintenance commits like this.  IMO, JME should use Ivy or Maven and let the build system retrieve specified versions of third party products instead of maintaining everything in the world.



If my current commit times out again (after making continuous progress for 40 minutes), I'll have to revert to committing the changes in batches.



Waah.



POSTNOTE:  Prominently posting the Subversion config file and a request to conform to standard filename suffixes and capitalization would prevent the need to repeat this work.

Done :slight_smile:



I also fixed the incorrect encoding in 15 source files.



For the benefit of anybody who gets here from searching the forum for "encoding":



If you add any non-ASCII characters to *.java files, make sure that your editor is using UTF-8 encoding.

If you add any non-ASCII characters to *.properties files, make sure that your editor is using ISO-8859-1 encoding (unicode escape codes can be used for characters not handled by ISO-8859-1).