From: Nikodemus Siivola <nikodemus@sb-studio.net>
Date: Mon, 6 Jun 2011 06:48:08 +0000 (+0300)
Subject: A git-only SBCL workflow
X-Git-Url: http://repo.macrolet.net/gitweb/?a=commitdiff_plain;h=bf5163e4f07f0666b53b32b6232a4bd81d0d548e;p=sbcl.git

A git-only SBCL workflow

 This updates the SBCL build and release process to be more compatible
 with distributed development -- to facilitate moving the upstream
 repository into Git.

 A detailed description of what is going on here is in
 doc/GIT-WORKFLOW.md.

 Some highlights:

  * Drop version.lisp-expr and branch-version.lisp-expr.

  * Auto-generate the version at build time using information
    from Git, incorporating:
    - Last release number.
    - Number of commits on origin/master since last release.
    - Current branch, if there are commits not on origin/master.
    - Number of commits not on origin/master.
    - SHA1 id of the last commit.
    - Optional -dirty marker.

  * Update release.sh to work with Git.

  * Make source-distribution.sh exclude the .git directory from tarballs.

  * Release tags contain NEWS for that release.
---

diff --git a/.gitignore b/.gitignore
index 76cc67c..7688f20 100644
--- a/.gitignore
+++ b/.gitignore
@@ -44,3 +44,4 @@ contrib/*/a.exe
 contrib/asdf/asdf-upstream
 contrib/sb-cover/test-output
 doc/manual/*.html
+version.lisp-expr
diff --git a/doc/GIT-WORKFLOW.md b/doc/GIT-WORKFLOW.md
new file mode 100644
index 0000000..e00225d
--- /dev/null
+++ b/doc/GIT-WORKFLOW.md
@@ -0,0 +1,92 @@
+# Git workflow for SBCL
+
+## Version numbering
+
+Historically each SBCL commit incremented the number in
+version.lisp-expr, and prepended that version number to the first line
+of the commit message. For CVS this served us well, but since Git
+makes it easier for anyone to create branches that run in parallel to
+the current "master" timeline, it destroys the illusion of a single
+official timeline defined through version.lisp-expr.
+
+In Git workflow, version.lisp-expr no longer exists in the repository,
+nor is the version number prepended to commit messages.
+
+Instead, we construct a version number as follows when building SBCL
+or generating a source tarball:
+
+For branch master:
+
+  release.commits-on-master-since-release.sha1
+
+  Eg. 1.0.48.20-152c97d
+
+                          Last release: 1.0.48
+       Commits on master after release: 20
+          SHA1 abbrev for current HEAD: 152c97d
+
+  If there are no commits on master since the last release, both the
+  count and the SHA1 are dropped.
+
+For other branches:
+
+  release.commits-on-branch-and-master-since-release.branch.commits-on-branch.sha1
+
+  Eg. 1.0.44.26.wip-pretty-backtraces.4-674f875
+
+                          Last release: 1.0.44
+       Commits on master after release: 26
+                                Branch: wip-pretty-backtraces
+   Commits on branch but not on master: 4
+          SHA1 abbrev for current HEAD: 674f875
+
+In both cases -dirty is appended to the version number if the tree
+isn't clean when building.
+
+Anyone who publishes binaries built using an altered version, should
+do so on a branch named appropriately, so that the binaries identify
+themselves as 1.0.50.debian.2 or whatever. If they wish to use a
+source release instead working from Git, they should identify their
+changes with an appropriate edit to version.lisp-expr.
+
+To cater for those whose existing processes really don't like the
+SHA1s part in version numbers, setting NO_GIT_HASH_IN_VERSION=t in the
+environment for make.sh will make the version number generator leave
+out the hash.
+
+## Making a release (release.sh)
+
+Short story: use `release.sh`.
+
+`release.sh` makes a release *locally.* This means it will perform all
+actions required for the release in the local git checkout, and will
+then instruct you how to proceed in order to push the release to the
+outside world.
+
+###Synopsis:
+
+    ./release.sh VERSION [-s]
+
+**VERSION** is the version to make a release for. Example: `1.0.46`.
+
+**-s** instructs `git tag` to create a gpg-signed tag for this
+release. Highly recommended.
+
+###Description:
+
+`release.sh` will perform these actions:
+
+* Check that the local checkout is clean.
+* Update NEWS and make a commit stating the release version number
+* Make an sbcl.<VERSION> tag and optionally sign it.
+* Build SBCL
+* Run tests
+* Build SBCL with the SBCL that just had tests pass
+* Build docs
+* Create source, binary, documentation tarballs
+* Sign these tarballs
+* Give you further instructions.
+
+After release.sh is done, you can inspect the results, and commence
+struggling with the SF.net file release system from hell. You are very
+brave.
diff --git a/doc/PACKAGING-SBCL.txt b/doc/PACKAGING-SBCL.txt
index 33abe83..3115549 100644
--- a/doc/PACKAGING-SBCL.txt
+++ b/doc/PACKAGING-SBCL.txt
@@ -1,19 +1,24 @@
 Packaging SBCL
 ==============
 
-If you package SBCL for a distribution, please edit version.lisp-expr,
-and append ".packaging-target-or-patch[.version]".
+If you package SBCL for distribution, we ask that you to take steps to
+make the version number reflect this. Our users often report bugs that
+are intimately tied up with configuration issues, and much confusion
+can result from mistaking a packaged SBCL for the upstream one.
+
+If you are working from a Git branch, all you need to do is make sure
+the branch name reflects the situation -- the build system will
+incorporate the it in the version string.
+
+If you are working from a release tarball, please edit
+version.lisp-expr, and append ".packaging-target-or-patch[.version]".
 
 Examples:
 
- "1.0.7.gentoo"
- "1.0.7.mikes-rpms.2"
+ "1.0.50.gentoo"
+ "1.0.50.mikes-rpms.2"
 
 This will make the startup banner, --version, and
 (lisp-implementation-version) all identify the packaged version
 correctly.
 
-We ask you to do this because users report bugs that are intimately
-tied up with configuration issues at regular intervals, and much
-confusion can result from mistaking a packaged SBCL for the upstream
-one.
diff --git a/generate-version.sh b/generate-version.sh
new file mode 100755
index 0000000..bfd7f1c
--- /dev/null
+++ b/generate-version.sh
@@ -0,0 +1,72 @@
+#!/bin/sh
+# Not a shell script, but something intended to be sourced from shell scripts
+git_available_p() {
+    # Check that (1) we have git (2) this is a git tree.
+    (which git >/dev/null 2>/dev/null && git describe >/dev/null 2>/dev/null)
+}
+
+generate_version() {
+    if ([ -f version.lisp-expr ] && ! git_available_p)
+    then
+        # Relase tarball, leave version.lisp-expr alone.
+        return
+    elif ! git_available_p
+    then
+        echo "Can't run 'git describe' and version.lisp-expr is missing." >&2
+        echo "To fix this, either install git or create a fake version.lisp-expr file." >&2
+        echo "You can create a fake version.lisp-expr file like this:" >&2
+        echo "    \$ echo '\"1.0.99.999\"' > version.lisp-expr" >&2
+        exit 1
+    fi
+    # Build it.
+    version_head=$(git rev-parse HEAD)
+    if [ -z "$SBCL_BUILDING_RELEASE_FROM" ]
+    then
+        version_root="origin/master"
+    else
+        version_root="$SBCL_BUILDING_RELEASE_FROM"
+    fi
+    version_base=$(git rev-parse "$version_root")
+    version_tag=`git describe --tags --match="sbcl*" --abbrev=0 $version_base`
+    version_release=`echo $version_tag | sed -e 's/sbcl[_-]//' | sed -e 's/_/\./g'`
+    version_n_root=`git rev-list $version_base --not $version_tag --count`
+    version_n_branch=`git rev-list HEAD --not $version_base --count`
+    if [ -z "$NO_GIT_HASH_IN_VERSION" ]
+    then
+        version_hash="-`git rev-parse --short $version_head`"
+    else
+        version_hash=""
+    fi
+    if git diff HEAD --no-ext-diff --quiet --exit-code
+    then
+        version_dirty=""
+    else
+        version_dirty="-dirty"
+    fi
+    # Now that we have all the pieces, put them together.
+    cat >version.lisp-expr <<EOF
+;;; This file is auto-generated using generate-version.sh. Every time
+;;; you re-run make.sh, this file will be overwritten if you are
+;;; working from a Git checkout.
+EOF
+    if [ "$version_base" = "$version_head" ]
+    then
+        if [ "0" = "$version_n_root" ]
+        then
+            printf "\"%s%s\"\n" \
+                $version_release $version_dirty >>version.lisp-expr
+        else
+            printf "\"%s.%s%s%s\"\n" \
+                $version_release $version_n_root \
+                $version_hash $version_dirty >>version.lisp-expr
+        fi
+    else
+        echo "base=$version_base"
+        echo "head=$version_head"
+        version_branchname=`git describe --contains --all HEAD`
+        printf "\"%s.%s.%s.%s%s%s\"\n" \
+            $version_release $version_n_root \
+            $version_branchname $version_n_branch \
+            $version_hash $version_dirty >>version.lisp-expr
+    fi
+}
diff --git a/make.sh b/make.sh
index 39cf802..e054c72 100755
--- a/make.sh
+++ b/make.sh
@@ -192,6 +192,9 @@ export DEVNULL
 . ./find-gnumake.sh
 find_gnumake
 
+. ./generate-version.sh
+generate_version
+
 # If you're cross-compiling, you should probably just walk through the
 # make-config.sh script by hand doing the right thing on both the host
 # and target machines.
diff --git a/release.sh b/release.sh
index 6a56997..8387759 100755
--- a/release.sh
+++ b/release.sh
@@ -1,71 +1,180 @@
 #! /bin/sh
 
-set -ex
+set -e
+
+usage () {
+    if ! [ -z "$1" ]
+    then
+        echo $1
+    fi
+    cat <<EOF
+
+usage: $0 [-s] VERSION-NUMBER [REV]
+
+  This script frobs NEWS, makes a "release" commit, builds, runs tests
+  and creates an annotated tag for the release with VERSION-NUMBER.
+
+  If -s is given, then use the gpg-sign mechanism of "git tag". You
+  will need to have your gpg secret key handy.
+
+  if REV is given, it is the git revision to build into a release.
+  Default is origin/master.
+
+  No changes will be pushed upstream. This script will tell you how to
+  do this when it finishes.
+
+EOF
+    exit 1
+}
+
+if [ "-s" = "$1" ] || [ "--sign" = "$1" ]
+then
+    sign="-s"; shift
+else
+    sign=""
+fi
+
+if [ -z "$1" ]
+then
+    usage "No version number."
+else
+    version=$1; shift
+fi
+
+if [ -z "$1" ]
+then
+    rev=origin/master
+else
+    rev=$1; shift
+    type=$(git cat-file -t "$rev" 2> /dev/null || echo "unknown")
+    if ([ "tag" != "$type" ] && [ "commit" != "$type" ])
+    then
+        usage "$rev is $type, not a tag or a commit."
+    fi
+fi
+
+if ! [ -z "$@" ]
+then
+    usage "Extra command-line arguments: $@"
+fi
+
+sbcl_directory="$(cd "$(dirname $0)"; pwd)"
+tmpfile=$(mktemp -t sbcl-build-$(date +%Y%m%d)-XXXXXXXXX)
+tmpdir="$(mktemp -d -t sbcl-build-tree-$(date +%Y%m%d)-XXXXXXXXX)"
+
+## Check for messy work dirs:
+
+echo "Fetching updates."
+git fetch
+
+branch_name="release-$(date '+%s')"
+original_branch="$(git describe --all --contains HEAD)"
+trap "cd \"$sbcl_directory\" ; git checkout $original_branch" EXIT
+git checkout -b $branch_name $rev
+
+echo "Checking that the tree is clean."
+if ! [ $(git status --porcelain | wc -l) = 0 ]
+then
+    echo "There are uncommitted / unpushed changes in this checkout!"
+    git status
+    exit 1
+fi
+
+## Perform the necessary changes to the NEWS file:
+
+echo "Munging NEWS"
+sed -i.orig "/^changes relative to sbcl-.*:/ s/changes/changes in sbcl-$version/ " NEWS
+rm -f NEWS.orig
+if ! grep "^changes in sbcl-$version relative to" NEWS > /dev/null
+then
+    echo "NEWS munging failed!"
+    exit 1
+fi
 
-cd "$1"
+cd "$sbcl_directory"
 
-sbcl_directory="$(pwd)"
+echo "Committing release version."
+git add NEWS
+git commit -m "$version: will be tagged as \"sbcl-$version\""
 
-cd "$sbcl_directory"
+relnotes=$tmpdir/sbcl-$version-release-notes.txt
+awk "BEGIN { state = 0 }
+ /^changes in sbcl-/ { state = 0 }
+ /^changes in sbcl-$version/ { state = 1 }
+ { if(state == 1) print \$0 }" < NEWS > $relnotes
 
-tmpfile=$(mktemp --tmpdir sbcl-build-$(date +%Y%m%d)-XXXXXXXXX)
+tag="sbcl-$version"
+echo "Tagging as $tag"
+git tag $sign -F $relnotes "$tag"
 
+SBCL_BUILDING_RELEASE_FROM=HEAD
+export SBCL_BUILDING_RELEASE_FROM
+echo "Building SBCL, log: $tmpfile"
 ./make.sh >$tmpfile 2>&1
 
-./src/runtime/sbcl --version | grep '^SBCL [1-9][0-9]*\.[0-9]\+\.[1-9][0-9]*$'
+if [ "SBCL $version" != "$(./src/runtime/sbcl --version)" ]
+then
+    echo "Built version number doesn't match requested one:" &>2
+    echo &>2
+    echo "    $(./src/runtime/sbcl --version)" &>2
+    exit 1
+fi
 
-version=$(./src/runtime/sbcl --version | awk '{print $2}')
-grep "^changes in sbcl-$version relative to" NEWS
+built_version=$(./src/runtime/sbcl --version | awk '{print $2}')
 
+echo "Running tests, log: $tmpfile"
 cd tests
 sh ./run-tests.sh >>$tmpfile 2>&1
 cd ..
 
-cp ./src/runtime/sbcl /tmp/sbcl-$version
-cp ./output/sbcl.core /tmp/sbcl-$version.core
+cp ./src/runtime/sbcl "$tmpdir"/sbcl-$version-bin
+cp ./output/sbcl.core "$tmpdir"/sbcl-$version.core
+
+echo "Self-building, log: $tmpfile"
+./make.sh --xc-host="$tmpdir/sbcl-$version-bin --core $tmpdir/sbcl-$version.core --no-userinit --no-sysinit --disable-debugger" >>$tmpfile 2>&1
 
-./make.sh "/tmp/sbcl-$version --core /tmp/sbcl-$version.core" > /tmp/sbcl-$version-build-log 2>&1
-cd doc && sh ./make-doc.sh
+echo "Building docs, log: $tmpfile"
+cd doc && sh ./make-doc.sh >$tmpfile 2>&1
 
 cd ..
 
-rm /tmp/sbcl-$version /tmp/sbcl-$version.core
+rm -f "$tmpdir"/sbcl-$version-bin "$tmpdir"/sbcl-$version.core
 
-cp -a "$sbcl_directory" /tmp/sbcl-$version
+cp -a "$sbcl_directory" "$tmpdir"/sbcl-$version
 
-ln -s /tmp/sbcl-$version /tmp/sbcl-$version-x86-linux
-cd /tmp/
-sh sbcl-$version/binary-distribution.sh sbcl-$version-x86-linux
-sh sbcl-$version/html-distribution.sh sbcl-$version
+echo "Building tarballs, log $tmpfile"
+ln -s "$tmpdir"/sbcl-$version "$tmpdir"/sbcl-$version-x86-linux
+cd "$tmpdir"/
+sh sbcl-$version/binary-distribution.sh sbcl-$version-x86-linux >$tmpfile 2>&1
+sh sbcl-$version/html-distribution.sh sbcl-$version >$tmpfile 2>&1
 cd sbcl-$version
-sh ./distclean.sh
+sh ./distclean.sh >$tmpfile 2>&1
 cd ..
-sh sbcl-$version/source-distribution.sh sbcl-$version
-
-awk "BEGIN { state = 0 }
- /^changes in sbcl-/ { state = 0 } 
- /^changes in sbcl-$version/ { state = 1 }
- { if(state == 1) print \$0 }" < sbcl-$version/NEWS > sbcl-$version-release-notes.txt
+sh sbcl-$version/source-distribution.sh sbcl-$version >$tmpfile 2>&1
 
 echo "The SHA256 checksums of the following distribution files are:" > sbcl-$version-crhodes
 echo >> sbcl-$version-crhodes
 sha256sum sbcl-$version*.tar >> sbcl-$version-crhodes
-bzip2 /tmp/sbcl-$version*.tar
+bzip2 "$tmpdir"/sbcl-$version*.tar
 
-echo Bugs fixed by sbcl-$version release > sbcl-$version-bugmail.txt
-for bugnum in $(egrep -o "#[1-9][0-9][0-9][0-9][0-9][0-9]+" sbcl-$version-release-notes.txt | sed s/#// | sort -n)
+echo "Building bugmail."
+bugmail=sbcl-$version-bugmail.txt
+echo Bugs fixed by sbcl-$version release > $bugmail
+for bugnum in $(egrep -o "#[1-9][0-9][0-9][0-9][0-9][0-9]+" $relnotes | sed s/#// | sort -n)
 do 
-  printf "\n bug %s\n status fixreleased" $bugnum >> sbcl-$version-bugmail.txt
+  printf "\n bug %s\n status fixreleased" $bugnum >> $bugmail
 done
-echo >> sbcl-$version-bugmail.txt
-
-set +x
+echo >> $bugmail
 
+echo SBCL distribution has been prepared in "$tmpdir"
 echo TODO:
 echo
-echo cvs commit -m "\"$version: will be tagged as sbcl_$(echo $version | sed 's/\./_/g')\""
-echo cvs tag sbcl_$(echo $version | sed 's/\./_/g')
-echo gpg -sta /tmp/sbcl-$version-crhodes
+echo "Sanity check: git show $tag"
+echo
+echo "git merge $branch_name && git push && git push --tags"
+echo "git branch -d $branch_name"
+echo "cd \"$tmpdir\""
+echo gpg -sta sbcl-$version-crhodes
 echo sftp crhodes,sbcl@frs.sourceforge.net
 echo \* cd /home/frs/project/s/sb/sbcl/sbcl
 echo \* mkdir $version
@@ -89,3 +198,4 @@ echo \* check and send sbcl-$version-bugmail.txt to edit@bugs.launchpad.net
 echo \ \ '(sign: C-c RET s p)'
 echo \* update \#lisp IRC topic
 echo \* update sbcl website
+
diff --git a/source-distribution.sh b/source-distribution.sh
index 7a2505b..4b77e3a 100755
--- a/source-distribution.sh
+++ b/source-distribution.sh
@@ -4,4 +4,4 @@ set -e
 # Create a source distribution. (You should run clean.sh first.)
 
 b=${1:?"missing base directory name argument"}
-tar cf $b-source.tar $b
+tar cf $b-source.tar --exclude .git $b
diff --git a/src/code/misc.lisp b/src/code/misc.lisp
index 59f5f56..8a32863 100644
--- a/src/code/misc.lisp
+++ b/src/code/misc.lisp
@@ -16,8 +16,4 @@
   "SBCL")
 
 (defun sb!xc:lisp-implementation-version ()
-  #.(format nil "~A~@[.~A~]"
-            (sb-cold:read-from-file "version.lisp-expr")
-            (let ((pathname "branch-version.lisp-expr"))
-              (when (probe-file pathname)
-                (sb-cold:read-from-file pathname)))))
+  #.(sb-cold:read-from-file "version.lisp-expr"))
diff --git a/version.lisp-expr b/version.lisp-expr
deleted file mode 100644
index 674f732..0000000
--- a/version.lisp-expr
+++ /dev/null
@@ -1,23 +0,0 @@
-;;; This is the master value for LISP-IMPLEMENTATION-VERSION. It's
-;;; separated into its own file here so that it's easy for
-;;; text-munging make-ish or cvs-ish scripts to find and tweak it. For
-;;; the convenience of such scripts, only a simple subset of Lisp
-;;; reader syntax should be used here: semicolon-delimited comments,
-;;; possible blank lines or other whitespace, and a single
-;;; double-quoted string value alone on its own line.
-;;;
-;;; Local additions to the version number can be effected via
-;;; branch-version.lisp-expr, which is not stored in the CVS.
-;;;
-;;; ANSI says LISP-IMPLEMENTATION-VERSION can be NIL "if no
-;;; appropriate and relevant result can be produced", but as long as
-;;; we control the build, we can always assign an appropriate and
-;;; relevant result, so this must be a string, not NIL.
-;;;
-;;; Conventionally a string like "0.6.6", with three numeric fields,
-;;; is used for released versions, and a string like "0.6.5.xyzzy",
-;;; with something arbitrary in the fourth field, is used for CVS
-;;; checkins which aren't released. (And occasionally for internal
-;;; versions, especially for internal versions off the main CVS
-;;; branch, it gets hairier, e.g. "0.pre7.14.flaky4.13".)
-"1.0.49"