Preparing an upgrade ==================== :Author: Edward Z. Yang .. highlight:: sh Wizard is designed to make pushing upgrades as painless as possible. In the best case scenario, adding a new version to a Wizard repository is as simple as running a few commands. Even when upstream makes backwards incompatible changes, or some of your patches conflict with other changes, Wizard aims to make resolving these changes tractable. Summary ------- ``$VERSION`` is a version number i.e. ``1.2.4``, ``$APPLICATION`` is the official application name i.e. ``MediaWiki``, and ``$APP`` is our internal name i.e. ``mediawiki``. The key thing to note is that we use ``wizard prepare-pristine`` in order to simulate the upstream upgrade, and then we piggy back off of Git's merge machinery to do everything else. First you prepare the pristine copy:: git checkout pristine wizard prepare-pristine $APP-$VERSION git commit -sm "$APPLICATION $VERSION" git tag $APP-$VERSION Next, you merge those changes to the scriptsified ``master`` copy:: git checkout master git merge pristine --no-commit # resolve conflicts git commit -sm "$APPLICATION $VERSION-scripts" .. note:: If you are creating a fix for a previous scripts version, you should bump the version to ``$VERSION-scripts2``. If the files associated with installation (for example, the install script) have changed, we need to double check that the configuration files are in order. On a scripts server with Wizard pointed at the latest version, run:: cd tests env WIZARD_NO_COMMIT=1 ./$APP-install-test.sh cd testdir_$APP_install_head wizard prepare-config With any luck, there will be no differences. However, if there are changes, manually restore any custom changes we may have made to the configuration file (a ``git checkout -p`` should allow you to selectively back out the relevant bits). Furthermore, make sure that no upstream changes broke our regular expressions for matching. The merge your changes back (preferably via your local machine, since you probably don't have appropriate author credentials setup to be accessible on Scripts):: git commit --amend -a git tag $APP-$VERSION-scripts git push --force git push --force --tags .. note:: If you have a split AFS and local Wizard setup, you may prefer to instead create a dummy commit and do the merging in your local copy. The flow looks like:: # from the scripts copy git commit -asm "Dummy" git push # from your copy git pull git reset HEAD~ git commit --amend -a git tag $APP-$VERSION-scripts git push -f Be sure to verify that your commit is the correct one; you can check with ``git show``, which should show the changes you made when amending the commit. Be especially careful to make sure you don't nuke any in configuration scripts changes. Implementation -------------- If it is the first time you have pushed an upgrade for an application, you will have to write a few more methods in your :class:`wizard.app.Application` class: :meth:`~wizard.app.Application.upgrade`, :meth:`~wizard.app.Application.checkWeb`, :meth:`~wizard.app.Application.backup` and :meth:`~wizard.app.Application.restore`. The latter three may not seem so useful for just pushing an upgrade, but are helpful for integrity checking installations post-upgrade, and rolling back if something went wrong. :meth:`~wizard.app.Application.checkWeb` is a method that should check whether or not an application is running properly. We use this to prevent us from trying to upgrade an install that is not publically accessible, or was broken from the very start, and we use it to automatically determine if our upgrade was successful or not. A common and easy way to perform this check is to use the :meth:`~wizard.app.Application.checkWebPage` method, which, along with the parameters :meth:`~wizard.app.Application.checkWeb` accepts, accepts two more: ``page`` and ``output``, which correspond to the page to grab from the web and the output string to match for in this page, .. warning:: We still haven't quite figured out a good combination of in-depth error checking and robustness against skin changes. This section should be further developed with tips about this. :meth:`~wizard.app.Application.backup` and :meth:`~wizard.app.Application.restore` perform backup and restoration of non-filesystem contents; the most common application is for the database. :func:`wizard.app.backup_database` and :func:`wizard.app.restore_database` implement this common functionality, and for most application implementing these methods is as simple as: .. code-block:: python def backup(self, deployment, backup_dir, options): app.backup_database(backup_dir, deployment) def restore(self, deployment, backup_dir, options): app.restore_database(backup_dir, deployment) Finally, :meth:`~wizard.app.Application.upgrade` actually performs an upgrade, and will most frequently call a shell script or fetches a web page that will perform a schema upgrade. .. note:: When migrating an old-style autoinstall, it is neither expected nor required for upgrade scripts for the intervening versions to be created. Troubleshooting --------------- Merge failed '''''''''''' When a merge fails, it's often good to refresh your memory about what particular patch was made to that file. You can find out with:: git diff :1:$FILE :2:$FILE If you are performing a repository conversion, a failed merge likely means that there is an updated patch lying around in :file:`/mit/scripts/deploy/$APP-$VERSION`. You can then revert the files to the pristine version:: git checkout --theirs $FILE And then apply the patch. If the patch is complicated, you may get warnings about hunks already being applied; you can ignore those warnings (don't assume ``-R``!) Going retro ----------- Under certain circumstances, you may need to splice in older versions of the application into your history. Do not rebase: you should never rebase published history. Instead, use the following procedure: Identify the version that, with regards to version numbering, directly precedes the version you'd like to add. For example, if you have the following commit tree: .. digraph:: original_dag node [shape=square] subgraph cluster_master { bs -> as as [label="1.0-scripts"] bs [label="2.0-scripts"] label = "master" color = white } subgraph cluster_pristine { b -> a a [label="1.0"] b [label="2.0"] label = "pristine" color = white } bs -> b as -> a And you are adding the 1.1 version, 1.0 and 1.0-scripts are the tags immediately preceding this version. Create temporary branches (we'll name them ``tmaster`` and ``tpristine``) pointing to these tags:: git checkout -b tmaster git reset --hard 1.0-scripts git checkout -b tpristine git reset --hard 1.0 Find the committer date associated with the ``tmaster`` commit using ``git show tmaster`` and note it somewhere:: DATE=`git show tmaster --pretty="format:%cd" | head -n1` Next, begin performing ordinary procedure for preparing the pristine copy. There are two caveats: you will need to use ``--force`` to make ``wizard prepare-pristine`` not complain about not being on the ``pristine`` branch, and you will need to falsify the author and committer time on the commits to be the times we noted down previously:: # on the tpristine branch wizard prepare-pristine $APP-$VERSION --force env GIT_AUTHOR_DATE="$DATE" GIT_COMMITTER_DATE="$DATE" git commit -asm "$APPLICATION $VERSION" git tag $APP-$VERSION .. note:: The date falsification is necessary to make Git prefer the later version tag when a commit has this (newer) commit and the most up-to-date version as merge parents. By default Git prefers the temporally closest commit. Next, merge the changes to the scriptsified ``tmaster`` (not ``master``!) copy, and falsify the dates as well:: git checkout tmaster git merge tpristine --no-commit # resolve conflicts env GIT_AUTHOR_DATE="$DATE" GIT_COMMITTER_DATE="$DATE" git commit -asm "$APPLICATION $VERSION-scripts" git tag $APP-$VERSION-scripts Note that we are creating a tag, because otherwise there is not an easy way to refer to this non-HEAD tag. On a scripts server with Wizard pointed at the latest version, run:: cd tests env WIZARD_NO_COMMIT=1 ./$APP-install-test.sh $VERSION cd testdir_install_$APP_$VERSION wizard prepare-config Note that ``$VERSION`` is specified explicitly. If there are changes, manually restore any custom changes we may have made, then amend your commit and push back:: # you probably lost your environment variable DATE=`git show HEAD --pretty="format:%ad" | head -n1` env GIT_AUTHOR_DATE="$DATE" GIT_COMMITTER_DATE="$DATE" git commit --amend -a git tag -d $APP-$VERSION-scripts git tag $APP-$VERSION-scripts git push --force --tags And on your now invalid version, grab the new version:: git fetch --tags --force $REMOTE Note that there is no need to reset the master branch, which hasn't changed.