MSBuild and including extra files from multiple builds

Note I’ve edited this blog post, as the original version had whitespace in the DestinationRelativePath element, which doesn’t work – see http://stackoverflow.com/questions/8218374/msbuild-build-package-including-extra-files/ for more detail

I was involved in changing an existing web application to be packaged by MSDeploy recently.

The package had to include files from external directories, as there are images, CSS and Javascript files that come outside the web application project. I needed to work out how to do this with MSDeploy.

My starting point was Sayed’s excellent article on extending CopyAllFilesToSingleFolder. The rest of this article assume you’ve already read his article. My other major reference was MSBuild: By Example, particularly Understanding the Difference Between @ and %.

Sayed’s article didn’t give me two things I needed.

  • The ability to specify multiple sources, each with different target subdirectories.
  • The ability to check that the file doesn’t already exist in the target directory.

Working out just how to make multiple sources with different target subdirectories possible took quite some investigation, and trial and error with $, @ and %. I ended up with the following approach.

I have a Common.Targets target file that defines a number of useful shared targets, that I import into my projects as required.

Within the projects I need to copy custom files, I add the following two pieces of XML after the import of my Common.Targets file.

Firstly, I extend the CopyAllFilesToSingleFolderForPackageDependsOn PropertyGroup in the following way:

<PropertyGroup>
  <CopyAllFilesToSingleFolderForPackageDependsOn>
    DefineCustomFiles;
    CustomCollectFiles;
    $(CopyAllFilesToSingleFolderForPackageDependsOn);
  </CopyAllFilesToSingleFolderForPackageDependsOn>
</PropertyGroup>

Edit: CopyAllFilesToSingleFolderForPackageDependsOn has been renamed to CopyAllFilesToSingleFolderForMsdeployDependsOn in Visual Studio 2012. Thanks to Scott Stafford for pointing this out in his comment on the post.

This is very similar what Sayed does, except I have two targets defined in here. DefineCustomFiles creates an ItemGroup containing the files to be copied, and is defined in each project. An example looks like this.

<Target Name="DefineCustomFiles">
  <ItemGroup>
    <CustomFilesToInclude Include="$(IncludeRootDir)\images\**\*.*">
      <Dir>images</Dir>
    </CustomFilesToInclude>
    <CustomFilesToInclude Include="$(IncludeRootDir)\css\**\*.css">
      <Dir>css</Dir>
    </CustomFilesToInclude>
    <CustomFilesToInclude Include="$(IncludeRootDir)\includes\**\*.js">
      <Dir>includes</Dir>
    </CustomFilesToInclude>
  </ItemGroup>
</Target>

This defines an ItemGroup CustomFilesToInclude, that includes the files in each of the given directories, with each file having the metadata Dir set as shown.

CustomCollectFiles is defined in Common.Targets. It uses the CustomFilesToInclude ItemGroup defined in DefineCustomFiles to define FilesForPackagingFromProject, as Sayed’s example shows.

<Target Name="CustomCollectFiles">
  <ItemGroup>
    <FilesForPackagingFromProject Include="@(CustomFilesToInclude)">
      <DestinationRelativePath>%(CustomFilesToInclude.Dir)\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
    </FilesForPackagingFromProject>
  </ItemGroup>
</Target>

This looks very simple, but the combination of @ and % syntax was the hard part of the exercise. I couldn’t use <FilesForPackagingFromProject Include="%(CustomFiles.Identity)"> as per Sayed’s example; using the Identity metadata prevents access to the Dir metadata previously defined to specify the destination. I ended up needing to use Include="@(CustomFilesToInclude)" so I could access the metadata. It took some more trial and error to find the correct syntax to reference the metadata of each of the items of CustomFilesToInclude, using the % syntax as shown.

I then extended the CustomCollectFiles task to check for files that already existed in the target directory.

<Target Name="CustomCollectFiles">
  <ItemGroup>
    <FilesForPackagingFromProject Include="@(CustomFilesToInclude)">
      <DestinationRelativePath>%(CustomFilesToInclude.Dir)\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
    </FilesForPackagingFromProject>
    <FilesForPackagingFromProject Include="@(CustomFilesToIncludeSkipExistingCheck)">
      <DestinationRelativePath>%(CustomFilesToIncludeSkipExistingCheck.Dir)\%(RecursiveDir)%(Filename)%(Extension)</DestinationRelativePath>
    </FilesForPackagingFromProject>
  </ItemGroup>
  <Error Text="Custom file exists in project files already: %(CustomFilesToInclude.FullPath)"
    Condition="Exists('$(MainProjectRootDir)\%(CustomFilesToInclude.Dir)\%(RecursiveDir)%(Filename)%(Extension)')" />
</Target>

You can see how the syntax I use in the Condition of the Error is the same as the syntax used in the DestinationRelativePath relative path above.

I toyed with the idea of adding another piece of metadata to the items within CustomFilesToInclude to indicate whether the Exists check is applicable for that item or not. But after a little experimentation I decided it was simpler to use two item groups. Any items that do not require the check go into CustomFilesToIncludeSkipExistingCheck.

So at the end of this journey I have learnt a few things: Sayed is always your first resource to search if you have MSBuild questions; the MSDeploy pipeline is extensible in a useful fashion; MSBuild can be devilishly confusing and take a fair amount of trial and error for those who don’t intimately understand it, especially when you try and do anything complex with groups of files.

To use MSDeploy’s extensibility, you need to use MSBuild. However, when not using MSDeploy, I’d like to avoid MSBuild. So the next time I start a project that promises to have any complexity, I’ll be looking for a build framework that makes it easy to leave complex behaviour outside of MSBuild. I investigated psake after seeing that Rhino.Mocks uses it, and liked what I saw. I also like that you don’t have to learn a specific “make” language – psake’s decision to leverage an existing scripting language is smart and practical.