Trawling through Microsoft.TeamFoundation.Build I found the flag for IncrementalGet. Not getting full sources every time has cut 40 seconds out of every build. This is particularly useful when debugging a build script.
This is what I added to my root level PropertyGroup just before DropLocation:
<!-- INCREMENTAL GET
Setting this flag will set: