Do not use Duplicacy on macOS

Duplicacy is a backup tool. It may possibly have good uses, but if you are using it on a Mac it is probably not actually making backups.

The architecture of the application

The Duplicacy application¹ on the Mac presents itself as a little web server which you can then talk to (only via localhost, which is good) to configure, run and monitor backups.

What it does behind the scenes is more complicated. Other than some keychain entries (perhaps only one keychain entry) for a master password which is used to encrypt all the other sensitive data, all of its state lives in ~/.duplicacy-web. This includes all the configuration, logs and so on and, critically, an executable which is the actual program which runs backups, which lives in ~/.duplicacy-web/bin and has a name like duplicacy_osx_x64_2.6.1. The application simply invokes this program to run backups for it. The application will also update this executable when it notices a new one.

This itself is mildly terrifying: where did this executable come from? How safe is it? Can you be sure that the place it comes from will never be compromised? This executable is about to read all your files and copy them somewhere: you probably want to be a bit more sure about it than this.

(This is very different than the case of updating the application itself: this is, or should be, something done under human control. At least in principle you can, and should, check that the thing you have just downloaded actually is what it says it is, and if you don’t, well, that’s a risk you are conciously taking.)

It gets worse: the default configuration of the application will fetch the latest executable, not a stable one (however that is defined), thus maximising the chance that you will be running something that doesn’t work to do your backups, and also maximising the chance that you’ll get a compromised executable. If you are not frightened by now, you will be in a minute.

The annoyances of macOS

From, I think, 10.14, macOS has developed a complicated and annoying protection system which is completely orthogonal to file permissions. I do not understand this system at all, but it essentially involves various policies about what programs can read and write to what. The intention seems to be that, for instance, some application you install should not be able to read or write personally-sensitive data without your explicit permission, even if the filesystem or other permissions would allow it to do so.

‘Personally-sensitive data’ includes things like your email, your contacts, location information and so on. You can see these permissions in the ‘Privacy’ pane of the ‘Security & Privacy’ entry in ‘System Preferences’ and presumably there is some configuration file somewhere which backs all this, and the tccutil command can be useful as well. The protection system also controls various APIs, such as the one that provides location information.

Although this system is irritating in the usual Apple way, I think it’s well-motivated: my email contains personally-sensitive data about me if no-one else, and I definitely don’t want some random program I run snooping on it, or finding out where I am, without explicitly asking me first.

A place where this protection system really gets in the way is for backup tools. Backup tools really need to be able to, well, make backups, and the most important things they need to back up are often the most sensitive. I really want my backup program to be able to back up my email, for instance, as well as my calendar configuration and so on, and all the other stuff that the macOS protection mechanism would not normally let it read.

So, Apple have thought of this. If you trust some application you can grant it ‘full disk access’ which lets it read (and write, probably) the whole filesystem, only limited by filesystem permissions. This is exactly what you need for a backup program.

The first disaster

So, obviously, when you get Duplicacy, you anoint it suitably in the Privacy pane so that it can have full disk access. (It does not tell you to do this, which is a bad sign in itself.)

This doesn’t work. I think it doesn’t work because the program that is doing the backups is not the Duplicacy application, but this little executable which it downloaded. And, in fact, that’s a good thing: I would really rather not allow an application to secretly download some executable which can read (and write) all my files and send them who-knows-where. It may be that the reason it does not work is that the executable is not signed, although it does appear to be signed, so I am not sure.

In any case, what happens is that the executable fails to read sensitive data and thus fails to back it up. And it dutifully logs this, in ~/.duplicacy-web/logs/backup-*.log:

2020-08-21 15:27:40.769 WARN LIST_FAILURE Failed to list subdirectory: open /Users/tfb/Library/Application Support/com.apple.TCC: operation not permitted
2020-08-21 15:27:40.955 WARN LIST_FAILURE Failed to list subdirectory: open /Users/tfb/Library/Calendars: operation not permitted
[...]
2020-08-21 15:27:43.830 WARN LIST_FAILURE Failed to list subdirectory: open /Users/tfb/Library/Containers/com.apple.mail: operation not permitted
[...]
2020-08-21 16:26:53.142 WARN BACKUP_SKIPPED 23 directories and 20 files were not included due to access errors

In other words: the backup worked, partially, but it didn’t succeed in reading some of the the most critical data. If you need to restore from this backup, all your email will be gone.

Well, perhaps you could suitably anoint the downloaded executable? You could do that, if you could work out how to get the Finder to let you see directories whose names have leading .s, which is possible but fiddly. And it would work, for a while, until a new version with a new name appears, and then it will all break again and you’ll have to do it all again.

So that’s a disaster. But it’s not the most serious one.

The second disaster

So, you are configuring this thing via the web interface, like a good person. And you’ve thought to anoint the application so it can read everything, even though at no point did it tell you to do this (unlike other, competently-written, backup tools). And you run backups, and the executable dutifully logs that they failed. And there is no indication of this, at all in the web interface, which simply tells you that the backup completed, by which it apparently means ‘the program ran, and after a while it stopped running, and that means everything must be OK’.

In other words: if you are using a recent macOS, then Duplicacy is almost certainly not making good backups for you, and it is certainly not telling you about it when it does not.

Don’t use Duplicacy

I don’t understand how this happened other than that, very clearly, a lot of testing simply was never done. I do understand that it tells you something very, very bad about Duplicacy. I certainly would not, ever, use it on a Mac, and I find it so alarming that I would not in fact use it on any system at all.

Backup tools need to work, because when you need them you really need them. Duplicacy is backup theatre: something that looks like a backup tool but in fact is not.