Published: Sun 29 September 2013
By viraptor
In misc .
tags: backup security devops
Any long-term data archives you keep - backups, copies of logs, code
repositories, audit entries, etc. Are they append-only? I don’t mean
from the perspective of the account owner. Of course the operator is
able to do whatever he wants with those files, including deleting them.
But what about your servers? I’ve seen many backup manuals, or even
published scripts which pretty much require you to setup the S3 account
and that’s it. They’ll manage the files inside and make sure everything
is updated as needed. But there’s a problem with that strategy - what if
someone else gains control of your server? What can they do to your archives?
Can they, in one go, wipe both your current database and all the
available backups using credentials stored locally? Or all of the logs
from this and other services? Can they wipe the code repositories using
your deployment keys? Can they read all your database backups with every
past and present account still in there? Guessing from many tutorials
and articles, there are lots of production services out there where the
answer is “yes” to all. Sometimes it’s not even an issue with the admins
not trying hard enough - sometimes services themselves are really not
prepared for protection from that kind of behaviour. Many providers
simply don’t offer any other ways of accessing their accounts apart from
a single user+key combination giving global access to everything.
There are ways to provide some level of protection even without any
support from the service provider, but it’s much easier if the access
control is already built-in. For example when uploading data into S3,
everything that’s needed for good protection is already provided. IAM
gives you the possibility to create an account per service, or even per
host. User permissions allow you to give only as much access as needed
for your use case. That means your remote backup script shouldn’t need
any more privileges than PutObject on some specific object (in case
you’re using versioned buckets), or on a bucket (without versioning).
The second case requires that you also assign unpredictable names (for
example random suffixes) to the files, so that they cannot be destroyed
by overwriting.
Here’s an example of a user policy for S3 bucket without versioning:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:PutObject"
],
"Sid": "Stmt1375779806000",
"Resource": [
"arn:aws:s3:::blah-database-backups/*"
],
"Effect": "Allow"
}
]
}
But that’s not all. Even if you have a single account, there’s still a
way to limit the potential damage to some extent. Instead of using the
same account for upload and long-term storage of the files, get two
accounts. Upload to the first one and keep it around only for that
purpose. For long-term storage, set up a machine which is completely
separated from the users (maybe even rejects all incomming connections)
and make sure all files are moved to the second account as soon as possible.
That kind of setup is not perfect and still allows the attacker to
replace files that have not been moved, or download files which wouldn’t
normally be accessible from the local filesystem. But the time window
and amount of data that may be impacted is much lower. There’s also
still the possiblity to encrypt the data locally using a public key
(which can be safely stored on the server) in order to protect the information.
So, if I got access to one of your servers and its backup system… what
would I be able to achieve?