Home › Blogs › Uwe Hermann's blog

Benchmarking an encrypted dm-crypt/LVM/ext3/SELinux hard drive with bonnie++ and hdparm

Fri, 2006-09-15 23:45 — Uwe Hermann

I'm going to set up a new laptop system soonish (more on that later) which shall have a completely encrypted hard drive. Hence, I'm testing a few setups wrt security, performance, manageability and fault-tolerance.

Here's a few performance tests I did on an 80 GB laptop hard drive (in an Intel Celeron based laptop, 1.7 GHz, 256 MB RAM, Linux 2.6.17, Debian unstable).
I ran bonnie++ (with no options) and hdparm as hdparm -tT /dev/hda each time. I haven't put too much thought into the test setup, so if I made some stupid mistakes, please let me know.

Unencrypted plain ext3 partitions:

  • Extra partitions for /, /boot, /usr, /var, /tmp, /home, and swap (no LVM).
  • Optionally, SELinux enabled on that system (targeted policy in permissive mode).

bonnie++:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
forest         432M 19857  84 21831  10  9536   4 16355  58 22165   3 148.8   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1650  98 +++++ +++ +++++ +++  1734  98 +++++ +++  3820  96
forest,432M,19857,84,21831,10,9536,4,16355,58,22165,3,148.8,0,16,1650,98,+++++,
+++,+++++,+++,1734,98,+++++,+++,3820,96

bonnie++ with SELinux:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
forest         432M 20321  90 21036  13  9473   5 16742  61 21978   4 148.1   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1398  98 +++++ +++ +++++ +++  1473  98 +++++ +++  3305  98
forest,432M,20321,90,21036,13,9473,5,16742,61,21978,4,148.1,0,16,1398,98,+++++,
+++,+++++,+++,1473,98,+++++,+++,3305,98

hdparm:

 Timing cached reads:   1416 MB in  2.00 seconds = 707.48 MB/sec
 Timing buffered disk reads:   82 MB in  3.06 seconds =  26.80 MB/sec

hdparm with SELinux:

 Timing cached reads:   1404 MB in  2.00 seconds = 700.59 MB/sec
 Timing buffered disk reads:   80 MB in  3.02 seconds =  26.53 MB/sec

Ext3 partitions on top of LVM on top of dm-crypt:

  • One partition which is encrypted using dm-crypt (aes-cbc-essiv:sha256 mode, AES, 256 bit key size)
  • On top of that an LVM2 system, with extra partitions for /, /boot, /usr, /var, /tmp, /home, and swap.
  • Optionally, SELinux enabled on that system (targeted policy in permissive mode).

bonnie++:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
forest         464M 11149  54 16660  20  6461   5  7472  58 11129   5 136.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1564  98 +++++ +++ +++++ +++  1650  98 +++++ +++  2640  97
forest,464M,11149,54,16660,20,6461,5,7472,58,11129,5,136.4,0,16,1564,98,+++++,
+++,+++++,+++,1650,98,+++++,+++,2640,97

bonnie++ with SELinux:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
forest         464M  9878  52 12138  11  5457   6  6834  56 11037   5 137.2   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1426  97 +++++ +++ +++++ +++  1451  98 +++++ +++  2433  97
forest,464M,9878,52,12138,11,5457,6,6834,56,11037,5,137.2,0,16,1426,97,+++++,
+++,+++++,+++,1451,98,+++++,+++,2433,97

hdparm:

 Timing cached reads:   1408 MB in  2.00 seconds = 704.01 MB/sec
 Timing buffered disk reads:   80 MB in  3.02 seconds =  26.53 MB/sec

hdparm with SELinux:

 Timing cached reads:   1396 MB in  2.00 seconds = 698.06 MB/sec
 Timing buffered disk reads:   82 MB in  3.07 seconds =  26.69 MB/sec

So yes, there is some overhead, but it's nothing too serious, IMHO. And quite honestly, I don't care too much about performance here — security is more important than performance. I think you'll agree; if you don't agree now, you will agree with me on the very day someone steals your laptop ;-)

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
Sat, 2006-09-16 12:34 — Russell Coker (not verified)

multiple runs

In some of the tests the differences were small enough that they are probably less than the random difference you might see between consecutive runs (as an example the per-char tests running faster under SE Linux in the first set).

Currently the best I've done in terms of analysing this is in Bonnie++ 1.9x where the bon_csv2html program applies colors to the numbers, put in 5 tests with one setting and 5 tests with another into bon_csv2html and it should be easier to see what the difference is.

PS For encrypting my laptop hard drive I do the encryption of the LV (IE encryption is on top of LVM not below it). The reason for this is that I often copy large files that are not secret to my laptop (EG ISO files of Linux distributions). Having LVM volumes that are not encrypted for such data is convenient.

Sat, 2006-09-16 14:25 — Uwe Hermann

dm-crypt over LVM, or vice versa

Hi Russell,

The results I listed here are the third run of each program (i.e. I ran hdparm three times without SELinux and pasted the results into the website, then three times with SELinux enabled, pasting run number three into the website, and so on; same for bonnie++)...

I might try bon_csv2html if I get around to it, but it's not that important anymore for me — I now know the performance impact is definately not going to stop me from setting up this system.

As for dm-crypt over LVM or LVM over dm-crypt: I'll do it the other way around for paranoia reasons (performance doesn't matter). If dm-crypt is the lowest layer, an attacker doesn't see different partitions of various sizes (which might give away some information, e.g. he could guess which of them is my /var, etc. etc.).
Also, I do not want anything besides my MBR to be unencrypted on the disk...

Do you notice real differences when putting ISOs on an unencrypted partition rather than an encrypted one very often?

Cheers, Uwe.

Sat, 2006-09-16 08:34 — blindcoder (not verified)

Encryption on laptop

Moin uwe.

I've been running dm-crypt filesystem encryption on my laptop for quite a while now with no noticable speed-impact. Even an encrypted root-filesystem doesn't slow the system down to the point of noticing.

Have fun :-)

Mon, 2009-03-23 12:52 — sileNT (not verified)

it does slow dow the things for me

I have a SATA drive which is capable of reading ~120 MB/s.

When using dm-crypt, read is only about ~55 MB/s. This is because kcryptd is taking most of CPU time - CPU just can't decrypt any faster.

Has anyone made any tests with comparison of different ciphers used with dm-crypt?

Sat, 2006-09-16 14:42 — Uwe Hermann

dm-crypt

Yeah, I'm running all my external USB hard drives on dm-crypt, it's about time to do the same on my laptop...

I'm currently trying to find a way to have all of the drive encrypted. An unencrypted /boot is not an option, and having /boot on CDROM is inconvenient when you do kernel upgrades (needs a new CD-R every time, I guess). Booting from USB thumb drive is not an option, my laptop doesn't support that.

I was thinking about putting /boot on the encrypted partition, too, and building a LiveCD which asks for a passphrase, opens the encrypted partition, and then boots from the /boot on there... That makes kernel upgrades easy, and /boot is encrypted when the laptop is shut down. I've never seen a HOWTO which does it this way, though, so I'll have to test it first and figure out if it really works...

Do you have any practical tips on running the system? What can you do to prevent data loss (e.g. if your battery goes empty and the laptop shuts down the hard way)? I've tested this three times, and it booted up fine after that, but I still have a bad feeling that it might not always work that well...

Running smartd is probably a good idea. Any other tips?

Cheers, Uwe.

Sun, 2006-09-17 11:36 — blindcoder (not verified)

Shutting down the hard way

Shutting down the hard way with dm-crypt is the same as without. I've had this happen to me so often that I stopped counting (both the battery contacts and the power plug are worn out by now and sometimes they both shut down at the same time).
device-mapper does not cache any data.

Anyway, I've recently read about something called kexec: http://www.xmission.com/~ebiederm/files/kexec/README

This might help you on your endeavour. I'd be eager to hear about it in any case :-)