I'm currently creating a backup of my domain martin-thoma.de. It is pretty small, but the MySQL database behind it grew extremely as I stored every drawing from write-math.com in there. For this reason, I want to store that specific table in a compressed way. But which compression format is the best for which use case?
This article is only about loss-less compression. There are many more things to say about lossy compression.
My System
| Thinkpad T460p | |
|---|---|
| CPU (comparison on ark.inten.com) | 8x Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz |
| RAM | 8 GB |
| Video Card | Nvidia GeForce 940MX |
| System | Ubuntu 18.04.3 LTS |
| Disk | SAMSUNG MZ7LN512HCHP (PM871 series) |
Usage
| Format | (Un)compress | Command | Time |
|---|---|---|---|
| ZIP | compress | zip output.zip input.sql |
92s |
| ZIP | extract | unzip output.zip -d outputdir |
29s |
| gzip | compress | gzip -1 -k backup.sql |
51s |
| gzip | compress | gzip -2 -k backup.sql |
50s |
| gzip | compress | gzip -3 -k backup.sql |
54s |
| gzip | compress | gzip -4 -k backup.sql |
58s |
| gzip | compress | gzip -5 -k backup.sql |
65s |
| gzip | compress | gzip -6 -k backup.sql |
89s |
| gzip | compress | gzip -7 -k backup.sql |
110s |
| gzip | compress | gzip -8 -k backup.sql |
319s |
| gzip | compress | gzip -9 -k backup.sql |
372s |
| gzip | extract | gzip -d backup.sql.gz (-1 to -6) |
58s to 62s |
| gzip | extract | gzip -d backup.sql.gz (-7 to -9) |
49s to 50s |
| bz2 | compress | bzip2 -1 -k -z backup.sql |
287s |
| bz2 | compress | bzip2 -2 -k -z backup.sql |
285s |
| bz2 | compress | bzip2 -3 -k -z backup.sql |
294s |
| bz2 | compress | bzip2 -4 -k -z backup.sql |
301s |
| bz2 | compress | bzip2 -5 -k -z backup.sql |
363s |
| bz2 | compress | bzip2 -6 -k -z backup.sql |
356s |
| bz2 | compress | bzip2 -7 -k -z backup.sql |
364s |
| bz2 | compress | bzip2 -8 -k -z backup.sql |
407s |
| bz2 | compress | bzip2 -9 -k -z backup.sql |
369s |
| bz2 | extract | bzip2 -d backup.sql.bz2 (-1 to -9) |
102s to 109s |
| 7z | compress | 7z a backup.sql.7z backup.sql |
466s |
| 7z | extract | 7z e backup.sql.7z |
49s |
| 7z (ultra) | compress | 7z a out.7z -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on backup.sql |
1562s |
7z Benchmark
$ 7z b
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (506E3),ASM,AES-NI)
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (506E3)
CPU Freq: 1737 2731 3166 3349 3271 3108 3181 3202 3157
RAM size: 7404 MB, # CPU hardware threads: 8
RAM usage: 1765 MB, # Benchmark threads: 8
Compressing | Decompressing
Dict Speed Usage R/U Rating | Speed Usage R/U Rating
KiB/s % MIPS MIPS | KiB/s % MIPS MIPS
22: 15592 618 2455 15168 | 175880 731 2053 15002
23: 12156 632 1958 12386 | 171362 738 2008 14829
24: 13697 643 2289 14727 | 164813 754 1918 14465
25: 13105 654 2287 14964 | 151047 742 1812 13443
---------------------------------- | ------------------------------
Avr: 637 2247 14311 | 741 1948 14435
Tot: 689 2098 14373
and
$ 7z b -mm=\* -mmt=\*
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (506E3),ASM,AES-NI)
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (506E3)
CPU Freq: 2661 2917 3208 3272 3177 3302 3269 3246 3188
RAM size: 7404 MB, # CPU hardware threads: 8
RAM usage: 225 MB, # Benchmark threads: 1
Method Speed Usage R/U Rating E/U Effec
KiB/s % MIPS MIPS % %
CPU 100 3186 3186
CPU 100 3180 3180
CPU 100 3129 3129 100 100
LZMA:x1 11260 100 4121 4116 132 132
33725 100 2750 2746 88 88
LZMA:x5:mt1 2596 100 3245 3244 104 104
32537 100 2745 2744 88 88
LZMA:x5:mt2 5446 180 3780 6804 121 217
34839 100 2939 2939 94 94
Deflate:x1 38939 100 4944 4944 158 158
112657 100 3500 3500 112 112
Deflate:x5 8530 100 3285 3284 105 105
102640 100 3189 3186 102 102
Deflate:x7 5016 100 5558 5558 178 178
121815 100 3780 3780 121 121
Deflate64:x5 11477 100 4961 4960 159 159
118711 100 3713 3713 119 119
BZip2:x1 6447 100 3895 3895 125 125
30463 100 3303 3302 106 106
BZip2:x5 5066 100 4228 4228 135 135
15084 100 2961 2961 95 95
BZip2:x5:mt2 9708 194 4167 8103 133 259
42347 160 5185 8311 166 266
BZip2:x7 1648 100 4271 4271 137 137
27652 100 5423 5423 173 173
PPMD:x1 4516 100 4672 4671 149 149
4234 100 4987 4987 159 159
PPMD:x5 3289 100 5575 5574 178 178
3070 100 5753 5753 184 184
Delta:4 863808 100 5309 5307 170 170
953462 100 5858 5858 187 187
BCJ 1672730 100 6852 6852 219 219
1654731 100 6778 6778 217 217
AES256CBC:1 201192 100 4946 4945 158 158
189151 100 4650 4649 149 149
AES256CBC:2 854545 100 7002 7000 224 224
3411324 100 6986 6986 223 223
CRC32:1 398309 100 2901 2900 93 93
CRC32:4 1302032 100 2907 2906 93 93
CRC32:8 2657729 100 3604 3604 115 115
CRC64 1196491 100 2451 2450 78 78
SHA256 201949 100 4121 4120 132 132
SHA1 553725 100 5190 5183 166 166
BLAKE2sp 325681 100 7167 7165 229 229
CPU 100 3364 3363
------------------------------------------------------
Tot: 111 3690 4120 119 132
RAM usage: 901 MB, # Benchmark threads: 4
Method Speed Usage R/U Rating E/U Effec
KiB/s % MIPS MIPS % %
CPU 399 3084 12318
CPU 400 3090 12353
CPU 400 3090 12354 100 400
LZMA:x1 49593 397 4571 18130 148 587
127780 399 2607 10406 84 337
LZMA:x5:mt1 11043 396 3482 13796 113 447
123224 399 2602 10391 84 336
LZMA:x5:mt2 15408 694 2774 19250 90 623
115385 394 2472 9730 80 315
Deflate:x1 127310 396 4079 16165 132 523
377145 399 2934 11718 95 379
Deflate:x5 39249 399 3791 15112 123 489
371312 399 2892 11527 94 373
Deflate:x7 15420 398 4291 17085 139 553
370053 399 2880 11483 93 372
Deflate64:x5 32242 388 3596 13933 116 451
364650 399 2855 11407 92 369
BZip2:x1 19452 399 2949 11753 95 381
101048 399 2743 10955 89 355
BZip2:x5 15092 400 3150 12595 102 408
61031 400 2996 11979 97 388
BZip2:x5:mt2 16446 783 1754 13726 57 444
93754 727 2530 18402 82 596
BZip2:x7 4855 397 3166 12578 102 407
62094 400 3046 12177 99 394
PPMD:x1 14300 400 3701 14790 120 479
13009 399 3836 15320 124 496
PPMD:x5 9590 399 4075 16253 132 526
8516 394 4050 15960 131 517
Delta:4 2229140 399 3429 13696 111 443
2407149 397 3727 14790 121 479
BCJ 4218363 398 4344 17278 141 559
4682966 393 4880 19181 158 621
AES256CBC:1 516685 391 3250 12698 105 411
488100 397 3018 11996 98 388
AES256CBC:2 2548047 399 5228 20874 169 676
9436324 397 4874 19326 158 626
CRC32:1 1304042 400 2375 9493 77 307
CRC32:4 4019189 399 2246 8971 73 290
CRC32:8 7320978 400 2484 9927 80 321
CRC64 3549391 399 1824 7269 59 235
SHA256 505381 395 2607 10310 84 334
SHA1 1255910 380 3093 11755 100 381
BLAKE2sp 451934 377 2638 9943 85 322
CPU 360 2275 8188
------------------------------------------------------
Tot: 440 3063 13276 98 430
RAM usage: 1802 MB, # Benchmark threads: 8
Method Speed Usage R/U Rating E/U Effec
KiB/s % MIPS MIPS % %
CPU 596 2352 14024
CPU 759 2489 18901
CPU 727 2477 18016 110 800
LZMA:x1 20619 590 1277 7538 57 335
110780 616 1466 9023 65 401
LZMA:x5:mt1 9602 633 1895 11997 84 533
135560 734 1557 11432 69 508
LZMA:x5:mt2 10958 696 1968 13690 87 608
114711 686 1411 9674 63 430
Deflate:x1 120232 707 2158 15267 96 678
368073 728 1572 11437 70 508
Deflate:x5 39116 707 2131 15061 95 669
391901 708 1719 12168 76 540
Deflate:x7 15543 736 2339 17222 104 765
416000 756 1707 12910 76 573
Deflate64:x5 33348 703 2050 14411 91 640
381221 742 1607 11926 71 530
BZip2:x1 19466 743 1583 11761 70 522
107349 705 1650 11637 73 517
BZip2:x5 14640 757 1614 12219 72 543
77475 738 2061 15207 92 675
BZip2:x5:mt2 15561 731 1776 12987 79 577
80522 758 2085 15805 93 702
BZip2:x7 5127 775 1714 13284 76 590
69233 782 1737 13577 77 603
PPMD:x1 13343 739 1867 13800 83 613
11466 720 1876 13503 83 600
PPMD:x5 9060 740 2074 15355 92 682
5011 648 1449 9392 64 417
Delta:4 1425436 592 1478 8758 66 389
1668622 575 1783 10252 79 455
BCJ 1763558 544 1328 7224 59 321
2872698 569 2070 11767 92 522
AES256CBC:1 473162 723 1609 11628 71 516
482136 744 1592 11849 71 526
AES256CBC:2 3391242 638 4356 27781 193 1234
7308622 649 2307 14968 102 665
CRC32:1 2005570 744 1961 14601 87 648
CRC32:4 5340310 769 1550 11920 69 529
CRC32:8 8750400 749 1585 11866 70 527
CRC64 4884587 707 1416 10004 63 444
SHA256 469742 751 1277 9583 57 426
SHA1 1292792 725 1669 12101 74 537
BLAKE2sp 810867 764 2335 17839 104 792
CPU 775 2390 18537
------------------------------------------------------
Tot: 696 1710 11921 76 529
Compression Rates
| File | File Size | .zip | .gz | .bz2 | .7z |
|---|---|---|---|---|---|
| MySQL DB dump | 3.5 GB | 634 MB | 1: 737 MB 2: 729 MB 3: 722 MB 4: 654 MB 5: 638 MB 6: 634 MB 7: 610 MB 8: 606 MB 9: 606 MB |
1: 534 MB 2: 525 MB 3: 521 MB 4: 519 MB 5: 518 MB 6: 518 MB 7: 518 MB 8: 518 MB 9: 518 MB |
default: 471 MB ultra: 398 MB |
Features
| Format | Archiving | Compression | Integrity Check | Partial Extraction |
|---|---|---|---|---|
| 7z (man) | ✔ | ✔ | ✔ | ✔ |
| zip (man) | ✔ | ✔ | ✔ | ✔ |
| tar (man) | ✔ | ✗ | ||
| gzip (man) | ✗ | ✔ | ||
| bzip2 (man) | ✗ | ✔ |
See also
- Wikipedia:
- Martin Thoma: Data Backup Strategies, 2013.
- Jeff Atwood:
- File Compression in the Multi-Core Era, 2009.
- Don't Use ZIP, Use RAR, 2007.
- Compression and Cliffs, 2005.
- Lossy compression:
- Jeff Atwood: A Comparison of JPEG Compression Levels and Recompression
- Jeff Atwood: Getting the Most Out of PNG, 2007.
- What is the maximum size of a zip file on Windows 10 Pro 64-Bit?, 2018