linux multi core 압축

gzip 으로 하는데 느려서 반디집 처럼 멀티코어 지원하는 압축프로그램 찾는 중 발견

p만 붙이면 되는거였나 싶은데 pigz는 이름이 독특하네 ㅋㅋㅋ

아무튼 gzip을 많이 썼으니 pigz를 외우면 될 듯

 

PXZ - Parallel XZ is a compression utility that takes advantage of running LZMA compression of different parts of an input file on multiple cores and processors simultaneously. Its primary goal is to utilize all resources to speed up compression time with minimal possible influence on compression ratio. sudo apt-get install pxz PLZIP - Lzip is a lossless data compressor based on the LZMA algorithm, with very safe integrity checking and a user interface similar to the one of gzip or bzip2. Lzip decompresses almost as fast as gzip and compresses better than bzip2, which makes it well suited for software distribution and data archiving. Plzip is a massively parallel (multi-threaded) version of lzip using the lzip file format; the files produced by plzip are fully compatible with lzip. Plzip is intended for faster compression/decompression of big files on multiprocessor machines, which makes it specially well suited for distribution of big software files and large scale data archiving. On files big enough, plzip can use hundreds of processors. sudo apt-get install plzip PIGZ - pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that takes advantage of multiple processors and multiple cores when compressing data. sudo apt-get install pigz PBZIP2 - pbzip2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 (ie: anything compressed with pbzip2 can be decompressed with bzip2). sudo apt-get install pbzip2 LRZIP - A multithreaded compression program that can achieve very high compression ratios and speed when used with large files. It uses the combined compression algorithms of zpaq and lzma for maximum compression, lzo for maximum speed, and the long range redundancy reduction of rzip. It is designed to scale with increases with RAM size, improving compression further. A choice of either size or speed optimizations allows for either better compression than even lzma can provide, or better speed than gzip, but with bzip2 sized compression levels. sudo apt-get install lrzip

[링크 : https://askubuntu.com/questions/258202/multi-core-compression-tools]

 

--use-compress-program 을 통해서 압축 프로그램을 지정가능한 듯?

tar -c --use-compress-program=pigz -f tar.file dir_to_zip

 

[링크 : https://stackoverflow.com/.../utilizing-multi-core-for-targzip-bzip-compression-decompression]

 

+ 21.12.03

use-compress-program="xz -t4" 식으로 주면 되려나?

$ tar cvf archive.tar.xz --use-compress-program='xz -1' *.csv

[링크 : https://unix.stackexchange.com/.../how-to-use-multi-threading-for-creating-and-extracting-tar-xz]

[링크 : https://www.baeldung.com/linux/xz-compression]