RtlCompressBuffer and RtlDecompressBuffer

[Patrice Terrier]

Has anyone used the RtlCompressBuffer NT API?

José, it is undocumented in your WinAPI header.

...

[José Roca]

 
It is not part of the SDK, but of the DDK (Driver Development Kit), and I think it's only available statically linking to Ntoskrnl.lib.

[Patrice Terrier]

It is part of the Runtime Library Routines
http://msdn.microsoft.com/en-us/library/bb981783.aspx

Available in Microsoft Windows XP and later versions of all Windows operating systems.

...

[José Roca]

 
RTL means Runtime Library, but RtlCompressBuffer belongs to Windows Driver Kit -> Device and Driver Technologies -> Installable File System Drivers.

Headers: Declared in Ntifs.h. Include Fltkernel.h or Ntifs.h.
Library: Contained in Ntoskrnl.lib.

You need to install the DDK if you want the headers, and the headers aren't worth a penny if the function is only available in a static library.

[Patrice Terrier]

José,

Here is the correct declaration to use it.

Code Select Expand


%COMPRESSION_FORMAT_LZNT1    = &H0002
%COMPRESSION_ENGINE_STANDARD = &H0000 '// Standart compression
%COMPRESSION_ENGINE_MAXIMUM = &H0100 '// Maximum compression

%STATUS_BAD_COMPRESSION_BUFFER    = &HC0000242&
%STATUS_UNSUPPORTED_COMPRESSION   = &HC000025F&
%STATUS_INVALID_PARAMETER         = &HC000000D&

DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer" ( _
       BYVAL CompressFormat  AS WORD, _
       BYVAL ptrSrceBuffer   AS DWORD, _
       BYVAL SrceBufferSize  AS LONG, _
       BYVAL ptrDestBuffer   AS DWORD, _
       BYVAL DestBufferSize  AS LONG, _
       BYVAL ChunkSize       AS LONG, _
       BYREF CompressedSize  AS LONG, _
       BYVAL WorkspaceBuffer AS DWORD) AS LONG

DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize" ( _
       BYVAL CompressFormat    AS WORD,  _
       BYREF pNeededBufferSize AS LONG, _
       BYREF pFragmentSize     AS LONG) AS LONG

DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer" ( _
       BYVAL CompressionFormat AS WORD, _
       BYVAL ptrDestBuffer     AS DWORD, _
       BYVAL DestBufferSize    AS LONG, _
       BYVAL ptrSrceBuffer     AS DWORD, _
       BYVAL SceBufferSize     AS LONG, _
       BYREF pDestinationSize  AS LONG ) AS LONG

It works very well, except in one case, where i got %STATUS_BAD_COMPRESSION_BUFFER and i have no idea why.
Hence the reason why i was asking if someone else was using it.

My goal was to use it to produce lossless compression, and a good alternative to zLib, that would not require an extra DLL as it is used by the OS itself to perform file compression/decompression.

...

[José Roca]

 
Anyway, the headers are not included in the Platform SDK, but in the Windows Driver Kit, that I don't have: http://www.microsoft.com/downloads/details.aspx?FamilyID=2105564e-1a9a-4bf4-8d74-ec5b52da3d00&displaylang=en

[Patrice Terrier]

José,

It has been available for a long time, since Windows XP, however it was an undocumented feature. Microsoft started to document it, as well as many other API, with the new SDK for Seven.

I thought it was worth mentioning it for the community.

...

[Patrice Terrier]

When using RtlDecompressBuffer with very small files, it is VERY IMPORTANT to use the same size for the resulting buffer than the original before compression, or you may encounter a %STATUS_BAD_COMPRESSION_BUFFER error!

The best solution is to use a header to store the information about the original buffer size, and add it to the resulting compressed buffer. If you do this, then you will have a very reliable alternative to ZIP, producing small and fast lossless compressed file.

...

[Jim Dunn]

Patrice, I was able to use your fragments from here and the PB forum and got this working (below).

Now I'm trying to figure out how to open an existing "file.zip" file.

Thx!  : )

Code Select Expand


#COMPILE EXE
#DIM ALL

DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer" ( _
                              BYVAL CompressionFormat       AS LONG, _
                              BYVAL SourceBuffer            AS DWORD PTR, _
                              BYVAL SourceBufferLength      AS LONG, _
                              BYVAL DestinationBuffer       AS DWORD, _
                              BYVAL DestinationBufferLength AS DWORD PTR, _
                              BYVAL Unknow                  AS LONG, _
                              BYREF pDestinationSize        AS DWORD, _
                              BYVAL WorkspaceBuffer         AS DWORD PTR ) AS LONG

DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize" ( _
                              BYVAL CompressionFormat       AS LONG,  _
                              BYREF pNeededBufferSize       AS LONG, _
                              BYREF pUnknown                AS LONG )  AS LONG

DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer" ( _
                              BYVAL CompressionFormat       AS LONG, _
                              BYVAL DestinationBuffer       AS DWORD PTR, _
                              BYVAL DestinationBufferLength AS LONG, _
                              BYVAL SourceBuffer            AS DWORD PTR, _
                              BYVAL SourceBufferLength      AS LONG, _
                              BYREF pDestinationSize        AS LONG ) AS LONG

FUNCTION RTL_DECOMPRESS ( BYVAL input_Buffer AS STRING, BYREF output_buffer AS STRING ) AS LONG
  LOCAL pOutBuffSize AS LONG

  ' make maximal destination buffer
  IF LEN( output_buffer ) = 0 THEN
     output_buffer = STRING$( LEN( input_buffer ) * 12.5, 0 )
  END IF

  ' decompress buffer
  RtlDecompressBuffer ( BYVAL &H02, _
              BYVAL STRPTR( output_buffer ), _
              BYVAL LEN(output_buffer), _
              BYVAL STRPTR( input_buffer ), _
              BYVAL LEN( input_buffer), _
                pOutBuffSize )

  output_buffer    = LEFT$(output_buffer, pOutBuffSize )

  ' function returns output buffer size, zero if error
  FUNCTION         = pOutBuffSize
END FUNCTION

FUNCTION RTL_COMPRESS ( BYVAL input_Buffer AS STRING, BYREF output_buffer AS STRING ) AS LONG
    LOCAL ComprFormat, L, pOutBuffSize, NeedBuffSize AS LONG
    LOCAL Need_Buff AS STRING

    ' set compression format
    ' use LZNT1 with MAXIMUM compression
    ComprFormat = MAK ( LONG, &H02, &H0100 ) '

    ' Calculate the workspace for compression engine
    RtlGetCompressionWorkSpaceSize ( ComprFormat, NeedBuffSize, &h4000 )

    ' define compress parameters
    Need_Buff = STRING$( NeedBuffSize, 0 )
    L         = LEN( input_buffer )

    IF LEN( output_buffer ) = 0 THEN output_buffer = STRING$ ( L, 0 )

    ' compress input buffer ...
    RtlCompressBuffer ( BYVAL ComprFormat, _
              BYVAL STRPTR( input_buffer ), _
              BYVAL L, _
              BYVAL STRPTR( output_buffer ), _
              BYVAL L, _
              BYVAL &h2000, _
                  pOutBuffSize, _
              BYVAL STRPTR(Need_Buff) )

    output_buffer  =    LEFT$( output_buffer, pOutBuffSize )

    ' function returns output buffer size, zero if error
    FUNCTION       =    pOutBuffSize
END FUNCTION


FUNCTION PBMAIN () AS LONG
  LOCAL sz, rt AS LONG, file, src_buff, dest_buff AS STRING

  file = "data.txt"

  ' Compression test
  OPEN file FOR BINARY AS 1: sz = LOF(1): GET$ 1, sz, src_buff: CLOSE 1
  rt = RTL_COMPRESS ( src_buff, dest_buff )
  OPEN file+".lz" & file FOR BINARY AS 1: PUT$ 1, dest_buff: CLOSE 1
  ? "Compression Done." & $CR & "Output buffer size=" & STR$(rt)

  ' Decompression test
  RESET src_buff
  rt = RTL_DECOMPRESS ( dest_buff, src_buff )

  OPEN file+".unlz" FOR BINARY AS 1: PUT$ 1, src_buff: CLOSE 1
  ? "DeCompression Done." & $CR & "Output buffer size=" & STR$(rt)
END FUNCTION

[Patrice Terrier]

If you want to open a ZIP file, then you must use code from Jean-Loup Gailly.

http://gailly.net/

...

[Pierre Bellisle]

I did some experiment with RtlCompressBuffer & co.

In the example above the output_buffer size is set to LEN(input_buffer) * 12.5
I do not know where 12.5 come from but the compression ratio can be
a lot bigger than that, like if we compress STRING$(1,000,000 "A").
So I used a loop to grow the buffer step by step to near
the PowerBASIC string size limit if needed.

Also, Patrice, about very small string compression,
if one want to avoid using a header.
As we know, the compressed result may be longer than the uncompressed source,
normal with a one byte string source.
But only by 3 bytes according to my tests under XP-32 and Se7en-64.
So increasing unCompressedBufferSize by 3 bytes resolve the problem
and keep compressed string structure intact.

Code Select Expand


'http://www.jose.it-berater.org/smfforum/index.php?topic=3467.0

#COMPILE EXE '#Win 8.04#  Thank to Aslan Babakhanov
#DIM ALL
#OPTION VERSION5 'Not 2000 but XP in fact, to be lazy or not to be lazy...
#INCLUDE "WIN32API.INC"
#INCLUDE "NTSTATUS.INC"

%MaxStringLen = 2147483000

GLOBAL sBuffer AS STRING

DECLARE FUNCTION RtlCompressBuffer LIB "ntdll.dll" ALIAS "RtlCompressBuffer"( _
       BYVAL CompressionFormat       AS LONG, _
       BYVAL SourceBuffer            AS DWORD PTR, _
       BYVAL SourceBufferLength      AS LONG, _
       BYVAL DestinationBuffer       AS DWORD, _
       BYVAL DestinationBufferLength AS DWORD PTR, _
       BYVAL Unknow                  AS LONG, _
             pDestinationSize        AS DWORD, _
       BYVAL WorkspaceBuffer         AS DWORD PTR) AS LONG

DECLARE FUNCTION RtlGetCompressionWorkSpaceSize LIB "ntdll.dll" ALIAS "RtlGetCompressionWorkSpaceSize"( _
       BYVAL CompressionFormat       AS LONG,  _
             pNeededBufferSize       AS LONG, _
             pUnknown                AS LONG)  AS LONG

DECLARE FUNCTION RtlDecompressBuffer LIB "ntdll.dll" ALIAS "RtlDecompressBuffer"( _
       BYVAL CompressionFormat       AS LONG, _
       BYVAL DestinationBuffer       AS DWORD PTR, _
       BYVAL DestinationBufferLength AS LONG, _
       BYVAL SourceBuffer            AS DWORD PTR, _
       BYVAL SourceBufferLength      AS LONG, _
             pDestinationSize        AS LONG) AS LONG
'______________________________________________________________________________

FUNCTION RTL_DECOMPRESS(BYVAL sCompressedBuffer AS STRING, BYREF sUncompressedBuffer AS STRING) AS LONG
LOCAL FinalUncompressedSize  AS DWORD
LOCAL unCompressedBufferSize AS DWORD
LOCAL RetVal                 AS LONG

unCompressedBufferSize = LEN(sCompressedBuffer) * 5
IF unCompressedBufferSize = 0 THEN 'Nothing to decompress,  0 * 5 = 0
  RetVal = %STATUS_NO_DATA_DETECTED '%STATUS_INVALID_PARAMETER
ELSE
  DO
    sUncompressedBuffer = NUL$(unCompressedBufferSize)
    RetVal = RtlDecompressBuffer(%COMPRESSION_FORMAT_LZNT1, _
                                 BYVAL STRPTR(sUncompressedBuffer), _
                                 BYVAL unCompressedBufferSize, _
                                 BYVAL STRPTR(sCompressedBuffer), _
                                 BYVAL LEN(sCompressedBuffer), _
                                 FinalUncompressedSize)
    SELECT CASE RetVal

      CASE %STATUS_INVALID_PARAMETER
        sBuffer = sBuffer & "%STATUS_INVALID_PARAMETER" & $CRLF

      CASE %STATUS_UNSUPPORTED_COMPRESSION
       sBuffer = sBuffer & "%STATUS_UNSUPPORTED_COMPRESSION" & $CRLF

      CASE %STATUS_SUCCESS
        sBuffer = sBuffer & "STATUS_SUCCESS" & $CRLF
        EXIT LOOP

      CASE %STATUS_BAD_COMPRESSION_BUFFER
        sBuffer = sBuffer & "%STATUS_BAD_COMPRESSION_BUFFER with " & _
                  FORMAT$(unCompressedBufferSize, "#,###") & " bytes." & $CRLF & _
                  "Increasing buffer size and retry..." & $CRLF
        'Dynamic strings can contain up to approximately 2 Gb (2^31) characters.
        IF unCompressedBufferSize >= %MaxStringLen THEN EXIT LOOP
        unCompressedBufferSize = MIN(unCompressedBufferSize * 20, %MaxStringLen)

    END SELECT
  LOOP
END IF

IF RetVal = %STATUS_SUCCESS THEN
  sUncompressedBuffer = LEFT$(sUncompressedBuffer, FinalUncompressedSize)
  FUNCTION = FinalUncompressedSize
ELSE
 sUncompressedBuffer = ""
 FUNCTION = 0
END IF

END FUNCTION
'______________________________________________________________________________

FUNCTION RTL_COMPRESS(BYVAL sUncompressedBuffer AS STRING, BYREF sCompressedBuffer AS STRING) AS LONG
LOCAL RetVal                        AS LONG
LOCAL CompressionFormatAndEngine    AS DWORD
LOCAL unCompressedBufferSize        AS DWORD
LOCAL CompressedBufferSize          AS DWORD
LOCAL FinalCompressedSize           AS DWORD
LOCAL CompressBufferWorkSpaceSize   AS DWORD
LOCAL CompressFragmentWorkSpaceSize AS DWORD
LOCAL unCompressedChunkSize         AS DWORD
LOCAL CompressBufferWorkSpace       AS STRING

CompressionFormatAndEngine = MAK(DWORD, %COMPRESSION_FORMAT_LZNT1, %COMPRESSION_ENGINE_MAXIMUM)
RtlGetCompressionWorkSpaceSize(CompressionFormatAndEngine, _      '%COMPRESSION_ENGINE_STANDARD
                               CompressBufferWorkSpaceSize, _
                               CompressFragmentWorkSpaceSize)
CompressBufferWorkSpace = NUL$(CompressBufferWorkSpaceSize)

unCompressedBufferSize = LEN(sUncompressedBuffer)
CompressedBufferSize = unCompressedBufferSize + 3 'See line below
'Add 3 in case output is bigger than input witch appen witth small buffer
sCompressedBuffer = NUL$(CompressedBufferSize)

unCompressedChunkSize = 4096
'Can be 512, 1024, 2048, or 4096. The os uses 4096 and it's also the recommended value.
RetVal = RtlCompressBuffer(BYVAL CompressionFormatAndEngine, _
                           BYVAL STRPTR(sUncompressedBuffer), _
                           BYVAL unCompressedBufferSize, _
                           BYVAL STRPTR(sCompressedBuffer), _
                           BYVAL CompressedBufferSize, _
                           BYVAL unCompressedChunkSize, _
                           FinalCompressedSize, _
                           BYVAL STRPTR(CompressBufferWorkSpace))
SELECT CASE RetVal

  CASE %STATUS_SUCCESS
    sBuffer = sBuffer & "STATUS_SUCCESS" & $CRLF

  CASE %STATUS_BUFFER_ALL_ZEROS
    sBuffer = sBuffer & "%STATUS_BUFFER_ALL_ZEROS" & $CRLF

  CASE %STATUS_INVALID_PARAMETER
    sBuffer = sBuffer & "%STATUS_INVALID_PARAMETER" & $CRLF

  CASE %STATUS_UNSUPPORTED_COMPRESSION
    sBuffer = sBuffer & "%STATUS_UNSUPPORTED_COMPRESSION" & $CRLF

  CASE %STATUS_NOT_SUPPORTED
    sBuffer = sBuffer & "%STATUS_NOT_SUPPORTED" & $CRLF

  CASE %STATUS_BUFFER_TOO_SMALL
    sBuffer = sBuffer & "STATUS_BUFFER_TOO_SMALL"  & STR$(FinalCompressedSize) & $CRLF

END SELECT

sCompressedBuffer = LEFT$(sCompressedBuffer, FinalCompressedSize)
FUNCTION = FinalCompressedSize

END FUNCTION
'______________________________________________________________________________

FUNCTION PBMAIN() AS LONG
LOCAL hFile           AS DWORD
LOCAL UnCompressedLen AS DWORD
LOCAL DeCompressedLen AS DWORD
LOCAL CompressedLen   AS DWORD
LOCAL sUncompressed   AS STRING
LOCAL sCompressed     AS STRING
LOCAL sDeCompressed   AS STRING
LOCAL zFileName       AS ASCIIZ * %Max_Path

zFileName = "C:\Windows\System32\Shell32.DLL" '<-- Change
hFile = FREEFILE : OPEN zFileName FOR BINARY AS hFile : GET$ hFile,  LOF(hFile), sUncompressed : CLOSE hFile
UnCompressedLen = LEN(sUncompressed)

sBuffer = zFileName & $CRLF & "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "2 first bytes = " & $DQ & LEFT$(sUncompressed, 2) & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "2 first bytes = " & $DQ & LEFT$(sDeCompressed, 2) & $DQ
MSGBOX sBuffer

sUncompressed = "A"
UnCompressedLen = LEN(sUncompressed)
sBuffer = "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sUncompressed & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes "
sBuffer = sBuffer & "(" & HEX$(ASC(sCompressed, 1), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 2), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 3), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 4), 2) & ")" & $CRLF
sUncompressed = ""
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sDeCompressed & $DQ & $CRLF
MSGBOX sBuffer

sUncompressed = STRING$(150, "B") 'sUncompressed = STRING$(155, 0) 'sUncompressed = ""
UnCompressedLen = LEN(sUncompressed)
sBuffer = "unCompressed size = " & FORMAT$(UnCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sUncompressed & $DQ & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_COMPRESS" & $CRLF
CompressedLen = RTL_COMPRESS(sUncompressed, sCompressed)
sBuffer = sBuffer & "Compression done" & $CRLF & "Compressed = " & FORMAT$(CompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "(" & HEX$(ASC(sCompressed, 1), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 2), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 3), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 4), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 5), 2) & $SPC & _
                          HEX$(ASC(sCompressed, 6), 2) & ")" & $CRLF
sBuffer = sBuffer & $CRLF & "Calling RTL_DECOMPRESS" & $CRLF
DeCompressedLen = RTL_DECOMPRESS(sCompressed, sDeCompressed)
sBuffer = sBuffer & "Decompression done" & $CRLF & "deCompressed size = " & FORMAT$(DeCompressedLen, "#,###") & " bytes" & $CRLF
sBuffer = sBuffer & "String = " & $DQ & sDeCompressed & $DQ & $CRLF
MSGBOX sBuffer

END FUNCTION
'______________________________________________________________________________
'


[Patrice Terrier]

Pierre,

The header i am using, is part of my encryption protection, and lossless image compression

...