Problem with saving string array with SaveFile statement

[Chris Chancellor]

Hello Charles

i refer to your code in http://www.oxygenbasic.org/forum/index.php?topic=1575.15

about the Saving and Loading a binary file, you were using numeric arrays.
so i have modified it to do  string arrays but was not successful

Code Select Expand


' Save_LoadFile.o2bas
' http://www.oxygenbasic.org/forum/index.php?topic=1575.15
  $ filename "Save_LoadFile.exe"
       uses rtl64

'  this program save array data into a random access file and  then load
'  the file data into another array

uses corewin

function LoadFile(string name, any*data, int e)
===============================================
sys f
f=fopen name,"r"    'open for reading
e=fread @data,1,e,f 'load data
fclose f            'close file
return e
end function


function SaveFile(string name, any*data, int e)
===============================================
sys f
f=fopen name,"w"      'open for writing
e=fwrite @data,1,e,f  'save data
fclose f              'close file
return e
end function

'TEST
=====
string  v[100], w[100]

' Save the data from array V[ ] into the file myfil.bin
for i=1 to 100 : v[i]= str(i*1.5)   : next
V[61] = "test string1"
V[71] = "test string2"
SaveFile "myfil.bin",v,bytesof v


' Load the data from myfil.bin to array W[  ]
LoadFile "myfil.bin",w,bytesof v

print w[51]
print w[61]
print w[69]
print w[71]




particularly the codes which i have changed from numeric arrays to string arrays

maybe the statement  SaveFile "myfil.bin",v,bytesof v      is incorrect as i'm using strings

what would be its replacement?   Thanxx and appreciate your help

[Charles Pegge]

Hi Chris,

The problem is that LoadFile and SaveFile will only work for flat data. An array of strings is essentially an array of pointers - so you end up saving a bunch of pointers without the accompanying data. For random access in a file, you need a table of locations and lengths, followed by the string content of each element.

As this is a fairly common situation, let me see if I can come up with a generic solution...

[James C. Fuller]

I think I may have posted this before?
It takes data from a csv file and creates a binary file.

James

Code Select Expand


'=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*
' not needed with O2RadAsm
'% filename "cFileIO.exe"
'=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*
use rtl64
#autodim off
use console
use corewin
'------------------------------------------------------------------------------
'GetFields function from modified OxygenBasic\examples\DataProcessing\GetFields.o2bas app
include "GetFields.o2bas"
'------------------------------------------------------------------------------
Type AddressType
    FirstName     As zstring * 32
    LastName      As zstring * 32
    Company       As zstring * 64
    Address       As zstring * 64
    City        As zstring * 32
    County        As zstring * 32
    State        As zstring * 4
    Zip           As zstring * 12
    Phone1         As zstring * 24
    Phone2         As zstring * 24
    Email         As zstring * 64
    Url           As zstring * 64
End Type
'==============================================================================
Macro LineInput string (s,c)
    s = nuls 1024
    s = (char*)fgets(s,1024,c)
End Macro
'==============================================================================
Function main() As sys
    string sfileIn = "Myus-500tab.csv"
    string sfileOut  = "Address50.bin"
    string sLine
    sys fp1,fp2,fields,Count
    Dim As AddressType Address
    int bytes,where
    fp1 = fopen sfileIn,"r"
    If not fp1 Then
        printl "PROBLEM #1"
        waitkey
        exit function
    End If
    fp2 = fopen sfileOut,"w"
    If not fp2 Then
        printl "PROBLEM #2"
        fclose fp1
        waitkey
        exit function
    End If

    Do
        sLine = LineInput(fp1)
        If Len(sLine) = 0 Then
            fclose(fp1)
            fclose(fp2)
            printl "PROBLEM #3"
            waitkey
            exit function
        End If
        Count++
        If Count > 50 Then
            break
        End If
        fields = GetFields(sLine,chr(9))
        With Address
            .FirstName    = arg(1)
            .LastName    = arg(2)
            .Company     = arg(3)
            .Address    = arg(4)
            .City        = arg(5)
            .County        = arg(6)
            .State        = arg(7)
            .Zip        = arg(8)
            .Phone1        = arg(9)
            .Phone2        = arg(10)
            .Email        = arg(11)
            .Url        = arg(12)
        End With   
        fwrite(Address,sizeof(Address),1,fp2)       
    loop
    fclose fp1
    fclose fp2
    printl "Is it done?"
    Printl "Now lets read the first record from the binary file"  cr  "press any key"
    waitkey
    fp1 = fopen sfileOut,"r+"
    If not fp1 Then
        printl "PROBLEM #4"
        waitkey
        exit function
    End If
    fread Address,sizeof(Address),1,fp1
    Printl "First Name -> " + Address.FirstName
    Printl "Last  Name -> " + Address.LastName
    Printl "State      -> " + Address.State
    Printl ""
    Printl "Now lets read the last record" + cr + "any key to continue"
     waitkey

     where = 49 * sizeof(Address)
     fseek fp1,where,0
     fread Address,sizeof(Address),1,fp1
    Printl "First Name -> " + Address.FirstName
    Printl "Last  Name -> " + Address.LastName
    Printl "State      -> " + Address.State
    fclose fp1
    Printl cr "any key to continue"
     
    waitkey
End Function
main



[Chris Chancellor]

Thanxx a lot  Charles and James

Yes James, you did send me a similar file name cFilieIO.o2bas  together   with its csv  files

in this case, i was checking whether an alternative statement such as SaveFile can be use or not.

i think your random file method is a better one.   i won't explore an alternative statement anymore.

[Charles Pegge]

This is what I had in mind. It converts an array of strings into a single string containing a table of lengths and offsets, followed by the raw strings' characters.

In this format, string arrays are easy to save and load, or to access strings directly from disk file:

Code Select Expand


'2018-04-30 T 21:19:28

  function FlatTable(string s[], int n) as string
  ===============================================
  'FORMAT:
  'number of strings
  'length of char data block
  'for each string:
  '  length of string in bytes
  '  offset of first string character
  'char data block
  '
  indexbase 1
  string bt       'to hold the table
  string bu       'to hold the char data
  int *bd         'dynamic table
  int lu          'length of char data buffer
  int di=2        'index for the table elements
  int offs=8+n*8  'base offset to the char data
  int ii          'offset to the char data
  int i           'iterator etc
  bt=nuls 8+n*8   'create table buffer
  @bd=strptr bt   'map table to its buffer
  for i=1 to n
    le=len s[i]
    if ii+le+2>lu then
      bu+=nuls le+0x1000 'expand buffer
      lu=len bu
    end if       
    mid bu,ii+1,s[i]
    di++
    bd[di]=le 'data length
    di++
    bd[di]=ii+offs 'char index (base 0)
    ii+=le+2 'next position, allowing 2 null bytes
  next
  bd[1]=n   'count of elements
  i=8+n*8   'length of table
  bd[2]=ii  'length of data block
  return left(bt,i)+left(bu,ii)
  end function

  'TESTS
  ======
  string s[100]
  for i=1 to 100 : s[i]="#" i*100 : next
  string t=FlatTable(s[],100)
  'DECODE AND EXTRACT STRINGS
  int d at strptr t
  redim string da(d)
  int j=2
  int le
  for i=1 to d
    j++ : le=d[j]
    j++ : da[i]=mid t,d[j]+1,le
  next
  print da[99] '#9900


[Chris Chancellor]

Thanxx a lot Charles

Beauty, when added the last  lines  as below,  it just came out instantly

Code Select Expand



  print "  at index 99 : " +da[99]   + chr(13,10) +
           "  at index 19  : " da[19] + chr(13,10) +
            "  at index 3  : " da[3]


[Chris Chancellor]

Yup this is a much faster method than read and write  into a disk file  for each record
just use the entire string to do it in memory.  so the modus operandi would be :

  1.  at the begining of the program, load the entire file into a string

  2.  do some data processing like update records etc

  3.  at the end of the program just save this single string into the disk file

i wonder how big would be file size that can be loaded into a single string.  what would be the file size limit?

[Charles Pegge]

Megabyte files, no problem. With Gigabyte files, you would have to handle at low-level, to avoid large intermediate strings.