APPEND

In this article, we look at the APPEND CUSP, which is used to append multiple files together. First the user documentation, then we will look at code that we have extracted and generalized from previous CUSPs, which have been moved out to CUSPLib - a unit that contains utility routines specifically for CUSPs. The routines we cover here have been adapted from the COPY CUSP. We won't go into great detail on any of them since they are merely generalized versions of code that we've covered in the past, but we include them here because we use them in the APPEND CUSP - and we will use them in the future. Yes, I should have done this back when we covered COPY, but I was under a time crunch. As the programming maxim goes: "there's never time to do it right the first time, but always time to do it over."

APPEND
Add the contents of one or more specified input files to the end of the specified output file.

Format

APPEND input{,...} output

Arguments

input
The file specification for a file to append to the output. Multiple input files can be specified by delimiting them with commas (,) or plus signs (+). Note that if any of the file specifications has embedded spaces, plus signs, or commas, you must enclose the specification within quotes. Wildcards are allowed in the input file specification(s). When wildcards are used, the files are appended in alphanumeric order. Note that if multiple input files are used with wildcards and the different specifications overlap, the same file(s) might be appended multiple times. For instance if *.TXT and A.* are both used for input and a file named A.TXT is found, it will be included twice.

output
The file specification for a file into which to append the input file(s). The current default device and/or path is used if not provided in the output file specification. The asterisk wildcard (*) can be used in the output specification, in which case the wildcard field(s) are defaulted from the first input file. Missing fields in the output specification are treated as wildcards.

Description

APPEND is similar in syntax and operation to COPY. Normally no backups or new file versions are created. An existing output file has the input files appended to the end of the file. /NEW_VERSION causes the creation of a new output file if the output file doesn't already exist.

Qualifiers

/ALLOCATION{=bytes}
Forces the initial allocation of the output file to the specified number of bytes. If the number of bytes is not specified, the initial allocation is determined by the size of the input file(s). This switch is ignored if the output file already exists.

/BACKUP
Uses the backup date when doing file comparisons using /BEFORE or /SINCE. This is incompatible with /CREATED, /EXPIRED, and /MODIFIED.

/BEFORE(=time)
Used to filter input files by date and time. The time specification can be an absolute or delta time, or one of the following keywords: BOOT, JOB_LOGIN, LOGIN, TODAY (default), TOMORROW, or YESTERDAY. This can be used in conjunction with /BACKUP, /CREATED, /EXPIRED, and /MODIFIED switches. If none of those switches are used, the file creation date is used.

/BUFFER_SIZE=n
Overrides the default buffer size used by the data transfer, where n is the number of bytes of memory to allocate (this is rounded up to a multiple of 512 bytes).

/BY_OWNER{=name}
Selects only files which match the specified user name or UIC. If the name is numeric, it is assumed to be a UIC. If the name is omitted, the default user is the one running the CUSP.

/CONFIRM
/NOCONFIRM(default)
Used to determine if the user is prompted before each append operation to confirm the operation. The following responses are valid (case is ignored and each of them can be abbreviated):

To confirm	To reject	To cancel	To confirm all
YES	NO	QUIT	ALL
TRUE	FALSE	Ctl-Z
1	0

Note that cancelling the operation will not undo any append operations that have already occurred.

/CONTIGUOUS
/NOCONTIGUOUS
Specifies whether a newly created output file is contiguous or not. By default, APPEND will create the output file in the same format as the first input file. If making the file contiguous fails due to a lack of contiguous space, it will be created noncontiguous.

/CREATED
Uses the file creation date when doing file comparisons using /BEFORE or /SINCE. This is incompatible with /BACKUP, /EXPIRED, and /MODIFIED.

/EXCLUDE=(filespec,{...})
Excludes the specified file(s) from the append operation. The parentheses are only required if multiple file specifications are used.

/EXPIRED
Uses the file expiration date when doing file comparisons using /BEFORE or /SINCE. This is incompatible with /BACKUP, /CREATED, and /MODIFIED.

/EXTENSION=bytes
Specifies the number bytes to be added to the output file each time the file is extended. Use of this switch implies the /NEW_VERSION switch, which is not required if this switch is used. The switch has no effect if the output file already exists.

/LOG
/NOLOG (default)
Controls whether or not each append operation is logged reported. If /LOG is used, the input and output files are displayed as well as the amount of data appended.

/MODIFIED
Uses the file expiration date when doing file comparisons using /BEFORE or /SINCE. This is incompatible with /BACKUP, /CREATED, and /EXPIRED.

/NEW_VERSION
/NONEW_VERSION (default)
Controls whether or not a new output file is created if it doesn't already exist. If the output file already exists, this switch is ignored.

/PROTECTION=(ownership{:access}{,...})
Specifies the file protection applied to the output file if it is created. "ownership" is one of the following:

Value	Meaning
S	System
O	Owner
G	Group
W	World

"access" is one or more of the following:

Value	Meaning
R	Read
W	Write
E	Execute
D	Delete

If not specified, the current default file protection is applied to the file.

/READ_CHECK
/NOREAD_CHECK (default)
/READ_CHECK causes APPEND to read the input file twice to ensure that is is read correctly.

/SINCE(=time)
Used to filter input files by date and time. The time specification can be an absolute or delta time, or one of the following keywords: BOOT, JOB_LOGIN, LOGIN, TODAY (default), TOMORROW, or YESTERDAY. This can be used in conjunction with /BACKUP, /CREATED, /EXPIRED, and /MODIFIED switches. If none of those switches are used, the file creation date is used.

/WRITE_CHECK
/NOWRITE_CHECK (default)
/WRITE_CHECK causes APPEND to read the output file after each write to verify that has been written correctly and to verify that the file can be read without error.

Examples

$ APPEND FILE1.DAT FILE.DAT

The contents of FILE1.DAT are appended at the logical end of FILE.DAT.

$ APPEND *.TXT FULL.TXT

The contents of all files with the .TXT extension are appended to the end of FULL.TXT. Note that if FULL.TXT already exists, it is excluded from the potential input files.

$ APPEND/EXCLUDE=A*.TXT *.TXT COMPLETE.SUM

The contents of all files with the .TXT extension are appended to the end of COMPLETE.TXT, except for any files whose name starts with "A".

$ APPEND 1.LIS,2.LIS,0.LIS FINAL.LIS

The contents of 1.LIS are appened to the end of FINAL.LIS, then the contents of 2.LIS are appended to the end of FINAL.LIS, and then 0.LIS is appened to the end of FINAL.LIS.

type TFile_Date_Match = ( Date_Backup, Date_Created, Date_Expired, Date_Modified ) ;
type TFile_Date_Mode = ( Date_None, Date_Before, Date_Since ) ;

const UIC_Default = 0 ;
const UIC_Input = -1 ;

// Returns false if the passed file exists and matches the filters.  If file
// does not exist or cannot be opened, function returns true.
//     S = File name
//     File_Date_Mode = How/if the file date is compared
//     File_Date_Match = Which file date is used for date comparison
//     Exclusions = List of file specifications that are excluded.  If the file
//       matches any of these, the function returns True.  If nil, there are no
//       exclusions.
//     Owner = Owner UIC to match against.  Ignored if 0.  Otherwise, the
//       function returns True if the owner doesn't match
//     Comparison_Time = Timestamp to compare to for before/since time
//       comparisons.
function Skip_File( S : string ; File_Date_Mode : TFile_Date_Mode = Date_None ;
    File_Date_Match : TFile_Date_Match = Date_Created ;
    Exclusions : TStringList = nil ; Owner : int64 = 0 ;
    Comparison_Time : int64 = 0 ) : boolean ;

var F : TCOM_UOS_File ;
    File_Time : int64 ;
    Index1 : integer ;

begin
    Result := False ;
    if( Exclusions <> nil ) then
    begin
        for Index1 := 0 to Exclusions.Count - 1 do
        begin
            if( Filename_Match( S, Exclusions[ Index1 ] ) ) then
            begin
                Result := True ;
                break ;
            end ;
        end ;
    end ;

    // Skip files not matching date criteria...
    F := Open_Binary_File( S, FAB_V_GET or FAB_V_SHRALL ) ;
    if( F = nil ) then
    begin
        Result := True ;
        exit ;
    end ;
    if( File_Date_Mode <> Date_None ) then
    begin
        case File_Date_Match of
            Date_Backup : File_Time := F.Last_Backup ;
            Date_Created : File_Time := F.Creation ;
            Date_Expired : File_Time := F.Expiration ;
            Date_Modified : File_Time := F.Last_Modified ;
        end ;
    end ;
    if( File_Date_Mode = Date_Since ) then
    begin
        if( File_Time < Comparison_Time ) then
        begin
            F.Free ;
            Result := True ;
            exit ;
        end ;
    end else
    if( File_Date_Mode = Date_Before ) then
    begin
        if( File_Time >= Comparison_Time ) then
        begin
            F.Free ;
            Result := True ;
            exit ;
        end ;
    end ;
    if( Owner <> 0 ) then
    begin
        if( Owner <> F.Owner ) then
        begin
            Result := True ;
        end ;
    end ;
    F.Free ;
end ; // Skip_File

function Translate_Time( S : string ) : int64 ;

var P : TPID ;

begin
    S := lowercase( S ) ;
    if( MinMatch( S, 'boot', 1 ) ) then
    begin
        S := Get_SYI( SYI_BOOTTIME, '', 0 ) ;
        move( PChar( S )[ 0 ], Result, sizeof( Result ) ) ;
    end else
    if( MinMatch( S, 'login', 1 ) ) then
    begin
        S := GetJPI( 0, JPI_LOGINTIM ) ;
        move( PChar( S )[ 0 ], Result, sizeof( Result ) ) ;
    end else
    if( MinMatch( S, 'job_login', 1 ) ) then
    begin
        P := Parent_PID( 0 ) ;
        S := GetJPI( P, JPI_LOGINTIM ) ;
        move( PChar( S )[ 0 ], Result, sizeof( Result ) ) ;
    end else
    begin
        Result := BINTIM( S ) ;
    end ;
end ;

function Read_File( F : TCOM_UOS_File ; Buffer : PChar ;
    Count : int64 ) : boolean ;

var Res : int64 ;

begin
    F.Blockread( Buffer[ 0 ], Count, Res ) ;
    if( Res < Count ) then
    begin
        Count := Res ;
    end ;
    Result := ( F.IO_Error = nil ) or ( F.IO_Error.Get_Error = 0 ) ;
end ;


function Write_File( Outfile : TCOM_UOS_File ; Buffer : PChar ;
    Count : int64 ) : boolean ;

var Res : int64 ;

begin
    Outfile.Blockwrite( Buffer[ 0 ], Count, Res ) ;
    if( Res < Count ) then
    begin
        Count := Res ;
    end ;
    Result := ( Outfile.IO_Error = nil ) or ( Outfile.IO_Error.Get_Error = 0 ) ;
end ;

procedure Copy_Stream( F, Outfile : TCOM_UOS_File ; In_Stream, Out_Stream : integer ) ;

var Buffer : array[ 0..1023 ] of byte ;
    Count, P, Size : int64 ;

begin
    P := 0 ;
    Size := F.Stream_Length( In_Stream ) ;
    while( Size > 0 ) do
    begin
        if( Size > sizeof( Buffer ) ) then
        begin
            Count := sizeof( Buffer ) ;
        end else
        begin
            Count := Size ;
        end ;
        Size := Size - Count ;
        F.Read_Stream( In_Stream, P, Count, int64( @Buffer ) ) ;
        OutFile.Write_Stream( Out_Stream, P, Count, int64( @Buffer ) )  ;
        P := P + Count ;
    end ;
end ;

// Compare filename F1 with wildcard specification F2
function Filename_Match( F1, F2 : string ) : boolean ;

var Node1, Access1, Node21, Device1, Path1, Name1, FType1, Version1 : string ;
    Node2, Access2, Node22, Device2, Path2, Name2, FType2, Version2 : string ;

begin
    Result := True ; // Assume match
    Parse_Filename( F1, Node1, Access1, Node21, Device1, Path1, Name1, FType1, Version1 ) ;
    Parse_Filename( F2, Node2, Access2, Node22, Device2, Path2, Name2, FType2, Version2 ) ;
    if( ( Node1 <> Node2 ) and ( not WC_Match( Node2, Node1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Access1 <> Access2 ) and ( not WC_Match( Access2, Access1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Node21 <> Node22 ) and ( not WC_Match( Node22, Node21, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Device1 <> Device2 ) and ( not WC_Match( Device2, Device1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Path1 <> Path2 ) and ( not WC_Match( Path2, Path1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Name1 <> Name2 ) and ( not WC_Match( Name2, Name1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( FType1 <> FType2 ) and ( not WC_Match( FType2, FType1, TT_UTF8 ) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
    if( ( Version1 <> Version2 ) and ( not WC_Match( Version2, Version1, TT_UTF8) ) ) then
    begin
        Result := False ;
        exit ;
    end ;
end ;

function Invalid_Wildcard( const S : string ) : boolean ;

begin
    Result := False ;
    if( S = '*' ) then // Only a wildcard
    begin
        exit ;
    end ;
    if( pos( '*', S ) = 0 ) then // No wildcard embedded
    begin
        exit ;
    end ;
    Result := True ; // Must be a wildcard embedded with other characters
end ;

// Open a file for output.  Parameters:
//    O = File spec
//    Protection = File protection for newly created files
//    Protection_Mask = Which bits of Protection are to be applied
//    Use_Protection = True to use Protection and Protection Mask, False to
//      ignore those parameters
//    OFlags = Open flags
//    Allocation_Size = Size to allocate for output file, ignored if 0
//    Symlink = True to create a symbolic link
//    Versions = True to assign new version to output file
//    Overlay = True to supersede existing file of same name
//    UIC = UIC of output file.  0 = default (current user), -1 = use UIC from
//      input file.  Cannot vary from current user unless process has privs.
//
// Returns file object on success or nil if error.
function Open_Output( O : string ; Protection, Protection_Mask : integer ;
    Use_Protection : boolean ; OFlags : integer ; Allocation_Size : int64 ;
    Versions : boolean = false ; Overlay : boolean = false ;
    UIC : int64 = 0 ) : TCOM_UOS_File ;

var A : int64 ;
    Buffer : array[ 0..1023 ] of byte ;
    D : TCOM_UOS_File ;
    DS : string ;
    Exists : boolean ; // True if file already exists
    FilePrivs : int64 ;
    O_Version : string ;
    Priv : integer ; // User privileges
    Res : int64 ;
    S : string ;
    This_Protection : int64 ;
    Node, Access, Secondary_Node, Device, Path, Name, Extension, Version : string ;

begin
    Result := nil ;
    S := GetJPI( 0, JPI_CURPRIV ) ;
    move( PChar( S )[ 0 ], Priv, sizeof( Priv ) ) ;
    Exists := File_Exists( O ) ;
    This_Protection := Protection and ( not Protection_Mask ) ;
    if( Exists ) then
    begin
        Result := Open_Binary_File( O, FAB_V_UFO ) ;
        if( Result = nil ) then
        begin
            exit ;
        end ;
        FilePrivs := Result.Privileges ;
        if( ( Priv and SETPRV ) = 0 ) then // No SETPRV privelege
        begin
            FilePrivs := FilePrivs and Priv ; // Limit file privileges to user's current privileges
        end ;
        Result.Free ;
        Result := nil ;
        if( ( Priv and ( BYPASS or SYSPRV ) ) <> 0 ) then
        begin
            Result := Open_Binary_File( O, FAB_V_UFO ) ;
            if( Result = nil ) then
            begin
                exit ;
            end ;
            if( UIC = UIC_Input ) then
            begin
                UIC := Result.Owner ;
            end ;
            Result.Free ;
            Result := nil ;
            Parse_Filename( O, Node, Access, Secondary_Node, Device, Path,
                Name, Extension, Version ) ;
            if( Versions ) then
            begin
                if( O_Version = '' ) then
                begin
                    O := O + ';1' ;
                end else
                begin
                    repeat
                        O_Version := ';' + inttostr( strtoint( copy( O_Version, 2, length( O_Version ) ) ) + 1 ) ;
                    until not File_Exists( O ) ; // Find non-existent version
                    O := Node + Access + Device + Path + Name + Extension + O_Version ;
                end ;
            end else
            begin
                DS := Node + Access + Device + Path ;
                D := Open_Binary_File( DS, 0 ) ;
                if( UIC = UIC_Input ) then
                begin
                    UIC := D.Owner ;
                end ;
                D.Free ;
            end ;
        end ; // if( ( Priv and ( BYPASS or SYSPRV ) ) <> 0 )
    end ; // if( Exists )
    if( Overlay ) then
    begin
        Result := Open_Binary_File( O, FAB_V_CIF or FAB_V_UFO or OFlags ) ;
    end else
    begin
        Result := Open_Binary_File( O, FAB_V_CIF or FAB_V_SUP or FAB_V_UFO or OFlags ) ;
    end ;
    if( Result = nil ) then
    begin
        exit ;
    end ;
    if( UIC > 0 ) then
    begin
        Result.Owner := UIC ;
    end ;
    if( Use_Protection ) then
    begin
        if( Exists ) then
        begin
            Result.Protection := This_Protection or ( Result.Protection and Protection_Mask ) ;
        end else
        begin
            Result.Protection := This_Protection or ( Get_Default_File_Protection( 0 ) and Protection_Mask ) ;
        end ;
    end ;
    if( ( Allocation_Size > 0 ) and ( Allocation_Size > Result.Get_Size ) ) then
    begin
        Result.Seek( Result.Get_Size ) ;
        A := Allocation_Size - Result.Get_Size ;
        fillchar( Buffer, sizeof( Buffer ), 0 ) ;
        while( A > 0 ) do
        begin
            if( A > sizeof( Buffer ) ) then
            begin
                Result.Blockwrite( Buffer, sizeof( Buffer ), Res ) ;
            end else
            begin
                Result.Blockwrite( Buffer, A, Res ) ;
            end ;
            A := A - sizeof( Buffer ) ;
        end ;
        Result.Seek( 0 ) ;
    end ;
end ; // Open_Output

In the next article, we will look at the code for the APPEND CUSP.