The File Processor and File System Management

In the previous article, we discussed the first job of the File Processr, which is device management. In this article we will discuss the second job of the File Processor: File System Management.
The method used to prepare a file system for use is the Mount method of that class. The FiP also has a mount method. Except in Init and the Kernel startup, the rest of UOS doesn't directly access file system objects - it accesses them through the FiP.

Historial note: why do we call the process of preparing a store for use "mounting"? The term comes from the ancient days of mainframe computers which had banks of tall magentic reel-to-reel tape drives. When someone needed to read or write data from/to a tape, they would request an operator to mount the tape for them. The tapes were put on spindles that stuck out from the wall of tape drives and the tape hung there like a picture on a wall. It is not hard to see how this was similar to mounting a photograph on the wall. Eventually, operators went away and the person who would have requested a tape to be mounted would do it themselves, but they still had to tell the operating system to prepare to use the tape. The term "mount" thus switched from being a request to an operator to a notice to the O/S. Later, (some) disks became removable and the term "mount" was expanded to encompass the preparation of any removable media for use. And "dismount" became the opposite operation. We will stick with these traditional terms even if they don't exactly fit modern technology.

The first order of business is the capability to mount a store's file system. This is done with the Mount method of FiP:

function TUOS_FiP.Mount( Device : PChar ; FS : TUOS_File_System ;
    Flags : longint ; Lab, Pass : PChar ) : TUnified_Exception ;

var D : TDevice ;
    Dummy : integer ;
    I : int64 ;
    Index : integer ;
    M : TUOS_Managed_Store ;
    Mount_Flags : longint ;
    N : string ;
    U : TUOS_String ;

begin // TUOS_FiP.Mount
    // Setup...
    Result := nil ;

    // Parse and normalize name...
    N := Device ;
    Dummy := pos( ':', N ) ;
    if( Dummy = 0 ) then // No terminating colon
    begin
        N := N + ':' ;
    end else
    begin
        N := copy( N, 1, Dummy ) ;
    end ;
    if( copy( N, 1, 1 ) <> '_' ) then
    begin
        N := '_' + N ;
    end ;

The first part of the mount process is to normalize the device name. Note that we do not call USC.Resolve_Symbol to handle a logical device. It is required that any request from the caller be run through that process before this routine is called, so we are assuming a physical device. The normalization process is to make sure the name conforms to the names in our device list - namely that it starts with an underscore and ends with a colon. We also handle the situation of a complete path or file specification being used by trimming off anything past the device name (after the colon).

    // Find device and see if already mounted...
    D := Get_Device( N ) ;
    if( D = nil ) then
    begin
        Result := Create_Error( UOSErr_Device_Not_Found ) ;
        exit ;
    end ;
    if( D.Info.Device_Type <> DT_Store ) then // Only stores can be mounted
    begin
        Result := Create_Error( UOSErr_Invalid_Operation ) ;
        exit ;
    end ;
    if( D.Mounted ) then
    begin
        Result := Create_Error( UOSErr_Already_Mounted ) ;
        exit ;
    end ;

Next, we retrieve the device from our device list. If it isn't found we return an error. Otherwise, we make sure the device is actually a store or we return an error. It makes no sense to mount a terminal device, for instance, since it is impossible to have a file system on such a device. Then we check to see if the device is already mounted. If so, we return an error.

    // Default to UOS file system if no file system passed...
    if( FS = nil ) then
    begin
        M := TUOS_Managed_Store.Create ;
        M.Store := D.Store ;
        FS := TUOS_Native_File_System.Create ;
        FS.Store := M ;
        I := 0 ;
        if( not FS.Valid_File_System( I ) ) then
        begin
            FS.Free ;
            M.Free ;
            Result := Create_Error( UOSErr_Device_Not_File_Structured ) ;
            exit ;
        end ;
    end ;

A file system instance can be passed to the Mouth method. If one is, it is used to access the store. Otherwise, we assume that the store contains a UOS file systems and we construct an instance of one. In the future, we will support other file systems and try to detect which one is on the store. For now, we are only dealing with the UOS file system. If a file system is passed in, we don't validate the file system. But if we create one, we make sure that the store contents are valid for the file system and, if not, return an error.

    // Try mounting the store...
    Mount_Flags := 0 ;
    if( ( Flags and Fip_Force ) <> 0 ) then
    begin
        Mount_Flags := UOSFSM_Force ;
    end ;
    if( FS <> nil ) then
    begin
        Result := FS.Mount( Lab, Pass, Mount_Flags ) ;
        if( Result <> nil ) then
        begin
            if(
                ( Result.Get_Error = UOS_File_System_Is_Dirty )
                and
                ( ( Flags and FiP_Rebuild ) <> 0 )
              ) then
            begin
                Result := Rebuild( Device, FS ) ;
                if( Result <> nil ) then
                begin
                    exit ;
                end ;
                Result := FS.Mount( Lab, Pass, Mount_Flags ) ;
                if( Result <> nil ) then
                begin
                    exit ;
                end ;
            end else
            begin
                exit ;
            end ;
        end ; // if( Result <> nil )
    end ; // if( FS <> nil )
    D.Mounted := True ;
    D.FS := FS ;

Next we tell the file system to mount the store. If we get an error about the file system being dirty, we return the error. Unless the caller passed in the FiP_Rebuild flag, in which case we call the Rebuild method and try to mount the file system again. If that fails, we return its error. If either file system mount works, we proceed on. We won't be covering the rebuild method in this article since all it does at present is call the file system's rebuild. In the future, we will be discussing performance optimizations - part of which will involve an expanded Rebuild method.

By default, mount uses the file system flags. But the caller can override the flags in the parameters. However, how can the routine tell if the caller is passing his own flags or wants to use the defaults? For instance, if 0 is passed for Flags, does that mean that the caller wants to use the defaults or that they desire to indicate all flags being clear? For that reason, the FiP_Flags value can be used to indicate that the Flags value is to be used in the place of the file system flags.

    // Determine mount flags...
    if( ( FS <> nil ) and ( ( Flags and FiP_Flags ) = 0 ) ) then // Not passing flags
    begin
        Flags := FS.Flags ; // Use flags from file system
    end ;

    if( ( Flags and UOSFSF_Private ) = 0 ) then // Mount as public
    begin
        if( D.Removable and ( ( Flags and FiP_Flags ) <> 0 ) ) then
        begin
            
        end else
        begin
            // Find first unused SYSn: logical...
            Index := 0 ;
            while( SSC.Get_Symbol( LNM_System, 0, PChar( 'SYS' + inttostr( Index ) + ':' ) ) <> nil ) do
            begin
                inc( Index ) ;
            end ;

            // Define new SYSn: logical...
            Set_Symbol( 'sys' + inttostr( Index ) + ':', Device ) ;

            // Add new logical to SYS: path...
            U := SSC.Get_Symbol( LNM_System, 0, '_sys:' ) ;
            if( U = nil ) then
            begin
                N := '_sys' + inttostr( Index ) + ':' ;
            end else
            begin
                N := U.Contents ;
                N := N + ';' + '_sys' + inttostr( Index ) + ':'
            end ;
            Set_Symbol( 'sys:', N ) ;
        end ;
    end ;
end ; // TUOS_FiP.Mount

The first few lines of code deal with the flags. What happens next depends upon if the store is being mounted publically or privately.
In article 13 we discussed the file system flags, one of which was UOSFSF_Private. As promised, we will now explain the meaning of that flag. If set, the store is treated as a private store. Otherwise it is treated as a public store. There are some exceptions that we will address in a future article, but most stores will be handled in the following standard manner.
Public stores take part in a pool of storage that is available to all users of the system. Private stores don't take part in this public pool. The public pool is referenced via the _SYS: logical. This logical contains a path containing all stores that are mounted as public. When a file is looked up, the FiP will search all stores in _SYS:. When a file is created, it is created on one of these stores (usually the one with the most free space). The consequence of this is that the public pool is treated as one large logical file system, even though each store in the public pool has its own physical file system. This means that every public store that is added to the system increases the size of the public pool - essentially expanding the size of your system disk. If you run low on space, just add a new disk and you don't need to worry about manually moving files between disks to make room for new or existing applications.
So, in the mount method, we add public stores to the _SYS: logical path. Note that we don't do this with removable devices, unless specifically told to do so. But the handling of removable media is a discussion for a future article. First we determine which _SYSn: logical to define for the store. We do this by setting the Index variable to 0 and then incrementing it each time we find a matching logical. When we fail to find one, we use that index to create the _SYSn: logical (where "n" is the index). Then we add that logical to the end of the contents of the existing _SYS: logical. If _SYS: isn't defined, then this is the first public store that is being mounted and we just set _SYS: to "_SYS0:".
You may recall that sys$system: is defined to point to the physical boot store, which means it doesn't take part in the public pool. This is intentional. Since only the boot store is available during kernel startup, it could be problematic to have some of the UOS files on other, as-of-yet-unmounted, stores. So, sys$system: is set and used to ensure that all UOS files are available during startup. Note, also, that the boot store becomes part of the public pool (as _SYS0:) by default. On single-disk systems, this is exactly what one would want. However, the system startup can change this if it is desired that the boot device be excluded from the public pool. Usually this is done when there is a small but fast device that is used exclusively for fast boots, and in similar circumstances. Another example would be the case when there is a local disk for general use but UOS is booted from a read-only store.

One might ask how UOS handles File systems that don't have a public/private flag. There are two ways to handle the situation. First, it is possible to include a "header" file on a foreign file system that UOS can use to determine if it is public/private. But even without that, all foreign file systems are assumed to be public. Of course, this is only the default handling - a file system can always be mounted as private. We will discuss these options in a future article.

Here is the local Set_Symbol function:

    procedure Set_Symbol( const S, S1 : string ) ;

    var Str : TString ;

    begin
        Str := TString.Create ;
        try
            Str.P := Pchar( S1 ) ;
            Str.Len := length( S1 ) ;
            SSC.Set_Symbol( LNM_System, 0, PChar( S ), Str ) ;
        finally
            Str.Free ;
        end ;
    end ;

This is similar to the local function called by the Kernel startup method. It simply creates the UOS string and passes it to the System Services component.

Spanned Sets vs UOS Public Stores
You may recall from our previous discussion of RAID, that there is a type of disk array that essentially concatenates several physical disk's into a single large virtual disk. Technically, this is known as a spanned set, and not a RAID.
You might think that a spanned set and a UOS public storage pool (or, simply, "public pool") are the same thing, but there are significant differences. A spanned set is treated as a single store with a single file system that spans multiple physical stores. Public pools consist of multiple stores which each have their own file system. They are treated as a single file system by the FiP. This has several important ramifications. First, a public pool is less susceptible to failures of individual stores. When a spanned set loses a store, it loses part of the file system for the entire spanned file system. If this happens to be the part of the spanned file system with the root folder or allocation table, then the whole file system is lost. If a store fails in a public pool, only that portion of the overall file system is lost - the rest remains available. The following diagram illustrates the difference between spanned sets and public pools.

Additional stores can be added to either spanned sets or public pools, but removing a store has the same effects mentioned above - whether removed intentionally or by a hardware failure. While it isn't recommended that stores be removed from public pools, it can be done without corrupting the whole pool (although it ought only be done when shutting down the system). Further, adding stores to spanned sets requires that the store be formatted for the file system used on the spanned set. But any file system that UOS recognizes can be added to the public pool without needing to be reformatted.
Because a spanned set logically concatenates the stores (the first cluster of the second store immediately follows the last cluster of the first store), a file can span multiple stores, even if contiguous. This allows files larger than any of the individual stores in the set. Public pools limit the size of files to the largest store in the pool. However, this restriction is compensated for by the UOS "spanned files" feature, which will be discussed in a future article.
Although UOS public pools are more reliable and flexible than spanned sets, there are a couple disadvantages as well. First, it is possible to have multiple files with the same name in a given directory if the file exists on more than one store. This won't happen under normal circumstances, but if the file is created on both stores before they are part of the public pool, one can end up with the duplicate. In such a case, the user would have to refer to the specific store - either by the physical device name or by the _SYSn: logical. To avoid the confusion that might result from this situation, it is recommended that all public stores be mounted before allowing general access to the system, and that otherwise only empty stores be added to an existing pool.
There is also a performance difference between spanned sets and UOS public pools - assuming identical performance characteristics of the stores used in both approaches. In the case of a file lookup on a spanned set, there needs to be but a single lookup. On a public pool, a lookup must happen in each file system that is part of the pool. This is only partly compensated for with caching (a topic for future article). However, a public pool has an advantage over a spanned set because while only a single I/O operation can occur at a time on a spanned set, multiple I/Os can happen on a public pool since each store can have an I/O operation concurrently.

The following is a comparison of the advantages/disadvantages between spanned sets and UOS public pools.

Advantage/Disadvantage Spanned set UOS Public Pool

Reliability X

Flexibility X

File System performance X

I/O performance X

No duplicate names X

Larger files X

There are other methods of the FiP related to file system management, but we're saving those for a general discussion of files in a future article. In the next article, we will discuss the remainder of the Kernel startup routine.