Preparing for execution

There are several things to take into consideration when it comes to executing a program on any operating system. On older OSes, only a single program could run at a time so running a program was nothing more than loading the file into memory and starting execution. Modern operating systems can run multiple programs simultaneously. If the CPU has multiple cores, the programs can literally run simultaneously. But if the CPU has only a single core (or simply has more programs to run than it has cores), then the operating system has to switch between the programs. We will discuss task switching in a future article. Right now, what concerns us is how loading a program in a multi-tasking environment affects memory management. Obviously, simultaneously running programs cannot occupy the same memory, so we need to keep track of where the program image is loaded into memory, and how much memory it requires.

A more subtle issue has to do with the shell (UCL being the default shell). The shell is nothing more than a program itself, but it can run other programs. But since a process only runs a single program at a time, how do we run both the shell and the program selected with the RUN command? There are any number of solutions, and we will cover a few of them here before we choose our approach.

1) We could terminate the shell, run the program, and then rerun the shell when the shell when the program terminates. This is a simple and straightforward approach, but there are consequences. UCL can execute programs at any point, including within loops and if/then blocks, these require context. Also recall that UCL has context in terms of open files, and multiple levels of command files. Thus, we would have to save that context before terminating the shell, then restore it all when we reopened the shell. Further, if a program has an exception, a UOS shell can provide an interactive environment where the user could fix the problem behind the exception and then continue the program, which means the program context has to be kept in these situations. Suddenly, this doesn't seem so simple. But it would work.

2) We could create a subprocess to run the program so that it runs in a process different than the the process running the shell. This is actually an elegant way to handle the situation. However, there are drawbacks to this approach as well. First, process creation is a relatively expensive operation and if being done in a loop in a command file, could significantly slow down a script, not to mention the system (due to the additional resources of new symbol tables, etc). Second, it counts additionally against user quotas since the subprocess quotas are counted in addition to the shell's process quotas. It is conceivable that a user could be unable to simply run a program if he used up his process quota. Third, because a subprocess has a different process ID, every time the user ran a program, it would run under a different PID - all of which are different than the shell's PID. If the user requested to see his PID, and then later tries to use it (for instance, to kill the process from another terminal), the PID he originally saw would not match the PID currently running the program that the user wishes to reference. This is both non-intuitive and unhelpful.

3) We could share the process ID for both the shell and the running program while saving the process contexts for both so that we could swap between them. Essentially, this is like the subprocess approach above, but we have two processes that take turns using the same PID. But just because we use the same PID as far as the user is concerned, we'd still have to have two processes internally that we'd have to keep track of somehow, so we're dealing with two processes for each process ID. This sounds needlessly complicated to me.

4) We could share the same memory for both the shell and the program images, they would simply be in different parts of the process address space. In fact, this is how some operating systems worked in the past. The drawback here is that a malicious - or merely buggy - program would have access to the image of the shell allowing it to modify it or run it at arbitrary addresses, which could corrupt the shell context or make it completely fail. Needless to say, we don't want errant programs being able to blow up the shell.

So how does VMS handle the situation? Honestly, I have no idea. This is an internal implementation detail that I don't know has been publically documented anywhere. Not that UOS has to operate like VMS under the hood. We are using the VMS specification as a way of determining the way it appears to the programmer/user. We aren't slavishly following in their algorithmic path. As mentioned more than once before, some of the ways they did things are problmatic and we intentionally don't follow those VMS examples. So, in this case we will make a choice based on what makes most sense for UOS.

We will go with something close to approach 3. But didn't I say that was needlessly complex? Fortunately, there is a feature of UOS that helps simplify the situation. Remember that UOS segregates itsself into multiple rings. Ring 0 is for the executive. Ring 1 is for drivers. Ring 3 is for user programs. What about ring 2? As it turns out, ring 2 (also referred to as the Supervisor ring, or Supervisor Mode) works very well as a place to run shells. Rings 0 and 1 run in the context of the executive and do not execute in the context of a process, although they perform services on behalf of processes. Rings 2 and 3 are reserved for use by the user. Processes can run program in both rings, although ring 2 is reserved for shells and utility code. Because the memory and resources are separated between the rings, we essentially have two processes that can operate within the context of a single PID. There is no need to do context switching other than what is normally done to switch between rings. Because the rings are separated by the hardware, we don't have to worry about a user program corrupting the shell. Granted, on a CPU without hardware support for rings, the program could still corrupt the shell, but it could also corrupt the executive. Obviously, we can't help in that situation.

As a reminder, although there are similarities between UOS and VMS in terms of rings, we are not following VMS in this area. For instance, VMS drivers are part of ring 0. Likewise, use of Supervisor mode in VMS differs from UOS.

        UOS_FIP_Run:
            begin
                UE := Enter_System_Call( Request, SReq, PID, MMC,
                    sizeof( S2I1_Request ) - sizeof( SReq ), Address ) ;
                if( UE <> nil ) then
                begin
                    Set_Last_Error( UE ) ;
                    exit ;
                end ;
                try
                    S2I1_Request := PString2I1_Request( Address ) ;

                    Status := Run_Program( S2I1_Request.String1,
                        S2I1_Request.String2, S2I1_Request.Integer1, IOSB ) ;
                    Write_User_int64( Kernel, PID, S2I1_Request.Request.Status,
                        IOSB.r_io_64.r_bcnt_32.l_bcnt ) ;
                finally
                    Exit_System_Call( integer( S2I1_Request ), PID, MMC,
                        sizeof( S2I1_Request ) - sizeof( SReq ) ) ;
                end ;
            end ;

This code is added the the kernel's API routine. This resembles other code in the routine such that I won't go into detail about it.

function TUOS_FiP.Run_Program( _Name, _Command : TSRB ; Flags : integer ;
    var IOSB : TIOSB ) : int64 ;

var Command, Name : string ;
    Handle : int64 ;
    Info : TUOS_File_Info ;
    S : string ;
    Size : int64 ;
    _WS_MAX : int64 ;

begin
    // Get parameters...
    Name := Get_String( Kernel, Kernel.PID, _Name, IOSB.r_io_64.w_status ) ;
    if( IOSB.r_io_64.w_status <> 0 ) then
    begin
        exit ;
    end ;
    Command := Get_String( Kernel, Kernel.PID, _Command, IOSB.r_io_64.w_status ) ;
    if( IOSB.r_io_64.w_status <> 0 ) then
    begin
        exit ;
    end ;
    if( not _File_Exists( Kernel.PID, Name, @Info ) ) then
    begin
        IOSB.r_io_64.w_status := UOSErr_File_Not_Found ;
        Generate_Exception( IOSB.r_io_64.w_status ) ;
        Result := IOSB.r_io_64.w_status ;
        exit ;
    end ;

    USC.Run( Kernel.PID, PChar( Name ), PChar( Command ), Info.Privileges, False ) ;
end ; // TUOS_FiP.Run_Program

This new method simply unpacks the passed parameters and calls the Run method of the USC.

function TUOS_FiP.Get_String( Kernel : TUOS_Kernel ; PID : TPID ; SRB : TSRB ;
    var status : longint ) : string ;

var US : TUOS_String ;

begin
    Result := '' ;
    US := Get_User_String( Kernel, PID, SRB, status ) ;
    if( status <> 0 ) then
    begin
        US.Free ;
        exit ;
    end ;
    Result := US.Contents ;
    US.Free ;
end ;

This wraper function simply makes obtaining strings from the user space easier.

function TUSC.Run( PID : cardinal ; _Executable, Command_Line : PAnsiChar ;
    Privileges : int64 ; Supervisor : boolean = False ) : TUnified_Exception ;

var B : PBTMemoryModule ;
    Buffer : array[ 0..1023 ] of byte ;
    Dummy, Index : integer ;
    F : TUOS_File ;
    MapA : int64 ;
    Process : TProcess ;
    Ring : integer ;
    S : Ansistring ;
    Size : int64 ;
    _WS_MAX : int64 ;

begin
    // Setup...
    Result := nil ;
    Set_Last_Error( nil ) ;
    Process := Get_Process( PID ) ;
    if( Process = nil ) then // Process doesn't exist
    begin
        Generate_Exception( UOSErr_Nonexistent_Process ) ;
        exit ;
    end ;

We've covered the Run method in the past, but we've changed the routine so much that it bears a complete review including the updated code. We start by obtaining the process and exiting if there is an error. The privileges parameter indicates the privileges to assign to the image while it executes. This is typically taken from the privilege field of the file. The optional Supervisor parameter indicates if the image should execute within ring 2 (supervisor). If false, the image executes in ring 3 (user).

    // Verify that executable exists...
    F := FiP.Open_File( PID, _Executable, FM_RW ) ;
    if( F = nil ) then
    begin
        Set_Last_Error( FiP.Last_Error ) ;
        exit ;
    end ;
    S := _Executable ;

    // Update process executable name...
    if( Supervisor ) then
    begin
        Process._Executable[ 0 ] := S ;
    end else
    begin
        Process._Executable[ 1 ] := S ;
    end ;

We open the indicated file, exiting on error. Then we set the appropriate index of the _Executable array.

    // Set the image name...
    if( Process.Shell = '' ) then
    begin
        Process.Shell := S ;
    end ;
    Dummy := length( S ) ;
    while( ( Dummy > 0 ) and ( pos( S[ Dummy ], ':\' ) = 0 ) ) do
    begin
        dec( Dummy ) ;
    end ;
    Process.Image_Name := copy( S, Dummy + 1, length( S ) ) ;

If no shell is set for the process, we set it to this program, since the first program for a process should be the shell. Shell should only be null if the the user associated with this process has no shell defined for it. Next we trim all but the file name from the executable string and assign it to the process' image name.

    // Get first 1K of the file...
    if( F.Read( 0, 0, 1024, Buffer, 0 ) <> 1024 ) then
    begin
        Set_Last_Error( FiP.Last_Error ) ;
        exit ;
    end ;

    // Validate the program...
    if( not BTMemoryValid_Image( @Buffer ) ) then
    begin
        F.Detach ;
        Result := Generate_Exception( UOSErr_Invalid_File_Format ) ;
        exit ;
    end ;

    // Check against quotas and memory...
    Size := BTMemoryImage_Size( @Buffer ) ;
    _WS_MAX := Kernel.Parameter( WS_MAX ) * HAL.RAM_Page_Size ;
    if( ( Size > _WS_MAX ) and ( _WS_MAX > 0 ) ) then // No program can exceed WS_MAX, if set
    begin
        F.Detach ;
        Result := Generate_Exception( UOSErr_Quota_Exceeded ) ;
        exit ;
    end ;

Next, we do a quick sanity check on the target executable. There are some old executable file formats that are essentially memory images that can simply be loaded into memory, as is, and then executed. However, modern executable files are more complex and the memory image must be constructed from the meta information in the file. Most executable files, including those UOS that runs natively, contain identifying information at the beginning of the file. This should all be well within the first 1K bytes of the file, so we read in the first 1024 bytes and then perform a couple of checks on it to make sure it is valid and that the image size fits within the value defined by the system's WS_MAX parameter. We will talk about system parameters in the future. For now, just know that WS_MAX is a value indicating the maximum number of pages of memory that any process can use for a program (a value of 0 means there is no limit). If the program image exceeds that, we exit with a quota exceeded error. Note that we convert from WS_MAX pages to bytes for the comparison. RAM page size is CPU-dependent, so we have to ask the HAL for that value.

Note that we make a couple calls to BTMemory* routines. These are part of a public domain package used to load images into memory for Windows. We've made a copy of that and modified it to work with UOS. We won't be going into that code since is it very specific to the "PE Executable" file format and is beyond the scope of what I want to be covering in these articles. Feel free to look in the PE Excecutable file format or obtain the BTMemory source files and peruse them at your leisure. We made reference to these routines in the past, where we used them to load the UOS components.

    if( Supervisor ) then
    begin
        Ring := 2 ;
        Index := 10 ;
    end else
    begin
        Ring := 3 ;
        Index := -1 ;
    end ;

    // Rundown current image, if any...
    Terminate_Image( PID, Supervisor ) ;

Next we determine which ring we will execute in, based on the Supervisor parameter, and we set a flag for the purposes of any simulator that might be running UOS. Then we terminate the current image (covered later in this article).

    // Load image...
    MapA := Kernel.MMC.Map_File( 0, F ) ;
    if( MapA = 0 ) then // Couldn't map the file
    begin
        F.Detach ;
        Result := Generate_Exception( UOSErr_Exhausted_Resource ) ;
        exit ;
    end ;
    B := BTMemoryLoadLibary( Kernel, pointer( MapA ), Size ) ;
    if( B = nil ) then
    begin
        Kernel.MMC.Unmap_File( PID, MapA, F ) ;
        F.Detach ;
        Result := Generate_Exception( UOSErr_Image_Load_Failure ) ;
        exit ;
    end ;

    Kernel.MMC.Unmap_File( PID, MapA, F ) ;
    F.Detach ;

Now we load the file. In order to do this, we will be using the BTMemory routines, as described above. These routines require that the entire executable file be mapped into memory for processing. As it processes the file, it will create a memory image of the exectuable that can be run by the CPU. Unfortunately, on non-paged systems, that means that there will essentially be two copies of the executable in memory as the BTMemoryLoadLibrary routine loads the image. This means that we cannot load a program that is larger than half of the available memory on non-paged systems. After the image is loaded, we release the mapped file. If the loading fails, we exit with an error. In any case, we close the file.

    // Execute the file...
    Process.Mode := Ring ;
    Process.Command_Line := Command_Line ;
    Process.Current_Privileges := Process.Current_Privileges or Privileges ;
    Process.State := SCH_C_COM ; // Computable
    Process.HAL_Context[ Ring ] := 
        HAL.Execute( Process.HAL_Context[ Ring ], B.headers.OptionalHeader.AddressOfEntryPoint, 
        Index, Ring ) ;
end ; // TUSC.Run

Finally, we set the process' mode, update the command line, add in image privileges to the current process privileges, set the state to computable (SCH_C_COM) so the program will begin executing as soon as the scheduler is ready to run it, and then get an execution context from the HAL. This context is used to manage task switching. The address we pass to the HAL is the starting execution address as determined by BTMemoryLoadLibrary from the executable file.

function TUSC.Terminate_Image( PID : cardinal ; Supervisor : boolean = False ) : TUnified_Exception ;

var AST : TAST ;
    Dummy : integer ;
    H : THandle ;
    I : integer ;
    Mode : byte ;
    Process : TProcess ;
    Ring : integer ;
    S : string ;

begin
    // Setup...
    Result := nil ;
    Set_Last_Error( nil ) ;
    Process := Get_Process( PID ) ;
    if( Process = nil ) then // Process doesn't exist
    begin
        Generate_Exception( UOSErr_Nonexistent_Process ) ;
        exit ;
    end ;
    if( Supervisor ) then
    begin
        Ring := Ring_Supervisor ;
    end else
    begin
        Ring := Ring_User ;
    end ;
    Process.State := SCH_C_LOAD ;
    Mode := Process_Mode( PID ) ;

This method is used to end the currently running image (if any). This is also known as "image rundown". We change the process state to loading/unloading (SCH_C_LOAD) which tells the scheduler that the process is not yet ready to run. We determine which ring to affect, given the Supervisor parameter.

    // Release resources...
    Kernel.MMC.Release_Allocation( PID, '*', 0, Ring ) ; // Release all memory
    Process.Command_Line := '' ;
    for I := Process._Handles.Count - 1 downto 0 do
    begin
        H := Process._Handles[ I ] ;
        if(
            ( H <> Process._Sys_Input )
            and
            ( H <> Process._Sys_Output )
            and
            ( H <> Process._Sys_Command )
            and
            ( H <> Process._Sys_Error )
            and
            ( TResource( H ).Mode = Mode )
          ) then
        begin
            Kernel.FIP.Close_Handle( H ) ;
        end ;
    end ;
    for I := Process.ASTs.Count - 1 downto 0 do
    begin
        AST := TAST( Process.ASTs[ I ] ) ;
        if( AST.Mode = Mode ) then
        begin
            Process.Remove_AST( AST.AST_Type, AST.Context ) ;
        end ;
    end ;

First, we free all resources associated with the image. Note that we skip the sys$input, sys$output, sys$command, and sts$error handles, since these persist across image startup/rundown and are shared between rings 2 and 3.

    // Terminate image...
    Kernel.Terminate( PID, Ring ) ;
    HAL.Terminate( Process.HAL_Context[ Ring ], Ring ) ;
    Mode := Ring_Supervisor ;

    // Restore process privileges...
    Process.Current_Privileges := Process.Authorized_Privileges ;
    
    Process.Image_Name := Extract_Filename( Process.Shell ) ;
end ; // TUSC.Terminate_Image

Next we notify the kernel of the image termination, and also the HAL. Whether the image being terminated is in the user or supervisor ring, we want to set the mode to supervisor. Then we set the privileges back to the authorized privileges (which is what the privileges were before the image's privileges were added to the current set of process privileges). Lastly, we make sure the process image name is set to that of the shell.

In the next article, we'll look at the next UCL command and the CUSP that it runs.