Hardware Abstraction and UOS Architecture

The HAL
With the completion of our File System class, it would be nice to be able to make use of it before we write all the rest of the operating system. In fact, it would be nice to check out all of our operating system code without having to have it bootable. Not to mention that it will be a while until UOS has enough support and utilities to allow development while it is running. So, we will create a UOS simulator that will allow us to test out the running UOS code. You may ask, "what is the use of simulated UOS?" But, in fact, it isn't UOS that is simulated - it is the hardware environment in which is operates. The UOS code is the actual operating system code.
So how do we simulate the hardware and how does that help us with "simulating" UOS?
You may recall that one of the design goals of UOS is for it to be able to run across many different hardware platforms. There are two ways to accomplish this goal. 1: we could completely rewrite much of UOS to support different hardware, or 2) we can isolate the hardware-specific code to a single class that is used by the rest of UOS. We will be taking the second approach. In order for the rest of the UOS code to operate without platform-specific changes, we must "abstract" away from the hardware. In essence, the class will provide a high-level interface to whatever hardware we are running on such that the rest of the code can operate without changes. We call this class the HAL. No, this is not a reference to the rogue computer in the movie 2001; it stands for "Hardware Abstraction Layer".
All hardware access is via the HAL, therefore the HAL is the only thing that needs to be re-written for different platforms. Any hardware platform that has a HAL written for it can run UOS. Of course, this requires that the the Pascal compiler we use can target that hardware. The Free Pascal compiler can target numerous types of hardware, so our use of Pascal provides a great number of potential target platforms that can run UOS. Our simulator provides a HAL that simulates hardware, and it provides a means of configuring that simulated hardware. But, using a HAL provides us with even more than the ability to target multiple hardware platforms. For instance, we can provide a virtualized UOS environment even on hardware that don't support machine virtualization. This is something that we will discuss in a future article.

How do we abstract the hardware in such a way that the HAL looks the same to UOS no matter what machine it is working on? First, we identify the possible hardware aspects that we could find on any computer system. In general, they are:

I/O Ports
Memory
Interrupts
Ring (protection) levels
Hardware devices
Numeric coprocessing

Second, we provide a non-hardware-specific interface that UOS can call to perform logical operations. For instance, we ask for a numeric coprocessor operation without having to know how it works (or even knowing if there is a coprocessor). The HAL maps the UOS requests into the platform-appropriate code.
Not every system will have every one of these hardware features. For instance, the PDP-11 hardware doesn't support I/O ports, but it does have memory-map ports. But the HAL will provide a unified interface regardless of the underlying hardware. We will not discuss every aspect of the HAL all at once. Rather, we will discuss various topics as they come up.

The UOS Architecture
Many times when people talk about an operating system, what they are really talking about is the "kernel". The Kernel is the core of an operating system. Beyond that, there is no official definition of the term. Some kernels out there are so stripped down in features that they would be better termed "micro Kernels". We will call the UOS kernel the "Executive" for reasons that will become clear later. But there is more to UOS than the executive. Here is a basic layout of the parts of UOS and its relation to the hardware and programs:

The UOS Executive consists of several components. The central component is what we call the UOS Kernel. It is the Kernel that interfaces the UOS features with the programs that run on UOS. The Kernel doesn't do much more than coordinate the other components in the Executive and interface with the HAL. One of the components is the File Processor, which handles all the UOS file and device I/O. There are other components which provide support for users, security, programming, etc.

Bootstrapping
There is one other part of UOS that isn't covered by the above diagram. To start up any operating system requires a small bit of code called a bootstrap. The BIOS (or its equivalent) loads the bootstrap from the boot device (usually a DVD, disk, or USB stick). The bootstrap, in turn, loads the operating system itself. Bootstraps do very little other than load a program into memory. In some cases, the bootstrap loads a larger bootstrap program which prepares for, and loads, the actual operating system. In windows, this secondary bootstrap is called the "Boot loader"; in RSTS/E it is called Init. In the case of UOS, we call our secondary loader the Init program.

When starting up a PC, the BIOS starts first and performs memory tests, determines device configuration, and boots the Operating System. If you enter a specific key code, you can enter the BIOS configuration to configure the hardware options. Init will perform an analogous operation for UOS. That is, Init will load UOS unless the user types something on the keyboard, in which case the Init prompt is shown and allows the user to configure UOS before loading it. Init also goes to the prompt if there is no UOS to load. Some examples of what Init will provide for the user is the ability to initialize and/or partition disks, install UOS from one device (such as an install DVD) to another (such as the hard disk), logically enable/disable devices, etc.

For now, the simulator will operate as if it boots directly from ROM. This is how many embedded systems operate. We will deal with booting from other devices later. Note that I will not bother covering any of the simulator code itself in these articles, although I'll provide the sources for it along with everything else - you can pretend that you are like UOS and all you know about is the HAL that is provided to you.

Init
Since the HAL is the interface to the hardware, Init will need an instance of the HAL before it can do anything. It is the job of the primary bootstrap to indicate where Init can find an instance of the HAL on the boot device, what the boot device is, and how large the instance of the HAL is. The HAL image must be a contiguous memory image so that Init can easily load it. For now, we will limit its understanding to that of a ROM boot only.

A simple tutorial on using the simulator: When you run the program, you will see a blank screen and a menu with two items: File and Test. Test contains options for running regression tests on our code, but I caution you, many of these tests go into a stress test that won't end until the program is killed - and you often have to kill the task from the task manager. Trust me, stick to the File menu.
The File menu has three options:

Boot: boot Init
Configure: Change the configuration of the simulated hardware
Exit: exit the program

Normally when you power-on a computer, it boots automatically. However, the simulator requires you to use File | Boot. That way, you can first alter what hardware is being simulated.
To configure the simulator, choose File | Configure. The dialog has tabs for different simulated hardware. At present, the Devices tab allows you to define the various devices and the Memory tab allows you to define how much memory is simulated. The default memory is 256 Mb. On the devices tab, you can define devices of type DISK or SERIAL (serial devices include terminals, keyboards/monitors, etc). There can be any number of controllers, each with any number of units. For example, disk controllers could include IDE, SATA, or SCSI disks. Serial controllers could include a single serial line controller, or a terminal multiplexer, or a PC motherboard's COM ports. A typical IDE controller could include four disks/CDROMS. In that case, each drive on the IDE is a different unit number (0 through 3). You can associate a file with a disk device - the file becomes a virtual disk. Note that the simulator stores the configuration and loads it the next time it starts.

There is a bit of a chicken-and-the-egg problem that we need to address. The HAL is the sole means of access to the hardware. But to load the HAL, Init needs to access the hardware directly. There are two possible ways of addressing this: 1) we could have the primary bootstrap load the HAL and the HAL loads Init, or 2) we could have the primary bootstrap load Init, which then loads the HAL. In the first case, the HAL would have to know about UOS (at least, in part). In the second case, Init would have to directly access the hardware instead of through the HAL. It might seem like it doesn't matter which approach we take, but this is where our coding philosophy comes to our aid. As you recall, we do not allow code at one level to know anything about the code that calls it. This means that that HAL shouldn't know anything about Init - including loading it. Our general philosophy trumps our desire to force Init to use the HAL. Fortunately, this minor indiscretion is limited to the initial load of the HAL. After it is loaded, all further interaction between the hardware and Init will be through the HAL.

In terms of actual code, Init is a function that is called with parameters that indicate the source of the HAL. Here is the function definition:

var HAL : THAL = nil ;

procedure Init( Typ, _HAL, Value : int64 ) ;

begin
    HAL := Get_HAL( Typ, _HAL, Value ) ;
    if( HAL = nil ) then // Couldn't get the HAL
    begin
        halt ;
    end ;

The primary bootstrap passes three items to the Init program after loading it into memory (unless it is already there, such as in ROM). The first item indicates the source of the HAL, the location of the HAL on that source, and a value that we reserve for future use (the bootstrap will set it to 0 for now). Usually the Value parameter will include a length value that indicates how large the HAL image is. For example, if the HAL is being loaded from disk, the location will be the offset on the disk, and the value will be the size of the image in bytes. Init passes these values to the Get_HAL function, which returns an instance of the HAL object. If Get_HAL returns nil, then it was unable to load the HAL and we halt since there is no way around this problem. Thus, we isolate the hardware-specific code of Init to a single function. The contents of that function must be customized for each hardware platform that UOS runs on.

Here is the Get_HAL function, minus hardware-specific code:

type TCreate_HAL = function() : THAL ;

function Get_HAL( Typ, _HAL, Value : int64 ) : THAL ;

var Creator : TCreate_HAL ;

begin
    case Typ of
        0 : // Firmware
            begin
                Creator := pointer( _HAL ) ;
                Result := Creator() ;
            end ;
        // Code to handle all other sources goes here
        else Result := nil ;
    end ; // case Type
end ;

A type of 0 indicates that we are loading the HAL from memory (probably a ROM). This is supported on all hardware. In that case, the _HAL parameter is the address of the code in memory. This must be a function as defined by the TCreate_HAL type. A call to this function returns an instance of a THAL object, which we then return to the Init function. If we don't recognize the source type, we return nil.

The first thing we will ask the HAL for is the console. The console is a "terminal" that is used during system startup. No other terminals are usuable in Init. In the old days, a terminal was a stand-alone video or printing mechanism with a keyboard. Typically they communicated via a serial interface over a cable that connected to the computer. The computer sends characters to display to the terminal and receives the characters typed on the keyboard. On a PC, the display is memory-mapped (that is, the computer writes directly to the video memory), although the keyboard is still a serial device. So, on a PC, the display and keyboard are separate devices. But we will treat them as a single logical terminal, which is the system console. On most PCs, this will be the only terminal. However, it is possible to connect other terminals via the COM ports. Also, some systems support multiple monitors and multiple keyboards via USB ports. Usually multiple monitors are used as a single larger logical monitor, and all keyboard input is treated as the same keyboard no matter how many keyboards are plugged in. In fact, this will be our default behavior in UOS. However, a user could tell UOS to treat each keyboard/monitor combination as a separate logical terminal. In any case, Init will treat the terminal as a character-oriented (non-graphical) device. The reason is that on non-PC systems, such as embedded systems, the console is likely a serial (or USB) port that a terminal can be plugged into. In these cases, the terminal may be a handheld device, or a tablet, or some such. In any case, Init knows nothing more about the console than that it is a serial I/O device. Because of this, the HAL class has a specific method that returns a TTerminal instance for the console. If there are no physical devices that can serve as a terminal, the Console method will return nil. This would only happen on an embedded system, although most will have some means of connecting a physical terminal to the computer.

type THAL = class
                public
                    function Console : TTerminal ; virtual ; stdcall ; abstract ;
                    function Heap : PSHAL_Heap ; virtual ; stdcall ; abstract ;
                    function Timestamp : int64 ; virtual ; stdcall ; abstract ;
            end ;

These are the first three HAL methods we will use. Console returns an instance of a TTerminal class the represents our console. Heap returns a simple heap manager. Timestamp returns a timestamp value that we can use to time the wait for the user's input.

Here is the portion of the TTerminal class that we will use in the initialization:

type TDevice = class
                   public // API...
                       procedure Cancel_Input ; virtual ; stdcall ; abstract ;
                       procedure Cancel_Output ; virtual ; stdcall ; abstract ;
                       function Pending_Input : boolean ;
                           virtual ; stdcall ; abstract ;
                       function Pending_Output : boolean ;
                            virtual ; stdcall ; abstract ;
               end ;

type TTerminal = class( TDevice )
                     public // API...
                         procedure Output( const S : string ) ;
                             virtual ; stdcall ; abstract ;
                 end ;

Note that TTerminal is a descendant of TDevice, as are all devices. Cancel_Input will clear any pending input from the device. Cancel_Output will clear any data waiting to be output to the device. Pending_Input returns true if there are any characters in the input buffer. Pending_Output returns true if the output buffer has any data in it. Output writes the contents of a string to the terminal.

Next in the Init function, we set up the heap:

    // Setup up heap...
    HAL_Heap := HAL.Heap ;
    SetMemoryManager( New_MM ) ;

As we've discussed in the past, we cannot use the default memory management routines because they call Windows functions, which don't exist. One advantage to Delphi/FPC is the ability to easily drop in our own heap management routines. The heap manager from the HAL is only intended for the use of the Init program - it isn't intended for use while UOS is running. The above lines reference this code:

var HAL_Heap : PHAL_Heap = nil ;


function NewGetMem( Size : Integer ) : Pointer ;

begin
    Result := HAL_Heap.getmem( Size ) ;
end ;


function NewFreeMem( P : Pointer ) : Integer ;

begin
    Result := 0 ;
    HAL_Heap.Free( P ) ;
end ;


function NewReallocMem( P : Pointer ; Size : Integer ) : Pointer ;

begin
    Result := HAL_Heap.Realloc( P, Size ) ;
end ;


const New_MM : TMemoryManager = (
                                   GetMem : NewGetMem ;
                                   FreeMem : NewFreeMem ;
                                   ReallocMem : NewReallocMem ;
                                ) ;

Back to the Init routine:

    // Get the console and see if the user wants the command interface...
    Console := HAL.Console ;
    if( Console <> nil ) then
    begin
        Console.Cancel_Input ;
        Console.Output( BEL+BEL ) ;
        Start := HAL.Timestamp ;
        while( HAL.Timestamp - Start < 20000000 ) do
        begin
            if( Console.Pending_Input ) then
            begin
                break ;
            end ;
        end ;
        if( not Console.Pending_Input ) then
        begin
            Boot ;
        end ;
        Console.Cancel_Input ; // Clear the user's typing

        Command_Mode ;
        halt ;
    end ;
    Boot ;
    halt ;
end ;

We request the console from the HAL. If nil is returned, we have no console and we will skip the following check. Otherwise, we make sure that any previous keystrokes are cleared from the input buffer, output two beeps (BEL is a constant for ASCII code 7, which is a "bell" or "beep"). On some systems/terminals, two consecutive beeps may sound like one long beep. The two beeps are the indication that Init is ready for the user to type something to request the command mode. We take the current timestamp and wait until two seconds have passed. Note that the timestamp is in 100 ns increments. Thus, 10000000 = 1 second. If the user types anything on the keyboard while we are waiting, we immediately exit the loop. Either way, after the wait loop we see if there are any keystrokes waiting. If not, we call our Boot function to autoboot. If we successfully boot UOS, the function will not return to us. Otherwise the boot failed (perhaps because UOS isn't installed) and we clear the console's input buffer before continuing to the command mode. Anything the user types from this point forward will be acted on. We then call our Command_Mode function. It shouldn't return, but if it does, we halt. If there is no console, we try to boot. If that fails, we halt. Note that we can display no message about this failure since there is no console to output the error to.

In the next article we will discuss the Init command interface and finally make use of our file system.