Symbol Substitution

In the previous article we described the UCL symbol substitution feature. In this article, we will examine the source code that implements it. We want to follow the VMS specification as closely as possible at this level. Unfortunately, the VMS documentation is somewhat vague as to the DCL parsing algorithm. In fact, I did some experiments with DCL and found that, under some conditions, the behavior of DCL was inconsistent with the documentation. Which is wrong? The documentation or the implementation? I have no idea. I have to confess some disappointment with the - otherwise very good - VMS documentation. For instance, consider the following situation:

A = "1"

B = "'A'"

C = "'"

Given the above symbol definitions, what should be the operation of the following assignment?

D = "'C'"

Symbol D ought to contain a single apostrophe ('), which it does. So far so good. Likewise, the following:
D='B'
results in D containing "1" due to iterative substitution of B. But now let's consider this assignment:
D:='C''B'
There are two possible ways to interpret how this would be parsed. The first way would be a simple substitution of 'C' and then 'B'. Since B would be iteratively substituted to 'A' and then to "1", D would contain the following:
'1
The other way to interpret the parsing would be to substitute 'C' which is an apostrophe, then use that as part of the following substitution (essentially an intermediate value of D=''B') resulting in a non-iterative substitution of B and resulting in D having the value:
'A'
But what result do we actually get from DCL? Symbol D has the following value:
B
Good grief! This simply doesn't correspond to anything in the documentation. I've tried other odd combinations, such as:
D="''C'''B'"
which resulted in D containing:
B'
Which only makes sense if the terminating apostrophe for C was also considered the first part of three apostrophes (it plus the following two). Now, there were some examples in the VMS literature that seemed to indicate that this was the intended behavior, but they might also have been typos. In the absence of any DCL documentation about this situation, we are left scratching our heads.

I could provide some other odd examples, where DCL behavior appears decidedly non-deterministic. But, frankly, I don't have the time to try to reverse engineer the DCL parsing - and I'm not sure that I'd want to. All we can do is aim at being the most compatible with the documentation as we can be, and with the actual behavior when it provides consistent insight into the interpretation of the documentation.

We could handle the issue of substitution a couple of ways. We could make several passes through the command line, substituting symbols as we come across them and repeating until no more substitutions occur. Or we could make a single pass through the command line, doing subsitutions as we encounter them. Personally, I'd prefer the first since that is somewhat cleaner code to implement. However, I think that the second approach provides a result that is closer to the observed behavior and to a strict reading of the documentation. UCL might not be 100% compatible with DCL, but it is close and it is consistent with the documentation.

function Parse( S : string ) : string ;

var C : char ;
    E, I : integer ;
    In_Quotes : boolean ;
    Single_Apostrophe : boolean ;
    Name : string ;

begin
    // Phase I substitution...
    In_Quotes := False ;
    Result := '' ;
    I := 1 ;
    while( I <= length( S ) ) do
    begin
        C := S[ I ] ; // Get next character

We start with phase 1 subsitution, obviously. First, we clear the result - which we will build as we process through the passed string. We start at the first string offset, and then loop until our index (I) exceeds the end of the string. The first thing we do in the phase 1 substitution loop is get the character at the current index.

        if( C = '"' ) then
        begin
            if( copy( S, I + 1, 1 ) = '"' ) then // Two quotes = single literal quote
            begin
               inc( I ) ;
               Result := Result + '"' ;
            end else
            begin
                In_Quotes := not In_Quotes ;
            end ;
            Result := Result + '"' ;
        end else

If the character is a quote ("), we toggle the In_Quotes flag, and add the quote to the result string. Note that two quotes together indicate a single literal quote character, thus they do not change the state of In_Quotes.

        if( C = #39 ) then // Apostrophe (substitution)
        begin
            Single_Apostrophe := ( copy( S, I + 1, 1 ) <> #39 ) ;
            if( not Single_Apostrophe ) then
            begin
                inc( I ) ;
            end ;

If the current character is an apostrophe, we are possibly processing a symbol substitution. As we discussed in the previous article, apostrophes delimit symbol names, but within quotes, the symbol name must be preceeded by two apostrophes. Single_Apostrophe is true if only a single apostrophe is found and false if there are multiples. If there are two apostrophes, we move the index past the first one.

            E := PosEx( #39, S, I + 1 ) ;
            if(
                ( E = 0  )
                or 
                (
                  ( E > PosEx( '"', S, I + 1 )
                  and
                  In_Quotes
                ) 
              ) then // No end quote to the name, must not be one
            begin
                Result := Result + #39 ; // Treat as literal
                if( not Single_Apostrophe ) then
                begin
                    Result := Result + #39 ;
                end ;
                inc( I ) ;
                continue ;
            end ;
            if( In_Quotes ) then
            begin
                if( Single_Apostrophe ) then // Symbol translation within strings must use 2 apostrophes
                begin
                    Result := Result + #39 ;
                    inc( I ) ;
                    continue ;
                end ;
            end ;

Once we've found the symbol name, we look for the terminating apostrophe. If there is none (or the next one is outside of the quote we are in), we know that this is not a symbol. In that case, we add the apostrophe to the result and increment the index. Within quotes, we make sure that there are two apostrophes.

            Name := copy( S, I + 1, E - I - 1 ) ; // Get symbol name
            I := E ;
            Name := LIB_Get_Symbol( Name ) ;
            if( Single_Apostrophe ) then
            begin
                // Iteratively substitute...
                while(
                       ( copy( Name, 1, 1 ) = #39 )
                       and
                       ( copy( Name, length( Name ), 1 ) = #39 )
                     ) do
                begin
                    Name := LIB_Get_Symbol( Name ) ;
                end ;
            end ;
            Result := Result + Name ;
        end else

Next we extract the name from the string and move the index to the end of the name. Then we translate the symbol's name to a value. Note that if two apostrophes in a row are found (or three in a row within quotes), that is parsed as a null name, which translates to a null string.

        begin
            Result := Result + C ;
        end ; // if( C = #39 )
        inc( I ) ;
    end ; // while I <= length( S )

Next we process any character which is not a quote or an apostrophe simply by adding it to the result. Finally we increment the index and loop back to process the next character. When we reach the end of the string, we exit the loop.

Now we are ready for the phase II substitution.

    // Phase II substitution...
    In_Quotes := False ;
    S := Result ;
    Result := '' ;
    I := 1 ;
    while( I <= length( S ) ) do
    begin
        C := S[ I ] ;
        if( C = '"' ) then
        begin
            In_Quotes := not In_Quotes ;
            Result := Result + '"' ;
        end else
        if(
            ( C = '&' )
            and
            ( pos( copy( S, I - 1, 1 ), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789$_-' ) = 0 )
            and
            ( not In_Quotes )
          ) then // Substitution
        begin
            E := I + 1 ;
            while(
                   ( E <= length( S ) )
                   and
                   ( pos( copy( S, I - 1, 1 ), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789$_-' ) = 0 )
                 ) do
            begin
                inc( E ) ;
            end ;
            Name := copy( S, I + 1, E - I - 1 ) ;
            I := E ;
            Name := LIB_Get_Symbol( Name ) ;
            Result := Result + Name ;
        end else
        begin
            Result := Result + C ;
        end ; // if( C = #39 )
        inc( I ) ;
    end ; // while I <= length( S )
end ; // Parse

Similar to the phase 1 substitution loop, we go through the string, using I as an index into the string. We grab the current character. We check for a quote and set our In_Quotes appropriately. The other special case is the amperand (&), which is our substitution indicator. Finally, any other character is copied to the result. Then we increment the index and loop back.
For the ampersand substitution, we check to make sure we are not inside quotes, and that the preceeding character wasn't one a valid symbol name character. If either of those is true, the ampersand is treated as any other character. Otherwise, we look for the end of the symbol name, which will terminate at the first character that doesn't match the list of symbol name characters. We then grab that symbol name, translate the name to the symbol value, and set the index to the last character of the symbol name so that the end-of-loop index increment will move us to the character after the symbol name. When the loop is done, we return the string with substitutions to the caller.

In the next article, we will look at the Process procedure, which handles the execution of UCL scripts.