ANS Forth extension proposals etc.

<title>Buffering clarification</title>

<h3><a name="buffering">Buffering clarification</a></h3>

[<a href="proposals.html">Other proposals</a>]

<h4>Problem</h4>

The standard does not specify whether buffering is allowed for words

dealing with the user input and output devices, and the file

words. The existence of FLUSH-FILE indicates that buffering is allowed

to some extent.

<h4>Proposal</h4>

No buffering is allowed for the user input device. No buffering is on

the user output device, if it is a terminal. Buffering is allowed for

the user output device, if it is no terminal (e.g., a file), and for

the file wordset. The amount of buffering is system-defined.

<pre>

OUTFILE-ID ( -- file-id ) file-ext

</pre>

the file-id for the current user output device

<h4>Remarks</h4>

Allowing buffering for non-terminal user output device may increase

the performance for filters.

<p>Flushing the user output device can be achieved with

<pre>

OUTFILE-ID FLUSH-FILE ... 

</pre>

<h4>Experience</h4>

Gforth <=0.3.0 used no buffering for KEY, line buffering for

outputting to a terminal, and block buffering for outputting on

non-terminals. Several users complained about the buffering for the

user output device (they used a terminal), so starting with 0.4.0 we

use no buffering for the user output device if it is a

terminal. Nobody has complained about the file buffering.

<p>Gforth has implemented OUTFILE-ID since 0.2, and it is in use

(mainly internally).

<p>Experiments with a filter we had on Gforth under Linux showed no

significant performance advantage for buffering the user output

device.  We should repeat this experiment with a more I/O-intensive

filter (the one we measured produced only 94KB of output).

<p>Another experiment with Gforth under Linux-Alpha resulted in a

worst-case slow-down factor of 25 for turning off buffering. The

benchmark used in this experiment was:

<pre>

: foo

    10000000 0 +do

        [char] x emit

    loop ;

</pre>

<hr><a href="/anton/">Anton Ertl</a>

<title>CONST-DOES&gt; proposal</title>

<h3><a name="const-does">CONST-DOES&gt;</a></h3>

[<a href="proposals.html">Other proposals</a>]

<h4>Problem</h4>

Many uses of <code>create</code>...<code>does&gt;</code> are just for

shifting data from the <code>create</code> time to the execution time

of the code after <code>does&gt;</code>; i.e., after the word is fully

defined, the data remains constant.  A prototypical example of this

use is the definition

<pre>

: constant ( n "name" -- )

  create ,

does> ( -- n )

  @ ;

42 constant answer

</pre>

Here, <em>n</em> is just shifted from <code>create</code> time to

<em>name</em> execution time.

<p>It would be nice if a native-code compiler could optimize a use of

<code>answer</code> in the same way that it would optimize a use of

<code>42</code>.  However, this is not possible, because the compiler

has to consider the following possibility:

<pre>

5 ' answer >body !

</pre>

I.e., the data in a

<code>create</code>...<code>does&gt;</code>-defined word can change at

almost any time.  So at best a compiler can compile

<code>answer</code> to the same code as <code>[ ' answer &gt;body ]

literal @</code>.

<p>The effects of this difference on the resulting code depend on the

context.  E.g., consider the code <code>answer cells + @</code>: If

the compiler could optimize <code>answer</code> to <code>42</code>, it

could compile this sequence to one instruction on the MIPS

architecture:

<pre>

lw        v0,168(a0) ; 42 cells + @

</pre>

Without this optimization, it needs at least five instructions:

<pre>

lui       v1,...    ; [ ' answer >body ] literal

lw        v0,...(v1); @

sll       v0,v0,2   ; cells

addu      v0,v0,a0  ; +

lw        v0,0(v0)  ; @

</pre>

So, the problem is how to communicate to the compiler that the data in

<code>answer</code> will not change.

<h4>Proposal</h4>

(See <em>Remarks</em> for an informal, easier-to-understand

description.)

<p>

<code>CONST-DOES&gt;</code> ``const-does'' core

<dl>

<dt>Interpretation:<dd> Interpretation semantics for this word are undefined.

<dt>Compilation:<dd>( C: colon-sys1 -- colon-sys2 )

<p>Append the run-time semantics below to the current definition. Whether

or not the current definition is rendered findable in the dictionary

by the compilation of <code>const-does&gt;</code> is implementation defined.

<dt>Run-time:<dd> ( u1*x u2*r u1 u2 ``name'' R: nest-sys1 -- )

<p>Create a word <em>name</em> with execution semantics given below.

Return control to the calling definition specified by

<em>nest-sys1</em>.  The <em>u1</em> cells and <em>u2</em> floats can

be interleaved in any order.

<dt><em>name</em> execution:<dd> ( ... -- ... ) 

<p>Perform initiation semantics below.  Transfer control to the code

right after the <code>const-does&gt;</code>.

<dt>Initiation:<dd> ( -- u1*x u2*r R: nest-sys2 )

<p>Save information <em>next-sys2</em> about the calling definition.

After pushing the <em>u1</em> cells and <em>u2</em> floats, they are

in the same order as they were at the start of the run-time semantics.

</dl>

<h4>Typical use</h4>

<pre>

: constant ( n "name" -- )

1 0 const-does> ( -- n )

  ;

: fconstant ( r "name" -- )

0 1 const-does> ( -- r )

  ;

: simple-field ( n "name" -- )

1 0 const-does> ( addr1 -- addr2 )

  + ;

</pre>

Note that the stack comments after <code>const-does&gt;</code> reflect the

total stack effect of <em>name</em> (including initiation semantics),

not the stack effect of the following code.

<h4>Remarks</h4>

The ANS-Forth-style formal proposal may be a bit hard to penetrate, so

here are the essentials: <code>Const-does&gt;</code> defines a word

(the role of <code>create</code>) and its behaviour (the role of

<code>does&gt;</code>).  The main other thing it does is to shift

<em>u1</em> cells and <em>u2</em> floats from the definition time of

<em>name</em> to its execution time.  As a consequence, a simple

definition like <code>constant</code> specifies just how many cells

and floats it wants to shift, and needs to do nothing else.

<p>Note that this works for both separate and combined data/FP stacks: On

a system with separate stacks <code>const-does&gt;</code> shifts <em>u1</em>

cells and <em>u2</em> floats from definition to execution.  On system

with a combined stack is just shifts as many cells as these cells and

floats take.

<p>An optimizing native code compiler could compile a word defined with

<code>const-does&gt;</code> by compiling the <em>u1</em> cells and <em>u2</em>

floats as literals, and then compiling (and possibly inlining) a call to

the code behind the <code>const-does&gt;</code>.  The compiler would know that

these literals are constant, and could optimize accordingly.

<p>There are several alternative approaches to attack the problem:

<ul>

<li>An ANS-compliant way is to use colon definitions and

literals. E.g.:

<pre>

: constant ( n "name" -- )

\ name execution: ( -- n )

  >r : r> POSTPONE literal POSTPONE ; ;

: fconstant ( r "name" -- )

\ name execution: ( -- r )

\ environmental dependency: separate FP stack

  : POSTPONE fliteral POSTPONE ; ;

: simple-field ( n "name" -- )

\ name execution: ( addr1 -- addr2 )

  >r : r> POSTPONE literal POSTPONE + POSTPONE ; ;

</pre>

A good compiler could inline the resulting colon definitions and

optimize the results.

<p>The downside of this approach is that it is hard to read (even with

<code>]]</code>...<code>[[</code> to eliminate the

<code>POSTPONEs</code>).  And to avoid the environmental dependency

for passing floats, we would have to save them in global variables,

which would make the code even harder to read.

<li>An alternative syntax would make <code>does-const&gt;</code> behave

like <code>does&gt;</code>, but it would remove the ability to apply

<code>&gt;body</code> to the execution token of words it is applied

to.  Advantage: The syntax would be familiar and thus easier to learn.

Disadvantages: Optimizing this would require a more complex compiler.

The defining word is cluttered with code for saving and restoring the

data to be shifted between definition and execution, making the code

harder to read.  Simple implementations would not enforce the

restriction on not applying <code>&gt;body</code>, leading to

portability bugs (in contrast, the proposed

<code>const-does&gt;</code> does not expose the placement of the data,

so a programmer cannot even access the data without learning

implementation details).

<li>In a similar vein, Stephen Pelc suggested defining the idiom

<code>constant</code>...<code>does&gt;</code> (currently non-standard,

but widely used) to mean that the body must not be changed.  The

advantages and disadvantages are similar to

<code>does-const&gt;</code> above.  Additional disadvantages: hard to

implement in Forth systems where constants are not implemented in the

traditional way; the syntax becomes quite unnatural when more than two

cells or one float have to be transferred from definition-time to

run-time.

<li>Stephen Pelc also suggested a word <code>invariant ( addr u --

)</code> that would just tell the compiler that a memory area would

not change; this could be used to solve the problem like this:

<pre>

: constant ( n "name" -- )

  create here >r ,

  r> 1 cells invariant

does> ( -- n )

  @ ;

</pre>

The advantage of this approach is that it is more general

(<code>invariant</code> could also be used to optimize accesses to

other unchanging memory areas).  The disadvantages are decreased

readability and more complexity in optimizing (it requires a new

optimization, whereas <code>const-does&gt;</code> can be optimized

through inlining).

<li>One implementation issue for this proposal is that <em>u1</em> and

<em>u2</em> are supplied at run-time.  If the numbers were supplied at

compile-time (with a syntax like <code>[ 1 0 ]does-code&gt;</code>), a

native-code compiler using this for optimization could be slightly

simpler (and the reference implementation below could be faster).

However, a native-code compiler probably can use the same mechanism

here that it uses for optimizing <code>pick</code>, and supplying the

number at run-time is easier for the user (both syntactically and

mentally) and more flexible.

</ul>

<h4>Reference implementation</h4>

This ANS Forth implementation of <code>const-does&gt;</code> behaves as it

should, but does not give you the performance advantages

(rather to the contrary).

<pre>

: const-does>-prelude ( u1*x u2*r u1 u2 ``name'' -- )

    \ create name and store u1*x u2*r there

    create 2dup 2,

    over cells allot here >r

    falign dup floats allot here ( u1*x u2*r u1 u2 addr2 )

    swap 0 ?do

        -1 floats + dup f!

    loop

    drop r> ( u1*x u1 addr1 )

    swap 0 ?do

        -1 cells + tuck !

    loop

    drop ;

: const-does>-postlude ( addr -- u1*x u2*r )

    \ fetch u1*x u2*r from addr

    dup 2 cells +

    swap 2@ >r

    0 ?do

        dup @ swap cell+

    loop

    faligned

    r> 0 ?do

        dup f@ float+

    loop

    drop ;

: const-does> ( compilation: colon-sys1 -- colon-sys2 )

    \ run-time: ( u1*x u2*r u1 u2 ``name'' R: nest-sys1 -- )

    \ name initiation: ( -- u1*x u2*r R: nest-sys2 )

    POSTPONE const-does>-prelude

    POSTPONE does>

    POSTPONE const-does>-postlude

; immediate

</pre>

The following implementation is used in Gforth; it translates the

defined words into colon definitions containing literals and a call,

like this:

<pre>

\ input:

: simple-field ( n "name" -- )

1 0 const-does> ( addr1 -- addr2 )

    + ;

8 simple-field field1

\ SEE output:

: simple-field  

  1 0 249544 (const-does>) ;

\ 249544 xt-see

noname : 

  + ;

: field1  

  8 <249544> ;

</pre>

The implementation code itself is not quite ANS Forth compliant, but

porting it to other systems should not be hard:

<pre>

: compile-literals ( w*u u -- ; run-time: -- w*u ) recursive

    \ compile u literals, starting with the bottommost one

    ?dup-if

	swap >r 1- compile-literals

	r> POSTPONE literal

    endif ;

: compile-fliterals ( r*u u -- ; run-time: -- w*u ) recursive

    \ compile u fliterals, starting with the bottommost one

    ?dup-if

	{ F: r } 1- compile-fliterals

	r POSTPONE fliteral

    endif ;

: (const-does>) ( w*uw r*ur uw ur target "name" -- )

    \ define a colon definition "name" containing w*uw r*ur as

    \ literals and a call to target.

    { uw ur target }

    header docol: cfa, \ start colon def without stack junk

    ur compile-fliterals uw compile-literals

    target compile, POSTPONE exit reveal ;

: const-does> ( run-time: w*uw r*ur uw ur "name" -- )

    here >r 0 POSTPONE literal

    POSTPONE (const-does>)

    POSTPONE ;

    noname : POSTPONE rdrop

    lastxt r> cell+ ! \ patch the literal

; immediate

</pre>

<h4>Experience</h4>

<code>Const-does&gt;</code> is implemented in Gforth since November 2000.

<h4>Comments</h4>

Stephen Pelc pointed out that <code>const-does&gt;</code> is not

practical for defining words that contain larger tables (you don't

want to pass the table from definition-time to use-time on the stack).  

<p>This is true, but such words are rare in my experience, and uses of

such words where the fetches can be optimized are probably even more

rare (usually you will index the table with a value that has to be

computed at run-time, and then you cannot optimize the fetch anyway).

For the exceptions a solution based on <code>invariant</code> would

help.

Guido Draheim:

<pre>

`const-does>` is good stuff - although I would like to have an alias `does>@`

where just the specification says that `to` won't work.

</pre>

<hr><a href="/anton/">Anton Ertl</a>

\ example implementation and test cases

: defer ( "name" -- )

  create ['] abort ,

does> ( ... -- ... )

  @ execute ;

: defer@ ( xt1 -- xt2 )

  >body @ ;

: defer! ( xt2 xt1 -- )

  >body ! ;

: is

  state @ if

    POSTPONE ['] POSTPONE defer!

  else

    ' defer!

  then ; immediate

: action-of

 state @ if

   POSTPONE ['] POSTPONE defer@

 else

   ' defer@

then ; immediate

\ test cases

require test/tester.fs

{ defer defer1 -> }

{ : is-defer1 is defer1 ; -> }

{ : action-defer1 action-of defer1 ; -> }

{ ' * ' defer1 defer! -> }

{ 2 3 defer1 -> 6 }

{ ' defer1 defer@ -> ' * }

{ action-of defer1 -> ' * }

{ action-defer1 -> ' * }

{ ' + is defer1 -> }

{ 1 2 defer1 -> 3 }

{ ' defer1 defer@ -> ' + }

{ action-of defer1 -> ' + }

{ action-defer1 -> ' + }

{ ' - is-defer1 -> }

{ 1 2 defer1 -> -1 }

{ ' defer1 defer@ -> ' - }

{ action-of defer1 -> ' - }

{ action-defer1 -> ' - }

<title>Exception proposal</title>

<h3><a name="exception">EXCEPTION</a></h3>

[<a href="proposals.html">Other proposals</a>]

<h4>Problem</h4>

Libraries cannot introduce throw values, because they don't know which

values are used by other libraries or the application.

<p>The system does not know how to report a new exception.

<h4>Proposal</h4>

<pre>

EXCEPTION ( c-addr u -- n ) exception

</pre>

n is a previously unused THROW value in the range

{-4095...-256}. Consecutive calls to EXCEPTION return consecutive

decreasing numbers.

<p>The system may use the string denoted by c-addr u when reporting

that exception (if it is not caught).

<p>After a marker is executed or a word is forgotten that was defined

before EXCEPTION was called, THROWing n is an ambiguous condition.

<h4>Typical Use</h4>

<pre>

s" Out of GC-managed memory" EXCEPTION CONSTANT gc-out-of-memory

...

... gc-out-of-memory THROW ...

</pre>

<h4>Remarks</h4>

The restriction to values in the range {-4095...-256} ensures that

existing standard programs continue to work.

<p>The requirement to return consecutive decreasing THROW values makes it

possible to check for whole classes of exceptions with WITHIN:

<pre>

... CATCH ?DUP IF

  DUP lib-last-exception lib-first-exception 1+ WITHIN IF

    ... \ deal with exceptions from lib

  ELSE

    THROW \ just pass the ones on that we don't know how to handle

  THEN

</pre>

<p>The ambiguous condition after forgetting allows systems to reclaim

exception numbers and the memory taken up by the strings on

forgetting. Systems are not required to do this.

<p>Andrew Haley also voiced concerns about the interaction with

multitasking systems.  This proposal can be integrated with

multitasking systems in several ways:

<ul>

<li>Exception numbers just can be treated like dictionary space: Each

task/user gets a piece of the available range and allocates numbers

within this space independently.

<li>Each task maintains its own mapping of exceptions.  The same

exception number could be used for different purposes in different

tasks, but exception number usage would be restricted to the task and

could not be used across tasks.

</ul>

<h4>Experience</h4>

<code>EXCEPTION</code> is implementend in Gforth since before release

0.4.0.

<p>An approximation in ANS Forth is included in the Gforth compat library

which is also included in the <a

href="../garbage-collection.zip">garbage collector</a>.

<p><code>EXCEPTION</code> is used in the <a

href="../garbage-collection.zip">garbage collector</a>.

<h4>Comments</h4>

Michael L. Gassanenko on experience:

<pre>

I used:

CREATE not-ready

... IF not-ready THROW ...

BTW, the system(s) that I have to work on have no built-in CATCH .

</pre>

Peter Knaggs:

<pre>

another part of your "library" model, most useful.

</pre>

<hr><a href="/anton/">Anton Ertl</a>