README for tokenizer 3.1,  97/03/27.

The files in this directory implement the "tokenizer", the program
used to convert FCode source code into FCode binary (byte codes).
Current version of tokenizer is 3.1, and is displayed when you
tokenize any Fcode source code with appropriate flags. 
The tokenizer program is not a part of an Open Boot CPU PROM.  It
runs under an operating system (usually Solaris), reading its input
(FCode source) from a file, and writing its output to a file.

This README file describes the three available tools to facilitate the
fcode program development process.

  1. tokenizer
  2. detokenizer
  3. makearray

The manual "Writing Fcode 3.x Programs" describes 3.x FCodes and related
information.
It may be accessed from Sun's Driver Development Site  at:
       http://www.sun.com/developers/driver/

       From above website, follow links to "Driver Documentation", 
                        "Driver Developer Site AnswerBook", 
                         and then to "Writing Fcode 3.x Programs" 

Same manual may also be accessed from:
	http://docs.sun.com/ 
	by doing a search for above title.

To understand OpenBoot from user point of view, please refer to 
the manual "OpenBoot 3.x Command Reference Manual" (part# 802-5837)  and 
"OpenBoot 3.x Quick Reference Card" (part# 802-5835-10). 
These manuals may also be accessed from:
        http://docs.sun.com/
        by doing a search for above title(s).
	Above manuals are usually in System Administrator's AnswerBook vol. 2.

The updated manual "Writing Fcode 3.x Programs" 802-6287-10 obsoletes these manuals: 
Writing Fcode 3.x Programs #802-5895-10 
Writing Fcode 3.x Programs #802-3239-10 

Look at test-pci.fth sample program to see how to add PCI specific ROM header
in front of fcode. 

detokenize doesn't strip off PCI header while detokenizing, so user
will need to use some utility to remove PCI header before detokenizing
a fcode image that has PCI header.
(NOTE:
       1) user can use Solaris/Unix utility "dd" and do (say if a.out header
                is attached to PCI fcode image) :
         % dd if=mypci.fcode of=my.fcode bs=84 skip=1
       2) emacs's hexl-mode allows you to edit binary files.
    or 3) User can come up with utility to remove PCI header by modifying fakeboot.c)


If user uses "Fcode-version3" as initial fcode, behavior of
"#" "#s" and "#>" will be as per 1275 specified.
i.e. "#" will generate "c7", "#s" will generate "c8" and "#>" will 
generate "c9"
If user uses "Fcode-version2" as initial fcode, behavior of
"#" "#s" and "#>" will be as specified in "Writing Fcode 2.x Programs"
i.e. "#" will generate "99", "#s" will generate "9a" and "#>" will 
generate "97"


1. TOKENIZE


   The tokenize program converts Forth source code to FCode 3.x binary format.

   This program is supplied "as-is".


	 usage: tokenize  [ -8 ] [ -s ] [ -n ] [ -D sym ] prog[.fth] [ out[.fcode] ] 

         -8 : use 8-bit offsets ( Used only in Fcode-version1 programs )

         -s : silent mode ( Warnings etc. will not be displayed)

         -n :  Output file will not have a.out(5) header.

         -D symbol : Define a symbol named "symbol".  Use with "ifdef"

         prog[.fth] : name of the FORTH program to be tokenized 
                      .fth extension is optional.

         out[.fcode]  : name of the output file,
                      default name is the same as input name and,
                      default extension is .fcode.

         example:
              tokenize the program my-prog.fth and create my-prog.fcode
                 tokenize my-prog
                 tokenize my-prog.fth  my-prog.fcode

         example:
              tokenize the program my-prog.fth and create result.fcode
                 tokenize my-prog  result
                 tokenize my-prog.fth  result.fcode



2. DETOKENIZE

   The detokenize program does the opposite of tokenize.  It converts
   FCode binary format to Forth source code.

   detokenize doesn't strip off PCI header while detokenizing, so user
   will need to use some utility to remove PCI header before detokenizing
   a fcode image that has PCI header.
   (NOTE:
       1) user can use Solaris/Unix utility "dd" and do (say if a.out header
                is attached to PCI fcode image) :
         % dd if=mypci.fcode of=my.fcode bs=84 skip=1
       2) emacs's hexl-mode allows you to edit binary files.
    or 3) User can come up with utility to remove PCI header by modifying fakeboot.c)
 

   This program is supplied "as-is".

     usage: detokenize  prog.fcode [ >  out.fth ]

     prog.fcode : name of the FCode binary program to be detokenized

	out.fth : name of the output file,
	          if no output file is specified, the output will be
	          displayed on the screen.

	example	: detokenize the program my-prog.fcode and create out.fth

                  detokenize my-prog.fcode  >  out.fth


	example : detokenize the program my-prog.fcode and display
		  the result on the screen.

		  detokenize my-prog.fcode


	example : detokenize the FCode PROM of the SBus card plugged into
                  the slot 2 from UNIX.

		1. Become super user

		   su


		2. Run the detokenize command and direct the output
			to a file called /tmp/slot2.fcode

		   detokenize  /dev/sbus2 > /tmp/slot2.fcode

3. MAKEARRAY


   The makearray program converts a binary data file to Forth source code.

   This program is supplied "as-is".

     usage: makearray  data.bin > [ data.fth ]

	data.bin : name of the binary format data file to be converted into
		   Forth source code.

	data.fth : name of the output file
		   if not output file is specified, the output will be
		   displayed on the screen.

	example : convert a file named sunlogo.icon to forth source code.

                makearray sunlogo.icon > sunlogo.fth


Here are several examples of incorporating array data into an FCode
program.

In all cases, assume that the "makearray" tool has already been
used to convert a binary file into a "-string Forth file named bindata.fth .
The contents of this file are multiple lines of the form:

" "( 6663 6f64 652d 7665 7273 696f 6e31 0a22 )" compile-bytes
" "( 0124 58df 2340 0000 12ff 1944 23fa 2300 )" compile-bytes
etc.

Note that interpretation of this form of "binary string" data will only
operate on Tokenizer versions 2.3 or greater.  The resulting FCode,
however, will execute properly on all Open Boot systems.

	Example #1:

Create an initialized array in the Forth dictionary.  Note: because Forth
dictionary space is often limited, this method is not recommended for
arrays larger than several hundred bytes.

Later execution of "my-array" leaves on the stack the address of the first
byte of data.

---------------------------------------------
\ Compile string into the dictionary
: compile-bytes  ( str.adr str.len -- )
   bounds ?do  i c@  c,  loop
;

create my-array
fload bindata.fth
---------------------------------------------

	Example #2:

Create an initialized array in DMA memory.  Also automatically preserves
the byte count of the array.

Later execution of "my-array" leaves on the stack the address of the first
byte of data.  The constant "array-size" holds the total array size.

---------------------------------------------
\ Copy string into next unoccupied region of memory.
: compile-bytes  ( count  mem.adr  str.adr len -- count'  mem.adr' )
\ First, update the count
   -rot  >r >r		( count len )  		( rs: str.adr mem.adr )
   tuck + swap		( count+len  len )  	( rs: " )
\ Next, calculate mem.adr'
   r@ over + swap	( count+len  mem.adr+len  len )  ( rs: " )
\ Finally, copy the data
   r> r> swap rot	( count+len  mem.adr+len  str.adr mem.adr len )
   move			( count+len  mem.adr+len )
;

\ Allocate memory region with any convenient method.  Here, "dma-alloc" is
\ used.  2000 bytes is used as upper limit for size in this example.
2000 dma-alloc   ( virt.adr )  CONSTANT  my-array

0  my-array   \ Initial count and memory address
fload bindata.fth    	( total.size  next.mem.adr )
drop			( total.size )
CONSTANT array-size
---------------------------------------------
