TOKENIZE.EXE by Bob Pigford July, 2010 ============================================================= TOKENIZE.EXE is a program that translates an ASCII listing of a BASIC program into tokenized form for use in the Tandy M100, T102, and T200 portable computers, also referred to as "Model T's". My compiled PowerBASIC code mimics many elements of tokenization code within VirtualT written by Ken Pettit. Ken generously shared his C++ source code and patiently answered my questions about its operation. Since Ken was so gracious and helpful to me, I also will share my source code. If you are interested in it, please email me at bob@pigford.org. In addition, please send me any helpful comments you may have about this program. PURPOSE ------- The purpose of TOKENIZE.EXE is to make a ready-to-LOAD-and-RUN BASIC program for the ModelT by translating its ASCII program listing, and to do so using the Windows platform. This program does essentially the same thing that the ModelT operating system would do if you were using a real ModelT itself (or VirtualT, the ModelT emulator written by Ken Pettit). If you were using the real ModelT, you would go to BASIC, then type LOAD"FILNAM" where that FILNAM was an ASCII listing with a .DO extension. With TOKENIZE.EXE you can have that program already tokenized by your WinXP computer and ready to use in the ModelT. TOKENIZE.EXE can be useful for converting the many program listings out there that show a .BA extension, but are really ASCII program listings (i.e.: .DO text files). Many programs are listed this way in the libraries and downloads available from Club100. In addition, by tokenizing your program listing files outside the ModelT, larger program listing files can be accomodated as they do not need to be in the ModelT memory when they are tokenized in BASIC via the LOAD"FILNAM.DO" command. OPERATION --------- Upon entry, you will see a scroll box of .DO files. Use the UP/DN arrows to highlight your choice for ASCII listing input file. Filenames longer than 9 characters will not be found. Ex: FILNAM.DO is ok, FILNAME.DO is not. Note: ASCII listings are the same text files that ModelT BASIC produces when using the SAVE"FILNAM",A option. You may also create your own ASCII program listing file as a simple text file using a word processor in your Windows or other computer. Save it as an MSDOS file using standard ModelT format filenaming convention. You will be asked if this is correct or not. If not (not Y), the screen clears and you can start again. If your input filename is OK, then you will be asked to type in a name for your output file (that will become a TOKENIZED version of your ASCII program listing input file). You should give it an extension of .BA so that the ModelT will know that it is ready to run. If the output filename does not already exist, TOKENIZE.EXE will create it or overwrite any existing file of that name. When your output filename is OK, pressing Y will start the program execution. The screen will change and show the results. RESULTS ------- The .BA output file that you named will be ready to LOAD and RUN in your ModelT computer. Note also that you can use any output name and extension that is permissible with Windows. But remember to rename it using the ModelT filename convention before you transfer it into the ModelT: Six or fewer characters, a dot, then the BA extension. The first character may not be a number. NOTES 1. This program requires the ASCII listing file to end each line with a CR and LF (Carriage Return and Line Feed). This is the standard format for ModelT programs saved as ASCII files [SAVE"FILNAM",A] or MS-DOS files saved with CR/LF end-of-line markers. 2. If Windows tries to use the whole screen for this program, just hit Alt-Enter to get a window, then right click on the top bar, go to properties and adjust the window size. I use 70 columns wide by 25 rows high. 3. Your ASCII listing file, TOKENIZE.hlp, and TOKENIZE.EXE must all be in the same directory. The resulting .BA file will also be found in this same directory. 4. If there is a blank line of 0 length in your input file, you will receive an error message about TEXT being ILL FORMED. You must exit TOKENIZE then fix your input file. 5. Please review your input file for programming and syntax errors before tokenizing as any programming errors will not show up until you load and RUN the tokenized file in your ModelT. Hard crashes might occur if the ASCII program listing file contains serious errors. Other than the basic check for empty lines, TOKENIZE.EXE does not do any error checking or correcting of your ASCII listing code. 6. You may use lower case and upper case letters for the output filenames, and for BASIC commands in your input file, because TOKENIZE.EXE checks for lowercase and changes the text to uppercase for the output file where required. 7. TOKENIZE.EXE is written to display in a console (text) window on the Windows OpSys screen. It has been tested to run under WinXP, 7, and 10. It should run OK under other Windows, but I have not tested those. 8. TOKENIZE.EXE was compiled using PowerBASIC Console Compiler version 6.03 sold by PowerBASIC.com. It is a powerful language and compiles to stand alone executables that do not need special run-time assistance.