snb wrote: ↑14 Mar 2025, 17:01
I very much doubt whether I understand what you are up to (since a lot of procedures seem to be lacking ...
Your client wants to invent a language and specifies its BNF. (Or asks you to design a language ...)(or asks you to document a parser/tokeniser for which the original BNF has been mislaid ...)
In all events you want to start from a BNF description of the grammar. Commonly this involves specifying lexical units and the syntax of the language.
The definition of floating-point literals in Python may be an exemplar of mixing several notations –
Code: Select all
floatnumber ::= pointfloat | exponentfloat
pointfloat ::= [intpart] fraction | intpart "."
exponentfloat ::= (intpart | pointfloat) exponent
intpart ::= digit+
fraction ::= "." digit+
exponent ::= ("e" | "E") ["+" | "-"] digit+
Suppose this to be the entire BNF grammar of a language (it's nowhere near it!)
If we can translate this BNF to a
State Transition Table (UK: Turing Machine) then we can have a definition of the language in the form of a State Transition Table. You have a copy of Rob Gelder's VBA code for his tokeniser.
I stared at his code and saw that it had been generated methodically from a State Transition Table. (I suspect
methodically by hand owing to anomalies in the coding, rather than
methodically by a computer program)
It is dead easy to translate from a State Transition Table to (almost) any language methodically, but especially wth a program.
Whew! If the client will write out (or get someone to write out) a BNF grammar for his purposes in analysing Excel, and if I can translate that BNF to a State transition table, and if I can translate that State transition Table to VBA code, THEN I can take in the BNF and
automatically emit (TaDa!) a tokeniser/parser written in VBA.
Which is what the client wants.
I did all this in a crude version of APl 50+ years ago, but have forgotten the process from BNF to State Transition table; it will come back to me.
Over the past two days I wrote the code to translate from the State Transition Table to VBA code and am now putting it through its tests.
The point of this exercise is to show that IF I am given a BNF grammar of a language THEN I can automatically spew out a tokeniser/parser.
My premise is that the original BNF can be very simple. I am using a floating point number as the entire grammar. Nowhere near Excel, yet, but if the client beefs up the BNF grammar to HIS specification of Excel formulas THEN we can generate the appropriate VBA code in under two minutes. And when (not if) the client adjusts the BNF, two minutes gives us an updated tokeniser, instead of man-months puzzling over that 500+ line procedure.
At this stage you just have to guess what procedures such as " PushRealString" and "InitialiseIntegerString" might do. I don't really know that myself; just that at some stage I will need to set up a string accumulator, and at some other time I will need to be able to Push (and Pop) something with a stack.
Did you need the full definition of those Class modules to comprehend what "objTokens.Add" did? I hadn't realized that they were there, I just had faith that something called "objTokens.Add" would add something to something called objTokens. (I further assumed that it might be a collection of Token objects ...)
Cheers, Chris
I have decided to document Rob Van Gelder's Excel Tokeniser. After close examination I believe that his program code (VBA) is based on (N.Amer) a State Transition Table a.k.a. (UK) a Turing Machine.