stde.pegen

A PEG parser generator based on the code of Pegen.

The "stde." in the name is from my GitHub username for namespacing only.

This project started as a PR for Pegen but ended up changing so much that I decided to make it a fork. The library is usable, but some project files (tutorial, reference, README, etc.) are outdated and incomplete.

Currently I would appreciate contributions in the following aspects:

• Documentation changes
• Adding new tests, especially those that improve test coverage

If you have a question about using the library, please ask in the Discussions area.

How this library is different from original Pegen

• Bug fixes
• A new base parser that matches character by character without forcing tokenization, compared to the default base parser that is limited to Python's tokenization
• An interface usable via Python code
• Implement skip_actions option
• A new grammar grapher that draws more readable graphs
• Support for rules that call arbitrary code (custom rules)

Installing

This library is not published to PyPI yet, so download the wheel from the Releases and install from it.

pip install the-wheel.whl

Note: The latest commit of the repository is not compatible with the v0.0.0 release because I am stocking up commits for a new major version and plan to not make releases before that.

How to generate a parser

As a library

from stde.pegen.v2.build import generate_parser_from_grammar
grammar = """
start: expr NEWLINE $ { expr }

expr:
    | a=expr2 "+" b=expr { a + b }
    | a=expr2 "-" b=expr { a - b }
    | expr2

expr2:
    | a=expr2 "*" b=NUMBER { a * int(b.string) }
    | a=expr2 "/" b=NUMBER { a / int(b.string) }
    | NUMBER { int(number.string) }
"""
parser_class = generate_parser_from_grammar(grammar).parser_class
parser = parser_class.from_text("1 + 2 * 3")
print(parser.start())  # Output: 7

Command line

Given a grammar file compatible with stde.pegen, e.g. with this saved as grammar.txt:

start: expr NEWLINE $ { expr }

expr:
    | a=expr2 "+" b=expr { a + b }
    | a=expr2 "-" b=expr { a - b }
    | expr2

expr2:
    | a=expr2 "*" b=NUMBER { a * int(b.string) }
    | a=expr2 "/" b=NUMBER { a / int(b.string) }
    | NUMBER { int(number.string) }

Generate a parser by running:

python -m stde.pegen grammar.txt -o parser.py

This will generate a file called parser.py in the current directory. This can be used to parse code using the grammar that we just used:

python parser.py file-to-parse.txt

and will print the result of parsing rule start.

The parser code can also be imported. You typically use the parser class which is named GeneratedParser by default.

from parser import GeneratedParser
parser = parser_class.from_text("1 + 2")
print(parser.start()) # Output: 3

Modes (legacy and v2)

stde.pegen has legacy mode (stde.pegen.legacy package) and v2 mode (stde.pegen.v2 package). The legacy mode is not maintained anymore, will probably never get documented, and is kept only for legacy reasons and as a reference for developing v2 (maybe). It may get removed in the future, so you should use v2 mode unless you know you really need to use legacy mode.

Note

Everything below is outdated.

Documentation

The documentation is available here.

Also an incomplete guide in doc.md.

How to contribute

See the instructions in the CONTRIBUTING.md file.

Differences with CPython's Pegen

This repository exists to distribute a version of the Python PEG parser generator used by CPython that can be installed via PyPI, with some improvements. Although the official PEG generator included in CPython can generate both Python and C code, this distribution of the generator only allows to generate Python code. This is due to the fact that the C code generated by the generator included in CPython includes a lot of implementation details and private headers that are not available for general usage.

The official PEG generator for Python 3.9 and later is now included in the CPython repo under Tools/peg_generator/. We aim to keep this repo in sync with the Python generator from that version of stde.pegen.

See also PEP 617.

Differences with original pegen

[TODO]

Repository structure

• The src directory contains the stde.pegen source (the package itself).
• The tests directory contains the test suite for stde.pegen.
• The data directory contains some example grammars compatible with stde.pegen. This includes a pure-Python version of the Python grammar (legacy mode: python.gram, v2 mode: python_v2.gram).
• The docs directory contains the documentation for the package.
• The scripts directory contains some useful scripts that can be used for visualizing grammars, benchmarking and other usages relevant to the development of the generator itself.
• The stories directory contains the backing files and examples for Guido's series on PEG parser.