#erlang #tokenizer #lexer

erl_tokenize

Erlang source code tokenizer

14 releases

0.3.7 Jun 23, 2017
0.3.6 Jun 15, 2017
0.2.0 Jun 7, 2017
0.1.4 Jun 7, 2017
0.1.1 May 31, 2017

#65 in Parser implementations

Download history 14/week @ 2018-08-22 2/week @ 2018-08-29 14/week @ 2018-09-05 3/week @ 2018-09-12 2/week @ 2018-09-19 15/week @ 2018-09-26 45/week @ 2018-10-03 1/week @ 2018-10-10 182/week @ 2018-10-17 14/week @ 2018-10-24 15/week @ 2018-10-31 126/week @ 2018-11-07

51 downloads per month
Used in 2 crates

MIT license

86KB
2K SLoC

erl_tokenize

erl_tokenize Documentation Build Status Code Coverage License: MIT

Erlang source code tokenizer written in Rust.

Documentation

Examples

Tokenizes the Erlang code io:format("Hello").:

use erl_tokenize::Tokenizer;

let src = r#"io:format("Hello")."#;
let tokenizer = Tokenizer::new(src);
let tokens = tokenizer.collect::<Result<Vec<_>, _>>().unwrap();

assert_eq!(tokens.iter().map(|t| t.text()).collect::<Vec<_>>(),
           ["io", ":", "format", "(", r#""Hello""#, ")", "."]);

Executes the example tokenize command:

$ cargo run --example tokenize -- /dev/stdin <<EOS
-module(foo).

-export([bar/0]).

bar() -> qux.
EOS

[Position { filepath: None, offset: 0, line: 1, column: 1 }] Symbol(Hyphen)
[Position { filepath: None, offset: 1, line: 1, column: 2 }] Atom("module")
[Position { filepath: None, offset: 7, line: 1, column: 8 }] Symbol(OpenParen)
[Position { filepath: None, offset: 8, line: 1, column: 9 }] Atom("foo")
[Position { filepath: None, offset: 11, line: 1, column: 12 }] Symbol(CloseParen)
[Position { filepath: None, offset: 12, line: 1, column: 13 }] Symbol(Dot)
[Position { filepath: None, offset: 13, line: 1, column: 14 }] Whitespace(Newline)
[Position { filepath: None, offset: 14, line: 2, column: 1 }] Whitespace(Newline)
[Position { filepath: None, offset: 15, line: 3, column: 1 }] Symbol(Hyphen)
[Position { filepath: None, offset: 16, line: 3, column: 2 }] Atom("export")
[Position { filepath: None, offset: 22, line: 3, column: 8 }] Symbol(OpenParen)
[Position { filepath: None, offset: 23, line: 3, column: 9 }] Symbol(OpenSquare)
[Position { filepath: None, offset: 24, line: 3, column: 10 }] Atom("bar")
[Position { filepath: None, offset: 27, line: 3, column: 13 }] Symbol(Slash)
[Position { filepath: None, offset: 28, line: 3, column: 14 }] Integer(BigUint { data: [] })
[Position { filepath: None, offset: 29, line: 3, column: 15 }] Symbol(CloseSquare)
[Position { filepath: None, offset: 30, line: 3, column: 16 }] Symbol(CloseParen)
[Position { filepath: None, offset: 31, line: 3, column: 17 }] Symbol(Dot)
[Position { filepath: None, offset: 32, line: 3, column: 18 }] Whitespace(Newline)
[Position { filepath: None, offset: 33, line: 4, column: 1 }] Whitespace(Newline)
[Position { filepath: None, offset: 34, line: 5, column: 1 }] Atom("bar")
[Position { filepath: None, offset: 37, line: 5, column: 4 }] Symbol(OpenParen)
[Position { filepath: None, offset: 38, line: 5, column: 5 }] Symbol(CloseParen)
[Position { filepath: None, offset: 39, line: 5, column: 6 }] Whitespace(Space)
[Position { filepath: None, offset: 40, line: 5, column: 7 }] Symbol(RightArrow)
[Position { filepath: None, offset: 42, line: 5, column: 9 }] Whitespace(Space)
[Position { filepath: None, offset: 43, line: 5, column: 10 }] Atom("qux")
[Position { filepath: None, offset: 46, line: 5, column: 13 }] Symbol(Dot)
[Position { filepath: None, offset: 47, line: 5, column: 14 }] Whitespace(Newline)
TOKEN COUNT: 29
ELAPSED: 0.007222 seconds

References

Dependencies

~1MB
~24K SLoC