class Drizzle::Lexer

Overview

The Lexer is the class in charge of reading in input from a file and converting the text into tokens.

It works similarly to a Python generator, with the #get_next_token method generating the next token from the source file.

Defined in:

drizzle/lexer.cr

Constructors

Instance Method Summary

Constructor Detail

def self.new(input : String) #

Create a Lexer instance using a String. This string will be split on newline characters and turned into an array of lines, as if it had come from a File.

Since this should only be called from the REPL, the file_name will be "<stdin>".


[View source]
def self.new(file : File) #

Create a Lexer instance using a File instance.

This will replace the filename and load the lines in to use as input.


[View source]

Instance Method Detail

def get_next_token : Token #

Generate the next Token instance from the given input.

This method first tries the current_char of the Lexer against all of the single character Tokens in Drizzle. If it does not match, it then attempts to build up identifiers / keywords or numbers, depending on what the character is.


[View source]
def peek_next_char : Char #

Check what the next character on the current line is, without updating all the pointers


[View source]
def read_identifier : String #

Builds up a possible identifier from the source.

This method is run whenever #get_next_token comes across a letter in the input.


[View source]
def read_next_char #

Update the char_num and read_char_num pointers, as well as the current_line and current_char values. If the end of the current line has been reached, move to the next line (if exists). If the end of the input has been reached, set the current char to be the Char::ZERO.


[View source]
def read_number : String #

Builds up an integer number from the source code.

The interpreter book only handles integer numbers when I move to creating an ANTLR parser then Drizzle will have other number types too :D The number is returned as a String still, to keep Token implementation simple


[View source]
def skip_comment #

Once a comment is found in the source, skip the lexer to the next line.

In Drizzle, there are currently only single line comments, so we can just skip the line.


[View source]
def skip_whitespace #

Skip whitespace characters in the input as it is unnecessary to turn them into Tokens for Drizzle

Whitespace characters include spaces, newlines, tabs, carriage returns, etc.


[View source]
def valid_identifier_char?(char : Char) : Bool #

Determine whether a given character is allowed to be used as part of an identifier name.

Since the #read_identifier method is only run when the #get_next_token finds a letter, we can safely allow numbers in this as well.


[View source]