first phase of a compiler

takes the modified source code from language preprocessors that are written in the form of sentences

breaks these syntaxes into a series of tokens by removing white space or comments in the source code

sequence of characters (alphanumeric) in a token

some predefined rules for every lexeme to be identified as a valid token

basic abstract unit of meaning

an object that represents something else

different types of tokens

predefined or reserved words in a programming language

used as the general terminology for naming of variables, functions and arrays

their values can't be modified by the program once they are defined

refer to fixed values

also called as literals

an array of characters ended with a null character

always enclosed in double quotes

used in programming languages having some special meaning

special symbols

used as array element reference

used to indicate function calls and function parameters

Mark the start and end of a block code

used to separate more than one statements

operator that essentially invokes an initialization list

create pointer variable

symbols that trigger an action when applied to variables and other objects

require only single operand

require two operands

requires three operands

notations for describing a set of character strings

used in search engines, search and replace dialogues of word processors

match zero or more quantifier

match one or more quantifier

match 0 or 1 quantifier

gives special meaning to the character following it

matches any character except the new line symbol

anchor for the start of the string or negation symbol

match a single character

anchor for the end of the string

separates a series of alternatives

used as range quantifiers

used for grouping characters or regular expressions

indicates a range in character class

formal language used to communicate algorithms both from programmer to programmer and from programmer to machine

set of symbols and rules for forming terms

The rules for transforming terms to terms

using the particular constructs of the language

define as a set of strings of symbols that may be constrained by specific rules

can generate context-free languages