Convert CVIR Tokens IDs, Labels and Syntax

Convert CVIR Tokens IDs, Labels and Syntax

is_keyword(token)

token_to_id(token)

id_to_token(id, convert_punctuation = TRUE)

Arguments

token

Token Label

id

Internal Token Byte Value

convert_punctuation

For `id_to_token()`: Convert Token ID to user-level syntax (`TRUE`) or internal label (`FALSE`); Default: `TRUE`

Value

a numeric vector of token IDs (`token_to_id()`), or a character vector of token labels or syntax (`id_to_token()`)

Examples

token_to_id("DERIVE")
#> DERIVE 
#>    141 

id_to_token(317)
#> [1] "%"