CEL-Go Codelab: Fast, safe, embedded expressions

1. Introduction

What is CEL?

CEL is a non-Turing complete expression language designed to be fast, portable, and safe to execute. CEL can be used on its own, or embedded into a larger product.

CEL was designed as a language in which it is safe to execute user code. While it's dangerous to blindly call eval() on a user's python code, you can safely execute a user's CEL code. And because CEL prevents behavior that would make it less performant, it evaluates safely on the order of nanoseconds to microseconds ; it's ideal for performance-critical applications.

CEL evaluates expressions, which are similar to single line functions or lambda expressions. While CEL is commonly used for boolean decisions, it can also be used to construct more complex objects like JSON or protobuf messages.

Is CEL right for your project?

Because CEL evaluates an expression from the AST in nanoseconds to microseconds, the ideal use cased for CEL are applications with performance-critical paths. Compilation of CEL code into the AST should not be done in critical paths; ideal applications are ones in which the configuration is executed often and modified relatively infrequently.

For example, executing a security policy with each HTTP request to a service is an ideal use case for CEL because the security policy changes rarely, and CEL will have a negligible impact on the response time. In this case, CEL returns a boolean, if the request should be allowed or not, but it could return a more complex message.

What's covered in this Codelab?

The first step of this codelab walks through the motivation for using CEL and its Core Concepts. The rest is dedicated to coding Exercises that cover common use cases. For a more in-depth look at the language, semantics, and features see the CEL Language Definition on GitHub and the CEL Go Docs.

This codelab is aimed at developers who would like to learn CEL in order to use services that already support CEL. This codelab doesn't cover how to integrate CEL into your own project.

What you'll learn

• Core concepts from CEL
• Hello, World: Using CEL to evaluate a String
• Creating variables
• Understanding CEL's short-circuiting in Logical AND/OR operations
• How to use CEL to build JSON
• How to use CEL to build Protobuffers
• Creating Macros
• Ways to tune your CEL expressions

What you'll need

• Go

Prerequisites

This codelab builds upon a basic understanding of Protocol Buffers and Go Lang.

If you're not familiar with Protocol Buffers, the first exercise will give you a sense of how CEL works, but because the more advanced examples use Protocol Buffers as the input into CEL, they may be harder to understand. Consider working through one of these tutorials, first. Note that Protocol Buffers are not required to use CEL, but they are used extensively in this codelab.

You can test that go is installed by running:

go --help

2. Key concepts

Applications

CEL is general purpose, and has been used for diverse applications, from routing RPCs, to defining security policy. CEL is extensible, application agnostic, and optimized for compile-once, evaluate-many workflows.

Many services and applications evaluate declarative configurations. For example, Role-Based Access Control (RBAC) is a declarative configuration that produces an access decision given a role and a set of users. If declarative configurations are the 80% use case, then CEL is a useful tool in rounding out the remaining 20% when users need to more expressive power.

Compilation

An expression is compiled against an environment. The compilation step produces an Abstract Syntax Tree (AST) in protobuf form. Compiled expressions are generally stored for future use to keep the evaluation as fast as possible. A single compiled expression can be evaluated with many different inputs.

Expressions

Users define expressions; services and applications define the environment where it runs. A function signature declares the inputs, and is written outside of the CEL expression. The library of functions available to CEL is auto-imported.

In the following example the expression takes a request object, and the request includes a claims token. The expression returns a boolean indicating if the claims token is still valid.

// Check whether a JSON Web Token has expired by inspecting the 'exp' claim.
//
// Args:
//   claims - authentication claims.
//   now    - timestamp indicating the current system time.
// Returns: true if the token has expired.
//
timestamp(claims["exp"]) < now

Environment

Environments are defined by services. Services and applications that embed CEL declare the expression environment. The environment is the collection of variables and functions that can be used in expressions.

The proto-based declarations are used by the CEL type-checker to ensure that all identifier and function references within an expression are declared and used correctly.

Three phases of parsing an expression

There are three phases in processing an expression: parse, check, and evaluate. The most common pattern for CEL is for a control plane to parse and check expressions at config time, and store the AST.

At runtime, the data plane retrieves and evaluates the AST repeatedly. CEL is optimized for runtime efficiency, but parsing and checking should not be done in latency critical code paths.

Warning: Parsing and checking should not be done in latency critical code paths.

CEL is parsed from a human-readable expression to an abstract syntax tree using an ANTLR lexer / parser grammar. The parse phase emits a proto-based abstract syntax tree where each Expr node in the AST contains an integer id that is used to index into metadata generated during parsing and checking. The syntax.proto produced during parsing faithfully represents the abstract representation of what was typed in the string form of the expression.

Once an expression has been parsed, it may be checked against the environment to ensure all variable and function identifiers in the expression have been declared and are being used correctly. The type-checker produces a checked.proto that includes type, variable, and function resolution metadata that can drastically improve evaluation efficiency.

Best practice: Perform type-checking to improve the speed and safety of parsed expressions, even with dynamic data like JSON where type inference is limited.

The CEL evaluator needs 3 things:

• Function bindings for any custom extensions
• Variable bindings
• An AST to evaluate

The function and variable bindings should match what was used to compile the AST. Any of these inputs can be re-used across multiple evaluations, such as an AST being evaluated across many sets of variable bindings, or the same variables used against many ASTs, or the function bindings used across the lifetime of a process (a common case).

3. Set Up

The code for this codelab lives in the codelab folder of the cel-go repo. The solution is available in the codelab/solution folder of the same repo.

Clone and cd into the repo:

git clone https://github.com/google/cel-go.git 
cd cel-go/codelab

Run the code using go run:

go run .

You should see the following output:

=== Exercise 1: Hello World ===

=== Exercise 2: Variables ===

=== Exercise 3: Logical AND/OR ===

=== Exercise 4: Customization ===

=== Exercise 5: Building JSON ===

=== Exercise 6: Building Protos ===

=== Exercise 7: Macros ===

=== Exercise 8: Tuning ===

Where are the CEL packages?

In your editor, open codelab/codelab.go. You should see the main function which drives the execution of the exercises in this codelab, followed by three blocks of helper functions. The first set of helpers assist with the phases of CEL evaluation:

• Compile function: Parses and checks and input expression against an environment
• Eval function: Evaluates a compiled program against an input
• Report function: Pretty-prints the evaluation result

Additionally, the request and auth helpers have been provided to assist with input construction for the various exercises.

The exercises will refer to packages by their short package name. The mapping from package to source location within the google/cel-go repo is below if you'd like to dig into the details:

Package	Source Location	Description
cel	cel-go/cel	Top-level interfaces
ref	cel-go/common/types/ref	Reference interfaces
types	cel-go/common/types	Runtime type values

4. Hello, World!

In the tradition of all programming languages, we'll start by creating and evaluating "Hello World!".

Configure the environment

In your editor, find the declaration of exercise1, and fill in the following to set up the environment:

// exercise1 evaluates a simple literal expression: "Hello, World!"
//
// Compile, eval, profit!
func exercise1() {
    fmt.Println("=== Exercise 1: Hello World ===\n")
    // Create the standard environment.
    env, err := cel.NewEnv()
    if err != nil {
        glog.Exitf("env error: %v", err)
    }
    // Will add the parse and check steps here
}

CEL applications evaluate an expression against an Environment. env, err := cel.NewEnv() configures the standard environment.

The environment can be customized by providing the options cel.EnvOption to the call. Those options are able to disable macros, declare custom variables and functions, etc.

The standard CEL environment supports all of the types, operators, functions, and macros defined within the language specification.

Parse and check the expression

Once the environment has been configured, expressions can be parsed and checked. Add the following to your function:

// exercise1 evaluates a simple literal expression: "Hello, World!"
//
// Compile, eval, profit!
func exercise1() {
    fmt.Println("=== Exercise 1: Hello World ===\n")
    // Create the standard environment.
    env, err := cel.NewEnv()
    if err != nil {
        glog.Exitf("env error: %v", err)
    }
    // Check that the expression compiles and returns a String.
    ast, iss := env.Parse(`"Hello, World!"`)
    // Report syntactic errors, if present.
    if iss.Err() != nil {
        glog.Exit(iss.Err())
    }
    // Type-check the expression for correctness.
    checked, iss := env.Check(ast)
    // Report semantic errors, if present.
    if iss.Err() != nil {
        glog.Exit(iss.Err())
    }
    // Check the output type is a string.
    if !reflect.DeepEqual(checked.OutputType(), cel.StringType) {
        glog.Exitf(
            "Got %v, wanted %v output type",
            checked.OutputType(), cel.StringType,
        )
    }
    // Will add the planning step here
}

The iss value returned by the Parse and Check calls is a list of issues which could be errors. If the iss.Err() is non-nil, there is an error in the syntax or semantics, and the program cannot proceed further. When the expression is well-formed the result of these calls is an executable cel.Ast.

Note: In this codelab, parse and check phases have been bundled into the compile() helper which can be used for the remainder of the exercises.

Evaluate the expression

Once the expression has been parsed and checked into a cel.Ast, it can be converted into an evaluable program whose function bindings and evaluation modes can be customized with functional options. Note, it is also possible to read a cel.Ast from a proto using either the cel.CheckedExprToAst or cel.ParsedExprToAst functions.

Once a cel.Program is planned, it can be evaluated against input by calling Eval. The result of the Eval will contain the result, evaluation details, and the error status.

Add planning and call eval:

// exercise1 evaluates a simple literal expression: "Hello, World!"
//
// Compile, eval, profit!
func exercise1() {
    fmt.Println("=== Exercise 1: Hello World ===\n")
    // Create the standard environment.
    env, err := cel.NewEnv()
    if err != nil {
        glog.Exitf("env error: %v", err)
    }
    // Check that the expression compiles and returns a String.
    ast, iss := env.Parse(`"Hello, World!"`)
    // Report syntactic errors, if present.
    if iss.Err() != nil {
        glog.Exit(iss.Err())
    }
    // Type-check the expression for correctness.
    checked, iss := env.Check(ast)
    // Report semantic errors, if present.
    if iss.Err() != nil {
        glog.Exit(iss.Err())
    }
    // Check the output type is a string.
    if !reflect.DeepEqual(checked.OutputType(), cel.StringType) {
        glog.Exitf(
            "Got %v, wanted %v output type",
            checked.OutputType(), cel.StringType)
    }
    // Plan the program.
    program, err := env.Program(checked)
    if err != nil {
        glog.Exitf("program error: %v", err)
    }
    // Evaluate the program without any additional arguments.
    eval(program, cel.NoVars())
    fmt.Println()
}

For brevity, we'll omit the error cases included above from future exercises.

Execute the code

On the command line, rerun the code:

go run .

You should see the following output, along with placeholders for the future exercises.

=== Exercise 1: Hello World ===

------ input ------
(interpreter.emptyActivation)

------ result ------
value: Hello, World! (types.String)

5. Use variables in a function

Most CEL applications will declare variables that can be referenced within expressions. Declarations of variables specify a name and a type. A variable's type may either be a CEL builtin type, a protocol buffer well-known type, or any protobuf message type so long as its descriptor is also provided to CEL.

Note: In the following example, the compile and eval helpers have been included as stand-ins for env.Parse() / env.Check() and program.Eval(). These helper functions are included to print the inputs and outputs of the exercises.

Add the function

In your editor, find the declaration of exercise2, and add the following:

// exercise2 shows how to declare and use variables in expressions.
//
// Given a request of type google.rpc.context.AttributeContext.Request
// determine whether a specific auth claim is set.
func exercise2() {
  fmt.Println("=== Exercise 2: Variables ===\n")
   env, err := cel.NewEnv(
    // Add cel.EnvOptions values here.
  )
  if err != nil {
    glog.Exit(err)
  }
  ast := compile(env, `request.auth.claims.group == 'admin'`, cel.BoolType)
  program, _ := env.Program(ast)

  // Evaluate a request object that sets the proper group claim.
  claims := map[string]string{"group": "admin"}
  eval(program, request(auth("user:[e-mail]", claims), time.Now()))
  fmt.Println()
}

Rerun and understand the error

Rerun the program:

go run .

You should see the following output:

ERROR: <input>:1:1: undeclared reference to 'request' (in container '')
 | request.auth.claims.group == 'admin'
 | ^

The type-checker produces an error for the request object which conveniently includes the source snippet where the error occurs.

6. Declare the variables

Add EnvOptions

In the editor, let's fix the resulting error by providing a declaration for the request object as a message of type google.rpc.context.AttributeContext.Request like so:

// exercise2 shows how to declare and use variables in expressions.
//
// Given a `request` of type `google.rpc.context.AttributeContext.Request`
// determine whether a specific auth claim is set.
func exercise2() {
  fmt.Println("=== Exercise 2: Variables ===\n")
  env, err := cel.NewEnv(
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )
  if err != nil {
    glog.Exit(err)
  }
  ast := compile(env, `request.auth.claims.group == 'admin'`, cel.BoolType)
  program, _ := env.Program(ast)

  // Evaluate a request object that sets the proper group claim.
  claims := map[string]string{"group": "admin"}
  eval(program, request(auth("user:[e-mail]", claims), time.Now()))
  fmt.Println()
}

Rerun and understand the error

Running the program again:

go run .

You should see the following error:

ERROR: <input>:1:8: [internal] unexpected failed resolution of 'google.rpc.context.AttributeContext.Request'
 | request.auth.claims.group == 'admin'
 | .......^

To use variables that refer to protobuf messages, the type-checker needs to also know the type descriptor.

Aside: The type descriptor is a protobuf message that describes the field declarations of a protobuf message. Descriptors can be thought of as a reflection type. Descriptors are used within the type-checker to determine the field type references. Descriptors can either be saved or read from a protobuf message instance. The cel.Types() helper is a cel.EnvOption that inspects messages instances to determine the descriptor set available to the type-checker.

Use cel.Types to determine the descriptor for request in your function:

// exercise2 shows how to declare and use variables in expressions.
//
// Given a `request` of type `google.rpc.context.AttributeContext.Request`
// determine whether a specific auth claim is set.
func exercise2() {
  fmt.Println("=== Exercise 2: Variables ===\n")
   env, err := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}), 
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )

  if err != nil {
    glog.Exit(err)
  }
  ast := compile(env, `request.auth.claims.group == 'admin'`, cel.BoolType)
  program, _ := env.Program(ast)

  // Evaluate a request object that sets the proper group claim.
  claims := map[string]string{"group": "admin"}
  eval(program, request(auth("user:[e-mail]", claims), time.Now()))
  fmt.Println()
}

Rerun successfully!

Run the program again:

go run .

You should see the following:

=== Exercise 2: Variables ===

request.auth.claims.group == 'admin'

------ input ------
request = time: <
  seconds: 1569255569
>
auth: <
  principal: "user:[e-mail]"
  claims: <
    fields: <
      key: "group"
      value: <
        string_value: "admin"
      >
    >
  >
>

------ result ------
value: true (types.Bool)

To review, we just declared a variable for an error, assigned it a type descriptor, and then referenced the variable in the expression evaluation.

7. Logical AND/OR

One of CEL's more unique features is its use of commutative logical operators. Either side of a conditional branch can short-circuit the evaluation, even in the face of errors or partial input.

In other words, CEL finds an evaluation order which gives a result whenever possible, ignoring errors or even missing data that might occur in other evaluation orders. Applications can rely on this property to minimize the cost of evaluation, deferring the gathering of expensive inputs when a result can be reached without them.

We'll add an AND/OR example, and then try it with different input to understand how CEL short-circuits evaluation.

Create the function

In your editor, add the following content to exercise 3:

// exercise3 demonstrates how CEL's commutative logical operators work.
//
// Construct an expression which checks if the `request.auth.claims.group`
// value is equal to admin or the `request.auth.principal` is
// `user:[e-mail]`. Issue two requests, one that specifies the proper 
// user, and one that specifies an unexpected user.
func exercise3() {
  fmt.Println("=== Exercise 3: Logical AND/OR ===\n")
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )
}

Next, include this OR statement that will return true if either the user is a member of the admin group, or has a particular email identifier:

// exercise3 demonstrates how CEL's commutative logical operators work.
//
// Construct an expression which checks if the `request.auth.claims.group`
// value is equal to admin or the `request.auth.principal` is
// `user:[e-mail]`. Issue two requests, one that specifies the proper 
// user, and one that specifies an unexpected user.
func exercise3() {
  fmt.Println("=== Exercise 3: Logical AND/OR ===\n")
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )

  ast := compile(env,
    `request.auth.claims.group == 'admin'
        || request.auth.principal == 'user:[e-mail]'`,
    cel.BoolType)
  program, _ := env.Program(ast)
}

And finally, add the eval case that evaluates the user with an empty claim set:

// exercise3 demonstrates how CEL's commutative logical operators work.
//
// Construct an expression which checks whether the request.auth.claims.group
// value is equal to admin or the request.auth.principal is
// user:[e-mail]. Issue two requests, one that specifies the proper user,
// and one that specifies an unexpected user.
func exercise3() {
  fmt.Println("=== Exercise 3: Logical AND/OR ===\n")
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )

  ast := compile(env,
    `request.auth.claims.group == 'admin'
        || request.auth.principal == 'user:[e-mail]'`,
    cel.BoolType)
  program, _ := env.Program(ast)

  emptyClaims := map[string]string{}
  eval(program, request(auth("user:[e-mail]", emptyClaims), time.Now()))
}

Pro-Tip: Make sure you read the request(), auth(), helper functions in this file. Along with time.Now(), they're useful to adapt to construct the input to the evaluation.

Run the code with the empty claim set

Rerunning the program, you should see the following new output:

=== Exercise 3: Logical AND/OR ===

request.auth.claims.group == 'admin'
    || request.auth.principal == 'user:[e-mail]'

------ input ------
request = time: <
  seconds: 1569302377
>
auth: <
  principal: "user:[e-mail]"
  claims: <
  >
>

------ result ------
value: true (types.Bool)

Update the evaluation case

Next, update the evaluation case to pass in a different principal with the empty claim set:

// exercise3 demonstrates how CEL's commutative logical operators work.
//
// Construct an expression which checks whether the request.auth.claims.group
// value is equal to admin or the request.auth.principal is
// user:[e-mail]. Issue two requests, one that specifies the proper user,
// and one that specifies an unexpected user.
func exercise3() {
  fmt.Println("=== Exercise 3: Logical AND/OR ===\n")
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
  )

  ast := compile(env,
    `request.auth.claims.group == 'admin'
        || request.auth.principal == 'user:[e-mail]'`,
    cel.BoolType)
  program, _ := env.Program(ast)

  emptyClaims := map[string]string{}
  eval(program, request(auth("other:[e-mail]", emptyClaims), time.Now()))
}

Run the code with a time

Rerunning the program,

go run .

you should see the following error:

=== Exercise 3: Logical AND/OR ===

request.auth.claims.group == 'admin'
    || request.auth.principal == 'user:[e-mail]'

------ input ------
request = time: <
  seconds: 1588989833
>
auth: <
  principal: "user:[e-mail]"
  claims: <
  >
>

------ result ------
value: true (types.Bool)

------ input ------
request = time: <
  seconds: 1588989833
>
auth: <
  principal: "other:[e-mail]"
  claims: <
  >
>

------ result ------
error: no such key: group

In protobuf, we know what fields and types to expect. In map and json values, we don't know if a key will be present. Since there's not a safe default value for a missing key, CEL defaults to error.

8. Custom Functions

While CEL includes many built-in functions, there are occasions where a custom function is useful. For example, custom functions can be used to improve the user experience for common conditions or expose context-sensitive state

In this exercise, we'll be exploring how to expose a function to package together commonly used checks.

Call a custom function

First, create the code to set up an override called contains that determines if a key exists in a map and has a particular value. Leave placeholders for the function definition and function binding:

// exercise4 demonstrates how to extend CEL with custom functions.
// Declare a `contains` member function on map types that returns a boolean
// indicating whether the map contains the key-value pair.
func exercise4() {
  fmt.Println("=== Exercise 4: Customization ===\n")
  // Determine whether an optional claim is set to the proper value. The
  // custom map.contains(key, value) function is used as an alternative to:
  //   key in map && map[key] == value
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    // Declare the request.
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
    // Add the custom function declaration and binding here with cel.Function()
  )
  ast := compile(env,
    `request.auth.claims.contains('group', 'admin')`,
    cel.BoolType)

  // Construct the program plan and provide the 'contains' function impl.
  // Output: false
  program, _ := env.Program(ast)
  emptyClaims := map[string]string{}
  eval(
    program,
    request(auth("user:[e-mail]", emptyClaims), time.Now()),
  )
  fmt.Println()
}  

Run the code and understand the error

Rerunning the code, you should see the following error:

ERROR: <input>:1:29: found no matching overload for 'contains' applied to 'map(string, dyn).(string, string)'
 | request.auth.claims.contains('group', 'admin')
 | ............................^

To fix the error, we need to add the contains function to the list of declarations which currently declares the request variable.

Aside: Declaring a function is similar to declaring a variable. A function must indicate it's common name and enumerate a set of overloads with unique signatures.

Declare a parameterized type by adding the following 3 lines. (This is as complicated as any function overload will be for CEL)

// exercise4 demonstrates how to extend CEL with custom functions.
// Declare a `contains` member function on map types that returns a boolean
// indicating whether the map contains the key-value pair.
func exercise4() {
  fmt.Println("=== Exercise 4: Customization ===\n")
  // Determine whether an optional claim is set to the proper value. The custom
  // map.contains(key, value) function is used as an alternative to:
  //   key in map && map[key] == value

  // Useful components of the type-signature for 'contains'.
  typeParamA := cel.TypeParamType("A")
  typeParamB := cel.TypeParamType("B")
  mapAB := cel.MapType(typeParamA, typeParamB)

  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    // Declare the request.
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),
    // Add the custom function declaration and binding here with cel.Function()
  )
  ast := compile(env,
    `request.auth.claims.contains('group', 'admin')`,
    cel.BoolType)

  // Construct the program plan.
  // Output: false
  program, _ := env.Program(ast)
  emptyClaims := map[string]string{}
  eval(program, request(auth("user:[e-mail]", emptyClaims), time.Now()))
  fmt.Println()
}

Add the custom function

Next, we'll add a new contains function that will use the parameterized types:

// exercise4 demonstrates how to extend CEL with custom functions.
// Declare a `contains` member function on map types that returns a boolean
// indicating whether the map contains the key-value pair.
func exercise4() {
  fmt.Println("=== Exercise 4: Customization ===\n")
  // Determine whether an optional claim is set to the proper value. The custom
  // map.contains(key, value) function is used as an alternative to:
  //   key in map && map[key] == value

  // Useful components of the type-signature for 'contains'.
  typeParamA := cel.TypeParamType("A")
  typeParamB := cel.TypeParamType("B")
  mapAB := cel.MapType(typeParamA, typeParamB)
 
  // Env declaration.
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    // Declare the request.
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),   
    // Declare the custom contains function and its implementation.
    cel.Function("contains",
      cel.MemberOverload(
        "map_contains_key_value",
        []*cel.Type{mapAB, typeParamA, typeParamB},
        cel.BoolType,
        // Provide the implementation using cel.FunctionBinding()
      ),
    ),
  )
  ast := compile(env,
    `request.auth.claims.contains('group', 'admin')`,
    cel.BoolType)

  // Construct the program plan and provide the 'contains' function impl.
  // Output: false
  program, _ := env.Program(ast)
  emptyClaims := map[string]string{}
  eval(program, request(auth("user:[e-mail]", emptyClaims), time.Now()))
  fmt.Println()
}

Run the program to understand the error

Run the exercise. You should see the following error about the missing runtime function:

------ result ------
error: no such overload: contains

Provide the function implementation to the NewEnv declaration using the cel.FunctionBinding() function:

// exercise4 demonstrates how to extend CEL with custom functions.
//
// Declare a contains member function on map types that returns a boolean
// indicating whether the map contains the key-value pair.
func exercise4() {
  fmt.Println("=== Exercise 4: Customization ===\n")
  // Determine whether an optional claim is set to the proper value. The custom
  // map.contains(key, value) function is used as an alternative to:
  //   key in map && map[key] == value

  // Useful components of the type-signature for 'contains'.
  typeParamA := cel.TypeParamType("A")
  typeParamB := cel.TypeParamType("B")
  mapAB := cel.MapType(typeParamA, typeParamB)

  // Env declaration.
  env, _ := cel.NewEnv(
    cel.Types(&rpcpb.AttributeContext_Request{}),
    // Declare the request.
    cel.Variable("request",
      cel.ObjectType("google.rpc.context.AttributeContext.Request"),
    ),   
    // Declare the custom contains function and its implementation.
    cel.Function("contains",
      cel.MemberOverload(
        "map_contains_key_value",
        []*cel.Type{mapAB, typeParamA, typeParamB},
        cel.BoolType,
        cel.FunctionBinding(mapContainsKeyValue)),
    ),
  )
  ast := compile(env, 
    `request.auth.claims.contains('group', 'admin')`, 
    cel.BoolType)

  // Construct the program plan.
  // Output: false
  program, err := env.Program(ast)
  if err != nil {
    glog.Exit(err)
  }

  eval(program, request(auth("user:[e-mail]", emptyClaims), time.Now()))
  claims := map[string]string{"group": "admin"}
  eval(program, request(auth("user:[e-mail]", claims), time.Now()))
  fmt.Println()
}

The program should now run successfully:

=== Exercise 4: Custom Functions ===

request.auth.claims.contains('group', 'admin')

------ input ------
request = time: <
  seconds: 1569302377
>
auth: <
  principal: "user:[e-mail]"
  claims: <
  >
>

------ result ------
value: false (types.Bool)

What happens when the claim exists?

For extra credit, try setting the admin claim on the input to verify the contains overload also returns true when the claim exists. You should see the following output:

=== Exercise 4: Customization ===

request.auth.claims.contains('group','admin')

------ input ------
request = time: <
  seconds: 1588991010
>
auth: <
  principal: "user:[e-mail]"
  claims: <
  >
>

------ result ------
value: false (types.Bool)

------ input ------
request = time: <
  seconds: 1588991010
>
auth: <
  principal: "user:[e-mail]"
  claims: <
    fields: <
      key: "group"
      value: <
        string_value: "admin"
      >
    >
  >
>

------ result ------
value: true (types.Bool)

Best practice: Declare overload ids according to their types and function names. e.g. targetType_func_argType_argType. In the case where argType is a type param, use a descriptive name instead of the simple type name.

Before moving on, it's worth inspecting the mapContainsKeyValue function itself:

// mapContainsKeyValue implements the custom function:
//   map.contains(key, value) -> bool.
func mapContainsKeyValue(args ...ref.Val) ref.Val {
  // The declaration of the function ensures that only arguments which match
  // the mapContainsKey signature will be provided to the function.
  m := args[0].(traits.Mapper)

  // CEL has many interfaces for dealing with different type abstractions.
  // The traits.Mapper interface unifies field presence testing on proto
  // messages and maps.
  key := args[1]
  v, found := m.Find(key)

  // If not found and the value was non-nil, the value is an error per the
  // `Find` contract. Propagate it accordingly. Such an error might occur with
  // a map whose key-type is listed as 'dyn'.
  if !found {
    if v != nil {
      return types.ValOrErr(v, "unsupported key type")
    }
    // Return CEL False if the key was not found.
    return types.False
  }
  // Otherwise whether the value at the key equals the value provided.
  return v.Equal(args[2])
}

In order to provide the greatest ease of extension, the signature for custom functions expects the arguments of ref.Val type. The tradeoff here is that the ease of extension adds a burden on the implementor to ensure all value types are handled properly. When the input argument types or count don't match the function declaration, a no such overload error should be returned.

The cel.FunctionBinding() adds a runtime type guard to ensure that the runtime contract matches the type-checked declaration in the environment.

Best practice: Use types.ValOrErr or types.MaybeNoSuchOverload to construct and propagate errors when the inputs don't match the expected runtime contract.

9. Building JSON

CEL can also produce non-boolean outputs, such as JSON. Add the following to your function:

// exercise5 covers how to build complex objects as CEL literals.
//
// Given the input now, construct a JWT with an expiry of 5 minutes.
func exercise5() {
    fmt.Println("=== Exercise 5: Building JSON ===\n")
    env, _ := cel.NewEnv(
      // Declare the 'now' variable as a Timestamp.
      // cel.Variable("now", cel.TimestampType),
    )
    // Note the quoted keys in the CEL map literal. For proto messages the
    // field names are unquoted as they represent well-defined identifiers.
    ast := compile(env, `
        {'sub': 'serviceAccount:[e-mail]',
         'aud': 'my-project',
         'iss': 'auth.acme.com:12350',
         'iat': now,
         'nbf': now,
         'exp': now + duration('300s'),
         'extra_claims': {
             'group': 'admin'
         }}`,
        cel.MapType(cel.StringType, cel.DynType))

    program, _ := env.Program(ast)
    out, _, _ := eval(
        program,
        map[string]interface{}{
            "now": &tpb.Timestamp{Seconds: time.Now().Unix()},
        },
    )
    fmt.Printf("------ type conversion ------\n%v\n", out)
    fmt.Println()
}

Run the code

Rerunning the code, you should see the following error:

ERROR: <input>:5:11: undeclared reference to 'now' (in container '')
 |    'iat': now,
 | ..........^
... and more ...

Add a declaration for the now variable of type cel.TimestampType to cel.NewEnv() and run again:

// exercise5 covers how to build complex objects as CEL literals.
//
// Given the input now, construct a JWT with an expiry of 5 minutes.
func exercise5() {
    fmt.Println("=== Exercise 5: Building JSON ===\n")
    env, _ := cel.NewEnv(
      cel.Variable("now", cel.TimestampType),
    )
    // Note the quoted keys in the CEL map literal. For proto messages the
    // field names are unquoted as they represent well-defined identifiers.
    ast := compile(env, `
        {'sub': 'serviceAccount:[e-mail]',
         'aud': 'my-project',
         'iss': 'auth.acme.com:12350',
         'iat': now,
         'nbf': now,
         'exp': now + duration('300s'),
         'extra_claims': {
             'group': 'admin'
         }}`,
        cel.MapType(cel.StringType, cel.DynType))

     // Hint:
     // Convert `out` to JSON using the valueToJSON() helper method.
     // The valueToJSON() function calls ConvertToNative(&structpb.Value{})
     // to adapt the CEL value to a protobuf JSON representation and then
     // uses the jsonpb.Marshaler utilities on the output to render the JSON
     // string.
    program, _ := env.Program(ast)
    out, _, _ := eval(
        program,
        map[string]interface{}{
            "now": time.Now(),
        },
    )
    fmt.Printf("------ type conversion ------\n%v\n", out)
    fmt.Println()
}

Rerun the code, and it should succeed:

=== Exercise 5: Building JSON ===

  {'aud': 'my-project',
   'exp': now + duration('300s'),
   'extra_claims': {
    'group': 'admin'
   },
   'iat': now,
   'iss': 'auth.acme.com:12350',
   'nbf': now,
   'sub': 'serviceAccount:[e-mail]'
   }

------ input ------
now = seconds: 1569302377

------ result ------
value: &{0xc0002eaf00 map[aud:my-project exp:{0xc0000973c0} extra_claims:0xc000097400 iat:{0xc000097300} iss:auth.acme.com:12350 nbf:{0xc000097300} sub:serviceAccount:[e-mail]] {0x8c8f60 0xc00040a6c0 21}} (*types.baseMap)

------ type conversion ------
&{0xc000313510 map[aud:my-project exp:{0xc000442510} extra_claims:0xc0004acdc0 iat:{0xc000442450} iss:auth.acme.com:12350 nbf:{0xc000442450} sub:serviceAccount:[e-mail]] {0x9d0ce0 0xc0004424b0 21}}

Note: Timestamps and durations are well-known types and have special convenience types within CEL. The google.protobuf.Timestamp and google.protobuf.Duration message types are equivalent to the convenience types; however, the fields of all well-known types are not directly accessible; instead, they are mediated by member functions.

The program runs, but the output value out needs to be explicitly converted to JSON. The internal CEL representation in this case is JSON convertible as it only refers to types that JSON can support or for which there is a well-known Proto to JSON mapping.

// exercise5 covers how to build complex objects as CEL literals.
//
// Given the input now, construct a JWT with an expiry of 5 minutes.
func exercise5() {
    fmt.Println("=== Exercise 5: Building JSON ===\n")
...
    fmt.Printf("------ type conversion ------\n%v\n", valueToJSON(out))
    fmt.Println()
}

Once the type has been converted using the valueToJSON helper function within the codelab.go file you should see the following additional output:

------ type conversion ------
  {
   "aud": "my-project",
   "exp": "2019-10-13T05:54:29Z",
   "extra_claims": {
      "group": "admin"
     },
   "iat": "2019-10-13T05:49:29Z",
   "iss": "auth.acme.com:12350",
   "nbf": "2019-10-13T05:49:29Z",
   "sub": "serviceAccount:[e-mail]"
  }

10. Building Protos

CEL can build protobuf messages for any message type compiled into the application. Add the function to build a google.rpc.context.AttributeContext.Request from an input jwt

// exercise6 describes how to build proto message types within CEL.
//
// Given an input jwt and time now construct a
// google.rpc.context.AttributeContext.Request with the time and auth
// fields populated according to the go/api-attributes specification.
func exercise6() {
  fmt.Println("=== Exercise 6: Building Protos ===\n")

  // Construct an environment and indicate that the container for all references
  // within the expression is `google.rpc.context.AttributeContext`.
  requestType := &rpcpb.AttributeContext_Request{}
  env, _ := cel.NewEnv(
      // Add cel.Container() option for 'google.rpc.context.AttributeContext'
      cel.Types(requestType),
      // Add cel.Variable() option for 'jwt' as a map(string, Dyn) type
      // and for 'now' as a timestamp.
  )

  // Compile the Request message construction expression and validate that
  // the resulting expression type matches the fully qualified message name.
  //
  // Note: the field names within the proto message types are not quoted as they
  // are well-defined names composed of valid identifier characters. Also, note
  // that when building nested proto objects, the message name needs to prefix 
  // the object construction.
  ast := compile(env, `
    Request{
        auth: Auth{
            principal: jwt.iss + '/' + jwt.sub,
            audiences: [jwt.aud],
            presenter: 'azp' in jwt ? jwt.azp : "",
            claims: jwt
        },
        time: now
    }`,
    cel.ObjectType("google.rpc.context.AttributeContext.Request"),
  )
  program, _ := env.Program(ast)

  // Construct the message. The result is a ref.Val that returns a dynamic
  // proto message.
  out, _, _ := eval(
      program,
      map[string]interface{}{
          "jwt": map[string]interface{}{
              "sub": "serviceAccount:[e-mail]",
              "aud": "my-project",
              "iss": "auth.acme.com:12350",
              "extra_claims": map[string]string{
                  "group": "admin",
              },
          },
          "now": time.Now(),
      },
  )

  // Hint: Unwrap the CEL value to a proto. Make sure to use the
  // `ConvertToNative(reflect.TypeOf(requestType))` to convert the dynamic proto
  // message to the concrete proto message type expected.
  fmt.Printf("------ type unwrap ------\n%v\n", out)
  fmt.Println()
}

Run the code

Rerunning the code, you should see the following error:

ERROR: <input>:2:10: undeclared reference to 'Request' (in container '')
 |   Request{
 | .........^

The container is basically the equivalent of a namespace or package, but can even be as granular as a protobuf message name. CEL containers use the same namespace resolution rules as Protobuf and C++ for determining where a given variable, function, or type name is declared.

Given the container google.rpc.context.AttributeContext the type-checker and the evaluator will try the following identifier names for all variables, types, and functions:

• google.rpc.context.AttributeContext.<id>
• google.rpc.context.<id>
• google.rpc.<id>
• google.<id>
• <id>

For absolute names, prefix the variable, type, or function reference with a leading dot. In the example, the expression .<id> will only search for the top-level <id> identifier without first checking within the container.

Try specifying the cel.Container("google.rpc.context.AttributeContext") option to CEL environment and run again:

// exercise6 describes how to build proto message types within CEL.
//
// Given an input jwt and time now construct a
// google.rpc.context.AttributeContext.Request with the time and auth
// fields populated according to the go/api-attributes specification.
func exercise6() {
  fmt.Println("=== Exercise 6: Building Protos ===\n")
  // Construct an environment and indicate that the container for all references
  // within the expression is `google.rpc.context.AttributeContext`.
  requestType := &rpcpb.AttributeContext_Request{}
  env, _ := cel.NewEnv(
    // Add cel.Container() option for 'google.rpc.context.AttributeContext'
    cel.Container("google.rpc.context.AttributeContext.Request"),
    cel.Types(requestType),
    // Later, add cel.Variable() options for 'jwt' as a map(string, Dyn) type
    // and for 'now' as a timestamp.
    // cel.Variable("now", cel.TimestampType),
    // cel.Variable("jwt", cel.MapType(cel.StringType, cel.DynType)),
  )

 ...
}

You should get the following output:

ERROR: <input>:4:16: undeclared reference to 'jwt' (in container 'google.rpc.context.AttributeContext')
 |     principal: jwt.iss + '/' + jwt.sub,
 | ...............^

... and many more errors...

Next, declare the jwt and now variables and the program should run as expected:

=== Exercise 6: Building Protos ===

  Request{
   auth: Auth{
    principal: jwt.iss + '/' + jwt.sub,
    audiences: [jwt.aud],
    presenter: 'azp' in jwt ? jwt.azp : "",
    claims: jwt
   },
   time: now
  }

------ input ------
jwt = {
  "aud": "my-project",
  "extra_claims": {
    "group": "admin"
  },
  "iss": "auth.acme.com:12350",
  "sub": "serviceAccount:[e-mail]"
}
now = seconds: 1588993027

------ result ------
value: &{0xc000485270 0xc000274180 {0xa6ee80 0xc000274180 22} 0xc0004be140 0xc0002bf700 false} (*types.protoObj)

----- type unwrap ----
&{0xc000485270 0xc000274180 {0xa6ee80 0xc000274180 22} 0xc0004be140 0xc0002bf700 false}

For extra credit, you should also unwrap the type by calling out.Value() on the message to see how the result changes.

Additional considerations for using CEL to build protos:

1. There is no native support for the conditional assignment for proto's oneof fields.
2. There are issues round-tripping to / from a google.protobuf.Any.

When a oneof needs to be set, test whether the desired value is present before constructing the message, or extend CEL to include a custom object building function such as a wither method, or perhaps something more abstract.

When an Any contains a wrapper type such as google.protobuf.IntValue, CEL automatically unpacks the Any to int or null_type depending on the contents. In the case where the wrapper type is unset, the null could not be assigned to an Any field and have its same original meaning. So far, this is the only roundtripping issue we have discovered, but it's worth noting.

11. Macros

Macros can be used to manipulate the CEL program at parse time. Macros match a call signature and manipulate the input call and its arguments in order to produce a new subexpression AST.

Macros can be used to implement complex logic in the AST that can't be written directly in CEL. For example, the has macro enables field presence testing. The comprehension macros such as exists and all replace a function call with bounded iteration over an input list or map. Neither concept is possible at a syntactic level, but they are possible through macro expansions.

Add and run the next exercise:

// exercise7 introduces macros for dealing with repeated fields and maps.
//
// Determine whether the jwt.extra_claims has at least one key that starts
// with the group prefix, and ensure that all group-like keys have list
// values containing only strings that end with '@acme.co`.
func exercise7() {
    fmt.Println("=== Exercise 7: Macros ===\n")
    env, _ := cel.NewEnv(
      cel.ClearMacros(),
      cel.Variable("jwt", cel.MapType(cel.StringType, cel.DynType)),
    )
    ast := compile(env,
        `jwt.extra_claims.exists(c, c.startsWith('group'))
                && jwt.extra_claims
                      .filter(c, c.startsWith('group'))
                      .all(c, jwt.extra_claims[c]
                                 .all(g, g.endsWith('@acme.co')))`,
        cel.BoolType)
    program, _ := env.Program(ast)

    // Evaluate a complex-ish JWT with two groups that satisfy the criteria.
    // Output: true.
    eval(program,
        map[string]interface{}{
            "jwt": map[string]interface{}{
                "sub": "serviceAccount:[e-mail]",
                "aud": "my-project",
                "iss": "auth.acme.com:12350",
                "extra_claims": map[string][]string{
                    "group1": {"[e-mail]", "[e-mail]"},
                    "labels": {"metadata", "prod", "pii"},
                    "groupN": {"[e-mail]"},
                },
            },
        })

    fmt.Println()
}

You should see the following errors:

ERROR: <input>:1:25: undeclared reference to 'c' (in container '')
 | jwt.extra_claims.exists(c, c.startsWith('group'))
 | ........................^

... and many more ...

These errors occur because macros are not yet enabled. To enable macros, remove cel.ClearMacros(), and run again:

=== Exercise 7: Macros ===

jwt.extra_claims.exists(c, c.startsWith('group'))
    && jwt.extra_claims
       .filter(c, c.startsWith('group'))
       .all(c, jwt.extra_claims[c]
              .all(g, g.endsWith('@acme.co')))

------ input ------
jwt = {
  "aud": "my-project",
  "extra_claims": {
    "group1": [
      "[e-mail]",
      "[e-mail]"
    ],
    "groupN": [
      "[e-mail]"
    ],
    "labels": [
      "metadata",
      "prod",
      "pii"
    ]
  },
  "iss": "auth.acme.com:12350",
  "sub": "serviceAccount:[e-mail]"
}

------ result ------
value: true (types.Bool)

Best practice: Disable macros using the cel.ClearMacros() option during environment construction if the use case doesn't require them.

These are the currently supported macros:

Macro	Signature	Description
all	r.all(var, cond)	Test if cond evaluates true for all var in range r.
exists	r.exists(var, cond)	Test if cond evaluates true for any var in range r.
exists_one	r.exists_one(var, cond)	Test if cond evaluates true for only one var in range r.
filter	r.filter(var, cond)	For lists, create a new list where each element var in range r satisfies the condition cond. For maps, create a new list where each key var in range r satisfies the condition cond.
map	r.map(var, expr)	Create a new list where each var in range r is transformed by expr.
	r.map(var, cond, expr)	Same as two-arg map but with a conditional cond filter before the value is transformed.
has	has(a.b)	Presence test for b on value a : For maps, json tests definition. For protos, tests non-default primitive value or a or a set message field.

When the range r argument is a map type, the var will be the map key, and for list type values the var will be the list element value. The all, exists, exists_one, filter, and map macros perform an AST rewrite that performs a for-each iteration that is bounded by the size of the input.

The bounded comprehensions ensure that CEL programs won't be Turing-complete, but they evaluate in super-linear time with respect to the input. Use these macros sparingly or not at all. Heavy use of comprehensions usually a good indicator that a custom function would provide a better user experience and better performance.

12. Tuning

There are a handful of features which are exclusive to CEL-Go at the moment, but which are indicative of future plans for other CEL implementations. The following exercise showcases different program plans for the same AST:

// exercise8 covers features of CEL-Go which can be used to improve
// performance and debug evaluation behavior.
//
// Turn on the optimization, exhaustive eval, and state tracking
// ProgramOption flags to see the impact on evaluation behavior.
func exercise8() {
    fmt.Println("=== Exercise 8: Tuning ===\n")
    // Declare the x and 'y' variables as input into the expression.
    env, _ := cel.NewEnv(
        cel.Variable("x", cel.IntType),
        cel.Variable("y", cel.UintType),
    )
    ast := compile(env,
        `x in [1, 2, 3, 4, 5] && type(y) == uint`,
        cel.BoolType)

    // Try the different cel.EvalOptions flags when evaluating this AST for
    // the following use cases:
    // - cel.OptOptimize: optimize the expression performance.
    // - cel.OptExhaustiveEval: turn off short-circuiting.
    // - cel.OptTrackState: track state and compute a residual using the
    //   interpreter.PruneAst function.
    program, _ := env.Program(ast)
    eval(program, cel.NoVars())

    fmt.Println()
}

Optimize

When the optimization flag is turned on, CEL will spend extra time to build list and map literals ahead of time and optimize certain calls such as the in operator to be a true set membership test:

    // Turn on optimization.
    trueVars := map[string]interface{}{"x": int64(4), "y": uint64(2)}
    program, _ := env.Program(ast, cel.EvalOptions(cel.OptOptimize))
    // Try benchmarking with the optimization flag on and off.
    eval(program, trueVars)

When the same program is evaluated many times over against different inputs, then optimizing is a good choice. However, when the program is only going to be evaluated once, optimizing will just add overhead.

Exhaustive Eval

Exhaustive Eval can be useful for debugging expression evaluation behavior as it provides insight into the observed value at each step in the expression evaluation.

    // Turn on exhaustive eval to see what the evaluation state looks like.
    // The input is structure to show a false on the first branch, and true
    // on the second.
    falseVars := map[string]interface{}{"x": int64(6), "y": uint64(2)}
    program, _ = env.Program(ast, cel.EvalOptions(cel.OptExhaustiveEval))
    eval(program, falseVars)

You should see a list of the expression evaluation state for each expression id:

------ eval states ------
1: 6 (types.Int)
2: false (types.Bool)
3: &{0xc0000336b0 [1 2 3 4 5] {0x89f020 0xc000434760 151}} (*types.baseList)
4: 1 (types.Int)
5: 2 (types.Int)
6: 3 (types.Int)
7: 4 (types.Int)
8: 5 (types.Int)
9: uint (*types.Type)
10: 2 (types.Uint)
11: true (types.Bool)
12: uint (*types.Type)
13: false (types.Bool)

Expression id 2 corresponds to the result of the in operator in the first. branch, and expression id 11 corresponds to the == operator in the second. Under normal evaluation, the expression would have short-circuited after 2 had been computed. If y had not been uint, then the state would have shown two reasons why the expression would have failed and not just one.

Note: Comprehension state is not yet properly tracked during exhaustive eval. See issues/121.

13. What was covered?

If you need an expression engine, consider using CEL. CEL is ideal for projects that need to execute user configuration where performance is critical.

In the previous exercises, we hope you became comfortable passing your data into CEL and getting the output or decision back out.

We hope you have a sense for the kind of operations you can do, anything from a boolean decision to generating JSON and Protobuffer messages.

We hope you have a sense of how to work with the expressions, and what they're doing. And we understand common ways to extend it.