Custom Transformers

FlyQL’s transformer system is extensible. You can register custom transformers that work identically to built-ins (upper, lower, len) across parsers, SQL generators, in-memory matcher, and editor autocomplete.

Transformer Interface

Every transformer implements 6 members:

Member	Type	Description
`name`	string	Unique lowercase identifier (e.g., `"firstoctet"`)
`inputType`	TransformerType	Expected input type (`string`, `int`, `float`, `bool`, `array`)
`outputType`	TransformerType	Output type after transformation
`argSchema`	ArgSpec[]	Declared argument types and arity (default: empty = no args)
`sql(dialect, columnRef, args)`	string	Generate dialect-specific SQL wrapping the column reference
`apply(value, args)`	any	Transform a runtime value in-memory

The sql() method receives the SQL dialect name ("clickhouse", "postgresql", "starrocks"), a column reference string, and the parsed argument list. It returns a SQL expression wrapping that reference. The apply() method receives a runtime value and the argument list, and returns the transformed value.

ArgSpec

Each entry in argSchema declares one expected argument:

Field	Type	Default	Description
`type`	TransformerType	—	Expected argument type (`string`, `int`, `float`, `bool`)
`required`	boolean	`true`	Whether the argument must be provided

Arity is derived from the schema length: required args must be present, trailing optional args may be omitted. Transformers with no arguments (like upper, lower, len) inherit the default empty schema.

Registry API

Method	Description
`defaultRegistry()`	Returns a fresh registry pre-loaded with built-in transformers
`registry.register(transformer)`	Add a custom transformer (raises error if name already registered)
`registry.get(name)`	Look up a transformer by name (returns null/None/nil if not found)
`registry.names()`	List all registered transformer names

The pattern: get a default registry, register your custom transformers, pass the registry to generators and matcher.

Python

Define a custom transformer by subclassing Transformer:

from flyql.transformers import Transformer, TransformerType, default_registry

class FirstOctetTransformer(Transformer):
    @property
    def name(self): return "firstoctet"
    @property
    def input_type(self): return TransformerType.STRING
    @property
    def output_type(self): return TransformerType.INT
    def sql(self, dialect, column_ref, args=None):
        if dialect == "clickhouse":
            return f"toUInt8(splitByChar('.', {column_ref})[1])"
        return f"CAST(SPLIT_PART({column_ref}, '.', 1) AS INTEGER)"
    def apply(self, value, args=None):
        return int(str(value).split('.')[0])

The arg_schema class attribute defaults to () (no arguments). To declare arguments, override it with a tuple of ArgSpec:

from typing import ClassVar, Tuple
from flyql.transformers.base import ArgSpec

class SubstringTransformer(Transformer):
    arg_schema: ClassVar[Tuple[ArgSpec, ...]] = (
        ArgSpec(type=TransformerType.INT),                    # start (required)
        ArgSpec(type=TransformerType.INT, required=False),    # length (optional)
    )
    # ... name, input_type, output_type, sql, apply ...

from flyql import parse
from flyql.generators.postgresql.generator import to_sql_where, Column

registry = default_registry()
registry.register(FirstOctetTransformer())

result = parse("src_ip|firstoctet > 192")
columns = {"src_ip": Column("src_ip", "text")}
sql = to_sql_where(result.root, columns, registry=registry)
# CAST(SPLIT_PART(src_ip, '.', 1) AS INTEGER) > 192

For in-memory matching:

from flyql.matcher.evaluator import Evaluator
from flyql.matcher.record import Record

evaluator = Evaluator(registry=registry)
record = Record({"src_ip": "193.0.0.1"})
evaluator.evaluate(result.root, record)  # True (193 > 192)

JavaScript

Define a custom transformer by extending Transformer:

import { Transformer, TransformerType, ArgSpec, defaultRegistry } from 'flyql/transformers'

class FirstOctetTransformer extends Transformer {
    get name() { return 'firstoctet' }
    get inputType() { return TransformerType.STRING }
    get outputType() { return TransformerType.INT }
    sql(dialect, columnRef, args = []) {
        if (dialect === 'clickhouse') return `toUInt8(splitByChar('.', ${columnRef})[1])`
        return `CAST(SPLIT_PART(${columnRef}, '.', 1) AS INTEGER)`
    }
    apply(value, args = []) { return parseInt(String(value).split('.')[0], 10) }
}

The argSchema getter defaults to [] (no arguments). To declare arguments, override it:

class SubstringTransformer extends Transformer {
    get argSchema() {
        return [
            new ArgSpec(TransformerType.INT, true),     // start (required)
            new ArgSpec(TransformerType.INT, false),    // length (optional)
        ]
    }
    // ... name, inputType, outputType, sql, apply ...
}

import { parse } from 'flyql'
import { generateWhere, newColumn } from 'flyql/generators/postgresql'

const registry = defaultRegistry()
registry.register(new FirstOctetTransformer())

const result = parse("src_ip|firstoctet > 192")
const columns = { src_ip: newColumn({ name: 'src_ip', type: 'text' }) }
const sql = generateWhere(result.root, columns, registry)
// CAST(SPLIT_PART(src_ip, '.', 1) AS INTEGER) > 192

For in-memory matching:

import { match } from 'flyql/matcher'

match("src_ip|firstoctet > 192", { src_ip: "193.0.0.1" }, registry) // true

For editor autocomplete — custom transformers appear alongside built-ins:

import { EditorEngine } from 'flyql-vue'

const engine = new EditorEngine(columns, { registry })
// After typing "src_ip|", suggestions include "firstoctet", "upper", "lower", "len"

Go

Define a custom transformer by implementing the Transformer interface:

package main

import (
    "fmt"
    "strconv"
    "strings"
    "github.com/iamtelescope/flyql/golang/transformers"
)

type FirstOctet struct{}

func (f FirstOctet) Name() string               { return "firstoctet" }
func (f FirstOctet) InputType() transformers.TransformerType  { return transformers.TransformerTypeString }
func (f FirstOctet) OutputType() transformers.TransformerType { return transformers.TransformerTypeInt }
func (f FirstOctet) ArgSchema() []transformers.ArgSpec        { return []transformers.ArgSpec{} }
func (f FirstOctet) SQL(dialect, colRef string, args []any) string {
    if dialect == "clickhouse" {
        return fmt.Sprintf("toUInt8(splitByChar('.', %s)[1])", colRef)
    }
    return fmt.Sprintf("CAST(SPLIT_PART(%s, '.', 1) AS INTEGER)", colRef)
}
func (f FirstOctet) Apply(value interface{}, args []any) interface{} {
    parts := strings.SplitN(fmt.Sprint(value), ".", 2)
    n, _ := strconv.Atoi(parts[0])
    return n
}

To declare arguments, return ArgSpec entries from ArgSchema():

func (s Substring) ArgSchema() []transformers.ArgSpec {
    return []transformers.ArgSpec{
        {Type: transformers.TransformerTypeInt, Required: true},   // start (required)
        {Type: transformers.TransformerTypeInt, Required: false},  // length (optional)
    }
}

import (
    flyql "github.com/iamtelescope/flyql/golang"
    postgresqlgen "github.com/iamtelescope/flyql/golang/generators/postgresql"
)

registry := transformers.DefaultRegistry()
registry.Register(FirstOctet{})

result, _ := flyql.Parse("src_ip|firstoctet > 192")
columns := map[string]*postgresqlgen.Column{
    "src_ip": postgresqlgen.NewColumn("src_ip", "text", nil),
}
sql, _ := postgresqlgen.ToSQLWhere(result.Root, columns, registry)
// CAST(SPLIT_PART(src_ip, '.', 1) AS INTEGER) > 192

For in-memory matching:

import "github.com/iamtelescope/flyql/golang/matcher"

matched, _ := matcher.Match("src_ip|firstoctet > 192", map[string]any{"src_ip": "193.0.0.1"}, registry)
// matched == true

Type Compatibility

Custom transformers participate in the same type chain validation as built-ins. If you chain field|firstoctet|upper, the system detects that firstoctet outputs int but upper requires string.

Available types: see flyql.Type — the canonical 11-value enum (String, Int, Float, Bool, Date, Duration, Array, Map, Struct, JSON, Unknown).

Validation with `diagnose()`

The diagnose() function walks a parsed AST against a column schema and transformer registry, returning positioned Diagnostic records without throwing. It checks:

Unknown column — base key not in the column list
Unknown transformer — transformer name not in the registry
Wrong arity — too few or too many arguments vs argSchema
Wrong argument type — argument value type does not match declared ArgSpec type (int-to-float widening is allowed)
Chain type mismatch — output type of one transformer does not match the input type of the next, or column’s normalized type does not match the first transformer’s input type

Each diagnostic carries a range (source position), message, severity ("error"), and code ("unknown_column", "unknown_transformer", "arg_count", "arg_type", "chain_type").

# Python
from flyql import parse, Column, ColumnSchema, Type, diagnose

ast = parse("hstt|upper=X").root
columns = ColumnSchema.from_columns([Column(name="host", column_type=Type.String)])
diags = diagnose(ast, columns)  # columns is a ColumnSchema
# [Diagnostic(range=Range(0, 4), message="column 'hstt' is not defined", severity='error', code='unknown_column')]

// JavaScript
import { parse, Column, ColumnSchema, Type, diagnose } from 'flyql'

const ast = parse('hstt|upper=X').root
const columns = ColumnSchema.fromColumns([new Column('host', false, Type.String)])
const diags = diagnose(ast, columns)  // columns is a ColumnSchema
// [Diagnostic { range: Range(0, 4), message: "column 'hstt' is not defined", severity: 'error', code: 'unknown_column' }]

// Go
import flyql "github.com/iamtelescope/flyql/golang"

result, _ := flyql.Parse("hstt|upper=X")
schema := flyql.FromColumns([]flyql.Column{flyql.NewColumn("host", flyql.TypeString)})
diags := flyql.Diagnose(result.Root, schema, nil) // nil = default registry
// []Diagnostic{{Range: {0, 4}, Message: "column 'hstt' is not defined", Severity: "error", Code: "unknown_column"}}

Pass a custom registry as the third argument if you have custom transformers registered. Pass nil/None/null to use the default registry.

Security

Custom transformers’ sql() methods only receive column references (from the schema), never user-provided values. User values are escaped by the generator. This preserves FlyQL’s injection-proof guarantee — custom SQL wraps column names, not user input.

Custom Transformers

Transformer Interface

ArgSpec

Registry API

Python

JavaScript

Go

Type Compatibility

Validation with diagnose()

Security

See also

Validation with `diagnose()`