Code Internals

GraphJin code is made up of a number of packages. We have done our best to keep each package small and focused. Let us begin by looking at some of these packages.

  1. qcode - GraphQL lexer and parser.
  2. psql - SQL generator
  3. serv - HTTP Endpoint, Configs, CLI, etc
  4. rails - Rails cookie and session store decoders


This package contains the core of the GraphQL compiler it handling the lexing and parsing of the GraphQL query transforming it into an internal representation called QCode.

This is the first step of the compiling process the func NewCompiler(c Config) function creates a new instance of this compiler which has its own config.

Keep in mind QCode has no knowledge of the Database structure it is designed to be a fast GraphQL parser. Care is taken to keep memory allocations to a minimum.

const (

type QCode struct {
  Type      QType
  Selects   []Select

type Select struct {
	ID         int32
	ParentID   int32
	Args       map[string]*Node
	Name       string
	FieldName  string
	Cols       []Column
	Where      *Exp
	OrderBy    []*OrderBy
	DistinctOn []string
	Paging     Paging
	Children   []int32
	Functions  bool
	Allowed    map[string]struct{}
	PresetMap  map[string]string
	PresetList []string

But before the incoming GraphQL query can be turned into QCode it must first be tokenized by the lexer lex.go. As the tokenizer walks the bytes of the query it generates tokens item structs which are then consumed by the next step the parser parse.go.

type item struct {
	typ  itemType
	pos  Pos
	end  Pos

For example a simple query like query getUser { user { id } } will be converted into several tokens like below.

item{itemQuery, 0, 4} // query
item{itemName, 6, 12} // getUser
item{itemObjOpen, 16, 20} // {

These tokens are then fed into the parser parse.go the parser does the work of generating an abstract syntax tree (AST) from the tokens. This AST is an internal representation (data structure) and is not exposed outside the package. Since the AST is a tree a stack stack.go is used to walk the tree and generate the QCode AST. The QCode data structure is also a tree (represented as an array). This is then returned to the caller of the compile function.

type Operation struct {
	Type    parserType
	Name    string
	Args    []Arg
	Fields  []Field

type Field struct {
	ID        int32
	ParentID  int32
	Name      string
	Alias     string
	Args      []Arg
	Children  []int32


This package is responsible for generating Postgres SQL from the QCode AST. There are various GraphQL query types (Query, Mutation, etc). And several more sub types like single root or multi-root queries, various types of mutations (insert, update delete, bulk insert, etc). This package is designed to be able to generate SQL for all of those types.

In addition to QCode variable data is also passed to the compile function within this package. Variables are decoded to derive what is being inserted and what kind of insert is it single or bulk. This information is not available in the GraphQL query its passed in seperatly via variables. This package is able to put all this together and generate the right SQL code.

The entry point of this package is in query.go. The database schema must be passed in the config object when creating a new compiler instance NewCompiler. The functions to extract this schema from the database are also part of this package tables.go. The GetTables functions fetches all the tables from the database and GetColumns fetches columns and relationship information.

func NewCompiler(conf Config) *Compiler {
	return &Compiler{conf.Schema, conf.Vars}

func (co *Compiler) Compile(qc *qcode.QCode, w io.Writer, vars Variables) (uint32, error) {
	switch qc.Type {
	case qcode.QTQuery:
		return co.compileQuery(qc, w)
	case qcode.QTInsert, qcode.QTUpdate, qcode.QTDelete, qcode.QTUpsert:
		return co.compileMutation(qc, w, vars)

	return 0, fmt.Errorf("Unknown operation type %d", qc.Type)

GraphQL, input is first converted to QCode.

query {
  user {
  posts {

SQL, in reality the generated SQL is far more complex single it has to be very efficient, leverage the power of Postgres, support RBAC (Role based access control) and all of this must be done in a single SQL query.

SELECT, posts.title FROM users, posts;


The serv package contains most of code that turns the above compiler into an HTTP service. It also includes authentication middleware, remote join resolvers, config parsing, database migrations and seeding commands.

Another big feature that this package handles is the allow.list management code. In production mode parsing the allow list file and registering prepared statements to adding GraphQL queries to this file in development mode.

Currently the following global variables are referenced across the package. In future I’d prefer to move these into a context struct and pass that around instead.

var (
	logger   zerolog.Logger  // logger for everything but errors
	errlog   zerolog.Logger  // logger for errors includes line numbers
	conf     *config         // parsed config
	confPath string          // path to the config file
	db       *pgxpool.Pool   // database connection pool
	schema   *psql.DBSchema  // database tables, columns and relationships
	qcompile *qcode.Compiler // qcode compiler
	pcompile *psql.Compiler  // postgres sql compiler


There are several unit tests and benchmark tests parse_test.go) included. There are also scripts included for memory and cpu profiling.

// Test to ensure synthetic tables generate the correct SQL
func syntheticTables(t *testing.T) {
	gql := `query {
		me {

	sql := `SELECT json_object_agg('me', json_0) FROM (SELECT row_to_json((SELECT "json_row_0" FROM (SELECT ) AS "json_row_0")) AS "json_0" FROM (SELECT "users"."email" FROM "users" WHERE ((("users"."id") = '{{user_id}}' :: bigint)) LIMIT ('1') :: integer) AS "users_0" LIMIT ('1') :: integer) AS "sel_0"`

	resSQL, err := compileGQLToPSQL(gql, nil, "user")
	if err != nil {

	if string(resSQL) != sql {

You can run tests within each package or across the entire app. It is usually the fastest to first write a test and then build the feature to satisfy it.

go test -v ./...

Memory profiling can help find where allocations are happening within the package code.

$ cd ./psql
$ ./
goos: darwin
goarch: amd64
BenchmarkCompile-8                 52567             19401 ns/op            3918 B/op         61 allocs/op
BenchmarkCompileParallel-8        219548              5684 ns/op            3938 B/op         61 allocs/op
ok       2.582s
Type: alloc_space
Time: Nov 29, 2019 at 11:59pm (EST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 880.59MB, 80.63% of 1092.14MB total
Dropped 33 nodes (cum <= 5.46MB)
Showing top 10 nodes out of 35
      flat  flat%   sum%        cum   cum%
      22MB  2.01%  2.01%   903.57MB 82.73%*Compiler).Compile
         0     0%  2.01%   862.98MB 79.02%
         0     0%  2.01%   862.98MB 79.02%  testing.(*B).RunParallel.func1
  461.95MB 42.30% 44.31%   760.53MB 69.64%*Compiler).compileQuery
  396.63MB 36.32% 80.63%   396.63MB 36.32%
         0     0% 80.63%   252.07MB 23.08%*Compiler).compileArgs
         0     0% 80.63%   228.15MB 20.89%  testing.(*B).runN
         0     0% 80.63%   227.63MB 20.84%
         0     0% 80.63%   227.63MB 20.84%  testing.(*B).launch
         0     0% 80.63%   187.04MB 17.13%*Compiler).Compile


Most packages contain benchmark tests to ensure new features don’t introduce a significant regression to performance.

$ cd ./psql
$ go test -v -run=xx -bench=.
goos: darwin
goarch: amd64
BenchmarkCompile-8                 60775             19076 ns/op            3919 B/op         61 allocs/op
BenchmarkCompileParallel-8        207847              5172 ns/op            3937 B/op         61 allocs/op
ok       2.530s

Reach out

If you’d like me to explain other parts of the code please reach out over Twitter or Discord. I’ll keep adding to this doc as I get time.

Developing GraphJin

If you want to build and run GraphJin from code then the below commands will build the web ui and launch GraphJin in developer mode with a watcher to rebuild on code changes. And the demo rails app is also launched to make it easier to test changes.

# yarn install dependencies and build the web ui
make build

# do this the only the time to setup the database
docker-compose run rails_app rake db:create db:migrate db:seed

# start super graph in development mode with a change watcher
docker-compose up

Developing on the GraphJin UI

# yarn is needed to build the web ui
brew install yarn

# this is where the react app for the ui lives
cd internals/serv/web

# launch it in development mode
yarn start