CGO

Introduction

The initial Sparta release supported running a normal Go binary alongside a NodeJS HTTP-based proxy.

With Sparta v0.11.0 it’s possible to transform your Sparta application into a CGO-library proxied by a Python 3.6 ctypes interface. This provides significant cold-start & hot exeecution performance improvements.

Requirements

  • Docker - Tested on OSX:
  $ docker -v
  Docker version 17.03.1-ce, build c6d412)
  • Ability to build CGO libraries
  • You MUST call cgo.Main(...) from your application’s main package. The CGO functionality depends on being able to rewrite your application code into a CGO-compabible source. This enables Sparta to transform your application’s main() into a library-equivalent init() function to initialize the internal function registry.

Usage

To enable CGO packaging , choose either Sparta/cgo.Main or Sparta/cgo.MainEx functions. These functions are signature compatible with the existing Main* functions that produce in a NodeJS package.

Example

Before

// File main.go
package main
import (
  // ================================================== //
	sparta "github.com/mweagle/Sparta"
  // ================================================== //

	"github.com/mweagle/SpartaPython"
)
// HelloWorld is a standard Sparta AWS λ function
func HelloWorld(event *json.RawMessage,
	context *sparta.LambdaContext,
	w http.ResponseWriter,
	logger *logrus.Logger) {
...
}

////////////////////////////////////////////////////////////////////////////////
// Main
func main() {
	lambdaFn, _ := sparta.Lambda(sparta.IAMRoleDefinition{},
		HelloWorld,
		nil)

	var lambdaFunctions []*sparta.LambdaAWSInfo
	lambdaFunctions = append(lambdaFunctions, lambdaFn)
  // ================================================== //
	sparta.Main("SpartaHelloNodeJSProcess",
		fmt.Sprintf("Test HelloWorld resource command"),
		lambdaFunctions,
		nil,
		nil)
  // ================================================== //

}

After

// File main.go
package main
import (
  // ================================================== //
	spartaCGO "github.com/mweagle/Sparta/cgo"
  // ================================================== //
	"github.com/mweagle/SpartaPython"
)
// HelloWorld is a standard Sparta AWS λ function
func HelloWorld(event *json.RawMessage,
	context *sparta.LambdaContext,
	w http.ResponseWriter,
	logger *logrus.Logger) {
...
}

////////////////////////////////////////////////////////////////////////////////
// Main
func main() {
	lambdaFn, _ := sparta.Lambda(sparta.IAMRoleDefinition{},
		HelloWorld,
		nil)

	var lambdaFunctions []*sparta.LambdaAWSInfo
	lambdaFunctions = append(lambdaFunctions, lambdaFn)
  // ================================================== //
	spartaCGO.Main("SpartaHelloPythonCGO",
		fmt.Sprintf("Test HelloWorld resource command"),
		lambdaFunctions,
		nil,
		nil)
  // ================================================== //

}

You should see log output that includes a statement similar to:

INFO[0000] Building `cgo` library in Docker              Args=[run --rm -v /Users/mweagle/Documents/gopath:/usr/src/gopath -w /usr/src/gopath/src/github.com/mweagle/SpartaPython/cgo -e GOPATH=/usr/src/gopath -e GOOS=linux -e GOARCH=amd64 golang:1.8.1 go build -o SpartaHelloPythonCGOUSEast.lambda.so -tags lambdabinary linux  -buildmode=c-shared -tags lambdabinary ] Name=SpartaHelloPythonCGOUSEast.lambda.so

Questions

How does Sparta determine the Docker image for the CGO build?

Sparta parses the output from your host machine to determine the container tag.

$ go version
go version go1.8.1 darwin/amd64

How can I pass environment variables to the Docker build

All host environment variables with a SPARTA_ prefix will be passed via the -e flag to the Docker run command.

How can I see what Sparta built?

Add a --noop command line argument to your provision command and examine the artifacts in the /.sparta workspace directory. You can also enable debug logging via --level debug for more verbose runtime logging.

How much of a performance increase should I expect?

Significant, particuarly at cold start times. In very limited testing I’ve seen times drop from 1500-2000ms to ~500ms. See also https://twitter.com/mweagle/status/854178789814882304 which was a comparison through an API-GW exposed function exercised by a load testing service.

What transformations are applied to my source?

See cgo_main_run.go for the set of changes applied to the input. If there is an error compiling the source, Sparta leaves the input file (with a sparta-cgo suffix) in the working directory for debugging.