Gitlab CI caching for Go projects
by Emile `iMil' Heitor - 2019-12-07
The reference documentation when it comes to couple golang and continuous integration in Gitlab is this one, it’s well put, easy to read and pretty accurate. Except for the caching
part, or at least nowadays with go modules. This is what happens when a commit is pushed with the .gitlab-ci.yml
given as an example in that document:
131 Creating cache default...
132 WARNING: /apt-cache: no matching files
133 WARNING: /go/src/github.com: no matching files
134 WARNING: /go/src/gitlab.com: no matching files
135 WARNING: /go/src/golang.org: no matching files
136 WARNING: /go/src/google.golang.org: no matching files
137 WARNING: /go/src/gopkg.in: no matching files
And as a matter of fact, the cache is empty for next stage.
Maybe I’m missing something, but the article is clear on these commands:
Then, you specify some folders of this image to be cached. The goal here is to avoid downloading the same content several times. Once a job is completed, the listed paths will be archived, and next job will use the same archive.
Well it does not.
So there’s this, and also another reason why I dug up on the topic; I like to locally test my commits before pushing, and it is possible to use gitlab-runner exec for that, which is great, but what’s not so great is that gitlab-runner exec
doesn’t support the artifact
feature, and thus I can’t bring a build result to a testing stage, which I need in my current project in order to test the program itself.
While the best practices say https://docs.gitlab.com/ee/ci/caching/:
cache: Use for temporary storage for project dependencies. Not useful for keeping intermediate build results, like jar or apk files. Cache was designed to be used to speed up invocations of subsequent runs of a given job, by keeping things like dependencies (e.g., npm packages, Go vendor packages, etc.) so they don’t have to be re-fetched from the public internet. While the cache can be abused to pass intermediate build results between stages, there may be cases where artifacts are a better fit.
artifacts: Use for stage results that will be passed between stages. Artifacts were designed to upload some compiled/generated bits of the build, and they can be fetched by any number of concurrent Runners.[…]
We can’t stick to these when using the exec
command because of its limitations. But we can usecache
!
So, thanks to this blog post which points a smart way of declaring GOPATH
and that article which explains how to actually preserve caching between stages, I came up with this .gitlab-ci.yaml
, which works and caches both locally with gitlab-runner exec
and on Gitlab jobs:
image: golang:1.12
stages:
- build
- test
before_script:
- export GOPATH=${CI_PROJECT_DIR}/.cache
- apt-get update -qq && apt-get -y -qq install jq xxd
build:
stage: build
cache: &build_cache
key: build
paths:
- .cache
- goxplorer
script:
- mkdir -p .cache
- make
unit_tests:
stage: test
cache: *build_cache
script:
- make test
- /bin/sh runtest.sh
dependencies:
- build
code_coverage:
stage: test
script:
- make coverage
Invoke the build phase like this:
$ gitlab-runner exec docker --docker-volumes $(pwd)/cache:/cache build
And finally the test phase:
$ gitlab-runner exec docker --docker-volumes $(pwd)/cache:/cache unit_tests
Happy testing ;)