I am using Tanka to template and deploy namespaces and network policies to Kubernetes. To do this in GitLab CI I am using the official Tanka image on Docker Hub, my .gitlab-ci.yml for it looked like this:

.tanka:
  image: grafana/tanka:0.13.0
  before_script:
    - cd tanka
    - tk env set --server-from-context='centralserver-k3s' environments/default/

Tanka Diff:
  extends: .tanka
  stage: tanka-diff
  script:
    - tk diff environments/default || if [ $? -eq 16 ]; then echo 0; else echo $?; fi
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" || ($CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH)
      changes:
        - tanka/**/*
    - if: $CI_PIPELINE_SOURCE == "web"

Tanka Apply:
  extends: .tanka
  stage: tanka
  script:
    - tk apply --dangerous-auto-approve environments/default
  rules:
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - tanka/**/*
      when: manual
    - if: $CI_PIPELINE_SOURCE == "web"

Until recently it worked fine like this, but suddenly it stopped working and started to produce the following error:

Executing "step_script" stage of the job script
Error: unknown subcommand `sh`
Usage:
  tk [command]
Available Commands:
[...]
Cleaning up file based variables
ERROR: Job failed: exit code 1

So, this is rather weird… it sounds like GitLab CI tries to run something like tk sh ..., why would it do that? There is no tk sh command in my .gitlab-ci.yml, it doesn’t seem to make any sense. After searching a bit I found this helpful explanation: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/65110#note_198073241. Looking at the Dockerfile of Tanka confirms that it calls tk in the entrypoint. As explained in the linked issue the combination of the entrypoint and the way the Docker executor starts the container leads to a start command like tk sh -c [rest of shell detection...]. With this knowledge it is even reproducible locally:

$ docker run --rm -it grafana/tanka:0.13.0 sh -c 'echo Hello World'
Error: unknown subcommand `sh`

Usage:
  tk [command]

Available Commands:
[...]

Overriding the entrypoint with an empty string, as described in the issue, fixes the problem:

$ docker run --rm -it --entrypoint "" grafana/tanka:0.13.0 sh -c 'echo Hello World'
Hello World

The same fix works in the .gitlab-ci.yml file:

.tanka:
  image:
    name: grafana/tanka:0.13.0
    entrypoint: [""]
  before_script:
    - cd tanka
    - tk env set --server-from-context='centralserver-k3s' environments/default/

This leaves me with one question: Why did it work before? I think when it worked the job was running with the Kubernetes executor and then it broke when I switched it to the Docker executor. So, my guess is that the Kubernetes executor already overrides the entrypoint by default.

Even though the initial error message was rather confusing the technical explanation makes a lot sense and the fix works fine. Also it is worth mentioning that this is not really a Tanka issue! This will happen with every Docker image with an ENTRYPOINT in the Dockerfile, the fix should also work for all of them.