Documentation
About Kubeapps
Tutorials
- Get Started with Kubeapps
- Using an OIDC provider
- Managing Carvel packages
- Managing Flux packages
- Kubeapps on TKG
- Kubeapps on TCE
How-to guides
- Using the dashboard
- Access Control
- Basic Form Support
- Custon App View Support
- Custom Form Component Support
- Multi-cluster Support
- Offline installation
- Private Package Repository
- Syncing Package Repositories
- Using an OIDC provider with Pinniped
Background
Reference
About the project
Understanding the CircleCI configuration ¶
Kubeapps leverages CircleCI for running the tests (both unit and integration tests), pushing the images and syncing the chart with the official Bitnami chart . The following image depicts how a successful workflow looks like after pushing a commit to the main branch.
The main configuration is located at this CircleCI config file . At a glance, it contains:
- Build conditions:
build_always
,build_on_main
,build_on_tag
andbuild_on_tag_or_prerelease
. They will be added to each job to determine whether or not it should be run. Whereas some should always be run, others only make sense when pushing to the main branch or when a new tag has been created. - Workflows: we only use a single workflow named
kubeapps
with multiple jobs. - Jobs: the actual commands that are run depending on the build conditions.
test_go
(always): it runs every unit test for those projects written in Golang (that is, it runsmake test
) as well as it runs some DB-dependent tests.test_dashboard
(always): it runs the dashboard linter and unit tests (yarn lint
andyarn test
)test_pinniped_proxy
(always): it runs the Rust unit tests of the pinniped-proxy project (cargo test
).build_go_images
(always): it builds the CI golang images forkubeops
,apprepository-controller
,asset-syncer
andkubeapps-apis
.build_dashboard
(always): it builds the CI node image fordashboard
.build_pinniped_proxy
(always): it builds the CI rust image forpinniped-proxy
.local_e2e_tests
(always): it runs locally (i.e., inside the CircleCI environment) the e2e tests. Please refer to the e2e tests documentation for further information. In this job, before running the script `script/e2e-test.sh , the proper environment is created. Namely:- Install the required binaries (kind, kubectl, mkcert, helm).
- Spin up two Kind clusters.
- Load the CI images into the cluster.
- Run the integration tests.
sync_chart_from_bitnami
(on main): each time a new commit is pushed to the main branch, it brings the current changes in the upstream bitnami/charts repository and merges the changes. This step involves:- Checking if the Bitnami chart version is greater than the Kubeapps development chart version. If not, stop.
- Deleting the local
chart/kubeapps
folder (note that the changes are already committed in git). - Cloning the fork kubeapps-bot/charts repository , pulling the latest upstream changes and pushing them back to the fork.
- Retrieving the latest version of the chart provided by Bitnami.
- Renaming the production images (
bitnami/kubeapps-xxx
) by the development ones (kubeapps/xxx
) with thelatest
tag. - Using
DEVEL
as theappVersion
. - Sending a draft PR in the Kubeapps repository with these changes (from a pushed branch in the Kubeapps repository).
push_images
(on main): the CI images (which have already been built) get re-tagged and pushed to thekubeapps
account.GKE_STABLE_VERSION_MAIN
andGKE_STABLE_VERSION_LATEST_RELEASE
(on tag or prerelease): there is a job for each Kubernetes version (stable and regular) supported by Google Kubernetes Engine (GKE). It will run the e2e tests in a GKE cluster (version X.XX) using either the code inprerelease
or in the latest released version. If a change affecting the UI is pushed to the main branch, the e2e test might fail here. Use a try/catch block to temporarily work around this.GKE_REGULAR_VERSION_MAIN
andGKE_REGULAR_VERSION_LATEST_RELEASE
(on tag or prerelease): the same as above, but using the Kubernetes regular version in GKE.sync_chart_to_bitnami
(on tag): when releasing, it will synchronize our development chart with the bitnami/charts repository and merge the changes. This step involves:- Checking if the Kubeapps development chart version is greater than the Bitnami chart version. If not, stop.
- Deleting the local
bitnami/kubeapps
folder (note that the changes are already committed in git). - Cloning the fork kubeapps-bot/charts repository , pulling the latest upstream changes and pushing them back to the fork.
- Retrieving the latest version of the chart provided by Kubeapps.
- Renaming the development images (
kubeapps/xxx
) by the production ones (bitnami/kubeapps-xxx
) with thevX.X.X
tag. - Using
vX.X.X
as theappVersion
. - Sending a draft PR to the Bitnami Charts repository with these changes (from the robot account’s personal fork)
release
(on tag): it creates a GitHub release based on the current tag by running the script script/create_release.sh .
Note that this process is independent of the release of the official Bitnami images and chart. These Bitnami images will be created according to their internal process (so the Golang, Node or Rust versions we define here are not used by them. Manual coordination is expected here if a major version bump happens to occur).
Also, note it is the Kubeapps team that is responsible for sending a PR to the
chart repository
each time a new chart version is to be released. Even this process is automatic (using the sync_chart_to_bitnami
workflow), Kubeapps maintainers must manually review the draft PR and convert it into a normal one once it is ready for review.
Credentials ¶
Besides other usual credentials or secrets passed through environment variables via the CircleCI user interface, it is important to highlight how we grant commit and PR access to our robot account kubeapps-bot <tanzu-kubeapps-team@vmware.com>
. The process is threefold:
- Create a
personal access token
with the robot account, granted, at least, with:
repo:status
,public_repo
andread:org
. This token must be stored as the environment variableGITHUB_TOKEN
(is where Github CLI will look for)- That will allow the GitHub CLI to create PRs from the command line on behalf of our robot account.
- Also, this token will be used for performing authenticated GitHub API calls.
- Add deployment keys to the repositories to which the CI will commit. Currently, they are
vmware-tanzu/kubeapps
andkubeapps-bot/charts
.- This step allows the robot account to push branches remotely. However, the CI will never push to the main branch as it always tries to create a pull request.
- Add the robot account GPG key pair in the
GPG_KEY_PUBLIC
andGPG_KEY_PRIVATE
environment variables.- The public key must be also uploaded in the robot account GPG settings in GitHub. It will be used for signing the commits and tags created by this account.
Generating and configuring the deployment keys ¶
This step is only run once, and it is very unlikely to change. However, it is important to know it in case of secret rotations or further events.
# COPY THIS CONTENT TO GITHUB (with write access):
## https://github.com/vmware-tanzu/kubeapps/settings/keys
ssh-keygen -t ed25519 -C "tanzu-kubeapps-team@vmware.com" -q -N "" -f circleci-kubeapps-deploymentkey
echo "Kubeapps deployment key (public)"
cat circleci-kubeapps-deploymentkey.pub
# COPY THIS CONTENT TO GITHUB (with write access):
## https://github.com/kubeapps-bot/charts/settings/keys
ssh-keygen -t ed25519 -C "tanzu-kubeapps-team@vmware.com" -q -N "" -f circleci-charts-deploymentkey
echo "Charts deployment key (public)"
cat circleci-charts-deploymentkey.pub
# COPY THIS CONTENT TO CIRCLECI (hostname: github.com):
## https://app.circleci.com/settings/project/github/vmware-tanzu/kubeapps/ssh
echo "Kubeapps deployment key (private)"
cat circleci-kubeapps-deploymentkey
echo "Charts deployment key (private)"
cat circleci-charts-deploymentkey
# COPY THE FINGERPRINTS TO ".circleci/config.yml"
## sync_chart_from_bitnami
echo "Charts deployment key (fingerprint) - edit 'sync_chart_from_bitnami'"
ssh-keygen -l -E md5 -f circleci-kubeapps-deploymentkey.pub
## sync_chart_to_bitnami
echo "Charts deployment key (fingerprint) - edit 'sync_chart_to_bitnami'"
ssh-keygen -l -E md5 -f circleci-charts-deploymentkey.pub
Debugging the CI errors ¶
As per the official CircleCI documentation , one of the best ways to troubleshoot problems is to SSH into a job and inspect things like log files, running processes, and directory paths. For doing so, you have to:
Ensure that you have added an SSH key to your GitHub account.
Start a job with SSH enabled, that is, select the Rerun job with SSH option from the Rerun Workflow dropdown menu.
- To see the connection details, expand the Enable SSH section in the job output where you will see the SSH command needed to connect.
The build will remain available for an SSH connection for 10 minutes after the build finishes running and then automatically shuts down. After you SSH into the build, the connection will remain open for two hours.