Background
eBay.com is powered by applications that are built with Jenkins as the opinionated choice of build-machinery. We run Jenkins instances as a cloud native workload on the Kubernetes Clusters with server-agent mode, all of which are managed by the eBay cloud team.
The Jenkins servers provided to eBay development teams have a default credential plugin installed. The credentials stored on these Jenkins instances are used for various purposes, such as connecting to GitHub and third-party APIs. The credential plugin stores the user credentials, as well as the encryption key, on the disk in an encrypted format.
These credentials are very sensitive in nature and need to be stored safely and securely. However, storing the key and the credential on the same disk poses a risk. Jenkins instances are also generally shared across a team with ADMINISTER privileges, which increases the chances of team members knowing each other’s credentials.
Initially, we considered setting up a Vault1 store, which is generally used in the community to secure the credentials in Jenkins. However, that would be a vendor lock-in. To remain vendor-neutral, we chose a different strategy.
However, eBay has its own proprietary key management platform, which integrates with Kubernetes cluster’s control-plane and mounts secrets in a secure manner to the containers leveraging Kubernetes API standards. Since we are running Jenkins as workloads on Kubernetes, a simple solve was to standardize by using Kubernetes Secrets. Another benefit of this approach is that the same secrets can also be backed up to the in-house key management solution.
The challenge here was to integrate the Jenkins application with this proprietary key management platform. To solve the challenge, we developed a new plugin - Jenkins Credentials Secrets Plugin - which replaces the default credential plugin and is now available as an eBay open source project.
This article explains the design and process of a more secure way of storing Jenkins credentials as secrets on Kubernetes Clusters.
Scale of Operation
We have enabled this plugin to store credentials as Kubernetes Secrets on ~6000 Jenkins instances at eBay. We have also migrated all the credentials currently stored in these instances to Kubernetes secrets with ZERO down time for the end users - eBay application developers.
Assumptions
1. The Jenkins master runs as a container in Kubernetes Cluster.
2. The Jenkins master has a Kubernetes service account mounted to it.
Limitations
Only credentials under the Jenkins Global domain are supported by the plugin at this moment.
Design
When a credential is created by the user from a Jenkins UI or API call, a Kubernetes Secret is generated in a namespace (as specified by the plugin user) in the cluster with the required information from the credentials. The Secret specification’s “data” will hold the credential’s “sensitive” information. The Secret will have labels and annotations in the spec to store the below details.
The secret name is generated as “sec” + “-” + “UUID” + “-” + “JENKINS_INSTANCE_NAME”
Example: sec-<<UUID>>-myCI.
We use UUID because the Kubernetes Secret object’s naming convention does not support all the characters allowed by Jenkins credentials (e.g. whitespaces and underscores). The credentials ID stored by the user is captured in the Secret’s labels, as shown in the above chart. We also capture other credential information, like description and type, and this list can be extended to attributes added in the future as well.
Secrets-To-Credentials Converter
The core component of this approach is the “Secrets-to-credentials converter” module. On Jenkins start up, the plugin gets the list of secrets that have the Jenkins name label selector. This is necessary so only the secrets belonging to the Jenkins in question are pulled. Converters for all of the below Jenkins supported credentials are implemented.
- Username with password
- Docker Host Certificate authentication
- Kubernetes Service Account
- OpenShift OAuth token
- OpenShift Username and Password
- SSH Username with Private Key
- SSH Username with Private Key and Passphrase
- Secret file
- Secret text
- Certificate
The credential type of a secret is identified from the label (jenkins.io/credentials-type), and the corresponding converter implementation is invoked to convert a secret to a credential. This credential is added to the Jenkins credentials Map offered by the Jenkins credentials plugin. This makes the approach transparent to the users, as they just see the same credential information on Jenkins UI, and all the secret conversion happens in the background.
Lifecycle Management of Secrets in Jenkins
➔ Create: When users add credentials in the Jenkins UI, secrets will be created on the corresponding Kubernetes cluster in the namespace provided, and credentials will no longer be stored in `credentials.xml` on the disk.
➔ Read: On Jenkins startup, credentials of the particular CI are loaded from the Kubernetes cluster (using label selector “jenkins.io/ci-name”).
➔ Update: When a credential is updated, the corresponding secret is updated (using label selectors “jenkins.io/ci-name” and “jenkins.io/credential-id”).
➔ Delete: When a credential is deleted, the corresponding secret is deleted (using label selectors “jenkins.io/ci-name” and “jenkins.io/credential-id”).
Secrets YAML
Username Password credential Type:
The rest of the credential type YAML specs are available in the source code repository here:
https://github.com/eBay/kube-credentials-plugin/tree/master/credentialspecsamples
- Base Credentials plugin - 2.1.19
- Kubernetes plugin (dependency) - 1.1.4
We have open sourced this project, and the git repo can be found here:
https://github.com/eBay/kube-credentials-plugin
We welcome any PullRequests (PR) or github-issues on this repo. We have listed to-do items on the repo and welcome PRs to address them.
1 Vault is a product from HashiCorp: https://www.vaultproject.io/