22. Secret Management For User Workloads
Date started: 2023-03-25 Date revised: 2023-08-29
# Status
Accepted
Relates to:
# Context
- When user workloads are deployed to environments, the system should be able to provide a way to inject values that are specific to the environment. Currently, this is done through environment variables that are managed as overlays on the GitOps repository for the application. However, this method does not provide a good way to manage
Secret
. This ADR addresses the secret management of user workloads for different environments. - As a StoneSoup component, I expect to have the ability to securely upload a secret, associate it with the component, and propagate it to the target environment.
- It is important to distinguish the process of uploading confidential information and deploying it in the destination environment. This should/can be performed by different roles: first, creating a searchable
RemoteSecret
with linkedSecretData
that is stored remotely onSecretStorage
, and second, finding thisRemoteSecret
and creating a working Secret withSecretData
linked to it. - It should be possible to set multiple destination targets.
- Any consumer should be able to perform a search of existing
RemoteSecret
by target environment/component/application.
# Decision
# Architecture Overview
# Terminology
- Upload Secret: A short-lived Kubernetes
Secret
used to deliver confidential data to permanent storage and link it to theRemoteSecret
CR. The Upload Secret is not a CRD. SecretData
: An object stored in permanent SecretStorage. Valid SecretData is always linked to a RemoteSecret CR.RemoteSecret
: A CRD that appears during upload and linksSecretData
+DeploymentTarget(s)
+ K8sSecret
.RemoteSecret
is linked to one (or zero) SecretData and manages its deleting/updating.- K8s
Secret
: What appears at the output and is used by consumers. SecretId
: A unique identifier of SecretData in permanent SecretStorage.SecretStorage
: A database eligible for storingSecretData
(such as HashiCorp Vault, AWS Secret Manager). That is an internal mechanism. Only spi-operator will be able to access it directly.
# Architecture
TODO update diagram
The proposed solution is to create a new Kubernetes Custom Resource (CR) called RemoteSecret
. It serves as a representation of the Kubernetes Secret that is stored in permanent storage, which is also referred to as SecretStorage
. This Custom Resource includes references to targets, like Kubernetes namespaces, that may also contain the required data to connect to a remote Kubernetes. To perform an upload to permanent storage, a temporary Upload Secret is utilized, which is represented as a regular Kubernetes Secret with special labels and annotations that the SPI controller recognizes. Different SecretStorage
implementations, like AWS Secret Manager or HashiCorp Vault, can be used. It is simpler to create the RemoteSecret first and then use the linked Upload Secret to upload secret data. However, in simple cases, the Upload Secret can be used to perform both uploading and the creation of RemoteSecret in a single action. It’s worth noting that the Upload Secret is not a core component of the framework, but rather a convenient way of creating secrets. In the future, it is possible that new methods of uploading SecretData to RemoteSecret may be added.
The design specifically allows for separating the upload of the secret data (using the UploadSecret
) and delivering the secret to the target. The list of targets of the RemoteSecret
can be initially empty and can be updated as time evolves.
Apart from the remote secret supporting the delivery of the secret, it is also optionally able to link this secret to the service accounts in the target. These service accounts are either managed by the remote secret (i.e. share its lifecycle) or are required to pre-exist in the target cluster and namespace. This is similar to the features of the SPIAccessTokenBinding
which also delivers secrets albeit sourced from a different workflow.
The remote secret is a template of a single secret that can be delivered to multiple targets. Because the target is identified by the namespace (and the URL of the cluster, if provided), there can be at most one secret delivered to a certain namespace by a single remote secret.
The linking of a remote secret to an application, environment and/or component, is done using labels. The exact behavior of where to deploy when the remote secret is labeled in a certain way is still TBD but the current thinking is lack of component label means that the secret should be available to all components and lack of environment means that the remote secret should be deployed to all environments. There is a specialized controller that watches SnapshotEnvironmentBinding
s and updates the targets of remote secrets matching the application/environment/components that correspond to the SEB with appropriate targets.
# Example: Delivering the secrets interatively
At first, the targets to which the secret should be deployed might not yet be known. Nevertheless, the remote secret can be created at this point in time. The creator just doesn’t declare any targets.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: db-credentials
targets: []
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: AwaitingData
status: "False"
type: DataObtained
targets: []
After creating the remote secret, the secret data may be associated with it, still without any targets. The user creates an UploadSecret
that associates the data with the remote secret.
apiVersion: v1
kind: Secret
metadata:
name: upload-secret-data-for-remote-secret
namespace: default
labels:
appstudio.redhat.com/upload-secret: remotesecret
annotations:
appstudio.redhat.com/remotesecret-name: test-remote-secret
type: Opaque
stringData:
username: u
password: passw0rd
After this step, the data is associated with the RemoteSecret
which is reflected in its status.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: db-credentials
targets: []
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: DataFound
status: "True"
type: DataObtained
targets: []
The targets of the RemoteSecret
can be defined at any time. If the secret data is not yet associated with the RemoteSecret
, nothing is delivered to the targets. If there is secret data associated with the secret, it is immediatelly delivered to the targets, if any are defined.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: db-credentials
targets:
- namespace: ns
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: DataFound
status: "True"
type: DataObtained
targets:
- namespace: ns
secretName: db-credentials
# Example: Define the structure of the secrets in the targets
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
targets: []
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: AwaitingData
status: "False"
type: DataObtained
targets: []
This example illustrates that we can prescribe the name
of the secret in the targets. If not specified, as in this case, the type of the secret defaults to Opaque
.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
generateName: secret-from-remote-
targets:
- namespace: ns
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: DataFound
status: "True"
type: DataObtained
targets:
- namespace: ns
secretName: secret-from-remote-sdkfl
Here, we merely illustrate that the secret might have a dynamic name when using the generateName
property. To learn the actual name of the secret when created in the target, the user can inspect the status of the remote secret.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
type: kubernetes.io/basic-auth
targets: []
status:
...
It is also possible to declare the required annotations and labels that the secret should have in the targets:
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
type: kubernetes.io/basic-auth
labels:
key: value
annotations:
key: value
targets: []
status:
...
# Example: Associating the secret with a service account in the targets
The spec of the RemoteSecret
can specify that the secret should be linked to a service account in the targets. This is identical to the feature present in the SPIAccessTokenBinding
.
The secret may be linked to a service account that must be already present in the target namespace. When deleting the RemoteSecret
, such service account is kept in place and only the link to the secret that is being deleted is removed from it.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
type: kubernetes.io/basic-auth
linkedTo:
- serviceAccount:
reference:
name: app-sa
targets: []
status:
...
It is also possible to create a managed service account. Such service account shares the lifecycle of the RemoteSecret
.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
type: kubernetes.io/basic-auth
linkedTo:
- serviceAccount:
managed:
name: app-sa
targets: []
status:
...
It is possible to link the secret to the service account either as an ordinary secret but also as an image pull secret.
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
name: secret-from-remote
type: kubernetes.io/basic-auth
linkedTo:
- serviceAccount:
as: imagePullSecret
managed:
name: app-sa
targets: []
status:
...
# Example: Inspecting the state of the deployment to targets
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret
namespace: default
spec:
secret:
generateName: secret-from-remote-
linkedTo:
- serviceAccount:
managed:
generateName: sa-from-remote-
targets:
- namespace: "test-target-namespace-1"
- namespace: "test-target-namespace-2"
- namespace: "test-target-namespace-3"
- namespace: "test-target-namespace-rainbow"
apiUrl: "over-the-rainbow"
clusterCredentialsSecret: "team-a--prod-dtc--secret"
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: DataFound
status: "True"
type: DataObtained
- lastTransitionTime: "..."
message: "some of the targets were not deployed to"
reason: PartiallyInjected
status: "False"
type: Deployed
targets:
- namespace: "test-target-namespace-1"
secretName: secret-from-remote-lsdjf
serviceAccountNames:
- sa-from-remote-llrkt
- namespace: "test-target-namespace-2"
secretName: secret-from-remote-lemvs
serviceAccountNames:
- sa-from-remote-lkejr
- namespace: "test-target-namespace-3"
secretName: secret-from-remote-kjfdl
serviceAccountNames:
- sa-from-remote-lmval
- namespace: "test-target-namepace-rainbow"
apiUrl: "over-the-rainbow"
error: "Connection refused"
There are 2 conditions in the status expressing the state of data readiness (
DataObtained
condition type withAwaitingData
andDataFound
as possible reasons) and the overall deployment status (Deployed
condition type with the condition either missing altogether if there are no targets orPartiallyInjected
orInjected
reasons). Additionally, the status contains the details of the deployment of each of the targets in the spec. The entries might not come in the same order as in the spec but correspond to each entry in the spec by thenamespace
+apiUrl
compound key (we don’t support 2 targets of a single remote secret pointing to the same namespace atm). The status of the target contains the actual names of the secret and the (optional) service accounts (this is important in case of usinggenerateName
for the secret or the service account(s)) and optionally also anerror
that explains why certain target was not deployed to.
# Example: If RemoteSecret has to be created with target namespace and Environment
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret-secret
labels:
appstudio.redhat.com/environment: prod
appstudio.redhat.com/component: m-service
appstudio.redhat.com/application: coffee-shop
spec:
secret:
name: test-remote-secret-secret
target:
- namespace: jdoe-tenant
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: AwaitingData
status: "False"
type: DataObtained
# Example: If RemoteSecret has to be created all Environments of certain component and application
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret-secret
labels:
appstudio.redhat.com/component: m-service
appstudio.redhat.com/application: coffee-shop
spec:
secret:
name: test-remote-secret-secret
target:
- namespace: jdoe-tenant
status:
conditions:
- lastTransitionTime: "..."
message: ""
reason: AwaitingData
status: "False"
type: DataObtained
# Internal RemoteSecrets
The RemoteSecret Custom Resource (CR) is a crucial implementation detail of “Secret Management For User Workloads”. It is also utilized in other features within RHTAP, such as “pull” secrets, among others.
To differentiate RemoteSecrets that are provided by end-users from RemoteSecrets that are generated by RHTAP’s components, a label appstudio.redhat.com/internal: true
is introduced. The presence of this label signifies that the RemoteSecret is intended for internal usage, and any unauthorized changes to such secrets may result in unpredictable outcomes.
This label helps to identify secrets that are meant for internal purposes, preventing accidental or unauthorized modifications that could disrupt the functioning of RHTAP’s components.
Example:
apiVersion: appstudio.redhat.com/v1beta1
kind: RemoteSecret
metadata:
name: test-remote-secret-secret
labels:
appstudio.redhat.com/component: m-service
appstudio.redhat.com/application: coffee-shop
appstudio.redhat.com/internal: true
spec:
...
It’s important to note that this label is not intended to alter the behavior of the controller or permissions. The primary effect is on the UI behavior, which is designed to hide these labeled RemoteSecrets from the list.
# Security and Access Control
- Access to the secret management backend should be restricted to authorized users and components. See the table for the access levels for different roles.
- The default backend should use authentication and authorization mechanisms provided by the underlying service, such as Kubernetes or AWS IAM.
- Workspace maintainers should have the ability to create and update secrets for their workspace on the default secret management backend.
- Access to the secret management backend should be audited to detect and prevent unauthorized access or misuse.
# Additions to Roles and Permissions
Role | Permissions | API Groups | Verbs | Resources |
---|---|---|---|---|
Contributor | RemoteSecret | appstudio.redhat.com | get, list, watch | remotesecret |
Maintainer | RemoteSecret | appstudio.redhat.com | get, list, watch | remotesecret |
Admin | RemoteSecret | appstudio.redhat.com | * | remotesecret |
Only roles with create
privileges to Secret
s can create and update secrets. At the time if this ADR this is limited to Admin role only.
Note : Initially, we decrease Maintainer’s capabilities to the Contributor level. Later we might introduce instruments, so they can modify RemoteSecret.
# Monitoring and Auditing
- The secret management system should be monitored and audited to detect and prevent unauthorized access or misuse.
- Access and usage of secrets should be monitored and tracked to identify potential security issues or incidents.
- An audit log of all secret access and usage should be maintained to provide a record of who accessed and used secrets, when, and for what purpose.
- The secret management system should also provide alerting and notification mechanisms to alert security administrators and other stakeholders of potential security issues or incidents. This may include alerts for unauthorized access or misuse of secrets, as well as alerts for other security-related events or issues.
# Consequences
- The new secret management system will provide a more secure and scalable way for users to manage secrets for their workloads. It will also require some additional work to integrate with existing systems and processes. A new UI will need to be created to allow users to create and update secret values for their components and provide environment-specific overrides to them.
- Further work to be addressed, mechanisms for determining the exact target namespace in case if Environment name is used as a destination for K8s Secret, and determining the trigger when the K8s Secret has to be delivered. The use of ArgoCD’s Resource ArgoCD’s Resource Hooks may be helpful in this regard.