v2.24.0 Armory Continuous Deployment Release (Spinnaker™ v1.24.2)

Release notes for v2.24.0 Armory Continuous Deployment

2021/01/21 Release Notes

Note: If you’re experiencing production issues after upgrading Spinnaker, rollback to a previous working version and please report issues to http://go.armory.io/support.

For information about what Armory supports for this version, see the Armory Continuous Deployment v2.24 compatibility matrix.

Required Halyard or Operator version

To install, upgrade, or configure Armory 2.24.0, use one of the following tools:

  • Armory-extended Halyard 1.10 or later
  • Armory Operator 1.2.1 or later

Security

Armory scans the codebase as we develop and release software. Contact your Armory account representative for information about CVE scans for this release.

Breaking changes

Breaking changes are kept in this list for 3 minor versions from when the change is introduced. For example, a breaking change introduced in 2.21.0 appears in the list up to and including the 2.24.x releases. It would not appear on 2.25.x release notes.

Suffix no longer added to jobs created by Kubernetes Run Job stage

Spinnaker no longer automatically appends a unique suffix to the name of jobs created by the Kubernetes Run Job stage. Prior to this release, if you specified metadata.name: my-job, Spinnaker updates the name to my-job-[random-string] before deploying the job to Kubernetes. As of this release, the job’s name will be passed through to Kubernetes exactly as supplied.

To continue having a random suffix added to the job name, set the metadata.generateName field instead of metadata.name, which causes the Kubernetes API to append a random suffix to the name.

This change is particularly important for users who are using the preconfigured job stage for Kubernetes or are sharing job stages among different pipelines. In these cases, jobs often running concurrently, and it is important that each job have a unique name. In order to retain the previous behavior, manually update your Kubernetes job manifests to use the generateName field.

Previously, this behavior was opt-in.

Impact

As of Armory 2.22, this behavior is the default. Users can still opt out of the new behavior by setting kubernetes.jobs.append-suffix: true in clouddriver-local.yml. This causes Spinnaker to continue to append a suffix to the name of jobs as in prior releases.

The ability to opt out of the new behavior will be removed in Armory 2.23 (OSS 1.23). The above setting will have no effect, and Spinnaker will no longer append a suffix to job names. We recommended that 2.22 users note which jobs are using the old behavior and prepare to remove the setting before upgrading to Armory 2.23 in the future.

Introduced in: Armory 2.22

Zombie Executions

Starting in Spinnaker 2.23.0, ManifestForceCacheRefreshTask was removed, as Kubernetes manifest related stages now do live lookups. While upgrading to Spinnaker 2.23.0 or later, if there is a running pipeline that contains a Kubernetes manifest related stage, it becomes a zombie execution. This causes Orca, Spinnaker’s orchestration service, to fail to complete any Kubernetes manifest related stage in that pipeline.

Workarounds:

To resolve the issue, cancel any zombie executions. For information about how to cancel them, see the Orca Zombie Execution runbook.

Affected versions: 2.23.0 and later

ManifestForceCacheRefreshTask removed from Orca

When you upgrade to 2.23.0 or later, you might encounter the following error:

2021-01-29 23:57:19.691 ERROR 1 --- [    scheduler-2] c.netflix.spinnaker.q.redis.RedisQueue   : Failed to read message 8f072714f1df6dbf3af93a4f4fe4cae2, requeuing...
com.fasterxml.jackson.databind.JsonMappingException: No task found for 'com.netflix.spinnaker.orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask' (through reference chain: com.netflix.spinnaker.orca.q.RunTask["taskType"])

The ManifestForceCacheRefreshTask task is no longer a required task when deploying a manifest. In earlier releases, forcing the cache to refresh was part of the deployment process for manifests. Because of this change, if a task was running or retried before the upgrade, the error shows up in logs as an exception.

Workaround

Before starting, make sure that you have access to the Redis instance that Orca uses.

To resolve this issue, delete the message from the queue:

  1. Verify that there are pipeline execution failure messages that contain ManifestForceCacheRefreshTask:

    Redis

    hgetall orca.task.queue.messages

    The command returns information similar to the following:

    1) "93ac65e03399a4cfd3678e1355936ab2"
2) "{\"kind\":\"runTask\",\"executionType\":\"PIPELINE\",\"executionId\":\"01EVFCCDG3Q2209E0Z1QTNC0FS\",   \"application\":\"armoryhellodeploy\",\"stageId\":\"01EVFCCDG3TJ7AFPYEJT1N8RDJ\",\"taskId\":\"5\",\"taskType\":\"com.netflix.spinnaker.   orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask\",\"attributes\":[{\"kind\":\"attempts\",\"attempts\":1}],   \"ackTimeoutMs\":600000}"

  2. Delete the message(s):

    Redis

    hdel orca.task.queue.messages

    The command returns information similar to the following:

    93ac65e03399a4cfd3678e1355936ab2
(integer) 1

Known issues

Bake failures

The Packer version included with Rosco disregards package overrides that use the -var-file= option. This may cause bakes to fail.

Affected versions: 2.22.2 and later

Lambda UI issue

There is a UI bug related to the caching agent that prevents Lambda functions from being displayed in the UI when there are no other clusters associated with the Application. In other words, in order for the function to show up in “Functions” tab, there needs to be a cluster (such as an AWS ASG/EC2 instance) deployed for that application.

Affected versions: 2.23.0 (1.23.0) - 2.26.2 Fixed version: 2.26.3

Pipelines-as-Code fails unexpectedly when updating modules

The container for the Dinghy service that Pipelines-as-Code uses fails when updating pipelines using modules stored in GitHub. The error you encounter references a failure related to GitHub, such as one of the following:

422 Validation Failed [{Resource:CommitComment Field:body Code:custom Message:body is too long (maximum is 65536 characters)}]

or

422 No commit found for SHA: <SHA for a commit> []

This results in only some pipelines in your deployment getting updated when a module gets updated.

Workaround:

  1. Use the arm CLI to render the JSON for your dinghyfiles.
  2. Update pipelines manually using the UI.

Affected versions: 2.22.x, 2.23.x, 2.24.0 Fixed versions: 2.25.0

Fixed issues

  • Fixed an issue where Clouddriver fails to cache the images for the account that comes first (alphabetically) in each region.

Highlighted updates

Baking

  • You can now use git/repo artifacts when baking a helm chart.
  • Rosco now supports using a secure connection to Redis

Deployment targets

AWS ECS

Improvements to ECS include the following:

  • Tag-based Moniker naming strategies are now supported for ECS. For more information, see Moniker.
  • Better ECS performance.

Clouddriver

You can now use PostgresSQL 10 or later as the backing store for Clouddriver.

Cloud Foundry

Improvements to the Cloud Foundry provider include the following:

  • Droplets do not have to be restaged after a service gets bound anymore.
  • The endpoints now use /v3 to provide more granular control over READ/WRITES.
  • You can now filter based on location and space. This gives you the ability to scope your Spinnaker to a specific organization or space, limiting the resources, operations, and caching to a specific organization or space.

Pipelines-as-Code

Pipelines-as-Code now supports using MySQL as the backing store, which can provide more durability and scalability than Redis. This feature is currently in early access.

For information about how to configure the backing store for Pipelines-as-Code, see Configuring SQL.

Terraform Integration

The Terraform Integration now supports the following Terraform versions: 0.13.4, 0.13.5, 0.14.0, 0.14.1, 0.14.2

Additionally, the integration is now more resilient. It will retry if it fails to fetch artifacts from Clouddriver. Configure this feature in your terraformer-local.yml file.

The following example enables retries and sets the minimum wait between attempts to 4 seconds, the maximum wait between attempts to 8s, and the maximum number of retries to 5.

# terraformer-local.yml
clouddriver:
  retry:
    enabled: true
    minWait: 4s # must be a duration, such as 4s for 4 seconds
  maxWait: 8s # must be a duration, such as 8s for 8 seconds
  maxRetries: 5

UI

Nested pipelines now have a breadcrumb trail that shows parent executions, making it easier to see and navigate the context.

Breadcrumbs for nested pipeline execution context

Spinnaker Community Contributions

There have also been numerous enhancements, fixes, and features across all of Spinnaker’s other services. See the Spinnaker v1.24.2 changelog for details.

Detailed updates

Bill Of Materials (BOM)

Here’s the BOM for this version.

Expand
version: 2.24.0
timestamp: "2021-01-19 18:09:21"
services:
    clouddriver:
        commit: 4c5d79fc
        version: 2.24.8
    deck:
        commit: d0859de2
        version: 2.24.1
    dinghy:
        commit: 16f3f547
        version: 2.24.5
    echo:
        commit: 3335cceb
        version: 2.24.3
    fiat:
        commit: e21ee877
        version: 2.24.3
    front50:
        commit: 9d1b8844
        version: 2.24.3
    gate:
        commit: 599b2365
        version: 2.24.3
    igor:
        commit: 213c430c
        version: 2.24.4
    kayenta:
        commit: 0a93d760
        version: 2.24.6
    monitoring-daemon:
        version: 2.24.0
    monitoring-third-party:
        version: 2.24.0
    orca:
        commit: cef289ed
        version: 2.24.4
    rosco:
        commit: cb8dc475
        version: 2.24.3
    terraformer:
        commit: f7d5096e
        version: 2.24.2
dependencies:
    redis:
        version: 2:2.8.4-2
artifactSources:
    dockerRegistry: docker.io/armory

Armory

Armory Echo - 2.23.6…2.24.3

  • fix(GHA): set-env deprecated (#244)

Armory Clouddriver - 2.23.28…2.24.8

  • fix(GHA): set-env deprecated (#224)
  • feat(docker): adding ecr utility to get an ecr token (#235) (#238)
  • fix(dep/springboot): downgrade spring-boot to SR4 (#249) (#250)
  • fix(config): reverted spring.profile sytax to SR4 syntax (#251) (#252)
  • fix(dependencies): downgrade resilience4j to match OSS (#253) (#254)

Armory Fiat - 2.23.5…2.24.3

  • fix(GHA): set-env deprecated (#126)

Armory Gate - 2.23.5…2.24.3

  • feat(dinghy): add api endpoints for dinghy. (#184)
  • fix(GHA): set-env deprecated (#186)
  • feat(dinghy): update dinghy events endpoint. (#187)

Armory Igor - 2.23.5…2.24.4

  • fix(GHA): set-env deprecated (#141)

Armory Front50 - 2.23.6…2.24.3

  • fix(GHA): set-env deprecated (#165)
  • fix(chore): typo for regex autobump (#151)

Armory Deck - 2.23.15…2.24.1

  • feat(kayenta): extending kayenta module to add dynatrace UI (#675)
  • feat(dinghy): add dinghy header. (#679)
  • fix(GHA): set-env deprecated (#682)
  • feat(dinghy): add dinghy component. (#680)
  • fix(dinghy): fix NPE for commits and files. (#684)
  • feat(kayenta): add cloudwatch UI (#686)
  • fix(cloudwatch): change textbox template to json editor (#688)
  • fix(terraform): save the default terraform version (#692)
  • chore(deps): update to 1.23.4 base (#698)
  • chore(deps): bump base to OSS 1.23.5 (#701)
  • chore(deps): bump base to OSS 1.24.0 (#704)
  • chore(deps): bump to new OSS 1.24.2 bom (#713) (#714)

Dinghy™ - 2.23.8…2.24.5

  • fix(nilcommits): fix nil commits for deck ui (#286)
  • fix(nofiles): only show file changes in logevents (#289)
  • feat(mainbranch): main branch support switch when master fails (#292)
  • fix(slacknotifications): slack notifications fix for 2.23 (#298)
  • feat(github): add GitHub notifier. (#297)
  • feat(yaml): update yaml support to be on par with json (#291)
  • feat(sqlsupport): full sql support (#300) (#308)
  • feat(parser): sprig functions support (#311) (#312)
  • fix(liquibase): liquibase will be executed by dinghy since it has the secrets (#314) (#315)
  • fix(validations): pr validations fixes (#317) (#318)

Armory Kayenta - 2.23.8…2.24.6

  • feat(cloudwatch): adding cloudwatch config and integration test (#142)
  • fix(GHA): set-env deprecated (#145)
  • fix(cloudwatch): fixes duplicate config (#148)
  • fix(cve): fix 2020-5408 force security-core to 5.2.4 (#163) (#164)
  • Update gradle.yml

Armory Orca - 2.23.12…2.24.4

  • fix(GHA): set-env deprecated (#170)

Armory Rosco - 2.23.10…2.24.3

  • Fix(fargate-job-executor): dont mutate original config when making orphan token clients (#146)
  • fix(fargate-job-executor): handle duplicate files when recursing config dir, because operator symlinks files more than once (#142)
  • fix(fargate-job-executor): specify cluster when canceling job (#150)
  • feat(fargate): Enable IAM authentication with Vault for Fargate Job Executor (#154)
  • fix(remote-jobs-image): fix tag command (#159)

Terraformer™ - 2.23.6…2.24.2

  • feat(retry): adds retries to clouddriver service (#283)
  • feat(retry): add customizable options (#284)
  • chore(versions): add 0.13.4, 0.13.5, 0.14.0 (#287)
  • chore(logging): improve logging when clouddriver returns 503 (#285)
  • chore(deps): update tf to add 0.14.1 & 0.14.2 (#290)
  • fix(clouddriver): fix short-write of git/repo file (#294)
  • refactor(git/repo): improve git/repo perf (#295)

Last modified May 26, 2023: (a7d5a9eb)