Automatically refresh caches when build dependencies are updated

Bitbucket Pipelines provides a caching feature that provides the ability to cache external build dependencies and directories e.g. 3rd-party libraries. This means faster builds and fewer consumed build minutes.

However, caches currently live for 7 days and are invalidated at the end of that period. And updates to build dependencies do not cause the cache to be refreshed which means that builds can sometimes use old versions of dependencies, leading to downstream errors. We have this highly voted suggestion where customers are looking for an option to invalidate and refresh the cache automatically whenever there is a change in the build dependency.

In this blog post, we are sharing a solution that can help to invalidate and refresh the cache automatically whenever there is a change in the build dependency.

Here’s how it works:

atlassian/bitbucket-clear-cache Pipe

We use the atlassian/bitbucket-clear-cache pipe to delete the cache in the pipeline. Here’s the yaml for this pipe.

pipe: atlassian/bitbucket-clear-cache:3.1.1
 variables:
 BITBUCKET_USERNAME: $BITBUCKET_USERNAME
 BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
 CACHES: [ "node"]

Changesets condition

Next, the condition changeset is used to execute a step only if there is a change in a specific file. If not, the step will be skipped.

We will add the changeset condition to indicate the file to be verified for the changes before actually deleting the cache. The structure in your YAML file will be something like this:

pipelines:
  default:
    - step:
        name: delete cache if changes in the build dependencies
        script:
          - pipe: atlassian/bitbucket-clear-cache:3.1.1
            variables:
                 BITBUCKET_USERNAME: $BITBUCKET_USER_NAME
                 BITBUCKET_APP_PASSWORD: $BITBUCKET_APP_PASSWORD
                 CACHES: ["node"]
        condition:
          changesets:
            includePaths:
               - package.json
    - step:
        name: build and create/reuse the cache
        caches:
          - node
        script:
          - npm install
          - npm test

That’s it. This process will check your dependencies for changes, and if there is a change, it will delete the existing cache and create a new cache so your builds are always using the most current versions of dependencies.

Limitations:

• Caches are currently global and shared across multiple branches. When the cache is invalidated it will clear the cache for all branches, until it is cached again.
• Changesets only find files in the last commit, so if multiple commits are pushed, only the last one will be checked for the includePaths section.

Watch and share your feedback in BCLOUD-16314 feature request as we will look into implementing out of the box support and better caching / invalidation strategies in the future.