[SOLVED] Puppet does not update package after changing the repo

Issue

According to https://docs.gitlab.com/runner/install/linux-repository.html#upgrading-to-gitlab-runner-10 the repos changed, so we are trying to update the gitlab-runner of all nodes. We need to delete the old repo and add the new one and then update the package.

On our puppet manifest[1] we update the repo, ensure the latest version of the package and after this update we have to run a script before ensuring that a service is running. Our problem is that we should only run this script after an update.

Right now, even if the repo is updated, the package is not. The package is only updated if distro_sync runs or if we run "yum update gitlab-runner", never through a puppet run. It seems that the package is never updated as if it was checking the latest version of the old repo instead of comparing against the recently added repo.

[1]

# Installs a GitLab-CI runner for CERN GitLab
class gitlab::gitlab_ci_runner (
  String $ensure = 'latest', # passed to the gitlab-runner package. Can be used to force a version
) {
  ensure_resource('yumrepo', 'gitlab-runner', {
      descr    => 'gitlab-runner for EL6/7',
      baseurl  => "http://packages.gitlab.com/runner/gitlab-runner/el/${::operatingsystemmajorrelease}/${::architecture}",
      gpgcheck => 0,
      enabled  => 1,
      exclude  => absent,
  })

  ensure_packages(['gitlab-runner'], {
    ensure  => $ensure,
    require => Yumrepo['gitlab-runner'],
  })

  exec {"post-install":
    command => 'sudo /usr/share/gitlab-runner/post-install',
    provider => shell,
    onlyif  => 'test -e /usr/share/gitlab-runner/post-install',
    refreshonly => true,
    subscribe => Package['gitlab-runner'],
  }

  service { 'gitlab-runner':
    ensure  => running,
    enable  => true,
    require => [Package['gitlab-runner'], Exec["post-install"]]
  }

}

Solution

The reason you are experiencing this behavior is because puppet is prefetching the repository metadata for the package resource prior to catalog application of resources. Note the relevant source code for the current state of the source code here for yum and here for the general provider. Note that the broad functionality of the linked code has not changed over time, so while the subtleties may change, the overall behavior has/will not.

Due to this, the determination of the latest version of a package occurs prior to the application of the resource. Therefore, in your situation the repositories you are subscribed to at catalog compilation will determine the latest version of a package. Your repository subscription change during catalog application will not impact the behavior of ensure => latest.

As you can probably guess, ensuring a specific version will still have the desired effect as it will utilize the new repository and no resource prefetching regarding latest occurs (other prefetching does still occur). Alternatively, a consecutive catalog application will have the desired effect for ensure => latest. To summarize, your workaround options here are:

• Apply the Puppet catalog twice in succession.
• ensure => <version> where you specify an exact version or version-release, such as 10.0.5-el7.

As one can expect, a great additional source of information about resource prefetching is Gary's blog article here. Scroll down to the title that begins with "Prefetching, flushing, caching, and other". Note the normal caution about Gary's blog having "strong language". The official documentation for it is essentially useless.

Answered By - Matt Schuchard