Wednesday, January 5, 2022

[SOLVED] AWS AMI: Need to explicitly remove apt locks when provisioning AMI from bionic

Issue

I am trying to provision via packer and its ansible provisioner a custom AMI, based off the official Ubuntu 18.04 ami as provided by AWS.

My packer provisioning that searches for the appropriate base ami is as follows:

  "source_ami_filter": {
    "filters": {
      "virtualization-type": "hvm",
      "name": "*{{user `ami-os`}}*",
      "root-device-type": "ebs",
      "architecture": "x86_64"
    },
    "owners": ["amazon"],
    "most_recent": true
  },

where I pass on the fly the variable: 

-var "ami-os=ubuntu-bionic-18.04-amd64-server"

Since the ami does not have python2 installed and I want to provision using ansible, I have to perform via raw:

- name: pre_tasks --> Install python2 for Ansible
  raw: bash -c "test -e /usr/bin/python || (apt -qqy update && apt install -qqy python-minimal)"
  become: yes
  register: output
  changed_when: output.stdout != ""
  when: ansible_isbionic

However, in most cases the above fails with the message that the process apt is locked:

amazon-ebs: fatal: [default]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 100, "stderr": "\nWARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n\n\nWARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n\nE: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?\nShared connection to 127.0.0.1 closed.\r\n", "stderr_lines": ["", "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.", "", "", "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.", "", "E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)", "E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?", "Shared connection to 127.0.0.1 closed."], "stdout": "190 packages can be upgraded. Run 'apt list --upgradable' to see them.\n", "stdout_lines": ["190 packages can be upgraded. Run 'apt list --upgradable' to see them."]}

To overcome this, I am explicitly performing:

- name: pre_tasks.yml --> Kill any apt commnds running
  raw: bash -c "killall apt apt-get || echo 'no apt-related processes found'"
  become: yes
  when: ansible_isbionic


- name: pre_tasks.yml --> Remove apt lock files
  raw: bash -c "rm -f /var/lib/apt/lists/lock"
  become: yes
  when: ansible_isbionic


- name: pre_tasks.yml --> Remove apt cache lock files
  raw: bash -c "rm -f /var/cache/apt/archives/lock"
  become: yes
  when: ansible_isbionic


- name: pre_tasks.yml --> Remove apt cache lock files
  raw: bash -c "rm -f /var/lib/dpkg/lock"
  become: yes
  when: ansible_isbionic

Any idea why the apt process gets locked?


Solution

I can advice you to wait for boot/update to complete but not kill lock-file.

We use this tricks in our cloud provisioning scripts:

while [ ! -f /var/lib/cloud/instance/boot-finished ]; do
  # wait...
done

while fuser /var/lib/apt/lists/lock >/dev/null 2>&1 ; do
  # wait...
done

When using Ansible you can pack this instructions into bash script and use script module instead of many raw executions. script module also doesn't need Python to be present on the managed host.



Answered By - Konstantin Suvorov
at