Package Management, part 2: examples and gotchas

2012/02/05 at 6:20 pm · Filed under Uncategorized

In the course of getting CFEngine 3 package management to do what I needed, I expanded upon some very nice ideas by others in the community and ran into a few frustrations with RPM and CFEngine that I’ve been able to overcome. Wanted to take some time to contribute those back.

The examples below are written against the package_method bodies in the Yale Cfengine 3 library.

Examples

yum_rpm_byname – install a package, version-agnostically

 packages:

 "wireshark"
    package_policy  => "add",
    package_method  => yum_rpm_byname,
    action          => actionsettings_fix_inform("inform");
view raw pkg_mgmt1.txt This Gist brought to you by GitHub.

Notes:

With policy “update” or “addupdate”, this will auto-update clients when new versions appear in repos exposed to them. Avoid this if you care about version specificity.

yum_rpm_exact – install a package at a particular version

packages:

  "wireshark"
    package_policy  => "add",     
    package_method  => yum_rpm_exact,
    package_version => "1.0.15-1.el5_6.4",
    package_architectures => { "x86_64" },
    action          => actionsettings_fix_inform("inform");

view raw pkg_mgmt1.txt This Gist brought to you by GitHub.

Notes:

You must specify “package_version” in any promises where this body is used. Otherwise, the version gets set to “*”, which causes Cfengine to refer to the package in terms of package-* (see package_name_convention), which could match on (and install/delete, depending on package_policy) all kinds of packages unintentionally.

Cfengine expects the package_version’s format to match version-release (see “Specifying package names” in the yum manpage). All other formats will fail to properly match and yield poor results.

$ /bin/rpm -qa --qf '%{version}-%{release}\n' wireshark
1.0.15-1.el5_6.4

Note that ‘version’ and ‘release’ are distinct pieces of metadata describing a package revision (in this case, Red Hat’s 6.4 release for RHEL5 of wireshark version 1.0.15).

yum_rpm_exact_enablerepo(repo) – install a package at a particular version, against an otherwise-disabled repository

packages:

      "rhm-docs"
        package_policy  => "add",
        package_method  => yum_rpm_exact_enablerepo("yale"),
        package_version => "0.7.946106-8.el5",
        package_architectures => { "noarch" },
        action          => actionsettings_fix_inform("inform");

view raw pkg_mgmt3.txt This Gist brought to you by GitHub.

This turns on an otherwise-disabled Yum repository temporarily for the purpose of installing a particular package. We use this to make way for per-package overrides from a local Yum repository.

Notes:

‘repo’ must already be defined in your yum configuration – otherwise yum will scream.

Caveats for yum_rpm_exact apply to this one too – see above.

yum_rpm_byname_enablerepo(repo) – like yum_rpm_byname, but with repository
“repo” enabled

packages:

  "wireshark"
    package_policy  => "add",
    package_method  => yum_rpm_byname_enablerepo("yale-testing"),
    action          => actionsettings_fix_inform("inform");

view raw pkg_mgmt4.txt This Gist brought to you by GitHub.

Notes:

Repo must already be defined in a file in yum.repos.d – otherwise yum will scream.

Caveats for yum_rpm_byname apply to this one too – see above.

Gotchas

Version comparator problems

As of 3.1.2, there’s a bug in Cfengine’s logic for comparing installed and requested package versions; it is unable to ever detect that the correct version is installed and will attempt to update in perpetuity. This is supposed to be fixed in later versions (>=3.2.0 RC); I haven’t tested newer versions myself but use a workaround for agents running versions affected by this bug.

packages:

  # First we want wireshark 1.0.11

       "wireshark"
         package_policy  => "add",
         package_method  => yum_rpm_exact,
         package_version => "1.0.11-1.el5_6.4",
         package_architectures => { "x86_64" },
         action          => actionsettings_fix_inform("inform");

  # Later we want wireshark 1.0.15.  Swap above policy for:

       "wireshark"
         package_policy  => "addupdate",
         package_method  => yum_rpm_exact,
         package_version => "1.0.15-1.el5_6.4",
         package_architectures => { "x86_64" },
         action          => actionsettings_fix_inform("inform");
view raw pkg_comparator_example.txt This Gist brought to you by GitHub.

Side effect of workaround: CFEngine will attempt to “update” wireshark to 1.0.15 on every agent run, until the cows come home.

On alternative methods of version formatting, like epochs

Some packages’ versions may be odd in some way that makes referring to them by “version-release” unworkable. In that case we would need to add a new package_method for them in library.cf. We decided to break from our usual approach for library design and not create a full-featured package_method body, because it would have required passing several long, gory regular expressions as parameters.

The most likely case of need for a different version format is the addition
of epochs [1]. Per Red Hat, no two packages release by them have the same “version-release” string with different epochs, so epochs are spurious for anything that comes from them. This may *not* be the case for EPEL, 3rd party RPMs, etc.

Originally, I hoped to refer to version numbers in terms of “epoch:version-release” in our packages promises and underlying package_method bodies, to facilitate CFEngine’s comparison of version numbers (which, remember, must be a universal algorithm, since Cfengine treats package managers as black boxes). Unfortunately, the means of passing versions by Cfengine promises and yum formatting requirements make this a pain. Per the yum manpage, any formats that includes the epoch also require architecture:

       Specifying package names
        A package can be referred to for install,update,list,remove etc with
        any of the following:


              name
              name.arch
              name-ver
              name-ver-rel
              name-ver-rel.arch
              name-epoch:ver-rel.arch
              epoch:name-ver-rel.arch
view raw yum_manpage_excerpt.txt This Gist brought to you by GitHub.

Because of the mutually-exclusive two means of specifying version and architecture for a package (see below), we’d have to pass
“name-epoch:ver-rel.arch” as the promiser and craft regexes for drawing out
the name, version (in this case, by format “epoch:ver-rel”) and arch out
of the promiser to pass to Cfengine. This gets seriously icky because of
inconsistent naming of packages and versions.

Cfengine uses complex logic for comparing version formats, so any diversion from “ver-rel” as the formatting scheme should be well-tested; this is the widely-used format in the forum and bug tracker, so there’s also more community weight behind this as opposed to “ver” or “epoch:ver-rel”.

Two methods of specifying intended version/architecture

We pass the package version as the promiser and add package_version and
package_architectures as promises attributes.

It is also possible to define regular expressions for drawing n,v,a (name, version and architecture) out of the (more detailed) promiser, as an alternative to passing package_version and package_architectures explicitly. I think the regexes get dorky for this, so I avoid it.

If one should decide to do this, know that the two approaches to divining version and architecture are mutually exclusive within package_method (that is, don’t try to pull n,v out of “wireshark-1.0.5-1.el5_6.4″ in a promise and set promise_architectures to “x86_64″ – this will yield strange results). Either divine all three (n,v,a) from the promiser or just n.

[1] For more on epochs, see “When Version Numbers Aren’t Enough” at rpm.org

Leave a Comment

Intro to package management with Cfengine 3

2012/02/05 at 4:34 pm · Filed under Cfengine, library.cf, What I Do

I’ve updated the public copy of our CFEngine 3 library to include the variants of package management bodies we’ve been using for several months now in production.

Cfengine’s approach to supporting multiple package managers

Cfengine treats package managers as black boxes – rather than natively define what it means for a package promise to be kept, these pieces are left as an exercise to the reader for his package manager of choice:

  • Command to list packages installed on the system
  • Regexes for retrieving names, versions, and architectures (n,v,a) from that list
  • The means by which to extract (n,v,a) from the promiser passed in the package promise, so Cfengine can compare against the above to determine whether the promise has been kept
  • Commands for installing, deleting, and verifying packages

So, there’s some legwork to be done to make sure package promises work as you’d expect, but there’s serious power in that flexibility – CFEngine packages promises can be made to work with arbitrary managers, even ‘dumb’ managers like tarballs. Fortunately, for the most common package managers, this work is already done in the Community Open Promise Body Library (COPBL).

Building blocks

There are two components to implementation of package management policy in CFEngine 3. Packages promises describe what specific package states are desired (“libkate should be installed at version 0.3.8-3″, “wireshark should be installed at at least version 1.0.15-1″; “httpd should not be installed”). Within the packages promise, ‘package_policy‘ defines what Cfengine should do when a package matching the name, version and architecture of a given package is not found to be installed on the system.  

We use:

  • add # if package not installed, add by way of package_add_command
  • update # if package version installed is less than version requested, update by way of package_update_command
  • addupdate # do both of the above, in order
  • delete # if package installed, remove by way of package_delete_command

(There are additional options, which I’ve opted not to use.  reinstall: delete then add; patch: wholesale system update by way of package_patch_command; verify: confirm a package is “okay” by way of package_verify_command)

Example:

 packages:

 "wireshark"
    package_policy  => "add",
    package_method  => yum_rpm_byname,
    action          => actionsettings_fix_inform("inform");
view raw pkg_mgmt1.txt This Gist brought to you by GitHub.

Packages promises call package_method bodies, which are reusable components called by packages promises that define how CFEngine will interact with package managers to determine whether promises are kept and how to go about keeping them. Many such bodies are available in the Community Open Promise Body Library (COPBL).

Though we are a RHEL shop, we did need slightly different functionality than the yum and rpm packages bodies provided by the COPBL, so I wrote our own package_method bodies to meet internal needs. More on these in my next post.

Package_method bodies tell CFEngine how to extract names, versions, and architectures (n,v,a) from both the package promise itself and from the list of installed packages on the system, provided by a package manager.  CFEngine compares the requested package (n,v,a) against the those in the installed package list to determine what actions are needed to bring the system into a convergent state.

Leave a Comment

Cfengine bundles for local user management

2011/09/12 at 11:07 pm · Filed under Cfengine, What I Do

I uploaded new policy to my git repo tonight to share work in local user management I think others may find helpful.  Here’s what they allow you to do:

  • Define a list of users that should exist on the system.  Create ones that don’t exist, making backups of current state first
  •  Copy users’ public SSH keys; enforce permissions on their home dirs and .ssh dirs
  •  Subscribe users to local Unix groups.  Create groups that don’t exist.
  •  Change users’ shells to something nonstandard

The *bundles.cf files are where the reusable components live.

userInfo.cf contains mappings of users’ UIDs and GECOS fields.

users.cf is a real-life, slightly scrubbed, example of how to use these components.

Comments (3)

Meet our library, or, Where I’m keeping the bodies

2011/04/03 at 10:48 pm · Filed under Cfengine, library.cf, What I Do

I’ve been working recently on a project to upgrade Yale’s existing Cfengine 2 installation to Cfengine 3.  Since we’ve had production hosts running cf3 for a few months now, I want to take some time to start sharing our experiences with the community.

I think the most compelling advantage of Cfengine 3 over its predecessor is its set of language features to support code reuse, which allow the user to create a library of building blocks for use in policy.  My team has been running version 2 for several years, and our cf2 codebase is active and sizable (over 11k lines); while this makes the job of converting production policy a challenge, it also means we had a bounty of real-world use cases to inform the design of  reusable components in our own collection.

Stability was a central design goal for our library.  We aimed to create as comprehensive but concise a set of bodies as possible, both because several administrators develop Cfengine policy within our group, so standardization makes sense; and because we plan to federate pieces of our Cfengine policy and infrastructure to other groups of system administrators at Yale.  To that end, our strategy in library development was to create, for each type of body we knew we needed to use, a general-purpose body (in some cases, a few of them) and a full-featured body, which takes in all body attributes as parameters (except, sometimes, in cases where enabling the passing of a parameter implied turning on behavior that was likely undesirable).

I’m happy to say this strategy is serving us well, so far – as we convert more v2 policy to v3, very few additions/modifications to the library have been necessary, and not due to overuse of the full-featured bodies – almost all of our v3 promises are written against the general purpose variants.  I’ve published it here:

https://github.com/jlgreer/yale_cfengine3/blob/master/library.cf

 

Feedback welcome!

Leave a Comment