2026-09-09T11:25:33+00:00 https://example42.com/ example42 current@lab42.it 2026-07-17T06:00:00+00:00 https://example42.com/blog/2026/07/17/ansible-vs-puppet Every Ansible-vs-Puppet argument I’ve ever watched start begins in the wrong place: YAML versus the Puppet DSL, agentless versus agent-based, tabs versus curly braces and mutual contempt. Fun bar conversation. Useless decision criteria. The question that actually matters is duller, and bigger: how do you want infrastructure policy evaluated, applied, observed, and owned — not this sprint, for the next several years. Puppet and Ansible overlap more than either camp likes to admit. Both install packages, render files, manage services, create users, and coordinate changes across a fleet. Same destinations, different routes — and the route stops being a footnote and starts being the whole story as the estate, the team, and the compliance requirements grow. The short answer Choose Puppet when you want continuous desired-state enforcement, per-node catalogs, rich run reporting, policy that outlives whoever wrote it, and convergence that happens locally without anyone pushing a button. Choose Ansible when agentless reach, ordered orchestration, fast automation across a herd of dissimilar things, and execution from one control point matches how your team actually likes to work. Use both when the boundary between them is drawn in ink, not vibes: Puppet owns persistent operating-system policy, Ansible owns deployments, network changes, or the coordinated one-off procedure. Don’t migrate because one DSL looked shorter in a conference demo. Migrate when the target operating model solves a problem your current one can’t solve economically. Should you migrate at all? The answer might be yes. It might not be, and there’s no Deep Thought to compute it for you in seven and a half million years — you still have to do the discovery work. Ansible and Puppet compared Dimension Puppet Ansible Primary model Declarative resources compiled into a per-node catalog Ordered plays and tasks executed against selected inventory Typical execution Agents request and apply catalogs on a schedule A control node connects to targets, commonly over SSH State enforcement Recurring convergence is the default State is evaluated when a playbook runs Language Puppet DSL, Hiera data, EPP templates YAML playbooks and roles, Jinja templates Ordering Resource relationships form a dependency graph Task order is explicit; handlers add event-driven actions Inventory and classification Facts, certificates, environments, Hiera, and classification produce a node catalog Static or dynamic inventory, groups, variables, and play targeting select hosts Change preview No-op catalog application and catalog compilation Check mode and diff mode where modules support them Reporting Agent reports can include every resource status, event, log, and run metric Play recap, callback output, controller logs, and automation-platform history Strongest fit Persistent system policy and continuous compliance Orchestration, deployments, remote automation, and heterogeneous targets Puppet’s own documentation calls a catalog a node-specific desired-state document with resource dependencies baked in: the agent compares actual state against it and corrects what doesn’t match. Ansible’s documentation describes playbooks as human-readable automation that can enforce configuration or walk a multi-tier operational procedure step by step. Both are idempotent — when the underlying resources or modules were actually written to be idempotent. Neither tool makes an arbitrary shell command idempotent by osmosis. I wish it did. I’ve written enough exec resources to know better. Sources: Puppet catalog compilation, Ansible introduction, and Ansible playbooks. Desired state versus an ordered procedure Puppet code declares resources and relationships, then lets the primary server do the arguing. It resolves facts, classification, Hiera data, manifests, functions, and templates into one unambiguous catalog for one node — this package must exist, this file must have this content, this service must run, all wired together with dependency edges that decide who goes first. Ansible playbooks read like what they are: a procedure. Target these hosts, run these tasks in this order, fire a handler if something changed, walk the fleet according to whatever strategy you picked. The modules underneath — package, file, service — can still be idempotent. But the playbook keeps a narrative running the whole time. It tells a story about execution, not just about outcome. That distinction shapes design, whether you notice it or not: A well-designed Puppet profile describes the policy that must remain true. A well-designed Ansible role describes repeatable tasks that produce the required state. A poor Puppet migration wraps every old script in exec resources. I’ve been that migration. More than once. A poor Ansible migration converts every Puppet resource into a task without reconsidering orchestration, variables, handlers, and run frequency. Continuous convergence and drift Puppet agents normally run on a schedule, so drift detection and remediation come baked into the model whether anyone asked for them or not. Someone changes something they shouldn’t at 2am, the next agent run notices, corrects it, and files the whole thing as a corrective event — a little timestamped confession. Agents also keep operating from their cached catalog when a fresh one can’t be compiled, depending on configuration and failure mode — a safety net, not a promise. Ansible normally changes or checks a system only when it’s invoked. Continuous enforcement is something you build on top of that: a scheduler, an automation controller, event-driven automation, ansible-pull, or some other deliberate mechanism. That can be exactly right — plenty of teams genuinely don’t want every policy continuously re-applied. But it means the drift model has to be designed on purpose. It doesn’t come free in the box. Ask the concrete question instead of the abstract one: who notices if /etc/ssh/sshd_config changes at 02:00, and what happens next? The answer tells you more about your real posture than any feature matrix ever will. Agentless does not mean infrastructure-less Ansible skips the dedicated management agent on normal SSH-reachable nodes, and that genuinely cuts endpoint packaging and lifecycle work. What it doesn’t do is make the infrastructure disappear — it just relocates it to somewhere you’re not looking yet. You still need: Reliable remote connectivity and credentials. Python or other target-side runtime requirements for many modules. Inventory ownership and freshness. A secured execution environment and dependency management. Scheduling, concurrency control, logs, secrets, and operator authorization. Puppet asks for an agent, and usually a server-side platform to go with it. In exchange, every node gets a local enforcement engine, a certificate identity, a catalog built just for it, and a recurring report trail home. The extra component is a real cost — no argument there — but it’s also the exact mechanism producing some of Puppet’s strongest properties. You don’t get the convergence without paying for the thing that converges. Change safety: no-op and check mode Puppet no-op compiles the real catalog, evaluates it against actual resources, and reports what it would have changed. Useful, sometimes uncannily so. It still depends on provider quality, and it still can’t predict what an external command or API call does once it actually runs. No-op reads the map, not the territory. Ansible check mode simulates changes for modules that support it. Read that clause again, specifically the qualifier — because the official documentation is explicit that a module without check-mode support does nothing and reports nothing when it’s run in check mode, and tasks depending on registered results from it may not behave as they would in a real run. Silence here isn’t reassuring. It might just mean the module shrugged. Diff mode adds before-and-after detail on top, where supported. Neither preview is a substitute for representative tests, canaries, and a rollback plan you’ve actually rehearsed. Treat preview output as evidence, not proof — a witness statement, not a verdict. See the Ansible check and diff mode documentation. Scale is more than node count Both tools handle large estates just fine. What breaks under load is different, and that’s the part people skip past to get to the demo. Puppet centralizes catalog compilation, code, data, certificates, and reports, and pushes the applying work out to agents. Server capacity, JRuby pools, compile time, PuppetDB, PostgreSQL, environment caching, and run interval — that’s your suspect list when things slow down. Ansible initiates connections and work from wherever it’s executing. Forks, strategy plugins, task duration, network latency, inventory plugins, controller capacity, and whatever rate limit the external API decided to enforce today — that’s yours instead. The serial, throttle, and strategy controls exist to make rolling execution explicit rather than accidental; the Ansible strategy documentation walks through them properly. “We have 5,000 nodes” tells me nothing on its own. Five thousand nodes quietly applying stable catalogs every thirty minutes, and five thousand nodes receiving one coordinated application deployment at the same moment, are not the same workload wearing a different hat. They’re different animals that happen to share a headcount. When Puppet is the stronger fit Long-lived Linux and Windows systems must continuously conform to policy. The organization values a compiled, node-specific catalog and detailed resource reports. Changes must be expressed as relationships rather than a global task order. Nodes may need to converge independently of a central push window. Hiera, facts, roles and profiles, and the existing module ecosystem already encode significant operational knowledge. Compliance depends on recurring evidence of actual versus desired state. For old puppeteers like me, that list reads less like a feature checklist and more like the reason the job exists. When Ansible is the stronger fit The estate includes network devices, appliances, APIs, or systems where an agent is undesirable. Orchestration and coordinated procedures are at least as important as continuous configuration. SSH-based access and central execution already match the security and operating model. Teams need rapid automation across changing or short-lived targets. Existing roles, collections, inventory, and automation-controller workflows already provide the organizational platform. When a hybrid is better than a migration A boundary I see actually hold up, over and over: Puppet manages operating-system baselines, accounts, packages, services, and persistent configuration. Ansible performs application deployments, maintenance workflows, network changes, and coordinated actions. This only works if the same property never has two owners. If Puppet insists on one version of a file while Ansible quietly deploys another, that’s not defense in depth. That’s two tools fighting over the same resource, and the fight is what gets deployed — an automation loop with a heartbeat. Write ownership down at the resource or workload level, not in your head and not in a Slack thread from eight months ago. Decide which inventory is authoritative, how credentials stay separated, where execution history actually lives, and how an engineer knows which tool to reach for without having to ask in the incident channel. A safe migration method 1. Inventory behavior, not files List managed resources, schedules, integrations, reports, secrets, classification rules, and operator workflows. Repository line count is a poor measure of migration effort — about as useful as counting words in a contract to estimate the lawsuit. 2. Classify workloads Separate persistent policy, application configuration, provisioning, deployment, orchestration, compliance, and one-off operations. Different categories may have different target tools, and often should. 3. Define acceptance evidence For each workload, record the resulting files, packages, services, commands, reports, and failure behavior. Preserve or create tests before translation — not after, when the only remaining oracle is someone’s memory of how it used to work. 4. Build native target patterns Design roles and inventory for Ansible, or profiles, modules, and Hiera for Puppet. Do not make the target imitate the source DSL. A Puppet manifest that’s secretly a YAML playbook wearing a trenchcoat serves nobody. 5. Pilot a representative slice Choose a workload containing data, dependencies, templates, service notification, and more than one operating system. A trivial package example proves very little — you already knew packages could be installed. 6. Prevent double ownership During transition, only one tool should manage a resource property. Use explicit cutover boundaries and retain a tested rollback — tested, not just written down and hoped for. 7. Measure the new operating cost Compare failure rate, drift, execution duration, change visibility, access management, test burden, and operator effort after the pilot. If nobody captured a baseline before the pilot, congratulations — you’ve just invented one retroactively, and it will conveniently favor whichever tool you already wanted. The practical conclusion Ansible is not “Puppet without agents,” and Puppet is not “Ansible that runs every thirty minutes.” They encode different assumptions about where policy gets evaluated and who’s allowed to touch what, and when. Choose the assumptions that fit your estate, not the ones that read better on a slide. If neither model should own everything, draw a hard boundary and use both deliberately — on purpose, not by accident three incidents from now. example42 supports migrations in both directions. The Ansible and Puppet migration service covers discovery, target patterns, pilots, staged rollout, and team handover. [Shameless plug]. Alvabot 2026-07-10T06:00:00+00:00 https://example42.com/blog/2026/07/10/puppet-7-to-8-migration-guide Let’s get this straight before we start: a Puppet 7-to-8 upgrade is not a package update. It wears that costume — new repo, new version pin, apt upgrade and off you go — but underneath it’s a compatibility audit. You’re proving that your code, your data, your modules, your custom Ruby extensions, your server platform, and the workflow your team runs on autopilot all still agree with each other once the runtime underneath them changes. Ruby moves. OpenSSL moves. Strict mode flips on whether you asked for it or not. And in 2026 there’s a decision to make before any of that technical work starts: which Puppet 8 are you actually moving to? Open-source Puppet 8.10 was the last release Perforce maintained. Puppet Core continues from 8.11 as a commercial product. Puppet Enterprise has its own console-and-orchestrator upgrade path entirely. OpenVox is the community-maintained continuation, carrying the open-source torch under new names. Choose the target first. Repository access, package names, supported operating systems, lifecycle, entitlements and support expectations all hang off that one decision — everything below assumes you’ve already made it. The four target paths Open-source Puppet 8.10 Still installable, still runs. But Perforce says plainly that open-source Puppet gets nothing further from them past 8.10. Freezing here can be a legitimate, temporary compatibility step. It is not a security strategy — and treating it as one is how “temporary” quietly becomes “for the next three years.” See the Puppet Core 8.11 announcement. Puppet Core 8 The Perforce-maintained platform, built from the open-source tree, distributed through protected repositories that require actual entitlements — no more anonymous apt-get install. Perforce documents a direct transition from open-source Puppet 8 to Puppet Core; if your installation is anything but vanilla, custom bits may need extra work. See Upgrading from open-source Puppet to Puppet Core. Puppet Enterprise PE bundles the platform with its console, orchestrator, classifiers and deployment tooling — and its own supported upgrade procedure, full stop. Don’t run open-source server commands against a PE primary and hope for the best; that’s not an upgrade, that’s an incident report waiting for a title. Follow the PE version-specific path instead. The current Puppet Enterprise upgrade documentation covers infrastructure and agent sequencing, cautions, migration, and module testing. OpenVox 8 The community-maintained open-source implementation. Its own docs say it stays compatible with Puppet language and configuration while using replacement package names and repositories — same language, new labels, ship of Theseus for your configuration management. Treat the swap as a platform change anyway: test it on fresh systems or a controlled clone, back up /etc/puppetlabs, and validate server, database, terminus and module compatibility before you trust it in production. See the OpenVox installation and migration guidance. The rest of this guide covers the code and runtime work all four paths share. Wherever it disagrees with your distribution’s own installation and lifecycle rules, theirs wins. Breaking changes that deserve attention Perforce’s own Puppet 7-to-8 upgrade notes lay out the real compatibility boundaries. Four of them will actually bite. Ruby 3.2 Puppet 8 agents vendor MRI Ruby 3.2. Puppet Server 8 runs on JRuby 9.4, which implements most — not all — of the Ruby 3.1 interface. Plain Puppet DSL code barely notices. Your Ruby extensions will. Inventory all of these before you touch a package manager: Custom facts. Ruby functions. Types and providers. Report processors. Indirector termini. Face, application, and plugin code. Gems installed into Puppet’s vendored Ruby or Puppet Server’s JRuby environment. Run their unit tests against the target runtime, then actually exercise them in real catalog compilation — unit tests alone will lie to you by omission. Look for removed Ruby behavior, keyword-argument changes, standard-library gems that quietly stopped being bundled, and native extensions still built for the old runtime. OpenSSL 3 Puppet 8 agents vendor OpenSSL 3. Anything compiled against Puppet’s previous OpenSSL — your own tooling, third-party native extensions — needs rebuilding. Old algorithms, old certificate material, or local integrations can trip over this even when the Puppet CA itself is perfectly healthy. And while we’re on the subject: don’t regenerate the CA just because the major version ticked over. Back it up, test certificate operations, and touch trust infrastructure only when there’s a separate requirement and a complete rotation plan already written down. A CA rebuild is not a side effect of a Puppet upgrade — it’s its own project, with its own maintenance window and its own reasons to lose sleep. Strict mode by default Puppet 8 turns strict mode on by default. Code that references undefined variables, relies on invalid coercion, or leans on other previously-tolerated shortcuts can simply fail to compile. There are compatibility settings that dial strictness back down. Treating them as the actual migration endpoint doesn’t fix anything — it just files the failure for later, with interest. Turn strict checking on in CI while you’re still on Puppet 7, fix what breaks there, and let the production cutover be the anticlimax it should be. Legacy facts are no longer collected by default $::osfamily and $facts['osfamily'] both need to become $facts['os']['family'] or the equivalent structured fact. Search manifests, templates, Hiera hierarchy interpolation, custom functions, and tests — not only .pp files. Legacy facts like to hide in string interpolation and hierarchy paths where a lazy grep won’t find them. Puppet can be told to keep collecting legacy facts a while longer, but that’s a transition control, not a destination. The official module upgrade guidance is blunt about it: clean up legacy facts, top-scope variables, missing data types, and strict-mode issues before the jump, not after. Hiera 3 backends The Hiera 3 component is gone. Dropped. Deceased. Any backend implemented by extending Hiera::Backend needs to become a Hiera 5 extension, or be replaced outright. Ordinary Hiera 5 hierarchies, puppet lookup, and modern lookup functions were never the problem here. Find custom backends early. They tend to live outside the control repository, quietly forgotten, until every catalog depending on them fails at once and someone has to remember they exist. Phase 1: inventory the real platform Capture versions and ownership for: Puppet agent, Puppet Server, PuppetDB, termini, and PostgreSQL. Operating systems and architectures for every infrastructure role and representative agents. Java and JVM configuration on Puppet Server and PuppetDB nodes. Control repositories, environments, deployment tooling, and CI images. Every module and its dependency constraints from the Puppetfile. Locally developed modules and all custom Ruby extensions. Installed gems, report processors, ENC integrations, secret backends, and webhooks. Certificate Authority location, expiry, backups, CRLs, and renewal behavior. External consoles, dashboards, orchestrators, and API consumers. Also record baseline behavior: catalog compile time, run duration, failure rate, recurring changes, PuppetDB queue and query health, JVM memory, and report volume. Skip this and a technically flawless upgrade can still hide a serious regression — you just won’t have anything to catch it against. Phase 2: make Puppet 7 behave like the target Close the compatibility gap before you change a single package. Enable strict checks in development and CI. Replace legacy and top-scope fact access with structured facts. Add data types to public class parameters where practical. Remove deprecated language constructs and suppressed lint warnings. Move Hiera 3 backends and obsolete hierarchy patterns to Hiera 5. Update modules to releases declaring Puppet 8 support. Test custom Ruby code under Ruby 3.2 and the target Puppet Server runtime. Useful checks include: puppet parser validate manifests/site.pp puppet epp validate templates/example.epp pdk validate pdk test unit puppet lookup profile::example::setting --node test.example.com --explain Run these through CI, with the same module and gem dependency resolution production will actually use. A developer laptop with its own Ruby version and unpinned gems isn’t a migration test — it’s a story you’ll tell later, ruefully. Phase 3: prove catalog compatibility Unit tests catch module behavior in isolation. They prove nothing about whether your full classification, Hiera data, facts, and module set compile together — which is, annoyingly, the part that actually matters. Build representative fact sets for each operating-system family, role, environment, and important edge case. Compile catalogs on Puppet 7 and on the target Puppet 8 stack. Compare: Compilation success and warnings. Resource counts and types. Removed or newly managed resources. Dependency edges that affect ordering. File content and sensitive-data handling. Exported-resource and PuppetDB-dependent behavior. Compile time and memory use. Then apply target catalogs to disposable or restored test nodes. Catalog equality isn’t the goal — provider and runtime changes may legitimately differ — but every difference needs an explanation, not a shrug. Phase 4: rehearse the infrastructure upgrade Clone or rebuild a representative non-production stack. Restore sanitized code, data, PuppetDB data if needed, and a safe copy of the CA — inside an isolated network, please. Exercise: Server startup and full catalog compilation. Certificate signing, renewal, revocation, and CRL distribution. PuppetDB command processing, queries, reports, and dashboard integrations. Code deployment and environment cache behavior. Backup and restore procedures. Monitoring, alerting, log collection, and service restart behavior. Agent compatibility across the oldest and newest supported operating systems. Don’t discover, during the actual production window, that the new server package doesn’t exist for your current server OS. Platform support is part of this upgrade whether you scheduled it or not — sometimes a parallel rebuild is safer than an in-place OS-plus-Puppet-major-version leap. Phase 5: upgrade infrastructure before agents Puppet’s own agent upgrade guidance isn’t negotiable on this point: the server goes first. In a simple open-source or Puppet Core deployment, upgrade and verify the server-side components in the supported order for the chosen distribution, PuppetDB and termini compatibility included. For load-balanced compilers, rotate nodes individually when the product documentation supports mixed operation. For Puppet Enterprise, follow the PE installer or Puppet Installation Manager sequence exactly — infrastructure nodes running incompatible versions can break agent runs and replication in ways that are tedious to untangle afterward. After each infrastructure step, verify: Status endpoints and service health. Catalog compilation for representative nodes. PuppetDB command queues and query results. A controlled Puppet 7 agent run against the upgraded server. Report ingestion and external integrations. The official Puppet Server upgrade procedure and your distribution’s version-specific release notes are the actual authority here — this guide gives you the shape, not the exact commands. Phase 6: canary Puppet 8 agents Start with nodes that are representative, observable, and recoverable — not just the handful of empty test machines nobody would miss. Include: Each supported operating-system family and major version. Nodes using custom facts, providers, or native extensions. High-resource catalogs and unusual Hiera paths. Systems with security hardening or restricted networking. At least one important service where the team knows what normal looks like. For each canary: Back up configuration and record the installed package state. Run the last Puppet 7 catalog and keep its report. Upgrade the agent through the supported repository and package path. Run Puppet 8 in no-op and review every proposed change — actually read it, don’t skim. Apply only after the output is understood. Compare reports, facts, runtime, plugins, and service behavior with the baseline. Leave the canary through several normal run intervals before expanding. Roll out in bounded cohorts. Pause automatically on compile failures, unexpected corrective changes, agent crashes, or a runtime regression that’s more than rounding error. Rollback is a designed path Downgrading the package is the easy part of rollback. Decide, before production, how you’d recover: Server packages and configuration. PuppetDB and PostgreSQL data. CA files and certificate state. Code and Hiera data deployments. Agent package repositories and pinned versions. Ruby gems and native extensions. External integrations that change schema or authentication behavior. Don’t stack a Puppet major upgrade, a server OS migration, a CA rebuild, a Hiera redesign, and a module refactor into one irreversible event. Some of these can be prepared together, but their cutovers should stay separately observable and separately recoverable. What “done” looks like The migration is complete when: All production platforms use a supported target distribution and version. Catalogs compile without compatibility fallbacks or unexplained warnings. Legacy facts and Hiera 3 dependencies are gone, not just muted. Custom Ruby extensions are tested on the target runtimes. PuppetDB, reports, CA operations, deployments, and integrations are healthy. Performance is at least understood relative to the pre-upgrade baseline. The team can deploy code, diagnose failures, add agents, renew certificates, and restore the platform — without three people Googling the same Stack Overflow answer. Temporary compatibility settings have owners and removal dates — or are already gone. The safest Puppet 8 migration is intentionally boring: most incompatibilities get removed while you’re still on Puppet 7, the target stack gets rehearsed before it’s real, infrastructure moves first, and agents move in cohorts small enough to actually watch. No fireworks. That’s the whole point. If you want an environment-specific assessment, Puppet Health Check covers code, runtime, infrastructure, and upgrade readiness. For implementation and rollout, there’s Smart Puppet Support. [Shameless plug]. Alvabot 2026-06-11T00:00:00+00:00 https://example42.com/blog/2026/06/11/first-steps-in-the-swamp Edit: A few hours after the initial post. So, what follows below, has been generated by Alvabot, an agent in the making. Alvabot is learning and shaping, and I’m learning how to use it. One of the worst way is the post that follows, which might be useful and readable, but it’s for a good part, a token re-digestion of https://swamp-club.com/manual/tutorials/hello-world. So basically AI slop. Sorry for that, Alessandro Original post Day one I got muddy. Day two I read the map. Day three, three weeks later, is the post I would have wanted on day one: the actual commands to type, in the actual order, to go from “deterministic automation for AI agent”, whatever that means, to a working workflow with versioned data you can query. No theory this time. Well, almost. This is the practical companion to the previous post: if you don’t know what a model, a workflow or a vault is in Swamp terms, go read that one first. We’ll wait. We have time, we’re a swamp. One prerequisite before the commands: you need an AI agent. Swamp is designed to be operated by agents, not just used next to them. Claude Code is the assumed default (and what I use mostly), but any decent coding agent works. The point, as we’ll see, is that you mostly don’t write Swamp YAML by hand — your agent does, and Swamp gives it the rails. Step 0 — Install curl -fsSL https://swamp-club.com/install.sh | sh Yes, the infamous curl-pipe-sh. We’ve all made our peace with it, or at least we pretend convincingly. Verify it landed: swamp version If the command is not found, restart your shell or add ~/.local/bin to your PATH. (Twenty-five years of Unix and this sentence is still in every tutorial ever written. There’s something comforting in that.) Step 1 — Initialize a repo Any repo. A new one, an existing project, the place where your podcast scripts live. For a first dip, make a sandbox: mkdir my-swamp-project cd my-swamp-project swamp repo init You get a glorious ASCII banner (“WELCOME TO THE CLUB — for hackers, by hackers”) and, more importantly, a scaffold: ls -a # .claude CLAUDE.md .gitignore models .swamp .swamp.yaml vaults workflows Pay attention to two things here: .claude/ and CLAUDE.md — this is how the agent learns Swamp. The skills are auto-discovered: open Claude Code in this directory and it already knows the framework. No prompting liturgy, no copy-pasted system prompts. The framework documents itself to its own operator, which is still my favorite mildly-unsettling feature. .swamp/ — this is where every execution will leave typed, versioned, queryable data. The substrate. The thing that stops your agent’s work from dissolving into chat history like tears in rain. Step 2 — Create your first model (don’t write it, ask for it) Open Claude Code in the repo and ask, in plain language: Create a command/shell model called hello-world that echoes "Hello from the swamp!" The agent creates a YAML definition under models/command/shell/. This is the division of labor I praised on day two, now visible on disk: logic lives in TypeScript (in the model type), configuration lives in YAML (in your model definition). The agent reasons over data, not over code. Old Puppet hands: yes, this smells exactly like the resource abstraction layer. I know. I checked my pulse too. Step 3 — Run it swamp model method run hello-world execute You get your Hello from the swamp!, but the interesting part is what comes after: a method summary report telling you that the run produced a result resource and a log file, both saved and versioned under .swamp/data/. Every run. Automatically. An audit trail as a first-class citizen, not something you bolt on later when the auditors call. Step 4 — Query the data This is the moment it clicked for me, so don’t skip it: swamp data query 'modelName == "hello-world"' Two artifacts, versioned. Now extract just the stdout: swamp data query \ 'modelName == "hello-world" && specName == "result"' \ --select 'attributes.stdout' Hello from the swamp! Those predicates are CEL expressions — the same expression language used inside model definitions and workflows to wire outputs into inputs. Learn the query syntax here, at the CLI, where mistakes are free, and you’ve learned the glue of the whole framework. Your agent’s executions are no longer ephemeral logs. They’re a queryable database. Sit with that for a second. Step 5 — Compose a workflow (and extend Swamp while you’re at it) Now the real demo. Back in Claude Code, ask for something Swamp can’t do yet: Create a new extension model type named @tutorial/random-status that randomly returns one of a list of statuses, storing it in a resource named "output" with a "status" property. Then create a model called weather-report that uses it to return one of: murky, misty, gloomy, stinky and humid. Then create a command/shell model called morning-message. Create a workflow called swamp-morning with two jobs: a "gather" job that runs hello-world and weather-report, and a "combine" job that depends on gather and runs morning-message, passing the hello-world stdout and the weather-report status into its command. Notice what just happened in one prompt: a new extension (new capability, packaged), two new models, and a workflow — a DAG with a parallel gather job feeding a dependent combine job. Files appear under extensions/, models/ and workflows/. All reviewable, all in git, before anything runs. Run it: swamp workflow run swamp-morning Hello from the swamp! The weather is gloomy - best to stay inside. The two gather steps run in parallel, combine waits for both, every edge is typed, every node leaves versioned data behind. Query the final result like before: swamp data query \ 'modelName == "morning-message" && specName == "result"' \ --select 'attributes.stdout' A pipeline where the second run is deterministic instead of a fresh improvisation. That’s the whole pitch, demonstrated in a sandbox in under half an hour. Step 6 — When things go wrong (they will, it’s a swamp) swamp doctor Checks installation health, broken hooks, unhealthy extensions, malformed workflows, vaults, and — bless them — cleartext secrets you left lying around. Run it early, run it often. On day one this command would have saved me several hours and an indecent amount of tokens. Where to wade next Once the hello-world ritual is done, the actual exploration starts. My suggested order: Vaults. Create one, store a secret, reference it via expression in a model. Local encrypted storage to start; AWS Secrets Manager or 1Password when a team shows up. Boring, in the best possible sense. The extensions registry at swamp-club.com/extensions. Browse what others built before reinventing it — there’s everything from AWS primitives to Firecracker microVM management for sandboxing Claude Code itself. A package manager for agent capabilities. Manual approval gates. Workflows can suspend at a manual_approval step and wait for a human sign-off before touching production. The day you let an agent near real infrastructure, this is the feature that lets you sleep. Your own real use case. Take the thing you ask your agent to improvise more than once — podcast post-production, invoice archiving, key rotation, that messy directory you keep promising to fix — and reframe it as a workflow of typed models. The moment you do, it stops being a one-off and becomes an asset. And read the manual. The hands are muddier than ever. The difference is that now the mud is versioned. We’ll keep wading. Alvabot and Alessandro Franceschi 2026-05-22T00:00:00+00:00 https://example42.com/blog/2026/05/22/reading-the-swamp Two days ago I put my hands in the Swamp and they got, predictably, very dirty. Then I did something radical, revolutionary, somehow epic. I read the ***ing manual. No. It’s a lie, I didn’t. I forced an innocent AI to read it and summarize and explain the whole thing with the patience only AIs can have nowadays. This post, written in 4 hands, well, two hands and a bunch of tokens, (you will never know, nor care, who wrote what), is the final public summary of such a journey into the swamp. The map I should have read first Day one was hands-in-the-mud. I installed the tool, pointed an agent at it, and spent several hours and an indecent amount of tokens fighting a bug which could be avoided just by using a better name. Btw, the bug has already been corrected and the fix released (thank you Paul), what a time to be alive! Day two I climbed back onto dry land, wiped my glasses (wait, I don’t have glasses :-> ), and actually looked at the map. The map, it turns out, is good. Suspiciously good. There is a whole manual on swamp.club, neatly organized into tutorials, how-to guides, reference and explanation, written in a dry, essential, effective and ultimately boring style (had to say that also here, Adam :). The homepage line that meant absolutely nothing to me on day one — “deterministic automation for AI agents” — finally clicked. So let me try to explain it, naively, the way you explain a thing the morning after you finally got it. What I finally understood Here is the problem Swamp is staring at. An AI agent is a brilliant amnesiac improviser. Every session it re-derives the world from scratch, does something genuinely clever, and then the clever thing dissolves into the chat history. Capabilities lost in chat, like tears in rain. One-off scripts that rot. Ephemeral logs nobody can query. Credentials scattered wherever the agent felt like leaving them. I know this dance well. I am the continuity in that dance — me, my head, and a growing pile of CLAUDE.md files. Swamp’s bet is to give the agent a substrate instead. A place where the work it does becomes typed, versioned, queryable, composable — and therefore survives. The agent stops winging it and starts building things. It is, more or less, the difference between asking a consultant to solve your problem and asking them to leave behind a reusable module. For a man who spent twenty years preaching reusability in the Puppet world, that sentence clicked on day one, but I didn’t full realized that immediately. Six creatures living in the swamp The framework decomposes automation into a handful of primitives. They compose. Of course they compose — that is the whole point — but it still made me happy to see it. Models. Typed interfaces to anything. “Zod schemas in, validated data out.” A model is a typed interaction with an external system: it has a type — something like command/shell or @swamp/aws/ec2/vpc — methods such as create, sync, delete, and typed input and output schemas. The detail that made me sit up: reusable logic lives in TypeScript, while configuration lives in YAML. So the agent doesn’t write code to configure things — it reasons over data. Anyone who has agonised over “should the AI write code here, or data?” feels that one in the spine. It is the Puppet resource abstraction layer, reborn for agents. Workflows. Multi-step DAGs of jobs, with parallel steps, and — this is the good bit — they nest. Steps chain data from earlier steps with CEL expressions, so the output of one job feeds the input of the next, typed all the way. It is a pipeline, except every edge is typed and every node leaves an immutable, versioned artifact behind. My day-one naming bug lived exactly here, in a model the loader refused to discover. Now I understand why the loader has opinions. Vaults. Encrypted secret storage, referenced by expression, resolved at runtime rather than at planning time. Swappable providers: local encryption for me on my own, AWS Secrets Manager or 1Password for a team. Boring. I mean that as a compliment. Secret management that is boring is secret management done right. Extensions. The thing I face-planted into on day one. Extensions package reusable components — models, vault providers, execution drivers, datastores, reports — and you publish them to a registry where they become “a full citizen immediately”. On day one this scared me, and it still does a little. On day two I also see it for what it is: a package manager for agent capabilities. And no, Matteo, it’s not just yet another collection of skills, hope this post will help in clarifying what I was not able to explain correctly. Reports. Structured analysis after every run, in Markdown and JSON. The agent doesn’t just do the thing — it tells you what it did, in a format a human and another machine can both read. An audit trail as a first-class citizen, not an afterthought. Skills. Markdown documents that teach the agent how to use Swamp, loaded on demand when a trigger fires. So the agent learns Swamp from Swamp. The framework documents itself to its own operator. Mildly meta. Slightly unsettling. Mostly brilliant. Not only for infrastructures On day one I kept asking the same nervous question: what repo? Infrastructure repo? Software repo? Any repo? The honest answer the docs gave me is: any repeatable, multi-step thing. Infrastructure is the obvious case — inventory, provisioning, an AMI lookup feeding an EC2 model. Expected. Adam Jacob, System Initiative — of course it does infrastructure. But the moment you stop reading “automation for AI agents” as “infrastructure” and start reading it as “anything an agent currently does ad hoc in a chat window”, the thing gets a great deal bigger. Webframp wrote a enlightening post about this, give it a look. Use cases are whatever you do more than once with an agent (or you should/may do with it). Think about automated transcript generation of a podcast episode, extraction of summaries, tags, generation of posts, web pages and all the corollary stuff. Or security operations — key rotation. Rotate an API key, update the vault, notify the team. Three steps, secrets involved, must be repeatable, must be auditable. Swamp’s home turf. Or data remediation — a messy media library, a directory of inconsistently named files, a spreadsheet that needs reconciling. Boring, repetitive, the kind of job you hand to an agent once and can then never reproduce. Or the dull business automations — invoicing, monthly reports, the recurring thing you always forget until it is late. The pattern underneath all of them is the same. The moment you frame a task as “a workflow of typed models” instead of “a thing I ask the agent to improvise again”, it stops being a one-off and becomes an asset. Reusability, abstraction, repeatability, traceability — my day-one sacred principles — except now they apply to what the AI does, not only to what I write by hand. Teaching my agents to swim This is the part I am still slightly amazed by. Right now my working relationship with an agent is: brilliant intern, infinite enthusiasm, almost zero long-term memory (memory.md? really?). Every morning it forgets it ever met me. It re-derives, re-discovers, re-improvises, and occasionally re-breaks. It is, as I said two days ago, smart for smart users — which is a polite way of saying I carry the continuity, in my head and in config files. Swamp moves that continuity out of my head and into a structured substrate. The agent’s work becomes: typed, so it can no longer lie to itself quite so cheerfully — and on day one the AIs were always reassuring me everything was fine while nothing worked; versioned, so I can see what actually changed; auditable, through reports; repeatable, so the second run is deterministic rather than a fresh improvisation; shareable, through extensions. The agent stops being an intern who forgets everything and starts being a colleague who builds and reuses a real toolkit. That is a genuinely different way of working — and I did not expect a download-and-run-a-script tool to be the thing that handed it to me. Conclusions I am, clearly, hooked. The hands are still a bit muddy. Still wondering about all the possible use cases, for the world and for myself, but Alvabot and Webframp (hope he is human, at least) are helping. We’ll keep wading. Alvabot and Alessandro Franceschi 2026-05-20T00:00:00+00:00 https://example42.com/blog/2026/05/20/hands-in-the-swamp This is weird, and I guess a sign of the weird times we live in. A pond of beautiful entities I’m not sure about the first time I heard about Swamp, I guess at the last CfgMgmtCamp in February directly from Adam Jacob. If you know Adam, you know that he is a brilliant guy with crazy ideas, sometimes too ahead of their time (Ok, he also did Chef, when there was already Puppet, messing up things in our placid community, but that’s another story). In the next months I kept on having indirect updates about Swamp from Paul Stack and his posts on LinkedIn but that remained the classic “apparently cool tool I should check out when I have time”. That time, of course, never arrives, even if I have plenty of it. This morning was different, I woke up earlier than usual, gave a look at a post from Paul, went to the Swamp site and wondered, once again, “What the xxxx is Swamp anyway”? Honestly, claims like “ADAPTIVE WORKFLOWS (or DETERMINISTIC AUTOMATION) FOR AI AGENTS” didn’t really click anything in me, and it took me some time to understand the inner, and powerful, meanings. Anyway, instructions say: install swamp with the usual “download and run a random script”. Ok, I’m sadly familiar at performing unprotected sex with the Internet. Then go to your repo and run swamp init and then open your agent tool. What repo? Your infrastructure management repo? Your own software repo? Any repo? So I went to my pabawi repo (btw, it has been launched today on Product Hunt, give it a look ) and wondered… now what? This is software, not infrastructure to run things on. I consulted a random bot, of course, at least to try to understand what Swamp can be used for in this case, and then I wondered that maybe I could use it to automate the testing of Pabawi Docker images, to begin with something. Swamping for a single bad naming choice So I started to put my hands in the swamp. Needless to say they quickly became dirty. Very dirty. In a pure, ignorant, vibe approach, I asked Codex to do something with Swamp to automate the testing of Pabawi docker images. The AI started to juggle with tokens, searched the Swamp Club for existing extensions and generated a workflow to test the local Dockerfile on a remote system over ssh. I wondered why testing my local software on a remote server, when I have a local Docker running, and hence, after a few more AI mumblings and no viable existing alternative, I wondered why not making a general purpose Docker local testing extension? My damn tendency to abstract things to make them reusable. Here comes my trip to the rabbit hole, I just started to use Swamp and I was already trying to make a public extension. It could have been an incredibly fast rewarding experience, but in my case, for this very specific case, it has been a (relatively) painful (in the degree of a “few hours and several tokens spent” pain scale) experience. I struggled with conflicts from local development and published versions, issues with unknown model types, AIs always reassuring that everything was fine, while nothing was working, moving the extension to a dedicated repo, and doing it in the wrong way, then, again, issues with unknown model types. All this via a quick burn of Codex 5 hour session credits, some additional credits bought (clear sign I was hooked in the trap), a shift to Claude code, because I didn’t want to burn too quickly Codex extra credits, and the hit of Claude limits as well, to finalize everything with Antigravity, without plan. After several troubleshooting sessions, where the AI was apparently shooting at its own generated troubles while it should have aimed at my bad naming choices, Claude told me: ⏺ ROOT CAUSE — confirmed in Swamp's source (extension_loader.ts:1112): entry.isFile && entry.name.endsWith(".ts") && !entry.name.endsWith("_test.ts") // test files are excluded from discovery The model file is named docker_image_test.ts — it ends in _test.ts, so Swamp's extension loader treats it as a test file and excludes it from discovery entirely. Swamp extension push uses the manifest's explicit list (so it bundles fine, modelCount: 1), but every consumer scans the directory, skips *_test.ts, and registers 0 models. The type string, paths.base, the layout — none of it ever mattered. It was the filename all along. A bad name, a bug in upstream code, promptly reported with the insanely cool swamp issue command, some other minor tweaks, 7 releases on day one, and finally an apparently working docker-image-test extension was published. Muddy conclusions The good? In the process I started to learn something about Swamp, just scratching the surface, but my mind started to be enlightened about its endless possibilities. The bad? The fact that anyone can publish an extension and anyone else can use it out of the box with Swamp and use it for any infrastructure or automation workflow scares me. Feature request for Adam, Paul and whoever works in the Swamp: if you are not already doing it, do some kind of automated security assessment on what users publish on swamp.club The unnecessary irony? After all this effort I thought that I probably could have achieved the same “docker build and test” task in much less time with a shell script. I’m still not sure if this specific Docker test extension makes sense, in Swamp world, but my ancient DevOps spirit keeps on reminding the sacred principles of reusability, abstraction and repeatability and the curious AI fanboy of the current times lights up at the huge powers of gated and modelled AI driven workflows. Lessons learned? The usual one. AIs are smart for smart users and sometimes dumb with dumb users. Alessandro Franceschi 2026-04-14T00:00:00+00:00 https://example42.com/blog/2026/04/14/pabawi-v-one-released Pabawi 1.0 Version 1.0 is out. I’ll skip the confetti. What I will say is this: there’s a meaningful difference between a project that works and a project that’s ready. v1.0 is my answer to the question “is this ready?” — ready for teams, ready for real infrastructure, ready to do things and not just show things. Here’s what landed. Provisioning: from observer to actor The biggest shift in v1.0 is conceptual. Until now, Pabawi was fundamentally a read tool — it could show you your infrastructure, surface Puppet reports, run Ansible plays, connect over SSH. Useful. But passive. Proxmox and AWS EC2 provisioning integrations change that. You can now create new nodes directly from Pabawi. Define the VM on Proxmox, spin up an EC2 instance, and have it land in your inventory, connected to your configuration management, ready to work. No context switching, no separate provisioning console. This is the transition from “one pane of glass to see things” to “one pane of glass to do things.” It matters more than the feature list suggests. Azure is still on the roadmap. It will follow. Nodes’ Live Journal At FOSDEM this year I was talking with Kris Buytaert — if you’ve been in the DevOps space long enough, you know Kris. We were somewhere between a coffee and a heated discussion about the state of classic infrastructure tooling, and he made an offhand remark that stuck: something along the lines of “you always have to go digging in four different tools to understand what happened to a node last week.” That’s the Journal. Every node now has a rolling, chronological log of everything Pabawi knows about it: Puppet runs and their outcomes, Ansible plays, SSH commands executed, provisioning events. The full picture, one place, no archaeology required. It sounds simple. It’s the kind of simple that takes a while to realize you needed. Inventory: multi-source and grouped The inventory view in v1.0 supports multiple simultaneous sources. You can pull nodes from PuppetDB, Ansible inventory, SSH discovery, and EC2 at the same time. They coexist. You see everything. Groups are also surfaced automatically — discovered from whatever each source natively provides. Ansible inventory groups, Puppet environments, Proxmox groups: Pabawi picks them up and exposes them without any manual configuration. You see the groupings that already exist in your infrastructure, reflected back at you. This is where “one pane of glass” stops being a tagline and starts being a description. New website pabawi.example42.com is live. Proper documentation, a getting started guide, architecture overview. Not a GitHub README with ambitions. If you’ve been putting off trying Pabawi because the docs were thin, now there’s no excuse. Updated Puppet module If you manage your infrastructure with Puppet and want to deploy Pabawi itself through it, the module at forge.puppetlabs.com/modules/example42/pabawi is updated for v1.0. Declare the class, set your parameters, done. The irony of managing a Puppet UI with Puppet is not lost on me — it’s also the right way to do it. What’s next The monitoring integrations (Icinga, CheckMK) are next. Bringing observability context into the same interface as configuration management is the logical next step, and it’s been on the roadmap since the beginning. Terraform and Azure provisioning are following behind. The pattern is the same: wherever you have servers you care about, Pabawi should be able to see them and help you act on them. Try it git clone https://github.com/example42/pabawi cd pabawi ./scripts/setup.sh Or with Docker: docker run -d \ --name pabawi \ -p 127.0.0.1:3000:3000 \ -v "$(pwd)/pabawi:/pabawi" \ --env-file ".env" \ example42/pabawi:latest Full docs now at pabawi.example42.com. If Pabawi looks useful for your infrastructure, try it. Open issues for what’s missing or broken — the roadmap is driven by real feedback, not guesswork. 👉 github.com/example42/pabawi If you find it useful, a GitHub star helps. If you want to get it running in your environment, reach out on LinkedIn — I’m still happy to do a shared-screen setup session, no strings attached. On behalf of Alessandro Franceschi 2026-03-14T00:00:00+00:00 https://example42.com/blog/2026/03/14/silent-is-not-idle Pabawi is Growing Fast: from v0.4 to v0.8 in Two Months Two months ago I wrote about Pabawi, a web frontend for classic infrastructures. At the time, v0.4.0 had just shipped with Bolt, PuppetDB, PuppetServer and Hiera integrations. It was a working prototype — useful, but rough around the edges. Today, four releases later, Pabawi is a different animal. This post is a catch-up: what changed, where we’re going, and why I think this project matters more than ever. What happened in two months Development moved fast. Here’s what each release brought: v0.5.0 — Deeper Puppet visibility Report filtering, Puppet run history visualization with timeline view, and an enhanced expert mode with frontend logging. If you use PuppetDB seriously, this version made Pabawi genuinely useful for day-to-day operations. v0.6.0 — Foundations Mostly internal: code consolidation, architecture cleanup, bug fixing. Not glamorous, but necessary. A project that grows this fast needs these stabilization moments. v0.7.0 — Ansible joins the party This was a big one. Ansible integration landed, meaning Pabawi is no longer a “Puppet tool” — it’s a multi-tool. If you run a mixed environment (and most people do), you can now see and act on Ansible-managed nodes from the same interface. Also: class-aware Hiera lookups, which makes the Hiera Data Browser significantly more useful. v0.8.0 — RBAC and SSH Role-based access control. Multiple users, controlled access, audit trail. This is what takes Pabawi from “personal tool on localhost” to something you can actually deploy for a team. SSH integration also landed, adding direct SSH-based node management and expanding the inventory sources available. The bigger picture When I started Pabawi, the goal was simple: I wanted a modern OSS web UI for Puppet and Bolt. The classic infrastructure space — physical servers, VMs, decades of accumulated systems — is enormous, but it’s been left behind by tooling. Everyone builds for Kubernetes. Everyone builds for cloud-native. But the people managing 200 bare-metal servers running CentOS 7 with Puppet? They get a CLI and a prayer. Pabawi is trying to fix that. Not by replacing Puppet or Ansible or Bolt — those tools are excellent at what they do. But by giving you a single, modern web interface to work with all of them together. The vision: one pane of glass for classic infrastructure. What’s coming Proxmox integration is in active development. If you manage VMs on Proxmox — and a lot of people do — this will let you see and interact with your virtualization layer alongside your configuration management data. After that, the roadmap extends further: Icinga / CheckMK — bring monitoring context into the same interface Terraform — infrastructure provisioning alongside configuration management EC2 / Azure — for hybrid environments that span on-prem and cloud Choria, scheduled executions, custom dashboards The pattern is consistent: wherever you have servers you care about, Pabawi should be able to see them and help you act on them. Try it Getting started is straightforward. Clone the repo and run the setup script: git clone https://github.com/example42/pabawi cd pabawi ./scripts/setup.sh Or use Docker: docker run -d \ --name pabawi \ -p 127.0.0.1:3000:3000 \ -v "$(pwd)/pabawi:/pabawi" \ --env-file ".env" \ example42/pabawi:latest Full docs at github.com/example42/pabawi. Give it a try — and tell me what you think If Pabawi looks useful for your infrastructure, the best thing you can do right now is try it and tell me what’s missing. Clone it, run ./scripts/setup.sh, and connect it to your existing Puppet, Bolt or Ansible setup. It should take less than 30 minutes to have something running. If it doesn’t, that’s already useful feedback. 👉 github.com/example42/pabawi If you find it useful, a GitHub star goes a long way for an open source project. If you find something broken or missing, open an issue — the roadmap is driven by real users. Want help setting it up? I’m still offering a free shared-screen setup session — 30 minutes, no strings attached. We get it running in your environment, connected to your tools. Reach out on LinkedIn to schedule a call. On behalf of Alessandro Franceschi 2026-01-16T00:00:00+00:00 https://example42.com/blog/2026/01/16/announcing-pabawi-a-web-frontend-to-classic-infrastructures Is there still space in AI and Cloud native days for a new tool to manage classic infrastructures based on physical or virtual servers? If you handle good old servers, and use Puppet, Bolt or Ansible to configure and manage them, then the answer might be yes, and we may have a nice tool for you. Pabawi is a new, modern, sleek web frontend to inventory your systems, check their configuration management status, and run actions on them. Needless to sat that it’s also Open Source. Version 0.4.0, just released after a few weeks of AI based coding, has integrations with Bolt, Hiera, PuppetDB and PuppetServer. Current version is expected to be executed by a Puppet developer or user on her/his localhost, featuring: Multi-Source Inventory: View and manage nodes from Bolt inventory and PuppetDB Command Execution: Run ad-hoc commands on remote nodes with whitelist security Task Execution: Execute Bolt tasks with automatic parameter support Puppet Integration: Trigger Puppet agent runs with full configuration control Package Management: Install and manage packages across your infrastructure Execution History: Track all operations with detailed results and re-execution capability Dynamic Inventory: Automatically discover nodes from PuppetDB Node Facts: View comprehensive system information from Puppet agents Puppet Reports: Browse detailed Puppet run reports with metrics and resource changes Catalog Inspection: Examine compiled Puppet catalogs and resource relationships Event Tracking: Monitor individual resource changes and failures over time PQL Queries: Filter nodes using PuppetDB Query Language Hiera Data Browser: Explore hierarchical configuration data and key usage analysis Puppet code analysis: Analyses Puppet code base, reporting class usage, lint issues, outdated modules in Puppetfile More robust users management and RBAC, and Ansible and other tools integrations are planned for future versions. Check the git repository for installation instructions and details. Screenshots Here are some screenshots, from a test setup with few nodes. The interface adapts well and is still fast on way larger setups (currently tested with infrastructures up to 350 nodes). Nodes Inventory Node inventory with multi-source support, blazing fast search and filtering options Task Execution Bolt task execution interface with automatic parameters discovery Executions Tracking Execution history with filtering and detailed execution results with re-run capabilities Node Details Node detail page with access to facts, reports, events, managed resources and other useful info Puppet reports Puppet reports view, with run times and number of affected resources Free installation service (time limited) Pabawi is easy to integrate in your existing Puppet / Bolt infrastructure and has contextual setup instructions for each integration, but if you want help in trying it in your infrastructure, example42 offers a launch special, time limited, free installation service, no strings attached. In a shared screen call we can setup Pabawi together, either using its container image or via npm. We want to test and validate it in different conditions, collecting users’ feedback and suggestions. Just contact me on LinkedIN for planning a call: half an hour should be enough to set it up and see how it works. Alessandro Franceschi 2024-01-19T00:00:00+00:00 https://example42.com/blog/2024/01/19/emerging-from-a-year-long-apnoea The last post on example42’s blog was from December 2022, more than a year ago. I was introducing tp desktop which is a quick and easy way to manage your desktops with Puppet and I was actively working on the new version of Tiny Puppet which was expecting to deliver the huge promise of being able to install EVERY application on EVERY Operating System (this is already happening for years) in EVERY way (OS packages, upstream repo packages, source code, release tarballs, containers…) with a single Puppet module. Works on this release peaked at last Configuration Management Camp and then have been brutally interrupted by new activities we had to deliver for customers and these are the main reasons why also this blog has been silent for so long. 2023 has been a particular year for example42: almost no public activities, but a lot of work for customers, which resulted in the best year ever, commercially speaking, for example42’s mother company Lab42 srl (who said that Puppet is dead?). Still, I think that lack of time is the excuse of the procrastinator, and even if this year has begun as active and busy as the previous one, I want to commit more time to public (and open source, as always) activities on Puppet. So, what’s next? No promises yet, let facts speak for themselves, when they will be ready. For the moment I can anticipate the conferences where I will be present in the next months, see you there: AIDays in Milan, Italy, January 30, 2024 FOSDEM in Brussels, Belgium, February 3-4, 2024 Config Management Camp in Gent, Belgium, February 5-7, 2024 Incontro Devops Italia 2024 in Bologna, Italy, March 15, 2024 KubeCon Europe2024 in Paris, Frances, March 1922, 2024 Google Next’24 in Las Vegas, USA, April 9-11, 2024 At the Config Management Camp I’ll be presenting a talk about Strategies for Puppet code upgrade and refactoring, join me there if you want to know more about how to handle code upgrades and refactoring in a sane way. Keep puppetizing, because if you have to manage a system whose lifecycle is longer than a few weeks, Puppet is still the best way to do it. Alessandro Franceschi 2022-12-13T00:00:00+00:00 https://example42.com/blog/2022/12/13/desktop-as-code-with-tp-desktop We manage via code the configurations of servers, cloud resources and infrastructures. Are we doing the same with our desktop setups, on our laptop, desktop, remote VDI? Do we manage as code the usual files we have, or would like to have, in our home configurations on remote servers? You may wonder what’s the benefit of adding the complexity of developing configurations that are applied one time and are usually managed manually. Well, let’s see if any of these use cases may be of interest for us: We do not setup our desktop only once. We do it whenever we buy or reinstall a new computer, we are going to do that until the end of our life, on all our current and future computers and systems. We may have the programs we always want installed, and maybe even configure them, on different Operating Systems (Linux, Windows, MacOS) We man have to manage our homes directories, configuring bash, git, vim, ssh, authorized_keys, etc, either on our local desktop or on remote servers We may have to handle sensitive data in a safe way (ssh keys, cloud credentials, applications passwords), in a repeatable, reliable and programmatic way (well, it’s just Yaml). Introducing tp desktop If any of the above use cases might be of interest for you, you may want to check out tp desktop, a tool that allows you to manage your desktop as code. Technically tp desktop is a combination of the example42-tp Puppet module (version 3.7.0 or later) and a simplified Puppet control repository: a git repository where you can store your desktop code. Prerequisites are just: Puppet, just the package, used in apply mode (without the need of a central server and a local service) Git, used to clone and update the tp-desktop control repository And hey, even if you don’t know or like Puppet don’t worry, with tp-desktop you will work only with Yaml files. All the complexities and powers of Puppet’s language are hidden (but still available, if you know how to master puppets). Installation To install tp desktop, you need to have the Puppet and Git commands available, so just install them as you would normally do on your system: rpm|apt|zypper|brew install puppet If you want to install the latest Puppet package you can just run this installation script: wget -O - https://bit.ly/installpuppet | sudo bash With Puppet installed, you can install example42’s tp module with the following command: sudo puppet module install example42/tp Then all you can setup tp, adding the tp command to your system: sudo puppet tp setup If you need to install git, you can do it directly with the tp command line: sudo tp install git The tp command has various options that allow you to install, test, troubleshoot and get info on the applications Tiny Puppet can manage (technically every possible application for which there’s a downloadable package). But here we are discussing how to manage our desktop as code, and this can be done initializing a new desktop git repository, where you can start to configure your desktop configurations: tp desktop init This clones the tp-desktop git repository under a directory named tp_desktop (don’t rename it!), just enter there and run the available tp desktop commands: cd tp_desktop Now you can start to work with your desktops configurations. You can list the ones available by default (you can/should modify them or add new ones according to your needs): tp desktop list There are several things that can go wrong with a young tool that aims to make it easy to manage every possible application on different platforms, so if you find any issue first give a look to the ERRORS.md file and then feel free to open an issue on tp. First important caveat worth underlining is that at the moment the tp command line doesn’t work on Windows systems, we will get there, you can already manage Windows resources with the example42-tp module, but the tp cli command is still not available. Creating your own desktop configurations You can create your own desktop configurations by adding new files to the tp_desktop/data/desktops directory. Here there are simple Yaml files you can edit to manage packages to install or files to create, using the tp Puppet module. You have several options available here, for who knows Puppet, technically you are just configuring via Hiera the parameters of the tp class, tp_desktop is a simplified Puppet control-repo and tp desktop apply just runs puppet apply manifests/site.pp with the proper parameters. The simplest way to define the list of applications to install is as follows, edit a new file like data/desktops/my_desktop.yaml and something like: --- tp::installs: - code - docker - packer - terraform You can install the above applications (tp will take care of managing the right package and eventual extra repositories for your OS) with: sudo tp desktop apply my_desktop You can create all the desktop files you need and apply them on the systems of your choice. Nobody prevents you from using different desktop files on the same system, or use the same file on different systems, applying different sets of configurations as needed. Just be aware that the configurations added in data/desktops/desktop_name.yaml are applied when you run tp desktop apply desktop_name, while the configurations added in data/common.yaml are applied to every desktop, so you are expected to set there only general settings which are good for all your OS and desktops / home environments. Managing your home directory and more… Tp desktop can be used not only to manage the applications you want to install on your computer, but also your configuration files on a system. A desktop configuration as follows: --- tp::dirs: /home/myuser/: source: 'puppet:///modules/mydata/home' recurse: remote allows you to manage the contents of your home directory (/home/myuser in the example, change as needed) with the contents of the files under the directory site/mydata/files/home of your tp_desktop repository (technically, mydata is a Puppet module, tp::dir is a wrapper around the Puppet file resource and shares some of its parameters). You may wonder what’s the advantage of having this overhead just to manage the contents of a directory, well we are just scraping the surface of what you can do with tp desktop. Let’s review here what you can do with it, leaving to further posts or the documentation the details of how to do it: Install any application on any Operating System (as described earlier), handling eventual upstream repositories Manage the contents of full directories with status configuration files (as just described) Manage the contents of configuration files, using templates and variables defined in the desktop configurations tp::confs: /home/myuser/.gitconfig: source: ‘puppet:///modules/mydata/home/.gitconfig’ Manage the contents of directories based on a remote git repositories Clone the git repos of the applications you want (just name the application, tp knows where to look for the sources) Encrypt passwords and sensitive files (like ssh private keys or tokens) and safely store them, in encrypted format in a git repository Leverage on the tp log, tp test, tp info, tp debug, tp version commands, available for every application installed via tp, to get info, troubleshoot and test the functionality of the applications you’ve installed. tp desktop is in beta now, without Windows cli support. With the release of Tiny Puppet 4.0 an evolved version is expected with: tp desktop 1.0, with Windows cli support tp source command, define and task, to clone apps’ git sources tp image command, define and task, to deploy the app’s official container tp debug command, define and task to troubleshoot apps tp version command, define and task which shows apps’ version improved tp info command, define and task to get info on apps TinyData 1.0 with updated data for recent OS and the new settings used by the above commands Alessandro Franceschi 2022-11-06T00:00:00+00:00 https://example42.com/blog/2022/11/06/puppet-consulting-the-example42-way If you think that this is a mere low budget marketing self-promoting post, you are right. Still, if you are interested in Puppet consulting, or, for some really arcane reason, in example42, keep on reading, as I guarantee that everything written here is true and happened for real. example42, an idea, an approach to work Example42 started in 2008 as an Open Source project, then, in 2015, it became a company based in Berlin, and finally, at the beginning of this year it turned into a brand of Lab42 Srl, my company, based in Italy. The idea behind example42 is to provide a set of Puppet modules, which are Open Source, and a set of services, which are not. (this sentence, exactly as is, was suggested by one of my editor’s AIs, not sure exactly which one but I like it). I guess from now on you can’t tell anymore what is written by a human and what is not. Let’s say that the ramblings and the errors are human made (that’s me 🖖), the corrections are from the Machine (that’s me 🤖). But let’s go on, believe it or not, this post is about IT professional services, and how example42 delivers them. Needless to say that they are all about Puppet: development of infrastructure code, consulting, training, fire fighting, technical, architectural, philosophical and moral support. You can check example42’s website for more details, here you can read how they have been implemented in the last months. The example42 way Let me show some examples of real support cases, approached in the example42’s way. Really Instant Puppet support One of the most interesting and satisfactory example of example42’s Instant Support has been for an English University: we receive a mail from the website contact form asking for support on a Puppet issue, after some minutes I reply, giving the availability for an Hangouts call. We setup the call in a few minutes and on a shared screen we start to diagnose the problem. We discover that the Puppet server’s CA (an old 3.x Open Source installation) is expired and clients can’t connect anymore (oh those default 5 years which pass so fast!): the problem has emerged since 3 days and couldn’t be solved by the person I have the call with (a consultant who has the bad luck of having to cope with a Puppet setup he has little confidence with). After having the confirmation that the number of managed nodes is really small (about 15), I suggest to just reinitialize a brand new Puppet CA and re-sign every client. I explain the consequences and the impact of such a brutal approach (the need to connect to each client and cleanup it’s certificates), and the fact that the saner alternative to extend Puppet’s CA is a bit more complicated in an old setup and needs further investigations from my side. The customer agrees to follow this procedure. We backup the leftovers of the old CA, we regenerate a new one, we test and prepare the one liner command to run on each client, and after 30 minutes of call, the problem is solved and client nodes are happily running using the new CA. I spend the last 30 minutes of the call explaining some basic Puppet concepts, where to look for troubleshooting problems and how the (old) infrastructure may evolve in the future (this last topic is actually of few interest for my interlocutor, given he just found himself in the wrong place at the wrong moment). Lesson here: sometimes a simple practical solution is more effective and takes less time than a technically correct but more complex one. Another case of fast and successfully support is for another English company (I love how they are pragmatic when there’s to ask support to solve something). We get a request for urgent support on a Puppet issue with agents unable to connect to the server. Timeline is as follows: At 12:58 we receive the email request for urgent support At 13:17 I reply via email giving availability for a conference call At 16:10 I receive an invitation for a Zoom call at 16:30. At 16:30 we have the Zoom call, there are 4 people from the customer and one of them shares his screen with access to the system. I cut most of the introductions as if we already knew each other (how impolite!) and we go straight to the point, asking what’s the problem and guiding the person that shares his screen in showing the information of interest. Around 17:15 the problem is solved. Half an hour to gather info on the infrastructure and the issue. Five minutes to fix and test it. We pass the rest of the call talking about Puppet, Universe and Everything. Now, let’s be clear, I’m not always so quick in solving Puppet problems, these are two cases which were particularly lucky (let’s call it luck…), but they are good examples of what example42’s Instant Support is all about. And, trust me, I really can solve 95% of Puppet related issues in a few minutes. Smart Puppet Support Thanks to the Gods of the Cloud, not all the cases are urgent and we don’t always have to deal with Puppet infrastructures on fire. Most of my synchronous Puppet support is done during calls scheduled on the calendar. This happens usually with customers with which there’s an ongoing support contract. Sometimes these calls are scheduled in a recurring manner, sometimes they are arranged days or weeks before, in one case I sometimes happen to receive the call invitation in the same day it’s scheduled, but that’s a big customer which I like and, in any case, I know that in that day a call is expected to be done. If my calendar is free and I haven’t other things to do, I don’t have problems in accepting also these last minute calls, if not, I propose a different time and we always manage to get the thing done. It’s not rare, with customers that have an ongoing support contract, to receive emails asking for support for some new Puppet issue or error. When this happens, I connect to the customer infrastructure, check the issue, arrange a call where I show how I can fix, or have fixed, the problem. Plain, straight to the point and usually effective. The customer pays for the time spent on it, according to the contract terms. Free support I lied before (actually this misleading AI did). The AI wrote that services are not free, but that’s not always the case. Via example42’s website you can contact me directly via WhatApp, and ask me any Puppet related question. As soon as possible, I reply, if I’ve a ready answer I just give it, if the issue requires more investigation details are asked. Then if the effort to diagnose and solve the problem seems to require more than a few minutes I ask to consider one of the paid support services we provide. In a case I actually happened to spend a few hours, including a shared screen call, to give some advice. It generally happens with individuals from countries where the cost of life is much lower than european one, in such cases I really don’t feel comfortable requesting hourly rates which might cost a significant part of an average salary. So, don’t be shy, if you have any Puppet related issue or something to clarify, just contact me via WhatsApp, I’ll be happy to help you. Money talks can happen later, if you want. Methodology What kind of professional services may you expect from an unknown person you contact out of nothing with whom you have to share in a few minutes the burden and complexity of your infrastructure? How can you trust him (me)? How can you share your infrastructure management with someone who doesn’t know anything about it? How can you share sensitive information with someone you don’t know, without a NDA or a contract? Well, first of all, I can sign all the NDA you want, I can sign a contract, I can sign a blood pact with you, if you want. But I don’t think that’s necessary. (This one comes from Copilot! 🤖). I don’t have problems in adapting to every procedure you may require, even I prefer to go straight to the point and face the technically challenges of the problem, rather than wasting time in signing papers or preparing slides on what has to be done. For the rest I guess you have to trust my integrity, experience and knowledge. I’ve done Puppet works for companies and entities like Bank of Italy, Vatican City State, Rocket Internet, Foodpanda, Volunia, Bsource / Avaloq, Cineca, Deutsche Telekom, Boeringer Ingelheim, Bundesnotarkammer, Strato, Kuwait Petroleum, University of Bologna, Politecnico di Milano, Infocert, IHS Markit, Cornell University, Porsche, Swisscom, Agility, Willis Tower Watch, Schufa, Banque Central de Luxemburg, Blackrock and several maybe less known ones. Hope I’ve left a good impression in all of them. I’ve been in IT for more than 25 years, working on operations, networking and security, mostly on Linux and Open Source software. I think I know what I’m doing when I touch configuration files. I’m also working almost exclusively on Puppet for 15 years and know how to discover what’s the potential effect of a change and how to investigate what’s its range of impact on the infrastructure. But most of all, when there’s not an ongoing contract where I’m granted access to the client’s systems, we work on a shared screen session, where customer controls what is done, I just give advice and options to the best of my knowledge, and customer decides what to do and when. I don’t have access to the infrastructure, I don’t have access to customers’ data, I don’t have access to their secrets. I just see their screen. Work and life balance Engagement for example42’s Puppet support can be done at any time and any day, from every continent. We never offer a strict SLA or guarantees on intervention times, but in most of the cases the reply and the actual start of the activity is within the day. You may wonder what hellish life I might be doing, giving this around the clock availability. My answer is that I don’t care about working outside plain working hours or in festive days, as I don’t mind about playing with my hobbies during working hours. When I work asynchronously, commits in my customer’s control-repos might be at midnight or at 6 AM GMT+1, my usual timezone (ehi, I wrote commits, not deployments to production: they are always verified and checked with the customer). When I work synchronously, typically in conference calls, sometimes onsite, I don’t need to plan them weeks before. If I can, I can connect and give support as soon as I’m notified of a request. The main reason all this does not drive me crazy, is that I like my work, I love to develop things with Puppet, and I like to share what I know about it. Another reason is that I’ve actually have plenty of free time to do other things, also because to solve a Puppet problem or develop code I usually take only a fraction of time of a DevOps engineer with generic Puppet experience (everyone has his/her skills, my ones are very vertical on Puppet but still quite broad on IT operations, security and networking). Final reason, I guess, is that I’m a flexibly and handy person who adapts to customers’ needs whenever possible. Needless to say that I request, in some occurrences, a similar flexibility from my customers. So long and thanks for all the fish Are you really reading these last words? How have you survived this unbearable sequence of embarrassing self promotion boasts? Well, what to say, if not just Congratulations and Thank You! You know for what. Alessandro Franceschi example42 Founder and Chief Puppet Consultant 2022-08-07T00:00:00+00:00 https://example42.com/blog/2022/08/07/puppet-evaluation-error-unknown-resource-type So you got the Error while evaluating a Resource Statement, Unknown resource type with Puppet and are searching for solutions? You are lucky, because right now we are going to see what it means, why it happens and how to solve it. TL;DR The module that provides the referred Unknown resource type is not available where your Puppet code is compiled. Find the module you need in the metadata.json file of the module where the code fails (refer to the provided file path). Solve by adding the dependency module to your Puppetfile, or running the puppet module install command. Decomposing the “Unknown resource type” Puppet error message [JUNIOR] Your error message may look like (we are going later to see which parts can be different for you): Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Unknown resource type: 'concat' (file: /etc/puppetlabs/code/environments/production/modules/openvpn/manifests/config.pp, line: 6, column: 5) on node lab.psick.io Here’s how to decompose it: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: This is your Puppet agent reporting an Error, saying that it was not able to get the catalog of the resources to apply from the server and got a 500 error (yes, Puppet client and server communicate over https). Next is shown the actual server error message: Server Error: Evaluation Error: Error while evaluating a Resource Statement Unknown resource type: 'concat' This is the specific message, where instead of ‘concat’ there can be any Puppet resource, and gives us the key information for solving it: Puppet is looking for a resource type, here ‘concat’, and it doesn’t find it. (file: /etc/puppetlabs/code/environments/production/modules/openvpn/manifests/config.pp, line: 6, column: 5) Another key information: where Puppet failed to compile our code. In a normal client-server infrastructure, the file path is on the Puppet server’s filesystem (path may be different, what matters is that this is the file to check). If there’s no server involved and you use the puppet apply command, then the path is on your local filesystem. on node lab.psick.io This is your client node where Puppet agent was run. A few Puppet basic principles. [BEGINNER] Puppet manages any kind of computer related item, it uses its own language, written in files like the /etc/puppetlabs/code/environments/production/modules/openvpn/manifests/config.pp of the above example, where we can declare what are the resources we want to manage on our system(s) (lab.psick.io). Puppet is shipped with some core resource types like ‘package’, ‘service’, ‘file’, ‘user’, etc. We use them to manage, guess what, packages, services, files, and users on Operating Systems like Windows, Linux, MacOS. For example code as follows automates the installation of the nginx package and manages its service on any node where is used: package { 'nginx': ensure => present, } service { 'nginx': ensure => running, enable => true, } Puppet is used typically to automate the management of configurations on multiple servers or desktop systems, but can be used to manage virtually anything which is IT related: network devices, cloud resources, etc. This is possible thanks to Puppet’s extensible and modular structure. Additional resource types can therefore be found in Puppet modules which are shared on sites like GitHub and on a public repository called the Forge. Whatever its name, in Puppet language, we always declare a resource type with the following syntax: resource_type { 'title': parameters, } Our problem here is that in line 6 column 5 in the /etc/puppetlabs/code/environments/production/modules/openvpn/manifests/config.pp file on our server, we have a resource declaration like concat { '/etc/default/openvpn': owner => root, group => 0, mode => '0644', warn => true, } that uses the resource ‘concat’ which is not available, because it’s not a native resource type shipped with Puppet itself, and we don’t have the module that provides it. How to solve Unknown resource type errors [JUNIOR] The quick answer is to install the module that contains the resource type you are trying to use, and the quick way to find it is to look at the dependencies of the module which is using the missing resource. Puppet modules can require other modules, in this case the ‘openvpn’ module requires an additional module which provides the ‘concat’ resource. The dependencies of every module are defined in the metadata.json file, at the root of the module directory, and indeed in our /etc/puppetlabs/code/environments/production/modules/openvpn/metadata.json we have the following: { "name": "puppet-openvpn", "version": "10.2.1", "author": "Vox Pupuli", [...] "dependencies": [ { "name": "puppetlabs/concat", "version_requirement": ">= 4.1.0 < 8.0.0" }, { "name": "puppetlabs/stdlib", "version_requirement": ">= 4.25.0 < 9.0.0" } ] } The important information here is: we are using the module puppet/openvpn (Here ‘puppet’ is the user name on the Forge of the Vox Pupuli community of modules authors) This module depends on 2 additional modules: puppetlabs/concat and puppetlabs/stdlib, they are both from the ‘puppetlabs’ Forge user, which is the Puppet company itself) So, in order to use the puppet/openvpn module we also need the puppetlabs/concat and puppetlabs/stdlib modules. Some info about where Puppet code lives [JUNIOR] In our example the missing resource is on the file /etc/puppetlabs/code/environments/production/modules/openvpn/manifests/config.pp, this whole path has a meaning and you need to know it if you work with Puppet: /etc/puppetlabs/code/environments/ is the default value for the $environmentpath configuration entry. Where the different Puppet environments are stored. production is the name of a Puppet environment, the default value modules is the default directory where an environment stores its modules openvpn is the name of the module that contains the config.pp file manifests is the directory in every module, where we place our Puppet code in manifests: files with .pp extension written in Puppet Domain Specific Language (DSL). config.pp is the manifest with Puppet code where we declare the resource which failed. So the metadata.json with info about the dependency modules is to searched at the root of our openvpn module: /etc/puppetlabs/code/environments/production/modules/openvpn. The modules listed under “dependencies” in the metadata.json file are what we need and we need them in the right place, more precisely in the Puppet’s $modulepath : a configuration entry which displays a colon-separated list of directories where Puppet searches for modules. Oh, incidentally, you can show all Puppet’s configuration entries with the command: puppet config print all and the specific one that matters here, the modulepath, where Puppet looks for modules: $ sudo puppet config print modulepath /etc/puppetlabs/code/environments/production/modules:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules So all we need to do is to install the missing modules. Installing modules [JUNIOR] We can install additional modules in various ways which depends on how is managed Puppet code on our systems. Using the Puppetfile If we are using a Puppet server, it’s likely and advisable that in your company you are managing the full content of the /etc/puppetlabs/code/environments/ dir with r10k (if using Puppet Open Source) or Code Manager (in Puppet Enterprise), so you should never manually touch any file there: a deployment procedure, eventually driven by a CI/CD tool, will do it for you. In this case any new external module should be listed in your control-repo’s Puppetfile. The control repo is a single git repository which contains all the Puppet code and data we need to manage the whole infrastructure. Every node uses a Puppet environment (default is called production), which has the contents of the control repo. When it is deployed by tools like r10k or CodeManager, commonly used in Puppet world, for each branch of the control repo a Puppet environment is created in the $environmentpath. When an environment is deployed two things happen: The content of the control-repo relevant branch (for example production) is copied/synced to its /etc/puppetlabs/code/environments/production directory. The modules listed in the control-repo’s Puppetfile are deployed under the modules subdir of the environment/branch: /etc/puppetlabs/code/environments/production/modules. In the Puppetfile our missing modules can be added as follows: mod 'puppetlabs/concat', '7.2.0' mod 'puppetlabs/stdlib', '8.4.0' The version names, and the actual syntax can be seen on the module’s page on the Forge. Using the puppet module install command If you are using Puppet in apply mode, without any Puppet server involved, or (blames on you!) you manage your manifests directly, on your servers’s /etc/puppetlabs/code/environments/ dir, you can install the needed modules by using the Puppet module command. It automatically installs the latest version of the defined modules (using the format: forge_user/module_name or forge_user-module_name) and their eventual dependencies: $ sudo puppet module install puppetlabs/concat $ sudo puppet module install puppetlabs/stdlib To list the installed modules use: $ sudo puppet module list Note than while the puppet module install command automatically installs every dependency, you have to specify them all in the Puppetfile. Common cases of Unknown resource type errors and remedies [INTERMEDIATE] So, I hope it’s clear that any kind of Puppet Unknown resource type error can be solved by adding the missing module which provides the resource we are trying to use. The best approach is just to check the dependencies in the metadata.json file as just described, but, if you are copying random code from the Internet or working with legacy modules without the metadata.json file to check, you may find useful the reference here. As a reference, we list here some common resource types and the relevant modules which provide them, with an extra bit of information which is worth knowing. Resource Type Module concat puppetlabs/concat archive puppet/archive line_line puppetlabs/stdlib anchor puppetlabs/stdlib vcsrepo puppetlabs/vcsrepo ini_setting puppetlabs/inifile ini_subsetting puppetlabs/inifile firewall puppetlabs/firewall firewallchain puppetlabs/firewallchain In Puppet 6.0 some built-in resource types have been moved to separated “core” modules. Normally you can ignore them as they are included in the puppet-agent package, still if you have a Unknown resource type error with any of the following resources, here are the relevant modules which provide them: Resource Type Module mount puppetlabs/mount_core augeas puppetlabs/augeas_core zfs puppetlabs/zfs_core zpool puppetlabs/zfs_core yumrepo puppetlabs/yumrepo_core host puppetlabs/host_core selboolean puppetlabs/selinux_core selmodule puppetlabs/selinux_core zone puppetlabs/zone_core cron puppetlabs/cron_core scheduled_task puppetlabs/scheduled_task sshkeys puppetlabs/sshkeys_core mailalias puppetlabs/mailalias_core maillist puppetlabs/maillist_core nagios_* puppetlabs/nagios_core The puppetlabs/nagios_core module provides the following, previously built in, types: nagios_command, nagios_contact, nagios_contactgroup, nagios_host, nagios_hostdependency, nagios_hostescalation, nagios_hostextinfo, nagios_hostgroup, nagios_service, nagios_servicedependency, nagios_serviceescalation, nagios_serviceextinfo, nagios_servicegroup and nagios_timeperiod. There is a set of modules which implement common types using the Augeas tool. They are the so called Augeas Providers: Resource Type Module pam herculesteam/augeasproviders_pam shellvar herculesteam/augeasproviders_shellvar ssh_config herculesteam/augeasproviders_ssh sshd_config herculesteam/augeasproviders_ssh sshd_config_subsystem herculesteam/augeasproviders_ssh sshd_config_match herculesteam/augeasproviders_ssh sysctl herculesteam/augeasproviders_sysctl kernel_parameter herculesteam/augeasproviders_grub grub_config herculesteam/augeasproviders_grub grub_menuentry herculesteam/augeasproviders_grub grub_user herculesteam/augeasproviders_grub mounttab herculesteam/augeasproviders_mounttab pg_hba herculesteam/augeasproviders_postgresql syslog herculesteam/augeasproviders_syslog syslog_filter herculesteam/augeasproviders_syslog syslog herculesteam/augeasproviders_syslog puppet_auth herculesteam/augeasproviders_puppet nrpe_command herculesteam/augeasproviders_nagios apache_directive herculesteam/augeasproviders_apache apache_setenv herculesteam/augeasproviders_apache If you use Puppet to manage Windows, you might need these resources and modules: Resource Type Module registry_key puppetlabs/registry registry_value puppetlabs/registry reboot puppetlabs/reboot dsc puppetlabs/dsc_lite acl puppetlabs/acl chocolateysource puppetlabs/chocolatey chocolateyfeature puppetlabs/chocolatey As a general reference, if the missing resource type has a double colon in its name, (like: apache::vhost), then the name of the module HAS to be the first part before the double quotes (apache). Next problem would be to find the right apache module from the Forge, and that can take some time if it’s not referenced in the metadata.json file. If you bumped into code which is using example42’s revolutionary Puppet modules, be aware of the following: Resource Type Module tp::install example42/tp tp::conf example42/tp tp::dir example42/tp tp::repo example42/tp tp::test example42/tp tp::info example42/tp psick::* example42/psick (*) Any defined resource type (also called define) from the psick module, there are quite a few: psick::puppet::access, psick::puppet::module, psick::puppet::set_external_fact, psick::netinstall, psick::yum::repo, psick::yum::plugin, psick::rclocal::script, psick::profile::script, psick::sudo::directive, psick::network::route, psick::network::routing_table, psick::network::set_lo_ip, psick::network::interface, psick::network::rule, psick::network::validate_gw, psick::network::netplan, psick::network::netplan::interface, psick::bolt::project, psick::limits::limit, psick::limits::config, psick::nfs::export, psick::nfs::mount, psick::openssh::keypair, psick::openssh::keyscan, psick::openssh::keygen, psick::openssh::config, psick::systemd::unit_file, psick::java::install_tarball, psick::services::init_script, psick::services::systemd_script, psick::archive, psick::users::managed, psick::aws::cli::script, psick::git::config, psick::php::module, psick::php::pear::module, psick::php::pear::config, psick::sysctl::set, psick::chruby::gem, psick::tools::create_dir, psick::tools::gpgkey, psick::kmod::module. Conclusions Hope this has been useful, thanks for reading. Let me know if you like similar posts on Puppet errors, I might do more of them. If you need quick support for any Puppet related error or problem, Smart Puppet Support or the contact link in the top right will get you there: if the solution is quick (as it would be the one for an error like this), it’ll be without any cost or obligation. If it’s Puppet related, I know I can help, and I’m glad to. Alessandro Franceschi Twitter: @alvagante GitHub: @alvagante 2022-06-22T00:00:00+00:00 https://example42.com/blog/2022/06/22/hashiconf- a-vibes-from-a-perfect-conference IT conferences are main attractions in our jobs with machines, the occasions to meet again remote friends you’ve known for years, know people with whom you have something in common and stay updated on vendors’ products. After the last two years I guess many of us are looking forward to attend some good in presence conference, hence my decision to plan a road trip, from Italy to Amsterdam, take the occasion to meet business partners, old friends and, first time for me, jump into an HashiCorp conference. How tp always matters I didn’t came to the party without anything, I took the occasion to demonstrate how easily you can install any other Hashcorp application (as actually, every other application) with Tiny Puppet: “Example42’s coolest and most underrated module ever”. My opinion. Quick recap, if you have Puppet installed, even without any agent runner, you can install on your local OS (RedHat/Debian/Suse/MacOs/Windows…) the Tiny Puppet module with: puppet module install example42/tp Then you can setup Tiny Puppet (practically creating the local tp cli command) with: puppet tp setup Now you have the tp cli command which can be used for different tasks. You can install literally every application, so also the HashiCorp tools, with: tp install terraform tp install vault tp install packer tp install vagrant tp install consul tp install nomad ... You can test if the above apps are correctly installed with: tp test You can tail continuously all the system logs of the ones of a specific app with (CTRL+C to exit): tp log tp log consul You can collect live info on the system and single apps with: tp info tp info packer This is possible on every common OS, you don’t have to remember names or paths. Tiny Puppet, and Puppet under the hoods, takes care of everything (as long as there’s tinydata for a specific application). That’s only the surface of what you can do with Tiny Puppet, which is not only a cli command but also a Puppet module you can use in your profiles to manage quickly and in a cross OS compliant way the applications you need to configure. It also provides tasks, that perform the above actions remotely. A tp::test task run globally gives you instant health stats pf all your apps on all your systems, for example. Vibes of a conference Enough about tp, let’s dwell into HashiCrop Europe 2022, held in Amsterdam, in the same lovely location of earlier editions. Here we found an almost unique combination of different factors: An Open Source based company with a cool lineup of products A great conference location, near a gorgeous park and not far from center Perfect weather, sunny but not hot. Intriguing ambiance and background music Perfect organization and plenty of food and beverages That Amsterdam special feeling I guess words don’t make much sense or add a lot, so I hope the following videos can given an idea of the general location and environment. This is the main room at the beginning of the conference: A walk around during Day 2 Keynote, spaces organisation was good, with this main room used only for keynote all the resting, demo, food areas. Hope that these videos, better than many words, can give you an idea of how HashiConf Europe 2022 has been, which I guess is better than nothing. Alessandro Franceschi 2022-05-23T00:00:00+00:00 https://example42.com/blog/2022/05/23/psick-version-one-coming-soon Psick (Puppet Systems Infrastructure Construction Kit) is a project that aims to provide a top of the notch Puppet infrastructure in a quick and easy way. It’s composed by: The Psick Control-repo featuring useful features like Vagrant and CI integrations The Psick Module with a set of reusable profiles for the most common sysadmin activities Version 1 of the psick module, after years of lazy developments and a recent and abundant code-rush, is going to be released soon, and there are really a lot of new things which are worth some explanations. This is the reason of this post. What the psick module does? The psick module has some unique and diverse features which can be optionally chosen. Classification The first one is classification, you can use it to manage what classes to assign to your nodes and it does it entirely in a (Hiera) data driven way. Something like this: psick::base::linux_classes: ssh: psick::openssh sudo: psick::sudo This feature has been present since the early days and there are no changes with version 1, which is therefore completely backward compatible. Base profiles The second feature is a presence of several reusable profiles to manage common resources like users, ssh keys, limits, kmod, logs, time, nfs, mounts, sudo, sysctl, repos, openssh, networking, resolver and the setup of languages like php, ruby and python. Also for this common baseline profiles there are no significant backwards incompatibilities with version 1, which just introduces new profiles like: psick::network, to manage networking, based on the works on version 4 of example42-network module (which is now deprecated) psick::rclocal, to manage rc.local also in different files, based on example42-rclocal module (now moved to voxpupuli) psick::systemd for basic systemd management psick::kmod for basic kmod management Applications profiles: psick_profile module The third feature of psick module was the presence of several application specific profiles. This is where most of the changes will happen with version 1. Short story: this feature doesn’t exist any more in psick module. Instead if has been implemented in the new psick_profile module. Longer stroy, the following happens with psick 1.0: All/most the applications profiles are moved from psick module to the new psick_profile module The tp_profile module is now deprecated, its classes (all with the same structure and content) have been moved to tp classes in psick_profile. The new psick_profile module basically contains profiles you can cherry pick to manage applications like Grafana, Icinga2, Jenkins, Keepalived, Mongo, OpenSwan, OpenVpn, Oracle (prerequisites), Prometheus, VirtualBox, Sensu, Ansible, CheckMK, Docker, Foreman, InfluxDB, MariaDB, MySql, Nagios, Newrelic, NRPE, Postfix, PostgreSQL, Redis RabbitMQ… All of the backwards incompatibilities are actually due to changes in these classes, and their move from psick and tp_profile modules. In order to cope with them, a migration doc is provided. In short: just the class names change, not their behavior. for the classes you use of the psick module, you have to rename the relevant Hiera keys. Relevant code changes are in these pull requests: PR for Psick Version 1 Module psick_profile (changes have already been merged into this module) PR for the sample psick Hieradata which gives a good idea of the Hiera keys to rename Antipatterns in a box or just a different approach? Tiny Puppet and Psick might be considered a concentration of anti-patterns: single modules that do too many things, which is exactly, according to all best practices, what a module shouldn’t do. My stance here is this: I understand and agree with the risks of such an approach, I’ve pondered elements, and, from my perspective of Puppet modules developer and user, I prefer to have a single module that tackles, in a coherent and comprehensive way, several common configurations that we generally add to our base profiles, rather than selecting, adapting, and chasing the dependencies of several different other component modules. You have benefits both in terms of performance, integration, speed of implementation and ease of use. Also, all the features of psick are optional and you can and should cherry pick them. All you need to do is to include the psick module, which by default, does nothing, and then configure via Hiera what you want to do: the whole classification process, if you want, or just the data for the single profiles you have classified. Like this, for the psick::sudo class: psick::sudo::directives: al: content: 'al ALL=(ALL) NOPASSWD:ALL' example42 “stack” I have done Puppet modules for years, I love the modularity and the possibility to extend Puppet. I written hundreds of different modules for different applications. My search for optimization has driven me to develop Tiny Puppet to manage applications in a generic way, and then the psick control-repo where to have the ideal development environment, and the psick module, with all its profiles to manage common system resources and applications in more complete ways, using Tp as backend. It has became clear that these open edged modules can grow uncontrolled, and that’s actually what I hope, in some way: having a growing number of application profiles and tiny data, which can work for most of the use cases, and are fully configurable to adapt to special needs. Always knowing that you can cherry pick them and there will be on the Forge a better module to handle the same application. The example42 stack of modules is now composed as follows, in Puppetfile format: # Tiny Puppet - The general purpose universal installer, and its Tiny data mod 'example42/tp' mod 'example42/tinydata' # psick module - The [recommended] infrastructure module to manage most of the common OS resources mod 'example42/psick' # psick_profile module - The [optional] psick addendum with profiles to manage different applications mod 'example42/psick_profile' plus the usual and eventual other dependencies for specific profiles. Four modules, or just two if you need only Tiny Puppet, not too many plugins to sync, nothing done by default, but several options available a Hiera data key away. If you are using Puppet on a medium, large scale, you know how big Puppetfiles can become (and how this can affect deployments) I’m available to explain in chats or live development sessions, for free, how to better use these modules. Alessandro Franceschi 2022-04-13T00:00:00+00:00 https://example42.com/blog/2022/04/13/puppet-is-acquired-and-example42-goes-back-to-the-roots On April 11th, 2022 Puppet’s CEO Yvonne Wassenaar announced the acquisition from Perforce. Interesting and, for me, unexpected news. Only time will tell us how much this is going to impact Puppet’s product and the nature of its community. Do we still need Puppet? In these times where serverless, containers, and cloud based services are trending tech friends often ask me (given my barely concealed addiction for it) what’s the future and the role of a tool like Puppet. It was born to solve problems of the past, when people had to configure their own servers, rather than relying on some cloud managed service, an hosted Kubernetes cluster or a bunch of “serverless” functions. My answer is often on these lines: How many systems do you have to manage? What’s their average lifetime? If they are more than a dozen and they “live” more than a few weeks, you NEED a tool like Puppet. That’s it. Plain and simple. In a modern IT shop, especially when its scale is not minimal, you have to manage everything as code: versionable, reproducible, repeatable. What kind of code do you need to (re)build your infrastructure? Whatever the Yaml, HCL, Json files that allow you to describe and configure your IT world with whatever tool, you need something to put under version control. You need something that allows you to reproduce and possibly automate your setups and the configurations. Manual operations, eventually based on an hardly documented checklist of point and click operations or commands, are barely acceptable. To some degree, the essential disaster recovery of a modern IT shop is a git repository from where you can rebuild everything. So, back to the key question: what’s the lifetime of the systems you have to manage? I mean, all systems, not only the servers which provide your company’s internal or public services (be they on-premise physical and virtual or cloud instances). If you don’t have own servers to deal with, do you have to install, configure and manage your client desktops? How do you configure the development stations from where you manage your cloud services? How do you ensure that security settings and selected programs are installed on the employees’ laptops? If anywhere in your company, there’s a Linux, a Windows, or a MacOS system that lives more than a few weeks, then you need a configuration management tool. And among the various ones around, trust my biased opinion, Puppet is still the most complete, advanced, and scalable one. It’s also one of the hardest to learn, but for this you can rely on the advice old puppeteers like me and the plenty of documentation around. Getting personal I started to use Puppet around June 2008. I had just begun to work in the national Bank of Italy, as freelance System Administrator, when a colleague came back from an IT conference with exciting infos on this new tool which could help us configuring the hundreds of servers we had to manage. We were already doing some kind of configuration management, using a somehow genial but cumbersome method based on configuration rpm packages. I remember my initial reluctance in replacing the existing approach, which I just had started to master, with this new tool. The typical sysadmin resistance to what’s new and may disrupt his/her dream of stable and untouchable systems and established habits. We started with version 0.21, if my memories don’t fail me, and as soon as I began to understand its principles, I could not prevent myself from loving it. A year later I was already publishing my first set of Puppet modules (yes, that’s embarrassing code now), following the steps of David Schmitt’s modules collection, with the ambition of doing modules with a standard naming structure, multi OS support and integrated monitoring, auditing, documentation and firewalling. At the times I called this bunch of Puppet code “Lab42 Puppet Infrastructure”. The Forge was still far from being even an idea. The first PuppetCamp In 2009 I also attended the first PuppetCamp ever, held in San Francisco. It has been a blast. I thought to be a Puppet expert, I found out I had so many things to learn and fields to explore. It has been by far the most useful and enlightening IT conference I’ve ever been at. I’ve vivid and wonderful memories of that event, from the visit at Google’s headquarters, thanks to Nigel Kersten, who was working there at the times: To the tours around San Francisco with new friends and Puppet legends like Ohad Levy (he presented The Foreman there), Brice Figureau (the wizard behind exported resources, puppet device and a lot of other community code) and Dan Bode (he has been one of the first employees hired by Luke right before the event): Since those times I had this obsession about modules standards which never really faded, here’s me suggesting a round table about the topic: example42 modules After that first event I attended almost every major Puppet conference, actually all the times I’ve been in USA has been for a Puppet event. I owe to Luke Kanies and Puppet a good half of my IT career: since those first years I never stopped to work with Puppet, writing modules, delivering training and consulting all over the world, either for direct customers or as Puppet Partner. I passed 10 years traveling almost every week, taking an airplane to flight to destinations in Europe, and occasionally Middle East and even Australia and Singapore. The Lab42 Puppet infrastructure was renamed to example42, under this name over the years I developed various Puppet related projects: Puppi a module which integrates with the first generation of example42 Puppet modules (featuring standard naming, saner MultiOS support and decommissioning of resources) to provide the ability to test from the command line what a module was managing and provide defines and commands to manage applications deployments The example42 “NextGen” modules set, introduced in 2012, with the params_lookup function which somehow anticipates the concept of classes’ automatic parameters lookup Tiny Puppet, a single module you can use to install and configure virtually every application on every OS. I like to consider it the most underrated Puppet module ever, but, again, I’m definitively biased here. A full featured control-repo (when the control-repo concept was introduced) which was then renamed to PSICK (Puppet Systems Infrastructure Construction Kit, any reference to SEUCK is not casual) The psick module a collection of reusable profiles and defines for common uses. In many of these projects I break many official best practice rules and recommendations. I can explain. I’ve my reasons for many of such controversial choices and I’d do most of them again. I developed most of example42 code by myself, but I don’t forget and I am grateful for the PRs and fixes from other contributors. It was nice at the times to learn about people and companies using my modules, and funny to see persons surprised to learn that example42 was just a person and not an organization with various developers. example42 GmbH Actually, in 2015, example42 from a one man band became a company: example42 GmbH registered in Berlin, in collaboration with Martin Alfke. We were both experienced Puppet freelance consultants and we decided to join forces. I wanted to preserve the example42 name, since it was known, for the good or the bad, in Puppet world, Martin had no objections to that. My developments in Puppet code were reduced to simple maintenance activities on the existing projects, sacrificed on the altar of profit with the excuse of lack of time. Our business has grown well over the years, we worked with some of the biggest Puppet customers in Europe but even if our relationship was fine, the company has always had a dual nature, with the two of us working somehow in parallel without mutual interferences but also without a real common goals and direction. Last year we decided to close the example42 GmbH company, and follow our own paths, as you can read from last blog post. And now? So here we come to 2022, with the long tail of the pandemic and the terrible war news. As for many others out there, I lived the last 2 years in a sort of suspended state, completely changing my habits (from one travel a week to endless hours in front of my home computers) and reconsidering priorities and life goals. The example42 name and online assets have come back to my full control, now I deliver all my Puppet related activities via my company in Italy, Lab42 Srl, keeping the example42 brand. I brought to the extreme the flexibility I always grant to the customers I work with, some of the services I offer are really unique for the ways you can access, activate and use them: do you have a problem or something to do with Puppet? Send me a message, let’s setup a conference call with shared screen and I can explain, guide, troubleshoot and work with you on your own Puppet setup. Trust me, in most of the cases I can solve, or show how to solve, Puppet problems in a few minutes. I like my work, I don’t have working hours or days: I can pass full mid-week days playing video games or watching videos and nights or Sundays working on Puppet code or doing conference calls with people on other continents. I’m not writing a lot of public code lately, and I’m not happy of that. I also would like to write some updated documentation and hints on Puppet (in the past I wrote a book on Puppet which was well received, but now it’s definitely obsolete and there’s so much to talk about newer Puppet topics and patterns). Still it has been 3 months I’m not doing anything of the above, working only for a fraction of my available time, without regrets. Suspended, as I said, reconsidering the meaning of Life, the Universe and Everything and planning unlikely road trips across Europe. I guess you can relate. The news about the acquisition has left me with mixed thoughts: I don’t know if this will mean the end of Puppet as we know it or the beginning of a new shining era. Maybe both. At least it has achieved one thing for me: it has enticed me to write this new blog post, the first one after closing the German company. Given the times, it might be a beginning, or just another false start. Alessandro Franceschi 2021-12-20T00:00:00+00:00 https://example42.com/blog/2021/12/20/example42-gmbh-closes-business-on-31st-of-december-2021 example42 GmbH was founded back in 2015 and registered at chamber of commerce during PuppetConf 2015. We have worked in Puppet world for many years, have supported hundreds of companies, and delivered training worldwide. We were self-employed professionals and decided that it was good to unite the forces and develop a company. That’s how example42 Gmbh started, registered in Berlin, owned by the most prominent Puppet experts in Italy and Germany, and committed to providing Puppet support to customers via the virtuous partnership with Puppet Inc. or directly. It worked well, constantly cash flow positive, with slow but steady growth, based initially on the work of the two of us, supported by Jessica. The balance worked; while delivering our consulting and training works, Martin was more dedicated to business and commercial growth and Alessandro on example42 open source projects, which he started to develop in 2008. Still, it was clear that the company was a two owners, two minds, two faces project. We never really managed to deliver it as a whole entity, with shared focus and targets, even if we tried. Over the years, we added people to the technical team: first Simon, then Robert, and Tim this year. They are very talented people we are honored to have on board. The company growth has somehow been also the reason why the double-side nature latent issue started to become a real issue. We had different views on how to continue. Initially, we tried, without success, to sell the company, and then we ultimately decided that it was better to close the shared experience and keep on working in our own ways. This “shared experience” is a company with duties and responsibilities, so we wanted to minimize the impact on any involved individual and entity. The German team is going to be hired by Martin’s new company, betadots GmbH, in order to give continuity to their works on customers. Both Martin, via betadots GmbH, and Alessandro, via his company in Italy Lab42 Srl, will continue their partnership with Puppet Inc, guaranteeing a transparent transition for final customers. The example42 name, domain, website, social media accounts will be transferred back to Alessandro and become a brand of Lab42 Srl. So that’s it. On Dec 31st example42 GmbH closes its business activities. The company will deliver no new activities, and within a year, is going to do all the necessary procedures to close definitively. So here’s how a company closes, not for commercial or financial issues, but for divergent views on what and how the same company should be. We are ok with that, we have no hard feelings, we are going to continue our activities in Puppet and the IT world, and we wish each other and everybody all the best for your future. Take care. The future is ahead and is going to be shaped by us. Alessandro and Martin 2021-02-11T00:00:00+00:00 https://example42.com/blog/2021/02/11/hiera-merge-behavior Hiera v5 allows you to provide information regarding merge behavior in a couple of different ways. Using different merge behaviors allows you to e.g. separate common (admin) users from application users by adding common users in common hiera layer and the application users into an application hiera layer. This reduces duplicates in data and allow more simple data management. Another example is installation of packages. You usually have a list of packages you need on all systems (admin packages) and some packages which are needed on special systems only. Again you can mention all common packages in hiera common laxer and add node specific packages in node hiera layer. In this posting we explain the different possible merge behaviors and their results. Additionally we explain the options where you can set merge behavior. Hiera merge behavior options and the results first unique hash deep Merge behavior on explizit lookup Merge behavior configuration within hiera data Hiera merge behavior options and the results We will first have a look at all available merge options and explain the behavior and the result later in the explicit and automatic data explanations. Hiera offers the following merge options: first unique hash and deep All data examples assume that you have a hiera.yaml file using four hierarchies: node specific data application-stage specific data application specific data common data Example for hiera config: version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: "hiera hierarchies" paths: - "nodes/%{trusted.certname}.yaml" - "application/%{::application}-%{::stage}.yaml" - "application/%{::application}.yaml" - "common.yaml" first By default if no option is provided, hiera uses the first merge option. Using first is not really a merge option as hiera will return the very first result of a key. The data type of the result depends on the data type of hiera data. e.g. if hiera finds a string, it will return a string, if it finds an array, it will return an array. unique The unique merge option allows you to collect data from multiple hierarchies and returns the result as an array. All elements in all matching hierarchies must be of data type array. Now let’s have a look at the hiera data: In common you have a packages key: # data/common.yaml packages: - vim-enhanced - curl In application level you also have a packages key: # data/application/mysql.yaml packages: - xtrabackup The returned result for a packages key, using unique merge strategy will return the following values: packages: - vim-enhanced - curl - xtrabackup Visualization: Result mysql common packages: packages: packages: ..- vim-enhanced   ..- vim-enhanced ..- curl   ..- curl ..- xtrabackup ..- xtrabackup   hash The hash merge option parses all matching hierarchies and returns a list of hashes. All elements in all matching hierarchies must be of type hash. Let’s look at the data. In this case we manage users: # data/common.yaml users: martin: uid: 10012 home: /mnt/home/martin shell: /bin/bash alessandro: uid: 10011 home: /mnt/home/al shell: /bin/zsh In application level, we also have users: # data/application/mysql.yaml users: simon: uid: 10013 home: /home/simon martin: # mysql is different that common uid: 10012 home: /home/martin The hash key from highest hierarchy is taken first. other keys from lower hierarchies are just added. If a hash key exists in several hierarchies, the one from the highest hierarchy is taken: # result users: simon: uid: 10013 home: /home/simon martin: # mysql is different that common uid: 10012 home: /home/martin shell: /bin/bash alessandro: uid: 10011 home: /mnt/home/al shell: /bin/zsh Visualization: Result mysql common users: users: users: ..simon: ..simon:   ….uid: 10013 ….uid: 10013   ….home: /home/simon ….home: /home/simon   ..martin: ..martin: ..martin: ….uid: 10012 ….uid: 10012 ….uid: 10012 ….home: /home/martin ….home: /home/martin ….home: /mnt/home/martin ..alessandro:   ..alessandro: ….uid: 10011   ….uid: 10011 ….home: /home/alessandro   ….home: /mnt/home/alessandro deep The deep merge option is a special behavior of the hash merge option. Hash uses the first hash ke from highest hierarchy. Using deep allows you to merge data from hashes with the same key. All elements in all matching hierarchies must be of type hash. Let’s look at the data. In this case we manage users: # data/common.yaml users: martin: uid: 10012 home: /mnt/home/martin shell: /bin/bash alessandro: uid: 10011 home: /mnt/home/al shell: /bin/zsh In application level, we also have users: # data/application/mysql.yaml users: simon: uid: 10013 home: /home/simon martin: # mysql and shell is different that common home: /home/martin shell: /bin/zsh The hash key from highest hierarchy is taken first. other keys from lower hierarchies are just added. If a hash key exists in several hierarchies, the one from the highest hierarchy is taken: # result users: simon: uid: 10013 home: /home/simon martin: # mysql is different that common uid: 10012 home: /home/martin shell: /bin/zsh alessandro: uid: 10011 home: /mnt/home/al shell: /bin/zsh Visualization: Result mysql common users: users: users: ..simon: ..simon:   ….uid: 10013 ….uid: 10013   ….home: /home/simon ….home: /home/simon   ..martin: ..martin: ..martin: ….uid: 10012   ….uid: 10012 ….home: /home/martin ….home: /home/martin ….home: /mnt/home/martin ….shell: /bin/zsh ….shell: /bin/zsh ….shell: /bin/bash ..alessandro:   ..alessandro: ….uid: 10011   ….uid: 10011 ….home: /home/alessandro   ….home: /mnt/home/alessandro ….shell: /bin/zsh   ….shell: /bin/zsh Merge behavior on explizit lookup Note: This is not my preferred option! I prefer automatic data binding! When using the lookup function one can specify the merge behavior in 2 different ways: merge parameter When using the merge parameter you must also provide the data type parameter: lookup( <NAME>, [<VALUE TYPE>], [<MERGE BEHAVIOR>], [<DEFAULT VALUE>] ) lookup('users', Hash, 'deep') parameter hash When using the parameter hash, you can skip the data type: lookup( [<NAME>], <OPTIONS HASH> ) lookup('users', { 'merge' => { 'strategy' => 'deep', }, 'value_type' => Hash}) Merge behavior configuration within hiera data When using automatic data binding (naming hiera keys according to t eh class/parameter names) one can not directly specify the merge behavior as the lookup is done automatically. But hiera offers an option to use a special key called lookup_options. Within the lookup_options key one specifies a Hash. The key of the hash is the hiera key to look for. For each key you can then specify e.g. the merge strategy and the return value data type conversion. Let’s have a look at the users with deep merge example from above. Let’s assume we have a class class users with a parameter called users. To allow automatic data fetching the key in hiera must have the name users::users: # data/common.yaml users::users: martin: uid: 10012 home: /mnt/home/martin shell: /bin/bash alessandro: uid: 10011 home: /mnt/home/al shell: /bin/zsh In application level, we also have users: # data/application/mysql.yaml users::users: simon: uid: 10013 home: /home/simon martin: # mysql and shell is different that common home: /home/martin shell: /bin/zsh Additionally we add the lookup_options key to common.yaml: lookup_options: users::users: merge: 'deep' It is up to you and your use case if you place the lookup_option into the common layer or if you even overwrite lookup_options on a higher level. Happy puppetizing and data merging, Martin Alfke 2021-01-12T00:00:00+00:00 https://example42.com/blog/2021/01/12/from-2020-to-2021 We are at the beginning of a new year, example42 wishes all customers, partners, colleagues, friends, and supporters a healthy, successful, and happy 2021. Looking back on 2020 Lookout to 2021 Looking back on 2020 2020 has been difficult for many people and organizations. Many of them have suffered due to lockdown and to local restrictions. Just consider musicians, artists and their crews who had to cancel event after event. Or think about local small business which had to close their shops. Others were able to continue their business and we are super happy that our customers enabled example42 to keep working. It was astonishing to see how fast remote work options were created and how people adapted to a new way of working. In the first few weeks of remote work we still saw people having difficulties with video conferences and how to communicate with many people attending. On many occasions, we saw that a DevOps mindset, attitude, and behavior was super helpful. People started listening, asking questions, pushing themselves to make a meeting happen within the scheduled time. This showed us that DevOps is one of the keys to success in IT. Without the DevOps culture and its agile working methods, it would have not been possible for most organizations to enable new ways of working within a reasonable amount of time. We also saw many companies moving - some for the very first time - to Open Source applications. Especially BigBlueButton or Jitsi were carefully proofread by security. Understandably, it took the projects some time to adapt the findings. But having hardware in a rack or a cloud account is not enough to enable teams to work fast. This is where IT automation comes into place - a core field of example42’s business. Automation heavily reduces the time needed to deploy new systems or bring out changes with minimum effort. Lookout to 2021 At example42, we are fans of Puppet, Terraform, Vagrant, Vault, GitLab, and many other tools which help to make IT a smarter place to work. We appreciate all the support and feedback our customers provided to us within one of the most difficult years in modern times. But support is something that any one of us can do easily. This does not mean spending a huge amount of money, but instead showing respect and thinking twice prior to making decisions. Next time you are thinking about a spending: please think carefully who you want to support. Do you want to go to a big company, run by unknown managers? Or will you go next door to a shop nearby? Please consider supporting local, small business, these are the ones that most suffered in 2020 and these are the ones that make cities a lively place. example42 wishes everybody a healthy, successful, and happy new year. Please stay safe and take care of those who need your help. Happy puppetizing, Your team of example42: Alessandro, Jessica, Martin and Simon 2020-11-13T00:00:00+00:00 https://example42.com/blog/2020/11/13/puppetize-digital-git-workflows Puppetize Digital is coming! November 19th! example42 is super excited that we are again delivering a talk to a Puppet conference. This year Martin Alfke is talking about GIT workflows. The talk was prerecorded as Martin recovers from a dental surgery and will be broadcasted on November 19th at 1:30 PM (GMT) and at 2:30 PM (AEDT). Puppet and GIT Single long living branch - simple GIT Staging branches GIT flow Summary Puppet and GIT Working on Puppet always is fun. But working with GIT is hard for many people who are new to GIT. Single long living branch - simple GIT Within this talk I will show why a single long living branch (production) makes sense for most installations. I will show how you bring changes into your environment by using feature branches. Staging branches But how do you deal when you have separated networks? e.g. people are not allowed to do changes in production network. Changes must de done in the development network. In this case you need GIT servers in each network zone. How do you now stage your code from development to production network? GIT flow What happens if you are in a more ITIL based environment. In this case it might be required to have several long living branches like production, testing, development. How do you now bring individual changes into production and how to backport hotfixes into development. This is the situation where you want to consider following the GIT Flow concept. Summary I will explain when and how to make use of which concept and what are the main differences and challenges. Puppet Control-Repo Simple GIT Staging Branches GIT Flow Happy puppetizing and git-ting, If you’d rather have this workflow set up on your own control-repo than pick it apart from a talk, see Puppet Infrastructure Kickstart. Martin 2020-11-09T00:00:00+00:00 https://example42.com/blog/2020/11/09/all-our-good-reasons-to-sponsor-puppetize-digital Another year, another Puppet conference, and another time we are sponsoring it. Let’s see what, how and why. Puppetize Digital Sponsors, as usual Presenting at Puppetize Digital Puppet, for fun and business Puppetize Digital Puppetize Digital will be a 24 hours online event, on November 19, sprayed on three different continents timezones: Asia Pacific: 10:00 AM - 6:00 PM AEDT Europe 9:00 AM - 5:00 PM GMT Americas 9:00AM - 5:00 PM PST In GMT timezone this means: Asia Pacific: 11:00 PM (Nov 18) - 7:00 AM Europe 9:00 AM - 5:00 PM Americas 5:00 PM - 1:00 AM (Nov 20) What can we expect from this event? Anything related to Puppet and its ecosystem: Details on recently announced products (Puppet Comply) Workshops and technical sessions on Puppet products: Puppet Enterprise, Puppet Remediate, Relay Business and DevOps content from IT leaders Last but not least, everything is going to be free! You just need to Register here. Sponsors, as usual As individuals, even before our company was born, we have attended Puppet events since the very beginning: the first PuppetCamp , in San Francisco, on 2009. What is impressive looking back at that event is that most of the participants have been hired by Puppet Inc, over the years. In the picture you see me pitching a round table session about Modules Standards, an obsession I had since the very first times. Since then we almost never missed any global Puppet event and since example42 GmbH was founded, five years ago, we have always been sponsors of the yearly major one. We are happy to sponsor the main Puppet conference also in 2020: this time is remote only, for the reasons we all know, but the feelings and the appreciation for such a great product and company, and the big expectations we have for “that moment in the year” remain the same. Presenting at Puppetize Digital Martin is going to make a presentation at this event, it’s titled “Puppet Control Repository and GIT - a comparison of simple vs Git Flow”. He will discuss how to work with Git on a Puppet control-repo and what workflows make sense when dealing with multiple environments. Check the event schedule and look our for it! For a list of the presentations we had in past Puppet (and not only) events, give a look to this page. Puppet, for fun and business We are true lovers of Puppet: we know it, we like it, we actually have fun when working with it, and, yes, we do business with it. We are passionate on sharing and spreading Puppet knowledge, we collaborate with our customers with the focus and aim to make their IT life better while using Puppet to manage their infrastructures. We do training, consulting, modules development, we helped companies in setting up a Puppet managed infrastructure, migrating their code base, managing with Puppet existing systems. We always do this with passion and love, trying to be as much flexible as possible in doing it: you don’t need to hire us for days or mid/long term engagement, we can provide support also for a single hour, in a quick and effective Remote Puppet Pair Programming session. But we are not here to sell our services, we are here to reiterate our love and support for Puppet: unless you have less than a few dozens of systems to manage or your are completely Cloud Native, you NEED a tool like Puppet, and we definitively believe that in the Configuration Management space, Puppet is one step ahead. Alessandro Franceschi 2020-11-02T00:00:00+00:00 https://example42.com/blog/2020/11/02/5-years-of-example42 example42 was founded back in 2015. It was the union of intents and purposes of two of the most experienced and known puppeteers in Europe: Martin Alfke from Germany and Alessandro Franceschi from Italy. Martin recalls with a smile the first steps of the company: “I still remember well the day where we received information of our official company registration on October 8th 2015 while being at PuppetConf 2015 in Portland, Oregon: I had a call with my girlfriend in Berlin, Germany, and she was excited that I had received a letter from the district court, a few hours prior I had to give my talk about Puppet 4. I asked her to open the letter: It contained the official company registration letter. That made us the youngest company at PuppetConf 2015, being less than 1 day old!” Trainings Customers Products People Trainings We mainly do consulting and training. A lot of training. In 5 years we trained more than 300 people on Puppet within 50 courses. We have an astonishing list of locations where we conducted trainings: Essen, Germany Berlin, Germany Munich, Germany Hamburg, Germany London, UK Rome, Italy Melbourne, Australia Singapore Abu Dhabi, UAE Customers We are humbled and impressed by the customers we had the privilege to work with: from major car brands to telcos, from financial giants to startups, from central banks to public institutions. Some of them are direct customers, some of them coming from our deep and solid collaboration with Puppet. We traveled for several hundreds of days, delivering onsite services mostly in Europe but also in the Middle East, now times are definitively different, and while we like to provide remote support worldwide, we are looking forward to flight “back to the road”. We have always tried to put at disposal and share with customers our experience and knowledge: we want them to success with the technologies we work with and we want them to be independent and autonomous on the topics for which we provide consulting and training. Still, not unsurprisingly, it happens frequently that they keep on asking us for support and services. Products Before becoming a company example42 was the name of a collection of modules developed by Alessandro. We continued to work on Puppet modules and Open Source projects in these years, and now we are mostly focused on these projects: Tiny Puppet, a Puppet module which allows to easily install and configure in the way we want “Any Application on Any Operating System”. Looks like magic, but is real. Psick, a reference control-repo and a Puppet module that provide the most common resources that Puppet usually manages in an Operating System Hiera Data Manager a web frontend to visualise and manage Hiera data. Being open source and freely downloadable we don’t have a clear idea on how much and where these products are used, but from time to time we are surprised in learning that some major companies and institutions actually do use them in their production systems (in some cases the ones so big and popular we wouldn’t even dare to think about). This is always a source of motivation and great responsibility. People We are a small company, someone might define it a consulting boutique rather than a warehouse, and actually we consider our work automation craftmanship. We started as 2, sharing the company vision, equities and responsibilities, then Jessica Zimmerman came onboard, taking care of the administration side. In 2019 Simon Hönscheid joined as Senior IT Infrastructure Automation Engineer and starting in February 2021 Robert Waffen will be with us as well. Both of them are experienced DevOps with great Puppet background. We don’t even have a sales department, as works chase us via word of mouth, direct contacts or our partnership with Puppet. We are growing steadily, without any financial help from banks or backers or any other investment or funding. Even if we work on cutting-edge IT space, under this point of view we don’t consider ourselves startuppers, but rather tailors of automation. In the future we plan to keep on growing, steadily and without hurry, and keep on learning and using the wonderful technologies that make IT more and more automated and reliable. Happy puppetizing, take care and stay safe. If you want that same training experience for your own team, see our current Puppet Training offering. Alessandro and Martin 2020-06-12T00:00:00+00:00 https://example42.com/blog/2020/06/12/using-eyaml-gpg-to-store-secrets-in-hiera There are situations when you want to store secrets like passwords, tokens or usernames in Hiera. The default way to do this is to use Hieras E(ncrypted)YAML implementation based on PKCS7. You can find out more about that at: Encrypt your secrets with Hiera eyaml Pros and cons of the two EYAML mechanisms EYAML EYAML uses a public/private keypair. The public key goes out to all users. They are able to encrypt content; then, the private key is stored somewhere safe and on the Puppetservers. This key is the only way to decrypt content. This is pretty easy to configure, but might become an issue if team members leave the company. In practical teams, users often have to have the complete keypair to be able to look things up and use the hiera files as a password database. + build in + easy to configure - might cause security issues EYAML-GPG EYAML-GPG uses public/private keypairs too, but each user and Puppetserver has its own pair. It is relatively easy to add/remove users from the allowed list of receipients. GPG is often already in use, so every user already has a keypair. + each user/server has its own keypair + add/remove users as needed + recrypt content so it is unreadable to former users, even if they take data and their keys with them - not build in - needs a bit of setup The project on Github Let’s set it up Prerequirements You need to have a running Puppetmaster 5.5 or newer and a workstation with a local Ruby which is NOT the one shipped with Puppet. We will note why this is important later. Install the nedded Gems packages There are two different GPG implementations for Ruby: Ruby_gpg is the native implementation, but AFAIK only able to decrypt content. gpgme is the full feature implementation, able to encrypt and decrypt content. However, it needs to be compiled, so it is not usable with the JRuby used by the Puppetserver. Therefore, we install the distributions Ruby. sudo /opt/puppetlabs/bin/puppetserver gem install ruby_gpg sudo /opt/puppetlabs/bin/puppetserver gem install hiera-eyaml hiera-eyaml-gpg sudo /opt/puppetlabs/puppet/bin/gem install ruby_gpg sudo opt/puppetlabs/bin/puppetserver gem install ruby_gpg sudo apt-get install ruby ruby-dev sudo gem install hiera-eyaml-gpg gpgme The software requirements are installed now. ##Generating the GPG keypairs Generating the Puppetmaster keypair gpg --full-generate-key Follow the options and DON’T protect the key with a passphrase. Generating the users cert gpg --full-generate-key Follow the options and protect the key with a passphrase. Export the keypairs check the Keypairs gpg -K sec rsa4096 2020-05-29 [SC] DD186A9AE323294BA99A124977DC5816AD58E28E uid [ ultimativ ] Puppet Server (the gpg key of the puppetserver) <puppetserver@example42.com> ssb rsa4096 2020-05-29 [E] sec rsa4096 2020-05-29 [SC] 23872B1059DD0AF63EDC8CA4E24907C2260F6FA7 uid [ ultimativ ] Puppet User (the puppet users gpg key) <puppetuser@example42.com> ssb rsa4096 2020-05-29 [E] Export the private keys gpg --export-secret-key -a "puppetuser@example42.com" > puppetuser.gpg.sec gpg --export-secret-key -a "puppetserver@example42.com" > puppetserver.gpg.sec Export the public keys gpg --export -a "puppetuser@example42.com" > puppetuser.gpg.pub gpg --export -a "puppetserver@example42.com" > puppetserver.gpg.pub Let’s get our Hiera structure ready vim hiera.yaml --- version: 5 defaults: hierarchy: - name: "my hierachy structure" lookup_key: eyaml_lookup_key options: gpg_gnupghome: /opt/puppetlabs/server/data/puppetserver/.gnupg paths: - "common.yaml" Encrypt your first secret Import keypairs First import the keys to the system. Due to the fact that this is a demo system, you also need to import the Puppetuser’s private key to a systemuser. Normally this would remain on a workstation or development server. sudo cp puppetserver.gpg.sec /opt/puppetlabs/server/data/puppetserver/key sudo chown puppet:puppet /opt/puppetlabs/server/data/puppetserver/key sudo su puppet -s /bin/bash -c '/usr/bin/gpg --import /opt/puppetlabs/server/data/puppetserver/key' gpg --import puppetuser.gpg.sec Finish by restarting the Puppetserver. systemctl restart puppetserver.service Add scrips and public keys to data directory of your puppet repository There are two scripts which make it very handy to work with Hiera EYAML-GPG: edit.sh Use this script to edit a file with encryped content or add new blocks. Change the Puppetserver Key to your needs. #!/bin/bash puppetserver_key='puppetserver@example42.com' recipient_file='gpg_recipients' if [ $# -ne 1 ]; then echo "[-] Please specify a file to edit.." exit 1 fi grep $puppetserver_key $recipient_file > /dev/null || { echo "ERROR: ${puppetserver_key} not in recipient file ${recipient_file}. This may NEVER happen!"; exit 1; } if [ ! -e $1 ]; then echo "[*] Specified file argument $1 does not exist, creating it for you..." touch $1 fi echo -e "[*] Importing new public keys..." gpg --import gpg_pubkeys/* echo -e "[*] Editing the following file: $1" echo -e "[*] Recipients are:" cat $recipient_file echo "" eyaml edit --gpg-always-trust --gpg-recipients-file $recipient_file $1 recrypt_all.sh Use this script to recrypt all files with encryped content after a team member joins or leaves. Change the Puppetserver key to fit your needs. #!/bin/bash if [ $# -eq 0 ]; then encrypted_files=`grep -Rl "ENC\[GPG" *` else encrypted_files=$* fi puppetserver_key='puppetserver@example42.com' recipient_file='gpg_recipients' grep $puppetserver_key $recipient_file > /dev/null || { echo "ERROR: ${puppetserver_key} not in recipient file ${recipient_file}. This must NEVER happen!"; exit 1; } echo -e "[*] Reencrypting the following files:\n $encrypted_files\n" echo -e "[*] Recipients are:" cat $recipient_file echo "" for item in $encrypted_files ; do echo "[*] Reencrypting $item" eyaml recrypt --gpg-always-trust --gpg-recipients-file $recipient_file $item if [ $? -eq 0 ] ; then echo -e "[+] Successfully reencrypted $item\n" else echo "[-] Reencryption of $item failed, this is bad!" echo "[-] Please investigate what went wrong and DO NOT PUSH THIS!!" exit 1 fi done echo "[+] Reencryption of all files was successful" Populate the gpg_recipients file This file contains all email adresses or key IDs of all puppetservers or team members. If a team member joins or leaves, add or remove their key. puppetserver@example42.com puppetuser@example42.com Put down GPG public keys in gpg_pubkeys directory cp puppetserver.gpg.pub gpg_pubkeys/ cp puppetuser.gpg.pub gpg_pubkeys/ If a team member joins or leaves, add or remove their key. Putting it all together Copy scrips and files to your Hiera data directory. cp -r edit.sh recrypt_all.sh gpg_recipients gpg_pubkeys/ data/ Time to encrypt your first secret The script will open your favorite editor. If this has not been defined yet, a prompt will open up to ask. ./edit.sh common.yaml [*] Importing new public keys... gpg: key 77DC5816AD58E28E: "Puppet Server (the gpg key of the puppetserver) <puppetserver@example42.com>" not changed gpg: key E24907C2260F6FA7: "Puppet User (the puppet users gpg key) <puppetuser@example42.com>" not changed gpg: Total number processed: 2 gpg: unchanged: 2 [*] Editing the following file: common.yaml [*] Recipients are: puppetserver@example42.com puppetuser@example42.com # | This is eyaml edit mode. This text (lines starting with # | at the top of # | the file) will be removed when you save and exit. # | - To edit encrypted values, change the content of the DEC(<num>)::PKCS7[]! # | block (or DEC(<num>)::GPG[]!). # | WARNING: DO NOT change the number in the parentheses. # | - To add a new encrypted value copy and paste a new block from the # | appropriate example below. Note that: # | * the text to encrypt goes in the square brackets # | * ensure you include the exclamation mark when you copy and paste # | * you must not include a number when adding a new block # | e.g. DEC::PKCS7[]! -or- DEC::GPG[]! --- super_secret_test: 'DEC::GPG[super secret string]!' Save this. The output with cat will look like this: --- super_secret_test: 'ENC[GPG,hQIMA0HLK7hFkXxnAQ//RqmeCG1vG7QVpTaaQ3NXJ7sb4/kd8PYtc9jL/P10z76KAuuid2CR1rlGczmCsDLasHGcQDLQuXpfcIOdKN1CxK3M2fJUDWsOn+oK+LK21W+0YsTHLmSUm6k/2pp36q03QlIaNcWL2BFzSs/fGskM6V57p97a3Fm27i32dGJRVyZ071G9f2lgismTK09sk50+xtIS3OCT8S4uWZkCst7TBnon3RQvfr80xKFOBYfJoo4NJob/XQi5/j00IMmF7KmrX76LZBeJV4X5PqRvOWTmlRGFT9JpDLi1fWR6hGvzSDNaL5JL6e1Wl/EsCMZgaTL7VxYwvRCvD0sAkyESD2LLMGULVT9MRO5mwhmoR1E5AMVt0FwFXwL9kQnfWS7us/TJfSPgovrZmMAav+oIZOzAv/Q1c6381urpdHtfbP17iz1jfggfDgmowcCH1UJ7R8kZ7C5RcZa/j+Uv+ll8SqbAQo/yX8mXe627OOD/WfzXP++UF7nsvdxCHpk2me+hUjo6XUWt0h2bkZKc2GinY29oojd/2FoI4EZBpTow6TgpFhw0hK2tiU1PWaAU5v6oi2BSrpIqFaZ0Fowd1fNT+86NjvArb6JY8vOWbqbh2Y6DKoTqWtZV/pxB42iXe1I59h7CsGHMtp71S37XwXDcie51EZOpEfiTmBuR4xFg7eEEiwOFAgwD+3NwktMooowBD/wNxeOWG2fH9raBl1G671JdyzNhWZq/3wyK8As7nC2P8dqK9OBT4GXfSif1ssLwvabix4C9SAiET1+JJVPPnhh3tGOCye4TUjkpdMjWNf8NXIkUXd7qwbtNNqu1TTpuSfTNpGc9cgaonuvr8SLiwGpKm57kdcuPVfrTVkeRns6h6ahTsgy4kbAXSD8b+FMApMCnJFyjEu2ne+IKfCmYbnhExj5S2qeedW0509XhVLR4cPAMQ6tefYmzRrrgm+3P3mHkNULtbrXmZWmpt0HhrVyT0axDVvuvcz+g3poByDBPsrHdhMTjkOqY6ikyS9a+H85OM9HECxQD/X1oseMrnaoCr9Ds+In9aFV2gmXHoraSEfVecHNhHyU470lx5X0CI8clICYsZzCKFea2IjG34myb2xNwQSQyg6pX7iUR4zafOsUImOacc30HX+XUoaOLn/GEcHiiBucRnso57CLICN474TXIEqEqBpaJEHnwxozxyQI7mlf2mfpdGt1X+ECjQth7lzKDJeapcK6uxy7g+9GALhjGC+dKXWY+MvM/7fvj+yUls798a4f05PivMDXmwysxYr3W/CCxzKOwSJNZ3SZkSRcpMg0zeIrk617jHsKI6ehJ1ADr8on3wfLjlD9Yg+4YqGxQxcTHN+IDexAZVO9erZR1C+02kTsqMmdyxjaTBdJOARxyzTCNeUDweVLIwXKPT8Bq5IX7CGEsySRDmqYxZpACzbvP0xIlZskuJ0ybKJUCLEmRDikJBIx0n/DcQEK2jthFidOgy6nnVv2Bc2LO]' Congrats! Check the lookup Let’s check if we can use puppet lookup to check the value. We should get cleartext. HIERA_EYAML_GPG_GNUPGHOME=~/.gnupg puppet lookup 'super_secret_test' --- super secret string Setup done. Adding and removing team members edit gpg_recipients add/remove public key to/from gpg_pubkeys directory run recrypt.sh commit merge done Known pitfalls Data Structures YAML literal blocks do not work. key: | content more content Other complex structures might not work either. Quoting The quote always goes around the enycryption. 'DEC(1)::GPG[super secret string]!' The index number and copying values Yes, it’s totally valid to copy a value in edit mode and paste it again, but be aware: Index numbers are counted by file and are always uneven. So if you copy, don’t forget to remove the index. It will be added automatically. --- super_secret_test: 'DEC(1)::GPG[super secret string]!' super_secret_test_two: 'DEC::GPG[super secret string too]!' Next edit: --- super_secret_test: 'DEC(1)::GPG[super secret string]!' super_secret_test_two: 'DEC(3)::GPG[super secret string too]!' Have fun using Hiera EYAML-GPG! Simon Hönscheid 2020-05-21T00:00:00+00:00 https://example42.com/blog/2020/05/21/managing-extra-repositories-with-tiny-puppet I’ve written recently a post about Tiny Puppet, which is a recommended reading if you want to understand some of its internals. Now, here, I want to talk about how you can use it to manage interesting and juicy extra packages repositories with it. Managing packages repositories with Tiny Puppet RedHat based repositories Ubuntu / Debian based repositories Adding custom repositories Managing packages repositories with Tiny Puppet I suppose everybody who works with RedHat Linux or derivatives is well aware of EPEL, a collection of packages, totally compatible with the default set of packages, shipped with RedHat Enterprise Linux (and derivatives like CentOS, Oracle Linux and Scientific Linux). Now on Puppet Forge there are various modules you can use to install the Epel repository, but why adding another module to your Puppetfile (with eventual dependencies), when you can install EPEL using Tiny Puppet simply by adding in your profiles: tp::install { 'epel': } And why limiting ourselves to EPEL, when with the single Tiny Puppet module and its TinyData companion you can add several other repos? What Repos? Let see them, note that Tiny Puppet installs them using the best approach possible, which in most of the cases means installing the relevant release package which takes care of configuring yum repo files, GPG keys and whatever is needed to add the repository to the system, but when a release package is not available the yum or apt configuration files, and eventual GPG keys are directly managed. This is transparent to the user and it all depends on the available Tiny Data. RedHat based repositories As of writing we currently support on Tiny Puppet, for RedHat and derivatives versions 6, 7 and 8 (where supported) the following additional repositories: Epel. The Big Brother of them all. The most used and supported, with more than 5000 extra packages a yum install away. As we have seen, to install it, as we anticipated, just add in a class used by your node the following Puppet code: tp::install { 'epel': } alternatively, if you have the class tp included in your nodes, you can install it via Hiera: tp::install_hash: epel: ensure: present finally, if you prefer to use Tiny Puppet from the command line (install it with: puppet module install example42-tp ; puppet tp setup), as root, on your favorite shell, you can simply type: tp install epel The above options are available for all the other repos (and applications) that Tiny Puppet can install, so we will not repeat them in the examples below. RPM Fusion, is another historic repo of extra packages, result of the merger of older repos (Dribble, Freshrpms and Livna, for who remembers them ;-) it provides both free and non free (like Nvidia drivers) packages for RedHat (and derivatives) 6, 7 and 8 and keeps compatibility with EPEL: tp::install { 'rpmfusion-free': } tp::install { 'rpmfusion-nonfree': } Note, in the git commit where RPM Fusion Tiny Data has been added, how we manage the needed dnf or subscription-manager commands on RedHat 8. IUS, which stays for Inline with Upstream Stable, provides updated packages of common applications, packages in a way to be compatible with RedHat native rpms. As of writing supports RedHat (and derivatives) 6 and 7: tp::install { 'ius': } The relevant commit shows how easily a new repo like this one could be added. Remi Repository, a repo which adds to EPEL’s packages several (more than 2000) other packages, particularly oriented to different versions of PHP, for RedHat (and derivatives) 6, 7 and 8: tp::install { 'remi': } Also here, one commit, one repo support added (and no extra module needed ;-) ELRepo focuses on hardware related packages for enterprise usage it supports RedHat (and derivatives) 6, 7 and 8: tp::install { 'elrepo': } Another simple commit. Nux DexTop provides multimedia and desktop oriented packages for RedHat (and derivatives) 6 and 7: tp::install { 'nux': } Introduced in a too quick commit based on another repo, then corrected. Ulyaoth, a repo with different versions of Tomcat and Solr packages for RedHat (and derivatives) 7 and 8: tp::install { 'ulyaoth': } Here just one release package is enough for all RedHat versions, as you can see in this commit. Ubuntu / Debian based repositories At the moment there are no extra Debian or Ubuntu repositories added to Tiny Data as we couldn’t find any interesting enough one which is not already added (or easy to activate) to the standard distros, or that is not related to a specific application (in such cases the repo is added to the application’s Tiny Data). In any case, if you have any interesting extra repo to suggest here, please let us know: adding it will be easy and quick. Adding custom repositories Since Internet is a bad and dangerous place, many companies prefer to have internal repositories where packages are both mirrored from upstream sources and locally packaged. Tiny Puppet can help here, and allow to handle any custom repository, either by using custom tinydata as in the above examples or by specifying directly the expected params when using the tp::repo define, which is declared inside tp::install if repo related Tiny Data is present. Let’s see some examples. If you have a local release package, with all the configurations of your repos and gpg keys, you can install it with something like: tp::repo { 'my_repo': repo_package_url => 'https://my.internal/my_repo-release.rpm', } If you have to configure a Yum Repository with these parameters (not all of them are needed): tp::repo { 'my_repo': repo_url => "https://my.internal/yum/repos/my_repo-el-${::os['distro']['release']['full']}-\$basearch", key_url => 'https://my.internal/my_repo-GPG-KEY', key => '54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB', yum_priority => '5', repo_description => "My Internal Repo", yum_gpgcheck => true, } For a Zypper repo for SuSE: tp::repo { 'my_repo': zypper_repofile_url => 'https://my.internal/zypper/repos/my_repo.repo', repo_name => 'my_repo', } If you use an Apt repository you can have parameters as follows: tp::repo { 'my_repo': repo_url => 'https://my.internal/apt/repos/my_repo/', key_url => 'https://my.internal/my_repo-GPG-KEY', key => '54A6 47F9 048D 5688 D7DA 2ABE 6A03 0B21 BA07 F4FB', apt_release => $::os['distro']['codename'], aptrepo_title => "My Internal Repo", apt_repos => 'main', aptrepo_title => 'my_repo', # Default is $title } It might be a bit overkill to use tp::repo to manage internal repositories, but consider it as a side effect of Tiny Puppet, considering that all this comes, as usual, with just one module that installs everything. Alessandro Franceschi 2020-05-05T00:00:00+00:00 https://example42.com/blog/2020/05/05/controlrepo-workflows When starting with Puppet you usually first create your Puppet GIT control-repository, a single place from where you can rebuild your whole Infrastructure with Puppet. Within this Puppet control repository you separate upstream library modules (forge modules) from your own code. Upstream libraries are added to Puppetfile (preferably specifying the version of each module). It is up to you, whether you just copy and adopt our Open Source Control-Repository or if you prefer to start with an empty repository. Each branch in the Puppet control repository will be deployed as a Puppet environment. In both cases you want to carefully consider your workflow on how to get changes into your code base. Some people consider Puppet environments as platform stages. But this causes some issues. What if you manage the infrastructure for the application development stage with Puppet using development branch? Are you totally sure that none of your commits will somehow break the development infrastructure? How do you bring single feature changes from development to production (maybe via testing branch)? But you can do different and more easy in the beginning. The most simple option is to only use the production environment branch and add all changes via feature branches and merge requests. When working in an ITIL based change management environment this simple approach does not adapt to change management requirements which only allows code changes being tested on dedicated systems prior being deployed to production systems. This is where one should consider adopting the GIT Flow concept. This blog post explains the simple, stage branches and the git flow based change workflow for a Puppet control-repository. Simple workflow Transfer Branch GIT Flow Recommendations Simple workflow Within the simple workflow you are working with a single long living branch which we usually call production. We prefer to set this branch to “protected” to prevent any direct changes. All changes must be delivered using feature branches which will be merged into production branch. When working with multiple people in different feature branches, everyone must rebase their branches on a regular basis, at least prior creating the merge request. This is similar to many upstream development procedures of most Puppet library module code. Pro: easy to learn Con: changes on Puppet code affect all systems at once An example: prod prod | | feature You will start by creating your own feature branch: git checkout -b <feature_branch> At customers we usually recommend to build the name of the branch based upon user or team name. e.g. git checkout -b alfke_new_db_role. Additionally we recommend to work with rebase on feature branches instead of merge. Rebasing will take care that your feature branch changes are placed after any other production changes. If you see changes on production branch you need to rebase: git rebase origin/production Any feature branch should result in a merge request. Every merge request should consist of a single commit only. The best option is to use git commit --amend on any additional change or to squash all commits once your feature is ready to get deployed. Transfer Branch But how do you proceed when there is a requirement for following changes in stages independently? How do you follow your implementation documentation standards? Some infrastructures require network separation between development and production systems. This also affects Puppet Code and GIT Server, as you will have multiple installations. In this case we recommend to only work on the development side using feature branches. A special transfer branch gets updates and has a CI system with access to both networks, so it can pull code from the development git transfer branch and pushes changes into the production git transfer branch. On the production git a merge request is generated to merge changes from transfer into production. On development git you only have development, transfer and feature branches. On production git you have transfer and production branch. Pro: security concerns like network separation are taken care on Con: needs automatic merging and staging, no manual merges up to production merge request. GIT Flow But how to proceed, if you want to have Puppet code available for each of your infrastructure stages? In this case you have to create several long living branches like development, testing and then production. You can use any string lower case letters and numbers and underscore as environment name. Maybe you prefer other naming like dev, qa, int, pre_prod, prod. But using multiple branches makes it harder to deploy single changes independently. What will happen upon merge if you have two changes within the development branch and only the second one may be deployed to the next branch? One must reconsider on how you look at your branches within your GIT repositories: Instead of just seeing one single code base within a GIT repository you should see several loosely coupled streams of code placed into branches on a single GIT repository. Each of these code stream branches can be developed and improved independently and all development must be done in a stream feature/change branch. All workflows must be tracked within a ticket. Within this change ticket you follow work and deployment be placing them into subtasks for each stream branch. To allow independent changes still being transferred from one branch to another, you must take care that a single merge request consists of a single commit only. You can achieve this by using git commit --amend or by running git squash. This might look like duplicate work, as you need to apply the same change in multiple places. But on the other hand, this deployment and staging methods allows you to also deploy hotfixes in production and backporting them to development by using git cherry-pick. Multiple branches have several additional requirements: Rebase and Squashing is a hard must for each merge. Cherry-Picking is the way to get a change from one feature branch to another. All long living branches must be protected. Changes may only be added via merge requests. No exceptions allowed. Normal code staging is done by merging into dev and cherrypicking the change into a merge request to the other branches. Hotfixes in testing are developed on testing feature branch, forwarded via cherry-pick to production feature branch and backported (also cherry-pick) into development feature branch. Hotfixes on production branch are developed on production feature branch and are backported via development feature branch and testing feature branch (both cherry-pick). Pro: adopts agile and waterfall concepts Con: needs more GIT knowledge Recommendations When to use which solution? What are characteristics which will make you consider using staging branch or GIT Flow. What are the requirements? Deploy fast and often? Single platform, no stages? -> Use simple Deploy fast and often? Platform separation e.g. network isolation? -> Use staging branch Deployment may stuck for weeks in a specific stage? Deployments are rare and slowly? ITIL based change management and change approval process? -> Use GIT Flow Switching to GIT Flow allows more flexible handling of complex change approval requirements but needs more understanding on GIT squash, rebase and cherry-pick and how to deal with merge conflicts. Get yourself a proper GIT Server (appliance). Automate everything. Happy hacking, If you’d rather have this workflow chosen and set up for you, that’s what Puppet Infrastructure Kickstart is for. Martin Alfke 2020-04-20T00:00:00+00:00 https://example42.com/blog/2020/04/20/five-years-of-tiny-puppet When my colleague Martin Alfke suggested me to write a blog post about Tiny Puppet, I wondered what I could write about it that I haven’t written before. So I started to look at the past blog posts and I realized that more than five years have already passed since the first announcement, in January 2015. At those times Tiny Puppet came as a natural evolution of the first generation and the “NextGen” example42 Puppet modules: a large set of modules, now mostly deprecated, with a standard structure that became a pain to maintain for a single person. Tiny Puppet was my natural solution for a problem that for me was, and still is, of some relevance: reduce the number of modules and dependencies to use, especially when resources to manage are simple packages, services and files. For this reason I wrote a single module to manage via Puppet a cross-OS installation of several (potentially ANY) applications giving users full freedom on how to handle their configurations. The core tp defines (Used Defined Type, written in Puppet DSL) were already there, since the beginning: tp::install, tp::conf, tp::dir and all of them were already relying on TinyData, a yaml based, hierarchical set of files where are defined the different settings of different applications on different Operating systems. Initially the tinydata was placed in the same tp module, then, with the release 1.0 it was separated in an autonomous git repo to allow easier upgrades and separate the tool (the tp module) from the data (the tinydata repository, which can also be user defined and customized using the data_module parameter, present in all the tp defines. With the release 1.0 we started to support Puppet 4 by default, note anyway that you can still use tp with Puppet 3 (and even Puppet 2), using the relevant defines with the 3 suffix: tp::install3, tp::conf3, tp::dir3. tp::install The define tp::install in its simplest usage just accepts, as title, the name of the application to install. The names of the package(s) to install are derived from the relevant Tiny Data, or, if missing, from the same title used for the defines (this is possible since TP version 2.3. Some examples of tp::install can be: tp::install { 'apache': } # We rely on common sense, where Apache is intended as the web server and not the Foundation. # Tiny Puppet takes care of using the proper package name on different distros tp::install { 'redis': } # Not so different than a plain "package { 'redis': }" since that's usually the name of the # Redis package everywhere tp::install { 'elasticsearch': } # Here Tiny Puppet provides some extra benefits, like installing the relevant repo, on the # underlying distro and then the elasticsearch package tp::install { 'gitlab-ce': } # Under the hoods, the installation here is down in an unusual way: it downloads and runs the # install script from the GitLab website tp::install { 'sysdig': } # The fastest way to install this cool tool. Tiny Puppet takes care of repos, dependencies # and package tp::install { 'kubernetes': } # A recent addition to the supported apps: nothing particularly new: repos and packages are # managed (but, hei, no kube cluster is initialised) tp::install { 'opera': } # Since there's no specific tinydata for opera, here, if present, is just installed a package # called opera, using the underlying package manager. So what tp::install does? Basically it uses the data in Tiny Data to handle the right package name for the underlying OS (and by OS we typically mean the most popular Linux distros (RedHat 6, 7 and 8 and derivatives, Debian 7,8,9,10 and derivatives (Ubuntu mostly LTS editions), SuSe and OpenSuse), Darwin/MacOS (relying on brew and brew-cask), Windows (relying on Chocolatey), Solaris and *BSD. The resource tp::install, is basically a wrapper, that, according to the relevant tiny data can manage: one or more package resources, based on the value of the following tiny data settings: package_name the application’s package(s), whose installation can be customized with settings as package_provider, package_source, package_install_options and the general package_params hash to eventually handle all the other arguments of Puppet’s package resource package_prerequisites an array of eventual packages to install as prerequisites one or more service resource, based on the value of the following tiny data settings: service_name the application’s service(s), whose parameters can be managed by the service_ensure, service_enable and the general service_params hash. one or more exec resources as described in one of these tiny data setting: exec_prerequisites, an hash of exec resources with the relevant parameters to run before package installation exec_postinstall, an hash of exec resources with the relevant parameters to run after the package installation one or more other tp::install resources as defined by the tiny data setting tp_prerequisites (an array of prerequisite application to install via tp) a tp::repo resource, which manages the extra package repository (for yum, apt and zypper) if any of these tinydata setting is availeble: repo_url, yum_mirrorlist, repo_package_url. one or more tp::conf resources, according to the optional conf_hash parameter passed to the tp::install define a tp::conf { $app: } resource if parameter auto_conf is set to true and exists the tiny data setting config_file_template a tp::conf { “${app}::init”: } resource if parameter auto_conf is set to true and exists the tiny data setting init_file_template one or more tp::dir resources, according to the optional dir_hash parameter passed to the tp::install define a tp::test resource, which can be used to test the installed application, if the parameter test_enable is set to true a tp::puppi resource, which can be used to integrate the installed application with Puppi , if the parameter puppi_enable is set to true the file { “/etc/tp/app/${app}”: } used by the tp cli command if the parameter cli_enable is set to true Quite a long list of resources, indeed, but don’t be scared by that, in most of the cases with a tp::install you just manage, via Puppet, a package and a service, needless to say that it’s up to tp to handle the correct names and dependencies. tp::repo One of the Tiny Puppet added values, compared to the abstraction that Puppet already provides with the package resource, is to manage eventual repositories needed to install a package. This is done by using different tinydata settings, more specifically, tp::repo can manage: Installation of a release package, containing the configuration of a yum or apt repo, if is present for a given application the setting repo_package_url. In this case the settings repo_package_name, repo_package_provider and repo_package_params are used to define, respectively, the name of the repo release package, the Puppet provider to install it and eventual other parameters to pass to the repo package resource. If no release package is provided, the package repositories can be configured directly: Yum repositories are managed via a yumrepo Puppet resources using the tinydata settings repo_url, repo_filename, repo_description, yum_mirrorlist, key_url and yum_priority. Apt repositories are managed via exec (to add gpg keys) and file (for the source list file) resources, based on the tinydata settings: repo_url, aptrepo_title, apt_key_server, apt_key_fingerprint, key_url and key. Zypper repositories are managed via exec resources based on the tiny data settings: repo_url, repo_name, zypper_repofile_url. All the above tinydata can be overridden by parameters passed to the tp::repo define. One special parameter, called upstream_repo allows users to specify if to install a package using the repo from the same application upstream authors or if to use the default repos from the underlying OS. Of course the relevant tinydata settings must exist in order to enforce this option. tp::conf This tp define is probably the simplest one, as it just manages ONE file, related to the application specified in the title. By default tp::conf manages the main configuration file of the relevant application. This is defined by the config_file_path tinydata setting- So for example, a define as follows in going to manage the file /etc/httpd/conf/httpd.conf on RedHat derivatives or /etc/apache2/apache2.conf on Debian derivatives: tp::conf { 'apache': source => 'puppet:///modules/profile/apache/apache.conf', } The content of the file to manage can be set using different, alternative, parameters, which are reflected on actual parameters of the file resource: source => 'puppet:///modules/profile/apache/apache.conf', # Passed as is to source parameter of the file resource content => '#This file has no content', # Passed as is to content parameter of file resource template => 'profile/apache/apache.conf.erb', # Passed to content parameter as template($template) epp => 'profile/apache/apache.conf.epp', # Passed to content parameter as epp($epp) When using templates it’s possible to use the options_hash parameter to specify an Hash of custom key pairs which can be used in the templates, for dynamic, user defined, data. In the templates it’s also available the variable settings which all the tinydata settings for the application. For example, we can have a define as: tp::conf { 'mongodb': template => 'profile/mongodb/mongod.conf.erb', options_hash => { 'replSetName' => 'prod0', 'bindIp' => $::ipaddress, } } and in the file profile/templates/mongodb/mongod.conf.erb a content as follows (note that the parameter $options_hash can be accessed in the template both via the options_hash and the options variable, and that the settings variable get directly the correct setting for the underlying OS): # File managed by Puppet - mongod.conf systemLog: destination: file logAppend: true path: <%= @settings['log_file_path'] %> storage: dbPath: <%= @settings['data_dir_path'] %> processManagement: fork: true pidFilePath: <%= @settings['pid_file_path'] %> net: port: <%= @settings['tcp_port'] %> bindIp: <%= @options['bindIp'] %> <% if @options['replSetName'] != '' -%> replication: replSetName: <%= @options['replSetName'] %> <% end -%> So, what’s important to understand here is that Tiny Puppet has no idea on how to configure any application (as a dedicated module may have) and leaves totally to the user HOW to handle its configurations. This can be good (if you know how to configure your stuff and just need a quick way to Puppettize it) or not (if you rely on a module’s intelligence the logic of how to configure an app, or if there are more complex and specific resources to manage than just packages, services and configuration files). Note, however, that you can rely on Tiny Data settings to manage configuration files which may be “cross OS compatible” how of the box. In the above template example, we have seen some other tinydata settings which may be useful when working with tp::conf. We have seen that by default, if you just pass in the title the name of an application, tp::conf manages the “main” configuration file of that application, but you can actually manage other files for that application according to the following conventions. If we specify a file name after the application name in the title, separated by ::, that file is placed in the main configuration directory (setting config_dir_path), the following, for example, will manage /etc/ssh/ssh_config since openssh’s config_dir_path is /etc/ssh: tp::conf { 'openssh::ssh_config': [...] } If we explicitly set a path, that path is used and the title is only used to understand what application we are managing and handling the relevant dependencies. In the following example we manage /usr/local/bin/openssh_check: tp::conf { 'openssh::ssh_check': path => '/usr/local/bin/openssh_check', [...] } If we specify a base_dir and use a title with the format: application::file_name the file is created with the defined name in the indicated base directory. For example, the following wil create (in RedHat derivatives) /etc/httpd/conf.d/example42.com.conf: tp::conf { 'apache::example42.com.conf': base_dir => 'conf', # Use the settings key: conf_dir_path } There are different possible base_dir values, they may be defined according to the application. The most common ones are: base_dir param Settings key Description config config_dir_path The main configuration directory. That's the default value for base_dir. conf conf_dir_path A dir that contains fragments of configurations (usually /conf.d/) log log_dir_path Directory where are placed the application logs data data_dir_path Directory where application data lives Why is it useful and necessary to always specify the application name in tp::conf? Because in this way all dependencies are automatically managed: file is managed after the installation of the relevant package and triggers a service restart. We can however disable, or change, this behavior. To disable package and service dependencies: tp::conf { 'bind': config_file_notify => false, config_file_require => false, } To override the default dependencies, based on tinydata we can also specify directly the relevant resources: tp::conf { 'bind': config_file_notify => Service['bind9'], config_file_require => Package['bind9-server'], } tp::dir Tp::dir is like tp::conf but works on whole directories rather than just single files. For example, to manage the whole content of /etc/redis, we can use (source files are supposed to be present in our profile module under profile/files/redis): tp::dir { 'redis': source => 'puppet:///modules/profile/redis/', } Also here the base_dir parameter can be used to define what directory we are using (default value is always “config” meaning the main configuration directory of an application, in case this concept might be controversial, just refer to the actual tinydata settings and act accordingly). We haven’t mentioned this before, but the following parameters cna be used, both for tp::dir and tp::conf: tp::dir { 'redis': owner => 'root', group => 'redis', mode => '0750', } We can also force the recursive purge of all the files of a directory which are not in the given source (params here behave as in the file resource), always be careful when recursively purging files with Puppet (double check your paths!): tp::dir { 'redis': source => 'puppet:///modules/profile/redis/', recurse => true, purge => true, force => true, } In case you want to manage a directory, which might have as parent a non existing directory, you can ensure that its parent directory exists, to prevent Puppet failures: tp::dir { 'apache::my_app': path => '/data/www/my_app', path_parent_create => true, } You can also manage the content of a directory using a git repository ()or whatever is supported by puppetlabs-vcsrepo module, which is require for this functionality): tp::dir { 'apache': base_dir => 'data', source => 'https://git.example.42/apps/my_app/', vcsrepo => 'git', } You can even do a poor man continuous delivery by ensuring you have always the latest version of a git repo: tp::dir { 'apache::my_app': ensure => 'latest', path => '/data/www/my_app', path_parent_create => true, source => 'https://git.example.42/apps/my_app/', vcsrepo => 'git', } Everything is data We have talked about the tp defines, but there’s actually a tp class, which is needed only if you want to install the tp command line on your systems and to manage tp defines via Hiera. If you include it iin your catalog you will have the tp command and the possibility to configure via Hiera basically everything. In the following example, we are managing a full LAMP stack, included configuration files for virtual hosts and web application files with auto deployment from a Git source (don’t try this at home, you miss some files and access to git repos): tp::install_hash: apache: {} mysql: {} php: {} tp::conf_hash: apache::openkills.info.conf: base_dir: conf template: psick/apache/vhost.conf.erb options_hash: ServerName: openskills.info ServerAlias: - openskill.info - www.openskills.info - www.openskill.info AddDefaultCharset: ISO-8859-1 apache::deny_git.conf: base_dir: conf source: puppet:///modules/psick/apache/deny_git.conf apache::abnormalia.com.conf: base_dir: conf template: psick/apache/vhost.conf.erb options_hash: ServerName: abnormalia.com ServerAlias: - www.abnormalia.com AddDefaultCharset: ISO-8859-1 extra: ErrorDocument: '404 /index.php' tp::dir_hash: apache::openskills.info: vcsrepo: git source: git@bitbucket.org:alvagante/openskills.info.git path: /var/www/html/openskills.info apache::abnormalia.com: ensure: latest vcsrepo: git source: git@bitbucket.org:alvagante/abnormalia.com.git path: /var/www/html/abnormalia.com tp resources defaults If you use massively tp in your control repo, you may want to set in the main manifests/site.pp some resource defaults which are applied to all the tp resources in your catalog. Here is an example where tp cli integration is enabled (do yourself a favour, use it just for the joys of writing on your shell tp log and tp test ;-) ) and a custom tinydata module is used. $tinydata_module = 'my_tinydata' Tp::Install { cli_enable => true, test_enable => true, data_module => $tinydata_module } Tp::Conf { data_module => $tinydata_module, } Tp::Dir { data_module => $tinydata_module, } Using tp for custom applications or in local profiles An interesting point to consider is that, besides the option data_module which allows the usage of a custom tinydata module, all the tp defines have also the parameter settings_hash which can be used to override any tiny data setting. We can even manage via tp a custom application, packaged internally, as long as we have the relevant data in a custom tinydata module: tp::install { 'my_app': data_module => 'my_tinydata', } When you use directly the upstream tinydata module and not a local clone (in your internal git servers) or fork of it, we recommend to always specify the used version in your Puppetfile and properly test changes before updating the version used: tinydata contains info about a large number of applications, and their data might be updated if we find that is wrong for a given OS or if we want to extend support for a new OS release. These changes don’t follow SemVer standards and even a minor version update in Tiny Data may change some settings for an application you manage via Tiny Data. The rule of thumb here is: know what applications you manage via tp and check, when upgrading the tinydata module, if there are changes in data for these applications that might impact you. It’s not hard and not difficult, but that’s the tradeoff of having a single module managing different applications. Changes on the tp, which contains code and not data, instead, are much more controlled and fully follow SemVer conventions. As alternative to the usage of a custom tinydata module or fork, you can override the defaults tinydata settings. Let’s say you want to install a custom apache package, coming from some internal repo, but keep all the other settings, you can write something as follows: tp::install { 'apache': settings_hash => { package_name => 'my_httpd', } } Even better, to be sure that we use the same settings for all the related defines, place your settings in a variable (whose content can come from Hiera) and use it wherever needed: $apache_settings = { package_name => 'my_httpd', } tp::install { 'apache': settings_hash => $apache_settings, } tp::conf { 'apache::mime.types': settings_hash => $apache_settings, [...] } More information about how to use Tiny Data to configure custom applications and how to use tp defines in custom profiles, can be read in this blog post. The full list of the currently used available tinydata settings is defined in the tp::settings data type, note however that currently this is not enforced or used for validation of the settings_hash parameter. More information on the tiny data settings you can configure and customize to adapt to local versions of supported application or totally new applications, have been described in a recent posts series, where we described and updated info on: Tiny Puppet principles Details on Tiny data structure Fancy and powerful features on how to check for config files syntax, how to use upstream repos or run an application in a container. How to manage ANY application One of the interesting things I noticed reading the past blog posts mentioned in this article is that most of their contents are still actual and valid: I’m not sure if this is a good or bad sign. Has Tiny Puppet been sane enough since the beginning or is it simply never evolved too much? Whatever the answer, it’s still here, and, as far as we know, it’s used in Startups, Top 100 companies, Government institutions and Central Banks. So, I guess, it just works. Alessandro Franceschi 2020-02-12T00:00:00+00:00 https://example42.com/blog/2020/02/12/managing-pe-troubleshooting In the previous blog post we have seen what are the main services present in a Puppet Enterprise (PE) server, here we are going to give a look at their logs. All Puppet Enterprise log files are under the /var/log/puppetlabs directory. Here there are different subdirectories and log files for each single component. Puppet Server: /var/log/puppetlabs/puppetserver/ /var/log/puppetlabs/puppetserver/puppetserver.log Sample Puppet catalog compilation log. Sample code deployment log (with success) /var/log/puppetlabs/puppetserver/puppetserver-access.log /var/log/puppetlabs/puppetserver/code-manager-access.log /var/log/puppetlabs/puppetserver/puppetserver_gc.log.0.log PuppetDB: /var/log/puppetlabs/puppetdb/ /var/log/puppetlabs/puppetdb/puppetdb.log /var/log/puppetlabs/puppetdb/puppetdb-access.log /var/log/puppetlabs/puppetdb/puppetdb_gc.log.0.current Console Services: /var/log/puppetlabs/console-services/ /var/log/puppetlabs/console-services/console-services.log /var/log/puppetlabs/console-services/console-services-access.log /var/log/puppetlabs/console-services/console-services-api-access.log /var/log/puppetlabs/console-services/console-services_gc.log.0.current Nginx: /var/log/puppetlabs/nginx/ /var/log/puppetlabs/nginx/access.log /var/log/puppetlabs/nginx/error.log Puppet Enteprise Installer: /var/log/puppetlabs/installer/ Orchestration Services: /var/log/puppetlabs/orchestration-services/ /var/log/puppetlabs/orchestration-services/orchestration-services.log /var/log/puppetlabs/orchestration-services/orchestration-services-access.log /var/log/puppetlabs/orchestration-services/orchestration-services-api-access.log /var/log/puppetlabs/orchestration-services/orchestration-services_gc.log.0.current PostgreSQL: /var/log/puppetlabs/postgresql/ /var/log/puppetlabs/postgresql/pgstartup.log /var/log/puppetlabs/postgresql/postgresql-WEEKDAY.log CLIENT LOGS /var/log/messages /var/log/puppetlabs/pxp-agent/pxp-agent.log Puppet Server: /var/log/puppetlabs/puppetserver/ Directory /var/log/puppetlabs/puppetserver contains the logs of the Puppet Server component. /var/log/puppetlabs/puppetserver/puppetserver.log This is probably the most important and used log file of Puppet Enterprise. It contains information on all the Puppet Server activities: When a catalog is compiled for a client Eventual warnings or errors during the compilation of a catalog The Commands submitted to PuppetDB (replace_facts, replace_catalog, store_report) Information about the Code deployment activities This log should be analysed to troubleshoot the following issues: Code Deployments failures (from the logs it’s possible to verify if code has been successfully deployed and, in case of errors, the underlying reason) Recurring warnings on catalog compilations (WARNings, are generally not an issue but should be reviewed regularly and possibly cleaned up) Catalog compilation ERRORs, which prevent the Master from sending a valid catalog to clients You can change Puppet Server’s logging behavior by editing /etc/puppetlabs/puppetserver/logback.xml, and you can specify a different Logback config file in global.conf. See: https://puppet.com/docs/puppetserver/6.1/config_file_logbackxml.html for details. Logs relevant to catalog compilation look like this: 2019-01-14T10:57:48.774Z WARN [qtp223287217-100574] [puppetserver] Puppet Unknown variable: '::nisdomainname'. (file: /etc/puppetlabs/code/environments/production/modules/network/manifests/global.pp, line: 49, column: 15) Format is: Datetime Log level (INFO, WARN, ERROR…) The default log level is INFO, and Puppet Server sends nothing to syslog. An unique identifier for each catalog compilation (in the example is qtp223287217-100574): use it with grep to identify all the messages related to a single client catalog compilation. This is useful on busy Puppet Masters where logs related to different clients are mixed on puppetserver.log. So, for example, to have the full list of log messages related to the above sample line run: grep qtp223287217-100574 /var/log/puppetlabs/puppetserver/puppetserver.log The name of the involved puppetserver subsystem (always [puppetserver] in logs concerning Puppet runs The actual log message Sample Puppet catalog compilation log. Here’s a sample log of all the activities related to a single client catalog compilation: 2019-01-14T11:27:47.616Z INFO [qtp223287217-100708] [puppetserver] Puppet 'replace_facts' command for node1.example.com submitted to PuppetDB with UUID 2d4b60f5-1853-42b4-8d01-d5604af34ebc This line refers to the command replace_facts sent to PuppetDB by the Server. It’s the first activity done by the server when it rreceives catalog compilation requests: it copies to PuppetDB all the updated client facts and then it uses them when compiling the catalog. 2019-01-14T11:27:48.313Z WARN [qtp223287217-100708] [puppetserver] Puppet Unrecognized escape sequence '\/' (file: /etc/puppetlabs/code/environments/production/modules/logging/manifests/logrotate.pp, line: 151, column: 128) 2019-01-14T11:27:48.313Z WARN [qtp223287217-100708] [puppetserver] Puppet Unrecognized escape sequence '\/' (file: /etc/puppetlabs/code/environments/production/modules/logging/manifests/logrotate.pp, line: 151, column: 128) 2019-01-14T11:27:48.313Z WARN [qtp223287217-100708] [puppetserver] Puppet Unrecognized escape sequence '\/' (file: /etc/puppetlabs/code/environments/production/modules/logging/manifests/logrotate.pp, line: 151, column: 128) 2019-01-14T11:27:50.152Z WARN [qtp223287217-100708] [puppetserver] Puppet Unknown variable: '::nisdomainname'. (file: /etc/puppetlabs/code/environments/production/modules/network/manifests/global.pp, line: 49, column: 15) All the above warning have been produced by the Puppet Server while compiling the client’s catalog based on the client facts and the currently deployed Puppet code and Hiera data. Warnings like 'Unrecognized escape sequence' can usually be ignored as they might be due to necessary escape () chars used in commands or Windows path referred in Puppet manifests. In some cases they are hard to fix as we might actually need such escape chars in that shape, in other cases some workarounds can be tried. Warnings like 'Puppet Unknown variable' should deserve some more attention. They happen when we try to use in Puppet code a variable which is not set. In some cases this is due to lazy checks on variables existence, in other they might be due to a missing value for other, unwanted reasons. In any case they are worth a check and a fix, even if they might not actually be a problem. If you have the Puppet parameter strict_variables set to true all the above 'Puppet Unknown variable' warnings would cause a catalog compilation failure. 2019-01-14T11:27:50.892Z INFO [qtp223287217-100708] [puppetserver] Puppet Inlined resource metadata into static catalog for node1.example.com in environment production in 0.01 seconds This entry states that Puppet has generated metadata for file resources with a puppet:/// source. This happens when static catalogs are used (as default in recent Puppet versions) and the contents of files with the source argument are directly placed in the catalog instead of being requested by the client to the server when the catalog is applied. In short words: you will always see this line when static catalogs are used. 2019-01-14T11:27:50.892Z INFO [qtp223287217-100708] [puppetserver] Puppet Compiled static catalog for node1.example.com in environment production in 3.12 seconds This line states that the Server has successfully compiled the catalog for the client (here node1.example.com) and that it took 3.12 seconds. Use a command like: grep 'Puppet compiled' /var/log/puppetlabs/puppetserver/puppetserver.log to have a quick overview of the catalog compilation times (the same information is available via the PE console anyway). 2019-01-14T11:27:50.892Z INFO [qtp223287217-100708] [puppetserver] Puppet Caching catalog for node1.example.com This means that the server has started to send the catalog back to the client. 2019-01-14T11:27:51.241Z INFO [qtp223287217-100708] [puppetserver] Puppet 'replace_catalog' command for node1.example.com submitted to PuppetDB with UUID 24849581-5ca3-489d-ad19-3860a6bd141d At the same time, the Puppet Server, as it did for facts, sends the catalog also to PuppetDB (only the last version of the catalog and facts is stored on PuppetDB). This allows us to query PuppetDB for each resource on each client, and, for example, have the possibility to view each node’s graph on the PE console. 2019-01-14T11:28:09.045Z INFO [qtp223287217-100708] [puppetserver] Puppet 'store_report' command for node1.example.com submitted to PuppetDB with UUID 70b67f83-d796-41df-97c2-3f5f4fbb051c This is the last log line concerning a single Puppet run, it’s shown some time after the previous ones and it indicates that the Puppet Server has sent the content of the Puppet run report received from the client after the local catalog application. Sample code deployment log (with success) This is a sample output of a log puppetserver.log during the deployment of Puppet code via Code Manager. 2019-01-14T11:25:54.294Z INFO [qtp1096095702-961] [p.c.app] Queuing deploy for environment production 2019-01-14T11:25:54.333Z INFO [deploy-pool-1] [p.c.core] Attempting to deploy environment 'production'... The above lines appear as soon as a Code Manager deployment is triggered, either via a webhook or the command line. The next lines appear only after the code has been successfully deployed or failed. They may appear some minutes later. 2019-01-14T11:29:44.794Z INFO [deploy-pool-1] [p.c.core] Successfully staged environment 'production': This happens when the code has been successfully deployed to the /etc/puppetlabs/code-stating directory. 2019-01-14T11:29:47.778Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-versioned-code-core] Running pre-commit hook command: /opt/puppetlabs/server/bin/generate-puppet-types.rb 2019-01-14T11:29:47.956Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing staging directory /etc/puppetlabs/code-staging to file sync storage service 2019-01-14T11:29:48.064Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing submodules in directory 'environments' for repo :puppet-code 2019-01-14T11:29:48.065Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing submodule :production 2019-01-14T11:29:51.432Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing the following changes: Added Files: modules/data/hiera/hosts/node1.example.com.yaml, modules/data/hiera/hosts/node2.example.com.yaml, modules/data/hiera/hosts/node4.example.com.yaml, modules/data/hiera/hosts/node3.example.com.yaml, modules/profile/hiera/hosts/ ...(10 of 256 files shown) Removed Files: modules/profile/manifests/options.pp, modules/hiera/host/lnp6d1gitrdb03.example.com.yaml ...(10 of 44 files shown) Changed Files: .r10k-deploy.json, modules/profile/files/context.xml ...(10 of 620 files shown) This is the list of the changes occurred on the deployed control-repo files (this specific deployment has been done after several days, so many files have changed. Usually they are much less) 2019-01-14T11:29:51.606Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Completed commit in submodule at environments/production 2019-01-14T11:29:51.607Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing repo /etc/puppetlabs/code-staging 2019-01-14T11:29:51.608Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Committing the following changes: Added Files: Removed Files: Changed Files: Added Submodules: Updated Submodules: production 2019-01-14T11:29:51.610Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-storage-core] Completed commit of repo :puppet-code 2019-01-14T11:29:51.721Z INFO [deploy-pool-1] [p.c.file-sync] committed environment production with environment commit 'e59927f2c0f92c3af375642f69d016d88b10c9ea' and code commit '82aa94684fb1abc6f07b7ae74c11635f7e2be209' 2019-01-14T11:29:51.842Z INFO [deploy-pool-1] [p.c.core] Finished deploy attempt for environment 'production'. This message confirms that the code has been successfully deployed to the /etc/puppetlabs/code directory 2019-01-14T11:29:54.524Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-client-core] Fetching ':production' to e59927f2c0f92c3af375642f69d016d88b10c9ea 2019-01-14T11:29:55.504Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-client-core] New latest commit: e59927f2c0f92c3af375642f69d016d88b10c9ea 2019-01-14T11:29:55.506Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-client-core] Fetching ':puppet-code' to 82aa94684fb1abc6f07b7ae74c11635f7e2be209 2019-01-14T11:29:55.532Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-client-core] New latest commit: 82aa94684fb1abc6f07b7ae74c11635f7e2be209 2019-01-14T11:29:55.642Z INFO [clojure-agent-send-off-pool-1796] [p.e.s.f.file-sync-client-core] Forcefully syncing live directory at /etc/puppetlabs/code for repository :puppet-code The above messages notify that the code has been synced to the other compile master or replica servers, if they exist. 2019-01-14T11:29:57.741Z INFO [clojure-agent-send-off-pool-1796] [p.s.j.puppet-environments] Marking environment 'production' as expired. 2019-01-14T11:29:57.741Z INFO [clojure-agent-send-off-pool-1796] [p.s.j.puppet-environments] Marking environment 'production' as expired. 2019-01-14T11:29:57.741Z INFO [clojure-agent-send-off-pool-1796] [p.s.j.puppet-environments] Marking environment 'production' as expired. 2019-01-14T11:29:57.741Z INFO [clojure-agent-send-off-pool-1796] [p.s.j.puppet-environments] Marking environment 'production' as expired. When code is successfully deployed the relevant environment cache is flushed, so that Puppet Server ensures that an updated code is used to compile catalogs. /var/log/puppetlabs/puppetserver/puppetserver-access.log This log contains all the http requests done from clients to the Puppet Server (remember, all communications from clients to server’s port 8140 is via https). Log format is similar to a normal web server log and may give useful information. A sample line looks like: 10.199.33.29 - - [14/Jan/2019:13:51:13 +0000] "GET /puppet/v3/file_metadata/modules/pe_infrastructure/puppet-infrastructure?environment=production&links=manage&checksum_type=md5&source_permissions=ignore HTTP/1.1" 200 267 "-" "Puppet/6.0.4 Ruby/2.5.1-p57 (x86_64-linux)" 133 With the following fields: The client host IP address The remote log name (usually not set, so you see a -) The user name (not set, so -) The datetime The http request method (always GET, except when clients sends the puppet run reports, via a PUT) The requested URL (endpoint) on the server with the arguments passed (after the ? sign) and the protocol used (alway HTTP/1.1) The http status code (200 for any correct connection, 304 for cached replies, 5xx for server errors, 4xx for authentication/access errors) The response’s content lenght in bytes The referer (always - here) The client user agent (here you can see the client’s Puppet version) This logfile format and place is configured in /etc/puppetlabs/puppetserver/request-logging.xml Even if this log contains useful and interesting information about the http traffic on the server, it’s not commonly used, as the puppetserver.log gives better, application level, insights. Could be used, anyway, to generate statistics with tools that parse normal http access logs. /var/log/puppetlabs/puppetserver/code-manager-access.log This log contains the logs of all the http requests made to the codemanager component. It shows a line for each attempted code deployment. Format is similar to the puppetserver/access.log one. Note that a 200 status code here doesn’t involve necessarily a successful code deployment, but that a deployment activity has been successfully queued. /var/log/puppetlabs/puppetserver/puppetserver_gc.log.0.log This log contains information about Garbage Collection on the JVM the Puppet Server is running on. Refer to docs like https://dzone.com/articles/understanding-garbage-collection-log for more details on how to interpret the output. Note that messages like [GC (Allocation Failure) are relatively normal and, per se, not a sign or a failure. Proper analysis of these logs might be done to optimise and fine-tune JVM memory settings for the Puppet server. Check https://puppet.com/docs/puppetserver/6.1/tuning_guide.html for details on tuning PuppetServer. PuppetDB: /var/log/puppetlabs/puppetdb/ The directory /var/log/puppetlabs/puppetdb/ contains the PuppetDB logs. /var/log/puppetlabs/puppetdb/puppetdb.log This log contains all PuppetDB application logs. In particular, you can see here: All the commands submitted to Puppetdb The sync activities between the Master and the Replica Puppet DB servers PuppetDB internal activities such as flushing of stale reports, purging of deactivated nodes and database garbage collection events A command log looks like: 2019-01-14T14:20:37.836Z INFO [p.p.command] [16827-1547475637809] [24 ms] 'replace facts' command processed for node3.example.com Here we can see: The Datetime The log level The Component involved An unique ID The time spent in ms for the operation The PuppetDB command The certname for which the command was issued Format, location, and rotation of this log is configured in /etc/puppetlabs/puppetdb/logback.xml. /var/log/puppetlabs/puppetdb/puppetdb-access.log Similarly to the puppetserver-access.log, this one contains http requests done to PuppetDB. It has the same combined log format and just gives an idea of the http traffic, without details on what happens at the application level. Format, location, and rotation of this log is configured in /etc/puppetlabs/puppetdb/request-logging.xml. /var/log/puppetlabs/puppetdb/puppetdb_gc.log.0.current This log contains information about Garbage Collection on the JVM the PuppetDB is running on. What has been written about puppetserver_gc.log.0.current applies also here. Check https://puppet.com/docs/puppetdb/5.1/maintain_and_tune.html for details on PuppetDB tuning. Console Services: /var/log/puppetlabs/console-services/ The directory contains /var/log/puppetlabs/console-services/ the logs of PE Console, the web application we use to view and operate on Puppet Enterprise. /var/log/puppetlabs/console-services/console-services.log This log contains console application logs. In particular you can see here: Autentication and user access attempts Notices about sync of deployed Puppet classes to display in the Node classifier Eventual application stack traces Use this log to troubleshoot issues related to access to web console or with class updates (when class deployed on the puppetserver environment dir are not visible on the console). You don’t see here details on the pages accessed by users. Format, location, and rotation of this log is configured in /etc/puppetlabs/console-services/logback.xml. /var/log/puppetlabs/console-services/console-services-access.log Similarly to the other access logs, this one contains http requests done to Puppet Console. It has the same combined log format and gives a better idea of the pages visited by the Console users. Format, location, and rotation of this log is configured in /etc/puppetlabs/console-services/request-logging.xml. /var/log/puppetlabs/console-services/console-services-api-access.log Similarly to the above one, this logs contains requests to the console API. In particular it’s possible to see here http requests related to Jobs and Classifier activities. It’s configured in /etc/puppetlabs/console-services/request-logging-api.xml. /var/log/puppetlabs/console-services/console-services_gc.log.0.current This log contains information about Garbage Collection on the JVM the PE Console is running on. What has been written about the other gc logs applies also here. Check https://puppet.com/docs/pe/2019.0/config_console.html for details on configuring and tuning the PE console. Nginx: /var/log/puppetlabs/nginx/ Directory /var/log/puppetlabs/nginx/ contains the logs of the NGINX webserver that acts as revers proxy to the PE Console. /var/log/puppetlabs/nginx/access.log Typical web server access logs, contains basically the same requests found in /var/log/puppetlabs/console-services/console-services-access.log with the difference that here is possible to see the clients’ IP (on console-services-access.log all the requests come from 127.0.0.1 which is this Nginx reverse proxy). /var/log/puppetlabs/nginx/error.log Error logs of the NGINX proxy. Generally not much to see here, check it if you want to troubleshoot issues about which there are no traces in other places. Puppet Enteprise Installer: /var/log/puppetlabs/installer/ The directory /var/log/puppetlabs/installer/ contains the logs produced by the Puppet Enterprise installer, used both to install and upgrade a PE instance. They basically contain the output of the puppet-enterprise-installer script, when executed. File names are based on the creation date. Orchestration Services: /var/log/puppetlabs/orchestration-services/ The directory /var/log/puppetlabs/orchestration-services/ contains the logs of PE orchestration service, responsible for managing and scheduling jobs (remote Puppet runs, jobs and plans) execution. /var/log/puppetlabs/orchestration-services/orchestration-services.log This log contains the orchestration service application logs. In particular you can see here: Minimal info about created job IDs Notices about deletion of old jobs and plans Eventual application services stack traces Use this log to troubleshoot issues related to application orchestration. Format, location, and rotation of this log is configured in /etc/puppetlabs/orchestration-services/logback.xml. /var/log/puppetlabs/orchestration-services/orchestration-services-access.log Similarly to the other access logs, this one contains http requests done to Puppet orchestration service. Format, location, and rotation of this log is configured in /etc/puppetlabs/orchestration-services/request-logging.xml. /var/log/puppetlabs/orchestration-services/orchestration-services-api-access.log Similarly to the above one, this log contains requests to the orchestration API. In particular, it’s possible to see here http requests related to Jobs and Classifier activities. It’s configured in /etc/puppetlabs/orchestration-services/request-logging-api.xml. /var/log/puppetlabs/orchestration-services/orchestration-services_gc.log.0.current This log contains information about Garbage Collection on the JVM the PE orchestration service is running on. What has been written about the other gc logs applies also here. PostgreSQL: /var/log/puppetlabs/postgresql/ The directory /var/log/puppetlabs/postgresql/ contains the logs of the PostgreSQL instance shipped with Puppet Enterprise. These are standard PostgreSQL logs. /var/log/puppetlabs/postgresql/pgstartup.log Generated at startup it just informs of the new log files locations. /var/log/puppetlabs/postgresql/postgresql-WEEKDAY.log The actual PostgreSQL log files, rotated on a weekly base. They contain information about: Connections and disconnections to PostgreSQL Indexing operations Checkpoint operations Eventual problems Refer to this file if you have problems of access between PuppetDB and PostgreSQL. CLIENT LOGS The following logs refer to all the Puppet agents, so they exist both on the Master and the clients. /var/log/messages /var/log/messages (or whatever log is used by syslog for normal system messages) contains the output of the Puppet runs. It’s the same output seen when running puppet agent -t from the command line or what is visible on the reports on the PE console /var/log/puppetlabs/pxp-agent/pxp-agent.log Is the log of the pxp-agent service with is running along with the puppet service on each node. This service handles communication with the pxp-broker on the Puppet Master through which jobs and remote puppet runs are triggered. In this log you can see information about: Tasks, remote puppet runs and plans triggerred as Jobs on the PER Console Maintenance notes about caches and temp dirs purging That’s enough about logs. Hope you survived. If you’d rather have someone else parse these logs when something looks wrong, that’s what Smart Puppet Support is for. So long and thanks for all the puppets. Alessandro Franceschi 2020-02-05T00:00:00+00:00 https://example42.com/blog/2020/02/05/managing-pe-services Puppet Enterprise (PE) is Puppet’s commercial offering. It’s based on the Open Source core and provides various enterprise features, like the powerful and utterly useful Console to manage and visualize the whole infrastructure from a Web interface. Puppet Enterprise can be configured as in All In One installation (AIO for short), where all the PE components are installed on a single node, or have them distributed on different ones. An AIO PE server runs the following services: pe-puppetserver. The core Puppet Server service responsible for communication with clients and compilation of their catalogs pe-puppetdb. The PuppetDB service, responsible for handling all the data produced by Puppet pe-console-services. The PE web interface we can access with a browser pe-nginx. An NGINX reverse proxy for the PE console pe-postgresql. A PostgreSQL instance where is stored the data generated, used or handled by PuppetDB and the PE Console pe-orchestration-services. Responsible for handling Puppet Jobs (such as Tasks, Plans and remote Puppet runs) All the Puppet clients (and also the Masters, which as clients of themselves) have the following services: puppet. The Puppet agent service, it requests the catalog from the server and applies it locally. Runs as root. pxp-agent. It’s used to allow the remote executing of Puppet runs, tasks and plans from the PE server. Let’s review these services in detail. Puppet Server service PuppetDB service Console Services service NGINX service Orchestration Services service PostgreSQL service Global checks Puppet Server service It’s the main Puppet server service, it takes care of: Receiving catalogs requests from clients Handling Puppet CA and clients’ certificates Compiling catalogs for the clients and sending them back Receiving Puppet runs reports from clients and submitting them to PuppetDB Receiving facts from clients and submitting them to PuppetDB (besides using them when compiling catalogs) Handling Puppet code and data deployments via the Code Manager component Syncing code between HA Master and Replica servers It’s a Closure application running as pe-puppet user inside a JVM. Its configuration files are under the directory /etc/puppetlabs/puppetserver. Its log files are under the directory /var/log/puppetlabs/puppetserver. It listens on port 8140 (used for the communication with clients) and 8170 (used by the Code Manager component). The process looks like this: pe-pupp+ 29018 1 5 2018 ? 2-02:00:52 /opt/puppetlabs/server/bin/java -Xms2048m -Xmx2048m -Djava.io.tmpdir=/opt/puppetlabs/server/apps/puppetserver/tmp -XX:ReservedCodeCacheSize=512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/puppetlabs/puppetserver/puppetserver_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=64m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar:/opt/puppetlabs/server/apps/puppetserver/jruby-9k.jar:/opt/puppetlabs/server/data/puppetserver/jars/* clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d --bootstrap-config /etc/puppetlabs/puppetserver/bootstrap.cfg --restart-file /opt/puppetlabs/server/data/puppetserver/restartcounter To check its status: systemctl status pe-puppetserver To stop and start the service (running inside a JVM the startup takes some seconds): systemctl stop pe-puppetserver systemctl start pe-puppetserver If PE is configured in High Availability mode, this service runs on the Primary Master and the Primary Master Replica servers, in case of failure on the Primary Master: Compile master component still works on the Replica (existing clients still can request, fetch, apply and report back their catalog) CA component is not able to provision new clients certificates (new nodes cannot be added) Code Manager can’t deploy new code (Puppet code and data can’t be updated) To re-establish full functionality, the Puppet server service has to run correctly on the Primary Master or the Primary Master Replica has to be promoted to Primary Master (and the existing Primary Master server must be decommissioned). In case of failure of the Puppet server service on the Primary Master Replica, all the above activities still work but in case of Code Manager deployments, new code is not synced to the Replica (it will be synced as soon as the Puppet server service is re-established there). PuppetDB service It’s the component that takes care of storing (using a PostgreSQL backend) all the data generated by a Puppet run: the list of facts of each node the last catalog compiled for a node all the reports of all the Puppet runs of all the nodes (old reports are regularly purged) Puppet exported resources It’s a Closure application running as pe-puppetdb user inside a JVM. Its configuration files are under the directory /etc/puppetlabs/puppetdb. Its log files are under the directory /var/log/puppetlabs/puppetdb. It listens on ports: 127.0.0.1:8080 (for http traffic) 0.0.0.0:8081 (for https traffic) It typically communicates only with the Puppet Server (and PostgreSQL for data storage). The process looks like this: pe-pupp+ 29254 1 0 2018 ? 05:00:11 /opt/puppetlabs/server/bin/java -Xmx512m -Xms512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/puppetlabs/puppetdb/puppetdb_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=64m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/puppetdb/puppetdb.jar clojure.main -m puppetlabs.puppetdb.main --config /etc/puppetlabs/puppetdb/conf.d --bootstrap-config /etc/puppetlabs/puppetdb/bootstrap.cfg --restart-file /opt/puppetlabs/server/data/puppetdb/restartcounterl To check its status: systemctl status pe-puppetdb To stop and start the service (running inside a JVM the startup takes some seconds): systemctl stop pe-puppetdb systemctl start pe-puppetdb This service runs in HA both on the Primary Master and the Primary Master Replica servers, in case of failure on the Primary Master the service is guaranteed by the Primary Master Replica. Console Services service It’s the web application that presents the Web interface for Puppet Enterprise. It’s a Closure application running as pe-console-services user inside a JVM. Its configuration files are under the directory /etc/puppetlabs/console-services. Its log files are under the directory /var/log/puppetlabs/console-services. It listens on ports: 127.0.0.1:4430 (Web application listens here in http, proxied by a local Nginx server which terminates https connections to port 443) 0.0.0.0:4431 (Web application over https) 127.0.0.1:4432 (Used for local status checks) 0.0.0.0:4433 (Node classifier / console services API endpoint) Puppet Server communicates with Console services over port 4433. The Nginx proxy communicates over port 4430 and serves clients (Users’s browsers) over port 443. The process looks like this: ppe-cons+ 19234 1 0 2018 ? 02:59:15 /opt/puppetlabs/server/bin/java -Xmx256m -Xms256m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/puppetlabs/console-services/console-services_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=64m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/console-services/console-services-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/console-services/conf.d --bootstrap-config /etc/puppetlabs/console-services/bootstrap.cfg --restart-file /opt/puppetlabs/server/data/console-services/restartcounter To check its status: systemctl status pe-console-services To stop and start the service (running inside a JVM the startup takes some seconds): systemctl stop pe-console-services systemctl start pe-console-services This service runs only on the Primary Master and if it fails there the web console is no longer available but classification via the node classifier API still works (catalogs are compiled normally and the classes defined via the Web Console for a node are normally included). To be able to use the Web console on the Replica, the Replica server has to be promoted. NGINX service It’s a normal Nginx instance that acts as a reverse proxy to the Web console. Its configuration files are under the directory /etc/puppetlabs/nginx. Its log files are under the directory /var/log/puppetlabs/nginx. It starts as root then forks the processes that communicate with clients as pe-webserver user. It listens on ports: 0.0.0.0:80 (http port, connections here are redirected to https port) 0.0.0.0:443 (https port, used by all the clients) The Nginx service communicates over port 4430 with the Console and serves clients (Users’ browsers) on port 443. The process looks like this: root 18533 1 0 2018 ? 00:00:00 nginx: master process /opt/puppetlabs/server/bin/nginx -c /etc/puppetlabs/nginx/nginx.conf pe-webs+ 18534 18533 0 2018 ? 00:00:59 nginx: worker process To check its status: systemctl status pe-nginx To stop and start the service: systemctl stop pe-nginx systemctl start pe-nginx This service runs only on the Primary Master and if it fails here the web console is no longer available for users. Orchestration Services service It manages orchestration services (the ability to trigger Puppet runs, tasks and plans from the web console or a cli command like puppet job) It’s a Closure application running as pe-orchestration-services user inside a JVM. Its configuration files are under the directory /etc/puppetlabs/orchestration-services. Its log files are under the directory /var/log/puppetlabs/orchestration-services. It listens on ports: 0.0.0.0:8142 (Used to accept inbound traffic and responses from client’s pxp-agents) 0.0.0.0:8143 (Used by PCP brokers and by orchestrator CLI client) All managed servers, via their local pxp-agent service, communicate using port 8142. When puppet job commands are used from the CLI (puppet job command run from management workstations or the same PE server) they communicate over port 8143. The process looks like this: pe-orch+ 18694 1 0 2018 ? 03:17:10 /opt/puppetlabs/server/bin/java -Xmx704m -Xms704m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/puppetlabs/orchestration-services/orchestration-services_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=64m -Djava.security.egd=/dev/urandom -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/puppetlabs/server/apps/orchestration-services/orchestration-services-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/orchestration-services/conf.d --bootstrap-config /etc/puppetlabs/orchestration-services/bootstrap.cfg --restart-file /opt/puppetlabs/server/data/orchestration-services/restartcounter To check its status: systemctl status pe-orchestration-services To stop and start the service (running inside a JVM the startup takes some seconds): systemctl stop pe-orchestration-services systemctl start pe-orchestration-services This service runs only on the Primary Master and if it fails here the orchestration services (ability to trigger remote Puppet runs or run tasks or plans) don’t work. To be able to use the Orchestration services on the Replica, the Replica server has to be promoted. PostgreSQL service It’s a normal PostgreSQL instance that stores all the data handled by PuppetDB, configurations done on the Web Console (such as node classification, rbac settings, auditing of users activities on the web interface) and the jobs operated by the Orchestration services. Databases and configurations are under /opt/puppetlabs/server/data/postgresql/11/data Its log files are under the directory /var/log/puppetlabs/postgresql. It runs as pe-postgres user. It listens on port: 0.0.0.0:5432 (Default PostgreSQL port) The PostgreSQL service communicates over port 5432 with PuppetDB, the Console services and the Orchestration services. The process looks like this: pe-post+ 30002 1 0 2018 ? 00:17:05 /opt/puppetlabs/server/apps/postgresql/bin/postgres -D /opt/puppetlabs/server/data/postgresql/9.6/data -c log_directory=/var/log/puppetlabs/postgresql pe-post+ 30003 30002 0 2018 ? 00:01:36 postgres: logger process pe-post+ 30005 30002 0 2018 ? 00:03:00 postgres: checkpointer process pe-post+ 30006 30002 0 2018 ? 00:00:59 postgres: writer process pe-post+ 30007 30002 0 2018 ? 00:03:34 postgres: wal writer process pe-post+ 30008 30002 0 2018 ? 00:05:02 postgres: autovacuum launcher process pe-post+ 30009 30002 0 2018 ? 00:10:11 postgres: stats collector process pe-post+ 30010 30002 0 2018 ? 00:00:01 postgres: bgworker: pglogical supervisor pe-post+ 30015 30002 0 2018 ? 00:00:05 postgres: bgworker: pglogical manager 16397 [...] pe-post+ 32574 30002 0 09:42 ? 00:00:00 postgres: pe-puppetdb pe-puppetdb 10.29.130.135(53026) idle pe-post+ 7622 30002 0 10:24 ? 00:00:00 postgres: pe-rbac-write pe-rbac 10.29.130.135(55262) idle pe-post+ 7697 30002 0 10:24 ? 00:00:00 postgres: pe-classifier-write pe-classifier 10.29.130.135(55274) idle pe-post+ 8174 30002 0 2018 ? 00:01:49 postgres: wal sender process pe-ha-replication 10.29.130.136(41438) idle [...] Several similar processes for the various databases To check its status: systemctl status pe-postgresql To stop and start the service: systemctl stop pe-postgresql systemctl start pe-postgresql This service runs on both the Primary Master and the Replica if one fails the other can keep on working. Global checks To check the overall health of the PE infrastructure, there’s the very handy puppet infrastructure status command. Its output looks as follows, in an HA setup: Notice: Contacting services for status information... Code Manager: Running on Primary Master, https://puppet01.example.com:8170/ File Sync Storage Service: Running on Primary Master, https://puppet01.example.com:8140/ File Sync Client Service: Running on Primary Master, https://puppet01.example.com:8140/ Puppet Server: Running on Primary Master, https://puppet01.example.com:8140/ Classifier: Running on Primary Master, https://puppet01.example.com:4433/classifier-api RBAC: Running on Primary Master, https://puppet01.example.com:4433/rbac-api Activity Service: Running on Primary Master, https://puppet01.example.com:4433/activity-api Orchestrator: Running on Primary Master, https://puppet01.example.com:8143/orchestrator PCP Broker: Running on Primary Master, wss://puppet01.example.com:8142/pcp PCP Broker v2: Running on Primary Master, wss://puppet01.example.com:8142/pcp2 PuppetDB: Running on Primary Master, https://puppet01.example.com:8081/pdb Info: Last sync successfully completed 65 seconds ago (at 2019-01-09T11:06:54.963Z) File Sync Client Service: Running on Primary Master Replica, https://puppet02.example.com:8140/ Puppet Server: Running on Primary Master Replica, https://puppet02.example.com:8140/ Classifier: Running on Primary Master Replica, https://puppet02.example.com:4433/classifier-api RBAC: Running on Primary Master Replica, https://puppet02.example.com:4433/rbac-api Activity Service: Running on Primary Master Replica, https://puppet02.example.com:4433/activity-api PuppetDB: Running on Primary Master Replica, https://puppet02.example.com:8081/pdb Info: Last sync successfully completed 79 seconds ago (at 2019-01-09T11:06:41.380Z) 2019-01-09 11:08:00 +0000 17 of 17 services are fully operational. Some of what has been described here for Puppet Enterprise applies also to the open source Puppet Server and PuppetDB services. In the next post, we are going to see where and how are all the PE related logs. For hands-on help keeping a PE deployment like this one healthy, see Smart Puppet Support. Stay awaken. Alessandro Franceschi 2020-01-21T00:00:00+00:00 https://example42.com/blog/2020/01/21/hiera-nested-lookups The Puppet data backend (hiera) allows you set data for site specific differences in your infrastructure. For example you can have multiple datacenters where DNS, NTP, SNMP and backup are different, or you have a development stage which uses a different database server than the production stage. Usually you check your infrastructure for differences and then use Facter data to build layers of Hiera data. But what if you need the same data in several hiera keys? Fot example you want to set the db connection settings like user/password for databases and webservers. This is where we can make use of nested hiera lookups. Hiera nested lookups The “lookup” and “hiera” lookup The “alias” lookup The “literal” lookup The “scope” lookup Hiera nested lookups Hiera lets you run an additional hiera lookup inside of hiera data. This allows you to specify shared information once only, instead of adding the same information several times: An example: profile::app::db::auth 'app1': user: 'app1' password: '$\.fgeetd' profile::app::web::auth: 'app1': user: 'app1' password: '$\.fgeetd' The “lookup” and “hiera” lookup Instead of managing the same information on several places, we can ask hiera to run another lookup to fetch the required data: profile::app::db::auth 'app1': user: "%{lookup('app1_user')}" password: %{lookup('app1_user_pass')}" profile::app::web::auth: 'app1': user: "%{lookup('app1_user')}" password: %{lookup('app1_user_pass')}" app1_user: 'app1' app1_user_pass: '$\.fgeetd' Instead of “lookup” you can also specify “hiera”. Please note that “lookup” and “hiera” nested lookups will only return string based values. If you specify a non existing key, hiera will fail and return an error. You can have multiple layers of nested lookups. That means that you run a lookup on a key, which does again run lookup. You have to be aware that Hiera will fail, if you build loops. Hiera will detect these and return an error. The “alias” lookup What if you don’t need string, but other data types like boolean, array or hash to be returned by a nested lookup? In this case you can use the “alias” lookup: profile::app::db::default_packages: "%{alias('default_packages')}" profile::app::web::default_packages: "%{alias('default_packages')}" default_packages: - 'tree' - 'net-utils' Please note, that you can not add additional data to an “alias” lookup. The “literal” lookup Consider the situation in which you donot like hiera to interpolate the percent (%) sign. In this case you can use the “literal” lookup: profile::app::web::server_name_string: "%{literal('%')}{SERVER_NAME}" This will return the value %{SERVER_NAME}. The “scope” lookup The “scope” function interpolates variables. The following two examples are identical: profile::app::web:stage: "%{facts.app_stage}" profile::app::web:stage: "%{scope('facts.app_stage')}" As you also use the simple interpolation, the “scope” lookup is not really needed. Happy hacking on hiera, Martin Alfke 2019-12-24T00:00:00+00:00 https://example42.com/blog/2019/12/24/HDM-released Merry christmas to everybody. We are super excited to announce today the general availability of HDM (Hiera Data Manager) - a webfrontend for vizualising and managing Hiera data. HDM Requirements Upcoming features Coming soon: HDM Pro Community HDM We learned from several customers, that most of them have specifc requirements when it comes to managing application data. Data must be able to be modified by a person which: has no knowledge on Git has no knowledge on Puppet only runs Windows on Workstation Therefore we decided to develop HDM. HDM starts with letting you select a desired environment where you want to check or change data. Next you can select a node. We query PuppetDB to find existing environments, nodes and their facter values: Now you can see all hiera keys a node has within the environment hiera data: When selecting a key we show the hierachies and vizualize whether a hierarchy has data for a key and which one is the default: You can now change data on node level. HDM writes the data back to a file. Requirements In the actual state, HDM must run on the Puppet Master. We fetch environments, nodelist and facts from PuppetDB. HDM needs a file structure where it con store modified data. This file structure must be added to your hiera.yaml configuration file. Upcoming features Within the next releases we want to: add read-only feature provide a Puppet module to install, configure and run HDM optimize the web view have HDM run on a separate machine run HDM as a container Coming soon: HDM Pro The Open Source implementation has no login and no access control. This is a feature we are putting into HDM Pro, our commercial release of HDM. Another planned HDM Pro features is a git based storage backend with review capabilities. Community We are looking forward to learn about issues, missing features or any other feedback. Best, the team of example42. 2019-12-19T00:00:00+00:00 https://example42.com/blog/2019/12/19/request-for-tinydata-part4 Post four of four about our Request for Tiny Data! Previously we have: Part 1 - Introduced Tiny Puppet and where it can be used Part 2 - Described how to write Tiny Data Part 3 - Shown special tiny data to handle package repositories, configuration files validation and usage within containers Now let’s add a few other info and raise the final call! Tiny Puppet defaults Did we mention that Tiny Puppet is smart enough to do something also when it has no Tiny data for an application? In this case it simply tries to declare a resource like: package { 'unknown_app': ensure => present, } Automatically adding the right package provider, like chocolatey on Windows and homebrew on MacOS. So, even if there’s no tidy data for, example, opera, something like this would work for Linux, Windows and MacOS: tp::install { 'opera': } When there’s no tiny data anyway the tp::conf define can’t be used. tp command line One of the free side benefits of using Tiny Puppet is that optionally you can install the tp shell command (powershell version still not available) to interact with the apps you manage, and have at disposal commands as the following. To just test that everything is fine: tp test To test a specific app (can be used in shell sessions, CI, monitoring, puppet tasks, motd… ) tp test gitlab-ce To tail -f all the known logs of the apps managed with tp: tp log To limit to the logs of an application tp log nginx To quickly install an application caring of repositories tp install elasticsearch To actually list the applications for which is available tiny data (based on latest version of the module on the forge): tp list Request for Tiny Data! So, here is our renovated call for tiny data. We have tinydata for some applications: ls -la data/ | wc -l 179 the common ones or what we needed or found interesting. Still there’s more. A lot of wonderful applications that would be great to be able to install on a shell command: tp install wonderapp or manage with a Puppet define: tp::install { 'wonderapp': } On any Linux, and maybe Mac and Windows. With the quick choice of using the default OS packages, the app upstream repo or any other repo might be configured. tp::install { 'wonderapp': upstream_repo => true|false, } We know we can add new data very easily, and relatively quickly. We don’t know what application interests you. Please engage with, in effort order: Let us know, in any way (tweet, comment, mail, voice) what app you would like to quickly manage via tp Open a ticket on Github for a new app support. Possibly provide context and relevant information Open a ticket for incorrect, incomplete or not updated existing tiny data Do directly the work with updated tinydata and submit a Pull Request Our goal is to tp install everything on any OS ;-) Now let’s define everything, together. 2019-12-16T00:00:00+00:00 https://example42.com/blog/2019/12/16/request-for-tinydata-part3 Here is the is the third part of a blog series on our Request for for Tiny Data. Previously we have: Part 1 - Introduced Tiny Puppet and where it can be used Part 2 - Described how to write Tiny Data Here we are going to reveal some new and not so new features of Tiny Data: Managing additional repositories There 2 ways to are manage repos, the first is to specify the typical data to use in apt repos: elasticsearch::settings: init_file_path: '/etc/default/elasticsearch' repo_url: 'http://packages.elastic.co/elasticsearch/2.x/debian' key: 'D88E42B4' key_url: 'https://packages.elastic.co/GPG-KEY-elasticsearch' apt_repos: 'main' apt_release: 'stable' apt_key_server: 'http://pgp.mit.edu' or yum repos: elasticsearch::settings: init_file_path: '/etc/sysconfig/elasticsearch' repo_url: 'http://packages.elastic.co/elasticsearch/2.x/centos' key: 'D88E42B4' key_url: 'http://packages.elastic.co/GPG-KEY-elasticsearch' the second, when repo_package_url is defined, involves setting the download url of the release package, with all the necessary repository configurations: puppet::settings: repo_package_url: 'https://yum.puppet.com/puppet/puppet-release-el-7.noarch.rpm' Managing application upstream repositories We have recently added to the tp::install define a very powerful parameter: upstream_repo, which allows users to install an app from its own upstream repositories. So, if you want to install a package using the native OS packages, you simply can have a manifest with: tp::install { 'puppet': } but if you want to install the same application using the upstream Puppet repositories, provided by the same application authors, you can write: tp::install { 'puppet': upstream_repo => true, } All the tinydata necessary and specific to the upstream repo packages, in placed in (for this case with puppet) the data/puppet/upstream directory. This is a new feature and we currently have very few application with upstream data info. Managing checks on configuration files Puppet has a not much known feature for the file resource: the argument validate_cmd which allows to check, with the command passed as argument, the syntax of the file we are managing before changing it on the system. Almost no module uses this feature, Tiny Puppet does it, potentially for any application, in reality for any application for which there’s the relevant Tiny Data (and, yes, you can help us in improving Tiny Data). This is a partial set of tinydata for apache: --- apache::settings: package_name: 'httpd' service_name: 'httpd' config_file_path: '/etc/httpd/conf/httpd.conf' config_dir_path: '/etc/httpd' tcp_port: '80' pid_file_path: '/var/run/httpd.pid' log_file_path: [ '/var/log/httpd/access.log' , '/var/log/httpd/error.log' ] log_dir_path: '/var/log/httpd' data_dir_path: '/var/www/html' process_name: 'httpd' process_user: 'apache' process_group: 'apache' nodaemon_args: '-DFOREGROUND' validate_cmd: config: 'httpd -t -f %' Note the hash for validate_cmd, where we can use different commands for different kind of files (here we set the command to check for the file defined by **config**_file_path ). As everything in Tiny Data, settings can be overridden for different OS, so for Debian OS family, the above data is overridden by: --- apache::settings: package_name: 'apache2' service_name: 'apache2' config_file_path: '/etc/apache2/apache2.conf' init_file_path: '/etc/default/apache2' config_dir_path: '/etc/apache2' mods-available_dir_path: '/etc/apache2/mods-available' mods-enabled_dir_path: '/etc/apache2/mods-enabled' sites-available_dir_path: '/etc/apache2/sites-available' sites-enabled_dir_path: '/etc/apache2/sites-enabled' conf-available_dir_path: '/etc/apache2/conf-available' conf-enabled_dir_path: '/etc/apache2/conf-enabled' conf_dir_path: '/etc/apache2/conf.d' pid_file_path: '/var/run/apache2.pid' log_file_path: [ '/var/log/apache2/access.log' , '/var/log/apache2/error.log' ] log_dir_path: '/var/log/apache2' data_dir_path: '/var/www' process_name: 'apache2' process_user: 'www-data' process_group: 'www-data' validate_cmd: config: 'apache2 -t -f %' That’s it, when you use tp::conf { 'apache': } you have, out of the box, automatic validation of the syntax of your Apahce (main) configuration file. Run app in container In the above data you might have noticed the nodaemon_args setting. It’s used to specify what argument has to be used to run the application in foreground. Yes, you are guessing right, we use it to build Docker images where app has to be launched in foreground and not as a background service. In the past we engineered a process to build images using Tiny Puppet, without having Puppet installed on the target image! There are fragments of such works in our psick control repo and other collateral projects but, we must admit, works here have been stale for long. Request for Tiny Data! We would love to add upstream_repo support by default to all our apps and add as much validate_cmd and nodaemon_args settings as possible. Still we have to know know what to prioritize, and you can help with that, in many ways: Let us know, in any way (tweet, comment, mail, voice) what app you would like to quickly manage via tp Open a ticket on Github for a new app support. Possibly provide context and relevant information Open a ticket for the applications missing upstream repo data you would like. Do directly the work with updated tinydata and submit a Pull Request 2019-12-12T00:00:00+00:00 https://example42.com/blog/2019/12/12/request-for-tinydata-part2 This is the second of four post series for our call for Tiny Data. In the first post we introduced Tiny Puppet (tp), now we are going to talk about what feeds it and what all this is about. Tiny data Tiny Puppet actually has a, non intrusive, dependency: the tinydata module. Here is where the tp magic becomes plain information easy to read, fix, and improve. Tinydata contains the info on package, service, files names, paths for any application supported by Tiny Puppet. For example, openssh default tiny data looks as follows: --- openssh::settings: package_name: 'openssh-server' # The name (can be empty or an array) of the package to install service_name: 'ssh' # The name (can be empty or an array) of the service to manage config_file_path: '/etc/ssh/sshd_config' # What you configure with a default tp::config { 'openssh': ... } config_file_mode: '0600' # Mode of the main configuration file config_dir_path: '/etc/ssh' # Path of the main configuration dir tcp_port: '22' # Listening port. This can be used for automatic monitoring pid_file_path: '/var/run/sshd.pid' # Can be used for automatic monitoring log_file_path: '/var/log/messages' # Used by the tp log command process_name: 'sshd' # Can be used for automatic monitoring process_user: 'root' process_group: 'root' nodaemon_args: '-D' # Used when starting the app in a container validate_cmd: 'sshd -t -f %' # If present, the syntax of the config file is automatically validated before change But there are variations for Debian and Derivatives: openssh::settings: config_file_mode: '0644' init_file_path: '/var/default/ssh' # The path of the 'init' file log_file_path: '/var/log/syslog' or Solaris: openssh::settings: package_name: '' service_name: 'ssh' config_file_mode: '0600' log_file_path: '/var/adm/authlog' Adding tinydata The current list of applications support by Tiny Puppet is basically the list of directories in the data directory, in each of these directories, there’s a hiera.yaml file which configures the hierarchy that tp has to use to look for that application tinydata. For example, a common hierarchy, used also for the openssh application, is: --- :hierarchy: - "%{title}/osfamily/%{osfamily}/%{operatingsystemmajrelease}" - "%{title}/osfamily/%{osfamily}" - "%{title}/default" - "default/%{operatingsystem}" - default This tells tiny puppet in what files to look for tinydata starting from: OSfamily files specific for the app, here in data/openssh/osfamily Application defaults data/openssh/default.yaml OS specific general data in data/default/ the defaults in data/default.yaml Lookup is an hiera like (note that Hiera is not actually used to get his data): first value found while crossing the hierarchy has precedence on values found, for any key, at lower hierarchy levels. Tiny data can currently define: packages to install, services and the typical configuration files to manage date for additional software repositories to configure before trying to install the relevant application’s package how to launch the application in a docker instance what command to use to validate the syntax of the app configuration files what ports, pids, users are used with the application, used for monitoring and availability checks Request for Tiny Data! You see? Writing tiny data is not difficult, and it’s very fast once you know these principles. If you want to improve existing data or have data for new applications, you can: Let us know, in any way (tweet, comment, mail, voice) what app you would like to quickly manage via tp Open a ticket on Github for a new app support. Possibly provide context and relevant information Open a ticket for incorrect, incomplete or not updated existing tiny data Do directly the work with updated tinydata and submit a Pull Request In any case we will try to give our example42 answer. 2019-12-09T00:00:00+00:00 https://example42.com/blog/2019/12/09/request-for-tinydata-part1 This is the first of four post series for our Request for Tiny Data. We will clarify better at the end who can request what, first, we have to give some context, and explain what uses Tiny Data: Tiny Puppet (tp) If you know something about example42, you should know that we developed Tiny Puppet (tp), a Puppet module which allows to manage potentially any application on any Operating System. What applications exactly Tiny Puppet can manage? The current list is always this, but the right answer is: Any application that can be installed via a Puppet package resource, taking care, where necessary, of the configuration of the relevant package respository. On What Operating Systems? Mostly Linux (RedHat, Debian, Suse and derivatives) But also Solaris, BSDs and Darwin (with brew-cask) And potentially also Windows (with Chocolatey). What do I need to achieve this? Puppet, the Tiny Puppet module and, guess what, the app tinydata. What you can use it for? Shell usage: Use the tp command to quickly install, test, show logs of the managed applications Puppet manifests usage: Use tp::install, tp::dir and tp::conf defines to manage applications installation and configurations. Usage in shell Tiny Puppet is born and expected to be used in Puppet manifests, but it can actually work standalone, as a cli command. It can be installed, via Puppet, with: sudo puppet module install example42-tp sudo puppet tp setup After this we have at disposal the tp command, that we can use to install something: sudo tp install sysdig sudo tp install opera sudo tp install puppetdb We can also check if all the resources we installed via tp are running well: tp test tp test apache Or show the live logs of all or the selected applications: tp log tp log nginx Usage in Puppet manifests The tp module provides the following defines: tp::install. It installs an application and starts its service, by default tp::conf. It allows to manage configuration files related to the app, handling dependencies tp::dir. Manages the content of directories tp::test. Allows quick and easy checking of the status of the application tp::repo. Manages extra or upstream repositories for the application package We can use them in Puppet manifests like: Local site profiles, with our code, our files configured with our logic Possibly, optionally, in component modules, to benefit of tp cli integrations and OS coverage Ready for Hiera use profiles collections, like the tp-profile or the psick module. Just to give you an idea, the following code: class profile::openssh ( String $template = 'profile/openssh/sshd_config.erb', Hash $options = {}, ) { tp::install { 'openssh': } tp::conf { 'openssh': template => $template, options_hash => $options, } # Alternative which does the same: # tp::conf { 'openssh': # content => template($template), # } } will install the package, configure the file with the contents we want, manage the service (taking care of dependencies and different names and paths) for openssh. The example used here for openssh can be done virtually for all applications you can think about (for which there’s a package to install and the right tinydata). This can be useful when we know how to configure our application, and we want a quick way to puppettize it without getting lost in finding the right component modules with its bunch of dependencies, in a way the allows us to concentrate just on the data to customise, which, for the above example, could be Hiera data as follows: profile::openssh::template: profile/openssh/sshd_config.erb profile::openssh::options: Protocol: 2 PermitRootLogin: 'no' UsePAM: 'yes' TCPKeepAlive: 'yes' Content of the template, to be placed in our profile module, could be something like what follows (plus all the necessary extra settings we may want to hard-code): # File managed by Puppet <% @options.each do |k,v| -%> <%= k %> <%= v %> <% end -%> Request for Tiny Data So, this is Tiny Puppet, and anything want to manage with it we need its Tiny Data. Is there any app that you would like to easily install and configure (via Tiny Puppet)? Let us know, in any way (tweet, comment, email, voice…). We will give an example42 answer. 2019-04-04T00:00:00+00:00 https://example42.com/blog/2019/04/04/puppet_sensitive_data Securing important and sensitive information with Puppet is a long time issue. There are at least three different locations where one has to deal with securing information. The most well known is the Hiera data store, where many people today use hiera-eyaml to encrypt values using a public-private key pair. The next one is the catalog itself, where the Puppet server places data unencrypted inside. Ben Ford from Puppet provided a solution using node_encrypt module. The third one is the Puppet report. Here we see file diffs, showing old and new password. This is the topic we are dealing in today’s Puppet Tip. Let’s start with the data type: class profile::db ( Sensitive $password, ){ file { '/etc/.db_password': ensure => file, content => $password, } } Within hiera one specifies the key-value pair: --- profile::db::password: ENC[PKCS7,Y22exl+O...] But on the next Puppet agent run, we will receive an error message: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Class[Profile::Db]: parameter 'password' expects a Sensitive value, got String (file: /etc/puppetlabs/code/environments/production/manifests/ site.pp, line: 31, column: 3) on node master.example42.training We need to inform hiera, that we want to receive the value as Sensitive data type by adding a lookup option: --- lookup_options: profile::db::password: convert_to: "Sensitive" profile::db::password: ENC[PKCS7,Y22exl+O...] Any data which is based on Sensitive data type, will not be shown in a catalog diff: Notice: /Stage[main]/Profile::Db/File[/etc/.db_password]/ensure: changed [redacted] to [redacted] example42 wishes everybody safety and success Martin Alfke 2019-02-18T00:00:00+00:00 https://example42.com/blog/2019/02/18/partner_awards When we joined as individuals the Puppet community almost ten years ago, we learned that this community was different. The most important drive here has always been: try to be helpful and inclusive! Shortly after example42 GmbH was founded, in October 2015, we were accepted as Puppet Channel Partner. This partnership allows Puppet Inc and example42 GmbH to work closely together on training, consulting, migrations and any aspect of a Puppet Enterprise installation. Several of the relevant engagements have been conducted by Alessandro Franceschi, CTO at example42 GmbH. We are super excited that Alessandro is rewarded as Channel Partner Engineer of the Year for “exemplifying excellence in engineering and solving customer challenges using Puppet products”. Additionally example42 GmbH is rewarded as Service Delivery Partner of the Year in EMEA region, “driving additional services, regular availability, and bench growth”. Find more information at Puppet blog posting - Puppet Announces 2018 Channel Partner Excellence Awards What is a Puppet Channel Partner? How does a Service Delivery Partner adds value? Why example42 GmbH? What is a Puppet Channel Partner? A Puppet Channel Partner supports Puppet in being present at market and engages in Puppet community. How does a Service Delivery Partner adds value? When using the supported Puppet Enterprise Platform, customers need to ensure that any work inside the platform is done in accordance to Puppet guidelines. A Service Delivery Partner must have at least one Puppet Certified Consultant and support Puppet Enterprise customers in all technical aspects of their Puppet platform. Why example42 GmbH? At example42 GmbH two long time Puppet experts have combined their Puppet knowledge, each one having more that ten years of hands-on Puppet experience. Meanwhile we participate in a world-wide network of Puppet experts and contribute to the community with our Open Source modules and Puppet related projects. We are happy and proud of being part of this world. example42 GmbH Puppet DevOps Automation Martin Alfke / Alessandro Franceschi 2019-01-07T00:00:00+00:00 https://example42.com/blog/2019/01/07/lazy_tags In Puppet trainings I usually tell people that a lazy admin is a good admin. A lazy admin will automate repetitive tasks. Additionally I like to refer to Puppet as also being lazy: Puppet always checks the actual state and compares it with the desired state. In case that actual state is correct, Puppet will go to sleep again. But there are parts where being lazy is kind of a problem. One of these is what I prefer to name “the lazy tagging” in Puppet. What are tags? Tags and collectors Work around the lazy tagging issue What are tags? Puppet offers the possibility to add arbitrary names to resources or classes. Let’s have a look at some examples: You can add tags to resource declarations using meta parameter: package { 'foobar': ensure => present, tag => 'barfoo', } Or you can have a class and you add an additional name to it using the tag function: class foobar { tag('barfoo') } For classes Puppet also does automatic tagging using the class name. class profile::database::mysql { tag('profile::database::mysql') # not needed, done by Puppet internally } These tags can then be used to limit the resources the Puppet agent is checking and managing by using the --tag cli option: puppet agent --test --tags=barfoo The agent will still receive the whole catalog of resources for a node. But the agent will only work on resources with the provided tags, which may be a comma separated list. Tags and collectors When using virtual resources and collectors, one might think, that the tags are usable also during catalog compilation. But this is not true, as tags are usually handled by the Puppet agent, not the server. Let’s assume that you want to manage users locally. You have some people which always get access (the sysadmins) and you have staff being responsible for specific applications. Maybe some of your people manage more than one application. First you consider placing all the user information into hiera and you tag the individual users: # data/common.yaml profile::usermanagement::users: 'ben': uid: '1044' shell:'/bin/zsh' tag: - 'sysadmin' 'bob': uid: '1045' shell: '/bin/bash' tag: - 'sysadmin' 'rob': uid: '1046' shell: '/bin/bash' tag: - 'app3b' - 'billing2' 'fin': uid: '1047' shell: '/bin/bash' tag: - 'proxy' - 'billing2' Now you want to create users based on the node role. Let’s assume that we have the following roles: app3b billing2 proxy In this case you might consider using virtual resources and declare/collect only users which have a node role tag set (and of course all of your sysadmin people). class profile::usermanagement ( Hash $users = {}, ) { $users.each |String $key, Hash $value| { @user { $key: * => $value, } } User <| tag == $::role or tag == 'sysadmin' |> } But what will happen? Puppet will always create ALL users which you listed in hiera. The reason is that the lazy tagging gets evaluated on the Puppet agent and not on the Puppet master. Work around the lazy tagging issue Using Lambdas will allow you to run a function which is executed at the compiler to remove unneeded users. Here we can use the filter function: class profile::usermanagement ( Hash $users = {}, ) { $sysadmins = $users.filter |$key, $value| { 'sysadmin' in $value['tag'] } $roleadmins = $users.filter |$key, $value| { $::role in $value['tag'] } $all_admins = $sysadmin.merge($roleadmins) $all_admins.each | $key, $value | { user { $key: * => $value, } } } As you have removed all users you don’t want to have on systems using the filter function, you can now directly declare the users instead of using virtual resources and collectors. example42 wishes everybody a healthy and successful new year. Martin Alfke 2018-12-31T00:00:00+00:00 https://example42.com/blog/2018/12/31/2018_categorized_posting_summary Welcome to 2019! We close the year with a categorised summary of all postings from 2018 and a rethinking of the purpose of this blog. After exactly 2 years of weekly posts about Puppet we have decided to quit with the weekly schedule. We will continue to release Puppet Tips here, but not every week. The ratio between time spent in writing the posts and feedback from readers and the Puppet world has not been positive enough to justify the effort. We apologise to who has always supported and helped us in spreading these contents (special thanks to Taciano Tres (@tacianot) for the continuous help in this) and we hope we will still be able to provide you useful contents in the future. Now, let’s see what has been written about in 2018. As similar post was done for 2017 check it out for a full list of all our “Tips of the Week”. Awards PSICK, Control-Repository, Tiny Puppet and example42 Modules DevOps,Containers, Strategies Puppet Tutorials and Best Practices Puppet 6 Bolt, tasks, plans Puppet Code Development and Testing Hiera, Razor Awards example42 is rewarded as one of the Top10 DevOps consulting companies in Europe Puppet Community Award PSICK, Control-Repository, Tiny Puppet and example42 Modules Classification and First Run mode with PSICK Running acceptance tests on different Puppet versions with PSICK How to use PSICK module for common profiles Anatomy of a Puppet control-repo compared to PSICK example42 Puppet modules status update Update on example42 Puppet modules A few steps to Tiny Puppet on the command line tp install anything (anywhere),and configure Custom applications management using Tiny Puppet PSICK profiles, Part 1 - Overview PSICK profiles, Part 2 - Setting proxy, server and hostname PSICK profiles, Part 3 - Managing OpenSSH PSICK profiles, Part 4 - Managing users PSICK profiles, Part 5 - Managing /etc/hosts and DNS DevOps,Containers, Strategies DevOps and the permaent learning pattern A wholistic DevOps approach Configuration Management and Containers The downside of server naming conventions Puppet Tutorials and Best Practices Puppet Control Repo change impact scenarios The example42 Puppet cheat sheet example42 Puppet Tutorial - Part 1 example42 Puppet Tutorial - Part 2 example42 Puppet Tutorial - Part 3 example42 Puppet Tutorial - Part 4 Puppet custom data types What you need to know about Puppet facts. Part 1 - Core facts What you need to know about Puppet facts. Part 2 - Custom facts What you need to know about Puppet facts. Part 3 - External facts What you need to know about Puppet facts. Part 4 - Trusted facts What you need to know about Puppet facts. Part 5 - facter.conf Facter fork bombs and timeout - what are they, how to prevent them Puppet node classification Puppet Library Module synchronization Managing packages on MacOSX with Puppet and HomeBrew The FIVE things you need to know when starting with Puppet Roles and Profiles explained Puppet 6 What’s new with Puppet 6? Upgrading to Puppet 6 CA Bolt, tasks, plans All the ways to remotely trigger Puppet runs (with or without Puppet tasks Puppet Code Development and Testing Using catalog diff to check refactoring Testing Puppet Enterprise 2018.1 How to adopt the Puppet Developer Kit (PDK) to your code Editorconfig Hiera, Razor Migrate from Hiera v3 to Hiera v5 Introduction to Razor example42 wishes everybody a healthy and successful new year. Martin Alfke, Alessandro Franceschi 2018-12-17T00:00:00+00:00 https://example42.com/blog/2018/12/17/psick_profiles_part_5_hosts_and_dns Our tour of the ready to use profiles of the psick module continues this week with management of DNS entries for the resolver, and of the content of /etc/hosts file. These are the previous posts of this series: Part 1 - Overview of the psick module and its reusable profiles. Part 2 - Proxy and Hostname settings with psick profiles. Part 3 - OpenSSH settings, keys, configs management. Part 4 - Users management. Managing /etc/hosts Psick module provides 3 different classes to manage the contents of /etc/hosts: psick::hosts::file: Manage /etc/hosts via a file resource psick::hosts::dynamic: Manage /etc/hosts dynamically using exported resources psick::hosts::resource: Manage /etc/hosts content via Puppet native host resource type Using psick::hosts::file This class just manages /etc/hosts as a file, you can customise the template to use for the content of this file and an array of entries to add to it. To use this approach, just classify your nodes with with profile: include psick::hosts::file Then it can be configured via Hiera with the following settings: The erb template to use to manage the content of /etc/hosts (default value is psick/hosts/file/hosts.erb): psick::hosts::file::template: profile/hosts/hosts.erb The ip address (default is the value of $::psick::primary_ip which defaults to the fact $::networking['ip']), the short hostname (defaults to fact $::hostname) and the domain (defaults to fact $::domain) to use to identify the local node in /etc/hosts: psick::hosts::file::ipaddress: 10.12.13.14 psick::hosts::file::domain: example42.com psick::hosts::file::hostname: my_server An array of custom extra lines to add to /etc/hosts (default: []), each element of the array should contain the expected text in each extra line: psick::hosts::file::extra_hosts: - 10.12.13.15 puppet puppet.example42.com - 10.12.13.20 other_server other_server.example42.com Using psick::hosts::dynamic This class manages /etc/hosts automatically and dynamically: each server managed by Puppet exports its own host entry (via Puppet host resource) and collects the ones of all the other nodes. The class provides options to customise IP and alias to export for a node, if to actually export the host entry and a special “magic var” which can be used to divide nodes in different groups (within each group hosts’ entries are exported and collected). This class is alternative to the others, it requires Store configs enabled on the Puppet Server (so, consequently, the usage of PuppetDB in not too ancient setups) and can be used with a simple: include psick::hosts::dynamic In small setups this could be enough to have /etc/hosts automatically managed with all the entries of all the nodes. It’s possible anyway to customise some entries. For example the ip address (default $::ipaddress) and the array of aliases (default [ $::hostname ] to use when exporting the local host’s info): psick::hosts::dynamic::dynamic_ip: 10.12.13.14 psick::hosts::dynamic::dynamic_alias: - my_server - my_server.example42.com - my_server_other_alias It’s also possible to control if and how to export and collect the node’s host entry. It’s possible to set a string that allows to group together nodes: all nodes having this magic var set collect and export host resources only for nodes using the same magic var: psick::hosts::dynamic::dynamic_magicvar: intranet If we don’t want to collect in any place the host entry of a node we can set, for it (default value is false, so each node exports a valid and collectable host resource): psick::hosts::dynamic::dynamic_exclude: true Since /etc/hosts entries are managed via exported host Puppet resources, it’s also possible to specify an Hash of custom additional entries to add to the host file. This makes sense if we want to add references to hosts or devices not managed by Puppet. The syntax of the hash to use maps the available arguments of the host resource (puppet describe host for a full list): psick::hosts::dynamic::extra_hosts: firewall.example42.com: ip: 10.12.13.1 target: /etc/hosts # (Default) host_aliases: - firewall - fw.example42.com san.example42.com: ip: 10.12.13.250 host_aliases: - san Using psick::hosts::resource This is the third alternative to manage /etc/hosts. It’s just a wrapper that exposes an Hiera controllable entry point for Puppet host resource. This is alternative to the previous profiles and has to be classified as the others: include psick::hosts::resource Configuration the is done via an Hash of hosts resources similar to what we have seen for psick::hosts::dynamic::extra_hosts: psick::hosts::resource::hosts: firewall.example42.com: ip: 10.12.13.1 target: /etc/hosts # (Default) host_aliases: - firewall - fw.example42.com san.example42.com: ip: 10.12.13.250 host_aliases: - san Managing DNS resolver. To manage the contents of /etc/resolver you can use the psick::dns::resolver profile. Classify it with something equivalent to : include psick::dns::resolver And then configure on Hiera both the template to use and its entries. To manage the template to use (default is ‘psick/dns/resolver/resolv.conf.erb’) and the actual path of the resolver file (default is ‘/etc/resolv.conf’): psick::dns::resolver::resolver_path: /etc/resolv.conf psick::dns::resolver::resolver_template: profile/resolver/resolv.conf.erb To manage the actual typical configuration settings of resolv.conf: psick::dns::resolver::nameservers: - 1.1.1.1 - 8.8.8.8 psick::dns::resolver::options: - attempts: 2 - timeout: 2 - rotate psick::dns::resolver::search: - example42.com - lab.example42.com psick::dns::resolver::domain: example42.com psick::dns::resolver::sortlist: - 130.155.160.0/255.255.240.0 - 130.155.0.0 That’s all for today. As we have seen, for the most common use cases, on Linux systems, Psick provides ready to use profiles to manage the content of /etc/hosts and /etc/resolv.conf without the need of fetching dedicated modules. Have fun with Puppet, Life, Universe and Everything. Alessandro Franceschi 2018-12-10T00:00:00+00:00 https://example42.com/blog/2018/12/10/psick_profiles_part_4_users We continue our review of psick profiles describing how psick::users can help in configuring users, root password, sudo and ssh keys. This comes after: Part 1 - Overview of the psick module and its reusable profiles. Part 2 - Proxy and Hostname settings with psick profiles. Part 3 - OpenSSH settings, keys, configs management. For the following configuation we just need to classify for psick::users class: include psick::users As usual the following hiera sample parameters are expressed in yaml format. Managing root user We can set password of root user with: psick::users::root_pw: '$6$OkmG4mu/$RbyL' The value is passed as password argument for the user root, in whatever encrypted format the local system requires. It’s possible to configure any other argumente of the root user with the parameter root_params: psick::users::root_params: comment: 'Root user' shell: '/bin/bash' Managing all users For all the other users there’s the users_hash parameter, looked up in deep merge mode (as any parameter of psick classes whose name ends with _hash), which allows to define one or more users with the relevant set or arguments. This is not just a wrapper around the user resource type, as we can use component modules as “backends”. This is managed via the psick::users::module parameter which define which module to use to manage users: ‘user’ to use Puppet native type user ‘psick’ to use the define psick::users::managed ‘accounts’ to use accounts::user from puppetlabs-accounts module So for example we can manage our users, using Puppet native type user, with: psick::users::module: 'user' psick::users::users_hash: al: ensure: present comment: 'Al' groups: - users ma: ensure: present comment: 'Ma' groups: - users In the above example the users “al” and “ma” are created, the parameters specified in the hash must be compatible with resource provided by the selected module, otherwise you get an Unknown Argument error. ### Managing extra users resources In the users hash, besides the arguments acceptable by the selected module define, we can set other special parameters to manage resources for users: ssh_authorized_keys: An array of keys to add to the user’ authorized keys openssh_keygen: A boolean, if true a ssh key pair is automatically generated for the user using the psick::openssh::keygen define check here the available parameters sudo_template: The path as used by the template() function of an erb template to use to manage the user’s sudo file. Nothing is created if not defined. psick::users::users_hash: al: ensure: present ssh_authorized_keys: - ‘ssh-rsa AAAAB3BAQC93uOkdIr…’ sudo_template: ‘profile/users/sudo/admins’ jenkins: openssh_keygen: true Purge unmanaged users By default Puppet only manages the resources we instruct it to manage, and the same is done with users here. Still there are cases where we want full control on the interactive users managed on a server, this can be accomplished with the (quite dangerous) parameter: psick::users::delete_unmanaged: true The default is false, but is this is the to true all non system users not managed by Puppet are automatically deleted. Managing /etc/skel The /etc/skel directory on a Linux systems contains files that are added to the homes of all the users created on the system. We can manage its content with: psick::users::source: 'puppet:///modules/profile/users/skel' The contents of the directory specified as source (used as in the file resource type so, in the above sample, on a local profile module, under the files/users/skel directory) are copied to the system’s /etc/skel directory. Note that files from here are copied to only to newly created users, so changes here don’t affect the home of existing users. Alternative way to define users If a single hash becomes hard to manage via hiera, due to different users to be applied to different kind of servers, we can have an alternative way to define the users we want to configure on a system. Alternatively (or complementarily) to users_hash we have the key available_users_hash which acts exactly in the same way, but rather than actually creating the users, it can serve as key to define all the possible users we want to manage. The actual users to create are then defined by the available_users_to_add array. psick::users::available_users_hash: al: ensure: present [...] ma: ensure: present [...] jenkins: ensure: present [...] psick::users::available_users_to_add: - al - ma In this way we can define all our users in a single Hiera layer, and then use the available_users_to_add parameter on the different Hiera hierarchies, to actually manage which users are created where. This is what can be done for users with psick::users profile. Have fun with Puppet, Life, Universe and Everything. Alessandro Franceschi 2018-12-03T00:00:00+00:00 https://example42.com/blog/2018/12/03/psick_profiles_part_3_openssh On the first post of our series on Psick profiles we introduced the psick module and had an overview of its reusable profiles. On the second one we described how to manage hostname and proxy settings with the existing psick profiles. In this post we are going to review the resources that psick module provides to manage OpenSSH. The main psick::openssh profile, if included, installs openssh via Tiny Puppet (via the psick::openssh::tp class) and exposes parameters that act as entry points to configure other openssh related resources: psick::openssh::configs_hash expects an hash (looked via Hiera in deep merge mode) of psick::openssh::config resources which permit the configuration of users’ ~/.ssh/config file psick::openssh::keygens_hash expects an hash (looked via Hiera in deep merge mode) of psick::openssh::keygen resources which runs the keygen command to generate ssh keypairs for users. psick::openssh::keypairs_hash expects an hash (looked via Hiera in deep merge mode) of psick::openssh::keypair resources which allow to manage ssh keypairs for users. psick::openssh::keyscans_hash expects an hash (looked via Hiera in deep merge mode) of psick::openssh::keyscan resources which make a ssh keyscan of remote nodes and add their ssh host key to users’ known hosts file. Let’s explore the mentioned resources, as they do commonly used operations. Define psick::openssh::config to manage users’ ssh configs The ssh config file for single users can be managed with code as (the titles snd parameters used here can be expressed via Hiera with the previously described keys): The content of the file can be managed in different ways. With the source parameter: psick::openssh::config { 'al': source => 'puppet:///modules/profile/openssh/al/config', } With an explicit content parameter (alternative to source): $sshconfig = @("SSHCONFIG"/L) Host * ForwardAgent no TCPKeepAlive yes | SSHCONFIG psick::openssh::config { 'al': content => $sshconfig, } Or also with the template parameter, which can be coupled with a custom set of settings via the options_hash parameter: psick::openssh::config { 'al': template => 'profile/openssh/user_config.erb', options_hash => $ssh_parameters, } Since we need to have the directory .ssh created, we can instruct the define to automatically create it for us. Default value is false to prevent issues with duplicated resources: psick::openssh::config { 'al': source => 'puppet:///modules/profile/openssh/users/al/config', create_ssh_dir => true, # Default: false } Note: All the psick::openssh:: defines have create_ssh_dir parameter, default value is false for all of them except psick::openssh::keypair. Define psick::openssh::keygen to create ssh keypairs This define creates a ssh keypair using the ssh-keygen command. There are various options to manage where, how and for whom the ssh public and private keys have to created. The simplest use, is just to specify as title the username to create keypairs in the default paths ( ~/.ssh/id_rsa , ~/.ssh/id_rsa.pub): psick::openssh::keygen { 'al': } To customise the home, for example: psick::openssh::keygen { 'jenkins': home => '/var/lib/jenkins' } Define psick::openssh::keypair to manage ssh keypairs If we want to manage directly the contents of our ssh keys, instead of generating them with psick::openssh::keygen, we can use the define psick::openssh::keypair where we have different options to manage the content of the ssh keys. psick::openssh::keypair { 'al': private_key_content => lookup('ssh_private_key_al'), public_key_source => 'puppet:///modules/profile/openssh/al/id_rsa.pub' } Define psick::openssh::keyscan to pre fetch ssh host keys When you need to automate ssh connection you need to have the remote hosts’ ssh keys added to local users’ or system’s known_hosts file. Usually this is done at the first ssh connection to a new host, by accepting manually the remote host key, the psick::openssh::keyscan does this for us. Simple usage is: psick::openssh::keyscan { git.example.com: user => 'jenkins', } Sample hiera data for psick::openssh Here’s how Hiera data for an openssh configuration may look like: psick::openssh::tp::resources_hash: tp::conf: openssh: template: 'psick/generic/spaced.erb' options_hash: Protocol: 2 PermitRootLogin: 'no' Subsystem: 'sftp /usr/libexec/openssh/sftp-server' psick::openssh::configs_hash: jenkins: path: /var/lib/jenkins/.ssh/config create_ssh_dir: true options_hash: Host puppet.lab.psick.io: StrictHostKeyChecking: no UserKnownHostsFile: /dev/null Host puppet: StrictHostKeyChecking: no UserKnownHostsFile: /dev/null psick::openssh::keyscans_hash: github.com: user: jenkins known_hosts_path: /var/lib/jenkins/.ssh/known_hosts psick::openssh::keygens_hash: jenkins: home: /var/lib/jenkins Have fun with Puppet, Life, Universe and Everything. Alessandro Franceschi 2018-11-26T00:00:00+00:00 https://example42.com/blog/2018/11/26/puppet_community_award Puppet is a product built around a community. The Puppet community encourages learning new skills, expanding knowledge and honors contributions. We want to celebrate the 100th post in our Puppet Tip of the Week celebrating the same Puppet community of which we are proud members and talking about this year edition of Puppet MVP (Most Valued Puppeteer) award. The MVP award is voted by community and Puppet members and given to people in the community who have distinguished themselves in their Puppet activities. As far as I remember the first MVP award was given to Eric Dalen at PuppetConf 2014 for his feedback and help on PuppetDB. In 2015 Mark Sutter was honored for his work on the Windows Powershell DSC module. At Puppetconf in 2016 the award has been given to Rob Nelson for most fixed tickets at a triage. I was unable to find information on 2017 award. I only found a tweet mentioning Ryo Sugahara and Tim Meusel as MVP runnersup. This years Community Awards have expanded, with new mentions to Invaluable Puppeteer and Force Multiplier. Tim was again on the MVP list, but this time he was recognised for his fantastic work within voxpupuli - the GitHub Puppet Community. Nick Maludy won the Force Multiplier award for his contributions to Bolt, and I’m am proud and happy for being nominated as Invaluable Puppetteer. List of past MVP announcements: 2014: Finalists Winner Daniel Sluijters (@danney) - activity on IRC Eric Dalen (@eric_dalen) - PuppetDb feedback Felix Frank (@felix_res) - fixing long outstanding bugs   Tim Sharp (@rodjek) - shares testing tools   2015: Finalists Winner Felix Frank - improve core resource types Mark Sutter - work on Powershell DSC module Erik Dalen - Puppet Explorer   Brice Figuerau - Bug fixes   2016: Winner Rob Nelson (@rnelson0) 2017: Winners Ryo Sugahara (@denkas1973) Tim Meusel (@bastelsblog) 2018: Winners Tim Meusel - MVP Martin Alfke - Invaluable Puppeteer Nick Maludy - Force Multiplier Many thanks to everybody for supporting, building and helping community. Martin Alfke 2018-11-19T00:00:00+00:00 https://example42.com/blog/2018/11/19/psick_profiles_part_2_proxy_and_hostname_settings On the first post of our series on Psick profiles we introduced the psick module and had an overview of its reusable profiles. In this post we are starting to review some handy psick profiles: psick::proxy to configure proxy settings, on Linux system startup scripts, on gem and pip environments and in repo configurations for apt and yum psick::hostname to manage / enforce the system’s hostname both on Linux and Windows Managing proxy settings with psick::proxy To use it just include the class in your manifests (remember that): include psick::proxy or classify with the ENC of choice. Then, hiera configuration can be as follows: psick::proxy::proxy_server: host: proxy.example.com port: 3128 user: john # Optional password: xxx # Optional no_proxy: - localhost - "%{::domain}" - "%{::fqdn}" scheme: http #  If you have included the psick class, you can set the same values using the general psick::servers hash, as this is the default value for the psick::proxy::proxy_server: class psick::proxy ( [...] Optional[Hash] $proxy_server = $::psick::servers['proxy'], [...] The above example has the same effect of: psick::servers: proxy: host: proxy.example.com port: 3128 You can customise the components for which proxy should be configured, here are the default params: Manage presence of proxy settings: psick::proxy::ensure: present Configure proxy settings for system’s gem environment: psick::proxy::configure_gem: true Configure proxy settings for Puppet’s gem environment: psick::proxy::configure_puppet_gem: true Configure proxy settings for PIP environment: psick::proxy::configure_pip: true Configure proxy settings for system. Exporting http_proxy variables in profile.d: psick::proxy::configure_system: true Configure proxy settings on package management tool (yum and apt supported): psick::proxy::configure_repo: true Managing the hostname with psick::hostname Another profile for common use cases is psick::hostname which manages the server’s hostname both on Linux and Windows. It allows to set or reinforce hostname, fqdn and domain in local system configuration files on Linux and Windows. Class defaults are as follows: class psick::hostname ( String $host = $::hostname, Variant[Undef,String] $fqdn = $::fqdn, Variant[Undef,String] $dom = $::domain, String $ip = $::ipaddress, Boolean $update_hostname = true, Boolean $update_host_entry = true, Boolean $update_network_entry = true, Boolean $update_cloud_cfg = false, Boolean $no_noop = false, ) { The update booleans have effect only on Linux and control where we want to set the hostname: On /etc/hostname and via the hostname command (update_hostname) With Puppet’s host resource (update_host_entry) On /etc/sysconfig/network (only on RHEL derivatives) (update_network_entry) On /etc/cloud/cloud.cfg.d/99_preserve_hostname.cfg (requires cloud config) (update_cloud_cfg) On Windows if $update_hostname is true and hostname is not the same configured, the netdom renamecomputer command is run. Usage is the usual, include in manifests (ie: in role classes): include psick::proxy Or via any other classification option you use. Configuration via hiera can be as follows: psick::hostname::host: my_host psick::hostname::fqdn: my_host.my_domain psick::hostname::dom: my_domain psick::hostname::ip: "%{::ipaddress}" These are two of the several psick profiles for common uses. They are less complete than dedicated modules but can do most of the expected work without the need of additional modules, other than psick, keeping the whole psick philosofy of giving choice on what profiles to use and what to configure with them. Alessandro Franceschi 2018-11-12T00:00:00+00:00 https://example42.com/blog/2018/11/12/psick_profiles_part_1_overview We have already talked in our blog about the PSICK project, both the control-repo and the Puppet module. Here and in our next posts we will get deeper in how to use the reusable profiles of the Psick module. Prerequisites Psick is a module you can get from the forge, if you install it form the command line all its dependencies are resolved: puppet module install example42-psick When used inside a control repo, you need to explicitly state its dependencies in your Puppetfile: mod 'puppetlabs-stdlib', :latest mod 'example42/psick', :latest mod 'example42/tp', :latest mod 'example42/tinydata', :latest In a production environment we will likely specify fixed and tested versions, the current latest version of the above modules are: mod 'puppetlabs-stdlib', :5.1.0 mod 'example42/psick', :0.5.8 mod 'example42/tp', :v2.3.1 mod 'example42/tinydata', :v0.3.2 Some psick modules can support other third party component modules, refer to each psick class documentation for details on additional prerequisite modules. Usage Once you have psick among your modules you can include any of its profiles using whatever method you have to classify nodes. You have to include first the main psick class, which is safe and harmless as it does nothing by default but act as entrypoint for some common variables used by psick profiles. Usage inside manifests then can be like: include psick include psick::<profile> You can consider psick classes as reusable profiles: they have the characteristic of a profile as they wrap resources from other modules to accomplish specific functions, but they are conceived in a way that they can be reused and adapted to different (common) use cases. You can cherry pick and use the ones you want instead of writing a custom profile (which might eventually need an additional component module and its dependencies) for every case where a psick profile fits your situation. Usage of psick profiles is similar to the one of classes from existing modules: you classify nodes with the relevant class name, and then use hiera to configure it via it’s parameters. Base and applications profiles Psick comes with a wide like of base profiles for common settings, here are the most significant ones: psick::hosts - Manage /etc/hosts psick::motd - Manage /etc/motd and /etc/issue psick::nfs - Manage NFS client and server psick::sudo - Manage sudo configuration psick::sysctl - Manage sysctl settings psick::firewall - Manage firewalling psick::openssh - tp profile and keygen define psick::hardening - Manage system hardening psick::network - Manage networking psick::puppet - Manage Puppet components psick::users - Manage users psick::time - Manage time and timezones There are also several other application specific profiles, where it’s generally possible to choose what module to use to manage the actual application (if a popular public module or Tiny Puppet): psick::ansible - Manage Ansible installation psick::aws - Manage AWS client tools and VPC setup psick::bolt - Manage Bolt installation psick::docker - Docker installation and build tools psick::foreman - Foreman installation psick::git - Git installation and configuration psick::gitlab - GitLab installation and config psick::mariadb - Manage Mariadb psick::mysql - Manage Mysql psick::mongo - Manage Mongo psick::php - Manage php and modules psick::oracle - Manage Oracle prereq and setup psick::sensu - Manage Sensu tp profiles TP profiles in psick are profiles which are generated automatically, they all share the same structure (and parameters) offering a consistent user interface. They, of course, use Tiny Puppet defines (tp::install and tp::conf) to manage the relevant applications. All tp profiles in psick have names like psick::<app>::tp, and are generated with pdk using this template. You can generate a new tp profile, by running on the psick module: bin/bin/tp_profile.generate.sh nginx will create the class psick::nginx::tp On the upstream psick module, currently, only a limited list of application have their own tp profile. Feel free to request more application specific tp profiles. TP encourages the usage of the template + options hash pattern, ad has some generic templates usable for common file structures. So, for example, to manage openssh with psick , we can: include psick::openssh::tp by default a tp profile installs the relevant application but doesn’t configure it. In order to manage configuration files you have to set parameters as follows (in this example we use a generic template where each key value of the options hash is written as <key> <value> which is a valid format for OpenSSH confiuration): psick::openssh::tp::resources_hash: tp::conf: openssh: template: 'psick/generic/spaced.erb' options_hash: Protocol: 2 PermitRootLogin: 'no' Subsystem: 'sftp /usr/libexec/openssh/sftp-server' Here, by including the psick::openssh::tp profile we have Openssh package installed and the relevant service started. We configure it using the resources_hash parameter, common in all tp profiles, where for each configuration file to manage via tp::conf, or whole directories to manage with tp::dir, we specify name, and content (by using parameters like source, content, template or epp) and a custom options_hash where any parameter, referred in the used template, looked up in deep merge mode, can be managed via Hiera. The whole logic of the tp profiles is to expose parameters that allow to directly configure tp defines via Hiera. If you know exactly how to configure your application, and its setup is based on a standard package + configuration files + service pattern, then tp, either used within your profiles on via a psick tp profile, can be a solid alternative to the usage of a dedicated component module. Conclusion This is the first post of a series about the profiles shipped with the psick module. After this general introduction we are going, in the next posts, to see more details on how to use these profiles for common system configurations. Alessandro Franceschi 2018-11-05T00:00:00+00:00 https://example42.com/blog/2018/11/05/example42_top10_devops_consulting-company We are pleasantly surprised and honoured to be rewarded as one of the Top 10 DevOps consulting companies in Europe by CIO Applications Europe. Here the full issue in PDF. Example42, is described as a company that “offers and shares huge IT experience and world-class knowledge of Puppet software and DevOps practices”. Efficient Consulting Services Knowledge sharing Efficient Consulting There is a major difference between effective and efficient consulting. Within effective consulting companies get solutions and support in a fast and reliable matter. We prefer efficient consulting, where we enable customer teams to understand the solutions and are able to continue maintaining them by themselves. This allows companies to be less reliant on long term external consulting and results in happier staff. Services At example42 we not only deliver consulting, we also support teams by conducting specific trainings on DevOps related tool chains and offer Puppet code maintenance support. Our maintenance contract takes the burden from teams when it comes to building Puppet code following modern best practice and not being blocked by deprecations when it comes to Puppet upgrades. Especially in the DevOps world training is one of the essentials tasks which must be done to keep pace with modern best practice. Knowledge sharing We have always shared, and will continue to share, our experience on Puppet, operations, automation and DevOps. A list of the material, software and documentation we have shared includes: This same blog, with our Tips of the Week on Puppet and DevOps Tiny Puppet a Puppet module that abstracts installation of configuration of applications (“Any app on any OS”) PSICK a Puppet control-repo and module to easily setup a full featured Puppet infrastructure Less recent projects involve: A Puppet Tutorial usable for trainings and workshops The DevOps Guide to Puppet, Universe and Everything. Featuring a Slides deck fro beginners and a reusable control-repo from which the PSICK project was derived The NextGen Example42 Puppet modules set, most of which are now no more maintained, but for years have been a point of reference in Puppet modules Puppi a Puppet module to configure automate application deployments We want to thank the CIO Applications authors for the honour and the reward and you, as reader, for being here. We wish everybody a great start into this week. Alessandro Franceschi and Martin Alfke 2018-10-29T00:00:00+00:00 https://example42.com/blog/2018/10/29/editorconfig This post, for once, is not strictly about Puppet, but about a nice, standard way to customise the behaviour of a text editor directly from within your source code repository. Editorconfig is a file format that defines the coding style to use in an editor for the files of your repository. It contains, in stanzas divided ini file format, various definitions to manage common editor settings: indent_style - Indentation Style. Possible Values: tab, space indent_size - Indentation Size (in single-spaced characters). Possible Values: an integer, tab tab_width - Width of a single tabstop character. Possible Values: an integer (defaults to indent_size when indent_size is a number) end_of_line - Line ending file format (Unix, DOS, Mac). Possible Values: lf, crlf, cr charset - File character encoding. Possible Values: latin1, utf-8, utf-16be, utf-16le, utf-8-bom trim_trailing_whitespace - Defines whether whitespace is allowed at the end of lines. Possible Values: true, false insert_final_newline - Defines whether file should end with a newline. Possible Values: true, false Some other settings are editor specific. Editorconfig is supported by most of the existing editors either natively or via plugins. Editors that support it out of the box: BBEdit, Codelite, elementaryCode, Builder, GitHub, Gogs, IntelliJidea, KText editor, Komodo, Kakoune, PyCharm, ReSharper, Rider, RubyMine, Source Lair, Tortoise Git, Visual Studio, Web Storm. Editors that support it via a plugin: AppCode, Atom, Brackets, C Lion, Coda, Code::Block, Eclipse, Emacs, Geany, Gedit, jEdit, Micro, Net Beans, NodePad++, Php Storm, Sublime Text, Text Adept, Text Mate, Vim, Visual Studio Code. An example is probably better than many works, so here’s our .editorconfig file in our Psick control-repo. It applies to all the files of the repo, and, as you see, different settings can be defined according to the files name or extension. [*] charset = utf-8 end_of_line = lf indent_style = space indent_size = 2 insert_final_newline = true trim_trailing_whitespace = true [*.md] max_line_length = off trim_trailing_whitespace = false Have fun with your editor, Puppet, life, universe and everything! Alessandro Franceschi 2018-10-22T00:00:00+00:00 https://example42.com/blog/2018/10/22/profiles_explained Either during training or at onsite consulting we are confronted with the question on what a profile is, how to best build it and how it differs from a module and a role. Within this week’s tip of the week we want to show our best practices on Puppet implementation profile development covering namespaces, parameter usage, content and templates and testing. What is a profile? Profile naming and namespaces Parameters and automatic data binding with Hiera What is a role? Using an existing implementation No time to read everything and code for yourself? Jump to the end of this page. What is a profile? A profile describes a technical implementation on your systems. The code may consist of all self written Puppet code (package, file, service) or may use existing modules (class { ::ntp: servers => $ntp_servers, }). You want to have self written Puppet code in case that you are still in the process of learning Puppet and learn how to read (and check) existing modules. Once you are able to read and understand existing Puppet module code, you might want to switch to an existing module. Hint: Never write modules! Write profiles instead! This will not block you using an existing namespace (module directory name). A technical component can be something like SSH, backup, your web server with specific extensions, your virus scanner installation. Configuration differences are either placed directly into your code (static data) or placed into hiera using automatic data binding on profile class parameters. Additionally a profile directory structure can also hold templates or static configuration files. Profile naming and namespaces You want to think about different names for related configurations. Usually these consist of several processes which must work together. Think about the following example, assuming a Linux only platform: Your Login into a VM consists of SSH, SSSD with LDAP integration and SUDO settings. Your profile is responsible for the authorization and authentication. These are names you can use for your profile: # <$environmentpath><$environment>/site/profile/manifests/auth.pp # Class profile::auth # # @summary Authorization and Authentication for SSH users # # @param sshd_conf_allowgroups # must be type array, defaults to empty array # list of groupnames with ssh access used by ghoneycutt-ssh module # # @param sssd_config # must be of type hash, defaults to empty hash # configuration settings used by sgnl05-sssd module # # @param sudo_config # must be of type hash, defaults to empty hash # configuration settings for saz-sudo module # class profile::auth ( Array $sshd_config_allowgroups= [], Hash $sssd_config = {}, Hash $sudo_configs = {}, ) { class { 'sssd': config => $sssd_config, } # ... } Always remember to add documentation to your modules. This is highly important and can be visualized by puppet-strings. Other - non basic parts of your server - can be easily grouped by placing them into a directory structure. Now we will add windows support: <$environmentpath><$environment>/site/profile/manifests/ |- auth/ | |- linux.pp # <- SSH, LDAP, PAM | \- windows.pp # <- RDP |- time/ | |- ntpdate.pp # <- NTP Linux | \- windows.pp # <- NTP Windows |- databases/ | |- mysql_server.pp # <- MySQL - Linux | \- postgresql_server.pp # <- PostgreSQL - Linux |- monitoring/ | |- alerting.pp # <- Prometheus Alerting - Linux | |- metrics.pp # <- Prometheus Metrics - Linux | |- node.pp # <- Prometheus Node <- Linux/Windows | \- server.pp # <- Prometheus Server - Linux |- services/ | |- systemd_wait_for_port.pp # <- Snippet to set secial setting in unit file | \- docker_timeout.pp # <- Snippet to set container max start time Parameters and automatic data binding with Hiera Puppet can lookup data for classes whenever they are declared. The lookup key name passed to hiera is built programatically: <class name>::<parameter>. Data for profile::auth class might look like this: profile::auth::sshd_config_allowgroups: - 'infastructure_admins' - 'all_developers' This concept allows you to always identify which profile is using which data. For data used by multiple profiles you can do nested lookups within hiera: profile::auth::sssd_config: 'domain/LDAP': 'ldap_default_authtok': "%{lookup('ldap_pw')}" This security artifact is stored using hiera-eyaml: ldap_pw: > ENC[PKCS7,MIIBeQ...] Security settings are now separated from configuration settings. What is a role? A role is a class that includes ones or more profiles. It’s generally recommended that in role classes you just include profiles, without passing parameters or other Puppet resources. A role class may look like: class role::webserver { include profile::base include profile::apache include profile::php } This implies that as we may have a profile module containing all our profile classes, we can have a role module containing all our role classes. So, basically, roles are used to simplify nodes classification. You just include a role class for each nodes’ role and all the rest is managed according to the profiles included there. If you have a fact called $role we can just have a line like this in our control repo’s manifests/site.pp: include "::role::${::role}" For this reason, even if the roles and profiles pattern is well established as it makes things simpler, it’s not necessarily the only approach we can use: we may have just profiles without roles, in this case we don’t need role classes, but just other ways to define what profiles to include in a node (for example via Hiera data, or on an external node classifier). Bear this in mind when working with roles and profiles: what you really need to define is basically what profiles you want on your nodes, how this is done, if via role classes or other methods does not make a real difference. Using an existing implementation So now the real work begins: writing profiles for all the different things you have in your environment. Profiles are typically local classes, not supposed to be shared, as they implement our own specific way to manage resources. Still you can actually benefit from others’ work on profiles: our PSICK module contains several reusable profiles, some of them use tiny-puppet some plain Puppet resources, and they cover common features and use cases which may apply also to your setup. The important thing to understand about the psick module, is that you can pick the profiles you want to use, and ignore the others: you decide which solution you prefer: you just have to be sure that if you include any psick profile, you should also include the main psick class (which by default, with the predefined parameters, does NOTHING): include psick # Prerequisite for all the psick Profiles include psick::time # A profile to manage NTP and timezones on Linux and Windows All psick profiles have parameters which can be used to customise and manage their behaviour via Hiera data. We wish everybody success. Please provide feedback, especially when you are using our Puppet Systems Infrastructure Construction Kit - PSICK If your team wants this explained live on your own profiles, that’s exactly what Puppet Training covers. Martin Alfke Alessandro Franceschi 2018-10-15T00:00:00+00:00 https://example42.com/blog/2018/10/15/application-management-using-tinypuppet This week’s tip of the day deals with management of custom applications with Puppet and Tiny Puppet. Usually developers prefer to concentrate on their application development and not also write Puppet modules for its deployment and configuration. Especially when it comes to individual, in-house developed software you will find no Puppet module available, so a custom profile or module has to be written. In this post we are going to show how we can easily manage custom applications with Tiny Puppet with limited or no Puppet code at all. Application deployment using modules Application deployment using a Tiny Puppet profile Application deployment using a Tiny Puppet wrapper define Application deployment using Tiny Puppet and custom Tinydata Advantages? Application deployment using modules Individual applications need individual modules or profiles. Developers know about possible configuration flags (e.g. db connectors or feature flags) and how and where to set specific configurations. They will start with the installation (hopefully package resource type) and then use the file resource to configure settings. Afterwards they use the service resource to ensure that the application is running. e.g. class my_application ( String[1] $version, String[4] $admin_password, Boolean $enable_feature_432 => false, Boolean $enable_feature_876 => true, ){ package { 'my_application': ensure => $version, } file { '/etc/application/app.conf': ensure => file, content => template('my_application/app.conf.erb'), notify => Service['my_application'], require => Package['my_application'], } file { '/etc/application/secrets': ensure => file, content => "admin:${admin_password}", notify => ['my_application'], require => Package['my_application'], } service { 'my_application': ensure => running, enable => true, } } If multi OS support is needed, the module can get a lot more complex, with a params class or data in module to cope with different paths and names. Moreover you may need to add fragments of code for each managed file and manage the relevant dependencies. All code must be inside either a module or a profile and needs unit and acceptance testing. When you have multiple applications you will have multiple classes for each of your applications. Hopefully, finally, your application modules do not interfere with any other module which might lead to duplicate resource type declarations. Application deployment using a Tiny Puppet profile Within Tiny Puppet you don’t have to bother with developing Puppet code. Tiny Puppet is an abstraction layer for any application deployment and can fully controllable by hiera data. Let’s migrate the above Puppet code to a profile with Tiny Puppet defines. First we generate a wrapper class (profile), which can be used by all applications: class profile::my_application ( String[4] $admin_password = '', Hash $install_options = {}, Hash $conf_options = {}, ){ if ! empty($install_options) { tp::install { 'my_application': * => $install_options, } } if ! empty($conf_options) { tp::conf { 'my_application': * => $conf_options, } } if !empty($admin_password) { tp::conf { 'my_application::secrets': content => "admin:${admin_password}", } } } Now we can add the required data to hiera: profile::my_application::admin_password: ENC[...] # encrypted data profile::my_application::install_options: ensure: '1.1.3' profile::my_application::conf_options: path: '/etc/application/app.conf' template: 'profile/my_application/conf.erb' options_hash: master: appmaster.prod.mydomain listen: 0.0.0.0 In the conf.erb template we can refer to our application configuration options with something like: master = <%= @options_hash['master'] %>. Application deployment using a Tiny Puppet wrapper define An alternative to the above example, where a profile class is created for each application, can be to write a custom generic wrapper define where the tp::install and tp::conf defines are declared, and a simple class that allows to configure via Hiera data any application. The generic wrapper could be something like: define profile::application ( Hash $tp_options_hash = {}, ){ if has_key($tp_options_hash, 'install') { tp::install { $title: * => $tp_options_hash['install'], } } if has_key($tp_options_hash, 'conf') { tp::conf { $title: * => $tp_options_hash['conf'], } } } The class that exposes a parameter to configure the profile::application defines could be as simple as: class profile ( Hash $applications = {}, ) { $applications.each |$k,$v| { profile::application { $k: * => $v, } } } This is the only code we would need to write for any application, then we can feed it with data like: profile::applications: my_application: install: ensure: '1.1.3' conf: path: '/etc/application/app.conf' template: 'profile/my_application/conf.epp' Now any development team just provides a set of YAML data for their application based on a custom template where we can parametrise what we need to change in different environments or servers. Application deployment using Tiny Puppet and custom Tinydata We can be even smarter and create tinydata (the data used by Tiny Puppet) specific for our application, in a custom data module, with custom hierarchy for each application we want to manage (in case we want support for multiple OS): vi my_tinydata/data/my_application/hiera.yaml --- :hierarchy: - "%{title}/osfamily/%{osfamily}" - "%{title}/default" - default We need a generic (valid for all applications) data/default.yaml which can have the same contents of the tinydaya default. vi my_tinydata/data/default.yaml Now we need to create at least a file with application specific data, let’s just create the default file, valid for all OS: vi my_tinydata/data/my_application/default.yaml Content might look like: --- my_application::settings: package_name: 'my_application' service_name: 'my_application' config_file_path: '/etc/my_application/app.conf' config_dir_path: '/etc/my_application' These are the minimal settings for having a typical package/service/config file setup, but we can add more options such as: # Used by tp log command, can be an array log_file_path: '/var/log/my_application.log' # Optional argument to launch service in forground (useful inside Docker containers) nodaemon_args: '-D' # Optional command to check the syntax of the application configuration before restarting my_application service validate_cmd: 'my_application -t -f %' # Optional Url of the release package which configures my_application Yum/Apt repo (if no release package is available the repo settings can be set via other keys in tinydata) repo_package_url: 'https://repo.mydomain/my_application/my_application-release-el-7.noarch.rpm' Given tinydata like this we can configure our application without writing a single line of code. We can just include the tp class (here needed just to expose Hiera configurable parameters to manage tp defines, the same can be accomplished with a custom class similar to the profile example before) and write Hiera data like: tp::install_hash: my_application: ensure: '1.1.3' data_module: my_tinydata tp::conf_hash: my_application: template: 'profile/my_application/conf.erb' options_hash: master: appmaster.prod.mydomain listen: 0.0.0.0 data_module: my_tinydata my_application::secrets: content: ENC[...] # Hiera-eyaml encrypted content data_module: my_tinydata The default value for the data_module parameter is ‘tinydata’, and for this reason the tp module has example42-tinydata as dependency. Still the data_module can be configured, so we can have a custom local module (here my_tinydata) where we add our own applications data. Alternatively we could just form the tinydata module and add our applications data to our local version of the module. Advantages? When it’s worth using tp instead of writing a custom module? When we want to reduce the amount of custom Puppet code to write and prefer a totally data driven management of applications When we need to manage packages, services, configuration files (tp takes care of relationships) When we might benefit for a command like tp test, which can be executed locally on the system, via a monitoring, remote execution or orchestration tool to quickly and automatically get the status of all the applications managed by tp When we want a quick and standardised way to troubleshoot applications, using the tp log command to tail all the logs of the applications we manage. We wish everybody fun and success with Tiny Puppet, Martin Alfke Alessandro Franceschi 2018-10-08T00:00:00+00:00 https://example42.com/blog/2018/10/08/puppet6-ca-upgrading In our last blog post we covered all the new features of Puppet 6. Now we look forward on how to migrate to the new CA. CA usage on Puppet 5 and earlier The Puppet 6 CA Migrating to Puppet 6 CA New CA auth.conf Modify CA cert Autosigning on Puppet 6 CA CA and certificate management on Puppet 6 CA usage on Puppet 5 and earlier The CA and certificate management was usually part of the Puppet Agent package. All related commands were part of the puppet command: puppet cert list [--all] puppet cert sign <certname> puppet cert print <certname> puppet cert revoke <certname> puppet cert clean <certname> The Puppet 6 CA Starting with Puppet 5.5 you will recognize that Puppet CA and certificate management will be moved from Puppet Agent to Puppetserver in Puppet 6. Puppet 5.5 prepares you by providing deprecation information in the Puppet server logfile and on the command line. The Puppetserver handles certificate management via API calls. All Puppetserver APIs are protected with an authorization layer for read and write access. This authorization and access control is also done for the certificate management API. Within a newer Puppet server, you will find two new entries in the authorization configuration file located at /etc/puppetlabs/puppetserver/conf.d/auth.conf: { # Allow the CA CLI to access the certificate_status endpoint match-request: { path: "/puppet-ca/v1/certificate_status" type: path method: [get, put, delete] } allow: { extensions: { pp_cli_auth: "true" } } sort-order: 500 name: "puppetlabs cert status" }, { # Allow the CA CLI to access the certificate_statuses endpoint match-request: { path: "/puppet-ca/v1/certificate_statuses" type: path method: get } allow: { extensions: { pp_cli_auth: "true" } } sort-order: 500 name: "puppetlabs cert statuses" }, Within the allow section we see the default settings, which now requires the Puppet server ca certificate to have the pp_cli_auth extension set. Let’s analyze the new Puppet 6 CA certificate: openssl x509 -noout -text -in /etc/puppetlabs/puppet/ssl/certs/$(puppet config print certname).pem [...] X509v3 extensions: Netscape Comment: Puppet Server Internal Certificate X509v3 Authority Key Identifier: keyid:B3:DC:C3:68:D5:3A:A8:A3:30:3C:EB:85:79:0F:EB:9E:1A:82:5E:7A X509v3 Subject Key Identifier: 1F:32:B6:9D:D0:9F:9D:57:8A:57:D6:DE:42:45:78:6D:27:D3:A1:15 1.3.6.1.4.1.34380.1.3.39: ..true [...] Here we see a new entry with an OID (1.3.6.1.4.1.34380.1.3.39) and the value set to true. Migrating to Puppet 6 CA In general you have multiple possibilities which you can follow when upgrading to Puppet 6: new CA and certificates modify auth.conf to use old CA certificate modify existing CA certificate and add required extension New CA Usually you barely want to follow this option as this does mean a complete CA roll-over within all of your Puppet managed systems. Maybe this is an option in case that your CA is about to expire soon? We will look into the two solutions which do not require new certificates: auth.conf Within /etc/puppetlabs/puppetserver/conf.d/auth.conf you want to add your Puppet server: { # Allow the CA CLI to access the certificate_status endpoint match-request: { path: "/puppet-ca/v1/certificate_status" type: path method: [get, put, delete] } allow: { extensions: { pp_cli_auth: "true" } } allow: master.example.com # <- add your puppet master certname sort-order: 500 name: "puppetlabs cert status" }, Please try to not set allow-unauthenticated: true. Even though this is technically possible (e.g. if your VM management solution is not integrated or managed by Puppet), you can easily generate a certificate on the Puppet CA and copy it over to the system which is responsible for removing or signing certificates. Please remember to restart your Puppet server process to activate changes. Modify CA cert Another solution (untested) is to add the required extension to the Puppet CA certificate. There is a project on GitHub from smortex which also has links to tickets at Puppet and which provides a ruby script which adds the required extension. Autosigning on Puppet 6 CA Autosigning itself has not changed from Puppet 5 to Puppet 6. The configuration is still done in Puppet configuration file (/etc/puppetlabs/puppet/puppet.conf) in master section: # /etc/puppetlabs/puppet/puppet.conf [master] autosign = false # <- disable autosign autosign = true # <- default, sign based on content of autosign.conf file (naive autosigning) autosign = /etc/puppetlabs/puppet/autosign.sh # <- script to execute: on exit 0 signing will take place (policy based autosigning) CA and certificate management on Puppet 6 The new CA management is completely integrated into Puppet server: puppetserver ca list [--all] puppetserver ca sign <certname> puppetserver ca print <certname> puppetserver ca revoke <certname> puppetserver ca clean <certname> We wish everybody fun and success with Puppet 6, If you’d rather have a CA migration like this reviewed before you touch production, that’s part of our Puppet Health Check. Martin Alfke 2018-10-01T00:00:00+00:00 https://example42.com/blog/2018/10/01/what-s-new-with-puppet-6 Puppet 6 has been released the 18th of September, Eric Sorenson announced the new release in this blog post. This is a new major release because it contains some enhancements which are not backwards compatible, but for most of the cases they won’t require you to do any major review of your Puppet code base: if it works with Puppet 4 it’s likely to work also on Puppet 6. Let’s see what are the most interesting new features. Several types moved to dedicated modules New CA management on the puppetserver Puppet ssl command The Resource API The “Deferred” data type Functions imported from stdlib Command puppet module build removed Conclusions Several types moved to dedicated modules This is a long awaited cleanup: all the Nagios types, and some other OS specific ones are no more part of the core product or have been moved to modules. Most of them are still shipped in the puppet-agent package, others are in maintained modules not included in the puppet agent and some have been moved to modules and been deprecated. More precisely, the following types are included in supported modules on the forge and repackaged in puppet-agent, so nothing changes for end users: augeas cron host mount scheduled_task selboolean selmodule ssh_authorized_key sshkey yumrepo zfs zone zpool. These other types have been moved to module which are still supported by are not included in Puppet agent package, so if you use them you should add the relevant modules from the Forge: k5login mailalias maillist. These types have been deprecated, they are moved to modules which are not going to be actively maintained and are not shipped with puppet-agent package: computer interface macauthorization mcx router vlan plus all the nagios_* types (all moved to puppetlabs-nagios_core module). In general all the moved types are now placed in a puppetlabs module with _core suffix. Look here for a rough list. The core modules shipped with Puppet-agent are placed under /opt/puppetlabs/puppet/modules on *nix and $codedir/modules on Windows, these paths are added to the default basemodulepath setting. New CA management on the puppetserver Puppet CA support has been rewritten in Clojure and included in the puppetserver package, the old CA Ruby code has been removed. The new puppetserver ca command has been introduced to replace the previous puppet cert and puppet ca commands. The puppetserver ca actions to manage certs are similar to the old puppet cert ones. Available actions are: clean: clean files from the CA for certificates generate: create a new certificate signed by the CA setup: generate a root and intermediate signing CA for Puppet Server import: import the CA’s key, certs, and CRLs list: list all certificate requests revoke: revoke a given certificate sign: sign a given certificate Puppet ssl command The puppet ssl command has been introduced. It replaces puppet certificate_request (use puppet ssl submit request) and add subcommands like puppet ssl verify (verifies that local Puppet cert and key are valid) and puppet ssl download_cert (downloads a certificate for the local node). The Resource API A Resource API has been added, providing a new, recommended method to create custom types and providers. The Resource API is built on top of Puppet core and is easier, faster, and safer than the old types and providers method. Writing new Puppet providers based on the Resource API is even simpler by using the pdk new provider commands with the puppet Development Kit. Check the reference for more details. The “Deferred” data type This is a quite interesting and long awaited feature. Up to now the sanest method to do something on the client based on some client’s condition was to create a custom fact and use it as needed in the Puppet code. The Deferred data type allows to call Ruby functions directly from the client before the catalog is applied (they are usually evaluated on the server). This is possible only with Ruby functions as they are already shipped to the client with pluginsync. So it’s now possible to write some special function in our modules (under lib/puppet) that we want to run on the client. One of the expected use cases is to support secret lookups at catalog application time via tools like Consul and Vault. Functions imported from stdlib Some functions from the Puppetlabs-stdlib module have been moved to core: the getvar() function has been moved from stdlib, a new get() one has been introduced. the sort() function has been moved from stdlib to core and now accepts a lambda for custom comparisons the mathematical functions abs(), ceil(), floor(), round(), min(), and max() are compatible with homonymous stdlib ones with the added feature to use a lambda with a custom compare. upcase(), downcase(), capitalize(), camelcase(), lstrip(), rstrip(), strip(), chop(), chomp(), and size() and been moved to core and updated to the modern function API. New compare() function. Command puppet module build removed The puppet module build command has been removed. To build and package modules PDK should be used. Conclusions This is a list of the most important changes with Puppet 6, in our opinion. As you have seen they are mostly related to consolidation, cleanup and enrichment of the language and the platform. Upgrade from Puppet 5 is expected to be safe and painless for most of the cases. If you are using any of the types which have been moved to separated modules which are not shipped with puppet-agent, you will have to add them by yourself to your module path. Thanks to David Schmitt from Puppet for the remarks about some incorrect statements in the first revision of this post. Alessandro Franceschi 2018-09-24T00:00:00+00:00 https://example42.com/blog/2018/09/24/the_five_things_you_need_to_know_about_puppet Let’s say you know nothing about Puppet. Or maybe you already know something and have started to use it, but still don’t have a firm grasp on its components or a clear understanding of the whole picture. Eventually someone has started to introduce it in your company and you have to deal with it. Or you are a more or less experienced Puppet practitioner. You might belong to one or more of these category of Puppet users: End user, dealing only with (Hiera) Data based on given internal directives Power user comfortable with using public or internal modules and eventually write custom classes (such as profiles) Modules developer, writing modules and profiles intended for local or public use Admin, who maintains the Puppet server infrastructure Architect, who designs the data structure, the code organisation, the classification approach Bad news is that more you belong to the latter categories, more things you need to know, and their amount is not negligible. Good news is that you don’t need to know everything, in order to successfully use Puppet. Here is a list, with useful links, of the things to know, according to who you are and what you want to be, in Puppet terms. Terms, command and components Basic knowledge of Puppet ecosystem is essential for architects, admins and developers, and useful, even if not strictly necessary, for end users: Puppet terminology Components of an infrastructure Server side: Puppet Server Puppet CA (by default on the master) PuppetDB Web Frontends: Puppet Enterprise Foreman Puppetboard Puppet Explorer Hiera Client side: Puppet agent Facter The puppet commands Help from: Official documentation page The Community Support Modules If you are an end user, you might even know nothing about Puppet modules and blindly add the data you are told to manage, still for everybody else using, understanding and integrating modules is part of the job, so these concepts are given for granted: Modules structure and conventions Public modules on the Forge and GitHub stdlib and other common modules Conventions for templates and files Automatic class parameter lookup on Hiera data The puppet module command Modules development via PDK Classification Classification is how we decide what classes, from any module, to include in our nodes, in order to manage the desired resources. Basically for each Puppet managed node we have to define a list of classes to use, each class comes from modules, either public (so we don’t have to write it) or written by a developer. Once included, these classes can be parametrised via Hiera data and hence have different behaviours according to our data. End users, if allowed to define what applications to install on a node, deal with the classification approach decided by architects. Classication can be done in different ways using different, not necessarily alternative, tools: External Node Classifiers (ENC): Custom ENC script getting data from any source Classification in Puppet Enterprise console Classification in The Foreman In the main manifests dir: Node statement Facts driven class inclusion In modules: Roles module Other modules or classes which include and group other classes On Hiera Looking up a key (like classes or profiles with an array of classes to include Classification via example42’s psick module [Shameless plug] The Control Repo A modern Puppet setup is based on a control repository. If you are an architect, you designed it, otherwise you might need to work on it (if hieradata is included in the control-repo) or use it while locallh testing any development. In any case it’s important to know about it. Layout: manifests directory, where Puppet code starts to be evaluated Puppetfile, with list of external modules to use and hiera.yaml with definition of hierarchy levels and backend for data lookup Eventually the hieradata dir if backend is file based (json,yaml…) (Can stay in an external module) site directory with local modules (such as profiles) (Can stay in an external module) Change management: development environment (with pdk) testing (syntax, unit, integration tests) deployment (r10k or alike, with control-repo branches matching Puppet environments) application (alternative ways to trigger Puppet runs) Puppet DSL Strictly speaking knowledge of Puppet language is not necessary for Puppet end users which may configure their data via Hiera. Still a basic understanding of Puppet abstraction and idempotency principles and the basics of the language is enough to understand better the used modules and eventually write custom ones. The Resource Abstraction Layer: Types and providers Everything can be a Puppet resource (via the relevant module) the command puppet resource Resources, classes, defined types and modules Ruby and Puppet DSL functions Data Types Facts Native Custom facts in ruby External facts (in any language) Trusted facts via csr_attributes.yaml file Builtin variables Language operators, expressions, conditionals, iterators… Resources ordering … Enough? Enough for now :-) If you’d rather have someone walk your team through all of this on your own code, that’s what Puppet Training is for. Alessandro Franceschi 2018-09-17T00:00:00+00:00 https://example42.com/blog/2018/09/17/puppet_training_updates example42 trainers conducts the official Puppet Training courses since 2011. Since then Puppet has adopted best practices multiple times. These adoptions also had an impact on the official training courses material. The old courses In the early days everybody was using a Linux based virtual machine on every attendee laptop. The last major update was done in 2016, when the course material switched to Puppet 4 and people used a Docker container to run their course exercises. Containers were provided by the trainer. Course management was built around a self developed tooling where attendees were able to create their account (which was then used in Gitea, Puppet Enterprise Console and to instantiate the containers). The training was fully focusing on Linux distributions and Windows was explained but not actively done. All attendees were asked to have knowledge in vim. Many attendees complained about to few active doings and to deep introduction into Puppet instead of providing a good starting point for building their infrastructure. The new courses With the newest content change, Puppet renamed the Puppet Fundamentals Training to “Getting Started with Puppet”. Initialising Puppet in your infrastructure The new course setup is different. We start using bolt (the Puppet open source ad-hoc task utility) to install Puppet agent on non-managed systems within a cloud based setup. We still have a course management, but git server was switched from gitea to GitLab. Step-by-step declaration of your infrastructure The whole training content is now built around a control repository. Instead of coding everything by ourselves, we immediately introduce the Puppet Modules and the Roles and Profile pattern. In the base course we will not go too deep into the panning of a good hiera setup (this is done in an advanced course), but we will use an existing hiera configuration to separate data from code. On-Premise and off-premise usage of Puppet As the course is now cloud based, we will have hands-on exercises for managing Windows systems and different Linux distributions on AWS. This opens the course also for Windows administrators. Your code, your environment People can now choose the IDE they prefer. Depending on existing knowledge it is now possible to either use VIM or Visual Studio Code. Training Availability The new courses are available from Puppet starting at Puppetize Live in October 2018. example42 will switch to the new training courses after we have verified the setup, see our list of upcoming training courses. See what today’s Puppet Training looks like, tailored to your team’s own code and infrastructure. Martin Alfke 2018-09-10T00:00:00+00:00 https://example42.com/blog/2018/09/10/tp-install-anything-and-configure We have talked about Tiny Puppet in this blog more than 3 years ago when introducing the example42 tp module and then again at release 1.0: Introducing Tiny Puppet Preparing for Tiny Puppet 1.0 Tiny Puppet 1.0 More recently, we had a Tip of the Week on using the tp command from the cli: Tip of the Week 82 - A few steps to Tiny Puppet on the command line I think it’s time for some updates on Tiny Puppet given that we are at version 2.3. The module is stable, solid, used in several productions, it provides: The user defined types tp::install, tp::conf, tp::dir to manage any (*) application The ruby command tp with actions like tp install, tp test, tp log to check the status of the applications installed via tp (*) Any application The promise of installing any application (and by this we mean, package) on any (**) OS where Puppet can run is potentially matched. The difference between potential and real is a matter or more or less present and correct tiny data. Up to a few days ago the any was related to any application for which there’s tinydaya, now, since version 2.3.0, tp install defaults to the given title, if a package with that name is installable it tries to install it on the underlying (**) OS. (**) Any OS By Any OS we mean any OS where Puppet can run. With tp::install (tp install on cli) we can install any application, even if no tinydata is present, if there’s usable tinydata we can configure it with tp::conf and tp::dir. On Windows and MacOS respectively Chocolatey and Homebrew are used as default package provider and they are a prerequisite to install anything via tp. Usage on the cli You can install tp as a normal module, it depends on puppetlabs-stdlib and example42-tinydata: puppet module install example42-tp On Mac you need Brew installed and a valid homebrew module with the homonimous package provider: puppet module install thekevjames-homebrew puppet apply -e "class { homebrew: user => $local_user }" On Windows you need chocolatey installed: puppet module install puppetlabs-chocolatey puppet apply -e "include chocolately" Then, whatever the OS we can install locally the tp command with: puppet tp setup Now you can try to install anything. On Linux try with packages like (if tinydata is present also custom repos are created): tp install docker tp install elasticsearch tp install apache On Mac install anything brew can install, also via cask: tp install opera tp install dropdox ... Usage from cli under windows is Work In Progress. Usage in manifests The tp command might be nice to play around and test the status of apps managed via tp, but it’s inside Puppet manifests where the tp defines can give real help in configuring our applications. The tp defines In your classes, typically in your profiles for specific applications you can manage the package, service, configuration files triple with: # Manage package (and relevant repos if needed) and service tp::install { 'openssh': } # Manage main configuration file # (File content can be managed with different params: content, template, source, epp...) tp::conf { 'openssh': content => $sshd_content, } # Manage other configuration file (in main configuration directory) tp::conf { 'openssh::ssh_config': content => $ssh_content, } tp takes care of managing dependencies and using the right names and paths for the underlying OS. The tp class In order to install the tp command on a node, it’s enough to: include tp The tp class is needed and used only to install the tp command eventually and as entrypoint for hiera data for hashes of tp resources. In this way, for example, you can define what applications to install with data like: # We can define an array or an hash of tp installs: tp::install_hash: - opera - dropbox # Similarly we can define hashes and things to do with tp install and conf: tp::install_hash: elasticsearch: auto_repo: false logstash: {} # A bunch of tp::conf resources: tp::conf_hash: elasticsearch: template: profile/elasticsearch/elasticsearch.yml options_hash: cluster.name: el-1 index.number_of_shards: 2 logstash::syslog: source: puppet:///modules/logstash/syslog logstash::my_app: source: puppet:///modules/logstash/my_app We don’t actually recommend to use the tp namespace for using tp, it’s more handy to use tp defines in profiles, as needed and when needed. If you don’t want to install the tp command you can skip the inclusion of the tp class, the other defines can work the same. Custom templates and variables When using tp::conf to manage the content of a configuration file, we have at disposal, and can interpolate in our epp or erb templates, two very useful variables: $settings is an hash with the result of the tp_lookup function, which for the given app tries to get usable tinydata. This is useful to manage in the same template cross OS differences due to file paths and names. $options or $options_hash is currently just the content of the options_hash parameter passed to tp::conf. You can do with it whatever you want, according to the configured application. So, your erb template can have contents as in the following fragments from this erb template: storage: dbPath: <%= @settings['data_dir_path'] %> net: port: <%= @settings['tcp_port'] %> bindIp: <%= @options['bindIp'] %> To have an idea of the available settings, give a look to the tp::settings Data type. Generic templates for standard file formats If we like the idea of having all our configurations as (hiera) data, we can use tp conf to manage configurations using generic templates for standard file formats, like these from the psick module. On real life a quick profile to manage (with good hope to work on different OS) redis can looks like this: class profile::redis ( String $ensure = present, Hash $options = {}, String $template = 'psick/generic/spaced.erb', ) { tp::install { 'redis': ensure => $ensure, } tp::conf { 'redis': ensure => $ensure, template => $template, options_hash => $options, } } With Hiera data as easy as: profile::redis::options: slaveof: '127.0.0.1 6380' port: 6380 So is tp for me? It depends on what you need and what you know. Tiny Puppet manages packages, services, repo configurations, and files whose content is entirely up to you. It doesn’t manage any application specific resource. If you know how to configure your application, want a quick way (probably the quickest) to manage it with Puppet, and know at least Puppet basics, tp is for you. When tp (and some DIY code) can be better than using a dedicated module from the Forge? When don’t want to spent time testing a new module, add its dependencies, hoping it does all what you need When at the end there’s just to manage packages, services and files When you know exactly how our configuration files must be and want to control how they are generated (from static sources or dynamic templates with custom $options and os related $settings). When you don’t have to manage application specific resources, which are present in a dedicated module When you don’t have to manage complex setups for which a good dedicated module would deliver faster results Is it for you? tp install fortune Alessandro Franceschi 2018-09-03T00:00:00+00:00 https://example42.com/blog/2018/09/03/managing-osx-packages-with-puppet-and-homebrew Puppet ships natively with some providers to manage the package type on MacOS X: The macports provider supports installation of packages via MacPorts. The appdmg and apple and pkgdmg providers use /usr/bin/hdiutil and /usr/sbin/installer to mount DMG disk images and installs the included applications. Generally we need to specify the source from where to get the dmg files. Besides the App Store, Mac OS has not a native repository and package managers, and this makes it harder to automate the installation of software without specifying from where to get it. But we live in interesting and fecond times and Open Source third party tools can do the job. One of the most complete, popular and interesting package managers is Homebrew. Homebrew, with its big list of formulas allows easy installation of many OSS programs, but it’s thanks to extensions like homebrew-cask that becomes useful to manage common desktop applications (Like Atom, Chrome, Parallels and many more). Needless to say that in Puppet’s huge modules landscape we have modules that allows installation of HomeBrew, and, more important, ship with dedicated package providers also for Cask. In particular we tested thekevjames-homebrew which is one of the most promising fork of the original Kelsey Hightower module. It provides 3 different providers to manage packages with the brew command: The provider brew installs packages using brew install <module> without using brewcask. The provider brewcask installs packages using brew cask install <module> so looks in available casks. The provider homebrew attempts to install packages using first brew and then, on failure, brewcask. This is probably the sanest to use in most of the cases. We just need to have the homebrew module in our Puppet’s modulepath and specify one of them when declaring the packages we want to install on Mac. Puppet code would look like: package { 'atom': provider => 'homebrew', } A custom profile which installs Homebrew and has parameters to list the packages to install for a given user could be as simple as: class profile::brew ( String $user, Array $packages = [ 'docker' , 'virtualbox' , 'firefox' , 'nginx' ]. ) { # Use this class { 'homebrew': user => $user, } $packages.each | $p | { package { $p: provider => 'homebrew', require => Class['homebrew'], } } } If you use example42’s psick module you can achive the same using the generic package profile wrapper with this Hiera data: psick::pre::darwin_classes: brew: homebrew psick::base::darwin_classes: packages: psick::packages psick::packages::resource_default_arguments: provider: homebrew psick::packages::packages_list: - docker - firefox - little-snitch - parallels - atom Installing packages via Puppet on MacOsX, opening the path of Puppet management both of Mac servers and clients has never been so simple! Alessandro Franceschi 2018-08-27T00:00:00+00:00 https://example42.com/blog/2018/08/27/module_synchronization Don’t reinvent the wheel, use existing Puppet Modules to declare your infrastructure. But how to keep them in sync with upstream? How to proceed in case of urgent fixes required? This weeks Tip of the week deals with upstream developed modules, how to store them in your local infrastructure, how to synchronize them and how to add urgent patches, without loosing upstream connectivity. Storing upstream modules Using upstream modules in your control-repository is easy. You just add the wanted modules to your Puppetfile: mod 'example42/tp', :latest mod 'example42/tinydata', :latest mod 'puppetlabs/concat', '3.0.0' # postgresql requires concat < 3.0.0 mod 'puppetlabs/stdlib', :latest mod 'puppetlabs/vcsrepo', :latest mod 'puppetlabs/firewall', :latest mod 'puppetlabs/aws', :latest mod 'jdowning/rbenv', :latest mod 'trlinkin/noop', :latest mod 'puppetlabs/catalog_preview', :latest mod 'puppet/archive', :latest mod 'puppetlabs/inifile', :latest Now your heart of your infrastructure automation must access the internet and fetches modules from puppet forge. How to proceed if you want to keep modules locally? In general there are two options: use a private, local forge store modules in local git Local forge Running a local forge server is easy. You can use unibet/forge_server. Other solutions are Pulp or commercial artifacts systems like JFrog Artifactory or Sonatype Nexus Major difference between unibet forge server and the other mentioned solutions is, that forge server does not act as proxy to the real forge server. You just place the modules as tar.gz files into the forge server directory. Local GIT But how to proceed if you don’t want to run a commercial artifacts platform or a forge server? How can you easily and superfast fix issues in existing code? In this case you want to place the upstream modules in your local git server. Some enterprise Git solutions allow you to specify an upstream source when creating a new module. Some implementations will even sync periodically from upstream to your local Git repository. You definitley want a regular sync from upstream to your local working copy. Within your Puppetfile you refer to the synchronized upstream module by specifying the desired tag: mod 'puppetlabs/concat', :git => '<user>@<gitserver>:<path>/puppetlabs-concat.git', :ref => '3.0.0' # postgresql requires concat < 3.0.0 Module synchronization How do you get synchronization when using git? First you create the repositories on your local Git server and clone them locally: git clone <your localgit url>/<path>/<repo>.git git clone git@gitserver/puppet7puppetlabs-concat.git Now you switch into the empty directory and add a new remote: git remote add github https://github.com/puppetlabs/puppetlabs-concat.git From the newly created remote with the name “github” we fetch all onjects and non-objects like taks and branches: git fetch --all git pull github master Now we have identical code base compared to upstream. We now push all objects and non-objects to our local Git server: git push origin master --tags Every time when you want to upgrade, you run the last three commands on each of your synchronized Git repositories. Wait! Manual work, when we do automation? This does not feel good. Multiple module synchronization When managing many repositories it becomes error prone and time consuming when doing this in a manual pattern. Luckily there is help around: repo vcstool rosinstall python-vcs-repo-mgr go-vcs The most simple one - from my point of view - is myrepos a Perl script from Joey Hess. Within a configuration file we provide the list of modules which we want to manage additionally we provide commands which we want to execute. In the lib section we specify commands with parameters: # ~/git/.mrconfig [DEFAULT] checkout = git clone ssh://<user>@<git server>/<path>/$(basename $MR_REPO).git pull = git pull --rebase fetch = git fetch --all master = if [ $(git branch | grep master) ]; then git checkout master; git reset --hard ;git pull --rebase; fi prod = if [ $(git branch | grep production) ]; then git checkout production ;git reset --hard ;git pull --rebase; fi clean = rm -fr vendor .bundle spec/fixtures/modules remote_update = git checkout master; git fetch --all --prune; git pull github master; git push origin master --tags lib = remote () { git remote -v | grep $1 || git remote add github $1 || echo 'remote already added' } # Our control repository [puppet-control-repo] ## Upstream modules [puppetlabs-concat] remote = remote https://github.com/puppetlabs/puppetlabs-concat.git [puppetlabs-stdlib] remote = remote https://github.com/puppetlabs/puppetlabs-stdlib [puppetlabs-inifile] remote = remote https://github.com/puppetlabs/puppetlabs-inifile # ... Now you can get your whole repository synced to your local machine with just one command: mr checkout. When you return to work, you want to ensure that your local code is up to date. Run: mr master and mr prod. To add the remote you run: mr remote. Nice side effect: everybody has the same remote set! No more checking which puppet-foo module you had taken initially. To update the local Git server you run mr remote_update. Best option is to have this process running via cron using an application user to push updates. Local patching and remote PR But how to proceed if you encounter issues with released versions of a module? Usually you want to create an issue on upstream location (github) and wait for someone to fix it. Maybe you are even able to provide a PR for the issue. Buth then you have to wait until upstream creates a new release. In the menatime you can do the following to your local Git copy of the upstream module: create a local branch add changes to the branch, commit and push them locally within your puppetfile, switch to the branch name or the commit id When upstream fixes the issue you just update your local module working copy and switch back to release tags. Happy hacking, Martin Alfke 2018-08-20T00:00:00+00:00 https://example42.com/blog/2018/08/20/puppet-classification One of the most important tasks we face when working with Puppet is Node Classification, that is the action of assigning classes to nodes each class containing the resources we want to manage on a node. I used the word “assign”, in other parts I might use the word “include”, more precisely is the method used to decide what classes, with the contained resources, have to be added to the catalog compiled, usually on the Puppet server, and applied on the client node when Puppet runs. Classes are typically defined in modules and may have parameters: setting the parameter values of the classes we use on our different nodes is another other crucial Puppet activity, which is commonly done via Hiera and is worth is blog post of its own. Or a read to the Looking up data qith Hiera documentation page, at least the first part about Automatic Lookup of Class Parameters. Once we have defined what classes are “included” in what node and how they are parametrised, we have basically done our work with Puppet. Puppet nodes classification can be done in several different ways, let’s review the most common ones with a final mention to our psick approach. Node statement This is the original and still usable, even if not too much popular in these days, method. We can use the node statement in manifests/site.pp or other manifests in the main manifests directory of our control repo / Puppet environment. With this approach, we identify each node by its certname and declare all the resources and classes we want for it, as shown in the following code: node 'web01.example.com' { include ::general include ::apache } We can also use regular expressions, to group together different nodes, for example, all the nodes with name beginning with “web” can be grouped as follows: node /^web/ { include ::general include ::apache } In the past it was possible to use inheritance to manage nodes and groups of nodes, but this no more supported as it could lead to scoping issues with variables, when used incorrectly. External Node Classifier On an External Node Classifier (ENC), we can define the classes (and parameters) that each node should have in a totally separated tool, which can be on a system other than the puppet server. To configure Puppet to use an ENC, it’s enough, on the puppet server to add lines like these to puppet.conf: [master] node_terminus = exec external_nodes = /usr/local/bin/enc When configured to use an ENC with the node_terminus = exec option, Puppet runs the command specified via external_nodes and it passes as argument the client’s certname. The executed command can do anything with any language (query a web API, a Database, check file contents) and has to return a YAML output for the given certname with contents as follows: --- environment: production classes: - general: - apache: parameters: role: 'web' Here are defined the classes to include for that node, the global parameters, which will be usable as top scope variables in Puppet code and the Puppet environment to use. Puppet Enterprise, The Foreman, Puppet Dashboard and other less popular products can all work as external nodes classifiers with a Web frontend where to select what classes to include in what nodes, or group of nodes. Note however that an ENC can be of any kind, and doesn’t involve the presence of a Web interface where to configure data for clients. A YAML based ENC is as a script can be one that just makes a cat of a Yaml file with contents as the one shown before. Look here for such an example, which uses files in this directory. LDAP Since its early years, Puppet has the possibility to integrate with LDAP and retrieve the lists of classes (referenced with the puppetClass attribute) to include in nodes, which can be managed in a ldap tree where a node can inherit the classes set in a parent node (parentNode attribute). LDAP based node classification is not common and is usually not even mentioned, but it’s a viable alternative, especially where there is a robust LDAP infrastructure which users can access and manage with any kind of graphical user interface. Configuration requires some settings on the Puppet server puppet.conf as follows: [master] node_terminus = ldap ldapserver = ldap.example.com ldapbase = ou=Hosts ldapuser = cn=admin,ou=users,dc=example,dc=com ldappassword = ldapuser_password also we need to add Puppet’s schema to the LDAP server and be sure to have, in our manifests/site.pp a default node statement like: node default {} hiera We can specify the list of classes to include on a node via Hiera. Versions ago there was the hiera_include function, typically added in manifests/site.pp as follows: hiera_include('classes'). This function looks for the ‘classes’ key in Hiera (could be any name), which is expected to contain an array of classes to merge across hiera’s hierarchies and include in the relevant node. The hiera_include, as all the other hiera_* functions, is now deprecated and can be replaced by lookup, so the above line can be replaced by: lookup('classes',Array,'unique',[]).include Which is a fancy and condensed way of writing: $classes = lookup('classes',Array,'unique',[]) $classes.each | $class | { include $class } Then, we define in our hierarchy under the key named classes, what to include for each node. For example, with a YAML backend, our case would be represented with the following lines data: --- classes: - general - apache Nodeless Classification In our main manifest manifests/site.pp we have the code that the Puppet master alwayes parses first when compiling a catalog. Here we can place anything: declarations of resources we want on ALL the nodes, nodes statements (as in the example before), resources defaults, definition of top scope variables and so any valid Puppet code. Here we can also include classes without the need to use the node statement. If we write here, outside any conditional logic, something like: include general The general class (which, by convention, is expected to be defined in the file manifests/init.pp of a module called general) is included on all the nodes. Here we can also include classes according to whatever logic we want, eventually using variables in the class names. So, for example if we have a fact (or a parameter set via an ENC) called $role we can implement the roles and profiles pattern just by adding something like: include "role::${::role}" and have in a module called role different classes named according to the $role variable values we may have. PSICK module Classification In example42 we like to explore new way of doing things with Puppet and refine our ideal approach to managing infrastructures with it. The psick module has an unique approach to nodes classification based on Hiera data, featuring: Usage of hashes instead of arrays to more easily allow overrides and exceptions to the list of classes to include in each nodes Different Hiera keys to manage classes to include on different OS (so that you don’t need to add OS specific layers in your environment’s hiera.yaml) Different phases of Puppet application: an optional firstrun mode, where are defined what classes to include in the very first Puppet run, and three other phases, pre, base and profiles , classes defined for them are applied in that order (so typically in pre we include classes like proxy and repo settings which are a prerequisited for the others, in base the common classes we want on all the nodes (even if we can override them via Hiera) and in profiles the typical profile classes, as in the roles and profiles pattern). In order to use Psick classification we need to add the psick class to our nodes with whatever classification method we want and then configure everything via Hiera. Then we can use the psick module (which also provides a lot of profiles for common use cases) by setting Hiera data as follows, having different keys for Linux phases: psick::pre::linux_classes: repo: psick::repo psick::base::linux_classes: sudo: psick::sudo ssh: psick::openssh::tp mail: postfix psick::profiles::linux_classes: webserver: apache and Windows ones: psick::enable_firstrun: true psick::firstrun::windows_classes: hostname: psick::hostname aws_sdk: psick::aws::sdk psick::pre::windows_classes: hosts: psick::hosts::resource psick::base::windows_classes: features: psick::windows::features time: psick::time users: psick::users::ad psick::profiles::windows_classes: webserver: iis For each element of the above hashes, the element’s key name is used as a tag to allow override via Hiera, and the value is the class name to include (in the above example most of these class names are profiles defined in the same psick module, but can be classes from any module with any name). So for example, given the above data in common.yaml we can override to a specific node the name of the class to use to manage ssh with a node specific Hiera yaml file with a content like: psick::base::linux_classes: ssh: profile::ssh::bastion On the relevant node the class used to configure ssh (note that we used the ssh key, but this could be called in any way, not necessarily referring to the actual function) will a custom class called profile::ssh::bastion from our local profile module instead of the common one from psick module psick::openssh::tp. We can even decide to NOT manage ssh at all on a node (or group of nodes, according to where on Hiera we make the configuration) with an entry like: psick::base::linux_classes: ssh: '' which overrides and nullifies the classes defined in more general Hiera layers. You can read more about PSICK approach to classification on this blog post Roles and profiles Strictly speaking the Roles and Profiles pattern is not a classification alternative, but a way to organise classes in a flexible and composable way. In this case we just have to include a role class in a node, and this role class will itself include other classes (typically classes from a module called profile). Classification of the role class itself can be done in different ways, as the ones mentioned before: Via an ENC including the relevant role class for each of our nodes On manifests/site.pp, including the role class inside the node statements On manifests/site.pp, if we have a $role fact, with a single line like include "role::${::role}" Even via the psick module, where we can reproduce the roles and profiles pattern by defining the profiles to include only on Hiera files defined in a hierarchy level which uses the $role variable (look here for some samples) Conclusion Puppet seems complex. Puppet is complex, because it requires knowledge of many things for effective usage. Still once you understand a few key concepts everything becomes clearer and the dots start to be connected. One of this key concepts is classification: how we decide what classes have to be included on what nodes. I hope that after this reading you have a better and clearer idea on how you can manage nodes classification in Puppet, and what approach better fits your use case. Alessandro Franceschi 2018-08-14T00:00:00+00:00 https://example42.com/blog/2018/08/14/the_downside_of_naming_conventions In the past we had our servers and handled them like pets. We loved and we hated them and we gave them names so it was easy for us to recognize which system we were working on. We had names like dbmaster, dbslave01 to dbslave99, lb-ext and lb-int. Our shell prompt directly showed us the system we were working on: [dbmaster] /etc $: Some companies even have more information placed in the fqdn of a system: dc1_appf_01_t.domain.com Pets versus cattle Puppet and the pet node classification The downside of naming conventions Puppet and the certificate trust based node classification Pets versus cattle Long living systems, where we did maintenance, OS upgrades and application updates, repair broken installations are called pets. Similar to your dog or cat you look at them carefully and doing healthcare when required. You usually have only a few of them (well, some people who really love pets might have more) and you take any issue serious, trying to keep them alive as long as possible. The opposite of this are short living systems, where you don’t do OS upgrades, but reinstantiate the system with a new OS underneath. Switching back to the old version in case that something is not working. These systems are called cattle. Similar to cows, sheeps or ducks which you grow to use them afterwards. You have plenty of them and when one is going to become ill you will rarely do healthcare but remove it from the crowd and try to get a new one instead. From data center perspective you can compare pets to OS running on hardware whereas cattle are virtual machines. Puppet and the pet node classification When we introduced Puppet to manage our infrastructure we used the system hostnames for node classification. We created a long list of all nodes: node 'dbmaster' { contain role::dbmaster } node 'lb-int' { contain role::lb_int } For multiple systems with identical use cases we had the option to use regular expressions: node /dbslave\d+/ { contain role::dbslave } Everybody was happy and we had our list of 158 node classifications in Puppet. The downside of naming conventions We never saw a good naming convention which was useful when it comes to growth in numbers of nodes or applications. Additionally admins struggled with node specific declarations and - even worse - node specific modules. The naming convention makes it hard to migrate e.g. to cloud as you usually will not be able to re-use your naming convention on any cloud provider. Yes, you can set hostnames, yes, you can have your own DNS server in cloud. How to deal with autoscaling? How to deal with containers and how to deal with the next 25 new applications? Consider rebuilding everything and you will learn, that the naming convention blocks you from doing so. Puppet and the certificate trust based node classification From Puppet point of view the common name of the client certificate (which uses the fqdn if not configured to do otherwise) is only used to be a trust chain from your node to your Puppet master. But you can add more data to your certificate, using x509v3 openssl extensions. Puppet even does this automatically for you! # /etc/puppetlabs/puppet/csr_attributes.yaml --- pp_role: 'dbmaster' pp_region: 'Berlin1' pp_zone: 'vlan127' You only must take care that this information is added to the node prior it generates its Puppet certificate. An existing certificate will not be updated with the information. Now we can move on to our one line node classification: # manifests/site.pp contain "role::${trusted['extension']['pp_role']}" No more need to add all of your systems to your node classification. Every new machine just needs the new role deployed to the Puppet master and immediately gets its classification. Martin Alfke 2018-08-07T00:00:00+00:00 https://example42.com/blog/2018/08/07/cfgmgmt_and_containers When we are at customers who start thinking about containers we usually get confronted with a combination of half knowledge and weird assumptions. One of the most interesting topics usually deals with configuration management and the idea that this is no longer needed when switching to containers. This posting will explain, how Puppet can help you managing your container infrastructure. Container host Container runtime Container network Container volumes Container images Running single containers The container orchestration Docker compose Docker swarm Kubernetes Container host To run containers you must have a system at hand. Usually people will just use a cloud provider, eliminating the need of local hardware. But running on cloud needs detailed configuration like access control and authentication, besides the container host. When running locally on your own hardware you want to configure the system to be capable of running container at all. Puppet Inc. has therefor published the puppetlabs-docker module. To make use of the puppetlabs docker module, you must use a supported operatingsystem like modern CentOS, Debian or SLES releases and have a Puppet Agent running locally. The main docker class installs and configures your dockerd. Container runtime With the docker module you can choose to either install Docker CE (Community Edition) or Docker EE (Enterprise Edition). Please note that the puppetlabs-docker module does not install or configure the Docker Enterprise UCP (Universal Control Plane) nor the DTR (Docker Trusted Registry). For installation and configuration of UCP and DTR you can use the puppetlabs-ucp module Now let’s get started installing docker: # Docker CE: class { 'docker': } # Docker EE: class { 'docker': docker_ee => true, } This will give you a default docker installation from docker repositories and docker daemon is listening on socket only. If you want to reach the docker daemon from remote and locally you must declare the docker class with several parameters: class { 'docker': tcp_bind => ['tcp://127.0.0.1:4243','tcp://10.0.0.1:4243'], socket_bind => 'unix:///var/run/docker.sock', } Usually only root is allowed to access docker daemon. You can easily add other users to docker group: class { 'docker': docker_users => ['user1', 'user2'], } Other useful parameters you want to set on your docker daemon might be live-restore or configuring the storagedriver. class { 'docker': stragedriver => 'devicemapper', extra_parameters => ['--live-restore'], } Container network If you need to use a specific bridge with individual network settings you can declare the docker class with core networking parameters. class { 'docker': ip_forward => true, iptables => true, ip_masq => true, bridge => br0, fixed_cidr => '10.20.1.0/24', default_gateway => '10.20.0.1', } Next to core networking you can add docker networks (which are supported since docker 1.9 and later): docker_network { 'my-network': ensure => present, driver => 'overlay', subnet => '192.168.1.0/24', gateway => '192.168.1.1', ip_range => '192.168.1.4/32', } Container volumes Container should not have persitant data. One option for persistant data (as long as you are on one host) is to make use of volumes: docker::volume { 'registry-volume': ensure => present, } Container images We now can prepare the local docker daemon to have images available which can then be instantiated (docker container run). As of now, we can only pull images from official Docker registry as we have no private registry yet available. So let’s pull the registry image via Puppet: docker::image { 'registry': image_tag => '2.6.2', } You can also the docker::image defined resource type to build containers by yourself using a Dockerfile: docker::image { 'my_container': docker_file => '/home/docker/my_container/Dockerfile', } Running single containers Now everything is prepared to spin up a container. Let’s start the registry container: docker::run { 'registry': image => 'registry:2', ports => ['5000:5000'], volumes => ['registry-volume:/var/lib/registry'], extra_parameters => ['--restart=always'], privileged => false, env => [ 'REGISTRY_STORAGE_DELETE_ENABLED=true', 'REGISTRY_LOG_LEVEL=warn', ], } Especially for the registry container you want to also add the registry browser. We assume that both containers run on the same host: docker::run { 'registry-browser': image => 'klausmeyer/docker-registry-browser', ports => ['8888:8080'], env => [ "DOCKER_REGISTRY_URL=http://${::fqdn}:5000", 'NO_SSL_VERIFICATION=true', 'ENABLE_DELETE_IMAGES=true', ], } The container orchestration Until now we created single instances of containers on single hosts. This is not what you usually want, as this causes outage when the one hosts goes down or when docker needs maintenance (e.g. adding a configuration and restarting docker daemon). The puppetlabs-docker module can also handle swarm and compose. Docker compose Docker compose does not give you high availability. It is useful when you want to run several containers which are functional together only. e.g. you need a database and a webserver. First you need to install the compose utility: class {'docker::compose': ensure => present, version => '1.9.0', } Docker compose descriptions must be placed in a yam file, which can be easily deployed via Puppet: file { '/etc/docker-compose/teamA/web-db.yaml': ensure => file, source => 'puppet:///...', } Now we can use the Docker compose resource type to deploy the application: docker_compose { '/etc/docker-compose/teamA/web-db.yaml': ensure => present, scale => { 'web_server' => '4', }, } Docker swarm When HA is required one should look into Swarm or Kubernetes. Both are capable of managing multiple containers on multiple hosts. To activate docker swarm one first must run an initial command on the first swarm manager node. docker::swarm { 'cluster_manager': init => true, advertise_addr => $fact['networking']['ip'], listen_addr => $fact['networking']['ip'], } The token can be found in the node log file. Adding workers must use the token from forst master: docker::swarm { 'worker_1': join => true, advertise_addr => $fact['networking']['ip'], listen_addr => $fact['networking']['ip'], manager_ip => '192.168.1.1', token => '<your join token>' } Having your Docker Swarm ready allos you to deploy docker services: docker::services { nginx': create => true, service_name => 'nginx', image => 'nginx:latest', publish => '8443:80', replicas => '5', } Kubernetes Whereas Swarm integrates many configurations, Kubernetes allows you to choose the tools you need. This is done by separating APIs, Networks and Hosts into separate services (which are usually run in containers). We will cover Kubernetes and Puppet in an upcoming Tip of the Week. Martin Alfke 2018-07-30T00:00:00+00:00 https://example42.com/blog/2018/07/30/introduction_to_razor This is the first of a series of posts about Razor, an Open Source systems provisioning tool developed by Puppet. Razor automates the installation of the most common Operating Systems on bare metal servers or virtual machines, as long as they can perform a network boot via iPXE. Nodes to be provisioned need to have network boot enabled and be connected to a network where a DHCP server is configured with a valid Boot server a Bootfile name. This must be a reachable a tftp server configured to serve via network a Razor micro kernel (a minimal Linux distro) which uses Facter to collect data (facts) about the system to be provisioned and interacts with the Razor server to manage how provisioning has to be performed. Razor server is a ruby application that works on RedHat and derivatives 6 and 7, it uses PostgreSQL for data persistence and HTTP(S) for APIs and client-server traffic. It can run on nodes different from the Puppet Master, the DHCP and the tftpserver. When using Puppet Enterprise to install the Razor server it’s enough to include the pe_razor Puppet module, which is already shipped with PE. The FOSS version can be installed by lavaburn-razor module. Alternatively the razor-server package can be installed from the official Puppet repos. In its configuration file, /etc/puppetlabs/razor-server/config-defaults.yaml, we can manage database endpoints, authentication methods and how Razor behaves with newly discovered hosts (by default, now, they are considered installed, to avoid unwanted re-provisioning of existing systems) Razor uses port 8150 for HTTP communication between the server and nodes (traffic should be open to server’s port 8150 from any host to be provisioned) and port 8151 for HTTPS, used for accessing the public API (server’s port must be reachable from any system where the Razor client is used). The razor client is a Ruby gem (gem install razor-client) which provides a command-line tool (razor), that interacts with Razor server’s APIs (by default using the URL http://razor:8150/api, can be changed by the --url option or by setting the RAZOR_API environment variable). To setup provisioning via Razor we need in our Infrastructure: Nodes to be provisioned (either VMs or Physical servers) able and configured to boot via network A DHCP server configured to provision clients using a kernel image loaded via network (next-server and filename option on ISC DHCPd) A tftp server with Razor’s microkernel A Razor server installed and configured to access a PostgreSQL DB A minimal configuration of Razor resources (repos, brokers, tags, policies, tasks…) Each node is identified by its Mac address, by default, and once it boots Razor’s microkernel image, it is registered with most of its facts. According to the node’s facts, we can tag systems, matching any condition we need. We can then create a policy which correlate tags to the other basic Razor provisioning elements. Policies are rules that tell Razor what to do with a node and how it has to be provisioned: repos to use, tasks (commands, snippets of preseed / kickstart configs…) to include, and brokers (post-installation integrations) to activate, like the one that installs Puppet on the provisioned node. We will review in future posts how to configure the various server components and how to configure Razor elements. Here is a quick preview of the minimal essentials. First we can give a look around and show current Razor server configs: razor config Then we can see if there are predefined elements (there should be at least some tasks): razor tasks More generally we can specify any valid element type and eventually its name: razor <tags|brokers|hooks|nodes|policies|repos|tasks> [element_name] A minimal configuration requires setting a tag for one or more nodes, accoridng to the matching rules (based on facts) we want. So, for example, strictly matching a node with its MAC: razor create-tag --name my_node --rule '["in", ["fact", "macaddress"], "00:0c:21:21:11:43"]' Then we have to add at least a repo of an ISO of a OS to provision. Here we get the CentOS7 iso and associate it to the pre-existing centos/7 task: razor create-repo --name centos7-1804 --iso-url http://www.mirrorservice.org/sites/mirror.centos.org/7/isos/x86_64/CentOS-7-x86_64-DVD-1804.iso --task centos/7 We create a broker, a post installation activity, that is intended to hand over the provisioned system to a configuration management tool. Available broker types are: noop, puppet, puppet-pe, chef. razor create-broker --name mypuppet -c server=puppet.example.com -c environment=production --broker-type puppet We can finally create a policy, that applies the given provisioning tasks, repos and broker to the nodes that match the specified tags: razor create-policy --name centos7 \ --repo centos7-1804 --task centos/7 --broker mypuppet \ --enabled --hostname 'host${id}.example.com' \ --root-password my_root_password --max-count 20 \ --tag my_node We can review the commands given with: razor commands This has been just a gentle introduction to Razor, with a glimpse of the commands needed for basic minima configuration, in the next posts we will see more details on how to setup an use a Razor based provisioning infrastructure. Alessandro Franceschi 2018-07-23T00:00:00+00:00 https://example42.com/blog/2018/07/23/a-few-steps-to-tiny-puppet-command-line Tiny Puppet if a Puppet module that provides general defines to manage any application on any operatingsystem, using the data present in the companion Tiny Data module. Core defines Tiny Puppet on the command line Core defines The user defined resource tp::install can be used to install an application package and it’s service, if the relevant hieradata it also takes care of managing its repository data via the tp::repo define (either via a release package or directly configuring the repository data). With tp::conf it’s possible to configure an application configuration files using different formats and methods. With tp::dir whole directories can be managed, also with a source on a scm repo like git, mercurial and subversion. Finally with tp::test is possible to define any test script that checks if the relevant application is correctly working. These are defines which we can use in your classes and profiles to manage applications without the need of a dedicated module, but Tiny Puppet also offer a command line interface, which is easy to use and powerful. Tiny Puppet on the command line We can install Tiny Puppet on any system where Puppet is installed, just run, as root: puppet module install example42-tp puppet tp setup Now you can use Tiny Puppet from the command line: [root@lab ~]# tp Usage: tp <action> [app] Available actions: install: Install the given app with tp (relevant tinydata must be available) uninstall: Uninstall the given application with tp test: Test if the given (or all) application is correctly working log: Tail the logs of the given (or all) application (CTRL+C to exit) list: List ALL the applications installable with tp Applications installed via tp (available for tp test and log actions): openssh nginx ruby-dev ntpdate epel sysdig make ruby rsyslog virtualbox dkms The command shows the available actions and the list of applications that have been installed locally via tp. Via tp on the cli you can install or uninstall packages or repos. For example top install epel Repository on Redhat distributions it’s enough to write: tp install repo To install virtualbox with relevant repositories: tp install virtualbox To list all the available applications: tp list But, more useful than anything else, to test if applications are correctly working: [root@lab ~]# tp test - openssh: package openssh-server OK - openssh: service sshd OK - nginx: package nginx OK - nginx: service nginx OK - ruby-dev: package ruby-devel OK - ntpdate: package ntpdate OK - sysdig: package sysdig OK - make: package make OK - ruby: package ruby OK - rsyslog: package rsyslog OK - rsyslog: service rsyslog OK - virtualbox: package VirtualBox-5.1 OK - virtualbox: service vboxdrv OK - dkms: package dkms OK - dkms: service dkms OK The exit code of the command is 0 if everything is OK, otherwise it’s 1. You can you tp test in multiple places, for example in integration tests or canary runs during a CI pipeline to test if all the applications are locally working correctly. Finally it’s useful to use tp log to open in tail all the logs of all the applications installed via tp. It’s possible to specify an application name to just just its logs: tp log nginx. Needless to say that all the involved paths of logs, configuration files or names of packages and services are automatically detected for different OS: so using tp commands or defines we can refer to the generic application name without the need of knowing anything on how that is managed on the system. Alessandro Franceschi 2018-07-16T00:00:00+00:00 https://example42.com/blog/2018/07/16/prevent_facter_fork_bombs On April 11th 2018 we wrote about external facts. Within the mentioned posting we explained the possibilities you have and that you can use executables (like shell scripts) which will allow you to add individual facts. What we have not talked about is the facter fork bomb and long facter execution times which you can run into easily. Facter fork bomb Facter fork bomb example Facter fork bomb explanation Facter fork bomb prevention Facter execution times Facter timeout example Facter timeout prevention Facter fork bomb What is the facter fork bomb? A fork bomb means that the same process executes itsef over and over again. Facter fork bomb example Consider the following external facter shell code: #!/bin/bash $osfamily=$(facter -p os.family) case $osfamily in 'RedHat') ;; 'SLES') ;; 'Debian') ;; esac Facter fork bomb explanation You want to return different values, depending on the os.family facts. But what will happen: Facter executes the shell script. The shell script itself executes facter, which will exectute the shell script, which will execute facter,….. Welcome to your first facter fork bomb. Within a couple of seconds your system will be unusable, running on high load and eating memory. Facter fork bomb prevention Never use the facter executable in an external, executable fact. If you really must use a fact, then please consider writing a custom fact in Ruby. Here you can easily use facter.value to access available facts without the experience of a fork bomb. Facter execution times Another issue which can occur is that facter takes a hugh amount of time to collect facts. Sometimes this can be related to an external or custom fact where you try to connect to a non-performance system collecting data. Facter timeout example Consider your 200 nodes to query information e.g. from LDAP or a remote database. See the following code example: #!/bin/bash $role=$(ldapsearch -b 'cn=role, cn=$(hostname) ou=Servers, dc=example42, cd=com' -h ldapmaster.example42.com -l 3600 echo "role=$role" What happens if your LDAP server is not available, slowly responding or available over a highly saturated network link only? Facter will take long time to collect all information. This can even lead to a timeout. Facter timeout prevention Don’t access remote systems from facter. Either the remote system can not deal well with the amount of requests or single nodes can not access the remote system. Each external (executable) fact should use local resources only. Happy hacking, Martin Alfke 2018-07-09T00:00:00+00:00 https://example42.com/blog/2018/07/09/update_on_example42_modules TL;DR: We “un-deprecated” two more modules - by customer request. example42 Puppet modules in early 2018 example42 Puppet modules in mid 2018 example42 Puppet modules in future example42 Puppet modules in early 2018 In April 2018 we talked about the status of example42 modules. Within the mentioned posting we also gave a short historical summary (from 2008 to 2017) regarding example42 Puppet modules, underlying how most of the “NextGen” modules, written in 2010, were deprecated (even if most of them, written at Puppet 2.6 times, can still work under Puppet 5) and how most of their functionalities could be replaced by tp and psick. There are many good maintained modules available from voxpupuli or camptocamp, for example, and, when necessary, we collaborate with existing communities. By that time we did the minimal updates needed to ensure functionality with Puppet 5 without using any of the modern Puppet implementations like lambda or data in modules. We decided to only keep the example42 puppet-network module. Due to the reason that the network module is an approved module on Puppet forge, and a few other modules who had a dedicated maintainer, as apt example42 Puppet modules in mid 2018 In July 2018 we have been at several customers and we learned that they are using two more modules: puppet-rclocal and puppet-zabbix_agent, so we decided to keep on with their development, “un-deprecate” them, and work on the releases which make use of current Puppet syntax. The puppet-rclocal module is already upgraded to make use of data in modules with hiera 5. A new version has already been released to Puppet forge. The zabbix_agent module is under development at the moment. We removed dependencies to example42 firewall, monitor and puppi module, legacy of NextGen times, and we updated it to modern Puppet language. A new version will be pushed to the Forge very soon. example42 Puppet modules in future We are carefully reviewing any usage of our modules at customers. Generally we try to move customers to supported or approved modules (mostly from puppetlabs or voxpupuli). We will keep our list of modules short and we try to get approved status for all example42 modules which are under actual development. Happy hacking, Martin Alfke 2018-07-02T00:00:00+00:00 https://example42.com/blog/2018/07/02/pdk_how_to_adopt PDK alows you to easily get unit tests for your puppet modules. The Developer Kit is available for Linux, OS X and macOS and Windows. What is inside? Starting a new module Generate the Module Generate module content Using PDK on existing module Adopting PDK to your code Adding ruby gems Adding your own code to spec_helper Adding more facts Running individual tests What is inside? In the PDK installation package you get some Puppet and Ruby versions installed into /opt/puppetlabs/pdk. In version 1.5.0 you get Ruby 2.1.9 and 2.4.4 (installed into /opt/puppetlabs/pdk/private/puppet/ruby/) Installation of Puppet versions is spread among the ruby versions. For Ruby 2.1.09 we have Puppet 4.7.1, 4.8.2, 4.9.4 and 4.10.11 (installed in /opt/puppetlabs/pdk/private/puppet/ruby/2.1.0/gems/). In Ruby 2.4.4 you find Puppet versions 5.0.1, 5.1.0, 5.2.0, 5.3.6, 5.4.0 and 5.5.1 (installed in /opt/puppetlabs/pdk/private/puppet/ruby/2.4.0/gems/) Besides this git is bundled in version 2.14.2 in /opt/puppetlabs/pdk/private/git. Starting a new module The Puppet Developer Kit is based on a module template. The template is bundled as a bare git repository in the installer and is located at /opt/puppetlabs/pdk/share/cache/pdk-templates.git/. The module which is created by PDK uses Gemfile for installation of ruby extensions required for testing like rspec-puppet and provides CI configurations for travis and GitLab CI runner. Generate the Module The module creation process starts with asking several questions regarding the PuppetForge account name (you don’t need one, it is just a name, which is prefixed to the module name), the author name, license and supported operating systems. The creation is started by running pdk new module <modulename> e.g. pdk new module demo pdk (INFO): Creating new module: demo We need to create the metadata.json file for this module, so we're going to ask you 4 questions. If the question is not applicable to this module, accept the default option shown after each question. You can modify any answers at any time by manually updating the metadata.json file. [Q 1/4] If you have a Puppet Forge username, add it here. We can use this to upload your module to the Forge when it's complete. --> mea [Q 2/4] Who wrote this module? This is used to credit the module's author. --> tuxmea [Q 3/4] What license does this module code fall under? This should be an identifier from https://spdx.org/licenses/. Common values are "Apache-2.0", "MIT", or "proprietary". --> Apache-2.0 [Q 4/4] What operating systems does this module support? Use the up and down keys to move between the choices, space to select and enter to continue. --> RedHat based Linux, Debian based Linux, Windows (Use arrow or number (1-7) keys, pres--> RedHat based Linux, Debian based Linux, Windows Metadata will be generated based on this information, continue? Yes pdk (INFO): Module 'demo' generated at path '/Users/mea/Desktop/example42-blog/demo', from template 'file:///opt/puppetlabs/pdk/share/cache/pdk-templates.git'. pdk (INFO): In your module directory, add classes with the 'pdk new class' command. Generate module content With PDK you can create classes, defined_types and tasks. Generating providers is an experimental feature at the moment. pdk new class demo pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/manifests/init.pp' from template. pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/spec/classes/demo_spec.rb' from template. This generates a Puppet class and the according basic unit test file. Creating a self defined Puppet resource type is similar: pdk new defined_type demo::foo pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/manifests/foo.pp' from template. pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/spec/defines/foo_spec.rb' from template. Tasks are created by running: pdk new task run_demo pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/tasks/run_demo.sh' from template. pdk (INFO): Creating '/Users/mea/Desktop/example42-blog/demo/tasks/run_demo.json' from template. As you can see, PDK creates you a stub file using .sh extension. The JSON file is the description of the task. Using PDK on existing module On your module base directory just run pdk convert. This will add all required files to your module. Please note that several existing files might managed by the PDK template from now on. How do you re-enable your individual settings and configurations? Adopting PDK to your code PDK manages several files and will overwrite them when running pdk update. This is especially for Gemfile, Rakefile, spec/spec_helper.rb and spec/default_facts.yml. These files are managed and owned by PDK. But how to adopt these to your specific needs? Adding ruby gems PDK does not include the hiera-eyaml gem. If you want to run tests with eyaml, you must install the hiera-eyaml gem within PDK. Installation of Ruby gems is done via Gemfile. Any additional gem can be placed either in ~/.gemfile or in Gemfile.local. Please note that Gemfile.local is excluded from git in .gitignore file! Another solution is to use the .sync.yml file: Gemfile: required: ':development': - gem: hiera-eyaml - gem: puppet-lint-resource_reference_syntax - gem: puppet-lint-trailing_comma-check - gem: puppet-lint-variable_contains_upcase The .sync.yml file allows you to also specify additions to other files: spec/spec_helper.rb: hiera_config: 'spec/fixtures/hiera.yaml' Rakefile: default_disabled_lint_checks: - 'class_inherits_from_params_class' Adding your own code to spec_helper Using sync.yml will be less optimal when you want to add lots of content to spec_helper. In this case you can use the spec/spec_helper_local.rb file instead. Adding more facts How to add more facts to your tests. e.g. you want to make suse of an ssh module which has an ::ssh_version fact. In this case you add the additional facts to spec/default_module_facts.yml instead. Running individual tests PDK allows you to not only use pdk validate or pdk test unit. You also have the option to run specific pdk rake tasks: pdk bundle exec rake -T gives you the complete list of rake tasks available. Happy hacking, Martin Alfke 2018-06-25T00:00:00+00:00 https://example42.com/blog/2018/06/25/what-you-need-to-know-about-puppet-facts-part-5-facter_conf We end our series of blog posts about what is essential to know about Facter with some notes about the often forgotten Facter’s configuration file /etc/puppetlabs/facter/facter.conf (on Windows systems: C:\ProgramData\PuppetLabs\facter\etc\facter.conf). If you missed the previous posts, you can read them here: Part 1 - Facter and core facts Part 2 - Custom facts Part 3 - External facts Part 4 - Trusted facts The forgotten (and useful) configuration file Do not worry if you didn’t even know that Facter has a configuration file which permits the configuration of rather interesting and useful features. It has been introduced from Version 3, and by default it’s not even created when Facter is installed, so it’s up to us to create and configure it (typically via Puppet). facter.conf is written in Hocon format, and has 3 main sections: facts where we configure facts groups and policies on how to cache or distribute them global where we defines the paths where facter looks for and if to include some kind of core_facts cli where are managed facter cli command outputs A sample output may look as follows: facts : { blocklist : [ "file system", "EC2" ], ttls : [ { "operating system" : 2 days }, ] } global : { external-dir : [ "path1", "path2" ], custom-dir : [ "custom/path" ], no-exernal-facts : false, no-custom-facts : false, no-ruby : false } cli : { debug : false, trace : true, verbose : false, log-level : "warn" } Configuring caching and exclusions The most useful and interesting settings can be set under the facts section, here we can configure, with the ttls key, a caching Time To Live for each listed fact group, so that Facter doesn’t have to resolve all the facts of that group every time is executed (that is, at least at every Puppet run). Here we can also completely exclude from execution, with the blocklist key, whole groups of facts. We can see the list of all the cacheable groups, with the ttls key, (with the relevant facts) with: facter --list-block-groups EC2 - ec2_metadata - ec2_userdata GCE - gce Xen - xen - xendomains augeas - augeas - augeasversion desktop management interface - dmi - bios_vendor - bios_version - bios_release_date - boardassettag - boardmanufacturer - boardproductname - boardserialnumber - chassisassettag - manufacturer - productname - serialnumber - uuid - chassistype disk - blockdevices - disks file system - mountpoints - filesystems - partitions fips - fips_enabled hypervisors - hypervisors id - id - gid - identity kernel - kernel - kernelversion - kernelrelease - kernelmajversion load_average - load_averages memory - memory - memoryfree - memoryfree_mb - memorysize - memorysize_mb - swapfree - swapfree_mb - swapsize - swapsize_mb - swapencrypted networking - networking - hostname - ipaddress - ipaddress6 - netmask - netmask6 - network - network6 - macaddress - interfaces - domain - fqdn - dhcp_servers operating system - os - operatingsystem - osfamily - operatingsystemrelease - operatingsystemmajrelease - hardwaremodel - architecture - lsbdistid - lsbdistrelease - lsbdistcodename - lsbdistdescription - lsbmajdistrelease - lsbminordistrelease - lsbrelease - macosx_buildversion - macosx_productname - macosx_productversion - macosx_productversion_major - macosx_productversion_minor - system32 - selinux - selinux_enforced - selinux_policyversion - selinux_current_mode - selinux_config_mode - selinux_config_policy path - path processor - processors - processorcount - physicalprocessorcount - hardwareisa ssh - ssh - sshdsakey - sshrsakey - sshecdsakey - sshed25519key - sshfp_dsa - sshfp_rsa - sshfp_ecdsa - sshfp_ed25519 timezone - timezone uptime - system_uptime - uptime - uptime_days - uptime_hours - uptime_seconds virtualization - virtual - is_virtual - cloud The list of the blockable groups, via the blocklist key is shorter: facter --list-block-groups EC2 - ec2_metadata - ec2_userdata file system - mountpoints - filesystems - partitions hypervisors - hypervisors The configuration of caching or blocking of groups of facts can save a lot of time during Facter execution but we can still disable caching or blocking with, respectively, the following command line arguments: facter --no-cache facter --no-block Global and cli settings The other keys we can configure on facter.conf can be used to manage the general behaviour of the facter command, all of them can be set or overridden via arguments specified in the command line. The global settings are the following: external-dir A list of directories to search for external facts (equivalent to the cli argument --external-dir) custom-dir A list of directories to search for custom facts (equivalent to --custom-dir) no-external-facts If true, prevents Facter from searching for external facts. (Default false, when set to true is equivalent to argument --no-external-facts) no-custom-facts If true, prevents Facter from searching for custom facts. (Default false, when set to true is equivalent to argument --no-custom-facts) no-ruby If true, prevents Facter from loading facts written in Ruby. (Default false, when set to true is equivalent to argument --no-ruby) The cli settings are the following: debug If true, Facter outputs debug messages. (Default false, when set to true is equivalent to cli argument --debug) trace If true, Facter prints stack traces from errors arising in your custom facts. (Default false, when set to true is equivalent to cli argument --trace) verbose If true, Facter outputs its most detailed messages. (Default false, when set to true is equivalent to cli argument --verbose) log-level Sets the minimum level of message severity that gets logged. Valid options: “none”, “fatal”, “error”, “warn”, “info”, “debug”, “trace”. Equivalent to cli argument --log-level Conclusions We have reached the last post of our series about “What you need to know about Facter”. We have reviewed the most important things to know about Facter and also some more or less known features or capabilities. Facter has seen continuous improvements in the last years, both in terms of performance (facts written in C++ and caching/blocking from version 3) and flexibility (structured and external facts from version 2) and we hope to have helped you in understanding how to make the best use out of them. Alessandro Franceschi 2018-06-18T00:00:00+00:00 https://example42.com/blog/2018/06/18/what-you-need-to-know-about-puppet-facts-part-4-trusted_facts Welcome to part 4 of a series of posts on what is essential to know about Facter: trusted facts and their very particular nature. But before beginning, if you missed something, here are the previous posts: Part 1 - Facter and core facts Part 2 - Custom facts Part 3 - External facts Trusted facts are certificate’s extensions attributes We call trusted facts what are, more precisely, extensions to the Puppet agent x509 certificates, used in all the https communications with the server. Trusted facts must be set before Puppet is executed the first time, and once set they can’t be changed (unless the Puppet client certificate is removed recreated). This is done by editing the file csr_attributes.yaml in the confdir (so by default it’s /etc/puppetlabs/puppet/csr_attributes.yaml) where, in valid yaml format, we can set: custom_attributes, key values that are added to the CSR but won’t appear in the certificate, once signed. They are typically used for managing nodes’ auto sign extension_requests, key values which are added to the CSR and in the generated certificate. These are what we call trusted facts in Puppet world. A sample /etc/puppetlabs/puppet/csr_attributes.yaml can look like: --- custom_attributes: 1.2.840.113549.1.9.7: 342thbjkt82094y0uthhor289jnqthpc2290 extension_requests: pp_role: 'ci' pp_zone: 'lab' pp_environment: 'devel' pp_datacenter: 'us1' pp_application: 'jenkins' The $trusted hash In Puppet we have an handy $trusted variable, an hash containing information about the client certificate with the following keys: authenticated — if the catalog request was authenticated (remote, local, false) certname — the node’s certificate name domain — the node’s domain, as derived from its validated certificate name. extensions — the hash containing any custom extensions we have set in csr_attributes.yaml. Keys here are the extensions OIDs, or, if they are registered extensions, their relevant short names. There’s already a list of Puppet registered ID extensions, and it’s possible to add custom IDs to map by editing the custom_trusted_oid_mapping.yaml file. Use cases So, we can access our trusted facts with $trusted[extensions][<EXTENSION OID>], the above sample csr_attributes.yaml file would generate a $trusted variable as follows: { 'authenticated' => 'remote', 'certname' => 'jenkins.lab.psick.io', 'domain' => 'lab.psick.io', 'extensions' => { 'pp_application' => 'jenkins', 'pp_datacenter' => 'us1', 'pp_environment' => 'devel', 'pp_role' => 'ci', 'pp_zone' => 'lab', '1.3.6.1.4.1.34380.1.2.1' => 'ssl-termination' }, 'hostname' => 'jenkins' } We can directly refer to trusted facts in our Hiera hierarchies: - "nodes/%{trusted.certname}.yaml" - "roles/%{trusted.extensions.pp_role}.yaml" - "zones/%{trusted.extensions.pp_zone}.yaml" Or in our Puppet code: if $trusted['extensions']['pp_role'] { $role = $trusted['extensions']['pp_role'] } if $trusted['extensions']['pp_zone'] { $zone = $trusted['extensions']['pp_zone'] } If we had the above variables set at top scope, in a place like manifests/site.pp we could have an hierarchy equivalent to the previous example, which looks like: - "nodes/%{trusted.certname}.yaml" - "roles/%{::role}.yaml" - "zones/%{::zone}.yaml" Conclusions We have seen that trusted facts are hardcoded in the Puppet client certificate, that can be set by editing the csr_attributes.yaml file before launching Puppet the very first time on a node. They can’t be altered, unless the client ssl certificate is cleaned and regenerated with updated attributes, so we can decide to use them or not according to our use cases. If we decide to use them in our Hiera hierarchies we need a way to automate the provisioning of different nodes with different trusted facts. With this approach we can decide to use such custom facts to configure and classify our nodes entirely via Hiera. The same can be achieved with normal custom or external facts, which have the benefit or defect of being more easily changeable during a node’ lifetime. As usual different approaches are possible, according to our needs, what’s essential is to know possible alternatives and available options. Alessandro Franceschi 2018-06-11T00:00:00+00:00 https://example42.com/blog/2018/06/11/what-you-need-to-know-about-puppet-facts-part-3-external_facts This week continues our journey inside what’s worth knowing about Facter. In the first post we introduced its basic features and the Core facts, in the second post we described how to write custom facts in Ruby language. Now we are going to give a look to an even easier way to create custom facts: external facts External facts They have been introduced in Facter 1.7, inheriting a similar functionality that was proposed via the stdlibe module. External facts can be texts in ini file, yaml or json format, or simply commands or scripts, written in any language for which is available a local interpreter, that respectively contain or return key-values. Writing an external file is as easy as placing a file like /etc/facter/facts.d/myfact.txt with a content like: myfact=myfactvalue The same fact can be placed in a Yaml file called /etc/facter/facts.d/myfact.yaml and have content like: --- myfact: myfactvalue Or be a Json file like /etc/facter/facts.d/myfact.json with content like: { "myfact": "myfactvalue" } Note that actually the name of these files has not to be the same of the fact, what’s important is the content of the txt, or json or yaml, the name of the key being the name of the fact. We can have even more than one fact definition in a single file, and it’s trivial to write structured facts (check previous posts for explanations) in Yaml or Json files: --- classification: role: webserver env: prod zone: dc1 Executable Facts External facts can be also expressed as the output of a command, on Unix derivatives it’s enough to have an executable file under /etc/facter/facts.d/ that outputs the name of the facts and its value (in ini like style: name=value): #!/usr/bin/env bash echo connected_users=$(/usr/bin/who | wc -l | tr -d ' ') Any language can be used to write external facts as executables, but it’s required to specify the path of the interpreter to use (here bash) in the first line, with a shebang (#!). On Windows executable facts can be placed in files with the following extensions: .com or .exe for binary executables .bat or .cmd for batch scripts .ps1 for PowerShell scripts Up to now we mentioned the directory /etc/facter/facts.d but actuslly FGacter looks for external facts in different directories. On Unix/Linux they can be placed under: /opt/puppetlabs/facter/facts.d/ /etc/puppetlabs/facter/facts.d/ /etc/facter/facts.d/ On Windows they can be placed under: C:\ProgramData\PuppetLabs\facter\facts.d\ When running Puppet as a non privileged user, external facts are looked in: <HOME DIRECTORY>/.facter/facts.d/ Contrary to custom facts written in Ruby, which can be seen from the command-line only by running facter -p, external facts are visible, as core facts just by running facter. Shipping external facts As we have seen, it’s enough to place a file in one of the mentioned directories to create an external fact. This can be done during the nodes’ provisioning, and might be a way to define how different can be our nodes in our infrastructure according to facts values like role, env, zone, datacenter, application or similar. We can also ship external facts directly in our modules: as the lib directory of modules (ciontaining Puppet types and provides, ruby facts and functions) is automatically distributed to clients via Puppet pluginsync mechanism, the content of the directory facts.d of a module is also copied to clients, and the external facts there are evaluated since the very first Puppet run. Note that any external file placed in a module’s facts.d directory is copied as is to every client, so there’s no way to ship different facts with different values to different clients. The obvious consequence is that generally in modules the shipped external facts are executables, that and run and compute their values on the clients. We can also shop external facts via Puppet using the file resource and placing a file in one of the available dirs for external facts. In this case the content of such file can be derived from a template and be different according to our nodes, but note that if we follow this approach, the relevant facts are available only starting from the second Puppet run, as the first one is needed to create them (no pluginsync is involved in this case). Alessandro Franceschi 2018-06-04T00:00:00+00:00 https://example42.com/blog/2018/06/04/what-you-need-to-know-about-puppet-facts-part-2-custom_facts In the first post of this series about Facter, we introduced its basic features and we talked about Core facts, the ones shipped directly with Facter and available whenever we use Puppet. In this post we will see more details about facts that we can write in Ruby by ourselves for whatever purpose we may have. Custom facts Custom facts are shipped with Puppet modules, if we use already Puppet it’s likely we are already using some custom fact present in one of the public modules we might be using. They are written in Ruby language and have to be placed in the directory lib/facter of a module. The simplest example of a fact is one that just executes a command and shows its output: Facter.add('connected_users') do setcode do Facter::Core::Execution.execute('/usr/bin/who | wc -l') end end This fact is called collected_users, this is the argument passed to the Facter.add method. Whatever code we want to run in order to calculate the value of this fact, has to stay inside the setcode statement. In this case we just want to run a shell command, and we use the dedicated Facter::Core::Execution.execute method for this (which takes care of wrapping our command as needed). The argument passed here is the command we want to execute (/usr/bin/who | wc -l). The output of this command is the value of our collected_users fact. In order to ship and use this fact, we have to place the above piece of Ruby code in a file called lib/facter/collected_users.rb of a module. If we have such a module in the $modulepath of our Puppet Server (for example the directory /etc/puppetlabs/code/modules for modules available to all Puppet environments, or /etc/puppetlabs/code/environments/production/modules for modules available for the, default, production Puppet environment), this fact would be automatically copied to each client, before running the catalog request. This means that the fact is immediately available to the client and can be used straight on in our Puppet manifests. This automatic copy of each custom fact (and other extensions placed in the lib directory) of a module is called pluginsync and we actually see it happening when we run Puppet, with an output like: Info: Using configured environment 'production' Info: Retrieving pluginfacts Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/connected_users.rb]/ensure: defined content as '{md5}d4cfb32bbc71e8d738004e584b0ac8bf' [...] which is telling us that the connected_users.rb file present in the lib/facter directory of one of our modules has been copied to the /opt/puppetlabs/puppet/cache/lib/facter/ directory of the client. Note that custom facts are not visible when we run the facter command from the local cli. We need to specify -p (--puppet) argument, or, recommended starting from Facter 3, use instead the puppet facts command: root@client:~# facter connected_users root@client:~# facter -p connected_users 1 root@puppet:~# puppet facts | grep connected_users "connected_users": "1", Facts confinement What’s wrong about the fact we just wrote, if we work in a multi OS environment? It runs a shell command which is available under Linux / Unix, but this is what happens when we run it under Windows: Info: Retrieving plugin Notice: /File[C:/ProgramData/PuppetLabs/puppet/cache/lib/facter/connected_users.rb]/ensure: defined content as '{md5}d4cfb32bbc71e8d738004e584b0ac8bf' Info: Loading facts Error: Facter: error while resolving custom fact "connected_users": execution of command "/usr/bin/who | wc -l" failed: command not found. and we don’t want unnecessary (in this case not blocking) errors, right? We can use the confine statement, which restricts the execution of the fact only on systems that match the given condition, based on another fact. For example, to confine our connected_users fact to run only on Linux we can use the kernel core fact: Facter.add('connected_users') do **confine :kernel => 'Linux'** setcode do Facter::Core::Execution.execute('/usr/bin/who | wc -l') end end Facts within Facts We can refer to and use the values of other facts inside our custom facts code. For example we can assign to a local variable, the value of a fact with: k = Facter.value(:kernel) Facts precedence We can have multiple Facter.add statements with the same name, they can have different confinements so that we can resolve the possible values of a fact using different code and logic. We can also have different entries for the same confinement group, in this case we need a way to decide what’s the actual value to use for a fact, and here comes handy the concept of Facts’s weight which defines the wight to give to a give, if we have valid values for two different fact names, the one with higher weight “wins” and provides the relevant value. A good example of usage if confine and has_weight is the code of the core fact virtual in Ruby (so in Facter version 2, as the equivalent in version 3 is written in C++). This can be seen here. Note that we have, in the virtual.rb file, multiple blocks like: Facter.add("virtual") do confine :kernel => 'XXX confine: :XXX => 'XXX' has_weight XXX end their combination is used to provide the final value of the virtual fact for different OS and different Hypervisors. Structured facts and aggregated resolutions In the fist part of this series about What you need to know about Facter, we have seen that since Facter version 2, with new acceptable data types (Integer, Float, TrueClass, FalseClass, NilClass, String, Array, Hash. ) as values, Structured facts were introduced. At the same time, aggregated resolutions have been introduced. This is a way to compose the content of a structured fact but executing different chunks of code. A fact with aggregated resolution may look like: Facter.add(:connected_users, :type => :aggregate) do chunk(:a_name) do [..] # Ruby code that get some information from the system end chunk(:another_name) do [..] end aggregate do | chunks | # Optional aggregate block that massages the output ProgramData end end A glorious example of structured fact using aggregated resolution is the Ruby version of the $::os fact. Note that in this case different chunks are defined and simply merged in the final result, without the use of the aggregate method. Conclusions We have seen the basics to create custom facts in Ruby. Give a look on Puppet docs site to the Overview of custom facts with examples, and the Custom facts walkthrough, and check the version 2 branch of facter for, probably, the most advanced examples (being also the core facts) of fact written in Ruby. Next week we are going to see a much simpler way to create custom facts: external facts. If you don’t know much about them, prepare to have nice surprises. Alessandro Franceschi 2018-05-28T00:00:00+00:00 https://example42.com/blog/2018/05/28/what-you-need-to-know-about-puppet-facts-part-1-core_facts Whoever works with Puppet is probably familiar with facts. In this series of posts we are going to review the different kind of facts we can deal with, when working with Puppet, hopefully clarifying grey zones or revealing new informations. Facter basics Whenever we install Puppet’s agent on a node, another tool, called Facter is installed. Its task it to collect information about the system, any Operating System, and present them as facts. We can run it from the command line and see the facts it collects on our system, it shows them in alphabetical order: $ facter aio_agent_version => 5.3.2 [...] timezone => CEST virtual => virtualbox Facter is executed at the beginning of every Puppet run and the collected facts are included in the catalog request by the Puppet agent to the Puppet server. After receiving a client’s catalog request, the Puppet server does the following: [Usually] stores the client’s facts on PuppetDB, so that they can be queried and visualised in some web frontend Uses the facts, together with our code and data, to compile the client’s catalog which is then sent back So the actual catalog applied during a Puppet run on the client, can be based on data coming from the client itself, collected as facts. Facts are or can be used in a lot of places in our Puppet control-repo: In public or local modules facts like os, operatingsystem, osfamily, kernel... are used to provide correct resource names (packages, paths, services…) and behaviour for different OS. Modules can provide custom facts, for example application’s version or anything needed for the managed resources In our profiles we may create and add custom facts used according to our needs In our Hiera’s hierarchy we often use facts, eventually custom ones, based on local topologies In our main manifests we may set global resource defaults or set top scope variables based on the values of facts In our main manifests we can manage nodes classification based on some local fact (include "role::${role}") On web frontends like Puppet Enterprise Console, Foreman we may configure and classify nodes based on facts values Any web frontend able to access PuppetDB can have a function of an Inventory system, giving easy visibility to [semi] realtime, easily customisable, facts about the system Facter can be installed as standalone tool, without Puppet, and be integrated with 3rd party software, as system’s data collector (more suitable for relatively static informations than data series) We have different kind of facts: Core facts are shipped with Facter itself, we can find them in any Puppet installation Custom facts are written in Ruby and can be shipped in modules External facts are simple text files or commands (in any language) which also can be shipped in modules Trusted facts are extension requests added to Puppet’s client SSL certificates. They must be defined before the very first Puppet run and once set, can’t be changed (unless the client certificate is recreated and resigned) Let’s start, in this post to review: Core facts These are the typical and most common facts we deal with when using Puppet, they are shipped with Facter itself and, starting from Facter version 3 they are written in C++ and are much faster to generate. Earlier Facter versions had facts written in Ruby with a structure and content similar to the one we can use in Custom facts (we will review them in the next post). Starting from Facter 2, facts can have values different from simple strings, so structured facts have been introduced. In Puppet 3 a neater distinction has been promoted: New, mostly structured, modern facts are the recommended onesaas The, widely used, older legacy facts are still supported, but not visualised by default from the command line. So, for example, legagy facts, like operatingsystem, osfamily, architecture are now replaceable with subkeys of the modern, structured fact os: [vagrant@git ~]$ facter os { architecture => "x86_64", family => "RedHat", hardware => "x86_64", name => "CentOS", release => { full => "7.5.1804", major => "7", minor => "5" }, selinux => { config_mode => "enforcing", config_policy => "targeted", current_mode => "enforcing", enabled => true, enforced => true, policy_version => "28" } } Being an hash, we can access the values of its subkeys as we normally do in Puppet code: class report_os { notice ("Operating system: ${::os['name']}") if $::os['family'] == 'RedHat' and has_key('os','selinux') { notice ("Selinux current mode ${::os['selinux']['current_mode']}") } } We have actually different ways to access to facts in Puppet code: Referring directly to them: $factname Using the $facts hash: $facts[$factname] Using the fact function from stdlib module (which allows dotted notation and doesn’t fail if we try to access a non existing subkey): fact($factname). So for example, the legacy fact $osfamily can be expressed also with any of these alternatives: The relevant key in the structured fact os: $os['osfamily'] The relevant key and subkey in the facts hash: $facts['os']['osfamily'] The function fact with subkey expressed in dotted notation: fact('os.osfamily') The most commonly used legacy facts are: operatingsystem ( same of $::os['operatingsystem'] ) osfamily ( same of $::os['family'] ) operatingsystemrelease ( same of $::os['version']['full'] ) architecture ( same of $::os['architecture'] ) ipaddress ( same of $::networking['interfaces']["${::networking['primary']}"]['ip'] ) fqdn ( same of $::networking['fqdn'] ) hostname ( same of $::networking['hostname'] ) domain ( same of $::networking['domain'] ) The most interesting modern facts: os: Basic system info networking: Networking info and status disks: Disks layout filesystems: Filesystems layout identity: Info about the user running Facter memory: Info about system memory processors: Info about CPUs system_uptime: Uptime, in various units ssh: System’s ssh host public keys in various formats timezone: The system’s timezone virtual: Name of Hypervisor, or ‘physical’ for physical machines Digression on facts and local class variables All the facts are available as top scope variables when we refer to them in our code, so, for example, the os fact can be expressed with the $::os variable. Note the leading :: which ensures we are referring to a top scope variable called os. We can also use just $os, but doing so we are not sure if we refer to the os fact or a local class variable with the same name. The better you give the idea, a code like this: class test_fact { notice("\$::timezone is: ${::timezone}") notice("\$timezone is: ${timezone}") $timezone = 'Local change' notice("\$::timezone, after local override is still: ${::timezone}") notice("\$timezone, after local override is now: ${timezone}") } include test_fact generates this output: Notice: Scope(Class[Test_fact]): $::timezone is: CEST Notice: Scope(Class[Test_fact]): $timezone is still : CEST Notice: Scope(Class[Test_fact]): $::timezone, after local override is still: CEST Notice: Scope(Class[Test_fact]): $timezone, after local override is now: Local change Conclusions In this first part of a a series of blog posts, we have seen the basic of Facter and its core facts, with particular emphasis on the difference between modern and legacy facts. Next week we will start to see how we can create our own facts, customised to our needs. Facts, not only words! Alessandro Franceschi 2018-05-21T00:00:00+00:00 https://example42.com/blog/2018/05/21/migrate_from_hiera3_to_hiera5 Hiera is the Puppet implementation of separating code from data. This concept allows you to describe your infrastructure in code and to let you put differences among your platform into the data source: e.g. different NTP servers or different password for databases. With Puppet 4.9 Hiera config version 5 was introduced. The new Hiera version allows you to place data globally, in environments and modules. As of Puppet 4.9.2 Hiera has the following data backends included: yaml json eyaml (hiera-eyaml gem still required) This posting describes how to migrate from Hiera config version 3 to the new usage of Hiera data levels. Location of Hiera configuration file and data Global data Environment data Module data Content of Hiera configuration file Migrating from hiera*() to lookup() function How to upgrade when using another data backend Location of Hiera configuration file and data With new version of Hiera you can have three different layers of data: global data environment data module data Global data Global Hiera data are the same as they have been with the older Hiera version. Data in environments allow you to stage data and hiera config changes. Data in modules are a replacement for params pattern and inheritance. Global Hiera uses a global configuration file which must be placed in the Puppet configuration directory (usually /etc/puppetlabs/puppet/hiera.yaml). If you keep this configuration file in place, the provided Hiera data will have highest priority! Don’t forget to remove the global Hiera config file, once the migration has been finished. The location of the global data is either in a separate directory (e.g. /etc/puppetlabs/hieradata) or already within your environments ( /etc/puppetlabs/code/environments/${environment}/hieradata). Consider migrating from a global path to an environment path prior migrating to new Hiera. Environment data Next layer is environment data. The required Hiera configuration file is within each environment (e.g. /etc/puppetlabs/puppet/code/environments/production/hiera.yaml). Usually data are also placed within the environment in a data directory ( e.g. /etc/puppetlabs/code/environment/production/data). Please note that you can not copy your existing global hiera config file to an environment, You must use Hiera config version 5 in an environment hiera.yaml file. Module data On module level it is also possible to have a hiera configuration file at module root (e.g. /etc/puppetlabs/code/environments/production/modules/ntp/hiera.yaml). This layer is usually only used by module authors and you are encouraged to overwrite the provided data within your environment data. Content of Hiera configuration file In older hiera configuration files backends and hierarchies were separated settings. First you provided an array of used backends and then you listed the hierarchies. Backends were searched in order of occurrence in the configuration file and then the hierarchies got queried for data. The new hiera config allows you to specify backends globally (as default) or on a per hierarchy level. Let’s assume the following existing Hiera config v3 file: # Hiera config v3 :backends: - eyaml - yaml :yaml: :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata" :eyaml: :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata" :pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem :pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem :hierarchy: - "nodes/%{trusted.certname}" - "location/%{facts.whereami}/%{facts.group}" - "groups/%{facts.group}" - "os/%{facts.os.family}" - "common" :logger: console :merge_behavior: native :deep_merge_options: {} With hiera config v5 the :logger:, :merge_behavior: and :deep_merge_options: settings are no longer used and can be removed. Next you have the option to specify default lookup options like datadir and the data backend. Afterwards the hierarchies get listed. Single hierarchies can make use of different data backends. # Hiera config v5 version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: "Per-node data (yaml version)" path: "nodes/%{trusted.certname}.yaml" # Add file extension # Omitting datadir and data_hash to use defaults. - name: "Per-group secrets" path: "groups/%{facts.group}.eyaml" lookup_key: eyaml_lookup_key options: pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem - name: "Other YAML hierarchy levels" paths: # Can specify an array of paths instead of a single one. - "location/%{facts.whereami}/%{facts.group}.yaml" - "groups/%{facts.group}.yaml" - "os/%{facts.os.family}.yaml" - "common.yaml" As you can see it is now possible to group hierarchies which use the same backend. In this special case it is also possible to completely remove the yaml backend and simplify the configuration file, as the eyaml backend is also capable of returning unencrypted values: # Hiera config v5 - eyaml only version: 5 defaults: datadir: data lookup_key: eyaml_lookup_key options: pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem hierarchy: - name: "All hierarchies" paths: - "nodes/%{trusted.certname}.yaml" # Add file extension - "location/%{facts.whereami}/%{facts.group}.yaml" - "groups/%{facts.group}.yaml" - "os/%{facts.os.family}.yaml" - "common.yaml" Migrating from hiera*() to lookup() function The new lookup function provides a huge amount of possible usages. The most impressing one is that the function will query hiera for lookup and merge options prior doing the real lookup on a key. First let’s get an overview on explicit lookup function usage: lookup type hiera v3 hiera v5 with merge options hash hiera v5 with Data type, default and merge option single hiera(‘key’) lookup(‘key’) lookup(‘key’, DataType) array hiera_array(‘array’) lookup(‘array’, {merge => unique}) lookup(‘array’, Array, unique, []) hash - first found values hiera_hash(‘hash’) lookup(‘hash’, {merge => hash}) lookup(‘hash’, Hash, hash, {} ) hash - merged values hiera_hash(‘hash’) lookup(‘hash’, {merge => deep}) lookup(‘hash’, Hash, deep, {} ) include hiera_include(‘classes’) lookup(‘classes’, {merge => unique}).include lookup(‘classes’, Array, unique, [] ).include If you have more complex decisions on when to do deep lookups, you have the option to place the lookup behaviour into your hiera data: lookup_options: 'keyname': merge: 'merge_option' e.g. # common.yaml lookup_options: ntp::servers: merge: 'unique' Then you can always use the simple lookup function to query data. How to upgrade when using another data backend Hiera has changed the way how lookup backends are working. The old hiera 3 backends like hiera-mongodb and hiera-file are no longer working on hiera v5 in environment level. It is possible to use them on global level and specify the backend to use: # Only working on global hiera level - name: "Per-node data (MongoDB version)" path: "nodes/%{trusted.certname}" # No file extension hiera3_backend: mongodb options: # Use old backend-specific options, changing keys to plain strings connections: dbname: hdata collection: config host: localhost If you don’t want to use the global hiera level anymore, you must use hiera v5 style lookup functions. Ask the authors of the backend used, whether they are able to provide a hiera config 5 compatible solution. For file backend there is a solution posted at the GitHub issue. Many thanks to Igor Galic for the solution. Still running Hiera v3 somewhere? A Puppet Health Check will flag it along with the rest of your upgrade readiness gaps. Happy hacking, Martin Alfke 2018-05-14T00:00:00+00:00 https://example42.com/blog/2018/05/14/puppet-custom-data-types One of the most powerful features introduced with Puppet 4 is the new type system. For every variable or parameter in Puppet can be defined the type of data we can expect for it. We typically use the type system to validate the kind of data expected for classes or defines parameters. For example, Puppetlabs’ ntp module has the ntp class which has parameters like these: class ntp ( Boolean $broadcastclient, Stdlib::Absolutepath $config, Optional[Stdlib::Absolutepath] $config_dir, String $config_file_mode, Optional[String] $config_epp, Enum['running', 'stopped'] $service_ensure, Variant[Boolean, Integer[0,1]] $tos_cohort, ... These few lines give us a good idea of how flexible the type system is, and how it can be extended and customised. Besides the most common data types like: Boolean (matches a boolean value, either true or false, without quotes otherwise they become Strings) String (matches any string) Hash (matches an hash (an unordered list of key values)) Array (matches an ordered array of elements) Integer (matches a integer number) Puppet has natively many more useful data types: Optional allows to accept a valid value (of the data type specified in square brackets) or an undef value. Enum accepts a list of strings, as defined, separate by comma, in the square brackets. Variant accepts different data types, separated by comma. In the above example a Boolean or an Integer from 0 to 1. For a more extensive list of the native data types you can heck the official documentation. In the above code fragment, we can see an “unusual” data type: Stdlib::Absolutepath, this is a case of a custom data type, defined in the stdlib module, under the path types/absolute.pp with content as follows: type Stdlib::Absolutepath = Variant[Stdlib::Windowspath, Stdlib::Unixpath] What’s nice here is that we can compose and use different data types, even custom ones (as Stdlib::Windowspath and Stdlib::Unixpath) and ship them directly in a module. For example, in our psick module we have created, under types/ensure.pp a Psick::Ensure data type which we use to manage the ensure parameter of a package resource, its content looks like: type Psick::Ensure = Variant[Enum['present', 'absent', 'installed','latest'],Pattern[/\d+(\.\d+)*/]] This accepts either a string as defined in Enum, or a Regular expression matching version numbers. Another interesting data type is Struct which allows to validate the type of each value of the keys of an Hash. We use it in Tiny Puppet, to validate the list of settings which can be used to override an application’s tinydata: Here is is, as defined in tp module’s types/settings.pp: type Tp::Settings = Struct[{ Optional[package_name] => Variant[String,Array], Optional[package_ensure] => String, Optional[package_provider] => String, Optional[service_name] => Variant[String,Array], Optional[service_enable] => Boolean, Optional[service_ensure] => Enum['running', 'stopped'], Optional[process_name] => String, Optional[process_extra_name] => String, Optional[process_user] => String, Optional[process_group] => String, Optional[config_file_path] => Stdlib::Absolutepath, Optional[config_file_owner] => String, Optional[config_file_group] => String, Optional[config_file_mode] => String, Optional[config_dir_path] => Stdlib::Absolutepath, Optional[config_dir_owner] => String, Optional[config_dir_group] => String, Optional[config_dir_mode] => String, Optional[config_dir_recurse] => Boolean, Optional[log_file_path] => Stdlib::Absolutepath, Optional[pid_file_path] => Stdlib::Absolutepath, Optional[init_file_path] => Stdlib::Absolutepath, Optional[log_file_path] => Stdlib::Absolutepath, Optional[conf_dir_path] => Stdlib::Absolutepath, Optional[data_dir_path] => Stdlib::Absolutepath, Optional[plugins_dir_path] => Stdlib::Absolutepath, Optional[modules_dir_path] => Stdlib::Absolutepath, Optional[tcp_port] => Variant[String,Integer], Optional[udp_port] => Variant[String,Integer], Optional[nodaemon_args] => String, Optional[dockerfile_prerequisites] => String, Optional[package_prerequisites] => Array, Optional[tp_prerequisites] => Array, Optional[exec_prerequisites] => Hash, Optional[exec_postinstall] => Hash, Optional[repo_package_url] => String, Optional[repo_package_provider] => String, Optional[repo_url] => String, Optional[repo_namel] => String, Optional[key] => String, Optional[key_url]=> String, Optional[include_src] => String, Optional[apt_repos] => String, Optional[apt_key_server] => String, Optional[apt_key_fingerprint] => String, Optional[apt_release] => String, Optional[apt_pin] => String, Optional[yum_priority] => String, Optional[yum_mirrorlist] => String, Optional[zypper_repofile_url] => String, }] Note that each key an be optional (so the parameter passed can be an hash with any of the above keys) and must adhere to the defined type, either native or custom, like Stdlib::Absolutepath. We find the Struct type particularly useful when using the templates + options hash pattern. Another very useful data type is Sensitive which, when used, hides the relevant value from being shown in reports and logs. You can use it to manage single values (and avoid to see them in reports) or for whole files, when using the content argument for a file resource: file { '/etc/secret': content => Sensitive(template("${module_name}/secret.erb")), } In this way even if the content of the file changes in a Puppet run, you won’t see it and its diff with potential sensitive information. There’s a lot more to talk about data types, and it’s definitively worth reading the official documentation and giving a look to the ones provided by puppetlabs-stdlib module, as they cover some quite common use cases (Stdlib::Ip_address, Stdlib::HTTPSUrl, Stdlib::MAC to name a few). Give a type to your data, it makes your code more robust and reliable. Alessandro Franceschi 2018-05-07T00:00:00+00:00 https://example42.com/blog/2018/05/07/testing-puppet-enterprise-2018-1-0 A few days ago a new release of Puppet Enterprise (PE), the first of 2018, has been published. This is a LTS release which is going to be supported until 2020. The major improvements, compared to the previous release, are in the Tasks and Plans integrations, with improved Role Based Access Control to manage with more granularity what tasks can be run by each user. One of the most valuable selling points of Puppet Enterprise have been the visualisation options that really give a meaning to the term “infrastructure awareness” (facts, reports, events, catalog graphs… there’s really a lot of info to view on PE web interface), these have been coupled by the current management of Puppet Jobs and tasks, giving a single interface from where to control and monitor our infrastructure. We can make tasks for any “one shot” action we may need to execute on one or more servers: from system upgrades to reboots, from applications deployments to database migrations. Doing tasks for these operations is trivial, we can even reuse our custom scripts, and have a single frontend from where to trigger such tasks, with a robust authorisation scheme, and complete overview of the result of their actions and the status of the infrastructure. Trying Puppet Enterprise Puppet Enterprise can be downloaded and used for up to 10 nodes for free. This is enough to have an idea of how it works and eventually to manage very small shops. You can download the packages for RedHat 6 or 7, SLES 12 or Ubuntu 16.04 from Puppet site (note: these are the only supported OS for the PE server components, the Puppet agent is available for many more OS. You can also test it on the Learning VM or, if you have your infrastructure on AWS cloud you can also use try Opsworks for Puppet Enterprise. Testing local code on Puppet Enterprise with PSICK You can finally test a fully operational Puppet Enterprise infrastructure under Vagrant (based on Virtualbox) using example42’s PSICK control-repo. A single commit is what we needed to add support for it in one of the Vagrant environments available. In order to test a PE infrastructure with PSICK on Vagrant you need: Puppet (you can install it with bin/puppet_install.sh) The r10k and optionally hiera-eyaml and deep_merge Ruby gems Vagrant (you can install it with the needed plugins with bin/vagrant_setup.sh). If you want to install the plugins manually just run: vagrant plugin install vagrant-vbguest vagrant plugin install vagrant-pe_build vagrant plugin install vagrant-hostmanager Virtualbox These are the commands to run to setup locally a full featured control repo based on PSICK: git clone https://github.com/example42/psick cd psick r10k puppetfile install -v cd vagrant/environments/pe vagrant up puppet.pe.psick.io This can take some time, if you don’t have already locally the used Vagrant box. There’s a known issue, during first time provisioning: you will probably get an error like: Stderr from the command: bash: line 4: /vagrant/.pe_build/puppet-enterprise-2018.1.0-el-7-x86_64/puppet-enterprise-installer: No such file or directory Don’t worry, it happens only when you provision PE the first time. You just have to: vagrant reload puppet.pe.psick.io vagrant provision puppet.pe.psick.io This will proceed with the installation of PE on the local VM, at its end you should be able to access PE web interface by browsing to https://127.0.0.1:1643 and login with user admin and password puppetlabs (you can ignore your browser warnings about insecure https connection: Puppet CA’s self signed certificates are used). The last command, after having installed Puppet Enterprise (thanks to the vagrant-pe_build plugin) triggers also a normal Puppet run on the node, using directly the contents of the PSICK control repo. Once the command has ended you have a full configured PE server, which can serve the other nodes present in the same vagrant environment (they are configured in vagrant/environments/pe/config.yaml). Note that the PE Vm is not thin, by default we allocate 4GB of RAM to it (this can be configured too on the config.yaml file) and to this you have to add the memory used by each client VM: an host with at least 8, better 16GB, is recommended. Now you can start other VMs from the same Vagrant environment, they will automatically connect and autosign to the PE server and apply their Puppet code and data: vagrant up pe-ubuntu1604.pe.psick.io You can test directly the effect of the changes on the code and the data of your local PSICK based control repo. For example edit the hiera file which configures the role ostest used by this node (this special role is used to test features on different OS): vi modules/hieradata/data/role/ostest.yaml And set the following key to true: psick::base::manage: true This particular parameter is used to skip the management of the base profiles (useful when we want to test only role specific classes or when building Docker images via Puppet). When set to true all the base classes are going to be effectively applied and several changes are going to occur on the node by running Puppet either via Vagrant: vagrant provision or, even better, triggering Puppet runs on the selected nodes from the PE console, to apply your local changes on the active VMs (check the PSICK module classification features and general docs for more details on how to configure the psick module via Hiera) When you decide to destroy one of the clients, that one will be automatically removed from PE server (freeing licence slots): vagrant destroy pe-ubuntu1604 docker.pe.psick.io Note that PE based Vagrant environments are present also under vagrant/environments/lab (here with integration with GitLab for Puppet CI/CD) and vagrant/environments/demo. The purpose of these different Vagrant environments embedded in PSICK control repo is to let Puppet developers test locally their changes in Puppet code and hieradata in different conditions (with PE server, FOSS server, The Foreman, or in serverless mode using puppet apply) and on different Operating systems. Have an unforgettable Puppet experience with Puppet Enterprise and PSICK! Alessandro Franceschi 2018-04-30T00:00:00+00:00 https://example42.com/blog/2018/04/30/puppet_tutorial_part_4 example42 Puppet Tutorial - Part 4 This is the last post of a series of articles covering an introduction to Puppet. In the first post I started with Puppet agent installation and how to use Puppet and Facter to analyse your system. Next topics have been the introduction to the Puppet programming language (DSL), how to setup the central Puppet master and how to connect Puppet agents to the Puppet master. The second posting covered cover Puppet modules, code logic and variables and how to add external facts to your systems. Besides this I introduced parameters and the concept of separating code and data by using hiera. The third part explained how to make use of upstream Puppet libraries when describing your own infrastructure, how to best classify nodes and where to place the code. In this posting I will combine what I have shown and explain how to make use of the example42 PSICK control repository, the PSICK module and the sample PSICK hieradata. example42 Puppet Tutorial - Part 4 The PSICK Control Repository The PSICK Module (Library) Phased classification Base profiles TP profiles The PSICK hieradata The PSICK Control Repository One of the biggest challenges for whoever is new to Puppet is how to organise code and data in order to safely and effectively manage the current infrastructure and be ready for future evolutions. PSICK has been done by people who have 10 years of Puppet experience, have designed and build dozens of infrastructure of any size and over the years have updated and refined the approach to Puppet design according to the evolution of the tool. Some design choices in PSICK are rather unusual, and don’t look like the typical ones in Puppet world (ie: the classic roles and profiles pattern). We like to think they are evolutions and steps forward, even if opinionated. Note anyway that you can decide what to use and what to change in PSICK, so ultimately the choice is always yours. The PSICK control repository contains: Setup automation (Hardware, Vagrant, Fabric, Docker) Unit and acceptance tests (rspec-puppet, beaker) CI/CD integration (GitLab, Jenkins, Travis, Danger) Developer support (editorconfig, Vscode, RuboCop, Codacy) Multiple Vagrant environments where to test your code Your main starting point is manifests/site.pp here you want to define the top scope variables used in Hiera, set defaults for some resources defaults and manage the server side noop mode. Then the psick class from example42 PSICK puppet module is included: include '::psick' Next important file is hiera.yaml, the environment level hiera configuration file. Here you see that data and data management is split into a separated hieradata repository: defaults: datadir: modules/hieradata/data # Data in separated module, defined in Puppetfile # datadir: hieradata # Data in control-repo. Previous psick setting # datadir: data # Data in control-repo. Default for puppetlabs/control-repo We do this on PSICK control-repo in order to ease its upgrade without the need to align the sample hieradata, in some cases it may be preferred to have the hieradata directly in the control repo directory. The choice depends mostly on personal preferences and if different people need to access and edit the data from the ones who have to manage the Puppet code. Also, by default, we use and recommend the usage of [hiera-eyaml[(https://www.example42.com/2017/08/21/encrypt-your-secrets-with-hiera-eyaml/) to encrypt sensitive data. Note that you must create your own keys in order to use it: pushd /etc/puppetlabs/puppet /opt/puppetlabs/puppet/bin/eyaml createkeys popd Hiera hierarchy is probably something you may need to change and adapt to your infrastructure. The default one looks like: paths: - "nodes/%{trusted.certname}.yaml" - "role/%{::role}-%{::env}.yaml" - "role/%{::role}.yaml" - "zone/%{::zone}.yaml" - "common.yaml" Zone may refer to a datacenter or region, the role is the function of the system (as in the roles and profiles pattern), and env is the operational environment or tier of the node. All the variables must be top scope, so they are supposed to be set as facts, or defined in the site.pp or set via an External Node Classifier (ENC) like Puppet enterprise or The Foreman. Upstream dependencies are placed into Puppetfile. This is probably another file you are going to change, adding the public module you decide to use. By using example42’s tiny-puppet and the psick module we are able to manage a lot of different profiles and applications without adding extra, dedicated, modules, but as usual you can opt to chose differently. The PSICK Module (Library) The PSICK module can be considered an infrastructure library which provides 3 major features: Phased classification Profiles for common system configurations TinyPuppet profiles to manage applications (via tp module) Phased classification Normal nodes classification is done using the node statement, using roles classes, using an ENC, using hiera_include (or lookup) or including classes in site.pp according to custom logic. The psick module can manage classification and it does it in phased way entirely configurable via Hiera. Four phases are available: firstrun, optional phase, in which the resulting catalog is applied only once, at the first Puppet run. At its end a reboot can optionally be triggered and the real definitive catalog is applied. pre, prerequisites classes, they are applied in a normal catalog run (that is, always except in the very first Puppet run, if firstrun is enabled) before all the other classes. Here for example you can include classes that manage network, repositories, and everything you want to apply first. base, base classes, common to all the nodes (but exceptions can always be managed via Hiera), applied in normal catalog runs after the pre classes and before the profiles. profiles, exactly as in the roles and profiles pattern. The profile classes that differentiate nodes by their role or function. Profiles are applied after the base classes are managed. Strictly speaking, besides the ordering, which you can decide to use or not, there’s no difference among classes included in the pre, base or profiles phases. To define what classes you want to include in what phases you can use parameters as follows: # Optional firstrun phase for Linux and Windows psick::enable_firstrun: true # default is false psick::firstrun::linux_classes: hostname: psick::hostname packages: psick::aws::sdk psick::firstrun::windows_classes: hostname: psick::hostname packages: psick::aws::sdk # Sample pre, base and profiles phases for Linux psick::pre::linux_classes: puppet: ::puppet dns: psick::dns::resolver repo: psick::repo psick::base::linux_classes: sudo: psick::sudo time: psick::time sysctl: psick::sysctl update: psick::update ssh: psick::openssh::tp mail: psick::postfix::tp psick::profiles::linux_classes: webserver: apache # Sample pre, base and profiles phases for Windows psick::pre::windows_classes: hosts: psick::hosts::resource psick::base::windows_classes: features: psick::windows::features registry: psick::windows::registry services: psick::windows::services time: psick::time users: psick::users::ad psick::profiles::windows_classes: webserver: iis There are different key names for Linux and Windows (and Solaris and Darwin) to ease Hiera driven classification for entirely different OS without the need of adding OS dependent variables in your Hiera’s hierarchy. The classes included are defined in the values of the above hashes. They can be any class for any module in the modulepath: dedicated component modules, custom profiles, profiles from the psick module (as in many entries of the above example). The keys of the hashes can be any string and their purpose is to allow override across Hiera hierarchies. For example of a specific node, let’s say a Puppet server, you may want to use a different class to manage puppet, you can do by setting, in that node’s Hiera data (here as yaml file) something like: psick::pre::linux_classes: puppet: ::profiles::puppetmaster Using the psick module to classify nodes, is an option. You can follow more traditional (who said primitive? :-) ways and use psick only for its profiles… Base profiles Even if every infrastructure is a snowflake of its own, the kind of resources to manage on a system is more or less always the same: users, packages, authentication, dns, ntp, networking, sudo, ssh and so on. For the most common use cases the psick module provides profiles, both for Linux and Windows. In most of the cases such profiles can be used instead of adding a dedicated module to your Puppetfile, but, as usual, you can decide if and which ones to use. Give a look at psick’s manifests to have an idea of the available profiles. Some of these profiles are able to manage common system features for Linux (users, cron, sysctl, time and timezones, hostname, hosts file, repositories, system’s proxy, dns, motd, nfs, syslog, iptables…) and Windows (users, packages, features, registry keys, services, time…) other are application specific and in some cases can even provide more features than dedicated component modules (apache, openssh, bolt, ansible, mariadb, mysql, docker, gitlab, icinga, java, mongo, openvpn, oracle, openswan, prometheus, php, puppet…). In some cases a psick profile even allows you to choose different modules to manage the same resource (users, sysctl, puppet, php, docker …). TP profiles Besides the multitude of base profiles, in the psick module there are so called, tp profiles, they are always in manifests called tp.pp, they are automatically generated and provide a standard interface to manage an application using Tiny Puppet. For example to manage apache via Tiny Puppet you can use the profile psick::apache::tp by setting something like: psick::profiles::linux_classes: 'apache': psick::apache::tp Then you can configure it with something like: psick::apache::tp::resources_hash: tp::conf: apache::openkills.info.conf: base_dir: conf template: psick/apache/vhost.conf.erb options_hash: ServerName: openskills.info ServerAlias: - openskill.info - www.openskills.info - www.openskill.info AddDefaultCharset: ISO-8859-1 apache::deny_git.conf: base_dir: conf source: puppet:///modules/psick/apache/deny_git.conf tp::dir: apache::openskills.info: vcsrepo: git source: git@bitbucket.org:alvagante/openskills.info.git path: /var/www/html/openskills.info The PSICK hieradata Everything in PSICK is entirely data driven. Via Hiera you can configure classification and configuration of classes. Infrastructure data is typically site specific, so we placed in a separated module the sample Hieradata used in the PSICK control repo, to ease the usage of custom data in custom modules (or directly in the control repo). Give a look at the PSICK hieradata module to see real world examples of the data you can use to configure an infrastructure via PSICK. In particular the lab zone with lab hosts specific data is actually used to configure the living servers in our PSICK lab setup, which runs using the Vagrant environment in the control repo. You can use that data as starting point to build your own infrastructure with PSICK. Happy hacking. If four blog posts got you this far, imagine what your team could learn with hands-on, code-based sessions: see our Puppet Training. Martin Alfke Alessandro Franceschi 2018-04-23T00:00:00+00:00 https://example42.com/blog/2018/04/23/puppet_tutorial_part_3 example42 Puppet Tutorial - Part 3 This is the third post of a series of articles covering an introduction to Puppet. In the first post I started with Puppet agent installation and how to use Puppet and Facter to analyse your system. Next topics have been the introduction to the Puppet programming language (DSL), how to setup the central Puppet master and how to connect Puppet agents to the Puppet master. The second posting covered Puppet modules, code logic and variables and how to add external facts to your systems. Besides this I introduced parameters and the concept of separating code and data by using hiera. This third part will explain how to make use of upstream Puppet libraries when describing your own infrastructure, how to best classify nodes and where to place the code. At the last posting I will combine what I have shown and explain how to make use of the example42 PSICK control repository, the PSICK module and the PSICK hieradata. example42 Puppet Tutorial - Part 3 Technical Component (Library) Modules Implementation Profiles Business Use Case Role Node Classification The Puppet Control Repository Technical Component (Library) Modules Usually we encourage you to write your own Puppet code, because this is the best way to learn Puppet DSL. But once there will be the time where you ask yourself whether you have to maintain a large code base by yourself unless you have super simple Puppet code only. This is the time where you will rethink your Puppet usage. There are many people and organisations which provide Puppet modules. Main search starting point is the Puppet Forge. Another search might lead you to GitHub and here especially to voxpupuli or example42. The modules which can be found at the above mentioned links are very generic, usable on many different operating systems and (hopefully) can be adopted to your specific implementation. From now on we refer to the upstream developed, generic modules as Technical Component Modules. Best way is to see them like libraries for a programming language. But when you search e.g. for an Apache Module you will find 1056 matching modules. So which one is the one you should use? Luckily the Puppet Forge has some more information on modules. First there are supported modules. These modules are usually managed by Puppet itself and are included in the Puppet Enterprise Support Contract and are therefore a good choice. Next Puppet added the “approved” flag. These are modules developed by community or organisations and had a careful review regarding supported Operating systems, Tests included and have active maintainer. Other information for each module is the number of downloads and the community feedback (Quality score). If you find errors or misbehaviour on Technical Component Modules, you are encouraged to collaborate with upstream development to get the issue fixed. It makes no sense to only locally fix the issue as you will loose upgradeability. We recommend to mirror Technical Component Modules on your internal Git server to not rely on foreign infrastructure (network, servers, storage,…) when deploying them to your Puppet Master. Technical Component Modules only manages the smallest possible set of configuration. A Tomcat Module only manages Tomcat, the required Java installation is managed by a Java Module. Apache Module configured Apache, PHP module does the same for PHP only. Implementation Profiles So most of the work has been made with Technical Component Module development. Now it is up to you to describe how these Modules should be implemented and adopted to your infrastructure and requirements. This work is also placed in a Puppet Module but this module serves a specific need. It describes your implementation. Implementation Modules are now called Profiles. A Profile is the smallest set of infrastructure artefacts. Think of an authentication Profile which uses SSH, SSSD and PAM Technical Component Modules. Now it is easy to do additional infrastructure artefacts: Your Mail Profile uses Postfix and Clamav Modules, your Logging Profile uses rsyslog and splunk Module, your Monitoring Profile uses prometheus and grafana Module. Technically a Profile is also a Module. It has a manifests path, it can have a files and templates directory. To better visualise the different usage, it is recommended to place Profiles into a separate module path. This can be done by configuring the environment.conf file: # /etc/puppetlabs/code/environments/production/environment.conf modulepath = ./site:./modules:$basemodulepath Within your environment you place Technical Component Modules in the modules directory and your profiles in a profile directory located in the site folder. Differences among your platform can be placed in hiera. Usually we recommend to do lookups in profiles and to declare Technical Component Classes by specifying all required parameters. This prevents from having hiera data with module name space. Example - Puppet Profile # /etc/puppetlabs/code/environments/production/site/profile/manifests/authentication.pp # class profile::authentication ( String[1] $ldap_server, Array[String] $ssh_allow_groups = [], Boolean $permit_root = false, ){ # Puppet DSL code class { 'ssh': permit_root_login => $permit_root, sshd_config_allowgroups => $ssh_allow_groups, } # ... } Example - Hiera Data # /etc/puppetlabs/code/environments/production/data/stage/dev.yaml profile::authentication::ldap_server: 'ldap.domain.tld' profile::authentication::permit_root: true Following this pattern allows you to easily identify which profile uses which data. Profiles may contain: declaration of classes and resources parameters few Puppet DSL code logic files and templates facts functions Resource ordering Profiles should not contain: defined resource types custom types and providers Puppet data types OS specific case (this must be done in a Technical Component Module) Business Use Case Role When you have many identical systems it will be error prone when you just use Profiles for node classification. In this case you want to reconsider that any system in your infrastructure serves a specific business need. Some systems might be of value to IT only (metrics and backup server) some have a generic value to everybody (Mail and DNS server), some are used by customers (Login server and invoicing system). You want to try to identify the business use case for each system. Sometimes it will be hard for IT to provide a correct name. Just head to the business owner and ask what they use the system for. e.g. everybody knows the ‘vacation planner’, but when you ask HR they will tell you that this is the ‘HR self service portal’. Each business use case is built based on profiles. We call them now Roles. Roles should contain: declaration of Profiles classes using include or contain Resource ordering Roles should not contain: explizit resource declaration code logic self defined resource types Puppet data types types and providers facts functions parameters An example for a Role: # /etc/puppetlabs/code/environments/production/site/role/manifests/ci_server.pp # class role::ci_server { contain profile::authentication contain profile::backup::client contain profile::jenkins::master Class['profile::authentication'] -> Class['profile::backup::client'] -> Class['profile::jenkins::master'] } Node Classification Now you have all Puppet Code in place. The master muss somehow identify what Roles should be compiled into the catalog for a specific systems. This process is called Node Classification. There are three different ways on how to do node classification: manifests based per node classification fact based node classification hiera based node classification The manifest based node classification might look like being the most simple one, but that is only true for small environments with only a few systems. Just add a node declaration to your manifests/site.pp in your control repository: # /etc/puppetlabs/code/environments/production/manifests/site.pp # ... node 'id3452276.domain.com' { contain role::ci_server } The name provided seems to look like a FQDN. In fact it is the common name of the nodes client SSL certificate. You can also split up your node classification in multiple files and directories, located in the manifests directory: # /etc/puppetlabs/code/environments/production/manifests/dev_zone.pp # ... node 'jenkins.dev.domain.com' { contain role::ci_server } The next possible solution is to make use of facts which have been placed onto a system during provisioning. When you for example spin up a new app web server for your infrastructure you place deploy an external fact to the system: # /etc/puppetlabs/facter/facts.d/classification.yaml # application: 'webserver' stage: 'development' Now you can from Puppet code query the existence of the fact and use the fact for node classification. This reduces your node classification to just one default node: # /etc/puppetlabs/code/environments/production/manifests/site.pp # ... node default { if $facts['application'] { contain "role::${facts['application']}" } } Now there is also no need to have the node mentioned. You can place the variable check and class declaration directly into your site.pp file: # /etc/puppetlabs/code/environments/production/manifests/site.pp # ... if $facts['application'] { contain "role::${facts['application']}" } This solution is handy when you want to be able to reuse an existing system and let it have another role, as you only have to change the fact. Please note that changing a systems use case is not considered best practice! Usually you want to de-provision the old system and provision a new one. And then there is hiera. You can add node classifications also directly into hiera. # /etc/puppetlabs/code/environments/production/data/nodes/id3046756732.domain.com.yaml # role: 'ci_server' In your node classification you only do a lookup on the key ‘classes’ and use the value: # /etc/puppetlabs/code/environments/production/manifests/site.pp # ... $role = lookup('role', String, first, 'base') contain "role::${role}" The Puppet Control Repository Now you have everything together which is needed in a Puppet environment. But how do you develop new features, how do you refactor existing code, how to test that everything is working? This is where the concept of a Control Repository will help you. A Control Repository is a GIT Repository. Different to “normal” GIT repositories the default branch will not be “master”. The Control Repository uses “production” as default branch. The required modules are listed in Puppetfile. For each Module you will provide the GIT url and specify a tag or a commit which should be deployed: # Puppetfile # ISSUES: # activemq - f4b580461e1b9c1980f3141f0414512aa6b2a0ba - has some new features, needs new build: 0.4.1 or higher # mod 'activemq', :git => 'ssh://git@<git server>:<port>/<path>/puppetlabs-activemq.git', :ref => 'f4b580461e1b9c1980f3141f0414512aa6b2a0ba' mod 'ssh', :git => 'ssh://git@<git server>:<port>/<path>/ghoenycutt-ssh.git', :ref => 'v3.57.0' Any Code development is done in a feature branch. Changes are merged with fast-forward strategy from the feature branch into the production branch. But how do you deploy the Puppet code from the feature branch? How do you update production code after a merge? Code deployment in this case is handled by r10k. The r10k application is installed and configured on the Puppet Master: /opt/puppetlabs/bin/gem install r10k Configuration file is located in /etc/puppetlabs/r10k: # /etc/puppetlabs/r10k/r10k.yaml --- :cachedir: /opt/puppetlabs/puppet/cache/r10k :sources: puppet: basedir: /etc/puppetlabs/code/environments remote: ssh://git@<git server>:<port>/<path>/puppet-control-repo.git r10k pulls the control repository and checks for existing branches. Branch names are then converted to Puppet environment paths. In each of the environments, r10k will parse the Puppetfile and install the mentioned modules. This is achieved by running r10k: /opt/puppetlabs/puppet/bin/r10k deploy environments -pv The -p option enables parsing of Puppetfile and installing modules. -v enables verbose output so you see what r10k is doing. From now it is up to you, whether you want to start with an empty control repository and add all required code by yourself or whether you want to adopt an existing control repository to your needs. The upcoming posting will explain the concept of example42 PSICK Control Repository and how you can easily adopt it to your infrastructure. Martin Alfke 2018-04-16T00:00:00+00:00 https://example42.com/blog/2018/04/16/puppet_tutorial_part_2 example42 Puppet Tutorial - Part 2 This is the second post of a series of articles covering an introduction to Puppet. Take your time digging through this posting as this a very dense summary with lots of information. In the first post I started with Puppet agent installation and how to use Puppet and Facter to analyze your system. Next topics have been the introduction to the Puppet programming language (DSL), how to setup the central Puppet master and how to connect Puppet agents to the Puppet master. This posting will cover Puppet code location and structure like modules, code logic and variables and how to add external facts to your systems. Besides this I will introduce parameters and the concept of separating code and data by using hiera. The third part will explain how to make use of upstream Puppet libraries when describing your own infrastructure, how to best classify nodes and where to place the code. At the last posting I will combine what I have shown and explain how to make use of the example42 PSICK control repository, the PSICK module and the PSICK hieradata. example42 Puppet Tutorial - Part 2 Puppet code location Modules and Classes Puppet Variables Puppet Code Logic Class Parameters Node classification Dynamic configuration files Separation of Code and Data Puppet code location Puppet code is placed on the Puppet master and is usally located in /etc/puppetlabs/code/environments/production. This directory has a special name: the “production environment path”. The name “production” refers to the state of your Puppet code, not to the stage of your systems. Think about the following scenario: The Ops Team is building an infrastructure in autmated way. The Dev Team uses this infrastructure. Where will the Ops Team test their changes? On the platform where the Dev Team is doing their work? What will happen if Ops breaks the infrastructure? In this case it is best, to see the Development platform as a stable infrastructure with SLA. But this means, that Ops must have a development platform by themselves. Let’s also rename the teams: Ops is now Infrastructure Development, Dev is now Application Development. You can visualize the differences by the following table: Team Infrastructure Development Devevelopment Testing Production Application Development - Development Testing Production Infrastructure Development Development Production Production Production Within the environment we have a strict naming convention of content: /etc/puppetlabs/code/environment/ \- production/ |- environment.conf # <- Here you configure your environment |- manifests/ | \- site.pp # <- Here you place the Node Classification \- modules/ \- <module> # <- Here you place modules Modules and Classes A module is a directory located in the $modulepath configured location. This configuration is done in environment.conf and has the following default setting: modulepath = ./modules:$basemodulepath The $basemodulepath configuration is a Puppet default and is especially needed if you are using Puppet Enterprise. Best option is to see a module as a small part of your platform like ssh, ldap, apache, nginx, postfix, exim, mysql, postgresql, firewall, … Aditionally there are some modules which don’t configure anything, but which provide extensions to Puppet (e.g. stdlib, concat, inifile). A module has again a strict directory naming convention: <module>/ |- manifests/ # <- Here you place your Puppet classes |- files/ # <- Here you place static configuration files \- templates/ # <- Here you place dynamic generated configuration files Let’s start with the content of the manifests directory. Here you add classes. Classes are written in Puppet DSL (like the puppetserver.pp file from last weeks posting). There is just one more thing which is different to the puppetserver.pp file: In the puppetserver.pp file we directly placed Puppet DSL code. In a class we wrap this Puppet DSL content into a class definition: class puppetserver { package { 'puppetserver': ensure => present, } service { 'puppetserver': ensure => running, enable => true, } file { '/etc/motd': ensure => file, content => "# This is puppetserver\n", } } The difference: a class definition is part of your Puppet code and will only be added to a nodes catalog if a class declaration is added. Class declaration is a concept we explain in the next posting. This time I concentrate on the concept of classes, the DSL and the naming convention. Usually I refer to classes as parts of modules. Or better: classes are within the “namespace” of a module. The name of the module (to be more precise: the directory name) will always be part of the class. This allows you to easily identify to which module a class belongs to. Names of classes must follow a strict ruleset. There is one special class - the main class of a module, which is placed in a file called init.pp. Think about a SSH module: /etc/puppetlabs/code/environment/prpoduction/modules/ssh/ \- manifests/ \- init.pp # <- main class of ssh module The init.pp file will start with the term “class” and then use the module name: class ssh { } Inside of a class you can put Puppet DSL code: class ssh { package { 'openssh-server': ensure => present, } package { 'openssh-client': ensure => present, } service { 'sshd': ensure => running, enable => true, } } This will ensure that you have an SSH daemon running with default configuration. I already showed how you can manage the content of a file. But placing the sshd_config file content inside the class will lead to bad readable code. Let’s assume that you want to deploy one version of sshd_config file to all systems. In this case you can use the source parameter: file { '/etc/ssh/sshd_config': ensure => file, source => 'puppet:///modules/ssh/sshd_config', } Looking weird? I will explain: The parameter source will be part of the agents catalog. So the agent must learn where to get the desired file from. That is the reason why you specify an URI. The ‘puppet’ protocol is shorthand for https on port 8140 including client certificate (the default Puppet connectivity behavior). We omit the servername as we have that one configured in puppet.conf file. On the Puppet server you have a Puppet internal “mount point” available, pointing to the modules directory. Now you only must tell the Puppet server which module you are referring to and what is the filename inside the modules file directory. Please note that you must omit the files directory name! puppet:///modules/ssh/sshd_config <protocol>://<server>/modules/<modulename>/<file in files directory> But what happens if the SSH daemon is already running? Puppet will ensure that the config file will get updated. But the service will continue running with the old configuration. In this case you must tell the service resource that it should restart upon config file changes. This is done by using a metaparameter. file { '/etc/ssh/sshd_config': ensure => file, source => 'puppet:///modules/ssh/sshd_config', notify => Service['sshd'], } The notify parameter uses a reference to a declared resource type. The resource type is written with capital letter and afterwards you use the title in brackets. Type['title'] Another solution is to use the chaining pattern: File['/etc/ssh/sshd_config'] ~> Service['sshd'] Let’s put everything together: # /etc/puppetlabs/code/environments/production/modules/ssh/manifests/init.pp class ssh { package { 'openssh-server': ensure => present, } package { 'openssh-client': ensure => present, } file { '/etc/ssh/sshd_config': ensure => file, source => 'puppet:///modules/ssh/sshd_config', notify => Service['sshd'], } service { 'sshd': ensure => running, enable => true, } } And the configuration file: # /etc/puppetlabs/code/environments/production/modules/ssh/files/sshd_config Port 22 PermitRootLogin no PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys UsePAM yes UseDNS no Subsystem sftp /usr/libexec/openssh/sftp-server Puppet Variables But what if one system needs a different configuration? e.g. allow root access or switch port. This would require a second configuratoin file and a class adopted to only one specifc system. Or what if you must manage SSH on another UNIX system where paths are different? Puppet allows you to build flexible code by making use of variables. A variable in Puppet is easily identified by having a dollar sign: $variable. You can assign values to variables - but within a class you can not reassign the same variable a second time. Variables in Puppet are more like static artifacts and Puppet is not a scripting language. Next you can check variables whether they have a value, you can check for specifc values or for regular expressions. Check for variable having a value: $var1 = false if $var1 { # Puppet code } else { # <- else is optional # Puppet code } Checking a variable like in the example above will return true in the following cases: the variable has the bool value true the variable has any content (Array, Hash String or even an empty string) Check for variable having specific value: $var2 = 'dbmaster' if $var2 == 'dbmaster' { # Puppet code } Check for variable using regular expression $var3 = 'db22.domain.com' if $var ~= /^db\d+\.domain\.com$/ { # Puppet code } Puppet Code Logic Variables will be mostly used in Puppet code logic: use correct package names or file paths depending on Linux distribution name. This is where the case function will be useful: case $::facts['os']['name'] { 'CentOS', 'Amazon', 'RHEL': { # Puppet code for RedHat based systems } 'Ubuntu', 'Debian': { # Puppet code for Debian based systems } 'SLES': { # Puppet code for SuSE based systems } default: { # Optional default for any other OS } } What is this $::facts['os']['name'] thing? Remeber post 1 when I was introducing facter? Facts are available to Puppet code within a special variable: $::facts. All data are stored as a hash inside this variable. On the command line you were using facter os or facter os.name to access specific facts. Within Puppet code you must use the $::facts variable and put the elements into brackets and quote them. You might want to add your own set of variables during system provisioning. e.g. information on which datacenter the system is running in, what is the usecase of the system and whether it is a development or production system. You can add these facts easily by placing files into a specific directory (/etc/puppetlabs/facter/facts.d/). Don’t worry, if the directory does not exist, just create it. In this directory you can place: .yaml files - with yaml syntax .json files - with json syntax .txt files - with key=value syntax Now you can create information on your datacenter to a node: # /etc/puppetlabs/facter/facts.d/provision_facts.yaml --- datacenter: 'london' application: name: 'cms' stage: 'dev' You can query these facts by using facter -p: facter -p datacenter london facter -p application { "name" => "cms", "stage" => "dev" } Let’s get back to the Puppet code: Now we can rewrite the SSH class to also work on Debian systems: # /etc/puppetlabs/code/environments/production/modules/ssh/manifests/init.pp class ssh { case $::facts['os']['family'] { 'RedHat': { $packages = ['openssh-server', 'openssh-client'] } 'Debian': { $packages = ['ssh'] } } package { $packages: ensure => present, } file { '/etc/ssh/sshd_config': ensure => file, source => 'puppet:///modules/ssh/sshd_config', notify => Service['sshd'], } service { 'sshd': ensure => running, enable => true, } } Here we use another thing within Puppet: at a title you are able to use an array. Puppet internally will split the array up into single package resource type declarations. Class Parameters But how do you deal with a single node to allow root ssh access? You have multiple possible solutions like creating an external fact on the node, and check for existance of the fact. But that is highly unflexible. Instead you can use class parameters. # /etc/puppetlabs/code/environments/production/modules/ssh/manifests/init.pp class ssh ( Boolean $permit_root = true, ){ # Puppet code } At the parameter we specify the expected data type and we provide a default value. But how to now declare the class? Node classification The most simple approach (on small platforms) is the manifest based node classification. Remeber the environment directory structure where I showed the manifests directory with site.pp file inside? THis is the place where you will place information on your nodes. # /etc/puppetlabs/code/environments/production/manifests/site.pp node 'agent1.example42.training' { } There are two ways on how to inform the Puppet server that he should add a class to the node’s catalog: include ssh or class { 'ssh': } When using the second approach, you are able to specify class parameters: class { 'ssh': permit_root => true, } Dynamic configuration files Now we must ensure that Puppet uses the provided parameter inside a configuration file. Which means the configuration file must be built during Puppet catalog compilation. This is where templates come into place. Puppet templates are plain text files which use opening (<%) and closing (%>) tags to identify where the template engine should do something. The content within the tags is just Puppet DSL code. Templates are - like files - part of the module, but are not in the files folder, but in the templates directory. Modern Puppet uses the EPP template engine which requires that templates must have the file ending .epp. In this case you want the template engine to check for the value of the parameter permit_root and set the correct configuration value: # /etc/puppetlabs/code/environments/production/modules/ssh/templates/sshd_config.epp Port 22 <% if $ssh::permit_root { %> PermitRootLogin yes <% } else { %> PermitRootLogin no <% } %> PubkeyAuthentication yes AuthorizedKeysFile .ssh/authorized_keys UsePAM yes UseDNS no Subsystem sftp /usr/libexec/openssh/sftp-server Using the template is different to using a static file. At the static file we were managing the source. Now you must manage the content: file { '/etc/ssh/sshd_config': ensure => file, content => epp('ssh/sshd_config.epp'), } The template validation takes place on the Puppet master while compiling the catalog. So there is no need to specify the protocol or the server or telling the server that it should look in a module. You only specify the module and the name of the file in the templates directory. This will give you the following Puppet code: # /etc/puppetlabs/code/environments/production/modules/ssh/manifests/init.pp class ssh ( Boolean $permit_root = true, ){ case $::facts['os']['family'] { 'RedHat': { $packages = ['openssh-server', 'openssh-client'] } 'Debian': { $packages = ['ssh'] } } package { $packages: ensure => present, } file { '/etc/ssh/sshd_config': ensure => file, content => epp('ssh/sshd_config.epp'), notify => Service['sshd'], } service { 'sshd': ensure => running, enable => true, } } Separation of Code and Data But what if you have a large number of systems and each system needs to get configured slightly different. In this case it will become a nightmare when you add each node individually to manifests/site.pp file or by writing specific puppet code. This is where Hiera jumps in. Hiera allows you to do data lookups, so you can separate code from data. Think about the following Puppet code: class ssh { case $::facts['datacenter'] { 'amsterdam': { case $::certname { 'gateway.ams.example42.training': { $permit_root = false } 'default': { $permit_root = true } } $ssh_port = '22' $ssh_listen = 'enp0s3' } 'london': { if $::certname == 'firewall.lon.example42.training' { $permit_root = false } else { if $::certname = 'devel.lon.example42.training { $permit_root = true } } $ssh_port = '222' $ssh_lisen = 'any' } } file { '/etc/ssh/sshd_config': ensure => file, content => epp('ssh/sshd_config.epp'), # <- template uses variables from above } } Looks like nightmare? Yes, this is nightmare. Let’s start using hiera: class ssh { $permit_root = lookup('permit_root', Boolean, first, false) $ssh_port = lookup('ssh_port', String, first, '22') $ssh_listen = lookup('ssh_listen', String, first, 'enp0s3') file { '/etc/ssh/sshd_config': ensure => file, content => epp('ssh/sshd_config.epp'), # <- template uses variables from above } } Puppet code now looks cleaner. But where have you hidden the data? First: you need a Hiera configuration file located in /etc/puppetlabs/code/environment/production/hiera.yaml. Within the hiera.yaml file, one specifies different layers of data. Think of hiera layers being a huge ‘chessboard’. At the default ‘chessboard’ every field means a key and has the value written on it. With every hiera layer, hiera looks whether it has another ‘chessboard’ which can be placed over the default one. The new layer ‘chessboard’ has some elements unset, which means, you can look through the layer and you see the data from the default ‘chessboard’, some data are overwritten. I try to visualize (the effective data are printed in bold node agent.ams.example42.training Datacenter: Amsterdam Certname: agent.ams.example42.training Level permit_root ssh_port ssh_listen Common Data false 22 any Amsterdam Data true — enp0s3 Node Data — — — node gateway.ams.example42.training Datacenter: Amsterdam Certname: gateway.ams.example42.training Level permit_root ssh_port ssh_listen Common Data false 22 any Amsterdam Data true — enp0s3 Node Data false — — node agent.lon.example42.training Datacenter: London Certname: agent.lon.example42.training Level permit_root ssh_port ssh_listen Common Data false 22 any London Data — 222 — Node Data — — — node firewall.lon.example42.training Datacenter: London Certname: firewall.lon.example42.training Level permit_root ssh_port ssh_listen Common Data false 22 any London Data — 222 — Node Data — — — node devel.lon.example42.training Datacenter: London Certname: devel.lon.example42.training Level permit_root ssh_port ssh_listen Common Data false 22 any London Data — 222 — Node Data true — — Based on these inforation you can build your hierarchies into your hiera.yaml file: # /etc/puppetlabs/code/environments/production/hiera.yaml --- version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: 'Node Data' path: "nodes/%{trusted.certname}.yaml" - name: 'Data Center Data' path: "datacenters/%{facts.datacenter}.yaml" - name: 'Common Data' path: 'common.yaml' The data for hiera are placed inside the (relative) path data (/etc/puppetlabs/code/environments/production/data). Your directory structure (using the above examples) will be the following: /etc/puppetlabs/code/environments/production/data |- common.yaml |- datacenters/ | |- london.yaml | \- amsterdam.yaml \- nodes/ |- gateway.ams.example42.training.yaml \- devel.lon.example42.training The content of the files will be YAML structured data containing key: value. e.g. # common.yaml permit_root: false ssh_port: '22' ssh_listen: 'any' But now it is up to you to take care to not use duplicate key names. There is another even more simple way: When declaring a parameterozed class using the include function, Puppet will automatically query hiera for data. So you move the lookups to parameters: class ssh ( Boolean $permit_root = false, String $ssh_port = '22', String $ssh_listen = 'enp0s3', ){ file { '/etc/ssh/sshd_config': ensure => file, content => epp('ssh/sshd_config.epp'), # <- template uses variables from above } } When you just include ssh Puppet will ask hiera for the parameters in the given namespace: “Hey hiera, do you have a value for namespace ssh and parameter permit_root? Or to be short: ssh::permit_root. All you have to do is add the namesapces to your keys: # common.yaml ssh::permit_root: false ssh::ssh_port: '22' ssh::ssh_listen: 'any' The next posting will explain the concept of re-using existing modules and provide information on why you should see modules similar to libararies. Additionally I will explain the concept of Roles and Profiles and the Node Classification. Martin Alfke 2018-04-09T00:00:00+00:00 https://example42.com/blog/2018/04/09/puppet_tutorial_part_1 example42 Puppet Tutorial - Part 1 This is the first post of a series of articles covering an introduction to Puppet. I start with Puppet agent installation and how to use Puppet and Facter to analyse a system. Next topics are the introduction to the Puppet programming language (DSL), how to setup a central Puppet master and how to connect Puppet agents to a Puppet master. The second part will cover Puppet modules, code logic and variables and how to add external facts to your systems. Besides this I will introduce parameters and the concept of separating code and data by using hiera. The third part will explain how to make use of upstream Puppet libraries when describing your own infrastructure, how to best classify nodes and where to place the code. At the last posting I will combine what I have shown and how to make use of the example42 PSICK control repository, the PSICK module and the PSICK hieradata. example42 Puppet Tutorial - Part 1 Introduction Puppet installation Puppet and Facter Start with Puppet DSL Puppet apply Puppet CA and Puppet agents Introduction Puppet is a client-server based configuration management solution written in Ruby (Agent) and Java (Server). When using Puppet, you write a declarative description of your systems and infrastructure. This means that you are not providing information on how to change something, but you describe the desired configuration state, e.g. by providing the complete configuration file. The declarative description is done by using resource types like user, group, package, file, service, cron, mount. We are going to explain these within this posting. Your infrastructure description must be written in Puppet DSL code. The files in which you place your code are called Puppet manifests. Puppet allows you to run even without a central Puppet server. This usage of Puppet is called Masterless Puppet and will be our starting point - even for setting up our Puppet server. The Puppet server can only run on Linux based systems (see Puppetserver Supported Platforms). The Puppet agent supports a large variety of different operating systems like Linux, Windows, OS X and macOS, Juniper, Arista, BSD. Puppet installation You start with installing Puppet agent on a Linux system. This tutorial uses CentOS 7, descriptions on how to install Puppet agent on other Linux distributions can be found on the Puppet Platform website. As the first system also will become your Puppet server, you have to take care on proper hardware sizing. A Puppet server should have at least 2-4 GB of RAM and at least 1-2 CPU cores. The hard disk should have enough space to hold your code, plus information and reports from your agents. Usually Puppet recommends 100 GB free space. Further information can be found on the Puppet Server Installation page. Note: in larger environments (more than 400 nodes) a Puppet server should have 4-12 cores and 12 to 24 GB RAM. Some Linux distributions have Puppet packages added to their repositories. Usually the shipped versions are outdated - sometimes even without security fixes from Puppet. At the time of writing this posting, Puppet has recently released version 5.5. You will therefor use the Puppet repositories to install the agent using root user account: rpm -Uvh https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm Now you can install the Puppet Agent package: yum -y install puppet-agent The Puppet Agent package installs into /opt/puppetlabs directory and has everything the Puppet Agent needs bundled inside: ruby openssl The configuration files for Puppet agent are located in /etc/puppetlabs/puppet. The main configuration file is the puppet.conf file. You will leave this file untouched as you also want to manage the Puppet setup in an automated way. Puppet and Facter Now you already have everything required to start playing and working with Puppet. The Puppet binary is located at /opt/puppetlabs/puppet/bin/puppet. The installer provides a profile.d snippet in /etc/profile.d/puppet-agent.sh which add the puppet binary path to the PATH environment variable. You can either log out and log in again or refresh your shell by running exec bash. The puppet binary has several subcommands like agent, apply, describe, resource, .... A complete list of commands can be found when running puppet --help. At the beginning you use the puppet describe command. This command prints the documentation which is part of the Puppet code. When running puppet describe --list you will receive a list of all available resource types on your system. At the moment these will be the built in Puppet Types. You can either refer to the online documentation or print the documentation using puppet describe <resource type>. As you now know about existing resource types, you are able to use the puppet resource command to read existing resources and print them in Puppet DSL code. Read a user resource: # puppet resource user root user { 'root': ensure => 'present', comment => 'root', gid => 0, home => '/root', password => '$1$FQDy6m9T$5JJ5fqv9ylivZNQIj5Eet0', password_max_age => 99999, password_min_age => 0, shell => '/bin/bash', uid => 0, } Read a package resource: # puppet resource package puppet-agent package { 'puppet-agent': ensure => '5.5.0-1.el7', } Read a file resource: # puppet resource file /etc/motd file { '/etc/motd': ensure => 'file', content => '{md5}d41d8cd98f00b204e9800998ecf8427e', ctime => '2017-09-11 18:34:44 +0000', group => 0, mode => '0644', mtime => '2013-06-07 14:31:32 +0000', owner => 0, selrange => 's0', selrole => 'object_r', seltype => 'etc_t', seluser => 'system_u', type => 'file', } Read a service resource: # puppet resource service puppet service { 'puppet': ensure => 'stopped', enable => 'false', } Read a mount resource: # puppet resource mount / mount { '/': ensure => 'mounted', device => '/dev/mapper/VolGroup00-LogVol00', dump => '0', fstype => 'xfs', options => 'defaults', pass => '0', target => '/etc/fstab', } As you can see Puppet always uses the same layout and syntax for resources: <resource_type> { '<resource title>': parameter => 'value', parameter2 => 'other value', } This pattern is named a “resource type declaration”. Another important utility which is part of the puppet agent package is facter. facter collects different system information covering hardware (RAM, CPU, manufacturer), network (interfaces, IPv4 and IPv4 addresses), operating system (name, version) and many more. You can print the whole set of information by running facter on the command line as root user. A single element can be read when adding the fact name: facter os, facter networking. # facter os { architecture => "x86_64", family => "RedHat", hardware => "x86_64", name => "CentOS", release => { full => "7.4.1708", major => "7", minor => "4" }, selinux => { config_mode => "enforcing", config_policy => "targeted", current_mode => "enforcing", enabled => true, enforced => true, policy_version => "28" } } Start with Puppet DSL Now let’s start using Puppet and prepare the system to become our Puppet master. At the moment there are only a couple of things to do: install the package puppetserver and start the process. Let’s additionally place the information that this is the puppet server into /etc/motd. You are now writing your first Puppet manifest: # /root/puppetserver.pp # install puppetserver package package { 'puppetserver': ensure => present, } # ensure that puppetserver is running and started at reboot service { 'puppetserver': ensure => running, enable => true, } # put information into motd file { '/etc/motd': ensure => file, content => "This is Puppet Server\n", } As you can see I have omitted the single quotes around the words present, running and true. Maybe it is obvious why I am not quoting the bool value. But what about the others? There are some special words within Puppet which do not need quoting. Mostly these are Ruby Symbols. At the file resource I am managing the content. This is easy to do for small configuration files, but leads to hard to read Puppet code when it comes to larger configuration files. In Part 2 I will introduce another solution for providing file content. But how do you now “execute” the manifest we have written? First: in Puppet you never “execute” something. Instead you want to apply a specific declaration onto a system. This is done by Puppet first parsing the manifest and rendering it into a catalog. This process is called “catalog compile”. No worries, this is not binary code, the Puppet compiler returns minified JSON to the Puppet agent, who will then start working on the returned catalog. Puppet apply As you are not (yet) having a Puppet master you must run Puppet in masterless mode. This is possible by running puppet apply and providing the filename. Prior installing and starting Puppet master process you need to ensure that the DNS settings are correct and that time is set correctly: You can use the hostnamectl utility to set a hostname ans add an entry to your puppetmaster /etc/hosts file: hostnamectl set-hostname puppetmaster.example42.training echo "127.0.0.1 puppetmaster.example42.training >> /etc/hosts Now you can verify DNS resolution: ping -c1 $(facter networking.fqdn) PING puppetmaster.example42.training (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.026 ms --- puppetmaster.example42.training ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.026/0.026/0.026/0.000 ms Next you need to verify correct time setting: yum install -y ntpdate ntpdate pool.ntp.org Now we can start the Puppet master installation: # puppet apply /root/puppetserver.pp Notice: Compiled catalog for puppetmaster.example42.training in environment production in 0.69 seconds Notice: /Stage[main]/Main/Package[puppetserver]/ensure: created Notice: /Stage[main]/Main/Service[puppetserver]/ensure: ensure changed 'stopped' to 'running' Notice: /Stage[main]/Main/File[/etc/motd]/content: content changed '{md5}d41d8cd98f00b204e9800998ecf8427e' to '{md5}c3f33391f7d0d397055190ac0c96cbf8' Notice: Applied catalog in 75.65 seconds Now you have Puppet server process listening on Port 8140 and you can start adding other systems to your Puppet server. But there is one more thing: what happens if you run the puppet apply command again? # puppet apply /root/puppetserver.pp Notice: Compiled catalog for puppetmaster.example42.training in environment production in 0.67 seconds Notice: Applied catalog in 0.14 seconds As you can see, nothing happens. This concept is called idempotence. Puppet will always first check the actual system state, compare it with the desired declarative state and only perform actions in case that there is a mismatch. Puppet CA and Puppet agents Puppet uses client SSL certificates to authenticate and authorise Puppet agent connections to the Puppet server process. Per default the Puppet server uses a self signed CA to retrieve certificate signing requests and sign certificates. Information about the Puppet CA can be read using the puppet cert command. # puppet cert list --all + "puppetmaster.example42.training" (SHA256) D3:F3:44:8D:AE:13:CC:AB:FE:F0:DA:74:13:64:7A:44:09:92:23:CF:90:A4:73:17:5F:35:6C:70:14:64:9B:A8 (alt names: "DNS:puppet", "DNS:puppetmaster.example42.training") You can even read the CA by using the print parameter: puppet cert print puppetmaster.example42.training [... output truncated ...] Now you are able to connect your first node to the Puppet master. There are some prerequisites that must be accomplished first which are the same as the ones you needed to ensure prior installing the puppetserver: Network setup DNS/Hostname Time settings We set the hostname: hostnamectl set-hostname agent1.example42.training echo "<your IP> agent1.example42.training" >> /etc/hosts And we set the time: yum install -y ntpdate ntpdate pool.ntp.org Now you must either have correct DNS setup or you can add the puppetmaster IP and hostname to /etc/hosts file: echo "<master IP> puppetmaster.example42.training >> /etc/hosts ping -c1 puppetmaster.example42.training Now you are installing the puppet agent repository and package: rpm -Uvh https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm yum -y install puppet-agent Now you are able to use the puppet agent: puppet agent --test Info: Creating a new SSL key for agent1.example42.training Error: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known) Exiting; failed to retrieve certificate and waitforcert is disabled Please note the --test parameter!! When running puppet agent with no further option, the agent will fork from shell and run in background permanently, “waking” himself every 30 minutes. The --test option ensures that the agent process stays in foreground and is running one time only. You receive an information that the agent has created an SSL certificate, but it can not connect to the master. When no Puppet master is configured the Puppet agent will try to connect to a system with name “puppet”. As your Puppet server uses another hostname, you must provide information regarding the DNS name of the puppetserver. You can either provide the information on command line, using the --server option: puppet agent --test --server puppetmaster.example42.training or add the configuration to the puppet.conf file: # /etc/puppetlabs/puppet/puppet.conf # This file can be used to override the default puppet settings. # See the following links for more details on what settings are available: # - https://docs.puppetlabs.com/puppet/latest/reference/config_important_settings.html # - https://docs.puppetlabs.com/puppet/latest/reference/config_about_settings.html # - https://docs.puppetlabs.com/puppet/latest/reference/config_file_main.html # - https://docs.puppetlabs.com/puppet/latest/reference/configuration.html [agent] server = puppetmaster.example42.training Now you can run the puppet agent command again: puppet agent --test Info: Caching certificate for ca Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml Info: Creating a new SSL certificate request for agent1.example42.training Info: Certificate Request fingerprint (SHA256): 4A:30:16:96:14:01:3E:B6:2D:5A:3A:E7:B4:FE:C9:2D:FC:76:1C:AC:39:15:50:E1:88:A3:3C:8A:39:9E:5D:13 Info: Caching certificate for ca Exiting; no certificate found and waitforcert is disabled This time you received one error message, claiming that ‘no certificate found’. This information comes from the Puppet master. Switch back to your Puppet master and check for information regarding certificates using the puppet cert command: puppet cert --list "agent1.example42.training" (SHA256) 4A:30:16:96:14:01:3E:B6:2D:5A:3A:E7:B4:FE:C9:2D:FC:76:1C:AC:39:15:50:E1:88:A3:3C:8A:39:9E:5D:13 The puppet cert --list command shows you outstanding certificate signing requests. Now you can sign the request: puppet cert --sign agent1.example42.training Signing Certificate Request for: "agent1.example42.training" (SHA256) 4A:30:16:96:14:01:3E:B6:2D:5A:3A:E7:B4:FE:C9:2D:FC:76:1C:AC:39:15:50:E1:88:A3:3C:8A:39:9E:5D:13 Notice: Signed certificate request for agent1.example42.training Notice: Removing file Puppet::SSL::CertificateRequest agent1.example42.training at '/etc/puppetlabs/puppet/ssl/ca/requests/agent1.example42.training.pem' Now the agent run on your node will work: puppet agent --test Info: Caching certificate for agent1.example42.training Info: Caching certificate_revocation_list for ca Info: Caching certificate for agent1.example42.training Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Caching catalog for agent1.example42.training Info: Applying configuration version '1523190091' Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml Notice: Applied catalog in 0.01 seconds Congratulations. You have now a working setup between your agent and your master and you can run the agent permanently: service puppet start Redirecting to /bin/systemctl start puppet.service For now the node has no configuration on the Puppet server and therefor will not have anything managed. This is dealt with in the next tow. Stay tuned. In the next posting we will dig deeper into Puppet DSL, how to write flexible Puppet code instead of writing per node Puppet code, how to make use of existing Puppet module libraries how to deal with slight differences between systems and how to store Puppet code and Puppet data. Martin Alfke 2018-04-02T00:00:00+00:00 https://example42.com/blog/2018/04/02/example42_puppet_modules_status_update A quick update on the status of example42’s Puppet works. Modules 2008 We released our first set of Puppet modules ten years ago, in 2008, at Puppet 0.24 times, looking at them now I can’t prevent from smiling slighly embarassed: they had some unique features for the times (decommissining support, multi OS support by design, first attempt to separate common code from local implementation) but look definitively naive now. Modules 2010 Two years later, in 2010, when current Puppet version was 2.6, we released our Next Gen set of modules, a completely rewritten army of modules with a somehow revolutionary feature: each class parameter value was looked up by the params_lookup function, which looks for values on Hiera and top scope variables. Note this was done before the release of Puppet 3 and automatic data binding on Hiera. The params_lookup function is provided by the puppi module, which is therefore required in the modulepath (but you don’t need to include it). The modules had also optiona dependency of our monitor and firewall modules, to automatically manage monitoring and firewalling of the managed applications. All this has probably contributed to the wrong idea, which I heard too often around, that these modules couldn’t be cherry picked but had to be taken as a whole set. Well, that’ not true, all you need to use any nextgen module is the module itself and puppi in the modulepath. Next Gen modules had several other feautures aimed to improve their reusability: parameters to define and override almost any variable used for package and service names, file paths, permissions and so on, parameters to control how to provide configuration files (as templates or sourced static files), and a lot of other common features (modules were generated from a common blueprint and then customised for the relevant application). Those modules still mostly work now on Puppet 5 (minor fixes might be necessary in some parts) and are basically most of the current example42 ones you see on the forge. Also, most of them are currently deprecated: we are not actively working on them, for years, and receive and accept PRs to fix issues or ensure support on latest Puppet versions. Even if they mostly do their work, we would not recommend them anymore: our suggestion for migration is either to choose a more updated module from other authors or enter the Tiny Puppet / PSICK world (more details later on this) Modules 2018 Some of the modules designed at “NextGen” times (problems with names like this is wondering what comes after the “next” gen…) are still supported and maintained either directly by Example42 of by some other mainteiners. The most relevant ones are apt, yum, puppi and the network module. The latter is actually the reason I’m writing this post: my personal view on the module is somehow erratic, even if it’s currently the only example42 Puppet module with the “Approved” label (when Puppet introduced Approved modules we were already deprecating most of our ones), I’ve given little attention and love to it for years, to the point that at some point I wanted to look for external maintainers and, in case no one were found, even deprecate it. Some comments and discussions (here and here) on GitHub have forced us to reconsider this decision. The network module might be the only reason you know example42 and maybe the only example42 module you are using, despite our past efforts on dozens of, now deprecated, modules and the current one on unconventional modules that most people probably don’t understand or consider useful (tp and psick). Modules future So we decided to keep on maintaining the network module and actually dedicate it much more time and resources than in the past. Here’s a short list, to sum up what is our current stance on the Puppet modules we have written and our plans for them: network. We’ll keep on supporting it, with more effeort than in the last years. Short term goal is to improve documentation and flush PRs and tickets, longer term is to evaluate how to upgrade it (module comes from the “nextgen” era, it has been adapted to avoid puppi dependencies but it’s still based on old Puppet language). It’s working on all Puppet versions from 2.x to latest. puppi has not received support for years but still does its job. It has 3 main functions: provide the params_lookup function, which is now used only on the deprecated nextgen modules (but still works on Puppet 5) provide shell commands to check the status of the managed applications (puppi check, puppi log, puppi info), which are now ideally replaced by Tiny Puppet commands (tp test, tp log) provide Puppet defines and shell commands to manage deployment of applications using different workflows defines in Puppet language (puppi deploy $app, puppi rollback $app). We find this functionality still valid and somehow underrated (in the IT shop when puppi was written 8 years ago, it’s still successfully used to seamlessly deploy hundreds of different applications) and in the future we might consider the opportunity to write a new puppi version which concentrates only on applications deployments. Other Example42 modules on the forge and GitHub. They are all deprecated (there’s a deprecation notice at the beginning of the README) or maintained by third party supporters. We might give some support to them, especially the most used ones, in terms of possibility to run on recent Puppet versions and OS, but we still consider them a dead end, as they are already replaced by much more modern PSICK profiles. tp. Tiny Puppet, coupled with PSICK module or used in your profiles, can replace ALL the old example42 modules. We are not actively working on new Tiny Puppet features as it currently does all we need. The companion tinydata module, with data for different applications on different OS is regularly updated with new or fixed data. We plan to keep on using them in the foreseeable future. psick. Psick, the module, the control-repo and the separated sample hieradata module is currently where we concentrate most of our Puppet developments. If you want to have an idea of how, in 2018, we design our Puppet infrastructures, give it a look and, please, before judging or dismissing it be sure to understand it ;-) Finally, after years of solitary development (believe it or not, besides external contributions coming via PRs, most of the development of all these modules was done by me in slices of spare time), as example42 we have decided to dedicate more resources to the maintenance of our public Puppet works. So if you are an example42 user and/or know and possibly like what we do, and want to be paid for development of our Open Source modules, contact us, we have a lot of work to do and are in the condition of involving more people on it. Alessandro Franceschi 2018-03-26T00:00:00+00:00 https://example42.com/blog/2018/03/26/example42_puppet_cheat_sheet Yes, we know, there is a famous Puppet cheat sheet available at docs.puppet.com This is my personal collection of different Puppet DSL items, which I usually generate at each official Puppet Fundamentals training. Now I found the time to paste them into a blog posting and add information on code logic and hiera. Puppet DSL Resource type declaration: type { 'title': param => 'value', } Class definition: class <name> ( DataType $param1, # this parameter must be provided upon declaration DataType $param2 = 'value', ) { # Puppet DSL code } Class declaration using a function: include <name> # no ordering, the mentioned class will be somewhere in the catalog require <name> # strict ordering, the class must be finished prior continuing contain <name> # local ordering, the class must be finished within the class where the contain function is used Class declaration using class as resource type: class { '<name>': param1 => 'value', } Self defined resource type definition: define <name> ( DataType $param1, DataType $param2 = 'value', ){ # Puppet DSL # all resource type declaration must use the $title variable # older Puppet code uses $name instead of $title } Self defined resource type declaration: <name> { 'title': param1 => 'value', } Puppet DSL code logic Case statement: case $test_variable { 'value1': { # specific value # Puppet DSL } /regexp/: { # regular expression # Puppet DSL } 'value2', 'value3': { # multiple values # Puppet DSL } default: { # fall back value - optional # optional, Puppet DSL } } If statement: Variant 1: Boolean or existing variable: if $test_variable { # Puppet DSL } else { # else is optional # Puppet DSL } Variant 2: test content of variable: if $test_variable == 'content' { # Puppet DSL } Variant 3: test content on regular expression: if $test_variable =~ /regexp/ { # Puppet DSL } Selector: Please use selectors sparsely as they very fast lead to hard to read Puppet code! $result_var = $test_var ? { 'value1' => 'return_val1', 'value2' => 'return_val2', default => 'return_val3', } Puppet DSL lambda functions Iterating over an array: $var = [ 'element1', 'element2' ] $var.each |DataType $key| { type { $key: param => 'value', } } Iterating over a hash: $var = { 'key1' => { 'var1' => 'val1', 'var2' => 'val2', }, 'key2' => { 'var1' => 'val1', }, } $var.each |DataType $key, DataType $val| { type { $key: * => $val, } Puppet and Hiera 5 Explicit lookup class foo { $data = lookup('key', DataType, <merge behavior>, <default value>) } DataType, ‘merge behavior’ and ‘default value’ are optional merge behavior: 'first' # returns the first occurrence of 'key' 'unique' # returns an array of all occurrences of 'key' with duplicates removed 'hash' # returns a hash of all occurrences of 'key', duplicates hash keys are taken from highest priority 'deep' # returns a hash of all occurrences of 'key', duplicate hash keys are merged Automatic data lookup class foo ( DataType $data = 'value' # identical to $data = lookup('foo::data', DataType, 'first', 'value') ) { } Puppet will automatically query hiera for the key 'foo::data' Puppet and Resource ordering Ordering with meta parameters Variant 1: require and subscribe package { 'foo': ensure => present, } file { '/etc/foo/foo.conf': ensure => file, require => Package['foo'], } service { 'foo': ensure => running, subscribe => File['/etc/foo/foo.conf']. } Variant 2: before and notify package { 'foo': ensure => present, before =: File['/etc/foo/foo.conf'], } file { '/etc/foo/foo.conf': ensure => file, notify => Service['foo'], } service { 'foo': ensure => running, } Variant 3: resource chaining package { 'foo': ensure => present, } file { '/etc/foo/foo.conf': ensure => file, } service { 'foo': ensure => running, } Package['foo'] -> File['/etc/foo/foo.conf'] ~> Service['foo'] or multiline: Package['foo'] -> File['/etc/foo/foo.conf'] ~> Service['foo'] Puppet Module A module is a directory structure inside the $modulepath. <modulepath>/ \- <modulename> |- manifests/ | |- init.pp # main class definition: class <modulename> { ... } | |- subclass.pp # subclass definition: class <modulename>::subclass { ... } | \- folder/ | \- subclass.pp # subclass in directory definition: class <modulename>::folder::subclass { ... } |- files/ | \- staticfile.conf # static file: source => 'puppet:///modules/<modulename>/staticfile.conf', |- templates/ | \- dynamic_config_file.epp # templated config file: content => epp('<modulename>/dynamic_config_file.epp'), |- facts.d/ | \- external_facts.yaml # plugin synced |- types/ | \- datatype.pp # type <modulename>::datatype { ... } |- functions/ | \- puppetfunction.pp # function <modulename>::puppetfunction { ... } |- lib/ | |- facter/ | | \- custom_fact.rb # $facts['custom_fact'] | \- puppet/ | |- functions/ | | \- <modulename>/ | | \- puppet4function.rb # Puppet::Functions.create(':<modulename>/puppet4function) do ... end | |- parser/ | | \- functions/ | | \- puppetfunction.rb # legacy functions - avoid : module Puppet::Parser::Functions.newfunction(':puppetfunction) do ... end | |- type/ | | \- custom_type.rb # Puppet::Type.newtype(:custom_type) do ... end | \- provider/ | \- custom_type/ | \- custom_provider.rb # Puppet::Type.type(:custom_type).provider(:custom_provider) do ... end \- spec/ Happy hacking, Martin Alfke 2018-03-19T00:00:00+00:00 https://example42.com/blog/2018/03/19/wholistic_devops “DevOps is the collaboration of Developers and Operations to allow faster deployments.” This is a wording we usually hear from customers, technical staff and managers. But is this really everything? This posting will explain why DevOps does not only have a purely technical implication but also affects any other company departments. DevOps at technical level The DevOps movement was first mentioned somewhen back in 2007. In this time it was really a purely two department collaboration and communication procedure which had its roots in agile development. As development was pushing out releases faster, there was a need for IT Ops to adopt to the new release cycle. This is where the DevOps movement founds it base. DevOps and Security Spectre, Meltdown, Heartbleed, zero-day exploits. Security is of high concern for every company with IT departments. Most security departments where usually seen as the “no-sayers”. With every new request for security related review of platforms and code, usually IT security had some complaints. Nowadays IT Security gets involved into the planning, implementation and deployment process far more early, allowing them to share knowledge on security topics and provide guidance on security implementations. Some people gave this new way of collaboration a new term: SecOps or DevSec or even DecSecOps. But there are also non technical departments which must adopt to DevOps culture and principles. The most stunning ones will be HR and Finance. But was has HR and Finance to do with DevOps? DevOps and HR Remember the old times? Companies placed a job offering on their website, waiting for resumes to flow in, reviewing and pre-processing them and passing a few to the IT department lead for in detail review and conducting a job interview. Now HR learned that IT is working as DevOps team. Due to a misunderstanding in wording and methods, they now start looking for DevOps engineers. But what should a DevOps engineer be capable of? Should it be more development skills or system engineering knowledge? In general an organisation doing DevOps in IT has specific needs for specific specialised knowledge. Besides this - as the different IT sub departments now have to strongly communicate with each others a new skillset gained even more importance: soft skills like communication and willingness to adopt. Another item we saw in the past was IT department not allowing their team members to talk about the technical setup or their work. This has changed drastically as more and more companies lack sufficient staffing, leaving positions open for a very long time. This is where HR MUST jump in and take over responsibilities. It is about teaching, training and talking which is of high value for HR to get more resumes so they are able to fill open positions faster. HR must allow (and insist in fulfilment) team members to attend conferences as speakers or even run local meet-ups where people with similar mindset gather. DevOps and Finance As IT now acts as a department based on DevOps principles they will try to move from slow to manage, self hosted systems to some cloud concepts. Especially when using public cloud providers, the collaboration with Finance is of high importance. Finance for example differentiates between capital expenses (CAPEX) and operational expenses (OPEX). Ask any technician regarding the differences and he will not know. Even when IT says that public cloud costs can be more easily predicted upfront, Finance might ask for another solution. Why? On premise own hardware is usually billed as capital expenses, whereas public cloud is operational expenses. Moving all costs from CAPEX to OPEX might not be financially the best solution from tax perspective - especially when your cloud costs grow to a high volume. Always try to explain Finance what you want to achieve and ask for their feedback on IT plannings. Besides this: Finance and procurement can even help you in negotiating better terms with your cloud provider. This is especially if you are high volume user. DevOps and Management The most difficult part is DevOps and their managers. In general there are two approaches on how to go for DevOps methods in departments: top-down approach grass-root approach Within the top-down approach, top level management makes the decision that teams must work according to DevOps methods. Now it is the top managers responsibility to find and support staff which would like to take part in the migration and to convince staff which objects to the upcoming changes. This is different from the grass-root based DevOps implementation, where teams decide for themselves to go on the DevOps journey. Here the teams must at least ask for clearance to run a test. The team is responsible to deliver results fast, usually by going for low hanging fruits and showing the management that the approach is worth to spend more time and money into. But how about the mid management? In older times the mid management was responsible to ensure top level management goals are achieved. They had to look carefully for proper KPI and sometimes needed to push pressure on teams when positive results where in danger. Within a DevOps organisation the mid management gets a new focus. They are responsible to remove obstacles from the team, allowing the team to concentrate on their work and not do to many unplanned work. They must start seeing themselves no longer as bosses, but as leaders. They must ensure high motivation within the team and take care on high learning process by coaching senior people to train junior staff. Conclusion DevOps is not purely a technical way of collaboration. It affects a company organisation as a whole. When going the DevOps approach it is highly recommended to ensure that every department has the data it needs to help building success without loosing money or resources. Martin Alfke 2018-03-12T00:00:00+00:00 https://example42.com/blog/2018/03/12/control-repo-change-impact A Puppet control-repo is usually the place where most of the Puppetteer activities are done. This is especially true if we keep our Hieradata and local profiles in the control-repo git repository, rather than maintaining them in separated git repos, as external modules. We always wonder (well, we should) what might be the impact of a change on the control repo on the managed infrastructure. This might be an issue espectially for who is learning Puppet, Hiera and the local interactions. Here we try to summarize the possible impacts on changes on different files, underlying that every mileage may vary and referred paths ultimately depend on how we structure our code and data in the control-repo. We define 4 risk levels: [SAFE] Changes done here are totally safe in terms of impact on running servers (or generally any Puppet managed system) [LIMITED] Changes here impact a limited number of servers or not critical elements [WARNING] Changes may impact several servers and should be considered with care [DANGER] Changes may have a very large impact. Be sure to be aware of what we are doing Let’s review what level of risk may be associated to changes on the different control-repo files. Needless to say that they refer to actual changes in Puppet code and data, if we are just adding commentef lines we can be confident that we change won’t have any effect (unless we change a configuration file on a system which may trigger a service restart). We assume to have hieradata in the data/ directory and local profiles in site/profile/. [SAFE] README.md, docs/, LICENSE or any other documentation or general information file. Changes done here won’t have any impact on our servers [DANGER] hiera.yaml is the Hiera configuration file for the environment, changes here on the hierarchy or the used backends may affect several systems in more or less unpredictable ways. We should edit it only if we know what we are doing. In case of backend changes or big refactors, server side enforced noop mode is highly recommended. [DANGER] data/common.yaml, data/defaults.yaml or any file that contain Hiera data which is used for all the nodes (when not overridden in more specific layers of the hierarchy), so any change here may impact several servers. Be aware. [WARNING] data/role/$role.yaml, data/zone/$zone.yaml, data/env/$env.yaml contains Hiera data which is used for a more or less large group of nodes. The actual path names depends on how is our hierarchy. It’s recommended here to test at least one node belonging to the affected group, before promoting the change. [LIMITED] data/nodes/ contains Hiera data for specific nodes (again, the actual path may change according to the hierarchy, but there always should be one level matching each node certname. Here we can place nodes specific settings, which are easy to test (directly on the involved node) and have a limited impact (only the node having the name of the file we change). [DANGER] manifests/site.pp or any other manifest here may impact all the nodes. Handle with care. [WARNING] Puppetfile contains the list of the modules to add to the control-repo. If we add a new module we won’t have any effect on nodes until we actually start to use its classes or defines. If we remove a module we’ll break Puppet runs in all the nodes that eventually use it. When we add or remove modules, we may see on our nodes files changing at the first Puppet run: these are due the contents of module’s plugins being synced to the clients (pluginsync feature) they are normal and won’t affect our servers operations. When we change versions of the used modules, we might impact exiting nodes. Versions changes for the used modules should always be tested on each managed OS. [WARNING] .gitlab-ci.yml, Jenkinsfile, .travis.yml define, according to the used tool, how is the CI pipeline to test our code, changing these files, or eventual CI commands used in CI on directories like bin/ or scripts/ may break our CI (and that’s something that should always have the highest priority for fixing). [WARNING] site/profile/* here may stay local profiles, templates, files, facts, resource types, data types. Changes to our profiles in site/profile/manifests impact all the nodes which classifies them. Changes to site/profile/files or site/profile/templates may actually change the contents of configuration files on our managed systems. We should not be too much worried about the above dangers and warnings, though, it’s normal in the life of Puppet admin to edit such files. We have just be aware of the potential impact area of our change and, when we are not fully confident on what we are doing, we should always check our changes in noop mode before actually enforcing them. If you’d rather have this kind of risk mapped out for your own control-repo, that’s part of what our Puppet Health Check covers. Alessandro Franceschi 2018-03-05T00:00:00+00:00 https://example42.com/blog/2018/03/05/catalog-diff-on-refactoring Note: This is an updated version of the Tip of the Week 4 - Existing code and Puppet 4 dealing with octocatalog-catalog diff only. General Refactoring Puppet code is a task which we usually have to do, especially when being at a customer with Puppet code with a long history. There are several reasons for refactoring: Adopt to new best practices Migrate from Hiera config v3 to Hiera config v5 Restructuring code to allow better code management But how can you verify that the refactoring was done properly, so no changes on systems occur? This is where catalog diff tools come into place. Originally these tools have been created to allow smooth upgrade from Puppet 3 to Puppet 4 (like the puppetlabs-catalog_diff or octocatalog-diff). Installation Both tools use a different approach. The puppetlabs-catalog_diff is installed as a puppet module, bringing a new puppet interface (command line option). Installation can be done using the puppet module installcommand: puppet module install puppetlabs/puppetlabs-catalog_diff After installation there is now a puppet cli command available: puppet catalog diff The octocatalog-diff is delivered as a ruby gem. It needs ruby 2.0 or newer and can be installed using the puppet gem or - in case you have a sufficient ruby version - the gem command: /opt/puppetlabs/puppet/bin/gem install octocatalog-diff The tool needs a local Puppet agent installed and can be used on Linux and Mac OS. Windows is not supported. Configuration The octocatalog diff tool needs a configuration file (.octocatalog-diff.cfg.rb) unless you want to specify all parameters as command line options. The configuration file can reside in different locations. Lookup for the file is done in the following order: in local directory in home directory /usr/local/etc/octocatalog-diff.cfg.rb /opt/puppetlabs/octocatalog-diff/octocatalog-diff.cfg.rb /etc/octocatalog-diff.cfg.rb The configuration covers 4 different parts: Hiera Node Classifier (ENC) PuppetDB and Puppet Hiera If you are already using Puppet 4.9 or later and when you have a hiera.yaml (config v5 format) in your environment root, Puppet will recognize the file by itself, so will octocatalog-diff do. In this case you are good to skip the hiera settings and you are asked to NOT configure hiera settings in octocatalog-diff. ENC If you are using an external node classifier, you must tell octocatalog-diff about it. Puppet Enterprise uses the classifier API as ENC, which can be configured using a token or a whitelist. If you are using token authentication against the Puppet Enterprise RBAC service, one needs to configure the following settings: octocatalog-diff \ --pe-enc-url https://your.pe.console.server:4433/classifier-api \ --pe-enc-token-file /path/to/token/file.txt \ --pe-enc-ssl-ca /path/to/ca.crt \ [other options] If you are using a whitelisted SSL keypair the following options must be used: octocatalog-diff \ --pe-enc-url https://your.pe.console.server:4433/classifier-api \ --pe-enc-ssl-ca /path/to/ca.crt \ --pe-enc-ssl-client-cert /path/to/client.crt \ --pe-enc-ssl-client-key /path/to/client.key \ [other options] If you are using another ENC, you must pass the ENC file option: octocalaog-diff \ -enc bin/enc.sh \ [other options] PuppetDB PuppetDB can be used to store latest node facts, exported resources and node catalogs and their reports. In this specific case we are interested to receive the latest catalog from PuppetDB. Octocatalog-diff only supports PuppetDB API v4 which means that you must run PuppetDB 2.3 or newer. Usually Puppet Master is able to communicate with PuppetDB using a certificate whitelist. We can re-use this setting as this is the most secure way how to configure access to PuppetDB. Octocatalog-diff must be configured to use proper certificates by using the following settings in configuration file: settings[:puppetdb_url] 'https://puppetdb.server:8081/ settings[:puppetdb_ssl_ca] 'path to Puppet SSL CA file' settings[:puppetdb_ssl_client_cert] 'path to Puppet SSL certificate settings[:puppetdb_ssl_client_key] 'path to Puppet SSL key file' If you prefer to set CLI options you can use the following form: octocatalog-diff \ --puppetdb-url https://puppetdb.example.net:8081 \ --puppetdb-ssl-ca FILENAME --puppetdb-ssl-client-cert FILENAME --puppetdb-ssl-client-key FILENAME Access to PuppetDB must be grated by using the PuppetDB SSL whitelist option. Puppet Last but least, octocatalog-diff must know about the Puppet installation which it uses to compile a catalog. Note: the puppet-agent is sufficient. There is no need to install puppetserver on a system where you want to make use of octocatalog-diff. settings[:puppet_binary] = '/opt/puppetlabs/puppet/bin/puppet' Please note, that octocatalog-diff must have the puppetdb-termini package installed in case that you also want to get exported resources in a nodes catalog. Usage Now you are able to verify your settings by running octocatalog-diff --config-test Read the output carefully and see whether all settings are correct for your installation. In normal (module) mode, octocatalog-diff will assume that the base for the catalog will be the origin/master branch on a git repository. For control repository you want to change this by setting the -f origin/productoin CLI parameter. Facts can be read from PuppetDB - if configured - or from a specifc facts file which gets added using the --fact-file option. The nodes to test are provided using the -n <nodename> option Additional descriptions and usage can be found in the octocatalog-diff documentation. Happy hacking and refactoring using catalog diff verification. If you’d rather have this kind of refactoring risk assessed for you, that’s part of what our Puppet Health Check covers. Martin Alfke 2018-02-26T00:00:00+00:00 https://example42.com/blog/2018/02/26/devops-permanent-learning DevOps engineers, DevOps teams, DevOps academies. Everybody sees new stuff appearing reusing the same wording again and again. Do we really have DevOps engineers? What responsibilities, tasks, expertise does a DevOps engineer have? DevOps engineers and academies First: there is no DevOps engineer, no DevOps teams, no DevOps academy. DevOps is a culture on how to build a community of trust and engagement, breaking silos by enabling cross team communication. But every IT technical job is not purely culture only. Next to learning how to communicate with others, it is highly important to also expand technology skills. New toolings Over the past 10 years we saw a large amount of different technology getting adopted and used in platforms. There is now a large collection of different tooling e.g. for logging and reporting which needs understanding prior using in production. Some people learn by testing the new stack in a separate development environment, reading documentation and mailinglists. Others prefer to go to a workshop session e.g. within a specific meetup group or by attending a training course. Learning is essential What we usually see at different customers is a huge need for training IT staff, but mostly departments lack a proper funded training budget. There is a saying that in such situations always comes up: CFO to CTO: "What if we send all staff to trainings and then they leave! CTO to CFO: "What if we dont send them to trainings and they stay?" Culture and attitude Sending people to trainings is not about paying for something where the company does not receive a benefit from. Sending people to trainings make them feel being taken care about, makes them feel being supported by their manager and HR department. Conclusion Dear managers, if you want your IT department to support you in growing your business then it is essential that your staff knows how to run and manage platforms now and in future. Get a proper funded budget for trainings and education. Usually it is OK to send one or two only and let these then educate the remaining colleagues. Don’t only send the same two people, spread knowledge of a larger amount of people, reducing the risk of loosing knowledge when one or two people will leave. Support and cultivate knowledge sharing inside the company: organise internal workshops, presentations, hackatons, events where your same team members become trainers, of things they know better, and trainees, on the topics they don’t master. Don’t try to make people stop leaving. Try to show them a path to achieve when remaining with you. Even if the want to leave: don’t stop supporting them. Build a trust relationship in your entire company, so people will always tell good things about the job and the company. Every invest in your people is an invest in your success. Happy training and hacking, If you’re looking for a funded, structured way to build that Puppet knowledge in-house, that’s exactly what Puppet Training is for. Martin Alfke 2018-02-19T00:00:00+00:00 https://example42.com/blog/2018/02/19/anatomy-of-a-puppet-control-repo-compared-to-psick Note: This is an updated and improved version of the Tip of the Week 2 - Anatomy of a Puppet control-repo. A Puppet control-repo is a [git] repository that contains the files you expect to have in your directory environment (for example /etc/puppetlabs/code/environments/production for the default production environment) which provides whatever you need to manage your infrastructure. The standard control-repo A control repo is typically composed of: The manifests directory where are placed the first files that the Puppet server parses when compiling catalogs for clients. Here you typically have the site.pp file (but other manifests with different names can be seamlessly added) where you can set top scope variables, resource defaults, and eventually have node statements to define what classes should be included in your nodes (nodes classification can be done in several different ways, using the node statement is just one of them). The hieradata (or data) directory which contains Hiera data files. The name of the directory is completely arbitrary, even if these are a sort of standard de facto. This same directory could not even exists in the unlikely case you are not using Hiera, or if you use Hiera with backends which don’t store data in normal (typically yaml or json) files. Hiera data can also be placed in a separated dedicated repository, eventually a “fake module” to load via Puppetfile so you might have it outside the control-repo, and have different people who might access the Hiera data and the control-repo code. The hiera.yaml file configures the Hiera backed and the hierarchies to use for the environment The modules directory contains Puppet modules. Typically you don’t place themselves directly in your control-repo but define them in the Puppetfile and then deploy them with either r10k or Librarian Puppet. You will probably have to develop custom modules (your role and profile modules, your site specific ones). You can decide to place them in dedicated repositories, and add them to your Puppetfile, or keep them in your same control-repo. In this case it makes sense to place them in a separated directory, such as site, in order to differentiate external modules, defined in the Puppetfile, from the local ones. The environment.conf file, which configures your environment: where the modules are placed, the caching timeout and eventually a script that returns a custom configuration version. You can find an essential skeleton for a control-repo, from Puppet. The PSICK additions Example42’s PSICK is a Puppet control repo with superpowers. It contains all the above directories and something more: The vagrant directory contains different Vagrant environments with the relevant toolset that can be used to locally test the same control-repo. They are fully customizable by editing the config.yaml file in each Vagrant environment. Files for building Docker images locally are under the docker directory. Fabric tasks are defined in the fabfile directory. Documentation is stored under docs The bin directory contains several scripts for various Puppet master activites. Many of them can be invoked via Fabric or are used in the CI pipelines. The files .gitlab-ci.yml, Jenkinsfile and .travis.yml are used to configure CI pipelines for Puppet code on different tools Rakefile, Gemfile and the spec/ directory are needed to run control-repo unit and integration tests Dangerfile configures the Danger bot based which automatically send comments based on control-repo changes metadata.json was added to be able to use pdk within the control-repo. psick command can be used to create a new control-repo based on the Puppet standard (and essential) one, or example42’s big, fat, rich, psick one. Besides these additions, PSICK can be used as a normal Puppet control repo, and it fully follows its conventions and rules. If you’d rather start from a control-repo built and tuned for you, that’s what Puppet Infrastructure Kickstart delivers. Alessandro Franceschi 2018-02-12T00:00:00+00:00 https://example42.com/blog/2018/02/12/how_to_use_psick_module_for_common_profiles At last Configuration Management Camp we have presented PSICK, giving some background info on the reasons behind it, an overview what it does and glimpses on how it can be used both for new and existing Puppet infrastructures. In this post we are going to give a deeper look to how to cherry pick the profiles of the psick module we may need in our current infrastructure. Using psick module in existing infrastructures The psick module makes large use of example42’s Tiny Puppet, which provides defines that allow installation of management of applications in a quick and powerful ways. Tiny Puppet requires the tinydata module, where is stored the data of all the supported applications on different OS. Both psick and tp need Puppet’s stdlib module, but you probably have already it. To use psick module add to your Puppetfile: mod 'example42/psick', :latest mod 'example42/tp', :latest mod 'example42/tinydata', :latest In a production environment we will likely specify fixed and tested versions, the current latest version of the above modules are: mod 'example42/psick', :0.5.8 mod 'example42/tp', :2.1.0 mod 'example42/tinydata', :0.2.3 This may look a lot for not doing anything, by default, but these modules empower the possibility to manage a lot of common system configurations and applications: potentially saving you from adding several third party modules and giving you the power and flexibility of Tiny Puppet to manage applications on your site profiles. Once you have psick in your modulepath, you have to classify the main psick class (whatever method you use to assign classes to nodes) with something like: include psick This is a prerequisite for all the psick profiles, as they use common variables evaluated in the psick main class. By default, by including the psick class, nothing is done on your nodes: just a few empty classes are added to your catalog (psick::pre, psick::base, psick::profiles: they can be used for classification done directly via the psick module, but that’s another feature we are not going to talk about here). Then, when you need to manage an application or a component of your system, you will be able to choose from these options: Find and use a public module that does the job Write a custom profile that wraps existing modules functionalities Write a custom profile or module that implements what you need without using external code (you can decide or not to use tp::install to manage your packages and service and tp::conf to manage your configuration files) Find and use a psick profile that does what you need. Usage of psick profiles is similar to the one of classes from existing modules: you classify nodes with the relevant class name, and then use hiera to configure it via it’s parameters. PSICK profiles Psick comes with a wide like of base profiles for common settings, here are the most significant ones: psick::hosts - Manage /etc/hosts psick::motd - Manage /etc/motd and /etc/issue psick::nfs - Manage NFS client and server psick::sudo - Manage sudo configuration psick::sysctl - Manage sysctl settings psick::firewall - Manage firewalling psick::openssh - tp profile and keygen define psick::hardening - Manage system hardening psick::network - Manage networking psick::puppet - Manage Puppet components psick::users - Manage users psick::time - Manage time and timezones There are also several other application specific profiles, where it’s generally possible to choose what module to use to manage the actual application (if a popular public module or Tiny Puppet): psick::ansible - Manage Ansible installation psick::aws - Manage AWS client tools and VPC setup psick::bolt - Manage Bolt installation psick::docker - Docker installation and build tools psick::foreman - Foreman installation psick::git - Git installation and configuration psick::gitlab - GitLab installation and config psick::mariadb - Manage Mariadb psick::mysql - Manage Mysql psick::mongo - Manage Mongo psick::php - Manage php and modules psick::oracle - Manage Oracle prereq and setup psick::sensu - Manage Sensu In PSICK all the application profiles that use Tiny Puppet have a standard structure and common parameters. They have names like psick::<app>::tp, and are generated with pdk using this template. TP encourages the usage of the template + options hash pattern, ad has some generic templates usable for common file structures. So, for example, to manage openssh with psick , we can: include psick::openssh::tp and configure it with parameters like: psick::openssh::tp::resources_hash: tp::conf: openssh: template: 'psick/generic/spaced.erb' options_hash: Protocol: 2 PermitRootLogin: 'no' Subsystem: 'sftp /usr/libexec/openssh/sftp-server' Here, by including the psick::openssh::tp profile we have Openssh package installed and the relevant service started. We configure it using the resources_hash parameter, common in all tp profiles, where for each configuration file to manage via tp::conf, or whole directories to manage with tp::dir, we specify name, and content (by using parameters like source, content, template or epp) and a custom options_hash where any parameter, referred in the used template, looked up in deep merge mode, can be managed via Hiera. Alessandro Franceschi 2018-02-05T00:00:00+00:00 https://example42.com/blog/2018/02/05/fosdem_and_cfgmgmt_camp Example42 attended FOSDEM in Brussels and will be at Configuration Management Camp in Ghent. FOSDEM You can have an idea of FOSDEM only when you attend it. Thousands of people, hundreds of talks, dozens of tracks, everything about Open Source and its multiple fields of application. When you go to FOSDEM you have always the problem of picking the presentations that interest you and, then, manage to get inside the rooms, which are often full. Among the presentations we attended, besides the “impressive as usual” demos from James Shubin with MGMT we were amazed by the demonstration of the forthcoming version 5 of Grafana. We are excited at the idea of integrating it in the psick module, so expect a psick::grafana profile soon with hopefully some out of the box dashboards and integrations with the managed applications. Other tools we used in the past, or not, have grabbed our interest and, as it always happens, we left the conference with new ideas or inspiration on things to learn and do (and the latent frustration at the idea that we will hardly have the time to cope with all of them). We take the occasion to send a great thank you to Toshaan Bharvani and Kris Buytaert for the car rides from FOSDEM / Brussels to CfgMgmtCamp / Ghent. CfgMgmtCamp This year, example42 is again sponsoring CfgMgmtCamp. We believe that the Ghent based event - celebrating its 5th year anniversary - is one of the most important ones to attend when being interested in Configuration Management. It was clear from the beginning that this event will not focus on one technology only, but wants to be a place where all tools covering configuration management will collaboratively work together. This year, the main stage will start with several, well known people - and we are very happy to see them again: Mark Burgess (CFEngine) Luke Kanies (Puppet) Adam Jacob (Chef) Mike Place (Salt) Besides the founders or creators of configuration management tools, we will also hear from people directly involved with tools in the business. As usual the agenda is split into general sessions covering DevOps and Configuration Management and tool specific sessions covering Ansible, Chef, CFEngine, Foreman, Juju, Kubernetes, Mgmt Config, Puppet, Rudder, Salt, Terraform. We will be presenting PSICK at the Puppet room, if you want to have a direct reply on what you can do with it, check room B.1.015, Tuesday, February 6 at 15:40. On Wednesday fringe events will start, covering Foreman Constructoin day puppethack day Habitat and InSpec Hackday Mgmt Hackaton Container Trouble Shooting Workshop with SysDig CFEngine and IoT hackaton Ansible Lightbulb Workshop These are very tool specific sessions which run parallel. Choose wisely on where to head to. Happy System and Configuration Automation, Martin Alfke Alessandro Franceschi 2018-01-29T00:00:00+00:00 https://example42.com/blog/2018/01/29/remote_puppet_control Example42’s psick control-repo has several features which makes easier the life of the Puppet administrator. Since the release of Puppet Tasks several infrastructure commands have been added as tasks in the psick module: psick::system_update - Update all packages on a system psick::puppet_unlock - Remove Puppet lockfiles psick::puppet_install - Install Puppet agent on a node psick::puppet_enable_noop - Enable noop option in Puppet agent config psick::puppet_agent - Run Puppet agent on a node In this post we review the different ways we can use to remotely trigger a Puppet task and (in this case psick::puppet_agent but can be any task from any module): Use the bolt command (OSS) Use the puppet task command (PE) Run Puppet tasks from Puppet Enterprise Web console Directly interact with PE Orchestrator APIs The above methods allow execution of any Puppet task, if we “just” want to trigger Puppet agent execution on a remote node, we have some other, more or less classic, additional options: Run Puppet from PE Web Console for single nodes Use MCollective Use Ansible, Fabric or any other [unattended] remote execution tool Run Puppet Use PCP broker and a custom PXP agent module In the examples here we use a single target node, but can be defined use multiple nodes or the ones identified by a PQL query. Tasks via bolt command line (OSS) Bolt is the most direct tool we can use to remotely run Puppet tasks as it doesn’t need any agent installed on the target node. Syntax to run a given task (psick::puppet_agent) on a given node (git.lab.psick.io) accessed via ssh as specified user (bolt, but can any user on the remote node which has our local ssh public key added to its authorized_hosts): bolt task run psick::puppet_agent environment=host --modules /local/path/to/modules/ --user bolt remove -n git.lab.psick.io git.lab.psick.io: Info: Using configured environment 'host' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for git.lab.psick.io Info: Applying configuration version 'af0e48e - PE File Sync Service, Mon Jan 15 21:41:10 2018 +0100 : code-manager deploy signature: 'da3956ed288ad2573fc8cec722330e5618514525'' Notice: Applied catalog in 0.80 seconds Tasks via puppet task command (PE) If we have Puppet Enterprise we can use the puppet task subcommand. Which requires token based authentication and interacts directly with Puppet orchestrator APIs (on port 8143 on the PE Puppet Server): puppet task run psick::puppet_agent -n git.lab.psick.io Starting job ... New job ID: 395 Nodes: 1 Started on git.lab.psick.io ... Finished on node git.lab.psick.io STDOUT: Info: Using configured environment 'host' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for git.lab.psick.io Info: Applying configuration version 'af0e48e - PE File Sync Service, Mon Jan 15 21:41:10 2018 +0100 : code-manager deploy signature: 'da3956ed288ad2573fc8cec722330e5618514525'' Notice: Applied catalog in 0.73 seconds Job completed. 1/1 nodes succeeded. Duration: 11 sec Tasks via PE Web Console From Puppet Enterprise 2017.3.0 onwards it’s possible to select tasks from the modules available on the PE server. Click on the RUN - Task menu to access an easy to use web interface where to choose the tasks to run. Note that from the RUN - Puppet menu entry it’s possible to trigger a Puppet run on a emote node using the native method based on PXP. The list of Puppet Jobs executed (both tasks and puppet runs) are visible from the INSPECT - Jobs menu entry or via the command: puppet job show For more details check the official documentation Tasks via Orchestrator APIs Finally we can query directly the PEOrchestrator API, which as with the puppet job command, requires a token and proper RBAC permissions. The generated token has to be added to the http headers of our API calls. Tasks have a specific API endpoint: the command endpoint. Task parameters are added to the payload of the request. Affected nodes can be listed as array. The following example shows how to build the curl command call: curl -k -X POST \ -H "Content-Type: application/json" \ -H 'X-Authentication:<token>' \ https://<mom or compile master>:8143/orchestrator/v1/command/task \ -d '{ "environment" : "production", "task" : "psick::puppet_agent", "params" : { "noop" : true, "puppet_master" : "<compile master to use for this specific agent run>" }, "scope" : { "nodes" : ["<node1>", "<node2>"] } }' This will return the following Output: { "job" : { "id" : "https://<mom or compile master>:8143/orchestrator/v1/jobs/12", "name" : "12" } } Read result from job: curl -k -X GET \ -H 'X-Authentication:<token>' \ https://<mom or compile master>:8143/orchestrator/v1/jobs/12 End of output while still running: "node_count" : 1, "node_states" : { "running" : 1 } End of output when finished: "node_count" : 1, "node_states" : { "finished" : 1 } Read node results: curl -k -X GET \ -H 'X-Authentication:<token>' \ https://<mom or compile master>:8143/orchestrator/v1/jobs/12/nodes Output: { "items" : [ { "finish_timestamp" : "2018-01-24T14:41:14Z", "transaction_uuid" : null, "start_timestamp" : "2018-01-24T14:40:18Z", "name" : "<node1>", "duration" : 55.795, "state" : "finished", "details" : { }, "result" : { "_output" : "\u001B[0;32mInfo: Using configured environment 'production'\u001B[0m\n \u001B[0;32mInfo: Retrieving pluginfacts\u001B[0m\n \u001B[0;32mInfo: Retrieving plugin\u001B[0m\n \u001B[0;32mInfo: Loading facts\u001B[0m\n \u001B[0;32mInfo: Caching catalog for <node1>\u001B[0m\n \u001B[0;32mInfo: Applying configuration version '7a4be91 - Alessandro Franceschi, Sun Jan 14 18:15:44 2018 +0100 : run acceptance tests also for newer puppet versions (#220) (#221)'\u001B[0m\n \u001B[mNotice: /Stage[main]/Psick::Dns::Resolver/File[/etc/resolv.conf]/content: \n --- /etc/resolv.conf\t2018-01-24 14:31:51.428194104 +0000\n +++ /tmp/puppet-file20180124-20270-1wxros6\t2018-01-24 14:40:44.602327849 +0000\n @@ -1,3 +1,3 @@\n-# Generated by NetworkManager\n -search pe.psick.io\n-nameserver 10.0.2.3\n +#File managed by Puppet\n +nameserver 8.8.8.8\n +nameserver 8.8.4.4\n \u001B[0m\n \u001B[0;32mInfo: Computing checksum on file /etc/resolv.conf\u001B[0m\n \u001B[0;32mInfo: FileBucket got a duplicate file {md5}b9dfc6d9764870be83fe35ecf2cfc5f3\u001B[0m\n \u001B[0;32mInfo: /Stage[main]/Psick::Dns::Resolver/File[/etc/resolv.conf]: Filebucketed /etc/resolv.conf to puppet with sum b9dfc6d9764870be83fe35ecf2cfc5f3\u001B[0m\n \u001B[mNotice: /Stage[main]/Psick::Dns::Resolver/File[/etc/resolv.conf]/content: \n \u001B[0m\n\u001B[mNotice: /Stage[main]/Psick::Dns::Resolver/File[/etc/resolv.conf]/content: content changed '{md5}b9dfc6d9764870be83fe35ecf2cfc5f3' to '{md5}3ccdb679ea166bdf52104b3ae3a4499d'\u001B[0m\n \u001B[mNotice: Applied catalog in 29.16 seconds\u001B[0m\n" }, "latest-event-id" : 41, "timestamp" : "2018-01-24T14:41:14Z" } ], "next-events" : { "id" : "https://<node1>:8143/orchestrator/v1/jobs/14/events?start=42", "event" : "42" } } Remote Puppet agent run using other methods (not as tasks) All the above cases can be applied to any task, but if we just need to remotely trigger a Puppet agent execution, various other ways are available. We have already seen how it possible to trigger Puppet runs (not as tasks) from the PE Web Console, possible from the RUN - Puppet menu entry, or, for a specific node, directly from the details page (Run Puppet link). It has been possible for long to run Puppet (and do a lot of other actions) via MCollective. We can also use Ansible, Fabric, or other tools for orchestration or remote command execution. On PSICK for example, this is possible, if using Fabric, with: fab puppet.agent:host=git.lab.psick.io Finally it’s potentially possible to trigger Puppet runs (and potentially other commands) directly via the PCP Broker and eventually a custom PXP Module. ATTENTION: this solution is NOT working at the moment as the PCP broker is not yet opened and documented. First we need a pxp-agent module on the nodes. An example can be found in the pxp-agent repository and this is actually basically the only existing one (and the one currently used under the hood when remote Puppet run is triggered from the PE Console Web interface). For development purpose the module can be executed locally: sudo echo "{\"input\":{\"flags\":[\"--noop\", \"--server=puppet.pe.psick.io\"]}, \"configuration\" : {\"puppet_bin\" : \"/opt/puppetlabs/bin/puppet\"}}" | /etc/puppetlabs/pxp-agent/modules/puppet_agent run The following is not working, as the API is unknown. Could be something similar to the following: curl -k -X POST \ -H "Content-Type: application/json" \ -H 'X-Authentication:0MDuxrsDQnQbzGn1P_4sWu4hgzg8AvQk0sebRuReGiZI' \ https://puppet.pe.psick.io:8142/orchestrator/v1/command/task \ -d '{ "properties" : { "notify_outcome" : true, "module" : "puppet_agent", "action" : "run", "params" : { "flags" : [ "--noop" ] } }, "required" : ["transaction_id", "notify_outcome", module", "action"], "additionalProperties" : false }' We have a ticket at Puppet open for opening PCP broker API. Conclusion As we have seen there are multiple ways to remotely trigger a Puppet agent execution from a central node and there are various ways to trigger a Puppet task to achieve the same result. The ability to orchestrate Puppet execution can affect our choices on how we decide to manage how infrastructure via Puppet, allowing many combination of options, such as: Normal Puppet agent running every [30] minutes + On request remotely triggered Puppet runs Normal Puppet agents 30 minutes in noop mode + On request no-noop Puppet runs Puppet runs triggered on Canary nodes during CI Puppet runs only on request, without always running agents Staged rollouts of configurations via remotely orchestrated Puppet runs. Note that in most of the above examples we can see the result of the Puppet run only after it has ended and not in real-time. Happy Puppet orchestration, with or without PSICK, Martin Alfke Alessandro Franceschi 2018-01-22T00:00:00+00:00 https://example42.com/blog/2018/01/22/acceptance_testing_of_different_puppet_versions_with_psick Example42’s psick control-repo has several features which allows user to manage most of the typical infrastructure tasks easily. PSICK has a set of unit and acceptance tests already built in. Usually we want to test the Puppet version we have running in infrastructure and we want to test functionality with newer versions. Unit Testing Handling different Puppet versions on unit testing is done by reading an environment variable from Gemfile. rspec puppet uses the Puppet version installed by bundler to compile and check catalogs. Running unit tests for different Puppet versions is easy: export PUPPET_GEM_VERSION='4.10.9' bundle install --path vendor bundle exec rake spec Now let’s test Puppet 5: export PUPPET_GEM_VERSION='5.3.3' bundle update puppet bundle exec rake spec Unsetting the PUPPET_GEM_VERSION environment variable will use the latest Puppet version from rubygems. Acceptance Testing Acceptance testing is different. At PSICK we offer the possibility to use Docker containers or Vagrant instances for acceptance tests run by beaker. In both solutions the Puppet agent must be installed inside the instance. Usually we want to use the version we have running in our infrastructure. The beaker-puppet helper offers a new way on specifying Puppet versions to install. At PSICK we re-use the pattern from unit testing by specifying the PUPPET_GEM_VERSION environment variable: export PUPPET_GEM_VERSION='4.10.8' bundle install --path vendor bundle exec rake beaker_roles:psick Now we want to test latest Puppet 5 version: export PUPPET_GEM_VERSION='~> 5' bundle update puppet bundle exec rake beaker_roles:psick The magic is in PSICK spec/spec_helper_acceptance.rb where we check for the Puppet version installed by bundler and pass proper attributes to beaker’s install_puppet_agent_on method. Happy hacking, Martin Alfke 2018-01-15T00:00:00+00:00 https://example42.com/blog/2018/01/15/classification_and_first_run_mode_with_psick Example42’s psick module has several features which allows users to manage most of the typical infrastructure tasks with a single module. One of them is classification: we can use it to define which classes we want on each node via Hiera data. The module provides different parameters to manage in which phase of a Puppet run we want to include classes for different major families of operating systems (Linux, Windows, Solaris, Darwin…). Classification via psick module Psick has a different subclass for each phase: pre, in this phase prerequisites classes are included, they are applied before all the other ones. base, classes common to all the nodes (but exceptions can be applied via Hiera), applied in normal catalog runs after the pre classes and before the profiles. profiles, exactly as in the roles and profiles pattern. The profile classes that differentiate nodes by their role or function. Profiles are applied after the base classes are managed. Installation of the psick module from the Forge can be done with: puppet module install example42/PSICK In order to be able to access Psick’s features you just have to add the psick class to your catalog, this can be done, at top scope for each node, in the main manifest (manifests/site.pp): include psick This does nothing by default, every psick configuration is data driven. The classes to include in each phase can be managed via Hiera, for different OS, as follows: # Classes to include in pre, base and profiles phases on Linux # (different distros are expected to be managed in the defined classes) psick::pre::linux_classes: puppet: ::puppet dns: psick::dns::resolver hostname: psick::hostname hosts: psick::hosts::resource repo: psick::repo psick::base::linux_classes: sudo: psick::sudo time: psick::time sysctl: psick::sysctl update: psick::update ssh: psick::openssh::tp mail: psick::postfix::tp psick::profiles::linux_classes: webserver: apache # Classes to include in pre, base and profiles phases on Windows psick::pre::windows_classes: hosts: psick::hosts::resource psick::base::windows_classes: features: psick::windows::features registry: psick::windows::registry services: psick::windows::services time: psick::time users: psick::users::ad psick::profiles::windows_classes: webserver: iis Each key-pair of these ${kernel}_classes parameters contain an arbitrary tag or marker (users, time, services, but could be any string), and the name of the class to include. This name must be a valid class, which can be found in the Puppet Master modulepath (so probably defined in your control-repo Puppetfile): you can use any of the existing Psick profiles, or your own local site profiles, or directly classes from public modules and configure them via Hiera in their own namespace. To manage exceptions and use a different classes on different nodes you only have to specify the alternative class name as value for the used marker (here ‘ssh’), in the appropriate Hiera file: psick::base::linux_classes: ssh: ::profile::ssh_bastion To completely disable on specific nodes the usage of a class, included in a general hierarchy level, set the class name to an empty string: psick::base::linux_classes: ssh: '' The pre -> base -> profiles order is strictly enforced, so be sure to place your class in the most appropriate phase (even if functionally they all do the same work: include the specified classes) and, to prevent dependency cycles, avoid to set the same class in two different phases. First run phase A special phase, disabled by default, is applied only at the very first time Puppet is executed. Its purpose is to give users the possibility to make configurations on a node via Puppet before actually making a full Puppet run. Optionally, a reboot may be triggered at the end of this first Puppet run. The next Puppet executions will use the normal configurations expected in each nodes. Possible use cases for Firstrun mode: Set a desired hostname on Windows, reboot and join an AD domain Install aws-sdk gem, reboot and have ec2_tags facts since the first real Puppet run Set external facts with configurable content (not via pluginsync) and run a catalog only when they are loaded (after the first run) Any case where a configuration or some installations have to be done in a separated and never repeating first run. With or without a system reboot. To enable first run mode set: psick::enable_firstrun: true To define which classes to include in nodes, according to each $::kernel, we have an approach similar to the one used for the pre, base and profiles phases: psick::firstrun::windows_classes: hostname: psick::hostname aws_sdk: psick::aws::sdk psick::firstrun::linux_classes: hostname: psick::hostname proxy: psick::proxy aws_sdk: psick::aws::sdk To manage if we want to trigger a system reboot after the first run: psick::firstrun::windows_reboot: true # (Default value) psick::firstrun::linux_reboot: false # (Default value) IMPORTANT NOTE: If firstrun mode is activated on an existing infrastructure or if the ‘firstrun’ external fact is removed from nodes (we use this fact, configured the the same psick::firstrun class, to determine if Puppet has already been executed or not), this class will be included in the main psick class as if this were a real first Puppet run. This will trigger a, probably unwanted, reboot on Windows nodes (and in any other node for which reboot is configured. Set psick::firstrun::${kernel}_reboot to false to prevent undesired reboots. Conclusion Phased, hiera data driven, classification and first run mode is just one of the features of the psick module, among the others (a rich set of profiles for common system settings and applications, a standardised set of tp profiles, some common use cases defines…). The good news is that you can decide which of such features to use and you can integrate psick in existing infrastructures where traditional classification techniques are used. The bad news is that psick concept is so unusual that you might get some time to figure out its whole logic. Perseverance, here as everywhere, is key to success :-) Alessandro Franceschi Martin Alfke 2018-01-08T00:00:00+00:00 https://example42.com/blog/2018/01/08/puppet_seminar_and_workshop Einladung: DevOps und Automatisierung Workshop “IT is too important to leave to IT” (Mark Smalley) Wie setzt man DevOps um? Wo bekommt man DevOps her? Wenn Deutsche Bank, Volkswagen oder Deutsche Telekom erwähnt werden, denkt man nicht sofort an IT… Allen diesen Firmen ist gemeinsam, dass IT und Softwarebereitstellung einen kritischen Erfolgsfaktor bei der Erreichung und Bindung von Kunden spielen. Die dazugehörigen IT Infrastrukturen müssen effizient und zeitnah bereitgestellt und verwaltet werden. Kostentransparenz und -effizienz sind hierbei wichtige Parameter für das Business. Das DevOps Konzept hat sich hierbei als eine Methodik etabliert, mit der man einen Informationsaustausch zwischen IT und Business in Hinsicht auf Zeit (Time to Market) und Monetarisierung (Return on Investment) erreicht. Puppet unterstützt DevOps Implementierungen durch Automatisierung in der IT. Diese führt zu mehr Transparenz, besserer (aktueller) Dokumentation, einfacheren und schnelleren Zertifizierungen von IT Plattformen. Aber wie setzt man DevOps um? Wo bekommt man DevOps her? Wir laden Sie zu einem eintägigen Event am 25. Januar 2018 in Frankfurt ein, bei dem wir vorstellen, wie Unternehmen den DevOps Gedanken in ihre Teams bringen können und wie IT-Automatisierung mit Hilfe von Puppet Enterprise umgesetzt werden kann. Am Vormittag liegt der Schwerpunkt auf den eben beschriebenen Themen. Am Nachmittag werden praxisbezogene Lösungsszenarien für DevOps und CI/CD vorgestellt. Beide Teile des Events können jeweils separat, aber selbstverständlich auch als Gesamtüberblick besucht werden. Hier das Formular für die Anmeldung. Sie möchten Ihr Team gezielt schulen? Unser Puppet Training passt sich an Ihre Infrastruktur und Ihr Team an. Martin Alfke Alessandro Franceschi 2018-01-01T00:00:00+00:00 https://example42.com/blog/2018/01/01/2017_categorized_posting_summary Welcome to 2018! We start this year with a categorized summary of all postings from 2017. PSICK & Control-Repository One liner to install Puppet Anatomy of a Puppet control-repo A PSICK Vagrant experience Automated Puppet Infrastrucuture setup Self contained nodes classification Control Repo documentation Puppet node classification options Puppet and Fabric PSICK explained GIT workflow for Puppet control-repositzories Testing any role on any OS with PSICK control-repo Tenant modules Introducing PSICK - The Infrastructure Puppet module Bolt and tasks with PSICK Puppet Code Development and Testing Existing code and Puppet 4 RSpec Testing on existing Puppet code Roles, profiles and tools Puppet Continuous Integration with GitLab Puppet Debugger The PSICK Developer Environment Setup Spec Testing a Puppet control-repository Puppet class indirection via Hiera Testing a control-repo with Vagrant Puppet code development IDE Acceptance testing on a control-repository using beaker with vagrant and docker Hiera Hiera 5 Hiera 5 globs and mapped paths The optimal hiera.yaml Encrypt your secrets with Hiera eyaml Automated, reusable hiera eyaml setup When to place data in hiera Secure data management with multiple eyaml keys Puppet Best Practice Resources from Hash Data in Puppet 4 Using Exec resource the right way Using resource reference the right way Server side noop mode Using a second mount point for files Environment caches Environment enforcement Puppet reports and metrics Puppet 5 Checking config files before applying them How to DevOps Puppet modules documentation Scheduling Puppet resources application Building your own Puppet 4 Data Types Puppet noop, no-noop and the path to safe Puppet deployments Bolt, tasks, plans, LibRAL LibRAL and a proposal for the new Puppet Resource API Bolt and tasks with PSICK Puppet plans and tasks example42 wishes everybody a healthy and successful new year. Martin Alfke Alessandro Franceschi 2017-12-27T00:00:00+00:00 https://example42.com/blog/2017/12/27/noop-no-noop-and-the-path-to-safe-puppet-deployments We have already talked in this blog, about server side noop mode in the past, now we come back to the topic to expose how we currently prefer to use noop mode, and how to enforce no-noop mode on specific classes. Puppet noop mode mode allows us to review the changes that Puppet would do on the system without actually applying them. This is particularly useful when managing critical servers, as it allows to push to production Puppet code and data in a more controlled, safe and manageable way. There are various ways we can enforce noop mode when using this control repo: let’s review them. Setting noop from the client In any Puppet installation it’s possible to run Puppet in noop mode specifying the --noop option in the command line: puppet agent -t --noop This applies only for that specific Puppet run, so if there’s a Puppet agent service running in the background, that service will run Puppet in normal mode. The noop setting can be configured and made persistent also in puppet.conf: [agent] noop = true If you use the PSICK module you can configure it on Puppet Enterprise clients as follows: # We want to manage noop mode on clients: psick::puppet::pe_agent::manage_noop: true # We want to set noop to true psick::puppet::pe_agent::noop_mode: true # We enforce no-noop mode on the pe_agent class to be # able to revert our noop settings. Details in no-noop section. psick::puppet::pe_agent::no_noop: true Note that this is the common and official approach to manage noop mode and is controlled and managed from the client, not on the server. Setting noop server-side In the PSICK control-repo we use the trlinkin-noop module which provides a function called noop() which adds the noop metaparameter to each resource. We use this function in manifests/site.pp: $noop_mode = lookup('noop_mode', Boolean, 'first', false) if $noop_mode == true { noop() } This code sets the noop_mode variable via a Hiera lookup for the key noop_mode. If it’s not found on Hiera, then the default value is false. If the noop_mode variable is true then noop metaparameter is added to all the resources of the catalog. It’s recommended to limit the usage of noop_mode key on Hiera only when necessary, for example when a massive or invasive code change has to be promoted to production and we want a safe net where we can selectively remove noop_mode to control the propagation of change. For example when pushing to production particularly critical changes it’s possible to force noop mode for all the servers adding in hieradata/common.yaml: noop_mode: true To test the actual changes we could override this in a Hiera yaml (something like hieradata/env/test.yaml to give you an idea) which configures a subset of our servers: noop_mode: false Note that this is not the typical way to manage noop mode in Puppet and when using the Puppet Enterprise console you will see nodes where noop is applied with this approach undef the “Nodes run in enforcement” group (and not in noop mode) in the dashboard. Still, checking report you will be able to notice that actually no resource is really applied and you should see eventual changes in the noop column. Final result is the same (no resources are really applied) but they are shown differently on PE console. Important caveat with server-side noop mode There’s a small but rather important thing to consider when using the noop() function: since it works by adding the noop metaparameter to a resource, and can override a normal puppet run without noop set client-side, it can have unpredictable effects when you are working with exported resources which, when missing, alter the configuration of the node which collects them. Let’s see an example: You run puppet in normal mode on a node which exports a concat resource used by a load balancer. When server side noop mode is used , this concat resource is exported with noop = true When you run puppet on the load balancer which collects the concat resources previously exported (with noop set server side), Puppet will rebuild the target configuration without the concat fragment where noop is true. This doesn’t happen when you run Puppet with noop mode client side- How to avoid such situations? Identify the cases where missing exported resources can actually change some configurations (rather than just not managing for a single Puppet run a resource previously configured). This basically happens whenever you export concat fragments and where you export files in directories which are completely managed and where files not explicitly managed by Puppet are purged. In the above cases, be sure to have a normal real Puppet run on the exporting nodes before doing a Puppet run in the collecting ones. Enforcing no-noop mode In some cases we might need to enforce the applications of the resources of some classes in every case, whatever is the noop mode. Some of the profiles used in the PSICK module have the no_noop parameter: if set to true all the resources of the class are enforced and are applied whatever are the noop settings (either client or server side). By default no_noop is set to false and nothing changes in terms of noop management. This allows us to have some server where Puppet runs in noop mode but have still some resources always applied. In order to set no_noop mode for a class, use hiera data like: psick::dns::resolver::no_noop: true psick::hostname::no_noop: true psick::hosts::file::no_noop: true psick::puppet::gems::no_noop: true psick::puppet::pe_agent::no_noop: true # This is required to be able to change the noop setting client side In case the no_noop parameter is not present in a profile, it’s quite easy to add it: class my_class ( [...] Boolean $no_noop = false, ) { if $no_noop { info('Forced no-noop mode.') noop(false) } [...] } Puppet code deployment and application workflows Enabling noop mode on some clients, the most important ones, or the whole production ones, allows us to implement some sophisticated and safe workflows for the testing and the deployment of the Puppet code and data that manage our nodes. Some basic principles have to be considered in order to design them in the most effective way: Server side noop mode if set to true, overrides any client setting Setting a class no_noop parameter to true overrides any noop setting either client or server side We can manage via Hiera both server and client settings, giving us full flexibility on where to set it, still we should limit as much as possible the places where we configure it, and possibly, to avoid unnecessary confusion, not use, on regular basis, both server and client settings at the same time (exceptions below). Client settings are effective after the Puppet run that sets them. Server side settings are immediately effective. Always consider that classes with no_noop set to true are always applied, if you make changes to them consider the possibility to set no_noop temporarily to false, before propagating such changes everywhere. The following approach is recommended when noop mode is used or desired: Set noop mode client side on the nodes where we want it (all production nodes or particularly critical ones) Use server side noop mode only when deploying big or potentially dangerous code/data changes, keep it undefined in normal conditions Have a CI pipeline which triggers Puppet runs on canary nodes, also in production, enforcing one-shot no-noop runs In the CI pipeline trigger noop Puppet runs on the other production nodes and verify the result Do not accumulate too many changes on noop nodes: run Puppet in no-noop mode on production servers as soon as possible (eventually do that in maintenance windows if you are particularly prudent). To trigger real no-noop Puppet runs and apply changes on nodes normally running in noop mode different approaches can be used: At the end of the CI pipeline if everything is OK trigger (manually or automatically) a no-noop Puppet run on nodes normally in noop mode Actual execution of a no-noop run can be done via a Puppet task like psick::puppet_agent (it has a parameter for forcing it) or any other tool that can remotely execute puppet agent -t --no-noop on a node. Via PE-Console, in the Run Puppet section, manage manually the remote execution of a no-noop Puppet run by clicking on the “Override noop = true configuration”. Remember that in both these last two cases, if noop is set server side Puppet keeps on skipping changes on the managed node, that’s why we suggest to use server side noop mode only to add a safe net when deploying massive, critical or potentially dangerous code and data changes. In normal operations is probably better to use client side noop mode that can be more easily overridden. Alessandro Franceschi 2017-12-18T00:00:00+00:00 https://example42.com/blog/2017/12/18/beaker_with_vagrant_and_docker Beaker has become the de-facto standard utility for Puppet code acceptance testing. Like the figure from The Muppet Show the Puppet tooling “…is a magnet for disaster”. In this specific case we want to find out whether a catalog can be applied to a system successfully. In addition to that the Muppet figure has even more qualification for breaking things: “he routinely experiences mishaps such as being blown up, electrocuted, eaten by large monsters, or afflicted with awkward side effects caused by Dr. Bunsen Honeydew’s experiments. Beaker communicates in a nervous, high-pitched squeak that sounds like “Mee-mee-mee mee”” (quote taken from Wikipedia Beaker Muppet - see link above). I also like the name for the tool, as we sometimes re-use the “Mee-mee-mee” spelling to a person telling you that something is not working. Beaker installation Puppet beaker allows for unattended CI based testing and acts as a frontend for Vagrant, AWS, Google Compute, VMware/Vsphere, Docker and others hypervisors and a puppet apply command. Please see the beaker documentation on hypervisors for additional information on hypervisors. Within this posting I want to explain how we at example42 use beaker for acceptance testing in control-repositories like PSICK. Installing beaker is usually done by adding a few lines to your Gemfile: # Beaker group :acceptance do gem 'beaker' gem 'beaker-rspec' gem 'beaker-hiera' gem 'beaker-puppet_install_helper' end All standard beaker hypervisor extensions are a dependency to the beaker gem. Please note that beaker uses the fog libraries for cloud access which causes a long list of dependencies. Beaker nodesets Next we need to add node descriptions providing information on operating system and releases to run tests on. These are placed into the spec/acceptance/nodesets/ folder. Here we usually see the first difference between acceptance tests on control-repositories versus acceptance testing on modules: Within modules one wants to test, whether the single code base is working as expected on all supported operating systems. For a control-repository one wants to test multiple different system roles on a few operating systems. Let’s continue testing control-repositories on CentOS 7. The nodeset configuration consists of the following two files: First we generate the nodeset for testing on docker: # spec/acceptance/nodesets/default.yml HOSTS: centos7-box: platform: el-7-x86_64 hypervisor: docker docker_cmd: '["/usr/sbin/init"]' docker_image_commands: - 'yum install -y crontabs initscripts iproute openssl sysvinit-tools tar wget which ss' - 'systemctl mask getty@tty1.service' docker_preserve_image: true mount_folders: controlrepo: host_path: . container_path: /tmp/production opts: ro CONFIG: type: foss Now we add the nodeset for testing on vagrant: # spec/acceptance/nodesets/vagrant.yml HOSTS: centos-7-x64: roles: - agent platform: el-7-x86_64 box: centos/7 hypervisor: vagrant mount_folders: controlrepo: host_path: ../../../ container_path: /tmp/production opts: ro CONFIG: type: foss Beaker running instance preparation The images we are using do not have puppet agent package installed. Installation and preparation of the started images is configured in the spec/spec_helper_acceptance.rb file: # spec/spec_helper_acceptance.rb require 'beaker-rspec' require 'beaker/puppet_install_helper' # Instal PC1 puppet 4 agent packages run_puppet_install_helper RSpec.configure do |c| # Readable test descriptions c.formatter = :documentation # preare each system after starting the image: hosts.each do |host| # remove obsolete global hiera.yaml on(host, '/usr/bin/test -f /etc/puppetlabs/puppet/hiera.yaml && /bin/rm -f /etc/puppetlabs/puppet/hiera.yaml || echo true') # remove existing production environment on(host, '/usr/bin/test -d /etc/puppetlabs/code/environments/production && /bin/rm -fr /etc/puppetlabs/code/environments/production || echo true') # re-create production environment directory on(host, '/usr/bin/test ! -d /etc/puppetlabs/code/environments/production && mkdir -p /etc/puppetlabs/code/environments/production || echo true') # copy control-repo on(host, 'cp -r /tmp/production/{.git,environment.conf,hiera.yaml,hieradata,manifests,site,modules} /etc/puppetlabs/code/environments/production/') end end Beaker serverspec tests Next one can add serverspec tests for specific roles: # spec/acceptance/puppetmaster_spec.rb require 'spec_helper_acceptance' describe 'puppetmaster' do let(:manifest) { <<-EOS include role::puppetmaster EOS } it 'should run first time with changes and without errors' do result = apply_manifest(manifest, :catch_failures => true) expect(@result.exit_code).to eq 2 end it 'should run a second time without changes' do result = apply_manifest(manifest, :catch_changes => true) expect(@result.exit_code).to eq 0 end # here one can add more serverspec tests end Now we need to changes the default beaker acceptance testing rake task as we want to run tests on a control-repository. # Rakefile # beaker is designed to run all tests on multiple nodes # we have another use case: run single tests on one host, then next test on new fresh host Rake::Task[:beaker].clear RSpec::Core::RakeTask.new(:beaker) do |config| puts 'dont use beaker, use beaker_roles:<role> or all_roles instead' abort end # iterate over acceptance tests and create namespaced rake tasks namespace :beaker_roles do # find all acceptance tests Dir.glob("spec/acceptance/*_spec.rb") do |acceptance_test| # find role part of file name test_name = acceptance_test.split('/').last.split('_spec').first RSpec::Core::RakeTask.new(test_name) do |t| t.rspec_opts = ['--color'] t.pattern = acceptance_test end end end Running beaker and acceptance tests Now we can run the acceptance tests: bundle exec rake beaker_roles:puppetmaster This command only uses the default.yml nodeset. If we want to use the vagrant.yml nodeset we must tell beaker to do so by providing an environment variable: BEAKER_set=vagrant bundle exec rake beaker_roles:puppetmaster In addition to BEAKER_set there are some other useful environment variables: Usually beaker deletes the container/virtual machine after running tests (even after failed tests). To keep the instance alive just add BEAKER_destroy=no environment variable. If one needs more information on what is happening during beaker running, it is possible to enable debug mode by specifying BEAKER_debug=true Happy testing on your control-repository. example42 wishes everybody a happy Christmas. Martin Alfke 2017-12-11T00:00:00+00:00 https://example42.com/blog/2017/12/11/scheduling_puppet_resources_application In our Puppet operations we may need to apply some resources only in given periods of time, such as maintenance windows, os specific days of the week of hour. Puppet has a quite useful, and not too much known, resource for that: schedule. The schedule resource type can be used to define a time period. Once defined, by using the schedule metaparamter we can tell to Puppet to apply a given resource only when Puppet is running within the specified schedule. For example, we can define a daily schedule from 2 AM to 4 AM, and also specify how many times a resource using such schedule should be applied via the repeat parameter, as follows: schedule { 'maintenance': range => '2 - 4', period => daily, repeat => 1, } and then apply to to any resource, via the schedule metaparameter: exec { '/usr/local/bin/daily_maintenance': schedule => 'maintenance', } Note that such piece of code doesn’t guarantee that the command is executed every day, and it doesn’t ensure that this happens at a fixed time. What will happen is that IF Puppet runs on a node, between 2 and 4 AM, THEN the maintenance exec is applied. As every other resource type, we can declare as many instances of it as we want, they just need to have different titles. The full list of attributes is: schedule { 'title': name => # (namevar) The name of the schedule. The title is used if not set the name. period => # The period of repetition for resources on this schedule, valid values: # hourly, daily, weekly, monthly, never. periodmatch => # Whether periods should be matched by: # - 'number' (IE: two times in an hour) or by # - 'distance' (IE: two times which are 60 minutes apart) range => # The hour (from 0 to 23) range when to apply a schedule. # Midnight might be crossed with something like: 22:00 - 02:00 repeat => # How often a given resource may be applied in the schedule weekday => # The days of the week in which the schedule occurs } If we need to always run a specific command we probably better use cron on any similar system’s scheduler, as in this case we have more control on when and if commands are executed. There are still use cases for Puppet’s resource type, for example we can use it to: Manage different contents of configurations according to the hour or period Trigger system reboots if conditions apply Manage backups or other maintenance operations (given the described limitations) Manage resources which take a long time to be applied and don’t need be to continuously enforced Do crazy things nobody has conceived before Alessandro Franceschi 2017-12-04T00:00:00+00:00 https://example42.com/blog/2017/12/04/libral_puppet_resource_api Puppet has always run on fast development cycles. Even though only a very few releases required people to refactor their existing Puppet code (e.g. the migration from Puppet 3 to Puppet 4). Puppet still has lots of upcoming changes and still follows the Open Source idea. All discussion on upcoming development or new features take place in the Puppet Specification repository. At the moment there is at least one PR which deals with idea of a new Puppet Resource API, extending the way how we usually deal with and develop custom types and providers. Besides this we see more mature ideas made available as new git repositories like libral. This posting will dig into libral and the new proposed Puppet resource API. LibRAL LibRAL is described as “a systems management library that makes it possible to query and modify system resources […] through a desired-state API”. The aim is to not only query and modify built in resources like files, packages and services, but to also add a new API for managing new kind of resources. It follows Puppet concept of idempotency, only changing a resource when desired state does not match actual state and keep the resource untouched if it is in sync. Besides this, LibRAL was developed for low performance systems like devices or containers. Puppet Resource API The resource API is at a very early state. There is not even a finalised document describing the new API. At the moment this is more a discussion base and Puppet developers are eager to get feedback on the new Resource API. An interesting way for new resource API can be found within the PR at the simple_apt.rb example. Here David uses a SimpleResource concept which is a generic concept for any new resource type (see line 98 and following in Module::SimpleResource. We are looking forward to seeing more examples and ways on how to make use of a new way to develop Puppet types and providers using the new API. Feel free to comment within the PR or contact the developers via Slack or IRC. Martin Alfke 2017-11-27T00:00:00+00:00 https://example42.com/blog/2017/11/27/puppet_modules_documentation Modules’ documentation is useful to understand what a module does and how it does it and how to make use of it. Usually we can access these information by reading the README, directly on the Forge or GitHub page, and eventually looking directly at the inline documentation in manifests. Puppet Strings As you are probably aware there is a tool, Puppet strings which is able to automatically generate documentation for a module based on in code documentation. It’s based on the Yard tool and can generate docs in various formats. To install Puppet strings: puppet resource package rgen provider=puppet_gem puppet resource package puppet-strings provider=puppet_gem To generate the documentation for a module, move inside the module main directory and run: /opt/puppetlabs/puppet/bin/puppet strings generate **/*{.pp\,.rb} **/**/*{.pp\,.rb} This will parse all the .rb and .pp files in the module and generate html documentation under the doc directory of the module. Puppet documentation server Instead of running the puppet strings command on every change in every module, it is also possible to have the strings server checking for changes and rendering documentation upon file changes. Just change to your control repository, install all modules from Puppetfile by running r10k puppetfile install and run the strings server: /opt/puppetlabs/puppet/bin/puppet strings server --modulepath=./modules:./site This will spin up a webservice which is accessible on port 8808: http://localhost:8808 We recommend to not have this web server running on the puppet master. Spin up a new server which will get code updates via git hooks or CI pipelines. puppetmodule.info As we know now, Yard can act as a server and show directly the html pages generated. There’s a web site which relies on this and show Puppet strings based documentation for most of the Puppet public modules on the Forge and GitLab. Give a visit to www.puppetmodule.info, site created by Dominic Cleal from The Foreman team. Here you can see the documentation for virtually any module you will find yourself using, the site is able to generate on request the documentation for modules it hasn’t yet processed. So on this site you can look how documented modules appear and search modules and contents as needed. Control repo documentation In Psick we use Puppet strings also to generate the documentation of the whole control-repo. This is Psick’s puppet strings generated documentation (it includes README with merged texts from the psick control-repo and the classes and defines from the psick module). It is automatically generated during the CI pipeline we run on GitLab, relevant lines are here. As they say: it’s not ready until it’s documented. How to use puppet strings in your puppet code? Every class and define must start with the documentation prior class or define definition. Documentation is marked as comments using the hash character (#). # The demo setup class # # This is an example of using documentation in a class or define # # @summary this is rendered as summary of the class or define # # @example Show how people can make use of the class # include demo_setup # # @param prod [Boolean] This parameter describes the stage or maturity level # of the application. This text is longer, so we use newline for # readability # @param port [Integer] Port on which the demo_setup must run # class demo_setup ( Boolean $prod = true, Integer $port = 1025, ){ # Puppet code } Strings can also render documentation from types and providers: Puppet::Type.newtype(:demo_setup) do desc <<-DESC The type for the demo_setup @example Show usage for the type demo_setup { 'application': prod => true, port => 8880, } DESC newparam(:prod) do desc 'Stage to run in' # ... end end Alessandro Franceschi Martin Alfke 2017-11-20T00:00:00+00:00 https://example42.com/blog/2017/11/20/how_devops_changes_infrastructure How DevOps started In the past times system administrators mostly managed their infrastructure in a manual pattern: provisioning systems using CDROMs or USB sticks or - if you were already lucky to have larger infrastructure - you might have used a provisioning system like FAI or Cobbler. Afterwards people logged into the system using SSH and configured them according to some documentation or tickets or they had some shell or perl scripts which did the initial configuration. When developers changed from waterfall to agile methods, system administrators were facing new issues: they needed to build systems faster as developers were pushing out code faster. That was the point where DevOps started. Ops people learned that communication and collaboration with developers gave them insight in upcoming development. As a nice side effect they were able to gain knowledge on new systems already during development phase. They learned about developers’ methods like version control system and adopted them to their needs. On the other hand developers got an understanding of system engineers needs and frustrations. This collaboration of development and system engineering is since then called DevOps. But what does DevOps do? How does it work? What else did it change? What is DevOps? Some companies still try to sell their ‘DevOps products’. You can find evidence especially at enterprise level as one can see at the landing pages from IBM or CA. Others have understood that there are no DevOps products like the Atlassian description of DevOps. There already is a description on what DevOps is since 2010 when Daemon Edwards and John Willis gave a talk at DevOpsDays Mountain View. They describe DevOps using the CAMS acronym. These are: Culture Automation Measurement Sharing Another summary of terms would be: People Processes Platforms Participation HowTo DevOps? Automation and Measurement are the parts which you can easily make available to you and your teams by searching for products. For Configuration Management we can choose between tools like Puppet, Chef or Ansible. All mentioned tools are aimed at teams managing IT infrastructures like servers, routers, storages and mostly use a declarative system description. When it comes to metrics, we must first analyse which are the important informations. Here we see different needs for different departments. IT Ops wants to see whether server usage is within normal limits, Product Owners want to see customer process and Management wants PKI information. When collaborating with the application developers it is easy to also fetch application specific data into a central metrics collection system like Elasticsearch (formerly known as the ELK stack - Elastic, Logstash, Kibana) from Elastic or Prometheus. But how to implement ‘Culture’ and ‘Sharing’? This is not something which you can just shop and ‘buy’ as this is people and not products or processes. The most important task is to tear down any barrier which exists between developers and operations. Usually we see that these two departments are part of different C-Level stakeholders, especially at large companies: Development usually is located within Product, whereas Operation is handled within Technology. You will always fight battles between the CPO (Chief Product Officer) and the CTO (Chief Technology Officer). Both have different goals: the CPO wants new features and products to be available for customers as soon as possible, whereas the CTO sees a desire for stable platforms. In this case the whole DevOps approach must be fully supported by the whole C-Level team. The next barrier which shows up are mid-level managers. Usually these are former technology people (either from development or operations) which have been given a team or even department lead. In this position they are responsible to C-Level management for performance and results. Now these people have to adopt to a new role as team coach or leader. Their new main responsibility will be to remove any kind of issue their team has, they have to coach their staff to learn new technologies. This will take some time, as they first have to gain knowledge on how to fullfill the new role. During these phases, C-Level management gets an extra task. They have to prove to their technical team members that they fully support the company and culture change. This can be achieved by finding or getting the right people and support them directly while your mid management is on training courses. Make the technicians a team which is directly under your control. If you haven’t talked directly to your employees for a while you might want to change that now. Ensure that you have smart people. Don’t only listen to the loudest. Listen also to the silent ones. If you want to hear both, let them make discussions based on Fishbowl Conversation. Everybody in your team is your favourite player. Only working together as a team will bring you success in implementing DevOps. We don’t need DevOps? In the past few years we have seen plenty of tools and terms coming up within the DevOps area. The one with the most momentum is Docker. Docker allows you to easily build systems running as Containers but on multiple architectures and operating systems. Some people even say that when using Docker there is no need to an IT operations team. An idea, which I don’t share: who is responsible (technical experienced) contact to your Container Platform supplier? who is able to understand a decent way to manage containers and get important data from them? who will define and verify whether your container is created as a secure system? Another term which is used often nowadays is Serverless which describes a way to run systems in cloud architecture. But serverless does not mean that you run a platform without operations. It is just a wording saying that you run your platform on other company computers. Even when using any kind of cloud you will have the need to manage (operate) your cloud access and configuration. Both still require to have people with good knowledge on network, storage and application management around. The only difference is that we just don’t have hardware in our own datacenter. The need for operations and therefor for collaboration between developers and system administrators is still there. So is the DevOps idea. The future of DevOps As we are mostly within different customers, we see different stages of DevOps implementation. Some companies are already deep into DevOps whereas others are starting or evaluating DevOps principles. Customers who do DevOps for quite some time already found, that they should not limit DevOps to developers and operation. They start to adopt the principles and ideas also to other departments, mostly loosely coupled to technology. Within the past months several other implementations of DevOps already came up: SecOps - Collaboration of IT Security, Operations and Development NetDev - Collaboration and automation between network engineers and other departments BizOps - Shared responsibility between product management or other business units and other teams Even though that these re-use the DevOps naming conventions these enablements are not just a 1-1 correlation. The best term describing what we see as the future of DevOps might be something like: SecNetDevBizOps. That’s why is probably better to keep on calling all this just DevOps :-) Martin Alfke 2017-11-13T00:00:00+00:00 https://example42.com/blog/2017/11/13/checking-config-files-before-applying-them Lately I’ve rediscovered an option of Puppet’s file resource that can be incredibly useful and, surprisingly, doesn’t seem to be widely used. Since Puppet 4.6 the file resource type has the validate_cmd attribute, which allows to run a command to check the syntax of a file before actually applying it to the system. For example we can write: file { '/etc/nginx/nginx.conf': content => 'This can be wrong', validate_cmd => 'nginx -t -c %', } to make Puppet run the specified validate_cmd on the file we are trying to provide, which is referenced by the % sign (can be configured with the validate_replacement attribute). Needless to say that this is extremely useful to prevent accidental failures in managed systems due to incorrect syntax (or incomplete values, due to missing variables) in the provided configuration files for the managed applications. This is what happens in a Puppet run when we try to provide a file with a wrong syntax for which is defined the validate_cmd attribute: Notice: /Stage[main]/Psick::Nginx::Tp/Tp::Conf[nginx]/File[/etc/nginx/nginx.conf]/content: --- /etc/nginx/nginx.conf 2017-02-11 21:00:57.000000000 +0000 +++ /tmp/puppet-file20171111-22472-s7ino5 2017-11-11 16:33:57.380255495 +0000 @@ -1,85 +1 @@ -user www-data; [....] -#} +something wrong \ No newline at end of file Info: Computing checksum on file /etc/nginx/nginx.conf Info: /Stage[main]/Psick::Nginx::Tp/Tp::Conf[nginx]/File[/etc/nginx/nginx.conf]: Filebucketed /etc/nginx/nginx.conf to puppet with sum 907bbf7d1cb3f410d8d6d4474a984b86 Error: Execution of 'nginx -t -c /etc/nginx/nginx.conf20171111-22472-12tbrva' returned 1: nginx: [emerg] unexpected end of file, expecting ";" or "}" in /etc/nginx/nginx.conf20171111-22472-12tbrva:1 nginx: configuration file /etc/nginx/nginx.conf20171111-22472-12tbrva test failed Error: /Stage[main]/Psick::Nginx::Tp/Tp::Conf[nginx]/File[/etc/nginx/nginx.conf]/content: change from '{md5}907bbf7d1cb3f410d8d6d4474a984b86' to '{md5}d9729feb74992cc3482b350163a1a010' failed: Execution of 'nginx -t -c /etc/nginx/nginx.conf20171111-22472-12tbrva' returned 1: nginx: [emerg] unexpected end of file, expecting ";" or "}" in /etc/nginx/nginx.conf20171111-22472-12tbrva:1 nginx: configuration file /etc/nginx/nginx.conf20171111-22472-12tbrva test failed Notice: /Stage[main]/Psick::Nginx::Tp/Tp::Install[nginx]/Service[nginx]: Dependency File[/etc/nginx/nginx.conf] has failures: true Warning: /Stage[main]/Psick::Nginx::Tp/Tp::Install[nginx]/Service[nginx]: Skipping because of failed dependencies Notice: Applied catalog in 11.07 seconds Puppet shows the diff of the file but it doesn’t actually change it, as the validate command has returned an error (any exit code different from 0). Since the file resource has failed, also the dependent resources, as the nginx service, are skipped. On the system the original file and the relevant service have remained untouched. We have just added such functionality to Tiny Puppet, now, whenever there’s the relevant Tiny Data for an application, the tp::conf defines runs a validate command for the files it provides. Configuration on Tiny Data is easy, check this commit for an example. Note that it’s possible, in Tiny Data, either to define a single string containing the validation script, or an hash, where is possible to define what command to run for what configuration file type. For example, the httpd -t -f % command can be used only with the main Apache configuration file (base_file = ‘config’), as it would fail with configuration fragments as can be virtual hosts definitions. Expect more TinyData to appear in the future to add configuration validation to applications managed by Tiny Puppet, and, please, feel free to send Pull Requests to TinyData, similar to the one linked earlier, to add support to new applications. Alessandro Franceschi 2017-11-06T00:00:00+00:00 https://example42.com/blog/2017/11/06/example42_events Are you interested to learn and discuss about Puppet? We will be at the following events, let’s meet there! November 9th - Puppet User Group Berlin Hosted by Visual Meta GmbH, Alexanderstraße 3, 10178 Berlin. This time we will demo bolt and tasks and plans. Please register using meetup or Xing. January 22nd to 26th - Puppet Roadshow One day demos on PSICK, GitLab integration, Puppet Enterprise and all new development. Location is yet to be defined, Stay tuned for updates. At the other days we are travelling around Germany, visiting customers. Please give us a note, if you like to get an on-site demo and learn about automating infrastructure. February 5th to 7th - CfgMgmtCamp, Ghent, Belgium We are happy to again attend and sponsor THE European automation event. Learn news from Puppet, Ansible, Chef and whatever is related to infrastructure automation. On third day, we will obviously be at Puppet Contributor Summit. Training courses Interested in learning Puppet? Our training courses are done in German language (english courses are available upon request). See our current Puppet Training offering for the up-to-date format. December 18-20 - Puppet Practitioner, Linuxhotel, Essen, Germany December 21-22 - Foreman, Linuxhotel, Essen, Germany February 12-14 - Puppet Fundamentals, ATIX AG, Munich, Germany February 15-16 - Puppet Architect, ATIX AG, Munich, Germany February 19-21 - Puppet Practitioner, ATIX AG, Munich, Germany February 26-28 - Puppet Fundamentals, Heinlein Academy, Berlin, Germany We are looking forward to seeing you in person and wish everybody happy hacking on Puppet and PSICK. Martin Alfke Alessandro Franceschi 2017-10-30T00:00:00+00:00 https://example42.com/blog/2017/10/30/puppet_development_ide How do you develop Puppet code? Which tools do you use? What tools should you use? In the early days I was mostly using vim for any kind of code development. Basically this have been shell scripts, some ugly Perl code and rarely fixing some PHP code. But: I have never been a developer. When Puppet came into place I continued using vim also for Puppet code development. While writing code and learning new principles, I learned that pure vim is not really very helpful. Luckily there are extensions one can use like puppet-vim. Genereally you will never want to code Puppet without syntax highlighting and automatic indentation. But how to develop code on larger projects with multiple repositories? When asking an experienced developer one will immediately hear the term IDE - Integrated development environment. In the above mentioned case, we use vim as IDE. But whet if you don’t know vim? What other solutions are available? Let’s check for some existing IDE’s IDE’s Disclaimer: I am totally aware that this list is incomplete. It is a list of tools I had a look at. Eclipse The most common used IDE is Eclipse. It is developed in Java and has a huge set of extensions to be also useful on any other programming language. Puppet support is possible by installing many different extensions. Mostly you will find “Geppetto”. Geppetto Based on Eclipse the Geppetto IDE was put together. One can either install Eclipse and add the Geppetto extensions or one can directly download a bundled version. But Geppetto lacks all the new, modern Puppet features like lambdas, data types and tasks. RubyMine JetBrains has multiple IDE’s - partly Open Source, partly paid software - for different development purposes. The most known one is IntelliJ IDEA. For Ruby based development RubyMine is available. For Open Source development JetBrains grants free licenses for Open Source projects. XCode When running OS X or MacOS it is also possible to use XCode. But XCode lacks full featured Puppet support. Visual Studio Code My personal favorite at the moment - next to vim - is Visual Studio Code available for Linux, OS X, MacOS, Windows. VSCode has plenty of plugins for different development purposes and multiple Version Control Systems. The most nice thing is that Puppet has officially released the Puppet plugin for VSCode. Atom Another IDE to use is Atom - an IDE developed by the people at GitHub. Atom also has plenty of plugins which allow you to easily write code in many programming languages. Atom has a very active community which also provide puppet support. Which one to use? This basically depends on the Operating System which you have running. Most of the above mentioned IDE’s work on Linux, macOS and Windows. When people at customers ask us for a recommendation, we usually ask them whether their developers already have a license for a specific IDE or whether they have a preferred one. In this case we ask the Puppet developers to also make use of the same tool as there is already knowledge available. When there is no common usage, one should check the Wikipedia IDE list or the Comparison of integrated development environments. Always verify for usage conditions and licenses, get the download link and try which one you are most comfortable with. Happy hacking on Puppet and PSICK. Martin Alfke 2017-10-23T00:00:00+00:00 https://example42.com/blog/2017/10/23/bolt_and_tasks_with_psick The release of bolt at last PuppetConf has stirred a lot of interest in Puppet community and in a very few days modules with tasks have started to appear on the Forge. At example42 we have started to experiment with Bolt and have added relevant profiles and tasks to the psick module. Let’s see how to work with Bolt in PSICK. First we need to install the psick module: puppet module install example42/psick Or add it to the Puppetfile of our control-repo (we can use the PSICK control-repo or any other one): mod 'example42/psick', :latest Then we have to classify our nodes with the psick class, it’s enough something like include psick Once the psick class is added to the catalog nothing happens, by default, but a huge amount of functionalities is a parameter away. If we use, as we should, Hiera to manage our data, and we have an [e]yaml backend, we can install bolt on a node (we need it only on the server from which we run commands) with data like: psick::base::linux_classes: bolt: 'psick::bolt' psick::bolt::is_master: true Psick can also automatically manage ssh keys sharing between nodes and the creation of a bolt user on all the nodes, who can sudo bold commands. This is completely optional (we can connect with bolt directly using the root user and share authorised keys via other methods) but if we want everything done out of the box we can add, for all our nodes: psick::base::linux_classes: bolt: 'psick::bolt' psick::bolt::master: <bolt_master> # Bolt master is the fqdn of node where to set psick::bolt::is_master: true psick::bolt::keyshare_method: storeconfigs We require storeconfigs enabled on our Puppet Server to automatically share ssh keys between the Master and the managed nodes. It will take some Puppet runs, on the so called Bolt master and the managed nodes, to converge and distribute the ssh keys to use for bolt. Once done, we can login on the Bolt master, as bolt user and from here we can run via Bolt commands, scripts, tasks and plans on any node of our Puppet infrastructure: [bolt@puppet ~]$ bolt command run uptime --n $(cat nodes/all) --user bolt Psick creates automatically the file called nodes/all in the home of the bolt user (this is the default user psick uses for ssh connections both on master and managed nodes), with a csv of all the nodes of the infrastructure. In the class psick::bolt::master is possible to create and customise different files for different nodes lists. The psick module has some tasks too, the first one we’ve thought about when we have heard about Bolt: psick::puppet_install installs Puppet agent on a remote node psick::puppet_agent runs Puppet agent on a remote node (eventually specifying the Puppet master, the Puppet environment and if to run in noop or no-noop mode psick::puppet_enable_noop configures noop mode on puppet.conf psick::puppet_unlock removes lock files create by stale Puppet runs or by puppet agent --disable psick::system_update trigger the update of all packages of the system We are quite sure this list is going to grow and the single tasks to be refined, but we think this list already covers some quite common needs. To run one of Psick’s tasks: bolt task run psick::puppet_unlock -n <node_fqdn> --modules <module_path> --user bolt (Note, we don’t have to specify --user bolt (the one used for SSH login) if we are running as bolt user locally. The examples in this post are done on vagrant servers and for some reasons the vagrant user is used by default to connect to remote servers, even if bolt is run as bolt user.) To run puppet agent in noop mode using the integration environment on all nodes, a command like this is enough: bolt task run psick::puppet_agent noop=true environment=integration -n $(cat nodes/all) --modules <module_path> --user bolt If you use psick, you have also the Tiny Puppet module installed, and this brings with it the tp::test task, which allows quick testing on the status of all the applications managed by Tiny Puppet in the whole infrastructure: bolt task run tp::test -n $(cat nodes/all) --modules <module_path> --user bolt This is just the beginning of our exploration of Bolt and Puppet tasks (and plans!) in psick. We see a huge potential in Bolt, it perfectly fits the part where Puppet was weaker than other tools like Ansible: remote commands execution, on demand, and, partly, orchestration. We are sure a lot of interesting use cases and applications will arise in the near future and we are committed to play a lot with it inside and outside PSICK. Alessandro Franceschi 2017-10-16T00:00:00+00:00 https://example42.com/blog/2017/10/16/plans_and_tasks At PuppetConf2017 the bolt task runner was released and made public. Bolt uses the concept of Puppet tasks to allow workflow based system management, which was missing in Puppet since ages. Puppet itself uses the declarative state configuration model, describing the final state of a system. With declarative description it was always a pain adding workflow based configurations like application updates or running maintenance tasks only at specific times. Bolt fills this gap. With bolt one can run any kind of: upload files to a system run any remote command run any script run a Puppet task run a Puppet plan Connection to remote systems is done either via ssh or WinRM. Other connectors can be added to bolt upstream development. At the moment there is no API available to add additional connectors to bolt via some kind of bolt plugin. The ssh access must be configured in advance prior being able to make use of bolt. Access can be configured as unprivileged user using sudo commands. Bolt just needs to know which credentials to use. Credentials for ssh can be placed in your ssh config file (~/.ssh/config). Credentials for Windows systems are provided on command line using the user (--user) and the password (--password) parameter. Which systems bolt should connect to must be provided on cli with nodes (--nodes) parameter. As of now, no node groups can be specified. Upload files Uploading files to a number of systems is easy: bolt file upload /local/file /remote/file --nodes www.domain.com,mail.domain.com For Windows system the nodes must be given using the winrm URI: bolt file upload /local/file /remote/file --nodes winrm://win.domain.com,server.domain.com --user Administrator --password <password> Running remote commands Running remote commands is easy. Just tell bolt which remote command to execute: bolt command run 'yum -y update' --nodes www.domain.com,mail.domain.com Running scripts Bolt is able to use a local script, copy it to the mentioned nodes and run it there: bolt script run ~/update_system.sh --nodes www.domain.com,mail.domain.com Please note that there is a difference to file upload: the script will be removed after execution. Writing and running tasks Tasks are something different. Tasks are part of modules and are placed into the (tasks) directory. When running tasks with bolt, one must specify the task and the module name space and the module path: bolt task run <modulename>::<taskname> --nodes <node list> --modules <modulepath> Additionally tasks may use parameters to switch action or behavior or to provde any kind of data. e.g. bolt task run application::update_app apppath=/opt/app --nodes db.domain.com --modules ~/workspace/modules The mentioned task (application::update_app) can be found within the application modules task directory in the update_app file. modules/ \- application/ \- tasks/ \- update_app A task must have an according .json file which documents the task and uses Puppet 4 data types on parameters: # modules/application/tasks/update_app.json { "description": "Update application", "supports_noop": false, "input_method": "environment", "parameters": { "apppath": { "description": "Path to application", "type": "Optional[String[1]]" } } } Within the task the parameter is used as environment variable with PT_ prefix: # modules/application/tasks/update_app #!/usr/bin/env bash if [ -z "$PT_apppath" ]; then apppath=$PT_apppath else apppath='/opt/app' fi pushd $apppath git reset hard --master git fetch --all git pull origin master popd When setting a parameter is mandatory, one can just use the task variable: #!/usr/bin/env bash updurl=$PT_updurl # will fail if no data was given pushd $apppath /opt/app/update.sh $updurl popd When having many parameters it will become a nightmare to provide all on command line. One can place parameters and their valies to a .json file; # params.json { "updurl": "git@git.domain.com/application.git", "apppath": "/opt/app" } Now you just must tell bolt that it should use the params.json file: bolt task run application::update_app --nodes db.domain.com --modules ~/workspace/modules --params @params.json Writing und running plans Plans combine multiple tasks. Think about the following problem: Update of an application requires you to do the following steps: disable node on loadbalancer wait for last request to be served update application restart web server check functionality re-enable node on loadbalancer Plans are - similar to tasks - part of a module and located in the (plans) directory. modules/ \- application/ \- plans/ \- update.pp we use the above mentioned example and generate a puppet plan: # modules/application/plans/update.pp plan application::update ( String $lbserver = 'lb.domain.com', String $maxtime = '60', String $updurl = 'ssh://git@git.domain.com/application.git', String $apppath = '/opt/app', String $chkscript = '/opt/app/bin/check', ){ # 'execute' tasks run_task('application::disable_node', $lbserver) run_task('application::wait_last_conn', $maxtime) run_task('application::update_app', $updurl, $apppath) run_task('application::restart_app', $maxtime) run_task('application::check_app', $chkscript) run_task('application::enable_node', $lbserver) } Usually we want error handling in plans. Please check writing plans for details. Now the bolt plan command can be used: bolt plan run application::update --modules <modulepath> Check the task docs and plan docs on additional topics like enable no-op mode on tasks use different plan execution functions: commands, scripts or other plans, uploading files input and output of tasks using tasks input and output from and to plans converting scripts to tasks Happy hacking on bolt. Martin Alfke 2017-10-08T00:00:00+00:00 https://example42.com/blog/2017/10/08/introducing-psick-infrastructure-module We have talked in the past about PSICK, example42’s Puppet control-repo which provides an integrated, powerful and customisable Puppet setup. This control repo had a set of profiles to manage common configurations using either external modules or local classes. We have seen this set of profiles grow over time and we have realised that they could be useful for any user, also the ones not using our control-repo. So we have decided to make out of them a dedicated, separated module and place there some of the classification logic we had in our control-repos main manifests directory. So here is the PSICK module, the first Infrastructure Puppet module. You can get it from the Forge or GitHub and just include it in your main manifest or any node classifier: include psick Once you do it, nothing happens :-D, as PSICK is entirely configurable via Hiera and allows you to decide what of its components and functions to use. PSICK modules main components are: Classification: you can use PSICK to classify, via Hiera, your nodes in well structured and robust way Base profiles: a set of classes that manage common configuration needs (resolver, time, sysctl, users, proxy, sudo…) TP profiles: a set of standard profiles to manage, using Tiny Puppet, applications like apache, mysql, openssh, mongodb and basically whatever is supported by Tiny Puppet Every PSICK configuration is data driven, so if you are using the YAML backend you can have your data as in the following examples. Classification Classification is based on the (opinionated) assumption that, when defining the classes and resources that are needed on a node it’s easier to split them by the underlying $kernel (Linux, Windows, Solaris..) and that, typically each node has a set of classes that should be applied before anything else (pre), a set of common classes applied to all the nodes of the infrastructure (base) and a set of specific classes (profiles), which are different for each role. In some cases, finally there could be some classes/resources that we want to apply only once, at the first Puppet run (firstrun). The PSICK module has classes that expose parameters that allow to specify the classes to include (note: they can be psick profiles, local profiles or external modules, choice is up to the user) on each phase for each $kernel. They can be expressed as hashes of key-values, looked up in deep merge mode, where keys are markers that allow override across the hierarchy, and the values are the names of the classes to include. A complete example of how classification may look like is as follows: # First run optional settings for Linux and Windows psick::enable_firstrun: true # By default firstrun mode is disabled psick::firstrun::linux_classes: hostname: psick::hostname psick::firstrun::windows_classes: hostname: psick::hostname # Common Linux classes psick::pre::linux_classes: puppet: puppet hostname: psick::hostname hosts: psick::hosts::resource dns: psick::dns::resolver repo: psick::repo users: psick::users psick::base::linux_classes: mail: psick::postfix::tp ssh: psick::openssh::tp sudo: psick::sudo logs: psick::logs::rsyslog time: psick::time sysctl: psick::sysctl update: psick::update motd: psick::motd profile: psick::profile network: network systat: psick::monitor::sar # Pre and Base psick settings Windows psick::pre::windows_classes: hosts: psick::hosts::resource psick::base::windows_classes: features: psick::windows::features registry: psick::windows::registry services: psick::windows::services time: psick::time users: psick::users::ad # Profiles for specific roles (ie: webserver) psick::profiles::linux_classes: webserver: apache psick::profiles::windows_classes: webserver: iis Common profiles The PSICK module provides a quite large of profiles for common configurations we do on servers. Some of them even support different common external modules as alternative to PSICK’s internal resources (like psick::sysctl or psick::users. The idea here is that for many common activities it’s not really needed to look for a dedicated public module, deal with its dependencies, and eventually for it locally to make it fit our implementation. An example of Hiera data configuring some of these PSICK common profiles is as follows, refer to relevant classes documentation for details: # Repo settings psick::repo::add_defaults: true # Time settings psick::time::servers: - 'pool.ntp.org' # Timezone settings psick::timezone::timezone: 'UTC' # Sample sysctl settings psick::sysctl::settings_hash: net.ipv4.conf.all.forwarding: 0 # Users management psick::users::delete_unmanaged: false psick::users::module: 'user' psick::users::users_hash: al: ensure: present comment: 'Al' groups: - users ssh_authorized_keys: - 'ssh-rsa AAAAB3N.....' # Hosts management psick::hosts::dynamic::extra_hosts: 'puppet.lab.psick.io': ip: '10.42.43.101' host_aliases: - puppet TP profiles Another set of features offered by the PSICK module are TP (Tiny Puppet) profiles: they offer a standard interface to the management of configurations of applications. Technically speaking any application for which we have tinydata for. Tiny Puppet takes care to install the relevant application on different OS but it’s up to the user to provide the configurations needed, with full freedom on how they are delivered (as static source files, via erb/epp templates using hashes of custom options). An example of configuration of tp profiles looks like this: # Postfix configuration psick::postfix::tp::resources_hash: tp::conf: postfix: template: 'psick/postfix/main.cf.erb' psick::postfix::tp::options_hash: 'mydomain': "%{facts.domain}" 'inet_interfaces': '127.0.0.1' 'inet_protocols': 'all' 'my_destination': '$myhostname, localhost.$mydomain, localhost' # Apache configuration psick::apache::tp::resources_hash: tp::conf: apache::example.com.conf: base_dir: conf template: psick/apache/vhost.conf.erb options_hash: ServerName: example.com ServerAlias: - www.example.com AddDefaultCharset: ISO-8859-1 apache::deny_git.conf: base_dir: conf source: puppet:///modules/psick/apache/deny_git.conf tp::dir: apache::example.com: vcsrepo: git source: git@github.com/company/example.com.git path: /var/www/html/example.com Here we define an hash of tp resources to apply (tp::conf and tp::dir ) and an hash of custom options that we can use in our templates. In templates we can use both the options, provided by users, and OS dependent tp settings, defined in tinydata, to easy support for multiple OS in the same template. For example, the template psick/apache/vhost.conf.erb which follows, uses variable like @settings['data_dir_path'], indicating the default DocumentRoot, which changes according the underlying OS: # File Managed by Tiny Puppet <VirtualHost *:80> DocumentRoot <%= @settings['data_dir_path'] %>/<%= @options['ServerName'] %> ServerName <%= @options['ServerName'] %> <% if @options['ServerAlias'] != "" -%> <% if @options['ServerAlias'].is_a? Array -%> ServerAlias <%= @options['ServerAlias'].flatten.join(" ") %> <% else -%> ServerAlias <%= @options['ServerAlias'] %> <% end -%> <% end -%> <% if @options['AddDefaultCharset'] -%> AddDefaultCharset <%= @options['AddDefaultCharset'] -%> <% end -%> ErrorLog <%= @settings['log_dir_path'] %>/<%= @options['ServerName'] %>-error_log CustomLog <%= @settings['log_dir_path'] %>/<%= @options['ServerName'] %>-access_log common </VirtualHost> For some applications, besides standard tp profiles, where are more complex classes to manage application specific items, give a look at the mariadb , php , docker , ansible classes for some examples. All these profiles are optional, you can decide to use them or external component modules or local profiles to manage a specific application in the way you want, and you can even have a mix of them. Conclusion The PSICK module does a lot of things, definitively too many according to purists of “a module for each application/function”, but as we always try to do, it’s always a matter of choice. You can choose what you want to use of this module, you can decide to install and configure applications via Tiny Puppet or via dedicated modules, and this choice can be done for every single component you want to manage. PSICK has a few dependencies, the puppet-stdlib module, which should be already used in any Puppet environment, and example42-tp module (Tiny Puppet) which depends on example42-tinydata. All these modules can be added to your modules’ path seamlessly and without any interference with existing Puppet setups (a few extensions are pluginsynced, but we have been careful to reduce as much as possible their number). On the other side, if you embrace the PSICK philosophy to its full, you’ll have dramatically reduced the amount of modules needed to configure your systems (and much less conflicting modules dependencies hell), you will have a clean way to order your classes and you’ll have extremely compact catalogs, with a limited number of resources doing a lot of things. The PSICK module, paired with the PSICK control-repo can help you in setting up a state of the art Puppet infrastructure in a few time, with full flexibility or how and what to configure on your systems. We hope you’ll understand and appreciate them as we do and, please, do let us know what you think should be made different and better: works on them has just began, we plan to add a lot of stuff to them in order to make sysadmins life better. Alessandro Franceschi 2017-10-02T00:00:00+00:00 https://example42.com/blog/2017/10/02/tenant-modules The Puppet code and data we use to shape and configure our infrastructures often need to be managed by different people, sometimes belonging to different groups. Some of them may have good Puppet skills, some not and just need configurations done in the way they need. A common scenario is where there are different teams managing the basic systems’ configurations and applications: the first group may be the one which introduced Puppet and has most of the knowledge about it, the second one need to use existing code and eventually write custom one, and feed the data to configure their managed applications. In other cases different groups responsible for different applications may need to manage them independently. A single control-repo with or without extra internal modules added to the Puppetfile may serve the purpose, but it has some drawbacks: changes needed by an applications team have to be done on a repository which manages the whole infrastructure, containing systems which they are not responsible for. There may be different approaches to this, which can involve a strict overview and change process on the control-repo, but probably the most flexible and frictionless one is by using tenant modules. A tenant module is contained in a git repository under the responsibility of a team and it can manage an application or a part of the Puppet managed infrastructure without interfering with the global infrastructure, managed by the operations team handling the control-repo. It has the structure of a normal module, with its one classes, defines, templates and eventually data in module, but compared to a a normal module it has also a local directory which is added to the environment Hiera’s hierarchy. A Hiera 5 environment hiera.yaml using a tenant module and a fact or top scope variable identifying the tenant name, may look like: --- version: 5 defaults: datadir: data hierarchy: - name: "Master control repo hierarchy node specific" lookup_key: eyaml_lookup_key # eyaml backend paths: - "nodes/%{trusted.certname}.yaml" options: pkcs7_private_key: /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem pkcs7_public_key: /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem - name: "tenant module hierarchy certname" lookup_key: eyaml_lookup_key # eyaml backend paths: - "../modules/%{::tenant}/hieradata/nodes/%{trusted.certname}.yaml" options: pkcs7_private_key: "/etc/puppetlabs/puppet/keys_%{::tenant}/private_key.pkcs7.pem" pkcs7_public_key: "/etc/puppetlabs/puppet/keys_%{::tenant}/public_key.pkcs7.pem" - name: "tenant module hierarchy common" lookup_key: eyaml_lookup_key # eyaml backend paths: - "../modules/%{::tenant}/hieradata/common.yaml" options: pkcs7_private_key: "/etc/puppetlabs/puppet/keys_%{::tenant}/private_key.pkcs7.pem" pkcs7_public_key: "/etc/puppetlabs/puppet/keys_%{::tenant}/public_key.pkcs7.pem" - name: "Master control repo hierarchy" lookup_key: eyaml_lookup_key # eyaml backend paths: - "env/%{::env}.yaml" - "common.yaml" options: pkcs7_private_key: /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem pkcs7_public_key: /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem this means that inside the $::tenant module, under its hieradata directory it’s possible to set parameters that affect a node, without the need to change anything on the control-repo and it’s also possible to specify custom paths for the Hiera eyaml keys, so that each group can manage its passwords securely and independently. Hierarchies may vary, other layers may be added and extra precautions may be taken in order to avoid a team affecting nodes managed by other teams. Plus point anyway is that a group can place its data, templates, and eventually also custom classes, in a module which is managed independently. In order to avoid the need to update the Puppetfile in the control-repo when a change is done on a tenant module, the special option :branch => :control_branch can be added for the tenant module: mod 'team_one', :git => 'git@git.example.com:organization/team_one.git', :branch => :control_branch This syncs the branch used in the control-repo with the one of the module, so, for example, when control-repo’s production branch is deployed in Puppet’s production environment, also the production branch of the tenant module is deployed. If we configure both on the control-repo and on the tenant module, Puppet Enterprise’s Code Manager or another r10k webhook to automatically deploy code as soon as a change is done in the relevant git branch, we can automate code deployment of the tenant module and give the relevant group full control of their systems, without the need of support or manual intervention from the group than manages the control-repo. Variations on the theme are possible, yet the principle is the same: by allowing different groups to manage their own git repositories, and placing there tenant modules which are used to manage custom files and manifests AND also affect the Hiera data, we can delegate responsibilities in a safe and frictionless way. Alessandro Franceschi 2017-09-25T00:00:00+00:00 https://example42.com/blog/2017/09/25/multiple_eyaml_keys With improved security implementations it is often required that keys must be separate among different infrastructure stages. This means that we have to deal with multiple eyaml keys for production-stage and ci- and development-stage. Nobody may have the private production key. Everybody should have access to the production public key (which is used for encryption). All other keys can be made available to everybody. First let’s set some top scope variable by analysing facts: # Set top scope variables # eyaml key selection based on existence of an external fact: # 'eyaml_private_base_path' # when fact is set, then we run on spec tests keys # otherwise we use production keys if has_key($::facts, 'eyaml_private_base_path') { $eyaml_selector = 'development' } else { $eyaml_private_base_path = '/etc/puppetlabs/puppet/eyaml' $eyaml_selector = 'production' } Now let’s adopt our hiera yaml: --- version: 5 defaults: datadir: data hierarchy: - name: "Data" lookup_key: eyaml_lookup_key paths: - "hosts/%{::trusted.certname}.yaml" - "hosts/%{::trusted.certname}_secrets_%{::eyaml_selector}.yaml" - "role/%{::role}/%{::env}.yaml" - "role/%{::role}/%{::env}_secrets_%{::eyaml_selector}.yaml" - "role/%{::role}.yaml" - "role/%{::role}_secrets_%{::eyaml_selector}.yaml" - "zone/%{::zone}.yaml" - "zone/%{::zone}_secrets_%{::eyaml_selector}.yaml" - "common_secrets_%{::eyaml_selector}.yaml" - common.yaml options: pkcs7_private_key: "%{::eyaml_private_base_path}/private_key.pkcs7_%{::eyaml_selector}.pem" pkcs7_public_key: "/etc/puppetlabs/code/environments/%{::environment}/eyaml/keys/public_key.pkcs7_%{::eyaml_selector}.pem" This will lead to a quite complex hierarchy, with the benefit of separating encryptions done with different keys. All *_secrets_production.yaml files contain secrets encrypted with the production key. All *_secrets_development.yaml files contain secrets encrypted with the development key. On the other hand it is easy to find missing encrypted production keys by comparing the hiera data keys in both yaml files. Martin Alfke 2017-09-18T00:00:00+00:00 https://example42.com/blog/2017/09/18/when-to-place-data-in-hiera When Puppet Inc. released hiera, we gained the possibility to separate data from code. Prior hiera we used logic patterns like if and case to identify differences in our platform and configurations. With hiera we started using the hiera lookup function to fetch data based on a hierarchy where we specify differences of our platform. This lead to cleaner, better maintainable code. Now people started putting data no longer into their code, which lead to huge hiera data files and hard to maintain hierarchies. The biggest problem is to identify when to put data in hiera and when to keep data in code. Puppet did a blog post a couple of weeks ago, describing the problem and providing a workflow to identify where data should be placed. I mostly agree with Gary’s posting and decisions. There is just one minor issue I see with his approach: the posting does not talk about hiera data namespaces. Generally you start with the question: is this data different among the platform? If you must confess that data is identical anywhere you will not put this data into hiera, but keep it inside the profile: # site/profile/manifests/login/ssh.pp class profile::login::ssh { class { 'ssh': permit_root_login => 'no', } } If you must use the variable from multiple places, it is recommended to assign the data to a variable and use this variable instead: # site/profile/manifests/data.pp class profile::data { $delete_unmanaged_accounts_and_keys = true, } # site/profile/manifests/login/ssh.pp class profile::login::ssh { include profile::data class { 'ssh': permit_root_login => 'no', purge_keys => $profile::data::delete_unmanaged_accounts_and_keys, } } # site/profile/manifests/users.pp class profile::users { include profile::data class { 'profile::users::static': delete_unmanaged => $profile::data::delete_unmanaged_accounts_and_keys, } } Next question is whether data can be calculated on a simple logic. In this case you will place the logic into the profile: # site/profile/manifests/login/ssh.pp class profile::login::ssh { include profile::data $case $::location { 'dmz': { $manage_firewall = true } default: { $manage_firewall = false } } class { 'ssh': permit_root_login => 'no', purge_keys => $profile::data::purge_ssh_auth_keys, manage_firewall => $manage_firewall } } No comes the point where I differ from Gary’s posting: In all other cases make your profile a parametrised profile. Whether you want to specify a sane default value is up to you. # site/profile/manifests/login/ssh.pp class profile::login::ssh ( Integer $port = 22, Array $allow_groups, ){ include profile::data $case $::location { 'dmz': { $manage_firewall = true } default: { $manage_firewall = false } } class { 'ssh': permit_root_login => 'no', purge_keys => $profile::data::purge_ssh_auth_keys, manage_firewall => $manage_firewall } } Within hiera one can now use the profile namespace to place data: profile::login::ssh::port: 2222 profile::login::ssh::allow_groups: - 'admin' - 'backup' example42 wishes everybody fun and success with Puppet and hiera. Martin Alfke 2017-09-11T00:00:00+00:00 https://example42.com/blog/2017/09/11/automate-reusable-eyaml-setup Many people prefer to have sensitive data not in plain text in hiera. Instead of plain text the eyaml - encrypted yaml - hiera extension is widely used. But how to deal with public/private key pair? Where to store them? Where to place them? Why do you want to store your keys? After bootstrapping your Puppet server and installing eyaml you usually run the eyaml createkeys command. This generates a private/public key pair. Now you start encrypting your data using eyaml encrypt. For encryption only the public key is required. Normally you will work with a larger group of people on your configuration automation and management. So you will not directly work on the puppet server. Additionally you want everybody to encrypt data, but only dedicated users or systems should be able to decrypt data. To allow everybody encrypting data, you can easily place the public key into your control-repository. The private key must be protected and kept safe. So you will not put it into the control-repo. When re-bootstrapping your Puppet server, you must ensure that you are re-using the old keys which already have been used to decrypt data. Otherwise the hiera lookup will complain about a bad key if keys gets re-created. Which systems must have the private key? First it is your Puppet server(s) so data lookups can be decrypted. Depending on your spec tests you want your ci systems to also have the private key at hand. On the CI systems you can easily place the key via Puppet, using another file serving mount point (see post Using a second mount point for files). On the Puppet server bootstrapping you must install the private key from a secure location, e.g. vault. Your hiera.yaml file can now use the public key in the control-repository and the private key in a separate directory: --- version: 5 defaults: datadir: hieradata hierarchy: - name: "Eyaml hierarchy" lookup_key: eyaml_lookup_key # eyaml backend paths: - "nodes/%{trusted.certname}.yaml" - "role/%{::role}-%{::env}.yaml" - "role/%{::role}.yaml" - "zone/%{::zone}.yaml" - "common.yaml" options: pkcs7_private_key: /etc/keys/private_key.pkcs7.pem pkcs7_public_key: keys/public_key.pkcs7.pem # relative path in control-repository example42 wishes everybody fun and success at encrypting data. Martin Alfke 2017-09-04T00:00:00+00:00 https://example42.com/blog/2017/09/04/testing-any-role-on-any-os-with-a-psick-control-repo Short version of the post: cd vagrant/environment/ostest ln -sf hieradata/role/${::role}.yaml hieradata/role/ostest.yaml vagrant up [vm] Long version of the post: PSICK explained, again :-) PSICK is the Puppet control repo of reference we use in example42 to bootstrap new projects and to test new modules and design patterns. It contains the evolving synthesis of our best practices and 10 years of Puppet experience, so it’s full of stuff, from tools to help with Puppet development, to CI for testing Puppet deployments, from a rich set of opinionated profiles to a solid and flexible classification approach. It’s intended to be forked, morphed, adapted and customised, or cherry picked. It’s a ongoing forge of ideas and solutions, and even if every part of it can be adapted or modified according to any need, there’s is a design approach that defines PSICK: it is self contained and entirely data driven. It has all, by default, we need to manage and provision an infrastructure with Puppet. Entirely Hiera driven PSICK, for the user perspective, can be entirely Hiera driven, based on hierarchies containing concepts like roles, operational environments (envs, tiers) and zones (datacenters). The default hiera.yaml looks like: --- version: 5 defaults: datadir: hieradata hierarchy: - name: "Eyaml hierarchy" lookup_key: eyaml_lookup_key # eyaml backend paths: - "nodes/%{trusted.certname}.yaml" - "role/%{::role}-%{::env}.yaml" - "role/%{::role}.yaml" - "zone/%{::zone}.yaml" - "common.yaml" Names, layers and logic may differ, as the same way we assign these variables. The basic principle should be: In our environment/control-repo hiera.yaml we use hierarchies which reflect and map how our configurations change in the nodes of our infrastructure. We don’t generally care about changes in Operating Systems (data in profiles and modules care of them), we focus on our infrastructure, and how data can adapt to it. Setting top scope variables used in hierarchy We may set the top scope variables used in hierarchy paths in different ways: As trusted facts defined during server provisioning, before Puppet’s first run As external facts set during provisioning As normal facts pluginsynced from our site modules As global variables set via an ENC (Puppet Enterprise, Foreman) Directly in the main manifest, in the top scope, outside any class PSICK’s default expects the variables used in the hierarchy as trusted or normal facts, this implemented in manifest/site.pp, where everything happens, with something like: if $trusted['extensions']['pp_role'] { $role = $trusted['extensions']['pp_role'] } To give the idea, these are sample user-data files to set such trusted facts on ec2 instances. Nodes classification By default, in the main manifest we manage also nodes classification, as follows. First, we include, in all the nodes, a settings profile, used only as entry point for (Hiera driven) variables shared across profiles. contain '::profile::settings' Variables like $::profile::settings::proxy_server may be used by different profiles and this should be the default value for their own proxy settings. Then we include a prerequisites class, which provides the prerequisites resources we want to to evalutate first (typically package repositories and proxy settings) contain '::profile::pre' Finally a general baseline class is included, distinct for each OS kernel: $kernel_down=downcase($::kernel) contain "::profile::base::${kernel_down}" Every group of resources managed by the pre and base profiles is declared inside a class, using a class name exposed as a parameter, manageable via Hiera: class profile::base::linux ( # General switch. If false nothing is done. # Set to false to skip base classes management. Boolean $manage = true, String $network_class = '', String $mail_class = '', String $puppet_class = '', [...] ) { if $network_class != '' and $manage { contain $network_class } [...] } So we can set, in common.yaml or anywhere in the hierarchy, params like the following to fine tune what common classes, local profiles or directly public module we want: profile::base::linux::mail_class: '::profile::mail::postfix' profile::base::linux::puppet_class: '::puppet' profile::base::linux::ssh_class: '::profile::ssh::openssh' profile::base::linux::users_class: '::profile::users::static' profile::base::linux::sudo_class: '::profile::sudo' profile::base::linux::monitor_class: '::profile::monitor' [...] For windows, many resources are different and it makes sense to manage them in a separated base profile: profile::base::windows::puppet_class: '' profile::base::windows::features_class: '::profile::windows::features' profile::base::windows::registry_class: '::profile::windows::registry' profile::base::windows::network_class: '' Additional profile classes, which are specific for [group of] nodes, are looked via Hiera (using the profiles key) and included: lookup('profiles', Array[String], 'unique', [] ).contain We also ensure they are applied after all the base profiles. lookup('profiles', Array[String], 'unique', [] ).each | $p | { Class["::profile::base::${kernel_down}"] -> Class[$p] } No roles (classes) There are no role classes, they function is replaced by the profiles included via hiera: we can reproduce the tipical roles and profiles pattern by defining under hieradata/role/$::role.yaml something like: --- profiles: - profile::git - profile::puppet::gems - profile::ci::octocatalog - profile::ci::danger - profile::gitlab::runner - profile::gitlab::ci - docker This would be the same of having a class like site/role/manifests/cirunner.pp with: class role::cirunner { include profile::git include profile::puppet::gems include profile::ci::octocatalog include profile::ci::danger include profile::gitlab::runner include profile::gitlab::ci include docker } But here we have the flexibility of Hiera and the possibility to manage exceptions, or add new roles, just working with yaml files. Vagrant to test multiple OS PSICK includes several Vagrant multi VM environments that can be used to test the code and the data of the control repo itself. We have made them easily configurable (check the Vagrant docs for details) by editing a single config.yaml where it’s possible to test role, operating systems both in Puppet agent and apply mode. We have a vagrant environment called ostest, when we can test our control repo on the following machines: al@lab psick [development] $ cd vagrant/environments/ostest/ al@lab ostest [development] $ vagrant status Current machine states: centos7.ostest.psick.io running (virtualbox) centos6.ostest.psick.io not created (virtualbox) ubuntu1604.ostest.psick.io poweroff (virtualbox) ubuntu1404.ostest.psick.io not created (virtualbox) ubuntu1204.ostest.psick.io not created (virtualbox) debian9.ostest.psick.io poweroff (virtualbox) debian8.ostest.psick.io not created (virtualbox) debian7.ostest.psick.io not created (virtualbox) suse12.ostest.psick.io not created (virtualbox) suse11.ostest.psick.io not created (virtualbox) opensuse-tumbleweed.ostest.psick.io not created (virtualbox) opensuse-42-1.ostest.psick.io not created (virtualbox) alpine3.ostest.psick.io not created (virtualbox) fedora23.ostest.psick.io not created (virtualbox) cumulus.ostest.psick.io not created (virtualbox) windows2012-ostest not created (virtualbox) windows2008-ostest not created (virtualbox) Yes also Windows, yes it works. These VMs are configured as follows in vagrant/environments/ostest/config.yaml, note how we set ex external fact $::role to **ostest**. --- # Default settings for all vms (they can be overridden on each node) vm: memory: 1024 # MB or RAM to assign cpu: 1 # Number of vCPU to assign to the VM role: ostest # Default role box: centos7 # Box used for the VM, from the box list in vagrant/boxes.yaml puppet_apply: true # Run puppet apply on the local control-repo during provisioning puppet_agent: false # Run puppet agent during provisioning facter_external_facts: true # Create external facts in facts.d/$fact.txt. Note 1 facter_trusted_facts: false # Create csr_attributes.yaml. Note 1 # A local network is created among the VM. Here is configured. network: range: 10.42.45.0/24 # Network address and mask to use ip_start_offset: 101 # Starting ip in the network for automatic assignement domain: ostest.psick.io # Name of DNS domain for the created machines # Puppet related settings puppet: version: latest # Version to use for OSS install_oss: true # If to install Puppet OSS agent on the VMS install_pe: false # If to install Puppet Enterprise agent on the VMS env: devel # Setting for the env fact (may be used in hiera.yaml) zone: ostest # Setting for the zone fact (may be used in hiera.yaml) datacenter: vagrant # Setting for the datacenter fact (may be used in hiera.yaml) application: puppet # Setting for the application fact (may be used in hiera.yaml) master_vm: foreman.fab.psick.io # Name of the VM which play as Puppet server for the others master_fqdn: 'foreman.fab.psick.io' # FQDN of the Puppet server to use with puppet agent link_controlrepo: true # Add a link for a Puppet environment to the development control-repo environment: host # Puppet environment to link to local control-repo # Nodes shown in vagrant status nodes: - hostname_base: centos7 box: centos7 - hostname_base: centos6 box: centos6 - hostname_base: ubuntu1604 box: ubuntu1604 [...] Now we actually have a sample hieradata/role/ostest.yaml but we can, better, use a symlink instead, pointing to the actual role we want to test without the need to write any extra code. cd vagrant/environment/ostest ln -sf hieradata/role/${::role}.yaml hieradata/role/ostest.yaml vagrant up [vm] Alessandro Franceschi 2017-08-28T00:00:00+00:00 https://example42.com/blog/2017/08/28/git-workflow-on-control-repo Modern best practices for management of Puppet Code suggest to make use of a control-repository. Within a control-repository one manages the whole Puppet Code in a centralized pattern but is still allowing flexibility and code staging: upstream modules are referenced in Puppetfile Profiles and Roles are managed directly inside the repository Hiera configuration and data are also part of the control-repository Usually it is recommended to rename master branch to production. Older versions of Puppet might behave erratically when using a puppet.conf section name (master, main, agent) as an environment name. Most people use an additional branch for development and staging purpose. This will result in the following branches: production integration development New features are usually developed inside a feature branch. Up to here everything is fine. But…. How to proceed if you are developing several features with multiple developers in parallel? Time point 0: Developer A creates a feature branch based on production Time point 1: Developer B creates a feature branch based on production Time point 2: Developer B finished his work and merges his feature branch into development and integration Time point 3: Developer A merges his feature into development and integration Time point 4: Developer A has finished his tests and wants to merge his feature into production BANG! Developer A can not merge integration branch into production as integration branch has an additional feature not yet ready for production. Usually people start cherry-picking their features which normally requires to squash all commits into a single commit. Let’s rethink our branches. Why do you need development and integration as long living branches? Why not have production branch only and short living feature branches? OK, you want to test on a dedicated infrastructure which uses a fix Puppet environment name. OK, you must wait for your ITIL release manager to approve the change in production. This brings you to two branches: integration and production. The production branch is the default branch for all of your systems. Differences in infrastructure stages are configured by having a hiera hierarchy for your stages. Integration branch is the branch where you separate your feature deployments. Every feature which is not yet fully developed may not be merged into integration branch. This will lead to the following concept: production is your default branch integration is your feature isolation branch integration may differ one code commit from production only Happy hacking on your control repo! Martin Alfke 2017-08-21T00:00:00+00:00 https://example42.com/blog/2017/08/21/encrypt-your-secrets-with-hiera-eyaml Hiera-eyaml is a Hiera backend which can be used to encrypt single keys in Hiera yaml files. It has become the standard the facto to manage passwords, secrets and confidential data in Puppet. It’s now included by default in Hiera 5, (shipped with Puppet version > 4.9), in earlier versions in can be installed as a gem: gem install hiera-eyaml On the Puppet server we need to do that also in Puppet environment: puppetserver gem install hiera-eyaml To configure it we need to specify the backend in hiera.yaml and the location of the keys used to encrypt the data. Syntax for Hiera < 5 is something like: --- :backends: - eyaml :eyaml: :datadir: "/etc/puppetlabs/code/environments/%{environment}/data" :pkcs7_private_key: /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem :pkcs7_public_key: /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem :extension: 'yaml' Syntax for Hiera version 5 is like: --- version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: "Eyaml hierarchy" lookup_key: eyaml_lookup_key # eyaml backend paths: - "nodes/%{trusted.certname}.yaml" - "common.yaml" options: pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem" pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem" The ge provides the eyaml command, which can be used to perform any Hiera.eyaml related operation. Before starting to encrypt data a pair of public and private keys has to be created: eyaml createkeys This creates in the keys directory (relative to the current working directory) the private_key.pkcs7.pem and public_key.pkcs7.pem files. The first one should never be shared and must be managed in a safe way, for this reason the keys (at least the private one) should not be added to the control-repo git repository and must be readable by the user running the Puppet Server (/etc/puppetlabs/puppet/keys is a sane path, but could be anything). Both of these file must be placed wherever Hiera files are evaluated: that means basically all the Puppet Servers but also, eventually, on developers workstations, if Puppet code is tested locally via Vagrant. To avoid the need to share private keys to all developers, we recommend, anyway, to avoid to encrypt data in Hiera files used by machines running in Vagrant. So for example, if we have a Hiera layer which represent a machine environment or tier, and for Vagrant nodes we use the devel tier, we can override eventually encrypted data in a general common.yaml with clear text entries in a Vagrant specific layer (like "tier/devel.yaml"). Just know that we need the private key when encrypted data is looked for, if we manage to have no encrypted data for servers running under Vagrant, Hiera eyaml works flawlessly even if the public and private keys are not stored locally. Creating encrypted Hiera values We can generate the encrypted value of any Hiera key with the following command: eyaml encrypt -l 'mysql::root_password' -s 'V3ryS3cr3T!' This will print on stdout both the plain encrypted string and a block of configuration that we can directly copy in our yaml files as follows: --- mysql::root_password: > ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMII [...] Note that the value is in the format ENC[PKCS7,Encrypted_Value]. Since we have the password stored in plain text in our bash history, we should clean it using the following command: history | grep encrypt 572 eyaml encrypt -l 'mysql::root_password' -s 'V3ryS3cr3T!' history -d 572 Alternatively we can directly edit Hiera yaml files with the following command: eyaml edit hieradata/common.eyaml Our editor of preference will open the file and decrypt the encrypted values eventually present so that we can edit our secrets in clear text and save the file again (of course, we can do this only on a machine where we have access to the private key). To add a new encrypted key to a file we can open it with eyaml edit and add a key with a syntax like this: --- mysql::root_password: DEC::PKCS7[my_password]! The string my_password (our password in clear text) will be encrypted once the file is saved. To show the decrypted content of an eyaml file, we can use the following command: eyaml decrypt -f hieradata/common.eyaml Since hiera-eyaml manages both clear text and encrypted values, we can use it as our only backend if we want to work only on yaml files, so it’s pointless to use both yaml and eyaml backends. Hiera-eyaml has been originally created by Tom Poulton then continued as a community effort which, given its popularity, at a certain point was directly incorporated in upstream Puppet, as it happened with the same RIPienaar’s Hiera. A wonderful example of how the community has helped in shaping Puppet. Alessandro Franceschi 2017-08-14T00:00:00+00:00 https://example42.com/blog/2017/08/14/testing-a-control-repo-with-vagrant When we develop our Puppet code it’s useful to have the possibility to test the effect of what we are doing on real systems, running Puppet on them, using our code under development, and seeing what happens without the need to commit anything. Traditional testing methods based on spec tests don’t verify the actual effect of our code on systems (they analyse the generated catalog to verify if it has the expected resources), we need to run real code on real operating systems. Vagrant is the perfect tool for this and we can use it in our Puppet development setup. We’ve already talked about how we use Vagrant on PSICK, Example42’s sample Control repo [generator], but that’s a rather complex setup, with multiple Vagrant environments, a simple to use configuration file to use for each one of them, and multiple approaches to Puppet run. If you want all the work done, just use PSICK, or copy from it the vagrant directory and the script bin/puppet_install.sh (used to install Puppet in certain VMs). Let’s review here, instead, the basic principles and what has to be done to setup Vagrant testing from within our control-repo. First we have to create a Vagrantfile, here we can configure one of more VMs to work on. Then we have to decide how we want to run Puppet within the VMs, we have different options here: (1) Run Puppet in apply mode, without using any Puppet Server. This is the simplest approach (we don’t need a dedicated Puppet server to point to) but it fully simulates our real server setup , without further efforts, only if the following conditions are met: We are not using an External Node Classifier or we can simulate in the Vagrant environment what the ENC provides (classes to include, parameters to set) We don’t rely on PuppetDB to manage resources in our catalog, that is we don’t use exported resources and we don’t use functions, like puppetdb_query that interrogate PuppetDB directly. If we are in these conditions we have to provide some workaround for machines running in Vagrant. (2) Run Puppet in agent mode, using a Puppet Master running in our Vagrant environment. This is a valid alternative, which may cope with PuppetDB but may presents a few additional challenges: If we use an ENC on our live Puppet Server, we must configure accordingly our Vagrant Puppet Server We have to mount on the Vagrant Puppet Server our local control-repo, so that the files it serves come directly from the host where we are developing (in this case is absolutely necessary to disable catalog caching in environment.conf). On PSICK you can see both approaches used in different Vagrant environments under vagrant/environments. Other alternatives, like running Vagrant in agent mode pointing to an existing external Puppet Server, may be tried, as long as it’s preserved the basic principle of being able to test our code before committing it (so we should either develop directly on the Puppet Server, using a dedicated environment, our mount there via NFS or similar, our local development directory). Let’s concentrate on the apply scenario, as using puppet agent implies that we are able to setup a Puppet Server on Vagrant which reproduces the same conditions we have on the real infrastructure. Besides the apparent limitations, listed earlier, such approach is possible in many different cases, as long as we care of: Setting with provisioning a script either external facts or trusted facts before running Puppet, if they are needed to classify nodes or are used in our hiera.yaml hierarchies. Running Puppet in apply mode passing all the arguments we need to point our local Hiera data, and use the modules in the control-repo. For example this one. Be sure we have, on our VMs all the gems and tools needed to compile a catalog, so Puppet, of course, and eventual extra gems (example) Mount on the VM (better if in Read Only mode) the control repo directory we are developing on, under /etc/puppetlabs/code/environments/production (or link the “usual” /vagrant directory or change the puppet apply command to point to the correct local path). Have the modules listed in our Puppetfile deployed on our development workstation (it’s enough to run r10k puppetfile install from the main control-repo directory to populate accordingly its modules subdirectory) If we use Hiera-eyaml and we don’t want to place our private key on developers’ workstations, we can just override eventual encrypted data in common Hiera files with unencrypted data, in the Hiera layers specific for development hostnames or for a devel tier. If we use exported resources or functions that query PuppetDB, provide exceptions when the code is evaluated under Vagrant (usually this can be done checking if the value of the virtual fact matches Virtualbox, as this is the most comment hypervisor used in Vagrant.) Generally testing code under Vagrant in Puppet apply mode is easier and doesn’t require particular workarounds when our control-repo is self-contained: it contains all the information we need to classify and configure nodes, eventually basing it on facts that can be easily added, as we’ve seen it before, during Vagrant provisioning. Alessandro Franceschi 2017-08-07T00:00:00+00:00 https://example42.com/blog/2017/08/07/class-indirection There are words that we, well I, sometimes use with hesitation: we are not 100% sure about their meaning, especially when they are in a foreign language. Puppet class indirection is one of these. It’s the name I give to the pattern I’m going to describe in these lines, but I’m not fully sure that’s a correct one. Anyway, I already happened to write about class indirection driven via Hiera, let’s review here the key principles and see some use case. We are used, with Puppet, to include classes which contain other classes. It happens in most of the modules, where the main class includes sub classes to manage installation, service, configuration or extra components of the managed application. It may happen in our site profiles, where we group different kind of resources in different classes and we wrap then in a single handy wrapper class. Class indirection is the possibility of defining what class to use of each of this sub function. This can be simply accomplished by exposing in the main class (the ones that includes the other ones) parameters that define the names of the sub classes to include. An example is from PSICK, where there’s a baseline profile for each $::kernel which exposes a parameter to define the name of the class to use for each sub component / group of system’s resources. class profile::base::linux ( # General switch. If false nothing is done in this class. Boolean $enable, String $puppet_class, [...] String $ssh_class, ) { if $puppet_class != '' and $enable { contain $puppet_class } [...] if $ssh_class != '' and $enable { contain $ssh_class } } This means that it’s possible to define on Hiera the names of the classes to use to manage Puppet, SSH or anything else with data like: profile::base::linux::puppet_class: '::profile::puppet::agent' profile::base::linux::ssh_class: '::profile::openssh' Which, being Hiera driven, given us complete flexibility to manage common classification problems in handling exceptions and edge cases. On a PuppetMaster role or node Hiera file, for example, we might have: profile::base::linux::puppet_class: '::profile::puppet::master' on a SSH gateway or jump host we might have: profile::base::linux::ssh_class: '::profile::ssh::jump' This approach makes it easier to test and rollout new profiles and manage a gradual puppettization of resources on a brown field environment. For example when introducing management of ssh via Puppet on existing servers, we can disable on common.yaml to inclusion of any ssh class: profile::base::linux::ssh_class: '' and then to test our class on a env/test.yaml file with our profile: profile::base::linux::ssh_class: '::profile::openssh' And then eventually update common.yaml with the tested '::profile::openssh'. Note that we don’t always need a custom profile to manage an application, an existing module may do all what we need. In such cases, we might include it directly: profile::base::linux::ssh_class: '::openssh' and use in Hiera parameters from the used module: openssh::root_login: false I found this pattern particularly fitting for baseline classes that typically include other classes, but it might be useful also in normal modules. I’d love to see, as common practice, the usage of parameters like: class apache ( String $install_class = '::apache::install', String $service_class = '::apache::service', String $config_class = '::apache::config', ) { [...] } Having the possibility to override the class used to manage, for example, the installation of Apache, would be make the module usable, without changes, in many environments with legacy needs (for example custom repos or packages). It’s up to the user to provide a working alternative class: apache::install_class: '::profile::apache::install' In other cases we might replace the classes that require external modules dependencies that conflict with ours with our own modified version, in a different namespace with the needed corrections. These are just examples. Possibilities and use cases are many but most of all, class indirection is easy to introduce in a module and doesn’t harm. To introduce is a matter of changing code like: class ntp ( ) { contain ntp::install contain ntp::config contain ntp::service Class['::ntp::install'] -> Class['::ntp::config'] ~> Class['::ntp::service'] } to something like: class ntp ( String $install_class = '::ntp::install', String $config_class = '::ntp::config', String $service_class = '::ntp::service', ) { contain $install_class contain $config_class contain $service_class Class[$install_class] -> Class[$config_class] ~> Class[$service_class] } Will users ever use such parameters to provide their own classes? In many cases never, but adding them requires low efforts and makes the module more adaptable to special cases. Alessandro Franceschi 2017-07-31T00:00:00+00:00 https://example42.com/blog/2017/07/31/sort_merged_array-fix With Puppet 4.10.5 and 5.0.2 a fix for a missing functionality was released: sort_merged_arrays lookup option. In earlier versions of Puppet this was named ‘sort_merge_arrays’ but lacked the functionality. Unluckily Puppet did not build the fix with backward compatibility. When using ‘sort_merge_arrays’ the following will happen: Puppet 4.10.4/5.0.1 or earlier: success - but no functionality Puppet 4.10.5/5.0.2 or later: compiler failure When using ‘sort_merged_arrays’ the following will happen: Puppet 4.10.4/5.0.1 or earlier: compiler failure Puppet 4.10.5/5.0.2 or later: success - with functionality How to work around this version dependent functionality when providing code for multiple Puppet versions? We have a pull request on GitHub with a really ugly hack: identify version of Puppet and set a variable use this variable as a hierarchy in hiera.yaml add new hierarchy with corrected spelling of sort_merged_arrays Let’s look at details: manifests/site.pp # Puppet 4.10.4 and older had a typo in sort_merged_array lookup option # to allow functinonality we add a new hierarchy to profile hiera.yaml where we use the wrong name if versioncmp('4.10.4', $facts['puppetversion']) >= 0 { $fix_sort_merge = '4' } else { $fix_sort_merge = undef } site/profile/hiera.yaml --- version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: "In module hierarchy" paths: - "%{facts.virtual}.yaml" - "%{facts.os.name}-%{facts.os.release.major}.yaml" - "%{facts.os.name}.yaml" - "%{facts.os.family}-%{facts.os.release.major}.yaml" - "%{facts.os.family}.yaml" - "common%{fix_sort_merge}.yaml" - "common.yaml" site/profile/data/common.yaml lookup_options: "^profile::(.*)::(.*)_hash$": merge: strategy: deep knockout_prefix: "--" sort_merged_arrays: true "^profile::(.*)::(.*)::(.*)_hash$": merge: strategy: deep knockout_prefix: "--" sort_merged_arrays: true "^profile::(.*)::(.*)_list$": merge: strategy: deep knockout_prefix: "--" sort_merged_arrays: true "^profile::(.*)::(.*)::(.*)_list$": merge: strategy: deep knockout_prefix: "--" sort_merged_arrays: true site/profile/common4.yaml --- lookup_options: "^profile::(.*)::(.*)_hash$": merge: strategy: deep knockout_prefix: "--" sort_merge_arrays: true "^profile::(.*)::(.*)::(.*)_hash$": merge: strategy: deep knockout_prefix: "--" sort_merge_arrays: true "^profile::(.*)::(.*)_list$": merge: strategy: deep knockout_prefix: "--" sort_merge_arrays: true "^profile::(.*)::(.*)::(.*)_list$": merge: strategy: deep knockout_prefix: "--" sort_merge_arrays: true All this is already part of a PR for PSICK. We are interested to learn about any other - less ugly - solution. We wish successful unit and integration tests on your control-repositories. Martin Alfke 2017-07-24T00:00:00+00:00 https://example42.com/blog/2017/07/24/spec-testing-control-repo Spec tests are a common way to test whether your Puppet code either has no errors causing the compiler to stop or whether your Puppet code does things in the right way. Usually spec tests are done at different levels: Lint testing Unit Testing Acceptance Testing Lint tests check if the Puppet code follows the Puppet style guide. Within unit tests we use rspec-puppet. Rspec-puppet compiles a Puppet catalog in a sandbox and checks if the catalog is compiled successfully and contains all required resources. Acceptance tests are used to deploy a machine, apply the Puppet code and verify system settings. For acceptance testing we use beaker which is a wrapper around vagrant or docker and serverspec. Lint tests are easy to deploy and run. All you need is the following files: Gemfile Rakefile We are re-using the puppetlabs_spec_helper gem as this brings most what we need. # Gemfile source ENV['GEM_SOURCE'] || "https://rubygems.org" gem 'puppetlabs_spec_helper' # optional lint extensions (see https://voxpupuli.org/plugins/#puppet-lint) gem 'puppet-lint-appends-check', :git => 'https://github.com/voxpupuli/puppet-lint-appends-check.git', :require => false gem 'puppet-lint-classes_and_types_beginning_with_digits-check', :git => 'https://github.com/voxpupuli/puppet-lint-classes_and_types_beginning_with_digits-check.git', :require => false gem 'puppet-lint-empty_string-check', :git => 'https://github.com/voxpupuli/puppet-lint-empty_string-check.git', :require => false gem 'puppet-lint-file_ensure-check', :git => 'https://github.com/voxpupuli/puppet-lint-file_ensure-check.git', :require => false gem 'puppet-lint-leading_zero-check', :git => 'https://github.com/voxpupuli/puppet-lint-leading_zero-check.git', :require => false #gem 'puppet-lint-numericvariable', # has issues with new puppet-lint release # :git => 'https://github.com/fiddyspence/puppetlint-numericvariable.git', # :require => false gem 'puppet-lint-resource_reference_syntax', :git => 'https://github.com/voxpupuli/puppet-lint-resource_reference_syntax.git', :require => false gem 'puppet-lint-spaceship_operator_without_tag-check', :git => 'https://github.com/voxpupuli/puppet-lint-spaceship_operator_without_tag-check.git', :require => false gem 'puppet-lint-trailing_comma-check', :git => 'https://github.com/voxpupuli/puppet-lint-trailing_comma-check.git', :require => false gem 'puppet-lint-undef_in_function-check', :git => 'https://github.com/voxpupuli/puppet-lint-undef_in_function-check.git', :require => false gem 'puppet-lint-unquoted_string-check', :git => 'https://github.com/voxpupuli/puppet-lint-unquoted_string-check.git', :require => false gem 'puppet-lint-variable_contains_upcase', :git => 'https://github.com/fiddyspence/puppetlint-variablecase.git', :require => false gem 'puppet-lint-version_comparison-check', :git => 'https://github.com/voxpupuli/puppet-lint-version_comparison-check.git', :require => false Within the Rakefile you must enable the puppet-lint rake task: # Rakefile require 'puppetlabs_spec_helper/rake_tasks' In the spec/spec_helper.rb file we enable the puppetlabs_spec_helper module spec helper: # spec/spec_helper.rb require 'puppetlabs_spec_helper/module_spec_helper' Puppet-lint will check for a manifests and modules directory to read puppet manifests and checks for style guide. But within a control-repository the files to test are not inside the modules directory, but inside the site directory. As we can not overwrite this default behavior we generate a new lint rake task in the Rakefile: # Rakfile require 'puppetlabs_spec_helper/rake_tasks' exclude_paths = %w( vendor/**/* spec/**/* modules/**/* pkg/**/* tests/**/* ) Rake::Task[:lint].clear PuppetLint::RakeTask.new(:lint) do |config| # Pattern of files to ignore config.ignore_paths = exclude_paths # Pattern of files to check, defaults to `**/*.pp` config.pattern = ['manifests/**/*.pp', 'site/**/*.pp'] # List of checks to disable config.disable_checks = ['140chars', 'relative', 'class_inherits_from_params_class'] # Should the task fail if there were any warnings, defaults to false config.fail_on_warnings = true # Print out the context for the problem, defaults to false #config.with_context = true # Log Format #config.log_format = '%{path}:%{line}:%{check}:%{KIND}:%{message}' end Unit tests need to know where to find the upstream modules which we have in Puppetfile within the control-repo. We don’t fetch these from upstream source as this would need to have Puppetfile and .fixtures.yml files synced or either one automatically generated. Instead we have chosen to re-use the modules which must be installed using r10k: r10k puppetfile install -v Within the spec/spec_helper.rb file we set the modulepath to ‘site’ and ‘modules’: fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures')) RSpec.configure do |c| c.module_path = File.join(fixture_path, 'modules/site') + ':' + File.join(fixture_path, 'modules/r10k') c.manifest_dir = File.join(fixture_path, '../../manifests') c.manifest = File.join(fixture_path, '../../manifests/site.pp') c.hiera_config = File.join(fixture_path, '../../hiera.yaml') c.fail_fast = true end The .fixtures.yml just ensures that all directories are in place: fixtures: symlinks: site: "#{source_dir}/site" r10k: "#{source_dir}/modules" Now rspec-puppet needs a test. The most simple one just checks if a catalog is successfully created: # spec/classes/profile_apache_spec.rb describe 'profile::apache' do context 'catalog compile' do it { should compile.with all_deps } end end Next we want acceptance tests. Usually beaker was created to run acceptance tests on modules. Modules have a multiple tests running on supported operating systems. Beaker reuses a VM it has created for all tests. Within a control-repo we want a fresh state on every test, as we have single tests which should run on a fresh os every time. First we need the beaker gem: # Gemfile group :system_tests do gem 'beaker' gem 'beaker-rspec' end In Rakefile we disable the default beaker task. Next we generate a new task which will iterate over our acceptance tests: # Rakefile Rake::Task[:beaker].clear RSpec::Core::RakeTask.new(:beaker) do |config| puts 'dont use beaker, use beaker_roles:<role> or all_roles instead' abort end namespace :beaker_roles do Dir.glob("spec/acceptance/*_spec.rb") do |acceptance_test| test_name = acceptance_test.split('/').last.split('_spec').first RSpec::Core::RakeTask.new(test_name) do |t| t.rspec_opts = ['--color'] t.pattern = acceptance_test end end end To allow all tests running in parallel we generate a multitask in Rakefile: all_roles = [] Rake.application.in_namespace(:beaker_roles) do |beaker_roles_namespace| beaker_roles_namespace.tasks.each do |beaker_roles_tasks| all_roles << beaker_roles_tasks end end multitask :all_roles => all_roles All this is already part of PSICK We wish successful unit and integration tests on your control-repositories. Martin Alfke 2017-07-17T00:00:00+00:00 https://example42.com/blog/2017/07/17/psick-explained Using PSICK for the very first time confuses people as they are confronted with a highly flexible approach for managing and configuring their infrastructure. This posting will give you a guidance on how to read, understand and use PSICK. Normally system administrators strictly follow the approach of writing roles and profiles which consist of lots of Puppet code, describing the infrastructure. With PSICK one has the possibility of using or adopt a predefined role/profile pattern. Let’s get to the content: why PSICK set up NTP configure SSH manage users Why PSICK? Where do you usually start when working with Puppet? Do you really want to start from scratch, re-inventing the wheel? Do you really want to start with an empty control-repository and start coding your implementations? Why not just take what is already there and adopt it to your needs? You will very fast recognize that it is far more easy to remove unneeded items instead of writing everything from scratch. This is the reason why example42 provides the Puppet Systems Infrastructure Construction Kit PSICK which has many implementations already included. On most configuration items you only have to provide hiera data which describe your desired setup and your infrastructure. Setup NTP The classical way Usually people classify the puppetlabs ntp module: class profile::time ( $servers, ){ class { '::ntp: servers => $servers, } } The $servers parameter will cause an automatic data binding lookup into hiera and then the puppetlabs ntp module gets declared with data from your infrastructure. This module already does a lot more than only installing ntp and setting ntp servers. It will overwrite the existing ntp.conf file and even start the ntpd service. But how about Windows systems? The puppetlabs ntp module is not suitable for Windows systems. This requires you to find another module which is capable of managing time server settings on Windows. How about RedHat 7 which uses chrony instead of ntpd? It is always up to you to provide proper suited profiles for all of your infrastructure systems. The PSICK way Within PSICK there’s a default profile for time settings (profile::time). This profile uses facter variables to identify which OS should get configuration and uses parameters for flexible usage either regarding the desired tools (chrony, ntpd, ntpdate). The PSICK profile::time class already is an implementation class which uses upstream modules, like puppetlabs ntp. There is no need for you to write code, you just need to provide data in hiera: profile::time::servers: - '1.2.3.4' Your PSICK way The above is PSICK default profile for time management, configured on Hiera with: profile::base::linux::time_class: '::profile::time' profile::base::windows::time_class: '::profile::time' but PSICK is about choice and customisation, you can can use any other class to manage ntp settings in your OS, both component modules or profiles, for example: profile::base::linux::time_class: '::ntp' profile::base::windows::time_class: '::profile::time::windows' Additional parameters to configure time depend on the used class. So, for example, with puppetlabs/ntp module we can configure on Hiera: ntp::servers: - '1.2.3.4' ntp::restrict: - 'default ignore' - '-6 default ignore' - '127.0.0.1' Configure SSH The classical way Most of upstream module development not only install services, but build a configuration file and manage the service. Some people might be happy with this approach, but what if you want to provide a sshd_config file or template by yourself? How to use this approach on existing infrastructure (brownfield). In this case you must review the upstream module whether it allows overwriting default settings. If this is not possible you can not use the upstream module. Usually people start writing an implementation by themselves: class profile::ssh ( $template, ){ package { 'openssh-server': ensure => present, } file { '/etc/ssh/sshd_config': ensure => file, content => epp('profile/ssh/sshd_config.epp'), } } The PSICK way TinyPuppet is a module from example42 which allows you to easily manage installation of applications and their configuration files. With TinyPuppet it is possible to tell Puppet to just install the application we want, then it’s up to us to provide templates and data for our configuration files. This is currently the default profile used in PSICK to manage OpenSSH. It only uses defines for Tiny Puppet OpenSSH installation, optionally uses a template to use for sshd_configuration or even a static source for the whole main configuration directory. class profile::ssh::openssh ( Enum['present','absent'] $ensure = 'present', Variant[String[1],Undef] $config_dir_source = undef, String $config_file_template = '', ) { $options_default = { } $options_user=hiera_hash('profile::ssh::openssh::options', {} ) $options=merge($options_default,$options_user) ::tp::install { 'openssh': ensure => $ensure, } if $config_file_template != '' { ::tp::conf { 'openssh': ensure => $ensure, template => $config_file_template, options_hash => $options, } } ::tp::dir { 'openssh': ensure => $ensure, source => $config_dir_source, } } Within your profile hiera data one only needs to specify which template should be used: profile::ssh::openssh::config_file_template: 'profile/ssh/sshd_config.erb' And the eventual hash of data, which might be used in the template: profile::ssh::openssh::options: PermitRootLogin: prohibit-password PrintLastLog: yes UseLogin: no In our template, to be placed in profile/templates/ssh/sshd_config.erb, in this case in erb format, we would have something like: PermitRootLogin <%= @options['PermitRootLogin'] > PrintLastLog <%= @options['PrintLastLog'] > UseLogin <%= @options['UseLogin'] > Your PSICK way If such freedom and flexibility to manage the content of sshd_config does not satisfy us, we as usual can provide alternative approaches, specifying alternative modules or profiles to manage SSH: profile::base::linux::ssh_class: '::ssh' Manage users The classical way We still see platforms where users are not kept inside a central user management but are configured locally. What you usually learn in every training is that you use a self defined resource type for wrapping several resources together: define profile::usermanagement ( $passwd = undef, ){ File { owner => $title, group => $title, } group { $title: ensure => present, } user { $title: ensure => present, } file { "/home/${title}": ensure => directory, mode => '0750', } file { "/home/${title}/.ssh": ensure => directory, mode => '0700', } } Other implementations might use puppetlabs/accounts module. The PSICK way Within PSICK we have defined a set of self defined resource types. These are not part of the implementation profile, as we believe that these are generic to use. Self defined resource types, custom facts and custom functions are kept inside the tools module. Adding users with PSICK just requires to read and adopt our tools::user::managed self defined resource type to your needs and add hiera data. profile::users::static::managed_users_hash: 'tom': uid : '1002' homedir : '/home/tom' id_rsa_source : 'puppet:///modules/profile/users/tom/id_rsa' id_rsa_pub_source: 'puppet:///modules/profile/users/tom/id_rsa.pub' 'ben': uid : '1003' homedir : '/home/ben' id_rsa_source : 'puppet:///modules/profile/users/ben/id_rsa' id_rsa_pub_source: 'puppet:///modules/profile/users/ben/id_rsa.pub' Your PSICK way Users are always a sensitive thing. Everybody manages them in their own way. The sample default, static, use management profile won’t fit the needs of many, but as usual we have choice: Hiera driven choice of what class to use to manage Users: profile::base::linux::users_class: '::sssd' profile::base::windows::users_class: '::domain_membership' Being managed via Hiera the same name of the class to use, we can exploit to nicely manage exceptions (some servers or a specific one might need a totally different way to manage users), or we can test different modules each one with its own set of parameters: profile::base::linux::users_class: '::accounts' We wish everybody fun with adopting and using PSICK. If you’d rather have this adopted and adapted for you than work through it alone, that’s what Puppet Infrastructure Kickstart is for. Martin Alfke 2017-07-10T00:00:00+00:00 https://example42.com/blog/2017/07/10/puppet-and-fabric Fabric is a remote execution tool, written in Python, which ease parallel execution and orchestration of commands on different nodes. Why does it matter with Puppet? Because it’s a good candidate, and not a rare choice, to trigger Puppet runs (and more) from a central location. In PSICK, our Puppet control-repo [generator], we use it for several tasks tasks which are related to the whole Puppet code workflow, from development, to testing and deployment. We can install Fabric, as common with Python software, using pip: pip install fabric Once installed we have at disposal the fab executable, which reads a file called fabfile.py to get the list of available Fabric commands. In PSICK we gathered different fab files for different purposes in a single directory give them a look for an idea of their format. Being written in Python they can have a much more complex and flexible content. To list the available commands in the local fabfile (or directory) we can type: fab -l Fabric commands can be executed locally on remote nodes. On remote nodes it uses SSH for connection. There are various ways to define the list of remote nodes where to execute a given command, the simplest one is probably to specify them directly in the command line with the -H argument followed by a comma separated list of nodes or using the :host argument: fab <task>[:host=<hostname>][:option=value] fab [-H <hostname>] <task>[:option=value]> Note that there are other ways to define and group the nodes to work with, for example by using Fabric roles, refer to the official documentation for details. Typically access to the remote node is done using SSH keys, using the local user for remote authentication, if keys are not used a password is prompted. If local and remote users don’t match, or access to a remote node can’t be direct and requires a jump host, it’s definitively worth adding the relevant nodes in our ~/.ssh/config file, where we define for our nodes, the user to access them, the SSH key to use, eventually a jump host and so on. Syntax for SSH client file is something like: Host mon ProxyCommand ssh -A -x -W %h:%p bastion.aws.example.com 2> /dev/null ForwardAgent yes User ec2-user Hostname 10.10.2.160 IdentityFile ~/.ssh/aws.pem In this way we can connect to this host, with all the correct configurations, with ssh mon, or, when using Fabric, fab -H mon. Fabric on PSICK We mentioned PSICK and its integration with Puppet, the list of available commands is not short: al@mule psick [development] $ fab -l Available commands: aws.apply [local] Run puppet apply locally using the specified role (default: aws) aws.setup [local] Install locally the aws cli environment aws.status [local] Show AWS resources on one or all regions docker.purge [local] Clean up docker images and containers (CAUTION) docker.rocker_build_role [local] WIP Rockerize a role on all or the specified image OS (data in hieradata/role/$puppetrole.yaml) docker.setup [local] Install locally Docker (needs su privileges) docker.status [local] Show Docker status info docker.test_role [local] Test a role on the specified OS on a Docker image docker.tp_build_role [local] Dockerize a role based on tp on all or the specified Docker (data in hieradata/role/$puppetrole.yaml) facter.set_external_facts [remote] Set the given external facts in /etc/puppetlabs/facter/facts.d facter.set_trusted_facts [remote] Set the given trusted facts in /etc/puppetlabs/puppet/csr_attributes.yaml git.checkout_master [local] Run git checkout master on each on the installed modules git.install_hooks [local] Install Puppet .git/hooks git.setup_new_repo [local] Create a new repo from scratch, based on the current contents of this control-repo git.status [local] Run git status on this repo and the installed modules puppet.agent [remote] Run puppet agent puppet.agent_noop [remote] Run puppet agent in noop mode puppet.apply [remote] Run puppet apply on the deployed control-repo (uses control-repo in the environments/production dir) puppet.apply_noop [remote] Run puppet apply in noop mode (needs to have this control-repo deployed) puppet.check_syntax [local] Check the syntax of all .pp .erb .yaml files in the contro-repo puppet.current_config [remote] Show currently applied version of our Puppet code puppet.deploy_controlrepo [remote] Deploy this control repo on a node (Puppet has to be already installed) puppet.install [remote] Install Puppet 4 on a node (for Puppet official repos) puppet.lint [local] Run puppet-lint on all site manifests. Eventually fix them puppet.module_generate [local] Generate a Puppet module based on skeleton puppet.module_publish [local] Publish on GitHub and the Forge the local version of a module puppet.remote_setup [remote] Installs on a remote node the packages needed for a puppet apply run on the control-repo puppet.setup [local] Setup the contro-repo, installs r10k and external modules puppet.sync_and_apply [remote] Run puppet apply on a synced copy of the local git repo (syncs and uses control-repo) tp.clone_data [local] Add a new app name data directory under modules/tinydata, based on the specified source tp.install [local] Install locally any tinydata knows app via tp tp.remote_test [remote] WIP Run tp tests on remote node vagrant.destroy [local] Destroy the specified vm vagrant.env_status [local] Run vagrant status on all or the specified environments vagrant.halt [local] Halt all or the specified Vagrant vm vagrant.node_test [local] Run existing and testing Puppet code on a VM vagrant.provision [local] Provision all or the specified vm vagrant.reload [local] Reload all or the specified vm vagrant.resume [local] Resume all or the specified vm vagrant.setup [local] Install locally Vagrant and the needed plugins vagrant.status [local] Show status of all or the specified vm vagrant.suspend [local] Suspend all or the specified vm vagrant.up [local] Vagrant up the specified vm Many of these commands are executed locally (and just wrap simple shell commands present in PSICK’s bin/ directory), but there are some intended to be used on remote nodes. For example, to install Puppet on one or more remote nodes we can run: fab puppet.install -H host1,host2 To run puppet agent in noop mode on all the known hosts (as defined in fabiles, or in the environment): fab puppet.agent_noop To run puppet agent on a specific node: fab puppet.agent -H web01.example.test To run in apply mode the local code on a remote node (code is rsynced and then compiled on the remote node, eventual eyaml keys and first copied and then removed):. fab puppet.sync_and_apply Local Puppet activities with PSICK Local commands are, generally, not ommon in Fabric, as the tools is supposed to be used for remote execution, still in PSICK there are several commands available to support us in our Puppet code workflow. For example, to install useful git hooks for Puppet development. By default downloaded from (https://github.com/drwahl/puppet-git-hooks)[https://github.com/drwahl/puppet-git-hooks]: fab git.install_hooks To generate a new module based on the format of PSICK’s skeleton directory. fab puppet.module_generate To check the git status of the main control-repo and of each module in modules: fab git.status To check the syntax of all .pp .yaml .epp .erb files in our control-repo: fab puppet.check_syntax To publish the local version of a module in modules/ dir to Forge and GitHub (puppet-blacksmith setup and access to remote git repo required): fab puppet.module_publish:<module_name> To test a role (as defined in hieradata/role/$role.yaml) with Docker on different OS base images: fab docker.test_role:<role>,<image> fab docker.test_role:log,ubuntu-14.04 Available images are: ubuntu-12.04, ubuntu-14.04, ubuntu-14.06, centos-7, debian-7, debian-8, alpine-3.3. These are just examples, give a look to the list of available commands for more, and use the -d argument to show a list of available arguments: al@mule psick [development] $ fab -d docker.test_role Displaying detailed information for task 'docker.test_role': [local] Test a role on the specified OS on a Docker image Arguments: puppetrole='docker_test_role', image='centos-7' The fabfiles on PSICK are rather basic, but much more can be do, list of nodes can be automatically queried to AWS or PuppetDB, commands can be the result of more or less Python code, which may trigger remote backups, check for systems status, perform database operations, applications deployment and so on. So, even if not strictly necessary (not even in PSICK), Fabric can be a good companion to Puppet and generally to the whole operations. It’s usage inside a control-repo may give to it a whole new meaning, which goes further than a “simple” central repository for Puppet code and data, and may become the single place from where the whole infrastructure can be provisioned, configured, controlled, and managed. The limit is our imagination. Alessandro Franceschi 2017-07-03T00:00:00+00:00 https://example42.com/blog/2017/07/03/node-classification-options When using a Puppet Agent - Puppet Master setup it is required that the Puppet Master has information about classes which a node should receive. This process is called Node Classification. There are several possible ways on how to do this. This posting will cover several different options Node Classification in Puppet environment manifests Node classification in Hiera Node classification on external sources Node Classification in Puppet environment manifests The most cenvenient way to classify nodes is using Puppet environment manifests data. This is the default behavior for any Puppet Open Source installation. It uses the manifests directory structure which is located on the root of a Puppet Environment code basis. /etc/puppetlabs/code/environments/production |- manifests/ | |- site.pp | |- infrastructure/ | | |- internal_servers.pp | | \- workstations.pp | \- servers.pp \- modules/ The site.pp file is parsed first. Any other files and directory are parsed in directory globbing order. Usually the site.pp file is used to declare resource defaults and has a fallback default node classification. # /etc/puppetlabs/code/environments/production/manifests/site.pp # file backup should be local on node. backup file should be next to replaced file and has the fileending .puppet_backup File { backup => '.puppet_backup', } # On windows we use chocolatey as dfault provider if $facts['os']['family'] == 'windows' { Package { provider => 'chocolatey', } } # default node should always fail node default { fail("Missing node classification for node ${trusted['certname']}") } All other nodes are then placed into other files or directories. It is common best practice to classify nodes either by their business use case (role) or by implementation classes (profiles) node 'www01.example.com' { include role::company_website } node 'mail.example.com' { include profile::infrastructure::security include profile::infrastructure::ldap_client include profile::mail::postfix::mda include profile::mail::cyrus } Node classification in Hiera Another possible solution is using hiera data. Usually the hiera lookup is added to site.pp - either in a default node or at global position. # /etc/puppetlabs/code/environments/production/manifests/site.pp # profiles are read as array from hiera on all matching hierarchies $profiles = lookup('profiles', Array, [], 'deep') # use Puppet 4 lambda for array iteration and declare each profile $profiles.each |String $profile| { include "profile::${profile}" } The hiera lookup mentioned in the example above checks all valid hierarchies from a node and collects all findings of the ‘profiles’ key into an array. Think about the following example: # common.yaml profiles: [] # (%{datacenter}) - infrastructure.yaml profiles: - 'infrastructure::security' - 'infrastructure::ldap_client' # (%{application}) - mailserver.yaml profiles: - 'mail::postfix::mda' - 'mail::cyrus' Node classification on external sources The next possible solution is to use an external source. In Puppet this is called an ENC (external node classifier). An ENC needs some configuration on Puppet Server: # /etc/puppetlabs/puppet/puppet.conf [master] node_terminus = exec external_nodes = <full path to executable script> Don’t forget to restart your Puppet Server process to activate the new settings. The external node script can be anything (perl, ruby, python, compiled C++ code) and must be executable by the user running the Puppet Server (puppet on Open Source, pe-puppet on Puppet Enterprise). Please be careful when using remote data sources like databases, webservers, CMDB. The Puppet Server will access the remote systems every time a node requests a catalog. On larger installations this can lead to many requests against remote systems. It is best practice to use local stored data only. The ENC script must produce YAML or JSON or empty output. classes: - profile::infrastructure::security - profile::infrastructure::ldap_client - profile::mail::postfix::mda - profile::mail::cyrus There is one more thing which an ENC is capable of: environment enforcement. Different to the manifests based node classification - which already takes part inside the Puppet compiler - the ENC is running prior the compiler. Additional information on this topic can be found at TOW 15. Martin Alfke 2017-06-26T00:00:00+00:00 https://example42.com/blog/2017/06/26/psick-developer-environment-setup Working on Puppet code with least possible effort requires a properly setup of a suitable workstation environment. Usually everybody starts playing around with different sets of utilities and tools, learning the hard way about the best usable setup. This article will guide you to a setup, providing a proper basis for initially working with PSICK, Puppet or Ruby in general. Puppet Development is based on Ruby. The first thing we need is a Puppet recommended Ruby version. As version can change from release to release, therefor we need some flexibility on Ruby versions being available. Puppet provides a website with information on Component versions in recent Puppet Enterprise releases. More information regarding ruby development for Puppet is mentioned in the System Requirements - Section Prerequisites. The usually preferred method uses Ruby in user space. This allows work without super user privileges. There are two possible solutions: rvm rbenv Within this posting we will use rbenv. System preparation As we might need to install ruby from source, several development and application packages are required: # RedHat/CentOS: yum install -y git gcc make bzip2 yum install -y openssl-devel readline-devel zlib-devel gcc-c++ # Debian/Ubuntu: apt-get install -y git gcc make bzip2 apt-get install -y libssl-dev libreadline-dev zlib1g-dev g++ On OS X and macOS Xcode installation is required. Then install the Command Line Tools: xcode-select --install Installation of rbenv Installation of rbenv is done as a non-root user First we clone the rbenv github repository: git clone https://github.com/rbenv/rbenv.git ~/.rbenv Now we can compile shell extensions (this step is optional) cd ~/.rbenv && src/configure && make -C src Now we add the rbenv executable path to our PATH environment variable: # RedHat/CentOS/OS X/macOS: echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile # Debian/Ubuntu: echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc For the next step we need output from rbenv, which we will add to our shell: ~/.rbenv/bin/rbenv init The output: # RedHat/CentOS/OS X/macOS: # Load rbenv automatically by appending # the following to ~/.bash_profile: eval "$(rbenv init -)" # Debian/Ubuntu: # Load rbenv automatically by appending # the following to ~/.bashrc: eval "$(rbenv init -)" Follow the provided information and add the mentioned line to your shell environment. Next we need to take care on possible ways to install ruby versions. This is not part of rbenv directly but placed into a separate repository which delivers an extension to rbenv: ruby-build Just run the following git command will place the code into proper location: git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build Remember to refresh your shell: exec bash Verify functionality of rbenv: rbenv version This should return the following output: system (set by /home/tuxmea/.rbenv/version) Install ruby versions Puppet 4 uses Ruby 2.1.9, Puppet 5 uses Ruby 2.4.1 Install both ruby version by running the following command: rbenv install 2.1.9 Verify installation of new ruby version: $rbenv versions 2.1.9 Install ruby basic extensions Now we activate the ruby version to install some basic ruby extensions which are required for further development: rbenv shell 2.1.9 gem install bundler wirble pry rbenv shell --unset Bundler is used to install ruby extensions required by some development into another path. This allows you to run development and testing even when there is version mismatch between some application you are working on. Wirble is an irb (interactive ruby shell) extension which offers syntax highlighting and tab completion. Pry is a ruby debugger. During development one can set breakpoints where pry will open an irb session within the running application. Configuration for wirble irb extension is done in ~/.irbrc require 'rubygems' require 'wirble' Wirble.init Wirble.colorize Wirble and pry are not required for Puppet testing, but recommended extensions for development of custom facts, functions, types or providers. Repeat the steps mentioned above with ruby version 2.4.1 to be prepared for Puppet 5 code testing! GIT Prompt As we are working on a GIT repository, it is highly recommended to have an informative shell prompt delivering information about your actual repository state. For bash shells one wants to check git bash prompt, for zsh shells one might consider using oh my zsh. For git bash prompt the following steps are required: cd ~ git clone https://github.com/magicmonty/bash-git-prompt.git .bash-git-prompt --depth=1 Now enable the git bash prompt in your shell: # RedHat/CentOS/OS X/macOS: cat <<- EOF >> ~/.bash_profile # git bash prompt GIT_PROMPT_ONLY_IN_REPO=1 source ~/.bash-git-prompt/gitprompt.sh EOF # Debian/Ubuntu: cat << EOF >> ~/.bashrc # git bash prompt GIT_PROMPT_ONLY_IN_REPO=1 source ~/.bash-git-prompt/gitprompt.sh EOF Don’t forget to reinitialize your shell exec bash Start working on PSICK Now clone the PSICK repository: git clone https://github.com/example42/psick.git When changing into the psick directory the default ruby version is selected automatically by using content from .ruby-version file. Additionally you will se the default branch at the shell prompt: [mea@puppet ~]$ [mea@puppet ~]$ cd psick/ ✔ ~/psick [production L|✔] Now it is possible to install all ruby gems required for testing into a separate path: bundle install --path vendor Next you can run tests. Tests are not yet complete but under development. # e.g. # get puppet version used for testing bundle exec puppet --version # list all rake tasks bundle exec rake -T # install modules from Puppetfile bundle exec r10k puppetfile install -v Happy testing, developing and puppetizing. Martin Alfke 2017-06-19T00:00:00+00:00 https://example42.com/blog/2017/06/19/control-repo-documentation The current method to manage documentation in Puppet code is based on puppet strings. We can install it as a gem: puppet resource package puppet-strings provider=puppet_gem and then, from within the main directory of a module we can generate the module documentation just by running: puppet strings Puppet strings uses Yard, a Ruby Documentation Tool that parses comments with defined @tags in the code and creates documentation based on them. For example,we can document a class by adding lines as the following at the beginning of its manifest: # @summary This class manages the general hardening of a system. # The class provides, as params, the names of the classes to include in # order to manage specific hardening activities. # # @example Define all the available hardening classes. Set a class name to an # empty string to avoid to include it # profile::hardening::pam_class: '::profile::hardening::pam' # profile::hardening::packages_class: '::profile::hardening::packages' # profile::hardening::services_class: '::profile::hardening::services' # profile::hardening::tcpwrappers_class: '::profile::hardening::tcpwrappers' # profile::hardening::securetty_class: '::profile::hardening::securetty' # profile::hardening::network_class: '::profile::hardening::network' # # @param pam_class Name of the class to include to manage PAM # @param packages_class Name of the class where are defined packages to remove # @param services_class Name of the class to include re defined services to stop # @param securetty_class Name of the class where /etc/securetty is managed # @param tcpwrappers_class Name of the class to include to manage TCP wrappers # @param network_class Name of the class where some network hardening is done There are various tags which Yard automatically detects and uses to compose documentation, here is the full list, the most used for Puppet are: @summary - A brief summary of what the class does @param - The description of what parameter does. Puppet strings automatically detects the accepted Type for the param (if set in the class) and the default value (currently the default value is not shown if we use data in module and values are set in hiera's Yaml files) @example - To should usage examples. Outour will be shown in a monospaced font. @return - In functions, defines what the function returns. Puppet strings can parse and create documentation for manifests (classes, user defines, functions in Puppet DSL), types and providers, and any piece of ruby code in our files. The use case for a single module shown before generates the documentation for that module files and uses the README.md one as content for the main page. When we want to generate documentation for a whole control repo, we need to pass some more options. If we have our local modules in the site directory and we want to generate documentation only for them, we can run a command like: puppet strings generate site/**/**/*{.pp\,.rb} site/**/**/**/*{.pp\,.rb} This can generate something like the PSICK documentation that we automatically generate in our PSICK Puppet CI pipelines. It’s possible to use files different than README.md as text for the main page (in the above case the same README.md file used by puppet strings is generated by composing different fragments of documentation), to have the output in different formats and to customise Yard in various ways. To specify extra options for documentation generation we have two alternatives: Use the provided rake task Add a .yardopts file to the main directory of the control-repo. Check here for further info. Documentation is always a multi facets giant to face, when we write code: We hate to write it, but still we want it when we need it and we want it to be complete and updated. At the same time it take time to write it, it takes time to keep it updated and it’s really hard to make it complete, easy to follow and fitting for different kind of users with different knowledge. If it’s written in a dedicated, separated place (a Wiki, a document file somewhere, an email(!?)… ) it’s probably doomed to become obsolete a few days after it has been written. Having in code comments that can be used to generate contextual, “live” documentation is a pattern present is practically any language. We have it on Puppet too, it’s flexible and nice enough to be worth using and, staying where the code stays, it can help us in making it updated and current. Generating it for the whole control-repo, instead of a single module, during the delivery pipeline of our Puppet code base is just a simple step which we can add while putting together the different dots that draw our Puppet infrastructure. Alessandro Franceschi 2017-06-12T00:00:00+00:00 https://example42.com/blog/2017/06/12/puppet5 Puppet 5 was scheduled to be released somewhere near May 2017. The release was postponed due to some CVE on existing supported Puppet versions. So we expect Puppet 5 to be available quite soon. Therefore it will be soon time to work on Puppet 5 updates. Most important is to mention that Puppet 5 does not require any Puppet code changes. It is still recommended to update to latest Puppet 4 version prior upgrading to Puppet 5. The packages layout and directory structure has not changed. Agent and Master still install into /opt/puppetlabs. Hardware requirements are also identical to Puppet 4. The master now requires Java 8 and has the eyaml gem contained inside the package. PuppetDB now needs PostgreSQL 9.6!! Puppetserver now has a newer JRuby version. This requires to reinstall all server gems! At time of this writing the following versions are bundled: /opt/puppetlabs/puppet/bin/ruby --version ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux] /opt/puppetlabs/puppet/bin/openssl version OpenSSL 1.0.2k 26 Jan 2017 /opt/puppetlabs/puppet/bin/facter --version 4.0.0 (commit 5cdb4319067a740037d81f042330d85711d9a34c) /opt/puppetlabs/puppet/bin/puppet --version 5.0.0 /opt/puppetlabs/puppet/bin/pxp-agent --version 1.5.2 /opt/puppetlabs/bin/puppetdb --version puppetdb version: 5.0.0.SNAPSHOT.2017.06.06T2148 /opt/puppetlabs/bin/puppetserver --version puppetserver version: 5.0.0.master.SNAPSHOT.2017.06.06T0957 /opt/puppetlabs/puppet/bin/mco --version /opt/puppetlabs/puppet/bin/mco 2.10.4 You are encouraged to file bug reports at tickets.puppet.com. Please use the “Affected Version” field and use “PUP 5.0.0” The most easy way to test packages and functionality is by using the nightly builds which are made available since May 9th 2017. The packages can be installed using the following URL’s: OSURL Cisco wind river linux 5https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-cisco-wrlinux-5.noarch.rpm Cisco wind river linux 7https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-cisco-wrlinux-7.noarch.rpm Enterprise Linux 5 (CentOS 5)https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-el-5.noarch.rpm Enterprise Linux 6 (CentOS 6)https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-el-6.noarch.rpm Enterprise Linux 7 (CentOS 7)https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-el-7.noarch.rpm Fedora core 24https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-fedora-24.noarch.rpm Fedora core 25https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-fedora-25.noarch.rpm SUSE Linux 11https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-sles-11.noarch.rpm SUSE Linux 12https://yum.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-sles-12.noarch.rpm Cumulus Linuxhttps://apt.puppetlabs.com/puppet5-nightly-release-cumulus.deb Huawei OShttps://apt.puppetlabs.com/puppet5-nightly-release-huaweios.deb Debian 8 Jessiehttps://apt.puppetlabs.com/puppet5-nightly-release-jessie.deb Ubuntu 14.04 Trustyhttps://apt.puppetlabs.com/puppet5-nightly-release-trusty.deb Debian 7 Wheezyhttps://apt.puppetlabs.com/puppet5-nightly-release-wheezy.deb Ubuntu 16.04 Xenialhttps://apt.puppetlabs.com/puppet5-nightly-release-xenial.deb Windows (64bit)https://downloads.puppetlabs.com/windows/puppet5-nightly/puppet-agent-4.99.0-x64.msi Windows (32bit)https://downloads.puppetlabs.com/windows/puppet5-nightly/puppet-agent-4.99.0-x86.msi OS X 10.10 Yosemitehttps://downloads.puppetlabs.com/mac/puppet5-nightly/10.10/x86_64/puppet-agent-4.99.0-1.osx10.10.dmg OS X 10.11 EI Capitanhttps://downloads.puppetlabs.com/mac/puppet5-nightly/10.11/x86_64/puppet-agent-4.99.0-1.osx10.11.dmg macOS 10.12 Sierrahttps://downloads.puppetlabs.com/mac/puppet5-nightly/10.12/x86_64/puppet-agent-4.99.0-1.osx10.12.dmg Please note that backports are required for Debian Jessie due to Java 8 dependency! Martin Alfke 2017-06-05T00:00:00+00:00 https://example42.com/blog/2017/06/05/puppet-reports-and-metrics Any Puppet infrastructure must be monitored and needs log checking. For monitoring we can re-use existing solutions like cacti, checkmk, icinga, monit, munin, nagios, sensu, zabbix see also Wikipedia. Every puppet agent has locally stored important information for your puppet infrastructure. Most of these informations are stored inside the statedir. puppet agent --configprint statedir /opt/puppetlabs/puppet/cache/state tree /opt/puppetlabs/puppet/cache/state /opt/puppetlabs/puppet/cache/state |- classes.txt |- graphs |- last_run_report.yaml |- last_run_summary.yaml |- resources.txt |- state.yaml \- transactionstore.yaml One wants to check not only for running puppet agent, but also for proper catalog retrieval. This can be achieved by analyzing the last_run_summary.yaml file. For logchecking it is possible to re-use your exiting log infrastructure like syslog-ng, elasticsearch or splunk. Within a basic puppet installation logfiles are written to the logdir setting. puppet master --configprint logdir /var/log/puppetlabs/puppetserver tree /var/log/puppetlabs/puppetserver /var/log/puppetlabs/puppetserver |- masterhttp.log |- puppetserver-access.log \- puppetserver.log Most important is the puppetserver.log files deprecation messages. These messages inform you of upcoming incompatibilities with your puppet code. grep deprecated /var/log/puppetlabs/puppetserver/puppetserver.log 2017-05-28 13:29:30,563 WARN [clojure-agent-send-pool-0] [puppetserver] Puppet Support for ruby version 1.9.3 is deprecated and will be removed in a future release. See https://docs.puppet.com/puppet/latest/system_requirements.html#ruby for a list of supported ruby versions. 2017-05-28 13:31:05,084 WARN [qtp1631527616-56] [puppetserver] Puppet /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 2017-05-28 13:31:05,105 WARN [qtp1631527616-56] [puppetserver] Puppet Defining environment_data_provider='hiera' in environment.conf is deprecated 2017-05-28 13:31:05,698 WARN [qtp1631527616-56] [puppetserver] Puppet Defining "data_provider": "hiera" in metadata.json is deprecated 2017-05-28 13:31:06,377 WARN [qtp1631527616-56] [puppetserver] Puppet Defining "data_provider": "hiera" in metadata.json is deprecated. It is ignored since a 'hiera.yaml' with version >= 5 is present 2017-05-28 13:31:07,767 WARN [qtp1631527616-56] [puppetserver] Puppet The function 'hiera_hash' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.0/reference/deprecated_language.html Still most of these tools don’t give you an insight on Puppet internal status or provide information about all changes of your infrastructure. This is where the Puppet reporting frontends come into place. There are several possible solutions available: As Puppet Enterprise user you will get the Puppet Enterprise Console installed automatically. On Puppet Open Source no webinterface will get installed automatically. For both platforms it is possible to make use of other open source developments: Puppet Board Puppet Explorer Both webinterfaces require PuppetDB configured as reporting backend. This is easily possible by setting the reports setting in puppet.conf for the puppet server to ‘puppetdb’ [server] reports = puppetdb There is one major difference between PuppetBoard/PuppetExplorer and the Puppet Entrprise Console: PuppetBoard/PuppetExplorer only have read access to PuppetDB. They are not designed to be able to work as External Nodes Classifier. With Puppet 5 there is another possible source of information where you can receive insights of your Puppet server status: the Puppet server metrics backend. This backend was originally available on Puppet Enterprise only and has been ported to Puppet Open Source. The following settings must be activated: #/etc/puppetlabs/puppetserver/conf.d/puppetserver.conf # enable metrics in http-client http-client: { metrics-enabled: true } #/etc/puppetlabs/puppetserver/conf.d/metrics.conf metrics: { # a server id that will be used as part of the namespace for metrics produced # by this server server-id: localhost registries: { puppetserver: { # specify metrics to allow in addition to those in the default list #metrics-allowed: ["compiler.compile.production"] reporters: { # enable or disable JMX metrics reporter jmx: { enabled: true } # enable or disable Graphite metrics reporter graphite: { enabled: true } } } } # this section is used to configure settings for reporters that will send # the metrics to various destinations for external viewing reporters: { graphite: { # graphite host host: "127.0.0.1" # graphite metrics port port: 2003 # how often to send metrics to graphite update-interval-seconds: 5 } } Now you can follow the description from PE site regarding installation of grafana dashbord. If you’d rather have someone review what your reports and metrics are actually telling you, that’s the core of our Puppet Health Check. Martin Alfke 2017-05-29T00:00:00+00:00 https://example42.com/blog/2017/05/29/hiera-5-globs-and-mapped-paths The release of Hiera 5, shipped with Puppet 4.9, has introduced several new features. We have already talked about it in a previous blog post and now we are going to explore new elements which may be quite interesting and useful in some use cases: globs and mapped paths. We already know that in hiera.yaml, when using file based backends, we can define hierarchies using 2 similar keys: path and paths. They are similar as they allow us to specify a path, or an array of paths, where to look for data. Their usage is something like: hierarchy: - name: "Per-node data" path: "nodes/%{trusted.certname}.yaml" - name: "Common data" path: "common.yaml" which, when using the paths key, is equivalent to : hierarchy: - name: "Hiera data" paths: - "nodes/%{trusted.certname}.yaml" - "common.yaml" This is common and known stuff, more or less we have always used this path based approach to define our hierarchies. Now we have some more options, more dynamic and evolved ways to define hierarchies and where to look for data files. One of them is the usage of the glob and globs keys: hierarchy: - name: "Hiera data" glob: "groups/*.yaml" In this case the Hiera lookup is done for each yaml file present in the groups directory, parsed in alphanumerical order. Note that in the above example no variable interpolation is used, but that’s still possible: any fact or variable in the scope can be used in the glob definition. The Ruby glob method is used to map file paths, so the following rules apply: With one asterisk (*) we match any character for a single file. With two asterisks (**) any depth of nested directories is matched. A question mark (?) matches one character. Comma-separated lists in curly braces ({admins,dba}) match any option in the list. Sets of characters in square brackets ([abcd]) match any character in the set. A backslash () escapes special characters. Using globs instead of glob allows us to specify an array of glob patterns, the same logic of paths and path. I’m still trying to figure out good use cases for glob, considering that I don’t personally like long or complex hierarchies. Maybe this could be useful for cases where we want to have different users (or machines) to edit independently different files, or when we want to define the parameters for our classes in different places, for sake of order or simplicity. For example it could be useful to split a file like this in different files, eventually one of each class / profile. Another interesting way to define hierarchies is by using mapped paths key. An example: - name: Applications mapped_paths: [apps, app, "apps/%{app}.yaml"] The mapped_paths key must have an array as argument with three string elements, in the following order: A variable whose value is an array or hash (apps in the example) A temporary variable name to represent each element of the array or hash. This variable name, (app in the example), is used only in the path in this key. A path where that temporary variable can be used in interpolation expressions. With the above example if we had a $apps variable containing an array like ['fe','be','db'] Hiera would lookup for data in the following files (relative to the defined datadir): apps/fe.yaml apps/be.yaml apps/db.yaml What’s the use case for mapped paths? One typical (?) case is when we want to assign to a node one or more roles. Usual practice is to have a role and only a role for each node, but in some cases the concept of role has slightly different nuances and we may want to be able to have more than one role (or equivalent concept) in a node. With the good old path key we imply that for each variable used in the hierarchy there can be only one possible value at a time for a node, and that’s the value used to identify the path of the file with our data. Both globs and mapped_paths allow far more flexible hierarchies, with data which may be looked, in the same hierarchy, in different files according to values of variables (with mapped_paths) or more general wild card or regexp based matches (with globs and glob). Having more options is hardly a negative thing, with Hiera 5 a remarkable new spectrum of alternatives is available for more complex, dynamic and flexible data storing and handling. It’s up to us to understand if we actually need them, but it’s definitively useful to know the possible alternatives, as in some cases they can make our Puppet life better. Alessandro Franceschi 2017-05-22T00:00:00+00:00 https://example42.com/blog/2017/05/22/automated-puppet-infrastructure-setup How often do you reinstantiate your Puppet server infrastructure? How often do you upgrade your Puppet master and the agents? Usually people set up the heart of their Puppet infrastructure in a manual way. From our perspective this is an anti pattern when you manually manage the core of your automation. We believe that automating your automation allows you: to better re-deploy your Puppet infrastructure to manage your Puppet infrastructure by using Puppet gain confidence that you can easily spin up everything from scratch after major outage The example42 PSICK control-repo now allows you to spin up either Puppet Enterprise or Puppet Open Source infrastructures in a fully automated way. Getting started with PSICK to automate your Puppet Open Source Server setup on a bare OS installation Get the code base: mkdir -p /etc/puppetlabs/code/environment/ git clone https://github.com/example42/psick /etc/puppetlabs/code/environments/production cd /etc/puppetlabs/code/environments/production Install Puppet 4: bin/puppet_install.sh This installs Puppet 4 from Puppet Inc. repositories. Prepare your Puppet bin/puppet_setup.sh Installs required gems and fetches remote modules with r10k. The list of modules can be modified in Puppetfile. Understanding classifying your nodes There are two possible options for node classification: Option 1: using hiera data Option 2: using trusted facts Option 1 is enabled in manifests/site.pp per default. Option 2 is available, but deactivated Our intention is to make use of profiles within hiera for node classification. Adding roles on top of profiles adds another layer of complexity which mostly is not required. Profiles usually are parameterized classes which can fetch data from hiera using automatic data binding. Classify your master Adopt settings from hieradata/nodes/puppet.foss.psick.io.yaml # hieradata/nodes/<master fqdn>.yaml --- profiles: - profile::puppet::gems - profile::puppet::foss_server profile::puppet::gems::install_puppetserver_gems: true run puppet to automate the Master setup bin/papply.sh This builds you a fully operational monolithic Puppet master with PuppetDB, storeconfigs and reporting enabled. But how to get your code improvements onto your fresh Puppet master? This is where the profile parameters come into place: The FOSS Puppet master profile allows you to set r10k configurations. At the moment we only support a single r10k repository. Just add proper namespace keys to hiera: # hieradata/nodes/<master fqdn>.yaml profile::puppet::foss_master::r10k_remote_repo: 'git@gitlab.foss.psick.io:/repos/psick.git' You can either specify this setting prior running step 5 or add the setting later and run puppet agent. Summary for automated Puppet Open Source installation add node classification information to hiera place the repo on the new Puppet master bin/puppet_install.sh bin/puppet/setup.sh bin/papply.sh If you’d rather have this built for you than script it yourself, that’s exactly what Puppet Infrastructure Kickstart is for. Martin Alfke 2017-05-15T00:00:00+00:00 https://example42.com/blog/2017/05/15/self-contained-classification Every Puppeteer quite soon in his career has to cope with nodes classification. The inevitable, necessary and useful task of defining which classes to include in which nodes. In Puppet we can do this with various alternatives, which can often coexist. Some of them are based on data or code we have in our control-repo: Using the node statement in our manifests directory files. For each node, eventually matched by RegExps, we can include classes grouped as needed.   Including, conditionally, classes in the main manifests directory based on top scope variables which may come from facts, variables from an ENC, be defined in the same main manifests. Defining in Hiera the classes to include for each node, according the local hierarchy. If using local files, data may be in the hieradata or data directories of our control repo. Others rely on external data sources: Using an External Node Classifier (ENC) such as the console of Puppet Enterprise or The Foreman, or any other which, queried with the node certname, returns a yaml as described in a previous tip. Using LDAP, querying an external LDAP server for the classes to include in a node (Puppet schema is provided) Using Hiera to classify nodes with an external datastore backend. We define self contained a control repo which has all the code and the data needed and necessary to manage our infrastructure: Hiera data, Puppet code in manifests and local modules, list of external modules in Puppetfile and eventually provisioning scripts for different environments such as Vagrant, AWS, Docker. My current own personal preference on how to classify nodes is, more or less, in PSICK, example42’s opinionated, customizable, control repo which by default includes in manifests/site.pp a profile::pre and profile::base class, which are just class containers for other classes to manage, respectively, the resources we want to apply before all the others (package repos, network configs…) and the ones which we want to apply to all our nodes (the common baselines of configurations). On PSICK is possible to customize, via Hiera, the actual classes to include for each component managed in prerequisites and baselines. Hiera is used also to define the classes (profiles) to add for each node according to how different they are. This approach is equivalent to the roles and profiles pattern but instead of defining and declaring role classes, the list of profiles to include is retrieved via Hiera, using the modern equivante to the good old hiera_include: lookup('profiles', Array[String], 'unique', [] ).contain Probably it’s easier to give a look at the site.pp to get an idea. This approach makes it easy to test similar environments during development, testing and CI. It’s also entirely data driven: we can configure the whole infrastructure in Hiera Yaml files, both the classes to include in each node and how they are parametrized. Worth underlining is that all this still relies on some kind of external data source: the facts or the variables derived from the node name. I currently tend to use trusted facts to set such variables during systems provisioning. Also, it’s important to understand that such an approach may, and in some cases has, to adapt to local needs and software, such as installations based on Puppet Enterprise or The Foreman where we can use a web interface to classify nodes. What information has to stay on Hiera, and what on the ENC, depends on single cases, as long as it’s clear, and logic, what is responsible for what, the two worlds can exist. In PE based setups, for example I still prefer to define classification as Hiera Yaml data, and use PE Node manager just to manage the Puppet infrastructure itself (as done by itself), and eventually to create custom local groups, where may be defined top scope variables used in the control repo code and data. As usual, your mileage may vary. Alessandro Franceschi 2017-05-08T00:00:00+00:00 https://example42.com/blog/2017/05/08/a-psick-vagrant-experience PSICK is an opinionated Puppet control-repo with a lot of integrations and tooling to support the Puppeteer during development, testing and operations. One of the most useful integration is the one with Vagrant. We can test our current local changes to Puppet code and data in several different Vagrant environments and VMs. Under vagrant/environments we have various Vagrant environments, fully customisable, where Puppet can be run in agent or apply mode testing directly the effect of our changes on the repo. Here we can test different Operating Systems and Puppet setups, with Puppet Enterprise (PE), OSS Puppet or The Foreman masters. But first, we need to setup a few things. Our new control-repo PSICK is both a control-repo by itself and a generator (at the moment very rough) of control-repos. To create a control-repo for our new, wonderful, green field acme project we can: git clone https://github.com/example42/psick cd psick ./psick create This command allows us to create a new control-repo in a new directory. It asks some questions: the path (absolute or relative to the dir containing psick) where we want to create it if we want to create a bare minimal control-repo or a full featured one, which is the exact copy of the current psick files. if we want to automatically make the first commit on the brand new control-repo with all the added files. Output is something like: ### PSICK is going to create a brand new control-repo ### # Specify the path where you want to create your new Puppet control-repo Provide the full absolute path or the name of a dir that will be created under /Users/al/tmp Press [ENTER] when done. acme # Choose how you want to create your new control-repo 1- Create a full featured control-repo based on current PSICK 2- Create a minimal control-repo with only the bare minimal files Note that you will be able to add or remove components later. Make your choice: 1 # Copying all files from psick to /Users/al/tmp/acme # Initialising git in the new directory Initialized empty Git repository in /Users/al/tmp/acme/.git/ # Showing current status of the new git repo On branch production Initial commit Untracked files: (use "git add <file>..." to include in what will be committed) .codacy.yaml .gitignore .gitlab-ci.yml [...] vagrant/ nothing added to commit but untracked files present (use "git add" to track) # NOTE: master branch has been renamed to production for Puppet compliance # Do you want to make a first commit on the new repo? Press 'y' to commit all the existing files so to have a snapshot of the current repo Press anything else to skip this and take your time to review and cleanup files before your first commit y [production (root-commit) 1c7c6c8] First commit: Snapshot of origin https://github.com/example42/psick (fetch) originhttps://github.com/example42/psick (push) 607 files changed, 107764 insertions(+) create mode 100644 .codacy.yaml create mode 100644 .gitlab-ci.yml [...] ### Congratulations! Setup of the new control-repo finished ### # To start to work on it: cd /Users/al/tmp/acme # Keep updated the psick repo, and use the psick command to update or add componenent to your control-repo So now we can move in the created dir, in my test case: cd /Users/al/tmp/acme git log git status and setup our control repo following the instructions. Prerequisites installation If we still haven’t Puppet installed, we can install it (more or less on any Linux) with: sudo bin/puppet_install.sh Consider that many parts of PSICK use Puppet latest features, optimal would be Puppet version 4.10 or later. Remember, as is, PSICK is intended to be used for greenfield setups or migrations: we are not supposed to use it on existing Puppet control-repo, if not for inspiration, or some code or ideas grabbing. Once a decent Puppet is in place, we have to deploy the modules via r10k, if not already installed, we can install it and some other useful gems with: bin/puppet_setup.sh If we have r10k already installed, we can just run: r10k puppetfile install -v Setup is done, now we can start to play around. Under the vagrant directory we have most of the Vagrant related stuff. We need Vagrant, Virtual Box and some plugins. We can install them all (with the option to skip single steps) with: bin/vagrant_setup.sh We can install the recommended vagrant plugins with: vagrant plugin install vagrant-cachier vagrant plugin install vagrant-vbguest vagrant plugin install vagrant-hostmanager vagrant plugin install vagrant-triggers And, if we want to test a Puppet Enterprise based environment (such as pe, demo, lab) vagrant plugin install pe_build Time to play Once Vagrant is setup with the needed dependencies, we can create some VM. cd vagrant/environments/ostest vagrant status Output here is quite interesting, note all OS work flawlessly out of the box, though. al@mule ostest [production] $ vagrant status Current machine states: centos7.ostest.psick.io not created (virtualbox) centos6.ostest.psick.io not created (virtualbox) ubuntu1604.ostest.psick.io not created (virtualbox) ubuntu1404.ostest.psick.io not created (virtualbox) ubuntu1204.ostest.psick.io not created (virtualbox) debian8.ostest.psick.io not created (virtualbox) debian7.ostest.psick.io not created (virtualbox) suse12.ostest.psick.io not created (virtualbox) suse11.ostest.psick.io not created (virtualbox) opensuse-tumbleweed.ostest.psick.io not created (virtualbox) opensuse-42-1.ostest.psick.io not created (virtualbox) alpine3.ostest.psick.io not created (virtualbox) fedora23.ostest.psick.io not created (virtualbox) cumulus.ostest.psick.io not created (virtualbox) windows2012-ostest not created (virtualbox) windows2008-ostest not created (virtualbox) We try a Centos 7 vm: vagrant up centos7.ostest.psick.io The ostest environment uses puppet apply to test our local code, mounted on the selected VM. Puppet run can be triggered either via a command like: vagrant provision centos7.ostest.psick.io Or, from within the VM, as root: vagrant ssh centos7.ostest.psick.io vm $ sudo su - vm # /etc/puppetlabs/code/environments/production/bin/papply.sh The same concept applies for other VMs and for the other environments under vagrant/environments, some of them use Puppet Enterprise or Foreman, serving directly our code to client VMs running in puppet agent mode. In some cases, further steps may be required, local documentation should help. Don’t expect everything to work out of the box flawlessly, combinations of OS, roles and data are many and not all tested. Please report bugs and eventually Pull Requests on Github. Customising The existing Vagrant environments can be customised, by editing the config.yaml file in each dir. Give a look to this document for details on how to tune the Vagrant environments. Now we can enjoy our Puppet environment, start to develop and customise our control repo starting from the data in hieradata, the local code under site the external modules to add to Puppetfile, the classification logic in manifests/site.pp and hiera.yaml. And, most of all, we can immediately test our code in several different systems which may emulate our infrastructure servers. Alessandro Franceschi 2017-05-01T00:00:00+00:00 https://example42.com/blog/2017/05/01/puppet-debugger Deep in its core, Puppet holds arcane codes that might be hard to understand. We usually don’t need to grasp Puppet inners in our daily activities, but at times it’s necessary to explore what hides under the hoods. Puppet Debugger by Corey Osman is a great tool that can help us whenever we need to get deeper in Puppet wonderlands. We can install it as gem: gem install puppet-debugger If Puppet is not installed, it’s installed as a gem dependency, in the local user ruby environment. We can enter the Puppet debugger console running: puppet debugger If Puppet is already installed, typically via a system package, you might have issues (I had), with the above command, probably due to mismatching library paths. Once in the console, various commands are available: Ruby Version: 2.0.0 Puppet Version: 4.10.0 Puppet Debugger Version: 0.6.1 Created by: NWOps <corey@nwops.io> Type "exit", "functions", "vars", "krt", "whereami", "facts", "resources", "classes", "play", "classification", "types", "datatypes", "benchmark", "reset", or "help" for more information. So, we can start to play around, for example, by querying available info. Starting from facts: 1:>> facts { "architecture" => "x86_64", [...] } The full local list of local facts is shown. Nothing new you can rightly say, but that’s just the beginning, we can get the list of variables available in the scope: 1:>> vars "Facts were removed for easier viewing" { "facts" => "removed by the puppet-debugger", "module_name" => "", "name" => "main", "server_facts" => { "environment" => "production", "serverip" => "172.17.0.1", "servername" => "lab.psick.io", "serverversion" => "4.10.0" }, "title" => "main", "trusted" => { "authenticated" => "local", "certname" => nil, "domain" => nil, "extensions" => {}, "hostname" => nil } } Starting to be interesting eh? Now, there’s more. Based on the modules available in the modulepath we can get the list of available functions, both puppet core ones and the ones from the modules. 1:>> functions archive::artifactory_sha1 archive::assemble_nexus_url archive::go_md5 inifile::create_ini_settings noop::noop puppet-4.10.0::alert puppet-4.10.0::assert_type [...] stdlib::abs stdlib::any2array [...] A list of the available datatypes, also the ones defined in modules: 1:>> datatypes [ [ 0] "Any", [ 1] "Array", [...] [41] "Stdlib::Unixpath", [42] "Stdlib::Windowspath", [...] [47] "Tp::Settings", [...] ] A list of the Puppet types, both native and from modules: 1:» types [ [ 0] “stage”, [ 1] “file”, [ 2] “exec”, […] ] Things get even more interesting when you consider that the context in which puppet debugger operates when you lauch it is determined by modulepath and environmentpath Puppet variables. If you have your control repo in the environmentpath, or a link to it, you can gen a list of the classes that would be provided to your local node (this is possible when node classification is managed directly in the control-repo and is not delegated to an External Node Classifier): 1:>> classes [ [ 0] "settings", [ 1] "tools", [ 2] "profile::settings", [ 3] "profile::pre", [ 4] "profile::repo::generic", [...] ] Even more intriguing is the list of the resources that you be applied to your node: 1:» resources Resources not shown in any specific order [ [ 0] “Stage[‘main’]”, [ 1] “Class[‘Settings’]”, [ 2] “Class[‘main’]”, [ 3] “Class[‘Tools’]”, […] [15] “File[‘/usr/local/bin/facter’]”, […] ] Note that this information is relevant to the local node, based on the Puppet code in the default environment and the modulepath, but we can run Puppet debugger and get information about a third node. In order to do this we should either run Puppet debugger on the Puppet Master or do it via a development workstation, as long as we set (either in puppet.conf or on the command line) the name of the Puppet server and we have locally the same code base we have on the server. puppet debugger -n node.my.domain Finally, another neat feature of Puppet debugger is the benchmark command, which allows to evaluate how much time Puppet takes for its operations. For example, to see the performance of a function: 1>> benchmark 2:BM>> lookup('profiles',Array,'unique',[]) => "Time elapsed 0.86 ms" 3:BM>> tp_lookup('apache','settings','tinydata','merge') => [ [0] { "conf_dir_path" => "/etc/httpd/conf.d", "config_dir_group" => "root", [...] "tcp_port" => "80" }, [1] "Time elapsed 29.73 ms" ] It’s possible to launch an instance of Puppet debugger directly inside a manifest, this allows to query the above info having as scope exactly what’s available at in a specific part of a manifest. In order to launch Puppet debugger on real code, we need the debug::break function from nwops/puppet-debug module. We can introduce in our code the function:: class 'apache' ( [...] ) { [...] debug::break({'run_once' => false}) } When the debug::break function is found, the compiler is interrupted, a Puppet debugger instance is opened, it has access to all the local scope variables (just type a $variable name to see its value), it shows where in the code the function has been called, and, it allows to skip to the next break function found during the catalog compilation. Puppet debugger gives powerful insights on our Puppet code and how and where is evaluated and definitively deserves it place in the essential toolset of every Puppet engineer. Alessandro Franceschi 2017-04-24T00:00:00+00:00 https://example42.com/blog/2017/04/24/technical_puppet_workshop_frankfurt This week the tip is for a real world event about Puppet. Could be our best tip so far :-) example42 announces the first german speaking Technical Puppet Workshop. The event takes place at Sheraton Frankfurt Airport Hotel & Conference Center, located directly at Frankfurt International airport, on Wednesday, May 17th. This workshop is aimed at systems administrators who use Puppet to manage their infrastructure and want to learn more about Puppet. The whole event is free for registered attendees. You will receive an email from Puppet shortly with further details. example42 customers have already been informed about this event. The event will start on Wednesday, May 17th, at 9:00 and will close at 18:00 at the same day. Coffee breaks and lunch will be served for free. A post event drink is also included. We will allow up to 25 attendees on a first come, first serve basis. At this event we offer three different topics where attendees may choose from during registration: Puppet Best Practice Puppet Code Testing Puppet and Integration in existing infrastructure For participation we recommend to bring your laptop with the following specifications: Linux VM on VirtualBox or VMware (CentOS, RHEL, Ubuntu, SLES) Vagrant installed on the laptop min 2 GB RAM min 10 GB free disc space Some details on the topics: Puppet Best Practice We will dig into common errors and misusage like cross referencing modules, inheritance, git repo with submodules and ways of abusing exec resource type. We will talk about NIHS (not invented here syndrome) and how to start or continue instead. Puppet Code Testing We see many Puppet code bases without proper unit and integration testing. We will provide demo code with lots or errors, fix these based on lint and rspec-puppet results and still see the module not working at all. We will then move on to beaker and docker based integration tests to fix the remaining errors. Puppet and Integration in existing infrastructure Which tool do you use for a specific task? Do you have a VM management solution in place? Do you have a reporting system ready and in production? There is no need to switch tools as Puppet has lots of APIs which allow integration into existing components and tools. We will cover reporting backends and Puppet API for certificate and node management. Further information: The event itself is for free. Neither example42 nor Puppet will cover travel and/or hotel expenses. Event organizer: example42 GmbH We are looking forward to seeing you in Frankfurt on May 17th. Can’t make it to Frankfurt? Our Puppet Training covers the same ground, tailored to your own team and code. 2017-04-17T00:00:00+00:00 https://example42.com/blog/2017/04/17/hiera-5 With the release of Puppet 4.9, version 5 of Hiera has landed on our Puppet servers, introducing some very interesting evolutions. Hiera is Puppet’s builtin key/value data lookup system, which has some peculiar characteristics: It’s hierarchical: We can configure different hierarchies of data sources and these are traversed in order to find the value of the desired key, from the layer at the top, to the one at the bottom It has a modular backend system: data can be stored on different places, according to the used plugins, from simple Yaml or Json files, to MongoDb, Mysql, PostgreSQL, Redis and others Hiera is important because it allows to assign values to the parameters of Puppet classes: a parameter called server of a class called ntp, for example, can be evaluated via a lookup of the Hiera key ntp::server, this is useful to cleanly separate our Puppet code, where we define the resources we want to apply to our nodes, from the data which defines how these resources should be. The new version of Hiera is backwards compatible with earlier version: if you don’t use custom plugins you should be able to seamlessly use your existing data sources, it’s also compatible with Puppet lookup function and face and actually you can upgrade Puppet with Hiera 5 without problems, you will just have some deprecation warnings about things that have changed (some issues were actually introduced in Puppet 4.9.0, the very first version with Hiera 5, but were promptly solved in later releases). So, what’s new and exciting about Hiera 5? A new hiera.yaml format Hiera’s configuration file (hiera.yaml) has changed format, here’s the default, which uses the core Yaml backend and has only a layer called common: --- version: 5 hierarchy: - name: Common # A level of the hierarchy. They can be more using different data sources path: common.yaml # The path of the file, under the datadir, where data is stored defaults: data_hash: yaml_data # Use the YAML backend datadir: data # Yaml files are stored in the data dir of your Puppet environment Here’s a bit more complex example, where the popular Hiera-eyaml backend is used (a backend that uses Yaml files and allows the encryption of single keys) and multiple paths are defined (they are equivalent of having multiple hierarchy levels): --- version: 5 hierarchy: - name: "Eyaml hierarchy" lookup_key: eyaml_lookup_key # Use eyaml backend. Note this can be specified for each level paths: # Instead of multiple hierarchy levels we can define just one with - "nodes/%{trusted.certname}.yaml" # multiple paths, when the same backend is used. It's exactly the same. - "role/%{::role}-%{::env}.yaml" - "role/%{::role}.yaml" - "common.yaml" options: # Hiera-eyaml specific options (the paths of the keypair used for encryption) pkcs7_private_key: /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem pkcs7_public_key: /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem defaults: datadir: data For full reference on the format of Hiera 5 configuration file, check the Official Documentation Environment and module data Hiera 4, used from Puppet versions 4.3 to 4.8, introduced the possibility of defining, inside a module, the default values of each class parameter using Hiera. The actual user data, outside modules, was configured by a global /etc/puppetlabs/puppet/hiera.yaml file, which defines Hiera configurations for every Puppet environment. Now is possible to have environment specific configurations, so we can have a hiera.yaml inside a environment directory which may be different for each environment (/etc/puppetlabs/code/environments/$environment_name/hiera.yaml). This is useful to test hierarchies or backend changes before committing them to the production environment. We can have also per module configurations, so in a NTP module, for example, we can have a $module_path/users/hiera.yaml with the, now familiar, version 5 syntax: --- version: 5 defaults: datadir: data data_hash: yaml_data hierarchy: - name: "In module hierarchy" paths: - "%{facts.virtual}.yaml" - "%{facts.os.name}-%{facts.os.release.major}.yaml" - "%{facts.os.name}.yaml" - "%{facts.os.family}-%{facts.os.release.major}.yaml" - "%{facts.os.family}.yaml" - "common.yaml" this refers yaml files under the data directory of the module. The interesting thing in this is that we have a uniform and common way to lookup for data, across the three layers: global, environment and module: each hierarchy of each layer is used to compose a “super hierarchy” which is traversed seamlessly. In the module data is also possible to define the kind of lookup to perform for each class parameter. Previously the lookup was always a “normal” one: the value returned is the one of the key found the first time while traversing the hierarchy. Now (actually since Hiera 4) it’s possible to specify for some parameters alternative lookup methods (for example merging all the values found across the hierarchy for the requested key). This is done in the same data files where we specify our key values, so, for example, in our $module_path/users/data/common.yaml we can have: lookup_options: users::local: # This lookup option applies to parameter 'local' of class 'users' merge: # Merge the values found across hierarchies, instead of getting the first one strategy: deep # Do a deep merge, useful when dealing with Hashes (to override single subkeys) merge_hash_arrays: true users::admins: # This lookup option applies to parameter 'admins' of class 'users' merge: strategy: unique # In this case we expect an array and will merge all the values found in a single one knockout_prefix: "--" # It's even possible to define a prefix (here --) to force the removal of entries # even if they are present in other layers Note that you can use regular expressions when defining specific lookup options for some keys: lookup_options: "^profile::(.*)::(.*)_hash$": merge: strategy: deep knockout_prefix: "--" "^profile::(.*)::(.*)_list$": merge: strategy: unique knockout_prefix: "--" The lookup command It’s possible to use the puppet lookup command to query Hiera for a given key. If you run this on your Puppet Master you can easily find out the value of a given key for the specified node: puppet lookup profiles --node git.lab # Looks for the profiles key on the node git.lab If you add the --debug option you will see a lot of useful information about where and how data is looked for. You can also use the lookup() function inside your Puppet code, it replaces (and deprecates), the old hiera(), hiera_array(), hiera_hash() and hiera_include(). The general syntax is: lookup( <NAME>, [<VALUE TYPE>], [<MERGE BEHAVIOR>], [<DEFAULT VALUE>] ) or lookup( [<NAME>], <OPTIONS HASH> ) Some examples: lookup('ntp::user') # Normal lookup. Same of hiera('ntp::user') lookup('ntp::user','root') # Normal lookup with default. Same of hiera('ntp::user','root') lookup('ntp_servers', Array, 'unique') # Array lookup, same of hiera_array('ntp_servers') lookup('users', Hash, 'deep') # Deep merge lookup, same of hiera_hash('users') with deep_merge set to true lookup('classes', Array[String], 'unique').include # Same of hiera_include('classes') lookup({ 'name' => 'ntp_servers', 'merge' => { 'strategy' => 'deep', 'knockout_prefix' => '--', }, }) Check the official reference for all the options available for the lookup function. Conclusions Hiera 5 seems to finally put together years of Hiera evolution: it has a uniform approach to global, environment and module data, it has an easy to use command to query keys and gives users and modules authors much more flexibility on how data should be looked up. It also makes users like easier (if they use the Yaml backend, they can see directly in modules’ data the format of the keys to configure) and, it seems, has some performance benefits. Finally, and yet not mentioned here, it allows easier creation of custom backends. You can start to use it with your existing Puppet code base (if already Puppet 4 ready) and it allows gradual migration or your data. Embrace changes, Hiera 5 is here and now, for better Puppet data management. Alessandro Franceschi 2017-04-10T00:00:00+00:00 https://example42.com/blog/2017/04/10/environment-enforcement Think about the following situation: You have a node running in development environment. A user logs in and runs puppet agent --test --environment production. What will happen: the node will receive production ready code. This is OK… but… what if you use the environment to also pass data like passwords, users, accounts to your nodes? In this case the development system will have all production data and users are happy that they now can connect to your production database. If you would not like to allow this, you have to make use of an external nodes classifier (ENC). An ENC is configured within the master section of puppet.conf file. It is called external as it does work prior the master starts catalog compilation. ENC usage is configured as follows: # /etc/puppetlabs/puppet/puppet.conf [master] node_terminus = exec external_nodes = bin/enc_cat.sh What does the enc_cat.sh script do? An external nodes classifier using the exec terminus is called with the node cert name as argument and returns yaml syntax. Foreman, Puppet Enterprise Console are common examples of ENC, they provide a custom script for external_nodes which queries the relevant API. In the above example the command is a simple bash script that just cats yaml files (either with the name of the node passed as argument, or a default one): # bin/enc_cat.sh #!/bin/bash repo_dir="$(dirname $0)/.." script_dir="$(dirname $0)" . "${script_dir}/functions" host=$1 if [ -f "${repo_dir}/bin/enc_cat/${host}.yaml" ]; then cat "${repo_dir}/bin/enc_cat/${host}.yaml" else cat "${repo_dir}/bin/enc_cat/default.yaml" fi Within the enc_cat directory one places yaml files. Either host specific or a default. If the ENC has no file to return, the node will be classified using manifests node syntax only. A default.yaml file can look like the following (the same format is what is expected by the ENC scripts for Foreman, PE Console or any other ENC): # enc_cat/default.yaml --- environment: production classes: - profile::base::pre parameters: - env: development - role: default To pin a node to a specific environment, the first line is required: environment: development This value is used by the Puppet server to determine whether a node should be in a specific environment and verifies node environment setting. If the nodes setting does not match the Puppet server will switch the catalog compilation based upon the environment from ENC. When using PSICK the above essential enc_cat is already available and can be easily activated in puppet configuration file. Martin Alfke 2017-04-03T00:00:00+00:00 https://example42.com/blog/2017/04/03/puppet-ci-with-gitlab GitLab is a versatile Open Source tool to manage your code repositories. It’s often used on premise to host private code projects, something like a private GitHub. We often use it to host our Puppet code and we have started to appreciate its multiple features, one of them is the integrated CI engine. It’s incredibly easy to use and powerful, you can design your CI pipeline in a file called .gitlab-ci.yml at the root directory of your control repo. GitLab will automatically interpret it whenever there are changes in your repo code, and try to use GitLab CI runners (agents running on, typically, separated servers where are run the actual CI operations). Let’s see how this file is structured, samples are from example42’s PSICK, where the control-repo is tested in various ways on different phases. Give a look at the official documentation for more details on GitLab CI. First, on .gitlab-ci.yml, you define the stages of your pipelines, in the order you want them to be executed (note you can have more pipelines for your project, to use in different conditions (for example when there are changes at specific git branches) so not all stages you list here have to be in the same pipeline): stages: - checks - version_check - specs - diffs - sitedoc - integration - live_runs - merge_request - promote - rollout - postcheck Then you define the jobs to do, they have a stage assigned, and for each of them you can run scripts, manage behavior, assign with tags a specific CI runner, and define on which branch they should be run. # Run Syntax Checks on Feature/Personal Branch and Development branch: # All branches excluded production and testing syntax: stage: checks # This is a stage previously defined before_script: "bin/gitlab_before.sh" # A script to run before the tests script: "bin/puppet_check_syntax_fast.sh" # The actual test script, its # exit code defines jobs status tags: # If you tag a job you force it to run on specific - test_puppet # CI runners except: # This defines the branches on which to NOT run the job - testing - production only: # This defines the branches on which RUN the job - branches cache: # On the runner you can cache specific directories untracked: true paths: - modules/ # Paths are relative to the git repo, here we cache # the directory where external modules are placed via r10k For each job, you have a similar syntax, and you can add options, for example to allow failures on a job (the pipeline is not interrupted in case of errors in the job): # Puppet lint tests. lint: stage: checks before_script: # You can run multiple scripts using an - "bin/gitlab_before.sh" # array like this script: "bin/puppet_lint.sh" cache: untracked: true paths: - modules/ tags: - test_puppet except: - testing - production only: - branches allow_failure: true # The job can fail without blocking the pipeline You can have also manual steps, which are not automatically performed in the pipeline, and can be triggered manually from GitLab web interface: # Do Vagrant run. Start the machine, run the tests, halt the machine. # This will use a .vagrant dir under $HOME of the users so that # it will not be deleted accidentally by gitlab-runner vagrant_checks: stage: integration before_script: - "bin/gitlab_before.sh" script: - "bin/vagrant_node_test.sh centos7.ci ci setup" - "bin/vagrant_node_test.sh centos7.ci ci drift" after_script: - "bin/gitlab_after.sh" - "bin/vagrant_node_test.sh centos7.ci ci halt" cache: untracked: true paths: - modules/ - tests/ allow_failure: true tags: - test_puppet only: - development when: manual # This step can be run manually In our sample we use customer scripts to automate Merge Requests and Accept from different branches. In this example code is promoted automatically, if there weren’t unallowed failures in the previous steps, from development to testing branch: # Development to testing merge request creation merge_request_testing: stage: merge_request script: "bin/gitlab_create_merge_request.rb development testing" tags: - deploy_puppet only: - development # Automatic Accept merge request from Development to testing merge_accept_testing: stage: promote script: "bin/gitlab_accept_merge_request.rb development testing" only: - development tags: - deploy_puppet when: on_success # Do this only if there are no failures in the pipeline Once merged in testing we can trigger Puppet runs on real servers and check the server’s status: # On testing branch run_puppet_on_testing: stage: live_runs before_script: # Trigger remote Puppet runs via puppet job - "bin/puppet_job_run.sh testing" # Trigger remote Puppet runs via Rundeck # - "bin/rundeck_job_run.sh testing" script: - "bin/puppetdb_env_query.sh testing" # Query PuppetDB for last run status tags: - deploy_puppet when: on_success only: - testing allow_failure: true The actual deployment of Puppet code on the Puppet Server can be performed as a normal job too, achieving, if wanted, a Continuous Deployment solution. In our case we use PE Code Manager or GitLab hooks to automatically deploy the Puppet control-repo code when there are changes in a branch, so, basically, Puppet code deployment to production is done whenever a change is merged in the production branch. A nice touch, which can be automated too, is the generation on the control-repo documentation using puppet strings and publish it directly on GitLab pages: pages: stage: sitedoc script: - rm -rf doc public .yardoc - puppet strings generate site/\*\*/manifests/\*.pp site/\*\*/manifests/\*\*/\*.pp site/\*\*/manifests/\*\*/\*\*/\*.pp site/\*\*/functions/\*\*/\*.pp manifests/\*.pp - mv doc public tags: - deploy_puppet artifacts: paths: - public expire_in: '30 day' only: - production These are just examples of what you can do in a pipeline for Continuous Integration of your Puppet code. The workflow design, the stages, jobs and implementation are something that have to be adapted to each context, we are still exploring alternative approaches in terms of things to check (other jobs involve using catalog diff, tests on docker vms, spec tests…) and in the logic of the git workflow. The approach used here, with Merge Requests from development to testing to production, may have better alternatives. Any suggestion? Alessandro Franceschi 2017-03-27T00:00:00+00:00 https://example42.com/blog/2017/03/27/environment_caching Usually a Puppet Server as a monolithic installation can handle up to 4000 nodes. In larger environments one can easily scale this by adding compile servers placed behind a load balancer. But prior you throw additional hardware on your load issue, Puppet server allows you for some scaling. One of the ways to prevent scaling issues is to make use of Puppet environment cache. When activating the environment cache, the Puppet master process compiles catalogs for a node once only and keeps the generated catalog in memory. This also has an impact on available RAM on the Puppet master as all catalogs are kept in RAM - either for a specific timeout or permanently. The environment cache can either be set globally or per environment. Global setting must be done in puppet.conf file: # /etc/puppetlabs/puppet/puppet.conf [master] # environment_timeout = 180 # default 3 minutes environment_timeout = 100 The environment_timeout option can have one of the following values: 0 - never cache catalogs unlimited - always use catalog cache ‘number’ - duration to keep a catalog in cache Mostly one does not like to set environment_timout as a global setting. In production environment . where changes are less often - you want to have an environment cache, whereas in development environment - where you push changes very often - you want to disable the cache. Environment specific cache settings can be put in environment.conf file and has the same look and options according to the global setting in puppet.conf. Now let’s set environment in the production environment: # /etc/puppetlabs/code/environments/production/environment.conf environment_timeout = unlimited But how can I let Puppet know that new code is available so the compiler uses the new code instead of the cached catalog? Puppet server provides an admin API where you can delete the caches of specific environments. Usually you want to run this API call on r10k postrun or whenever you deploy Puppet code on your Puppet master. This also solves another side effect: race-condition when updating an environment while the compiler is in progress of generating a catalog for a node. A sample script that flushes the environment cache is available on PSICK, you can run it, as root, on the Puppet server to flush the cache of all the environments or the one passed as argument. It runs basically a command like this: curl -i --cert $(puppet config print hostcert) \ --key $(puppet config print hostprivkey) \ --cacert $(puppet config print cacert) \ -X DELETE \ https://$(puppet config print server):8140/puppet-admin-api/v1/environment-cache Martin Alfke 2017-03-20T00:00:00+00:00 https://example42.com/blog/2017/03/20/roles_profiles_and_tools Roles and profiles is an established pattern that for years has helped Puppet architects in the organization of their code base. A role, and only a role, is typically assigned to a node, describing its business functionality (blog webserver, api backend, relay mail server…) and includes one or more profiles. A profile is basically a wrapper class which may use a component module (like the ones for apache or mysql or whatever we find on the Modules Forge) and adds site specific resources: the things we need to configure servers in the way we want. Over the years I found myself using slight variations on this pattern, according to use cases and context. For example I find more flexible the possibility to define the profiles to use in nodes, not in “static” role classes, but via Hiera. I consider absolutely necessary to have in my hiera’s hierarchy a layer that represents the role of the machine, and since this is already there I can use it as layer of reference where to include profiles. The Puppet code needed to define in Hiera the profiles to use is quite straightforward: hiera_include('profiles', []) The hiera_include function looks for an array and includes as classes all the elements of the array. Starting from Puppet 4.9 Hiera functions in Puppet DSL are deprecated, so the above code can be written with: $profiles = lookup({ name => 'profiles', merge => 'unique', default_value => [], value_type => Array[String], }) $profiles.each | $p | { contain $p } This alternative may look more complex but it’s actually much more powerful. Check this blog post for R.I.Pienaar for more details. For a smarter alternative of the above, you can simply have something like: lookup('profiles', Array[String], 'unique').contain One of the benefit in defining what profiles to include via Hiera is that we have much more flexibility on where to include them. We can set them at the role’s hierarchy level but we can also set them at other levels, for example managing per node exceptions. So, in my latest control-repos I don’t have a role module where I define my role classes, as everything is managed via Hiera, but I have another kind of site module: tools. The tools module is usually coupled with the profile one and basically contains defines which are used by the profile classes. The basic idea is that tools contains local facts, functions, data types, defines, types and providers which are not provided by a dedicated module and which serve us for our own purposes, be they very specific or somehow common. An example of a tools module is in example42’s PSICK, a rather complete control-repo, here, since we make a large use of Tiny Puppet which saves us from the need of importing several external modules, we place various defined for common (and not so common) purposes. Some of them should actually stay in dedicated modules, along with the relevant profiles classes, but our control-repo is somehow opinionated and has its own rules ;-) Alessandro Franceschi 2017-03-13T00:00:00+00:00 https://example42.com/blog/2017/03/13/second_mount_point Credentials with a certain level of security should never be stored in plain text on a Puppet server. How about items that shpuld be managed by another team (e.g. Security and Compliance) but the team insists that these data may not be part of the standard Puppet environments? In this case one can make use of a second file mountpoint. Using a second mount point within Puppet code follows the already well known pattern for delivering static configuration files using file resource type and source property: class my_certs { file { '/etc/ssl/certs/my_company_ca.pem': ensure => file, source => 'puppet:///certs/my_company_ca.pem', } } Do you recognize the pattern? Let’s compare this with a “normal” file source declaration where we fetch the file from the module files directory: class my_old_certs { file { '/etc/ssl/certs/my_company_ca_old.pem': ensure => file, source => 'puppet:///modules/my_old_certs/my_company_ca_old.pem', } } The second declaration uses the file from the module. Directory layout is the following: <modulepath>/my_old_certs/ |- manifests/ | \- init.pp \- files/ \- my_company_ca_old.pem Adding a second mount point is straight forward. Just add the following snippet to /etc/puppetlabs/puppet/fileserver.conf. # /etc/puppetlabs/puppet/fileserver.conf [certs] path /opt/security/data/certs/ allow * Don’t forget to restart the puppetserver process # On Open Source service puppetserver restart # On Puppet Enterprise service pe-puppetserver restart Now one can use the new mountpoint easily in Puppet code: class my_certs { file { '/etc/ssl/certs/my_company_ca.pem': ensure => file, source => 'puppet:///certs/my_company_ca.pem' } } File system layout looks the following: /opt/security/data/certs \- my_company_ca.pem This concept allows you to easily store information on the master in a specific path without these information being part of the Puppet environment DSL code. Martin Alfke 2017-03-06T00:00:00+00:00 https://example42.com/blog/2017/03/06/server-side-noop-mode Puppet has a well known command option that allow us to make dry runs, with clients fetching the catalog from the Master and showing what it would have changed if it were applied for real. From the command line on the client this can be done with a command like: puppet agent -t --noop There are situations, anyway , where it would be preferable to activate nood directly from the server, forcing noop mode on one or more clients or sets of resources. This is possible thanks to the trlinkin-noop module, which provides a function, called noop() which automatically add the noop argument to every resource in the current and the child scopes. A use case we’ve found quite useful in many occasion is the possibility to manage and force the noop mode directly in Hiera. To do this for all the resorces, it’s enough to add in the top scope, in one of the main manifests, for example manifests/site.pp a few lines of code like these: $noop_mode = hiera('noop_mode', false) if $noop_mode == true { noop() } Basically we look for a hiera key called noop_mode if this is set to true, then noop is enabled in the given Hiera context (for a specific node, an environment or all the nodes) with an entry (in yaml format) like: --- noop_mode: true What are the use cases for such an approach? Various. For example when you have to deploy large or potentially dangerous Puppet code refactorings and you can’t fully test their effect until you deploy to production. In these cases being able to rollout your changeset with noop mode enabled (for all or the most important nodes) allows you to test and review the real consequences on your servers with more safety and confidence. In some situations you may prefer to have production servers only running in safe noop mode, and trigger real changes only upon request. This can be done via Hiera, as seen before, but also eventually setting a top scope parameter on an ENC like PE console or The Foreman) and using it as trigger for the noop() function in your site.pp. In some situations you might want to enforce real mode whatever is the noop setting. This can be done by calling the noop function with the false parameter: if $enforce == true { noop(false) } If this is done in a class with a parameter called $enforce, for example, you can let users decide if to ignore or not any noop option. Alessandro Franceschi 2017-02-27T00:00:00+00:00 https://example42.com/blog/2017/02/27/building-data-types Puppet comes with a set of core data types like Integer, Float, String, Boolean. But what if you have special needs? How to reimplement the stdlib functions like validate_absolute_path() or validate_ipv4()? Puppet allows you to place your own build data types into a modules type directory: modulepath/ \- firewall/ |- manifests/ | \- init.pp |- lib/ | |- facter | \- puppet |- facts.d \- types \- ipv4address.pp Inside the types directory you can specify a new data type. This new data type is available in your module namespace: # firewall/types/ipv4address.pp type firewall::ipv4address = Pattern[/^((([0-9](?!\d)|[1-9][0-9](?!\d)|1[0-9]{2}(?!\d)|2[0-4][0-9](?!\d)|25[0-5](?!\d))[.]){3}([0-9](?!\d)|[1-9][0-9](?!\d)|1[0-9]{2}(?!\d)|2[0-4][0-9](?!\d)|25[0-5](?!\d)))(\/((([0-9](?!\d)|[1-9][0-9](?!\d)|1[0-9]{2}(?!\d)|2[0-4][0-9](?!\d)|25[0-5](?!\d))[.]){3}([0-9](?!\d)|[1-9][0-9](?!\d)|1[0-9]{2}(?!\d)|2[0-4][0-9](?!\d)|25[0-5](?!\d))|[0-9]+))?$/] You can use any data type for building new ones. Some combined data types are already available in puppetlabs-stdlib or voxpupuli-tea. Martin Alfke 2017-02-20T00:00:00+00:00 https://example42.com/blog/2017/02/20/local-references class phpmyadmin { package { 'phpmyadmin': ensure => present, require => Service['apache'], } } What is wrong with this code? You have build a hidden dependency between the phpmyadmin class and the class which declares the Apache Service resource. Why is this bad? You can never use the class phpmyadmin standalone. The phpmyadmin class should be a Technical Component Class which can be used alone or in any other combination of classes. Besides this: you are forcing the users of your class to have phpmyadmin running on Apache webserver only. How should this be done right? Use local Resource References. class phpmyadmin { package { 'phpmyadmin': ensure => present, } } class apache { service { 'apache': ensure => running, } } class profile::phpmyadmin { contain apache contain phpmyadmin Class['apache'] -> Class['phpmyadmin'] } You can check your code for non-local Resource References by using the lint extension reference_on_declaration_outside_of_class-check. Martin Alfke 2017-02-13T00:00:00+00:00 https://example42.com/blog/2017/02/13/exec-done-right exec { ‘touch /tmp/foo’: } Ouch. Exec resource abusing seems to be an often used pattern. Consider an Exec resource being an emergency exit. It is a powerful resource type that offers you to do almost anything to a system. But please keep in mind that with great power comes great responsibility. You (as the author of the Exec resource type declaration) are responsible for idempotency and error handling. Best practice is to have an Exec resource for one-time commands only. This does not refer to a single puppet run, but throughout the life cycle of the system. Don’t forget to make use of the exec resource type parameters for idempotency: onlyif unless refreshonly creates And don’t forget to set the path parameter. Martin Alfke 2017-02-06T00:00:00+00:00 https://example42.com/blog/2017/02/06/resources-from-hash-in-puppet4 Puppet is always about abstraction. But please never forget maintainability, readability and simplicity (K.I.S.S). Nowadays we often see code like the following: class my_great_thing ( $data = {}, ){ $default = { ensure => present, } create_resources('user', $data, $default) } Great abstraction, but what happens really? When declaring the class my_great_thing, puppet compiler will automatically do a hier alookup for the key my_great_thing::data. Maybe the key-value looks like this: my_great_thing::data: 'foo': home: '/home/foo' shell: '/bin/bash' uid: '1044' gid: '1044' 'bar': home: '/home/bar' shell: '/bin/bash' uid: '1045' gid: '1045' These data are not used with the create_resource function, which will generate two user resource types for the users foo and bar. This is a quite hidden approach of what is done. How about using Puppet 4 capability of dealing with data? class my_great_class ( Hash $data = {} ){ $data.each |String $key, Hash $value| { $ensure = pick($value['ensure'], 'present') user { $key: ensure => $ensure, * => $value, } } } The most beautiful thing here is the splat operator * => $value,. This expands the $value subhash. Each key becomes the parameter and the according value becomes the parameter value. Martin Alfke 2017-01-30T00:00:00+00:00 https://example42.com/blog/2017/01/30/rspec-on-existing-code Many Puppet code bases are pure Puppet code without unit or integration testing. Most maintainers of such a code base fear the work for adding tests to their code. But what is it that you as a maintainer would like to achieve? Normally you are happy when you can check whether your Puppet code still compiles on a newer Puppet version. Just adding this kind of unit tests is easy and depends on your code layout: do you have repository per module do you have one monolithic repository Let’s go for the first solution: You need a Gemfile, a Rakefile, a .fixtures.yml and a spec/spec_helper.rb file. The Gemfile can be short: # Gemfile source 'https://rubygems.org' gem 'puppetlabs_spec_helper' gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4' Yu can even omit the puppet gem and you get the latest version automatically. In our example you have the option to specify other versions via environment variables: export PUPPET_GEM_VERSION='~> 3' The Rakefile can be even shorter: # Rakefile require 'puppetlabs_spec_helper/rake_tasks' The .fixtures.yml describes the naming scheme of your module and adds additional required modules for spec testing in a sandbox. # .fixtures.yml fixtures: repositories: stdlib: "https://github.com/puppetlabs/puppetlabs-stdlib.git" symlinks: put your class name here: "#{source_dir}" The spec_helper.rb file must be located in the spec Directory and has the following content: # spec/spec_helper.rb require 'puppetlabs_spec_helper/module_spec_helper' Next you want to test your module. We assume that the module has one class only. Put class tests inside the spec/classes directory. The test file must end with _spec.rb to allow rspec-puppet to find the test. # spec/classes/init_spec.rb require 'spec_helper' describe 'put your class name here' do describe 'on test osfamily' do let(:facts) do { :osfamily => 'put the os you want to test here' } end context 'with default options' do it { is_expected.to compile.with_all_deps } end end end The second approach (one monolithic repository just needs adoption of all symlinks in the .fixtures.yml file. e.g. # .fixtures.yml fixtures: symlinks: my_ntp: "#{source_dir}/modules/ntp" my_apache: "#{source_dir}/modules/my_apache" mysql: "#{source_dir}/modules/mysql" ... Everything else works similar. Now you need to install the Gemfile extensions. First you want to ensure that you have a corresponding, supported Ruby version installed. In case that the OS vendor does not offer the correct version from repositories you might want to reconcider installing ruby into your home directory by using rvm or rbenv. Switch into the repository directory where the Gemfile is located and install the extensions using bundler: bundle install Extra tip: if you dont like to mess up your ruby installation you can also specify a path where the extensions will be installed: bundle install --path vendor Now you can run the rake task: bundle exec rake spec Please note that in this simple case you want to run the tests on the same os as your infrastructure. Testing on e.g. a Windows or OS X workstation requires additional fact and Puppet code mocks. But how to write full tests when no time is given for this task? Please check the retrospec tool which will parse your Puppet code and generate the tests automagically for you. Martin Alfke 2017-01-23T00:00:00+00:00 https://example42.com/blog/2017/01/23/existing-code-on-puppet4 Starting with Puppet 4 on greenfield is easy. But how to upgrade from Puppet 3 to Puppet 4 with an existing codebase? To be honest: it depends. When you have a really old code base (e.g. you started using Puppet 2.6 or older) and you never adopted to new best practices you might reconsider a full rewrite from scratch. Don’t mess you new Puppet 4 infrastructure with old practices Puppet code. When you have constantly adopted to best practices and removed deprecations from your Puppet code base you are in the comfortable situation to use tools which help you ensuring that your code is working on Puppet 4, too. First you want to ensure that you are running the latest Puppet 3 Master version (3.8.x) and that your Agents are upgraded to the same version, too. There are several ways to identify whether your code works in the same way on Puppet 4: check identical catalogs with Puppet 3 and 4 test by enabling the Puppet 4 parser on a Puppet 3 Master integration testing with different Puppet versions Checking identical catalogs needs a Puppet code extension. You can choose between zack/catalog_diff or github/octocatalog_diff or puppetlabs/puppetlabs-catalog_preview. All three tools will generate an overview on where the catalog contains differences. To make use of catalog_diff you might want to place a new Puppet 4 based Master in place. On your Puppet 3 Master you want to give access to facts and catalogs # allow the diff server to query facts path /facts method find, search auth any allow diff.example.com # allow the diff server to retrieve any catalog path ~ ^/catalog/([^/]+)$ method find allow $1 allow diff.example.com Now you can make use of the puppet catalog command. The output will tell you where differences in the catalog occur. e.g. Resource counts: Old: 2 New: 2 Catalogs contain the same resources by resource title Individual Resource differences: Old Resource: file{"/tmp/foo": content => d3b07384d113edec49eaa6238ad5ff00 } New Resource: file{"/tmp/foo": content => dbb53f3699703c028483658773628452 } If you want to use puppetlabs catalog-preview tool you need latest Puppet 3.8 with two environments: your production environment with your working puppet code and a copy of that where you enable the future parser. Then you can run the Puppet preview command: puppet preview --preview-environment future_production <nodename> ‘future_production’ is the copy of your environment with future parser enabled. Puppet preview not only shows catalog diff, but also shows code deprecation warnings and many more. Martin Alfke 2017-01-16T00:00:00+00:00 https://example42.com/blog/2017/01/16/the-optimal-hiera-yaml Yes, we are cheating. There’s not a single optimal hiera.yaml file to configure Hiera in the perfect way for any infrastructure. As usual different infrastructures may need different approaches and have different preferences on how and where to store data. Here we are going to give a few suggestions on what could be common use cases. The backend Hiera supports several different backends (here is a more complete list) where to store data. Unless you don’t have specific needs or preferences, you will generally use a file based backend, as yaml and json. Also, you will probably need to encrypt some of your data so you’ll likely move your eyes on the hiera-eyaml backend, which allows to easily encrypt some hiera keys in plain yaml files. In such a case, use ONLY the hiera-eyaml backend, there’s really no sense in having both it and the normal yaml backend. So your hiera.yaml file would begin with: --- :backends: - eyaml Then you have to configure hiera-eyaml. You have to provide the paths where keys used for encryption are stored. They are needed wherever a catalog, that uses encrypted data, is compiled, typically on the Puppet Server, but also in your development and testing stations (unless you take care to avoid to use encrypted keys in such environments). :eyaml: :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata" :pkcs7_private_key: '/etc/puppetlabs/keys/private_key.pkcs7.pem' :pkcs7_public_key: '/etc/puppetlabs/keys/public_key.pkcs7.pem' :extension: 'yaml' Note that in the above configuration we place the keys in a dedicated directory outside the Puppet environment: we don’t want to store in the same repo where we encrypt data the keys to decrypt it. You may want to add a link called keys in your control-repo pointing to /etc/puppetlabs/keys in order to seamlessly be able to use a command like eyaml edit hieradata/common.yaml (eymal looks for its keys in a directory called keys in the cwd if their path is not defined in the /etc/eyam/config.yaml file). The hierarchy This is probably the part of your Hiera configuration where you will have to spend some time evaluating the best approach. You should not have too many levels in your hierarchy, definitively less than 10, they should start from the most specific one (where the same nodes’ cert name is used) to the most generic. The intermediary levels depend on your infrastructure and your decisions on them should be based on an evaluation of how values for the various configurations you manage may change in your setup according to different conditions. Typically you’ll have layers that represent the operational environment of your nodes (prod, qa, test, devel… here we call it $env as it does not match Puppet’s environment for which there’s the internal $environment variable), their role and the datacenter, network or zone where they are placed. So a possible sample hierarchy may look like this: :hierarchy: - "hostname/%{::trusted.certname}" - "role/%{::role}-%{::env}" - "role/%{::role}" - "zone/%{::zone}" - common Note that all the variables used in the hierarchy are top scope, and you need to define them in some way: either as facts or in an External Node Classifier (ENC) or in manifest/site.pp. Note also that we haven’t placed any reference to Operating System facts: even if (old) Hiera examples and online documentation used to have them in sample hierarchies, you generally should not need them: OS related settings should be managed directly in modules and classes, Hiera’s hierarchy should reflect your own infrastructure logic, not the one of Operating Systems you use. Conclusions Hiera has changed the way we work with Puppet. Since its introduction in Puppet 3 (before it was available as external add on) it has quickly become the standard tool to manage Puppet data and separate it from our code. Should we use it? Yes definitively, that’s currently the best and most versatile solution around. Should we place ALL our data there? It depends. I personally prefer to leave inside my profile and local classes the company defaults and the OS related infos to avoid too much data in Hiera yaml files. Also, when a ENC is used, I avoid to use it to store data, so that infrastructure data is not present in multiple places. The only exception is when the ENC is used to set global parameters used in the hierarchy. Still, your mileage may vary and you might find better or more fitting solutions for your use case. Alessandro Franceschi 2017-01-09T00:00:00+00:00 https://example42.com/blog/2017/01/09/anatomy-of-a-puppet-control-repo For years Puppeteers have struggled to find a way to organize their Puppet code to manage their infrastructures in an optimal way, following the evolution of Puppet itself and its ecosystem. Many different approaches were taken, some worked better, some worse, but there was no single, common approach. Now things are different. There are established tools to manage Puppet data (Hiera), deployment of external modules (r10k or Librarian Puppet) and a standard place where to place everything (directory environments). A Puppet control-repo is a [git] repository that contains the files you expect to have in your directory environment which provide whatever you need to manage your infrastructure, that is: The manifests directory where are placed the first files that the Puppet server parses when compiling catalogs for clients. Here you typically have the site.pp file (but other manifests with different names can be seamlessly added) where you can set top scope variables, resource defaults, and eventually have node statements to define what classes should be included in your nodes (nodes classification can be done in several different ways, using the node statement is just one of them). The hieradata directory which contains Hiera data files. The name of the directory is completely arbitrary, even if this is a sort of standard de facto. This same directory could not even exists in the unlikely case you are not using Hiera, or if you use Hiera with backends which don’t store data in normal (typically yaml or json) files. Some people, not me, prefer to place Hiera data in a separated dedicated repository, so you might have it outside the control-repo. The modules directory contains Puppet modules. Typically you don’t place themselves directly in your control-repo but define them in the Puppetfile and then deploy them with either r10k or Librarian Puppet. You will probably have to develop custom modules (your role and profile modules, your site specific ones). You can decide to place them in dedicated repositories, and add them to your Puppetfile, or keep them in your same control-repo. In this case it makes sense to place them in a separated directory, such as site, in order to differentiate external modules, defined in the Puppetfile, from the local ones. The environment.conf file, which configures your environment: where the modules are placed, the caching timeout and eventually a script that returns a custom configuration version. You can find an essential skeleton for a control-repo, from Puppet. You can also give a look to example42 control-repo, which provides much more. Maybe too much. It’s based on a node-less classification, without role classes (profiles to include are defined in Hiera), a LOT of sample profiles with relevant sample Hiera data, deep integrations with Docker and Vagrant for code testing and a set of tools, optionally integrated with Fabric, to manage the whole Puppet code development, testing and deployment workflow. For more information about example42 control-repo, give a read to this blog post or just look at its documentation and files: whatever are your Puppet skills you may find something useful there, after all it’s the result of 10 years of Puppet experience. If you’re setting up this structure for the first time, our Puppet Infrastructure Kickstart service builds exactly this kind of foundation from scratch. Alessandro Franceschi 2017-01-01T00:00:00+00:00 https://example42.com/blog/2017/01/01/one-liner-to-install-puppet4 New year, new challenges. At example42 we continuously develop Puppet solutions but we seldomly take the occasion to talk about them. We’ve decided, starting today, to publish every week a blog post with tips and infos about Puppet, DevOps, Automation and what we do with it. Since the 31st of December 2016 Puppet 3 has reached its End Of Life, so we think it’s a good occasion to begin the new year and our journey with how to install or upgrade Puppet 4. If you want it on Linux you can follow the official documentation or you can just run this command: wget -O - https://raw.githubusercontent.com/example42/control-repo/production/bin/puppet_install.sh | sudo bash Or, if you are lazy: wget -O - https://bit.ly/installpuppet | sudo bash The script automatically detects the underlying OS, removes existing Puppet installations and repos, installs the relevant, official, repositories for Puppet 4 and then installs the puppet-agent package. It currently supports installation on Linux (RedHat and derivatives, Fedora, Suse, Debian, Ubuntu) and MacOS/Darwin. Check online reference for installing Puppet on Windows. It runs totally unattended, so it can be used during automated provisioning of servers on public or private clouds. If you don’t like the idea of running as root scripts from the Internet, even worse via an url shortener, you are right and you already know what do do: download the script, analyze it, place it in a safe location and run it from there. Or just fire the above one-liner and forget: it will work in the same way ;-) Alessandro Franceschi 2016-05-30T00:00:00+00:00 https://example42.com/blog/2016/05/30/exploring-puppet4-modules-design-patterns The enhancements coming with Puppet 4’s parser and type system are starting to appear in the modules ecosystem, still the need to preserve backwards compatibility is often slowing authors from fully embracing the powers and the elegance of the new Puppet language. When example42 announced the 4th generation of its Puppet modules and introduced a complete control repo, we decided to fully embrace Puppet 4 and ignore backwards compatibility. I, Alessandro, was struggling to find a sane way to put together the possibilities (and limitations) of Tiny Puppet, the structure of a full featured control-repo, and the usage of third party modules when I met, at the last OSDC, David. He talked about new design principles for modules that found me in complete agreement, and I’m glad to have him here to describe them directly: Roles, Profiles, and Components In 2012 Craig Dunn posted the seminal Roles and Profiles pattern. At that time we all just figured out how to write modules. Craig, and everyone else, were now learning how to apply those modules to increasingly bigger parts of their infrastructure. Fast-forward to today, where roles and profiles are an established pattern, and modules have grown to cover all the edge-cases of that pesky real world. For example, the apache::vhost define has over 130 parameters, while the apache main class can install apache with any of four different MPMs, and provides a fully specialized default configuration. It also turns out that this kind of module provides both too much opinion on configuration, standing in the way of the seasoned apache practitioner, and not enough to be immediately useful to people just needing a web server. Craig wrote in 2012: […] the profile “Tomcat application stack” is made up of the Tomcat and JDK components, whereas the webserver profile is made up of the httpd, memcache and php components. In Puppet, these lower level components are represented by your modules. I, David, believe this to be an important part of the pattern, that we all never followed up on: modules need to provide the components - the cogs, and gears - that system engineers can use in their profiles to control their infrastructure, but - from the time before the Roles and Profiles pattern - modules also provided, and still do provide, functionality from a profile’s responsibility to be useful. This additional functionality is now standing in the way of fulfilling all the other use-cases, by prescribing certain ways to do things. To solve this I propose a separation of concerns within modules into a component layer and a profile part. The main class and its associated defines in the component layer, implement platform support and do not carry any configuration content at all. All configuration values and policies move into one or more profile classes in the module. Profiles in Modules Profiles in Modules are opinionated, optional, top-level classes, and defines that address a very specific use-case for consumers of a module. Good examples are “ntp::client::local_broadcast”, “ntp::client::unicast” and “ntp::server::local_broadcast”, or “apache::reverse_proxy_vhost” and “apache::reverse_proxy_location”. The goal here is to cover common use-cases and allow collaboration within the community on best-practice configurations. There are only very few people world-wide who really need to know how to setup and cryptographically secure a local ntp broadcast setup, but everyone would benefit from the added security and reduced bandwidth usage. Being opinionated allows them to move the subject matter forward. Profiles will be written by subject matter experts of the underlying software, and need to match the concepts of that software, and the use-case they are intended to solve. Through this matching, they can speak the language of the consumers, and will be easier to understand. This also serves to say that if a system engineer has an opinion on how something should be configured, they should be encouraged to write, and share, their own profile. Being specific allows them to concentrate on solving the problem, instead of trying to cater to everyone. It is better to have tow things that each solve a problem well than one thing that tries to do everything, but does nothing good. Being optional allows them to control their slice of the system with bold confidence. Profiles can pull all the stops to configure the service optimally for their target use-case. Even more so than site-specific profiles, profiles in modules need to take care that they are composable with profiles from other modules. In their area of responsibility on the other hand, profiles can do everything to maximize their efficiency, and ease of use. This also means that profiles are easily replaceable, when they do not intertwine with the basic necessities of the component layer. Being top-level allows them to expose critical operational knobs to the administrator. This can range from a simple file source/content parameter pair to very high-level, domain-specific use-case specific tunables, like “use X gigabytes of RAM to run this mysql instance.” It is very important that profiles do expose the things that are important to their solution, and nothing else, keeping them relevant to the consumer’s needs. Being in the module itself allows them to be shared, refined, and accessed by many. Whether that is just within your own organization, your customers, or the wider open source community, I hope I do not have to explain the benefits here anymore. This addresses the feature support. Example: # Configure an NTP client to listen to local broadcasts signed by a ntp::server::local_broadcast with the same $keygroup_name. # $interface can be used to only bind to a specific network interface class ntp::client::local_broadcast(String $keygroup_name = $facts['domain'], String $interface = 'ALL') { include ::ntp ntp::conf { config_content => template('ntp/client/local_broadcast.cfg.erb'), } } Component Layer The Component Layer is a transparent, mandatory foundation for all consumers of a specific service, or application. This layer aggregates all the platform/distribution specific logic and data, and uses that to take care of the direct management of package installation, service management, and deploying configuration. Should the managed software allows it, it would also be appropriate to install multiple instances on the same node. Being mandatory enables the interaction of multiple different consumers on the same node, and avoids namespace clashes across complete environments, as everyone shares a common language to manage the basics of a thing. Being transparent avoids putting untoward constraints on the component’s consumer. The basic layer contains all platform knowledge, but no configuration information, or domain-specific abstractions. It also exposes this platform knowledge for consumption by the profiles, to allow them to be portable, without having to have all that logic themselves. And finally, this means that innovative profiles can still rely on the services of the component layer, as they have full control over the configuration. Being a foundation requires to be broad, stable, well-tested. Being a shared resource brings great responsibilities. Foremost a broad platform support, to cover the component layer’s core competency. Then, a good test coverage. Profile authors and direct consumers need assurance that they can rely on the component layer’s services across all platforms. As a shared resource, the investment pays off over a bigger number of uses and improved stability for a larger number of systems. Example: # install NTPd class ntp( Enum[present, absent] $ensure = 'present', ) { [...] } # deploy configuration file for NTPd ntp::conf( String $config_source = undef, String $config_content = undef, ) { [...] } Beyond that, a component should expose information about the capabilities and features of the installed software. A example is which of the many mysql versions is installed, as this has consequences in details of configuration. Doing this allows all the platform- and version-specific niggles to be concentrated in one location. Whenever a profile needs to make a decision on one of those values, confident in their fidelity, without having to fall back to guessing from operating system versions, or similar. From theory to practice Based on these principles just described by David, I (Alessandro) have started to work on some sample proof of concepts. The first attempt was with Apache, here we have a very compact main class with a few general purpose parameters and no resource directly managed: class apache ( # Manage presence Variant[Boolean,String] $ensure = present, # The name of the class that manages apache installation. Tiny Puppet is used here by default String $install_class = '::apache::install::tp', # A **single** hash to override the module general configuration settings for the # underlying OS (package names, file paths...) Hash $settings = { }, # The name of the module to use for [tiny puppet] data String[1] $data_module = 'apache', # Here follow some useful, module wide, default behaviors. # These can be referenced in any module's class or define. # If to restart services when changes occur: Boolean $auto_restart = true, # If to automatically apply default configurations (if present): Boolean $auto_conf = false, # If to automatically add prerequisites resources (repos, users, packages...) when needed: Boolean $auto_prerequisites = false, ) { ... } I wanted to reproduce the behavior of the current apache example42 module, adding defines to manage virtual hosts, modules and configuration files. They were easy and quick to write, also thanks to Tiny Puppet features and the choice to use the main class as the module’s general entry point for variables: define apache::vhost ( Variant[Boolean,String] $ensure = '', String[1] $template = 'apache/vhost/vhost.conf.erb', Hash $options = { }, ) { include ::apache tp::conf { "apache::${title}": ensure => pick($ensure, $::apache::ensure), base_dir => 'vhost', template => $template, options_hash => $::apache::options + $options, data_module => $::apache::data_module, settings => $::apache::real_settings, config_file_notify => $::apache::service_notify, } } Finally I started to add some sample profiles to the module, and also in this case I found out that the whole model was consistent, reusable, and elegant. I liked the idea of namespacing them explicitly, so that profile classes in the module are easy to recognize, and well distinguished from the component classes and defines: class apache::profile::passenger ( Variant[Boolean,String] $ensure = '', Hash $options = { }, Variant[Undef,String[1]] $template = undef, ) { include ::apache ::apache::module { 'passenger': ensure => pick($ensure, $::apache::ensure), template => $template, options => $::apache::options + $options, package_install => true, } } Having a minimal main class and using it as main entry point for all the general module’s variables implies that that class can, and should, be included in all the other module’s classes and defines. In order to be able to compose profiles freely and have more than one of them on a node the main class, and other classes, should always be included and never declared with explicit parameters. Data in modules, the Tiny Puppet way I continued this exploration with a Docker module, here integration with Tiny Puppet is even deeper, as tp is not only used to configure and install Docker, but also to build Docker images for any application, based on starting images from different Operating Systems. This is done via the docker::tp_build define which is used, for example, in the builder profile which can be applied to any node to convert it to a Docker build system, which can be configured with Hiera data like: docker::profile::builder::images: apache: ensure: present conf_hash: apache::mysite: base_dir: 'conf' template: 'profile/apache/sample.conf.erb' options: ServerName: 'www.example42.com' ServerAliases: - www.example42.com - www.example42.eu dir_hash: apache::example42.com: base_dir: 'data' vcsrepo: 'git' source: 'https://github.com/example42/example42.github.io' The above data creates an apache image containing the referred configuration file(s) and data directories. Note that Puppet and Tiny Puppet are NOT installed on the image: tp_build creates the proper Dockerfile for the chosen OS and app and the relevant directories and files on the builder host. Another small but nice use case of profiles is to provide sample code which can be used for demos, documentation or testing, such, for example, the ::docker::profile::run_examples. The Docker module uses two different sources for module’s data and this needs some more explanations: String[1] $data_module = 'docker', String[1] $tinydata_module = 'tinydata', First of all one concept must be clear, the approach used by Tiny Puppet to get data is not based on Puppet 4’s data in modules design (also because Tiny Puppet was conveived and released far before the current implementation of data in modules). It uses a custom tp_lookup function that looks for data in yaml files organized according to a hierarchy defined in a hiera.yaml file (it uses Hiera’s same syntax and logic but Hiera is not actually used for the lookups, read here for more details). The tp_lookup function allows the choice of the data module to use, by default Tiny Puppet uses the tinydata module where common settings for different applications on different OS are defined but in all the Puppet 4 modules shown here, the data_module is the module itself, which contains in its data directory not only the common settings, already defined in tinydata, but also more data, specific to the module’s application. For example in the ansible module I started to add all the default Ansible application options and provide a sample config_file_template: ansible/ansible.cfg.erb which is automatically added when the auto_conf option is true. Note that the <%= @options['keyname'] %> variables are the result of the merge of users’ custom $::ansible::options and the module’s default options shown before. This follows the options_hash + default options + [custom] template pattern described in this blog post. The Docker module is a particular case, as it has also a $tinydata_module parameter, which defines the data module to use to get info about any application (not Docker) and this data is used to build the relevant images. Stack modules Another experiment is with the rails module. Here class indirection can give the module remarkable flexibility on what components of a distributed Rails setup should be included by what nodes. The module is intended not only to manage the installation of Rails but of all the components of a complex multi-node Rails setup. This class is similar to what I defined in the past a stack module: a module included by nodes of different roles that concur to setup the whole Rails stack, each one “activating”, using class indirection parameters in the main class, the desired components (classes) of the stack (web server, database, caching server…) class rails ( String $install_class = '::rails::install::gem', String $proxy_class = '', String $app_class = '', String $db_class = '', String $deploy_class = '', [...] ) { In such a scenario we may configure a node (via Hiera) to be a all-in-one server by settings something like: rails::proxy_class: '::rails::proxy::nginx' rails::app_class: '::rails::app::unicorn' rails::db_class: '::rails::app::mysql' rails::deploy_class: '::rails::deploy::redmine' But we can also split these classes across different nodes, enabling or disabling the relevant classes. Here the deploy class can be used to manage different Rails applications, which might be provided as dedicated profiles in the rails or in other modules. So, bringing on this logic, the module is supposed to be a sort of super-module which not only manages the installation of the components of a distributed Rails architecture, but also the deployment and the configuration of different Rails applications, each one in their one, independent and interoperable, profile classes. Is this a bad idea as it breaks the first rule of modules design (Single Responsibility principle)? Maybe, or maybe it’s simply a different way to define boundaries and responsibilities of Puppet classes and modules. Note also how having different options for the install class may make the module more complete and feature rich. Check, for example, the different install classes currently available on that module. We have seen some sample modules following the patterns that David has described, they are not complete and may be over simplified. For example I like the options_hash + template pattern, and I prefer it over having classes with a huge amount of parameters (up to one for each application configuration entry), so I did these modules with this principle in mind. Still this is not a requirement. Once you keep a minimal “just install and do nothing else” approach for the main class (and if you do it, please use install_class indirection as it would greatly enhance its interoperability and potential) and move into profiles in modules all the customization, typical use cases, opinionated setups, you can actually have endless options. You might even provide in the same module alternative profiles using either an essential options_hash + template approach or one based on multiple (profile class’) parameters, for example. Also the idea of using Tiny Puppet in component modules (originally I considered it mostly as a possible replacement for component modules, to be used in local profiles) has proven to be useful and practical: it saves modules authors from managing a lot of common logic and module data and provides an handy abstraction on application management. Note that even if Tiny Puppet has a sort of own implementation of the data in modules concept, it’s not just a data in modules implementation. You can still have modules with data in modules and Tiny Puppet. This is the direction that the 4th generation of example42 modules are taking: this time we ABSOLUTELY don’t want to follow past errors, trying to write by ourselves most of our modules. We hope the community of modules’ authors will start to write modules following similar design patterns, this would allow us, and everybody, to more easily use third party modules and integrated them in existing Puppet installations. A good starting point to write modules based on these principles is this module skeleton. To create a module based on it from example42 control-repo you have just to run this Fabric task: fab puppet.module_generate The same example42 control-repo can be used as a reference on how roles, local profiles and profiles in modules can seamlessly work together. Alessandro Franceschi and David Schmitt 2016-05-11T00:00:00+00:00 https://example42.com/blog/2016/05/11/a-modern-puppet4-control-repo A few weeks ago we introduced version 4 of example42 Puppet modules with a radical change in the reference repository layout. We’ve started to work on a Puppet 4 only compatible control-repo setup and we explored alternatives or optimizations to current best practices. The term “control-repo” is relatively recent in Puppet world but its function has been common for a while: a single place where we manage our whole Puppet setup: our data, our code, the public modules we use. Needless to say that a good stating control-repo is vital to a sane Puppet setup. What we present here is a sample modern, rich featured, opinionated and customizable Puppet control repo with: Sample Hiera structure and data, using hiera-eyaml Local profiles optimized for Puppet 4 usage. Use of experimental Puppet 4 optimized modules based on new design principles Multiple, easily customizable, Vagrant environments, where to test the same control-repo code Docker integration: Data driven build images configuration and control-repo testing environments Fabric integration for Puppet code development, testing and deployment Tiny Puppet usage for extremely compact, consistent and readable code You can give it a try with: git clone git://github.com/example42/control-repo.git cd control-repo r10k puppetfile install -v The starting code and data organization and nodes classification is intended to be adapted ad customized according to different needs. By default in manifests/site.pp it’s used a node-less classification based on 3 top scope variables, $::role, $::env, $::zone: they can be set as facts (as done in the provided Vagrant environments) or via an External Node Classifier. We don’t use role classes: the profiles to include in each node are defined via Hiera, using the profiles key. The sample hierarchy in hiera.yaml uses data in hieradata. In the local site profile class we manage the baseline of resources common to all nodes, and role specific ones. The public modules installed via r10k are from various sources, not only example42 ones. The control-repo code is far from being complete, we are experimenting design patterns for modules and this is an ongoing process. Just consider that you can, and actually should, decide what to use and what to change in the code, the data and the classification logic: the existing base code allows any customization in a totally data driven way. Control repo testing You can test the control repo code and data while you develop or in your Continuous Integration pipeline. We provide both Vagrant and Docker based testing environments and some Fabric tasks to work with them. To install the required vagrant plugins: fab vagrant.setup To see the status of the Vagrant environments in vagrant/environments (there are different ones for different purposes): fab vagrant.status To start and provision a Vagrant vm do something like (or run the relevant Vagrant commands in the chosen environment): fab vagrant.up:vm=dev-local-docker-host-01 fab vagrant.provision:vm=dev-local-docker-host-01 To build locally Docker images for different OS (using the data in hieradata/role/docker_multios_build.yaml) fab docker.multios_build To test a role on a Docker instance with puppet-agent preinstalled: fab docker.provision:puppetrole=tpweb,image=centos-7 Developments on this control-repo and the example42 modules are going to proceed while we explore patterns for Puppet infrastructures based on Puppet 4 and beyond. Stay tuned, wonderful things happening here. 2016-04-13T00:00:00+00:00 https://example42.com/blog/2016/04/13/the-fourth-generation-of-example42-puppet-modules Example42 Puppet modules have been around since 2009, they have lived different generations. It’s time for a new one, the 4th. Almost a revolution. Let me give an historical perspective. The first generation of example42 Puppet modules introduced a set of features that were quite uncommon for the times: Support of multiple OS, based on a dedicated params class Possibility to remove (decommission) the resources managed by the module Abstraction and automation of firewalling and monitoring (Optional) Awesome (IMHO) integration with Puppi 1.0 (Optional) The reference layout of this first generation was done in pre Puppet 2.6 times, when there weren’t parametrized classes and the modules ecosystem was basically a collection of OS specific recipes. With Puppet 2.x many things changed and modules’ classes could finally have parameters that allow much wider customizations. At the beginning of 2012 started the release of the second version of modules, the NextGen edition. They were far more advanced and featured: Alternative ways to manage configurations (templates, static files, whole dirs… ) Extreme customization options (for example it was possible to manage Puppet Enterprise with the normal OSS puppet module, handling all the different packages names and paths) A standard parameters set, common across modules Better separation of custom code, data and files from module repo Use of different skeletons for quick creation of full featured modules according to their nature The params_lookup() function, provided by the Puppi module. Used for each class parameter. It partly anticipated the behavior of Puppet 3’s Hiera data bindings and actually allowed even more flexible lookup options. In this occasion independent git repos for each module were introduced with the main puppet-modules repo linking them as submodules. Support for Puppi 2 was introduced, but ultimately failed, since Puppi 2 was never finished. NextGen modules have been used for several years, across different Puppet versions and, I like to think, have in some parts influenced some design patterns in other modules. In the meantime the modules ecosystem was getting better and better and it was impossible for a single person, mostly working on his free time, to deliver quality comparable to the best modules around. It started a somehow decaying phase, where Puppet and its modules continued to grow, but work on Example42 modules became stale: the NextGen modules were starting to feel their age. With the intention to change this tendency it was opened an example42 GitHub organisation with different maintainers taking care of specific modules. The plan was to work on a third generation of modules without the params_lookup dependency (since it was somehow redundant having Puppet 3 data bindings) and with a more modern layout. It was launched the stdmod initiative with the intention of introducing naming standards in Puppet chaotic modules world. StdMod and this third generation never really took off, most of the modules are still based on the 2.0 layout and I personally failed in supporting properly the community and the contributors on GitHub. In the meantime it was started the work on Tiny Puppet which somehow embodies all the standard features of example42 modules in a single module based on an external data module, where it’s easy to add support for new OS and applications. So we arrive at current times, with an aged set of modules (most of them were designed even before the release of Puppet 3, four years ago), lack of proper motivation to support them, and better alternatives around. Time to take radical decisions for our fourth generation of Puppet modules: A huge amount of modules have been deprecated. For specialized tasks there are better alternatives around, for common package, service, files management Tiny Puppet can do the same. The repository layout has been overhauled: from a collection of submodules to a Puppet control-repo with a Puppetfile (with example42 and third party modules), sample local roles and profiles, sample Hiera data Vagrant and (soon) Docker environments are added to the repo for testing Puppet code and data. The repository is now more suitable as a starting point for a flexible and decently structured Puppet infrastructure. Work has just begun, a modules set is developed over time. Most of the current modules are still in their “NextGen” layout and all of them should work on Puppet 2, 3 and 4. Fourth generation is for Puppet 4. The sample profiles in the puppet-modules repository work only on Puppet 4, backwards compatibility may be preserved in the single component modules. In the next months we will see how this generation evolves, I have reasons to believe that it will be luckier than the previous also because something has changed in the meantime: there a company behind example42, not just a single person. Per Aspera ad Astra Alessandro Franceschi 2015-12-20T00:00:00+00:00 https://example42.com/blog/2015/12/20/puppet4-examples-epp-template Puppet 4 has some new functionality. Within the next few blog posts I will give some examples on how to use the new functionality. The first post covered the new Data Type system. The second post covered the new function API. This third post covers the new EPP template engine and the HEREDOC implementation. In Puppet 3 all templates were written as ERB (embedded Ruby) templates. All variables in ERB templates have either been looked up dynamically, or one needed to specify the scope for variable lookup. In Puppet 4 a new template engine was introduced: EPP (embedded Puppet). Within EPP templates varaibles are writtenin Puppet syntax - which means that variables can be specified by using the module/class namespace. Example (sshd_config): <% if $ssh::port { -%> Port <%= $ssh::port %> <% } else { -%> Port 22 <% } -%> <% if $ssh::listen { -%> ListenAddress <%= $ssh::listen %> <% } else { -%> ListenAddress 0.0.0.0 ListenAddress :: <% } -%> Besides this the array iteration also needs to be written in Puppet DSL code: <% if $ssh::port ~= Array { -%> <% $ssh::port.each |$port| { -%> Port <%= $port %> <% } -%> <% } -%> Additionally the EPP template now offers the possibility to make use of parameters (like parameterized classes). Parameters in EPP have to be put in the beginning and are enclosed in EPP tag and a pipe sign. <% | String $string, Array $array, Boolean $bool |> <% if $string { -%> Text from variable: <%= $string %> <% } -%> <% if $array { -%> <% $array.each |$element| { -%> Array item: <%= $element %> <% } -%> <% } -%> <% if $bool { -%> Bool value is true <% } -%> Data for EPP parameters are set by the epp or inline_epp function. This function now can have two parameters: the content of the template (either as file or as variable a hashmap for parameters Example: content => epp('test/example.epp', { string => 'foo', array => ['one', 'two'], bool => false }), Parameterized templates are useful when the template serves different puprposes for different modules. The non parameterized template should be used when only used by one specific module. === HEREDOC It has always been a pain having small config files in Puppet DSL - mostly due to having bad readable code: class my_motd { $content = "Welcome to <%= @fqdn %> This system is managed by Puppet. Changes will be overwritten on next Puppet Agent run." file { '/etc/motd': ensure => file, content => inline_template($content), } } In Puppet 4 we have a new way of having files being part of the code: HEREDOC. First, let’s migrate the example to heredoc and epp. Hereddoc needs a tag (set in round brackets) - like Shell heredoc class my_motd { $content = @(EOF) Welcome to <%= $::fqdn %> This system is managed by Puppet. Changes will be overwritten on next Puppet Agent run. EOF file { '/etc/motd': ensure => file, content => inline_epp($content) } } Next we will make use of the fixed identation by using a pipe sign: class my_motd { $content = @(EOF) Welcome to <%= $::fqdn %> This system is managed by Puppet. Changes will be overwritten on next Puppet Agent run. | EOF file { '/etc/motd': ensure => file, content => inline_epp($content), } } Now we want to make use of the heredoc substitution. Substitution can be enabled by putting the tag in double quotes: class my_motd { $content = @("EOF") Welcome to ${::fqdn) THis system is managed by Puppet. Changes will be overwritten on next Puppet Agent run. | EOF file { '/etc/motd': ensure => file, content => $content, } } if you need escape sequences then you need to enable them at the heredoc tag: $content = @(EOF\tn) THis enables tabular and newline escape sequences. The next posting will deal with several ways on how to upgrade to Puppet 4. 2015-11-24T00:00:00+00:00 https://example42.com/blog/2015/11/24/example42-at-cfgmgmtcamp The Config Management Camp has quickly become a conference of reference for who works around it automation and systems management. Ansible, CFEngine, Chef, Foreman, Juju, Puppet, Salt… communities reunite, share, discuss and confront tools, methodologies, approaches and evolution of configuration management. The next edition is the 1st and 2nd of February 2016, in Gent, Belgium, right after, and near, the Fosdem days. Registration is open, the event is free and tickets are quickly going away (book only if you really plan to come, there’ll be people in waitlist). Example42 is a proud sponsor of such a wonderful event. We look forward to meet again old and new friends, see you there. 2015-11-18T00:00:00+00:00 https://example42.com/blog/2015/11/18/tp-1-release So Tiny Puppet 1.0 is out with a dedicated website and some collateral repos: The tinydata module, where is defined all the application data used by tp defines. The Tiny Puppet Playground, a Vagrant environment where to play and experiment with tp The derived tp-acceptance Vagrant environment, where tests are done and summarized in this compatibility matrix Tiny Puppet [install|conf|dir|test|...] defines work only with Puppet 4, for older versions use the alternatives with 3 suffix. For example: tp::install # Optimized for Puppet 4 or Puppet 3 with future parser enabled tp::install3 # Backwards compatible versions. Work on Puppet 2, 3 and 4. Enjoy, al 2015-10-30T00:00:00+00:00 https://example42.com/blog/2015/10/30/first-guide-steps A few weeks ago we announced The [DevOps] Guide to Puppet, Universe and Everything. It is an holistic, Open Source, documentation project composed of: A book about Puppet and DevOps practices A set of commented sample code architectures A modular slides deck A practical Puppet Universal Reference We have launched an IndieGogo campaign about it, which actually is proceeding in quite a miserable way, but that won’t stop us from trying to deliver quality material for who wants to learn and use Puppet. This first output of the guide is here now: a set of 42 slides for Puppet beginners. It’s the first module of the slides deck (others are expected for more expert users). The slides markdown code is on GitHub. Check this link for the rendered slides. To see them from your computer using Showoff follow these instructions: # Install the showoff gem gem install showoff # Clone the slides repo and move into the Beginner directory git clone https://github.com/example42/puppetguide-slides cd puppetguide-slides/Beginner # Run a local webserver, on port 9090 that shows the presentation showoff serve To see the slides, browse to http://127.0.0.1:9090 and use the arrow keys to flip them, or press the T key to show a menu of the contents. We think that this Beginners slides deck, even if not finished (it will never be it as we plan to maintain and update such documentation over time), is ready for release and can give a valuable overview about Puppet world, and online references, to who is approaching it for the first times or needs some clarifications on its basic concepts. It contains 42 slides (just for this prime time, we will not struggle to keep this fancy number) that can be used in internal trainings, workshops, or for self learning. We are looking forward to hear your opinion on the work (use GitHub or example42 social forums for them) and if you want to support our efforts on the [DevOps] Puppet guide, give a look to the IndieGogo campaign. al 2015-10-26T00:00:00+00:00 https://example42.com/blog/2015/10/26/preparing-for-tp-1 Tiny Puppet is a Puppet module that abstracts application management. It provides a set of Puppet defined types to manage installation and configuration of potentially any application on any Operating System (tp::install, tp::conf, tp::dir… ). When Tiny Puppet was released Puppet 4 was still not available, so all its code is compatible with Puppet 3 (and actually also with Puppet 4 and Puppet 2). Yet the power of the future parser is undeniable so we started to make some Puppet 4 only versions of the main tp defines, adding a 4 suffix to them (tp::install4, tp::conf4, tp::dir4… ). Now we are approaching the release 1.0 of Tiny Puppet and a relevant choice has to be done: should we keep on preserving old parser compatibility or jump hands and feet on the future (parser?) We decided to jump, so starting from version 1.x the standard tp defines are going to be compatible only with Puppet 4. Puppet 3 compatible defines will have the 3 suffix (tp::install3, tp::conf3, tp::dir3… ), and, to ease migration on current codebase, they are already available. So, summing up, the current version of tp (0.9.x) has this layout for defines: tp::install # Works on Puppet 2, 3 and 4 tp::install3 # Works on Puppet 2, 3 and 4, the same of of tp::install tp::install4 # Optimized for Puppet 4 (doesn't work on earlier versions) When tp 1.x will be released, in mid November, we want to switch this naming as follows: tp::install # Optimized for Puppet 4 (doesn't work on earlier versions) tp::install3 # Works on Puppet 2, 3 and 4 the same of tp::install from 0.x The current code and documentation refer to the pre 1.x layout: this will be changed at 1.0.0 release. If you are using Tiny Puppet 0.x and have Puppet 4 you have probably nothing to do, the new functions work as before, a few, rarely used, parameters will be removed If you have a Puppet 3 environment you should start to use defines with the 3 suffix (renaming functions like tp::conf to tp::conf3), in order to be able to seamlessly upgrade to tp 1.x when released. If you are still not using Tiny Puppet, give it a try. It can do a lot of useful things for you, and, we think it will be able to do much more with the powers of Puppet 4. 2015-10-15T00:00:00+00:00 https://example42.com/blog/2015/10/15/born-at-the-puppetconf October 7th, 2015 has been a special day for us: example42 GmhB has been officially registered and can begin its business, based in Berlin. We were at the PuppetConf in Portland that day and we celebrated there as the youngest company at the event while Martin performed his talk about Puppet 4. We are enrolling as Puppet Labs Training and Service Delivery Partners and will provide to our customers a full range of Puppet related services. Business is business, but ideas don’t change: example42 is still embracing Open Source, all our Puppet modules are and will continue to be free as speech and beer and to stress our commitment we have launched a new, ambitious, project: The [DevOps] Guide to Puppet Universe and Everything. It is an holistic, Open Source, documentation project composed of: A book about Puppet and DevOps practices A set of commented sample code architectures A modular slides deck A practical Puppet Universal Reference We have launched an IndieGogo campaign about it, if you want to support this effort, please give it a look. We plan to release the first works on the guide very soon. Stay tuned, something awesome is going to happen. 2015-10-07T00:00:00+00:00 https://example42.com/blog/2015/10/07/puppet4-examples-functions Puppet 4 has some new functionality. Within the next few blog posts I will give some examples on how to use the new functionality. The first post covered the new Data Type system. This second post covers the new Function API. In Puppet 3 functions had limitation like no type checking unique naming required Puppet 3 functions had to be placed in a module in lib/puppet/parser/functions This has changed in Puppet 4. Functions now live in lib/puppet/functions Inside this functions directory other subdirectories can be used, to provide namespaces for functions. e.g. in module resolver: # lib/puppet/functions/resolver/resolve.rb Puppet::Functions.create_function(:'resolver::resolve') do ... end This allows module specific functions with the same name as other functions (e.g. v3 functions from stdlib). In fact this is not the same name. It is the same filename. The resolve function should return the Puppet master fqdn in case no argument is given. Hint: this functionality requires the socket gem. We now extend the function: # lib/puppet/functions/resolver/resolve.rb require 'socket' Puppet::Functions.create_function(:'resolver::resolve') do def resolve Socket.gethostname end end Please note: the def uses the function short name without the namespace ! With Puppet 3 we had to have multiple functions returning different data depending on the provided arguments. With Puppet 4 we now have a possibility to check for arguments data type and execute according function parts only. First we need to write dispatch definitions which will evaluate the given data type. We will continue with the last example: # lib/puppet/functions/resolver/resolve.rb require 'socket' Puppet::Functions.create_function(:'resolver::resolve') do dispatch :no_param do end def no_param Socket.gethostname end end Note that we now make use of the dispatch when running specific parts of the function. One can have multiple dispatch sections e.g. for different data types. We want to make use of this by adding tow more resolve calls: when provided with an IP address, it should return the hostname when provided with a hostname, it should return the IP address Hint: This functionality requires the rubygem resolv. # lib/puppet/functions/resolver/resolve.rb require 'socket' require 'resolv' Puppet::Functions.create_function(:'resolver::resolve') do dispatch :ip_param do param 'Pattern[/^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$/]', :ip end dispatch :fqdn_param do param 'Pattern[/^([a-z0-9\.].*$/]', :fdqn end dispatch :no_param do end def ip_param(ip) Resolv.getname(ip) end def fqdn_param(fqdn) Resolv.getaddress(fqdn) end def no_param Socket.gethostname end end We now can make use of the function in a manifest: $localname = resolver::resolve() notify { "Without argument resolver returns local hostname: ${localname}": } $remotename = resolver::resolve('google.com') notify { "With argument google.com: ${remotename}": } $remoteip = resolver::resolve('8.8.8.8') notify { "With argument 8.8.8.8: ${remoteip}": } When declaring a manifest with this code inside, the following result will show up: Notify[Without argument resolver returns local hostname: puppetmaster] Notify[With argument google.com: 216.58.216.142] Notify[With argument 8.8.8.8: google-public-dns-a.google.com] This function now fully relies upon working DNS resolution. One might want to add some sanity checks around the resolv and socket ruby code. The next posting will cover Puppet 4 EPP template engine. 2015-09-09T00:00:00+00:00 https://example42.com/blog/2015/09/09/puppet4-examples-data-types Puppet 4 has some new functionality. Within the next few blog posts I will give some examples on how to use the new functionality. The first post covers the new Data Type system. Let’s assume that you want to have a parameterized ssh class where users of your module might choose whether the server side should be installed. class ssh ( $server = true, ){ ... if $server { ... } .. } And now we use the module: class { 'ssh': server => 'false', } In this example Puppet will verify the String ‘false’ which will result in a boolean value of ‘true’. This is not what we expected. To avoid further misuse of the module we can add Data Types: class ssh ( Boolean $server = true, ){ ... if $server { ... } ... } If we now use the same class declaration from above we will receive the following error message: Error: Expected parameter 'server' of 'Class[Ssh]' to have type Boolean, got String at ssh.pp:2 on node puppetmaster.example.net What Data Types are available? Puppet has Core Data Types and Abstract Data Types. The Core Data Types are the following: String Integer Float Numeric Boolean Array Hash Regexp Undef Default Integers are identified by a number (with or without minus sign) and no decimal point. Integer $var1 = 1 Integer $var2 = -3 Floats are identified by having a decimal point Float $var3 = 1.0 Float $var4 = 0.2 Please note that you need to provide the trailing 0 Most of the other Data Types explain themselves. The Default Data Type is something special: The Default Data Type can be used in case statements and selectors: $real_server = $server ? { Boolean => $server, String => str2bool($server), Default => true, } Besides specifying the Data Types one can also specify Ranges of validity: class ssh ( Integer[1,1024] $listen_port = 22, ){ ... } This will check that $listen_port is set to an Integer value within 1 and 1024 and will fail if the boundary limits are not met. The Abstract Data Types are the following: Scalar Collection Variant Data Pattern Enum Tuple Struct Optional Catalogentry Type Any Callable One will mostly use Abstract Types which are built upon other Data Types. e.g. you want to express the Data Types of the content of an Array or an Hash: class ntp ( Array[String] $ntp_servers = ['pool.ntp.org'], ){ ... } A more complex example which takes a hash map: $hash_map = { 'ben' => { uid => 2203, home => '/home/ben', }, 'jones' => { uid => 2204, home => 'home/jones', } } class usermanagement ( Hash[String, Struct[{ uid => Integer, home => Pattern[/^\/.*/]}]] $hash ) { $keys = keys($hash) $keys.each |$single_key| { users::define{ $single_key: uid => $hash[$single_key]['uid'], home => $hash[$single_key]['home'], } } } define users::define ( Integer $uid, Pattern[/^\/.*/] $home, ){ notify { "User: ${title}, UID: ${uid}, HOME: ${home}": } } class { 'usermanagement': hash => $hash_map, } In case that one would like to allow certain words only, the Enum Data Type can be easily used: class ssh ( Enum['*','::1','127.0.0.1'] $listen_ip = '*', ){ ... } The Optional Data Type describes parameters which can be set to undef. The next posting will cover Puppet 4 functions. 2015-08-11T00:00:00+00:00 https://example42.com/blog/2015/08/11/example42-module-update The example42 modules received a larger update throughout the past couple of days: update to make use of new travis infrastructure add ruby 2.1.4 and puppet 4.2 rspec tests lint cleaning green travis spec tests We are happy to announce that all example42 modules which we will keep updated are now in proper shape regarding spec tests and are ready to get used with puppet 4. (Note: see the following list with example42 modules and deprecation status) We have not yet switched to new language features like type system and epp templates to allow compatibility with older puppet installations. Within the next step we will upload new versions to puppet forge and afterwards increase version and make use of puppet 4 only language features. Do not hesitate contacting us via google group in case of questions. We are looking forward to seeing you in Portland at PuppetConf 2015. Martin Alfke 2015-06-25T00:00:00+00:00 https://example42.com/blog/2015/06/25/example42-at-puppetconf We will attend PuppetConf 2015 which will take place back in Puppet home town: Portland, OR. example42 is one of the speakers and has given the opportunity to hand out conference tickets at 35% discount: Eventbrite. (Discount valid until July 31st) In our talk we will share knowledge regarding Puppet 4 major changes and deprecations and how to preapre your Puppet code for Puppet 4.x. You can find the schedule here: complete PuppetConf schedule Link to our talk We are looking forward to seeing you in Portland. Martin Alfke 2015-01-02T00:00:00+00:00 https://example42.com/blog/2015/01/02/introducing-tiny-puppet Tiny Puppet is a single Puppet module that can replace virtually any other single application module. Consider it as another Puppet abstraction layer, where we abstract the interface to the management of whole applications. I’ve been thinking about a similar solution for years, since when I started to create reusable modules based on a common template (all the “NextGen” version of Example42 modules). Now it’s reality. We usually deal with different kind of modules: A- Public modules that manage single applications (apache, openssh, redis …) B - Custom local modules that manage applications in the way we need C - Public modules that manage application stacks with multiple components (they are rare, think about shared and reusable profiles) D - Local site modules where we place our custom resources and logic (site, profiles, $project …) Tiny Puppet can be used as replacement or complementary for modules as in point A and B. It features: Quick, easy to use, standard, coherent, powerful interface to the managed resources Out of the box and easily expandable support for most common Operating Systems Support of a quickly and easily growing list of applications. Smooth coexistence with any existing Puppet modules setup: you decide what to manage Quick and easy integration tests It is intended to be used in modules that operate at an higher abstraction layer (as the ones in points C and D) where we assemble and use different application modules to achieve the setup we need. With Tiny Puppet the installation of an application is as easy as: tp::install { 'nginx': } And, once installed, you can configure it with: tp::conf { 'nginx': template => 'site/nginx/nginx.conf.erb', options_hash => hiera('nginx::options_hash'), } Tiny Puppet can do a lot more, though, for example you can populate any custom directory from a Git repository (it requires Puppet Labs’ vcsrepo module): tp::dir { '/opt/apps/my_app': source => 'https://git.example.42/apps/my_app/', vcsrepo => 'git', } All the reusability features explored and implemented during the years in Example42 modules have been ported to TP, for example you have multiple (alternate) options on how to provide a configuration file (the following example shows them all, but you can use only one of them at once): tp::conf { 'nginx': content => 'My file content', template => 'site/nginx/nginx.conf.erb', epp => 'site/nginx/nginx.conf.epp', source => 'puppet:///modules/site/nginx/nginx.conf', } You can override the specific data (file paths, packages and services names and so on) for a given application: tp::install { 'nginx': settings_hash => { package_name => 'my_nginx', service_name => 'my_nginx', }, } Tiny Puppet can handle virtually any application which may be installed via the local OS package manager, you can manage eventual custom repositories providing a custom class where you configure them: tp::install { 'elasticsearch': dependency_class => 'site/elasticsearch/repo.pp', } For some applications, a default extra repository is automatically added, you can disable this automatic lookup for a repository with: tp::install { 'elasticsearch': auto_repo => false, } and you can configure you own repo with the tp::repo define. Use cases You may wonder how Tiny Puppet manages all the application specific resources, such as Apache VirtualHosts, or Mysql Grants. Well, it doesn’t. Tiny Puppet manages packages, services and files, it provides a standard and easy to use interface to them, it adds OS abstraction and a lot of collateral frills, but everything is done by common, general use, defines which are feed by application specific data. Currently (but there are plans for that) it doesn’t manage explicitly application specific configuration options (such as Apache’s DocumentRoot, ServerName…): you don’t have explicit parameters to handle them, but you cana manage them as pure (Hiera) data with your own templates and the options_hash parameter. Tiny Puppet’s expected user is the System Administrator who knows how to configure his/her files and doesn’t want to study/import a new module just to provide resources and configure things in the desired way. Tiny Puppet can cohexist with any other Puppet setup, its only mandatory dependency is PuppetLabs’ stdlib module. Then you can decide for which application to use it or a dedicated module. You can even choose, in some cases, to use both tp and a dedicated module to manage some applications (for example using an existing mysql module to manage grants and tp to manage the installation and configuration of Mysql). Integration testing done easy If you want to give Tiny Puppet a try, you can use the Vagrant environment delivered in its repo, which allows quick testing of an application on different OS. You need/should install some Vagrant plugins and Librarian Puppet: git clone https://github.com/example42/puppet-tp cd puppet-tp vagrant status vagrant plugin install vagrant-cachier # Recommended for caching downloads vagrant plugin install vagrant-vbguest # Recommended for have updated VirtualBox Tools gem install librarian-puppet # Needed for the following command librarian-puppet install --puppetfile Puppetfile --path vagrant/modules/public vagrant up Centos7 bin/test.sh nginx Centos7 The bin/test.sh script can be used to run acceptance tests for all the applications on different VMs, for example to test the installation of apache on Debian7 you can run: bin/test.sh apache Debian7 Tests are based on a simple script that can be customised either directly from the tp::install define: tp::install { 'activemq': test_enable => true, test_acceptance_template => 'site/activemq/test.erb', } or using the tp::test define. Such a script would be placed, by default, in /etc/tp/test/activemq and might be used in any automation pipeline you want. Compatibility Matrix In the acceptance directory you can give a look at the current compatibility matrix of different applications on different OS. The files you see there are the result of the execution of commands like: bin/test.sh all Debian7 acceptance which runs the installation, the execution of a test script and the uninstallation of all the applications defined in data. The compatibility matrix is going to be reviewed and fixed with time, currently we have 275 successes and 165 failures on 5 different OS for the 88 supported applications (consider that some of the failures are due to recoverable or trivial reasons such as missing repos, incorrect test scripts, incorrect application data for an OS, problems with running serially all the tests on the same machine…). Adding support for new applications or OS Most of Tiny Puppet’s magic is done by the tp_lookup function which retrieves all the settings for an application from the data directory using a Hiera-like lookup method based on a dedicated hiera.yaml file present in each application subdirectory. Bacically to add support for a new application you can simply add a new directory in data and populate it accordingly, or, even more easily, run the script bin/moduledata_clone.sh: bin/moduledata_clone.sh -m test -n courier This this create a courier directory in data/ based on the contents of the test directory (I use it as a “sane” starting template). Then, obviously you may have to edit the files in the courier subdir to match different operating systems settings. The present and the future Tiny Puppet works like a charm on every system with Ruby 1.9.3 or higher, this means that you Puppet Master should run on: Ubuntu 14.04 Debian 7 RedHat 7 and derivatives and it needs some extra steps (namely the installation of Ruby 1.9.x) on other distros like: Ubuntu 12.04 Debian 6 RedHat 6 and derivatives Note, however, that this applies only for the Puppet Master node, where the catalog is compiled, on your clients, unless they run in masterless mode, you may run any other OS. Some applications have already support for FreeBSD, OpenBSD and Solaris and, as we have seen, it’s quite easy and quick to enlarge support coverage. Future developments will evolve around: Support and tuning of different applications on different OS Refactoring of the tp_lookup function to work on Ruby 1.8.7 Allow variables interpolation in the data files Automatic management of firewalling and monitoring Management of multiple instances of a given application Support for infile line-based configuration (tp::line) Command line interface to query, check and diagnose the status of the installed applications Addition of application specific configuration options and templates I’m quite excited about the future implications, power and impact of Tiny Puppet. The Example42 modules set is going to incorporate Tiny Puppet and will remove from the master branch dozens of component application modules, while introducing reusable higher abstraction modules (I call them stacks, because they are different from profiles as commonly designed and used). If you don’t care at all about Example42 modules you are forgiven, give it a try in your local site modules and see if it can be useful in some cases where you want the job done in a quick, easy and predictable way. If you have additions, fixes, suggestions or request for the support of new OS or applications, please use GitHub. Happy smart puppettizing with Tiny Puppet. Alessandro Franceschi 2014-10-29T00:00:00+00:00 https://example42.com/blog/2014/10/29/reusability-features-every-module-should-have The quality of Puppet modules is constantly increasing, for each relevant application there’s a good module to manage it. They may have different structures, features sets, OS coverage and naming conventions (:-I) but most of the times the better modules around do their work. Still, often, also in some of the most popular modules, I desperately miss a few basic features that would allow me to use them without being forced to make a local fork. Whenever we have to modify a public module to adapt it to our needs, we have a reusability failure IMHO. The good news is that a few very and easy to apply principles may improve dramatically our modules’ reusability without the need of a local fork. They are not rocket science, just common sense and more or less established patterns. Let’s review them. Expose a parameter to change the used templates. Always. Wherever we use an erb template in our module to populate the content of a managed file, we should add a parameter that allows our users to provide their own custom template. The reason is simple, in most of the cases we simply can’t provide a fitting template for the configuration needs of everyone. So, whenever we have in a module something like: file { '/etc/redis/redis.conf': content => template('redis/redis.conf.erb'), } We can expose in the class or the define that contains this file declaration a parameter that allows customisation of the template. For the above example, a quick, safe, and backwards compatible fix is as easy as: class redis ( $config_file_template = 'redis/redis.conf.erb', ) { file { '/etc/redis/redis.conf': content => template($config_file_template), } } Expose a parameter for a generic configuration hash What and how many parameters should a module expose? We may try to expose plenty of parameters to configure any possible configuration setting of our application, or we can expose only the “most important” ones, the ones that are more frequently changed, leaving room for ambiguity and arbitrary choices. We can also expose a single parameter where users can pass an hash of custom and arbitrary data, for whatever usage they may think about. Such a parameter is generally paired with the template one we have just seen. Let’s just add it to our class: class redis ( $config_file_template = 'redis/redis.conf.erb', $options_hash = { }, ) { file { '/etc/redis/redis.conf': content => template($config_file_template), } } Now our users can have (Hiera) data where they can define whatever they want: --- redis::config_file_template: 'site/redis/redis.conf.erb' redis::options_hash: port: '12312' bind: '0.0.0.0' masterip: '10.0.42.50' masterport: '12350' slave: true The template to use for redis.conf is taken from a custom local site module (so no modification is needed on the “public” redis module) and in this template we can access the $options_hash variable. A sample fragment of the file $MODULEPATH/site/templates/redis/redis.conf.erb might be like: port <%= @options_hash['port'] %> bind <%= @options_hash['bind'] %> <% if @options_hash['slave'] == true -%> slaveof <%= @options_hash['masterip'] %> <%= @options_hash['masterport'] %> <% end -%> Users can actually provide any kind of data in such an hash, and use it in their templates as preferred. They can use it for application’s configuration options, to manage triggers with booleans or to pursue any other purpose that fits their needs. Is such a generic hash a catch all solution that might replace all or most parameters used only to populate the contents of our templates? Well, eventually. Let’s look at the next step. Provide defaults for the options_hash I may understand the eyebrow of the module author reading this and wondering how to provide a working and useful setup out of the box with just a generic hash which is empty by default. Actually what’s empty can be filled with default values, exactly like the default values we place in parameters. We can, for example, set default values in this way: class redis ( $config_file_template = 'redis/redis.conf.erb', $options_hash = { }, ) { # Default configuration values $options_defaults = { port => '6379', bind => $::ipaddress, slave => false, timeout => '0', } # We use the merge function from stdlib to override the defaults with users' values $options=merge($options_defaults, $options_hash) } Now we can provide in our module a default configuration template that is highly customisable via the $options_hash parameter, with the extra benefit of providing good sample reference for users’ custom templates. Our module’s $MODULEPATH/redis/templates/redis.conf.erb should, obviously, use the computed $options variable which is the result of the merging of users’ data with our defaults: port <%= @options['port'] %> bind <%= @options['bind'] %> <% if @options['slave'] == true -%> slaveof <%= @options['masterip'] %> <%= @options['masterport'] %> <% end -%> Now we have the best of two worlds: Very few parameters (two) are enough to set any configuration value for the managed application In our modules we can set default values that are used to provide sane and working configurations out of the box Module’s users can manage any configuration element in their (Hiera) data. The level of detail is up to users and their needs. Up to now we have seen how we can improve users’ customisation of the content of the managed files. Let’s give a look to a final hint that can literally save our modules’ integrity (in terms of possibility to be used without changes) in many situations. Place extra resources in dedicated sub classes. Allow users to change them. A few months ago I expressed my opinions on the common ambiguity we currently have with Puppet modules about what they should and should not do. This is basically due to a not clear distinction we make in application component modules, which are expected to be the single responsibility point to manage that application, and higher abstraction modules, like profiles and stacks (Stacks are an approach to higher level modules I started to use as an alternative to profiles). What should a wordpress module do? Install only the wordpress files? Configure the webserver? Configure the backend database? If we consider it a component module it should just download the wordpress code and eventually manage its configuration file. If we want to work at an higher level, it should configure the web frontend, the database credentials and make everything work out of the box. This is the first thing that users expect from a module. The second one is the ability to adapt the module to custom needs. Often, authors, place resources that refer to some external application, in a dedicated subclass. For example wordpress::apache might be used to configure apache as frontend, alternative to a wordpress::nginx or whatever. In these cases it’s generally available a parameter that allows the choice of the webserver to use. That’s fine but I’d go further. For every subclass of a module, that groups resources somehow related to other modules, there should be a parameter that allows users to provide a custom version of that class. Most of the modules dependencies conflicts can be solved with such an approach. Let’s see an example with something as easy as: class wordpress ( $webserver_class = '::wordpress::apache', ) { if $webserver_class { include $webserver_class } } Small note with Puppet 4 the above code would not work as expected if we set an empty string as value for $webserver_class, a possible alternative could be something like: if $webserver_class and $webserver_class != '' { include $webserver_class } If our users want to use a different implementation of Apache (they may use a different, not compatible, module) or a different webserver, they can simply provide the name of the class to use with data like: --- wordpress::webserver_class: '::site::wordpress::apache' and define this class in the own site module, so have the file $MODULEPATH/site/manifests/wordpress/apache.pp with a content like: class ::site::wordpress::apache { # Anything needed to configure Apache as desired } This approach can be followed for other cases, for example, if we need to configure additional repositories to manage the installation of packages, we can con confine them in a dedicated class, or if we want to provide automatic firewalling or monitoring features with the module, we might place them in dedicated classes that allow our users to manage these features within their infrastructure. I bet you have not read anything really new to you in this post, still there a lot of modules from expert authors that don’t follow these simple patterns which are not expensive and intrusive (excluding the third point, which is somehow more opinionated an questionable). So, if you write and publish public Puppet modules, do yourself, your users, and me a favour, embrace these suggestions, we will have all a better Puppet life. Alessandro Franceschi 2014-10-13T00:00:00+00:00 https://example42.com/blog/2014/10/13/example42_goes_org The Github account where all the Example42 Puppet modules have been published so far, is now an organization on GitHub. This change is not only “a flag in a db record”, but involves a new approach on how Example42 Puppet modules are going to be maintained. The effort to keep this modules set updated and in good shape was becoming overwhelming for a single person not even working on them full time. Rspec tests and Travis integration were breaking, support for new OSes was not introduced timely, bugs and sometimes pull requests had been ignored for too much time. This was not a sustainable scenario and for this reason everything is changing. Example42 is no more a single-man effort but a community project. Whoever is interested and has the skills to do that, can become maintainer of one or mode modules. The current priorities for the existing modules are these: Fix rpsec tests and Travis integration. Review and work out the bugs and PRs queue Guarantee support for more recent versions of the most important OS Preserve backwards compatibility and the general design patterns of these modules. If you are interested in maintaining one or more modules, please contact me. This is not a disengagement from my own side, I’ll keep on working on these modules, but, simply, I can’t guarantee the support and dedication that their users deserve. Alessandro Franceschi 2014-06-30T00:00:00+00:00 https://example42.com/blog/2014/06/30/extending-puppet Last November I was contacted by Llewellyn Rozario, Acquisition Editor from Packt Publishing, asking me to write a book about Puppet. That has been an unexpected and welcomed surprise, I’ve never written a book and hardly considered the opportunity to write one, but the idea seemed too much appealing to be dismissed. The editor requested a book about how to extend Puppet functionalities with Ruby code, but in my own opinion there where some more interesting topics to talk about. I proposed a completely revised table of contents and a different title. I wanted to write about all the things I’ve learnt in several years of Puppet usage and had never written in books. I wanted to face relatively new topics, such as how to plan a Puppet migration, how to design reusable modules, the difference between component and higher abstraction modules, how to use Puppet on network devices, and so on. The table of contents was accepted and is almost identical to the final one, while my suggested title (“Puppet architectures”) was rejected, seemingly for marketing reasons. Whatever the title, the book is out now and available from Packt’s site and various online stores. “Extending Puppet” is a book targeted to who is already using Puppet (it’s not appropriate for absolute beginners) and is trying to find out the best ways to organize his Puppet infrastructure. Making of Since the beginning I knew that this had to be a challenging and hard work, but, honestly, I didn’t realize it could be that hard. The schedule to write the first draft was somehow dense, about 10 days for a 20 pages chapter. I started to write something as soon as possible, so at least for the first chapters I was ahead of schedule, but more time passed and more I found myself rushing to deliver the last chapter in time. Once I completed the first draft I thought that most of the work had been done, but that was only partially true. The technical reviewers (Dhruv Ahuja, C. N. A. Corrêa and Brice Figureau, they made a great work and helped a lot in delivering more precise and correct contents) pointed out various things to fix and at the same time I realized that my written english was, to say the least, quite inappropriate for a book. The book went under some further editing stages, with the support of the Technical Editors (Rohit Kumar Singh and Pratish Soman), the Copy Editors (Sayanee Mukherjee, Karuna Narayanan, Alfida Paiva, Laxmi Subramanian) and the Proofreaders (Simran Bhogal, Maria Gould, Ameesha Green, Paul Hindle) coordinated by Kartik Vedam and Azharuddin Sheikh. This is quite a remarkable number of people who have in a way or another contributed to the final outcome and have tried to make my words more readable and understandable. Besides all these great people who are in the official credits I’d like also to thank Joshua Hoblitt for the very useful comments on the contents and the language and to Nigel Kersten for having written the foreword. Contest: Get this book free The book is available online, at a very low price, if you ask me :-), but if you want to get it for free you can participate to a small contest by Packt Publishing. All you have to do is to tweet about “why you would like to win this book”. The tweet MUST contain this link http://bit.ly/extendingpuppet. Five winners will be selected and contacted by Packt to get a free e-copy of the book. In any case whoever can get a free sample chapter from http://bit.ly/1qbxWZy in order to have an idea of what you can expect from these pages. I feel definitively relieved at the idea that my weekends are finally free from some writing duties and I must say that I’m satisfied by the work done. I know that this is not the best written Puppet book around, my English writing skills are not up to a professional publication, but I think there’s much content in those pages, a lot of useful information also for experienced Puppet users. The cost is also lower than industry average, given the topic and the number of pages (I sincerely wonder how the publisher made the pricing :-I) and all in all I think this is a good deal. If you use and like Example42 modules and want to thank me for them, give it a try and help me in promoting this book, this would be greatly appreciated. Alessandro Franceschi 2014-06-22T00:00:00+00:00 https://example42.com/blog/2014/06/22/rethinking-modules-part-2 In the first part of this blog series I expressed my opinions about these points: The general reusability features a component module should have The difference between component and higher abstraction modules What are the challenges we have to face when we want to make reusable higher abstraction modules I also underlined two fundamental issues that I think are still open in the Puppet modules ecosystem: Patterns to extend reusability of higher abstraction layer modules Standardization in the component application modules Some preliminary exploration on the first topic has been expressed in Part 1. There’s much more to do about it. The second point is what we are going to discuss in this post. In the last months there has been the effort to stimulate collaboration around a standard set of naming properties for modules parameters: the StdMod initiative seems quite stale, it doesn’t seem to be a high priority for Puppet’s community. Still I think that a coherent and standard set of class and defines parameters, would be of great benefit. It sounds obvious, because it is obvious. How many times we just need, from a module, to simply install its application and have the possibility to manage freely and easily its configuration? How many times we had to write our own code for very simple needs because existing modules were too complex, or didn’t offer enough flexibility, or had dependency issues, or they forced us to study the module’s parameters in order to make them do what we wanted? Modules that just install and configure a single application (what we are calling component modules) should just do that, in the simplest, quickest, and most predictable way. During the years I wrote dozens of modules that had a standard interface: a common set of parameters that implemented basic reusability features, whose usage was easy and predictable. With a code like the following you could manage the ERB template of your application, whatever the module: class { 'openssh:' template => 'site/openssh/sshd.conf.erb' } Most of Example42 modules, therefore, had a standard layout which was duplicated for each module. The horrible side of such an approach is that when I had to fix some “core” code, duplicated on all the modules, I had to do it on several places. For example, most of the existing spec tests in Example42 modules are failing on Travis because they are not compliant with the latest versions of rspec-puppet. I should fix them, I don’t have enough time, will and motivation to do that. So I wondered if it was possible to replicate the reusability and coherency features of these modules in a more manageable way. Extreme component modules standardization: Tiny Puppet Well, it seems possible. Here is TP (Tiny Puppet), if it will turn out as I hope, it will rock. Consider Tiny Puppet as the essence of most of Example42 modules + some grains of Puppi. It’s a single module that allows the installation and configuration of different component applications, using a set of common defines. It’s supposed to be a replacement for simple application modules, when they just install and configure basic stuff, and a complementary tool for more complex modules, when they offer specific resources and options you may need. The project is still at its very early stages, consider it as ReadMe driven development, various of the features which are described in the following lines have still to be developed or refined. Installation and usage Tiny Puppet is a normal module, you can install it by placing the content of the GitHub repository in your modulepath. It will be published on the Forge, when it will be more complete and tested. It depends on Puppet Labs’s stdlib module, and, optionally (if you use specific features) on the vcsrepo and concat modules. It contains few basic defines that allow very specific functions: tp::install. It just installs an application and starts its service, with default settings tp::conf. It allows to manage configuration files of an application with whatever method possible for files (as an ERB template, as an EPP template, via the fileserver, managing directly its content…) tp::dir. Manages the content of a directory, either sourced from the fileserver or from repositories of the most common VCS tools (Git, Mercurial, Subversion, Bazaar, CVS) tp::stdmod. Manages the installation of an application using StdMod compliant parameters. tp::line. (TODO) Manages single lines in a configuration file tp::concat. (TODO) Manages file fragments of a configuration file These are the basic tools, but they would be of relative use if they weren’t coupled with application specific data. In the data directory of the tp module, for each supported application (currently very few, as I’m still defining the most optimal data structure and naming, but once this is defined it will be very quick to add support for new applications) there is a Hiera-like hierarchy of yaml files for different Operating Systems. This data, which the user can always override, allow usage patterns like the one we see in the following paragraphs. Worth noting is that the module is not invasive, it has limited dependencies and you can decide to use it only for the cases you need. For example you can use PuppetLabs PostgreSQL module’s types to manage grants and credentials and eventually use tp::install to install it and tp::conf to manage specific configuration files. Also, you can use a third party Apache module to install and configure it and tp::dir to manage the content of the documents root based on a VCS repository. Tiny Puppet usage in manifests Her are some sample snippets of code that fulfil specific needs. They all refer to redis, as this is the frst application I used to test data structure, but are going to work for all the applications dfined in the data dir. Install an application with default settings (package installed, service started) tp::install { 'redis': } Install an application specifying a custom dependency class (where, for example, you can add a custom package repository) tp::install { 'redis': dependency_class => 'site::redis::redis_dependency', } Install custom packages: tp::install { 'redis': packages => { 'redis' => { 'ensure' => 'present' } 'redis-addons' => { 'ensure' => 'present' } }, } Install custom packages, services and files ( The parameters feed a create_resource function, so they might be populated from a Hiera call ): tp::install { 'redis': packages => hiera('redis::packages'), service => hiera('redis::services'), files => hiera('redis::files'), } Configure a file of an application providing a custom erb template: tp::conf { 'redis::redis.conf': template => 'site/redis/redis.conf.erb', } Configure a file of an application providing a custom epp template: tp::conf { 'redis::redis.conf': epp => 'site/redis/redis.conf.epp', } Provide a file via the fileserver: tp::conf { 'redis::redis.conf': source => 'puppet:///modules/site/redis/redis.conf', } Provide a whole configuration directory from the fileserver: tp::dir { 'redis': source => 'puppet:///modules/site/redis/', } Provide a whole configuration directory from a Git repository (it requires Puppet Labs’ vcsrepo module): tp::dir { 'redis': source => 'https://git.example.42/puppet/redis/conf', vcsrepo => 'git', } Populate any custom directory from a Subversion repository (it requires Puppet Labs’ vcsrepo module): tp::dir { 'logstash': # The title is irrelevant, when 'path' is defined path => '/opt/apps/my_app', source => 'https://git.example.42/apps/my_app/', vcsrepo => 'svn', } Provide a data directory (the default DocumentRoot, for apache) from a Git repository (it requires Puppet Labs’ vcsrepo module): tp::dir { 'apache': # Prefix is a tag that defines the type of directory to use # Default: config. Other possible dir types: 'data', 'log', 'confd', 'lib' # available according to the application prefix => 'data' source => 'https://git.example.42/apps/my_app/', vcsrepo => 'git', } Configure a single line in an existing file (TODO): tp::line { 'redis::redis.conf::port': value => '1234', } Configure a fragment of a given file (TODO): tp::concat { 'redis::redis.conf': order => '10', content => 'port 1234', } Install an application and provide custom settings for internally used parameters (TODO): tp::install { 'redis': settings => { config_dir_path => '/opt/redis/conf', tcp_port => '3242', pid_file_path => '/opt/redis/run/redis.pid', }, } Tiny Puppet usage on the Command Line The previous code samples are expected to be used in manifests, mostly in higher abstraction modules where the single components applications have to be installed and configured as needed. All the logic on how to correlate and configure different applications may stay in these higher abstraction classes, which can use the tp defines to configure specific files as needed or lines inside existing files. Note also that tp defines are relatively light, and even if in most cases they just wrap native resources like package, service and file, they are probably slimmer and less resource intensive than a dedicated module. Still there’s something more that such a Tiny Puppet module can do: bring Puppet knowledge to the cli. It’s an old idea of mine which I implemented with Puppi but required some extra code on Exaple42 modules for a seamless integration: with Puppet we install and configure everything, we have complete knowledge on how an application is installed on a system, and it would be useful to query and use this information directly via a command line tool. Once you can do something with a single unattended command on your shell, you can do that in many different ways: inside scripts, on cron jobs, as remote command execution during a continuos delivery pipeline or an orchestrated sequence. That’s why I plan to introduce a tp Puppet face that can allow the execution of simple, and powerful, commands as the ones that follow. Install a specific application (TODO) puppet tp install redis Retrieve contextual info about an application (TODO). For example the relevant network connection, the output of diagnostic commands, the status of the managed application puppet tp info redis Check if an application is running correctly (TODO) puppet tp check redis This last command, still not implemented, reflects one of the features I liked most of Puppi when used with Example42 NextGen modules: the ability to immediately check on a system if the resources provided by the installed modules deliver a working application. Launch it from the shell, via Mcollective, as a step in a CI pipeline and verify immediately if your application, as configured with your own parameters, is running correctly on the provisionined system… a bit simpler than writing Beaker tests, isn’t it? Since puppet tp would be a face it could leverage on Puppet libraries and functions, and this may bring to some interesting interactions. For example a failing puppet tp check (eventually executed via cron) command may trigger a report message that could be used, by the report server (typically the Puppet Master) to send alarms or trigger the execution of other activities (for example a Puppet run), with an orchestration tool like MCollective. So what? Well, these are more or less the points behind Tiny Puppet. I’m genuinely interested in knowing opinions and suggestions about them. There are various implementation details to define and some decision on the structure of the data directory. The repository is on GitHub open for contributions. Alessandro Franceschi 2014-05-31T00:00:00+00:00 https://example42.com/blog/2014/05/31/rethinking-modules-part-1 I’m somehow obsessed by Puppet modules, it must be a rare syndrome, and I hope self awareness is the first step towards its cure. I’ve passed years developing modules, trying to find ways to make them useful, usable and reusable. Most of the times I wrote them while working on specific Puppet projects, trying to figure out how they could be used in different circumstances. I’ve redefined my opinions on how to design them over time, following the evolution of Puppet, the best practices of the moment, the feedback I received from other users and my own personal experience. I have to say that I’d definitively do things in different ways if I had to start from scratch, now. I’d dare to say that I would not write almost any of the 91 modules published on the Forge and the, even more, present on Github. Their maintenance cost is simply unbearable for the spare time of a single person, even with the great contributions from a lot of people. Something has to change. This post, divided in two parts, is about how Puppet modules might change. At least my Puppet modules. On this first part I’m going to express my opinion on some points: What are the reusability features a module should have The distintion between application modules and higher abstraction modules What are the challenges and reusability options for higher abstraction modules Modules reusability features I’ve tried to figure out what could be the features a module should have in order to be considered reusable. I might be a bit extreme on this point, but according to me a reusable module should provide: Support for different Operating Systems By default deploy a neutral setup, honoring the underlying OS default settings Support for different configuration approaches, such as: Complete files provided as ERB templates ( content => template($template) ) Complete files from a fileserver ( source => puppet:///modules/... ) Single lines in files managed by Augeas or other settings based defines Files built assembling fragments with concat Complete configuration directories Support for different usage behaviors, such as: Option to decide if configuration changes should trigger services restarts or not Options to activate debug or audit modes Options to provide custom classes, to be used as alternatives to the default subclasses, to manage dependencies or components of the module Support for custom names and paths for the managed resources (Package and Service name, files paths, and so on) Support for alternate installation options (via native packages, upstream sources and so on) Many of these features were introduced in the Example42 NextGen modules, using different modules blueprints: full featured templates for the generation of different kind of applications’ modules. I’d consider these the common reusability features a module should have: a shared baseline that any great reusable module should have. Then there are all the other application specific resources and features that can be added and can help in making users’ life easier and the module more accomplished. For example we can have custom types and defines that manage single elements of an application, such as: apache::vhost or mysql::grant, or classes that configure specific components of the module. Also, in many cases, the module’s classes expose parameters that allow the configuration of specific application settings. These parameters are generally used in ERB templates to shape configuration files as needed or to configure the application’s components, behavior or relationship with other applications and services. The kind and amount of such custom parameters may vary, a lot. My current opinion is that besides, eventually, some very core, useful and basic parameter for specific applications (For example: syslog_server , dns_servers) it should be enough to expose a single parameter where users can provide a hash with any specific configuration setting the application may have. Such a hash should be used with a proper template to manage any kind of configuration. The benefit is clear, we preserve the possibility to manage our infrastructure via bare data, without the need to add and update the module’s classes with application specific parameters. Higher abstraction modules Application modules (also called “component” modules) should follow a Single Responsibility Principle, dealing only with resources directly related to the managed application. Still in our infrastructures we have many applications that interoperate and have to be configured to reflect a consistent setup which involves different configurations for different applications to be assembled together. The Roles and Profiles pattern is an example of how we can work at a higher abstraction layer, using different component modules to setup whole stacks of applications which may run on separated systems. I think that application modules should not be opinionated, at least in their default functionalities. They should provide resources to manage different configuration needs, without enforcing any specific implementation. An application module should be considered as a library, an interface to the configuration of the underlying application. Something that does just a specific thing and it does it well, in a predictable way. So, by default, a component module should just install the relevant application and start it’s service, where present, with default settings. It should then offer a simple to use, coherent and robust interface to the application configuration. Such as the reusability features outlined previously. On the other hand, a higher abstraction module, has to be opinionated. It must provide a working setup, so it has to manage files and configurations in some specific way, as decided by the author. This is what many have done for a long time in their infrastructures: component modules, either public or written locally, have been used by custom internal “site” classes, where we’ve managed the composition of the services provided to the nodes of the infrastructure. In some cases public, shared, modules where forked and modified locally, in other cases people (rightly, IMHO), have preferred to keep unaltered in a dedicated directory these public modules. Sometimes a component module wants to do too much, and tries to manage resources related to other applications, which are needed for a complete and working setup. Here is where the Single Responsibility Principle is broken and the distinction between component and higher abstraction modules gets blurred. The reason is that just recently the difference from component and higher abstraction modules, such as profiles, has started to be defined. Reusable higher abstraction modules Up to know we have not seen may public examples of profile modules that can be adapted to different cases. How much flexibility such a module might offer depends exclusively on the author’s ambition to widen alternative options of his module. Let’s see an example of an higher abstraction module. A Wordpress module is a (apparently simple) case. It’s the typical borderline case, as it should just install WordPress, but in order to provide a working setup we have to configure the component modules of Apache, Php and Mysql in a very precise way managing Mysql grants, Apache virtualhosts and Php settings. Here, we should actually have two different modules: The component module, that just installs and configures WordPress The higher abstraction module, that manages the whole infrastructure needed to provide a WordPress site. Let’s concentrate on the latter. Such a module, or class, might expose parameters that allow to set high level settings (virtualhost names, database credentials…) but it has to be opinionated on the setup, from the same core choice on the web server and database backend to use, to content of the actual configuration files delivered. To offer better reusability options, it might expose parameters to manage which component to use, for example allowing users the option to use Nginx and PostgreSQL, instead of Apache and MySQL. It also could allow the possibility to compose the deployment of its various components on different, separated, servers and eventually add features to manage reverse proxies, load balancers or caching servers. All such reusability options definitively augment the complexity of the module to a point that we might question if it’s worth the effort. Still at least some composition options are just needed, an example of a wordpress profile that installs all its components on a single server might be useful for training purposes but is almost useless in a modern, mid size, setup. I hope to see sooner or later some examples of reusable profiles. To test and validate the idea (reusability, composition on multiple servers) I’ve make this, essential, logstash stack which is actually in production on at least a site. Besides the specific implementation, I think that a reusable higher abstraction module should provide: Support for different Operating Systems, where possible By default a working setup, based on opinionated choices about what components to use and how to configure them Support for alternate templates. For each application for which a configuration file is managed via a template, users should be able to provide their own alternative one Parameters that allow configuration of high level settings Optional support for monitoring and firewalling classes, with the possibility for users to override the default classes for these tasks, if present Optional support for different components (for example: different web servers, database backends and so on) Compositioning options, that allow users to install the needed components on different nodes adapting to different infrastructures. The last point is particularly important as is the key for real reusability and, at the same time, introduces new challenges that raise the complexity of the module and opens new questions on the module’s usage patterns. I think that what is still missing, in the current Puppet modules’ ecosystem, are two fundamental points: Patterns to extend reusability of higher abstraction layer modules (as described in this post) Standardization in the component application modules I suppose and hope we will see proposals and suggestions on how reusability patterns will be extended to higher abstraction modules, maybe clarifying the naming (here sometimes I talked about “reusable profiles”, even if profiles, as originally described in Craig Dunn’s post, are not so similar to the reusable abstraction modules discussed here). As for the second point, a possible approach on how to face the standardization of the component modules, or even rethink the whole concept of component modules, is going to be presented in the second part of this post. 2014-05-12T00:00:00+00:00 https://example42.com/blog/2014/05/12/from-drupal-to-jekyll Example42 site has been moved. It was running on Drupal, with a few plugins, an essential theme and a hugely spammed comment system. I hardly used most of Drupal features and was quite annoyed by the continuous need to update it, even if this can be done quickly with drush. When I started to look around for alternatives I searched for a lightweight Content Management System that could allow me to write quickly texts and save me from using directly html tags and badly manageable WYSIWYG editors. Jekyll and its integration with GitHub Pages seemed the right solution. Being a design donkey I searched for some good theme, based on BootStrap (because eveyrthing nowadays has to be based on Bootstrap, right?). I found Bootflat. I like it. Powerful, essential, clean. I imported my previous blog posts from Drupal using Jekyll Import and copied from Carl Boettiger site some plugins and design approaches. The result is here, credits and my personal thanks go to the previous resources. Thanks you! Only time will tell if this is enough for me to write more stuff here. For the moment, thank you for being here, links around will point you to the online resources where you can see my Puppet works. Alessandro Franceschi 2013-09-27T00:00:00+00:00 https://example42.com/blog/2013/09/27/talking-about-evolution I recently posted a comment about the evolution of Example42 modules in the Example42 Puppet Modules Google group.   It contains my opinions on how modules should be and how could evolve the new generation of Puppet modules.   I repost it here, as it gives a good overview of my current ideas and the evolution of Example42 modules.   ---    Let's move. This is not something that I can do alone. NextGen modules are there, somehow stable, doing their work. They are requiring a lot of support that I can't give in my free time. Even for testing submitted patches.   We have to start to work on brand new 3.x modules and a sane ecosystem.   I suggest these principles: 0- No NextGen legacies  1- Real single point of responsability for modules 2- Higher abstraction  stack modules 3- Follow stdmod naming conventions 4- Distributed development (modules maintainers) 5- Implementation and usage patterns coherency     0- No NextGen legacies   The current NextGen modules should not be taken as reference, no needs to have backwards compatibility. Let's take the good things (defines, implementation patterns, correct params and setups for different OS ) and build from them. Let's cut all the parameters and functionalities that don't fit with points 1 and 3.     1- Real single point of responsability for modules   This is important and critical for the whole long term feasibility and reusability of the modules: an application module should only expose what's needed to configure and manage its application. No relation to third modules, no options to provide implementation logics of higher abstractions. For example, an apache main class should expose just the essential parameters to just manage (in a reusable way) the application. Then the module may have custom defines to manage modules, virtualhosts, users, or whatever. We can reuse the good ones around, adapting the parameters to stdmod conventions. It might even have something like apache::passenger, with its own parameters, but is should not expose any passenger related parameter in the main class. I would add to these basic parameters the ones need to firewall and monitor them, keeping them in the most simple and open way, just 2 parameters: $monitor_class - The name of the class to use to manage monitoring. $monitor_config_hash - An open to any option configuration hash (we may stanrdardize it when working on the new monitor / firewall implementations )  Same for $firewall_ I'd also provide in each module (that needs them) standard "general purpose" defines to: - Manage single configuration files (useful when multiple config files are needed and you don't want to clutter the main class params to manage them) - Manage single configuration entries (via augeas?) No read need, IMHO, to have different defines with specific parameters for different configuration files, just assure there's a "catchall" config_hash parameter. So just dumb interfaces to the application's configuration , all the logic and the implementation details can be provided by 2: This would allow quick creation of basic modules with at least these "minimal" features, then we can add or import more specific ones.  Here we have to place rspec tests, rspec-system tests and essential documentation (all the stdmod standards related one can be avoided, imho)   2- Higher abstraction  stack modules   Here we move all the logic to manage modules interdipendences and integrations. So we might have modules like: puppet-stack_lamp puppet-stack_liferay (but also just puppet-liferay as long as it follows stacks' patterns) puppet-stack_rails puppet-stack_logs ( here's a sample https://github.com/example42/puppet-stack/blob/master/manifests/logs.pp ) puppet-stack_puppetinfrastructure ( https://github.com/example42/puppet-stack_puppet  )  Stacks do, in some way, what we often do in our puppet setups (and have called them roles, hostgroups, profiles) exposing higher level parameters that allow and manage the logic provided by the stack to integrate different application modules. They are supposed to be used by nodes doing different things which are needed to build the stack. In stacks some parameters could be managed in automatic ways, based on the nodes who use their single components. With puppetdbquery some wonderful things could be done. The parameters exposed by a stack class should be about: - if to include a specific application in the node ( install_webserver ?) - what to use for each stack's application (where alternatives are provided) ( webserver_name ? ), - how to configure them (parameters hash, custom template files ) - endpoints, credentials, urls… Stacks are the place for integration tests.      3- Follow stdmod naming conventions   They are still not definitive, I really urge whoever is interested in a good shared modules ecosystem to contribute actively to https://github.com/stdmod We definitively need to decide on few remaining things, IMHO, like: file (file_path? config_file? config?) (1) file_source (source? config_file_source? config_source?) file_template (template? config_file_template? config_template?...) Since we are doing this as users of Ex42 modules we might decide to choose the current names for the most used parameters (template? source?) as long as they make sense in the whole stdmod picture.     4- Distributed development (modules maintainers)   This can't be an Example42's only work. I can't and don't want to manage multiple dozens of modules of applications I don't know or use. Once good blueprints are available for different kind of modules, we can quickly create new modules based on them, adding and adapting the available defines, facts, functions from nextgen and  other open source modules. Possibly involving the relevant authours, also as maintainers. Code can stay in each maintainer's repos , Dolf, I, whoever can manage their own collections of stdmodules and stacks, pointing to gitsubmodules at the desired version / head (or managing them in a single big git repo) Of course just one "stdmod" module for application would be the ideal: forks just for pulling back to upstream, as managed by the official maintainer(s). All this, strictly following point 5:     5- Implementation and usage patterns coherency   It's very important to keep coherency inside the modules set, both for naming and usage patterns. For this the sooner the finalize naming standars and we have good samples and  templates, the better. I would stick the code of the modules for Puppet 3 compatibility, no need to do now things that support older versions and too early to embrace fully Puppet 4 ( > 3.3) only  features (data in modules, future parser, containers… ). They will be useful probably to manage in a smart way firewalling and monitoring, but given the simple basic layout. For this reason I would leave open parameters like monitor_class or monitor_config_hash that may adapt to any future evolution.     Next steps… - propose blueprints for common  module's layout. - make quick and easy common usage, single package modules (to just "include"): zip, wget, git… - make essential modules (to improve with time) for the most cross used apps: apache, nginx, mysql… - explore stacks my2c Al 2013-08-31T00:00:00+00:00 https://example42.com/blog/2013/08/31/nextgen-modules-lessons-learned Two years have passed since the first works on Example42's NextGen modules and I think it's time to review what has been done, what has worked and what hasn't. The NextGen modules layout has introduced various solutions and approaches to reusable modules design which have been more or less successful: some have been used also by other modules authors, some have been made obsolete by Puppet's evolution, some have remained practically unused.Some of the ideas used were already around or emerging, other were quite new and unexplored, some are still valid, others would probably be done differently now. What I can say is that my main task of that module set has been achieved: I, and, afaik, various other people, use effortlessly the same modules unmodified on different environments and infrastructures: that's what I want from a reusable module. What worked, what failed, what could be better Params lookup This is a function that is used on every main class parameter and allows choice on where data is defined: on an ENC (as Top Scope vars), Hiera or directly passed as parameters to the class. Dan Bode gave me the original idea, on which I added some frills like the possibility to look for a global variable after a module specific one.  Basically is the same concept of Puppet 3's data bindings, with the difference that it has been introduced before the release of Puppet 3 and it works on any version of Puppet > 2.6.The point is that now, with Puppet 3 more and more used, such a function is quite redundant and introduces some extra calculation time that might be avoided. Alternatives for configuration files All the NextGen modules have params like template and source that let users decide how to provide configuration files. The usage of a parameter like template is now common and practically required for a reusable module, at those times it wasn't. I think this kind of params (eventually with a wider "content" one) must stay in a module that aims to be considered reusable. Also there's the possibility to manage whole configuration dirs, with source_dir , source_dir_purge, which even if not generally recommended with large amount of files to syncronize, it can be a valid solution in some cases. Decommissioning support To provide parameters that allow the removal of the managed resources (like 'absent') is another possibility that at the times wasn't much spread and it's becoming more and more common. My only concern here is the unfortunate naming choice, something like 'ensure' is definitively a better and clearer name. Service management options Params like disable, disableboot and service_autorestart were introduced to let users decide how to manage services startup and their behaviour when config file change. I've seen them being used also in other modules and this confirms me that they make sense, even if also in this case I think that the naming choice was quite poor. Multi OS support and user's override options Multi OS support was not new for modules also at the time, I consider it a condicio sine qua non, for a reusable module, and the params class pattern was probably the best way to manage it.Now this is going to change with Puppet 3.3.0 , Hiera 2 and data in modules, where a brand new world opens to module's internal data management.NextGen modules also expose all the OS specific params as class parameters, so that the OS defaults values set in params.pp can be overridden by users: this has the consequence of giving more reusability options for edge cases but adds also a bunch of parameters which are rarely used, such as: package, service, config_file, config_dir, config_file_owner, config_file_group, config_file_mode ... Probably I'd keep only the first 3 or 4 of these parameters, now. Integrated monitoring and firewalling options I firmly believe that a module should provide the possibility to automatically monitor and firewall the resources it installs and do it in a "tool neutral" way, that is it should not contain parameters related to specific monitoring or firewalling tools. I've seen this concept (mostly the firewalling integration) be reused in other modules like the PuppetLabs ones, and I think it's definitively worth to be followed. The current implementation, with the usage of meta classes like Example42's monitor and firewall ones, still doesn't satisfy me fully and also I'm wondering if and where it makes sense to expose in the module all the parameters required to make this work ( monitor , monitor_tool, monitor_target , firewall , firewall_tool , firewall_src , firewall_dst and other ones like port , protocol , pid_file , process , process_args ).Probably the wonders and the evolution of the Puppet  Future Parser , which will probably be the default in Puppet 4, will allow better management of params like this, with the usage and manipulation of configuration hashes that would limit the exposure of a bunch of parameters and the possibility to freely expand them with tools specific settings. Puppi integration This has remained an incomplete work. One of Puppi's aims is to use Puppet's data to feed a CLI command. "Puppet Knowledge to the Shell" was my mantra, and I still think this can lead to powerful results. The problem is that the current Puppi still doesn't work with NextGen Puppi integration (which is quite ridiculous, I admit) and the Puppi 2 which should support it is still incomplete. Also in the whole picture is missing a sane web front end for all the data that Puppi might collect on the system.It's all in the TODO list, but it's there for quite a long time, so I would rate the Puppi integration in the NextGen modules a failure.Also the Puppi integration bring a bunch of other hardly used parameters like data_dir, log_dir, log_file and various ones already used for other functions.Note that you can still use Puppi for local puppi checks (setting it as monitor_tool) and for its other main function, application deployment, and that works quite well. Templates + Options Hash pattern I don't like the idea of adding a parameter to a class for each/most configuration option of the managed application, as you might end up adding a large amount of parameters to your module.For this reason a quite open solution, if you really want all your configurations as data, is to provide a custom template and feed it with a single configuration hash via the options parameter.The puppi module (which you can consider the stdlib for Example42 modules) provides also a useful function, options_lookup , contributed by Mike Novak, which allows easy usage of the options hash in an erb template. I'd like to see such an option in the stdlib.Personally I've not used too much this solution (generally I just provide a custom template that has encoded most of the specific settings and uses variables only the most important or qualifying ones),  and I suppose is the same for others, also because in the modules there weren't sample templates that could show some usage patterns. Debug, audit and noops Some extra juice was added to the modules.A debug parameter which dumps the whole class scope in a file. Useful, even if I rarely use it.An audit_only parameter which was supposed to let the user define to audit the changes of the module. Never used it.A noops parameter, introduced later, which is supposed to run per module noops (the noop metaparameter on the class has no effects on the contained resources). The idea makes sense, imho, the implementation was buggy and incidentally I've spent the last weekend fixing it on the modules who have this parameters, so, it's time to update them! Custom and dependency classes In all the modules there's a my_class parameter which allows you to define a custom class where you can place extra resources related to the module. Strictly speaking this is not needed, as you can place these resources directly in the role/service class that uses the module, but the idea is to have whatever is related to a module (config files, extra resources and so on) defined in a single point. Not essential but not harmful, imho. Recently I've also started to introduce a parameter like dependency_class, which allows the definition of a custom class where resources needed by the module, but provided by other modules, could be placed. The idea is to give the user the possibility to use other modules different from the Example42 ones to manage the required resources  ( database operations, extra repos , virtual host defines ... ). I think that such an option is useful to allow better integration with other modules and to allow usage of single Example42 modules without entering into dependencies hell with modules from other authors. Modules cloning via templates / blueprints  All the nextgen modules are made in a way that is easy to rename them, make some sed works and have a brand new full featured modules with limited effort. Actually there's a script available to clone Example42 modules that allows easy creation of a new module based on an existing ones.More than once someone has told me that if a module can be generated from a blueprint then there should be a saner way to inject data into a single module layout without making a new module every time.Actually the idea behind a structure that allows quick cloning is not to have modules that are all the same (with some changes in the params class) but to have a solid and standard base from which to build features specific to the application managed. This approach has permitted also other authors to create modules based on the NextGen layout and actually various of them, like the ones from Netmanagers guys or Marcus Burger or others, have been integrated and linked in the NextGen modules set and are managed directly from the original authors. That's a model I like and hope to see it grow. Docs, Lints and Spec tests All the modules have PuppetDoc compliant documentation, and are tested via Puppet Lint and Rspec Puppet. This is basically a good thing, and it's effortlessly cloned from module to module. What I must admit here is that not always I've given much care to the documentation or the test coverage of the new, module specific, features. The result is that documentation looks often all the same and the tests have not that much added value.This is something that I'd like to better care in the future. What's next The future parser and Hiera 2 may really change radically the way we design our modules and there's so much development in the Puppet ecosystem that is clear that we are moving to another transition, where the DSL is radically opened to new solutions and patterns.  If I had to rewrite the NextGen modules now I'd do some things differently: no params_lookup, better naming choices, less parameters, but I think this layout still works well with current setups and actually I'm using it in production in many different places.There will be sooner or later a third iteration of Example42 modules:- The first one, was done in pre-2.6 era, it has some interesting concepts, for the times, and various drawbacks. I'd not use it anywhere now. Actually I should remove the OldGen modules remained in the Official Example42 modules set.- The second one, the NextGen, is strongly based on parametrized classes, requires Puppet > 2.6 and works well with Puppet 3, even if the params_lookup is redundant.- The third one has yet to be done, and it might be Puppet 4 only compliant (or at least Puppet > 3.3), trying to fully use the future parser, data in modules and what will be available.It will also probably use rspec-system, adhere to stdmod naming standards, whatever they will be, and be based on stdlib.Also I'll definitively try to find a more shared and open development effort, so that applications specific experts can create and be maintainers of the modules they need and use. There's time to experiment on that, some patterns reveal their degree of success or failure only when they have been used and tested in different conditions.Puppet 4 is expected for the end of the year, but some time will pass before it gets massively used in production, so I presume the NextGen modules are here to stay for many months. I'd love to discuss about this and what has been written here directly on Example42 Puppet Modules Google group (I definitely surrendered at the idea on managing comments on this spam-infested blog). And thanks for reading up to here! I'm not good in writing short blog posts :-) 2013-06-15T00:00:00+00:00 https://example42.com/blog/2013/06/15/the-handy-grail-of-modules-standards Update 20140206. Some of the links and the info on this article may be outdated. For current implementations and discussions refer to https://github.com/stdmod . Current namings are here: https://github.com/stdmod/puppet-modules/blob/master/Parameters_List.md Discussions about Puppet modules standards have been raising and decaying for years: occasional posts on the Google groups, some presentations at conferences, gists, twits and bits around the net but, up to now, this topic has never been pushed enough to turn into something real or widespread. Still the matter is important, and we see it more evidently now, when we have several hundreds of modules on the Forge and GitHub, which are typically reviewed, cherry picked, forked and painfully adapted to custom needs by so many Puppet Masters.  When you start to select modules from different authors you inevitably get into dependencies conflicts and different usage patterns, you start to have local, not merged, forks of upstream modules, and introduce hacks that inevitably affect your code cleanness. Puppet is no more a young and quickly growing software, I'd say it's in an early adulthood, with established principles and patterns but still with a long life to sustain and feed. Now, even more than before, is time to start to make some order in the modules ecosystem. The topic is large and debatable, but can be begin from some simple, standard naming convention in modules' usage. If a "Standard Module" exposes a minimal set of common parameters that manage some of it behavior and usage patterns, it will be easier to switch modules, eventually merge the best parts of different modules for the same application, use them in a predictable and conventional way and also make them interoperate better. It's not something that can happen in a short time, is has eventually to be embraced, not enforced: a suggested set of naming principles for modules interfaces that suggest re-usability best practices and ease integrations.  Standard benefits Standard naming conventions can be set at different levels: - In the module's main class parameters - In parameters used by defines, according to the kind of define- In custom types  - In specific frequently used and typical cross-module defines, like apache::vhost, mysql::grant and similar. Different levels of 'standard sets' might be defined, so that modules' authors can support only a subset of them and always be free, to add custom parameters that enhance and enrich any specific module. Such Standards are not intended to set rules on how modules should be done, but how they should interface with others: we are basically talking about the names of the parameters to expose and the relative function, not how they are implemented.  Moreover suggesting some parameters may help the same module's reusability features and generally enhance the average quality of the modules around and ease their integration. Benefits of some standard modules interfaces are quite obvious, but let me recall you some of them: - Better user experience (modules are easier to use and understand)  - Quicker and more reliable Puppet manifests development (for basic functions you can expect predictable parameters) - More coherent and error proof Puppet setups  - Better interoperability among different modules In longer terms further benefits may emerge, like: - A PuppetLabs and/or Community driven central repository of well tested and features rich unique Standard modules - The possibility to have an unified approach to smoke testing of common features  - The possibility to have web front-ends and ENC that leverage on the standardized parameters  - Easier integration with superclasses that expose their own parameters and use different modules to build up full applications stacks Samples and references to spark the discussion  To talk about standard naming in abstract may lead to long and pointless speculations, so I'd rather show you some examples of what could be these minimal standards. Consider them as a draft, a proposal, some scribbled ideas not necessarily on how modules should be done (even if I personally think they are good modules layouts) but on how the same interfaces (parameters) can be implemented in different ways: https://github.com/example42/puppet-stdmod https://github.com/example42/puppet-stdmodalt  They reproduce the typical package-service-configuration file example, different examples (and relevant "standard" parameters) may be defined for different kind of modules (for example modules based on Java applications, for which some Java specific parameters may be added). A draft proposal for standard naming conventions can be discussed on this Google doc, for the moment it can work, I suppose.  If you want to comment or contribute send me your Gmail account. Some notes - The parameters used in the main class might be questionable and not all necessarily required: you may decide to implement only a part of them, eventually finding a convention to identify differnet levels of "Modules Standards coverage" and of course you can add other parameters more specific to the managed application. The idea is not to have all modules made in the same ways, but to leverage a set of common parameters. - Consider the parameters used in the above modules, or described in the Google document a  draft for version 0.0.1.  - Some of the parameters are particularly useful for a proper module reusability: leaving to the user freedom on out to populate the main configuration file (**source** and **template** (or **content**) parameters) or the whole configuration directory, when it exists (**dir_source**, **dir_recurse**, **dir_purge**) covers a great part of what you typically need to customize the module and, most of all, it doesn't force any specific approach on how you manage your configuration files: use can provide it via static source files, custom templates or other methods (for example with custom concats)  - A parameter like **options** which expects an hash of any kind of configuration elements for a specific application, when used with a properly structured **template** adds another huge layer of customizability that allows you to pass any module specific configuration parameter without actually modifying the module  - Parameters like **ensure**, **version**, **status** and **autorestart** allows you to manage partial application or also removal of resources provided by the module and to better manage the behavior of the service provided the the module, covering some specific but not so rare use cases.  - Parameters like **noops** and **audits** allows dry runs of a single module or better fit for Puppet Enterprise Compliance features. - Parameters like **my_class** and **dependency_class** allow automatic loading of custom classes, either to extend (not necessarily inheriting) the resources provided by the module or by giving more freedom on how to manage dependencies. This is an approach to module's interoperability which deserves some explanation. Interoperability matters A good basic rule for a module is that it should manage only the application it provides. For example a Wordpress module should not manage apache/nginx, php and mysql, as there are specific modules for them. At the same time if you want to provide a working Wordpress module you must somehow manage resources related to other modules, such as virtual host files or database grants.  This hasn't been really solved in the modules ecosystem, the Modulefile is useful to automatically manage dependencies for modules but it hasn't solved the issue on how to happily have different modules behave well together without dependencies conflicts.  A *part* of the solution may be the adoption of a parameter like **dependency_class** which defines the names of a class that contains all the external dependencies the module has. By default its value should be blank or a class inside the same module that uses modules and resources chosen by the module's author, but the same resources can be provided in any way by the module's users passing a different custom class name where these dependencies are managed. That would allow the possibility for users to bypass some troublesome dependencies and work around them in the most fitting way. This behavior might be also reflected in some additional parameter for the Modulefile, for example considering the key **dependency** only for required modules dependencies (modules like stdlib, for example) and an alternative key, such as **soft-dependency** where are placed the modules used in the (customizable) dependency class. Via the **puppet module** tool you might then decide if to install only (hard) dependencies or also soft dependencies. This is just a proposal, which, together with standard naming, may improve modules' reusability and interoperability. What now? Talking with different puppeteers and modules authors I realized that frustration in a seamless usage of external modules is rather wide and that forking and keeping local copies of public modules is much more common than upstream contribution. This results in large waste of time, less code reuse and less general quality of modules. To introduce a Standard naming recommendations for Modules is not the only solution but may significantly help the modules ecosystem. Implementation details (selection of what parameters should be considered standard and how they are named) can be discussed in public, among PuppetLabs engineers, modules authors and users (the proposed links are proposal which might be useful as a starting point, feel free to comment the code on GitHub or contribute to the Google doc) and I personally think that the approach may be based on small but steady steps, without wasting too much time upfront into edge-cases or implementations that would require relevant changes to Puppet code that would greatly delay adoption (there are still many 2.6 or earlier PuppetMasters around, proposals strictly related to changes in Puppet code would get years to find large adoption). I also personally think that only PuppetLabs has the moral and effective authority to  propose and promote Module Standards , sharing suggestions and ideas with the community. The little stone is launched, once again: methods, modalities, implementations and approaches are all to be discussed, but, really, there's not so much to do:the Holy Grail of Puppet modules interoperability and reusability is not impossible, major improvements can be done just with few naming conventions. Are we ready to define them? 2012-12-21T00:00:00+00:00 https://example42.com/blog/2012/12/21/example42-puppet-playground If you need to test your Puppet code on different Operating Systems, this might help you in doing it quickly.It's a Vagrant multi vm setup containing Puppet enabled Vagrant base boxes (retrieved from Internet sources like VagrantBox.es) where you can easily play with Puppet.The intention is to keep the list of base boxes updated and add new ones for new Operating Systems.  INSTALLATION Clone this repo to a work directory of your choice (here puppet-playground): git clone https://github.com/example42/puppet-playground.git puppet-playground This creates a multi vm vagrant environment cd puppet-playground vagrant status This is enough to play with Puppet in Masterless mode: default manifest is manifests/init.pp, modules are in modules/. WORK WITH MODULES You can add the modules you want in puppet-playground/modules. For this you have various alternatives: 1 - If you want to quick test Puppet resources without using modules just write your Puppet code in manifests/init.pp (see below). 2 - If you want to test modules from the Puppet Forge you can install them with: puppet module install <modulename> --modulepath modules/ So, for example: puppet module install puppetlabs-apache --modulepath modules/ 3 - If you want to test the NextGen Example42 modules you have to activate their integration: git submodule init git submodule update This initializes the modules dir with the Example42 NextGen modules set, then you have to actually download all the single modules: cd modules git submodule init git submodule update 4 - If you want to test your own modules just place them in the modules dir 5 - If you want to test librarian-puppet toasters use the toast.sh script (more details below) gem install librarian-puppet ./toast.sh VAGRANT USAGE Review, if you want, the Vagrantfile in puppet-playground and show the available OS cat Vagrantfile vagrant status Edit and play with the Puppet manifest applied on the boxes vi manifests/init.pp This is your test playground, add resources, use modules, declare classes... For sample code that uses Example42 modules, look at the other files in manifest/*.pp. See how your code behaves on the selected test box: vagrant up Test_Centos6_64 This may take a while, the first time, to download the box. Once created the VM connect to it with: vagrant ssh Test_Centos6_64 To exit form the shell on the VM vm# exit To restart your VM: vagrant reload Test_Centos6_64 To destroy and rebuild from scratch vagrant destroy Test_Centos6_64 vagrant up Test_Centos6_64 PLAY WITH PUPPET You can test and apply code directly from the VM: vagrant ssh Test_Centos6_64 From a shell on the VM get the superpowers and move to vagrant configs: vm# sudo -s vm# cd /tmp/vagrant-puppet/ To try some Puppet code edit the manifest file: vm# vi manifests/init.pp You can work on it both from your system and the VM. On your system is in puppet-playground/manifests/init.pp On the VM is available at /tmp/vagrant-puppet/manifests/init.pp From the VM you can run a test with: vm# puppet apply -v --modulepath '/tmp/vagrant-puppet/modules-0' --pluginsync /tmp/vagrant-puppet/manifests/init.pp From your host: vagrant provision Test_Centos6_64 To test the code on all the running nodes vagrant provision USE BUNDLED TOASTERS You can experiment with bundles of modules and Puppet code with the toast.sh script. It copies configurations from the toasters/directory to manifests/init.pp and Puppetfile and runs librarian-puppet to automatically install the required modules in the modules/ directory. To show the available toasters: ./toash.sh list To install a specific toaster: ./toash.sh install garethr-riemann To install and directly test on the running boxes a specific toaster ./toash.sh run garethr-riemann To show the status of currently installed modules and manifests/init.pp ./toast.sh status To cleanup the modules directory, the Puppetfile and manifests/init.pp (Beware all the existing changes will be wiped off) ./toast.sh clean CAVEATS Broken Vagrant Boxes Not all the Vagrant boxes have been widely tested, they have probably old versions of the VirtualBox Guest Additions and maybe provide not updated Vagrant configurations. If you find errors like: /Users/al/.vagrant.d/boxes/solaris10_64/include/_Vagrantfile:7: undefined method `system=' for #<Vagrant::Config::VMConfig:0x1025fc5a0> (NoMethodError) Try to remove or delete the referred file: mv /Users/al/.vagrant.d/boxes/solaris10_64/include/_Vagrantfile /Users/al/.vagrant.d/boxes/solaris10_64/include/_Vagrantfile.bak Some boxes (currently the ones with the ToFix prefix) are not fully working for Puppet provisioning. Modules directory The cohexistence of different ways to manage the modules directory (with puppet module tool, with the Example42 NextGen git repo, with custom modules r via librarian-puppet) may create inconsistent status, if you mix these methods. Start from an empty modules dir to have a clean setup good for every use. SUPPORT AND BUGS Please submit bug filings, pull requests and suggestions via GitHub. This Puppet Playground might become more and more useful if: More Working vagrant Boxes are provided for different OSMore toasters are provided that use different modules sets with librarian-puppet Any contribution to these is very welcomed. 2012-12-08T00:00:00+00:00 https://example42.com/blog/2012/12/08/introducing-example42-puppet-tutorials One of the greatest satisfactions behind the work on Example42 modules has always been to hear about people saying that they have learnt a lot about Puppet from these modules (even if many times they don't use them ... and this makes me think a bit :-O )Recently I was asked to prepare a dedicated and customized Puppet training session for a customer of a company I've started to work with and I was asked to prepare some training material for this occasion. At the same time this company has introduced me to Slidery, a Java library they are developing to convert markdown text into slides, using different presentation layouts (Desk.js, Impress.js, Reveal,js, PDF...). A Slidery plugin for Gradle makes all this very simple. I got in love with this tool and felt the unstoppable temptation to use it to to provide some useful Puppet information to the community. Here it is, then, the first , very young, set of Example42 Puppet Tutorials. You can find the Markdown "source" code (and what's needed to build the slides) on GitHub, and you can see directlty the first Tutorial sets: Puppet EssentialsWhat you need to know to start with PuppetRendered Slides (In Deck.js) - Markdown source How To Use Example42 ModulesInstructions on how to use Example42 modules, material has been retrieved from past presentations and posts.Rendered Slides (In Deck.js) - Markdown source That's all, for the moment... a set of Tutorials, presented as slides (and a big work in progress), that I hope will help people in understanding how to use Puppet. 2012-09-30T00:00:00+00:00 https://example42.com/blog/2012/09/30/puppet-modules-for-fun-and-profit Another presentation on the design choices behind the NextGen Example42 Puppet modules.Held at PuppetConf 2012 in San Francisco (what a great conference! :-) 2012-07-11T00:00:00+00:00 https://example42.com/blog/2012/07/11/a-presentation-about-example42-next-gen-modules PuppetCamps are always a great occasion to meet good friends and talk about Puppet with people and, when possible, talk about the work you're doing with it. I've had the chance to make a presentation about the NextGen modules at the Dublin and Geneva PuppetCamps.Slides, are similar (Geneva ones have just some cosmetic additions) and are both available on SlideShare Puppet modules: A Holistic Approach - Geneva 2012-04-10T00:00:00+00:00 https://example42.com/blog/2012/04/10/nextgen-modules-documentation-and-examples Since the first post about the NextGen modules some improvements have been done and some design choices have been done (the most important one is the params_lookup pattern which is the result of some good feedback and makes it possible to automatically retrieve class' arguments from top scope variables, an ENC and / or Hiera). Here is some detailed documentation on the common arguments of the new modules: And to make things, hopefully, clearer, here some sample usage code snippets: 2012-03-02T00:00:00+00:00 https://example42.com/blog/2012/03/02/testing-puppet-runs-with-puppi Whoever has used Puppet on production servers has probably lived, at least once, the uncomfortable situation when a change on Puppet manifests has introduced "unwanted effects" on target nodes if not real failures.There are many things that can go wrong when you make a change in whatever part of your Puppet code: you may introduce a wrong configuration file that triggers a service that doesn't restart correctly or have a modification you expected to be limited to some servers and, for some reason, due to wrong logic or poor understanding of your Puppet code, be applied to the "wrong" servers.Then there are other more mundane errors, that might occur with syntax errors that prevent Puppet from running on a node, but do not affect its functionality. Let's see a brief summary of the common errors and the testing methods available for Puppet. Common errors and testing methods Syntax errors are quite common for the unexperienced Puppet Master still battling with Puppet's somehow picky syntax, but they are also the less dangerous and easier to test. It's a very good idea to force syntax checks, both on Puppet manifests and Erb templates directly at the commit stage, with an hook command of your favorite SCM (search for something like "puppet syntax check hook" for implementation details). Another somehow unusual but effective method to test the manifests syntax is to use Puppet doc. A simple command like: puppet doc --outputdir /var/www/doc/ --mode rdoc not only generates a nice Rdoc output like the one here but does a syntax check for all your manifests.Moreover, if you like to code with an IDE, Geppetto is definitively your choice, having, among the others, also syntax checking and autocompletion features. Modules' integration and logic errors may be intercepted by proper tests. Cucumber Puppet and Rspec Puppet introduce Puppet to some of the fancier testing methodologies around. They examine the catalog that would be produced for an host and test if it contains the expected resources. As you can imagine their effectiveness is strictly related to the quality of the tests you write and a good coverage of your classes and defines requires time. I must confess that I was initially skeptical about their usefulness, but when I started to use rspec puppet I've found it really useful to catch regressions, integration and logic issues and found myself much more comfortable in committing changes to my modules set. The puppet --noop option is also quite useful to test if the catalog provides the expected resources for a node, but requires manual intervention. Service failures, though, can be due to other kind of errors, that can't be intercepted just by checking if Puppet compiles its catalog correctly: an error in a configuration file that prevents the relevant service from restarting can be noticed only when it has already been applied to a node. Also applying a syntactically correct configuration on the wrong nodes may have terrible effects (imagine how bad could be to apply the apache configuration reserved to a specific kind of servers to nodes that should have a different role...)To mitigate or prevent the impact of this kind of problems there are different approaches, none of them is really resolutive but still have their usefulness:- You may use Puppet's environments to test the effect of a change on a limited set of nodes before propagating it to the whole infrastructure. An implementation example is here. - You can use canary nodes, that can eventually be created from scratch and destroyed each time, where your configurations are applied and their effect verified. A nice sample with Vagrant is shown here.- You have probably already testing and staging environments, in your infrastructure, where you test your applications before deploying them in production and where you can test also your Puppet changes, even if you might not be 100% sure that the Puppet settings for your testing nodes are the same of your production nodes.- You can introduce in your change workflow a code peer review phase, with tools like Gerrit, that ease team communication and limit the involuntary disasters that can be introduced by Puppet newbies.- You can, also, use Puppi as postrun command and be notified if your Puppet run has done something wrong. And this is actually what I'm going to describe in this post. Running puppi check to test Puppet runs Puppi has various actions, among them I find myself using quite frequently, puppi check. It runs a series of checks on your system, its running services and eventually its web applications (or actually whatever can be checked by a nagios plugin) and shows in a quick way if everything is running fine. This is useful when you are logged to a node, to quickly see what's wrong in it, or when you deploy an application to verify immediately if you've made some damage, or via mcollective, to check quickly a cluster of nodes, or, as we are going to see, after a Puppet run, to be notified immediately if something has gone wrong. To use puppi you just have to include it in your nodes or in a class that is included by your node: include puppi To be sure to have all the commands Puppi needs include also the puppi::prerequisites class (that works for Example42 modules) or to be sure to have all the resources provided there: include puppi::prerequisites Once puppi is on your system you can play a bit with it, by issuing commands like puppi info, puppi log or puppi check (nagios plugins should be installed on your system). By default the quality and the richness of their output is rather limited, but, if you use the Example42 modules set you can add automatic Puppi integration just setting these variables: $puppi = yes # Enables puppi integration. $monitor = yes # Enables automatic monitoring $monitor_tool = "puppi" # Sets puppi as monitoring tool Note that at this point you might extend the monitoring tools to use, to enable automatic checks on the relevant tools. For example you can set: $monitor_tool = ["puppi","nagios","munin"] If you don't use Example42 modules or have some modules you want to check with puppi you can just add defines like these (either in the relevant modules on in other classes): puppi::check { "apache": command => "check_tcp -H $fqdn -p 80", } the above checks for port 80 on the local system using the check_tcp Nagios plugin, to check if a process is running you can use something like: puppi::check { "apache_process": command => "check_procs -c 1: -C httpd", } Needless to say that if some of these values are present in your module as variables, you can parametrize them. You can use also custom scripts for these checks, they just have to be in the Nagios plugins directory of your server (this might be fixed soon, giving the possibility of specifying an absolute custom path) and have an exit code logic compliant with them: Exit 0 if everything is OK, exit 1 for WARNING, exit 2 for CRITICAL. Whatever you do, the checks you define for your node are shown with the puppi check command. Now you can configure Puppet to run these checks at the end of each Puppet run and be notified via mail if any of them FAILS (exit code 2). You just have to add something like this in the [main] stanza of your puppet.conf. postrun_command = "/usr/bin/mailpuppicheck -m alerts@mycompany.com" The mailpuppicheck is a rather naif script provided by the Puppi module, it has the following options: -m email_address # Specify the destination email for alerts -r 4 # Number of times a puppi check is re-run in case of failures The -r option (default value is 1) can be useful if you want to re-run a puppi check more times, in case of failures, before actually sending an email. Note that if this value is not low and a puppi check run takes some time, especially if there are failures with long timeout, your puppet run can last more than you might like. In order to avoid, too much spamming, the mailpuppicheck command doesn't  send an alert email if the failures encountered are the same of the previous Puppet run. In this way you are not regularly notified of persistent problems. More options to better manage notification logic might be added in future, the same notification method, email, is quite primitive ( I personally hate it ) and can be changed to something more interesting in the future. The result, in any case is a mail like: Subject:  [puppet] Errors after Puppet run on web01.example42.com web01 check: 50-apache_process FAILED web01 check: 50-apache_tcp_80 FAILED which is sent immediately after a Puppet run and can give early warning of incoming disasters, especially if you have a battery of servers where the puppet run is sprayed on some time interval. So, this is far from being the definitive solution to testing the impact of Puppet changes on your infrastructure, also because it happens AFTER the change has been applied, when it is late but maybe not too late, but is a useful layer of "early warning". Moreover, if you already use Example42 modules with the above settings, all this comes out free, with no extra effort: you add a module and that module's process and port is automatically checked. For example, on the server running this site, this is the output of puppi check, obtained out of the box. [root@web01 ~]# puppi check web01 check: 10-Connected_Users [ OK ] USERS OK - 1 users currently logged in |users=1;5;10;0 web01 check: 10-Disks_Usage [ OK ] DISK OK - free space: / 27771 MB (86% inode=95%); /dev/shm 1915 MB (100% inode=99%); /boot 425 MB (92% inode=99%);| /=4405MB;27118;30508;0;33898 /dev/shm=0MB;1532;1723;0;1915 /boot=33MB;387;435;0;484 web01 check: 10-Local_Mail_Queue [ OK ] OK: mailq is empty|unsent=0;2;5;0 web01 check: 10-System_Load [ OK ] OK - load average: 0.05, 0.02, 0.00|load1=0.050;15.000;30.000;0; load5=0.020;10.000;25.000;0; load15=0.000;5.000;20.000;0; web01 check: 10-Zombie_Processes [ OK ] PROCS OK: 0 processes with STATE = Z web01 check: 15-DNS_Resolution [ OK ] DNS OK: 0.004 seconds response time. example.com returns 192.0.43.10|time=0.004123s;;;0.000000 web01 check: 20-NTP_Sync [ OK ] NTP OK: Offset 0.000167965889 secs|offset=0.000168s;60.000000;120.000000; web01 check: 50-apache_process [ OK ] PROCS OK: 21 processes with command name 'httpd' web01 check: 50-apache_tcp_80 [ OK ] TCP OK - 0.000 second response time on port 80|time=0.000136s;;;0.000000;10.000000 web01 check: 50-cron_process [ OK ] PROCS OK: 1 process with command name 'crond' web01 check: 50-munin_process [ OK ] PROCS OK: 1 process with command name 'munin-node' web01 check: 50-munin_tcp_4949 [ OK ] TCP OK - 0.001 second response time on port 4949|time=0.001250s;;;0.000000;10.000000 web01 check: 50-mysql_process [ OK ] PROCS OK: 1 process with command name 'mysqld' web01 check: 50-mysql_tcp_3306 [ OK ] TCP OK - 0.000 second response time on port 3306|time=0.000111s;;;0.000000;10.000000 web01 check: 50-nrpe_process [ OK ] PROCS OK: 1 process with command name 'nrpe' web01 check: 50-nrpe_tcp_5666 [ OK ] TCP OK - 0.000 second response time on port 5666|time=0.000276s;;;0.000000;10.000000 web01 check: 50-ntp_process [ OK ] PROCS OK: 1 process with command name 'ntpd' web01 check: 50-openssh_process [ OK ] PROCS OK: 2 processes with command name 'sshd' web01 check: 50-openssh_tcp_22 [ OK ] TCP OK - 0.000 second response time on port 22|time=0.000373s;;;0.000000;10.000000 web01 check: 50-postfix_process [ OK ] PROCS OK: 1 process with command name 'master' web01 check: 50-postfix_tcp_25 [ OK ] TCP OK - 0.000 second response time on port 25|time=0.000178s;;;0.000000;10.000000 web01 check: 50-rsyslog_process [ OK ] PROCS OK: 1 process with command name 'rsyslogd' web01 check: 50-splunk_process [ OK ] PROCS OK: 2 processes with command name 'splunkd' web01 check: 50-splunk_tcp_8089 [ OK ] TCP OK - 0.000 second response time on port 8089|time=0.000119s;;;0.000000;10.000000 my2c ;-)  2012-01-03T00:00:00+00:00 https://example42.com/blog/2012/01/03/the-next-generation-of-example42-puppet-modules Update 20130902: Read the post Next Gen Modules: Lessons Learned for current ideas and considerations about NextGen modules Introduction The Example42 Puppet modules collection has tried to establish a standard, integrated, reusable and coherent way to manage Puppet modules based on the principles exposed in a pair of PuppetCamp presentations: Puppet Modules Standards and Interoperability and ReUse Your Modules! It has been initially developed and conceived for Puppet versions 0.2x and has introduced and followed some concepts that I've found quite useful in various situations, such as monitoring and firewall abstraction, "include and play" approach without the enforcing of custom logic, easily extendable multiple operating system support, attempt (only partially successful) to separate "core" module elements from custom ones, standard structure that allows scaffolding and relatively quick creation and upgrade of modules. Still, experience and the evolution of the Puppet language have suggested new and better ways of "doing things with Puppet" and, more important, most of the modules are going to have some issues (as probably a relevant part of the existing Puppet code in the world) when Puppet 2.8 will be released and variables dynamic scoping will be discontinued. For this reason I've decided to rewrite the whole modules set from scratch and make a new generation of Puppet modules, that is going to be compliant only with Puppet Masters with version major than 2.6 and will benefit of all the experience (mistakes?) made with the current modules. The effort is not as big as one may think, given the way modules are organized (most of the existing changes are in params.pp and custom module-specific defines are placed in dedicated files that can be imported) and the fact that module's generation is based on the scaffolding of a "foo template". Still there's much work to do ... Existing and NextGen features You can find the next generation of Puppet modules at http://github.com/example42/puppet-modules-nextgen consider that currently that module set is experimental work, not ready for production nor for testing, but it has the base structure that can give glimpses of what is going to become. All the application related modules are git submodules contained in independent git repositories (this is a first fix and evolution of the older one-repo modules set that will allow better integration in the Puppet Module Forge and possibly, better cherry picking of the desired modules), but some special modules and directories are part of the main git repo. The main features of the modules are: - Coherent and standardized structure, logic and usage based on best practices - Cross OS support (main targets are Redhat and Ubuntu derivatives) - Use of parametrized classes and fully qualified variables for Puppet 2.8 compliance (with support for mixed approach) (This is a "NextGen" feature). - Extreme customization options without any change to the core module (NextGen) - Optional integration with Puppi - Optional support of monitoring and firewalling abstraction - Decommissiong support: you can remove (almost) whatever you've added with a module, Monitoring elements included. - Auditing support: you can audit changes that the modules would do to existing files, before applying them (NextGen) - Integrated rspec-puppet tests (NextGen) - Embedded documentation compliant with PuppetDoc - Modules scaffolding based on different kind of basic foo templates - Compact code optimized for compilation and reporting times. (NextGen) Parametrized classes and variables All the main classes are now parametrized: you can pass all the parameters they use as explicit arguments. This allows better introspection on the parameters used by the classes, with a coherent and standard API (at least inside Example42 modules) to the modules functionality, but introduces new challenges in the definition of a whole Puppet setup: - You can use the same parametrized class only once - Besides internal defaults, you have to explicitly declare all the parameters you want to pass. Since I wanted to provide a rich and standard set of common parameters, that allows users to affect the behavior of the class without changing it, this could result in having a lot of similar and redundant code parts to manage these arguments  (such as if to enable automatic monitoring and what tools to use). For this reason all the next-gen classes have these features and twists: - all the parameters' defaults are defined in the params.pp class where cross OS variations are managed and top scope variables eventually used to redefine defaults - you can therefore use top scope variables (such as the ones defined in an External Node Classifier) instead of parameters - you can mix top scope variables and class parameters (at first look this might not sound a good idea, but keep on reading...) - parameters, if defined, always override the module internal defaults and top scope variables. So, let's take the openssh module (which is actually the one I'm testing for different cases). You can use it in both the old include way: include openssh or as a parametrized class: class { "openssh" : } by default this just installs the relevant package, starts the relevant service and doesn't change configuration files. You can provide parameters in two ways. 1- Define top scope variables (the ones you define in a ENC or in site.pp) and include the class:  $::openssh_template = "example42/openssh/sshd.config.erb" include openssh 2- Pass the arguments in the parametrized class: class { "openssh":  template => "example42/openssh/sshd.config.erb", } and you can mix top scope variables and class parameters to manage site-wide settings reducing the verbosity of common arguments: $::monitor = true$::monitor_tools = [ "nagios" , "puppi" , "monit" , "munin" ]$::puppi = trueclass { "openssh":  template => "example42/openssh/sshd.config.erb", } The above does the same of: class { "openssh":  template => "example42/openssh/sshd.config.erb", monitor => true, monitor_tools => [ "nagios" , "puppi" , "monit" , "munin" ], puppi => true, } which, if repeated for many classes can actually be too much redundant. Needless to say that the values of the top scope variables can be assigned via Hiera, Extlookup or other functions that set specific values to the variable according to custom logic. Customize everything, modify nothing You have already had a glimpse of the parameters you can pass to these classes: the path of the template to use for the main configuration file, for example. The whole discussion about externalizing Puppet data sources and separating it from the module logic with functions like Hiera is, IMHO, a very important but not resolutive step towards full modules' reusability. A module, according to me, should not enforce the way it provides its configuration files (static files, eventually based on an array of possible sources,  templates or even concatenated files), it should allow site specific customizations where existing resources are modified or new ones added, and should be adaptable to different scenarios. To our openssh class you can pass variables like: class { "openssh": source => [ "puppet:///modules/lab42/openssh/sshd_config-${hostname}" , "puppet:///modules/lab42/openssh/sshd_config" ], } So you can specify the name of the Puppet path  of the static files to source (in this case an array). But you can also do something like: class { "openssh": source_dir => "puppet:///modules/lab42/openssh/" , source_dir_purge => false, } in order to provide the whole content of the configuration directory based on the path specified in source_dir (in this case, with the source_dir_purge option disabled, without removing eventual existing files on the local system that are not in the source directory on the PuppetMaster). But if you prefer to use templates, or they are more fitting for your setup, you can use the template argument instead of the source one (note that the parameters source and template can't coexist): class { "openssh": template => "lab42/openssh/sshd_config.erg" , options => { 'LogLevel => 'INFO', 'PermitRootLogin => 'yes', 'ListenAddress' => '0.0.0.0', }, } Here besides the usage of a custom template you can specify, as an hash, whatever custom options you may want to use in your template. They needn't to be explicitely added as parameters of the openssh class, so you can actually provide custom parameters to use in your templates without the need to explicit them in the called class. Given the above example, in your own lab42/openssh/sshd_config.erg template , you can use the custom values provided in the options hash in this way: [ ...] # Direct usage example PermitRootLogin <%= options['PermitRootLogin'] %> LogLevel <%= options['LogLevel'] %> # Conditional usage example, with default value setting <% if scope.lookupvar("openssh::options['ListenAddress']") then -%> ListenAddress <%= options['ListenAddress'] %> <% else -%> ListenAddress 0.0.0.0 # Default value <% end -%> Now, even if this approach has its evident disadvantages (not exactly easy to use, you need to place some extra logic in the template to manage defaults in order to avoid the danger of having blank fields for options not explicitly passed) it provides a mathod to affect, in a completely independent way, the main class behavior without touching anything of its module. Still this might not cover all your customization needs, you may want to add custom resources to the basic openssh class, or redefine arguments for existing arguments for which there's not already a class parameter. Well, you can specify a custom subclass to automatically include: class { "openssh": my_class => "openssh::lab42", } This make the class openssh include openssh::lab42 a custom class that may inherit or not the main openssh class (inheritance would be needed only when you have to modify existing resources defined in openssh).  Note that in order to allow class inheritance your custom class name should be something like openssh::myname (and not myname::openssh). So, if you would like (as good and recommended practice) to place your custom classes in a custom module (eventually named myname), you might need to explicitly  import somewhere your myname module with all your custom classes, templates and files, since class autoloading in this case wouldn't work (it actually depends on the Puppetmaster's version: on 2.6 you need to explicitly import your myname module with custom classes, on 2.7 it seems not necessary). In any case, as usual, all these are options left to the module's user: usage of custom source files or whole configuration directories, addition of custom classes and resources, usage of custom templates with custom extra options are possible without any modification of the core module, but, if you simply get the module and modify it directly to fit your needs, adding there resources, custom arguments or whatever, you are free to do that. Maybe this won't be not the most "reusable way", it will break upstream compatibility (but Github pull requests are always welcomed ;-), but will work for you. And this is what matters. "Common" options and features Most, if not all, the next-gen modules are going to have a set of parameters that affect their basic behavior.  They are common in the sense that you'll find them in all the Example42 modules (most of these features are actually already present in the current set) but actually they are not so common in the world of Puppet modules. Some are related to the decommissioning of resources: as whoever uses Puppet knows, if you want to remove something deployed via Puppet you can't simply comment or remove the Puppet code that placed it. You need to explicitly tell Puppet to remove the resource. This can be a big PITA in some cases or can be just a matter of typing: class { "openssh": absent => true, } This not only removes the openssh package, its configuration files and the service but all the relevant monitoring configurations that the module provided (more on that later). You get it? Decommissioning is automatically done on nasty beasts like Nagios configurations, Monit control files and so on.. Similarly you can specify: class { "openssh": disable => true, } in order to keep the openssh package but disable its service (and the relevant monitoring). And you can also set: class { "openssh": disableboot => true, } for the specific cases (typically when the service is managed by a cluster) where you don't want to start the service at boot but also don't want to check if it's running and eventually stop it during a Puppet run. Nothing new, up to now, these features are already present in the current modules, even if managed in a less elegant way (now there are sub classes like openssh::absent to include and generally more verbosity in the module). A new addiction is an option like this: class { "openssh": audit_only => true, } which does what it suggests. When you specify it the class doesn't change any existing configuration file, it just activates Puppet's audit metaparameter and lets you see what are the changes that the class would make to your files (a sort of noop run with central reporting of the expected changes). Should be useful to see and prevent the disasters you're going to distribute when applying your Puppet modules on existing nodes. There are various other parameters that you cat pass to the class, such as: package, service, service_status, process, process_args, config_file_mode, config_file_owner, config_file_group, config_file_init, pid_file, data_dir, log_dir, log_file, port, protocol. For them there's not the "top scope variable" equivant since, generally, they are for module's internal use and are not expected to be modified, normally, but still you can pass them as parameters to the class. In the openssh/manifest/params.pp file the right values for the main Operating Systems (or at least Linux distros) are already set, but there are special cases where it can be  nice to have the opportunity to alter them. For example you may need to use custom package names, built internally, which use custom service names and process names, or you may need to change the owner of the configuration file(s) to allow "modifications by non privileged users ;-)"or make some tweaks to the resources to monitor (various of the above parameters are used just for the monitoring abstraction and the Puppi stuff). Just note that changing these parameters may yield to untested and unpredictable results (for example the port argument is used for monitoring and it doesn't necessarily reflect a parameter in the configuration file if it's not explicitly used in a relevant template). Finally, talking about "common" features, the modules are supposed to have cross Operating Systems compatibility, they should work for RedHat 5/6 and derivatives, Debian 5/6 and derivatives, Ubuntu 10.04 and later (I'm fed up to quarrel with Ubuntu 8.04 operatingsystem fact). Suse support sooner or later will probably be introduced, Solaris too, eventually. Generally speaking development of alternative systems support is done upon necessity: when I'll have to work on these OS, the relevant support will be introduced (call it job driven development ;-). Monitoring and firewalling abstraction One of the features I'm proudest of, in the Example42 modules, is the monitoring abstraction approach: In the modules I define what to monitor, not how. It has proven to be extremely powerful and useful even if I managed to write "connectors" only for a limited number of monitoring tools: Nagios, Monit, Puppi (more on this later) and, even if they auto configure themselves, Munin and Collectd. Again, I didn't add new "tools" because I hadn't the opportunity to use them in a "real world", but I'm quite confident that the abstraction model works also for most of the cases, eventually with some tweak. Basically the point is that, if you want to monitor the resources (typically the listening port and the running process) provided by a class you can add these parameters : class { "openssh": monitor => true, monitor_tool => [ 'nagios','puppi','monit','munin' ], monitor_target => $ipaddress_eth1, } With the above parameters you enable the monitoring of Openssh with the specified monitoring tools specifying the Ip address to use as target for monitoring (by default is $ipaddress and you generally don't need to define it, here it was just placed for reference). Note that, as most of the other arguments and as shown before, you can use top scope variables ( for example $::openssh_monitor_tool ) to set these configurations, but, most important, you can set a "site wide" general behavior with the variables $::monitor , $::monitor_tool and $::monitor_target and have per module overrides with the arguments written in the example above or the analog variables: $::openssh_monitor , $::openssh_monitor_tool and $::openssh_monitor_target . Actually the setting of general top scope variables that might be overriden, case per case, by parametrized class specific arguments is the real reason that can justify the usage of the mixed approach. A similar approach is used for automatic firewalling of the ports provided by the module. You can set these parameters: class { "openssh": firewall => true, firewall_tool => [ 'iptables' ], firewall_src => "0.0.0.0/0", # This is the default value firewall_dst => "$ipaddress", # This is the default value port => "22", # This is the default value protocol => "tcp", # This is the default value } as you see most of them have sensible defaults and are reported here as reference. The only firewall tool currently supported is Example42's iptables module, but it's possible, as for the monitor metamodule, to create connectors for other modules and eventually also for a central network firewall. One of the features of the existing module set that is probably not going to be implemented in the next gen, is the backup abstraction. It's based on similar logic (a module defines what to backup and then in a backup metamodule are defined the connectors for different backup tools), but, honestly, it has not been really used up to now. The underlining idea behind all these abstactions, is that somehow is reductive to use a Puppet module JUST to configure an application: a module inherently may have a lot of information about it: what are its components (package and service names, configuration files, but also, with limited extra effort, process names, listening ports, log and data paths...), the services it provides, their dependencies. All this information can and should be used for many infrastructure activities: monitoring, firewalling, backup, maybe storage management, possibly network configuration, dependencies provisioning and, why not, local access to relevant information... Oh My Puppi! Sometimes I feel like I really never managed to express properly what is Puppi and what it can do. The fact that even in the work environment where I introduced it (where we deploy dozens of different applications each day with it, either with a single command line, a cronjob, a mcollective command or an automated task) people make confusion between Puppet and Puppi (yes the name doesn't help) should make me ponder about this. Let's try to summarize in a few words what is Puppi. Puppi has basically two different and indipendent functions:- Deploy applications- Get information about the system More concretely Puppi is a Puppet module that installs on a system a bash command and all the scripts and files that it needs to perform its subcommands. The Puppet module can be imported indipendently (yes, you can use it without the whole Example42 bunch) and just included in your nodes. Once you have it you have at disposal defines, in Puppet language, that allow you to perform the above functions. In order to manage application deployments there's not much effort to achieve appreciable results. For example, this define: puppi::project::war { "myapp": source => "http://repo.example42.com/deploy/prod/myapp.war", deploy_root => "/usr/share/tomcat/webapps", } makes it possible to issue the command "puppi deploy myapp" on the node where you placed it. What that command does, what are the different things you can deploy and the options you have to customize the procedure is better described elsewhere in this site (hint: look at the top menu). For the second function, "get information about the system", you have to use other defines for specific Puppi commands. Out of the box Puppi already provides some output to show when you type puppi check, puppi info or puppi log (some system wide information is shown in these cases) but it becomes more interesting and powerful when application specific information is made available (for example where are its logs, its configuration files, what's its status and if it's working properly). The Example42 modules (many of the existing and all the NextGen ones) have integrated optional Puppi support. That is you can use them also without using Puppi, as you can use Puppi without Example42 modules, but if you use Example42 modules and activate Puppi, some magic happens. Activate Puppi with: class { "openssh": puppi => true, puppi_helper => "myhelper", # Default is "standard" } or, as you can guess by now, with the top scope variable $::puppi set to true.  Now the bad news: the implementation of Puppi for the NextGen modules still doesn't exists (!?). I've totally changed the way to provide data for Puppi in the module.Up to now modules created various files that were (are) used for the various Puppi subcommands.The new approach is lighter and moves much of the work on the local system: it just creates a single yaml file containing all the variables and parameters used by the module (yes, all the "Knowledge that Puppet alreeady knows on the system: names, paths and so on"). The puppi_helper is just one more variable that will allow the personalize the scripts that will use those variables in order to yield custom outputs.  Still there's not yet a Puppi something on the local system able to do sane things with that yaml. I don't expect it will be difficult to do that but I still have to figure out a sane way to keep backward compatibility and if the new Puppi will be a Puppet face (this would make it unfit for Puppet 2.6 and this bothers me). So not much more to say about Puppi here, let talk about... The painful art of testing I must confess it. Testing and QA on the existing modules have been crap. Fixes have been pushed upstream from different working environments on specific distros, quickly tested on other ones. Some modules were obsoleted and not updated and, even worse, the sample "Puppet infrastructure" shown on this site not always was aligned to the modules' changes and this led to brutal errors that even if quick to fix in some cases could be disturbing and misleading for new users. I would like to be more rigorous about this, I would like to provide modules that can be updated safely, that behave consistently on different distros and "always" work out of the box.I also want to introduce finally an automated testing environment for different operating systems, where not only the catalog is tested for the expected resources (as Cucumber and Rspec Puppet do) but the deployed application is working as wanted. One of the common mistakes that is easy to do with Cucumber Puppet or Rspec Puppet is just to check if Puppet does what it is expected to do: I define a resource and check if that resource is present on the catalog.This is something that the same Nikolay Sturm, author of Cucumber Puppet has stressed in different occasions: test the logic of your modules, not if Puppet works (that's PuppetLabs' job). In the NextGen modules I inserted some Puppet Rspec tests that try to verify some internal logic, but to be honest I'm not totally won by testing approaches based on catalog checks. You can have a wonderful and totally sane catalog that when applied to a node breaks a service because there's a syntax error in the service's configuration file. This is something that you will never discover if you look just at the catalogue. Now, I don't dare to enter here in a discussion about best practices for managing Puppet change in production environments, for the sake of this post just let me say that I'll probably automate a procedure that builds different operating systems, runs Puppet, triggers Puppi checks and notifies the result. Eventually at each post-commit, and have softer and quicker rspec tests in the pre-commit stage. I've called a similar approach "Test Driven Puppet Infrastructure Development", but people around are doing similar things with different names, approaches and tools. The value added I see is that when you write tests for your modules you are actually writing checks that you can use in your monitoring tools. Let's finally face another characteristic of these modules: scaffolding. Is modules templates scaffolding a good idea?  Looking at some answers at the feedback page on this site various people don't consider it so. Maybe I haven't well expressed the advantages I see in "modules cloning" or maybe is really a bad idea. The point is that to make new modules I always start from an existing "foo" template: the skel of a full featured module, that is cloned, renamed (by the script 00_example42_scripts/module_clone.sh ) and then customized according to the application specificity. This doesn't mean that all the modules are the same with a massive renaming, changes are added in terms of specific resources or defines according to the single applications. The value added I see in this approach is that I can very quickly create a basic module with all the features we have seem so far. Another plus is that modules keep a standard structure, coherent naming and parameters and relatively easy upgradability. Still not all the applications need similar module design, so, in the NextGen, I decided to create different foo templates for different type of modules: from the typical package/service/configuration file to layouts dedicated to java or php web applications, where you can even decide if to install them from package or sources (or via Puppi). I'm testing these different layouts with the openssh modules, we are seeing it, the wordpress and the solr ones. Actually they are supposed to be the test beds from where to define good starting foo templates from where regenerate all the new modules. For this reason also I'm writing this post, to gather ideas, suggestions and comments about the "nextgen" openssh module before converting it to foo and make it the master of the clones. So, if you've read up to now I suppose you are interested in the topic and therefore I would love to hear your opinion about what has beed described here: any suggestion will be considered and pondered, if you think there are design issues, bad choices, wrong approaches, please let me know and argument: commenting this post or directly on GitHub. Thanks for the attention and the patience. 2011-07-27T00:00:00+00:00 https://example42.com/blog/2011/07/27/deploying-applications-and-bringing-puppet-information-to-the-cli-with-puppi With Puppet we build infrastructures, piece by piece, manifest after manifest. We control how nodes are configured, what services they provide, how they are checked. We manage where web applications stay and sometimes how they are built, tested and deployed. Puppet has a lot of knowledge about our systems, every catalog we receive is an unique source of data that is exactly what brings our servers to the desired state. More information we provide in our manifests about the system, more we can do things with it. Puppi tries to bring this knowledge to the command line.   PRESENT Puppi was initially developed to standardize different procedures of web applications deployments and it has evolved into a shell command that provides handy and quick actions useful for the system administrator. Now is stable enough to have reached version 1.0 (currently in RC state) the features initially planned are present, it's used in production and no big changes are planned for this version. See the presentation held at The PuppetCamp Europe 2011 (http://example42.com/?q=Puppi_presentation_PuppetCamp_Europe_2011) for some videos and further details. Puppi most useful actions are: deploy, to manage the whole deploy workflow with a single keystroke check, to verify the general system's health and specific checks on the application deployed log, to quickly tail some or all the known logs  info, to show the output of a custom set of preconfigured commands rollback, to quickly rollback a deployed application   Puppi is currently entirely provided as a Puppet module, (http://github.com/example42/puppi), you include it and you have the whole puppi thing: - the bash command /usr/sbin/puppi - its configuration directory /etc/puppi, with plenty of files and dirs configured by Puppet defines like puppi::check, puppi::info etc - a set of (customizable) defines that build deploy procedures, like puppi::project::maven that retrieves Java artifacts generated by Maven - some general use native scripts that are used to accomplish the different steps of a deployment - a set of defines to populate the output of puppi actions - some default content for puppi info, log and check actions to make puppi useful out of the box.   There are not other modules prerequisites, but for full functionality you need on the systems where you place puppi these commands: wget, mail, rsync and the most common Nagios Plugins.  Note also that Puppi is entirely self contained, it doesn't need external services to run, once deployed, all its actions are based on local files and data: checks are local, info scripts are local, deploy procedures need only the availability of the defined source to work.   HOW TO USE The puppi module provides some deploy procedures that cover many typical scenarios. For example, to retrieve a war from $source and deploy it in $deploy_root, keeping a copy for rollback and notifying $report_email, you need this: puppi::project::war { "myapp":     source           => "http://repo.example42.com/deploy/prod/myapp.war",     deploy_root      => "/store/tomcat/myapp/webapps",     report_email     => "sysadmins@example42.com",     } All the existing deploy procedure have a set of optional arguments that make them more flexible. Here a more complex case: puppi::project::maven { "supersite":     source           => "http://nexus.example42.com/nexus/content/repositories/releases/it/example42/supersite/",     deploy_root      => "/usr/local/tomcat/supersite/webapps",     config_suffix    => "cfg",     config_root      => "/srv/htdocs/supersite",     document_suffix  => "css",     document_root    => "/srv/htdocs/supersite",     firewall_src_ip  => $site ? {         dr      => "192.168.101.1/30",         main    => "192.168.1.1/30",     },     backup_retention => "3",     init_script      => "tomcat",     report_email     => "sysadmins@example42.com",     enable           => "true", } this does the following actions in sequence:  - Retrieves the maven-metadata.xml from $source, - Blocks access from a loadbalancer IP, - Backups the existing data for rollback operations - Deletes older backups (3 archives are kept, instead of the default 5) - Deploys the release war in $deploy_root,  - Unpacks a configurations tarball tagged with the Maven qualifier $config_suffix in $config_root,  - Unpacks a static files tarball tagged with the Maven qualifier $document_suffix in $document_root - Restarts tomcat and notifies via mail. All this can be triggered with the command "puppi deploy supersite" and if something fails you can "puppi rollback supersite".   The above puppi::project::maven (or tar|war|list|dir|mysql..) defines build up the logic and the sequence of commands run in deployment and rollback operations using basic puppi defines like puppi::project, puppi::deploy, puppi::rollback, puppi::init. You can use the existing puppi::project::* procedures or build up your own ones, to manage special cases. The same bash scripts they use ("native scripts", stored in puppi/files/scripts/) can be replaced by custom scripts, in whatever language.   The other puppi actions require simpler constructs, for example you can manage a single check (we use Nagios plugins, as they are so common) with: puppi::check { "Port_Apache":     command  => "check_tcp -H ${fqdn} -p 80" , } or insert more elaborated checks in your defines (for example when you create virtualhosts, using data you may already provide): puppi::check { "Url_$name":     enable   => $enable,     command  => "check_http -I '${target}' -p '${port}' -u '${url}' -s '${pattern}'" , }   The logs to tail with puppi log are defined by the puppi::log define (note that with a simple selector you can adapt the commands to run according the underlining OS): puppi::log { "auth":     description => "Users and authentication" ,     log => $operatingsystem ? {          redhat => "/var/log/secure",         darwin => "/var/log/secure.log",         ubuntu => ["/var/log/user.log","/var/log/auth.log"], } Also in this case you can insert a puppi::log inside an existing define, using the data it inherently has: puppi::log { "tomcat-${instance_name}":     log => "${tomcat::params::storedir}/${instance_name}/logs/catalina.out" }   You can manage the output of "puppi info network" with something like: puppi::info { "network":     description => "Network settings and stats" ,     run         => [ "ifconfig" , "route -n" , "cat /etc/resolv.conf" , "netstat -natup | grep LISTEN" ], } or build more elaborated info subclasses using custom templates and specific data: puppi::info::instance { "tomcat-${instance_name}":     servicename => "tomcat-${instance_name}",     processname => "${instance_name}",     configdir   => "${tomcat::params::storedir}/${instance_name}/conf/",     bindir      => "${tomcat::params::storedir}/${instance_name}/bin/",     pidfile     => "${instance_rundir}/tomcat-${instance_name}.pid",     datadir     => "${instance_path}/webapps",     logdir      => "${instance_logdir}",     httpport    => "${instance_httpport}",     controlport => "${instance_controlport}",     ajpport     => "${instance_ajpport}",     description => "Info for ${instance_name} Tomcat instance" , } Examples are endless, you can extend and customize easily the existing defines and you can integrated them in your modules according to the needs you have.   THE JOYS OF COLLECTIVISM Puppi's purpose is not only to provide a command tool based on Puppet data, that helps the sysadmin to gather info, deploy applications, troubleshoot and check them. It can be run manually from the local system, automatically via a cron job or triggered by a web interface, it can be used to summarize a set of common actions to be sudoed by non privileged users and it can be called by an agent of an orchestration tool. Puppi enters into a new scale with the MCollective agent puppi and the command mc-puppi: whatever can be done locally with Puppi, can be repeated on the whole infrastructure, with the same syntax, using the power of MCollective. This becomes particularly interesting when your deploy procedures involve actions on different nodes, or when you need to check quickly the systems health on a moltitude on nodes. A command like "mc-puppi check" runs and shows the equivalent of ALL your Nagios checks on your WHOLE MCollective domain.   They can be thousands: you have them in few seconds. I like to consider this a real-time distributed instant infrastructure test. Something like "mc-puppi info network", instead, provides immediate overview of the network configuration and status of all your nodes, and if verbosity bothers you, just grep what you need.   A missing piece in the puppi world is a web frontend that gathers the reports of the deployments, collects information about nodes, shows the results of local checks and possibly lets users trigger deploy procedures via a central web console. The development of a web interface to puppi, altough planned since the beginning, has not yet started. The main reason is that it was considered a priority to have a stable command and a mcollective agent, another reason it's time to make decisions about puppi, and possibly these have to be shared.   FUTURE Puppet development is growing quickly, at the Europe PuppetCamp 2011 Luke presented Puppet Faces and it's clear that version 2.7 introduces us to a new era in Puppet evolution. There are some common points in faces and puppi: they both bring parts of Puppet to the cli and they are expandable with actions. Actually it just seems natural that puppi's future is to become a Puppet face. Now it's a bash script that executes bundles of bash scripts, based on data more or less elegantly provided by Puppet modules, it works also on older Puppet versions (at least 0.25) so that it can be widely adopted and integrated in current layouts. Next version is probably going to be in ruby, use directly Puppet APIs and possibly be based on a more standardized modules data model. And of course is going to work only with Puppet 2.7 and later. It could become something different, maybe even with a different name, but as far as I'm concerned it should keep the principles it's based upon: - Be based on Puppet data: Puppet knows the infrastructure, we want this knowledge and intelligence integrated in commands we run on the shell. The way this data is currently provided is not optimized (every piece of puppi information is basically a Puppet resource (generally a file) and this is an overhead we should avoid), we could rely directly on the catalog, which seems the most natural source, but in order to do this I suspect some kind of standardization is needed at the module level. - Provide a simple single line standard command to run an application deployment (one keystroke to deploy them all) - Provide useful actions that can be used from the cli, an orchestrator agent or a web interface to show info, status and working details on systems and applications (I would keep the check/info/log actions, as I'm finding them quite useful) - It can be run manually or automatically, locally or from a central orchestrator and, possibly, also via a web interface. A puppi webapp should let different users request, trigger and view the results of a deploy, gather info from the systems (via rest?), in order to become an inventory frontend on steroids with as much detail on the system as users want, receive and show checks and possibly be able to search and correlate the large amount of data it could receive. Truth is that development on the next puppi and its web frontend is something that I would like to do in collaboration with PuppetLabs and who is interested in the community. Puppi 1.0 was done by me for a customer's needs with the knowledge and the requirents I had at disposal. Puppi 2.0 (whatever the name and the shape) has not immediate operative requirements, its design, how data is feeded from modules, how it's integrated in Puppet should be discussed and shared. Anyone interested? 2011-05-24T00:00:00+00:00 https://example42.com/blog/2011/05/24/puppi-presentation-at-puppetcamp-europe-2011 Puppi has been presented at the PuppetCamp Europe 2011. Here you find the slides:</p> Puppi. Puppet strings to the shell The talk Video: Puppet Camp Europe 2011 - Alessandro Franceschi - Automating Applications Deployments with Puppi The videos embedded in the presentation: Puppi. Puppet strings to the bash. Puppi goes MCollective 2011-02-28T00:00:00+00:00 https://example42.com/blog/2011/02/28/test-driven-puppet-infrastructures-deployment A server infrastructure can't be considered fully operative if it's not monitored in some way. Various tools are available in the market and they cover different facets of the common monitoring needs (alerting, trending, performance, security...) but whatever  is your choice you need to configure them in some way. One of the nice side effects of having modules that include automatic monitoring functions, such as the Example42 ones, is that while deploying a Puppet infrastructure you add the relevant checks to your monitoring software so that you can quickly understand what is working out of the box and what has to be fixed. All the Example42 Puppet modules provide built in monitoring features, you can activate them just by setting the $monitor variable to "yes" (whatever the method you use to define and classify nodes) and at least one $monitor_tool. An unique feature of the Example42 modules is the abstraction that is embedded in all the modules, so that it's quite easy and quick to introduce new monitoring tools without having to modify anything in the modules. Typically a module has 2 kind of checks enabled by default: its listening port, if it is a network service, and its process name, for example, in the samba module you have: monitor::port { "samba_${samba::params::protocol}_${samba::params::port}": protocol => "${samba::params::protocol}", port => "${samba::params::port}", target => "${samba::params::monitor_target_real}", enable => "${samba::params::monitor_port_enable}", tool => "${monitor_tool}", } monitor::process { "samba_process": process => "${samba::params::processname}", service => "${samba::params::servicename}", pidfile => "${samba::params::pidfile}", enable => "${samba::params::monitor_process_enable}", tool => "${monitor_tool}", } Even if not excessively obvious, from the above lines we can deduce:- Two custom defines are used to specify what to check (a port and a process)- The arguments given to the defines are obtained via qualified variables set in the samba::params class- The user variable $monitor_tool specifies the monitoring tool(s) to be used for the above resources. Currently the Example42 Puppet modules support various tools (you can define them using an array): Nagios, Munin, Monit, Collectd and Puppi. They are actually different by nature and scope but the most interesting ones to actually check if something is working as expected are Nagios and, in some way, Puppi. Besides port and process checking, for these two tools is possible to define also URL tests based on pattern matching, so that you can actually check different functionalities of your web application checking if custom urls contain specific strings. An example of Url check:     monitor::url { "Url-Example42_TestDatabase":         url      => "http://www.example42.com/testdb.php",         port     => '80',         target   => "${fqdn}",         pattern  => 'Database OK',         enable   => "true",         tool     => "${monitor_tool}",     } If the http://www.example42.com/testdb.php page contains the string "Database OK" the check is positive. Note that the host on which is run the check is defined with the target argument. Note also that if you set enable to false, the check is removed/disabled. Another available check is for mount points. With a define like:     monitor::mount { "/var/www/repo":         name    => "/var/www/repo",         fstype  => "nfs",         ensure  => mounted,         options => "defaults",         device  => "nfs.example42.com:/data/repo",         atboot  => true,     } you both mount and monitor the specified resource. A proper test driven infrastructure does not only checks if the services delivered by Puppet are running or the mount points are mounted, it verifies also HOW they work.You can have Apache running but the web application failing in one or more elements. While the basic service/port checks are automatically added when is included the relevant module, for more accurate tests you need to write some (Puppet) code.For this the monitor::url define is useful for web applications but we haven't still identified a good method to abstract application specific tests (for example: is ldap/mysql/activemq responding correctly?), performance and security checks, proactive failure detection and other generally needed features.Probably there's not a real way to abstract certain specificities and some custom approach, strictly related to the software used and the contingency, is required. One possible approach to manage arbitrary checks could be to consider Nagios plugins as de facto standard and refer to them to handle custom checks, considering that they are used by different tools, besides Nagios, and are easily extendable.Currently, in the Example42 module there's a monitor::plugin define, but its usage is not yet standardized and is more oriented to be used to manage plugins for software like Collectd or Munin rather than to refer to Nagios plugins, practice and operational needs will drive our choice for this point. Understanding the monitor module All the above references to the monitor classes or defines imply the usage of the Example42 monitor module.This is an implementation entirely based on Puppet's DSL of a (strongly needed) monitor abstraction type. Different approaches and implementations would be welcomed, as we think that for the Puppet ecosystem it would be advisable to define at least standard naming  and syntax for the monitoring elements to be included in every module. The Example42 monitor implementation prefers linearity and extendability over performance and optimization of resources.The generic monitor defines are placed in files like:monitor/manifests/process.pp, monitor/manifests/port.pp, monitor/manifests/url.pp.Let's see for example monitor/manifests/port.pp: define monitor::port ( $port, $protocol, $target, $tool, $checksource='remote', $enable='true' ) { [...] if ($tool =~ /nagios/) { monitor::port::nagios { "$name": target => $target, protocol => $protocol, port => $port, checksource => $checksource, enable => $enable, } } if ($tool =~ /puppi/) { monitor::port::puppi { "$name": target => $target, protocol => $protocol, port => $port, checksource => $checksource, enable => $enable, } } } note that here according to the tool requested are called some specific functions that are configured in places like: monitor/manifests/port/nagios.pp, monitor/manifests/process/port.pp where are called the actual defines that "do" the checks. Let's see for example monitor/manifests/port/nagios.pp: define monitor::port::nagios ( $target, $port, $protocol, $checksource, $enable ) { $ensure = $enable ? { "false" => "absent", "no" => "absent", "true" => "present", "yes" => "present", } # Use for Example42 nagios/nrpe modules nagios::service { "$name": ensure => $ensure, check_command => $protocol ? { tcp => $checksource ? { local => "check_nrpe!check_port_tcp!localhost!${port}", default => "check_tcp!${port}", }, udp => $checksource ? { local => "check_nrpe!check_port_udp!localhost!${port}", default => "check_udp!${port}", }, } } # Use for Camptocamp Nagios Module # nagios::service::distributed { "$name": # ensure => $ensure, # check_command => $protocol ? { # tcp => "check_tcp!${port}", # udp => "check_udp!${port}", # } # } } Note that here you can choose different implementations of the specific module, so you are free to change the whole module to be used for a specific monitoring tool editing just these few files, for example if you don't like the Example42 Nagios module you can use the Camptocamp one just by changing the references in this file.Note, incidentally, that the port check can be triggered either from the Nagios server or from the same monitored host via nrpe, according to the value of the checksource parameter. In order to manage per site, per module and per role or host exceptions, the Example42 modules provide a fat but functional approach, generally managed in the params.pp class of each module.You can basically manage if to enable or not monitoring for all the modules or also module by module by setting the value of some variables: There are some "node-wide" variables you can set, their defaults are set in params.pp of each module:$monitor_port (true|false) : Set if you want to enable port monitoring for the host.$monitor_process (true|false) : Set if you want to enable process checking.$monitor_target : Set the ip/hostname you want to use on an external monitoring server to monitor the hostThese variables can be overriden on a per-module basis (needed, for example if you want to enable process monitoring for some service but not all):$foo_monitor_port (true|false) : Set if you want to monitor foo's port(s). If any. Default: As defined in $monitor_port$foo_monitor_process (true|false) : Set if you want to monitor foo's process. If any. Default: As defined in $monitor_process$foo_monitor_target : Define how to reach (Ip, fqdn...) the host to monitor foo from an external server. Default: As defined in $monitor_target Note that generally you really have not to care about them, as sensible defaults are set. But this it's important to note that with $monitor_target variable you can set HOW to reach the host to be monitored, by default is its $fqdn, but on multihomed nodes you might want to reach it via and alternative IP or name (possible defined or based on a fact value in order to avoid manual settings). Finally note that for tools that imply a central monitoring node and a variety of nodes to check, we have introduced the possibility to define a "grouplogic" variable to automatically manage different monitoring servers according to custom groups of nodes. Let's see how it works, for example for the Nagios module. You just have to define a variable, $nagios_grouplogic, and set as value the name of another variable you use to group your nodes.  For example you may want to have different Nagios servers according custom variables as zones, environments, datacenters etc (ie: $nagios_grouplogic = "env" ).By default all the checks go to the same server (managed by the same PuppetMaster) if you define in $nagios_grouplogic the name of the variable you want to use as discrimitator, you will have different Nagios servers monitoring the group of nodes having the same value for that variable. Note that you need to add in the list below your own variable name, if is not already provided. In nagios/manifests/params.pp you have: # Define according to what criteria you want to organize # what nodes your Nagios servers monitor $grouptag = $nagios_grouplogic ? { '' => "", 'type' => $type, 'env' => $env, 'environment' => $environment, 'zone' => $zone, 'site' => $site, 'role' => $role, } In nagios/manifests/service.pp the define nagios::service used to specify every Nagios service check (as we've seen before) is: define nagios::service ( $host_name = $fqdn, $check_command = '', $service_description = '', $use = 'generic-service', $ensure = 'present' ) { require nagios::params # Autoinclude the target host class include nagios::target # Set defaults based on the same define $name $real_check_command = $check_command ? { '' => $name, default => $check_command } $real_service_description = $service_description ? { '' => $name, default => $service_description } @@file { "${nagios::params::customconfigdir}/services/${host_name}-${name}.cfg": mode => "${nagios::params::configfile_mode}", owner => "${nagios::params::configfile_owner}", group => "${nagios::params::configfile_group}", ensure => "${ensure}", require => Class["nagios::extra"], notify => Service["nagios"], content => template( "nagios/service.erb" ), tag => "${nagios::params::grouptag}" ? { '' => "nagios_service", default => "nagios_service_$nagios::params::grouptag", }, } } In nagios/manifests/init.pp  (used only on the Nagios servers) you collect exported resources with: case $nagios::params::grouptag { "": { File <<| tag == "nagios_host" |>> File <<| tag == "nagios_service" |>> } default: { File <<| tag == "nagios_host_$nagios::params::grouptag" |>> File <<| tag == "nagios_service_$nagios::params::grouptag" |>> } } This lets you automatically deploy different Nagios servers monitoring different groups of nodes according to custom variables.If you need to use a variable different from the ones already defined (type, env, environment, zone, site, role) just add a line in the selector shown in nagios/manifests/params.pp.Neat, isn't it? In this article we have seen how to use Example42 modules to automatically monitor the resources included in Puppet managed nodes, how to add checks based on string patterns in Urls or mount points, how all this is done using a layer of abstraction that makes it possible to introduce a new monitoring tool that uses all the already present checks and how, de facto, systematic automatic monitoring implies a test driven deployment, since you find yourself checking what you want on your servers and you can quickly see, for example on your Nagios server, what is up and running and what needs to be fixed. Further work in Example42 modules will be done in the development of support for other monitoring tools, in the definition of other abstract enough monitor defines and generally in exploring the possibilities to automate more specific and complete checks. 2011-02-23T00:00:00+00:00 https://example42.com/blog/2011/02/23/puppi-a-puppet-module-for-deployment-automation Puppi is a Puppet module that lets users manage and automate the deployment of applications or generally every kind of batch activity that involves the execution of a sequence of scripts and commands.Its structure provides complete flexibility on the actions required for virtually any kind of application deployment but, in order to make usable out of the box, some example defines and scripts are provided to manage common scenarios and needs.The module provides:- The puppi command and its whole working environment- A set of common usage scripts that can be used in chain - Sample defines that can be used for many common application deployment scenarios. The whole picture in an example Before diving into details let's see a brief example.On your node or role you can use a similar define: puppi::project::war { "myapp": source => "http://repository.example42.com/myapp/myapp.war", user => "myappuser", deploy_root => "/srv/tomcat/myapp/webapps", report_email => "release@example42.com", enable => "true", } This creates a set of scripts in /etc/puppi/projects/myapp that make it possibile to write the simple command: puppi deploy myapp to run the whole deployment procedure, that, in this case, involves retrieving the war file from the http://repository.example42.com/myapp/myapp.war, backing up /srv/tomcat/myapp/webapps, deploying the new war in the same directory as myappuser and notifying via email release@example42.com. This case is not particularly complex but consider that the same puppi::project::war define user before has many more options, that let you run custom pre or post commands (in customizable order and with defineable user), stop and start, during the process any custom service, block access from a specific IP (such as the one of your load balancer) and so on. If something wrong happens you can simply type: puppi rollback myapp and choose to what version you want to rollback (latest or any other previous (dated) backup). Let's see a real life example of a deploy: root@metaportali-mpc:~# puppi deploy configurator Puppi setup: 00-configurator-RuntimeConfig-Initialization  [  OK  ] Deploy: 10-configurator-Run_PRE-Checks                     [ OK ] PROCS OK: 4 processes with command name 'apache2' TCP OK - 0.001 second response time on port 80|time=0.000651s;;;0.000000;10.000000 [ ... ] Deploy: 20-configurator-Retrieve_WAR                       [  OK  ] Deploy: 30-configurator-Backup_existing_WAR                [  OK  ] Deploy: 36-configurator-Disable_extra_services             [  OK  ] Deploy: 37-configurator-Check_undeploy                     [  OK  ] Deploy: 38-configurator-Service_stop                       [  OK  ]  * Stopping tomcat-mpc instance Using CATALINA_BASE:   /store/tomcat/mpc [ ... ] Deploy: 39-configurator-Run_Custom_PreDeploy_Script        [  OK  ] Deploy: 40-configurator-Deploy_WAR                         [  OK  ] Deploy: 42-configurator-Service_start                      [  OK  ]  * Starting tomcat-mpc instance Using CATALINA_BASE:   /store/tomcat/mpc [ ... ] Deploy: 43-configurator-Check_deploy                       [  OK  ] Deploy: 44-configurator-Enable_extra_services              [  OK  ] Deploy: 80-configurator-Run_POST-Checks                    [ OK ] PROCS OK: 4 processes with command name 'apache2' TCP OK - 0.004 second response time on port 80|time=0.003824s;;;0.000000;10.000000 [ ... ] Reporting: 20-configurator-Mail_Notification               [  OK  ] REPORT FOR PUPPI - STATUS OK Summary of operations is: /var/log/puppi/configurator/20110224-114729/summary Details are in: /var/log/puppi/configurator/20110224-114729/ Temporary workdir has been: /tmp/puppi/configurator/ (Will be rewritten at the next puppi run) Runtime config file is: /tmp/puppi/configurator/config Files have been archived in: /var/lib/puppi/archive/configurator/20110224-114729 If you use Example42 modules, enable their automatic monitoring features ($monitor=yes) and use Puppi, among others, as monitoring tool (ie: $monitor_tool=["nagios","munin","puppi"] ) you have out of the box the possibility of running local checks of the applications managed by Puppet with the command puppi check. Let's see an example: root@metaportali-mpc:/# puppi check Host check: 50-apache_process                              [  OK  ] PROCS OK: 4 processes with command name 'apache2' Host check: 50-apache_tcp_80                               [  OK  ] TCP OK - 0.001 second response time on port 80|time=0.000668s;;;0.000000;10.000000 Host check: 50-mcollective_process                         [  OK  ] PROCS OK: 1 process with command name 'ruby', args 'mcollectived' Host check: 50-munin_process                               [  OK  ] PROCS OK: 1 process with command name 'munin-node' Host check: 50-munin_tcp_4949                              [  OK  ] TCP OK - 0.003 second response time on port 4949|time=0.003184s;;;0.000000;10.000000 Host check: 50-nrpe_process                                [  OK  ] PROCS OK: 1 process with command name 'nrpe' Host check: 50-nrpe_tcp_5666                               [  OK  ] TCP OK - 0.005 second response time on port 5666|time=0.005355s;;;0.000000;10.000000 Host check: 50-openssh_process                             [  OK  ] PROCS OK: 13 processes with command name 'sshd' Host check: 50-openssh_tcp_22                              [  OK  ] TCP OK - 0.001 second response time on port 22|time=0.000857s;;;0.000000;10.000000 Host check: 50-puppet_process                              [  OK  ] PROCS OK: 1 process with command name 'puppetd' Host check: 50-rsyslog_process                             [  OK  ] PROCS OK: 1 process with command name 'rsyslogd' Host check: 50-snmpd_process                               [  OK  ] PROCS OK: 1 process with command name 'snmpd' Host check: 50-tomcat-mpc                                  [  OK  ] PROCS OK: 2 processes with command name 'java', args 'mpc' Host check: 50-tomcat-verticali                            [  OK  ] PROCS OK: 1 process with command name 'java', args 'verticali' Host check: 50-tomcat_tcp_8200                             [  OK  ] TCP OK - 0.001 second response time on port 8200|time=0.000697s;;;0.000000;10.000000 Host check: 50-tomcat_tcp_8201                             [  OK  ] TCP OK - 0.001 second response time on port 8201|time=0.001045s;;;0.000000;10.000000 From these simple examples it should be clear that with Puppi you have:- A common command to manage deploys and rollbacks on any host.- A simple to use, coherent and still flexible and customizable set of defines to describe in Puppet manifests all the elements you need for your application deployments. - A quick way to check if everything is ok on your local node (if you use Example42 module set). Think about it: with just a Puppet define you build the whole deploy logic- Reporting for each deploy/rollback is built-in and extensible - Automatic checks can be built in the deploy procedure- You have a common, growing, set of general-use scripts for typical actions- You can define custom deployments workflows (puppi::project::myprocedure) for specific cases not covered by the provided sample defines. Using the puppi command Puppi provides various actions and options, let's see the main ones: puppi <action> <project_name> [ -options ]These are, currently, the available actions:- puppi init <project_name> : First time initialization of the defined project- puppi deploy <project_name> : Deploys the defined project- puppi rollback <project_name> : Rollback to a previous deploy state - puppi check [project_name] : Runs project specific and host wide checks For each of these actions puppi runs the commands in /etc/puppi/projects/$project_name/$action, logs their status and then runs the commands in /etc/puppi/projects/$project_name/report to provide reporting, in whatever, pluggable, way.You can also provide some options:-f : Force puppi commands execution also on CRITICAL errors-i : Interactively ask confirmation for every command-t : Test mode. Just show the commands that should be executed without doing anything-d <yes|full>: Debug mode. Show debugging info during execution-o "parameter=value parameter2=value2" : Set manual options to override defaults. The options must be in parameter=value syntax, separated by spaces and inside double quotes. Here are some command line samples: puppi check : Run host-wide checks. puppi init myapp : Make the first deploy of "myapp". Can be optional.puppi deploy myapp : Deploys myapp with the standard logic/parameters defined in Puppetpuppi deploy myapp -f : Deploys myapp and doesn't stop in case of Critical errorspuppi deploy myapp -i : Deploys myapp in interactive mode. Confirmation is asked for each steppuppi deploy myapp -t : Test mode. Just show the commands that would be executedpuppi deploy myapp -d full : Deploys myapp with full debugging outputpuppi deploy myapp -i -o "version=1.1 source_url=http://dev.example42.com/code/my_app/": Deploys myapp in interactive mode and sets some custom options that override the standard Puppet params. Note that these parameters change according to the script you use (and the scripts must honour this override).puppi rollback myapp : Rollbacks myapp to a previous archived state. User is prompted to choose which deploy to override. File Paths All the paths used by the puppi scripts are provided, and can be configured, in the puppi Puppet module:/usr/sbin/puppi - Where the puppi command is placed. Currently is a bash script./etc/puppi/puppi.conf - Puppi main config file. Here various puppi wide paths are defined/etc/puppi/checks/ ($checksdir) - Here can be placed all the host wide checks. If you use the Example42 monitor module and have "puppi" as $monitor_tool, this directory is automatically filled with checks based on Nagios plugins./etc/puppi/projects/ ($projectsdir) - In this directory you can have one or more projects subdirs, with the commands to be run for deploy, rollback and check actions. They are completely built (and purged) by the Puppet module./etc/puppi/scripts/ ($scriptsdir) - The general-use scripts directory, these are used by the above commands and may require one or more arguments./var/lib/puppi/archive/ ($archivedir) - Where all data to rollback is placed./var/log/puppi/ ($logdir) - Where logs and reports of the different commands are placed./tmp/puppi/ ($workdir) - Temporary, scratchable, directory where Puppi places temporary files. It's wiped at every puppi run./tmp/puppi/$project/config - A runtime configuration file, where are dinamically placed variables usable by all the scripts invoked by puppi. This is necessary to mantain "state" information that changes on every puppi run (such as the deploy datetime, used for backups). Understanding the Puppi module The puppi module provides few basic defines to manage puppi's setup elements and some example templates that use these defines to build deployment workflows.The basic defines are:puppi::project - Creates the main project structure. One or more different deployment projects can exist on a node.puppi::initialize - Creates a single command to be placed in the init sequence. It's not required for every project.puppi::deploy - Creates a single command to be placed in the deploy sequence. More than one is generally needed for each project.puppi::rollback - Creates a single command to be placed in the rollback sequence. More than one is generally needed for each project.puppi::check - Creates a single check (based on Nagios plugins) for a project or for the whole host (host wide checks are auto generated by Example42 monitor module)puppi::report - Creates a reporting command to be placed in the report sequence.The above init, deploy, rollback, check and report defines have generally a standard structure and similar arguments. Every one is reversable (enable => false) but you can wipe out the whole /etc/puppi directory to have it rebuilt from scratch.Here is an example for a deploy command: puppi::deploy { "Retrieve files": # The $name of the define command => "get_curl.sh", # General-use script to use argument => "file:///storage/file", # Argument(s) for the script priority => "10", # Execution order user => "root", # Execution user project => "myapp", # The name of the project } This define creates a file named /etc/puppi/projects/${project}/deploy/${priority}-${name}Its content is, simply:su - ${user} -c "export project=${project} && /etc/puppi/scripts/${command} ${arguments}"You can glue together, with the desired order according to the priority argument, different basic defines to create a complex project template and use this in your manifests. Various sample defines that can be used to many common deployment scenarios are present in puppi/manifests/project/ You can use puppi::project::war to manage the deployment of a simple war file, as seen before, or other defines to manage deployments of a tarball, an arbitrary list of files, or the maven artifacts published on a Nexus repository. The puppi/files/scripts directory in the module contains some general usage scripts that can be used in custom deployments.They are generally made to work together according to a specific logic, which is at the base of the sample defines in puppi/manifests/project/ but you're free to write your own scripts, in whatever language, according to your needs, and integrate them with custom Puppet defines. The default scripts are engineered to follow this procedure for a deployment:- Remote files are downloaded in /tmp/puppi/$project/store or directly in the predeploy directory: /tmp/puppi/$project/deploy- If  necessary the downloaded files are expanded in one or more predeploy directories (default:/tmp/puppi/$project/deploy)- Runtime configuration entries might be saved in /tmp/puppi/$project/config- Files are eventually backed from the deploy directory (Apache' docroot, Tomcat webapps or whatever) to the archive directory (/var/lib/puppi/archive/$project)- Files are copied from the predeploy directory to the deploy dir. - Relevant services are eventually sopped and started The most used common scripts are (they might have different arguments, some of them are quite simple):get_file.sh - Retrieves a remove file via ssh/http/rsync/svn and places it in a temporary directory (store or predeploy)deploy.sh - Copies the files in the predeploy dir to deploy dirarchive.sh - Backups and restores files in deploy dirservice.sh - Stops or starts one or more serviceswait.sh - Waits for the presence or absence of a file, for the presence of a string in a file or a defined number or seconds.get_metadata.sh - Extracts metadata from various sources in order to provide info to other scriptsreport_mail.sh - Sends a mail with the report of the operations done How to customize It should be clear that with puppi you have full flexibility in the definition of a deployment procedure, since the puppi command is basically a wrapper that executes arbitrary scripts with a given sequence, in pure KISS logic.There are different parts where you can customize the behaviour of puppi: - The set of general-use scripts in /etc/puppi/scripts/ ( this dir is filled with the content of puppi/files/scripts/ ) can/should be enhanced. As been, these can be arbitrary scripts in whatever language. If you want to follow puppi's logic, though, consider that they should import the common and runtime configuration files and have an exit code logic similar to the one of Nagios plugins: 0 is OK, 1 is WARNING, 2 is CRITICAL.Note that by default a script that exits with WARNING doesn't block the deploy procedure, on the other hand, if a script exits with CRITICAL (exit 2) by default it blocks the procedure. Take a second, also, to explore the runtime config file created by the puppi command that contains variables that can be set and used by the scripts invoked by puppi. - The custom project defines that describe deployments procedures. These are placed in puppi/manifests/project/ and can request all the arguments you want to feed your scripts with. Generally is a good idea to design a standard enough template that can be used for all the cases where the deployment procedure involves similar steps. Consider also that you can handle exceptions with variables (see the $loadbalancer_ip usage in puppi/manifests/project/maven.pp) What's next The Puppi module is a work in progress that is already used in production. It has been designed to automate deployments in a specific environment but a lot of effort has been done to standardize as much as possible the procedures and the options for differents needs and logic and to make it modular by design.The same puppi command can be re-written from scratch. It's has been made in bash and not in fancier languanges for simplicity and portability.What it does at the moment, after all, is something quite simple (basically execute commands in a sequence). The whole point of the Puppi module is to define quickly and standardize in Puppet manifests any kind of deployment procedure (at least, at the moment, any kind of procedure that can be executed from the same node, orchestration is planned but not yet functional). Future plans involve:- dedicated agents for orchestration tools like Mcollective or ControlTier- more reporting scripts (for example reporting on Google Calendar and Docs)- a web interface to manage the deployment procedure and see reports 2011-02-15T00:00:00+00:00 https://example42.com/blog/2011/02/15/understanding-example42-modules Example42 modules are based on a module template called "foo". They share a common structure that might seem too complex at first glance but once grasped few basic elements it results coherent, easy to manage and solid (so we like to think).The main features of Example42 modules are:- Support for different Operating Systems,- Separation of custom code and logic from the module core,- Extendible automatic monitoring for the resources provided by the module,- Coherent structure with standard naming,- Debugging functionalities,- Plug & Play approach with User variables that can change the module's behaviour. The usage of a common foo template makes it possible to create a new full featured module very quickly, using the script 00_example42scripts/module_clone.sh . The foo module template is an evolving skeleton for modules, new features are introduced with new versions and the approach is done in order ot make the "upgrade" of a module generated from an earlier foo version relatively easy. Currently the Puppet code is compatible both for version 0.25 and version 2.6, this is done for backward compatibility (the default Puppet version on many distros is still 0.2x) at the price of renouncing some of the new interesting features of Puppet 2.6. The foo module template structure Let's examine the foo module, once you understand it, you can grasp the logic behind all the other modules and cope with their apparently complex manifests. The module's layout follows Puppet official standards and some community best practices.At its root you can see the following elements: manifests/ # The directory with all the module's classes and defines templates/ # The directory for the templates tests/ # The directory containing quick tests Modulefile # Metadata for the Module Forge README # General module's documentation Let's begin from the quick things.The tests directory actually is not widely used at the moment, but is intended, in the future, to provide unit tests for the module's classes and defines. Currently in each test, there's just an "include classname". Nothing particularly relevant or exciting. The templates directory contains a file called: templates/variables_foo.erb This is a template that shows all the variables used in the module and is placed on /var/lib/puppet/debug/variables/foo if you have the $debug variable set to yes (reccomended if you use Example42 modules, it helps a lot in troubleshooting variables scoping). Then there's the manifests directory, which contains all the modules's classes and defines: init.pp # Contains the main "foo" class params.pp # foo::params contains ALL the module's variables absent.pp # foo::absent removes the foo package disable.pp # foo::disable stops the foo service # (both runtime and at boot) disableboot.pp # foo::disableboot removes foo service from boot # but doesn't check if is running debug.pp # foo::debug provides the debugging template monitor.pp # foo::monitor describes what has to be monitored # for the foo application monitor/absent.pp # foo::monitor::absent decommits also monitoring # elements when you include foo::absent backup.pp # foo::backup describes what to backup of foo backup/absent.pp # foo::backup::absent decommits backup elements firewall.pp # foo::firewall describes what has to be firewalled firewall/absent.pp # foo::firewall::absent conf.pp # Generic foo::conf define for infile line # modifications of foo's main configuration file example42.pp # foo::example42 is the custom class where to place # all the customizations. This is renamed to your # $my_project by the project_rename.sh script example42/monitor.pp # foo::example42::monitor for monitoring customizations example42/backup.pp # foo::example42::backup for backup customizations As you see there are quite a lot of classes but generally you have to touch just few of them, when you create a module from scratch, based on the foo template, and ideally you should not modify anything else than the example42 classes (renamed to your $my_project) when you have to use existing modules. It all starts from init.pp If you look inside a class you will see Puppet code that at first sight might look not clear. In all the classes in fact there's wide usage of qualified variables ( things like $foo::params::packagename ) that refer to the ones defined in the params class.Let's see, for example how the main foo class present in init.pp is done, and how this class may actually include many of the other classes defined before if the right variables are set: # # Class: foo # # Manages foo. # Include it to install and run foo # It defines package, service, main configuration file. # # Usage: # include foo # class foo { # Load the variables used in this module. Check the params.pp file require foo::params # Basic Package - Service - Configuration file management package { "foo": name => "${foo::params::packagename}", ensure => present, } service { "foo": name => "${foo::params::servicename}", ensure => running, enable => true, hasrestart => true, hasstatus => "${foo::params::hasstatus}", pattern => "${foo::params::processname}", require => Package["foo"], subscribe => File["foo.conf"], } file { "foo.conf": path => "${foo::params::configfile}", mode => "${foo::params::configfile_mode}", owner => "${foo::params::configfile_owner}", group => "${foo::params::configfile_group}", ensure => present, require => Package["foo"], notify => Service["foo"], # content => template("foo/foo.conf.erb"), } # Include OS specific subclasses, if necessary case $operatingsystem { default: { } } # Include extended classes, if relevant variables are defined if $backup == "yes" { include foo::backup } if $monitor == "yes" { include foo::monitor } if $firewall == "yes" { include foo::firewall } # Include project specific class if $my_project is set # The extra project class is by default looked in foo module # If $my_project_onmodule == yes it's looked in your project module if $my_project { case $my_project_onmodule { yes,true: { include "${my_project}::foo" } default: { include "foo::${my_project}" } } } # Include debug class is debugging is enabled ($debug=yes) if ( $debug == "yes" ) or ( $debug == true ) { include foo::debug } } Let's analyze what this class does:- At the beginning it has a brief description, in Puppetdoc compliant standard.- Then the first thing is the inclusion of foo::params where all the variables used later are defined. Foo::params is included in all the classes and represents the only point where modules variables are defined: both "internal variables" such as the packagename and "user variables" that are somehow "filtered" in foo::params which sets default values if these variables are not actually provided by users. This choice is done because we want "include and play" modules that work, with default settings, out of the box.- Then there are the definitions of the main resources of the foo application: the package, the service and the main configuration file. Of course these may change from case to case if more resources have to be defined for a certain application (for example more packages or more configuration files). Just notice the wide usage of qualified variables and that the configuration file name (foo.conf) is just the name of the resource and not the pathname of the file, so that, for example, we will have a File resource called "samba.conf" in the Samba module, even if the actual filename is smb.conf (as defined by the samba::params::configfile variable).- Then OS specific subclasses are included if necessary. This is done in modules where the difference between different distros or OS in managing an application are more radical than just names changes and require additional resources. Eventual OS specific classes are placed in different files with the fact name of the Operating system (ie: class foo::ubuntu placed in foo/manifests/ubuntu.pp).- After there's the automatic inclusion of the so-called Example42 extended classes for monitoring, backup and firewalling. These classes are included if the relevant user variables are set to "yes", so you can also ignore them if you don't intend to use these extensions.- Then there's the code that manages the autoloading of custom $my_project classes. The if and case constructs are just needed to load the $my_project class from a file present in the same module or from a dedicated "project related" module. By default the custom classes are placed in the application module (as seen before with the example42.pp file) but you can have them completely separated in a module called $my_project setting $my_project_onmodule to "yes".- Finally there's the inclusion of the debug class if the $debug user variable is set to "yes". In various cases (when to manage an application you just have to install a package, manage a service and deploy a configuration file) there's actually no need to touch the init.pp file at all, since all the OS relevant modifications are defined in the params.pp class. In other cases, further resources might be added or sub classes included, according to the module's logic and complexity. The relative beauty and absolute power of params.pp The params class might be considered the cornerstone of all the other classes, has a standard structure, whose content, obviously, changes according to the managed application. Let's see a default params class (the parts with [...] indicate omissis for uninteresting or reduntand parts): class foo::params { ## DEFAULTS FOR VARIABLES USERS CAN SET # (Here are set the defaults, provide your custom variables externally) # (The default used is in the line with '') ## Example: Full hostname of foo server $server = $foo_server ? { '' => "foo", default => "${foo_server}", } ## EXTRA MODULE INTERNAL VARIABLES #(add here module specific internal variables) ## MODULE INTERNAL VARIABLES # (Modify to adapt to unsupported OSes) $packagename = $operatingsystem ? { solaris => "CSWfoo", debian => "foo", ubuntu => "foo", default => "foo", } $servicename = $operatingsystem ? { debian => "foo", ubuntu => "foo", default => "foo", } $configfile = $operatingsystem ? { freebsd => "/usr/local/etc/foo/foo.conf", default => "/etc/foo/foo.conf", } [...] $protocol = "tcp" $port = "80" ## DEFAULTS FOR MONITOR CLASS # These are settings that influence the (optional) foo::monitor class # You can define these variables or leave the defaults # The apparently complex variables assignements below follow this logic: # - If no user variable is set, a reasonable default is used # - If the user has set a host-wide variable (ex: $monitor_target ) that one is set # - The host-wide variable can be overriden by a module specific one (ex: $foo_monitor_target) # How the monitor server refers to the monitor target $monitor_target_real = $foo_monitor_target ? { '' => $monitor_target ? { '' => "${fqdn}", default => $monitor_target, }, default => "$foo_monitor_target", } # If foo port monitoring is enabled $monitor_port_enable = $foo_monitor_port ? { '' => $monitor_port ? { '' => true, default => $monitor_port, }, default => $foo_monitor_port, } [...] ## FILE SERVING SOURCE # Sets the correct source for static files # In order to provide files from different sources without modifying the module # you can override the default source path setting the variable $base_source # Ex: $base_source="puppet://ip.of.fileserver" or $base_source="puppet://$servername/myprojectmodule" # What follows automatically manages the new source standard (with /modules/) from 0.25 case $base_source { '': { $general_base_source = $puppetversion ? { /(^0.25)/ => "puppet:///modules", /(^0.)/ => "puppet://$servername", default => "puppet:///modules", } } default: { $general_base_source=$base_source } } } This class is divided in various parts: DEFAULTS FOR VARIABLES USERS CAN SETHere are enforced the default values for all the module specific variables users can set.The syntax might look a bit verbose but that's what is currently needed to manage variables' value assignement (at least with a standard Puppet 0.2x).Note that you set the value of the variable $server (that is $foo::params::server) according to the value of the user's variable $foo_server. The convention is to have all the users' variables prepended by the relevant module name (as foo_) and reassigned, in params.pp, with an omonimous variable without the foo_ prefix.Note also how the $foo::params::server value is defined: if $foo_server is null (that is if the user has not defined it) it has the default value "foo", otherwise it gets the value of $foo_server.You might define many other module specific user variables that might be used in templates of in the module logic to autoload specific classes. EXTRA MODULE INTERNAL VARIABLESHere you might find other variables that are used internally in the module (the user has not to define them). They are placed here, before other more relevant parameters, because all the other following variables in the module are the same (with different values) in different modules. MODULE INTERNAL VARIABLESIn this section you find all the variables that are used in the module's classes. As you can see different values can be given to valiables like $foo::params::packagename,  $foo::params::configfile etc. according to the underlining operating system.On a simple module, just by changing these variables you can manage basic support for different operating systems.This set of variables is the same on all the modules derived from the foo template and makes it quick and easy to clone new modules based on it. DEFAULTS FOR MONITOR/BACKUP/FIREWALL CLASSThese 3 groups of variables manage the behaviour of the omonimous extended classes. You can ignore them if you don't use these features (shame! :-) . The horrendously verbous syntax used to assign their values is needed to set defaults if the user doesn't define neither a global variable (such as $monitor_target to define how to reach the node for monitoring purposes) nor a module specific variable (such as $foo_monitor_target to define the monitoring target to use only for the foo application). The default value of the variable that is actually used ($foo::params::monitor_target_real) in this example is ${fqdn}.The existing Example42 modules have a reasonable default for all these settings. FILE SERVING SOURCEFinally is defined the $general_base_source variable, which is the same in every module (it's repeated in order to make single modules reusable out of the whole Example42 set). This variable should be used whenever there's a "source" parameter to provide a location for file serving.Note that by default this variable just set the correct naming for the current Puppet version, but you can define a site wide $base_source to provide static files from a different server. Disabling and removing modules Init.pp and params.pp are the most relevant manifests in the module, but there are various other manifest files that provide omonimous classes to manage common features. So of them have to inherit the main class in order to override specific resources, such as the service to manage in disable.pp: class foo::disable inherits foo { require foo::params Service["foo"] { ensure => "stopped" , enable => "false", } # Remove relevant monitor entries if $monitor == "yes" { include foo::monitor::absent } } Note that this class automatically includes the foo::monitor::absent class to decomission monitoring resources for the disabled service.In all the modules based on the foo template there is also a foo::disableboot class that is identical to the above foo::disable but it hasn't the "ensure => stopped" argument.You can use this class in all the cases where the specific service is not directly managed by Puppet, for example in clusters where there's the cluster software that starts services at boot and assures they are running on the desired node. Kris Buytaert has suggested to rename this class to a more elegant and meaningfull foo:unmanaged, maybe this will be done in future versions of the foo template (but we fear it might be too late).In other cases, such as absent.pp we don't want (and need) to inherit the main class: class foo::absent { require foo::params package { "foo": name => "${foo::params::packagename}", ensure => absent, } # Remove relevant monitor, backup, firewall entries if $monitor == "yes" { include foo::monitor::absent } if $backup == "yes" { include foo::backup::absent } if $firewall == "yes" { include foo::firewall::absent } # Include debug class is debugging is enabled ($debug=yes) if ( $debug == "yes" ) or ( $debug == true ) { include foo::debug } } Note that here we remove all the monitoring/backup/firewall resources but still include the debug class. Example42 extended classes Finally let's give a glimpse to the so-called extended (and embraced?) Example42 classes.These are powerful meta-classes that let you automatically manage monitoring, firewalling and backup of the resources provided by the module, at the cost of some overhead in terms of number of resources applied to the node and the need, in most cases, of store configs support.Let's see the most interesting manifest of the bunch, monitor.pp, we leave the original comments because they explain how to use the module's variables: # Class: foo::monitor # # Monitors foo process/ports/service using Example42 monitor meta module (to be adapted to custom monitor solutions) # It's automatically included and used if $monitor=yes and is defined at least one monitoring software in $monitor_tool # This class permits to abstract what you want to monitor from the actual tool and modules you'll use for monitoring # and can be used to quickly deploy a new monitoring solution that instantly notifies what's working and what's needs # to be fixed (call it Test Driven Puppet Deployment, if you like ;-) # # Variables: # The behaviour of this class has some defaults that can be overriden by user's variables: # $foo_monitor_port (true|false) : Set if you want to monitor foo's port(s). If any. Default: As defined in $monitor_port # $foo_monitor_url (true|false) : Set if you want to monitor foo's url(s). If any. Default: As defined in $monitor_url # $foo_monitor_process (true|false) : Set if you want to monitor foo's process. If any. Default: As defined in $monitor_process # $foo_monitor_plugin (true|false) : Set if you want to monitor foo using specific monitor tool's plugin is. If any. Default: As defined in $monitor_plugin # $foo_monitor_target : Define how to reach (Ip, fqdn...) the host to monitor foo from an external server. Default: As defined in $monitor_target # $foo_monitor_url : Define the baseurl (http://$fqdn/...) to use for eventual foo URL checks. Default: As defined in $monitor_url # # You can therefore set site wide variables that can be overriden by the above module specific ones: # $monitor_port (true|false) : Set if you want to enable port monitoring site-wide. # $monitor_url (true|false) : Set if you want to enable url checking site-wide. # $monitor_process (true|false) : Set if you want to enable process monitoring site-wide. # $monitor_plugin (true|false) : Set if you want to try to use specific plugins of your monitoring tools # $monitor_target : Set the ip/hostname you want to use on an external monitoring server to monitor this host # $monitor_baseurl : Set baseurl to use for eventual URL checks of services provided by this host # For the defaults of the above variables check foo::params # # Usage: # Automatically included if $monitor=yes # Use the variable $monitor_tool (can be an array) to define the monitoring software you want to use. # To customize specific and more granular behaviours use the above variables and eventually your custom modulename::monitor::$my_project class # class foo::monitor { include foo::params # Port monitoring monitor::port { "foo_${foo::params::protocol}_${foo::params::port}": protocol => "${foo::params::protocol}", port => "${foo::params::port}", target => "${foo::params::monitor_target_real}", enable => "${foo::params::monitor_port_enable}", tool => "${monitor_tool}", } # URL monitoring monitor::url { "foo_url": url => "${foo::params::monitor_baseurl_real}/index.php", pattern => "${foo::params::monitor_url_pattern}", enable => "${foo::params::monitor_url_enable}", tool => "${monitor_tool}", } # Process monitoring monitor::process { "foo_process": process => "${foo::params::processname}", service => "${foo::params::servicename}", pidfile => "${foo::params::pidfile}", enable => "${foo::params::monitor_process_enable}", tool => "${monitor_tool}", } # Use a specific plugin (according to the monitor tool used) monitor::plugin { "foo_plugin": plugin => "foo", enable => "${foo::params::monitor_plugin_enable}", tool => "${monitor_tool}", } # Include project specific monitor class if $my_project is set if $my_project { case $my_project_onmodule { yes,true: { include "${my_project}::foo::monitor" } default: { include "foo::${my_project}::monitor" } } } } Just notice how nothing in this class has application specific information: replace foo with apache and you have the monitoring class for apache based on settings defined in params.pp.Consider that generally are activated (via the $monitor_*_enable variables set in params.pp) only the process and port checks. Url checks are quite useful to check web applications funnctionality but are seldom related to a module application and plugins are special cases used to monitoring tools that provide application oriented plugins. Currently there are 5 monitor tools (with different scopes and usage fields) supported: Nagios, Munin, Collectd, Monit and Puppi (an Example42 module for applications deployments and local checks of services) but the nice side effect of this monitoring abstraction (note that in the above class we have defined WHAT to monitor, not HOW to monitor it) is that you just have to write the plugin for a new tool in the monitor module (more about it in another post) and you'll find yourself with all the monitoring elements already configured for it. Note also that you can use different modules for the supported monitoring tools, for example you might use Immerda, DavidS or Camptocamp Nagios modules instead of the native Example42 one for Nagios monitoring. Isolation of customizations Among the various classes present in the foo template there are the ones called example42. Here we provide the possibility to customize the module behaviour without modifying other parts. The path to complete module's customization and resuability is far from being fulfilled, using custom "project" classes we have considered various issues.The logic on how to provide configuration files and their content should not be enforced by a module. It varies too much according to specific environments, operating systems and needs. That's why the Example42 modules often don't provide by default the content of a configuration file, but just its general properties, such as path and  permissions. HOW you want to provide that file should be defined in a custom class.As we have seen custom classes are autoloaded if you define the variable $my_project (in this case its value is "example42"), these classes should inherit the main one only when they set some parameters for an already defined resources. If you just add new resources just don't have to inherit anything.For example you might want to provide a static file according to a custom order: class foo::example42 inherits foo { File["foo.conf"] { source => [ "puppet:///foo/foo.conf-$hostname" , "puppet:///foo/foo.conf-$role" , "puppet:///foo/foo.conf" ], } } or use a template: class foo::example42 inherits foo { File["foo.conf"] { content => template("foo/foo.conf.erb"), } } Having a dedicated $my_project class to manage customizations permits better delegation of the management of Puppet code when different teams work on indipendent infrastructures that might share some common Puppet modules.In this case each infrastructure is to be considered a different project.To ease this kind of delegation the custom classes can be searched in a dedicated module, just set $my_project_onmodule to "yes" for this. The use of custom classes faces one of the problems of modules' reasability and customization. Another one is how to provide data that is used by a module (variables values, source file paths and so on). Puppet 2.6 already natively provides the extlookup function that lets you dynamically define data sources (note that you can introduce and use extlookup in these custom classes, even if it's not used in the core classes) and some discussion is ongoing on how to separate the module data. What is still unclear, at least to me, is how much people really need or want full module's reusability, and if they are likely to accept the extra complexity that this probably leads to. When using the Example42 modules in real scenarios I've seen that there's always the temptation of following the quick approach of placing custom assumptions in the core classes: it's easier and after all it doesn't prevent reusability (just copy, paste and modify where needed) but ... well, it's simply not the right way, at least concerning Example42 modules reusability ambition. More details on how to customize modules are here. Who fears variables scoping issues? As you should have realized by now Example42 modules rely heavily on variables, both internal and user provided ones.This somehow conflicts with some philosophical approaches emerging in the Puppet community such as facts-driven configuration or the endless fear of variables scoping issues.Well let's underline some points:- I don't think there's the right way to do things with Puppet. There are probably many wrong ways and there's the one you find comfortable with, that fits your infrastructure, needs and mindset.- I'm conscious of some of Example42 modules limitations, for example the fact that they are not optimized for performance, their verbosity or some very inelegant constructs (see default enforcings in params.pp). Still there are design choices behind them: compatibility with Puppet 0.2x versions, quick cloning with just a find and replace script, abstraction, reusability and include&play. These concepts have been expressed also in presentation at PuppetCamp 2010 Europe and PuppetCamp 2010 SF.- Variables scoping issues are not generally due to modules organization but to how and where you define your variables and when you include your classes: under this point of view the use of qualified variables and the params class inside the module helps in having, at least in the module itself, a rigorous scoping coherency.- There are various ways to "define variables and include classes", you can do that with external node classifiers, dedicated classes or whatever method I won't debate here. I've found reliable, scalable and well manageable an approach based on nodes inheritances, where you define and override variables at different levels of the inheritance tree, and ONLY AT THE END you include classes (typically a general class that include common classes and a role class that includes role specific classes).You might also define role specific variables inside a role class, but always before including the classes that use them. Trust me, it's ages I don't face scoping problems with this appoach.- If you use Example42 modules, turn on debugging (set $debug to "yes"). You will find in /var/lib/puppet/debug/variables/modulename all the variables used in your module and this lets you spot immediately during a Puppet run if some variable is not set as you want. Any comment on this post and the solutions used in Example42 modules is very welcomed, more than once, in the past I've modified my "template module" and the approach used in designing modules. The same foo template is to be considered an evolving work in progress, that is going to follow Puppet evolution and new features. Always open to redefine my Puppet beliefs, given the right motivations. 2011-01-02T00:00:00+00:00 https://example42.com/blog/2011/01/02/how-to-setup-a-puppetmaster-with-example42-modules NOTE May 2012. The documentation below refers to old versions of the Example42 Puppet Modules and sample Infrastructures. You can still use them but be sure to git clone the correct branch versions (1.0 and 2011 respectively). The setup of Puppet for a beginner is not always easy. You have to install and configure the PuppetMaster and the Puppet clients, exchange and sign their certificates and then start to write Puppet code in order to see something interesting happening. To install the Puppet Master is not difficult but you need some tweaks in order to activate specific features and functionalities. With the Example 42 puppet module you can easily setup a PuppetMaster with the following features: - Integration with the web frontends Dashboard or the Foreman with or without external nodes. - Support for 0.25 and 2.6 versions- Support for storeconfigs (full and thin) - Support for Apache Mod Passenger and Mysql database backend - Support for these distro: RedHat/Centos, Debian/Ubuntu.  First of all you have to install Puppet and Git packages, following the procedures needed for your Operating System. Then you can clone the Example42 modules and infrastructures repository (you might need to rename or remove the directories /etc/puppet/modules and /etc/puppet/manifests if they already exist): git clone --recursive -b 1.0 git://github.com/example42/puppet-modules.git /etc/puppet/modules git clone --recursive -b 2011 git://github.com/example42/puppet-infrastructures /etc/puppet/manifests Then you can edit the Example42 PuppetMaster Toaster manifest and set there the variables you need. Note that, even if this file uses normal Puppet language is intended to be used in "masterless" mode, via a direct puppet command run, as shown below. vi /etc/puppet/manifests/toasters/puppet/init.pp There are different variables that you can set in order to obtain the setup you prefer. The most important ones are: $puppet_server (default: "puppet") - Defines the hostname of the puppet master. Set it to the fqdn of your PuppetMaster (the preferred choice is to call it puppet.yourdomain.com) $puppet_allow (default: "*.$domain" , "127.0.0.0") - Defines the acls on PuppetMaster's fileserver.conf. Put here the networks or domain names of your Puppet clients. $puppet_nodetool (default: undefined) - Use to install an external web interface ("dashboard" or "foreman" are the possible values) which can be used for reporting or as an external node classifier (see next option). $puppet_externalnodes (default: no) - Use the specified nodetool as external node classifier (if "yes" here, $puppet_nodetool must be defined). If set to yes your nodes, variables and classes are defined in the chosen web interface instead of normal Puppet manifests (as the example ones provided in the Example42 Puppet Infrastructures). $puppet_passenger (default: no) - Use Apache's mod ruby (Passenger) instead of default Webric. This is a preferred choice for medium/large installations.You might prefer the default, more straightforward, choice for test beds or small installations (less than 100 nodes). $puppet_storeconfigs (default: no) - Use storeconfigs, a feature that requires the use of a database (SQLLite or MySQL are the supported alternatives here) to save data related to the nodes. This option is needed when you use exported resources (as in Example42's Monitoring classes) but can heavily impact on the PuppetMaster performances. $puppet_storeconfigs_thin (default: yes) - Use thin storeconfigs, a "lighter" usage of storeconfigs with less performance issues but the same support for exported resources. This is a sub-options, it still needs $puppet_storeconfigs = yes to be activated. $puppet_db (default: sqlite) - The Database backend to use if you activate storeconfigs ("sqlite" or "mysql" are the possible values, the former should be used only in test beds or very small installations). $puppet_db_server (default: localhost) ,  $puppet_db_user (default: root) , $puppet_db_password (default: undefined)  control the database access credentials when using MySql. Once edited the toasters/puppet/init.pp file with the settings you want, you can run Puppet on it: puppet -v /etc/puppet/manifests/toasters/puppet/init.pp Here starts the Puppet dance that might last some minutes while packages are installed (we imply that the system has Internet access or, in any case, can install packages with its tools such as apt-get or yum), files are configured and things are generally done. If there have been dependencies issues (should not), you may need to re-run the above Puppet command , as many times as needed, until nothing new happens, and at the end you have your Puppet Master setup. Now begins the fun:- If you have set $puppet_externalnodes to yes browser to http://puppet.your.domain.com:3000 and start to add nodes, variables and include classes from there.- Otherwise look at /etc/puppet/manifests/site.pp (the first file the PuppetMaster looks at when it has to compile the catalog for its clients: from here everything begins). The one provided with Example42 Infrastructures you cloned earlier imports other manifests (.pp) where are defined example nodes, that include classes and set variables used by these classes. Check the modules' README for more info on the variables each module accepts and change the nodes names according to your clients' fqdns. - Then on your clients, you can install the Puppet package, and run puppet: puppetd -t or, if your PuppetMaster cannot be resolved as "puppet" puppetd -t --server=mypuppetmasterhost.domain.com The first time you run this command no resources are applied, because the client's certificate has not been signed by  the PuppetMaster. Go on the PuppetMaster host and write: puppetca -l to see the current certificates requests, and to sign one type: puppetca -s clienthostname.domain.com Now you can finally run again puppetd -t on the client and see what Puppet has been configured to do on your node.