Deploying hosts is one of the core functions of orcharhino along with configuration management and lifecycle management. It provides various ways to provide information to new hosts during installation and eliminates the need for human interaction.

Value and Vision frequently need to deploy new hosts, often in large numbers, in response to customer requests. Usually, these are virtual machines, while some customers prefer bare metal servers. In all cases, the V&V employees prepare the new host in the web interface of orcharhino. For example, with virtual machines, they define their performance characteristics (CPU, RAM, networking, storage). Of course, it is also determined in which network the host should be located so that customers can access it. Further criteria are the operating system or the user configuration management including roles/class/states. Value and Vision must be prepared for any case, which is why they keep data and clients for almost all operating systems that orcharhino can manage:

• CentOS 6, 7 and 8
• CloudLinux 7
• Debian 9, 10
• Oracle Linux 7 and 8
• Red Hat Enterprise Linux (RHEL) 7 and 8
• SuSE Linux Enterprise Server (SLES) 12 (SP3 and SP4) sowie 15 (inkl. SP1)
• Ubuntu 16.04 LTS and 18.04 LTS

Linux hosts require a kernel and initial ramdisk, which they get either by PXE booting from a boot image. Windows hosts can boot from a special minimal image. Various mechanisms are then used to create the operating system automatically. Under Linux the following have been established: Kickstart/Anaconda (Red Hat family), AutoYAST (SUSE) and Preseed (Debian family). In all cases a program queries the basic configuration via HTTP. The orcharhino-admins have previously written templates, which now create the Kickstart, AutoYAST or Preseed definition from the configuration parameters. The rendered template then contains, for example, the network configuration of the respective host, password hashes, a basic packet selection and how to prepare the configuration management.

orcharhino can start virtual machines directly by addressing the hypervisor API. Most bare metal servers are started by orcharhino via BMC (iLO/IPMI/iDRAC) after their network interfaces have been attached to the switch in the correct VLAN. At Value and Vision, all hosts (physical and virtual) hang on at least two networks: a productive network and an installation network. At the beginning only the installation network plays a role. The orcharhino or its orcharhino proxies act as DHCP servers and deliver all necessary files for PXE booting via TFTP.

After a few minutes, the new hosts are basically set up and reboot. The central configuration management then takes further steps and installs additional software. For example, some customers want web servers. An Ansible role written for this purpose installs the necessary packages, creates configuration files and sets up firewall rules. From now on customers can use their system.

At Finesse-Bank, the process is similar, but more manageable, because only SuSE Linux Enterprise Server is intended as the operating system. And there are fewer application scenarios. In addition, there is a test environment parallel to the production environment, in which adjustments to the infrastructure are simulated. Only when everything is running smoothly there, the admins take the changes to the next level.

The University of Quality, on the other hand, must dig a little deeper for the trick up its sleeve. Some applications require Windows Server to run. These are also installed and managed from orcharhino. On the one hand, there is a Golden Image, which is stored on the virtualization hosts and can be directly integrated. On the other hand, there is also a customizable network installation that uses templates, similar to kickstarting or preseeding under Linux.

A core function of orcharhino is the provision of software repositories – besides configuration management and the deployment of new hosts. However, it is more than a simple mirror server, because it can also version repositories. At Finesse-Bank there is a group of repositories for the operating system, which are updated every night. There is a similar situation for third-party software. At the same time, Finesse Bank develops its own software. This is also packaged in repositories, where the upload is done manually from the developer system. Connected systems do not automatically get access to the latest version of the packages, but a version at a given time.

orcharhino-admins combine several repositories into products. The product SLES 15 SP1, for example, includes repositories for the operating system and updates. Different repositories – not necessarily from a single product – can be grouped together as content views. This allows the relevant content for a target system to be aggregated, for example, at Finesse Bank the content view SLES15_Puppet combines the repositories for SuSE Linux Enterprise Server 15 together with the client tools for orcharhino and the configuration management tool Puppet.

At the file level, all packages are in a pool, possibly in multiple versions. When administrators create a new version of a content view, a new branch is created in the directory tree that contains links to the current packages – a snapshot. At Finesse Bank, the latest version of SLES15_Puppet 11.0 is currently available and was released a few days ago. Version 10.0 is already almost half a year old but has been updated several times in the meantime to fix security holes (more on this later).

At Finesse Bank, the software goes through several development stages that are classically called dev, test, and prod – the Lifecycle Path. Each of these environments is assigned exactly one version. Hosts in test and prod are currently attached to version 10.8 of the content view SLES15_Puppet, in dev version 11.0 is used because there are some new functions available, which the developers would like to use in their software. When the latest version is ready for testing, the version level in the test environment is raised.

Companies in the financial sector usually have high security standards. In order to meet these requirements, IT managers must maintain an overview of any weaknesses in their systems. Under Red Hat the Errata have been established for this purpose – this concept is now also available under other distributions. Behind it hides meta-information about published vulnerabilities. In addition to a code to identify and describe the vulnerability, this includes a list of affected packages and a solution, i.e. in which newer packages the bug is fixed. Theorcharhino gets lists with errata from the vendor repositories. At the same time, it keeps the information about which packages are installed on which systems. ATIX provides Errata for CentOS, Debian and Ubuntu to customers.

For each host, you can easily see with orcharhino if such errata affect the respective system and import them directly. The previously mentioned minor release of the Content View is due to the fact that security updates are in packages that are not yet present in SLES15_Puppet 10.0. In this case, it is possible to update only individual packages and create minor versions of a content view.

In the case of the University of Quality, this process is generally simpler, as there are no in-house developments, but only upstream repositories are maintained. At Value and Vision, the situation is similar, except that the experts there have additionally inserted “Lifecycle Fast Paths” in which minor releases of the content views are tested before they land on customer systems.

orcharhino combines several tools for automation in data centers. Configuration management plays a central role here – in addition to the deployment of new hosts (see ‘orcharhino as installation source’) and the versioned provision of repositories (see ‘Lifecycle Management’) this can be considered as one of its core tasks. orcharhino-users can choose between Ansible, Puppet or SaltStack for configuration management, which are all integrated and can be used via the Web GUI. The Finesse Bank relies entirely on Puppet. The tool has established itself in areas where configuration drifts should be avoided. For this purpose, agents run on the connected systems, which regularly request their target configuration from the puppet master. If their status differs, they take all necessary steps to achieve the desired status. For example, packages could be installed or uninstalled, services configured and started. Puppet corrects any changes made by users or local software. Usually – also in the case with Finesse Bank – the orcharhino assumes the role of the puppet master. Thus the reports of the individual Puppet Agents automatically end up there. For each host, you can see in detail what changes have been made. If there was an error during the run, the host is marked accordingly in the overview.

Value and Vision uses Ansible for configuration management. With Ansible, sequential processes can be better mapped and it can also be used to avoid configuration drifts, but this is not so important for Value and Vision, because after handover, the customers themselves are responsible for their machines. In orcharhino the hosts are linked to Ansible roles for this purpose. After installation, these are automatically executed once and install certain packages, make configurations or execute certain commands.

At the University of Quality, the managed system is altogether more heterogeneous, since in the past the institutes were responsible for their own infrastructure. orcharhino now offers them a common interface, but according to the respective LDAP affiliation, the admins only have access to their own systems. Part of this heterogeneity is also the fact that different configuration management tools are used. Some institutions also use a mix. The orcharhino-team has therefore activated the plug-ins for Ansible, Puppet, and SaltStack. The reports show that all three are in use.

The University of Quality now uses the orcharhino (see ‘How does orcharhino arrive at the customer?’) for the entire campus. Originally a pure project for the administration of central services, almost all institutes are now involved. Thus a collection of tools and approaches has been concentrated in one tool. In the course of this process, it has become apparent that it is important for the individual institutions involved (department chairs, administrative units) to be able to continue to map the original separation between institutes and departments.

Mechanisms to map this separation in orcharhino are primarily known as organizations and locations. As organizations, for example, there are universities for the central services and the department infrastructure as well as administration and the hospital. The locations are one level below. The administration does not make any further distinctions here; the hospital has opted for a division into infrastructure and teaching operations. For the organization university, there are locationswith the names of the individual departments, data centers, clusters and media. All objects in orcharhino are assigned an organization and a location. This allows to filter lists on the one hand and to assign rights on the other hand.

For user authentication, orcharhino is connected to the central LDAP server of the university and also to that of the hospital. When a user logs on to the web frontend, the system assigns him or her to a user group. The assignment is made according to the group assignment in LDAP. Department administrators, for example, only see their own computers and can only manage these.

The managed objects are primarily the hosts because this is where the administrators work most, but subnets, for example, can also be restricted in their access permissions. Thus the admins can register newly created hosts only in certain IP ranges.

A class C network is available to each administration unit. Almost all computers receive public IPs, only the computing cluster is located in a private network and there is a separate DMZ for certain servers. The cluster network is highly isolated, similar to a DMZ. Only two nodes are accessible from the university network and can establish connections to the outside: a login node for users and the orcharhino proxy. The latter is a separate instance and belongs to the orcharhino infrastructure. It does not have its own web-GUI but only serves as a link for networks to which orcharhino has no direct access or mirrors repositories if this is recommended to minimize the used bandwidth. It also receives puppet reports and serves as a proxy for Ansible Calls. Services running on the orcharhino proxy are similar to orcharhino, but in a subordinate role. Pulp synchronizes selected repositories from orcharhino – in this case Debian – and supplies the nodes with packages, it is DHCP server and name server and provides via PXE and TFTP the necessary data to install new nodes.

Besides the cluster, there are currently 8 subnets attached to orcharhino. This includes a central network in which the university’s server services are located (intranet, mail server, etc.) and several institute networks if these operate their own central services or small local computing clusters, as in physics or chemistry.

In all networks, a orcharhino proxy runs as a link between the hosts and orcharhino. Since the network load is manageable so far, almost all hosts use orcharhino irectly as packet source, but they have to use their respective orcharhino proxy as HTTP proxy, because there is no direct route – der orcharhino itself hangs in an internal network. Workstations are so far excluded from the central administration by the orcharhino, although this is being considered at least for the Linux pools.

Value and Vision operates a comparable infrastructure but has predominantly private networks that customers access via VPN. The administrators automatically create a separate organization for each customer. Hosts authenticate themselves to orcharhino via SSL certificates and therefore only see allowed repositories. Some customers deposit self-developed software and can thus ensure that no competitor can easily access it.

Finesse-Bank has a central network in the main branch and subnets in the remote locations. The orcharhino proxies there mirror the repositories of orcharhino. The operation of orcharhino is completely in the hands of the infrastructure team, so that no fine granular rights management on user level is necessary. This requires more stringent guidelines for the network. This is actually outside the range that orcharhino manages. However, the hardware can be controlled via Ansible, which in turn can be managed via the orcharhino.

Value and Vision operates several data centers worldwide. Customers typically require computing capacity for certain project phases but do not want to keep it permanently available or need to manage it. Usually, they then order virtual machines, which are made available to them preconfigured. Once they have completed their task, you can simply delete them again. This is why the data centers are primarily equipped with virtualization hosts. In the beginning, these were just VMware ESXi hosts, but more recently there has been an innovation: some datacenters are switching to Proxmox clusters.

The API endpoints and credentials of the virtualization hosts can be entered in orcharhino as “Compute Resources”. This allows the orcharhino-GUI to create, switch on/off and delete virtual machines on these hosts. For VMware, snapshots can also be managed with orcharhino.

So when a Value and Vision employee creates a new host, they select the virtualization host and virtual machine specification in addition to the operating system. After clicking on “Create Host” the VM is automatically created and booted. It then appears as a new host in the list in the web interface – both in orcharhino and the hypervisor.

Other customers manage Value and Vision bare metal servers. They also depend on orcharhino, which takes over configuration management and lifecycle management. Thanks to “Intelligent Platform Management Interface” admins can also control the power settings for these machines via orcharhino.

Recently the admins have also installed the plugins for the cloud providers Amazon Web Services, Google Cloud Engine, and Microsoft Azure. So these are also available as Compute Resources and orcharhino can create instances there. Originally the idea was to be able to absorb peak loads dynamically. Originally the idea was to be able to absorb peak loads dynamically.

Initially, Finesse-Bank only had physical servers in use, but gradually replaced them with virtual machines. The flexibility extended the deployment scenarios and allowed a better-adapted utilization of the data centers. The fact that these can be managed so easily via orcharhino has almost completely displaced physical hosts.

At the University of Quality, on the other hand, it’s almost exclusively about physical servers – too great a concern of the scientists that they will lose CPU cycles to hypervisors that they actually need for their calculations. Only central network services run virtualized. A special feature has recently been established at the Institute for Information Systems. There, a new professor gives courses on IT infrastructure. In the first semester, it was not possible to spontaneously provide enough resources to provide training environments for course participants. In the plans, someone from the data center suggested that cloud instances should be used and that orcharhino was already prepared for this. He demonstrated how to create hosts in the Azure cloud that were set up after installation using Ansible. During the first semester, Azure instances were started for the courses (block courses) and deleted directly afterward – again through orcharhino. This way has been maintained until today and the course content is now also infrastructure management with the orcharhino.

When the University of Quality was founded in the 1970s, there was not yet an IT infrastructure at the individual departments; everything was managed centrally via the data center. In the meantime, however, computers have become cheaper and Windows and Linux administration easier. Knowledge and financial resources no longer accumulated in the data center and the IT landscape fragmented. With the emergence of configuration management tools, a discourse about a renewed consolidation started. Over time, isolated solutions established themselves despite efforts to standardize the infrastructure. Then the data center began to set up the so-called Gondor cluster and thus offer HPC resources centrally. In the course of this, various management tools were evaluated. The University of Quality decided to use orcharhino. ATIX was assigned to build a proof of concept environment. For this an ATIX consultant installed an orcharhino in the quality university computer center including orcharhino proxy in the cluster network, prepared it to provision Debian hosts. In addition, he included some ready-made puppet modules that can be used to add hosts to the cluster queue system and install scientific packages. These tasks on-site extended over a week, during which the computer center employees had the opportunity to look at individual points in the same way again and again.

After the set-up week, the quality university had its first orcharhino and could start to build up the cluster completely. Its active test phase began soon afterward. The PoC quickly became a productive system and it was decided to add further systems to orcharhino. First other central servers, which also ran on Debian Linux. Then it quickly became clear that most isolated applications could also be integrated without much effort. The biggest hurdle was the different distributions or even operating systems in use. For this, the University of Quality agreed with ATIX on an on-site training to let more employees work with orcharhino. Furthermore, an ATIX consultant should come regularly to the University of Quality for the next months to support the migration process there. First, a central Git server was built to house the definitions for the configuration management, package sources for the different distributions were synchronized and based on this, content views and activation keys were created. The central user administration was integrated and finally, a use for a compute resource from the cloud was found, as a new professor used training infrastructure in Azure for a course.

The visits of the ATIX consultant would be more frequent at the beginning of course – two days every two weeks – and went back to once per month towards the end when the computer center employees became more familiar with the system.

For the basic installation of the productive systems at Value and Vision and Finesse Bank, the companies got support from ATIX. However, both had previously conducted tests with orcharhino on their own initiative. There is a test license for this. Incidentally, the proceeds from this will go to the “Save the rhino” conservation project. After the system was found suitable, Finesse Bank ordered someone to install the productive system. Value and Vision have no true central location, so the installation was done via remote sessions. As a result, the whole process took a little longer, but was much more flexible.