Table of Contents
Micro Cloud (Hybrid) Architecture DRAFT
Disposable HW and SW architecture requied! Vendor-lock-in may be kept at minimum levels. SDDC (Software-Defined Data Center): A data storage facility where networking, storage, CPU and security are virtualized and delivered as a service. Stable, cheap, good enough.
| Tier | Description |
|---|---|
| I | A Tier 1 data center is a basic server room implementing the general guidelines for computer system installations. This first level runs within a 99.671 percent availability through one non-redundant distribution path with non-redundant capacity components. |
| II | A Tier 2 data center Includes all requirements of Tier 1, plus a guarantee of 99.741 percent availability with redundant site infrastructure capacity components. |
| III | In addition to fulfilling requirements of Tiers 1 and 2, Tier 3 data centers provide dual-powered IT equipment to receive data from multiple independent distribution paths with an increased availability of 99.982 percent guaranteed. |
| IV | Tier 4 data centers include the components of the first three Tiers with the addition of independently dual-powered cooling equipment. The site infrastructure is fault-tolerant with distribution capability and the capacity to store electrical power. An availability of 99.995 percent is guaranteed. |
Basic Topology Important parts
| Command and conquer | Openstack Horizon, Proxmox Datacenter Manager, Nutanix Prism etc. |
| Compute Nodes | KVM |
| Storage Nodes | File, Block, Objects |
AWS Basic knowledge (Introduction by Chetan)
This is My Architecture playlist from AWS
Schematics
Basic Topology
Firewalls are templates linked to VPS instance
Integration Schema
Service Flow
CEPH SD Storage Schema
Ceph Storage at CERN (IT Department)
Exabyte storage scaleout with Geocluster and autoscaling
© 2015 Jian Zhang Intel CEPH software optimizations for cloud workloads
Replication HA schema
- root
- ↘ datacenter
- ↘↘ row
- ↘↘↘ rack
- ↘↘↘↘ host
- ↘↘↘↘↘ osd
AZ recommendation
| EBS | 1x AZ |
|---|---|
| EFS | ~3x AZ (Full region) |
| S3 | ~3x AZ (Full region) |
HA granularity
| By service | No HA | Single AZ | MultiAZ |
|---|---|---|---|
| VPS | Yes | No | No |
| CephFS | No | Yes | may |
| Blockstorage - CEPH | No | Yes | may |
| Blockstorage - Minio | Yes | may | No |
| Blockstorage - Garage | Yes | may | No |
| GalleraCluster | No | Yes | may |
| ScyllaDB | No | Yes | may |
| MariadDB | Yes | Yes | may |
| Postgresql | Yes | Yes | may |
| Citus | No | Yes | may |
Software Stack
Cloud Core Infrastructure (IaaS)
| Type | Cloud Service Description | CPU support |
|---|---|---|
| Compute Nodes and Management | AWS EC2 replacement with cloudinit support | ARM support |
| Openstack | Full, but as LEGO style | YES |
| Proxmox | Light, but EASY to implement | YES |
| – LXC may deploy container templates – both have native CEPH support | ||
| SD Storage (Object, Block, File) Scaleout | AWS S3 compatible and EBS replacement | |
| AWS EBS - CEPH SD Block Storage | Hyperconverged or PetaSAN | YES |
| AWS EFS - CEPH SD File Storage | Hyperconverged or PetaSAN | YES |
| SD Load Balancer, Cache | AWS ELB / Citrix Netscaler replacement | |
| NGINX | or AlibabaCloud clone | YES |
| Integration + Enterprise Service Bus | IBM WebSphere replacement | |
| WSO2 Integrator | YES | |
| Mass webhosting | ||
| ISP Config | YES | |
| MQTT Platform | AWS IoT core replacement | |
| Mosquitto | YES | |
| Cloud Data Synchronization / Backup | ||
| Rclone | YES | |
| Restic | YES | |
| Infrastructure Monitoring System | AWS CloudWatch replacement | |
| NetXMS | agent only | |
| Grafana | YES | |
| Kubernetes Orchestrator | AWS EKS replacement | |
| Mikro K8s | ||
| Microsoft Windows | Windows Infrastructure | $$$ not recommended |
| 1x Datacenter - dedicated HW | Xx Server Standard on KVM | no/LIMITED |
| XXX users | CAL (client access licence) per user | |
| Cloudinit for Windows | ||
Deployment
| Ansible |
|---|
| Terraform |
| Juju |
Security as a Service (SECaaS)
| Type | Cloud Service Description | CPU support |
|---|---|---|
| WAF (Application Firewall) | ||
| OWASP® ModSecurity | +nginx connector | YES |
| SD Firewall | AWS VPC, Security groups replacement | |
| Integrated firewall with templates | inside hypervisor | YES |
| pfSense | +platform integrated (VPS sandboxing) | LIMITED |
Software as a Service (SaaS)
| Type | Info | Service Description | Location |
|---|---|---|---|
| Services and Management | API catalog | WSO2 API Manager | KVM or LXC |
| DB | PostgreSQL | CitusData (Sharding Cluster) | KVM or LXC |
| DB | PostgreSQL | Postgresql | KVM or LXC |
| DB | PostgreSQL | Neon | KVM or LXC |
| DB | MariaDB | MariaDB with GalleraCluster | KVM or LXC |
| DB | AWS DynamoDB replacement | ScyllaDB Alternator | KVM or LXC |
| Storage | AWS S3 | Ceph version | CEPH Cluster |
| Storage | AWS S3 | Garage | Garage |
| Storage | AWS S3 | Minio version | Minio Cluster |
Platform as a Service (PaaS)
| Type | Name | Service Description | Location |
|---|---|---|---|
| Web Publishing | Info | Wordpress | ISPConfig |
| Web Storage | Network Drive | NextCloud * can use AWS S3 | ISPConfig, LXC or KVM |
| Videoconferencing | Talk | Talk * WebRTC | Inside Nextcloud + coTurn Server |
| Bug Tracking | Bug Tracking | Mantis Bug Tracker | ISPConfig |
| Forum | Discussion Forum | VanillaForums | ISPConfig |
| Knowledge Base | Wikipedia | DokuWiki | ISPConfig |
* WebRTC is decentralized protocol it may needs some help from CDN
Cloud-Init
KVM - Virtual drive with configuration
LXC - scripts for distribution executed directly in container
API + CLI
Multiuser setups
You may use pools inside Proxmox to separate users and namespaces inside Kubernetes
Kubernetes (MicroK8s)
Hardware Draft
| Compute nodes |
|---|
| AMD Epyc |
| EBS - SD Storage |
| Ampere / ARM 4x core+ |
Security
Biometric Readers + card
Man-Trap
Cage or room with Biometric Readers + card
Zero-Trust-Security
Every microservice must not trust to anyone else
Main-Distribution-Frame (MDF)
Incoming and outgoing communication and power lines within a building
A panel or set of panels where all communication cables from different parts of the building are both terminated and connected.
Cage or room with Biometric Readers + card
Intermediate Distribution Frame (IDF)
A wall-mounted or free-standing rack used to manage and interconnect a telecommunications cable between end-user devices and the main distribution frame (MDF).
Meet-Me-Room (MMR)
Network interchange place
Cage or room with Biometric Readers + card
Carrier Hotel
Internet exchange points for an area.
East-West Traffic
From server to server within a network's data center
North-South Traffic
Data flowing from or to a system physically residing outside the data center
CDN
Content Delivery Network with NGINX or TENGINE
Hardware suppliers & Datacenter vendors
Reuse what's possible and make sense
Custom boards from 1000+ pcs
| Hardware | Datacenter builders |
|---|---|
| ASRock Rack | Equinix |
| Gigabyte Enterprise | Digital Realty |
| Ingrasys | Cologix |
| Wiwynn | Aligned |
| Quanta Cloud Technology | DataBank |
| Tyan | NTT Data |
| Inventec | Digital Edge |
| ZT systems | EdgeConneX |
| Supermicro | |
| Aivres |
Remarks
Add Storage type
Backup
Storage CEPH
Replication schema
CEPH One SSD backbone network connnectivity
CEPH Pools (replication config)
Create VM
Running VM
Firewall Template (SECaaS)
MultiCluster Proxmox Dashboard
MultiCluster Kubernetes Dashboard
Proxmox on RaspberryPI 5 / with Win11 ARM inside
Proxmox direct import from VMWare ESXi
AWS Load Balancer
AWS S3 (replicated across AZs within location), S3 Express One Zone (faster, cheaper only one zone)
Software stack overview
Sorted by my recommendation
| Openstack | Pros | Cons |
|---|---|---|
| OpenSource,cost-effective | Hard to setup | |
| Modular design | Modular design | |
| Ultimate solution for very large clouds | Higher maintenance costs | |
| NASA Hybrid Cloud Amazon AWS and OpenStack, SEZNAM.cz | ||
| AT&T runs its mobile core network on an OpenStack cloud, serving millions of subscribers. | ||
| China Mobile, one of the world’s largest telecom providers, is building the biggest NFV network based on OpenStack with over 50,000 servers. | ||
| OpenStack is more alive than ever with 40 million cores in production and over 300 public cloud data centers worldwide. | ||
| Proxmox | Pros | Cons |
| OpenSource,cost-effective | Scalability issues for very large environments | |
| Easy to setup | approx. 32 servers in group more needs MultiCluster manager | |
| Multiple HV KVM, LXC | Limited enterprise support | |
| LXC is very lightweight | Upgrade on OS level | |
| Integrated SD storage, SD firewall (like AWS sg) | ||
| CEPH and ZFS implemented, IPAM support | ||
| Can consume CEPH from a dedicated cluster | ||
| Open vSwitch alternative to vSphere Distributed Switch | ||
| SDN Network alternative to NSX-T Data Center | ||
| OpenNebula | Pros | Cons |
| Multiple HV KVM, LXC, vCenter | ||
| Kubernetes included | ||
| Nutanix AHV | Pros | Cons |
| Kubernetes Support | Vendor-lock-in (Closed opensource products) | |
| Integrated storage and networking solutions | Messy Architecture | |
| Rocky Linux, Cassandra | No external storage support | |
| Microsoft Hyper-V | Pros | Cons |
| Easy to setup | Cost, limited functionality | |
| Additional licensing costs! | ||
| Vendor-lock-in | ||
| VMWare | Pros | Cons |
| Easy to setup | Higher cost | |
| Additional licensing costs! | ||
| Vendor-lock-in | ||
| CEPH | Pros | Cons |
|---|---|---|
| No single point of failure | Complex | |
| Data durability via replication or erasure coding | minimum 8nodes+ recommended | |
| No interruption of service from rolling upgrades, online expansion, etc. | QoS on pool | |
| A single cluster can serve object, block, and file | ||
| Compatibility with Openstack, S3, K8s, Proxmox | ||
| Notable known references | ||
| Approx 3500 clusters worldwide with more than 1.5EB capacity | ||
| CERN storage services and its support of experiments, 37000 users 5100 projects in data centers. | ||
| SAMSUNG | CHINA MOBILE | DigitalOcean |
| My experience from 7.3.2018 in production 18TB MLC AllFlash | ||
Competitors
Dell Isilon
Dell Powerflex
You may connect storage nodes with one network card and compute nodes with two (in LACP mode)
Openstack
MINIO S3 service for small networks
CEPH 3node 10gbit performance (KVM-VirtIO)
3 servers with 3xSSD (9 OSD)
Block Size | 4k (IOPS) | 64k (IOPS)
------ | --- ---- | ---- ----
Read | 49.45 MB/s (12.3k) | 220.80 MB/s (3.4k)
Write | 49.54 MB/s (12.3k) | 221.96 MB/s (3.4k)
Total | 98.99 MB/s (24.7k) | 442.76 MB/s (6.9k)
| |
Block Size | 512k (IOPS) | 1m (IOPS)
------ | --- ---- | ---- ----
Read | 597.15 MB/s (1.1k) | 663.25 MB/s (647)
Write | 628.88 MB/s (1.2k) | 707.43 MB/s (690)
Total | 1.22 GB/s (2.3k) | 1.37 GB/s (1.3k)