Whats the need ?

Developers love OpenShift ❤️
Developers need local OpenShift environment (Solution : CRC (CodeReady Containers) ) 💻
Developers build Applications, that need Block/File/Object storage 🔨
ODF provides Block/File/Object storage to OpenShift 👌
ODF Nano deploys ODF on CRC 🙌 👏

Introducing ODF-Nano

tldr; Watch introduction & Demo Video here

ODF-Nano lets you deploy OpenShift Data Foundation on your Laptop (CRC).

For dev/test experimentation developers ofter need persistent storage with CRC.
Currently CRC lacks clean & simple persistent storage solution, ODF-Nano solves this problem for CRC.
Devs can now develop/test their apps locally using CRC+ODF-Nano. Once the app is ready, it could be deployed in production OCP+ODF without any change

Setup

Prerequisites

Host OS

RHEL 8,
Fedora 34 (tested) [ Feel free to test with other releases ]
Ubuntu 20.04 (tested) [ Feel free to test with other releases ]
MacOS ( Need more tests )

CRC / OC Binaries

Download CRC and OC binaries from [cloud.redhat.com]((https://cloud.redhat.com/openshift/create/local)
Create CRC directlry mkdir ~/.crc
Also get CRC pull secret from [cloud.redhat.com]((https://cloud.redhat.com/openshift/create/local) and save it as ~/.crc/pull-secret.txt

Step -1 :: Deploy CRC - Linux

Watch Demo Video here

Note : If you have already deployed CRC using OpenSpot project, you can skip step-1 and move directly to step-2

mkdir ~/.crc
cd ~/.crc
# Get CRC pull secret from [cloud.redhat.com]((https://cloud.redhat.com/openshift/create/local) and save it as `~/.crc/pull-secret.txt`
crc config set consent-telemetry no
crc config set enable-cluster-monitoring true # Enable only if you have enough memory, needs ~4G extra
crc config set cpus 15 #Change as per your HW config
crc config set memory 60000 #Change as per your HW config
crc config set pull-secret-file ~/.crc/pull-secret.txt
crc config view
crc setup
alias crcssh='ssh -i ~/.crc/machines/crc/id_ecdsa core@"$(crc ip)"'
crc start
crcssh uptime
crc console --credentials  > crc-creds.txt

Access https://console-openshift-console.apps-crc.testing from client machine

Step -2 :: Deploy ODF-Nano on CRC - Linux

Prerequisites

SSH into the host machine running CRC VM
Create a few raw devices that ODF-Nano will use

## Don't worry this is thin provisioned
sudo -S qemu-img create -f raw ~/.crc/vdb 100G
sudo -S qemu-img create -f raw ~/.crc/vdc 100G

Attach these devices to CRC VM

crc stop
sudo virsh list --all
sudo virsh dumpxml crc > ~/crc.xml
vim ~/crc.xml

Add the following section to crc.xml
Make sure to set the correct disk path

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='~/.crc/vdb' index='1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='~/.crc/vdc' index='2'/>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>

Apply XML file and start CRC

sed -i "s|~|$HOME|g" ~/crc.xml
sudo virsh define ~/crc.xml
crc start

List devices to verify

crcssh lsblk

Step -1 :: Deploy CRC - MACOS

Watch Demo Video here

mkdir ~/.crc
cd ~/.crc
# Get CRC pull secret from [cloud.redhat.com]((https://cloud.redhat.com/openshift/create/local) and save it as `~/.crc/pull-secret.txt`
crc config set consent-telemetry no
crc config set enable-cluster-monitoring true # Enable only if you have enough memory, needs ~4G extra
crc config set cpus 9 #Change as per your HW config
crc config set memory 32768 #Change as per your HW config
crc config set disk-size 250 #Don't worry this is thin provisioned
crc config set pull-secret-file ~/.crc/pull-secret.txt
crc config view
crc setup
alias crcssh='ssh -p 2222 -i ~/.crc/machines/crc/id_ecdsa core@"$(crc ip)"'
crc start
crcssh uptime
crc console --credentials  > crc-creds.txt

Access https://console-openshift-console.apps-crc.testing from client machine

Step -2 :: Deploy ODF-Nano on CRC - MACOS

Prerequisites

SSH into the host machine running CRC VM
Create a few loopback devices that ODF-Nano will use

## Don't worry this is thin provisioned
sudo -i
mkdir -p /var/lib/storage
truncate --size 220G /var/lib/storage/disk1
losetup -P /dev/loop1 /var/lib/storage/disk1
pvcreate /dev/loop1
vgcreate odf /dev/loop1
lvcreate -n disk1 -L 105G odf
lvcreate -n disk2 -L 105G odf

lsblk

cat << EOF > /etc/systemd/system/lvm-odf-losetup.service
[Unit]
Description=LVM ODF loopback device setup
DefaultDependencies=no
Conflicts=umount.target
Requires=lvm2-lvmetad.service systemd-udev-settle.service
Before=local-fs.target umount.target
After=lvm2-lvmetad.service systemd-udev-settle.service
[Service]
Type=oneshot
ExecStart=/sbin/losetup -P /dev/loop1 /var/lib/storage/disk1
ExecStop=/sbin/losetup -d /dev/loop1
RemainAfterExit=yes
[Install]
WantedBy=local-fs-pre.target
EOF

systemctl enable lvm-odf-losetup

Deploy ODF-Nano on CRC

Login to CRC using kubeadmin oc login -u kubeadmin -p <PWD> https://api.crc.testing:6443
Get odf-nano

git clone https://github.com/ksingh7/odf-nano.git
cd odf-nano

Install odf-nano - LINUX

sh deploy_odf.sh

Install odf-nano - MacOS

sh deploy_odf_macos.sh

Sample output

Setting up environment for ODF - this will take a few minutes
subscription.operators.coreos.com/ocs-subscription created
Waiting for operators to be ready
No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
...
.No resources found in openshift-storage namespace.
...............................
Operators are ready now
Finished up preparing the local storage
ODF is installing now, please be patient
ocsinitialization.ocs.openshift.io/ocsinit patched
pod/rook-ceph-tools-7d95854fb8-b78s2 condition met
ODF is installed now

Verify ODF setup

oc get sc

You now have File/Block/Object Persistent Storage Classes from ODF. Deploy and Test your app locally, like you do in production (OCP & ODF)

Miscelleanous

ODF-Nano Resource Footprint & Components

Resource Footprint

CPU	Memory
3 vCPU	2.5G

ODF Components

Component	Count
MON	1
MGR	1
OSD	2
MDS	2
RGW	1
Ceph-Tools	1
ocs-metrics-exporter	1
ocs-operator	1
noobaa-operator	1

** Reducing MDS count to 1 is WIP **

Access CRC from a remote client

By default CRC cluster is reachable from localhost. Inorder to access a CRC cluster remotely, we need to add a proxy layer. This setup is useful, when you want to deploy CRC on a remote machine (Home server or a Cloud bare metal), there has to be a way for you to acces CRC cluster remotely. This procedure help you access your CRC remotely.

Execute on the Host running CRC VM

SERVER_IP=0.0.0.0
CRC_IP=$(crc ip)
sudo cp /etc/haproxy/haproxy.cfg{,.bak}
sudo semanage port -a -t http_port_t -p tcp 6443
sudo tee /etc/haproxy/haproxy.cfg &>/dev/null <<EOF

global
    log /dev/log local0

defaults
    balance roundrobin
    log global
    maxconn 100
    mode tcp
    timeout connect 5s
    timeout client 500s
    timeout server 500s

listen apps
    bind 0.0.0.0:80
    server crcvm $CRC_IP:80 check

listen apps_ssl
    bind 0.0.0.0:443
    server crcvm $CRC_IP:443 check

listen api
    bind 0.0.0.0:6443
    server crcvm $CRC_IP:6443 check

sudo systemctl restart haproxy
sudo systemctl status haproxy

sudo netstat -plunt  | grep -i haproxy

Example output from netstat

$ sudo netstat -plunt  | grep -i haproxy
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      9291/haproxy
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      9291/haproxy
tcp        0      0 0.0.0.0:6443            0.0.0.0:*               LISTEN      9291/haproxy

MACos Client Node

https://www.stevenrombauts.be/2018/01/use-dnsmasq-instead-of-etc-hosts/

brew install dnsmasq
sudo brew services start dnsmasq
mkdir -p /usr/local/etc/dnsmasq.d
touch /usr/local/etc/dnsmasq.d/crc.conf
vim /usr/local/etc/dnsmasq.d/crc.conf

address=/.testing/192.168.1.6

verify dns resolution

sudo brew services restart dnsmasq
dig apps-crc.testing @127.0.0.1
dig console-openshift-console.apps-crc.testing @127.0.0.1

sudo mkdir /etc/resolver
sudo vim /etc/resolver/testing
nameserver 127.0.0.1

scutil --dns

ping -c 1 foo.api.crc.testing
ping -c 1 foo.apps-crc.testing
ping -c 1 console-openshift-console.apps-crc.testing

Uninstall ODF-Nano

bash uninstall_odf.sh

Troubleshooting

If running out of space, create a symlink for .crc

mkdir /mnt/hdd_space1/.crc
ln -s /mnt/hdd_space1/.crc ~/.crc

To ssh into crc vm

ssh -i ~/.crc/machines/crc/id_ecdsa core@"$(crc ip)"

Deep clean previous instance of crc

crc delete -f
crc cleanup
rm -rf ~/.crc/vd* ~/.crc/crc* ~/.crc/bin ~/.crc/machines
sudo virsh list --all
sudo virsh destroy crc
sudo virsh undefine crc
virsh vol-list --pool crc
#virsh pool-destroy crc  # generally you can skip this
# virsh vol-list --pool crc # generally you can skip this

Increase root disk spaced of CRC VM

By defautl CRC vm uses 30G of root disk, you definately need to increase that

crcssh lsblk
# Identify partition name of /sysroot
vda    252:0    0   31G  0 disk
|-vda1 252:1    0    1M  0 part
|-vda2 252:2    0  127M  0 part
|-vda3 252:3    0  384M  0 part /boot
`-vda4 252:4    0 30.5G  0 part /sysroot
vdb    252:16   0  100G  0 disk
vdc    252:32   0  100G  0 disk

crc stop

CRC_MACHINE_IMAGE=${HOME}/.crc/machines/crc/crc.qcow2

# This resize is thin-provisioned
sudo qemu-img resize ${CRC_MACHINE_IMAGE} +20G
sudo cp ${CRC_MACHINE_IMAGE} ${CRC_MACHINE_IMAGE}.ORIGINAL

#increase the /dev/sda4 (known as vda4 in the VM) disk partition size by an additional 20GB
sudo virt-resize --expand /dev/sda4 ${CRC_MACHINE_IMAGE}.ORIGINAL ${CRC_MACHINE_IMAGE}
sudo rm ${CRC_MACHINE_IMAGE}.ORIGINAL
crc start

To-Do

Refer issue#3

Errors while creating odf-nano MVP

There is something in this code block that is causing this

mohit@beast:/mnt/hdd_space1/git/odf-nano$ ./deploy_odf.sh
Setting up environment for ODF - this will take a few minutes
subscription.operators.coreos.com/ocs-subscription created
Waiting for operators to be ready
No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.No resources found in openshift-storage namespace.
.......Operators are ready now
Finished up preparing the local storage
Creating custom ODF configuration for CRC
ODF is installing now, please be patient
ocsinitialization.ocs.openshift.io/ocsinit patched
pod/rook-ceph-tools-7d95854fb8-lpdf2 condition met
Now configuring your ODF cluster
Configure your block environment
Configuring your file environment
error: You must provide one or more resources by argument or filename.
Example resource specifications include:
   '-f rsrc.yaml'
   '--filename=rsrc.json'
   '<resource> <name>'
   '<resource>'
error: You must provide one or more resources by argument or filename.
Example resource specifications include:
   '-f rsrc.yaml'
   '--filename=rsrc.json'
   '<resource> <name>'
   '<resource>'
command terminated with exit code 2
storageclass.storage.k8s.io/ocs-storagecluster-cephfs created
ODF is installed now

error: unable to recognize "STDIN": no matches for kind "VolumeSnapshotClass" in version "snapshot.storage.k8s.io/v1beta1" because we are using ocs 4.6 and not v4.7 ?

This code block is causing this

I had to remove the Limits/Requests block from the OSD, because OSD containers are failing. May be they were not getting enough resources. After removing limits/requests OSDs deployed successfully

ksingh7 / odf-nano Goto Github PK

odf-nano's Introduction

Whats the need ?

Introducing ODF-Nano

Setup

Prerequisites

Host OS

CRC / OC Binaries

Step -1 :: Deploy CRC - Linux

Watch Demo Video here

Step -2 :: Deploy ODF-Nano on CRC - Linux

Prerequisites

Step -1 :: Deploy CRC - MACOS

Watch Demo Video here

Step -2 :: Deploy ODF-Nano on CRC - MACOS

Prerequisites

Deploy ODF-Nano on CRC

Miscelleanous

ODF-Nano Resource Footprint & Components

Access CRC from a remote client

MACos Client Node

Uninstall ODF-Nano

Troubleshooting

To-Do

odf-nano's People

Contributors

Stargazers

Watchers

Forkers

odf-nano's Issues

Recommend Projects

Recommend Topics

Recommend Org