Raspberry pi cluster homelab. In this tutorial, we will show you how to build your own Raspberry Pi Cluster Computer. Feb 9, 2023 - Apr 10, 2023 (60 days) Ivan Kuleshov is raising funds for Compute Blade: Your rack-mountable ARM cluster on Kickstarter! Feature-rich enterprise-level Raspberry Pi Compute Module 4 carrier board by Uptime Lab for large companies or home labs. Sep 20, 2020 · Great! Now we have to do the same on our second cluster node. When you try the Argo CD CLI command again, it will most likely fail cause k3s-rpi. 1 LB, 2 Dev/Prod Node, and 4 Prod Nodes. 5 hexa screws and maybe an acrylic base, on Amazon. An entire Raspberry Pi with 4x cores running at 1. If you prefer, you can use Etcher to do this. On the chosen one, run the following command: This command will generate a connection string in the form of . I'd like to power it using my Teltonika PoE switch, so I bought some original Raspberry Pi PoE+ hats. Basic architecture is: 4gb and 8gb PI4 Bs. Want to see the entire build? Check it out here Sep 27, 2021 · Step 5. 1. Welcome to your friendly /r/homelab, where techies and sysadmin from everywhere are welcome to share their labs, projects, builds, etc. Still have not installex proxmox on the two Mac Minis. A bare-metal cluster, such as a Kubernetes cluster installed on Raspberry Pis for a private-cloud homelab, or really any cluster deployed outside a public cloud and lacking expensive professional hardware, needs another solution. Currently, it has Raspberry Pi 3B+s, but will be upgraded to RPi 4s 8gig of ram. Updated May 7, 2024. And if you google pi cluster case I am sure google will help you. Check out the Mini PiRack project. Aug 23, 2020 · To fix this, edit your kubeconfig and replace the endpoint https://192. Raspberry Pi K8s cluster homelab Hey guys, I want to get your opinions and maybe some answers to questions I have about what I want to do with my homelab. After upgrading my Raspberry Pi cluster storage by attaching SSD drives I migrated my Kubernetes persistent storage from the default Rancher Local Storage provisioner that comes with K3s to the new Rancher Longhorn storage provisioner. Certificate manager dynamically generates certificates from the root certificate when pods or ingresses request a certificate upon creation. 2+GHz is about $30-something, and if you bulk-buy including SD cards and multi-port hubs you can defray the per-unit cost even further; and it's cute, and it's quiet, and if you want you can split it into chunks and distribute them around the country to make a geographically-distributed Horcrux cluster: https://raw. So if you use 4x Raspberry Pi 4 1GB, you’ll get 16 cores Sep 30, 2022 · OctoPrint Live Stream. Advertisement. Prometheus/Grafana/Alert Manager for metrics alerting. Save. And I had not the willpower, nor the tool power, to cut him in half. 18. I have been working professionally in technology most of my life and have done fairly well without a I added a Raspberry Pi cluster with a 12 port PoE injector for a Kubernates teaching lab. This exercise will install a Kubernetes 1. Raspberry Pi vs Intel NUC for a Low-Power, Low-Noise Homelab opinions. But so far I haven't been able to get it up and running, so I'm experimenting with OpenFaaS. The PoE injector will run the Pis and my access points. Part 2 focuses on monitoring the cluster for disk space, CPU usage and network I/O. MotionEye: Manages IP cameras flashed with RSTP firmware. Adding a node. Pi-Hole: Network-level ad-blocking; moved from a Raspberry Pi to Proxmox. Now, you need to run the command from another Pi you want to add to the cluster: merocle. Hardware Choices and Initial Configuration. It comes completely disassembled - there's the metal frame, 4x buttons, 4x POE hats, 4x HDMI boards, 4x OLED screens, 4x of a few other bits, and a bag of screws and nuts. So that brings us to the MOD. •. The upside is you can build a ceph node for ~$75. I need the latest version of OpenStack that will run in my distribution because I don't think the latest versions have an AArch64 image. Traefik for LB and Ingress handling. Boot the Raspberry Pis. I posted to r/Raspberry_Pi, but it dawned on me this morning that there might be interest over here as well. My current homelab consists of one raspberry pi and one ubuntu server (reused my old gaming PC). Way more resources. These Pi's are really not powerful enough to run the Control Plane components, so I've ended up setting them up as worker nodes and running a single master node on a VM on my Mac. 3 drive vis USB and Have seen “open cases” which are basically some m2. It takes 8 raspi 5 to equal just one i5-13500T in Now that you have MicroK8s installed on all boards, pick one is to be the master node of your cluster. Adjust some configuration. So far, I installed all 4 screens and then put 1 Pi in place. Dec 30, 2020 · Introduction At home, I run my own k3s cluster on 4 Raspberry Pi 4Bs. As a self-proclaimed novice in the realm of networking and security, I am seeking advice on which security applications would be beneficial to run on my Raspberry Pi cluster operating with Kubernetes Jan 3, 2022 · Building a Raspberry Pi Cluster using the Turing Pi 2. You can start with different OSes like Ubuntu Server, CentOS, FreeNAS, OPNsense etc. I run a little 3 node Proxmox cluster at home, and I recently decided I'd like to add a simple 3 node Raspberry Pi cluster to it. In this part we build a Kubernetes cluster with Raspberry Pis and deploy OpenFaaS to test it. Each one is a Raspberry Pi 4B+, the 8GB model. Here's my review. To simplify the initial setup, read Modify a disk image to create a Raspberry Pi-based homelab to add a user and SSH authorized_keys to the Ubuntu image before writing it to an SD card and installing on the Raspberry Pi. Using my favorite flat file CMS, Bludit. When I heard that the Raspberry Pi 4 8GB model was launched, I immediately went to go order some. Downsides of a Raspberry pi cluster for home server Hello, I’m a massive noob and want a new project, but need some advice before I start I am wanting to build a home server, it will be a Plex, Nas, host a offline virtual assistant, a Vpn host and I’m sure I will add to this list as time goes on (so need flexibility). Jun 30, 2022 · Run "systemctl reboot" to start a reboot. Even if I didn’t hide the cables just yet. If your cluster has reached its capacity, you'll find it cheaper and more efficient for a lightweight PC. All nodes running Ubuntu OS. 2 cluster on three or more Raspberry Pi 4s running Ubuntu 20. If you're just having fun, Pis are Nov 13, 2019 · Think Raspberry Pi computers are no match for their bigger cousins? Think again, because there’s strength in numbers. 459K views 1 year ago #RaspberryPi #Homelab. Looking for new applications for your Raspberry Pi 4 or the new Raspberry Pi 5? Check our list of 50 cool Raspberry Pi projects that are useful and fun. Part 2: Monitoring. And in fact, that’s almost the only reason why you might want to build a Raspberry Pi Cluster. Now, you can build, run, and learn microservices, containers, and ML apps on Kubernetes using Raspberry Pi 3/3+ compute modules. And with us being geeks, we knew that the next logical bigger number was 1,024 My homelab Kubernetes cluster using bare-metal ARM and x86 nodes. My main requirements are low-power consumption and low noise levels. I created a movie to demonstrate how you can setup a kubernetes homelab in few minutes. While experimenting and learning how routing protocols, VLANs, and spanning-trees work can keep a network engineer busy for hours, at some point you are probably going to want to see some traffic from clients on your network. Nov 5, 2020 · The VM will start after cluster and vCenter are online. Some have an outer shell made to fit a small switch. 168. While browsing this subreddit, I've noticed that many of you have recommended ENTER TO WIN a custom Raspberry Pi (pre-built with K3s): https://bit. 100:6443 with https://k3s-rpi. What I'm hoping to find is something maybe a bit more powerful than a Pi 4 BUT with the same exact dimensions (and preferably mount points) so I can use the existing bracket/tab things that the PI's would mount to. Mar 26, 2021 · Create SD Card for nodes. I updated the roles below to be compatible with the latest Debian release: Debian 12 bookworm. The discussion I had with a colleague what is the better solution, running the multiple RiP's with the extra switch, or a single X86 server running virtual machines. com/TheSudo ☕☕The first part of a series where I document me configuring a Raspberry P To toy around running VMs on your deskop is probably more practical then using a raspberry pi. . The DS14MK2 practically laughed at my naïveté. Apr 13, 2024 · However, it may not be as well-suited for resource-constrained environments like Raspberry Pi due to its higher resource requirements and complexity. I bought this awhile ago with the intention of something like a Docker Swarm/Kubernetes on a bunch of Pi's. Raspberry Pi 3: $ 35. A tiny cluster based on 4x Raspberry Pi Zero 2 W. I am also planning to put a 3U drawer between the Pi cluster and the minis to store any networking related stuff. Aside from the obvious benefits of learning how they work by being hands on with the technology, what are you using your clusters for? Obvious things that come to mind first: -Media servers -Home automation -Crypto mining/stake pools -Self hosted web server. 2/NVME drive. Was planning for k3s. Each node is a Raspberry Pi 4 Model B 2019 4GB Quad Core with a 3KV isolated PoE hat. And with the growing interest in ARM, this might just set you up well for developing in the Oct 12, 2017 · Walk-through — install Kubernetes to your Raspberry Pi in 15 minutes. Monitoring and Observability The first level of monitoring in the home cluster occurs inside of the service mesh. 04 with microk8s. Jun 14, 2021 · The home cluster utilizes a self-signed ECDSA certificate to serve as the root certificate. For me on a Pi once you get enough daemomsets (essentially containers running on ever node) you are 75% saturated before any real workloads go onto it. State of the Homelab - with a new iteration on the Raspberry Pi Cluster. PJ Evans shows you how to create a cluster computer from several Raspberry Pi computers. Apr 11, 2023 · A Raspberry Pi Cluster Computer is a set of interconnected Raspberry Pi devices that work together as a single unit to perform a task. Today, the Rpi4 is released! With 3x the processing power, potentially 4x the memory and still being $35/ea, we have a serious candidate for building out a reasonably beefy local Kubernetes cluster. And created a few ansible to provision the virtual machines with cloud image with cloud-init and deploy k3s on it. by Anatol Locker. It's been a great learning lab for k8s, and now runs most of my service stack really well. Two fun starter projects that run fantastic on a Raspberry Pi: Pi-Hole - Network-wide ad blocking (as simple as entering your pi's IP address as your router's DNS target) Home Assistant - Home automation and washboarding solution (useful even if you don't have any "smart" devices) Sep 23, 2022 · Share. " New machines can just be plugged in, turned on, and automatically become part of your homelab. Then you configure services you might be interested: DNS, DHCP, NTP, Webservers (Apache, NGINX), file servers (Samba, NFS), LDAP etc. Not only is this gentler on your wallet when it comes to your energy bills, it’s also kinder to the planet. Examples of MPI software include OpenMPI or MPICH. Import the Vagrantfile file to this directory; Sep 23, 2022 · It's time to upgrade to my rackmount Pi cluster!You can buy the UCTRONICS Pi Rack Pro on Amazon: https://amzn. I’ve been pleasantly surprised with how much you can run on these small machines Are you running an OS like Raspberry Pi OS with k8s on top like k3s or microk8s or are you running one of those minimal k8s image/OS? Mind shooting a link, whatever it is? I currently have an old PC and run a 1 node k8s on ubuntu 18. The goal of this first article is to set up a running K3s cluster on multiple Raspberry Pi units, each with its own static IP. With that, we had a clear goal: whatever we do, it had to go beyond 750 nodes. Raspberry Pi Homelab. Both of these permit the programmer to divide a task among a group of networked computers, and collect the results of processing. Use it to evaluate your current setup, or to plan a new build including: hardware options, HA, Ingress, storage and Operating System choices. I've seen quite a few post about Raspberry Pi Clusters and I'm confused as to what the point of owning a cluster. 04. dsb. ssh into your selceted Quorum host and run May 5, 2023 · To get started, open your terminal and run the command below (or check the specific command based on your operating system, which can be found in the Ansible documentation. There are additional MPI implementations available. In addition, because these are NFS volumes, they can support multi-read/multi-write operations, so multiple pods can have the volumes mounted at the same time. Most application code is elixir/phoenix. Join other Consul agents. 99. This is based on Cluster HAT and Raspberry Pi 4 with 3 Gb of Ram (I re-soldered an 8GB module on the Raspberry CM4 earlier, so I installed 3GB chip on the Raspberry Pi Jul 21, 2023 · I use the lightweight Kubernetes K3s on a 3-node Raspberry Pi 4 cluster. homelab. The Homelab in general is still a work in progress, but getting near my planned architecture. Raspberry Pi 3 case: $ 8. Documented everything on my blog and, most importantly, as a set of Aug 9, 2020 · In summary, the following steps are what it takes to end with a fully functional cluster: Prepare SSH keys and SD cards. $60 x 8 is $480 for the boards alone. I can understand that power usage, noise factor, small space usage, and cool factor of all the blinking lights, but other than that I don't see it as a great idea. That's the thing, people build these to look cool, and then don't run anything on them! My 4 pi cluster runs grafana/prometheus metrics, flexget/tranmission downloader, pihole, gogs (git server), plex and homeassistant. Cluster Setup. 0. Grocy: Manages groceries and household items, meal plans, and more. Getting a local cluster of reasonable machines was too expensive. 15. About all Raspberry Pi models have a quad-cores CPU (the only exception are Raspberry Pi 1 and Zero). Join other Nomad clients. It knew — it knew that it could not be split in half. Used for educational purposes to learn about kubernetes, IaC and GitOps. Until then wild things can happen in a vSphere cluster without DNS. ARM hasn't taken over the server and when I did a Pi cluster for fun 3 years ago, the container image ecosystem was; interesting. to/3BDZD5Y May 6, 2024 · Goal. However, I don't know if I should build my own Pi cluster or just buy a NUC and create a couple of VMs on it. Mar 3, 2021 · ☕☕ Buy me a coffee and help support the content https://buymeacoffee. Click on “Write” to create the same SD card. Building a Pi cluster sounds like a neat way to learn kubernetes the hard way but if you want to use k8s as you likely would working for a company that uses a cloud service provider than Id go spin up a GKE cluster with the free $300 credit. Hi, I need to have somewhere to host all my personal projects (websites, backends, databases, custom smart home system etc) and also want to learn how to use Kubernetes. It's time to upgrade to my rackmount Pi cluster! You can buy the UCTRONICS Pi Rack Pro on Amazon: https://amzn. I have finally gotten back to 3D Printing. Then factor in 8x case, x8 power brick, x8 whatever hats you need (i'm assuming you'll use the nvme hat for storage since the microsd slot is not reliable enough for work related or mission critical use cases). Mar 11, 2022 · TACACS Server with tac_plus. My newly acquired equipment is all rack-mounted, and I even found a u1 cluster rack to accommodate my four Raspberry Pi 4s with 4GB of RAM each. Dec 23, 2020 · The aim of this post is to show you how to build a Kubernetes cluster with Raspberry Pi 4s for self-hosting APIs, websites and functions so that you can expose them on the Internet and serve traffic to users. Jan 20, 2023 · Drive upgraded to 256GB M. May 17, 2021 · Philips Hue bridge, Raspberry Pi 4, Ruuvitag wireless sensor, Unifi 8-port managed switch; Unifi UDM-Pro with uplink and downlink; APC 1400VA UPS; I added some Philips Hue lights, just to make it more appealing. If you are more interested in just learning virtualization and the various server/service packages, then a more conventional server is probably more cost effective. And with the growing interest in ARM, this might just set you up well for developing in the Jun 5, 2020 · Create dynamic persistent volumes on a Raspberry Pi Kubernetes cluster with the NFS-client provisioner. Save the planet. My other Pi4s are in use and I didn't want to power them down since my POE switch hasn't Feb 27, 2023 · This is the first in a series of articles about a Kubernetes cluster I built on the RaspberryPi platform for my homelab. Install a Kubernetes cluster on Raspberry Pis. ssh into the second cluster node, node2 and do this: apt update && apt install corosync-qdevice. Hardware: Two or more Raspberry Pi 5 boards (older boards work as well) Stable power supply to each boards; High-quality microSD cards (16GB or higher) Jul 13, 2020 · How Raspberry Pi 4 8GB Spawned this Project. The Pis are small and low power. Mounting plates for the Pis and mounting brackets for the switches are also 3d-printed. 2-to-USB adapter (with UASP support) 1 Raspberry Pi for Homeassistant. python3 -m pip install The goal was to build an affordable, super low power, plug-and-play cluster solution with a focus on performance and Kubernetes ecosystem support. I’ve been pleasantly surprised with how much you can run on these small machines Raspberry pi 1 B OpenVPN server Now I know that I could consolidate the setup to a single RiP4 but I had the hardware and wanted to learn about distributed computing. Samsung 32 GB Micro-SD card: $ 15. A homelab is a self-contained network of computer systems used for testing, experimentation, and learning about technology. A modern 4 core with hyperthreading CPU system will run at a minimal load with the same power consumption as a 5 node pi cluster with a high load. Reason #2: Raspberry Pi network clients. Any ideas on Raspberry Pi heatsinks that work with the Raspberry Pi 4B and the original Raspberry Pi 4 PoE+ hat? Discussion. Feb 9, 2021 · Raspberry pi 4 does indeed support network booting. It is custom built with 3d-printed frame (except for the metal rack posts). Proxmox on a Pi4 (2G) Ubuntu Server (arm64) running on a Pi 4 (4G) Background. I’m using a Netgear GC108P managed PoE switch. Repository containing random notes on building my Raspberry Pi cluster and K3s deployment - sleighzy/raspberry-pi-k3s-homelab Running a local Kubernetes cluster on cheap Raspberry Pi hardware is a great way to gain experience managing and developing on a true cloud technology giant. The whole thing is mounted on a pair of sliding rails to access the back. Bootstrap Nomad server node. Install K3s on Master Pi. I figured it was time to put Proxmox onto the Raspberry Pi 4, so here we go -. Once the image is ready, you need to create the SD card for each node of your cluster: Insert the new SD card into your computer. 8 gigs of RAM and 8 cores is plenty of horsepower for a compute node. 10 Node Raspberry Pi kubernetes cluster. k3s. 104 GB total ram, 104 total cores. ly/2VHhfwtLearn more about SUSE Rancher: https://bit. 3rd Image: Cluster 1. Mar 7, 2019 · MacOS and Linux: mkdir -p kuberverse/kv-cluster-s cd kuberverse/kv-cluster-s Windows: md kuberverse\kv-cluster-s cd kuberverse\kv-cluster-s. githubusercontent. Any suggestions? A Homelab Server on Raspbery Pi 4 with Docker. Members Online Need help building a raspberry pi cluster for learning Kubernetes Beowulf clusters normally run a Unix-like operating system, such as BSD, Linux, or Solaris. So we are looking for a small, energy-saving, inexpensive and configurable hardware solution as DNS server for our homelab. In order to access Tagged with k3s, tailscale, cloudflare, certmanager. Not listed above are a NAS and Firewall (might go over those at a later date). . io | sh - Check the status by running: sudo systemctl In interested in diving into my first raspberry pi cluster build. [root@microshift The absolute bottom end as other people have suggested is a Raspberry Pi 3. For anyone that is thinking of building a small cluster of nodes, the Raspberry Pi ecosystem is awesome. With MicroShift and cri-o installed, I next rebooted the Raspberry Pi 4 and it booted into the new OSTree with the packages installed. I have 3 of the 4 8 gigs, but need to order the other 2. k3s from Rancher to simplify building on ARM. First of eight clusters each slated to have 26 nodes. Octoprint: To control my Prusa MK3S 3D printer. External SSD connected with M. May 25, 2020 · With Cloud-init, there is no need to add custom data to the image. They tend to be fairly inexpensive, small, and low power which makes integration extremely easy. Contribute to akkupy/Homelab development by creating an account on GitHub. Dec 2, 2020 · Whether to use Ubuntu 64-bit or CentOS 64-bit for Raspberry Pi to boot headless; Raspberry Pi OS will not suffice, even as a Debian derivative, because there are no OpenStack packages for it. The latest version 1. Jun 5, 2020 · The NFS-client provisioner gives the benefit of dynamically provisioned volumes and makes a homelab cluster behave similarly to a Kubernetes cluster in the cloud. MagicMirror: Moved my smart mirror setup to Proxmox for centralized IoT management. Currently running a 6 node NGINX cluster. Of course this means I need another live stream! This live stream is utilizing a Wyze Pan Cam with the RTSP firmware. I had 5 3B+s laying around, so I put those in. Part 3: OpenFaaS. Configuration and deployment of services automated with Cloud-init, Ansible and ArgoCD. com So, you have to understand what you're trying to accomplish - if you really want to learn clustering/K8S, then the Pi 4B cluster is potentially attractive. Now running Proxmox on a Raspberry Pi 4 cluster. dev on any device I have connected to my Tailscale network and access my own password manager instance. On the left you can see a server, that isn’t rack mountable. Bootstrap Consul server node. You get 7 compute nodes interconnected with Ethernet. Dec 30, 2020 · This allows me to go to https://bitwarden. The last idea is want to explore is to have a huge amount of CPU cores with a cluster. Monitor your Raspberry Pi cluster with Prometheus. The rest of the application code is typically Go or Python, but other experimental I use my collection of raspberry pi devices for environmental monitoring (humidity and temperature) throughout my house and outside, as well as running monitoring processes that watch my VM infrastructure, and UPS health. Another option is raspberrypi compute module 4 cluster with a daughter board or base board, some have switch chip on the Feb 24, 2020 · The largest Raspberry Pi cluster that we could find was done by the Los Alamos National Laboratory's High-Performance Computing Division with a skyrocketing 750-node Raspberry Pi cluster. I built this cheap 10-inch rack to put my tiny homelab into a dedicated space. With the above hardware (excluding Raspberry Pi) I’ve created a ProxmoxVE HA cluster with a CEPH storage cluster built on There are more cost effective things to do. Down the road, I might want to make a single-board computers multi-node cluster like you did. I use Restreamer to push to YouTube Live. Requirements. Plex runs in a container on my 2018 Mac mini, and I run Tautulli to export Plex Metrics, and Varken to push those plus Sonarr, Radars, and Ombi stats into Influx which Grafana then displays. Initial version of the cluster using only Raspberry PI 4B nodes. I'd like to add heatsinks to the Raspberry Pis. to/3BDZD5YOther products mentioned in the video It can be installed on any Kubernetes cluster with Helm, with kubectl, or with the Rancher UI. This cluster is using 7 Raspberry Pi 4B’s with 8gb of RAM each for a total of 56gb of RAM. 0 Mar 26, 2024 · Nurgaliyev Shakhizat took three Raspberry Pi 5s and smashed (technical term) them all together to create a magical Ceph cluster. Step 1: First, I had to pop the rivets holding the innards in place. It sucks when your VM storage node goes down, and you lose access to all your tools servers. Once it is enabled in your images, your virtual machines, physical servers, even tiny Raspberry Pis can behave like cloud instances in your own "private cloud at home. This stream will run 24/7. In Win32 Disk Imager, select the image filename and the device letter. inlets will not resolve on your local machine. Sounds like the Swiss-Army knive, but it can be easily realized with a Raspberry Pi. Now run the following on your Raspberry Pi that you will utilize as the Master: curl -sfL https://get. inlets:6443 which is the cluster internal endpoint for our private cluster. Lets say you have 10 RPis which will cost around $400 Spent the last weekend trying to setup an HA Kubernetes cluster on three of my Pi Zero 2 Ws. ly/3rb8Ih0Need help with Kubernetes I'm building a Raspberry Pi 4B cluster for learning Kubernetes (on bare metal). My k3s cluster consists of four nodes. Test the Nomad cluster. ssh into your selceted Quorum host and run Raspberry Pi offers computers to fit a huge range of needs and applications of all shapes and sizes. At 15 watts, Raspberry Pi uses just a fraction of the power draw of traditional PCs. May 7, 2024 · 50 Cool Raspberry Pi Projects for May 2024. For high performance applications you can hook up a M. Yet another Raspberry Pi 4 Cluster. This lowers the price tag of a homelab but also means you can test and build on the Raspberry Pi’s ARM architecture and x86, at the same time. So I did the next best thing: I gutted the little m@#&f#&$#ker like a fish. Getting a Kubernetes cluster in Azure, AWS or Google Cloud was too expensive. 00. Save money. until you cluster. Oct 9, 2020 · The LXD appliance targets the Raspberry Pi 4 and Intel NUCs and supports mixed architecture deployments. Want to see the entire build? Building a Raspberry Pi Cluster using the Turing Pi 2. Perfect, now we have to also configure our third "fake" cluster node, in my case a 24/7 running raspberry pi, thats taking the place of node3. This is a very quick way to get a lab capable of running both contai Jul 1, 2020 · Kubernetes does not have a built-in network load-balancer implementation. For this Let's take 3 Raspberry Pi 4 (8GB) and assemble a home lab environment using LXD cluster. Hello everyone, I'm planning to create a homelab to experiment with Kubernetes, Jenkins jobs, and some IoT apps I've developed. bc cd eg cy nl xx jj cm xx fl