Mirroring IoT Data from Raspberry Pi to Local Cluster
This advanced tutorial requires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes.
Raspberry Pi to Local Cluster
This section will use Raspberry Pi v3 running Ubuntu 32-bit as the edge device and our local machine for the home cluster. Let's start with installing and configuring the home cluster.
Install Home Cluster on Local Machine
Installing the home cluster on Linux or Mac requires Kubernetes. Use the following instructions to set up Kubernetes on your local machine.
Install Kubernetes, then use the instructions below to install the experimental fluvio binaries.
Create a new directory
Create a clean directory for the configuration and metadata files:
mkdir -p local/projects/mirror; cd local/projects/mirror
Download fluvio binary
Use curl to download and install:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Make sure to add .fluvio/bin to the $PATHas specified in the installation script.
Start home cluster
Use the fluvio binary to start the cluster:
fluvio cluster start --k8
Check the result with:
fluvio cluster status
Register Edge clusters
Use the remote CLI to register the edge clusters (edge1 and edge2) with the upstream cluster:
Edge 1:
fluvio remote register edge1
Edge 2:
fluvio remote register edge2
Create the mirror topic
Mirror topics on the upstream clusters has multiple partitions, where each partition has a 1-to-1 relationship with the edge cluster.
Create a partition assignment file to define the edge devices:
echo '[
"edge1", "edge2"
]' > assignment_file.json
Apply the configuration file to create the topic:
fluvio topic create edge-topic --mirror-apply assignment_file.json
List partitions to check the assignment:
fluvio partition list
It should display all partitions:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 edge1 [] Online 0 B 0 0 0 0 []
edge-topic 1 5001 edge2 [] Online 0 B 0 0 0 0 []
List remote clusters to check their status:
fluvio remote list
It should show the following:
REMOTE SC STATUS SPU STATUS LAST SEEN ERRORS
edge1 Waiting Waiting - -
edge1 Waiting Waiting - -
Generate Metadata for Edge Clusters
Each edge cluster requires a unique metadata file that gives the edge cluster the information to connect to the upstream cluster and the topic/mirror where the data is synchronized.
Generate a metadata file for each cluster:
Edge 1:
The home edge device is a Virtual Machine emulating an IoT device:
fluvio remote export edge1 --public-endpoint host.orb.internal --file edge1.json
Edge 2:
The home edge device is a Raspberry Pi device. You may skip this if you don't have such a device.
The IP address of our machine where the upstream server is running is 192.168.79.252. Please identify your own IP address and replace it in the command below.
fluvio remote export edge1 --public-endpoint 192.168.79.252 --file edge2.json
We'll transfer these files to edge devices in the following sections.
Install Edge Cluster on Raspberry Pi (optional)
We'll use the same procedure as before to mirror from Raspberry Pi to the same upstream cluster. The test below was performed on a Raspberry Pi v3 running Ubuntu 32-bit image.
Download metadata file
We'll use the metadata file edge2.json that we've exported above to provision this device.
Identify the IP address of your Raspberry Pi device and it replace below
Using the upstream terminal, let's scp the edge2.json file to the edge device:
scp edge2.json fluvio@192.168.79.139:~
Login into the edge device
Spawn a new terminal and login into the Raspberry Pi:
ssh fluvio@192.168.79.139
Download fluvio binaries
On the raspberry pi, run the following command:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Run fluvio version to double check.
Start cluster
First we will start the cluster:
fluvio cluster start
Then, we'll use the metadata file on the Raspberry Pi to connect:
fluvio home connect --file edge2.json
Let's check the partitions:
fluvio partition list
The edge device should show the following partition::
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
edge-topic 0 5001 home_name:0:public_endpoint [] Online 0 B 11 11 11 0 []
Test 1: Mirror from Raspberry Pi Edge to Upstream
Let's produce on the Raspberry Pi and consume from the upstream cluster.
Produce to edge cluster
Produce on the pi terminal:
fluvio produce edge-topic
> A
Ok!
> B
Ok!
Consume from upstream
Consume on the upstream terminal:
fluvio consume edge-topic --mirror edge2 -B
A
B
Mirror test is successful.
Test 2: Upstream Cluster Offline
Shutdown the upstream cluster and check that the edge cluster can continue receiving records. Then, resume the upstream cluster and ensure the data is synchronized and can consumed on both sides.
Shutdown upstream cluster
On the upstream terminal, shutdown the cluster:
fluvio cluster shutdown --local
kubectl delete spu custom-spu-5001
Ensure the cluster is shutdown:
fluvio cluster status
On edge cluster
Produce a few more records on the pi terminal:
fluvio produce edge-topic
C
D
E
Reconnect upstream cluster & consume from topic
On the upstream terminal, restart the cluster:
fluvio cluster upgrade --local
The topic on the upstream cluster should automatically synchronize with the edge cluster.
Wait for the connection retry interval to trigger for the new records to arrive.
Let's consume:
fluvio consume edge-topic --mirror edge2 -B
A
B
C
D
E
The disconnect test was successful.
Test 3: Edge Cluster Offline
This test ensures that the edge cluster will not lose data following a power loss.
Restart edge cluster
On the edge terminal, shutdown the cluster:
fluvio cluster shutdown --local
Restart the cluster:
fluvio cluster upgrade
Consume from edge cluster
First, on the pi terminal, check the status of the target cluster:
fluvio cluster home
HOME ROUTE STATUS SC STATUS SPU STATUS LAST SEEN ERRORS
home localhost:30003 Online Connected Connected 1s -
Then, consume from the edge cluster:
fluvio consume edge-topic -B
A
B
C
D
E
Produce records and observe that the mirror will resume the synchronization.
🎉 Congratulations! You have successfully tested edge mirroring using Raspberry Pi. It is now time to roll it out in your environment.