README.md 13.1 KB
Newer Older
1
2
3
4
# Edge-IoT Simulator

The following project contains a small edge-IoT simulator.

Sebastian Böhm's avatar
Sebastian Böhm committed
5
6
## Components and Architecture

7
8
9
10
11
12
13
14
15
Components:

* `core`: IoT device simulators, e.g., `temperature` simulator
* `messaging`: MQTT messaging components to send data to MQTT broker
* `web`: A small flask application which provides access to selected data provided by `core` and `messaging`.

Architecture:

```bash
Sebastian Böhm's avatar
Sebastian Böhm committed
16
web <-> core <-> queue <-> mqtt_client
17
18
```

Sebastian Böhm's avatar
Sebastian Böhm committed
19
* `web`: The web application has a reference to the core services (e.g., temperature) and has currently three endpoints:
20
21
  * `/`: Dashboard which shows the latest temperature value
  * `/temperature`: Returns the latest temperature value as json
Sebastian Böhm's avatar
Sebastian Böhm committed
22
23
  * `/cpu-load`: Returns a UI to manage CPU load jobs
* `core, queue, mqtt_client`: The core services (e.g., `temperature`, `cpu-load`) are decoupled from the `mqtt_client` by a shared memory `threadsafe` queue. The `mqtt_client` forwards the message with a corresponding topic and payload.
24
25
26
27
28
29
30
31
32
33
34
35
36
37

For example:

```bash

temperature <-> TemperatureMeasurement (timestamp=XXX,value=XXX,unit=XXX)
                            |
                            v
                MqttMessage(topic=sensors/id/data payload=TemperatureMeasurement)
                            |
                            v
                          queue
                            ^
                            |
Sebastian Böhm's avatar
Sebastian Böhm committed
38
39
40
41
42
43
44
45
46
                mqtt_client.client.publish(MqttMessage.topic, MqttMessage.payload)
```

Other available services follow the same decoupled architecture.

## Services

## TemperatureService (`core.temperature_svc.py`)

Sebastian Böhm's avatar
Sebastian Böhm committed
47
This service is continuously transmitting temperature values to an mqtt endpoint.
Sebastian Böhm's avatar
Sebastian Böhm committed
48

Sebastian Böhm's avatar
Sebastian Böhm committed
49
### HTTP-Endpoints
Sebastian Böhm's avatar
Sebastian Böhm committed
50
51
52

* (UI) `/` \[GET\]: Current temperature value
* (API) `/temperature` \[GET\]: Current temperature value as json
Sebastian Böhm's avatar
Sebastian Böhm committed
53
* (API) `/device-info` \[GET\]: Current MQTT_CLIENT_ID as json 
Sebastian Böhm's avatar
Sebastian Böhm committed
54

Sebastian Böhm's avatar
Sebastian Böhm committed
55
56
57
All of the endpoints are protected with login.
You can specify the initial user by defining `FLASK_WEB_USER` and `FLASK_WEB_PASSWORD` in the `env` file.
Just have a look at section *Run the project locally*.
Sebastian Böhm's avatar
Sebastian Böhm committed
58
59
By default, the web ui is running under `https://<<your-ip>>:5000`.
You need to accept the self-signed certificate in order to proceed.
Sebastian Böhm's avatar
Sebastian Böhm committed
60
61

### MQTT-Topics
Sebastian Böhm's avatar
Sebastian Böhm committed
62
63
64

**TemperatureMessage:** `sensors/<MQTT_CLIENT_ID>/data`

Sebastian Böhm's avatar
Sebastian Böhm committed
65
66
Payload:

Sebastian Böhm's avatar
Sebastian Böhm committed
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
```json
{
  "timestamp": 1637751270.438974, 
  "value": 33, 
  "unit": "celsius"
}
```

## CPULoadService (`core.cpu_load_svc`)

This service allow to stress the CPU of the edge device for a period of time.
It is possible to define the duration and the target load for the cpu.

For example, all logical cores can be utilized to run at 50% CPU utilization for 15min.

### HTTP_Endpoints

Sebastian Böhm's avatar
Sebastian Böhm committed
84
* (UI) `/cpu-load` \[GET\]: Allow to create new cpu load jobs and to manage already existing ones
Sebastian Böhm's avatar
Sebastian Böhm committed
85
86
87
88
89

### MQTT_TOPICS

**Request Creation of CPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/create/req`

Sebastian Böhm's avatar
Sebastian Böhm committed
90
91
Payload:

Sebastian Böhm's avatar
Sebastian Böhm committed
92
93
94
95
96
97
98
99
100
101
102
103
104
```json
{
  "duration": 200,
  "target_load": 0.50
}
```

**Response Creation of CPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/create/res`

On Success:

```json
{
Sebastian Böhm's avatar
Sebastian Böhm committed
105
106
107
108
  "success":{
    "duration":200,
    "target_load":0.5
  }
Sebastian Böhm's avatar
Sebastian Böhm committed
109
110
111
112
113
114
115
116
117
118
119
}
```

On Error:

```json
{
  "error": "Illegal message received: ('Duration must be between 30 and 3600',)"
}
```

Sebastian Böhm's avatar
Sebastian Böhm committed
120
**Request Reading all QueuedCPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/read/req`
Sebastian Böhm's avatar
Sebastian Böhm committed
121
122
123

Payload intentionally left blank.

Sebastian Böhm's avatar
Sebastian Böhm committed
124
**Response Reading all QueuedCPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/read/res`
Sebastian Böhm's avatar
Sebastian Böhm committed
125
126
127

```json
{
Sebastian Böhm's avatar
Sebastian Böhm committed
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
  "success":{
    "1":{
      "duration":60,
      "target_load":0.25,
      "id":1,
      "pid":30040,
      "state":"RUNNING"
    },
    "2":{
      "duration":60,
      "target_load":0.55,
      "id":2,
      "pid":null,
      "state":"CREATED"
    },
    "3":{
      "duration":60,
      "target_load":0.55,
      "id":3,
      "pid":null,
      "state":"CREATED"
    }
  }
Sebastian Böhm's avatar
Sebastian Böhm committed
151
}
152
```
Sebastian Böhm's avatar
Sebastian Böhm committed
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178

**Request delete specific QueuedCPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/delete/req`

Payload:

```json
{
  "id":1
}
```

**Response delete specific QueuedCPULoadJobAllCores:** `services/cpuLoadSvc/<MQTT_CLIENT_ID>/jobs/delete/res`

```json
{
  "success":{
    "duration":200,
    "target_load":0.5,
    "id":4,
    "pid":13767,
    "state":"ABORTED"
  }
}
```

PLEASE NOTE:
Sebastian Böhm's avatar
Sebastian Böhm committed
179
180
181
182
183

* The **CPULoadService** is running a SQLite database to store the username and password for the web login.
* The submitted CPULoadJobs are only stored in-memory. That means, a shutdown of the application will lead to data loss. If you are restarting your application, you need recreate your job, respectively your job profile.

### General Information
184
185
186
187

All components can easily be configured with the following `.env` file with should be present in the root of this project.

```bash
Sebastian Böhm's avatar
Sebastian Böhm committed
188
189
190
191
192
193
194
195
196
197
198
#
# Example env file - DO NOT USE IN PRODUCTION!!!
# 
# MAKE SURE YOU ARE OVERRIDING AT LEAST
# 
# - MQTT_CLIENT_ID
# - FLASK_WEB_USER
# - FLASK_WEB_PASSWORD
# - FLASK_SECRET_KEY
# - FLASK_SECURITY_PASSWORD_SALT

199
200
201
# messaging
MQTT_SERVER_NAME=localhost
MQTT_PORT=1883
Sebastian Böhm's avatar
Sebastian Böhm committed
202
203
204
205
206
MQTT_TLS=False
MQTT_CA_CERTS=
MQTT_CERTFILE=
MQTT_KEYFILE=
MQTT_TLS_INSECURE=False
207
208
209
210
211
MQTT_USERNAME=
MQTT_PASSWORD=
MQTT_MAX_CONNECT_RETRIES=5
MQTT_RECONNECT_TIMEOUT=30
MQTT_CLIENT_ID=ab444537-cf67-47aa-a5ef-3548292e225b
Sebastian Böhm's avatar
Sebastian Böhm committed
212
MQTT_PUBLISH_QOS=2
Sebastian Böhm's avatar
Sebastian Böhm committed
213
MQTT_SUBSCRIBE_QOS=2
214
215
216
217
MQTT_TOPIC_PUBLISHER=publisher
MQTT_TOPIC_PUBLISHER_STATE=state

# core
Sebastian Böhm's avatar
Sebastian Böhm committed
218
219
220
221
MQTT_TOPIC_REQ_TYPE_CREATE=create/req
MQTT_TOPIC_RES_TYPE_CREATE=create/res
MQTT_TOPIC_REQ_TYPE_READ=read/req
MQTT_TOPIC_RES_TYPE_READ=read/res
Sebastian Böhm's avatar
Sebastian Böhm committed
222
223
MQTT_TOPIC_REQ_TYPE_DELETE=delete/req
MQTT_TOPIC_RES_TYPE_DELETE=delete/res
Sebastian Böhm's avatar
Sebastian Böhm committed
224

Sebastian Böhm's avatar
Sebastian Böhm committed
225
## temperature-svc
226
227
228
MQTT_TOPIC_SENSORS=sensors
MQTT_TOPIC_SENSORS_DATA=data

Sebastian Böhm's avatar
Sebastian Böhm committed
229
## cpu_load_svc
Sebastian Böhm's avatar
Sebastian Böhm committed
230
231
232
233
234
235
236
237
238
239
MQTT_TOPIC_SERVICES=services
MQTT_TOPIC_CPU_LOAD_SERVICE=cpuLoadSvc
MQTT_TOPIC_CPU_LOAD_SERVICE_DATA=jobs

CPU_LOAD_SVC_MIN_DURATION_SECONDS=30
CPU_LOAD_SVC_MAX_DURATION_SECONDS=3600

CPU_LOAD_SVC_MIN_TARGET_LOAD=0.25
CPU_LOAD_SVC_MAX_TARGET_LOAD=0.75

240
241
# web
WEB_HOSTNAME=localhost
Sebastian Böhm's avatar
Sebastian Böhm committed
242
243
244
245
FLASK_WEB_USER=test@test.com
FLASK_WEB_PASSWORD=kaq29387ckj
FLASK_SECRET_KEY=pf9Wkove4IKEAXvy-cQkeDPhv9Cb3Ag-wyJILbq_dFw
FLASK_SECURITY_PASSWORD_SALT=146585145368132386173505678016728509634
Sebastian Böhm's avatar
Sebastian Böhm committed
246
FLASK_SQLITE_DB_PATH=sqlite:///../../flask-db.db
Sebastian Böhm's avatar
Sebastian Böhm committed
247

Sebastian Böhm's avatar
Sebastian Böhm committed
248
249
250
251
252
253
254
255
256
257
# MQTT topics overrides
CUSTOM_MQTT_CLIENT_PUBLISHER_STATE=
CUSTOM_TOPIC_TEMPERATURESVC_TEMPERATURE_DATA=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_CREATE_REQ=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_CREATE_RES=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_READ_REQ=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_READ_RES=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_DELETE_REQ=
CUSTOM_TOPIC_CPULOADSVC_CPULOADJOB_DELETE_RES=

258
259
```

Sebastian Böhm's avatar
Sebastian Böhm committed
260
261
262
NOTE:

* Please make sure that you provide at least 4 characters for `FLASK_WEB_USER` and `FLASK_WEB_PASSWORD`.
Sebastian Böhm's avatar
Sebastian Böhm committed
263
* **NEW!** If you want to override the predefined MQTT topics, just define your desired custom topics for the different services under *MQTT topics overrides*. Then, Edge-IoT simulator will take your custom topic instead of the default configuration.
Sebastian Böhm's avatar
Sebastian Böhm committed
264

265
266
## Prerequisites

Sebastian Böhm's avatar
Sebastian Böhm committed
267
* Make sure that you installed `python3, python3-venv, python3-pip` on your system
268
269
270
271
* Make sure that you install `docker` on your system

## Run the project locally

Sebastian Böhm's avatar
Sebastian Böhm committed
272
273
274
* Install necessary packages: `sudo apt-get install python3-flask python3-venv python3-wheel`
* Clone/Pull this repository: `git clone https://gitlab.rz.uni-bamberg.de/sebastian.boehm/edge-iot-simulator`
* Go into the root directory of the repository, switch then to (`cd edge_iot_simulator`)
Sebastian Böhm's avatar
Sebastian Böhm committed
275
* Create the env file `.env` with the contents above and change the values according to your needs. Alternatively, you can copy the `.env-example` with `cp .env-example .env`. DO NOT FORGET TO ALTER THE SENSITIVE VARIABLES!!!
Sebastian Böhm's avatar
Sebastian Böhm committed
276
277
* Go back to the root directory of the repository: `cd ..`
* Create a virtual environment: `python3 -m venv venv`
278
* Change to the virtual environment: `source venv/bin/activate`
Sebastian Böhm's avatar
Sebastian Böhm committed
279
* Update pip: `pip3 install --upgrade pip`
280
281
* Install the necessary dependencies: `pip3 install -r requirements.txt`
* Change the working directory: `cd edge_iot_simulator`
Sebastian Böhm's avatar
Sebastian Böhm committed
282
* Run: `python3 main.py` (all services will start).
283

Sebastian Böhm's avatar
Sebastian Böhm committed
284
NOTE: If you want disable the `TemperatureService` (not burdening your broker with a lot messages), start the application with the following command: `python3 main.py --service cpu_load_svc`.
Sebastian Böhm's avatar
Sebastian Böhm committed
285
286
Then, the `TemperatureService` is effectively disabled.

287
288
289
290
You should see the following output:

```bash
11/11/2021 02:48:48 PM main Please any key to interrupt...
Sebastian Böhm's avatar
Sebastian Böhm committed
291
11/11/2021 02:48:48 PM mqtt_client Successfully started mqtt publisher...
292
293
294
11/11/2021 02:48:48 PM temperature_svc Successfully started temperature sensor...
11/11/2021 02:48:48 PM temperature_svc New value measured {"timestamp": 1636638528.2080338, "value": 26, "unit": "celsius"}
11/11/2021 02:48:48 PM app Successfully started WebbApp...
Sebastian Böhm's avatar
Sebastian Böhm committed
295
296
11/11/2021 02:48:48 PM mqtt_client Error establishing connection to localhost:1883[Errno 111] Connection refused
11/11/2021 02:48:48 PM mqtt_client mqtt publisher received shutdown signal
297
298
299
300
11/11/2021 02:48:51 PM temperature_svc New value measured {"timestamp": 1636638531.2109501, "value": 20, "unit": "celsius"}
11/11/2021 02:48:54 PM temperature_svc New value measured {"timestamp": 1636638534.2141488, "value": 7, "unit": "celsius"}
11/11/2021 02:48:57 PM temperature_svc New value measured {"timestamp": 1636638537.2166193, "value": 59, "unit": "celsius"}
11/11/2021 02:48:58 PM main Unknown error occurred: Could not establish the connection to the mqtt broker
Sebastian Böhm's avatar
Sebastian Böhm committed
301
11/11/2021 02:48:58 PM mqtt_client mqtt publisher received shutdown signal
302
303
304
305
306
307
308
309
11/11/2021 02:48:58 PM temperature_svc Temperature service received shutdown signal...
11/11/2021 02:48:58 PM app WebApp received shutdown signal....
11/11/2021 02:48:58 PM main Wait for graceful termination...
11/11/2021 02:49:00 PM main Successfully terminated edge-iot simulator
```

We can see the first measurements of the sensor. However, the edge-IoT simulator is terminating because no mqtt message broker is reachable.

Sebastian Böhm's avatar
Sebastian Böhm committed
310
For testing purposes, start a mqtt message broker via docker: `sudo docker run -d -p 1883:1883 -p 9001:9001 eclipse-mosquitto:1.6.15`
311
312
313
314
315
316
317

Run the application again: `python3 main.py`

You should see the following output:

```bash
11/11/2021 02:55:39 PM main Please any key to interrupt...
Sebastian Böhm's avatar
Sebastian Böhm committed
318
11/11/2021 02:55:39 PM mqtt_client Successfully started mqtt publisher...
319
320
321
11/11/2021 02:55:39 PM temperature_svc Successfully started temperature sensor...
11/11/2021 02:55:39 PM temperature_svc New value measured {"timestamp": 1636638939.4815965, "value": 46, "unit": "celsius"}
11/11/2021 02:55:39 PM app Successfully started WebbApp...
Sebastian Böhm's avatar
Sebastian Böhm committed
322
323
11/11/2021 02:55:39 PM mqtt_client Publisher connected with result code 0
11/11/2021 02:55:39 PM mqtt_client Successfully published message {"timestamp": 1636638939.4815965, "value": 46, "unit": "celsius"}
324
11/11/2021 02:55:42 PM temperature_svc New value measured {"timestamp": 1636638942.4844544, "value": 14, "unit": "celsius"}
Sebastian Böhm's avatar
Sebastian Böhm committed
325
11/11/2021 02:55:42 PM mqtt_client Successfully published message {"timestamp": 1636638942.4844544, "value": 14, "unit": "celsius"}
326
11/11/2021 02:55:45 PM temperature_svc New value measured {"timestamp": 1636638945.4878645, "value": 36, "unit": "celsius"}
Sebastian Böhm's avatar
Sebastian Böhm committed
327
11/11/2021 02:55:45 PM mqtt_client Successfully published message {"timestamp": 1636638945.4878645, "value": 36, "unit": "celsius"}
328
^C11/11/2021 02:55:48 PM main Received user's shutdown signal...
Sebastian Böhm's avatar
Sebastian Böhm committed
329
11/11/2021 02:55:48 PM mqtt_client mqtt publisher received shutdown signal
330
11/11/2021 02:55:48 PM temperature_svc New value measured {"timestamp": 1636638948.4884603, "value": 1, "unit": "celsius"}
Sebastian Böhm's avatar
Sebastian Böhm committed
331
332
11/11/2021 02:55:48 PM mqtt_client Successfully published message {"timestamp": 1636638948.4884603, "value": 1, "unit": "celsius"}
11/11/2021 02:55:48 PM mqtt_client mqtt publisher received shutdown signal
333
334
335
336
337
338
339
11/11/2021 02:55:48 PM temperature_svc Temperature service received shutdown signal...
11/11/2021 02:55:48 PM app WebApp received shutdown signal....
11/11/2021 02:55:48 PM main Wait for graceful termination...
11/11/2021 02:55:51 PM main Successfully terminated edge-iot simulator
```

Now the edge-IoT simulator is connected to the local mqtt message broker.
Sebastian Böhm's avatar
Sebastian Böhm committed
340
Press `CTRL+C` to abort the program.
341

Sebastian Böhm's avatar
Sebastian Böhm committed
342
343
Network architecture:

344
345
```bash
-----------------------localhost----------------------
Sebastian Böhm's avatar
Sebastian Böhm committed
346
edge-IoT simulator <-> docker daemon <-> mqtt message broker
347
348
349
------------------------------------------------------
```

Sebastian Böhm's avatar
Sebastian Böhm committed
350
351
352
By default web application is available under `https://<your ip>:5000`.
Since the web application contains a self-signed certificate, you need to accept this certificate in your browser.

353
354
## Run with docker

Sebastian Böhm's avatar
Sebastian Böhm committed
355
To run edge-IoT simulator with docker, do the following:
356
357

* Change to the root directory of this repository
Sebastian Böhm's avatar
Sebastian Böhm committed
358
* Build the image: `sudo docker build -t edge-iot-simulator:1.0 .`
359
* Change the working directory to the directory: `cd edge_iot_simulator`
Sebastian Böhm's avatar
Sebastian Böhm committed
360
361
* Create and adjust the `.env` file! **Make sure that you take the right IP address for the message broker. If you are running the mqtt broker on the same system (physical host), obtain the broker's ip address via `sudo docker inspect <containername>`**.
* Run the image: `sudo docker run -p 5000:5000 --env-file=.env edge-iot-simulator:1.0`, choose `sudo docker run -d -p 5000:5000 --env-file=.env edge-iot-simulator:1.0`
362

Sebastian Böhm's avatar
Sebastian Böhm committed
363
364
365
366
```bash
-----------------------localhost----------------------
docker daemon <-> edge-IoT-simulator <-> mqtt message broker 
------------------------------------------------------
Sebastian Böhm's avatar
Sebastian Böhm committed
367
```