- job_name: 'node-exporter'
  static_configs:
    - targets: ['<Controller-IP>:9100']

- job_name: 'cAdvisorr'
  static_configs:
    - targets: ['<Controller-IP>:9100']

Replace with the actual IP address of your Controller & Migration Engine machine.

For this machine, we have to configure two subfolders.

Kafka Cluster

1. Change the current working directory to the Kafka cluster folder.
2. Edit docker-compose.yml :
   
   In docker compose change these environment variables by changing 192.168.122.145 with your machine's public ip address. KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://192.168.122.145:9092,DOCKER://host.docker.internal:29092 KAFKA_JMX_HOSTNAME: 192.168.122.145.
3. Run pip install -r requirements.txt.

Logs Reporter

1. Change the current working directory to the logsParser folder.

2. Open the file config.ini; you have to edit the following parameters:

   • host = 192.168.122.1: Change this with the IP address of your databases IP.
   • user = root: This is the default username used to run the NoSQL MongoDB. If you want to change it, you also have to change MONGO_INITDB_ROOT_USERNAME in docker-compose.yml on the databases machine.
   • password = example: This is the default password used to run the NoSQL MongoDB. If you want to change it, you also have to change MONGO_INITDB_ROOT_PASSWORD in docker-compose.yml on the databases machine.

The Controller utilizes a pivotal configuration file named "config.ini," crucial for providing essential settings to the Migration Engine. This configuration holds paramount importance, guiding users in the dockerization of their migration engine.

The "config.ini" file consists of two integral parts:

1. First Part:

   • This section is transmitted unaltered to the Migration Engine. Its content remains intact when creating config.ini for the migration engine.

   • [[targetServer]]

     • In this section, the user can put any information needed to connect to the target Server.
       • host
       • user
       • password

   • [[sourceServer]]

     • In this section, the user can put any information needed to connect to the source Server.
       • host
       • user
       • password

   • [[KafkaCluster]]

     • In this section, the user should only change the value of the IP address of the reporter machine. The other variables should remain with the default values.
       • host=192.168.122.143; this should be changed with the reporter IP
       • port=9092
       • performanceBenchmarkTopic=performanceBenchmark
       • frameworkTopicName=framework

   • [[migrationEnvironment]]

     • In this section, the user should choose to put information needed for the migration.
       • migrationEngineDockerImage: the name of the docker image the user created for the migration engine.
       • loggingId: In case the user needs all the logs and information collected during the monitoring to be assigned to a certain Id; this can be left empty.
       • numberofexperiments: how many times each experiment is repeated with the same configuration (for the accuracy of the results).

2. Second Part:

   • The second part encompasses all conceivable parameters for the migration scenarios users wish to evaluate. Each parameter combination is systematically chosen by the Controller, which then conveys these specific parameters to the Migration Engine one at a time.

   • [[experiment]]

     • In this section, this is an example for parameters for a file migration engine, the user can put parameters according to his engine.
       • file = file1.csv, file2.txt, file3.java
       • limit = 1048576, 1048576
       • compressiontype = None, gzip, lz4
       • stream = 3, 2, 1

   The Controller is responsible for examining all possible combinations when generating the configuration file for the Migration Engine. As an illustration of the second part of the configuration file, consider the following example:

   • [[experiment]]
     • file = file1.csv
     • limit = 1048576
     • compressiontype = None
     • stream = 3

• On the Databases machine, change the current working directory to the databases folder.
• Run the following command:

  docker-compose up

• This will initiate Prometheus and MongoDB database along with Grafana. Grafana serves as a dashboard designed to help you monitor your migration engine's resource consumption in real-time.

• On the Kafka cluster machine, change the current working directory to the reporter/kafkacluster folder.
• Run the following command:

  docker-compose up

• On the Kafka cluster machine, change the current working directory to the reporter/kafkacluster folder.
• Run the following command:

  python consumer.py

Finally, initiate the Controller, which will orchestrate and commence all experiments.

• On the controller machine, change the current working directory to the controller folder.
• Run the following command:

  docker pull "<your migration engine image>"

  docker compose up

• This will initiate the controller along with cAdvisor and node-exporter.
  • cAdvisor serves as a daemon that collects, aggregates, processes, and exports information about running the controller and the migration engine.
  • node-exporter is designed to monitor the host system where all the containers are deployed on.

• Subsequently, the parser needs to be executed. Throughout the experiment, resource consumption logs were promptly stored in Prometheus. However, the logs pertaining to performance benchmarks remain localized on the Kafka machine.
• Running the parser becomes imperative at this juncture. Its role is twofold: to render the performance benchmark logs human-readable and to facilitate their exportation into JSON and CSV files. Furthermore, the parser ensures the archival of these logs in the MongoDB database for comprehensive analysis and reference.
• On the Kafka cluster machine, change the current working directory to the reporter/logsParser folder.
• Run the following command:

  python main.py