Bureaucrats, dave_user, Administrators
4,650
edits
Changes
→Results
* Local evaluation: accuracy and F1-score at the end of each round of FL were tested.
* Training evaluation: it was computed loss, accuracy, and F1-score.
Experiments were run both in the local and cloud environments. Detailed results are illustrated in [1]. In essence, the results are very similar for both frameworks. Thus, they can be considered equivalent from the point of view of the metrics considered. It is also very important to note that for all the results regarding the cloud environment, there are '''very similar values between the testbed based on virtual machines and the one based on embedded devices'''. This is not obvious because moving from virtualized, x86-powered clients to ARM64-powered clients entails several issues that can affect the results of the FL application [34]. Among these, it is worth to remember the following:
* Limited hardware resources: Embedded devices often have limited hardware resources, such as CPU, memory and computing power. This restriction can affect the performance of FL, especially if models are complex or operations require many resources.
* Hardware variations: Embedded devices may have hardware variations between them, even if they belong to the same class. These hardware differences may lead to different behaviors in FL models, requiring more robustness in adapting to different devices.
* Variations in workload: Embedded device applications may have very different workloads from those simulated in a virtual environment. These variations may lead to different requirements for FL.
== Privacy and security ==
In the comparison between the two FL frameworks, NVFlare and Flower, a crucial aspect that was assessed is the level of privacy and security offered by each framework. Both NVFlare and Flower implement strategies to ensure data privacy and security during the training process. These strategies involve techniques such as data encryption, secure communication protocols, and differential privacy mechanisms. By analyzing the privacy and security features, it was possible to evaluate how well each framework protects sensitive user data while enabling effective collaborative model training. Factors such as the strength of encryption algorithms, the robustness of communication channels against potential breaches, and the extent to which the frameworks adhere to privacy regulations were considered. The assessment aimed to determine whether these frameworks implement state-of-the-art security measures, maintain data confidentiality, and provide assurances against potential attacks or data leaks. Additionally, a comparison of the frameworks’ approaches to preserving user privacy within a collaborative training environment was conducted. This analysis not only provides insights into the level of privacy and security that NVFlare and Flower offer but also shows how ready these two frameworks are for industrial use and how their strategies can prevent some of the main attacks of the FL landscape. Specifically, the following items were investigated. For more details, please refer to [1]:
* Secure communication
* Differential privacy
* Secure aggregation
* Federated authorization
Privacy and security are two of the most important features to consider when analyzing FL frameworks. This is because FL itself was born out of the concern that classical ML brought about due to the lack of data security and privacy. In this regard, both frameworks have various techniques for data security and privacy, enabling a secure implementation of a FL architecture avoiding a lot of possible attacks. In spite of this, the analysis we conducted led us to the conclusion that newer NVFlare framework is more complete both in terms of the tutorials provided for a possible implementation example and in terms of the provided features. Indeed, it can be seen how greater attention was paid to a possible development in a real-world scenario thanks for example to the Federated Authorization system recently implemented by the framework.
== Ease of use ==
Another important feature to consider the ease of use of a framework. This is an important aspect because it influences the practicality and circulation of a framework.
== Execution time ==