__FORCETOC__

{| class="wikitable" border="1"

!Version

|}

According to Wikipedia, [https://en.wikipedia.org/wiki/Federated_learning Federated Learning] (FL) is defined as ''a machine learning technique that trains an algorithm via '''multiple independent sessions, each using its own dataset'''. This approach stands in contrast to traditional centralized machine learning techniques where local datasets are merged into one training session, as well as to approaches that assume that local data samples are identically distributed.''

A detailed dissertation of the work that led to this Technical Note is available here TBD [1].

For selecting the frameworks, several factors were taken into account:

* ML frameworks flexibility: The adaptability of the framework to manage different ML frameworks.

In the next sections, the aforementioned factors will be treated individually, justifying the reasons behind the discard of some frameworks rather than others.

Flexibility in ML frameworks is crucial when choosing a FL framework as it allows adapting the system to diverse use cases and data distributions. A flexible framework can support various ML algorithms, models, and data types, accommodating the specific requirements of different scenarios. This adaptability enhances the framework’s applicability in real-world deployments, ensuring it can effectively handle the heterogeneity and dynamic nature of distributed data sources across clients. It is noteworthy to mention that most of the frameworks discussed earlier, including NVFlare, FATE, Flower, PySyft, IBM, OpenFL, and FedML, are designed to be agnostic to the underlying ML framework used. This agnosticism provides users with the flexibility to employ various ML libraries, such as TensorFlow, PyTorch, and scikit-learn, among others, based on their preferences and project requirements. However, it is important to highlight that one exception to this trend is TFF. Unlike the other frameworks, TFF is specifically tailored for the TensorFlow ecosystem. While it harnesses the powerful capabilities of TensorFlow, it inherently limits the utilisation of other ML libraries in the FL context. As a result, users opting for TFF should be aware of this framework’s dependency on TensorFlow for their FL endeavors. For that reason TFF was discarded from the potential frameworks to be considered for comparison.

The choice of a suitable license is of paramount importance for any FL framework. A well-crafted license provides a legal foundation that governs the usage, distribution, and modification of the framework’s source code and associated components. A permissive license, like the MIT License or Apache License, allows users

to use, modify, and distribute the framework with relatively few restrictions. This encourages widespread adoption, fosters innovation, and facilitates contributions from a broader community of developers and researchers. The permissiveness of these licenses empowers users to incorporate the framework into their projects, even if hey have proprietary components. On the other hand, copyleft licenses, like the GNU GPL, require derived works to be distributed under the same terms, ensuring that any modifications or extensions to the framework remain open-source. While this may be more restrictive, it encourages a collaborative ecosystem where improvements are shared back with the community. A clear and well-defined license also provides legal protection to both developers and users, helping to mitigate potential legal risks and disputes. It ensures that contributors have granted appropriate rights to their work and helps maintain a healthy and sustainable development environment. Most of the frameworks previously described are under the Apache-2.0 license except one: IBMFL. In fact, it is under an unspecified license that makes the framework not suitable for commercial use. For that reason, IBMFL was discarded from the comparison too.

Ratings in public repositories such as "stars" in GitHub are important because they serve as a measure of popularity and community interest in the project. When a repository achieves a good rating, it indicates that more developers and users find the project valuable and relevant. This can lead to several benefits:

* Visibility: Repositories with good ratings are likely to appear higher in platform's search results, making it easier for others to discover and use the project.

These characteristics, although they certainly have a bearing on the choice of frameworks, were not enough to go so far as to discard any of the selected frameworks.

High quality documentation and well-crafted tutorials are essential considerations when selecting a FL framework. In fact, there are several reasons that are presented here below:

* Accessibility and Ease of Use: Comprehensive documentation allows users to understand the framework’s functionalities, APIs, and usage quickly. It enables developers, researchers, and practitioners to get started with the framework efficiently, reducing the learning curve.

Regarding this aspects, there are a lot of frameworks that still don’t have good documentation and tutorials. Among the latter, there are: PySyft, OpenFL and FedML. PySyft is still under construction, as the official repository says, and for that reason often the documentation is not up to date and is not complete. OpenFL, on its side, has very meager documentation and only a few tutorials that don’t explore a lot of ML frameworks or a lot of scenarios. The FedML framework also has, like PySyft, incomplete documentation because the project is born very recently and is still under development. Finally, the FATE framework has a complete and well-made documentation but very few tutorials and, because of its complex architecture, would have taken too much time. Because of these reasons, these four frameworks were discarded from the comparison.

In the context of FL, the significance of a framework being ready for commercial use cannot be overstated. As businesses increasingly recognize the value of decentralized ML solutions, the demand for robust and production-ready frameworks has intensified. A FL framework geared for commercial use offers several crucial advantages. Firstly, it provides a stable and scalable foundation to deploy large-scale FL systems across diverse devices and platforms. This ensures that businesses can seamlessly integrate the framework into their existing infrastructure, minimizing disruption, and optimizing efficiency. Moreover, a commercially viable framework emphasizes security and privacy measures, a non-negotiable aspect when dealing with sensitive data across distributed environments. Advanced encryption techniques, secure communication protocols, and differential privacy methods guarantee that user data remains safeguarded, mitigating potential risks of data breaches or unauthorized access. Of the frameworks covered, only a few are ready to be used commercially. Among them are: Flower, NVFlare, FATE, OpenFL, and TFF. On the other hand, there are some frameworks that are not yet ready. Among the latter are: FedML, PySyft and IBMFL. The first two are in fact still under development and not yet ready for commercial-level use, while the third has a private license that does not allow the framework to be used for commercial-level applications. As explained in the previous sub-sections, these frameworks were already discarded from comparison.

At the beginning of this section, a total of eight frameworks were considered. Each framework was assessed based on various aspects and after an in-depth

These two remaining frameworks are then: Flower and NVFlare. They demonstrated the potential to address the research objectives effectively and were well-aligned with the specific requirements of the FL project. Later, these two selected frameworks will be rigorously compared, examining their capabilities in handling diverse ML models, supporting various communication protocols, and accommodating heterogeneous client configurations. The comparison will delve into the frameworks’ performance, ease of integration, and potential for real-world deployment. By focusing on these two frameworks, this research aims to provide a detailed evaluation that can serve as a valuable resource for practitioners and researchers seeking to implement FL in a variety of scenarios. The selected frameworks will undergo comprehensive testing and analysis, enabling the subsequent sections to present an informed and insightful comparison, shedding light on their respective strengths and limitations.

=== Introduction ===

* Firstly, classification problems are a widely studied and well-understood domain in ML, making them suitable for benchmarking and evaluation purposes.

* Secondly, classification tasks are versatile and can be applied to various real-world scenarios, such as image classification, natural language processing, and medical diagnosis, making the evaluation results applicable to a wide range of applications.

|}

TBD

TBD

**https://arxiv.org/pdf/1805.03911.pdf

* [1]