Bureaucrats, dave_user, Administrators
4,650
edits
Changes
→Choosing Federated learning frameworks
== Choosing Federated learning frameworks ==
=== Criteria and initial, long list ===
For selecting the frameworks, several factors were taken into account:
* ML frameworks flexibility: The adaptability of the framework to manage different ML frameworks.
* Readiness for commercial usage: The readiness of the framework to be developed in a real-world scenario. In order to establish the readiness, it was checked the version of the framework and the license.
According to the previous criteria, an initial list including the most promising FL frameworks was completed. It comprised of the following products:
* [https://github.com/NVIDIA/NVFlare NVIDIA FL Application Runtime Environment ] (NVFlare)* [https://github.com/FederatedAI/ FATE Federated AI Technology Enabler ] (FATE)* [https://github.com/adap/flower Flower]* [https://github.com/OpenMined/ PySyft PySyft]* [https://github.com/IBM/federated-learning-lib IBM Federated Learning ] (IBMFL)* [https://github.com/ securefederatedai/openfl OpenFL]* [https://github.com/tensorflow/federated TensorFlowFederated ] (TFF)* [https://github.com/FedML-AI/ FedML FedML].
The limitation in selecting only eighth FL frameworks arises from the evolving nature of the field. As a relatively recent and rapidly evolving technique, FL continues to witness the emergence of various frameworks, each with its unique features and capabilities. In this context, the choice to focus on these frameworks reflects the attempt to capture the current state of the art and provide an analysis of the most prominent and well-established options available. This selection aims to offer valuable insights into the leading frameworks that are currently considered among the best choices in the evolving landscape of FL.
In the next sections, the aforementioned factors will be treated individindividually, justifying the reasons behind the discard of some frameworks rather than others. ==== ML frameworks flexibility ====Flexibility in ML frameworks is crucial when choosing a FL framework as it allows adapting the system to diverse use cases and data distributions. A flexible framework can support various ML algorithms, models, and data types, accommodating the specific requirements of different scenarios. This adaptability enhances the framework’s applicability in real-world deploy- ments, ensuring it can effectively handle the heterogeneity and dynamic na- ture of distributed data sources across clients. It is noteworthy to mention that most of the frameworks discussed ear- lier, including NVFlare, FATE, Flower, PySyft, IBM, OpenFL, and FedML, are designed to be agnostic to the underlying ML framework used. This ag- nosticism provides users with the flexibility to employ various ML libraries, such as TensorFlow, PyTorch, and scikit-learn, among others, based on their preferences and project requirements. However, it is important to highlight that one exception to this trend is TFF. Unlike the other frameworks, TFF is specifically tailored for the TensorFlow ecosystem. While it harnesses the powerful capabilities of Ten- sorFlow, it inherently limits the utilisation of other ML libraries in the FL context. As a result, users opting for TFF should be aware of this frame- work’s dependency on TensorFlow for their FL endeavors. For that reason TFF was discarded from the potential frameworks to be considered for comparison. ==== Licensing ====The choice of a suitable license is of paramount importance for any FL frame- work [19]. A well-crafted license provides a legal foundation that governs the usage, distribution, and modification of the framework’s source code and as- sociated components. A permissive license, like the MIT License or Apache License, allows users to use, modify, and distribute the framework with relatively few restrictions. This encourages widespread adoption, fosters innovation, and facilitates con- tributions from a broader community of developers and researchers. The per- missiveness of these licenses empowers users to incorporate the framework into their projects, even if they have proprietary components. On the other hand, copyleft licenses, like the GNU GPL, require derived works to be distributed under the same terms, ensuring that any modifica- tions or extensions to the framework remain open-source. While this may be more restrictive, it encourages a collaborative ecosystem where improve- ments are shared back with the community. A clear and well-defined license also provides legal protection to both developers and users, helping to mitigate potential legal risks and disputes. It ensures that contributors have granted appropriate rights to their work and helps maintain a healthy and sustainable development environment. Most of the frameworks previously described are under the Apache-2.0 license except one: IBMFL. In fact, it is under an unspecified license that makes the framework not suitable for commercial use. For that reason, IBMFL was discarded from the comparison too. ==== Repository stars and releases ====Stars in a GitHub repository are important because they serve as a measure of popularity and community interest in the project. When a repository receives more stars, it indicates that more developers and users find the project valuable and relevant. This can lead to several benefits:* Visibility: Repositories with more stars are likely to appear higher in GitHub search results, making it easier for others to discover and use the project.* Credibility: High-starred repositories are often perceived as more trust- worthy and reliable, as they are vetted and endorsed by a larger user base.* Contributions: Popular repositories tend to attract more contribu- tions from developers, leading to a more active and vibrant community around the project.* Feedback: Projects with many stars are more likely to receive feedback, bug reports, and feature requests, helping the developers improve the software.* Maintenance: Higher stars can also stimulate the maintainers to keep the project updated and actively supported. Other important aspects, which are related to the stars obtained by the framework, are the first and latest releases. Thanks to the latter, it is possible respectively to see the maturity of the framework and also how often it is updated, and thus the support behind it. Obviously, a framework that was born earlier than others is much more likely to have more stars. Having this in mind, at the time of writing this thesis, the ranking in terms of star ratings received correlated with the first release for each framework is as follows:* PySyft 8.9k stars Jan 19, 2020 2. FATE 5.1k stars Feb 18, 2019 3. FedML 3.1k stars Apr 30, 2022 4. Flower 2.8k stars Nov 11, 2020 5. TFF 2.1k stars Feb 20, 2019 6. OpenFL 567 stars Feb 1, 2021 7. IBMFL 438 stars Aug 28, 2020 8. NVFlare 413 stars Nov 23, 2021 These characteristics, although they certainly have a bearing on the choice of frameworks, were not enough to go so far as to discard any of the selected frameworks. ==== Documentation and tutorials ====* High quality documentation and well-crafted tutorials are essential consid- erations when selecting a FL framework. In fact, there are several reasons that are presented here below:* Accessibility and Ease of Use: Comprehensive documentation allows users to understand the framework’s functionalities, APIs, and usage quickly. It enables developers, researchers, and practitioners to get started with the framework efficiently, reducing the learning curve. • Accelerated Development: Well-structured tutorials and examples demon- strate how to use the framework to build practical FL systems. They provide step-by-step guidance on setting up experiments, running code, and interpreting results. This expedites the development process and encourages experimentation with different configurations. • Error Prevention: Clear documentation and good examples help users avoid common mistakes and errors during implementation. It provides troubleshooting tips and addresses frequently asked questions, reducing frustration and increasing user satisfaction. • Reliability and Robustness: A well-documented framework indicates that developers have invested time in organising their code and ex- plaining its functionalities. This attention to detail suggests a more reliable and stable framework. • Maintenance: Higher stars can also stimulate the maintainers to keep the project updated and actively supported.Regarding this aspects, there are a lot of frameworks that still don’t have good documentation and tutorials. Among the latter, there are: PySyft, OpenFL and FedML. PySyft is still under construction, as the official repos- itory says, and for that reason often the documentation is not up to date and is not complete. OpenFL, on its side, has very meager documentation and only a few tutorials that don’t explore a lot of ML frameworks or a lot of scenarios. The FedML framework also has, like PySyft, incomplete documentation because the project is born very recently and is still under development. Finally, the FATE framework has a complete and well-made documentation but very few tutorials and, because of its complex architec- ture, would have taken too much time. Because of these reasons, these four frameworks were discarded from the comparison. ==== Readiness for commercial usage ====In the context of FL, the significance of a framework being ready for commer- cial use cannot be overstated. As businesses increasingly recognise the value of decentralised ML solutions, the demand for robust and production-ready frameworks has intensified. A FL framework geared for commercial use offers several crucial ad- vantages. Firstly, it provides a stable and scalable foundation to deploy large-scale FL systems across diverse devices and platforms. This ensures that businesses can seamlessly integrate the framework into their existing infrastructure, minimising disruption and optimising efficiency. Moreover, a commercially viable framework emphasises security and pri- vacy measures, a non-negotiable aspect when dealing with sensitive data across distributed environments. Advanced encryption techniques, secure communication protocols, and differential privacy methods guarantee that user data remains safeguarded, mitigating potential risks of data breaches or unauthorised access. Of the frameworks covered, only a few are ready to be used commer- cially. Among them are: Flower, NVFlare, FATE, OpenFL and TFF. On the other hand, there are some frameworks that are not yet ready. Among the latter are: FedML, PySyft and IBMFL. The first two are in fact still under development and not yet ready for commercial-level use, while the third has a private license that does not allow the framework to be used for commercial-level applications. As explained in the previous sub-sections, these frameworks were already discarded from comparison. ==== Final choice ====At the beginning of this section, a total of eight frameworks were considered. Each framework was assessed based on various aspects and after an in-depth analysis, six frameworks were deemed unsuitable due to some requisites not being met. The requirements that were met and not met by the framework were summarised in Table 3.1: TBD These two remaining frameworks are: Flower and NVFlare. They demon- strated the potential to address the research objectives effectively and were well-aligned with the specific requirements of the FL project. In chapter 5, these two selected frameworks will be rigorously compared, examining their capabilities in handling diverse ML models, supporting vari- ous communication protocols, and accommodating heterogeneous client con- figurations. The comparison will delve into the frameworks’ performance, ease of integration, and potential for real-world deployment. By focusing on these two frameworks, this research aims to provide a detailed evaluation that can serve as a valuable resource for practitioners and researchers seeking to implement FL in a variety of scenarios. The se- lected frameworks will undergo comprehensive testing and analysis, enabling the subsequent sections to present an informed and insightful comparison,
== Testing the selected frameworks ==