Difference between revisions of "MISC-TN-015: Yocto and git protocol error"

From DAVE Developer's Wiki
Jump to: navigation, search
(→‎Yocto Manifest)
Line 2: Line 2:
 
{{AppliesToAXEL Lite TN}}
 
{{AppliesToAXEL Lite TN}}
 
{{AppliesToAXEL ULite TN}}
 
{{AppliesToAXEL ULite TN}}
−
{{AppliesTo ORCA TN}}
 
−
{{AppliesTo MITO 8M Mini TN}}
 
 
{{AppliesToBORA TN}}
 
{{AppliesToBORA TN}}
 
{{AppliesToBORA Xpress TN}}
 
{{AppliesToBORA Xpress TN}}
Line 25: Line 23:
  
 
==Introduction==
 
==Introduction==
−
Recently, some months ago, there was a policy change on accessing git repositories for security issues. The Github security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
+
Starting from beginning of 2022, there was a policy change on most git hosting services to restrict "unsecure" git protocol for security issues.
 +
 
 +
GitHub security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
  
 
As reported :
 
As reported :
Line 31: Line 31:
 
  ''We’re changing which keys are supported in SSH and removing unencrypted Git protocol. Only users connecting via SSH or git:// will be affected. If your Git remotes start with https://, nothing in this post will affect you. If you’re an SSH user, read on for the details and timeline.''  
 
  ''We’re changing which keys are supported in SSH and removing unencrypted Git protocol. Only users connecting via SSH or git:// will be affected. If your Git remotes start with https://, nothing in this post will affect you. If you’re an SSH user, read on for the details and timeline.''  
  
−
Since March 15, 2022 unencrypted git protocol has been disabled:
 
  
−
[[File:Git_protocol_security_on_GitHub.png | center|600px]]
+
This is already applied in DAVE BSPs published after March 2022 but older BSPs, published prior this change, may need to be fixed manually to use ssh instead of <code>git://</code> to access some repositories
  
−
To mitigate this issue, it will be required to change the protocol access to the repo starting with <code>git://</code>
+
This Technical Note shows an example on how to update those older releases to solve this issue
  
−
This Technical Note shows an example on how to modify <code>repo</code> manifest to correctly run the Yocto build. Some ''tips & tricks'' can be found also in this [https://stackoverflow.com/questions/70663523/the-unauthenticated-git-protocol-on-port-9418-is-no-longer-supported stackoverflow] question.
+
== BSP repo Manifest ==
 +
DAVE Yocto BSPs uses [https://gerrit.googlesource.com/git-repo/ repo] to track the multiple layers required to setup the BSP itself
  
−
== Yocto Manifest ==
+
If <code>git</code> protocol is used to clone some these layers, user will encounter the above issue
−
Before starting the Yocto build, it is required to check the <code>default.xml</code> manifest file for the git access to the layers repositories.
 
  
−
Here below there is an example using the [https://wiki.dave.eu/index.php?title=DESK-MX6-L/Development/Building_the_Yocto_BSP&oldid=14300 DESK-MX6-L-1.0.0] wiki page instructions for building the overall BSP.
+
Here there is an example using the {{OldRevision|page=DESK-MX6-L/Development/Building the Yocto BSP|revision=14300|text=DESK-MX6-L-1.0.0}}page instructions for building the overall BSP while fixing the manifest
  
−
=== Orginal ''default.xml''===
+
=== Setup with original manifest===
−
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the '''git''' protocol for accessing the Yocto, freescale and Qt repositories:
+
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the <code>git</code> protocol to clone some layers:
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 63: Line 62:
 
...
 
...
 
</pre>
 
</pre>
−
in this case, multiple access error will be prompted by repo sync:
+
in this case, multiple access error will be prompted while running <code>repo sync</code>:
 
   
 
   
 
  dvdk@vagrant:~/yocto$ ./repo sync
 
  dvdk@vagrant:~/yocto$ ./repo sync
Line 99: Line 98:
 
  ...
 
  ...
  
−
=== Fixed ''default.xml''===
+
=== Fixing repo manifest===
−
The modified file, as per <code>desk-mx6-l-1.0.2</code> tag, uses the '''https''' protocol for accessing the git repositories:
+
<code>repo</code> stores the current manifest in <code>.repo/manifest.xml</code>: this is the file that needs to be patched to solve this issue locally
 +
 
 +
After <code>repo init</code> and prior <code>repo sync</code>, user need to edit <code>.repo/manifest.xml</code> to change all occurrence of <code>git://</code> protocol specifier to <code>https://</code>
 +
 
 +
This can be also applied with a simple <code>sed</code> command:<syntaxhighlight lang="bash">
 +
sed -i 's/git:\/\//https:\/\//g' .repo/manifest.xml
 +
</syntaxhighlight>
 +
Please note that, for <code>DESK-MX6-L-1.x.x</code>, this is already fixed in <code>desk-mx6-l-1.0.2</code> release:
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 119: Line 125:
 
</pre>
 
</pre>
  
−
== Other Yocto layers or recipes ==
+
To be precise the <code>diff</code> between the two releases is:<syntaxhighlight lang="diff">
−
This issue can be present also on other layers or recipes: in this case, the related <code>git://</code> access should be changed or fixed.
+
diff --git a/default.xml b/default.xml
−
 
+
index f4d7443..1a4d7fb 100644
−
== Workaround ==
+
--- a/default.xml
 +
+++ b/default.xml
 +
@@ -3,12 +3,12 @@
 +
 +
  <default sync-j="2"/>
 +
 +
-  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
 +
-  <remote fetch="git://github.com/Freescale" name="freescale"/>
 +
-  <remote fetch="git://git.openembedded.org" name="oe"/>
 +
-  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
 +
-  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
 +
-  <remote fetch="git://github.com/meta-rust"  name="rust"/>
 +
+  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
 +
+  <remote fetch="https://github.com/Freescale" name="freescale"/>
 +
+  <remote fetch="https://git.openembedded.org" name="oe"/>
 +
+  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
 +
+  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
 +
+  <remote fetch="https://github.com/meta-rust"  name="rust"/>
 +
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
 +
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
  
−
As an workaround for the <code><nowiki>git://github.com/</nowiki></code> access, it is possible to replace the ''git'' access to the ''https'' access using the [https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf git configuration] command:
+
</syntaxhighlight>
  
−
<pre>
+
== Layer recipes ==
−
<nowiki>
+
While most of Yocto recipes used tar (or similar) archive to get source code, some of them download source code using git.
−
git config --global url."https://github.com/".insteadOf git://github.com/
 
−
</nowiki>
 
−
</pre>
 
  
−
See more info [https://stackoverflow.com/questions/1722807/how-to-convert-git-urls-to-http-urls here]
+
Usually <code>https</code> protocol is used for this, but in some cases <code>SRC_URI</code> may need to be updated if plain <code>git</code> protocol is used (and if the git server implements GitHub like security policies)

Revision as of 14:41, 22 April 2022

HOME SOMs SBCs ToloMEO Embedded Assistant Sale-tag.pngGET A QUOTE Customer-service.pngONLINE HELPDESK
Roadmap IoT Services ML/AI services Embedded Design Services
Info Box
Yocto-logo.png Applies to Yocto


History[edit | edit source]

Version Date Notes
1.0.0 Apr 2022 First public release

Introduction[edit | edit source]

Starting from beginning of 2022, there was a policy change on most git hosting services to restrict "unsecure" git protocol for security issues.

GitHub security access has been changed for accessing the git repositories via ssh: more information can be found in the following news Improving Git protocol security on GitHub.

As reported : 

We’re changing which keys are supported in SSH and removing unencrypted Git protocol. Only users connecting via SSH or git:// will be affected. If your Git remotes start with https://, nothing in this post will affect you. If you’re an SSH user, read on for the details and timeline.


This is already applied in DAVE BSPs published after March 2022 but older BSPs, published prior this change, may need to be fixed manually to use ssh instead of git:// to access some repositories

This Technical Note shows an example on how to update those older releases to solve this issue

BSP repo Manifest[edit | edit source]

DAVE Yocto BSPs uses repo to track the multiple layers required to setup the BSP itself

If git protocol is used to clone some these layers, user will encounter the above issue

Here there is an example using the DESK-MX6-L-1.0.0 page instructions for building the overall BSP while fixing the manifest

Setup with original manifest[edit | edit source]

The original file, as per desk-mx6-l-1.0.1 tag, uses the git protocol to clone some layers: 

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
  <remote fetch="git://github.com/Freescale" name="freescale"/>
  <remote fetch="git://git.openembedded.org" name="oe"/>
  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="git://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

in this case, multiple access error will be prompted while running repo sync: 

dvdk@vagrant:~/yocto$ ./repo sync
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0remote: Counting objects: 101, done.        
remote: Compressing objects: 100% (99/99), done.        
remote: Total 101 (delta 27), reused 0 (delta 0)        
Receiving objects: 100% (101/101), 15.88 KiB | 7.94 MiB/s, done.
Resolving deltas: 100% (27/27), done.
From ssh://git.dave.eu/desk-mx-l/desk-mx-l-bsp
 * [new branch]      hardknott             -> DAVE/hardknott
 * [new branch]      sumo                  -> DAVE/sumo
 * [new tag]         desk-mx6-l-1.0.1      -> desk-mx6-l-1.0.1
 * [new tag]         desk-mx6-l-1.0.0      -> desk-mx6-l-1.0.0
 * [new tag]         desk-mx6-l-1.0.2      -> desk-mx6-l-1.0.2
 * [new tag]         desk-mx6-l-3.0.0      -> desk-mx6-l-3.0.0
 * [new tag]         desk-mx6ul-l-1.0.0    -> desk-mx6ul-l-1.0.0
 * [new tag]         desk-mx6ul-l-1.0.1    -> desk-mx6ul-l-1.0.1
 * [new tag]         desk-mx6ul-l-3.0.0    -> desk-mx6ul-l-3.0.0
 * [new tag]         desk-mx8m-l-2.0.0-rc2 -> desk-mx8m-l-2.0.0-rc2
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
fatal: remote error: 
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
100   145  100   145    0     0    147      0 --:--:-- --:--:-- --:--:--   147
100 15.5M  100 15.5M    0     0  2051k      0  0:00:07  0:00:07 --:--:-- 2660k
Receiving objects: 100% (182040/182040), 15.58 MiB | 37.81 MiB/s, done.
Resolving deltas: 100% (126131/126131), done.
From /home/dvdk/yocto/.repo/projects/sources/meta-fsl-bsp-release.git/clone.bundle
 * [new branch]      warrior-4.19.35-1.1.0     -> CAF/warrior-4.19.35-1.1.0
 * [new branch]      thud-4.19.35-1.0.0        -> CAF/thud-4.19.35-1.0.0
 * [new branch]      sumo-4.14.98-2.3.0        -> CAF/sumo-4.14.98-2.3.0
...
...

Fixing repo manifest[edit | edit source]

repo stores the current manifest in .repo/manifest.xml: this is the file that needs to be patched to solve this issue locally

After repo init and prior repo sync, user need to edit .repo/manifest.xml to change all occurrence of git:// protocol specifier to https://

This can be also applied with a simple sed command:

sed -i 's/git:\/\//https:\/\//g' .repo/manifest.xml

Please note that, for DESK-MX6-L-1.x.x, this is already fixed in desk-mx6-l-1.0.2 release: 

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
  <remote fetch="https://github.com/Freescale" name="freescale"/>
  <remote fetch="https://git.openembedded.org" name="oe"/>
  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="https://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

To be precise the diff between the two releases is:

diff --git a/default.xml b/default.xml
index f4d7443..1a4d7fb 100644
--- a/default.xml
+++ b/default.xml
@@ -3,12 +3,12 @@
 
   <default sync-j="2"/>
 
-  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
-  <remote fetch="git://github.com/Freescale" name="freescale"/>
-  <remote fetch="git://git.openembedded.org" name="oe"/>
-  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
-  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
-  <remote fetch="git://github.com/meta-rust"  name="rust"/>
+  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
+  <remote fetch="https://github.com/Freescale" name="freescale"/>
+  <remote fetch="https://git.openembedded.org" name="oe"/>
+  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
+  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
+  <remote fetch="https://github.com/meta-rust"  name="rust"/>
   <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
   <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>

Layer recipes[edit | edit source]

While most of Yocto recipes used tar (or similar) archive to get source code, some of them download source code using git.

Usually https protocol is used for this, but in some cases SRC_URI may need to be updated if plain git protocol is used (and if the git server implements GitHub like security policies)

HOME SOMs SBCs ToloMEO Embedded Assistant Sale-tag.pngGET A QUOTE Customer-service.pngONLINE HELPDESK
Roadmap IoT Services ML/AI services Embedded Design Services
Retrieved from "https://wiki.dave.eu/index.php?title=MISC-TN-015:_Yocto_and_git_protocol_error&oldid=16443"