Difference between revisions of "MISC-TN-015: Yocto and git protocol error"

From DAVE Developer's Wiki
Jump to: navigation, search
(Introduction)
(4 intermediate revisions by 2 users not shown)
Line 25: Line 25:
  
 
==Introduction==
 
==Introduction==
Starting from beginning of 2022, there was a policy change on most git hosting services to restrict "unsecure" git protocol for security issues.
+
Recently, some months ago, there was a policy change on accessing git repositories for security issues. The Github security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
 
 
GitHub security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
 
  
 
As reported :
 
As reported :
Line 34: Line 32:
  
 
Since March 15, 2022 unencrypted git protocol has been disabled:
 
Since March 15, 2022 unencrypted git protocol has been disabled:
 +
 
[[File:Git_protocol_security_on_GitHub.png | center|600px]]
 
[[File:Git_protocol_security_on_GitHub.png | center|600px]]
  
This is already applied in DAVE BSPs published after March 2022 but older BSPs, published prior this change, may need to be fixed manually to use ssh instead of <code>git://</code> to access some repositories
+
To mitigate this issue, it will be required to change the protocol access to the repo starting with <code>git://</code>
  
This Technical Note shows an example on how to update those older releases to solve this issue
+
This Technical Note shows an example on how to modify <code>repo</code> manifest to correctly run the Yocto build. Some ''tips & tricks'' can be found also in this [https://stackoverflow.com/questions/70663523/the-unauthenticated-git-protocol-on-port-9418-is-no-longer-supported stackoverflow] question.
  
== BSP repo Manifest ==
+
== Yocto Manifest ==
DAVE Yocto BSPs uses [https://gerrit.googlesource.com/git-repo/ repo] to track the multiple layers required to setup the BSP itself. If <code>git</code> protocol is used to clone some these layers, user will encounter the above issue.
+
Before starting the Yocto, build it is required to check the <code>default.xml</code> manifest file for the git access to the layers repositories.
  
Here there is an example using the {{OldRevision|page=DESK-MX6-L/Development/Building the Yocto BSP|revision=14300|text=DESK-MX6-L-1.0.0}}page instructions for building the overall BSP while fixing the manifest
+
Here below there is an example using the [https://wiki.dave.eu/index.php?title=DESK-MX6-L/Development/Building_the_Yocto_BSP&oldid=14300 DESK-MX6-L-1.0.0] wiki page instructions for building the overall BSP.
  
=== Setup with original manifest===
+
=== Orginal ''default.xml''===
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the <code>git</code> protocol to clone some layers:
+
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the '''git''' protocol for accessing the Yocto, freescale and Qt repositories:
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 64: Line 63:
 
...
 
...
 
</pre>
 
</pre>
in this case, multiple access error will be prompted while running <code>repo sync</code>:
+
in this case, multiple access error will be prompted by repo sync:
 
   
 
   
 
  dvdk@vagrant:~/yocto$ ./repo sync
 
  dvdk@vagrant:~/yocto$ ./repo sync
Line 100: Line 99:
 
  ...
 
  ...
  
=== Fixing repo manifest===
+
=== Fixed ''default.xml''===
<code>repo</code> stores the current manifest in <code>.repo/manifest.xml</code>: this is the file that needs to be patched to solve this issue locally
+
The modified file, as per <code>desk-mx6-l-1.0.2</code> tag, uses the '''https''' protocol for accessing the Yocto, freescale and Qt repositories:
 
 
After <code>repo init</code> and prior <code>repo sync</code>, user need to edit <code>.repo/manifest.xml</code> to change all occurrence of <code>git://</code> protocol specifier to <code>https://</code>
 
 
 
This can be also applied with a simple <code>sed</code> command:<syntaxhighlight lang="bash">
 
sed -i 's/git:\/\//https:\/\//g' .repo/manifest.xml
 
</syntaxhighlight>
 
Please note that, for <code>DESK-MX6-L-1.x.x</code>, this is already fixed in <code>desk-mx6-l-1.0.2</code> release:
 
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 127: Line 119:
 
</pre>
 
</pre>
  
To be precise the <code>diff</code> between the two releases is:
+
== Other Yocto layers or recipes ==
<syntaxhighlight lang="diff">
+
This issue can be present also on other layers or recipes: in this case, the related <code>git://</code> access should be changed or fixed.
diff --git a/default.xml b/default.xml
 
index f4d7443..1a4d7fb 100644
 
--- a/default.xml
 
+++ b/default.xml
 
@@ -3,12 +3,12 @@
 
 
  <default sync-j="2"/>
 
 
-  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
 
-  <remote fetch="git://github.com/Freescale" name="freescale"/>
 
<remote fetch="git://git.openembedded.org" name="oe"/>
 
-  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
 
<remote fetch="git://github.com/meta-qt5"  name="QT5"/>
 
-  <remote fetch="git://github.com/meta-rust"  name="rust"/>
 
+  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
 
+  <remote fetch="https://github.com/Freescale" name="freescale"/>
 
+  <remote fetch="https://git.openembedded.org" name="oe"/>
 
+  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
 
+  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
 
+  <remote fetch="https://github.com/meta-rust"  name="rust"/>
 
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
 
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
 
  
</syntaxhighlight>
+
== Workaround ==
  
== Layer recipes ==
+
As an workaround for the <code>git://github.com/</code> access, it is possible to replace the ''git'' access to the ''https'' access using the [https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf git configuration] command:
While most of Yocto recipes used tar (or similar) archive to get source code, some of them download source code using git.
 
 
 
Usually <code>https</code> protocol is used for this, but in some cases <code>SRC_URI</code> may need to be updated if plain <code>git</code> protocol is used (and if the git server implements GitHub like security policies)
 
 
 
== Workaround ==
 
 
As an workaround for the <code><nowiki>git://github.com/</nowiki></code> access, it is possible to replace the ''git'' protocol to ''https'' using the [https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf git configuration] command:
 
  
 
<pre>
 
<pre>
Line 168: Line 131:
 
</nowiki>
 
</nowiki>
 
</pre>
 
</pre>
 +
 
See more info [https://stackoverflow.com/questions/1722807/how-to-convert-git-urls-to-http-urls here]
 
See more info [https://stackoverflow.com/questions/1722807/how-to-convert-git-urls-to-http-urls here]

Revision as of 13:12, 22 April 2022

Info Box
Yocto-logo.png Applies to Yocto


History[edit | edit source]

Version Date Notes
1.0.0 Apr 2022 First public release

Introduction[edit | edit source]

Recently, some months ago, there was a policy change on accessing git repositories for security issues. The Github security access has been changed for accessing the git repositories via ssh: more information can be found in the following news Improving Git protocol security on GitHub.

As reported :

We’re changing which keys are supported in SSH and removing unencrypted Git protocol. Only users connecting via SSH or git:// will be affected. If your Git remotes start with https://, nothing in this post will affect you. If you’re an SSH user, read on for the details and timeline. 

Since March 15, 2022 unencrypted git protocol has been disabled:

Git protocol security on GitHub.png

To mitigate this issue, it will be required to change the protocol access to the repo starting with git://

This Technical Note shows an example on how to modify repo manifest to correctly run the Yocto build. Some tips & tricks can be found also in this stackoverflow question.

Yocto Manifest[edit | edit source]

Before starting the Yocto, build it is required to check the default.xml manifest file for the git access to the layers repositories.

Here below there is an example using the DESK-MX6-L-1.0.0 wiki page instructions for building the overall BSP.

Orginal default.xml[edit | edit source]

The original file, as per desk-mx6-l-1.0.1 tag, uses the git protocol for accessing the Yocto, freescale and Qt repositories:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
  <remote fetch="git://github.com/Freescale" name="freescale"/>
  <remote fetch="git://git.openembedded.org" name="oe"/>
  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="git://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

in this case, multiple access error will be prompted by repo sync:

dvdk@vagrant:~/yocto$ ./repo sync
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0remote: Counting objects: 101, done.        
remote: Compressing objects: 100% (99/99), done.        
remote: Total 101 (delta 27), reused 0 (delta 0)        
Receiving objects: 100% (101/101), 15.88 KiB | 7.94 MiB/s, done.
Resolving deltas: 100% (27/27), done.
From ssh://git.dave.eu/desk-mx-l/desk-mx-l-bsp
 * [new branch]      hardknott             -> DAVE/hardknott
 * [new branch]      sumo                  -> DAVE/sumo
 * [new tag]         desk-mx6-l-1.0.1      -> desk-mx6-l-1.0.1
 * [new tag]         desk-mx6-l-1.0.0      -> desk-mx6-l-1.0.0
 * [new tag]         desk-mx6-l-1.0.2      -> desk-mx6-l-1.0.2
 * [new tag]         desk-mx6-l-3.0.0      -> desk-mx6-l-3.0.0
 * [new tag]         desk-mx6ul-l-1.0.0    -> desk-mx6ul-l-1.0.0
 * [new tag]         desk-mx6ul-l-1.0.1    -> desk-mx6ul-l-1.0.1
 * [new tag]         desk-mx6ul-l-3.0.0    -> desk-mx6ul-l-3.0.0
 * [new tag]         desk-mx8m-l-2.0.0-rc2 -> desk-mx8m-l-2.0.0-rc2
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
fatal: remote error: 
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
100   145  100   145    0     0    147      0 --:--:-- --:--:-- --:--:--   147
100 15.5M  100 15.5M    0     0  2051k      0  0:00:07  0:00:07 --:--:-- 2660k
Receiving objects: 100% (182040/182040), 15.58 MiB | 37.81 MiB/s, done.
Resolving deltas: 100% (126131/126131), done.
From /home/dvdk/yocto/.repo/projects/sources/meta-fsl-bsp-release.git/clone.bundle
 * [new branch]      warrior-4.19.35-1.1.0     -> CAF/warrior-4.19.35-1.1.0
 * [new branch]      thud-4.19.35-1.0.0        -> CAF/thud-4.19.35-1.0.0
 * [new branch]      sumo-4.14.98-2.3.0        -> CAF/sumo-4.14.98-2.3.0
...
...

Fixed default.xml[edit | edit source]

The modified file, as per desk-mx6-l-1.0.2 tag, uses the https protocol for accessing the Yocto, freescale and Qt repositories:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
  <remote fetch="https://github.com/Freescale" name="freescale"/>
  <remote fetch="https://git.openembedded.org" name="oe"/>
  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="https://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

Other Yocto layers or recipes[edit | edit source]

This issue can be present also on other layers or recipes: in this case, the related git:// access should be changed or fixed.

Workaround[edit | edit source]

As an workaround for the git://github.com/ access, it is possible to replace the git access to the https access using the git configuration command:


 git config --global url."https://github.com/".insteadOf git://github.com/

See more info here