Difference between revisions of "MISC-TN-015: Yocto and git protocol error"

From DAVE Developer's Wiki
Jump to: navigation, search
(→‎Fixed default.xml)
 
(2 intermediate revisions by 2 users not shown)
Line 25: Line 25:
  
 
==Introduction==
 
==Introduction==
−
Recently, some months ago, there was a policy change on accessing git repositories for security issues. The Github security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
+
Starting from beginning of 2022, there was a policy change on most git hosting services to restrict "unsecure" git protocol for security issues.
 +
 
 +
GitHub security access has been changed for accessing the git repositories via ssh: more information can be found in the following news [https://github.blog/2021-09-01-improving-git-protocol-security-github/ Improving Git protocol security on GitHub].
  
 
As reported :
 
As reported :
Line 32: Line 34:
  
 
Since March 15, 2022 unencrypted git protocol has been disabled:
 
Since March 15, 2022 unencrypted git protocol has been disabled:
−
 
 
[[File:Git_protocol_security_on_GitHub.png | center|600px]]
 
[[File:Git_protocol_security_on_GitHub.png | center|600px]]
  
−
To mitigate this issue, it will be required to change the protocol access to the repo starting with <code>git://</code>
+
This is already applied in DAVE BSPs published after March 2022 but older BSPs, published prior this change, may need to be fixed manually to use ssh instead of <code>git://</code> to access some repositories
  
−
This Technical Note shows an example on how to modify <code>repo</code> manifest to correctly run the Yocto build. Some ''tips & tricks'' can be found also in this [https://stackoverflow.com/questions/70663523/the-unauthenticated-git-protocol-on-port-9418-is-no-longer-supported stackoverflow] question.
+
This Technical Note shows an example on how to update those older releases to solve this issue
  
−
== Yocto Manifest ==
+
== BSP repo Manifest ==
−
Before starting the Yocto, build it is required to check the <code>default.xml</code> manifest file for the git access to the layers repositories.
+
DAVE Yocto BSPs uses [https://gerrit.googlesource.com/git-repo/ repo] to track the multiple layers required to setup the BSP itself. If <code>git</code> protocol is used to clone some these layers, user will encounter the above issue.
  
−
Here below there is an example using the [https://wiki.dave.eu/index.php?title=DESK-MX6-L/Development/Building_the_Yocto_BSP&oldid=14300 DESK-MX6-L-1.0.0] wiki page instructions for building the overall BSP.
+
Here there is an example using the {{OldRevision|page=DESK-MX6-L/Development/Building the Yocto BSP|revision=14300|text=DESK-MX6-L-1.0.0}}page instructions for building the overall BSP while fixing the manifest
  
−
=== Orginal ''default.xml''===
+
=== Setup with original manifest===
−
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the '''git''' protocol for accessing the Yocto, freescale and Qt repositories:
+
The original file, as per <code>desk-mx6-l-1.0.1</code> tag, uses the <code>git</code> protocol to clone some layers:
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 63: Line 64:
 
...
 
...
 
</pre>
 
</pre>
−
in this case, multiple access error will be prompted by repo sync:
+
in this case, multiple access error will be prompted while running <code>repo sync</code>:
 
   
 
   
 
  dvdk@vagrant:~/yocto$ ./repo sync
 
  dvdk@vagrant:~/yocto$ ./repo sync
Line 99: Line 100:
 
  ...
 
  ...
  
−
=== Fixed ''default.xml''===
+
=== Fixing repo manifest===
−
The modified file, as per <code>desk-mx6-l-1.0.2</code> tag, uses the '''https''' protocol for accessing the git repositories:
+
<code>repo</code> stores the current manifest in <code>.repo/manifest.xml</code>: this is the file that needs to be patched to solve this issue locally
 +
 
 +
After <code>repo init</code> and prior <code>repo sync</code>, user need to edit <code>.repo/manifest.xml</code> to change all occurrence of <code>git://</code> protocol specifier to <code>https://</code>
 +
 
 +
This can be also applied with a simple <code>sed</code> command:<syntaxhighlight lang="bash">
 +
sed -i 's/git:\/\//https:\/\//g' .repo/manifest.xml
 +
</syntaxhighlight>
 +
Please note that, for <code>DESK-MX6-L-1.x.x</code>, this is already fixed in <code>desk-mx6-l-1.0.2</code> release:
 
<pre>
 
<pre>
 
<?xml version="1.0" encoding="UTF-8"?>
 
<?xml version="1.0" encoding="UTF-8"?>
Line 119: Line 127:
 
</pre>
 
</pre>
  
−
== Other Yocto layers or recipes ==
+
To be precise the <code>diff</code> between the two releases is:
−
This issue can be present also on other layers or recipes: in this case, the related <code>git://</code> access should be changed or fixed.
+
<syntaxhighlight lang="diff">
 +
diff --git a/default.xml b/default.xml
 +
index f4d7443..1a4d7fb 100644
 +
--- a/default.xml
 +
+++ b/default.xml
 +
@@ -3,12 +3,12 @@
 +
 +
  <default sync-j="2"/>
 +
 +
-  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
 +
-  <remote fetch="git://github.com/Freescale" name="freescale"/>
 +
-  <remote fetch="git://git.openembedded.org" name="oe"/>
 +
-  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
 +
-  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
 +
-  <remote fetch="git://github.com/meta-rust"  name="rust"/>
 +
+  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
 +
+  <remote fetch="https://github.com/Freescale" name="freescale"/>
 +
+  <remote fetch="https://git.openembedded.org" name="oe"/>
 +
+  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
 +
+  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
 +
+  <remote fetch="https://github.com/meta-rust"  name="rust"/>
 +
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
 +
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
 +
 
 +
</syntaxhighlight>
 +
 
 +
== Layer recipes ==
 +
While most of Yocto recipes used tar (or similar) archive to get source code, some of them download source code using git.
 +
 
 +
Usually <code>https</code> protocol is used for this, but in some cases <code>SRC_URI</code> may need to be updated if plain <code>git</code> protocol is used (and if the git server implements GitHub like security policies)
  
 
== Workaround ==
 
== Workaround ==
−
 
+
−
As an workaround for the <code><nowiki>git://github.com/</nowiki></code> access, it is possible to replace the ''git'' access to the ''https'' access using the [https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf git configuration] command:
+
As an workaround for the <code><nowiki>git://github.com/</nowiki></code> access, it is possible to replace the ''git'' protocol to ''https'' using the [https://git-scm.com/docs/git-config#Documentation/git-config.txt-urlltbasegtinsteadOf git configuration] command:
  
 
<pre>
 
<pre>
Line 131: Line 168:
 
</nowiki>
 
</nowiki>
 
</pre>
 
</pre>
−
 
 
See more info [https://stackoverflow.com/questions/1722807/how-to-convert-git-urls-to-http-urls here]
 
See more info [https://stackoverflow.com/questions/1722807/how-to-convert-git-urls-to-http-urls here]

Latest revision as of 14:56, 22 April 2022

HOME SOMs SBCs ToloMEO Embedded Assistant Sale-tag.pngGET A QUOTE Customer-service.pngONLINE HELPDESK
Roadmap IoT Services ML/AI services Embedded Design Services
Info Box
Yocto-logo.png Applies to Yocto


History[edit | edit source]

Version Date Notes
1.0.0 Apr 2022 First public release

Introduction[edit | edit source]

Starting from beginning of 2022, there was a policy change on most git hosting services to restrict "unsecure" git protocol for security issues.

GitHub security access has been changed for accessing the git repositories via ssh: more information can be found in the following news Improving Git protocol security on GitHub.

As reported : 

We’re changing which keys are supported in SSH and removing unencrypted Git protocol. Only users connecting via SSH or git:// will be affected. If your Git remotes start with https://, nothing in this post will affect you. If you’re an SSH user, read on for the details and timeline.

Since March 15, 2022 unencrypted git protocol has been disabled:

Git protocol security on GitHub.png

This is already applied in DAVE BSPs published after March 2022 but older BSPs, published prior this change, may need to be fixed manually to use ssh instead of git:// to access some repositories

This Technical Note shows an example on how to update those older releases to solve this issue

BSP repo Manifest[edit | edit source]

DAVE Yocto BSPs uses repo to track the multiple layers required to setup the BSP itself. If git protocol is used to clone some these layers, user will encounter the above issue.

Here there is an example using the DESK-MX6-L-1.0.0 page instructions for building the overall BSP while fixing the manifest

Setup with original manifest[edit | edit source]

The original file, as per desk-mx6-l-1.0.1 tag, uses the git protocol to clone some layers: 

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
  <remote fetch="git://github.com/Freescale" name="freescale"/>
  <remote fetch="git://git.openembedded.org" name="oe"/>
  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="git://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

in this case, multiple access error will be prompted while running repo sync: 

dvdk@vagrant:~/yocto$ ./repo sync
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0remote: Counting objects: 101, done.        
remote: Compressing objects: 100% (99/99), done.        
remote: Total 101 (delta 27), reused 0 (delta 0)        
Receiving objects: 100% (101/101), 15.88 KiB | 7.94 MiB/s, done.
Resolving deltas: 100% (27/27), done.
From ssh://git.dave.eu/desk-mx-l/desk-mx-l-bsp
 * [new branch]      hardknott             -> DAVE/hardknott
 * [new branch]      sumo                  -> DAVE/sumo
 * [new tag]         desk-mx6-l-1.0.1      -> desk-mx6-l-1.0.1
 * [new tag]         desk-mx6-l-1.0.0      -> desk-mx6-l-1.0.0
 * [new tag]         desk-mx6-l-1.0.2      -> desk-mx6-l-1.0.2
 * [new tag]         desk-mx6-l-3.0.0      -> desk-mx6-l-3.0.0
 * [new tag]         desk-mx6ul-l-1.0.0    -> desk-mx6ul-l-1.0.0
 * [new tag]         desk-mx6ul-l-1.0.1    -> desk-mx6ul-l-1.0.1
 * [new tag]         desk-mx6ul-l-3.0.0    -> desk-mx6ul-l-3.0.0
 * [new tag]         desk-mx8m-l-2.0.0-rc2 -> desk-mx8m-l-2.0.0-rc2
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
fatal: remote error: 
  The unauthenticated git protocol on port 9418 is no longer supported.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
100   145  100   145    0     0    147      0 --:--:-- --:--:-- --:--:--   147
100 15.5M  100 15.5M    0     0  2051k      0  0:00:07  0:00:07 --:--:-- 2660k
Receiving objects: 100% (182040/182040), 15.58 MiB | 37.81 MiB/s, done.
Resolving deltas: 100% (126131/126131), done.
From /home/dvdk/yocto/.repo/projects/sources/meta-fsl-bsp-release.git/clone.bundle
 * [new branch]      warrior-4.19.35-1.1.0     -> CAF/warrior-4.19.35-1.1.0
 * [new branch]      thud-4.19.35-1.0.0        -> CAF/thud-4.19.35-1.0.0
 * [new branch]      sumo-4.14.98-2.3.0        -> CAF/sumo-4.14.98-2.3.0
...
...

Fixing repo manifest[edit | edit source]

repo stores the current manifest in .repo/manifest.xml: this is the file that needs to be patched to solve this issue locally

After repo init and prior repo sync, user need to edit .repo/manifest.xml to change all occurrence of git:// protocol specifier to https://

This can be also applied with a simple sed command:

sed -i 's/git:\/\//https:\/\//g' .repo/manifest.xml

Please note that, for DESK-MX6-L-1.x.x, this is already fixed in desk-mx6-l-1.0.2 release: 

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="2"/>

  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
  <remote fetch="https://github.com/Freescale" name="freescale"/>
  <remote fetch="https://git.openembedded.org" name="oe"/>
  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
  <remote fetch="https://github.com/meta-rust"  name="rust"/>
  <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
  <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>
...
...

To be precise the diff between the two releases is: 

diff --git a/default.xml b/default.xml
index f4d7443..1a4d7fb 100644
--- a/default.xml
+++ b/default.xml
@@ -3,12 +3,12 @@
 
   <default sync-j="2"/>
 
-  <remote fetch="git://git.yoctoproject.org" name="yocto"/>
-  <remote fetch="git://github.com/Freescale" name="freescale"/>
-  <remote fetch="git://git.openembedded.org" name="oe"/>
-  <remote fetch="git://github.com/OSSystems" name="OSSystems"/>
-  <remote fetch="git://github.com/meta-qt5"  name="QT5"/>
-  <remote fetch="git://github.com/meta-rust"  name="rust"/>
+  <remote fetch="https://git.yoctoproject.org" name="yocto"/>
+  <remote fetch="https://github.com/Freescale" name="freescale"/>
+  <remote fetch="https://git.openembedded.org" name="oe"/>
+  <remote fetch="https://github.com/OSSystems" name="OSSystems"/>
+  <remote fetch="https://github.com/meta-qt5"  name="QT5"/>
+  <remote fetch="https://github.com/meta-rust"  name="rust"/>
   <remote fetch="https://source.codeaurora.org/external/imx" name="CAF" />
   <remote fetch="ssh://git@git.dave.eu/" name="DAVE"/>

Layer recipes[edit | edit source]

While most of Yocto recipes used tar (or similar) archive to get source code, some of them download source code using git.

Usually https protocol is used for this, but in some cases SRC_URI may need to be updated if plain git protocol is used (and if the git server implements GitHub like security policies)

Workaround[edit | edit source]

As an workaround for the git://github.com/ access, it is possible to replace the git protocol to https using the git configuration command: 


 git config --global url."https://github.com/".insteadOf git://github.com/

See more info here

HOME SOMs SBCs ToloMEO Embedded Assistant Sale-tag.pngGET A QUOTE Customer-service.pngONLINE HELPDESK
Roadmap IoT Services ML/AI services Embedded Design Services
Retrieved from "https://wiki.dave.eu/index.php?title=MISC-TN-015:_Yocto_and_git_protocol_error&oldid=16444"