Difference between revisions of "ML-TN-003 — AI at the edge: visual inspection of assembled PCBs for defect detection — Part 3"
| Line 146: | Line 146: | ||
===Generative adversarial networks=== | ===Generative adversarial networks=== | ||
===Progressive GAN implementation=== | ===Progressive GAN implementation=== | ||
| + | When using proGANs to synthesize images at high resolutions, instead of attempting to train all layers of the generator and discriminator at once to generate samples at the target resolution as it is usually done, the networks are initially created with a block containing only a bunch of layers and are progressively grown to output higher resolution versions of the images by adding more and more blocks, one at a time, after completing the training of the previous one, as illustrated in the figure below. | ||
| + | |||
[[File:Training progression.png|center|thumb|500x500px|ProGAN: training progression]] | [[File:Training progression.png|center|thumb|500x500px|ProGAN: training progression]] | ||
| + | |||
| + | This approach leads to a series of advantages: | ||
| + | *the incremental learning process greatly stabilizes training and reduces the chance of mode collapse, since the networks gradually learn a much simple piece of the overall problem. | ||
| + | *the low-to-high resolution trend forces the progressively grown networks to focus on high-level structure first and fill in the details later, resulting in an improvement of the quality of the final images | ||
| + | *increasing the network size gradually is more computationally efficient w.r.t. the classic approach of using all the layers from the start (fewer layers are faster to train because there are fewer parameters). | ||
[[File:Fade-in progression.png|center|thumb|500x500px|ProGAN: growing progression of the model during training]] | [[File:Fade-in progression.png|center|thumb|500x500px|ProGAN: growing progression of the model during training]] | ||
Revision as of 14:31, 15 April 2021
| HOME | SOMs | SBCs | ToloMEO Embedded Assistant |  GET A QUOTE
|
 ONLINE HELPDESK
|
| Roadmap | IoT Services | ML/AI services | Embedded Design Services |
Info Box
|
Contents
History[edit | edit source]
| Version | Date | Notes |
|---|---|---|
| 1.0.0 | March 2021 | First public release |
Introduction[edit | edit source]
Building the dataset[edit | edit source]
Defects generation and acquisition[edit | edit source]
For this first attempt, 5 panels are prepared, each one containing 4 PCBs of the same type and with same template project i.e. the bill of materials (BOM) for the DIVA series 2. In this case, it has been decided to mount components only on top side of the PCB. Furthermore, in order to reduce complexity and simplify the problem for this test, anomalies generation is restricted to 2 contacts SMD passive components i.e. resistors, capacitors, and inductors. The table below reports all plausible anomalies that can be generated by editing the standard template project containing all the positions of the components for the P&P machine, decreasing or increasing the quantity of solder deposited by the serigraphy on the panel or that can be generated directly by the operator.
| Anomaly | P&P | Serigraphy | Manual | Numerosity |
|---|---|---|---|---|
| Missing | ✓ | 21 | ||
| Manhattan | ✓ | 4 | ||
| Shift x-axis | ✓ | 51 | ||
| Shift y-axis | ✓ | 58 | ||
| Shift&Rotation (shift x+z-axes) |
✓ | 57 | ||
| Rotation (z-axis) |
✓ |
55 | ||
| Under soldering | ✓ | All | ||
| Over soldering | ✓ | All |
Following it is reported a brief description for all the anomalies:
- Missing: the component is not in place according to the PCB design and indeed is not mounted on the board.
- Manhattan: the component is in place but is erected horizontally (the operation has to be performed manually by the operator).
- Shift x-axis: the component is in place according to the PCB design but shows a small shift along its x-axis.
- Shift y-axis: the component is in place according to the PCB design but shows a small shift along its y-axis.
- Shift&Rotation (x+z-axes): the component is in place according to the PCB design but is shifted along its x-axis and rotated along its z-axis.
- Rotation (z-axis): the component is in place according to the PCB design but is rotated along its z-axis.
- Under soldering: the amount of deposited solder is higher than normal.
- Over soldering: the amount of deposited solder is less than normal.
The same template for the P&P machine is used for all the 5 panels, but some of them use a different configuration for the program of the serigraph machine:
- 2 panels use the normal amount of solder paste;
- 1 panel uses more than the normal amount of solder. In this case solder is re-applied several times;
- 1 panel uses less than the normal amount of solder;
- 1 panel uses the normal amount of solder, but shifts along x and y axes are applied too.
After completing the assembly of all 5 panels, a visual inspection was performed with the AOI machine. A total amount of 832 anomalies were found and the corresponding unmarked images were saved for dataset building.
Class subdivision and labelling[edit | edit source]
In order to build a dataset for training a ML model for a classification application, a set of classes has to be defined. By looking at the standard IPC-A-610E-2010 developed by IPC for the Acceptability of Electronic Assemblies, and at the features of collected images, the classes identified are the following ones:
- Acceptable: the AOI machine signals the component as a possible anomaly because the component image doesn’t respect the color constraints specified by the software of the machine but in truth, there is no defect. Generally, this occurs when the component is correctly soldered on both pads, but the amount of red and green color is too high with respect to the amount of blue. In this case, quality is not the target one but still is acceptable.
- Missing: the component is not in place, hence only the pads with applied solder are visible.
- Tombstoning: the component is lifted from a pad of the PCB; this class also comprehends all the cases for which the component is lifted and rotated by a certain amount.
- Under soldering: the component is soldered on both pads, but the amount of solder is too low. By looking at picture of the reported component, this is clearly visible when on a pad or both there is a higher amount of red with respect to the blue one.
To simplify the problem, all the classes into which the images are divided are mutually exclusive. manhattan and over soldering defect typologies are no longer included among the possible classes because their generation is too difficult hence the quantity of examples obtained is too low for training a model.
For labeling the images makesense−ai tool was used.
| Acceptable | Missing | Tombstoning | Under soldering |
|---|---|---|---|
| 17 | 408 | 376 | 24 |
It is evident that this dataset is unbalanced because it has a relatively low number of acceptable and under soldering defect images. Nevertheless, taking into account the overall results achieved, this first attempt can be considered a successful one.
Soldering regions extraction[edit | edit source]
By looking at the acquired images, it is interesting to note that most defect features are distributed in the solder region in the component and in particular two defects belonging to two different classes can be easily distinguished by looking at the two soldering regions. This means that both can be potentially used for training a ML model for a classification problem, while all the other parts in the image can be discarded without losing information and accuracy.
To address this issue a methodology was developed for extracting the soldering regions from all the collected images. The algorithm is designed to find the correct position of a rectangular window i.e. a region of interest (ROI) by employing an adaptive approach that uses OpenCV image processing functions. Note that this approach can be used for all the different classes of defects.
Data augmentation with image synthesis[edit | edit source]
Generative adversarial networks[edit | edit source]
Progressive GAN implementation[edit | edit source]
When using proGANs to synthesize images at high resolutions, instead of attempting to train all layers of the generator and discriminator at once to generate samples at the target resolution as it is usually done, the networks are initially created with a block containing only a bunch of layers and are progressively grown to output higher resolution versions of the images by adding more and more blocks, one at a time, after completing the training of the previous one, as illustrated in the figure below.
This approach leads to a series of advantages:
- the incremental learning process greatly stabilizes training and reduces the chance of mode collapse, since the networks gradually learn a much simple piece of the overall problem.
- the low-to-high resolution trend forces the progressively grown networks to focus on high-level structure first and fill in the details later, resulting in an improvement of the quality of the final images
- increasing the network size gradually is more computationally efficient w.r.t. the classic approach of using all the layers from the start (fewer layers are faster to train because there are fewer parameters).
Training configuration and hyperparameters setup[edit | edit source]
A total of 6 proGAN models were built and trained, each one with the required number of straight-through and fade-in model stages, to generate several typologies of synthesized images at the desired target resolution, belonging to missing and tombstoning classes, more specifically full images (512 × 512 resolution), upper and lower soldering region images (256 × 256 resolution). The training was executed mainly on cloud, initially with the free services provided by Google Colab and finally on AWS SageMaker.
Discriminators and generators were trained both with a learn rate of 5 × 10-4 for all fade-in stages and with a smaller learn rate of 1 × 10-4 for all straight-through stages, in order to guarantee a smooth and slow enough fine-tuning for all layers. Each stage of each created model was trained for 100 epochs, with the sole exception of the last stage for the 512×512 full images, which needed 50 more epochs to generate satisfying results. The batch size is progressively reduced as the resolution increases, starting from a batch of 64 images for the first three resolutions (4 × 4, 8 × 8 and 16 × 16 resolutions), decreasing to 32 (32 × 32, and 64 × 64 resolutions) and 16 (128 × 128 and 256 × 256 resolutions). In the last stage for full typology (512 × 512), batch size is further reduced to 8 images for each train step.
| Class | Synth image | Resolution (pixel) |
Google Colab (min) |
AWS SageMaker (min) |
|---|---|---|---|---|
missing |
full | 512 × 512 | ~480 | ~410 |
| upper/lower | 256 × 256 | ~435 | ~310 | |
tombstoning |
full | 512 × 512 | ~460 | ~390 |
| upper/lower | 256 × 256 | ~420 | ~300 |
Results validation[edit | edit source]
In the two figures below there are shown some examples taken from the generated sets of missing and tombstoning classes. For each image, from left to right, there are displayed samples of 512×512 and 256×256 resolution full typology and further on the right, from top to bottom, 256×256 upper and lower soldering region typology.
The synthesized images can be effectively used to train a SMC defects classification model for a future ML-based PCB-AVI application, only if they have a similar probability distribution function with respect to the original ones. In particular, it is interesting to verify if there is a clear separation in the data. To this end, it is possible to employ the t-distributed stochastic neighbor embedding (t-SNE), which is a ML algorithm for visualization, based on Stochastic Neighbor Embedding (SNE) algorithm. Since the way this algorithm operates is computationally expensive, it is highly recommended to use another dimensionality reduction technique before applying t-SNE algorithm, in order to reduce the number of dimensions to a reasonable amount. For this purpose, it is possible to employ an autoencoder (AE).
The two soldering regions are the most important parts concerning the generated images and, by using the methodology proposed in the previous section, they can be easily extracted and evaluated too with the t-SNE algorithm. Since the two of them are also correlated for the same image, clearly strongly correlated for tombstoning class samples, they must be concurrently compressed by the AE. Therefore, a dual-stream autoencoder, with two input/output layers, is required.
The values obtained by applying the t-SNE algorithm on the compressed data are displayed into several 3D-plots reported below respectively for missing and tombstoning synthesized images. Interestingly, the 4 plots related to full typology do not show a separation in the data between fakes (red dots) and reals (blue dots), sign that the generated images do not differ too much from the original ones, used to train the proGANs.
The two figures to the utmost right report the plots showing the results of t-SNE on the compressed data, obtained from the dual stream encoder for missing and tombstoning synthesized images. In the first case, for missing class, there is actually no separation between fakes and real of upper and lower region sides, a good sign that the generated images are no very different from the original ones. This is quite understandable since the two regions look alike. The situation is visibly different in the plot for the tombstoning class, where two distinct clouds of dots can be seen, confirming a well defined separation between the data related to the upper and lower typologies. It is also possible to see that fakes and reals dots, belonging to the same soldering region, are very close to each other, while dots belonging to a different soldering region are distant in the plot, confirming once again that the image generation with the proGAN was successful.
Useful links[edit | edit source]
- Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, Progressive Growing of GANs for Improved Quality, Stability, and Variation, February 2018.
- Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville, Improved Training of Wasserstein GANs, December 2017.
- IPC-A-610E: Acceptability of Electronic Assemblies, a standard developed by IPC, April 2010.
