ML-TN-003 — AI at the edge: visual inspection of assembled PCBs for defect detection — Part 3

Info Box Applies to Machine Learning Applies to Machine Learning Technical Notes

History

Version	Date	Notes
1.0.0	March 2021	First public release

Building the dataset

Defects generation and acquisition

For this first attempt, 5 panels are prepared, each one containing 4 PCBs of the same type and with same template project i.e. the bill of materials (BOM) for the DIVA series 2. In this case, it has been decided to mount components only on top side of the PCB. Furthermore, in order to reduce complexity and simplify the problem for this test, anomalies generation is restricted to 2 contacts SMD passive components i.e. resistors, capacitors, and inductors. The table below reports all plausible anomalies that can be generated by editing the standard template project containing all the positions of the components for the P&P machine, decreasing or increasing the quantity of solder deposited by the serigraphy on the panel or that can be generated directly by the operator.

Anomalies, generation process and numerosity

Anomaly	P&P	Serigraphy	Manual	Numerosity
Missing	✓			21
Manhattan			✓	4
Shift x-axis	✓			51
Shift y-axis	✓			58
Shift&Rotation (shift x+z-axes)	✓			57
Rotation (z-axis)	✓			55
Under soldering		✓		All
Over soldering		✓		All

Following it is reported a brief description for all the anomalies:

• Missing: the component is not in place according to the PCB design and indeed is not mounted on the board.
• Manhattan: the component is in place but is erected horizontally (the operation has to be performed manually by the operator).
• Shift x-axis: the component is in place according to the PCB design but shows a small shift along its x-axis.
• Shift y-axis: the component is in place according to the PCB design but shows a small shift along its y-axis.
• Shift&Rotation (x+z-axes): the component is in place according to the PCB design but is shifted along its x-axis and rotated along its z-axis.
• Rotation (z-axis): the component is in place according to the PCB design but is rotated along its z-axis.
• Under soldering: the amount of deposited solder is higher than normal.
• Over soldering: the amount of deposited solder is less than normal.

The same template for the P&P machine is used for all the 5 panels, but some of them use a different configuration for the program of the serigraph machine:

• 2 panels use the normal amount of solder paste;
• 1 panel uses more than the normal amount of solder. In this case solder is re-applied several times;
• 1 panel uses less than the normal amount of solder;
• 1 panel uses the normal amount of solder, but shifts along x and y axes are applied too.

After completing the assembly of all 5 panels, a visual inspection was performed with the AOI machine. A total amount of 832 anomalies were found and the corresponding unmarked images were saved for dataset building.

Class subdivision and labelling

In order to build a dataset for training a ML model for a classification application, a set of classes has to be defined. By looking at the standard IPC-A-610E-2010 developed by IPC for the Acceptability of Electronic Assemblies, and at the features of collected images, the classes identified are the following ones:

• Acceptable: the AOI machine signals the component as a possible anomaly because the component image doesn’t respect the color constraints specified by the software of the machine but in truth, there is no defect. Generally, this occurs when the component is correctly soldered on both pads, but the amount of red and green color is too high with respect to the amount of blue. In this case, quality is not the target one but still is acceptable.
• Missing: the component is not in place, hence only the pads with applied solder are visible.
• Tombstoning: the component is lifted from a pad of the PCB; this class also comprehends all the cases for which the component is lifted and rotated by a certain amount.
• Under soldering: the component is soldered on both pads, but the amount of solder is too low. By looking at picture of the reported component, this is clearly visible when on a pad or both there is a higher amount of red with respect to the blue one.

To simplify the problem, all the classes into which the images are divided are mutually exclusive. manhattan and over soldering defect typologies are no longer included among the possible classes because their generation is too difficult hence the quantity of examples obtained is too low for training a model.

Examples of four types of defects

For labeling the images makesense−ai tool was used.

Total number of samples for each defect category

Acceptable	Missing	Tombstoning	Under soldering
17	408	376	24

It is evident that this dataset is unbalanced because it has a relatively low number of acceptable and under soldering defect images. Nevertheless, taking into account the overall results achieved, this first attempt can be considered a successful one.

Data augmentation with image synthesis

Progressive GAN implementation

ProGAN: training progression

ProGAN: growing progression of the model during training

Results validation

A total of 6 proGAN models were built and trained, each one with the required number of straight-through and fade-in model stages, to generate several typologies of synthesized images at the desired target resolution, belonging to missing and tombstoning classes, more specifically full images (512 × 512 resolution), upper and lower soldering region images (256 × 256 resolution). The training was executed mainly on cloud, initially with the free services provided by Google Colab and finally on AWS SageMaker.

ProGANs training time on Google Colab and AWS SageMaker

Class	Synth image	Resolution (pixel)	Google Colab (min)	AWS SageMaker (min)
missing	full	512 × 512	~480	~410
	upper/lower	256 × 256	~435	~310
tombstoning	full	512 × 512	~460	~390
	upper/lower	256 × 256	~420	~300

In the two figures below there are shown some examples taken from the generated sets of missing and tombstoning classes. For each image, from left to right, there are displayed samples of 512×512 and 256×256 resolution full typology and further on the right, from top to bottom, 256×256 upper and lower soldering region typology.

Synthesized images for missing class

Synthesized images for tombstoning class

The synthesized images can be effectively used to train a SMC defects classification model for a future ML-based PCB-AVI application, only if they have a similar probability distribution function with respect to the original ones. In particular, it is interesting to verify if there is a clear separation in the data. To this end, it is possible to employ the t-distributed stochastic neighbor embedding (t-SNE), which is a ML algorithm for visualization, based on Stochastic Neighbor Embedding (SNE) algorithm. Since the way this algorithm operates is computationally expensive, it is highly recommended to use another dimensionality reduction technique before applying t-SNE algorithm, in order to reduce the number of dimensions to a reasonable amount. For this purpose, it is possible to employ an autoencoder (AE).

IMMAGINE AUTOENCODER 5 × 10−4

The values obtained by applying the t-SNE algorithm on the compressed data are displayed into several 3D-plots reported below respectively for missing and tombstoning synthesized images. Interestingly, the 4 plots related to full typology do not show a separation in the data between fakes (red dots) and reals (blue dots), sign that the generated images do not differ too much from the original ones, used to train the proGANs.

T-SNE algorithm results for missing class synthesized images

512 × 512 resolution full images

256 × 256 resolution full images

256 × 256 resolution upper and lower region images

T-SNE algorithm results for tombstoning class synthesized images

512 × 512 resolution full images

256 × 256 resolution full images

256 × 256 resolution upper and lower region images

Useful links

• Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, Progressive Growing of GANs for Improved Quality, Stability, and Variation, February 2018.
• Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville, Improved Training of Wasserstein GANs, December 2017.
• IPC-A-610E: Acceptability of Electronic Assemblies, a standard developed by IPC, April 2010.