In a device with STM32F407, system clock is derived from a 25MHz crystal connected to HSE. This in PLL is divided to 1MHz, multiplied by 336 and then post-divided by 2 to provide the 168MHz system clock. This is then subdivided and distributed throughout the chip as usually; among other modules also to timers driving several LEDs with PWM at 260Hz.

In one specimen of the device, a couple of minutes after having passed an initial test and been programmed by the final application, all LEDs started flashing violently and synchronously at a frequency of cca 13Hz. The whole functionality of the device was off, indicating, that it run at a significantly lower system frequency. As the PWM should have been 260Hz and LEDs were blinking at around 13Hz, system frequency at that point was probably around 8MHz instead of 168MHz.

Why? Solution below.

Investigation revealed, that in given specimen, the 25MHz crystal was mechanically damaged (this probably had happened when board had been populated at a contractor), and as the device has been moved around the test positions, it ceased to oscillate.

The surprising thing in this was, that even if there was no HSE oscillation, the system kept running at a surprisingly low frequency (would it have switched to HSI, it would run at around 16MHz, but the observed system frequency was significantly lower). This was probably consequence of using PLL, where output of the phase detector probably went to one of its extremes and the VCO responded by going at its lowest frequency. It is rather surprising that the VCO which nominally operates between 100MHz and 432MHz, can oscillate (albeit in unspecified mode) at as low as 16MHz (PLL had a post-divider of 2).

The STM32F407 does have CSS (Clock Security System) which detects HSE oscillation and in case of failure it switches to HSI and throws an unmaskable interrupt (NMI); however, operation of this device is not critical and it also does not have any reasonable means of backup operation, so the solution in this case was simply exchanging the faulty crystal for a new one.