shared irq handler code assumes that a chain of handlers is only used by one core. #730

New issue

Open

opened 2022-02-10 11:23:41 -06:00 by kilograham · 1 comment

kilograham commented

2022-02-10 11:23:41 -06:00

(Migrated from github.com)

There are some race conditions in irq.c that are likely broken if a shared IRQ handler is removed from the the chain by one core when the other core happens to be at a particular point in the invocation of handlers from the same chain.

In practice, i don't believe anyone including me has ever seen this, and indeed it is likely a rare case that anyone actually hooks the enables the same IRQ on both cores. and perhaps even less likely that they remove IRQ handlers.

Still i'm making the issue for reference as I do know the problem exists. Sadly the fix is non-trivial.

These IRQ handler chains are rooted in the core's vector table, and therefore are to some extent local to the core, however today (though maybe not at the time) the VTOR is by default shared between the two cores. There are some race conditions in irq.c that are likely broken if a shared IRQ handler is removed from the the chain by one core when the other core happens to be at a particular point in the invocation of handlers from the same chain. In practice, i don't believe anyone including me has ever seen this, and indeed it is likely a rare case that anyone actually hooks the enables the same IRQ on both cores. and perhaps even less likely that they remove IRQ handlers. Still i'm making the issue for reference as I do know the problem exists. Sadly the fix is non-trivial.

akohlsmith commented

2026-01-24 16:59:14 -06:00

(Migrated from github.com)

Interesting.

I am currently running into an issue where I am trying to use the doorbell interface to signal the other core when a core hits an exception. Once the exception handler finishes recording state of the crashed core, it executes multicore_doorbell_set_other_core(). My doorbell handler is very simple: it triggers an NMI on the non-crashing core which I use to save state:

static void doorbell_handler(void)
{
	if (crash_doorbell_num >= 0 && multicore_doorbell_is_set_current_core(crash_doorbell_num)) {
		scb_hw->icsr |= (1 << 31);
	}
}

(yes I know I can do the state recording of the non-crashing-core in the doorbell handler, this is just how I've got it set up for now)

I'm intending on using this with FreeRTOS, and FreeRTOS also uses a doorbell bit for its own inter-core signaling. Its doorbell handler is also very small and looks to do the right thing:

static void prvDoorbellInterruptHandler()
{
	if (cDoorbellNum >= 0 && multicore_doorbell_is_set_current_core(cDoorbellNum))
	{
		multicore_doorbell_clear_current_core(cDoorbellNum);
		#if ( configNUMBER_OF_CORES != 1 )
			portYIELD_FROM_ISR( pdTRUE );
		#elif ( configSUPPORT_PICO_SYNC_INTEROP == 1 )
			BaseType_t xHigherPriorityTaskWoken = pdFALSE;
			uint32_t ulSave = spin_lock_blocking( pxCrossCoreSpinLock );
			EventBits_t ulBits = uxCrossCoreEventBits;
			uxCrossCoreEventBits = 0;
			spin_unlock( pxCrossCoreSpinLock, ulSave );
			xEventGroupSetBitsFromISR( xEventGroup, ulBits, &xHigherPriorityTaskWoken );
			portYIELD_FROM_ISR( xHigherPriorityTaskWoken );
		#endif /* configNUMBER_OF_CORES != 1 */
	}
}

As of today (Jan 2026) FreeRTOS uses irq_set_exclusive_handler() to assign the doorbell interrupt. I changed it to irq_add_shared_handler() and as long as I don't install my own shared handler, FreeRTOS works fine. However the momentn I add my shared handler, FreeRTOS runs at approximately 1/2 to 3/8 speed, even if I never trigger my own doorbell.

I'm currently setting both of the shared handler priorities to PICO_SHARED_IRQ_HANDLER_DEFAULT_ORDER_PRIORITY, but have also experimented with setting the priority of one to be higher (or lower) than the other but without any change in the outcome.

If I do not install my own shared handler but literally copy my code into FreeRTOS' doorbell handler, it works great, so it's not a code issue in either FreeRTOS or my own handlers, it's gotta be something with how the shared handler code is doing things. I wonder if the shared IRQ handler code is doing something goofy when it's possible that the IRQ is being triggered on different cores.

Interesting. I am currently running into an issue where I am trying to use the doorbell interface to signal the other core when a core hits an exception. Once the exception handler finishes recording state of the crashed core, it executes `multicore_doorbell_set_other_core()`. My doorbell handler is very simple: it triggers an NMI on the non-crashing core which I use to save state: ```c static void doorbell_handler(void) { if (crash_doorbell_num >= 0 && multicore_doorbell_is_set_current_core(crash_doorbell_num)) { scb_hw->icsr |= (1 << 31); } } ``` (yes I know I can do the state recording of the non-crashing-core in the doorbell handler, this is just how I've got it set up for now) I'm intending on using this with FreeRTOS, and FreeRTOS also uses a doorbell bit for its own inter-core signaling. Its doorbell handler is also very small and looks to do the right thing: ```c static void prvDoorbellInterruptHandler() { if (cDoorbellNum >= 0 && multicore_doorbell_is_set_current_core(cDoorbellNum)) { multicore_doorbell_clear_current_core(cDoorbellNum); #if ( configNUMBER_OF_CORES != 1 ) portYIELD_FROM_ISR( pdTRUE ); #elif ( configSUPPORT_PICO_SYNC_INTEROP == 1 ) BaseType_t xHigherPriorityTaskWoken = pdFALSE; uint32_t ulSave = spin_lock_blocking( pxCrossCoreSpinLock ); EventBits_t ulBits = uxCrossCoreEventBits; uxCrossCoreEventBits = 0; spin_unlock( pxCrossCoreSpinLock, ulSave ); xEventGroupSetBitsFromISR( xEventGroup, ulBits, &xHigherPriorityTaskWoken ); portYIELD_FROM_ISR( xHigherPriorityTaskWoken ); #endif /* configNUMBER_OF_CORES != 1 */ } } ``` As of today (Jan 2026) FreeRTOS uses `irq_set_exclusive_handler()` to assign the doorbell interrupt. I changed it to `irq_add_shared_handler()` and as long as I don't install my own shared handler, FreeRTOS works fine. However the momentn I add my shared handler, FreeRTOS runs at approximately 1/2 to 3/8 speed, even if I never trigger my own doorbell. I'm currently setting both of the shared handler priorities to `PICO_SHARED_IRQ_HANDLER_DEFAULT_ORDER_PRIORITY`, but have also experimented with setting the priority of one to be higher (or lower) than the other but without any change in the outcome. If I do not install my own shared handler but literally copy my code into FreeRTOS' doorbell handler, it works great, so it's not a code issue in either FreeRTOS or my own handlers, it's gotta be something with how the shared handler code is doing things. I wonder if the shared IRQ handler code is doing something goofy when it's possible that the IRQ is being triggered on different cores.