Debugging with an In-Circuit Emulator

This article presents various case studies that can aid in the understanding and use of in-circuit emulators.

Case Study 1

Every 24 hours or so the system crashes. After the first crash, the ICE was connected to the system, put into trace mode, and the program restarted. The next morning it was determined that the system had crashed some time during the night, and the trace buffer was inspected. The trace buffer showed that when a certain subroutine executed a RET instruction, it returned to not the caller, but to an unused portion of RAM memory and eventually executed an illegal instruction and crashed, which implied that somehow the stack had been corrupted. It was decided that any writes into the stack area of memory should generate a break. However, it was quickly realized that this would not work because PUSH, POP, RET, and similar instructions cause a legitimate write to the stack area, so the break would have to be more discriminating. It was then decided to set a break on a write to the stack area but only on MOV instructions, the rationale being that any direct move to the stack area would be a problem. This approach was successful and it was determined that the bounds of an array had been exceeded which resulted in writes to stack memory.

Case Study 2

Every hour or so one or more incoming RS-232 characters are lost. A serial line analyzer was connected to the line and it showed no loss of data, implying that the characters were getting into the system but were being dropped sometime after that. Further investigations implied that the problem was that the RS-232 isr was, under certain conditions, being blocked from executing fast enough to handle the interrupt. A test was set up in which the system was sent a continuous stream of characters, and when a character was missed, a breakponit would be hit. The test was run with trace enabled. When the breakpoint was hit the trace buffer was inspected to see what activity was occurring prior to the RS-232 isr being entered. It was found that the network receive packet isr had taken too much time to process a particular packet, and had forced a data overrun condition on the RS-232 line. The problem was solved by running the RS-232 isr at a higher priority hardware interrupt than the receive network packet isr.

Case Study 3

After running for hours the system hangs. An ICE was connected and when the system again hung, the ICE was instructed to halt program execution. By inspecting the trace buffer it was found that a link in a linked list been corrupted, which resulted in a corrupted node in the list. Since each node contains a pointer to a function, the system halted on an illegal instruction exception when the function was called indirectly through the corrupted node. Evidently the memory pool of linked list nodes was being erroneously written into. The ICE was instructed to break whenever a memory access to the linked list node memory pool was made and to simulate a software interrupt to a specified address (this is a feature that not all ICE's may have) at which code was loaded (by downloading to the ICE) to traverse the linked list and check it for any bad links. If a bad link was found then the code would jump to a particular address at which a breakpoint was set. Although this technique compromised the real-time nature of the system (because the list was traversed on each access) it was still thought to be worth a try. The system was restarted, and the breakpoint was reached, indicating that the list was corrupted. To determine how the list was corrupted, the trace buffer was inspected for the most recent access to the linked list node memory pool. This lead to the fact that a pointer somehow got corrupted so that it erroneously pointed to the linked list memory, however, it was not clear from the trace buffer how the pointer had been corrupted. The ICE was then instructed to break when a value in the range of the linked list node memory pool was written into the pointer. When the breakpoint was hit it was seen that the pointer, which was a local stack variable, was being corrupted because the stack itself had been corrupted by an interrupt service routine which had a local variable declaration of "char buf[512]", which pushed the stack of the current thread beyond its bounds into the stack of the thread with the corrupted pointer, thus corrupting the pointer. (Some detail has been left out of this explanation; for full details please send us email.)

Summary and Comments

As the above case studies show, the ICE can find problems that would be very difficult to find otherwise. However, for varying reasons, an ICE is sometimes not available. The article "Debugging without an In-Circuit Emulator" presents some techniques that can be useful in this situation.

We welcome comments. Let us know what subjects you would like written up. Send comments to mike@Ticsrealtime.com