Definition - What does Stack Smashing mean?
Stack smashing is a form of vulnerability where the stack of a computer application or OS is forced to overflow. This may lead to subverting the program/system and crashing it.
A stack, a first-in last-out circuit, is a form of buffer holding intermediate results of operations within it. To simplify, stack smashing putting more data into a stack than its holding capacity. Skilled hackers can deliberately introduce excessive data into the stack. The excessive data might be stored in other stack variables, including the function return address. When the function returns, it jumps to the malicious code on the stack, which might corrupt the entire system. The adjacent data on the stack is affected and forces the program to crash.
Techopedia explains Stack Smashing
If the program affected by stack smashing accepts data from untrusted networks and runs with special privileges, it is a case of security vulnerability. If the buffer contains data provided by an untrusted user, the stack may be corrupted by injecting executable code into the program, thus gaining unauthorized access to a computer. An attacker can also overwrite control flow information stored in the stack.
As stack smashing has grown into a very serious vulnerability, certain technologies are implemented to overcome the stack smashing disaster. Stack buffer overflow protection changes the organization of data in the stack frame of a function call to include canary values. These values when destroyed indicate that a buffer preceding it in memory has been overflowed. Canary values monitor buffer overflows and are placed between the control data and the buffer on the stack. This ensures that a buffer overflow corrupts the canary first. A failed verification of canary data signifies an overflow in the stack. The three types of canary are Random, Terminator, and Random XOR.
The terminator canary is based on the fact that stack buffer overflow attack depends on string operations ending at terminators. Random canaries are generated randomly from an entropy gathering daemon, which prevents attackers from knowing values. Random canaries are generated at program initialization and stored in global variables. Random XOR canaries are random carriers that are XOR scrambled using control data. It is similar to random canaries except that the "read from stack method" to get the canary is complex. The hacker needs the canary, algorithm, and control data to produce the original canary. They protect against attacks involving overflowing buffers in a structure into pointers to change pointer to point at a piece of control data.