Homework 1

CS155, Spring 2007

Due: Thursday, Apr 26

Problem 1

(a) In class we described how LibSafe defends against stack buffer overflows. Give a short sample C code that is vulnerable to buffer overflows even though LibSafe is used.

(b) Can you adapt the LibSafe mechanism to work in the heap? If so explain how and what additional functionality you would need from the memory manager. If not, explain why not.

(c) Suppose the processor stack grows upwards (as opposed to downwards as on the x86). Give sample C code that is vulnerable to a buffer overflow attack and briefly explain how the attack works.

Clarification (posted 4/23/2007): You don't need to assume that LibSafe is used for part (c).

Problem 2

Consider the following code snippet:

  if (!stat("./file.dat", buf)) return;   // Abort if file exists.
  sleep(10);                              // Sleep for 10 seconds.
  fp = fopen("./file.dat", "w" );
  fprintf(fp, "Hello world" );

a. Suppose this code is running as a setuid root program. Give an example of how this code can lead to unexpected behavior that could cause a security problem. (try using symbolic links.)

b. Suppose the sleep(10) is removed from the code above. Could the problem you identified in part (a) still occur? Please explain.

c. How would you fix the code to prevent the problem from part (a)?

Problem 3

Recall that modern processors have an on chip data cache (called an L2 cache) used to reduce the number of accesses to main memory. Reading from the L2 cache is about 10 times faster than reading from main memory. Explain how the L2 processor cache can result in a covert channel on the local system. Give a sample short (pseudo) code to exploit this channel.
Hint: use memory access timing information.

Clarification (posted 4/23/2007): Assume that processes have separate address spaces (no shared memory), although they share the underlying hardware. You don't need to worry about making the covert channel fast.

Problem 4

One of the complications of sandboxing x86 (as opposed to RISC) code is that the variable-length x86 instructions make it hard for a code verifier to parse the binary code. Suppose, instead, that sandboxed code were shipped around as ASCII assembly language source. In this scheme, the verifier would check the assembly language source. In particular it ensures that register %edx is only used for segment matching, and that every single instruction that modifies memory either:

  1. is relative to the stack or frame pointer (with a small enough offset to stay within guard zones), or
  2. uses the %edx register for segment matching with the following code:
             mov DEST, %edx
             bound %edx, data_bounds
             INST SRC, (%edx)

    Here DEST is the memory location that is being written, data_bounds is the location of pointers to the beginning and end of the fault domain's data segment, the bound instruction traps if %edx is not within that range, and INST and SRC can be any instruction and source that modify the memory pointed to by DEST.

Explain how, even with this scheme, malicious code could escape the sandbox and modify other regions of a process's address space.

Clarification (posted 4/23/2007): Assume that the verifier allows relative branches to any byte within the fault domain's code segment, but that control transfers outside the code segment use a jump table that vectors to stubs not in the actual code segment.

Problem 5

(a) Recall that a polymorphic virus uses an evolving decryption engine and a fixed (encrypted) virus body. Anti virus software detects polymorphic viruses by first emulating the executable in question for a short time and then looking for a virus signature in memory. Explain how polymorphic viruses can evade this detection technique. Give at least two examples.

Clarification (posted 4/23/2007): The two examples you provide should be examples of how polymorphic viruses can evade this detection technique (not names of real viruses). Your examples should be examples of polymorphic behavior (not metamorphic).

(b) Can you suggest a more robust mechanism for preventing polymorphic viruses from spreading? You may assume that the anti-virus vendor already obtained a copy of the virus and fully analyzed how it works.