Homework 1

CS255, Spring 2008

Due: Tuesday, Apr. 29

Problem 1

(b) Can you adapt the LibSafe mechanism to work in the heap? If so explain how and what additional functionality you would need from the memory manager. If not, explain why not.
Hint: it will help to review the inner workings of the heap memory manager used in malloc and free.

(c) Suppose the OS marks the stack memory pages as non-execute. Can a stack overflow be used to get a root shell on the machine? If so, briefly explain how. If not, explain why not.

Problem 2

Consider the following code snippet:

  if (!stat("./file.dat", buf)) return;   // Abort if file exists.
  sleep(10);                              // Sleep for 10 seconds.
  fp = fopen("./file.dat", "w" );
  fprintf(fp, "Hello world" );
  close(fp);

a. Suppose this code is running as a setuid root program. Give an example of how this code can lead to unexpected behavior that could cause a security problem. (try using symbolic links.)

b. Suppose the sleep(10) is removed from the code above. Could the problem you identified in part (a) still occur? Please explain.

c. How would you fix the code to prevent the problem from part (a)?

Problem 3

1. Based on slides 10-14 of lecture 1, what kind of vulnerability do you think was found? Explain how you came to this conclusion.
2. Many security experts caution against "security by obscurity." Was open source useful to the attackers? (You may want to look at slides 35-39 of Charlie Miller's slides posted as reading for lecture 5).
3. For each of the following security measures, explain whether Apple's use of the measure could have made it harder to find the iPhone vulnerability, or harder exploit it to take control of the iPhone. If the measure is good security practice, but not related to the iPhone example, say so.
   1. Address randomization
   2. Non-executable heaps
   3. Run applications as an unprivileged user
   4. chroot applications to prevent access to unrelated data

Problem 4

(a) Recall that a polymorphic virus uses an evolving decryption engine and a fixed (encrypted) virus body. Anti virus software detects polymorphic viruses by first emulating the executable in question for a short time and then looking for a virus signature in memory. Explain how polymorphic viruses can evade this detection technique. Give at least two examples. Your examples should be examples of polymorphic behavior (not metamorphic).

(b) Can you suggest a more robust mechanism for preventing polymorphic viruses from spreading? You may assume that the anti-virus vendor already obtained a copy of the virus and fully analyzed how it works.

Problem 5

Recall that modern processors have an on chip data cache (called an L2 cache) used to reduce the number of accesses to main memory. Reading from the L2 cache is about 10 times faster than reading from main memory. Explain how the L2 processor cache can result in a covert channel on the local system. Give a sample short (pseudo) code to exploit this channel.

Assume that processes have separate address spaces (no shared memory), although they share the underlying hardware. You don't need to worry about making the covert channel fast. You may make any reasonable assumption about the L2 cache, as long as you state your assumptions.