atp

We had to clean up some left over cgroups after another set of experiments with LXC.The guys doing it encountered problems, as the logic is the opposite of what you expect from your experience on a normal unix filesystem.

Specifically the problem happens when you have a nested cgroup - for example /cgroup/foo/bar/

The problem happens because this is a virtual filesystem with its own rules. 

If you create a cgroup (in this case we'll just mount the net_cls cgroup because its smaller)

# mount -t cgroup /dev/cgroup /cgroup -o net_cls 
# cd /cgroup
# ls
cgroup.procs net_cls.classid notify_on_release release_agent tasks
# mkdir foo
# ls
cgroup.procs foo net_cls.classid notify_on_release release_agent tasks
# ls foo
cgroup.procs net_cls.classid notify_on_release tasks

There are a set of files created to manipulate the cgroup. If the tasks entry is empty, then the cgroup is unused. 

However the files can't be removed. 

# rm -f *
rm: cannot remove `cgroup.procs': Operation not permitted
rm: cannot remove `foo': Is a directory
rm: cannot remove `net_cls.classid': Operation not permitted
rm: cannot remove `notify_on_release': Operation not permitted
rm: cannot remove `release_agent': Operation not permitted
rm: cannot remove `tasks': Operation not permitted

So here's where the confusion arose. To remove a directory hierarchy on a normal file system you should first remove all files from each subdirectory and then remove the parent directories. 

This is what "rm -fr  directory" will usually do for you. 

However, as the files cannot be removed, that approach will fail in this virtual filesystem. "rm -rf" is the wrong approach.  

Instead, you need to remove just the cgroup directories. If you want to do this recursively here's the magic incantation (we first make a sub-cgroup to give it more than one level of depth);

# mkdir -p foo/bar
# ls foo/
bar/ cgroup.procs net_cls.classid notify_on_release tasks
# find foo -depth -type d -print -exec rmdir {} \;
foo/bar
foo
# ls
cgroup.procs net_cls.classid notify_on_release release_agent tasks

 As the files aren't real, they don't need to be removed - we only need to remove the cgroup directories. The "-depth" argument tells find to do a depth first recursion, so we remove sub cgroups first. 

If that fails, then you'll most likely have a task occupying one of the cgroups somewhere.

 You can find it like this;

# mkdir -p foo/bar
# echo $$ > foo/bar/tasks
# find foo -depth -type d -print -exec cat {}/tasks \;
foo/bar
21413
21956
21957
foo

And to remove those tasks from the subcgroup add them into the top level. 

# echo $$ > tasks
# find foo -depth -type d -print -exec cat {}/tasks \;
 foo/bar
foo
#

The last gotcha with cgroups relates to mounting subsets of functionality.

 If you have created a cgroup, and then attempt to mount or remount /dev/cgroup with different cgroup subsystems active, the mount will fail with a rather unhelpful error message "mount: /dev/cgroup already mounted or /cgroup busy" 

 The answer is to remove the cgroups you have created, so that there are no cgroups on the system, and then you can mount with different subsystems active.  

 As in the following example;

# umount /cgroup
# mount -t cgroup /dev/cgroup /cgroup -o net_cls,cpuacct
mount: /dev/cgroup already mounted or /cgroup busy
# mount -t cgroup /dev/cgroup /cgroup -o net_cls
# find /cgroup/foo -depth -type d -print -exec rmdir {} \;
/cgroup/foo/bar
/cgroup/foo
# umount /cgroup
# mount -t cgroup /dev/cgroup /cgroup -o net_cls,cpuacct
# ls /cgroup
cgroup.procs cpuacct.usage net_cls.classid release_agent
cpuacct.stat cpuacct.usage_percpu notify_on_release tasks

Which is probably why fedora mounts them all separately. Even though that looks messy.