Working of System Calls in Operating Systems

System call is used in user application to request some hardware/Operating system service access.
When user call a system call it actually call some user level library (glibc/ulibc) which is in between User and Operating System.They are implemented as short pieces of assembly code named as system call stub.Which actually place system call number associated to the particular system call to a register (For x86 EAX) , place the arguments other registers (for x86 EBX,ECX etc.)  and issue a trap instruction ( in X86 calling assembly call INT 0x80, "sc" for powerpc) to switch from user mode to kernel privileged mode.


In trap handler kernel code copies arguments to kernel stack and call corresponding system call handler.


According to system call dispatcher a new system call may be attached to kernel in compile time or run time.


After running system code handler it return by putting return value in some register. (In PPC r3 register)


Note: While passing arguments from user can pass args in several way.
Some implementation in x86
1) push to stack

open:
    push    dword mode
    push    dword flags
    push    dword path
    mov eax, 5
    push    eax     ; Or any other dword
    int 80h
    add esp, byte 16

2) linux way - Put in registers

open:
    mov eax, 5
    mov ebx, path
    mov ecx, flags
    mov edx, mode
    int 80h

System call implementation in i386

http://www.tldp.org/HOWTO/html_single/Implement-Sys-Call-Linux-2.6-i386/
http://cse.yeditepe.edu.tr/~moergin/courses/cse331/addsystemcall.html


On ARM:
Its similar to x86/PPC ,It passes arguments through r0-r6,r7 is used for passing system call number.We call 'SWI' instruction to similar to "INT 80" in x86.

movl r7,#378

mov  r0,#25

mov r1,#35

SWI 0

mov r7,#1 

SWI  0 

Dynamic attachment

http://linux.die.net/lkmpg/x978.html


Update: in x86 __asmlinkage modifier is used in system side system call function implementation because we tell the function to use stack instead of registers for arguments of the function.
Post P4 processor use sysenter to enter in syscall mode instead of int 80,

In very detail :
http://12000.org/my_notes/system_calls_in_linux/system_call_in_linux/index.htm