Home | Projects | Notes > Embedded Linux > BBB Linux Boot Sequence - Step 4: Linux Kernel

BBB Linux Boot Sequence - Step 4: Linux Kernel

BBB Linux Boot Sequence

software-components-necessary-for-linux-booting

Linux Kernel

After kernel decompression is done, the control is transferred to another head.S file that is a part of the Linux kernel.

control-flow-during-linux-boot-3

Go to linux/arch/arm/kernel/ where Linux kernel's architecture dependent files are located. This head.S is the generic startup code for ARM processor (not SoC vedor specific) that does the ARM specific initializations:
1. CPU specific initialization
2. Checks for valid processor architecture
3. Page table inits
4. Initialize and prepare MMU for the identified processor architecture
5. Enable MMU to support virtual memory
6. Calls start_kernel function of the main.c (Architecture dependent code)
This head.S executes lots of architecture specific initialization code implemented in different intermediate files.
It is the responsibility of the Bootstrap Loader (i.e., glued to the Linux kernel image) to decompress and relocate the Linux kernel image. (These are NOT the responsibilities of U-boot)
Again, don't be confused about those two head.S files. One belongs to the Bootstrap Loader, and the other belongs to the Linux kernel. And both are architecture dependent; in this case, ARM.

`head.S`

Belongs to the Linux kernel architecture specific code.
It does CPU (e.g., ARM Cortex-A8) specific initialization of the SoC.
1. It first tries to identify CPU architecture information (e.g., ARM Cortex-A8)
2. When the architecture type is identified, it creates initial page table entries and enables the MMU for the processor's virtual memory support.
3. Then, transfers the control to the generic main.c of the Linux kernel.

Kernel startup entry point:


xxxxxxxxxx
43
1
/* linux/arch/arm/kernel/head.S */
2

3
/*
4
 * Kernel startup entry point.
5
 * ---------------------------
6
 *
7
 * This is normally called from the decompressor code.  The requirements
8
 * are: MMU = off, D-cache = off, I-cache = dont care, r0 = 0,
9
 * r1 = machine nr, r2 = atags or dtb pointer.
10
 *
11
 * This code is mostly position independent, so if you link the kernel at
12
 * 0xc0008000, you call this at __pa(0xc0008000).
13
 *
14
 * See linux/arch/arm/tools/mach-types for the complete list of machine
15
 * numbers for r1.
16
 *
17
 * We're trying to keep crap to a minimum; DO NOT add any machine specific
18
 * crap here - that's what the boot loader (or in extreme, well justified
19
 * circumstances, zImage) is for.
20
 */
21
 
22
 ...
23
 
24
     /*  
25
     * The following calls CPU specific code in a position independent
26
     * manner.  See arch/arm/mm/proc-*.S for details.  r10 = base of
27
     * xxx_proc_info structure selected by __lookup_processor_type
28
     * above.
29
     *
30
     * The processor init function will be called with:
31
     *  r1 - machine type
32
     *  r2 - boot data (atags/dt) pointer
33
     *  r4 - translation table base (low word)
34
     *  r5 - translation table base (high word, if LPAE)
35
     *  r8 - translation table base 1 (pfn if LPAE)
36
     *  r9 - cpuid
37
     *  r13 - virtual address for __enable_mmu -> __turn_mmu_on
38
     *
39
     * On return, the CPU will be ready for the MMU to be turned on,
40
     * r0 will hold the CPU control register value, r1, r2, r4, and
41
     * r9 will be preserved.  r5 will also be preserved if LPAE.
42
     */
43
...

L7: Here the decompressor code means the code of the Bootstrap Loader.
L8-L9: MMU and cache off since we are still in the initialization phase of hardware. This code uses machine id (r1) to carry out the machine specific initializations. r2 holds the address of Device Tree Binary which is located in RAM.
L25: Processor architecture specific initialization code this comment is referring to is found in arch/arm/mm/.
L26: r10 holds the based address of xxx_proc_info structure that is specific to the processor detected by the __lookup_processor_type call.
L39: Tells that all the processor architecture specific initializations have to be completed before MMU can be turned on! Before transferring control to the Linux generic code, MMU must be turned on. This is one of the important duties of hardware architecture specific code.

One important assembly subroutine is __enable mmu:

It initializes page table pointers and turn on MMU so that kernel can start running with the virtual address support.


xxxxxxxxxx
73
1
/* linux/arch/arm/kernel/head.S */
2

3
...
4

5
/*
6
 * Setup common bits before finally enabling the MMU.  Essentially
7
 * this is just loading the page table pointer and domain access
8
 * registers.  All these registers need to be preserved by the
9
 * processor setup function (or set in the case of r0)
10
 *
11
 *  r0  = cp#15 control register
12
 *  r1  = machine ID
13
 *  r2  = atags or dtb pointer
14
 *  r4  = TTBR pointer (low word)
15
 *  r5  = TTBR pointer (high word if LPAE)
16
 *  r9  = processor ID
17
 *  r13 = *virtual* address to jump to upon completion
18
 */
19
__enable_mmu:
20
#if defined(CONFIG_ALIGNMENT_TRAP) && __LINUX_ARM_ARCH__ < 6
21
    orr r0, r0, #CR_A
22
#else
23
    bic r0, r0, #CR_A
24
#endif
25
#ifdef CONFIG_CPU_DCACHE_DISABLE
26
    bic r0, r0, #CR_C
27
#endif
28
#ifdef CONFIG_CPU_BPREDICT_DISABLE
29
    bic r0, r0, #CR_Z
30
#endif
31
#ifdef CONFIG_CPU_ICACHE_DISABLE
32
    bic r0, r0, #CR_I
33
#endif
34
#ifdef CONFIG_ARM_LPAE
35
    mcrr    p15, 0, r4, r5, c2      @ load TTBR0
36
#else
37
    mov r5, #DACR_INIT
38
    mcr p15, 0, r5, c3, c0, 0       @ load domain access register
39
    mcr p15, 0, r4, c2, c0, 0       @ load page table pointer
40
#endif
41
    b   __turn_mmu_on
42
ENDPROC(__enable_mmu)
43

44
/*
45
 * Enable the MMU.  This completely changes the structure of the visible
46
 * memory space.  You will not be able to trace execution through this.
47
 * If you have an enquiry about this, *please* check the linux-arm-kernel
48
 * mailing list archives BEFORE sending another post to the list.
49
 *
50
 *  r0  = cp#15 control register
51
 *  r1  = machine ID
52
 *  r2  = atags or dtb pointer
53
 *  r9  = processor ID
54
 *  r13 = *virtual* address to jump to upon completion
55
 *
56
 * other registers depend on the function called upon completion
57
 */
58
    .align  5
59
    .pushsection    .idmap.text, "ax"
60
ENTRY(__turn_mmu_on)
61
    mov r0, r0
62
    instr_sync
63
    mcr p15, 0, r0, c1, c0, 0       @ write control reg
64
    mrc p15, 0, r3, c0, c0, 0       @ read id reg
65
    instr_sync
66
    mov r3, r3
67
    mov r3, r13
68
    ret r3
69
__turn_mmu_on_end:
70
ENDPROC(__turn_mmu_on)
71
    .popsection
72

73
...

From architecture dependent code to architecture independent code:
linux/arch/arm/kernel/head-common.S $\to$ linux/init/main.c
At this point, all the processor architecture dependent initializations are over, CPU is ready with MMU support and kernel is ready to perform architecture independent initializations.

entry-to-architecture-independent-code

`main.c`

Does all the startup work for the Linux kernel from initializing very first kernel thread all the way to mounting the Root File System (RFS) and executing the very first user-space Linux application program.
start_kernel() function is a huge function which calls other various initialization functions to initialize subsystems before mounting the RFS. For example, mm_init() (memory management init), sched_init() (scheduler init), etc.

rest_init() function creates two kernel threads; kernel_init (pid = 1), kthreadd (pid = 2).

kernel_init spawns the very first user application called init. This init application is inheriting the pid of its kernel thread.
kthreadd spawns other kernel threads.


xxxxxxxxxx
33
1
/* linux/init/main.c */
2

3
noinline void __ref rest_init(void)
4
{
5
    struct task_struct *tsk;
6
    int pid; 
7

8
    rcu_scheduler_starting();
9
    /*   
10
     * We need to spawn init first so that it obtains pid 1, however
11
     * the init task will end up wanting to create kthreads, which, if
12
     * we schedule it before we create kthreadd, will OOPS.
13
     */
14
    pid = kernel_thread(kernel_init, NULL, CLONE_FS);
15
    /*   
16
     * Pin init on the boot CPU. Task migration is not properly working
17
     * until sched_init_smp() has been run. It will set the allowed
18
     * CPUs for init to the non isolated CPUs.
19
     */
20
    rcu_read_lock();
21
    tsk = find_task_by_pid_ns(pid, &init_pid_ns);
22
    tsk->flags |= PF_NO_SETAFFINITY;
23
    set_cpus_allowed_ptr(tsk, cpumask_of(smp_processor_id()));
24
    rcu_read_unlock();
25

26
    numa_default_policy();
27
    pid = kernel_thread(kthreadd, NULL, CLONE_FS | CLONE_FILES);
28
    rcu_read_lock();
29
    kthreadd_task = find_task_by_pid_ns(pid, &init_pid_ns);
30
 
31
    rcu_read_unlock();
32
    ...
33
}

kernel_init()


xxxxxxxxxx
55
1
static int __ref kernel_init(void *unused)
2
{
3
    int ret; 
4

5
    /*   
6
     * Wait until kthreadd is all set-up.
7
     */
8
    wait_for_completion(&kthreadd_done);
9

10
    kernel_init_freeable();
11
    /* need to finish all async __init code before freeing the memory */
12
    async_synchronize_full();
13

14
    system_state = SYSTEM_FREEING_INITMEM;
15
    kprobe_free_init_mem();
16
    ftrace_free_init_mem();
17
    kgdb_free_init_mem();
18
    exit_boot_config();
19
    free_initmem();
20
    mark_readonly();
21
    
22
    ...
23
        
24
    /*
25
     * We try each of these until one succeeds.
26
     *
27
     * The Bourne shell can be used instead of init if we are
28
     * trying to recover a really broken machine.
29
     */
30
    if (execute_command) {
31
        ret = run_init_process(execute_command);
32
        if (!ret)
33
            return 0;
34
        panic("Requested init %s failed (error %d).",
35
              execute_command, ret);
36
    }
37

38
    if (CONFIG_DEFAULT_INIT[0] != '\0') {
39
        ret = run_init_process(CONFIG_DEFAULT_INIT);
40
        if (ret)
41
            pr_err("Default init %s failed (error %d)\n",
42
                   CONFIG_DEFAULT_INIT, ret);
43
        else
44
            return 0;
45
    }
46

47
    if (!try_to_run_init_process("/sbin/init") ||
48
        !try_to_run_init_process("/etc/init") ||
49
        !try_to_run_init_process("/bin/init") ||
50
        !try_to_run_init_process("/bin/sh"))
51
        return 0;
52
    
53
    panic("No working init found.  Try passing init= option to kernel. "
54
          "See Linux Documentation/admin-guide/init.rst for guidance.");
55
}

L19: free_initmem(); The memory consumed by initialization functions so far are reclaimed here, because those functions were just for initialization and nobody will call them afterwards. Find the message (e.g., "Freeing init memory: xxxK") printed out to the screen during botting.
L47: Try to run the init program one by one from the different location until one succeeds. If no init application succeeds, it executes the shell application which is located in the RFS and just returns.
Note that you can also pass the path of the customized init program as a kernel commandline argument which will be stored in execute_command.
L53: If none of these program works, throws the message "No working init found..."

This is finally how the Linux kernel first launches the first user program. This init program is responsible for launching other user programs or kernel services.

control-flow-during-linux-boot-4

References

Nayak, K. (2022). Embedded Linux Step by Step Using Beaglebone Black [Video file]. Retrieved from https://www.udemy.com/course/embedded-linux-step-by-step-using-beaglebone/