Honestly, LuaJIT is not such ugly and complex as you expect. Check out this talk about LuaJIT.

• Lua:
  • Language basics: "global" and local variables, conditions, loops
  • Plain types, functions as first-class values
  • Table as a single complex data structure
  • Metatables and metamethods
  • Coroutines
  • Userdata
• LuaJIT (implementation):
  • Extensions
  • FFI + cdata
  • JIT compiler

• Build system overview
• DynASM:
  • What is DynASM?
  • First DynASM example
  • Jumping with DynASM
  • Calling external functions with DynASM
• buildvm for "preprocessing" the standard library

• Type system (src/lj_obj.h): primitives, numbers, strings, tables, coroutines, prototypes (including list of literals and upvalues)
• Interpreter (src/vm_x86.dasc):
  • Bytecode reference
  • Types of callable objects: Lua functions, C functions, fast functions
  • Dispatch table
  • Host frame vs. guest frame (guest frame types): Lua, C, continuation, vararg, protected frames
  • Error handling incl. DWARF2 stack unwinding

• Overview
• Gotchas: object chain, reachability rules, entry points, colors, phases

• Content of the old missing LuaJIT Wiki page

• Talk by Peter Cawley about LuaJIT history and LJ_GC64
• Several talks by Roberto Ierusalimschy about Lua per se and its internals related to LuaJIT
  • Why (and why not) Lua
  • Garbage Collection in Lua
  • Functions in Lua
• You can find lots of talks related to LuaJIT here
• Brief dive into LuaJIT by Nick Zavaritsky