64-bit Solutions on

SQL Server 2005

Dean Bethke
Technology Architect
Microsoft Technology Team

Agenda
64-bit Platforms
64Benefits of 6464-bit SQL
64--Bit Challenges and Best Practices
64
Memory Configuration
NUMA
Performance

64-bit Platforms
The mainstream highhigh-volume server
platforms are now 6464-bit OS ready
Your choice is whether to run 64 bit SQL
on these platforms, or to run 3232-bit SQL on
the 6464-bit or 3232-bit operating system
For highhigh-scale 6464-bit systems, you also
need to choose between Itanium (IA64)
and x64 offerings

Properties of Itanium (IA64)

Runs 6464-bit Windows, drivers and software
specifically compiled for the Itanium instruction
set
Runs 3232-bit software without being recompiled
New 3232-bit execution layer

http://www.microsoft.com/windowsserver2003/64bit/ipf/ia32el.mspx

Cannot act like an x86 processor or boot 3232bit Windows

Does not run versions of Windows or 6464-bit
drivers compiled for x64

Intel Itanium-2 offerings

4 socket and above servers (
( 64 sockets)
1.6Ghz and 9MB L3 cache (at 6.4 gigabits/sec system
bus bandwidth)
Recent bump to 1.66Ghz but faster 667Mhz FSB (at
10.6 gigabits/sec)

Dual--core now available

Dual
Montecito
Dual--core, huge cache (24MB L3), but still 1.66 Ghz
Dual

Montvale+
Multi--core and massive cache, slated 2007
Multi

Massive scalability database target market

Main Vendors
HP, Unisys, NEC, Fujitsu

Properties of x64
Runs 6464-bit Windows, drivers and software
specifically compiled for X64 instruction set
Can act like an x86 processor when an X64
system is booted into a 3232-bit operating system
and as such runs all 3232-bit versions of Windows
commercially available today
Runs 3232-bit software without recompilation
Does not run Itanium versions of Windows nor
drivers compiled for Itanium

x64 Offerings
Two chip vendors, same OS required
AMD (Opteron)
Eliminates FSB and uses HyperTransport
Currently limited to 1MB L2 cache

Intel (Xeon EM64T)

Large cache, NetBurst & Hyperthreading
Features 4MB and 8MB L3 cache versions

Generally aimed at <= 44-so cke t, h o w e ve r

Unisys - 32 socket (8 socket interconnect)
IBM 16 socket (4 socket interconnect)
Sun 8 socket (AMD)

HP offers both AMD & Intel (max 44-socket)

Dell currently offers Intel for Dell servers

Multicore and Hyperthreading

Multicore chips scale very effectively
Dual-core offers >> 50% performance benefit relative
Dualto single core
S Q L S e rve rs o p e ra te s a s th o u g h e a ch co re is a
separate CPU
Separate SQL Scheduler assigned to each core

But SQL is priced per socket, not perper-core

Hyperthreading does not benefit typical SQL

workloads
Recommend disabling Hyperthreading
Potentially overloads a single core with multiple
concurrent scheduler tasks
Multiple threads can thrash the CPU cache

Windows on Windows (WOW)

What is it?

WOW allows 3232-bit applications to run on 6464-bit Operating Systems

SQL Server will NOT support both 3232-bit SQL in WOW and native 6464-bit
SQL coco-existing on same server
Example: Once you install 3232-bit SQL Server Engine the server Cannot run 6464-bit
Engine on same hardware
32-bit App
running on
32-bit OS

64-bit App
running on
64-bit OS

32-bit App
running on 64-bit
OS (in WOW)

32-bit App

64-bit App

32-bit App

32-bit App

32-bit App

64-bit App

32-bit App

WOW

WOW

32-bit OS

64-bit OS

64-bit OS

x64 Hardware

x64 Hardware

x64 Hardware

x64 Hardware = AMD64 and Intel EM64T

64-bit OS

x64 HW

What is 64-bit SQL Server

2005?

One code base for all platforms of SQL

Server
Flat memory addressing
IA64 and X64
Data files compatible with 3232-bit SQL
Easy Database Migration & Integration
Detach/Attach, Log shipping, Replication etc.

Massive ScaleScale-up support

64-way HP*, 326432-way NEC, 3232-way Unisys
>1,000,000 TPC number*

8-node FailFail-over clustering support

64-bit gaps in SQL 2005

VS not supported on Itanium
Although management tools are supported

So BIDS not supported on Itanium affects SSIS

Must develop and debug packages on a 3232-bit server and deploy
to 6464-bit
Watch for driver compatibility

x64 BIDS runs in the WOW

SSIS needs 3232-bit drivers at pipeline construction time
and 6464-bit drivers at execution time
Some x64 drivers for external RDBMS are still problematic

32-bit SQL Server on x64

32-bit SQL Server 2005 (server & tools) supported
32on x64 running 6464-bit Windows
Under WOW64, SQL Server can access FULL 4GB of
RAM, as well as AWE (no need to enable /3GB)
Only without CLR enabled; with CLR enabled only 3GB

Why would you want to run 3232-bit OS instead?

Drivers
C o e xiste n ce w ith o th e r a p p lica tio n s o r to o ls th a t a re n t
WOW--certified
WOW

SQL Server on X64, Itanium

x64
SQL Server Edition

32--bit
32
OS

64--bit
64
OS

Itanium
64--bit
64
OS

WOW64

64--bit
64
OS

64--bit
64
OS

WOW64

SQL Server 2000 3232-bit (SP4)

Yes

Yes

No

No

No

SQL Server 2000 6464-bit Itanium

No

No

No

No

Yes

SQL Server 2005 3232-bit

Yes

Yes

No

No

No

SQL Server 2005 6464-bit x64*

x64*

No

No

Yes

No

No

SQL Server 2005 6464-bit Itanium*

No

No

No

No

Yes

SQL Server Express 3232-bit

Yes

Yes

No

No

No

SQL Server Mgmt Studio 3232-bit

Yes

Yes

No

Yes

No

SQL Server BI Dev Studio 3232-bit

Yes

Yes

No

No

No

Agenda
64-bit Platforms
64Benefits of 6464-bit SQL
64--Bit Challenges and Best Practices
64
Memory Configuration
NUMA
Performance

Memory Primer
32--bit applications
32
All 3232-bit applications are limited to a 4GB virtual
address space (VAS)
An application can utilize 2GB (3GB using /3gb, or
4GB in x64 WOW). Remainder is reserved for OS
addresses
Using AWE, some applications can use larger
amounts of memory by mapping / unmapping
additional memory into the VAS

64--bit applications
64
Flat, huge virtual address space
No mapping needed for large memory access

DB Page Cache

Plan Cache

Query Workspace.

Buffer Pool
(8KB buffers)

Locks

Other

SQL Server Memory

Thread stacks, DLLs,

CLR GC heap, etc

AWE
Addressable
Memory

Where Will 64-Bit SQL Help?

Relational Server
Memory hungry workloads aside from DB
buffer pages consuming:
Plan Cache: Lots of stored procedures and dynamic
SQL batches (OLTP)
Workspace Memory: Large number of concurrent
hash joins or aggregates; largelarge-scale sorts and index
builds (Data Warehouse)
Connection memory: Large numbers of connections
Thread memory: High Concurrency OLTP
CLR GC Heap memory: CLR SP allocations
Lock Memory: Large scale OLTP
All are common scenarios for scaledscaled-up OLTP and Data
Warehouse applications, utilizing many CPUs

Identifying Memory Pressure

Relational Server
Workspace Memory
Queries queued waiting on Memory Grants
With large percentage of VM devoted to
workspace memory

Plan Cache
Large number of compiles/sec
Low cache hit ratio

Locks
Large percentage of VM occupied by locks

Where Will 64-Bit SQL Help?

Analysis Services
Dimensions
AS 2005 queries perform much better when dimension members
can be cached in memory
Individual sets of related attributes must fit in memory during
dimension processing
Key attributes for at least the largest dimension must fit in
memory during partition processing

Processing Buffers and Data Cache

Windows Filesystem Cache
Can grow very large on 6464-bit and boost performance accessing
the files containing aggregates and measures

NOTE: No AWE support in Analysis Services

Identifying Memory Pressure

Analysis Services
Inadequate process buffers in AS2005
Processing becomes single threaded during
aggregation phase
CPU utilization very low during partition processing
Temp files used (monitor in Filemon or Perfmon
counters)

Indicates need to increase Process Buffer

Size

Pressure on AS Data Cache

Perfmon: Agg Cache Hit Ratio, Evictions/sec

Where Will 64-Bit SQL Help?

Integration Services 2005
Specific largelarge-scale transformations can
consume a lot of virtual memory:
Sort
Aggregate
Key Lookup (cached)

Packages may fail on 3232-bit when 3GB is

insufficient virtual address space to
process large streams using the above
operations

Where Will 64-Bit SQL Help?

Scale-Up and Performance
Ita n iu m p la tfo rm s a re to d a ys ch o ice fo r
workloads requiring > 8 CPUs
Very large Data Warehouses
Massive OLTP
Itanium offers excellent scaling
characteristics, multimulti-node bus architecture
X64 Xeon scaledscaled-up servers have just arrived

X64 offers fastest CPU performance today

Single-threaded performance on X64 can
Singlesurpass Itanium today

Agenda
64-bit Platforms
64Benefits of 6464-bit SQL
64--Bit Challenges and Best Practices
64
Memory Configuration
NUMA
Performance

64-Bit Challenges
64--bit drivers for Data Access
64
Analysis Services and Integration Services require the
64--bit versions of 3rd party OleDB drivers to support
64
data sources such as Oracle, Informix, etc.
No 6464-bit ODBC access supported
Verify before purchasing your server!

Memory Configuration of Components

Multiple SQL Instances on server
Multiple Service components on server, e.g. AS, SQL
Relational, SSIS
Both scenarios were simpler on 3232-bit memory was
isolated between apps

Memory Configuration
Alternatives
Use separate hardware partitions or
separate servers for solutions that
combine Relational, AS and SSIS
Eliminates risks of crosscross-application memory
contention
But this may underutilize CPU resources

OR
Manage memory use explicitly for each
component running on a single server

64-bit Memory Configuration

Relational Server
If multiple SQL Server instances share the
server
OR
If multiple applications (AS, SSIS) coexist on the
server
Best Practice is to place a memory ceiling each SQL
instance
Or even use a fixed memory size

Lock pages in memory to prevent OS paging of SQL

under memory pressure

64-bit Memory Configuration

Relational Server
Min, Max Server Memory determines how
d yn a m ic S Q L s B u ffe r P o o l m e m o ry is
More Dynamic:
+ Allows SQL to balance memory consumption against
demands of other applications
- Reduces predictability of performance for critical memorymemorydependent workloads
- SQL Performance problems likely when there are sudden
external memory demands

Less dynamic
+ Guarantees resources for a critical workload
- May leave insufficient memory for other applications
- May leave insufficient memory for nonnon-buffer pool
components (CLR GC heap, DLLs, etc)

64-bit Memory Configuration

Relational Server
To limit SQL RDBMS memory consumption
Determine desired size of total RDBMS memory footprint = Msql
Ensure that
Msql < Total Server Memory ~0.5GB for OS
Max memory required by other apps or instances

Reserve enough room outside of buffer pool for

SQL Threads
Threads will consume 4MB each on Itanium, 2MB on X64
(= 1GB for 256 Worker Threads on Itanium!)

XPs, InIn-Proc OleDB drivers, CLR GC heap

Calculate & set the resulting size of buffer pool desired

Bpool = Msql - Threads - XPs - CLR GC heap - etc
S p _ co n fig u re M a x S e rve r M e m o ry = B p o o l
C o n sid e r: S p _ co n fig u re M in S e rve r M e m o ry = B p o o l

NOTE: Memory outside of Buffer Pool cannot be limited

64-bit Memory Configuration

Relational Server
SQL responds to external memory pressure
If memory is dynamic (Max <> Min) SQL will try to reduce its
working set when notified by OS
S o m e tim e s S Q L d o e sn t re sp o n d fa st e n o u g h a n d th e O S w ill p a g e
o u t p a rt o f S Q L s a d d re ss sp a ce

If memory is fixed (Max = Min), the OS may page out part of

S Q L s a d d re ss sp a ce u n d e r m e m o ry p re ssu re

Either way, OS paging of SQL is very bad

System becomes unresponsive
Query times can become impossibly long
Launching an unexpected new process or growing an existing
one can cause OS to page SQL

OS Paging of other apps on the server is disk and CPU

intensive
Can also degrade SQL performance

64-bit Memory Configuration

Relational Server
Prevent OS paging of Buffer Pool by
Locking Pages in Memory
Automatic if SQL Service Account has Lock
Pages in Memory privilege
Control Panel / Admin Tools / Local Security Policy

SQL Memory can still be dynamic

Note: Other parts of SQL may still be
paged, along with other applications

Lock Pages Permission

Verifying Locked Pages

Minimize Filesystem Cache

Potential for
enormous
filesystem cache
on 6464-bit can
create external
memory pressure
Configure OS to
favor memory for
programs, not
filesystem

SQL Memory Acquisition

By default SQL engine will not acquire all
requested memory upon startup
Grows buffer pool gradually as it needs the
memory

If another application on the server grabs

memory first, SQL may not be able to
obtain all the memory you have configured
it to use
Rare cases where you want to start SQL first
& run a workload to acquire memory prior to
launching other memorymemory-consuming apps

64-bit Memory Configuration

Analysis Services
Consumes memory in two ways
Virtual Memory for Analysis Services Service
Bounded by configuration settings

OS Memory used as Filesystem cache

Potentially unbounded

If AS is running on a server with other critical

services (SQL or SSIS):
Best Practice to establish a maximum memory size for
Analysis Services process and limit growth of
Filesystem Cache

64-bit Memory Configuration

Analysis Services
To limit Analysis Services memory
consumption, set
Analysis Services Properties: Advanced
or Config File
Memory \ TotalMemoryLimit

No mechanism to Lock Pages in Memory prior

to SP2
SP2 can prepre-allocate memory with
<PreAllocate> config entry

64-bit Memory Configuration

Analysis Services
Filesystem Cache filled by AS buffered IO
Can grow without limit and even page out other applications
Was limited to ~1GB max on 3232-bit
If SQL has Locked Pages, Relational Server will not be paged out as
cache grows
However, Analysis Services process and other components can page
out if this grows too large

System Cache Trimming

Built-in to AS2005
BuiltAS2005,, but off by default
Set Advanced Server Properties in SSMS or msmdsrv.ini
LimitSystemFileCacheSizeMB
(MB)
LimitSystemFileCacheSizePeriod (Milliseconds)
Takes immediate effect without a restart

AS2005 Properties to Limit

Filesystem Cache

64-bit Memory Configuration

Integration Services
You cannot impose explicit memory limits within SSIS
Simple pipelines or control packages will not be
significant consumers of memory
Large Sort, Aggregation and Cached Lookup operations
in a pipeline can consume a lot of virtual memory
Hint: Retrieve minimum necessary fields for lookup operations
If these create too much memory pressure, consider using DB
operations instead, or a separate server / partition

Key Memory Counters

NOTE: sqlservr Private Bytes and Working Set are not correct
in Perfmon when Lock Pages are used.

Agenda
64-bit Platforms
64Benefits of 6464-bit SQL
64--Bit Challenges and Best Practices
64
Memory Configuration
NUMA
Performance

NUMA Considerations
All 6464-bit servers that scalescale-up are NUMA
platforms
Non Uniform Memory Access
Longer latencies for accessing memory on remote
nodes
Server
Pod/Node

Pod/Node

RAM

CPU

CPU

CPU

Pod/Node

RAM

CPU

CPU

Bus

CPU

CPU

RAM

CPU

CPU

CPU

CPU

CPU

NUMA Servers
All NUMA servers can be configured in BIOS to
expose to the OS either
N U M A M e m o ry, o r
In te rle a ve d M e m o ry

NUMA memory OS associates a specific

memory address range with each pod
Interleaved memory memory addresses are
spread, fine grained, across all pods

SQL2005 and NUMA

NUMA memory configuration is the preferred
choice on SQL2005 servers with > 4 cores
Some OLTP workloads are 60% faster when using
NUMA instead of Interleaved memory

A SQL connection will remain on the same

NUMA node for its lifetime
Potential for leveraging local cache for all of its tasks
Can direct specific connections to specific nodes

SS2005 RDBMS
NUMA Memory Management
SQL RDBMS instantiates an abstraction for each
NUMA node on the server
A S Q L N o d e in clu d in g la zyw rite r, b p o o l, sch e d u le rs,
IO port, etc.

Each SQL Node obtains memory from Windows,

proportionally
e.g. memory per node = SQL memory/#Nodes

Windows will try to allocate to a SQL Node

memory to a that is local to its NUMA node
If other processes have grabbed local memory, the
S Q L N o d e w ill h a ve fo re ig n p a g e s a llo ca te d

NUMA Memory
Management Continued
SQL Tasks running on a node will use pages
allocated to that node (local or foreign) when
possible
B u t w ill u se a n o th e r n o d e s p a g e s if D B p a g e s a re
already cached there
W ill u se a n o th e r n o d e s p a g e s to sa tisfy a m e m o ry
grant when necessary

N o d e a llo ca tio n o f fo re ig n p a g e s ca n b e
monitored in Perfmon:

Some NUMA Gotchas

AS and SSIS are mostly not NUMA aware
Pre--SP2
Pre
No guarantees of evenly distributed memory allocation
O S w ill te n d to fill u p a sin g le n o d e s C P U s w ith w o rk b e fo re
exercising other CPUs
Common to see only 1 busy set of nodes and remainder of server
idle until a node saturates

SP2 Improvements
SP2 has NUMA improvements for processing
SP2 can prepre-allocate memory per NUMA node with config entry

Consider starting up SQL Server and warming cache

before launching AS
May be effective in very large cube environments
Avoids AS potentially grabbing all local memory for one node
Avoids filesystem cache from stealing pages disproportionately
from a single node which could create a SQL node starved of
local pages

Agenda
64-bit Platforms
64Benefits of 6464-bit SQL
64--Bit Challenges and Best Practices
64
Memory Configuration
NUMA
Performance

Performance Considerations
SQL Server 6464-bit working set is larger than 3232bit SQL
Long pointers
Code path less likely to be found in processor cache
than equivalent 3232-bit code

Result: Apps that are not memory constrained

on 3232-bit (especially OLTP) may run ~10% less
efficiently using 6464-bit edition than 3232-bit edition
on the same machine
May observe more CPU cycles to do equivalent work
CPUs spin waiting for nonnon-cached memory access

Performance Considerations
If you are not under memory pressure,
pressure, and do not
need to scale up,
up, and you want the absolutely
fastest singlesingle-thread performance
Use X64 instead of Itanium
Consider using 3232-bit SQL in the WOW or on 3232-bit OS
X e o n s la rg e r L 3 ca ch e m a y h a ve a n o tice a b le im p a ct
relative to Opteron

If you need to scale up for greater throughput

Use Itanium with large number of CPUs
or
Consider newest scaledscaled-up X64 servers by Unisys

Summary: 64-b it g o tch a s

Multiple components competing for
memory
Potential NUMA effects
Potential performance penalty due to
larger working set
Driver availability
Management tool support

Resources
SQLCAT Team Blog: http://blogs.msdn.com/sqlcat
SQL Server memory management, 6464-bit and NUMA
Slava Oks Blog
http://blogs.msdn.com/slavao

Analysis Services Configuration and Performance

Project REAL
http://www.microsoft.com/sql/bi/projectreal

Integration Services
Blogs by Ashvini Sharma, Donald Farmer at sqljunkies.com
http://www.sqljunkies.com/WebLog/donald_farmer
http://sqljunkies.com/WebLog/ashvinis

 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S.
and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because
Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any
information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.