MySQL Databases at RightMedia
Intro To RightMedia
RightMedia is an advertising exchange
Publishers, Advertisers, Networks compete and co-operate Rules and links between networks, publishers and advertisers lead to auctions for every ad call for fairness.
Over 7 Billion impressions per day!
Almost 8,000 ad servers in SP1,SK1,AC4,CH1,JP1
At peak, servers capable of processing 100K impressions/second With pixels, cookie requests, etc, 250K calls/second
-2-
Yahoo! Confidential
Data Cycle
Adserver data is aggregated and flushed every 10 minutes
Multiple aggregation layers per colo
Multiple data pipelines (similar to Data Highway) Reporting/Impression Log data stored in MySQL (rpt_prod and logdb) Customers use hourly reporting data to make decisions on campaigns and spending (trf_prod) Cache generation cycle every 2 hours
Traffic cache Predict cache Budget cache
-3-
Yahoo! Confidential
Data Cycle (Cont.)
Caches are pushed to each colo and distributed to ad servers. Cycle starts again
-4-
Yahoo! Confidential
MySQL Databases @ RightMedia
Rpt_prod Reporting Data Warehouse
51 database servers : 16 masters, 32 slaves (3 for archive) Data sharded over 16 partitions Middle ware server ReportWare converts XML requests to SQL and returns data in XML.
Reportware aggregates the 16 result set queries into one
DB size is 6.5TB All Dell PE 2950s 32GB memory, 2xQuad CPU, 6x146Gb SAS drives
Current schema very flat all tables are similar to materialized views
New version (AC4) Star/Snowflake schema
-5-
Yahoo! Confidential
MySQL Databases @ RightMedia (Cont.)
Trf_prod Heart of RightMedia
7 database servers: 1 master 6 slaves per colo Provides operational store for YieldManager UI, API, Creative Tester, Analytics, Predict, Budget processes. Very high transaction rate : nearly Half a billion per day
Relatively small but important
Downtime leads to larger failures
Ideal candidate for MySQL Cluster (Fault-tolerant system)
-6-
Yahoo! Confidential
MySQL Databases @ RightMedia (Cont.)
Predict Databases
Data store for adserver learning Prediction engine uses this data to lower CPM and maximize revenue Key system for LZ1.1, LZ2, LZ3
-7-
Yahoo! Confidential
MySQL Databases @ RightMedia (Cont.)
Budget Databases
-8-
Yahoo! Confidential
MySQL Architecture
Features and Components
The Big Picture
- 10 -
Yahoo! Confidential
Features MySQL vs Oracle
Features/Functionality
Strenths
MySQL
Price/Performance Great performce
Oracle
Capable of running large OLTPs and VLDBs
Database Server Instance
Instance stores global memory in mysqld background process
Made up of database schemas. Each storage engine stores information differently.
Instance has numerous background processes.
System SGA shared by procs. Tablespaces used for system metadata. $$$ Lots of options $$$ More complex, but lots of options. RMAN Advanced features
Physical Storage
Partitioning Replication
Free, in Version 5.1 Limited functionality Free, relatively simple setup. Great horizontal scalability
Backup/Recovery Stored Procedures
No online backup Home-grown/3rd party Very basic
- 11 -
Yahoo! Confidential
Query Cache
Global memory area used to cache queries and result sets.
SQL query (SELECT) and result stored in hash table Query is checked before parsing and optimization and result set returned immediately WICKED FAST PERFORMANCE!!
Excellent for static tables or tables with low change rate
Any INSERT,UPDATE,DELETE or any other DML operation on a table will invalidate all cache entries for that table.
- 12 -
Yahoo! Confidential
Storage Engine Architecture
Most flexible (and coolest) part of MySQL server.
Allows the freedom of choice to mix and match storage engines within the same database and server instance.
Storage engines are pluggable and can be dynamically loaded into the server instance.
Storage engine API allows developers to enhance and create own storage engines.
The MySQL community offers different pluggable engines for different business solutions.
- 13 -
Yahoo! Confidential
MySQL Storage Engines
MyISAM
Default storage engine for MySQL Disk Based storage - BTREE Table level locking No transactional support
Very low memory footprint
Full Text searching capability Index cache only OS buffers for data cache Extremely high performance Prone to corruption on server crash
- 14 -
Yahoo! Confidential
MySQL Storage Engines (Cont.)
InnoDB
ACID Compliant Transaction/XA support Row level locking Clustered index
Both data and index cached
Recovery can be time consuming can very slow Not scalable (Can be fixed with patches)
- 15 -
Yahoo! Confidential
MySQL Storage Engines (Cont.)
Memory (HEAP)
Similar to MyISAM, but all data/index resides in RAM Extremely fast for quick reference lookups Has finite storage limits (Can be configurable with max_heap_table_size)
Supports both BTREE and Hash indexes
No TEXT or BLOB columns allowed
- 16 -
Yahoo! Confidential
MySQL Storage Engines (Cont.)
Archive
Highly compressed data files Useful for storage of historical data Only allowed operations are SELECT and INSERT No indexes
MySQL 5.1 now allows index on AUTO_INCREMENT columns
- 17 -
Yahoo! Confidential
