sqlonlinux code snacks commit

Author: tarasha

samples/features/in-memory/code-snacks/SQLonLinux/In-Memory-Columnstore/test.txt

@@ -0,0 +1,49 @@
+# use glob syntax.
+syntax: glob
+*.ser
+*.class
+*~
+*.bak
+#*.off
+*.old
+
+# eclipse conf file
+.settings
+.classpath
+.project
+.manager
+.scala_dependencies
+
+# idea
+.idea
+*.iml
+
+#visual studio
+.suo
+
+# building
+target
+build
+null
+tmp*
+temp*
+dist
+test-output
+build.log
+out
+packages
+
+# other scm
+.svn
+.CVS
+.hg*
+
+# switch to regexp syntax.
+#  syntax: regexp
+#  ^\.pc/
+
+#SHITTY output not in target directory
+build.log
+
+#dropbox
+.DS_Store

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/.gitignore

@@ -0,0 +1,125 @@
+# In-Memory & Columnar Store Code Snack
+In this code snack, developers will experience the benefit of performing real-time operation analytics enabled by leveraging a memory optimized table in combination with a columnstore index. The Visual Studio project contains a load generator that will be used to simulate a write heavy workload. They will initially run the simulator against a disk based table with a clustered index (btree) and take note of the rows inserted per second, and will measure the performance of a provided analytics query while the system is under the heavy write load. They will then author the T-SQL to create the memory optimized table with a columnstore index, update the load generator to target the memory optimized table and observe the improved performance characteristics. This code snack targets SQL Server on Linux.
+
+## Requirements
+- Visual Studio 2015 with Update 3 (or later)
+- [SQL Server Data Tools for Visual Studio 2015](https://msdn.microsoft.com/en-us/mt186501) 
+- [SQL Server on Linux](https://www.microsoft.com/en-us/sql-server/sql-server-on-linux) running in [Docker](https://docs.docker.com/engine/installation/#/on-macos-and-windows)
+- Your SQL on Linux Server should have at least 4 GB of RAM
+
+## Clone the provided project
+Clone this repo on to your local machine.
+The recommended path is C:\In-Memory and Columnar\
+
+## Download the sample data
+This project requires a sample set of data you will load into SQL Server.
+Download the data from: [http://bit.ly/2envb8m](http://bit.ly/2envb8m)
+
+## Copy the sample data to your SQL on Linux host
+1. On your Docker host machine, first grab the container ID of your SQL Server on Linux container. The Container ID is the value present in the first column.
+```
+docker ps
+```
+2. Copy over the sample data from the host to the container running SQL on Linux by using docker cp as follows (replace ContainerID with the container ID your retrieved). Note that the path /var/opt/mssql will map to C: in any T-SQL scripts executed against this server.
+```
+docker cp datapoints.bcp [ContainerID]:/var/opt/mssql/data/datapoints.bcp
+```
+3. Verify that the copy was successful. By connecting to bash within the container. Connect to your container (substitute your container ID in the command below) and list out the files:
+```
+docker exec -t -i [ContainerID] /bin/bash
+ls /var/opt/mssql/data -1 -s -h
+```
+4. You should see datapoints.bcp in the listing
+```
+-rw-r--r-- 1 root root  256 Nov 13 22:49 Entropy.bin
+-rw-r--r-- 1 root root  14M Nov 13 22:50 MSDBData.mdf
+-rw-r--r-- 1 root root 768K Nov 14 22:48 MSDBLog.ldf
+-rw-r--r-- 1 root root 161M Nov 11 04:18 datapoints.bcp
+-rw-r--r-- 1 root root 4.0M Nov 14 22:55 master.mdf
+-rw-r--r-- 1 root root 768K Nov 14 23:07 mastlog.ldf
+-rw-r--r-- 1 root root 8.0M Nov 14 23:00 model.mdf
+-rw-r--r-- 1 root root 8.0M Nov 14 23:00 modellog.ldf
+-rw-r--r-- 1 root root 264M Nov 14 23:07 tempdb.mdf
+-rw-r--r-- 1 root root 8.0M Nov 14 23:07 templog.ldf
+drwxr-xr-x 3 root root 4.0K Nov 14 23:01 xtp
+```
+5. You are all set to continue the lab in Visual Studio 2015.
+
+## Create the database and tables
+1. Open the SqlLoadgenerator solution using Visual Studio 2015.
+2. From Solution Explorer, expand the SqlGenerator solution, then SQL Resources folder and open "Create Database.sql".
+3. Adjust the file paths for the FILENAME attributes if you installed SQL Server to a different location.
+4. Select the Execute button
+5. In the Connect dialog, provide your server name, authentication mode, username and password (as appropriate).
+6. Wait for the script to complete successfully.
+7. Within Visual Studio, open "Create Table- Disk Based.sql"
+8. Execute the script to create the DataPointsDiskBased table.
+This table  will be used to store simulated IoT device telemetry, using traditional disk based table as well as clustered and non-clustered indexes on the fields commonly used in both point queries and analytic queries.
+9. Within Visual Studio, open "Create Table- In Memory.sql"
+10. Execute the script to create the DataPointsInMem table.
+This table will be used to store the same simulated IoT device telemetry, but this time using a memory optimized table as well as clustered column store index against all fields (which will support analytic queries) and non-clustered hash indexes on the id field (which will support point lookups common to transactional queries).
+```
+CREATE TABLE [DataPointsInMem] (
+	-- ID should be a Primary Key, fields with a b-tree or hash index
+	Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
+	[Value] decimal(18,5),
+	[TimestampUtc] datetime,
+	DeviceId int,
+	--  This table should have a columnar index
+	INDEX Transactions_CCI CLUSTERED COLUMNSTORE
+) WITH (
+	--  This should be an in-memory table
+	MEMORY_OPTIMIZED = ON
+);
+
+--  In-memory tables should auto-elevate their transaction level to Snapshot
+ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ; 
+```
+
+## Load initial data
+
+1. Within Visual Studio, open "Load Sample Data.sql"
+2. Adjust the path to the DataPoint.bcp file so it matches the location of your BCP file within the SQL on Linux container (if necessary) and save the script. Recall the path /var/opt/mssql will map to C: in any T-SQL scripts executed against SQL Server on Linux.
+3. Execute the script to load each table with 4 million rows worth of sample data. This will take some time to complete.
+
+## Execute the sample analytics query
+
+1. Within Visual Studio, open "SampleQueries - DiskBased.sql".
+2. Execute the script to summarize the time series data stored in the disk based table.
+3. When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute. Query time is shown in the bottom right of the document window in Visual Studio.
+![alt text][Disk Based Results]
+
+[Disk Based Results]: images/DiskBasedResults.png "Disk Based Results"
+4. Now, execute the script to summarize the time series data stored in the memory-optimized table, in "SampleQueries - InMemory.sql".
+When the script completes, observe that 334 rows were returned.Take note of how long the query took to execute.
+You should notice that the performance of the query against the memory-optimized table runs between 2x-10x faster than the same query, running against the same data stored in a disk based table. Query time is shown in the bottom right of the document window in Visual Studio.
+![alt text][In-Memory Results]
+
+[In-Memory Results]: images/InMemoryResults.png "In-Memory Results"
+
+## Execute the queries under load
+
+1. Within Visual Studio, Solution Explorer, expand the SqlLoadGenerator project and then open "App.config".
+2. Locate the connection string with the name "SqlConnection" and modify it so it points to your instance of SQL Server on Linux.
+3. Save the App.config.
+4. From the Debug menu, select Start Without Debugging.
+5. At the prompt, choose option 1 to target the disk based table.
+You should see log entries when every 1000 rows are inserted.
+Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
+6. Open "SampleQueries - DiskBased.sql".
+7. Execute the script to summarize the time series data stored in the disk based table.
+8. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
+9. Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
+10. Close the console load generator.
+11. Run the SqlLoadGenerator again.
+This time at the prompt, choose option 2 to target the memory-optimized table.
+You should see log entries when every 1000 rows are inserted.
+12. Leave the console running (it should run for about 3 minutes) and return to Visual Studio.
+13. Open "SampleQueries - In Memory.sql".
+14. Execute the script to summarize the time series data stored in the disk based table.
+15. Observe that more than 334 rows were returned.Take note of how long the query took to execute.
+Repeat the query a few times, waiting a few seconds in between queries to get a sense of how long the query takes, even as new rows are inserted by the load generator.
+16. Close the console load generator.
+
+## Conclusion
+You should observe that while neither query was affected by the heavy insert load, the query against the analytics query continued to run 2x-10x faster than the same query against the disk-based table.

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/Images/DiskBasedResults.png

@@ -0,0 +1,15 @@
+
+-- For SQL Server 2016, create the Database with a master data file (mdf), 
+-- the log data file (ldf) and a separate filegroup to support memory-optimized tables.
+CREATE DATABASE [Telemetry]
+ CONTAINMENT = NONE
+ ON  PRIMARY 
+( NAME = N'Telemetry', FILENAME = N'C:\data\Telemetry.mdf' , SIZE = 128MB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB ), 
+ FILEGROUP [TelemetryInMem] CONTAINS MEMORY_OPTIMIZED_DATA  DEFAULT
+( NAME = N'Telemetry_mem', FILENAME = N'C:\data\Telemetry_mem' , MAXSIZE = UNLIMITED)
+ LOG ON 
+( NAME = N'Telemetry_log', FILENAME = N'C:\data\Telemetry_log.ldf' , SIZE = 128MB , MAXSIZE = 2048GB , FILEGROWTH = 65536KB )
+GO
+
+ALTER DATABASE [Telemetry] SET COMPATIBILITY_LEVEL = 130
+GO

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/Images/InMemoryResults.png

@@ -0,0 +1,24 @@
+USE Telemetry;
+GO
+
+DROP INDEX IF EXISTS dbo.DataPointsDiskBased.IX_DeviceId;
+DROP INDEX IF EXISTS dbo.DataPointsDiskBased.IX_Timestamp;
+DROP TABLE IF EXISTS dbo.DataPointsDiskBased;
+
+CREATE TABLE [DataPointsDiskBased] (
+	Id bigint IDENTITY NOT NULL PRIMARY KEY CLUSTERED,
+	[Value] decimal(18,5),
+	[TimestampUtc] datetime,
+	DeviceId int,
+);
+
+CREATE NONCLUSTERED INDEX IX_DeviceId   
+    ON dbo.DataPointsDiskBased (DeviceId);   
+GO  
+
+CREATE NONCLUSTERED INDEX IX_Timestamp   
+    ON dbo.DataPointsDiskBased (TimestampUtc);   
+GO  
+
+
+

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/README.md

@@ -0,0 +1,22 @@
+USE Telemetry;
+GO
+
+DROP INDEX IF EXISTS dbo.DataPointsInMem.IX_DeviceId;
+DROP INDEX IF EXISTS dbo.DataPointsInMem.IX_Timestamp;
+DROP TABLE IF EXISTS dbo.DataPointsInMem;
+
+CREATE TABLE [DataPointsInMem] (
+	-- ID should be a Primary Key, fields with a b-tree or hash index
+	Id bigint IDENTITY NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 30000000),
+	[Value] decimal(18,5),
+	[TimestampUtc] datetime,
+	DeviceId int,
+	--  This table should have a columnar index
+	INDEX Transactions_CCI CLUSTERED COLUMNSTORE
+) WITH (
+	--  This should be an in-memory table
+	MEMORY_OPTIMIZED = ON
+);
+
+--  In-memory tables should auto-elevate their transaction level to Snapshot
+ALTER DATABASE CURRENT SET MEMORY_OPTIMIZED_ELEVATE_TO_SNAPSHOT=ON ; 

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/SQL Scripts/Create Database.sql

@@ -0,0 +1,17 @@
+USE Telemetry;
+GO
+
+BULK INSERT Telemetry.dbo.DataPointsDiskBased
+    FROM 'C:\data\datapoints.bcp'
+    WITH (
+        DATAFILETYPE = 'native'
+    );
+GO
+
+BULK INSERT Telemetry.dbo.DataPointsInMem
+    FROM 'C:\data\datapoints.bcp'
+    WITH (
+        DATAFILETYPE = 'native'
+    );
+GO
+

samples/features/in-memory/code-snacks/sqlonlinux/in-memory-columnstore/SQL Scripts/Create Table- Disk Based.sql

@@ -0,0 +1,21 @@
+USE Telemetry;
+
+SELECT 
+	Count(*) Counted, 
+	Max([Value]) MaxValue,
+	Avg([Value]) AvgValue,
+	Min([Value]) MinValue,
+	DatePart(YYYY, TimestampUtc) [year],
+	DatePart(MM, TimestampUtc) [month],
+	DatePart(DD, TimestampUtc) [day],
+	DatePart(hh, TimestampUtc) [hour],
+	DatePart(mi, TimestampUtc) [minute],
+	DatePart(ss, TimestampUtc) [second]
+FROM DataPointsDiskBased
+GROUP BY
+	DatePart(YYYY, TimestampUtc),
+	DatePart(MM, TimestampUtc),
+	DatePart(DD, TimestampUtc),
+	DatePart(hh, TimestampUtc),
+	DatePart(mi, TimestampUtc),
+	DatePart(ss, TimestampUtc)