troubleshooting Question

Sql Server column store compression for row store queries

Avatar of websss
websssFlag for Kenya asked on
DatabasesMicrosoft SQL ServerPostgreSQLMicrosoft SQL Server 2005
10 Comments2 Solutions134 ViewsLast Modified:
I have a time series database with the the following columns

  id BIGINT  -- PRIMARY KEY,
  imei_number BIGINT NOT NULL,
  gps_datetime datetime  NOT NULL,
  event_id INT,
  speed INT,
  latitude decimal(10,6),
  longitude decimal(10,6),
  raw_data varchar NULL

The are currently 4 billion rows in this one table, and growing by 1 billion a month.
I need to do a couple of things

1) I would like to partition / split the data onto different drives, perhaps "now to 3 months" on current drive, and "3+ months and >" to a different drive.
2) I would like to compress the older data  "3+ > months" for example,

On the compression point, all queries I do are virtually the same
for example, these are typical queries
Where imei_number = 1234
And gps_datetime between 'date1' and 'date'
and event_id in 1,5,14 --this is sometimes not used

My Clustered index is on the following (inc order)
imei_number
gps_datetime
id

I also have a non clustered index on
imei_number
event_id
id

and a non clustered index (unique primary key) on
Id


I dont have much experience with compression, but I've been trying out postgres with timescaledb and they have a column store compression option where you define a segment by key, i've used imei_number as SEGMENT BY (so it sticks everything in b-tree/disc by imei number first) and the compression is outstanding (180 x), and query performance the same as normal row store.

On my SQL server I've run some stats on page/row compression and the saving are not groundbreaking (about 20-30% based on an SP that estimates)
Therefore I'd like to try column store compression but not sure if it has something similar to Segment by where it orders everything by IMEI_NUMBER in b-tree/disc

Can anyone suggest some direction and options for me to try please, i need to have a much higher compression rate but not kill performance
Luckily data older than 3 months is rarely queried

its on Sql server 2017 enterprise.
Join the community to see this answer!
Join our exclusive community to see this answer & millions of others.
Unlock 2 Answers and 10 Comments.
Join the Community
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 2 Answers and 10 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros