Handling SQL servers for IP Ranges - SQL Server

Hello all, I have the following challenge and need some help:

I have a SQL Server 2008 Database Table that has around 10-20 million records added per month. The current table size is around 200+ million records.

This table is used to keep track of Usage Data Records.

Each month we take the associated data and slice it and dice it for a number of reasons; reporting and billing. As part of the billing aspect we need to bill usage for IP addresses within a given IP range.

Note, this field is also currently not indexed.

For example only bill users with IP address range from 192.168.x.100 to 150

I'm at a loss at how to best process this efficiently. Does anyone have any recommendations on a good way to do this?

I thought about converting all the IP's into decimal, having an index on that and having another table with the full list of all possible matches. However, at the same time this seems inefficient. Especially if i was asked the following:

Get all uses against the following 10.x.x.100-150. This would have hundreds of thousands of combinations.

Look forward to responses. Thank you in advance
xathras1982Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ste5anSenior DeveloperCommented:
converting all the IP's into decimal

Huh? How do you store it right now? An IP is a number, thus it should be stored as such. You should always store data in its domain data type.

Especially as working with masks produces consecutive numbers. It just simple math.
1
xathras1982Author Commented:
Currently the data is stored in a varchar 15. e.g. 192.168.1.100.
Please can you provide an example of how this would work?
0
ste5anSenior DeveloperCommented:
IPv4.
1
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

xathras1982Author Commented:
I understand that mysql has functions that will convert it. But how can i handle queries to be specific in the examples

Storing as a number helps but i need to see a practical applied example

What if i needed to return the following:

10.21.x.100-200
10.22.x.101-200
10.x.x.100-105
0
ste5anSenior DeveloperCommented:
You need to apply just two masks. On for the first fix part and on for the second part. btw, you've tagged it as MSSQL..

E.g.

USE tempdb;
GO

/* Some math for understanding:
DECLARE @IP BIGINT = CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 1 * POWER(256, 0);

DECLARE @MaskOctet1 BIGINT = CAST(255 AS BIGINT) * POWER(256, 3);
DECLARE @MaskOctet2 BIGINT = CAST(255 AS BIGINT) * POWER(256, 2);
DECLARE @MaskOctet3 BIGINT = CAST(255 AS BIGINT) * POWER(256, 1);
DECLARE @MaskOctet4 BIGINT = CAST(255 AS BIGINT) * POWER(256, 0);

SELECT  CAST(192 AS BINARY(1)) ,
        CAST(168 AS BINARY(1));

SELECT  CAST(@IP AS BINARY(4)) ,
        CAST(@IP & @MaskOctet1 AS BINARY(4)) ,
        CAST(@IP & @MaskOctet2 AS BINARY(4)) ,
        CAST(@IP & @MaskOctet3 AS BINARY(4)) ,
        CAST(@IP & @MaskOctet4 AS BINARY(4));
*/

CREATE  TABLE SampleData
    (
      IP BIGINT NOT NULL
                PRIMARY KEY
    );

INSERT  INTO SampleData
VALUES  ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 1 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 2 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 3 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 4 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 5 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 6 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 7 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 8 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 9 * POWER(256, 0) ),
        ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 10 * POWER(256, 0) ),
        ( CAST(172 AS BIGINT) * POWER(256, 3) + 21 * POWER(256, 2) + 1 * POWER(256, 1) + 9 * POWER(256, 0) ),
        ( CAST(172 AS BIGINT) * POWER(256, 3) + 21 * POWER(256, 2) + 1 * POWER(256, 1) + 10 * POWER(256, 0) );

-- Filter for 192.168.x.2-5
DECLARE @PrefixMask BIGINT = 0xFFFF0000;
DECLARE @SuffixMask BIGINT = 0xFF;

-- Scan :(
SELECT  CAST(S.IP AS BINARY(4))
FROM    SampleData S
WHERE   S.IP & @PrefixMask = 0xC0A80000
        AND S.IP & @SuffixMask BETWEEN 2 AND 5;

-- Seek :)
SELECT  CAST(S.IP AS BINARY(4))
FROM    SampleData S
WHERE   S.IP BETWEEN 0xC0A80000 AND 0xC0A8FFFF
        AND S.IP & @SuffixMask BETWEEN 2 AND 5;

DROP TABLE SampleData;

Open in new window

1

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
xathras1982Author Commented:
Thank you. I will take a look and test this out today.
0
xathras1982Author Commented:
Ste4an, thank you for your help today. Took me a little while to understand what is going on here.
Essentially the current IP field we have today needs to be replaced into an IP as per your example. I will then introduce the query per your suggestion to run against the octet range I need.

I had a plan in a small environment will be interesting to see results on our 200m + table
0
ste5anSenior DeveloperCommented:
It will be faster than any string based approach. Just ensure that the IP column is indexed.

As were using two masks, you may consider a reverse approach for the suffix mask. E.g.

USE tempdb;
GO

CREATE  TABLE SampleData
    (
      IP BIGINT NOT NULL
                PRIMARY KEY ,
      ReverseIP BIGINT NOT NULL
    );
GO

CREATE INDEX IX_Reverse ON dbo.SampleData ( ReverseIP );

INSERT  INTO SampleData
VALUES  ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 1 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 1 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 2 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 2 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 3 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 3 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 4 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 4 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 5 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 5 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 6 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 6 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 7 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 7 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 8 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 8 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 9 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 9 * POWER(256, 3) ) ),
        ( ( CAST(192 AS BIGINT) * POWER(256, 3) + 168 * POWER(256, 2) + 1 * POWER(256, 1) + 10 * POWER(256, 0) ),
          ( CAST(192 AS BIGINT) * POWER(256, 0) + 168 * POWER(256, 1) + 1 * POWER(256, 2) + 10 * POWER(256, 3) ) ),
        ( ( CAST(172 AS BIGINT) * POWER(256, 3) + 21 * POWER(256, 2) + 1 * POWER(256, 1) + 9 * POWER(256, 0) ),
          ( CAST(172 AS BIGINT) * POWER(256, 0) + 21 * POWER(256, 1) + 1 * POWER(256, 2) + 9 * POWER(256, 3) ) ),
        ( ( CAST(172 AS BIGINT) * POWER(256, 3) + 21 * POWER(256, 2) + 1 * POWER(256, 1) + 10 * POWER(256, 0) ),
          ( CAST(172 AS BIGINT) * POWER(256, 0) + 21 * POWER(256, 1) + 1 * POWER(256, 2) + 10 * POWER(256, 3) ) );

-- Filter for 192.168.x.2-5
DECLARE @PrefixMask BIGINT = 0xFFFF0000;
DECLARE @SuffixMask BIGINT = 0xFF;

-- Scan :(
SELECT  CAST(S.IP AS BINARY(4))
FROM    SampleData S
WHERE   S.IP & @PrefixMask = 0xC0A80000
        AND S.IP & @SuffixMask BETWEEN 2 AND 5;

-- Seek :)
SELECT  CAST(S.IP AS BINARY(4))
FROM    SampleData S
WHERE   S.IP BETWEEN 0xC0A80000 AND 0xC0A8FFFF
        AND S.ReverseIP BETWEEN 0x02000000 AND 0x05FFFFFF;

DROP TABLE SampleData;

Open in new window


But this will require twice the space on disk, ~ 2GB extra. But on the other hand you should have gained enough from moving the string (15 bytes) to BIGINT 8 bytes. So it should not be a problem
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.