Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

Querying a Disorganized Database

Posted on 2011-09-19
8
Medium Priority
?
504 Views
Last Modified: 2012-05-12
Hi there, I have a disgustingly organized database to extract data from.  I have some experience in SQL, but not quite what I would require in order to get data in the exact form that I require for an automated extraction to work.  I have built a barebones query, but it functions under an assumption that is not always true.  I think I need help building a loop, but since I have never seen anything quite this bad, I think I need another opinion before I go crazy trying to build something.  Any help would be appreciated.

The database is built up in a series of tables where each table represents an entity in an oil field (a well).  Each of these tables has a timestamp (point type char(30) for reasons beyond my comprehension), and three variables to pull regarding the entity (one shown in the example).  The trick, and the most frustrating part of this, is that each of the three variables is set up across 40 columns in the table, each with different data through time.  That is, there is a variable for pressure, and it has 40 separate data points in a single row.  The output I require has a timestamp for each data point, and has each data point in a single column, as such, the data has to be reorganized and 39 of the 40 data points require a synthesized timestamp (see attached).

My current query functions based on a 15 second space between all data points and therefore does not display all data properly.  It also lives on the wrong side of a VPN so I had to screenshot it, I apologize.  It sets up to retrieve an external table name (for the well name) and a timestamp for the where statement, and then builds a table out of a bunch of union statements.  I imagine that this is sub-optimal, but as mentioned, I am inexperienced with SQL.

I guess what I really need help with is, is this a ridiculous way of pulling data/is there a better way, and if not really, how do I apply the interpolation of data?  I have set up the math for interpolation but have no idea how to make the logic occur in a structured manner.

Advice and instruction is appreciated.

Example.xlsx
Interoplation.PNG
Query.PNG
0
Comment
Question by:MeraGroup
  • 4
  • 3
8 Comments
 
LVL 42

Expert Comment

by:dqmq
ID: 36562128
The union structure looks OK to me.

To minimize dynamic SQL, I would consider creating a view for each table that does all the unions and returns the desired structure.

As for the timeinterval, can you describe what you need?
0
 

Author Comment

by:MeraGroup
ID: 36562651
Hi dqmq, thank you for the response.  I guess what I really need is a way to interpolate dates and times for all of the parameters that I union based on interpolation between the row's timestamp and the next row's timestamp.

Thanks!
0
 
LVL 50

Expert Comment

by:Lowfatspread
ID: 36564617
what control over the database do you have?

ie could you introduce triggers on the individual tables to maintain 1 common table with data for all the wells, and possible sort out your problem with the 40 columns as well?

or could you reorganise the database to just have the one well table and implement the existing solution as a set of views for each individual well , possibly with an instead of trigger to handle the updating?
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:MeraGroup
ID: 36567248
While a fantastic idea, Lowfatspread, I do not have sufficient control of the database to do something like that, the database is in a control network.  The most crucial piece of this is synthesizing a timestamp to be associated with each data point in the row, and being able to view the timestamps and data points together.  Thanks for the suggestion!
0
 
LVL 42

Expert Comment

by:dqmq
ID: 36568344
So you are saying, the time interval between row 1 and row 2 should be divided evenly into 120 time intervals, one for each column?
0
 

Author Comment

by:MeraGroup
ID: 36568394
The time interval should be divided by 40, as there are 40 columns per parameter, each division of 40 will be applied to the three variables - does that make sense?

Thank you again.
0
 
LVL 42

Accepted Solution

by:
dqmq earned 2000 total points
ID: 36568838
--this view returns the duration of all but the last row
Create View Well1View1
as
select t1.*, t2.timestamp - t1.timestamp as duration
from Well1Table T1, Well1Table T2
where
t2.timestamp = (select min(t3.timestamp) from Well1Table1 t3 where t3.timestamp > T1.timestamp)


--this view synthesizes the timestamp
Create View Well1View2
as
Select (timestamp + (duration/40*(1-1))) as timestamp, presA1, presB1, presC1 from Well1View1
union all
Select (timestamp + (duration/40*(2-1))) as timestamp, presA2, presB2, presC2 from Well1View1
...
Select (timestamp + (duration/40*(40-1))) as timestamp, presA40, presB40, presC40 from Well1View1
...


Declare @SQL1 varchar(4000)
SET @SQL1 = Select * from ' + @WellName + 'View2'
insert into Temptable
   exec (@SQL)

 
 
0
 

Author Closing Comment

by:MeraGroup
ID: 36571952
Done.  After organizing the data, the database had even worse quality information than I had originally suspected.  Regardless, the query is set up using a similar form to what's above.  Thank you so much for taking the time to assist me with that.  If you find yourself in Saskatchewan, Canada, I owe you drink.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever needed a SQL 2008 Database replicated/mirrored/log shipped on another server but you can't take the downtime inflicted by initial snapshot or disconnect while T-logs are restored or mirror applied? You can use SQL Server Initialize from Backup…
In part one, we reviewed the prerequisites required for installing SQL Server vNext. In this part we will explore how to install Microsoft's SQL Server on Ubuntu 16.04.
Via a live example, show how to shrink a transaction log file down to a reasonable size.
Viewers will learn how to use the SELECT statement in SQL to return specific rows and columns, with various degrees of sorting and limits in place.

926 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question