Comments on: NoSQL and the Relational Model: don’t throw the baby out with the bathwater http://matthew.yumptious.com/2009/07/databases/nosql-and-the-relational-model-dont-throw-the-baby-out-with-the-bathwater/ Geek musings touching on music, maths, applied computer science, and software development with Ruby, Javascript and others Tue, 06 Apr 2010 17:03:27 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Matthew http://matthew.yumptious.com/2009/07/databases/nosql-and-the-relational-model-dont-throw-the-baby-out-with-the-bathwater/comment-page-1/#comment-315 Matthew Tue, 06 Apr 2010 17:03:27 +0000 http://matthew.yumptious.com/?p=37#comment-315 Nathan - that sounds like a good PhD thesis topic :) might be worth trying to address some of the sub-problems with a reasonable degree of rigour before attempting to solve them all at once though? Eg one big concern with that approach would be the scope for DoS vulnerabilities if remote users are allowed to construct arbitrary queries. You'd need a way to sandbox things in terms of computational resources, not just in terms of authorisation for the logical model. And I guess the general problem is that of the 'leaky abstraction'. It's not just the physical data layout that needs to be optimised beneath the relational abstraction, it's dynamic policies relating to dataflow, indexing, cache expiry policies, other trade-offs between being consistent and up-to-date and being scalable, especially in distributed scenarios. If you could find a clean formalism for expressing these kinds of policies and trade-offs and optimisations for a logical model that's expressed relationally, that would be a big part of the puzzle. Probably it has been done already to some extent, but like you say, buried in old papers behind pay-walls and a Really Big Problem to take all that research and turn it into tools that engineers can understand and respect and use. Nathan – that sounds like a good PhD thesis topic :) might be worth trying to address some of the sub-problems with a reasonable degree of rigour before attempting to solve them all at once though?

Eg one big concern with that approach would be the scope for DoS vulnerabilities if remote users are allowed to construct arbitrary queries. You’d need a way to sandbox things in terms of computational resources, not just in terms of authorisation for the logical model.

And I guess the general problem is that of the ‘leaky abstraction’. It’s not just the physical data layout that needs to be optimised beneath the relational abstraction, it’s dynamic policies relating to dataflow, indexing, cache expiry policies, other trade-offs between being consistent and up-to-date and being scalable, especially in distributed scenarios. If you could find a clean formalism for expressing these kinds of policies and trade-offs and optimisations for a logical model that’s expressed relationally, that would be a big part of the puzzle. Probably it has been done already to some extent, but like you say, buried in old papers behind pay-walls and a Really Big Problem to take all that research and turn it into tools that engineers can understand and respect and use.

]]> By: Nathan Sobo http://matthew.yumptious.com/2009/07/databases/nosql-and-the-relational-model-dont-throw-the-baby-out-with-the-bathwater/comment-page-1/#comment-314 Nathan Sobo Tue, 06 Apr 2010 16:40:35 +0000 http://matthew.yumptious.com/?p=37#comment-314 You are right on. I've developed an as yet unreleased relational modeling API (called Monarch) that allows you to build up relations with algebraic operations (select, join, project, etc). There's a server side component in Ruby and a client-side component in JavaScript. You can actually construct a relation on the client and send it to the server, where it is re-interpreted in terms of a set of user-specific views that sandbox the queries to a permitted subset of the database. The tree of algebraic operations can of course be compiled to SQL and run against a relational db (as has already been done by Arel and SQL Alchemy), but you can also do other things with it. For example, you can subscribe to a relation to receive updates on its contents from the browser. I'm basically recapitulating the concept of a materialized view at the application level. It's a great model for subscribing to real time updates on arbitrary slices of the domain. Anyway, my thought is that by capturing the relational model at the application level, it may be possible to leverage it in a manner less monolithic than just sending SQL to a relational database. I have this idea that I could use the event propagation system to maintain my own materialized views that are stored in Cassandra. By making the application itself aware of the relational model, perhaps physical data layout and access patterns can be better controlled. Some relations can be maintained automatically. Others may need programmer intervention that explicates ad hoc optimizations. Etc. But bringing the theoretical underpinnings closer to the programming language means the language can be leveraged in making these choices. Just dipping my toe in the water of working with relations has unveiled a daunting array of theoretical challenges. For instance, if I do maintain a tree of interdependent materialized views, how do I detect and exploit any redundancy shared between the relations that define them? My research quickly leads me into heavy math and lots of papers behind academic pay walls. But the research exists and is ongoing... why limit the knowledge to the innards of proprietary databases? Anyway, drop me a line and I'd love to add you as a collaborator on the repository if you're interested. You are right on. I’ve developed an as yet unreleased relational modeling API (called Monarch) that allows you to build up relations with algebraic operations (select, join, project, etc). There’s a server side component in Ruby and a client-side component in JavaScript. You can actually construct a relation on the client and send it to the server, where it is re-interpreted in terms of a set of user-specific views that sandbox the queries to a permitted subset of the database. The tree of algebraic operations can of course be compiled to SQL and run against a relational db (as has already been done by Arel and SQL Alchemy), but you can also do other things with it. For example, you can subscribe to a relation to receive updates on its contents from the browser. I’m basically recapitulating the concept of a materialized view at the application level. It’s a great model for subscribing to real time updates on arbitrary slices of the domain. Anyway, my thought is that by capturing the relational model at the application level, it may be possible to leverage it in a manner less monolithic than just sending SQL to a relational database. I have this idea that I could use the event propagation system to maintain my own materialized views that are stored in Cassandra. By making the application itself aware of the relational model, perhaps physical data layout and access patterns can be better controlled. Some relations can be maintained automatically. Others may need programmer intervention that explicates ad hoc optimizations. Etc. But bringing the theoretical underpinnings closer to the programming language means the language can be leveraged in making these choices. Just dipping my toe in the water of working with relations has unveiled a daunting array of theoretical challenges. For instance, if I do maintain a tree of interdependent materialized views, how do I detect and exploit any redundancy shared between the relations that define them? My research quickly leads me into heavy math and lots of papers behind academic pay walls. But the research exists and is ongoing… why limit the knowledge to the innards of proprietary databases? Anyway, drop me a line and I’d love to add you as a collaborator on the repository if you’re interested.

]]> By: SQL is Dead. Long Live SQL. : Dataspora Blog http://matthew.yumptious.com/2009/07/databases/nosql-and-the-relational-model-dont-throw-the-baby-out-with-the-bathwater/comment-page-1/#comment-151 SQL is Dead. Long Live SQL. : Dataspora Blog Wed, 25 Nov 2009 10:58:37 +0000 http://matthew.yumptious.com/?p=37#comment-151 [...] others have noted, some balance is in [...] [...] others have noted, some balance is in [...]

]]>