Data Masking with Teiid and Delphix/DMSuite

[omarbkhan]

Hi,

I saw a demonstration of Delphix DMSuite which seems to have some compelling masking capabilities.

few links:

Masking Engine Admin Guide

Delphix Masking Quick Start Guide

Masking Data Source Support Matrix

Delphix Community

I'm not familiar with the product and it was just introduced to me, but was hoping we could use Teiid with it as it's Data Virtualization capabilities are excellent from what we have been leveraging to date?

It seems as Delphix provides two mechanisms to perform the data masking.

#1  A job based approach

I was wondering since it looks like the DMSuite product could use Teiids JDBC with the sources in a VDB and then run an job to insert back into PostgresSQL either through a VDB or directly to PostgresSQL. From my understanding is this product will do the data masking in memory.

#2  Some daemon/engine that runs and collects changes and masked them as changes come in. I'm not sure how the daemon works. Kind of sounds like Debezium with data masking (I only have high level understanding of it)

Right now we are doing some redaction from data sources in SQL and not using Teiid's column masking (which may be helpful as well) and materializing the redacted data in PostgresSQL.

I believe the product could work well together, but again I'm not sure. Any feedback appreciated as we evaluate our options.

[rareddy]

I am not familiar with Delphix, but their usage of "Data Virtualization" is different from Teiid's. I believe it is much more about re-playing the production data to the development environment. The Delphix case you are storing the masked copy back to Postgres database, where as in Teiid there is no intermediate storage (unless materialized). Also Teiid's column masking works based on the individual user credentials not common to all (you could make it all)

Ramesh..

[omarbkhan]

Thanks Ramesh for the quick response!

There terminology does seem different they use the term VDBs too, but in a very different manner so I'm even more confused.

I don't see why it would not work for Teiid to serve as a JDBC source and virtualize our various federated source system and target back to Teiid to a Delphix based job? For example It looks like (at the time of this writing, Oracle 12R2  and Hadoop/Hive not supported in their matrix for Delphix out of the box, but Teiid I believe does just fine - so there are a couple value add(s) there for leveraging Teiid for data virtualization  )

It seems feasible given Teiid offers both a Postgres port and Teiid custom ?

Given their compatibility matrix , at least the Postgres port might work. They also have Generic JDBC, so we probably could use Teiids custom driver there too. Not sure, but it seems possible.

Hmm... Interesting, how could we extend the column masking to all users?

[rareddy]

Yeah, we had same experience, Delphix overloads terminology of "Data Virtualization" from what is traditionally defined by other vendors. To top it off, look like they also have VDB concept

Yes, it should work fine with Teiid, if they provide any RDBMS source that has JDBC driver. The Generic JDBC may be better to try out first.

>>Hmm... Interesting, how could we extend the column masking to all users?

The masking in the Teiid means, show certain sensitive data to a user with some particular role, if not mask it. Where as in Delphix case might be just white wash the data itself. Teiid can do that too, just do not expose that column, or if you expose it put in a random data or null there in the view definition. If you want you can even materialize it.

IMO, you can think of Delphix as source control system of data, where you can move to particular version data to another. For example, slide the time to yesterday noon time of your database and run test, then move the slider the back to current and run test again. If there is regression then it is exposed by the data. Also, these systems are very clever not to copy whole data, but only deltas in between. Thus fast replication etc.

[himsofficial]

HI Omer,

I have worked with Delphix , the masking can plug into any jdbc source , its biggest USP is that it maintains  Referential Integrity between different sources so any downstream integration works.

--HM

[dbakevlar]

Hi Omar,

I'll start out by letting you know I work for Delphix...

We do differ in our technology from what you commonly refer to as data virtualization.  We actually create a "golden copy" that is continually updated with the transactional data via the log technology native to the RDBMS and then we can apply that to as many virtual copies as required for non-production environments.  This can be done for the database or application tier, so we have many customers who also virtualize the database installation, not just the datafiles.  With that said, you can use the masking engine to mask whatever data you wish, it's quite robust and would mask the data to any or all your virtual databases.  Best practice would recommend encrypting production data to secure it, decrypting it to those that need to see it without the encryption, (and have the key.)  Masking is irreversible and does maintain the referential integrity, so joins still function, performance is maintained and we can even mask flat files, (if you have source data outside the RDBMS.)  If you have any questions, please feel free to email me at kellyn at delphix.com

Thanks!

Kellyn Pot'Vin-Gorman

Technical Intelligence Manager, Delphix

[omarbkhan]

Thanks HM!

[rareddy]

Kellyn,

Do you guys provide the log scraping for each individual database or do you require products like golden gate in case of Oracle? Do you guys use any messaging products underneath?

This looks much like event source based aggregation or propagation of data, very neat.

Ramesh..

[dbakevlar]

Hi Ramesh,

Yes, the Delphix engine consumes the archive logs directly, keeping the "golden copy" up to date and able to refresh the vdbs and recover any PIT or scn for unique vdb.

No other product is required for the logs.  Can you elaborate on messaging?  We work with qmail  and others.

There's no aggregation -  full read/write copies that can be refreshed in just minutes.

Thanks!

Kellyn

[rareddy]

Kellyn,

Interesting. Typically every RDMS transactions logs do vary in information access and format. Can you tell me what databases you have support for? On messaging, what I was saying, is when you read the transaction log, you need to pass on that event to other downstream applications or databases who are keeping in sync with "golden copy", for that do you use any AMQP/Kafka kind of messaging or your own proprietary ways to propagate the event?

If you are not aware there is a project called Debezium which also does similar event capture using Kafka, but does not have support for many databases yet. But what you guys do with that events is pretty useful. I suspect none of your technologies are open source.

Ramesh..

[dbakevlar]

Hi Ramesh,

Database platforms we're certified on:  Oracle, Microsoft, ( Sql Server, Azure) Amazon, DB2, Sybase, Hana and MySQL.  We also support Hadoop and application tier. 

Messaging with Kafka could be added as hooks, (post refresh scripting) just as you do now, but it'd be automated into our "template".

We are an enterprise solution, so not open source, but that just means our product does what it says it does and our support is fantastic....:)

Kellyn

[rareddy]

My comment about open source about reuse of your technology elsewhere for other use cases. For example in Teiid. BTW, Teiid does not do change data capture, Debeizuim does. They are totally independent projects. Teiid does have an interest in integrating with such technologies.

Teiid is also basis for Redhat Data Virtualization product, that comes with Red Hat "enterprise" support. Teiid originated from commercial ventures in the past life before it became Open Source, so it is a very mature technology.

Thank you for the information about Delphix.

Ramesh..

[dbakevlar]

Absolutely!  Let me know if you have any other questions, Glad to answer them!

Kellyn