For the week ending 26 March 2021
A Resonance Calendar
A collection of information sources that have caught my attention and from which I’ve taken away something (that feels) important. In other words, something what resonates.
Data Policy
There is untapped potential in data that, if harnessed in a rights-respecting manner, could have positive implications for helping to solve societies’ biggest problems or creating new economic value.
Currently, though, much of the data needed to tackle these pressing challenges remains siloed within both public and private sources. At the same time, many jurisdictions, both in the real and digital world, lack comprehensive data protection and data security regulations to protect people’s rights and create sustainable mechanisms for data usage. …
Data Husbandry: Integration and Data Quality
In this episode of the Data Exchange, (Ben Lorica) speaks with Ryan Wisnesky, CTO and co-founder of Conexus, a startup that uses techniques from mathematics and incorporates them into novel tools for data integration, data management, and knowledge management.
Data Lineage
OpenLineage
- GitHub — OpenLineage/OpenLineage
- Datakin launches OpenLineage initiative for data lineage industry standards | VentureBeat
OpenLineage is an Open standard for metadata and lineage collection designed to instrument jobs as they are running. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.
DLT — Digital Ledger Technologies
… & Smart Contracts
The basic idea of smart contracts is that many kinds of contractual clauses (such as liens, bonding, delineation of property rights, etc.) can be embedded in the hardware and software we deal with, in such a way as to make breach of contract expensive (if desired, sometimes prohibitively so) for the breacher. A canonical real-life example, which we might consider to be the primitive ancestor of smart contracts, is the humble vending machine. Within a limited amount of potential loss (the amount in the till should be less than the cost of breaching the mechanism), the machine takes in coins, and via a simple mechanism, which makes a beginner’s level problem in design with finite automata, dispense change and product fairly. Smart contracts go beyond the vending machine in proposing to embed contracts in all sorts of property that is valuable and controlled by digital means. Smart contracts reference that property in a dynamic, proactively enforced form, and provide much better observation and verification where proactive measures must fall short. And where the vending machine, like electronic mail, implements an asynchronous protocol between the vending company and the customer, some smart contracts entail multiple synchronous steps between two or more parties.
… & Decentralized Data Markets
In other words, a TCR (token-curated registry) is curated list which can be monetized for those who provide content. TCRs are built atop smart contracts, where a DApp (decentralized application) creates the UX needed to reach a broader audience of customers.
… & Oracles
An oracle is a mechanism that summons real-world data to the blockchain, for example, the usd price of some asset, an outcome of an election, or an output of some costly computation. It’s difficult to imagine blockchain applications breaking out of its speculative nature and doing something truly useful without making use of oracles.
The existing projects sometimes employ centralized oracles, i.e. a trusted source that commits to sending the correct information to the blockchain. Even the projects that are in part oracles themselves, TrueBit¹ and Augur, still employ a centralized oracle within their design. The reason we’re not seeing applications adopting a fully decentral approach is that the problem of creating a functional decentralized oracle is genuinely tough.
TCRs — Token Curated Registries
Token-curated registries are increasingly common cryptosystems apparently applicable to solving problems in a number of domains. In this document we will provide a more formal but less-than-mathematical view of token-curated registries.
… We believe there is a “right” way to do token-curated registries and that wholesale reuse of a canonical implementation should be possible.
Service Mesh
Luke Kysow, software engineer at HashiCorp, discusses service mesh and Hashicorp’s open source service mesh, Consul. Luke and host Priyanka Raghavan conducted a deep dive of the features of a service mesh, including service discovery, health monitoring, infrastructure support, and security. The last segment focuses on how Consul talks to Envoy and also compares Consul to other service meshes in the industry.
DSML Commercial Ecosystem
Gartner’s Magic Quadrant report on data science and machine learning (DSML) platform companies assesses what it says are the top 20 vendors in this fast-growing industry segment.
Data scientists and other technical users rely on these platforms to source data, build models, and use machine learning at a time when building machine learning applications is increasingly becoming a way for companies to differentiate themselves.
Cloud … Public? Hybrid? Multi-? All of the above?
In every enterprise, companies are consuming services from multiple clouds. It’s becoming almost irrelevant where the workloads run, as long as there is a way to integrate the services and (with smart choices) manage them without a plethora of tools. The proliferation of high-quality cloud services allows us to consume services from the best provider for our specific needs. … Loosely coupled services create applications that are an amalgamation of cloud services — combined over networks — to become cloud native applications.
The design pattern is very similar to what was championed in the late 1990s as service-oriented architecture (SOA). Though instead of merely being API-driven, they are becoming event-driven.
Authorization
- Oso announces $8.2M Series A to simplify authorization for developers
- Using Open Policy Agent for cloud-native app authorization
- OPA: A general-purpose policy engine for cloud-native
As your organization embraces the cloud, you may find that the dynamism and scale of the cloud-native stack requires a far more complicated security and compliance landscape. For instance, with container orchestration platforms like Kubernetes gaining traction, developers and devops teams have new responsibility over policy areas like admission control as well as more traditional areas like compute, storage and networking. Meanwhile, each application, microservice or service mesh requires its own set of authorization policies, for which developers are on the hook.
It’s for these reasons that the hunt is on for a simpler, more time-efficient way to create, enforce and manage policy in the cloud.
Zero Trust Architecture
Part 1 of a 3 blog series.
Organizations have placed a lot of time, effort and capital spend on security initiatives in an effort to prevent security breaches and data loss. Even the most advanced “next generation” application layer firewalls filtering malicious traffic at the network perimeter has only revealed equal if not greater threats within
CI/CD for the Dataset
Machine Learning applications are becoming popular in our industry. However, the process for developing, deploying, and continuously improving them is more complex (when) compared to more traditional software, such as a web service or a mobile application. They are subject to change in three axes: the code itself, the model, and the data. Their behaviour is often complex and hard to predict, and they are harder to test, harder to explain, and harder to improve. Continuous Delivery for Machine Learning (CD4ML) is the discipline of bringing Continuous Delivery principles and practices to Machine Learning applications.
—
For about 15 years, I’ve created a weekly resonance calendar for myself, based on the the past week’s articles, blog posts, reports, books, presentations, podcasts, videos, maps and scrawls on (paper) placemats. It’s usually a list that goes into my notebook, annotated with a few descriptions, highlights and epithets. It never looks as thoughtfully created or well-tended as this list. But I thought it worthwhile to pare it down and post the result as a means of starting or continuing conversations. You can find me on LinkedIn and Twitter. If some of this speaks to you … or irritates you … , let me know.
