Data Modeling Techniques for big data

Big data challenges traditional data modeling techniques

The onslaught of big data, with its high data volumes and diverse data structures, has given rise to new technologies in the form of NoSQL, Hadoop, Spark and the like. NoSQL, particularly, calls for changes in established data modeling techniques.

Some basic learning is order, too, when it comes to NoSQL databases, such as MongoDB, Cassandra and Redis advised at least one data veteran during the recent Enterprise Data World (EDW) 2016 conference in San Diego.

“Nobody is born knowing NoSQL,” said Ted Hills, an enterprise data architect at information provider LexisNexis, based in New York. Data modelers should realize everything they know about logical modeling is still true, he continued, but they should also realize that “NoSQL gives a richer toolbox,” with which developers can work.

Data pros should be ready to the accept change, and to embrace the new capabilities of big data tools, Hills said, even though the tools lead to changes in existing modeling methods.

Hills, the author of the recently released NoSQL and SQL Data Modeling, suggested a need for new modeling notations that embrace NoSQL functionality.

Data modeling techniques meet schema-on-write

One effect of the NoSQL side of big data development has been to delay schema creation. The early definition of the data schema was once a lynchpin of data quality practices and a prerequisite for just getting a project going.

Schema creation may be moving to a different stage in the development cycle, according to Karen Lopez, data architect and principal consultant at InfoAdvisors.

“It’s not that we don’t care about quality. It’s that we are not caring about the schema upfront,” she said. This doesn’t mean designs become “schema-less.” Instead, they come to support something akin to “schema-on-read” model, she said.
Hills concurred, saying people’s enthusiasm for NoSQL becomes tempered as they wonder “what is all that stuff I slammed into my [database management system]?” He expects data modeling to move from a solely prescriptive mode to one that includes some descriptive modeling, where data schemas are created after initial data development.

Embrace the Agile – Data modeling techniques.

The term descriptive is apt for data architecture today, according to Lakshmi Randall, an independent analyst and consultant on hand at EDW 2016. “Now, as far as data modeling is concerned, things are more descriptive.
Instead of trying to plan everything out ahead of time, you see use cases developed sort of on the fly,” she said.

One type of NoSQL database may be particularly related to this style of data design, Randall said. That is the graph database.
The NoSQL graph database has the ability to capture information on the many interactions that occur in, for example, Web and customer relationship systems, she said. In this way, it can be helpful in creating a descriptive model of a wide-ranging application, she added.

For his part, LexisNexis’ Hills emphasized the challenge of NoSQL design is part of a larger trend to make both businesses and software development more flexible.

The trend bears the name of Agile methodology, and many of its principles — for example, projects organized around early delivery, short but frequent iterations and moderate use of upfront schema – are alien to traditional modeling.

Agile means teams work in smaller project “chunks” than before, he said, and the business side of the organization is involved in every step.

“Data modelers should learn from Agile development, working in small sprints. Our traditional data modeling world has been more of a Waterfall process,” Hills said, referring to a project style that has become associated with
multiyear development.

Hills said there is nothing wrong with beginning to store data in NoSQL prior to schema creation. He urged data modelers to be open-minded to the value of the new technology.

“Don’t view that as an enemy — view it as an opportunity,” he said. “You should be on that Agile development team, and talking to the business as much as your development colleagues are.”

Contact Musato Technologies to learn more about our innovative and intelligent solutions that empower business and foster digital transformation.

An article by Jack Vaughan

Author: Gideon E. M

Gideon Ebonde M. is the CEO and Chief Software Architect at Musato Technologies. He is experienced Software developer with a demonstrated history of working in the information technology and services industry. He has a strong engineering professional skilled in Mobile Application Development, Enterprise Software, AI, Robotics, IoT, Servers, Cloud and business application. He is an accomplished DevOps software engineer and a visionary computer scientist and engineer.