As with all project and product methodologies and frameworks, Agile and Scrum are creating new roles within organizations. Three of the more prominent roles are

• Product Owner (PO)
• Product Manager (PM)
• Scrum Master

All three roles are higher-level jobs and come with an attractive salary.

People are frequently confused about the differences between Product Owner, Project Manager, and Scrum Master in an Agile environment. This articles aims to answer that question.

Product Managers, Product Owners, and Scrum Masters are separate roles on an Agile team, and in the Product Manager’s case, outside the Agile team. Each role has its own part to play and can generally be distinguished by these characteristics:

We’ll go into more specific job, skill, and salary information later.

Product Managers can exist anywhere, anytime. Product Owners and Scrum Masters, however, are specific roles in the Scrum Framework.

Product Owners & Scrum Masters are specifically tied to Scrum

Scrum is an Agile Development system that:

• Focuses on goals, small and large
• Takes place in 1- or 2-week-long product development periods, known as sprints
• Often uses Kanban boards to create and organize tasks

Because Scrum is a specific system, it has particular roles. The roles on the team are:

• Product Owners
• Scrum Masters
• Developers

In the Agile mindset, the Scrum team is meant to be self-organized, and all team members are responsible for getting the work done. The Product Owner and Scrum Master are critical parts of developing product capability through using Scrum.

When Scrum teams do not exist, the Product Owner and Scrum Master identities fade away. Many of the tasks performed by these roles may be absorbed in over-arching Product Management roles or by Assistant PMs.

Product Manager: jobs, skills, salary

The scope of a Product Management role varies depending on the stage of the company, the maturity of the Product Management team, and other factors including job location. At its most mature, the PM is primarily responsible for:

• Talking to users
• Organizing strategic path of product
• Creating product development timelines
• Communicating between engineering and business teams

When a product is in its initial stages, or the team is in its infancy, the Product Management team can be found wearing mile-high hats, participating in everything from UX Designing, backend engineering, and design budgeting, along with all the customer communications that are required.

PMs tend to make good money. In the U.S., the average annual salary for a Product Manager is about $107,000 estimated base pay, according to Glassdoor. However, Glassdoor reports an extremely wide salary range for PMs of between $52K and $276K, with possible salary ranges up to $600K.

Specific PM salaries are dependent on what industry the PM works in (tech PMs seem to average in the $130K range and up) and other factors such as company and location. Your mileage may vary, so research the average pay in the industries and companies you would like to work for.

Product Owner: jobs, skills, salary

Where a Product Manager might wear several hats, Product Owner responsibilities in Scrum become very narrow. Like a second basemen on a baseball field, the Product Owner has an extremely specific piece of land to cover with specific people to speak to.

Scrum utilizes a system of tasks and keeps score, often with the help of a product management tool like the Kanban chart or even a simple Excel file. Throughout the sprint, engineers will claim tasks. The role of the Product Owner is to organize and prioritize the tasks for the engineers.

• If there are no tasks, the engineers wait around for the ball to get hit to them.
• If the tasks aren’t prioritized, the engineers develop features that aren’t crucial to the product’s mission.

Generally, the engineers, like the computer systems humankind develops, will build whatever task is on that task list, regardless of its direct impact on the end-product. Thus, it is especially important for the PO to maintain a good list, or those engineers might end up building a whole different product than what the company claims to sell.

Product Owners must do several things to maintain the Scrum backlog. The PO’s primary responsibilities are:

• Translate PMs’ vision to actionable tasks
• Determine day-to-day tasks
• Write user stories for development team
• Prioritize work in the Scrum backlog

Like PMs, Product Owners earn a solid salary. Glassdoor’s research indicates an average U.S. salary of just under $101,000 estimated total base pay for POs. Like PMs, POs have an extremely wide salary range of between $38K and $389K on Glassdoor. Also, like PMs, your PO salary mileage may vary so be sure to compare salaries for target industries and locations.

Scrum Master: jobs, skills, salary

Per ScrumAlliance.org, a Scrum Master helps the Scrum team perform at their highest levels. They protect the team from internal and external distractions so that all project members—especially the development team—can focus on their work.

Scrum Masters facilitate team coordination and support project processes by performing the following roles:

• Ensuring actionable tasks designated by the Product Owner are performed accordingly
• Communicating between team members about evolving planning and requirements
• Facilitating daily Scrum and Sprint Initiatives and other Scrum events
• Conducting meetings
• Managing administrative tasks
• Eliminating external and internal project hurdles
• Other items to help the team perform at their highest level

Scrum Masters also coach team members on delivering results. They are responsible for ensuring that team members understand, execute, and follow Agile principles, processes, and practices throughout the project.

Finally, a Scrum Master communicates with the Product Owner and others within the organization for effectively implementing the Scrum Framework during the project.

Scrum Masters also earn a solid salary. Glassdoor Research specifies an average U.S. salary around $111,000 total base pay for a Scrum Master, with the most likely pay range between $27K and $537K. But like PMs and POs, this is an average that may vary significantly by industry, company, and locations.

PM, PO, and Scrum Master certifications

There are ways to set yourself apart from the crowd by getting a certification in one of these areas. These certifications indicate your specialty and experience, so you can often expect to command a higher salary.

• The Product Manager can take one of a number of tests for certification
• The larger umbrella of Agile Development certifications will teach both PM and PO roles and responsibilities. Among them, the PMI-ACP is the top certification that acts as a catch-all for agile development roles. You may also want to consider obtaining the Scrum Alliance Certified Product Owner (CSPO).
• ScrumAlliance.org offers several certifications on both a Scrum Master track and on a Product Owner track. Each track offers Certified, Advanced Certified, and Certified Professional certificates.

Product Managers, Product Owners, & Scrum Masters: The outlook is good

With companies across all sectors and geographies adopting agile product development or blending it with traditional project management—such as the predictive, agile, and hybrid approaches now included with Project Management Institute (PMI) Project Management Professional certification—it’s likely that Product Managers, Product Owners and Scrum Masters will be around for a long while.

As technology continues its expansion, more people will be needed to explain innovative concepts and applications to the business. Great PMs, POs, and Scrum Masters will continue to contribute to companies’ ongoing innovation efforts in order to stay ahead of competition.

Related reading

• BMC Business of IT Blog
• BMC DevOps Blog
• What’s An App Owner? Application Owner Roles & Responsibilities
• IT Project Management: Concepts, Solutions & Best Practices
• Agile ITSM: How Agile & Service Management Can Work Together
• Today’s Best IT/Tech Certifications: A Complete Guide

Big Data has been labeled the new oil—parallels that describe the value of big data to our economy and business. Like oil, data has degrees of value based on its quality, integrity, and refining process. Different aspects of its quality encompass data accuracy, completeness, consistency, timeliness, validity, and uniqueness.

The terms quality and integrity can get mixed, but for data-driven businesses, the parameters and metrics that define the quality and integrity of data have vastly different implications. And that’s why we put together this brief primer—so you can fully understand the differences between Data Quality and Data Integrity.

What is data quality?

Data Quality characterizes how reliable the information is to serve some intended purpose. This purpose might be:

• Planning
• Decision-making
• Operations

When the data is complete, full of all features and attributes, it is usable information to address specific real-world circumstances.

(Learn all about data quality management.)

What is data integrity?

Data Integrity characterizes how reliable the information is in terms of its physical and logical validity. Data Integrity is based on parameters such as:

• Accuracy
• Validity
• Consistency of the data across its lifecycle

Integrity is the absence of unintended change to the information between two successive updates or modification to data records. Let’s consider Data Integrity as the polar opposite to data corruption, which renders the information ineffective for fulfilling desired data requirements.

Data quality vs data integrity

Data that has integrity can be quality data, but not all quality data has integrity. Below, we describe some characteristics of quality and integrity:

1. Completeness

Completeness is an indication of the comprehensiveness of the available data. It can be measured as a percentage to state how much of the entire possible data set is available. The proportion required to classify the data as “complete” determines on the business and particular reason for it.

For instance, consider a list of health records of patients visiting the medical facility between specific dates and sorted by first and last names. The data resource will be considered 100% complete as long as it includes all necessary health records, and the first and last names, within specific dates. In fact, even if it doesn’t include the address or phone numbers of the patients, we can consider it complete because the ask did not include this information.

Conversely, the percentage of completeness reduces as any critical data item(s) are absent.

2. Uniqueness

Uniqueness is a measurement of duplication. Does the data, or similar data, exist multiple times within the same dataset? It is a discrete measure on particular items within a data set.

Example: Consider the same list of health records as mentioned earlier. There are 100 patients in a hospital. If the list contains more than 100 patients, then one or more patients must have had their data duplicated and listed as a separate entity. Whichever patient’s list record is duplicated is considered not unique. Depending upon the circumstances and business requirements for the data analysis, this duplication could lead to skewed results and inaccuracies.

Mathematically, uniqueness may be defined as 100% if the number of data items in the real-world context is unique and equal to the number of data items identified in the available data set.

3. Timeliness

Timeliness is the degree to which data is up-to-date and available within an acceptable time frame, timeline, and duration.

The value of data-driven decisions not only depends on the correctness of the information but also on quick and timely answers. The time of occurrence of the associated real-world events is considered as a reference and the measure is assessed on a continuous basis. The value and accuracy of data may decay over time.

For instance, data about the number of traffic incidents from several years ago may not be completely relevant to make decisions on road infrastructure requirements for the immediate future.

4. Validity

Data validity is a test of whether the data is in the proper format. Does the data input match the required input format? Examples include:

• Is a birth date written as Month, Day, Year or as Day, Month, Year?
• Are times based on local time zones, user device time, or the global UTC time?

The scope of syntax may include the allowable type, range, format, and other attributes of preference.

In particular, validity is measured as a percentage proportion of valid data items compared to the available data sets (i.e.; “The 90% of the data is valid.”) In the context of Data Integrity, the validity of data also includes the relationships between data items that can be traced and connected to other data sources for validation purposes. Failure to establish links of valid data items to the appropriate real-world context may deem the information as inadequate in terms of its integrity.

Data validity is one of the critical dimensions of Data Quality and is measured alongside the related parameters that define data completeness, accuracy, and consistency—all of which also impact Data Integrity.

5. Accuracy

Accuracy is the degree to which the data item correctly describes the object in context of appropriate real-world context and attributes.

The real-world context may be identified as a single version of established truth and used as a reference to identify the deviation of data items from this reference. Specifications of the real-world references may be based on business requirements and all data items that accurately reflect the characteristics of real-world objects within allowed specifications may be regarded as an accurate piece of information.

Data accuracy directly impacts the correctness of decisions and should be considered as a key component for data analysis practices.

6. Consistency

Consistency measures the similarities between data items representing the same objects based on specific information requirements. The data may be compared for consistency within the same database or against other data sets of similar specifications. The discrete measurement can be used as an assessment of data quality and may be measured as a percentage of data that reflects the same information as intended for the entire data set.

In contrast, inconsistent data may include the presence of attributes that are not expected for the intended information. For instance, a data set containing information on app users is considered inconsistent if the count of active users is greater than the number of registered users.

So, timeliness and uniqueness of data are useful to understand the overall quality of data instead of the integrity of information. Data completeness, accuracy, and consistency are good measurements of data integrity.

And, what each data item will actually be is unique to each organization. That responsibility is left up to you. Now let’s turn to look at data integrity in the real world.

Data integrity in practice

Data quality and integrity are important in the machine learning and analytics worlds. When data is the resource from which all decisions are based, then quality data allows for quality decisions. But what happens when your data is invalid, inaccurate, or inconsistent?

Let’s see!

Decisions on invalid data…

Maybe for legal reasons. Differing formats. Anomaly detection.

Invalid data changes the actual input data, and, if left as is, the decisions that are made are completely wrong. It’d be like determining to feed a person a hearty breakfast because they always eat dinner at 7pm, but their time data is invalid and set to a different time zone, and, really, they should be eating their dinner.

When data points are invalid, it makes decisions around detected anomalous data points weaker, and decisions about the actions to take weaker as well. Invalid data can sometimes be redeemed by interpreting it (such as converting errors to what you think they should be), but that comes at the cost of time, and labor, and the fact that the truth gets lost in translation.

Decisions on inaccurate data…

Accurate data is easy. Generally, when gathering data, people ask questions relevant to their domain—they understand what is useful to their business and what is not.

Where inaccuracy most commonly occurs is when data comes from an outside source and is retrofitted to suit one’s needs. In machine learning, this is called Domain Switching, and, with no surprise, a machine learning model trained to predict someone’s opinion of a movie from a dataset of IMDB movie reviews will perform poorly when set to predict someone’s opinion of their Friday night date.

When the purpose of the model has switched, the dataset has become inaccurate, and the model’s performance suffers.

Decisions on inconsistent data…

Like in the real world, if the data is inconsistent, then the outcomes are unpredictable. All models and decisions, if they are to be modeled, require patterns in behavior. Consistency is a prerequisite to pattern detection, and if the data is inconsistent no patterns can be detected.

In the early days of AI imaging, converting handwriting was a hard task because everyone’s penmanship was inconsistent. The Palm Pilot, one of the first handheld touchscreen devices, developed its own written alphabet to help its users and its device communicate with one another.

To combat data inconsistency, the solution can vary. You might:

• Increase the size of the data set.
• Redefine the data required so that what is collected can be consistent.

Related reading

• BMC Machine Learning & Big Data Blog
• Data Architecture Explained: Components, Standards & Changing Architectures
• Artificial Intelligence (AI) vs Machine Learning (ML): What’s The Difference?
• Data Streaming Explained: Pros, Cons & How It Works
• What Is a Canonical Data Model? CDMs Explained
• DataOps Explained: Understand how DataOps leverages analytics to drive actionable business insights

Among the most used machine learning (ML) frameworks, two are quite popular:

• PyTorch is Facebook’s ML package
• TensorFlow is from Google

Both allow you to build Machine Learning models, both have easy out-of-the-box models, and both are highly customizable. Whether you are new to the field of an expert, these libraries can satisfy all your needs—from testing to deployment. Let’s take a look at the differences between them.

Note: For those looking to choose a programming language, both libraries are written in Python. I machine learning is the direction you intend to go, learning Python is a common denominator. TensorFlow now has TensorFlow JS, so it can be used with JavaScript as well.

(This tutorial is part of our Guide to Machine Learning with TensorFlow & Keras. Use the right-hand menu to navigate.)

Getting started

TensorFlow is older and more widespread. Initially, it had a lot more community support and tutorials out on the web, but PyTorch has caught up. There is now extensive documentation for both, so learning one will not be any easier than the other. The key is to just get started.

With modelling, too, you rarely need to start from scratch. You can both find pre-built models, and models with pre-trained weights:

• TensorFlow has a model garden or Keras
• PyTorch has its PyTorch Hub
• Huggingface.co has a good repo of both

Huggingface, too, is a great ML community that uses both PyTorch and TensorFlow models interchangeably (they literally have a tool to convert models and weights between the two), and their new Hugging Face course can walk you through how to get started with Machine Learning.

Additional resources include:

• PyTorch tutorials
• TensorFlow tutorials
• Repository of TensorFlow Demos on Colab (Bonus: Use Google’s Colab Notebooks for a free GPU to train your models)

Code style

The code for each looks different. PyTorch is more pythonic and uses the Object-Oriented Programming styles. TensorFlow has made things easy for coders, but, in doing so, it has removed some of the normal kind of coding developers are used to.

Tensors

Both frameworks use:

• Tensors to pass data through to its models
• Graphs to define their models

TensorFlow has a statically defined graph that gets exposed to the user through the commands tf.session and tf.Placeholder. PyTorch has dynamically defined graphs that become useful for some more complex models. TensorFlow Fold enables dynamism.

Comparing TensorFlow vs PyTorch

Now, let’s turn to a comparison of both frameworks.

Performance tests

At the beginning of 2020, OpenAI standardized on PyTorch. They found it easier and quicker to test new ideas.

PyTorch reports slower training times, which, given some models these companies use, training can take thousands of hours of compute so the training time is directly associated with costs. PyTorch tends to take more memory during training.

Accuracy

Both model frameworks can get the same accuracy. (A Comparison of Two Popular Machine Learning Frameworks)

This comparison is a little dated (2017), but, from a report, its results appear to still hold.

Distributed training

Distributed training is getting closer to a must for large models. The big idea is to train a model on a Kubernetes cluster. On PyTorch, implementing a distributed training model is easy:

import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel
dist.init_process_group("nccl", world_size=2)
model = DistributedDataParallel(
        model)

On TensorFlow, you’ll have to do a lot of the coding yourself.

Deploying models

Models are great for research and backend purposes, but they need to work out in the world, too. This means the model and its weights need to get either:

• Loaded onto a mobile or edge device
• Turned into an API so it can be used for inference

At first, TensorFlow was the best framework for this, but as time has passed, both frameworks can be used to deploy models into production and there is plenty of documentation on how to do so.

Now, given Google has its own cloud framework, if you are using Google Cloud, TensorFlow can be integrated with Google’s services pretty easily, such as saving a TF Lite model onto its Firestore account and delivering the model to a mobile application. Though some of the TF stuff integrates well with Google’s services, there are plenty of workarounds to get PyTorch models working, too.

Data visualization

TensorFlow has TensorBoard which lets you see multiple aspects around the machine learning model from the model graph to the loss curve. It has lots of options to explore your model, as you can see:

(Source)

PyTorch has no native visualization tool. Instead, there exists a graph visualization package called Visdom.

If needed, there is a way to display your PyTorch models on Tensorboard.

That concludes this comparison of TensorFlow and PyTorch. Browse the Guide to learn more.

Related reading

• BMC Machine Learning & Big Data Blog
• Anomaly Detection with Machine Learning: An Introduction
• Bias & Variance in Machine Learning: Concepts & Tutorials
• Interpretability vs Explainability: The Black Box of Machine Learning
• React JavaScript Library: Concepts & Tutorials for Getting Started
• The Data Visualization Beginner’s Guide

The data engineer equips the business with the ability to move data from place to place, known as data pipelines. Data engineers provide data to the data science teams.

The data scientist consumes data provided by the data engineers and interprets it to say something meaningful to decision-makers in the company.

In this article, let’s dive a little deeper into the roles of data engineer and data scientist.

What is a Data Engineer?

In general, the data engineers are responsible for building pipelines, architecting the back-end databases, creating queries, and more.

Responsibilities

The data engineer will often possess a degree in computer science or engineering. Their skills involve building and working with computers directly. They build databases, queries to interact with the databases, move data from one database to another, transform the data to be sent as the right type to its end point. They are the ones who build APIs.

Data engineers will use a number of computer languages to get the job done. At their level, the best language depends on the task and the equipment they are working with. Java, Scala, C++, Go, and Python may be used.

Skills

• Writing database queries
• Building database pipelines
• Building APIs
• Coding language: Java, Scala, C++, Go, Julia, Python

What is a Data Scientist?

The data scientists might have to know only a little computer coding to ingest the data from the engineer’s sources, and to transform it to fit their needs.

Data scientists’ skillset is founded more on having good reasoning and communication skills. Their job tends to be highly mathematical and statistical. They need to be able to:

• Create hypotheses around the data sets
• Test the hypotheses
• Put what they learn into communicable information to decision-makers

Responsibilities

Data scientists are responsible for consuming data from a source, and finding valuable information from it. Then, they are tasked with presenting the information often through a visualization.

Data scientists will have to:

• Identify relevant data sources
• Filter through structured and unstructured data
• Analyze the data for trends
• Prepare a reports and visualizations

Skills

• Strong mathematical and statistical skills
• Source, filter, clean, and verify data
• Excellent ability to reason and communicate
• Build visualizations and data dashboards

Increasingly, data scientists are adding machine learning to their skillsets, too.

(Find out why Python is the predominant coding language for big data.)

Salary comparison

Demand for both positions is high and will be around long enough to build a career around. Their pay varies with skill-level and company. Data engineers used to make more money than data scientists, but in recent years, with the ever growing supply of people entering the field, and the growing ease to work with cloud data infrastructures, both jobs have evened out to roughly the same. Even their overall salary has dropped about $30,000 in the past 5 years.

While engineers’ average salary is reported to be a little less than a data scientist’s, their distribution curve is a little flatter, meaning there are more people in the position making higher salaries.

Source

Data engineers & data scientists working together

The role each plays in the company is essential. The data engineers tend to be better programmers and have a far better grasp on moving data around—after all, that’s their sole job. The scientists will specialize more in data analytics and all the statistical and reporting strategies to extract meaningful information from data.

Data scientists have the more popular role because, in a way, they are the journalists of data, and create the reports for people to read. Thus, they become the face of data while the engineers are behind the scenes and make access to all the data possible for the data scientist’s reports.

Data scientists’ reports can also influence the data engineering team’s data collection efforts. If the analysts determine a new source of information is needed, then they can tell the engineers to build a pipeline to gain access to the information, and the engineers can go develop the pipeline to give the scientists access to that information. It may be discovered that access to the information is impossible, has security issues, cost restrictions, too many inputs or unclear definitions, and they can tell the data science team that access to the information is impossible or needs to be approached differently.

Whichever path you choose, data scientists and data engineers will be around for a long time.

Related reading

• BMC Machine Learning & Big Data Blog
• BMC Business of IT Blog
• Database Administrator (DBA) Roles & Responsibilities in The Big Data Age
• Data Science Certifications: An Introduction
• Best Books on Big Data & Data Science
• Machine Learning for Data Management

Event stream processing is a reactive programming technique that filters, analyzes, and processes data from a data streaming source as the data comes through the pipe. It is used for a number of different scenarios in real-time applications.

As we rely more and more on data generated from our phones, tablets, thermostats, and even cars, the need for it to be analyzed while still streaming only increases. The aim is two-fold:

1. To process data in real-time.
2. To act on those data signals in as close to real-time as possible.

One way the Internet of Things (IoT) data can be evaluated while it is in motion is with event stream processing. The IT community is adopting this technique at greater rates, and it is because of all the advantages and applications event stream processing offers to the community.

What is event stream processing?

Sometimes shortened to ESP, event stream processing is composed of three simple terms: event + stream + processing.

Event

An event is anything that happens at a clearly defined time and can be specifically recorded. The possible types of events can be vast! Events can be created from an event source. These could be systems, business processes, sensors, data streams.

On an application, there could be a login event. On a thermometer, there could be a “too hot” event. On a fire alarm, there is a smoke-detected event. On databases, there can be data uploaded events.

Stream

A stream is a constant and continuous flow of objects that navigate into and around companies from thousands of connected devices, IoT, and any other sensors. An event stream is a sequence of events ordered by time. Enterprises typically have three different kinds of event streams:

• Business transactions like customer orders, bank deposits, and invoices
• Information reports like social media updates, market data, and weather reports
• IoT data like GPS-based location information, signals from SCADA systems, and temperature from sensors

Processing

Processing is the final act of analyzing the incoming data. How the data is processed is dependent on the function of the processor. It could trigger alerts that a sensor needs attention. It could define the next path the data is supposed to travel down along the pipe. It could classify the data and tag it with metadata.

Goal of event stream processing

Event stream processing, then, is a form of reacting to a stream of event data and processing the data in a near, real-time manner.

The ultimate goal of ESP deals with identifying meaningful patterns or relationships within all of these streams in order to detect things like event correlation, causality, or timing.

ESP is a useful real-time processing technique for events that either/both:

• Occur frequently and close together in time
• Require immediate attention

Strong examples of when ESP would be necessary are in the areas of e-commerce, fraud detection, cybersecurity, financial trading, and any other type of interaction where the response should be immediate.

How event stream processing works

Steps to event processing:

1. Event Source
2. Event Processing
3. Event Consumer

Event stream processing requires the existence of a streaming data source. There is nothing to process if there is no data source.

Once the source is created, the source must emit events to the processor. Likewise, the processor needs a way to listen and receive the output of the event source. This can occur through an API.

And, finally, like the sound of a man screaming help in a forest, what is the point of processing data if there is no audience to hear it? An event processor needs a consumer to output its processes to. It can be a data dashboard, another database, a user analytics report.

The landscape of available tooling to process data streams is large and growing, and the best solutions, as usual, depend on the use case, your tech stack, your budget, and your team’s skill levels. Possible cloud-based solutions are:

• Amazon MSK
• Amazon Kinesis
• Apache Kafka
• Azure Stream Analytics
• Google Pub/Sub

Benefits of event stream processing

Event stream processing processes events at the moment they occur, completely changing the order of the traditional analytics procedure. It allows for a faster reaction time and creates the opportunity for proactive measures to be taken before a situation is over.

Processing data in this way is advantageous because it:

• Creates opportunity for a real-time response time. For example, a potential new business service.
• Requires less memory of the service to process individual data points rather than whole datasets.
• Extends a company’s ability to process data to a greater number of potential data sources, such as IoT devices, edge devices, and distributed event sources

Event stream processing is a smart solution to many different challenges and it gives you the ability to:

• Analyze high-velocity big data while it is still in motion, allowing you to filter, categorize, aggregate, and cleanse before it is even stored
• Process massive amounts of streaming events
• Respond in real-time to changing market conditions
• Continuously monitor data and interactions
• Scale according to data volumes
• Remain agile and handle issues as they arise
• Detect interesting relationships and patterns

When to use event stream processing

Event stream processing can be found in virtually every industry where stream data is generated, whether it be from people, sensors, or machines. As IoT continues to expand its technologies, it will continue to see dramatic increases in the real-world applications of stream processing.

Systems are growing more varied and dynamic. Applications consist of an ever-growing number of micro-decisions, processing growing numbers of events sources and events.

AI has entered the picture, too, and it is becoming the consumer for event processes—instead of solely people—and AI is taking on the burden of responsibility to address real-time events, reaching far and beyond the limits of the human attention span, care to detail, and ability to react to limited number of problems.

(Learn about AI/human augmentation.)

Instances where event stream processing can solve business problems include:

• Ecommerce
• Fraud detection
• Network monitoring
• Financial trading markets
• Risk management
• Intelligence and surveillance
• Marketing
• Healthcare
• Pricing and analytics
• Logistics
• Retail optimization

Though far from exhaustive, this list begins to show the huge variety of uses for event stream processing as well as how wide-reaching it can go.

Event stream processing here to stay

Overall, event stream processing is here to stay and will only prove itself more crucial as the demand by consumers, desires the data to be computed and interpreted in real-time. ESP is a powerful tool which allows people to get closer to their customers, their companies, and their events, in order to take their analytics to the next level. As devices become more and more connected, having the ability to continuously stream and analyze big data will become vital.

Will your organization be ready?

Related reading

• BMC Machine Learning & Big Data Blog
• BMC AIOps Blog
• Real Time vs Batch Processing vs Stream Processing
• Creating & Using Snowflake Streams
• What Is Pub/Sub? Publish/Subscribe Messaging Explained
• Machine Learning for Data Management

Data Center Colocation (aka “colo”) is a rental service for enterprise customers to store their servers and other hardware necessary for daily operations. The service offers shared, secure spaces in cool, monitored environments ideal for servers, while ensuring bandwidth needs are met. The data center will offer tiers of services that guarantee a certain amount of uptime.

The decision to move, expand, or consolidate your data center is one that must be weighed in the context of cost, operational reliability and of course, security. With these considerations in mind, more companies are finding that colocation offers the solution they need without the hassle of managing their own data center.

Data center colocation works like renting from a landlord: Customers rent space in the center to store their hardware.

(This article is part of our Data Center Operations Guide. Use the right-hand menu to navigate.)

Benefits of data center colocation

Data center colocation could be the right choice for any business of any size, in any industry. Let’s look at the benefits.

Uptime

Server uptime is a big advantage enterprise businesses have in data center colocation. By buying into a specific tier, each enterprise server client is guaranteed a certain percentage of uptime without the payroll cost to maintain or other maintenance fees.

Risk management

Utilizing a colocation facility ensures business continuity in the event of natural disasters or an outage. This means that if your business location loses power, your network traffic will not be affected.

Its key to success is redundancy. The layers of redundancy offered at a data center colocation are far more complex than many companies can afford in-house.

Some enterprise companies will consider the off-site location as their primary data storage location while maintaining onsite copies of data as backup.

(Read about enterprise risk management.)

Security

Data centers are equipped with the latest in security technology including cameras and biometric readers, check-in desks that welcome inbound visitors, and checks for security badges are commonplace.

These facilities are monitored 24/7/365, both in the physical world and on the cloud to ensure that unauthorized access does not occur.

Cost

One of the main advantages of colocation is that it results in significant cost savings especially when measured against managing a data center in-house. This means that for many companies, renting the space they need from a data center offers a practical solution to ever-shrinking IT budgets. With colocation, there is no need to worry about planning for capital expenditures such as:

• UPS (uninterrupted power sources)
• Multiple backup generators
• Power grids
• HVAC units (and the ongoing cost of cooling)

Apart from these capital expenditures, there are also ongoing maintenance costs associated with maintaining and managing an in-house server.

Bandwidth

Colos issue the bandwidth that enterprise client servers need to function properly. With large pipes of bandwidth to power multiple companies, data center colocations are primed to support businesses in a way their office location likely cannot—something that’s increasingly important to remote work.

Support & certifications

Data center colocation offers the benefit of peace of mind.

When you partner with a data center colocation, your enterprise business may be able to reduce potential payroll costs by relying on the data center experts to manage and troubleshoot major pieces of equipment. Enterprise businesses can rely on expert support from experts who are certified to help.

Scalability

As your business grows, you can easily expand your IT infrastructure needs through colocation.

Different industries will have different requirements in terms of the functionalities they need from their data center as it relates to space, power, support and security. Regardless, your service provider will work with you to determine your needs and make adjustments quickly.

In-house data center vs data center colocation

While data center outsourcing offers many benefits, some enterprise organizations may still prefer to manage their own data centers for a few reasons.

Control over data

Whenever you put important equipment in someone else’s charge, you run the risk of damage to your equipment and even accidental data loss. Fortunately, data centers are set up with redundancy and other protocols to reduce the likelihood of this occurring, as discussed above.

But some enterprise businesses with the knowledge and resources to handle data in-house, feel more comfortable with being liable for their own servers.

They also benefit from being able to fix server issues immediately when they occur. Enterprise businesses who seek to outsource instead must work closely with their service providers to ensure issues are resolved in a timely manner.

Contractual constraints

Enterprise business owners may find that they are unpleasantly surprised by the limitations of the contract between their company and a colo facility. Clauses that include:

• Vendor lock-in
• Contract termination or nonrenewal
• Equipment ownership

Choosing a data center

Here are eight considerations enterprise IT Directors should think about before moving their data to a co-located data facility.

1. Is the agreement flexible to meet my needs?
2. Does the facility support my power needs, current and future?
3. Is the facility network carrier neutral? Or does it offer a variety of network carriers?
4. Is it the best location for my data? Accessible? Out of the way of disaster areas?
5. Is the security up to my standards?
6. Is the data center certified with the Uptime Institute?
7. Does my enterprise business have a plan for handling transitional costs?
8. Is this data center scalable for future growth?

If an enterprise business leader can answer ‘yes’ to the above questions, it may be the right time to make the change.

Cloud services vs colocation

The cloud is another option over data center colocation:

• A cloud services provider will manage all elements of the data: servers, storage, and network elements.
• An enterprise’s only responsibility will be to work with their services and use it.

Cloud services are great for allowing a business to focus more on their business requirements and less on the technical requirements for warehousing their data. In this case, cloud services can be cheaper, and enable new businesses to get off the ground quicker.

More established businesses are considered to be better suited to handle their own data center needs through colo or in house means, and the costs to establish and maintain their colo will be cheaper in the long run than cloud services options.

Cloud services also allow access to quick start-up times, less technical knowledge required to get going, easily scalable (both up and down) server needs, and then integrated services with all the other options a cloud service provider might offer such as:

• Integrated monitoring
• Data storage and querying tools
• Networking tools
• Machine learning tools

(Accurately estimate the cost of your cloud migration.)

What’s next for data center colocation?

The biggest push in the industry comes from cloud service providers who use colo as a way to meet their hefty equipment storage needs. At the same time, the industry has been and will continue to remain fluid as laws change with regard to cloud storage requirements.

While soaring demand from cloud service providers has made the need for data center colocation increase, new technology offers rack storage density options that allow colo facilities to mitigate the demand for hardware space.

Related reading

• BMC IT Operations Blog
• BMC AIOps Blog
• Data Center Migration: Creating a DC Inventory
• What Is a Software-Defined Data Center? SDDCs Explained
• DAS vs NAS vs SAN: Choosing the Right Storage Solution
• Major Network Outages

“The medium is the message.”

Data visualization is the graphical representation of data. It is a way to communicate the overall meaning of data points in a simple and meaningful way. Where a picture is worth a thousand words, a data visualization is worth a thousand data points.

In a world of big data, where organizations and entities such as businesses, the weather, traffic, and customer acquisition might contain hundreds or thousands of data points, your message is more impactful if you use a graph rather than simply displaying an Excel spreadsheet.

The data points can be presented in multiple forms to give a different message. It takes some discipline to understand the message you wish to convey in order to organize the data points into the proper visualization.

(This article is part of our Data Visualization Guide. Use the right-hand menu to navigate.)

Data Visualization is an everyday skill

Creating a data visualization is an important career skill as more businesses are making data-driven decisions. Everyone is a data scientist. Whether you are a data scientist filtering through a bunch of data to find out what makes it unique or you are on a sales team, attempting to understand your target customer, you will likely use some form of data visualization to explain what is happening.

Fortunately, making a visualization is very accessible. Because it is an everyday skill, the visualization tools and articles explaining how to use them are abundant—for all data sources, and most common kinds of data types. The most common types of visualizations are:

• Charts
• Tables
• Graphs
• Maps
• Dashboards

Each of these can be explored for more specific representations. The best things to do to improve your skills are exploring the infographics that come across your path, read through the data charts, and practice creating some of your own at your own job.

(Get inspired with this gallery of beautiful data visualizations. Check out Nadieh Bremer’s portfolio at Visual Cinnamon.)

The importance of visualization

Data visualization is so essential in communicating what is happening with one’s data. In fact, it is the consumer’s expectation that a service should offer some form of visualization tool in order for the consumer to:

• Interact with their platform
• Give insights into what is happening with their activity

Whatever platform you might be using will generally have a data visualization element to it. Amazon, Azure, and Google all have their built-in data visualization tools to monitor all kinds of usage metrics within their platforms.

Companies like Salesforce, web hosting services, and social media companies, too, have created visualization tools for their content creators over the past few years so those people who create content on their service have easy access to understand what is happening between themselves and their audiences so they can better craft their message.

Top data visualization tools

Whether working individually or as part of a business, here are the top visualization tools available today:

• Tableau
• Grafana
• Google Charts
• d3.js

Data visualization best practices

When looking at data and determining how to visualize it, here are some key questions to ask yourself.

Good questions to ask

• How can I represent the information before me in a graphical representation?
• Can I say it all in one representation or do I need a few?
• Is the information time-sensitive?
• Does it represent growth of something?
• Does it represent frequency of occurrences?

Does it need to represent

• Dependence on time
• Growth
• Frequency of occurrence
• Dependence on location

Does your audience prefer beautiful or functional charts?

Craigslist’s design has remained relatively the same for a couple decades, and its success is in large part due to its functional use. It is a boring design—sure—but one that works to get information to people by its own design staying out of the way.

Charting is similar.

Some of the visualizations the big platforms use are very basic because they just have to communicate something simple, but visualization dashboards that have users come back to it every single day often require a little more design considerations because it’s users want to work with something that is pleasant on the eyes.

What message does the audience want?

The best visualizations are often the simplest ones that serve one purpose:

• User activity charts display user activity over time, either through a heatmap or a line chart.
• Sales volumes show cumulative or daily line charts.
• User activity based on region illustrates a geographical map with color-based activity overlaid.

But some users require visualizations that display information at a high level. They need to be able to dig deeper into it. Dashboards are particularly handy for:

• Displaying lots of data
• Allowing the user to dig deep when they see something on the chart that strikes them as interesting

Get involved with a community

“Imitation is the sincerest form of flattery.”

Messages are bits of communication from one person to another. The information takes a form and garners a particular style. If you wish to be an effective communicator when crafting your visualization, become familiar with how other people are already crafting their message.
Join communities of people who are sharing visualizations with one another. Look outside that community and observe how other people are talking with one another. Like any type of communication, getting better at representing your data requires:

• Practice
• Mimicry
• Exposure to what options are out there

(Gain exposure with Data Is Beautiful & these blogs.)

Seek feedback

Finally, visualizations rely on communication, so it is a good idea to get feedback from your audience. Pay attention to whether your audience is asking a lot of questions when you share your visualizations. Things as simple as decreasing line widths and using a bold font can improve your visualization’s effectiveness. If it’s not working, change it.

Good luck getting going!

Related reading

• BMC Machine Learning & Big Data Blog
• Data Engineer vs Data Scientist: What’s the Difference?
• How To Use Jupyter Notebooks with Apache Spark
• 3 Simple Data Strategies for Companies
• Data Ethics for Companies
• Data Architecture Explained: Components, Standards & Changing Architectures

As technology has become imperative to businesses large and small, two executive-level roles have become standard:

• Chief information officer (CIO)
• Chief technology officer (CTO)

But the distinction between the two can be confusing, as “information” and “technology” typically go hand-in-hand. So, what is the difference between these two roles? How does one focus on technology while the other focuses on information?

A simple distinction is that the CIO typically looks inward, aiming to improve processes within the company, while the CTO looks outward, using technology to improve or innovate products that serve the customers.

Let’s take a look at the difference between CIO and CTO roles as well as whether your company should employ one or both.

What is a CIO?

Short for chief information officer, the overall role of the CIO is ensuring business processes run efficiently, with a goal of promoting the productivity of individual employees and business units as a whole.

The CIO is responsible for managing and ensuring ongoing operations, mission critical systems, and overall security, from help desks and enterprise systems to service delivery and program management. The explicit impact of a CIO can be determined with a variety of metrics, though improving the company’s bottom line is a must.

The CIO can be seen as the ultimate cheerleader for all in-house technology and digital processes. IT traditionally has a nebulous reputation with other business units – so it is the role of the CIO to improve the image and reputation of IT services within the company.

But the CIO isn’t only tech-focused: a good CIO integrates the entire IT department with other business units, so knowledge of the common as a whole is imperative. For example, if a business unit seeks technology to digitize, improve, or even automate processes, the CIO is responsible for managing these processes, even if a specific IT team performs the actual implementation.

Responsibilities of a CIO

Some responsibilities of a CIO include, but are certainly not limited to:

• Managing all technology infrastructure
• Overseeing IT operations and departments
• Aligning and deploying technology to streamline business processes
• Increasing the company’s bottom line
• Focusing on the requirements of internal employees and internal business units
• Collaborating with ISPs and vendors to drive productivity

For a CIO to be successful in the role, general knowledge of a wide variety of technology is essential, though a CIO can’t be expected to have expert knowledge of every system. Management and communication skills are also essential: a CIO can oversee dozens of IT employees and a variety of IT teams, and the CIO must also be able to communicate needs and strategies with other executives and department managers.

What is a CTO?

The chief technology officer focuses on creating and using technology in order to help the business grow – typically improving offerings that the company’s customers purchase with the help of new technologies.

The CTO focuses on external customers: those who are buying your company’s products, even if the product itself isn’t digital or technology-based. As customers become savvier and more knowledgeable about the products they use, the CTO must stay innovative and on the cutting edge of technology to ensure the company is offering the best products.

To that end, the CTO is often responsible for the engineer and developer teams who focus on research and development to improve and innovate company offerings.

Responsibilities of a CTO

A CTO may be responsible for the following:

• Owning the company’s tech offerings and external products
• Using and reviewing technology to enhance the company’s external products
• Managing the engineering and developer teams
• Understand and touch all technologies the company deploys
• Increasing the company’s top line
• Aligning product architecture with business priorities
• Collaborating with vendors on supply solutions

Though a CTO is tech-focused, staying abreast of tech developments and relying on a background in computing or software engineering, a successful CTO must also embrace right-brain skills like creativity. Innovation might start with a simple question of “How can I use this technology differently from everyone else?”

Collaboration is also an essential skill, as a CTO will need to partner with in-house engineers and external vendors to achieve something that hasn’t been done before.

CIO & CTO salaries

CIOs and CTOs are skilled and have a lot of responsibilities, so they earn a larger income. The average salary for both are roughly equal, according to Payscale. As you will learn as you go throughout your career, the real differences in income won’t be found in salaries so much as they will be found in the benefits, equity, commissions, and overall working experiences you have.

As of October 2021, CIO and CTO Salaries in the United States were:

CIOs & CTOs work together

CIOs and CTOs have different strategies of success for their jobs.

• CIOs want to increase bottom-line numbers, and CTOs increase top line numbers.
• CIOs mediate between internal IT teams and other departments, and CTOs develop relationships outside the company.

But, at the end of the day, the two’s strategies are housed under the same corporate roof, and they will have to work together to see their strategies through to a successful finish.

The two don’t always get along, however. Their strategies come at a conflict of interests, but the tension between the two is what helps increase innovation in the organization.

CTOs want to keep developing technologies, bringing to the table new ones to use, are constantly experimenting with new tech stacks, possibly spending money on new projects that don’t always pan out. These are the appropriate behaviors of innovation. Unfortunately, when judged by the bottom-line monitoring CIO, these activities can appear reckless, costly, and as wastes of time.

The CIO will make rules to try and reign in the CTO, and make their behaviors more efficient and align with the business goals.

To the CTO, the CIO looks to be an inhibitor of innovation. They always look at what is instead of what could be, and their risk-averse tendencies are draining on the innovation process, and constantly keep the development of the organization lagging behind what its true potential could be.

The CTO/CIO feud is one of those battle-of-the-sexes conflicts that keeps an organization alive. To make it more durable, each party should focus on what their responsibilities are, and acknowledge the success of the whole organization when they occur. Creation does come with tension, and the day-to-day can be a grind, with lots of tedious bickering and negotiations.

But, by looking up once in a while and setting sights on the overall mission to see where you’ve come from, and where you are now, and where you are headed, suddenly it is possible to see value in the dance along the way.

CIO and CTO: Do you need both?

Though CIOs and CTOs may be confused by less tech-minded people, both roles are vital to your company’s success. It can be tempting to think of one role as superior or more of a priority – especially for smaller companies lacking the funding for both. As chief-level positions, one is generally not more senior or junior than the other. In fact, successful companies are often marked by strong presence from both the CIO and the CTO. Do what you can to make both positions a reality as soon as possible.

If you’re looking to hire or create the position of a CTO or CIO, consider these guiding questions to determine which role you’re actually in need of:

• Are you aiming to improve or digitize a business process or a product?
• Are you catering to your organization or to external customers?

Importantly, from the individual’s perspective, these are independent career paths: you don’t train for years towards a CIO role and switch to a CTO position on a whim. CIOs may come more from the IT operations side of the business, where CTOs may have more software engineering experience.

Companies need one person to support and promote productive employees and business processes just as much they require an innovator and creative solver who can leverage technology to improve business offerings.

How to become a CTO/CIO

Becoming a CTO or CIO requires the education and technical skills to know what kind of technology is needed to cater to the business’ products and technical infrastructure. So, if you really enjoy technology and engineering entire ecosystems of computer operations, then you will need to be ready to learn.

Take chances to extend the knowledge you have into wider domains of expertise:

• Network architectures
• Information security management
• Big data engineering
• Software development

When you don’t know something, take the time to figure it out. If your learning growth has stalled, or staled, reach out to learn something new

Knowledge is not the only thing. Plenty of people know things. You need to prove the capacity to execute plans, lead teams, and communicate well with others. Not only do you need to prove this to others—you might have already proven it to yourself—but you need to prove it over and over again. Getting to be a C-level executive takes time. Your reputation as a leader of technological know-how, is something for you to take care of, tend to, invest in, and watch grow over time.

“You’re only a success for the moment that you complete a successful act.”—Phil Jackson, Former NBA Coach

Doors will open here and there for you as your reputation is acknowledged—those are opportunities for you to reap what you sow. You must show up in those moments. So, at every stage of your career, when a door opens to accept greater responsibilities, take it.

Related reading

• BMC Business of IT Blog
• 4 Essential Leadership Qualities
• The State of the CIO
• The Chief Information Security Officer (CISO) Role Explained
• What Is a Chief Service Officer?
• IT Director: Requirements, Skills & Salary

Recently, I was using a company’s chatbot to get an answer about their software. It was a complex scenario where I thought direct contact with the company’s support team would be easier than googling a solution.

Their chatbot connected within seconds. I asked my question in natural language: that uncertain description of trying to say what is wrong, even though I have no access to the precise language of what was occurring. Fortunately, their chatbot picked up on exactly what I was talking about and gave me instructions to troubleshoot the error.

It was so easy I actually left the browser window open so I could use it on far simpler questions—instead of using Google.

Accessibility, simplicity, and effective communication are at the heart of a good chatbot experience. Enterprise chatbots help to deliver the intuitive, seamless, omni-channel experiences today’s employees expect while improving IT speed, efficiency, and productivity. Chatbots reduce cost, improve satisfaction, and free resources for higher priorities like multi-cloud management—helping the organization adapt to the dynamic demands of digital business.

As with adopting and implementing any new technology, it’s important to go about implementing your chatbot the right way to get the best results. Here are eight operational best practices—broken into three phases—that we’ve developed to ensure the success of your implementation.

Phase 1: Chatbot planning

1. Define business outcomes & success

Before it begins, make sure you’ve identified measurable, quantifiable business outcomes to track from beginning to end.

You could create a great, friendly chatbot that engages users for hours, but if its function was to get users access to information and fade into the background so the user could get back to their life, then the chatbot has failed.

Every successful IT project begins with a clear business case. What are the metrics you’re seeking to improve with your chatbot? How will you measure the success of your implementation? Possible success outcomes are:

• Reduce costs
• Process improvements
• Process efficiencies
• Transform user experience
• Improve user productivity
• Accelerate resolution
• Boost productivity or accuracy of IT staff

2. Identify use cases

To set the scope for your implementation, first decide what you want the chatbot to help you with, and how it will do so. It’s best to start small, with 5-10 key services that are:

• Frequently requested so you can achieve both high visibility and high impact
• Easily automated so you can gain skills and experience for more complex use cases to follow
• Conversational as suits the natural language nature of chatbot interaction
• Pervasive so you can drive transformation broadly across your organization and user base

Make sure the services you select have up-to-date content and questions in your service catalog, including accurate knowledge for less frequent requests.

(Understand natural language in our NLP vs NLU primer.)

3. Make it accessible

Put your chatbot where your users need it. Consider who your users are and the channels they’re already using. The goal is to make it accessible.

Once you’ve chosen where this is, tell them where it is. Let them know about the new service and what it is used for. Then, demonstrate some examples of the chatbot in use!

4. Design to communicate

The chatbot needs to communicate. There is a lot of developed theory around what makes chatbots easy to use and good to interact with that you can pull from and don’t need to feel like you’re reinventing the wheel.

Search terms for that information include:

• Chatbot conversation designers
• Chat UX

Anyone working with chatbots says their success depends on how well they were planned and designed, so these Phase 1 steps are important.

Let’s look at a few takeaways from chatbot communication design:

Design the dialogue flow. A conversation can go in many direction. That means the chatbot conversation needs to be carefully planned in much the same way that web designers plan how people will travel through their websites.

• What is the basic information to present, that can lead to deeper information?
• What is the conversation journey you wish to take the user on?
• How do they enter the conversation? How do they leave it?

Create a persona. People act with personalities and you should create one for your conversation experience. The persona can steer the language that presents the message. Based on the persona, the welcome message could vary. Consider the differences:

• “Howdy y’all!”
• “What may I do for you today?”
• “VA Roger reporting for duty.”

Tighten message length. Say what needs to be said and be done. Bullet points and numbered lists convey more information in less words.

A huge part of unsuccessful chatbots? When the user tries to go off-script. The tighter you keep the message, the narrower the channel of communication is, and the better you lead your user through a dialogue.

Provide user response options. You should prompt a user with accepted prompts. You train them to know what inquiries are appropriate for your system.

This is sometimes overlooked, and/or undervalued, but this same communication works well in other domains as well. Comedians are more successful when they queue the audience to laugh, and street performers are more successful when they give the audience optional answers after they ask a question.

This works because it puts constraints around the user response by setting an expectation of what kinds of responses are reasonable.

Build in feedback loops. Remember, communication is a two-way street—use employee feedback to assess and improve the effectiveness of your messaging.

(Read more about feedback loops.)

See how BMC engineers build chatbots:

Phase 2: Implement the chat agent

5. Train your chatbots

There are tools out there to design chatbots, like Voiceflow, and there are services to actually create a chat agent like Google’s DialogFlow or BMC Helix.

Training a chatbot often requires labeling data, which is handled through the software. You can start by using the bot yourself by working through some test scripts you’ve written out, and progress to sending the bot to other beta testers.

Most chatbot software has labeling features in it that allow you to tag pieces of text as particular entities or intents. After you and your beta testers have used the chat, you can go through the logs of the conversations and label the data.

Chatbot data has a couple essential labels:

• Entities are things like cities, names, times. This is very similar to part-of-speech tagging.
• Intents are used to identify the motive of the user. They classify whether the user is requesting the time, to order a pizza, to report a bug, or to inquire about the current usable version of the software.

For example, “Tell me the time,” would be classified as the “requesting the time” intent, and “A large with cheese” should get classified as a pizza order.

There are machine learning models, then, that train on this dataset you have entered that then work during the chatbot’s use to predict the user’s text inputs as one of these intent “buckets”. Once the chat agent knows what the user’s intent is, then it can follow the dialog flow you have designed.

6. Deploy your bot

Next step: deploy your bot.

Often, there are API keys required to put your bot on chat UIs, like your company’s Facebook or Instagram, for instance. Wherever you are building your bot, they should offer clear instructions on how to integrate your newly formed bot with the interface you require.

Phase 3: Operations

7. Monitor & improve

Finally, chatbots are not a set-it-and-forget-it technology. To make sure you’re meeting expectations, continue to monitor usage and measure key benchmarks against the targets you’d initially set. If you see sub-par metrics in some areas, revise the learning procedure or the conversation flowchart to improve chatbot performance.

As both users and IT gain experience with your chatbot, you can begin to add new use cases. Users will likely ask about or suggest areas where chatbot functionality would be especially valuable—let their needs and expectations guide you. And as always, keep collecting feedback to make sure you’re providing the right experiences in the right ways to meet your goals.

Chatbots fuel transformation

As a high-visibility, high-value project, chatbot deployment can help set the tone and build momentum for your entire digital transformation agenda. In this light, getting it right will deliver powerful benefits for both users and IT. I hope these best practices help!

Related reading

• BMC Service Management Blog
• Chatbot vs Virtual Agent: What’s The Difference?
• ChatOps Explained: How ChatOps Supports Collaboration
• Machine Learning Use Cases & Business Benefits
• Introduction to SMART Goals: Examples & Tips
• Choosing IT Metrics That Matter

Many predictions to the outcome of the humans and artificial intelligence take an either/or approach.

Skynet is determined to end the human race in The Terminator. In The Matrix, the machines have learned to farm humans for battery power. Then, there are the reports that AI beats the best Go and Chess players in the world, and there is no shot for a human victory.

These views make it seem that in order to win, humans must exist free of computer control. A more likely outcome—a better picture of success—is to say, “We found our peace through AI/human augmentation.”

AI/human augmentation is a view that sees the story of humans and machines as one of cooperation. It puts the human in the driver seat and focuses on how AI becomes assistive to enhancing human capabilities—like with hearing, seeing, and making decisions. Communication is an important factor to AI human augmentation. Communication can happen through sensors, human-in-the-loop learning, user surveys. Finally, our ability to create, detect, and communicate with these AIs, depends on how we believe they work.

AI/human augmentation is the most plausible look at the future of human computer interactions and deserves more attention.

What is human augmentation?

“Computers are like a bicycle for our minds.”–Steve Jobs

Computers were a leap in tool-building that made people far more capable than before. Computers allowed people to:

1. Solve a problem once.
2. Write lines of code to perform it.
3. Let a computer perform the task over and over.

The development of artificial intelligence (AI) has inserted itself as a prologue to these steps in problem solving. It is in this step of human decision-making that AI is having its profound impact.

AI has allowed a developer to solve how to make a decision around the same problem. Before, when a similar problem presented itself, a developer would have to sit down and write another program to tackle this new variation in the problem. But with AI, the developer is tasked with defining the scope of the problem, and then allowing the AI to make the decision for what the outcome should be.

AI/human augmentation example

In a brief example, an engineer must meet the requirements to construct the design of a building. An engineer needs to know whether 2x6s or 2x12s can cover the span from wall to wall. If there is a second floor, can the joists support the weight of the people walking on the floor? If there is a roof load passed to the floor system, are the beams and joists large enough to support the weight passed to them?

Given the design, the engineer has options on how to best engineer the ceiling and roof system in order to:

• Cost the least amount of money in timber costs
• Continue to support the structure

If the engineer were a computer developer, however, perhaps they could create a program with a guided set of rules to engineer the house for them. The limitation in previous computer programs has been that, for each new house the engineer would build, he would have to create a new program to do the engineering of the house. Every new design will be governed by the same rules (a 2×6 can only support so much weight—its variable is constant across houses).

Here is where the AI comes into play: AI lets the engineer write a piece of software that can do the engineering for all houses instead of having to write a new one each time a new design comes across their table.

AI helps humans

When the AI is created to make decisions around common problems, and used to assist in making decisions, the relationship between person and AI actually transforms who people are.

The very things people can accomplish changes. The day-to-day activities people perform changes. People are no longer set to fixate in one domain, solving the same problem over and over. Instead, people can:

1. Travel across many domains.
2. Understand a domain’s problem.
3. Create an AI to tackle that problem.

Benefits of AI (Does AI destroy jobs—or do we?)

In the human-AI dance, aside from human extinction, the primary concern people have is that AI is going to eliminate many jobs.

I must say, jobs have always had a lifecycle. Airline pilots used to be a top-of-society, high-end job. Banking and finance took a top spot in the ‘80s. These things change based on:

• Skill levels
• Competition
• Available technologies

Next, having old jobs persist into the future is not an appropriate standard by which to measure society.

If, suddenly, there were no more oil jobs because everyone worked at nuclear plants—because that is the energy in demand—that is not a bad thing. It is an adjustment to the times. If there were still floors of telephone switchboard operators in existence in today’s labor market, it’d be easy to wonder, “Why is that company still doing things that way?”

Instead of being concerned about AI eliminating jobs, the real questions we ask should be:

• Are there enough new jobs for people to get into?
• Are training and education easily accessible and affordable?
• Is there plenty of investment money in circulation to take risks on new industries and new ideas that lead to creating places for new workers to work?

The effect AI/human augmentation has on the market is, of course, eliminating old ones—but that is not the only thing, nor is that a bad thing.

AI helps focus people, work

AI’s entering the marketplace will also remove the remedial day-to-day tasks and let people work on the things that matter.

It can remove those parts of the workday nobody looks forward to doing. Whether it is because it is a communication hell or it is a lot of minute, detailed work, alone in a dark room which has to get done so food can be put on the table…all those things, AI is ripe to assist with!

People can use AI to eliminate the dull times, and place greater human attention on those things that matter. People can perfect their roles in certain areas. It is the elimination of simple tasks. It is more quality time spent doing the things that matter. Some will be gone, but new ones will also be created.

In photo or video editing, for instance, instead of spending hours editing a photo to include a person who was not present in the original photo, the editing can just be done by using the Photoshop technology with perfect blurring, lighting, and blending techniques. Honestly, you might not even know there is AI being used in the background.

The features and the things that you are capable of will increase—and they’ll increase in complexity. (But only for those who knew how it used to be done. Young eyes get to start with this tech and see how it transforms over time.)

The photo’s creator just has to give direction to what exactly they want. The photographer can spend more time trying to communicate their message, and less time doing the actual engineering.

AI enables people as creative curators

If an AI can fill in a lot of the processing steps, people have more time to iterate over different designs before selecting their final one.

In architecture, the design process might work similar to photos. The designers can say, we want walls here, here, and here. Then the AI, which can do all the structural engineering to ensure the building stands and passes city codes, can figure out how to construct the layout. If it is not liked, the designers can undo it, and try another variation.

In a perfect setup, people’s roles can shift from doing the grunt work to actually designing. Industries where people’s roles turn into creative curators:

• Architecture
• Painting
• Photography
• Music
• Writing
• Coding

Perfect elimination of the grunt work is a look at the ideal, but there will always be some form of new thing. As people relieve themselves from one challenging endeavor, they start using their newly freed time to push the boundaries of the new technology, ultimately creating a new problem.

Animation has been happening through various technological forms for 100+ years, and, even with computers and AI used to help, there are now more people working on a team to create an animated film than there were 100 years ago.

People as relatable entities

People are still the best at being people. This is both good and valuable. It sets the parameters around the human entity.

While machines might assist with making decisions, there still exist lots of emotional tensions, struggles, and hardships that people help each other get through. There will always be this necessary component.

Human augmentation example: AI in education

I think one opportunity where AI/human augmentation can present a clear and present shift is with teachers. I am the son of a high school teacher, and I only speak about this from the heart.

Like how an AI can make better moves across a chess board to defeat an opponent, an AI might become the best at selecting the “when and how” to dispense knowledge to educate a student. The AI could do the laborious tasks:

• Creating a curriculum
• Designing a syllabus
• Grading tests

The value of using an AI is to define the problem space and allow the AI to create many unique solutions. That means an AI can create a unique educational path for each student—which is far beyond the capabilities of any educational institution today.

The best organizational structure schools have found, thus far, is to create two or three different paths with a base-level path, then one or two accelerated path options (i.e.; AP and IB classes). More variation comes in the extracurriculars students get themselves into.

For education, this use of AI actually sounds great: Students can take a path suited to them, driven by their curiosity, and move down it at their own pace.

With AI in the mix, the teacher’s role is necessary because the presentation of new information is always met with resistance by a learner, and students handle the moment in various ways. The teacher becomes the students’ guide to instill discipline when approaching new topics. They:

• Encourage to persist through the struggle
• Act as a soundboard for feedback

And, with their lesson planning offloaded, teachers have the time to tend to this core responsibility. Teachers act as guides to knowledge acquisition, not dispensers of knowledge.

AI supports focused problem solving

The clear definition of success makes the use of AI in education possible. Papers are graded, and, by the end, the goal is to have a higher-level understanding of the maths, sciences, and the humanities.

Outside the educational world, however, once a person enters life—“the real world”—the path to success is not so clearly defined. It is not agreed upon, and it is not measured by passing grades, nor has another level of achievement.

Life is harder, maybe even impossible, to define what a “win” might be, yet we continue attempting to define the game. For example, there are paths through college, careers, family-planning, homeownership, and probably… I have lost a few people already in what qualifies as a successful path through life.

What is a successful life? This is far too broad of a problem for AI to solve. And it’s likely the wrong way of thinking about this.

Because AIs have limitations, the kinds of applications that come from AI/human augmentation are therefore limited. An AI cannot assist with broad problems like designing choices through life, but it can assist with narrow scopes like:

• Improving your grammar
• Making your choices on a chess board better
• Knowing when a person has a grasp on algebra, and what math comes next

Challenges of AI (AIs require new work)

AI presents its own challenges to operate. AI is not a perfectly running machine. It has all its own maintenance requirements.

In the new, booming AI industry, new jobs arrive that have both low-skill and high-skill labor options. The upkeep requirements to build and maintain a good AI in turn creates other jobs.

(Learn more about data annotation, technical debt & performance monitoring.)

AI/human augmentation is the step towards AGI

Artificial General Intelligence (AGI) is the holy grail of the AI world right now. AGI is built around the idea that a single AI can have the general intelligence of a human. But, because of how varied the human intellect is, and how fragmented all data collection and uses are, at present, it is infeasible to build an AGI.

The success of an AGI is dependent on:

• The number of points of contact between AI and human
• The kinds of data which passes between the two

It only makes sense, then, that AI/human augmentation is a necessary step towards building an AGI because it creates more points of contact though the apps that get created, and the data flow will continue to increase as the complexity of the interfaces increase. Whether an AGI is the goal or not, AI applications will continue to be developed to enhance human ability—which only helps the cause of AGI and makes its outcome seem certain.

• AGI requires more data to map human behaviors and decision-making.
• AI/human interfaces provide exactly this.

So, AGI is going to be built at some point. Whether there exists any fear over this depends on how you choose to grant an entity the right to communicate and engage with negotiation, and, in a similar vein, but from another angle, which devil you choose to assume exists.

The science of communication

The tech isn’t itself the problem. The blame comes down to the person designing it. An AGI isn’t necessarily bad, but, through its design, maybe it comes off as tyrannical.

Norbert Wiener is considered the originator of cybernetics, defined as the science of communication as it relates to living things and machines. The key here being communication. The communication between machines and living things has many potential states: It can exist, not exist, grow dull, be adversarial, and so much more.

(Source)

His book Human Use of Human Beings (1950) has stood the test of time, and rings with clarity today. He articulates a particular aspect of human-computer interactions which define the human, how communication occurs, and how machines learn. Furthermore, it defines the kinds of cultural visions, the goods and bads, of what should be believed in if we are to be successful in our human pursuits to live.

Wiener places limits on chaos and what the nature of that chaos should be if a scientist is to make any progress in their pursuit to understand it. This is precisely the consideration that needs to take place if people are to create a superintelligent AI. If AGI is to ever reach beyond the point of interpretability, then designers need to create aims towards making it the AI’s goal to make itself communicable and predictable so scientific judgements can be made.

In such a case, Wiener defines two kinds of opponents— ‘devils’ as he calls them—a person can face when trying to communicate. In Wiener’s view, there are the Manichaen and Augustinian devils.

• The Manichean Devil is the tricky and crafty opponent. He keeps his policy of confusion secret. If we show we are ever close to figuring out its secret, it will change its policy to keep us in the dark.
• The Augustinian Devil can be understood and does not actively try to confuse.

In communication between entities, if the result is to reach understanding and an agreement, it is the Augustinian Devil people wish to be in a game with. The intent of the AGI will be to reach some understanding. But, if the opponent is Manichaen, the two parties can never reach any kind of understanding because the Manichaen devil will actively change if it ever sees us as getting close to understanding. Wiener says, the devil the scientist faces “is the devil of confusion, not of wilful malice”.

So, the question we must ask is:

Will the AI that is created work towards an understanding (be Augustinian) or work towards mystifying and confusion (be Manichean)?

The answer people give to this question defines whether the AI and human relationship will work in harmony or with contention.

(See how people harness AI for cyberattacks.)

When is AI too far?

In an AI-assisted world, through the view of augmented AI, it is not AI or AGI which should cause fear. Likely the “willful malice” that could get baked into an AI would be built by a person, and not an outcome determined by the AI itself.

Caution, then, is not of the technology itself, but of people who do not know when to stop. The caution is about those who are trying to build the path to God. Those who wish to extend far beyond their own limitations and make the world a better place. They think they can tread on old grounds, people’s homes, and show a better way.

As much as computer technology is advancing, so too can progress be made in cultural and societal technologies so people grow an understanding for:

• Who and what they are
• How to partake in societies
• What their limits are
• Where their freedoms lie

No matter how great or successful a person, or a people, becomes, they must always operate in time. Time is the great equalizer. And, in the great song of life, as much as you think you can pull people along, and accelerate their lives and their situations, you can only rush the beat.

AGI is not the threat, but people who think their technological answer is everyone’s answer run the risk of destroying the harmonies of the song. Ensembles can gather to a new beat once multiple locations recognize where the beat should be, and more and more people dance to its pulse.

Related reading

• BMC Machine Learning & Big Data Blog
• 4 Types of Artificial Intelligence
• Data Architecture Explained: Components, Standards & Changing Architectures
• Supervised, Unsupervised & Other Machine Learning Methods
• DataOps Explained: Understand how DataOps leverages analytics to drive actionable business insights
• Data Ethics for Companies