What do you think of when you think of "big data"?
For many, it's a nebulous term that invokes images of huge server farms humming away. Or perhaps you think of receiving some kind of personalized advertisement from a retailer.
But big data is so much deeper and broader than that. I believe there are 10 major areas in which big data is currently being used to excellent advantage in practice - but within those arenas, data can be put to almost any purpose.
1. Understanding and Targeting Customers
This is one of the biggest and most publicized areas of big data use today. Here, big data is used to better understand customers and their behaviors and preferences. Companies are keen to expand their traditional data sets with social media data, browser logs as well as text analytics and sensor data to get a more complete picture of their customers. The big objective, in many cases, is to create predictive models.
You might remember the example of U.S. retailer Target, who is now able to very accurately predict when one of their customers will expect a baby. Using big data, Telecom companies can now better predict customer churn; Wal-Mart can predict what products will sell; and car insurance companies understand how well their customers actually drive.
Ski resorts are even using data to understand and target their patrons. RFID tags inserted into lift tickets can cut back on fraud and wait times at the lifts, as well as help ski resorts understand traffic patterns, which lifts and runs are most popular at which times of day, and even help track the movements of an individual skier if he were to become lost.
Imagine being an avid skier and receiving customized invitations from your favorite resort when there's fresh powder on your favorite run, or text alerts letting you know when the lift lines are shortest. They've also taken the data to the people, providing websites and apps that will display your day's stats, from how many runs you slalomed to how many vertical feet you traversed, which you can then share on social media or use to compete with family and friends.
Even government election campaigns can be optimized using big data analytics. Some believe Obama's win after the 2012 presidential election campaign was due to his team's superior ability to use big data analytics.
2. Understanding and Optimizing Business Processes
Big data is also increasingly used to optimize business processes. Retailers are able to optimize their stock based on predictions generated from social media data, web search trends and weather forecasts.
One particular business process that is seeing a lot of big data analytics is supply chain or delivery route optimization. Here, geographic positioning and radio frequency identification sensors are used to track goods or delivery vehicles and optimize routes by integrating live traffic data, etc. HR business processes are also being improved using big data analytics.
This includes the optimization of talent acquisition - Moneyball style - as well as the measurement of company culture and staff engagement using big data tools. For example, one company, Sociometric Solutions, puts sensors into employee name badges that can detect social dynamics in the workplace. The sensors report on how employees move around the workplace, with whom they speak, and even the tone of voice they use when communicating.
One of the company's clients, Bank of America, noticed that its top performing employees at call centers were those who took breaks together. They instituted group break policies and performance improved 23 percent.
You may have seen the RFID tags you can attach to things like your phone, your keys, or your glasses, which can then help you locate those things when they inevitably get lost. But suppose you could take that technology to the next level and createsmart labels that could stick on practically anything. Plus, they can tell you a lot more than just where a thing is; they can tell you its temperature, the moisture level, whether or not it's moving, and more.
Suddenly, this unlocks a whole new realm of "small data;" if big data is looking at vast quantities of information and analyzing it for patterns, then small data is about looking at the data for an individual product - say, a container of yogurt in a shipment - and being able to know if it's likely to go off before it reaches the store.
This part of the Internet of Things holds incredible promise for improving everything from logistics to health care, and I believe we're still just on the cusp of understanding what this incredible technology can do - as when electricity was only used to power light bulbs.
3. Personal Quantification and Performance Optimization
Big data is not just for companies and governments but also for all of us individually. We can now benefit from the data generated from wearable devices such as smart watches or smart bracelets. Take the Up band from Jawbone as an example: the armband collects data on our calorie consumption, activity levels, and our sleep patterns. While it gives individuals rich insights, the real value is in analyzing the collective data.
In Jawbone's case, the company now collects 60 years worth of sleep data every night. Analyzing such volumes of data will bring entirely new insights that it can feed back to individual users.
The other area where we benefit from big data analytics is finding love - online this is. Most online dating sites apply big data tools and algorithms to find us the most appropriate matches.
4. Improving Healthcare and Public Health
The computing power of big data analytics enables us to decode entire DNA strings in minutes and will allow us to find new cures and better understand and predict disease patterns. Just think of what happens when all the individual data from smart watches and wearable devices can be used to apply it to millions of people and their various diseases. The clinical trials of the future won't be limited by small sample sizes but could potentially include everyone!
Apple's new health app, called ResearchKit, has effectively just turned your phone into a biomedical research device. Researchers can now create studies through which they collect data and input from users phones to compile data for health studies. Your phone might track how many steps you take in a day, or prompt you to answer questions about how you feel after your chemo, or how your Parkinson's disease is progressing. It's hoped that making the process easier and more automatic will dramatically increase the number of participants a study can attract as well as the fidelity of the data.
Big data techniques are already being used to monitor babies in a specialist premature and sick baby unit. By recording and analyzing every heartbeat and breathing pattern of every baby, the unit was able to develop algorithms that can now predict infections 24 hours before any physical symptoms appear. That way, the team can intervene early and save fragile babies in an environment where every hour counts.
What's more, big data analytics allow us to monitor and predict the developments of epidemics and disease outbreaks. Integrating data from medical records with social media analytics enables us to monitor flu outbreaks in real-time, simply by listening to what people are saying, i.e. "Feeling rubbish today - in bed with a cold".
Of course, while much has been made in the past of Google's ability to predict flu outbreaks based on search traffic, their model didn't work in 2014. Google itself admits that just because you search for "flu symptoms," it doesn't mean you're sick.
5. Improving Sports Performance
Most elite sports have now embraced big data analytics. We have the IBM SlamTracker tool for tennis tournaments; we use video analytics that track the performance of every player in a football or baseball game, and sensor technology in sports equipment such as basket balls or golf clubs allows us to get feedback (via smart phones and cloud servers) on our game and how to improve it. Many elite sports teams also track athletes outside of the sporting environment - using smart technology to track nutrition and sleep, as well as social media conversations to monitor emotional wellbeing.
The NFL has developed its own platform of applications to assist all 32 teams in making the best decisions based on everything from the condition of the grass on the field, to the weather, to statistics about an individual player's performance while in university. It is all in the name of strategy as well as reducing player injuries.
One of the really cool new things I have come across is a smart yoga mat: sensors embedded in the mat will be able to provide feedback on your postures, score your practice, and even guide you through an at-home practice.
6. Improving Science and Research
Science and research is currently being transformed by the new possibilities big data brings. Take, for example, CERN, the nuclear physics lab with its Large Hadron Collider, the world's largest and most powerful particle accelerator. Experiments to unlock the secrets of our universe - how it started and works - generate huge amounts of data.
The CERN data center has 65,000 processors to analyze its 30 petabytes of data. However, it uses the computing powers of thousands of computers distributed across 150 data centers worldwide to analyze the data. Such computing powers can be leveraged to transform so many other areas of science and research.
The computing power of big data could also be applied to any set of data, opening up new sources to scientists. Census data and other government collected data can more easily be accessed and analyzed by researchers to create bigger and better pictures of our health and social sciences.
7. Optimizing Machine and Device Performance
Big data analytics help machines and devices become smarter and more autonomous. For example, big data tools are used to operate Google's self-driving car. The Toyota Prius is fitted with cameras, GPS as well as powerful computers and sensors to safely drive on the road without the intervention of human beings. We can even use big data tools to optimize the performance of computers and data warehouses.
Xcel Energy initiated one of the first ever tests of a " smart grid" in Boulder, Colorado, installing smart meters on customers' homes that would allow them to log into a website and see their energy usage in real time. The smart grid would also theoretically allow power companies to predict usage in order to plan for future infrastructure needs and prevent brown out scenarios.
In Ireland, grocery chain Tescos has its warehouse employees wear armbands that track the goods they take from the shelves, distributes tasks, and even forecasts completion time for a job.
8. Improving Security and Law Enforcement.
Big data is applied heavily in improving security and enabling law enforcement. I am sure you are aware of the revelations that the National Security Agency (NSA) in the U.S. uses big data analytics to foil terrorist plots (and maybe spy on us). Others use big data techniques to detect and prevent cyber attacks. Police forces use big data tools to catch criminals and even predict criminal activity and credit card companies use big data use it to detect fraudulent transactions.
In February 2014, the Chicago Police Department sent uniformed officers to make "custom notification" visits to individuals they had identified as likely to commit a crime through a computer generated list. The idea was to prevent crime by providing certain individuals with information about job training programs, or let them know about increased penalties for people with certain backgrounds. But many community groups cried foul and called the practice profiling.
9. Improving and Optimizing Cities and Countries
Big data is used to improve many aspects of our cities and countries. For example, it allows cities to optimize traffic flows based on real time traffic information as well as social media and weather data. A number of cities are currently piloting big data analytics with the aim of turning themselves into Smart Cities, where the transport infrastructure and utility processes are all joined up. Where a bus would wait for a delayed train and where traffic signals predict traffic volumes and operate to minimize jams.
The city of Long Beach, California is using smart water meters to detect illegal watering in real time and have been used to help some homeowners cut their water usage by as much as 80 percent. That's vital when the state is going through its worst drought in recorded history and the governor has enacted the first-ever state-wide water restrictions.
Los Angeles uses data from magnetic road sensors and traffic cameras to control traffic lights and thus the flow (or congestion) of traffic around the city. The computerized system controls 4,500 traffic signals around the city and has reduced traffic congestion by an estimated 16 percent.
A tech startup called Veniam is testing a new way to create mobile wi-fi hotspots all over the city in Porto, Portugal. More than 600 city buses and taxis have been equipped with wifi transmitters, creating the largest free wi-fi hotspot in the world. Veniam sells the routers and service to the city, which in turn provides the wi-fi free to citizens, like a public utility. In exchange, the city gets an enormous amount of data - with the idea being that the data can be used to offset the cost of the wi-fi in other areas. For example, in Porto, sensors tell the city's waste management department when dumpsters are full, so they don't waste time, man hours, or fuel emptying containers that are only partly full.
10. Financial Trading
My final category of big data application comes from financial trading. High-Frequency Trading (HFT) is an area where big data finds a lot of use today. Here, big data algorithms are used to make trading decisions. Today, the majority of equity trading now takes place via data algorithms that increasingly take into account signals from social media networks and news websites to make, buy and sell decisions in split seconds.
Computers are programmed with complex algorithms that scan markets for a set of customizable conditions and search for trading opportunities. The programs can be designed to work with no human interaction or with human interaction, depending on the needs and desires of the client.
The most sophisticated of these programs are now also designed to change as markets change, rather than being hardcoded.
For me, the 10 categories I have outlined here represent the areas in which big data is applied the most. Of course there are so many other applications of big data and there will be many new categories as the tools become more widespread.
Some opinions expressed in this article may be those of a guest author and not necessarily Analytikus. Staff authors are listed http://www.datasciencecentral.com/profiles/blogs/spark-or-hadoop-which-is-the-best-framework
