Big Data: What Is Spark - An Easy Explanation For Absolutely Anyone

Like Hadoop, Spark is open-source and under the wing of the 

Apache  Software Foundation. Essentially, open-source 

means the code can be freely used by anyone. Beyond

 that, it can also be altered by anyone to produce custom 

versions aimed at particular problems, or industries.

Volunteer developers, as well as those working at 

companies which produce custom versions, constantly 

refine and update the core software adding more 

features and efficiencies. In fact Spark was the most 

active project at Apache last year. It was also the most 

active of all of the open  source Big Dataapplications, with 

over 500 contributors from more than 200 

organizations.

Spark is seen by techies in the industry as a more 

advanced product than Hadoop - it is newer, and designed 

to work by processing data in chunks "in memory". 

This means it transfers data from the physical, 

magnetic hard discs into far-faster electronic memory where 

processing can be carried out far more quickly - up to 

100 times faster in some operations.

Spark has proven very popular and is used by many 

large companies for huge, multi-petabyte data storage 

and analysis. This has partly been because of its speed. 

Last year, Spark set a world record by completing a 

benchmark test involving sorting 100 terabytes of data 

in 23 minutes - the previous world record of 71 

minutes being held by Hadoop.

Additionally, Spark has proven itself to be highly suited to 

Machine Learning applications. Machine Learning is one of

 the fastest growing and most exciting areas of 

computer science, where computers are being taught to 

spot patterns in data, and adapt their behaviour based 

on automated modelling and analysis of whatever task 

they are trying to perform.

Unlike Hadoop, Spark does not come with its own file system - 

instead it can be integrated with many file systems including 

Hadoop's HDFS, MongoDB and Amazon's S3 system.

Another element of the framework is Spark Streaming, which 

allows applications to be developed which perform analytics 

on streaming, real-time data - such as automatically 

analyzing video 

Source :- https://www.bernardmarr.com/default.asp?contentID=1079

Artificial Intelligence

Machine Learning And Artificial Intelligence In Business: Year In Review, 2018

Much has happened in artificial intelligence (AI) this year. From NVIDIANVDA +0.75%, Intel INTC +2.34% and a host of startups announcing new chips focused on both training and run time inference, to announcements of a wide variety of new algorithms, much of the news has been focused on research and academia. The challenge is to notice what is happening with AI and machine learning (ML) that business can look to for short term adoption to aid in performance.

The technologies and techniques of AI and ML are still so new that the main adopters of the techniques are the large software companies able to hire and to invest in the necessary expertise. Those companies come into focus by looking at two pairs of companies which show the two angles of attack.

There are Google GOOGL +3.21% and Amazon, early movers due to their cloud foundation who are working to figure out how to generalize techniques developed for their products, in order to attract the wider business market. On the other side are Microsoft MSFT +3.46% and IBM IBM +0.11%, companies with long histories of addressing business needs who are layering AI across product offerings. Yes, there are plenty of other companies also working to add machine learning to their product lines, and an even larger body of startups focusing on building solutions with AI and ML techniques at the core, but those four companies are well known and display the two key methods to attack the problem.

Within the new technologies, there are certain areas that have gained more traction than others that are even earlier in their lifecycles. Let us discuss a few of the areas.

Natural Language

For years, search engines have been getting better at understanding more natural syntax, both written and spoken, for questions people ask. The ability to understand more human language is natural language processing (NLP). While the two previous years saw attention on personal assistants, 2018 was the year that serious inroads into business analytic applications have been made in NLP.

Just about every business intelligence (BI) and enterprise software company released versions of NLP for their analytic applications. The ability for business line management and employees to type or speak “what’s the sales last quarter for region X?” rather than have to deal with drop down list boxes and other more technical UX features is helping move BI from shelf-ware to regular use.

On the other side, reporting is almost coming full circle. From early full text reports on mainframes, to basic graphics being added, to the current world of advanced visualizations, images have taken over in much of the analytics world. Yet people see things differently, so combining visualizations and text better serves a wider audience.

Source

Are You A Better Statistician Than A Data Analyst?

There is nothing better than choosing the right data for your analytics, every data analyst has to curtail a problem statement by inferring the data, right selection can bring value to the further journey. One of the easiest ways to understand is through fact-finding using basic statistics, data visualisation, and other techniques to understand and infer the meaning, this is where the statistician in you will be able to help. While there are different ways to be able to build this skill in your journey, I believe these 3 R’s will help you succeed.

1. Recognise: Data classification using Descriptive, Inferential Statistics and various sampling techniques
2. Ratify: Approve your thought process for each find you have in the process, never carry assumptions, have consultations with business stakeholders and confirm the findings. Incorrect data decisions have a straight influence on the impact it creates further down the chain.
3. Reinforce: At every step in assessing your data, there will be loads of new learnings uncovered on how to approach a data problem, at times this is quite unique to the kind of business problem you are trying to solve, maintaining a log of learnings and ensuring you reinforce it back into the data processing lifecycle later is crucial as not only it speeds up the things, will also help you provide transparency into the model and build a sustainable solution.

In this article, I have provided my view on various statistical techniques available at hand that you will be applying to recognise your data better.

It all starts with identifying a data series for a problem which is typically a collection of observations often collected as a sample space, a random subset from a given population, descriptive statistics helps us understand and compare the various dimensions of sample space. Here are a few quick measures available to assess the data better.

• Distribution helps to identify the range of values within the data set for a given numeric feature and allow us to mark each observation to one of the available buckets, it gives a quick classification view of where it belongs and easily helps in finding an outlier.
• Central tendency helps identify the correlation of each observation against what we called a nominated central value. There are different means of adopting a central value, the quite common ones that you hear are mean (average), median (middle half) and mode (most recurring value).
• Dispersion provides a view on the spread of given data series from their mean, it is quite common that when you compare two different series they look alike and might be completely different at their spread, widely used methods to measure dispersion is standard deviation, it provides a scaled-down view of the deviations.

Understanding the data spread is as well a typical need, helps to make a decision on the data and to understand the correlation. There are different facets to it if distributed to three equal bins, each is termed as a quartile and median lies on the second Quartile (Q2), Q1 and Q3 are referred to as Lower and Upper Quartiles. Difference between first (Q1) and third quartile (Q3) is called Interquartile Range (IQR). If you choose to split into to 5 equals, each is referred as Quintile, If 10 then it’s a decile. Sample space at times is divided into n-subsets where each nth split is referred to as quantile.

Ensure you have a fair view on the data by looking at what we call a normaliSation graph, an example is here and it is quite possible that you have a skewed distribution due to the presence of an outlier observation in a sample space. An analyst in you will then have to treat these data points.

Mean is heavily influenced by an outlier observation, replace any missing values in continuous data through its mean and categorical choose a mode (most recurring value.)

As it is quite not possible to understand the entire population. Inferential Statistics helps us validate a set of samples and review if they have the right information. Most widely used is the Hypothesis or significance testing.

To conclude this article, nature of data is what signifies the direction of your outcome hence it is important as an analyst you gather and analyse data for your data lifecycle and it will influence the decision making process.

I hope this article helped you in getting started with basic statistics and thumb rules of how to improve the thinking behind data selection, this is the first building block behind all modelling techniques used in a machine learning solution.

What is Data Analytics (DA)?

Definition of Data Analytics:

Data analytics (DA) is the process of examining data sets in order to draw conclusions about the information they contain, increasingly with the aid of specialized systems and software. Data analytics technologies and techniques are widely used in commercial industries to enable organizations to make more-informed business decisions and by scientists and researchers to verify or disprove scientific models, theories and hypotheses.

Data analytics initiatives can help businesses increase revenues, improve operational efficiency, optimize marketing campaigns and customer service efforts, respond more quickly to emerging market trends and gain a competitive edge over rivals -- all with the ultimate goal of boosting business performance. Depending on the particular application, the data that's analyzed can consist of either historical records or new information that has been processed for real-time analytics uses. In addition, it can come from a mix of internal systems and external data sources.

Types of data analytics applications:

At a high level, data analytics methodologies include exploratory data analysis (EDA), which aims to find patterns and relationships in data, and confirmatory data analysis (CDA), which applies statistical techniques to determine whether hypotheses about a data set are true or false. EDA is often compared to detective work, while CDA is akin to the work of a judge or jury during a court trial -- a distinction first drawn by statistician John W. Tukey in his 1977 book Exploratory Data Analysis.

Data analytics can also be separated into quantitative data analysis and qualitative data analysis. The former involves analysis of numerical data with quantifiable variables that can be compared or measured statistically. The qualitative approach is more interpretive -- it focuses on understanding the content of non-numerical data like text, images, audio and video, including common phrases, themes and points of view.

At the application level, BI and reporting provides business executives and other corporate workers with actionable information about key performance indicators, business operations, customers and more. In the past, data queries and reports typically were created for end users by BI developers working in IT or for a centralized BI team; now, organizations increasingly use self-service BI tools that let execs, business analysts and operational workers run their own ad hoc queries and build reports themselves.

Source: searchdatamanagement.techtarget.com