Online analytical processing

Online analytical processing, or OLAP (/ˈoʊlæp/), is an approach to answering multi-dimensional analytical (MDA) queries swiftly in computing, .[1] OLAP is part of the broader category of business intelligence, which also encompasses relational database, report writing and data mining.[2] Typical applications of OLAP include business reporting for sales, marketing, management reporting, business process management (BPM),[3] budgeting and forecasting, financial reporting and similar areas, with new applications coming up, such as agriculture.[4] The term OLAP was created as a slight modification of the traditional database term online transaction processing (OLTP).[5]

OLAP tools enable users to analyze multidimensional data interactively from multiple perspectives. OLAP consists of three basic analytical operations: consolidation (roll-up), drill-down, and slicing and dicing.[6] Consolidation involves the aggregation of data that can be accumulated and computed in one or more dimensions. For example, all sales offices are rolled up to the sales department or sales division to anticipate sales trends. By contrast, the drill-down is a technique that allows users to navigate through the details. For instance, users can view the sales by individual products that make up a region's sales. Slicing and dicing is a feature whereby users can take out (slicing) a specific set of data of the OLAP cube and view (dicing) the slices from different viewpoints. These viewpoints are sometimes called dimensions (such as looking at the same sales by salesperson or by date or by customer or by product or by region, etc.)

Databases configured for OLAP use a multidimensional data model, allowing for complex analytical and ad hoc queries with a rapid execution time.[7] They borrow aspects of navigational databases, hierarchical databases and relational databases.

OLAP is typically contrasted to OLTP (online transaction processing), which is generally characterized by much less complex queries, in a larger volume, to process transactions rather than for the purpose of business intelligence or reporting. Whereas OLAP systems are mostly optimized for read, OLTP has to processes all kinds of queries (read, insert, update and delete).

Overview of OLAP systems

At the core of any OLAP system is an OLAP cube (also called a 'multidimensional cube' or a hypercube). It consists of numeric facts called measures that are categorized by dimensions. The measures are placed at the intersections of the hypercube, which is spanned by the dimensions as a vector space. The usual interface to manipulate an OLAP cube is a matrix interface, like Pivot tables in a spreadsheet program, which performs projection operations along the dimensions, such as aggregation or averaging.

The cube metadata is typically created from a star schema or snowflake schema or fact constellation of tables in a relational database. Measures are derived from the records in the fact table and dimensions are derived from the dimension tables.

Each measure can be thought of as having a set of labels, or meta-data associated with it. A dimension is what describes these labels; it provides information about the measure.

A simple example would be a cube that contains a store's sales as a measure, and Date/Time as a dimension. Each Sale has a Date/Time label that describes more about that sale.

For example:

 Sale Fact Table
+-------------+----------+
| sale_amount | time_id  |
+-------------+----------+            Time Dimension
|      2008.10|     1234 |----+     +---------+-------------------+
+-------------+----------+    |     | time_id | timestamp         |
                              |     +---------+-------------------+
                              +---->|   1234  | 20080902 12:35:43 |
                                    +---------+-------------------+

Multidimensional databases

Multidimensional structure is defined as "a variation of the relational model that uses multidimensional structures to organize data and express the relationships between data".[8] The structure is broken into cubes and the cubes are able to store and access data within the confines of each cube. "Each cell within a multidimensional structure contains aggregated data related to elements along each of its dimensions".[9] Even when data is manipulated it remains easy to access and continues to constitute a compact database format. The data still remains interrelated. Multidimensional structure is quite popular for analytical databases that use online analytical processing (OLAP) applications.[10] Analytical databases use these databases because of their ability to deliver answers to complex business queries swiftly. Data can be viewed from different angles, which gives a broader perspective of a problem unlike other models.[11]

Aggregations

It has been claimed that for complex queries OLAP cubes can produce an answer in around 0.1% of the time required for the same query on OLTP relational data.[12][13] The most important mechanism in OLAP which allows it to achieve such performance is the use of aggregations. Aggregations are built from the fact table by changing the granularity on specific dimensions and aggregating up data along these dimensions. The number of possible aggregations is determined by every possible combination of dimension granularities.

The combination of all possible aggregations and the base data contains the answers to every query which can be answered from the data.[14]

Because usually there are many aggregations that can be calculated, often only a predetermined number are fully calculated; the remainder are solved on demand. The problem of deciding which aggregations (views) to calculate is known as the view selection problem. View selection can be constrained by the total size of the selected set of aggregations, the time to update them from changes in the base data, or both. The objective of view selection is typically to minimize the average time to answer OLAP queries, although some studies also minimize the update time. View selection is NP-Complete. Many approaches to the problem have been explored, including greedy algorithms, randomized search, genetic algorithms and A* search algorithm.

Types

OLAP systems have been traditionally categorized using the following taxonomy.[15]

Multidimensional OLAP (MOLAP)

MOLAP (multi-dimensional online analytical processing) is the classic form of OLAP and is sometimes referred to as just OLAP. MOLAP stores this data in an optimized multi-dimensional array storage, rather than in a relational database.

Some MOLAP tools require the pre-computation and storage of derived data, such as consolidations – the operation known as processing. Such MOLAP tools generally utilize a pre-calculated data set referred to as a data cube. The data cube contains all the possible answers to a given range of questions. As a result, they have a very fast response to queries. On the other hand, updating can take a long time depending on the degree of pre-computation. Pre-computation can also lead to what is known as data explosion.

Other MOLAP tools, particularly those that implement the functional database model do not pre-compute derived data but make all calculations on demand other than those that were previously requested and stored in a cache.

Advantages of MOLAP

Disadvantages of MOLAP

Products

Examples of commercial products that use MOLAP are Cognos Powerplay, Oracle Database OLAP Option, MicroStrategy, Microsoft Analysis Services, Essbase, TM1, Jedox, and icCube.

Relational OLAP (ROLAP)

ROLAP works directly with relational databases and does not require pre-computation. The base data and the dimension tables are stored as relational tables and new tables are created to hold the aggregated information. It depends on a specialized schema design. This methodology relies on manipulating the data stored in the relational database to give the appearance of traditional OLAP's slicing and dicing functionality. In essence, each action of slicing and dicing is equivalent to adding a "WHERE" clause in the SQL statement. ROLAP tools do not use pre-calculated data cubes but instead pose the query to the standard relational database and its tables in order to bring back the data required to answer the question. ROLAP tools feature the ability to ask any question because the methodology does not limit to the contents of a cube. ROLAP also has the ability to drill down to the lowest level of detail in the database.

While ROLAP uses a relational database source, generally the database must be carefully designed for ROLAP use. A database which was designed for OLTP will not function well as a ROLAP database. Therefore, ROLAP still involves creating an additional copy of the data. However, since it is a database, a variety of technologies can be used to populate the database.

Advantages of ROLAP

Disadvantages of ROLAP

Performance of ROLAP

In the OLAP industry ROLAP is usually perceived as being able to scale for large data volumes, but suffering from slower query performance as opposed to MOLAP. The OLAP Survey, the largest independent survey across all major OLAP products, being conducted for 6 years (2001 to 2006) have consistently found that companies using ROLAP report slower performance than those using MOLAP even when data volumes were taken into consideration.

However, as with any survey there are a number of subtle issues that must be taken into account when interpreting the results.

Downside of flexibility

Some companies select ROLAP because they intend to re-use existing relational database tables—these tables will frequently not be optimally designed for OLAP use. The superior flexibility of ROLAP tools allows this less than optimal design to work, but performance suffers. MOLAP tools in contrast would force the data to be re-loaded into an optimal OLAP design.

Hybrid OLAP (HOLAP)

The undesirable trade-off between additional ETL cost and slow query performance has ensured that most commercial OLAP tools now use a "Hybrid OLAP" (HOLAP) approach, which allows the model designer to decide which portion of the data will be stored in MOLAP and which portion in ROLAP.

There is no clear agreement across the industry as to what constitutes "Hybrid OLAP", except that a database will divide data between relational and specialized storage.[16] For example, for some vendors, a HOLAP database will use relational tables to hold the larger quantities of detailed data, and use specialized storage for at least some aspects of the smaller quantities of more-aggregate or less-detailed data. HOLAP addresses the shortcomings of MOLAP and ROLAP by combining the capabilities of both approaches. HOLAP tools can utilize both pre-calculated cubes and relational data sources.

Vertical partitioning

In this mode HOLAP stores aggregations in MOLAP for fast query performance, and detailed data in ROLAP to optimize time of cube processing.

Horizontal partitioning

In this mode HOLAP stores some slice of data, usually the more recent one (i.e. sliced by Time dimension) in MOLAP for fast query performance, and older data in ROLAP. Moreover, we can store some dices in MOLAP and others in ROLAP, leveraging the fact that in a large cuboid, there will be dense and sparse subregions.[17]

Products

The first product to provide HOLAP storage was Holos, but the technology also became available in other commercial products such as Microsoft Analysis Services, Oracle Database OLAP Option, MicroStrategy and SAP AG BI Accelerator. The hybrid OLAP approach combines ROLAP and MOLAP technology, benefiting from the greater scalability of ROLAP and the faster computation of MOLAP. For example, a HOLAP server may allow large volumes of detail data to be stored in a relational database, while aggregations are kept in a separate MOLAP store. The Microsoft SQL Server 7.0 OLAP Services supports a hybrid OLAP server

Comparison

Each type has certain benefits, although there is disagreement about the specifics of the benefits between providers.

Other types

The following acronyms are also sometimes used, although they are not as widespread as the ones above:

APIs and query languages

Unlike relational databases, which had SQL as the standard query language, and widespread APIs such as ODBC, JDBC and OLEDB, there was no such unification in the OLAP world for a long time. The first real standard API was OLE DB for OLAP specification from Microsoft which appeared in 1997 and introduced the MDX query language. Several OLAP vendors – both server and client – adopted it. In 2001 Microsoft and Hyperion announced the XML for Analysis specification, which was endorsed by most of the OLAP vendors. Since this also used MDX as a query language, MDX became the de facto standard.[18] Since September-2011 LINQ can be used to query SSAS OLAP cubes from Microsoft .NET.[19]

Products

History

The first product that performed OLAP queries was Express, which was released in 1970 (and acquired by Oracle in 1995 from Information Resources).[20] However, the term did not appear until 1993 when it was coined by Edgar F. Codd, who has been described as "the father of the relational database". Codd's paper[1] resulted from a short consulting assignment which Codd undertook for former Arbor Software (later Hyperion Solutions, and in 2007 acquired by Oracle), as a sort of marketing coup. The company had released its own OLAP product, Essbase, a year earlier. As a result, Codd's "twelve laws of online analytical processing" were explicit in their reference to Essbase. There was some ensuing controversy and when Computerworld learned that Codd was paid by Arbor, it retracted the article. OLAP market experienced strong growth in late 90s with dozens of commercial products going into market. In 1998, Microsoft released its first OLAP Server – Microsoft Analysis Services, which drove wide adoption of OLAP technology and moved it into mainstream.

Product comparison

OLAP Clients

OLAP clients include many spreadsheet programs like Excel, web application, sql,dashboard tools, etc.

Market structure

Below is a list of top OLAP vendors in 2006, with figures in millions of US Dollars.[21]

Vendor Global Revenue Consolidated company
Microsoft Corporation 1,806 Microsoft
Hyperion Solutions Corporation 1,077 Oracle
Cognos 735 IBM
Business Objects 416 SAP
MicroStrategy 416 MicroStrategy
SAP AG 330 SAP
Cartesis (SAP) 210 SAP
Applix 205 IBM
Infor 199 Infor
Oracle Corporation 159 Oracle
Others 152 Others
Total 5,700

Open-source

See also

Bibliography

References

  1. 1 2 Codd E.F.; Codd S.B. & Salley C.T. (1993). "Providing OLAP (On-line Analytical Processing) to User-Analysts: An IT Mandate" (PDF). Codd & Date, Inc. Retrieved 2008-03-05.
  2. ↑ Deepak Pareek (2007). Business Intelligence for Telecommunications. CRC Press. pp. 294 pp. ISBN 0-8493-8792-2. Retrieved 2008-03-18.
  3. ↑ Apostolos Benisis (2010). Business Process Management:A Data Cube To Analyze Business Process Simulation Data For Decision Making. VDM Verlag Dr. Müller e.K. pp. 204 pp. ISBN 978-3-639-22216-6.
  4. ↑ Abdullah, Ahsan (November 2009). "Analysis of mealybug incidence on the cotton crop using ADSS-OLAP (Online Analytical Processing) tool". Computers and Electronics in Agriculture. 69 (1): 59–72. doi:10.1016/j.compag.2009.07.003.
  5. ↑ "OLAP Council White Paper" (PDF). OLAP Council. 1997. Retrieved 2008-03-18.
  6. ↑ O'Brien & Marakas, 2011, p. 402-403
  7. ↑ Hari Mailvaganam (2007). "Introduction to OLAP – Slice, Dice and Drill!". Data Warehousing Review. Retrieved 2008-03-18.
  8. ↑ O'Brien & Marakas, 2009, pg 177
  9. ↑ O'Brien & Marakas, 2009, pg 178
  10. ↑ (O'Brien & Marakas, 2009)
  11. ↑ Williams, C., Garza, V.R., Tucker, S, Marcus, A.M. (1994, January 24). Multidimensional models boost viewing options. InfoWorld, 16(4)
  12. ↑ MicroStrategy, Incorporated (1995). "The Case for Relational OLAP" (PDF). Retrieved 2008-03-20.
  13. ↑ Surajit Chaudhuri & Umeshwar Dayal (1997). "An overview of data warehousing and OLAP technology". SIGMOD Rec. ACM. 26 (1): 65. doi:10.1145/248603.248616. Retrieved 2008-03-20.
  14. ↑ Gray, Jim; Chaudhuri, Surajit; Layman, Andrew; Reichart, Don; Venkatrao, Murali; Pellow, Frank; Pirahesh, Hamid (1997). "Data Cube: {A} Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals". J. Data Mining and Knowledge Discovery. 1 (1): 29–53. Retrieved 2008-03-20.
  15. ↑ Nigel Pendse (2006-06-27). "OLAP architectures". OLAP Report. Archived from the original on January 24, 2008. Retrieved 2008-03-17.
  16. 1 2 3 Bach Pedersen, Torben; S. Jensen, Christian (December 2001). "Multidimensional Database Technology". Distributed Systems Online. IEEE: 40–46. ISSN 0018-9162.
  17. ↑ Owen Kaser and Daniel Lemire, Attribute Value Reordering for Efficient Hybrid OLAP, Information Sciences, Volume 176, Issue 16, pages 2279-2438, 2006.
  18. ↑ Nigel Pendse (2007-08-23). "Commentary: OLAP API wars". OLAP Report. Archived from the original on May 28, 2008. Retrieved 2008-03-18.
  19. ↑ "SSAS Entity Framework Provider for LINQ to SSAS OLAP".
  20. ↑ Nigel Pendse (2007-08-23). "The origins of today's OLAP products". OLAP Report. Archived from the original on December 21, 2007. Retrieved November 27, 2007.
  21. ↑ Nigel Pendse (2006). "OLAP Market". OLAP Report. Retrieved 2008-03-17.
  22. ↑ Yegulalp, Serdar (2015-06-11). "LinkedIn fills another SQL-on-Hadoop niche". InfoWorld. Retrieved 2016-11-19.
This article is issued from Wikipedia - version of the 11/30/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.