Principal Component Analysis And Randomness Test For Big Data Analysis

Author : Mieko Tanaka-Yamawaki,Yumihiko Ikura
Publisher : Springer Nature
Page : 153 pages
File Size : 49,5 Mb
Release : 2023-05-23
Category : Business & Economics
ISBN : 9789811939679

Get Book

Principal Component Analysis and Randomness Test for Big Data Analysis by Mieko Tanaka-Yamawaki,Yumihiko Ikura Pdf

This book presents the novel approach of analyzing large-sized rectangular-shaped numerical data (so-called big data). The essence of this approach is to grasp the "meaning" of the data instantly, without getting into the details of individual data. Unlike conventional approaches of principal component analysis, randomness tests, and visualization methods, the authors' approach has the benefits of universality and simplicity of data analysis, regardless of data types, structures, or specific field of science. First, mathematical preparation is described. The RMT-PCA and the RMT-test utilize the cross-correlation matrix of time series, C = XXT, where X represents a rectangular matrix of N rows and L columns and XT represents the transverse matrix of X. Because C is symmetric, namely, C = CT, it can be converted to a diagonal matrix of eigenvalues by a similarity transformation SCS-1 = SCST using an orthogonal matrix S. When N is significantly large, the histogram of the eigenvalue distribution can be compared to the theoretical formula derived in the context of the random matrix theory (RMT, in abbreviation). Then the RMT-PCA applied to high-frequency stock prices in Japanese and American markets is dealt with. This approach proves its effectiveness in extracting "trendy" business sectors of the financial market over the prescribed time scale. In this case, X consists of N stock- prices of length L, and the correlation matrix C is an N by N square matrix, whose element at the i-th row and j-th column is the inner product of the price time series of the length L of the i-th stock and the j-th stock of the equal length L. Next, the RMT-test is applied to measure randomness of various random number generators, including algorithmically generated random numbers and physically generated random numbers. The book concludes by demonstrating two applications of the RMT-test: (1) a comparison of hash functions, and (2) stock prediction by means of randomness, including a new index of off-randomness related to market decline.

Author : Mieko Tanaka-Yamawaki,Yumihiko Ikura
Publisher : Springer
Page : 0 pages
File Size : 53,7 Mb
Release : 2022-09-11
Category : Business & Economics
ISBN : 4431559043

Get Book

Principal Component Analysis and Randomness Tests for Big Data Analysis by Mieko Tanaka-Yamawaki,Yumihiko Ikura Pdf

This book presents the novel approach of analyzing large-sized numerical data (so-called big data). The essence of this approach is to grasp the "meaning" of the data instantly, without getting into the details of individual data. Unlike conventional approaches of principal component analysis, randomness tests, and visualization methods, the authors' approach has the benefits of universality and simplicity of data analysis, regardless of data types, structures, or specific field of science. First, mathematical preparation is described. The RMT-PCA and the RMT-test utilize the cross-correlation matrix of time series, C = XXT, where X represents a rectangular matrix of N rows and L columns and XT represents the transverse matrix of X. The RMT-PCA uses N samples of time series of length L. The RMT-test uses N elements of length L by cutting a single data to N pieces. Because C is symmetric, namely, C = CT, it can be converted to a diagonal matrix of eigenvalues by a similarity transformation SCST using an orthogonal matrix S. When N is significantly large, the histogram of the eigenvalue distribution can be compared to the theoretical formula derived in the context of the random matrix theory (RMT, in abbreviation). Then the RMT-PCA is applied to high-frequency stock prices in Japanese and American markets. This approach proves its effectiveness in extracting "trendy" business sectors of the financial market over the prescribed time scale. In this case, X consists of N stock- prices of length L, and the correlation matrix C is an N by N square matrix, whose element at the i-th row and j-th column is the inner product of the price time series of the length L of the i-th stock and the j-th stock of the equal length L. Next, the RMT-test is applied to measure randomness of various random number generators, including algorithmically generated random numbers and physically generated random numbers. The book concludes by demonstrating three applications of the RMT-test: (1) a comparison of hash functions, (2) choice of safe stocks, and (3) prediction of stock index by means of a sudden change of randomness.

Author : Robert C. Qiu,Paul Antonik
Publisher : John Wiley & Sons
Page : 632 pages
File Size : 42,9 Mb
Release : 2017-02-08
Category : Technology & Engineering
ISBN : 9781118716793

Get Book

Smart Grid using Big Data Analytics by Robert C. Qiu,Paul Antonik Pdf

This book is aimed at students in communications and signal processing who want to extend their skills in the energy area. It describes power systems and why these backgrounds are so useful to smart grid, wireless communications being very different to traditional wireline communications.

Author : Mohammad Shamsul Arefin
Publisher : Springer Nature
Page : 1053 pages
File Size : 41,8 Mb
Release : 2024-06-30
Category : Electronic
ISBN : 9789819989379

Get Book

Proceedings of the 2nd International Conference on Big Data, IoT and Machine Learning by Mohammad Shamsul Arefin Pdf

Author : Robert Qiu,Michael Wicks
Publisher : Springer Science & Business Media
Page : 633 pages
File Size : 55,6 Mb
Release : 2013-08-04
Category : Technology & Engineering
ISBN : 9781461445449

Get Book

Cognitive Networked Sensing and Big Data by Robert Qiu,Michael Wicks Pdf

Wireless Distributed Computing and Cognitive Sensing defines high-dimensional data processing in the context of wireless distributed computing and cognitive sensing. This book presents the challenges that are unique to this area such as synchronization caused by the high mobility of the nodes. The author will discuss the integration of software defined radio implementation and testbed development. The book will also bridge new research results and contextual reviews. Also the author provides an examination of large cognitive radio network; hardware testbed; distributed sensing; and distributed computing.

Author : Tsun Se Cheong,Xunpeng (Roc) Shi,Yanfei Li,Yongping Sun
Publisher : Frontiers Media SA
Page : 485 pages
File Size : 45,9 Mb
Release : 2022-07-25
Category : Technology & Engineering
ISBN : 9782889765966

Get Book

Application of Big Data, Deep Learning, Machine Learning, and Other Advanced Analytical Techniques in Environmental Economics and Policy by Tsun Se Cheong,Xunpeng (Roc) Shi,Yanfei Li,Yongping Sun Pdf

Author : Bengt Sunden,Jin-yuan Qian,Junhui Zhang
Publisher : MDPI
Page : 246 pages
File Size : 54,8 Mb
Release : 2020-12-29
Category : Technology & Engineering
ISBN : 9783039365111

Get Book

Smart Flow Control Processes in Micro Scale Volume 2 by Bengt Sunden,Jin-yuan Qian,Junhui Zhang Pdf

In recent years, microfluidic devices with a large surface-to-volume ratio have witnessed rapid development, allowing them to be successfully utilized in many engineering applications. A smart control process has been proposed for many years, while many new innovations and enabling technologies have been developed for smart flow control, especially concerning “smart flow control” at the microscale. This Special Issue aims to highlight the current research trends related to this topic, presenting a collection of 33 papers from leading scholars in this field. Among these include studies and demonstrations of flow characteristics in pumps or valves as well as dynamic performance in roiling mill systems or jet systems to the optimal design of special components in smart control systems.

Author : Saeid Pourroostaei Ardakani,Ali Cheshmehzangi
Publisher : Springer Nature
Page : 143 pages
File Size : 41,7 Mb
Release : 2023-10-29
Category : Science
ISBN : 9789819955435

Get Book

Big Data Analytics for Smart Urban Systems by Saeid Pourroostaei Ardakani,Ali Cheshmehzangi Pdf

Big Data Analytics for Smart Urban Systems aims to introduce Big data solutions for urban sustainability smart applications, particularly for smart urban systems. It focuses on intelligent big data which takes the benefits of machine learning to analyse large and rapidly changing datasets in smart urban systems. The state-of-the-art Big data analytics applications are presented and discussed to highlight the feasibility of big data and machine learning solutions to enhance smart urban systems, smart operations, urban management, and urban governance. The key benefits of this book are, (1) to introduce the principles of machine learning-enabled big data analysis in smart urban systems, (2) to present the state-of-the-art data analysis solutions in smart management and operations, and (3) to understand the principles of big data analytics for smart cities and communities. Endorsements ‘Over the many years of collaboration between academia and industry, we noticed the common language is ‘big data’; with that, we have developed novel ideas to bridge the gaps and help promote innovation, technologies, and science’.- Tian Tang, Independent Researcher, China ‘Big Data Analytics is a fascinating research area, particularly for cities and city transformations. This book is valuable to those who think vigorously and aim to act ahead’.- Li Xie, Independent Researcher, China ‘For urban critiques, knowledge trains aspiring opportunities toward outstanding manifestations. Smartness has evolved or/ advanced rambunctious & embracing realities along (with) novel directions and nurturing integrated city knowledge’.- Aaron Golden, SELECT Consultants, UK

Author : Jules J Berman
Publisher : Academic Press
Page : 480 pages
File Size : 51,5 Mb
Release : 2018-07-23
Category : Computers
ISBN : 9780128156100

Get Book

Principles and Practice of Big Data by Jules J Berman Pdf

Principles and Practice of Big Data: Preparing, Sharing, and Analyzing Complex Information, Second Edition updates and expands on the first edition, bringing a set of techniques and algorithms that are tailored to Big Data projects. The book stresses the point that most data analyses conducted on large, complex data sets can be achieved without the use of specialized suites of software (e.g., Hadoop), and without expensive hardware (e.g., supercomputers). The core of every algorithm described in the book can be implemented in a few lines of code using just about any popular programming language (Python snippets are provided). Through the use of new multiple examples, this edition demonstrates that if we understand our data, and if we know how to ask the right questions, we can learn a great deal from large and complex data collections. The book will assist students and professionals from all scientific backgrounds who are interested in stepping outside the traditional boundaries of their chosen academic disciplines. Presents new methodologies that are widely applicable to just about any project involving large and complex datasets Offers readers informative new case studies across a range scientific and engineering disciplines Provides insights into semantics, identification, de-identification, vulnerabilities and regulatory/legal issues Utilizes a combination of pseudocode and very short snippets of Python code to show readers how they may develop their own projects without downloading or learning new software

Author : Mayer Alvo
Publisher : Springer Nature
Page : 442 pages
File Size : 54,5 Mb
Release : 2022-11-30
Category : Mathematics
ISBN : 9783031067846

Get Book

Statistical Inference and Machine Learning for Big Data by Mayer Alvo Pdf

This book presents a variety of advanced statistical methods at a level suitable for advanced undergraduate and graduate students as well as for others interested in familiarizing themselves with these important subjects. It proceeds to illustrate these methods in the context of real-life applications in a variety of areas such as genetics, medicine, and environmental problems. The book begins in Part I by outlining various data types and by indicating how these are normally represented graphically and subsequently analyzed. In Part II, the basic tools in probability and statistics are introduced with special reference to symbolic data analysis. The most useful and relevant results pertinent to this book are retained. In Part III, the focus is on the tools of machine learning whereas in Part IV the computational aspects of BIG DATA are presented. This book would serve as a handy desk reference for statistical methods at the undergraduate and graduate level as well as be useful in courses which aim to provide an overview of modern statistics and its applications.

Author : Fei Hu
Publisher : CRC Press
Page : 463 pages
File Size : 41,5 Mb
Release : 2016-04-27
Category : Computers
ISBN : 9781498734875

Get Book

Big Data by Fei Hu Pdf

Although there are already some books published on Big Data, most of them only cover basic concepts and society impacts and ignore the internal implementation details-making them unsuitable to R&D people. To fill such a need, Big Data: Storage, Sharing, and Security examines Big Data management from an R&D perspective. It covers the 3S desi

Author : Momiao Xiong
Publisher : CRC Press
Page : 400 pages
File Size : 51,7 Mb
Release : 2018-06-14
Category : Mathematics
ISBN : 9781351172622

Get Book

Big Data in Omics and Imaging by Momiao Xiong Pdf

Big Data in Omics and Imaging: Integrated Analysis and Causal Inference addresses the recent development of integrated genomic, epigenomic and imaging data analysis and causal inference in big data era. Despite significant progress in dissecting the genetic architecture of complex diseases by genome-wide association studies (GWAS), genome-wide expression studies (GWES), and epigenome-wide association studies (EWAS), the overall contribution of the new identified genetic variants is small and a large fraction of genetic variants is still hidden. Understanding the etiology and causal chain of mechanism underlying complex diseases remains elusive. It is time to bring big data, machine learning and causal revolution to developing a new generation of genetic analysis for shifting the current paradigm of genetic analysis from shallow association analysis to deep causal inference and from genetic analysis alone to integrated omics and imaging data analysis for unraveling the mechanism of complex diseases. FEATURES Provides a natural extension and companion volume to Big Data in Omic and Imaging: Association Analysis, but can be read independently. Introduce causal inference theory to genomic, epigenomic and imaging data analysis Develop novel statistics for genome-wide causation studies and epigenome-wide causation studies. Bridge the gap between the traditional association analysis and modern causation analysis Use combinatorial optimization methods and various causal models as a general framework for inferring multilevel omic and image causal networks Present statistical methods and computational algorithms for searching causal paths from genetic variant to disease Develop causal machine learning methods integrating causal inference and machine learning Develop statistics for testing significant difference in directed edge, path, and graphs, and for assessing causal relationships between two networks The book is designed for graduate students and researchers in genomics, epigenomics, medical image, bioinformatics, and data science. Topics covered are: mathematical formulation of causal inference, information geometry for causal inference, topology group and Haar measure, additive noise models, distance correlation, multivariate causal inference and causal networks, dynamic causal networks, multivariate and functional structural equation models, mixed structural equation models, causal inference with confounders, integer programming, deep learning and differential equations for wearable computing, genetic analysis of function-valued traits, RNA-seq data analysis, causal networks for genetic methylation analysis, gene expression and methylation deconvolution, cell –specific causal networks, deep learning for image segmentation and image analysis, imaging and genomic data analysis, integrated multilevel causal genomic, epigenomic and imaging data analysis.

Author : Zehui Zhan,Bin Zou,William Yeoh
Publisher : Springer Nature
Page : 1364 pages
File Size : 46,5 Mb
Release : 2023-01-20
Category : Mathematics
ISBN : 9789464630343

Get Book

Proceedings of the 2022 3rd International Conference on Big Data and Informatization Education (ICBDIE 2022) by Zehui Zhan,Bin Zou,William Yeoh Pdf

This is an open access book. The 2022 3rd International Conference on Big Data and Informatization Education (ICBDIE2022) was held on April 8-10, 2022 in Beijing, China. ICBDIE2022 is to bring together innovative academics and industrial experts in the field of Big Data and Informatization Education to a common forum. The primary goal of the conference is to promote research and developmental activities in Big Data and Informatization Education and another goal is to promote scientific information interchange between researchers, developers, engineers, students, and practitioners working all around the world. The conference will be held every year to make it an ideal platform for people to share views and experiences in international conference on Big Data and Informatization Education and related areas.

Author : Bruno Carpentieri,Paola Lecca
Publisher : John Wiley & Sons
Page : 437 pages
File Size : 40,6 Mb
Release : 2024-05-31
Category : Medical
ISBN : 9781119846550

Get Book

Big Data Analysis and Artificial Intelligence for Medical Sciences by Bruno Carpentieri,Paola Lecca Pdf

Big Data Analysis and Artificial Intelligence for Medical Sciences Overview of the current state of the art on the use of artificial intelligence in medicine and biology Big Data Analysis and Artificial Intelligence for Medical Sciences demonstrates the efforts made in the fields of Computational Biology and medical sciences to design and implement robust, accurate, and efficient computer algorithms for modeling the behavior of complex biological systems much faster than using traditional modeling approaches based solely on theory. With chapters written by international experts in the field of medical and biological research, Big Data Analysis and Artificial Intelligence for Medical Sciences includes information on: Studies conducted by the authors which are the result of years of interdisciplinary collaborations with clinicians, computer scientists, mathematicians, and engineers Differences between traditional computational approaches to data processing (those of mathematical biology) versus the experiment-data-theory-model-validation cycle Existing approaches to the use of big data in the healthcare industry, such as through IBM’s Watson Oncology, Microsoft’s Hanover, and Google’s DeepMind Difficulties in the field that have arisen as a result of technological changes, and potential future directions these changes may take A timely and up-to-date resource on the integration of artificial intelligence in medicine and biology, Big Data Analysis and Artificial Intelligence for Medical Sciences is of great benefit not only to professional scholars, but also MSc or PhD program students eager to explore advancement in the field.

Author : Yuji Aruka
Publisher : Springer Nature
Page : 434 pages
File Size : 52,6 Mb
Release : 2022-05-23
Category : Business & Economics
ISBN : 9789811909375

Get Book

Digital Designs for Money, Markets, and Social Dilemmas by Yuji Aruka Pdf

An innovative feature of this book is its econocentric structure, focusing on digital designs. From the outset, econocentrism is assumed to be a core engine of capitalism, like money. The new coronavirus pandemic has changed lifestyles worldwide, which are unlikely ever to return in their original form. This great transformation will change the nature of the socio-economic system itself and will be centered on digital designs. At present, money already is beginning to undergo a major revolution in that sense. Many books dealing with digital designs and innovations have been published, but few if any of them focus on monetary and analytical methods in the way that this present volume does.The book then contains 6 parts: Evolution of money and thinking complexities in the AI era; Goods market and the future of labor market; Computational social approaches to social dilemmas, smart city, cryptocurrencies; Artificial market experiments; The randomness and high frequencies in financial data; Other trading strategy issues and the effects of AI usage. These issues may be indispensable subjects in our age. Study these subject, and have a step forward to the future society!