Markov Processes And Controlled Markov Chains

Author : Zhenting Hou,Jerzy A. Filar,Anyue Chen
Publisher : Springer Science & Business Media
Page : 501 pages
File Size : 45,7 Mb
Release : 2013-12-01
Category : Mathematics
ISBN : 9781461302650

Get Book

Markov Processes and Controlled Markov Chains by Zhenting Hou,Jerzy A. Filar,Anyue Chen Pdf

The general theory of stochastic processes and the more specialized theory of Markov processes evolved enormously in the second half of the last century. In parallel, the theory of controlled Markov chains (or Markov decision processes) was being pioneered by control engineers and operations researchers. Researchers in Markov processes and controlled Markov chains have been, for a long time, aware of the synergies between these two subject areas. However, this may be the first volume dedicated to highlighting these synergies and, almost certainly, it is the first volume that emphasizes the contributions of the vibrant and growing Chinese school of probability. The chapters that appear in this book reflect both the maturity and the vitality of modern day Markov processes and controlled Markov chains. They also will provide an opportunity to trace the connections that have emerged between the work done by members of the Chinese school of probability and the work done by the European, US, Central and South American and Asian scholars.

Author : Harold Kushner,Paul G. Dupuis
Publisher : Springer Science & Business Media
Page : 480 pages
File Size : 55,9 Mb
Release : 2013-11-27
Category : Mathematics
ISBN : 9781461300076

Get Book

Numerical Methods for Stochastic Control Problems in Continuous Time by Harold Kushner,Paul G. Dupuis Pdf

Stochastic control is a very active area of research. This monograph, written by two leading authorities in the field, has been updated to reflect the latest developments. It covers effective numerical methods for stochastic control problems in continuous time on two levels, that of practice and that of mathematical development. It is broadly accessible for graduate students and researchers.

Author : Wendell H. Fleming,Halil Mete Soner
Publisher : Springer Science & Business Media
Page : 436 pages
File Size : 43,5 Mb
Release : 2006-02-04
Category : Mathematics
ISBN : 9780387310718

Get Book

Controlled Markov Processes and Viscosity Solutions by Wendell H. Fleming,Halil Mete Soner Pdf

This book is an introduction to optimal stochastic control for continuous time Markov processes and the theory of viscosity solutions. It covers dynamic programming for deterministic optimal control problems, as well as to the corresponding theory of viscosity solutions. New chapters in this second edition introduce the role of stochastic optimal control in portfolio optimization and in pricing derivatives in incomplete markets and two-controller, zero-sum differential games.

Author : Evgenij B. Dynkin,Aleksandr A. Juškevič
Publisher : Unknown
Page : 289 pages
File Size : 45,7 Mb
Release : 1975
Category : Electronic
ISBN : OCLC:631627380

Get Book

Controlled Markov Processes by Evgenij B. Dynkin,Aleksandr A. Juškevič Pdf

Author : Tomas Prieto-Rumeau
Publisher : World Scientific
Page : 292 pages
File Size : 43,8 Mb
Release : 2012
Category : Mathematics
ISBN : 9781848168497

Get Book

Selected Topics on Continuous-time Controlled Markov Chains and Markov Games by Tomas Prieto-Rumeau Pdf

This book concerns continuous-time controlled Markov chains, also known as continuous-time Markov decision processes. They form a class of stochastic control problems in which a single decision-maker wishes to optimize a given objective function. This book is also concerned with Markov games, where two decision-makers (or players) try to optimize their own objective function. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science, among other areas. An extensive, self-contained, up-to-date analysis of basic optimality criteria (such as discounted and average reward), and advanced optimality criteria (e.g., bias, overtaking, sensitive discount, and Blackwell optimality) is presented. A particular emphasis is made on the application of the results herein: algorithmic and computational issues are discussed, and applications to population models and epidemic processes are shown. This book is addressed to students and researchers in the fields of stochastic control and stochastic games. Moreover, it could be of interest also to undergraduate and beginning graduate students because the reader is not supposed to have a high mathematical background: a working knowledge of calculus, linear algebra, probability, and continuous-time Markov chains should suffice to understand the contents of the book.

Author : E. B. Dunkin,A. A. Jukevi#%c
Publisher : Unknown
Page : 0 pages
File Size : 48,6 Mb
Release : 1979
Category : Electronic
ISBN : OCLC:770919848

Get Book

Controlled Markov processes by E. B. Dunkin,A. A. Jukevi#%c Pdf

Author : Jerzy A. Filar
Publisher : Now Publishers Inc
Page : 95 pages
File Size : 52,9 Mb
Release : 2007
Category : Mathematics
ISBN : 9781601980885

Get Book

Controlled Markov Chains, Graphs and Hamiltonicity by Jerzy A. Filar Pdf

"Controlled Markov Chains, Graphs & Hamiltonicity" summarizes a line of research that maps certain classical problems of discrete mathematics--such as the Hamiltonian cycle and the Traveling Salesman problems--into convex domains where continuum analysis can be carried out. (Mathematics)

Author : Marius Iosifescu
Publisher : Courier Corporation
Page : 305 pages
File Size : 49,7 Mb
Release : 2014-07-01
Category : Mathematics
ISBN : 9780486150581

Get Book

Finite Markov Processes and Their Applications by Marius Iosifescu Pdf

A self-contained treatment of finite Markov chains and processes, this text covers both theory and applications. Author Marius Iosifescu, vice president of the Romanian Academy and director of its Center for Mathematical Statistics, begins with a review of relevant aspects of probability theory and linear algebra. Experienced readers may start with the second chapter, a treatment of fundamental concepts of homogeneous finite Markov chain theory that offers examples of applicable models. The text advances to studies of two basic types of homogeneous finite Markov chains: absorbing and ergodic chains. A complete study of the general properties of homogeneous chains follows. Succeeding chapters examine the fundamental role of homogeneous infinite Markov chains in mathematical modeling employed in the fields of psychology and genetics; the basics of nonhomogeneous finite Markov chain theory; and a study of Markovian dependence in continuous time, which constitutes an elementary introduction to the study of continuous parameter stochastic processes.

Author : Evgeniĭ Borisovich Dynkin,Alexander Adolph Yushkevich
Publisher : Springer
Page : 320 pages
File Size : 42,5 Mb
Release : 1979
Category : Mathematics
ISBN : UOM:39015013837011

Get Book

Controlled Markov Processes by Evgeniĭ Borisovich Dynkin,Alexander Adolph Yushkevich Pdf

This book is devoted to the systematic exposition of the contemporary theory of controlled Markov processes with discrete time parameter or in another termi nology multistage Markovian decision processes. We discuss the applications of this theory to various concrete problems. Particular attention is paid to mathe matical models of economic planning, taking account of stochastic factors. The authors strove to construct the exposition in such a way that a reader interested in the applications can get through the book with a minimal mathe matical apparatus. On the other hand, a mathematician will find, in the appropriate chapters, a rigorous theory of general control models, based on advanced measure theory, analytic set theory, measurable selection theorems, and so forth. We have abstained from the manner of presentation of many mathematical monographs, in which one presents immediately the most general situation and only then discusses simpler special cases and examples. Wishing to separate out difficulties, we introduce new concepts and ideas in the simplest setting, where they already begin to work. Thus, before considering control problems on an infinite time interval, we investigate in detail the case of the finite interval. Here we first study in detail models with finite state and action spaces-a case not requiring a departure from the realm of elementary mathematics, and at the same time illustrating the most important principles of the theory.

Author : Evgeniĭ Borisovich Dynkin,Alexander Adolph Yushkevich
Publisher : Unknown
Page : 320 pages
File Size : 42,9 Mb
Release : 1979
Category : Control systems
ISBN : UCAL:B4407152

Get Book

Controlled Markov Processes by Evgeniĭ Borisovich Dynkin,Alexander Adolph Yushkevich Pdf

Author : Onesimo Hernandez-Lerma,Jean B. Lasserre
Publisher : Springer Science & Business Media
Page : 223 pages
File Size : 52,9 Mb
Release : 2012-12-06
Category : Mathematics
ISBN : 9781461207290

Get Book

Discrete-Time Markov Control Processes by Onesimo Hernandez-Lerma,Jean B. Lasserre Pdf

This book presents the first part of a planned two-volume series devoted to a systematic exposition of some recent developments in the theory of discrete-time Markov control processes (MCPs). Interest is mainly confined to MCPs with Borel state and control (or action) spaces, and possibly unbounded costs and noncompact control constraint sets. MCPs are a class of stochastic control problems, also known as Markov decision processes, controlled Markov processes, or stochastic dynamic pro grams; sometimes, particularly when the state space is a countable set, they are also called Markov decision (or controlled Markov) chains. Regardless of the name used, MCPs appear in many fields, for example, engineering, economics, operations research, statistics, renewable and nonrenewable re source management, (control of) epidemics, etc. However, most of the lit erature (say, at least 90%) is concentrated on MCPs for which (a) the state space is a countable set, and/or (b) the costs-per-stage are bounded, and/or (c) the control constraint sets are compact. But curiously enough, the most widely used control model in engineering and economics--namely the LQ (Linear system/Quadratic cost) model-satisfies none of these conditions. Moreover, when dealing with "partially observable" systems) a standard approach is to transform them into equivalent "completely observable" sys tems in a larger state space (in fact, a space of probability measures), which is uncountable even if the original state process is finite-valued.

Author : Xianping Guo,Onésimo Hernández-Lerma
Publisher : Springer Science & Business Media
Page : 240 pages
File Size : 42,6 Mb
Release : 2009-09-18
Category : Mathematics
ISBN : 9783642025471

Get Book

Continuous-Time Markov Decision Processes by Xianping Guo,Onésimo Hernández-Lerma Pdf

Continuous-time Markov decision processes (MDPs), also known as controlled Markov chains, are used for modeling decision-making problems that arise in operations research (for instance, inventory, manufacturing, and queueing systems), computer science, communications engineering, control of populations (such as fisheries and epidemics), and management science, among many other fields. This volume provides a unified, systematic, self-contained presentation of recent developments on the theory and applications of continuous-time MDPs. The MDPs in this volume include most of the cases that arise in applications, because they allow unbounded transition and reward/cost rates. Much of the material appears for the first time in book form.

Author : Onesimo Hernandez-Lerma
Publisher : Springer Science & Business Media
Page : 160 pages
File Size : 41,7 Mb
Release : 2012-12-06
Category : Mathematics
ISBN : 9781441987143

Get Book

Adaptive Markov Control Processes by Onesimo Hernandez-Lerma Pdf

This book is concerned with a class of discrete-time stochastic control processes known as controlled Markov processes (CMP's), also known as Markov decision processes or Markov dynamic programs. Starting in the mid-1950swith Richard Bellman, many contributions to CMP's have been made, and applications to engineering, statistics and operations research, among other areas, have also been developed. The purpose of this book is to present some recent developments on the theory of adaptive CMP's, i. e. , CMP's that depend on unknown parameters. Thus at each decision time, the controller or decision-maker must estimate the true parameter values, and then adapt the control actions to the estimated values. We do not intend to describe all aspects of stochastic adaptive control; rather, the selection of material reflects our own research interests. The prerequisite for this book is a knowledgeof real analysis and prob ability theory at the level of, say, Ash (1972) or Royden (1968), but no previous knowledge of control or decision processes is required. The pre sentation, on the other hand, is meant to beself-contained,in the sensethat whenever a result from analysisor probability is used, it is usually stated in full and references are supplied for further discussion, if necessary. Several appendices are provided for this purpose. The material is divided into six chapters. Chapter 1 contains the basic definitions about the stochastic control problems we are interested in; a brief description of some applications is also provided.

Author : Oliver Ibe
Publisher : Newnes
Page : 515 pages
File Size : 51,5 Mb
Release : 2013-05-22
Category : Mathematics
ISBN : 9780124078390

Get Book

Markov Processes for Stochastic Modeling by Oliver Ibe Pdf

Markov processes are processes that have limited memory. In particular, their dependence on the past is only through the previous state. They are used to model the behavior of many systems including communications systems, transportation networks, image segmentation and analysis, biological systems and DNA sequence analysis, random atomic motion and diffusion in physics, social mobility, population studies, epidemiology, animal and insect migration, queueing systems, resource management, dams, financial engineering, actuarial science, and decision systems. Covering a wide range of areas of application of Markov processes, this second edition is revised to highlight the most important aspects as well as the most recent trends and applications of Markov processes. The author spent over 16 years in the industry before returning to academia, and he has applied many of the principles covered in this book in multiple research projects. Therefore, this is an applications-oriented book that also includes enough theory to provide a solid ground in the subject for the reader. Presents both the theory and applications of the different aspects of Markov processes Includes numerous solved examples as well as detailed diagrams that make it easier to understand the principle being presented Discusses different applications of hidden Markov models, such as DNA sequence analysis and speech analysis.

Author : G. George Yin,Qing Zhang
Publisher : Springer Science & Business Media
Page : 354 pages
File Size : 52,5 Mb
Release : 2005-10-04
Category : Mathematics
ISBN : 9780387268712

Get Book

Discrete-Time Markov Chains by G. George Yin,Qing Zhang Pdf

This book focuses on two-time-scale Markov chains in discrete time. Our motivation stems from existing and emerging applications in optimization and control of complex systems in manufacturing, wireless communication, and ?nancial engineering. Much of our e?ort in this book is devoted to designing system models arising from various applications, analyzing them via analytic and probabilistic techniques, and developing feasible compu- tionalschemes. Ourmainconcernistoreducetheinherentsystemcompl- ity. Although each of the applications has its own distinct characteristics, all of them are closely related through the modeling of uncertainty due to jump or switching random processes. Oneofthesalientfeaturesofthisbookistheuseofmulti-timescalesin Markovprocessesandtheirapplications. Intuitively,notallpartsorcom- nents of a large-scale system evolve at the same rate. Some of them change rapidly and others vary slowly. The di?erent rates of variations allow us to reduce complexity via decomposition and aggregation. It would be ideal if we could divide a large system into its smallest irreducible subsystems completely separable from one another and treat each subsystem indep- dently. However, this is often infeasible in reality due to various physical constraints and other considerations. Thus, we have to deal with situations in which the systems are only nearly decomposable in the sense that there are weak links among the irreducible subsystems, which dictate the oc- sional regime changes of the system. An e?ective way to treat such near decomposability is time-scale separation. That is, we set up the systems as if there were two time scales, fast vs. slow. xii Preface Followingthetime-scaleseparation,weusesingularperturbationmeth- ology to treat the underlying systems.