Programming Massively Parallel Processors, 3rd Edition

Programming Massively Parallel Processors, 3rd Edition

by David B. Kirk, Wen-mei W. Hwu
Released November 2016
Publisher(s): Morgan Kaufmann
ISBN: 9780128119877

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Programming Massively Parallel Processors: A Hands-on Approach, Third Edition shows both student and professional alike the basic concepts of parallel programming and GPU architecture, exploring, in detail, various techniques for constructing parallel programs.

Case studies demonstrate the development process, detailing computational thinking and ending with effective and efficient parallel programs. Topics of performance, floating-point format, parallel patterns, and dynamic parallelism are covered in-depth.

For this new edition, the authors have updated their coverage of CUDA, including coverage of newer libraries, such as CuDNN, moved content that has become less important to appendices, added two new chapters on parallel patterns, and updated case studies to reflect current industry practices.

  • Teaches computational thinking and problem-solving techniques that facilitate high-performance parallel computing
  • Utilizes CUDA version 7.5, NVIDIA's software development tool created specifically for massively parallel environments
  • Contains new and updated case studies
  • Includes coverage of newer libraries, such as CuDNN for Deep Learning

Table of contents

  1. Cover image
  2. Title page
  3. Table of Contents
  4. Copyright
  5. Dedication
  6. Preface
    1. Target Audience
    2. How to Use the Book
    3. Illinois–NVIDIA GPU Teaching Kit
    4. Online Supplements
  7. Acknowledgements
  8. Chapter 1. Introduction
    1. Abstract
    2. 1.1 Heterogeneous Parallel Computing
    3. 1.2 Architecture of a Modern GPU
    4. 1.3 Why More Speed or Parallelism?
    5. 1.4 Speeding Up Real Applications
    6. 1.5 Challenges in Parallel Programming
    7. 1.6 Parallel Programming Languages and Models
    8. 1.7 Overarching Goals
    9. 1.8 Organization of the Book
    10. References
  9. Chapter 2. Data parallel computing
    1. Abstract
    2. 2.1 Data Parallelism
    3. 2.2 CUDA C Program Structure
    4. 2.3 A Vector Addition Kernel
    5. 2.4 Device Global Memory and Data Transfer
    6. 2.5 Kernel Functions and Threading
    7. 2.6 Kernel Launch
    8. 2.7 Summary
    9. References
  10. Chapter 3. Scalable parallel execution
    1. Abstract
    2. 3.1 CUDA Thread Organization
    3. 3.2 Mapping Threads to Multidimensional Data
    4. 3.3 Image Blur: A More Complex Kernel
    5. 3.4 Synchronization and Transparent Scalability
    6. 3.5 Resource Assignment
    7. 3.6 Querying Device Properties
    8. 3.7 Thread Scheduling and Latency Tolerance
    9. 3.8 Summary
  11. Chapter 4. Memory and data locality
    1. Abstract
    2. 4.1 Importance of Memory Access Efficiency
    3. 4.2 Matrix Multiplication
    4. 4.3 CUDA Memory Types
    5. 4.4 Tiling for Reduced Memory Traffic
    6. 4.5 A Tiled Matrix Multiplication Kernel
    7. 4.6 Boundary Checks
    8. 4.7 Memory as a Limiting Factor to Parallelism
    9. 4.8 Summary
  12. Chapter 5. Performance considerations
    1. Abstract
    2. 5.1 Global Memory Bandwidth
    3. 5.2 More on Memory Parallelism
    4. 5.3 Warps and SIMD Hardware
    5. 5.4 Dynamic Partitioning of Resources
    6. 5.5 Thread Granularity
    7. 5.6 Summary
    8. References
  13. Chapter 6. Numerical considerations
    1. Abstract
    2. 6.1 Floating-Point Data Representation
    3. 6.2 Representable Numbers
    4. 6.3 Special Bit Patterns and Precision in IEEE Format
    5. 6.4 Arithmetic Accuracy and Rounding
    6. 6.5 Algorithm Considerations
    7. 6.6 Linear Solvers and Numerical Stability
    8. 6.7 Summary
    9. References
  14. Chapter 7. Parallel patterns: convolution: An introduction to stencil computation
    1. Abstract
    2. 7.1 Background
    3. 7.2 1D Parallel Convolution—A Basic Algorithm
    4. 7.3 Constant Memory and Caching
    5. 7.4 Tiled 1D Convolution with Halo Cells
    6. 7.5 A Simpler Tiled 1D Convolution—General Caching
    7. 7.6 Tiled 2D Convolution With Halo Cells
    8. 7.7 Summary
    9. 7.8 Exercises
  15. Chapter 8. Parallel patterns: prefix sum: An introduction to work efficiency in parallel algorithms
    1. Abstract
    2. 8.1 Background
    3. 8.2 A Simple Parallel Scan
    4. 8.3 Speed and Work Efficiency
    5. 8.4 A More Work-Efficient Parallel Scan
    6. 8.5 An Even More Work-Efficient Parallel Scan
    7. 8.6 Hierarchical Parallel Scan for Arbitrary-Length Inputs
    8. 8.7 Single-Pass Scan for Memory Access Efficiency
    9. 8.8 Summary
    10. 8.9 Exercises
    11. References
  16. Chapter 9. Parallel patterns—parallel histogram computation: An introduction to atomic operations and privatization
    1. Abstract
    2. 9.1 Background
    3. 9.2 Use of Atomic Operations
    4. 9.3 Block versus Interleaved Partitioning
    5. 9.4 Latency versus Throughput of Atomic Operations
    6. 9.5 Atomic Operation in Cache Memory
    7. 9.6 Privatization
    8. 9.7 Aggregation
    9. 9.8 Summary
    10. Reference
  17. Chapter 10. Parallel patterns: sparse matrix computation: An introduction to data compression and regularization
    1. Abstract
    2. 10.1 Background
    3. 10.2 Parallel SpMV Using CSR
    4. 10.3 Padding and Transposition
    5. 10.4 Using a Hybrid Approach to Regulate Padding
    6. 10.5 Sorting and Partitioning for Regularization
    7. 10.6 Summary
    8. References
  18. Chapter 11. Parallel patterns: merge sort: An introduction to tiling with dynamic input data identification
    1. Abstract
    2. 11.1 Background
    3. 11.2 A Sequential Merge Algorithm
    4. 11.3 A Parallelization Approach
    5. 11.4 Co-Rank Function Implementation
    6. 11.5 A Basic Parallel Merge Kernel
    7. 11.6 A Tiled Merge Kernel
    8. 11.7 A Circular-Buffer Merge Kernel
    9. 11.8 Summary
    10. Reference
  19. Chapter 12. Parallel patterns: graph search
    1. Abstract
    2. 12.1 Background
    3. 12.2 Breadth-First Search
    4. 12.3 A Sequential BFS Function
    5. 12.4 A Parallel BFS Function
    6. 12.5 Optimizations
    7. 12.6 Summary
    8. References
  20. Chapter 13. CUDA dynamic parallelism
    1. Abstract
    2. 13.1 Background
    3. 13.2 Dynamic Parallelism Overview
    4. 13.3 A Simple Example
    5. 13.4 Memory Data Visibility
    6. 13.5 Configurations and Memory Management
    7. 13.6 Synchronization, Streams, and Events
    8. 13.7 A More Complex Example
    9. 13.8 A Recursive Example
    10. 13.9 Summary
    11. References
    12. A13.1 Code Appendix
  21. Chapter 14. Application case study—non-Cartesian magnetic resonance imaging: An introduction to statistical estimation methods
    1. Abstract
    2. 14.1 Background
    3. 14.2 Iterative Reconstruction
    4. 14.3 Computing FHD
    5. 14.4 Final Evaluation
    6. References
  22. Chapter 15. Application case study—molecular visualization and analysis
    1. Abstract
    2. 15.1 Background
    3. 15.2 A Simple Kernel Implementation
    4. 15.3 Thread Granularity Adjustment
    5. 15.4 Memory Coalescing
    6. 15.5 Summary
    7. References
  23. Chapter 16. Application case study—machine learning
    1. Abstract
    2. 16.1 Background
    3. 16.2 Convolutional Neural Networks
    4. 16.3 Convolutional Layer: A Basic CUDA Implementation of Forward Propagation
    5. 16.4 Reduction of Convolutional Layer to Matrix Multiplication
    6. 16.5 cuDNN Library
    7. References
  24. Chapter 17. Parallel programming and computational thinking
    1. Abstract
    2. 17.1 Goals of Parallel Computing
    3. 17.2 Problem Decomposition
    4. 17.3 Algorithm Selection
    5. 17.4 Computational Thinking
    6. 17.5 Single Program, Multiple Data, Shared Memory and Locality
    7. 17.6 Strategies for Computational Thinking
    8. 17.7 A Hypothetical Example: Sodium Map of the Brain
    9. 17.8 Summary
    10. References
  25. Chapter 18. Programming a heterogeneous computing cluster
    1. Abstract
    2. 18.1 Background
    3. 18.2 A Running Example
    4. 18.3 Message Passing Interface Basics
    5. 18.4 Message Passing Interface Point-to-Point Communication
    6. 18.5 Overlapping Computation and Communication
    7. 18.6 Message Passing Interface Collective Communication
    8. 18.7 CUDA-Aware Message Passing Interface
    9. 18.8 Summary
    10. Reference
  26. Chapter 19. Parallel programming with OpenACC
    1. Abstract
    2. 19.1 The OpenACC Execution Model
    3. 19.2 OpenACC Directive Format
    4. 19.3 OpenACC by Example
    5. 19.4 Comparing OpenACC and CUDA
    6. 19.5 Interoperability with CUDA and Libraries
    7. 19.6 The Future of OpenACC
  27. Chapter 20. More on CUDA and graphics processing unit computing
    1. Abstract
    2. 20.1 Model of Host/Device Interaction
    3. 20.2 Kernel Execution Control
    4. 20.3 Memory Bandwidth and Compute Throughput
    5. 20.4 Programming Environment
    6. 20.5 Future Outlook
    7. References
  28. Chapter 21. Conclusion and outlook
    1. Abstract
    2. 21.1 Goals Revisited
    3. 21.2 Future Outlook
  29. Appendix A. An introduction to OpenCL
    1. A.1 Background
    2. A.2 Data Parallelism Model
    3. A.3 Device Architecture
    4. A.4 Kernel Functions
    5. A.5 Device Management and Kernel Launch
    6. A.6 Electrostatic Potential Map in OpenCL
    7. A.7 Summary
  30. Appendix B. THRUST: a productivity-oriented library for CUDA
    1. B.1 Background
    2. B.2 Motivation
    3. B.3 Basic Thrust Features
    4. B.4 Generic Programming
    5. B.5 Benefits of Abstraction
    6. B.6 Best Practices
  31. Appendix C. CUDA Fortran
    1. C.1 CUDA Fortran and CUDA C Differences
    2. C.2 A First CUDA Fortran Program
    3. C.3 Multidimensional Array in CUDA Fortran
    4. C.4 Overloading Host/Device Routines with Generic Interfaces
    5. C.5 Calling CUDA C via ISO_C_Binding
    6. C.6 Kernel Loop Directives and Reduction Operations
    7. C.7 Dynamic Shared Memory
    8. C.8 Asynchronous Data Transfers
    9. C.9 Compilation and Profiling
    10. C.10 Calling Thrust from CUDA Fortran
  32. Appendix D. An introduction to C++ AMP
    1. D.1 Core C++ AMP Features
    2. D.2 Details of the C++ AMP Execution Model
    3. D.3 Managing Accelerators
    4. D.4 Tiled Execution
    5. D.5 C++ AMP Graphics Features
    6. D.6 Summary
    7. Reference
  33. Index

Product information

  • Title: Programming Massively Parallel Processors, 3rd Edition
  • Author(s): David B. Kirk, Wen-mei W. Hwu
  • Release date: November 2016
  • Publisher(s): Morgan Kaufmann
  • ISBN: 9780128119877