ROSE Tutorial - ROSE compiler infrastructure

Loading...

ROSE Tutorial: A Tool for Building Source-to-Source Translators Draft Tutorial (version 0.9.10.151) Daniel Quinlan, Markus Schordan, Richard Vuduc, Qing Yi Thomas Panas, Chunhua Liao, and Jeremiah J. Willcock Lawrence Livermore National Laboratory Livermore, CA 94550 925-423-2668 (office) 925-422-6278 (fax) {dquinlan,panas2,liao6}@llnl.gov [email protected] [email protected] [email protected] [email protected] Project Web Page: www.rosecompiler.org UCRL Number for ROSE User Manual: UCRL-SM-210137-DRAFT UCRL Number for ROSE Tutorial: UCRL-SM-210032-DRAFT UCRL Number for ROSE Source Code: UCRL-CODE-155962 ROSE User Manual (pdf) ROSE Tutorial (pdf) ROSE HTML Reference (html only) February 4, 2019

ii February 4, 2019

Contents 1 Introduction 1.1 What is ROSE . . . . . . . . . . . . . . . . . 1.2 Why you should be interested in ROSE . . . 1.3 Problems that ROSE can address . . . . . . . 1.4 Examples in this ROSE Tutorial . . . . . . . 1.5 ROSE Documentation and Where To Find It 1.6 Using the Tutorial . . . . . . . . . . . . . . . 1.7 Required Makefile for Tutorial Examples . . .

I

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

Working with the ROSE AST

1 1 2 2 3 10 11 11

13

2 Identity Translator

15

3 Simple AST Graph Generator

19

4 AST Whole Graph Generator

23

5 Advanced AST Graph Generation

29

6 AST PDF Generator

31

7 Introduction to AST Traversals 7.1 Input For Example Traversals . . . . . . . . . . . . . . . . . 7.2 Traversals of the AST Structure . . . . . . . . . . . . . . . . 7.2.1 Classic Object-Oriented Visitor Pattern for the AST 7.2.2 Simple Traversal (no attributes) . . . . . . . . . . . 7.2.3 Simple Pre- and Postorder Traversal . . . . . . . . . 7.2.4 Inherited Attributes . . . . . . . . . . . . . . . . . . 7.2.5 Synthesized Attributes . . . . . . . . . . . . . . . . . 7.2.6 Accumulator Attributes . . . . . . . . . . . . . . . . 7.2.7 Inherited and Synthesized Attributes . . . . . . . . . 7.2.8 Persistent Attributes . . . . . . . . . . . . . . . . . . 7.2.9 Nested Traversals . . . . . . . . . . . . . . . . . . . . 7.2.10 Combining all Attributes and Using Primitive Types iii

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

35 35 36 37 37 37 38 45 48 49 52 55 57

iv

CONTENTS . . . . . .

58 64 67 67 69 71

8 Graph Processing Tutorial 8.1 Traversal Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75 75

9 Scopes of Declarations 9.1 Input For Examples Showing Scope Information . . . . . . . . . . . . . . . . . . 9.2 Generating the code representing any IR node . . . . . . . . . . . . . . . . . . . .

79 79 80

10 AST Query 10.1 Simple Queries on the AST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Nested Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83 83 83

11 AST File I/O 11.1 Source Code for File I/O . . . . . . . . . . . 11.2 Input to Demonstrate File I/O . . . . . . . 11.3 Output from File I/O . . . . . . . . . . . . 11.4 Final Code After Passing Through File I/O

. . . .

89 89 89 89 89

12 Debugging Techniques 12.1 Input For Examples Showing Debugging Techniques . . . . . . . . . . . . . . . . 12.2 Generating the code from any IR node . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Displaying the source code position of any IR node . . . . . . . . . . . . . . . . .

93 93 94 94

II

97

7.3

7.2.11 Combined Traversals . . . . . . . . . . . . . . . . . . . . . 7.2.12 Short-Circuiting Traversals . . . . . . . . . . . . . . . . . Memory Pool Traversals . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 ROSE Memory Pool Visit Traversal . . . . . . . . . . . . 7.3.2 Classic Object-Oriented Visitor Pattern for Memory Pool 7.3.3 ROSE IR Type Traversal (uses Memory Pools) . . . . . .

Complex Types

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

. . . . . .

. . . .

13 Type and Declaration Modifiers 99 13.1 Input For Example Showing use of Volatile type modifier . . . . . . . . . . . . . 99 13.2 Generating the code representing the seeded bug . . . . . . . . . . . . . . . . . . 100 14 Function Parameter Types

103

15 Resolving Overloaded Functions

107

16 Template Parameter Extraction

111

17 Template Support 115 17.1 Example Template Code #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 17.2 Example Template Code #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

CONTENTS

III

v

Program Analyses

117

18 Generic Dataflow Analysis Framework 18.1 Basics of DataFlowAnalysis . . . . . . . . . . . . . 18.2 ROSE Dataflow Framework . . . . . . . . . . . . . 18.2.1 Call and Control-Flow Graphs . . . . . . . 18.2.2 Analyses . . . . . . . . . . . . . . . . . . . . 18.2.3 Dataflow . . . . . . . . . . . . . . . . . . . 18.2.4 Transferring Information Between Analyses 18.2.5 CFG Transformations . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

19 Recognizing Loops

119 119 121 121 121 123 132 134 137

20 Virtual CFG 20.1 CFGNode Index values . . . . . . . . . . . . 20.2 Important functions . . . . . . . . . . . . . 20.2.1 Node methods . . . . . . . . . . . . 20.2.2 Edge methods . . . . . . . . . . . . . 20.3 Drawing a graph of the CFG . . . . . . . . 20.4 Robustness to AST changes . . . . . . . . . 20.5 Limitations . . . . . . . . . . . . . . . . . . 20.5.1 Fortran support . . . . . . . . . . . 20.5.2 Exception handling . . . . . . . . . . 20.5.3 Interprocedural control flow analysis 20.6 Node filtering . . . . . . . . . . . . . . . . . 20.6.1 “Interesting” node filter . . . . . . . 20.6.2 Arbitrary filtering . . . . . . . . . . 20.7 Static CFG . . . . . . . . . . . . . . . . . . 20.7.1 Class methods . . . . . . . . . . . . 20.7.2 Drawing a graph of the CFG . . . . 20.8 Static, Interprocedural CFGs . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

21 Generating Control Flow Graphs

141 141 142 142 142 143 150 150 150 150 150 150 150 151 151 151 152 152 155

22 Graph Processing Tutorial 159 22.1 Traversal Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 23 Dataflow Analysis 23.1 Def-Use Analysis . . . . . . . . . . . . . 23.1.1 Def-use Example implementation 23.1.2 Accessing the Def-Use Results . 23.2 Liveness Analysis . . . . . . . . . . . . . 23.2.1 Access live variables . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

163 163 163 164 167 167

24 Generating the Call Graph (CG)

173

25 Dataflow Analysis based Virtual Function Analysis

177

vi

CONTENTS

26 Generating the Class Hierarchy Graph

181

27 Database Support 185 27.1 ROSE DB Support for Persistent Analysis . . . . . . . . . . . . . . . . . . . . . . 185 27.2 Call Graph for Multi-file Application . . . . . . . . . . . . . . . . . . . . . . . . . 185 27.3 Class Hierarchy Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 28 Building Custom Graphs

191

IV

193

Program Transformations and Optimizations

29 Generating Unique Names for Declarations 29.1 Example Code Showing Generation of Unique Names . . . . . . . . . 29.2 Input For Examples Showing Unique Name Generation for Variables 29.3 Example Output Showing Unique Variable Names . . . . . . . . . . 29.4 Input For Examples Showing Unique Name Generation for Functions 29.5 Example Output Showing Unique Function Names . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

195 196 196 197 197 197

30 Command-line Processing Within Translators 203 30.1 Commandline Selection of Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 31 Tailoring The Code Generation Format 31.1 Source Code for Example that Tailors the Code Generation . . . . . . . . . . . . 31.2 Input to Demonstrate Tailoring the Code Generation . . . . . . . . . . . . . . . . 31.3 Final Code After Tailoring the Code Generation . . . . . . . . . . . . . . . . . .

207 207 207 207

32 AST Construction 32.1 Variable Declarations . . . . . . . . . . . 32.2 Expressions . . . . . . . . . . . . . . . . 32.3 Assignment Statements . . . . . . . . . 32.4 Functions . . . . . . . . . . . . . . . . . 32.5 Function Calls . . . . . . . . . . . . . . 32.6 Creating a ’struct’ for Global Variables .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

211 211 215 217 219 224 224

33 Parser Building Blocks 33.1 Grammar Examples . . . . . . . . . . . . . . . 33.2 AstAttribute to Store results . . . . . . . . . . 33.3 The AstFromString Namespace . . . . . . . . . 33.4 Write your parsers using parser building blocks 33.5 Limitations . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

235 236 237 238 238 240

. . . . . .

. . . . . .

. . . . . .

34 Handling Comments, Preprocessor Directives, And Adding Arbitrary Text to Generated Code 241 34.1 How to Access Comments and Preprocessor Directives . . . . . . . . . . . . . . . 241 34.1.1 Source Code Showing How to Access Comments and Preprocessor Directives242 34.1.2 Input to example showing how to access comments and CPP directives . 242

CONTENTS

vii

34.1.3 Comments and CPP Directives collected from source file (skipping headers)242 34.1.4 Comments and CPP Directives collected from source file and all header files242 34.2 Collecting #define C Preprocessor Directives . . . . . . . . . . . . . . . . . . . . 242 34.2.1 Source Code Showing How to Collect #define Directives . . . . . . . . . . 242 34.2.2 Input to example showing how to access comments and CPP directives . 243 34.2.3 Comments and CPP Directives collected from source file and all header files243 34.3 Automated Generation of Comments . . . . . . . . . . . . . . . . . . . . . . . . . 243 34.3.1 Source Code Showing Automated Comment Generation . . . . . . . . . . 243 34.3.2 Input to Automated Addition of Comments . . . . . . . . . . . . . . . . . 243 34.3.3 Final Code After Automatically Adding Comments . . . . . . . . . . . . . 243 34.4 Addition of Arbitrary Text to Unparsed Code Generation . . . . . . . . . . . . . 244 34.4.1 Source Code Showing Automated Arbitrary Text Generation . . . . . . . 244 34.4.2 Input to Automated Addition of Arbitrary Text . . . . . . . . . . . . . . 244 34.4.3 Final Code After Automatically Adding Arbitrary Text . . . . . . . . . . 244 35 Partial Redundancy Elimination (PRE) 255 35.1 Source Code for example using PRE . . . . . . . . . . . . . . . . . . . . . . . . . 255 35.2 Input to Example Demonstrating PRE . . . . . . . . . . . . . . . . . . . . . . . . 256 35.3 Final Code After PRE Transformation . . . . . . . . . . . . . . . . . . . . . . . . 257 36 Calling the Inliner 259 36.1 Source Code for Inliner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 36.2 Input to Demonstrate Function Inlining . . . . . . . . . . . . . . . . . . . . . . . 259 36.3 Final Code After Function Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . 259 37 Using the AST Outliner 37.1 An Outlining Example . . . . . . . . . . . . . . . . . . . . 37.2 Limitations of the Outliner . . . . . . . . . . . . . . . . . 37.3 User-Directed Outlining via Pragmas . . . . . . . . . . . . 37.4 Outlining via Abstract Handles . . . . . . . . . . . . . . . 37.5 Calling Outliner Directly on AST Nodes . . . . . . . . . . 37.5.1 Selecting the outlineable if statements . . . . . . . 37.5.2 Properly ordering statements for in-place outlining 37.6 Outliner’s Preprocessing Phase . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

263 263 264 266 266 267 267 267 268

38 Loop Optimization 38.1 Example Loop Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . 38.2 Matrix Multiply Example . . . . . . . . . . . . . . . . . . . . . . . . . 38.3 Loop Fusion Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.4 Example Loop Processor (LoopProcessor.C) . . . . . . . . . . . . . . . 38.5 Matrix Multiplication Example (mm.C) . . . . . . . . . . . . . . . . . 38.6 Matrix Multiplication Example Using Linearized Matrices (dgemm.C) 38.7 LU Factorization Example (lufac.C) . . . . . . . . . . . . . . . . . . . 38.8 Loop Fusion Example (tridvpk.C) . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

281 281 284 286 286 289 291 293 295

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

viii

CONTENTS

39 Parameterized Code Translation 297 39.1 Loop Unrolling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 39.2 Loop Interchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 39.3 Loop Tiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302

V

Correctness Checking

303

40 Code Coverage

305

41 Bug Seeding 313 41.1 Input For Examples Showing Bug Seeding . . . . . . . . . . . . . . . . . . . . . . 313 41.2 Generating the code representing the seeded bug . . . . . . . . . . . . . . . . . . 314

VI

Binary Support

317

42 Instruction Semantics

319

43 Binary Analysis 43.1 DataFlow Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.1.1 Def-Use Analysis . . . . . . . . . . . . . . . . . . . . . . . . 43.1.2 Variable Analysis . . . . . . . . . . . . . . . . . . . . . . . . 43.2 Dynamic Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.3 Analysis and Transformations on Binaries . . . . . . . . . . . . . . 43.3.1 Source-to-source transformations to introduce NOPs . . . . 43.3.2 Detection of NOP sequences in the binary AST . . . . . . . 43.3.3 Transformations on the NOP sequences in the binary AST 43.3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

321 321 321 321 321 322 323 323 325 325

44 Binary Construction 44.1 Constructors . . . . . . . . . . . . . . . . . 44.2 Read-Only Data Members . . . . . . . . . . 44.3 Constructing the Executable File Container 44.4 Constructing the ELF File Header . . . . . 44.5 Constructing the ELF Segment Table . . . 44.6 Constructing the .text Section . . . . . . . . 44.7 Constructing a LOAD Segment . . . . . . . 44.8 Constructing a PAX Segment . . . . . . . . 44.9 Constructing a String Table . . . . . . . . . 44.10Constructing an ELF Section Table . . . . . 44.11Allocating Space . . . . . . . . . . . . . . . 44.12Produce a Debugging Dump . . . . . . . . . 44.13Produce the Executable File . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

331 331 331 332 332 333 333 335 336 336 336 337 337 337

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

CONTENTS

ix

45 Dwarf Debug Support 339 45.1 ROSE AST of Dwarf IR nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 45.2 Source Position to Instruction Address Mapping . . . . . . . . . . . . . . . . . . 340

VII

Interacting with Other Tools

46 Abstract Handles to Language Constructs 46.1 Use Case . . . . . . . . . . . . . . . . . . . 46.2 Syntax . . . . . . . . . . . . . . . . . . . . . 46.3 Examples . . . . . . . . . . . . . . . . . . . 46.4 Reference Implementation . . . . . . . . . . 46.4.1 Connecting to ROSE . . . . . . . . . 46.4.2 Connecting to External Tools . . . . 46.5 Summary . . . . . . . . . . . . . . . . . . .

345 . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

347 348 348 349 350 350 356 359

47 ROSE-HPCToolKit Interface 47.1 An HPCToolkit Example Run . . . . . . . . . . 47.2 Attaching HPCToolkit Data to the ROSE AST 47.2.1 Calling ROSE-HPCT . . . . . . . . . . 47.2.2 Retrieving the attribute values . . . . . 47.2.3 Metric propagation . . . . . . . . . . . . 47.3 Working with GNU gprof . . . . . . . . . . . . 47.4 Command-line options . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

361 361 368 368 368 368 369 370

. . . . . . .

48 TAU Instrumentation 375 48.1 Input For Examples Showing Information using Tau . . . . . . . . . . . . . . . . 375 48.2 Generating the code representing any IR node . . . . . . . . . . . . . . . . . . . . 375 49 The Haskell Interface 379 49.1 Traversals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 49.2 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380

VIII

Parallelism

383

50 Shared-Memory Parallel Traversals

385

51 Distributed-Memory Parallel Traversals

389

52 Parallel Checker 393 52.1 Different Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 52.2 Running through PSUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 53 Reduction Recognition

395

x

IX

CONTENTS

Tutorial Summary

54 Tutorial Wrap-up

397 399

Appendix 401 54.1 Location of To Do List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 54.2 Abstract Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 Glossary

409

List of Figures 1.1

2.1 2.2 2.3

Example Makefile showing how to use an installed version of ROSE (generated by make install). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Source code for translator to read an input program and generate an object code (with no translation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to identity translator. . . . . . . . . . . . . . Generated code, from ROSE identity translator, sent to the backend (vendor) compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

16 16 17

3.1 3.2 3.3

Example source code to read an input program and generate an AST graph. . . . Example source code used as input to generate the AST graph. . . . . . . . . . . AST representing the source code file: inputCode ASTGraphGenerator.C. . . . .

20 20 21

4.1 4.2

Example source code to read an input program and generate a whole AST graph. Example tiny source code used as input to generate the small AST graph with attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AST representing the tiny source code file: inputCode wholeAST 1.C. This graphs shows types, symbols, and other attributes that are defined on the attributed AST. Example source code used as input to generate a larger AST graph with attributes. AST representing the small source code file: inputCode wholeAST 2.C. This graph shows the significantly greater internal complexity of a slightly larger input source code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

4.3 4.4 4.5

6.1 6.2 6.3

7.1 7.2 7.3 7.4

Example source code to read an input program and generate a PDF file to represent the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to generate the PDF file of the AST. . . . . . Example output from translator which outputs PDF representation of AST. The generated PDF file makes use of the bookmark mechanism to expand and collapse parts of the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to program in traversals shown in this chapter. Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . Output of input file to the visitor pattern traversal over the memory pools. . . . Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . xi

24 25 26

27

31 32

33 36 38 40 41

xii

LIST OF FIGURES 7.5 7.6 7.7 7.8

41 42 42

7.34 7.35

Output of input file to the visitor traversal. . . . . . . . . . . . . . . . . . . . . . Example source showing simple pre- and postorder pattern. . . . . . . . . . . . . Output of input file to the pre- and postorder traversal. . . . . . . . . . . . . . . Example source code showing use of inherited attributes (passing context information down the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the inherited attribute traversal. . . . . . . . . . . . . . . Example source code showing use of synthesized attributed (passing analysis information up the AST). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the synthesized attribute traversal. . . . . . . . . . . . . . Example source code showing use of accumulator attributes (typically to count things in the AST). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the accumulator attribute traversal. . . . . . . . . . . . . Example source code showing use of both inherited and synthesized attributes working together (part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the inherited and synthesized attribute traversal. . . . . . Example source code showing use of persistent attributes used to pass information across multiple passes over the AST. . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the persistent attribute traversal showing the passing of information from one AST traversal to a second AST traversal. . . . . . . . . . . Example source code showing use nested traversals. . . . . . . . . . . . . . . . . . Output of input file to the nested traversal example. . . . . . . . . . . . . . . . . Input code with nested loops for nesting info processing . . . . . . . . . . . . . . Example source code showing use of inherited, synthesized, accumulator, and persistent attributes (part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code showing use of inherited, synthesized, accumulator, and persistent attributes (part 2). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output code showing the result of using inherited, synthesized, and accumulator attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source showing the combination of traversals. . . . . . . . . . . . . . . . Output of input file to the combined traversals. Note that the order of outputs changes as execution of several analyzers is interleaved. . . . . . . . . . . . . . . Input code with used to demonstrate the traversal short-circuit mechanism. . . . Example source code showing use of short-circuit mechanism to avoid traversal of full AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output code showing the result of short-circuiting the traversal. . . . . . . . . . Example source showing simple visit traversal over the memory pools. . . . . . . Output of input file to the visitor traversal over the memory pool. . . . . . . . . Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . Output of input file to the visitor pattern traversal over the memory pools. . . . Example source showing simple visit traversal over each type of IR node (one only) in the memory pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input file to the IR Type traversal over the memory pool. . . . . . . . Example of output using -rose:verbose 2 (memory use report for AST). . . . . . .

8.1 8.2

Source CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . Binary CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . .

76 77

7.9 7.10 7.11 7.12 7.13 7.14 7.15 7.16 7.17 7.18 7.19 7.20 7.21 7.22 7.23 7.24 7.25 7.26 7.27 7.28 7.29 7.30 7.31 7.32 7.33

43 44 46 47 48 49 50 51 53 54 55 56 57 59 60 61 62 63 64 65 66 68 68 69 70 72 73 73

LIST OF FIGURES 9.1 9.2 9.3

xiii

Example source code used as input to program in codes used in this chapter. . . Example source code showing how to get scope information for each IR node. . Output of input code using scopeInformation.C . . . . . . . . . . . . . . . . . . .

80 81 82

10.1 Example source code for translator to read an input program and generate a list of functions in the AST (queryLibraryExample.C). . . . . . . . . . . . . . . . . . 10.2 Example source code used as input to program in figure 10.1 (queryLibraryExample.C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Output of input file to the AST query processor (queryLibraryExample.C). . . . 10.4 Example source code for translator to read an input program and generate a list of access functions in the AST (nestedQueryExample.C). . . . . . . . . . . . . . 10.5 Example source code used as input to program in figure 10.4 (nestedQueryExample.C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Output of input file to the AST query processor (nestedQueryExample.C). . . .

88 88

11.1 11.2 11.3 11.4

90 91 91 92

Example source code showing how to use the AST file I/O support. . . . . . Example source code used as input to demonstrate the AST file I/O support. Output of input code after inlining transformations. . . . . . . . . . . . . . . Output of input code after file I/O. . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

. . . .

12.1 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node. . . . 12.3 Output of input code using debuggingIRnodeToString.C . . . . . . . . . . . . . . 12.4 Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node. . . . 12.5 Output of input code using debuggingSourceCodePositionInformation.C . . . . .

84 85 86 87

93 95 96 96 96

13.1 Example source code used as input to program in codes used in this chapter. . . 99 13.2 Example source code showing how to detect volatile modifier. . . . . . . . . . . 100 13.3 Output of input code using volatileTypeModifier.C . . . . . . . . . . . . . . . . . 101 14.1 Example source code showing how to get type information from function parameters.104 14.2 Example source code used as input to typeInfoFromFunctionParameters.C. . . . 105 14.3 Output of input to typeInfoFromFunctionParameters.C. . . . . . . . . . . . . . . 106 15.1 Example source code showing mapping of function calls to overloaded function declarations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 15.2 Example source code used as input to resolveOverloadedFunction.C. . . . . . . . 109 15.3 Output of input to resolveOverloadedFunction.C. . . . . . . . . . . . . . . . . . . 109 16.1 Example source code used to extract template parameter information. . . . . . . 112 16.2 Example source code used as input to templateParameter.C. . . . . . . . . . . . 113 16.3 Output of input to templateParameter.C. . . . . . . . . . . . . . . . . . . . . . . 113 17.1 Example source code showing use of a C++ template. . . . . . . . . . . . . . . . 115

xiv

LIST OF FIGURES 17.2 Example source code after processing using identityTranslator (shown in figure 2.1).116 17.3 Example source code showing use of a C++ template. . . . . . . . . . . . . . . . 116 17.4 Example source code after processing using identityTranslator (shown in figure 2.1).116 18.1 18.2 18.3 18.4 18.5 18.6 18.7

Example of a constant propagation analysis. . . . . . . . Example of a dataflow analysis with abstraction of affine Example of simple analyses . . . . . . . . . . . . . . . . Each variable’s lattice for constant-propagation analysis Example of Transformation on the CFG . . . . . . . . . Example of the Transformation on the Source Code . . Code Replacement Transformation . . . . . . . . . . . .

. . . . . . . constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

120 121 122 123 134 135 136

19.1 19.2 19.3 19.4

Example source code showing loop recognition (part 1). . . . . . Example source code showing loop recognition (part 2). . . . . . Example source code used as input to loop recognition processor. Output of input to loop recognition processor. . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

138 139 139 140

20.1 Example source code showing visualization of virtual control flow graph. . . . . . 20.2 Example source code used as input to build virtual control graphs. . . . . . . . . 20.3 The debug virtual control flow graph for function main() shows all virtual CFG nodes and edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4 The virtual control flow graph for function main() shows only interesting virtual CFG nodes and edges. Each CFGNode’s caption tells associated source line number and CFGNode index value (@line-num:index-value) . . . . . . . . . . . . 20.5 The debug virtual control flow graph for function testIf() shows all virtual CFG nodes and edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.6 The virtual control flow graph for function testIf() shows only interesting virtual CFG nodes and edges. Each CFGNode’s caption tells associated source line number and CFGNode index value (@line-num:index-value) . . . . . . . . . . . . 20.7 Example source code showing visualization of static control flow graph. . . . . .

144 145

. . . .

. . . .

146

147 148

149 152

21.1 Example source code showing visualization of control flow graph. . . . . . . . . . 156 21.2 Example source code used as input to build control flow graph. . . . . . . . . . . 157 21.3 Control flow graph for function in input code file: inputCode 1.C. . . . . . . . . 157 22.1 Source CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 22.2 Binary CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 23.1 23.2 23.3 23.4 23.5 23.6 23.7 23.8

Example input code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code using def use analysis . . . . . . . . . . . . . . . . . Output of the program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Def-Use graph for example program. . . . . . . . . . . . . . . . . . . . . . Example source code using liveness analysis . . . . . . . . . . . . . . . . . Example input code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control flow graph annotated with live variables for example program. . . Example code retrieving live variables based on virtual control flow graph

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

163 164 165 166 168 169 170 171

LIST OF FIGURES

xv

24.1 Example source code showing visualization of call graph. . . . . . . . . . . . . . . 174 24.2 Example source code used as input to build call graph. . . . . . . . . . . . . . . . 175 24.3 Call graph for function in input code file: inputCode BuildCG.C. . . . . . . . . . 176 25.1 25.2 25.3 25.4

Source code to perform virtual function analysis . . . . . . . . . . . . . . . . . . Example source code used as input for Virtual Function Analysis. . . . . . . . . . Call graph generated by Call Graph Analysis for input code in inputCode vfa.C. Call graph resulted from Virtual Function Analysis for input code in inputCode vfa.C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

178 179 179 180

26.1 Example source code showing visualization of class hierarchy graph. . . . . . . . 181 26.2 Example source code used as input to build class hierarchy graph. . . . . . . . . 182 26.3 Class hierarchy graph in input code file: inputCode ClassHierarchyGraph.C. . . . 183 27.1 27.2 27.3 27.4

Example translator (part 1) using database connection to store function names. . Example translator (part 2) using database connection to store function names. . Example source code used as input to database example. . . . . . . . . . . . . . . Output from processing input code through database example dataBaseTranslator27.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29.1 Example source code showing the output of mangled name. The string represents the code associated with the subtree of the target IR node. . . . . . . . . . . . . 29.2 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29.3 Output of input code using generatingUniqueNamesFromDeclaration.C . . . . . 29.4 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29.5 Output of input code using generatingUniqueNamesFromDeclaration.C . . . . . 30.1 Example source code showing simple command-line processing within ROSE translator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.2 Output of input code using commandlineProcessing.C . . . . . . . . . . . . . . . 30.3 Example source code showing simple command-line processing within ROSE translator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.4 Output of input code using commandlineProcessing.C . . . . . . . . . . . . . . .

186 187 188 189

198 199 200 201 202

204 204 205 205

31.1 Example source code showing how to tailor the code generation format. . . . . . 208 31.2 Example source code used as input to program to the tailor the code generation. 209 31.3 Output of input code after changing the format of the generated code. . . . . . . 210 32.1 AST construction and insertion for a variable using the high level interfaces . . . 32.2 Example source code to read an input program and add a new variable declaration at the top of each block. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Example source code used as input to the translators adding new variable. . . . . 32.4 Output of input to the translators adding new variable. . . . . . . . . . . . . . . 32.5 Example translator to add expressions . . . . . . . . . . . . . . . . . . . . . . . . 32.6 Example source code used as input . . . . . . . . . . . . . . . . . . . . . . . . . .

212 213 214 214 215 216

xvi

LIST OF FIGURES 32.7 Output of the input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.8 Example source code to add an assignment statement . . . . . . . . . . . . . . . 32.9 Example source code used as input . . . . . . . . . . . . . . . . . . . . . . . . . . 32.10Output of the input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.11Addition of function to global scope using high level interfaces . . . . . . . . . . 32.12Addition of function to global scope using high level interfaces and a scope stack 32.13Example source code shows addition of function to global scope (part 1). . . . . 32.14Example source code shows addition of function to global scope (part 2). . . . . 32.15Example source code used as input to translator adding new function. . . . . . . 32.16Output of input to translator adding new function. . . . . . . . . . . . . . . . . . 32.17Example source code to instrument any input program. . . . . . . . . . . . . . . 32.18Example source code using the high level interfaces . . . . . . . . . . . . . . . . . 32.19Example source code instrumenting end of functions . . . . . . . . . . . . . . . . 32.20Example input code of the instrumenting translator for end of functions. . . . . . 32.21Output of instrumenting translator for end of functions. . . . . . . . . . . . . . . 32.22Example source code shows repackaging of global variables to a struct (part 1). . 32.23Example source code shows repackaging of global variables to a struct (part 2). . 32.24Example source code shows repackaging of global variables to a struct (part 3). . 32.25Example source code shows repackaging of global variables to a struct (part 4). . 32.26Example source code shows repackaging of global variables to a struct (part 5). . 32.27Example source code used as input to translator adding new function. . . . . . . 32.28Output of input to translator adding new function. . . . . . . . . . . . . . . . . .

216 217 217 218 219 220 221 222 223 223 225 226 227 228 228 229 230 231 232 233 234 234

34.1 Example source code showing how to access comments. . . . . . . . . . . . . . . 34.2 Example source code used as input to collection of comments and CPP directives. 34.3 Output from collection of comments and CPP directives on the input source file only. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.4 Output from collection of comments and CPP directives on the input source file and all header files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.5 Example source code showing how to access comments. . . . . . . . . . . . . . . 34.6 Example source code used as input to collection of comments and CPP directives. 34.7 Output from collection of comments and CPP directives on the input source file and all header files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.8 Example source code showing how automate comments. . . . . . . . . . . . . . . 34.9 Example source code used as input to automate generation of comments. . . . . 34.10Output of input code after automating generation of comments. . . . . . . . . . . 34.11Example source code showing how automate the introduction of arbitrary text. 34.12Example source code used as input to automate generation of arbitrary text. . . 34.13Output of input code after automating generation of arbitrary text. . . . . . . .

245 246 246 247 248 249 250 251 252 252 253 253 254

35.1 Example source code showing how use Partial Redundancy Elimination (PRE). 255 35.2 Example source code used as input to program to the Partial Redundancy Elimination (PRE) transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 35.3 Output of input code after Partial Redundancy Elimination (PRE) transformation.258 36.1 Example source code showing how to instrument using Tau.

. . . . . . . . . . . 260

LIST OF FIGURES

xvii

36.2 Example source code used as input to program to the inlining transformation. . . 261 36.3 Output of input code after inlining transformations. . . . . . . . . . . . . . . . . 262 37.1 inputCode OutlineLoop.cc: Sample input program. The #pragma directive marks the nested for loop for outlining. . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 37.2 rose outlined-inputCode OutlineLoop.cc: The nested for loop of Figure 37.1 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 37.3 outline.cc: A basic outlining translator, which generates Figure 37.2 from Figure 37.1. This outliner relies on the high-level driver, Outliner::outlineAll(), which scans the AST for outlining pragma directives (#pragma rose outline) that mark outline targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 37.4 inputCode OutlineLoop2.c: Sample input program without pragmas. . . . . . . . 270 37.5 rose inputCode OutlineLoop2.c: The loop at line 12 of Figure 37.12 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 37.6 rose inputCode OutlineLoop2b.c: The 2nd loop within a function named initializefrom Figure 37.12 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . 272 37.7 outlineIfs.cc: A lower-level outlining translator, which calls Outliner::outline() directly on SgStatement nodes. This particular translator outlines all SgIfStmt nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 37.8 inputCode Ifs.cc: Sample input program, without explicit outline targets specified using #pragma rose outline, as in Figures 37.1 and 37.12. . . . . . . . . . . . . 274 37.9 rose inputCode Ifs.cc: Figure 37.8, after outlining using the translator in Figure 37.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 37.10outlinePreproc.cc: The basic translator of Figure 37.3, modified to execute the Outliner’s preprocessing phase only. In particular, the original call to Outliner::outlineAll() has been replaced by a call to Outliner::preprocessAll(). . . . . . . . . . . . 276 37.11rose outlined pp-inputCode OutlineLoop.cc: Figure 37.1 after outline preprocessing only, i.e., specifying -rose:outline:preproc-only as an option to the translator of Figure 37.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 37.12inputCode OutlineNonLocalJumps.cc: Sample input program, with an outlining target that contains two non-local jumps (here, break statements). . . . . . . . . 278 37.13rose outlined pp-inputCode OutlineNonLocalJumps.cc: The non-local jump example of Figure 37.12 after outliner preprocessing, but before the actual outlining. The non-local jump is handled by an additional flag, EXIT TAKEN , which indicates what non-local jump is to be taken. . . . . . . . . . . . . . . . . . . . . 279 37.14rose outlined-inputCode OutlineNonLocalJumps.cc: Figure 37.12 after outlining. 280 38.1 Example source code showing use of loop optimization mechanisms. . . . . . . . 38.2 Example source code used as input to loop optimization processor. . . . . . . . . 38.3 Output of loop optimization processor showing matrix multiply optimization (using options: -bk1 -fs0). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.4 Example source code used as input to loop optimization processor. . . . . . . . . 38.5 Output of loop optimization processor showing loop fusion (using options: -fs2). 38.6 Detailed example source code showing use of loop optimization mechanisms (loopProcessor.C part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.7 loopProcessor.C source code (Part 2). . . . . . . . . . . . . . . . . . . . . . . . .

283 284 285 286 286 287 288

xviii

LIST OF FIGURES

38.8 Example source code used as input to loopProcessor, show in figure 38.6. . . . . 38.9 Output of loopProcessor using input from figure 38.8 (using options: -bk1 -fs0). 38.10Example source code used as input to loopProcessor, show in figure 38.6. . . . . 38.11Output of loopProcessor using input from figure 38.10 (using options: -bk1 -unroll nvar 16). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.12Example source code used as input to loopProcessor, show in figure 38.6. . . . . 38.13Output of loopProcessor using input from figure 38.12 (using options: -bk1 -fs0 -splitloop -annotation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.14Example source code used as input to loopProcessor, show in figure 38.6. . . . . 38.15Output of loopProcessor input from figure 38.14 (using options: -fs2 -ic1 -opt 1 ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39.1 39.2 39.3 39.4 39.5 39.6 39.7

Example source code used as input to loopUnrolling . . . . . . . Output for a unrolling factor which can divide the iteration space Output for the case when divisibility is unknown at compile-time Example source code used as input to loopInterchange . . . . . . Output for loop interchange . . . . . . . . . . . . . . . . . . . . . Example source code used as input to loopTiling . . . . . . . . . Output for loop tiling . . . . . . . . . . . . . . . . . . . . . . . .

. . . . evenly . . . . . . . . . . . . . . . . . . . .

40.1 Example source code shows instrumentation to call a test function from of each function body in the application (part 1). . . . . . . . . . . . . . 40.2 Example source code shows instrumentation to call a test function from of each function body in the application (part 2). . . . . . . . . . . . . . 40.3 Example source code shows instrumentation to call a test function from of each function body in the application (part 3). . . . . . . . . . . . . . 40.4 Example source code used as input to translator adding new function. . 40.5 Output of input to translator adding new function. . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

the top . . . . . the top . . . . . the top . . . . . . . . . . . . . . .

289 290 291 292 293 294 295 296 298 299 300 301 301 302 302

308 309 310 311 312

41.1 Example source code used as input to program in codes used in this chapter. . . 313 41.2 Example source code showing how to seed bugs. . . . . . . . . . . . . . . . . . . 315 41.3 Output of input code using seedBugsExample arrayIndexing.C . . . . . . . . . . 316 43.1 Source-to-source transformation to introduce NOP assemble instructions in the generated binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.2 Header file for the traversal used to identify the NOP sequences in the binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.3 Example code to identify the NOP sequences in the binary executable. . . . . . . 43.4 Main program using the traversal to identify the NOP sequences in the binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.5 Example code showing the transformation of the binary executable. . . . . . . . 45.1 45.2 45.3 45.4

Example source code used to generate Dwarf AST for analysis. . . . Dwarf AST (subset of ROSE binary AST). . . . . . . . . . . . . . . Example source code (typical for reading in a binary or source file). Example source code (typical for reading in a binary or source file).

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

324 325 327 328 329 339 341 342 343

LIST OF FIGURES 46.1 Example 1: Generated handles for loops: using constructors with or without a specified handle type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46.2 Example 1: Example source code with some loops, used as input. . . . . . . . . . 46.3 Example 1: Abstract handles generated for loops. . . . . . . . . . . . . . . . . . . 46.4 Example 2: Generated handles from strings representing handle items. . . . . . . 46.5 Example 2: Source code with some language constructs. . . . . . . . . . . . . . . 46.6 Example 2: Handles generated from string and their language constructs. . . . . 46.7 Example 3: A simple data structure used to represent a loop in an arbitrary tool. 46.8 Example 3: A test program for simple loops’ abstract handles. . . . . . . . . . . 46.9 Example 3: Output of the test program for simple loops’ abstract handles (as strings). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47.1 profiled.c (part 1 of 2): Sample input program, profiled using the HPCToolkit. . 47.2 profiled.c (part 2 of 2): Sample input program, profiled using the HPCToolkit. . 47.3 XML schema for HPCToolkit data files: This schema, prepended to each of the HPCToolkit-generated XML files, describes the format of the profiling data. This particular schema was generated by HPCToolkit 1.0.4. . . . . . . . . . . . . . . . 47.4 PAPI TOT CYC.xml: Sample cycle counts observed during profiling, generated from running the HPCToolkit on profiled.c (Figures 47.1–47.2.) These lines would appear after the schema shown in Figure 47.3. . . . . . . . . . . . . . . . . . . . . 47.5 PAPI FP OPS.xml: Sample flop counts observed during profiling, generated from running the HPCToolkit on profiled.c (Figures 47.1–47.2.) These lines would appear after the schema shown in Figure 47.3. . . . . . . . . . . . . . . . . . . . . 47.6 attachMetrics.cc: Sample translator to attach HPCToolkit metrics to the AST. . 47.7 Sample output, when running attachMetrics.cc (Figure 47.6) with the XML inputs in Figures 47.4–47.5. Here, we only show the output sent to standard output (i.e., cout and not cerr). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47.8 Sample PDF showing attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . .

xix

351 352 353 354 355 355 356 357 358 363 364

365

366

367 372

373 373

48.1 Example source code used as input to program in codes used in this chapter. . . 376 48.2 Example source code showing how to instrument using Tau. . . . . . . . . . . . 377 48.3 Output of input code using tauInstrumenter.C . . . . . . . . . . . . . . . . . . . 378 49.1 Haskell version of identity translator. . . . . . . . . . . . . . . . . . . . . . . . . . 379 49.2 Haskell version of constant folding transformation. . . . . . . . . . . . . . . . . . 381 50.1 Example source showing the shared-memory parallel execution of traversals. . . . 386 50.2 Output of input file to the shared-memory parallel traversals. Output may be garbled depending on the multi-threaded behavior of the underlying I/O libraries. 387 51.1 Example source demonstrating the use of the distributed-memory parallel analysis framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 51.2 Example output of a distributed-memory analysis running on four processors. . . 392 53.1 Example source code showing reduction recognition. . . . . . . . . . . . . . . . . 395 53.2 Example source code used as input to loop reduction recognition processor. . . . 396 53.3 Output of input to reduction recognition processor. . . . . . . . . . . . . . . . . . 396

xx

LIST OF FIGURES

Chapter 1

Introduction 1.1

What is ROSE

ROSE is an open source compiler infrastructure for building tools that can read and write source code in multiple languages (C/C++/Fortran) and/or analyze binary executables (using the x86, Power-PC, and ARM instruction sets). The target audience for ROSE is people building tools for the analysis and transformation of software generally as well as code generation tools. ROSE provides a library (librose) that can be used to support the universal requirements of tools that do custom analysis and/or transformations on source code and custom analysis of binary forms of software. ROSE is portable across and expanding range of operating systems and work with an growing number of compilers. ROSE provides a common level of infrastructure support to user-defined tools, so that they need not implement the complex support required for software analysis and transformation operations. For source code based tools these include parsing, common forms of compiler analysis, common transformations, and code generation. For binary analysis based tools these include disassembly, function boundary detection, and common forms of analysis. User defined tools may also mix source code and binary analysis to form more interesting tools for specialized purposes. ROSE is part of research work to unify the analysis of both source code and binaries within general compiler research and define mixed forms of static and dynamic analysis. ROSE works by reading the source code and/or binary and generating an Abstract Syntax Tree (AST). The AST forms a graph representing the structure of the source code and/or binary executable and is held in memory to provide the fastest possible means of operating on the graph. The nodes used to define the AST graph are an intermediate representation (IR); common within compiler research as a way of representing the structure of software absent syntax details (commas, semi-colons, white-space, etc.). ROSE provides mechanisms to traverse and manipulate the AST. Finally, in the case of source code, ROSE provides mechanisms to regenerate source code from the AST. As a trivial example, if the input source code program contains a variable declaration for an integer, all of this information will be available in the AST generated from the input code passed on the command line to any tool built using ROSE. Similarly, an automated transformation of the variable declaration held in the AST would be expressed using a traversal over the AST 1

2

CHAPTER 1. INTRODUCTION

and code semantic actions to mutate the AST. Then the transformed source code would be generated (unparsed) from the AST. In the case of binaries (including executables, object files, and libraries), the AST will represent the structure of the binary. The AST for a binary also includes the binary file format (symbol table, debug format, import tables, etc.), disassembled instructions, all instruction operands, etc. ROSE provides a rich set of tools to support the analysis of software including the support for users to build their own forms of analysis and specialized transformations. As an example, ROSE includes a full OpenMP compiler built using the internal ROSE infrastructure for analysis and transformation. A wide assortment of AST traversals are provided to express both analysis and transformations of the AST. A set of common forms of analysis are provided (call graph, control flow, etc.) most work uniformly on both source code and binary executables. Visualization support in included to help users understand and debug their tools. GUI support is available to support building professional level tools using ROSE. ROSE is actively supported by a small group at LLNL and is used as a basis for compiler research work within DOE at LLNL. Technically, ROSE is designed to build what are called translators, ROSE uses a source-tosource approach to define such translators. Note that translators are significantly more sophisticated than preprocessors but the terms are frequently confused. A translator must understand the source code at a fundamentally deeper level using a grammar for the whole language and on the whole source code, where as a preprocessor only understands the source code using a simpler grammar and on a subset of the source code. It is loosely the difference between any language compiler and the C preprocessor (cpp).

1.2

Why you should be interested in ROSE

ROSE is a tool for building source-to-source translators. You should be interested in ROSE if you want to understand or improve any aspect of your software. ROSE makes it easy to build tools that read and operate on source code from large scale applications (millions of lines). Whole projects may be analyzed and even optimized using tools built using ROSE. For example, ROSE is itself analyzed nightly using ROSE. To get started immediately consult the ROSE User Manual, chapter Getting Started for details).

1.3

Problems that ROSE can address

ROSE is a mechanism to build source-to-source analysis or optimization tools that operate directly on the source code of large scale applications. Example tools that have been built include: • OpenMP translator, • Array class abstraction optimizer, • Source-to-source instrumenter, • Loop analyzer, • Symbolic complexity analyzer,

1.4. EXAMPLES IN THIS ROSE TUTORIAL

3

• Inliner and outliner, • Code coverage tools, • and many more... Example tools that can be built include: • Custom optimization tools, • Custom documentation generators, • Custom analysis tools, • Code pattern recognition tools, • Security analysis tools, • and many more...

1.4

Examples in this ROSE Tutorial

This tutorial lays out a set of progressively complex example programs (located in /tutorial/*) that serve as a tutorial for the use of ROSE. Translators built using ROSE can either just analyze (and output results) or compile the input programs just like a compiler (generating object files or executables). Many of the examples in this tutorial just do simple analysis of the input source code, and a few show the full compilation of the input source code. Where the translators generate either object files of executables, the vendor’s compiler is used to compile the final ROSE-generated code. Within ROSE, the call to generate source code from the AST and call the vendor’s compiler is referred to as the backend processing. The specification of the vendor’s compiler as a backend is done within the configuration step within ROSE (see options for configure in the ROSE User Manual). Within the example programs below, the user can provide alternative input programs for more complex evaluation of the tutorial examples and ROSE. The end of the chapter, section 1.7, shows the makefiles required to compile the tutorial programs using an installed version of ROSE (compiled using make install). This example makefile is run as part of the testing using the make installcheck rule. Chapters are organized in topics including simple ROSE AST visualization, dealing with complex data types, program analysis, program transformation and optimization, correctness checking, binary support, interacting with other tools, and parallelism. We hope readers can easily find the information they want. Specific chapters in this tutorial include: FIXME: We should constantly update this

• Introduction 1. Introduction (this chapter) 2. Problems that ROSE can address 3. Getting Started This chapter covers where to find ROSE documentation and how to install ROSE.

4

CHAPTER 1. INTRODUCTION 4. Example Makefiles demonstrating the command lines to compile and link the example translators in this tutorial are found in /tutorial/exampleMakefile. • Working with the ROSE AST: 1. Identity Translator This example translator reads a C or C++ application, builds the AST internally, generates source code from the AST (unparsing), and calls the backend vendor compiler to compile the generated C or C++ application code. Thus the translator acts like and can be used to replace any compiler since it takes in source code and outputs an object code (or executable). This example also shows that the output generated from and ROSE translator is a close reproduction of the input; preserving all comments, preprocessor control structure, and most formating. 2. Scopes of Declarations (scopeInformation.C) This example shows the scopes represented by different IR nodes in the AST. 3. AST Graph Generator This translator reads a C or C++ application code and builds the AST, internally. The translator does not regenerate code from the AST and so does not call the backend vendor’s compiler. This shows how simple it could be to build source code analysis tools; the code calls an internal ROSE function to generate a dot graph of the AST, the makefile has the details of converting the dot graph into a postscript file (also shown). 4. AST PDF Generator This translator reads an C or C++ application code builds the AST internally. The translator does not regenerate code from the AST and so does not call the backend vendor’s compiler. This shows how simple it could be to build source code analysis tools, the code calls an internal ROSE function to generate a pdf file with bookmarks representing the AST. The pdf file show as output is in this case a previously generated figure of a screen shot obtained by viewing the output pdf file using acroread. 5. Introduction to AST Traversals and Attributes This collection of examples show the use of the simple visitor pattern for the traversal of the AST within ROSE. The simple visitor pattern permits operations to be programmed which will be invoked on different nodes within the AST. To handle communication of context information down into the AST and permit communication of analysis information up the AST, we have provided inherited and synthesized attributes (respectively). Note that an AST is most often represented as a tree with extra edges and with shared IR nodes that make the full graph (representing all edges) not a tree. We present two styles of traversal, one over the tree representing the AST (which excludes some types of IR nodes) and one over the full AST with all extra nodes and shared nodes. Extra nodes are nodes such as SgType and SgSymbol IR nodes. (a) AST traversals These traversals visit each node of the tree embedded within the AST (excluding shared SgType and SgSymbol IR nodes). These traversals visit the IR nodes is

1.4. EXAMPLES IN THIS ROSE TUTORIAL

5

an order dependent upon the structure of the AST (the source code from which the AST is built). i. Classic Object-Oriented Visitor Patterns This example, classicObjectOrientedVisitorPatternMemoryPoolTraversal.C, show the use of a classic visitor patterns. At the moment this example uses the AST’s memory pools as a basis but it is identical to a future traversal. The ROSE visitor Pattern (below) is generally more useful. The classic visitor pattern traversals are provided for completeness. ii. Visitor Traversal (visitorTraversal.C) Conventional visitor patterns without no attributes. This pattern can explicitly access global variables to provide the effect of accumulator attributes (using static data members we later show the handling of accumulator attributes). iii. Inherited Attributes (inheritedAttributeTraversal.C) Inherited attributes are used to communicate the context of any location within the AST in terms of other parent AST nodes. iv. Synthesized Attributes (synthesizedAttributeTraversal.C) Synthesized attributes are used to pass analysis results from the leaves of the AST to the parents (all the way to the root of the AST if required). v. Accumulator Attributes (accumulatorAttributeTraversal.C) Accumulator attributes permit the interaction of data within inherited attributes with data in synthesized attributes. In our example program we will show the use of accumulator attributes implemented as static data members. Accumulator attributes are a fancy name for what is essentially global variables (or equivalently a data structure passed by reference to all the IR nodes in the AST). vi. Inherited and Synthesized Attributes (inheritedAndSynthesizedAttributeTraversal.C) The combination of using inherited and synthesized attributes permits more complex analysis and is often required to compute analysis results on the AST within a specific context (e.g. number of loop nests of specific depth). vii. Persistent Attributes (persistantAttributes.C) Persistent attributes may be added the AST for access to stored results for later traversals of the AST. The user controls the lifetime of these persistent attributes. viii. Nested traversals Complex operations upon the AST can require many subordinate operations. Such subordinate operations can be accommodated using nested traversals. All traversals can operate on any subtree of the AST, and may even be nested arbitrarily. Interestingly, ROSE traversals may also be applied recursively (though care should be take using recursive traversals using accumulator attributes to avoid over accumulation). (b) Memory Pool traversals These traversals visit all IR nodes (including shared IR nodes such as SgTypes and SgSymbols). By design this traversal can visit ALL IR nodes without the worry

6

CHAPTER 1. INTRODUCTION of getting into cycles. These traversals are mostly useful for building specialized tools that operate on the AST. i. Visit Traversal on Memory Pools This is a similar traversal as to the Visitor Traversal over the tree in the AST. ii. Classic Object-Oriented Visitor Pattern on Memory Pools This is similar to the Classic Object-Oriented Visitor Pattern on the AST. iii. IR node Type Traversal on Memory Pools This is a specialized traversal which visits each type of IR node, but one one of each type of IR nodes. This specialized traversal is useful for building tools that call static member functions on each type or IR node. A number of memory based tools for ROSE are built using this traversal. 6. AST Query Library This example translator shows the use of the AST query library to generate a list of function declarations for any input program (and output the list of function names). It can be trivially modified to return a list of any IR node type (C or C++ language construct). 7. Symbol Table Handling (symbolTableHandling.C) This example shows how to use the symbol tables held within the AST for each scope. 8. AST File I/O (astFileIO GenerateBinaryFile.C) This example demonstrates the file I/O for AST. This is part of ROSE support for whole program analysis. 9. Debugging Tips There are numerous methods ROSE provides to help debug the development of specialized source-to-source translators. This section shows some of the techniques for getting information from IR nodes and displaying it. Show how to use the PDF generator for AST’s. This section may contain several subsections. (a) Generating the code representing any IR node (b) Displaying the source code position of any IR node • Complex Types 1. Getting the type parameters in function declaration (functionParameterTypes.C) This example translator builds a list to record the types used in each function. It shows an example of the sort of type information present within the AST. ROSE specifically maintains all type information. 2. Resolving overloaded functions (resolvingOverloadedFunctions.C – C++ specific) The AST has all type information pre-evaluated, particularly important for C++ applications where type resolution is required for determining function invocation. This example translator builds a list of functions called within each function, showing that overloaded function are fully resolved within the AST. Thus the user is not required to compute the type resolution required to identify which over loaded functions are called. 3. Getting template parameters to a templated class (templateParameters.C – C++ specific)

1.4. EXAMPLES IN THIS ROSE TUTORIAL

7

All template information is saved within the AST. Templated classes and functions are separately instantiated as specializations, as such they can be transformed separately depending upon their template values. This example code shows the template types used the instantiate a specific templated class. • Program Analysis 1. Recognizing loops within applications (loopRecognition.C) This example program shows the use of inherited and synthesized attributes form a list of loop nests and report their depth. The inherited attributes are required to record when the traversal is within outer loop and the synthesized attributes are required to pass the list of loop nests back up of the AST. 2. Generating a CFG (buildCFG.C) This example shows the generation of a control flow graph within ROSE. The example is intended to be simple. Many other graphs can be built, we need to show them as well. 3. Generating a CG (buildCallGraph.C) This example shows the generation of a call graph within ROSE. 4. Generating a CH (classHierarchyGraph.C) This example shows the generation of a class hierarchy graph within ROSE. 5. Building custom graphs of program information The mechanisms used internally to build different graphs of program data is also made externally available. This section shows how new graphs of program information can be built or existing graphs customized. 6. Database Support (dataBaseUsage.C) This example shows how to use the optional (see configure --help) SQLite database to hold persistent program analysis results across the compilation of multiple files. This mechanism may become less critical as the only mechanism to support global analysis once we can support whole program analysis more generally within ROSE. • Program Transformations and Optimizations 1. Generating Unique Names for Declarations (generatingUniqueNamesFromDeclaration.C) A recurring issue in the development of many tools and program analysis is the representation of unique strings from language constructs (functions, variable declarations, etc.). This example demonstrated support in ROSE for the generation of unique names. Names are unique across different ROSE tools and compilation of different files. 2. Command-line processing ROSE includes mechanism to simplify the processing of command-line arguments so that translators using ROSE can trivially replace compilers within makefiles. This example shows some of the many command-line handling options within ROSE and the ways in which customized options may be added. (a) Recognizing custom command-line options

8

CHAPTER 1. INTRODUCTION (b) Adding options to internal ROSE command-line driven mechanisms 3. Tailoring the code generation format: how to indent the generated source code and others. 4. AST construction: how to build AST pieces from scratch and attach them to the existing AST tree. (a) Adding a variable declaration (addingVariableDeclaration.C) Here we show how to add a variable declaration to the input application. Perhaps we should show this in two ways to make it clear. This is a particularly simple use of the AST IR nodes to build an AST fragment and add it to the application’s AST. (b) Adding a function (addingFunctionDeclaration.C) This example program shows the addition of a new function to the global scope. This example is a bit more involved than the previous example. (c) Simple Instrumentor Translator (simpleInstrumentor.C) This example modifies an input application to place new code at the top and bottom of each block. The output is show with the instrumentation in place in the generated code. (d) Other examples for creating expressions, structures and so on. 5. Handling source comments, preprocessor directives. 6. Calling the inliner (inlinerExample.C) This example shows the use of the inliner mechanism within ROSE. The function to be inlined in specified and the transformation upon the AST is done to inline the function where it is called and clean up the resulting code. 7. Calling the outliner (outlinerExample.C) This example shows the use of the outliner mechanism within ROSE. A segment of code is selected and a function is generated to hold the resulting code. Any required variables (including global variables) are passed through the generated function’s interface. The outliner is a useful part of the empirical optimization mechanisms being developed within ROSE. 8. Call loop optimizer on set of loops (loopOptimization.C) This example program shows the optimization of a loop in C. This section contains several subsections each of which shows different sorts of optimizations. There are a large number of loop optimizations only two are shown here, we need to add more. (a) Optimization of Matrix Multiply (b) Loop Fusion Optimizations

Does this tutorial still exist?

9. Parameterized code translation: How to use command line options and abstract handles to have the translations you want, the order you want, and the behaviors you want. 10. Program slicing (programSlicingExample.C) This example shows the interface to the program slicing mechanism within ROSE. Program slicing has been implemented to two ways within ROSE. • Correctness Checking

1.4. EXAMPLES IN THIS ROSE TUTORIAL

9

1. Code Coverage Analysis (codeCoverage.C) Code coverage is a useful tool by itself, but is particularly useful when combined with automated detection of bugs in programs. This is part of work with IBM, Haifa. 2. Bug seeding: how to purposely inject bugs into source code. • Binary Support 1. Instruction semantics 2. Binary Analysis 3. Binary construction 4. DWarf debug support • Interacting with Other Tools 1. Abstract handles: uniform ways of specifying language constructs. 2. ROSE-HPCT interface: How to annotate AST with performance metrics generated by third-party performance tools. 3. Tau Performance Analysis Instrumentation (tauInstrumenter.C) Tau currently uses an automate mechanism that modified the source code text file. This example shows the modification of the AST and the generation of the correctly instrumented files (which can otherwise be a problem when macros are used). This is part of collaborations with the Tau project. 4. The Haskell interface: interacting with a function programming language. • Parallelism 1. Shared-memory parallel traversals 2. Distributed-memory parallel traversals 3. Parallel checker 4. Reduction variable recognition Other examples included come specifically from external collaborations and are more practically oriented. Each is useful as an example because each solves a specific technical problem. More of these will be included over time. FIXME: The following tutorials no longer exist?

1. Fortran promotion of constants to double precision (typeTransformation.C) Fortran constants are by default singe precision, and must be modified to be double precision. This is a common problem in older Fortran applications. This is part of collaborations with LANL to eventually automatically update/modify older Fortran applications. 2. Automated Runtime Library Support (charmSupport.C) Getting research runtime libraries into use within large scale applications requires automate mechanism to make minor changes to large amounts of code. This is part of collaborations with the Charm++ team (UIUC).

Add make installcheck e.am to build example ors using the installed libraries.

10

CHAPTER 1. INTRODUCTION (a) Shared Threaded Variable Detection Instrumentation (interveneAtVariables.C) Instrumentation support for variables, required to support detection of threaded bugs in applications. (b) Automated Modification of Function Parameters (changeFunction.C) This example program addresses a common problem where an applications function must be modified to include additional information. In this case each function in a threaded library is modified to include additional information to a corresponding wrapper library which instruments the library’s use.

1.5

ROSE Documentation and Where To Find It

There are three forms of documentation for ROSE, and also a ROSE web Page and email lists. For more detailed information on getting started, see the ROSE User Manual, chapter Getting Started for more details). 1. ROSE User Manual The User Manual presents how to get started with ROSE and documents features of the ROSE infrastructure. The User Manual is found in ROSE/docs/Rose directory, or at: ROSE User Manual (pdf version, relative link) 2. ROSE Tutorial The ROSE Tutorial presents a collection of examples of how to use ROSE (found in the ROSE/tutorial directory). The ROSE Tutorial documentation is found in ROSE/docs/Rose/Tutorial directory. The tutorial documentation is built in the following steps: (a) actual source code for each example translator in the ROSE/tutorial directory is included into the tutorial documentation (b) each example is compiled (c) inputs to the examples are taken from the ROSE/tutorial directory (d) output generated from running each example is placed into the tutorial documentation Thus the ROSE/tutorial directory contains the exact examples in the tutorial and each example may be modified (changing either the example translators or the inputs to the examples). The ROSE Tutorial can also be found in the ROSE/docs/Rose/Tutorial directory (the LaTeX document; ps or pdf file) : ROSE Tutorial (pdf version, relative link), 3. ROSE HTML Reference: Intermediate Representation (IR) documentation This web documentation presents the detail interfaces for each IR nodes (documentation generated by Doxygen). The HTML IR documentation is found in ROSE/docs/Rose directory (available as html only): ROSE HTML Reference (relative link) 4. ROSE Web Page The ROSE web pages are located at: http://www.rosecompiler.org

1.6. USING THE TUTORIAL

11

5. ROSE Email List The ROSE project maintains an external mailing list (see information at: www.roseCompiler.org and click on the Mailing Lists link for how to join).

1.6

Using the Tutorial

First install ROSE (see ROSE User Manual, chapter Getting Started for details). Within the ROSE distribution at the top level is the tutorial directory. All of the examples in this documentation are represented there with Makefiles and sample input codes to the example translators.

1.7

Required Makefile for Tutorial Examples

This section shows an example makefile 1.1 required for the compilation of many of the tutorial example programs using the installed libraries (assumed to be generated from make install). The build process can be tested by running make installcheck from within the ROSE compile tree. This makefile can be found in the compile tree (not the source tree) for ROSE in the tutorial directory.

FIXME: The exampleMakefile needs to be modified to include the support for compiling all of the tutorial examples.

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

CHAPTER 1. INTRODUCTION

# # # # # #

Example Makefile for ROSE users This makefile is provided as an example of how to use ROSE when ROSE is installed (using "make install"). This makefile is tested as part of the "make distcheck" rule (run as part of tests before any SVN checkin). The test of this makefile can also be run by using the "make installcheck" rule (run as part of "make distcheck").

# Location of include directory after "make install" ROSE_INCLUDE_DIR = /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/install_tree/include # Location of Boost include directory BOOST_CPPFLAGS = -pthread -I/nfs/casc/overture/ROSE/opt/rhel7/x86_64/boost/1_57_0/gcc/4.8.5/include # Location of Dwarf include and lib (if ROSE is configured to use Dwarf) ROSE_DWARF_INCLUDES = @[email protected] ROSE_DWARF_LIBS_WITH_PATH = @[email protected] # Location of library directory after "make install" ROSE_LIB_DIR = /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/install_tree/lib CC CXX CPPFLAGS #CXXCPPFLAGS CXXFLAGS LDFLAGS

= gcc = g++ = = @[email protected] = -g -O2 -Wall -Wall =

ROSE_LIBS = $(ROSE_LIB_DIR)/librose.la # Location of source code ROSE_SOURCE_DIR = \ /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial executableFiles = identityTranslator ASTGraphGenerator \ visitorTraversal inheritedAttributeTraversal \ synthesizedAttributeTraversal \ inheritedAndSynthesizedAttributeTraversal \ accumulatorAttributeTraversal persistantAttributes \ queryLibraryExample nestedTraversal \ loopRecognition \ typeInfoFromFunctionParameters \ resolveOverloadedFunction templateParameter \ instrumentationExample addVariableDeclaration \ addFunctionDeclaration loopOptimization \ buildCFG debuggingIRnodeToString \ debuggingSourceCodePositionInformation \ commandlineProcessing \ loopNestingInfoProcessing # Default make rule to use all: $(executableFiles) @if [ x$${ROSE_IN_BUILD_TREE:+present} = xpresent ]; then echo "ROSE_IN_BUILD_TREE should not be set" >&2; exit 1; fi

# Example of how to use ROSE (linking to dynamic library, which is must faster # and smaller than linking to static libraries). Dynamic linking requires the # use of the "-L$(ROSE_LIB_DIR) -Wl,-rpath" syntax if the LD_LIBRARY_PATH is not # modified to use ROSE_LIB_DIR. We provide two example of this; one using only # the "-lrose -ledg" libraries, and one using the many separate ROSE libraries. $(executableFiles): # g++ -I$(ROSE_INCLUDE_DIR) -o [email protected] $(ROSE_SOURCE_DIR)/[email protected] -L$(ROSE_LIB_DIR) -Wl,-rpath $(ROSE_LIB_DIR) $(ROSE_LIBS) # g++ -I$(ROSE_INCLUDE_DIR) -o [email protected] $(ROSE_SOURCE_DIR)/[email protected] $(ROSE_LIBS) # /bin/sh ../libtool --mode=link $(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) -I$(ROSE_INCLUDE_DIR) $(BOOST_CPPFLAGS) -o [email protected] $(ROSE_SOUR /bin/sh ../libtool --mode=link $(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) -I$(ROSE_INCLUDE_DIR) $(BOOST_CPPFLAGS) $(ROSE_DWARF_INCL

Figure 1.1: Example Makefile showing how to use an installed version of ROSE (generated by make install).

Part I

Working with the ROSE AST

Getting familiar with the ROSE AST is the basis for any advanced usage of ROSE. This part of tutorial collects examples for AST visualization, traversal, query, and debugging.

13

Chapter 2

Identity Translator What To Learn From This Example This example shows a trivial ROSE translator which does not transformation, but effectively wraps the the backend vendor compiler in an extra layer of indirection. Using the input code in Figure 2.2 we show a translator which builds the AST (calling frontend()), generates the source code from the AST, and compiles the generated code using the backend vendor compiler1 . Figure 2.1 shows the source code for this translator. The AST graph is generated by the call to the frontend() using the standard argc and argv parameters from the C/C++ main() function. In this example code, the variable project represents the root of the AST2 . The source code also shows what is an optional call to check the integrity of the AST (calling function AstTests::runAllTests()); this function has no side-effects on the AST. The source code generation and compilation to generate the object file or executable are done within the call to backend(). The identity translator (identityTranslator ) is probably the simplest translator built using ROSE. It is built by default and can be found in ROSE BUILD/exampleTranslators/documentedExamples/simpleTranslatorExamples or ROSE INSTALL/bin. It is often used to test if ROSE can compile input applications. Typing identityTranslator –help will give you more information about how to use the translator. Figure 2.3 shows the generated code from the processing of the identityTranslator build using ROSE and using the input file shown in figure 2.2. This example also shows that the output generated from and ROSE translator is a close reproduction of the input; preserving all comments, preprocessor control structure, and most formating. Note that all macros are expanded in the generated code. In this trivial case of a program in a single file, the translator compiles the application to build an executable (since -c was not specified on the command-line).

1 Note:

that the backend vendor compiler is selected at configuration time. The AST is technically a tree with additional attributes that are represented by edges and additional nodes, so the AST is a tree and the AST with attributes is a more general graph containing edges that would make it technically not a tree. 2

15

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

CHAPTER 2. IDENTITY TRANSLATOR

// Example ROSE T r a n s l a t o r : u s e d f o r

t e s t i n g ROSE i n f r a s t r u c t u r e

#i n c l u d e ” r o s e . h” i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // Run i n t e r n a l c o n s i s t e n c y t e s t s on AST AstTests : : runAllTests ( p r o j e c t ) ; // I n s e r t your own m a n i p u l a t i o n o f t h e AST h e r e . . . // G e n e r a t e s o u r c e c o d e from AST and c a l l r e t u r n backend ( p r o j e c t ) ; }

t h e vendor ’ s c o m p i l e r

Figure 2.1: Source code for translator to read an input program and generate an object code (with no translation).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

// Example i n p u t f i l e #i n c l u d e typedef

f o r ROSE t u t o r i a l

f l o a t Real ;

// Main f u n c t i o n i n t main ( ) { int x = 0; bool value = f a l s e ; // f o r l o o p f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 2.2: Example source code used as input to identity translator.

17

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

// Example i n p u t f i l e #i n c l u d e t y p e d e f f l o a t R e al ; // Main f u n c t i o n

f o r ROSE t u t o r i a l

i n t main ( ) { int x = 0; bool value = f a l s e ; // f o r l o o p for ( int i = 0; i < 4; int x ; } return 0; }

i ++) {

Figure 2.3: Generated code, from ROSE identity translator, sent to the backend (vendor) compiler.

18

CHAPTER 2. IDENTITY TRANSLATOR

Chapter 3

Simple AST Graph Generator What To Learn From This Example This example shows how to generate a DOT file to visualize a simplified AST from any input program. DOT is a graphics file format from the AT&T GraphViz project used to visualize moderate sized graphs. It is one of the visualization tools used within the ROSE project. More inforamtion can be readily found at www.graphviz.org/. We have found the zgrviewer to be an especially useful program for visualizing graphs generated in the DOT file format (see chapter 5 for more information on zgrviewer). Each node of the graph in figure 3.3 shows a node of the Intermediate Representation (IR); the graphs demonstrates that the AST is formally a tree. Each edge shows the connection of the IR nodes in memory. The generated graph shows the connection of different IR nodes that form the AST for an input program source code. Binary executables can similarly be vizualized using DOT files. The generation of such graphs is appropriate for small input programs, chapter 6 shows a mechanism using PDF files that is more appropriate to larger programs (e.g. 100K lines of code). More information about generation of specialized AST graphs can be found in 5 and custom graph generation in 28. Note that a similar utility program named dotGenerator already exists within ROSE/exampleTranslators/DOTGenerator. It is also installed to ROSE INS/bin. The program in figure 3.1 calls an internal ROSE function that traverses the AST and generates an ASCII file in dot format. Figure 3.2 shows an input code which is processed to generate a graph of the AST, generating a dot file. The dot file is then processed using dot to generate a postscript file 3.3 (within the Makefile). Figure 3.3 (../../../tutorial/test.ps) can be found in the compile tree (in the tutorial directory) and viewed directly using ghostview or any postscript viewer to see more detail. Figure 3.3 displays the individual C++ nodes in ROSE’s intermediate representation (IR). Each circle represents a single IR node, the name of the C++ construct appears in the center of the circle, with the edge numbers of the traversal on top and the number of child nodes appearing below. Internal processing to build the graph generates unique values for each IR node, a pointer address, which is displays at the bottom of each circle. The IR nodes are connected for form a tree, and abstract syntax tree (AST). Each IR node is a C++ class, see SAGE III reference for 19

20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

CHAPTER 3. SIMPLE AST GRAPH GENERATOR

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // G e n e r a t e a DOT f i l e t o u s e i n generateDOT ( ∗ p r o j e c t ) ;

v i s u a l i z i n g t h e AST graph .

return 0; }

Figure 3.1: Example source code to read an input program and generate an AST graph. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; }

Figure 3.2: Example source code used as input to generate the AST graph. details, the edges represent the values of data members in the class (pointers which connect the IR nodes to other IR nodes). The edges are labeled with the names of the data members in the Use this first example classes representing the IR nodes.

e to explain the use of ader files (config.h and d the code to build the SgProject object.

21

0:79 SgSourceFile child_count:4 0x7f508ffec010 inputCode_ASTGraphGenerator.C:1:1 (physical line=1) (raw line:col=1:1)

globalScope

package import_list class_list

1:78 SgGlobal isModified = false containsTransformation = false isTransformation = false child_count:434 0x7f509c266120 inputCode_ASTGraphGenerator.C:0:0 (physical line=0) (raw line:col=0:0) inputCode_ASTGraphGenerator.C:0:0 (physical line=0) (raw line:col=0:0) containsTransformationToSurroundingWhitespace == false

*[431]

2:25 SgTemplateClassDeclaration templateClass isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:2 0x7f508f12c010 inputCode_ASTGraphGenerator.C:3:1 (physical line=3) (raw line:col=3:1) inputCode_ASTGraphGenerator.C:11:5 (physical line=11) (raw line:col=11:5) containsTransformationToSurroundingWhitespace == false comments/directives (before) = 1 comments/directives (inside) = 0 comments/directives (after) = 0

definition

decoratorList

parameterList

4:7 SgVariableDeclaration isAssociatedWithDeclarationList = false variableDeclarationContainsBaseTypeDefiningDeclaration = false isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:2 0x7f508efa8010 inputCode_ASTGraphGenerator.C:7:11 (physical line=7) (raw line:col=7:11) inputCode_ASTGraphGenerator.C:7:15 (physical line=7) (raw line:col=7:15) containsTransformationToSurroundingWhitespace == false

baseTypeDefiningDeclaration

initptr

*[2]

parameterList

9:12 SgFunctionParameterList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fecb460 compiler generated compilerGenerated:0:0 (physical line=9) (raw line:col=9:11) compilerGenerated:0:0 (physical line=9) (raw line:col=9:23) is NOT output in generated code containsTransformationToSurroundingWhitespace == false

*[0]

decoratorList definition

16:23 SgTemplateMemberFunctionDeclaration foo isFriend = false isForward isModified = false containsTransformation = false isTransformation = false child_count:4 0x7f508ebc9468 inputCode_ASTGraphGenerator.C:10:11 (physical line=10) (raw line:col=10:11) inputCode_ASTGraphGenerator.C:10:26 (physical line=10) (raw line:col=10:26) containsTransformationToSurroundingWhitespace == false

CtorInitializerList

13:14 SgCtorInitializerList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:0 0x7f508eaa4010 NULL_FILE:0:0 (physical line=0) (raw line:col=0:0) NULL_FILE:0:0 (physical line=0) (raw line:col=0:0) containsTransformationToSurroundingWhitespace == false

parameterList

17:20 SgFunctionParameterList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fecb910 compiler generated compilerGenerated:0:0 (physical line=10) (raw line:col=10:11) compilerGenerated:0:0 (physical line=10) (raw line:col=10:26) is NOT output in generated code containsTransformationToSurroundingWhitespace == false

*[0]

10:11 SgInitializedName

18:19 SgInitializedName

isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbed68 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) IS output in generated code containsTransformationToSurroundingWhitespace == false

isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbeff8 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) IS output in generated code containsTransformationToSurroundingWhitespace == false

initptr

initptr

decoratorList definition

32:77 SgFunctionDeclaration foo isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:3 0x7f508fb54b78 inputCode_ASTGraphGenerator.C:15:1 (physical line=15) (raw line:col=15:1) inputCode_ASTGraphGenerator.C:25:4 (physical line=25) (raw line:col=25:4) containsTransformationToSurroundingWhitespace == false

parameterList

37:76 SgFunctionDefinition isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508e986010 inputCode_ASTGraphGenerator.C:16:4 (physical line=16) (raw line:col=16:4) inputCode_ASTGraphGenerator.C:25:4 (physical line=25) (raw line:col=25:4) containsTransformationToSurroundingWhitespace == false

body

28:29 SgInitializedName

34:35 SgInitializedName isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbf660 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) IS output in generated code containsTransformationToSurroundingWhitespace == false

21:22 SgCtorInitializerList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:0 0x7f508eaa4268 NULL_FILE:0:0 (physical line=0) (raw line:col=0:0) NULL_FILE:0:0 (physical line=0) (raw line:col=0:0) containsTransformationToSurroundingWhitespace == false

definition

*[0]

isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbf288 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) IS output in generated code containsTransformationToSurroundingWhitespace == false

CtorInitializerList

decoratorList

33:36 SgFunctionParameterList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fecc4c8 inputCode_ASTGraphGenerator.C:15:1 (physical line=15) (raw line:col=15:1) inputCode_ASTGraphGenerator.C:15:16 (physical line=15) (raw line:col=15:16) containsTransformationToSurroundingWhitespace == false

*[0]

8:15 SgTemplateMemberFunctionDeclaration foo isFriend = false isForward isModified = false containsTransformation = false isTransformation = false child_count:4 0x7f508ebc9010 inputCode_ASTGraphGenerator.C:9:11 (physical line=9) (raw line:col=9:11) inputCode_ASTGraphGenerator.C:9:23 (physical line=9) (raw line:col=9:23) containsTransformationToSurroundingWhitespace == false

*[1]

5:6 SgInitializedName x isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbec20 inputCode_ASTGraphGenerator.C:7:11 (physical line=7) (raw line:col=7:11) inputCode_ASTGraphGenerator.C:7:15 (physical line=7) (raw line:col=7:15) containsTransformationToSurroundingWhitespace == false

*[1]

*[433]

decoratorList definition

27:30 SgFunctionParameterList isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fecbdc0 compiler generated compilerGenerated:0:0 (physical line=14) (raw line:col=14:1) compilerGenerated:0:0 (physical line=14) (raw line:col=14:13) is NOT output in generated code containsTransformationToSurroundingWhitespace == false

3:24 SgTemplateClassDefinition isModified = false containsTransformation = false isTransformation = false child_count:3 0x7f508f2b7010 inputCode_ASTGraphGenerator.C:4:1 (physical line=4) (raw line:col=4:1) inputCode_ASTGraphGenerator.C:11:5 (physical line=11) (raw line:col=11:5) containsTransformationToSurroundingWhitespace == false

*[0]

*[432]

26:31 SgFunctionDeclaration foo isFriend = false isForward isModified = false containsTransformation = false isTransformation = false child_count:3 0x7f508fb543a8 inputCode_ASTGraphGenerator.C:14:1 (physical line=14) (raw line:col=14:1) inputCode_ASTGraphGenerator.C:14:13 (physical line=14) (raw line:col=14:13) containsTransformationToSurroundingWhitespace == false comments/directives (before) = 1 comments/directives (inside) = 0 comments/directives (after) = 0

initptr

38:75 SgBasicBlock isModified = false containsTransformation = false isTransformation = false child_count:3 0x7f508ea1b010 inputCode_ASTGraphGenerator.C:16:4 (physical line=16) (raw line:col=16:4) inputCode_ASTGraphGenerator.C:25:4 (physical line=25) (raw line:col=25:4) containsTransformationToSurroundingWhitespace == false

initptr

*[0]

39:46 SgVariableDeclaration isAssociatedWithDeclarationList = false variableDeclarationContainsBaseTypeDefiningDeclaration = false isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:2 0x7f508efa82a8 inputCode_ASTGraphGenerator.C:17:6 (physical line=17) (raw line:col=17:6) inputCode_ASTGraphGenerator.C:17:10 (physical line=17) (raw line:col=17:10) containsTransformationToSurroundingWhitespace == false

baseTypeDefiningDeclaration

*[1]

40:45 SgInitializedName x isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbf7a8 inputCode_ASTGraphGenerator.C:17:6 (physical line=17) (raw line:col=17:6) inputCode_ASTGraphGenerator.C:17:10 (physical line=17) (raw line:col=17:10) containsTransformationToSurroundingWhitespace == false

initptr

41:44 SgAssignInitializer isModified = false containsTransformation = false isTransformation = false lvalue = false child_count:1 0x7f508e920010 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) inputCode_ASTGraphGenerator.C:17:14 (physical line=17) (raw line:col=17:14) IS output in generated code containsTransformationToSurroundingWhitespace == false

operand_i

42:43 SgIntVal isModified = false containsTransformation = false isTransformation = false value = 1 lvalue = false child_count:0 0x7f508e953010 inputCode_ASTGraphGenerator.C:17:14 (physical line=17) (raw line:col=17:14) inputCode_ASTGraphGenerator.C:17:14 (physical line=17) (raw line:col=17:14) containsTransformationToSurroundingWhitespace == false

*[1]

*[2]

47:50 SgVariableDeclaration isAssociatedWithDeclarationList = false variableDeclarationContainsBaseTypeDefiningDeclaration = false isFriend = false !isForward isModified = false containsTransformation = false isTransformation = false child_count:2 0x7f508efa8540 inputCode_ASTGraphGenerator.C:18:6 (physical line=18) (raw line:col=18:6) inputCode_ASTGraphGenerator.C:18:10 (physical line=18) (raw line:col=18:10) containsTransformationToSurroundingWhitespace == false

baseTypeDefiningDeclaration

51:74 SgIfStmt isModified = false containsTransformation = false isTransformation = false child_count:3 0x7f508e88b010 inputCode_ASTGraphGenerator.C:21:6 (physical line=21) (raw line:col=21:6) inputCode_ASTGraphGenerator.C:24:14 (physical line=24) (raw line:col=24:14) containsTransformationToSurroundingWhitespace == false comments/directives (before) = 1 comments/directives (inside) = 0 comments/directives (after) = 0

*[1]

conditional

48:49 SgInitializedName y isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508fdbf8f0 inputCode_ASTGraphGenerator.C:18:6 (physical line=18) (raw line:col=18:6) inputCode_ASTGraphGenerator.C:18:10 (physical line=18) (raw line:col=18:10) containsTransformationToSurroundingWhitespace == false

52:57 SgExprStatement isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508e7b7010 inputCode_ASTGraphGenerator.C:21:10 (physical line=21) (raw line:col=21:10) inputCode_ASTGraphGenerator.C:21:10 (physical line=21) (raw line:col=21:10) containsTransformationToSurroundingWhitespace == false

initptr

true_body

expression

53:56 SgCastExp isModified = false containsTransformation = false isTransformation = false lvalue = false child_count:1 0x7f508e7e6010 compiler generated compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) compilerGenerated:0:0 (physical line=0) (raw line:col=0:0) IS output in generated code containsTransformationToSurroundingWhitespace == false

operand_i

54:55 SgVarRefExp isModified = false containsTransformation = false isTransformation = false name = x lvalue = false child_count:0 0x7f508e858010 inputCode_ASTGraphGenerator.C:21:10 (physical line=21) (raw line:col=21:10) inputCode_ASTGraphGenerator.C:21:10 (physical line=21) (raw line:col=21:10) containsTransformationToSurroundingWhitespace == false

false_body

58:65 SgExprStatement isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508e7b7070 inputCode_ASTGraphGenerator.C:22:9 (physical line=22) (raw line:col=22:9) inputCode_ASTGraphGenerator.C:22:14 (physical line=22) (raw line:col=22:14) containsTransformationToSurroundingWhitespace == false

66:73 SgExprStatement isModified = false containsTransformation = false isTransformation = false child_count:1 0x7f508e7b70d0 inputCode_ASTGraphGenerator.C:24:9 (physical line=24) (raw line:col=24:9) inputCode_ASTGraphGenerator.C:24:14 (physical line=24) (raw line:col=24:14) containsTransformationToSurroundingWhitespace == false

expression

expression

59:64 SgAssignOp isModified = false containsTransformation = false isTransformation = false lvalue = false child_count:2 0x7f508e780010 inputCode_ASTGraphGenerator.C:22:9 (physical line=22) (raw line:col=22:9) inputCode_ASTGraphGenerator.C:22:13 (physical line=22) (raw line:col=22:13) containsTransformationToSurroundingWhitespace == false

lhs_operand_i

60:61 SgVarRefExp isModified = false containsTransformation = false isTransformation = false name = y lvalue = true child_count:0 0x7f508e858078 inputCode_ASTGraphGenerator.C:22:9 (physical line=22) (raw line:col=22:9) inputCode_ASTGraphGenerator.C:22:9 (physical line=22) (raw line:col=22:9) containsTransformationToSurroundingWhitespace == false

rhs_operand_i

62:63 SgIntVal isModified = false containsTransformation = false isTransformation = false value = 2 lvalue = false child_count:0 0x7f508e953078 inputCode_ASTGraphGenerator.C:22:13 (physical line=22) (raw line:col=22:13) inputCode_ASTGraphGenerator.C:22:13 (physical line=22) (raw line:col=22:13) containsTransformationToSurroundingWhitespace == false

67:72 SgAssignOp isModified = false containsTransformation = false isTransformation = false lvalue = false child_count:2 0x7f508e780080 inputCode_ASTGraphGenerator.C:24:9 (physical line=24) (raw line:col=24:9) inputCode_ASTGraphGenerator.C:24:13 (physical line=24) (raw line:col=24:13) containsTransformationToSurroundingWhitespace == false

lhs_operand_i

68:69 SgVarRefExp isModified = false containsTransformation = false isTransformation = false name = y lvalue = true child_count:0 0x7f508e8580e0 inputCode_ASTGraphGenerator.C:24:9 (physical line=24) (raw line:col=24:9) inputCode_ASTGraphGenerator.C:24:9 (physical line=24) (raw line:col=24:9) containsTransformationToSurroundingWhitespace == false

rhs_operand_i

70:71 SgIntVal isModified = false containsTransformation = false isTransformation = false value = 3 lvalue = false child_count:0 0x7f508e9530e0 inputCode_ASTGraphGenerator.C:24:13 (physical line=24) (raw line:col=24:13) inputCode_ASTGraphGenerator.C:24:13 (physical line=24) (raw line:col=24:13) containsTransformationToSurroundingWhitespace == false

Figure 3.3: AST representing the source code file: inputCode ASTGraphGenerator.C.

22

CHAPTER 3. SIMPLE AST GRAPH GENERATOR

Chapter 4

AST Whole Graph Generator What To Learn From This Example This example shows how to generate and visualize the AST from any input program. This view of the AST includes all additional IR nodes and edges that form attributes to the AST, as a result this graph is not a tree. These graphs are more complex but show significantly more detail about the AST and its additional edges and attributes. Each node of the graph in figure ?? shows a node of the Intermediate Representation (IR). Each edge shows the connection of the IR nodes in memory. The generated graph shows the connection of different IR nodes to form the AST and its additional attributes (e.g types, modifiers, etc). The generation of such graphs is appropriate for very small input programs, chapter 6 shows a mechanism using PDF files that is more appropriate to larger programs (e.g. 100K lines of code). More information about generation of specialized AST graphs can be found in 5 and custom graph generation in 28. Again, a utility program, called dotGeneratorWholeASTGraph is provided within ROSE to generate detailed dot graph for input code. It is available from ROSE BUILD/exampleTranslators/DOTGenerator or ROSE INS/bin. A set of options is available to further customize what types of AST nodes to be shown or hidden. Please consult the screen output of dotGeneratorWholeASTGraph -help for details. Viewing these dot files is best done using: zgrviewer at http://zvtm.sourceforge.net/zgrviewer.html. This tool permits zooming in and out and viewing isolated parts of even very large graphs. Zgrviewer permits a more natural way of understanding the AST and its additional IR nodes than the pdf file displayed in these pages. The few lines of code used to generate the graphs can be used on any input code to better understand how the AST represents different languages and their constructs. The program in figure 4.1 calls an internal ROSE function that traverses the AST and generates an ASCII file in dot format. Figure ?? shows an tiny input code which is processed to generate a graph of the AST with its attributes, generating a dot file. The dot file is then processed using dot to generate a pdf file 4.3 (within the Makefile). Note that a similar utility program already exists within ROSE/exampleTranslators (and includes a utility to output an alternative PDF representation (suitable for larger ASTs) as well). Figure ?? (../../../tutorial/test.ps) can be found in the compile tree (in the tutorial directory) and viewed directly using any pdf or dot viewer to see more detail (zgrviewer working with the dot file directly is strongly advised). Note that AST’s can get very large, and that the additional IR nodes required to represent 23

24

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

CHAPTER 4. AST WHOLE GRAPH GENERATOR

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // // // //

B u i l d t h e DOT f i l e t o v i s u a l i z e t h e AST w i t h a t t r i b u t e s ( t y p e s , symbols , To p r o t e c t a g a i n s t b u i l d i n g g r a p h s t h a t a r e t o o l a r g e an o p t i o n i s p r o v i d e d t o bound t h e number o f IR n o d e s f o r which a graph w i l l be generated . The l a y o u t o f l a r g e r g r a p h s i s p r o h i b i t i v e l y e x p e n s i v e . c o n s t i n t MAX NUMBER OF IR NODES = 2 0 0 0 ; g e n e r a t e A s t G r a p h ( p r o j e c t , MAX NUMBER OF IR NODES ) ;

etc . ) .

return 0; }

Figure 4.1: Example source code to read an input program and generate a whole AST graph. the types, modifiers, etc, can generate visually complex graphs. ROSE contains the mechanisms to traverse these graphs and do analysis on them. In one case the number of IR nodes exceeded 27 million, an analysis was done through a traversal of the graph in 10 seconds on a desktop x86 machine (the memory requirements were 6 Gig). ROSE organizes the IR in ways that permit analysis of programs that can represent rather large ASTs. 1 2 3 4 5 6 7 8

// T r i v i a l f u n c t i o n u s e d t o g e n e r a t e graph o f AST // w i t h a l l t y p e s and a d d i t i o n a l e d g e s shown . // Graphs o f t h i s s o r t a r e l a r g e , and can be // v i e w e d u s i n g ” z g r v i e w e r ” f o r d o t f i l e s . int foo () { return 0; }

Figure 4.2: Example tiny source code used as input to generate the small AST graph with attributes. Figure 4.3 displays the individual C++ nodes in ROSE’s intermediate representation (IR). Colors and shapes are used to represent different types or IR nodes. Although only visible using zgrviewer the name of the C++ construct appears in the center of each node in the graph, with the names of the data members in each IR node as edge labels. Unique pointer values are includes and printed next to the IR node name. These graphs are the single best way to develop an intuitive understanding how language constructs are organized in the AST. In these graphs, the color yellow is used for types (SgType IR nodes), the color green is used for expressions (SgExpression IR nodes), and statements are a number of different colors and shapes to make them more recognizable. Figure 4.5 shows a graph similar to the previous graph but larger and more complex because it is from a larger code. Larger graphs of this sort are still very useful in understanding how more

25

26

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

CHAPTER 4. AST WHOLE GRAPH GENERATOR

// L a r g e r f u n c t i o n u s e d t o g e n e r a t e graph o f AST // w i t h a l l t y p e s and a d d i t i o n a l e d g e s shown . // Graphs o f t h i s s o r t a r e l a r g e , and can be // v i e w e d u s i n g ” z g r v i e w e r ” f o r d o t f i l e s . int foo ( int x ) ; int globalVar = 42; void foobar A ( ) { int a = 4; int b = a + 2; int c = b ∗ globalVar ; int x ; x = foo ( c ) ; int y = x + 2; int z = globalVar ∗ y ; }

void foobar B () { int p; int i = 4; i n t k = globalVar ∗ ( i +2); p = foo (k ) ; i n t r = ( p+2) ∗ g l o b a l V a r ; }

Figure 4.4: Example source code used as input to generate a larger AST graph with attributes. significant language constructs are organized and reference each other in the AST. Tools such as zgrviewer are essential to reviewing and understanding these graphs. Although such graphs can be visualized, in practice this is only useful for debugging small codes in the construction of custom analysis and transformation tools. The graphs for real million line applications would never be visualized. Using ROSE one can build automated tools to operate on the AST for large scale applications where visualization would not be possible.

27

28

CHAPTER 4. AST WHOLE GRAPH GENERATOR

Chapter 5

Advanced AST Graph Generation What To Learn From This Example This example shows a maximally complete representation of the AST (often in more detail that is useful). Where chapter 3 presented a ROSE-based translator which presented the AST as a tree, this chapter presents the more general representation of the graph in which the AST is embedded. The AST may be thought of as a subset of a more general graph or equivalently as an AST (a tree in a formal sense) with annotations (extra edges and information), sometimes referred to as a ‘decorated AST. We present tools for seeing all the IR nodes in the graph containing the AST, including all types (SgType nodes), symbols (SgSymbol nodes), compiler generated IR nodes, and supporting IR nodes. In general it is a specific filtering of this larger graph which is more useful to communicating how the AST is designed and internally connected. We use these graphs for internal debugging (typically on small problems where the graphs are reasonable in size). The graphs presented using these graph mechanism present all back-edges, and demonstrate what IR nodes are shared internally (typically SgType IR nodes). First a few names, we will call the AST those nodes in the IR that are specified by a traversal using the ROSE traversal (SgSimpleTraversal, etc.). We will call the graph of all IR nodes the Graph of all IR nodes. the AST is embedded in the Graph of all IR nodes. The AST is a tree, while the graph of all IR nodes typically not a tree (in a Graph Theory sense) since it typically contains cycles. We cover the visualization of both the AST and the Graph of all IR nodes.

FIXME: This chapter brings more confusions. I suggest to remove it until it is done right . -Leo

• AST graph These techniques define ways of visualizing the AST and filtering IR nodes from being represented. – Simple AST graphs – Colored AST graphs – Filtering the graph The AST graph may be generated for any subtree of the AST (not possible for the graphs of all IR nodes). Additionally runtime options permit null pointers to be ignored. . 29

FIXME: Is this true?

Removed this example newer mechanism for the whole AST graphs needs to be presented.

30

CHAPTER 5. ADVANCED AST GRAPH GENERATION • Graph of all IR nodes These techniques define the ways of visualizing the whole graph of IR nodes and is based on the memory pool traversal as a means to access all IR nodes. Even disconnected portions of the AST will be presented. – Simple graphs – Colored graphs – Filtering the graph

Chapter 6

AST PDF Generator What To Learn From This Example This example demonstrates a mechanism for generating a visualization of the AST using pdf files. A pdf file is generated and can be viewed using acroread. The format is suitable for much larger input programs than the example shown in previous chapters using dot format 4. This mechanism can support the visualization of input files around 100K lines of code. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // G e n e r a t e a PDF f i l e f o r i n t e r a c t i v e generatePDF ( ∗ p r o j e c t ) ;

e x p l o r a t i o n o f t h e AST .

return 0; }

Figure 6.1: Example source code to read an input program and generate a PDF file to represent the AST. The program in figure 6.1 calls an internal ROSE function that traverses the AST and generates an ASCI file in dot format. Figure 3.2 shows an input code which is processed to generate a graph of the AST, generating a pdf file. The pdf file is then processed using acroread to generate a GUI for viewing the AST. A standalone utility tool, called pdfGenerator is provided within ROSE. It is available from ROSE BUILD/exampleTranslators/PDFGenerator or ROSE INS/bin. Users can use it to generate AST in a pdf format from an input code. Figure 6.3 displays on the left hand side the individual C++ nodes in ROSE’s intermediate representation (IR). The page on the right hand side shows that IR nodes member data. Pointers 31

32

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

CHAPTER 6. AST PDF GENERATOR

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; }

Figure 6.2: Example source code used as input to generate the PDF file of the AST. in boxes can be clicked on to navigate the AST (or nodes in the tree hierarchy can be clicked on jump to any location in the AST. This representation shows only the IR nodes that are traversed by the standard traversal (no SgSymbol or SgType IR nodes are presented in this view of the AST). The output of this translator is shown in figure 6.3. The left hand side of the screen is a tree with click-able nodes to expand/collapse the subtrees. The right hand side of the screen is a description of the data at a particular node in the AST (the node where the user has clicked the left mouse button). This relatively simple view of the AST is useful for debugging transformation and finding information in the AST required by specific sorts of analysis. It is also useful for developing an intuitive feel for what information is in the AST, how it is organized, and where it is stored.

33

Figure 6.3: Example output from translator which outputs PDF representation of AST. The generated PDF file makes use of the bookmark mechanism to expand and collapse parts of the AST.

34

CHAPTER 6. AST PDF GENERATOR

Chapter 7

Introduction to AST Traversals An essential operation in the analysis and construction of ASTs is the definition of traversals upon the AST to gather information and modify targeted internal representation (IR) nodes. ROSE includes different sorts of traversals to address the different requirements of numerous program analysis and transformation operations. This section demonstrates the different types of traversals that are possible using ROSE. ROSE translators most commonly introduce transformations and analysis through a traversal over the AST. Alternatives could be to generate a simpler IR that is more suitable to a specific transformation and either convert modification to that transformation specific IR into changes to the AST or generate source code from the transformation specific IR directly. These approaches are more complex than introducing changes to the AST directly, but may be better for specific transformations. Traversals represent an essential operation on the AST and there are a number of different types of traversals. The suggested traversals for users are explained in Section 7.2. Section 7.3 introduces specialized traversals (that traverse the AST in different orders and traverse types and symbols), typically not appropriate for most translators (but perhaps appropriate for specialized tools, often internal tools within ROSE). See the ROSE User Manual for a more complete introduction to the different types of traversals. The purpose of this tutorial is to present examples, but we focus less on the background and philosophy here than in the ROSE User Manual. This chapter presents a number of ways of traversing the AST of any input source code. These traversals permit operations on the AST, which may either read or modify the AST in place. Modifications to the AST will be reflected in the source code generated when the AST is unparsed; the code generation phase of the source-to-source process defined by ROSE. Note that for all examples, the input code described in section 7.1 is used to generate all outputs shown with each translator.

7.1

Input For Example Traversals

The code shown in figure 7.1 shows the input code that will be used to demonstrate the traversals in this chapter. It may be modified by the user to experiment with the use of the traversals on 35

FIXME: Add What to learn from this example paragraph to each example. FIXME: Add What is different from previous example paragraph to each example. FIXME: Add a table and/or graph at the end of this chapter to summarize the traversals.

36

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ;

for

t e s t i n g overloaded function

resolution

void foo ( double ) { int x = 1; int y ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 7.1: Example source code used as input to program in traversals shown in this chapter. alternative input codes.

7.2

Traversals of the AST Structure

This collection of traversals operates on the AST in an order which matches the structure of the AST and the associated source code. These types of traversals are the most common traversals for users to use. A subsequent section of this chapter demonstrated more specialized traversals over all IR nodes (more than just those IR nodes in the AST representing the structure of the source code) that are suitable for some tools, mostly tools built internally within ROSE.

7.2. TRAVERSALS OF THE AST STRUCTURE

37

Because the traversals in this section traverse the structure of the source code (see the AST graph presented in the first tutorial example) they are more appropriate for most transformations of the source code. We suggest that the user focus on these traversals which represent the interface we promote for analysis and transformation of the AST, instead of the memory pools traversals which are suitable mostly for highly specialized internal tools. The simple traversals of both kinds have the same interface so the user may easily switch between them with out significant difficulty.

7.2.1

Classic Object-Oriented Visitor Pattern for the AST

We show this example first, but it is rarely used in practice, and more useful traversals follow. It is however most closely similar to traversals that are available in other compiler infrastructures, and so a concept with which many people will be familar. In this case because this implementation is based on the memory pool infrstructure it will visit all node and not in any order based on the AST. The ASTSimpleProcessing traversal in section 7.2.2 is closer to a common visitor pattern that visits the IR nodes in the order in which they appear in the AST. Figure 7.2 shows the source code for a translator using the classic object-oriented visitor pattern to traverse the AST. This visitor pattern is only implemented for the memory pool based traversal. Thus it works on the whole of the attributed AST and does not work on a restricted subset of the AST (e.g. a subtree of the unattributed AST). Figure 7.3 shows the output from this traversal using the example input source from figure 7.1.

7.2.2

Simple Traversal (no attributes)

Figure 7.4 shows the source code for a translator which traverses the AST. The traversal object is from the type visitorTraversal derived from AstSimpleProcessing. The visit() function is required to be defined because it is defined as a pure virutal funciton in the AstSimpleProcessing base class. The member function traverseInputFiles() of AstSimpleProcessing is called to traverse the AST and call the visit() function on each IR node. Note that the function traverse() (not used) would visit each IR nodes while traverseInputFiles() will only visit those IR nodes that originate in the input source code (thus skipping all header files). For each node where the visit() function is called a SgNode pointer is to the node is passed into the visit function. Note that using this simple traversal the only context information available to the visit function is what is stored in its member variables (though access to other nodes is possible along any edges in the attributed AST graph). The only option is to traverse the AST in either pre-order or postorder. The atTraversalEnd() function may be defined by the user to do final processing after all nodes have been visited (or to perform preparations before the nodes are visited, in the case of the corresponding atTraversalStart() function). Figure 7.5 shows the output from this traversal using the example input source from figure 7.1.

7.2.3

Simple Pre- and Postorder Traversal

Figure 7.6 shows the source code for a translator that traverses the AST without attributes (like the one in the previous subsection), but visiting each node twice, once in preorder (before its children) and once in postorder (after all children). Figure 7.7 shows the output from this traversal using the example input source from figure 7.1.

38

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

#i n c l u d e ” r o s e . h” // C l a s s i c V i s i t o r P a t t e r n i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t h as no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s C l a s s i c V i s i t o r : p u b l i c ROSE VisitorPattern { public : // O v e r r i d e v i r t u r a l f u n c t i o n d e f i n e d i n b a s e c l a s s void v i s i t ( SgGlobal ∗ g l o b a l S c o p e ) { p r i n t f ( ” Found t h e S g G l o b a l IR node \n ” ) ; } void v i s i t ( SgFunctionDeclaration ∗ functionDeclaration ) { p r i n t f ( ” Found a S g F u n c t i o n D e c l a r a t i o n IR node \n ” ) ; } v o i d v i s i t ( SgTypeInt ∗ i n t T y p e ) { p r i n t f ( ” Found a SgTypeInt IR node \n ” ) ; } v o i d v i s i t ( SgTypeDouble ∗ doubleType ) { p r i n t f ( ” Found a SgTypeDouble IR node \n ” ) ; } };

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // C l a s s i c v i s i t o r p a t t e r n o v e r t h e memory p o o l o f IR n o d e s ClassicVisitor visitor A ; traverseMemoryPoolVisitorPattern ( v i s i t o r A ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 7.2: Example source showing simple visitor pattern.

7.2.4

Inherited Attributes

Figure 7.8 shows the use of inherited attributes associated with each IR node. Within this traversal the attributes are managed by the traversal and exist on the stack. Thus the lifetime of the attributes is only as long as the processing of the IR node and its subtree. Attributes such as this are used to communicate context information down the AST and called Inherited attributes. In the example the class Inherited Attribute is used to represent inherited attribute. Each instance of the class represents an attribute value. When the AST is traversed we obtain as output the loop nest depth at each point in the AST. The output uses the example input

7.2. TRAVERSALS OF THE AST STRUCTURE

39

source from figure 7.1. Note that inherited attributes are passed by-value down the AST. In very rare cases you might want to pass a pointer to dynamically allocated memory as an inherited attribute. In this case you can define the virtual member function void destroyInheritedValue(SgNode *n, InheritedAttribute inheritedValue) which is called after the last use of the inherited attribute computed at this node, i. e. after all children have been visited. You can use this function to free the memory allocated for this inherited attribute.

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

a SgTypeInt IR node a SgTypeDouble IR node t h e S g G l o b a l IR node t h e S g G l o b a l IR node a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration

IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR

node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” class {

visitorTraversal

:

public AstSimpleProcessing

public : visitorTraversal (); v i r t u a l v o i d v i s i t ( SgNode∗ n ) ; v i r t u a l void atTraversalEnd ( ) ; }; visitorTraversal : : visitorTraversal () { } v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” Found a f o r l o o p . . . \n ” ) ; } } void v i s i t o r T r a v e r s a l : : atTraversalEnd ( ) { p r i n t f ( ” T r a v e r s a l e n d s h e r e . \n ” ) ; } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; if

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( ” I n v i s i t o r T r a v e r s a l . C : main ( ) \n ” ) ;

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l f u n c t i o n ( member f u n c t i o n o f A s t S i m p l e P r o c e s s i n g ) // s t a r t i n g a t t h e p r o j e c t node o f t h e AST, u s i n g a p r e o r d e r t r a v e r s a l . exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 7.4: Example source showing simple visitor pattern.

1 2 3

41

Found a f o r l o o p . . . Found a f o r l o o p . . . T r a v e r s a l ends here .

Figure 7.5: Output of input file to the visitor traversal.

42

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” c l a s s PreAndPostOrderTraversal : p u b l i c AstPrePostProcessing { public : v i r t u a l v o i d p r e O r d e r V i s i t ( SgNode∗ n ) ; v i r t u a l v o i d p o s t O r d e r V i s i t ( SgNode∗ n ) ; }; v o i d P r e A n d P o s t O r d e r T r a v e r s a l : : p r e O r d e r V i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” E n t e r i n g f o r l o o p . . . \n ” ) ; } } v o i d P r e A n d P o s t O r d e r T r a v e r s a l : : p o s t O r d e r V i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” L e a v i n g f o r l o o p . . . \n ” ) ; } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; if

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( ” I n p r e P o s t T r a v e r s a l . C : main ( ) \n ” ) ;

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t PreAndPostOrderTraversal exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t ) ; return 0; }

Figure 7.6: Example source showing simple pre- and postorder pattern.

1 2 3 4

Entering f o r loop . . . Leaving f o r loop . . . Entering f o r loop . . . Leaving f o r loop . . .

Figure 7.7: Output of input file to the pre- and postorder traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

43

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” // B u i l d an i n h e r i t e d a t t r i b u t e class InheritedAttribute { public : // Depth i n AST i n t depth ; i n t maxLinesOfOutput ;

f o r the t r e e

t r a v e r s a l t o t e s t t h e r e w r i t e mechanism

// S p e c i f i c c o n s t r u c t o r s a r e r e q u i r e d I n h e r i t e d A t t r i b u t e ( i n t x ) : depth ( x ) , maxLinesOfOutput ( 2 0 ) { } ; I n h e r i t e d A t t r i b u t e ( c o n s t I n h e r i t e d A t t r i b u t e & X ) : depth (X . depth ) , maxLinesOfOutput ( 2 0 ) { } ; }; class {

visitorTraversal public : // v i r t u a l virtual

:

p u b l i c AstTopDownProcessing

f u n c t i o n must be d e f i n e d I n h e r i t e d A t t r i b u t e e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode∗ n ,

InheritedAttribute

}; InheritedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode∗ n , I n h e r i t e d A t t r i b u t e i n h e r i t e d A t t r i b u t e ) { s t a t i c int linesOfOutput = 0; i f ( l i n e s O f O u t p u t++ < i n h e r i t e d A t t r i b u t e . maxLinesOfOutput ) p r i n t f ( ” Depth i n AST a t %s = %d \n ” , n−>s a g e c l a s s n a m e ( ) , i n h e r i t e d A t t r i b u t e . depth ) ; r e t u r n I n h e r i t e d A t t r i b u t e ( i n h e r i t e d A t t r i b u t e . depth + 1 ) ; } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // DQ ( 1 / 1 8 / 2 0 0 6 ) : P a r t o f d e b u g g i n g S g F i l e & l o c a l F i l e = p r o j e c t −> g e t f i l e ( 0 ) ; l o c a l F i l e . g e t f i l e i n f o ()−> d i s p l a y ( ” l o c a l F i l e

information ”);

// B u i l d t h e i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute (0); // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , i n h e r i t e d A t t r i b u t e ) ; // Or t h e t r a v e r s a l o v e r a l l AST IR n o d e s can be c a l l e d ! exampleTraversal . t r a v e r s e ( project , i n h e r i t e d A t t r i b u t e ) ; return 0; }

Figure 7.8: Example source code showing use of inherited attributes (passing context information down the AST.

inheritedAttribute );

44

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

Inside of Sg File Info : : display ( l o c a l F i l e information ) of this pointer = 0 x7f3f38d5f068 isTransformation = false isCompilerGenerated = false isOutputInCodeGeneration = false isShared = false isFrontendSpecific = false isSourcePositionUnavailableInFrontend = f a l s e isCommentOrDirective = false isToken = false isDefaultArgument = false isImplicitCast = false f i l e n a m e = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m line = 1 column = 1 = 0 = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i physical file id physical line = 1 source sequence number = 0 Depth i n AST a t S g S o u r c e F i l e = 0 Depth i n AST a t S g G l o b a l = 1 Depth i n AST a t S g T e m p l a t e C l a s s D e c l a r a t i o n = 2 Depth i n AST a t S g T e m p l a t e C l a s s D e f i n i t i o n = 3 Depth i n AST a t S g V a r i a b l e D e c l a r a t i o n = 4 Depth i n AST a t S g I n i t i a l i z e d N a m e = 5 Depth i n AST a t S g T e m p l a t e M e m b e r F u n c t i o n D e c l a r a t i o n = 4 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 5 Depth i n AST a t S g I n i t i a l i z e d N a m e = 6 Depth i n AST a t S g C t o r I n i t i a l i z e r L i s t = 5 Depth i n AST a t S g T e m p l a t e M e m b e r F u n c t i o n D e c l a r a t i o n = 4 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 5 Depth i n AST a t S g I n i t i a l i z e d N a m e = 6 Depth i n AST a t S g C t o r I n i t i a l i z e r L i s t = 5 Depth i n AST a t S g F u n c t i o n D e c l a r a t i o n = 2 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 3 Depth i n AST a t S g I n i t i a l i z e d N a m e = 4 Depth i n AST a t S g F u n c t i o n D e c l a r a t i o n = 2 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 3 Depth i n AST a t S g I n i t i a l i z e d N a m e = 4

Figure 7.9: Output of input file to the inherited attribute traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

7.2.5

45

Synthesized Attributes

Figure 7.10 shows the use of attributes to pass information up the AST. The lifetime of the attributes are similar as for inherited attributes. Attributes such as these are called synthesized attributes. This code shows the code for a translator which does an analysis of an input source code to determine the presence of loops. It returns true if a loop exists in the input code and false otherwise. The list of synthesized attributes representing the information passed up the AST from a node’s children is of type SynthesizedAttributesList, which is a type that behaves very similarly to vector (it supports iterators, can be indexed, and can be used with STL algorithms). The example determines the existence of loops for a given program.

46

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” #i n c l u d e #i n c l u d e #i n c l u d e typedef bool SynthesizedAttribute ; class {

visitorTraversal public : // v i r t u a l virtual

:

p u b l i c AstBottomUpProcessing

f u n c t i o n must be d e f i n e d SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode∗ n , S y n t h e s i z e d A t t r i b u t e s L i s t c h i l d A t t r i b u t e s

);

}; SynthesizedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode∗ n , S y n t h e s i z e d A t t r i b u t e s L i s t { // Fold up t h e l i s t o f c h i l d a t t r i b u t e s u s i n g l o g i c a l or , i . e . t h e l o c a l // r e s u l t w i l l be t r u e i f f one o f t h e c h i l d a t t r i b u t e s i s t r u e . SynthesizedAttribute localResult = s t d : : a c c u m u l a t e ( c h i l d A t t r i b u t e s . b e g i n ( ) , c h i l d A t t r i b u t e s . end ( ) , f a l s e , s t d : : l o g i c a l o r ( ) ) ; if

( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” Found a f o r l o o p . . . localResult = true ; }

childAttributes )

\n ” ) ;

return localResult ; } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST SynthesizedAttribute r e s u l t = exampleTraversal . t r a v e r s e ( p r o j e c t ) ; if

( r e s u l t == t r u e ) { p r i n t f ( ” The program c o n t a i n s a t l e a s t one l o o p ! \ n ” ) ; }

return 0; }

Figure 7.10: Example source code showing use of synthesized attributed (passing analysis information up the AST).

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3

Found a f o r l o o p . . . Found a f o r l o o p . . . The program c o n t a i n s a t l e a s t one l o o p !

Figure 7.11: Output of input file to the synthesized attribute traversal.

47

48

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.2.6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

Accumulator Attributes

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” // B u i l d an a c c u m u l a t o r a t t r i b u t e , f a n c y name f o r what i s c l a s s AccumulatorAttribute { public : i n t forLoopCounter ;

essentially a global

v a r i a b l e : −).

// S p e c i f i c c o n s t r u c t o r s a r e o p t i o n a l AccumulatorAttribute ( ) { forLoopCounter = 0 ; } A c c u m u l a t o r A t t r i b u t e ( c o n s t A c c u m u l a t o r A t t r i b u t e & X ) {} A c c u m u l a t o r A t t r i b u t e & o p e r a t o r= ( c o n s t A c c u m u l a t o r A t t r i b u t e & X ) { r e t u r n ∗ t h i s ; } }; class {

visitorTraversal

:

public AstSimpleProcessing

public : s t a t i c AccumulatorAttribute accumulatorAttribute ; v i r t u a l v o i d v i s i t ( SgNode∗ n ) ; }; // d e c l a r a t i o n r e q u i r e d f o r s t a t i c d a t a member AccumulatorAttribute v i s i t o r T r a v e r s a l : : accumulatorAttribute ; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” Found a f o r l o o p . . . \n ” ) ; a c c u m u l a t o r A t t r i b u t e . f o r L o o p C o u n t e r ++; } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST // can be s p e c i f i e d t o be p r e o r d e r o r p o s t o r d e r ) . exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; p r i n t f ( ” Number o f

f o r l o o p s i n i n p u t a p p l i c a t i o n = %d \n ” , e x a m p l e T r a v e r s a l . a c c u m u l a t o r A t t r i b u t e . f o r L o o p C

return 0; }

Figure 7.12: Example source code showing use of accumulator attributes (typically to count things in the AST). Figure 7.12 shows the use of a different sort of attribute. This attribute has a lifetime equal

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3

49

Found a f o r l o o p . . . Found a f o r l o o p . . . Number o f f o r l o o p s i n i n p u t a p p l i c a t i o n = 2

Figure 7.13: Output of input file to the accumulator attribute traversal. to the lifetime of the traversal object (much longer than the traversal of any subset of IR nodes). The same attribute is accessible from each IR node. Such attributes are called accumulator attributes and are semantically equivalent to a global variable. Accumulator attributes act as global variables which can easily be used to count application specific properties within the AST. Note that due to the limitation that the computation of inherited attributes cannot be made dependent on the values of synthesized attributes, counting operations cannot be implemented by combining these attributes as is usually done in attribute grammars. However, the use of accumulator attributes serves well for this purpose. Therefore all counting-like operations should be implemented using accumulator attributes (= member variables of traversal or processing classes). Although not shown in this tutorial explicitly, accumulator attributes may be easily mixed with inherited and/or synthesized attributes. In this example we count the number of for-loops in an input program.

7.2.7

Inherited and Synthesized Attributes

Figure 7.14 shows the combined use of inherited and synthesized attributes. The example source code shows the mixed use of such attributes to list the functions containing loop. Inherited attributes are used to communicate that the traversal is in a function, which the synthesized attributes are used to pass back the existence of loops deeper within the subtrees associated with each function. List of functions containing loops.

50 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

// ROSE i s #i n c l u d e

a

tool

for

building

preprocessors ,

this

file

is

an e x a m p l e

preprocessor

built

w i t h ROSE .

” r o s e . h”

#i n c l u d e
#i n c l u d e #i n c l u d e typedef typedef class {

bool bool

InheritedAttribute ; SynthesizedAttribute ;

Traversal

:

public

SgTopDownBottomUpProcessing

public : // F u n c t i o n s r e q u i r e d InheritedAttribute evaluateInheritedAttribute SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute );

(

SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute , SubTreeSynthesizedAttributes synthesizedAttributeList

);

}; InheritedAttribute Traversal : : evaluateInheritedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute ) { i f ( i s S g F u n c t i o n D e f i n i t i o n ( astNode ) ) { // The i n h e r i t e d a t t r i b u t e i s t r u e i f f return true ; } return inheritedAttribute ; }

we a r e

inside

a

function .

SynthesizedAttribute Traversal : : evaluateSynthesizedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute , SynthesizedAttributesList childAttributes ) { i f ( i n h e r i t e d A t t r i b u t e == f a l s e ) { // The i n h e r i t e d a t t r i b u t e i s f a l s e , i . e . we a r e n o t i n s i d e any // f u n c t i o n , s o t h e r e c a n be no l o o p s h e r e . return f a l s e ; } else { // F o l d up t h e l i s t o f c h i l d a t t r i b u t e s u s i n g l o g i c a l o r , i . e . t h e // r e s u l t w i l l be t r u e i f f o n e o f t h e c h i l d a t t r i b u t e s i s t r u e . SynthesizedAttribute localResult = s t d : : a c c u m u l a t e ( c h i l d A t t r i b u t e s . b e g i n ( ) , c h i l d A t t r i b u t e s . end ( ) , f a l s e , s t d : : l o g i c a l o r ( ) ) ; i f ( i s S g F u n c t i o n D e f i n i t i o n ( a s t N o d e ) && l o c a l R e s u l t == t r u e ) { p r i n t f ( ” Found a f u n c t i o n c o n t a i n i n g a f o r l o o p . . . \ n ” ) ; } i f ( isSgForStatement ( astNode ) ) { localResult = true ; } return localResult ; } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

See

Rose : : i n i t i a l i z e

//

Build the a b s t r a c t syntax t r e e SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ;

//

Build the i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute = false ;

//

Define the t r a v e r s a l T r a v e r s a l myTraversal ;

//

C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t ( r o o t ) node o f t h e AST myTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t , i n h e r i t e d A t t r i b u t e ) ;

//

T h i s program return 0;

only

does

analysis ,

so

it

local

need

not

call

the

backend

to

generate

code .

}

Figure 7.14: Example source code showing use of both inherited and synthesized attributes working together (part 1).

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2

Found a f u n c t i o n c o n t a i n i n g a f o r l o o p Found a f u n c t i o n c o n t a i n i n g a f o r l o o p

... ...

Figure 7.15: Output of input file to the inherited and synthesized attribute traversal.

51

52

7.2.8

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

Persistent Attributes

Figure 7.16 shows the use of another form of attribute. This attribute has a lifetime which is controlled explicitly by the user; it lives on the heap typically. These attributes are explicitly attached to the IR nodes and are not managed directly by the traversal. There attributes are called persistent attributes and are not required to be associated with any traversal. Persistent attributes are useful for storing information across multiple traversals (or permanently within the AST) for later traversal passes. Persistent attributes may be used at any time and combined with other traversals (similar to accumulator attributes). Traversals may combine any or all of the types of attributes within in ROSE as needed to store, gather, or propagate information within the AST for complex program analysis.

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

53

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” class {

persistantAttribute

:

public AstAttribute

public : int value ; p e r s i s t a n t A t t r i b u t e ( i n t v ) : v a l u e ( v ) {} }; class {

visitorTraversalSetAttribute public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l S e t A t t r i b u t e : : v i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” Found a f o r l o o p ( s e t t h e a t t r i b u t e )

...

\n ” ) ;

// B u i l d an a t t r i b u t e ( on t h e heap ) A s t A t t r i b u t e ∗ n e w A t t r i b u t e = new p e r s i s t a n t A t t r i b u t e ( 5 ) ; ROSE ASSERT( n e w A t t r i b u t e != NULL ) ; // Add i t t o t h e AST ( s o i t can be f o u n d l a t e r i n a n o t h e r p a s s o v e r t h e AST) n−>addNewAttribute ( ” MyNewAttribute ” , n e w A t t r i b u t e ) ; } } class {

visitorTraversalReadAttribute public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l R e a d A t t r i b u t e : : v i s i t ( SgNode∗ n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( ” Found a f o r l o o p ( r e a d t h e a t t r i b u t e ) // // // // //

...

\n ” ) ;

Add i t t o t h e AST ( s o i t can be f o u n d l a t e r i n a n o t h e r p a s s o v e r t h e AST) A s t A t t r i b u t e ∗ e x i s t i n g A t t r i b u t e = n−>a t t r i b u t e [ ” MyNewAttribute ” ] ; DQ ( 1 / 2 / 2 0 0 6 ) : Added s u p p o r t f o r new a t t r i b u t e i n t e r f a c e . p r i n t f ( ” v i s i t o r T r a v e r s a l R e a d A t t r i b u t e : : v i s i t ( ) : u s i n g new a t t r i b u t e i n t e r f a c e \n ” ) ; A s t A t t r i b u t e ∗ e x i s t i n g A t t r i b u t e = n−>a t t r i b u t e ( ) [ ” MyNewAttribute ” ] ; A s t A t t r i b u t e ∗ e x i s t i n g A t t r i b u t e = n−>g e t A t t r i b u t e ( ” MyNewAttribute ” ) ; ROSE ASSERT( e x i s t i n g A t t r i b u t e != NULL ) ; p r i n t f (” Existing

a t t r i b u t e a t %p v a l u e = %d \n ” , n , d y n a m i c c a s t

( e x i s t i n g A t t r i b u t e )−> v a l u e

} } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t t o s e t p e r s i s t a n t AST a t t r i b u t e s visitorTraversalSetAttribute exampleTraversalSettingAttribute ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversalSettingAttribute . traverseInputFiles ( project , preorder ) ; // B u i l d t h e t r a v e r s a l o b j e c t t o r e a d any e x i s t i n g AST a t t r i b u t e s visitorTraversalReadAttribute exampleTraversalReadingAtribute ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversalReadingAtribute . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

54

1 2 3 4 5 6

Found a Found a Found a Existing Found a Existing

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

f o r loop ( s e t the a t t r i b u t e ) . . . f o r loop ( s e t the a t t r i b u t e ) . . . f o r loop ( read the a t t r i b u t e ) . . . a t t r i b u t e at 0 x7f168e717010 value = 5 f o r loop ( read the a t t r i b u t e ) . . . a t t r i b u t e at 0 x7f168e717138 value = 5

Figure 7.17: Output of input file to the persistent attribute traversal showing the passing of information from one AST traversal to a second AST traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

7.2.9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

55

Nested Traversals

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” class {

visitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; class {

nestedVisitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { i f ( i s S g F u n c t i o n D e c l a r a t i o n ( n ) != NULL) { p r i n t f ( ” Found a f u n c t i o n d e c l a r a t i o n

...

\n ” ) ;

// B u i l d t h e n e s t e d t r a v e r s a l o b j e c t ne s t e dV i si t o r Tr a v er s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST ( t r a v e r s e i n p o s t o r d e r j u s t t o be d i f f e r e n t ) // Note t h a t we c a l l t h e t r a v e r s e f u n c t i o n i n s t e a d o f t r a v e r s e I n p u t F i l e s , b e c a u s e we a r e n o t s t a r t i n g a t // t h e AST r o o t . exampleTraversal . t r a v e r s e (n , postorder ) ; } } v o i d n e s t e d V i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { i f ( i s S g F u n c t i o n D e f i n i t i o n ( n ) != NULL) { p r i n t f ( ” Found a f u n c t i o n d e f i n i t i o n } }

within the f u n c t i o n

declaration

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; if

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( ” I n v i s i t o r T r a v e r s a l . C : main ( ) \n ” ) ;

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 7.18: Example source code showing use nested traversals.

...

\n ” ) ;

56

1 2 3 4 5 6 7 8 9

Found Found Found Found Found Found Found Found Found

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

a a a a a a a a a

function function function function function function function function function

declaration . . . declaration . . . declaration . . . declaration . . . d e f i n i t i o n within the f u n c t i o n declaration . . . declaration . . . declaration . . . d e f i n i t i o n within the f u n c t i o n

declaration

...

declaration

...

Figure 7.19: Output of input file to the nested traversal example. Figure 7.18 shows the use of multiple traversals in composition. Figure 7.19 shows the output of the nested traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

7.2.10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

57

Combining all Attributes and Using Primitive Types

i n t main ( ) { i n t x =1; f o r ( i n t i =1; i <10; i ++) f o r ( i n t j=i ; j <10; j ++) f o r ( i n t k=i ; k <10; k++) f o r ( i n t l=i ; l <10; l ++) f o r ( i n t m=i ;m<10;m++) x++; i n t i =5 , j =7; w h i l e ( i >0) { w h i l e ( j >0) { x++; j −−; i −−; } } i =10; do { x++; i −−; } w h i l e ( i > 0); return x ; }

Figure 7.20: Input code with nested loops for nesting info processing The previous examples have shown cases where attributes were classes, alternatively attributes can be any primitive type (int, bool, etc.). This example demonstrates how to use AstTopDownBottomUpProcessing to compute inherited and synthesized attributes, generate pdf and dot output, how to accumulate information, and how to attach attributes to AST nodes in the same pass. The attributes are used to compute the nesting level and the nesting depth of for/while/dowhile loops: The nesting level is computed using an inherited attribute. It holds that nesting − level(innerloop) = nesting − level(outerloop) + 1 starting with 1 at the outer most loop. The nesting depth is computed using a synthesized attribute. It holds that nesting−depth(innerloop) = nesting − level(outerloop) − 1 starting with 1 at the inner most loop. To compute the values we use a primitive type (unsigned int). This example also shows how to use defaultSynthesizedAttribute to initialize a synthesized attribute of primitive type. The values of the attributes are attached to the AST using AstAttribute and the AST node attribute mechanism available at every AST node (which can be accessed with node->attribute). (see loopNestingInfoProcessing.C) For the entire program the maximum nesting level (= max nesting depth) is computed as accumulated value using member variable _maxNestingLevel of class LoopNestingInfoProcessing. We also demonstrate how to customize an AstAttribute such that the value of the attribute is printed in a pdf output. (by overriding toString, see LoopNestingInfo class) In the generated pdf file (for some C++ input file) the values of the attributes can be viewed for each node (see printLoopInfo implementation). Further more we also generate a dot file, to visualize the tree using the graph visualization tool dot. The generated file can be converted to

58

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

postscript (using dot) and viewed with gv.

7.2.11

Combined Traversals

Performing a large number of program analyses as separate traversals of the AST can be somewhat inefficient as there is some overhead associated with visiting every node several times. ROSE therefore provides a mechanism for combining traversal objects of the same base type and evaluating them in a single traversal of the AST. This is entirely transparent to the individual traversal object, so existing code can be reused with the combination mechanism, and analyzers can be developed and tested in isolation and combined when needed. The one requirement that is placed on traversals to be combined is that they be independent of each other; in particular, this means that they should not modify the AST or any shared global data. Any output produced by the analyzers will be interleaved. Figure 7.24 shows the source code for a translator that combines three different analyzers into one traversal, each one counting the occurrences of a different type of AST node (as determined by a VariantT value). First three traversals are run after each other, as usual; then three traversal objects are passed (by pointer) to an object of type AstCombinedSimpleProcessing using its addTraversal method. One then invokes one of the usual traverse methods on this combined object with the same effect as if it had been called for each of the traversal objects individually. Any operation on the list of analyzers is possible using the get traversalPtrListRef method of the combined processing class that returns a reference to its internal list of analyzers (an object of type vector). Any changes made through this reference will be reflected in further traversals. In addition to AstCombinedSimpleProcessing, there is also a combined class for each of the other types of traversals discussed above: AstCombinedTopDownProcessing, AstCombinedBottomUpProcessing, etc. Where traversals using attributes are combined, all of the combined traversals must have the same attribute types (i. e. the same template parameters). Attributes are passed to and returned from the combined traversal as a vector.

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

59

// Author : Markus Schordan , Vienna U n i v e r s i t y o f Technology , 2 0 0 4 . // $ I d : l o o p N e s t i n g I n f o P r o c e s s i n g . C, v 1 . 1 2 0 0 6 / 0 4 / 2 4 0 0 : 2 2 : 0 0 d q u i n l a n Exp $ // #i n c l u d e // #i n c l u d e #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; typedef typedef typedef typedef

unsigned i n t unsigned i n t NestingLevel NestingDepth

NestingLevel ; NestingDepth ; InhNestingLevel ; SynNestingDepth ;

/ ∗ ! T h i s c l a s s i s u s e d t o a t t a c h i n f o r m a t i o n t o AST n o d e s . Method ’ t o S t r i n g ’ i s o v e r r i d d e n and c a l l e d when a p d f f i l e i s g e n e r a t e d . T h i s a l l o w s t o d i s p l a y t h e v a l u e o f an AST node a t t r i b u t e ( a n n o t a t i o n ) i n a p d f f i l e . ∗/ c l a s s NestingLevelAnnotation : public AstAttribute { public : NestingLevelAnnotation ( N e s t i n g L e v e l n , NestingDepth d ) n e s t i n g L e v e l ( n ) , n e s t i n g D e p t h ( d ) {} : NestingLevel getNestingLevel () { return nestingLevel ; } NestingDepth getNestingDepth ( ) { r e t u r n nestingDepth ; } string toString () { o s t r i n g s t r e a m s s ; s s << n e s t i n g L e v e l <<”,”<< n e s t i n g D e p t h ; return ss . str ( ) ; } private : nestingLevel ; NestingLevel NestingDepth nestingDepth ; }; /∗ ! The l o o p n e s t i n g l e v e l and n e s t i n g depth f o r e a c h w h i l e / d o w h i l e / f o r l o o p n e s t i s computed . I t i s a t t a c h e d t o t h e AST a s a n n o t a t i o n and can be a c c e s s e d a s node−>a t t r i b u t e [ ” l o o p N e s t i n g I n f o ” ] a f t e r t h e p r o c e s s i n g has been p e r f o r m e d . The maximum n e s t i n g l e v e l o f t h e whole AST i s computed a s ” a c c u m u l a t e d ” v a l u e i n a member v a r i a b l e and can be a c c e s s e d w i t h getMaxNestingLevel ( ) . ∗/ c l a s s L o o p L e v e l P r o c e s s i n g : p u b l i c AstTopDownBottomUpProcessing { public : L o o p L e v e l P r o c e s s i n g ( ) : m a x N e s t i n g L e v e l ( 0 ) {} / ∗ ! P e r f o r m s a t r a v e r s a l o f t h e AST and computes l o o p −n e s t i n g i n f o r m a t i o n by u s i n g i n h e r i t e d and s y n t h e s i z e d a t t r i b u t e s . The r e s u l t s a r e a t t a c h e d t o t h e AST a s annotation . ∗/ v o i d a t t a c h L o o p N e s t i n g A n n o t a t o n ( S g P r o j e c t ∗ node ) { t r a v e r s e I n p u t F i l e s ( node , 0 ) ; } / ∗ ! R e t u r n s t h e maximum n e s t i n g l e v e l o f t h e e n t i r e AST ( o f t h e i n p u t R e q u i r e s a t t a c h L o o p N e s t i n g A n n o t a t i o n ( t o be c a l l e d b e f o r e ) ∗/ NestingLevel getMaxNestingLevel ( ) { return maxNestingLevel ; }

file ).

protected : / / ! computes t h e n e s t i n g l e v e l I n h N e s t i n g L e v e l e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode ∗ , I n h N e s t i n g L e v e l ) ; / / ! computes t h e n e s t i n g depth SynNestingDepth e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode ∗ , I n h N e s t i n g L e v e l , S y n t h e s i z e d A t t r i b u t e s L i s t ) ; / / ! p r o v i d e s t h e d e f a u l t v a l u e 0 f o r t h e n e s t i n g depth SynNestingDepth d e f a u l t S y n t h e s i z e d A t t r i b u t e ( I n h N e s t i n g L e v e l i n h ) ; private : NestingLevel maxNestingLevel ; };

NestingLevel L o o p L e v e l P r o c e s s i n g : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode∗ node ,

NestingLevel loopNestingLevel ) {

/ / ! compute maximum n e s t i n g l e v e l o f e n t i r e program i n a c c u m u l a t o r ( member v a r i a b l e ) i f ( loopNestingLevel > maxNestingLevel ) m a x N e s t i n g L e v e l=l o o p N e s t i n g L e v e l ; s w i t c h ( node−>v a r i a n t T ( ) ) { c a s e V SgGotoStatement :

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

c o u t << ”WARNING: Goto s t a t e m e n t f o u n d . We do n o t c o n s i d e r g o t o l o o p s . \ n ” ; // DQ ( 1 1 / 1 7 / 2 0 0 5 ) : Added r e t u r n s t a t m e n t t o a v o i d g++ w ar n in g : c o n t r o l r e a c h e s end o f non−v o i d f u n c t i o n return loopNestingLevel ; break ; c a s e V SgDoWhileStmt : c a s e V SgForStatement : c a s e V SgWhileStmt : r e t u r n l o o p N e s t i n g L e v e l +1; default : return loopNestingLevel ; } } SynNestingDepth LoopLevelProcessing : : d e f a u l t S y n t h e s i z e d A t t r i b u t e ( InhNestingLevel inh ) { / ∗ ! we do n o t need t h e i n h e r i t e d a t t r i b u t e h e r e a s d e f a u l t v a l u e f o r s y n t h e s i z e d a t t r i b u t e we s e t 0 , r e p r e s e n t i n g n e s t i n g depth 0 . ∗/ return 0; }

SynNestingDepth L o o p L e v e l P r o c e s s i n g : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode∗ node , I n h N e s t i n g L e v e l n e s t i n g L e v e l , S y n t h e s i z e d A t t r i b u { i f ( nestingLevel > maxNestingLevel ) m a x N e s t i n g L e v e l=n e s t i n g L e v e l ; // compute maximum n e s t i n g depth o f s y n t h e s i z e d a t t r i b u t e s SynNestingDepth n e s t i n g D e p t h =0; f o r ( S y n t h e s i z e d A t t r i b u t e s L i s t : : i t e r a t o r i=l . b e g i n ( ) ; i != l . end ( ) ; i ++) { i f ( ∗ i >n e s t i n g D e p t h ) n e s t i n g D e p t h=∗ i ; } s w i t c h ( node−>v a r i a n t T ( ) ) { c a s e V SgDoWhileStmt : c a s e V SgForStatement : c a s e V SgWhileStmt : { n e s t i n g D e p t h ++; c o u t << ” N e s t i n g l e v e l : ” << n e s t i n g L e v e l << ” , n e s t i n g depth : ” << n e s t i n g D e p t h << e n d l ; break ; } default : { // DQ ( 1 1 / 1 7 / 2 0 0 5 ) : Nothing t o do h e r e , but e x p l i c i t }

default in switch avoids

lots

of warnings .

} // add l o o p n e s t i n g l e v e l a s a n n o t a t i o n t o AST N e s t i n g L e v e l A n n o t a t i o n ∗ n l a = new N e s t i n g L e v e l A n n o t a t i o n ( n e s t i n g L e v e l , n e s t i n g D e p t h ) ; ROSE ASSERT( n l a != NULL ) ; // DQ ( 1 / 2 / 2 0 0 6 ) : Added s u p p o r t f o r new a t t r i b u t e i n t e r f a c e . // p r i n t f ( ” L o o p L e v e l P r o c e s s i n g : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( ) : u s i n g new a t t r i b u t e #i f 0 i f ( node−>g e t a t t r i b u t e ( ) == NULL) { A s t A t t r i b u t e M e c h a n i s m ∗ a t t r i b u t e P t r = new A s t A t t r i b u t e M e c h a n i s m ( ) ; ROSE ASSERT( a t t r i b u t e P t r != NULL ) ; node−>s e t a t t r i b u t e ( a t t r i b u t e P t r ) ; } #e n d i f // node−>a t t r i b u t e . add ( ” l o o p N e s t i n g I n f o ” , n l a ) ; // node−>a t t r i b u t e ( ) . add ( ” l o o p N e s t i n g I n f o ” , n l a ) ; node−>addNewAttribute ( ” l o o p N e s t i n g I n f o ” , n l a ) ; / / ! r e t u r n t h e maximum n e s t i n g depth a s s y n t h e s i z e d return nestingDepth ;

attribute

} i n t main ( i n t a r g c , c h a r ∗∗ a r g v ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // command l i n e p a r a m e t e r s a r e p a s s e d t o EDG // non−EDG p a r a m e t e r s a r e p a s s e d ( t h r o u g h ) t o ROSE ( and t h e v e n d o r c o m p i l e r ) S g P r o j e c t ∗ r o o t=f r o n t e n d ( a r g c , a r g v ) ;

i n t e r f a c e \n ” ) ;

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11

Output : Nesting Nesting Nesting Nesting Nesting Nesting Nesting Nesting Max l o o p

level :5 , level :4 , level :3 , level :2 , level :1 , level :2 , level :1 , level :1 , nesting

61

n e s t i n g depth : 1 n e s t i n g depth : 2 n e s t i n g depth : 3 n e s t i n g depth : 4 n e s t i n g depth : 5 n e s t i n g depth : 1 n e s t i n g depth : 2 n e s t i n g depth : 1 level : 5

Figure 7.23: Output code showing the result of using inherited, synthesized, and accumulator attributes.

62

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

#i n c l u d e c l a s s NodeTypeCounter : p u b l i c A s t S i m p l e P r o c e s s i n g { public : NodeTypeCounter ( enum VariantT v a r i a n t , s t d : : s t r i n g typeName ) : myVariant ( v a r i a n t ) , typeName ( typeName ) , c o u n t ( 0 ) { } protected : v i r t u a l v o i d v i s i t ( SgNode ∗ node ) { i f ( node−>v a r i a n t T ( ) == myVariant ) { s t d : : c o u t << ”Found ” << typeName << s t d : : e n d l ; c o u n t++; } } v i r t u a l void atTraversalEnd ( ) { s t d : : c o u t << typeName << ” t o t a l : ” << c o u n t << s t d : : e n d l ; } private : enum VariantT myVariant ; s t d : : s t r i n g typeName ; unsigned i n t count ; }; i n t main ( i n t a r g c , c h a r ∗∗ a r g v ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; s t d : : c o u t << ” s e q u e n t i a l e x e c u t i o n o f t r a v e r s a l s ” << s t d : : e n d l ; NodeTypeCounter f o r S t a t e m e n t C o u n t e r ( V SgForStatement , ” f o r l o o p ” ) ; NodeTypeCounter i n t V a l u e C o u n t e r ( V SgIntVal , ” i n t c o n s t a n t ” ) ; NodeTypeCounter v a r D e c l C o u n t e r ( V S g V a r i a b l e D e c l a r a t i o n , ” v a r i a b l e d e c l a r a t i o n ” ) ; // t h r e e c a l l s t o t r a v e r s e , e x e c u t e d s e q u e n t i a l l y forStatementCounter . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; intValueCounter . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; varDeclCounter . t r a v e r s e I n p u t F i l e s ( pr o je c t , preorder ) ; s t d : : c o u t << s t d : : e n d l ; s t d : : c o u t << ” combined e x e c u t i o n o f t r a v e r s a l s ” << s t d : : e n d l ; AstCombinedSimpleProcessing combinedTraversal ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V SgForStatement , ” f o r l o o p ” ) ) ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V SgIntVal , ” i n t c o n s t a n t ” ) ) ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V S g V a r i a b l e D e c l a r a t i o n , ” v a r i a b l e // one c a l l t o t r a v e r s e , e x e c u t i o n i s i n t e r l e a v e d combinedTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; }

Figure 7.24: Example source showing the combination of traversals.

declaration ”));

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

63

sequential execution of traversals Found f o r l o o p Found f o r l o o p f o r loop t o t a l : 2 Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t i n t c o n s t a n t t o t a l : 10 Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n variable declaration total : 9 combined e x e c u t i o n o f t r a v e r s a l s Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found f o r l o o p Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found f o r l o o p Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t f o r loop t o t a l : 2 i n t c o n s t a n t t o t a l : 10 variable declaration total : 9

Figure 7.25: Output of input file to the combined traversals. Note that the order of outputs changes as execution of several analyzers is interleaved.

64

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.2.12

Short-Circuiting Traversals

The traversal short-circuit mechanism is a simple way to cut short the traversal of a large AST once specific information has been obtained. It is purely an optimization mechanism, and a bit of a hack, but common within the C++ Boost community. Since the technique works we present it as a way of permitting users to avoid the full traversal of an AST that they might deam to be redundant of inappropriate. We don’t expect that this mechanism will be particularly useful to most users and we don’t recommend it. It may even at some point not be supported. However, we present it because it is a common technique used in the C++ Boost community and it happens to work (at one point it didn’t work and so we have no idea what we fixed that permitted it to work now). We have regarded this technique as a rather ugly hack. It is presented in case you really need it. It is, we think, better than the direct use of lower level mechanisms that are used to support the AST traversal. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// I n p u t f o r

t r a n s l a t o r t o show e x c e p t i o n −b a s e d e x i t i n g from a t r a n s l a t o r .

namespace A { go ; int struct B { static int }; };

stop

;

void foo ( void ) { e x t e r n void bar ( i n t ) ; go ); b a r (A : : stop ); b a r (A : : B : : }

Figure 7.26: Input code with used to demonstrate the traversal short-circuit mechanism. Figure 7.27 shows the example code demonstrating a traversal setup to support the shortcircuit mechanism (a conventional mechanism used often within the C++ Boost community). The input code shown in figure 7.26 is compiled using the example translator, the output is shown in figure 7.28. The output shown in figure 7.28 demonstrates the initiation of a traversal over the AST and that traversal being short-circuited after a specific point in the evaluation. The result is that there is no further traversal of the AST after that point where it is short-circuited.

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

65

// Example o f an AST t r a v e r s a l t h a t u s e s t h e Boost i d i o m o f t h r o w i n g // an e x c e p t i o n t o e x i t t h e t r a v e r s a l e a r l y . #i n c l u d e #i n c l u d e #i n c l u d e u s i n g namespace s t d ; // E x c e p t i o n t o i n d i c a t e an e a r l y e x i t from a t r a v e r s a l a t some node . c l a s s StopEarly { public : ( n ) {} S t o p E a r l y ( c o n s t SgNode∗ n ) : e x i t n o d e ( e . e x i t n o d e ) {} S t o p E a r l y ( c o n s t S t o p E a r l y& e ) : e x i t n o d e // P r i n t s i n f o r m a t i o n about t h e e x i t node . v o i d p r i n t ( o s t r e a m& o ) c o n s t { i f ( exit node ) { o << ’ \ t ’ << ( c o n s t v o i d ∗ ) e x i t n o d e << ” : ” << e x i t n o d e −>c l a s s n a m e ( ) << e n d l ; c o n s t SgLocatedNode ∗ l o c n = i s S g L o c a t e d N o d e ( e x i t n o d e ) ; i f ( loc n ) { c o n s t S g F i l e I n f o ∗ i n f o = l o c n −>g e t s t a r t O f C o n s t r u c t ( ) ; ROSE ASSERT ( i n f o ) ; o << ”\ tAt ” << i n f o −>g e t f i l e n a m e ( ) << ” : ” << i n f o −>g e t l i n e ( ) << e n d l ; } } } private : c o n s t SgNode∗ e x i t n o d e ; // Node a t e a r l y };

e x i t from t r a v e r s a l

// P r e o r d e r t r a v e r s a l t o f i n d t h e f i r s t SgVarRefExp o f a p a r t i c u l a r name . c l a s s VarRefFinderTraversal : public AstSimpleProcessing { public : // I n i t i a t e t r a v e r s a l t o f i n d ’ t a r g e t ’ i n ’ p r o j ’ . v o i d f i n d ( S g P r o j e c t ∗ p r o j , c o n s t s t r i n g& t a r g e t ) { target = target ; t r a v e r s e I n p u t F i l e s ( proj , preorder ) ; } v o i d v i s i t ( SgNode∗ node ) { c o n s t SgVarRefExp ∗ r e f = isSgVarRefExp ( node ) ; if ( ref ) { c o n s t S g V a r i a b l e S y m b o l ∗ sym = r e f −>g e t s y m b o l ( ) ; ROSE ASSERT ( sym ) ; c o u t << ” V i s i t i n g SgVarRef ’ ” << sym−>get name ( ) . s t r ( ) << ” ’ ” << e n d l ; i f ( sym−>get name ( ) . s t r ( ) == t a r g e t ) // E a r l y e x i t a t f i r s t match . throw S t o p E a r l y ( r e f ) ; } } private : string };

t a r g e t ; // Symbol r e f e r e n c e name t o f i n d .

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; VarRefFinderTraversal f i n d e r ; // Look f o r a r e f e r e n c e t o ” try { f i n d e r . f in d ( proj , ” s t o p c o u t << ”∗∗∗ R e f e r e n c e t o a } c a t c h ( S t o p E a r l y& s t o p ) { c o u t << ”∗∗∗ R e f e r e n c e t o a stop . p r i n t ( cout ) ; }

stop

”.

”); symbol



stop

’ n o t f o u n d . ∗∗∗” << e n d l ;

symbol



stop

’ f o u n d . ∗∗∗” << e n d l ;



go

// Look f o r a r e f e r e n c e t o ” g o ”. try { ”); f i n d e r . f in d ( proj , ” g o c o u t << ”∗∗∗ R e f e r e n c e t o a symbol } c a t c h ( S t o p E a r l y& go ) {

’ n o t f o u n d . ∗∗∗” << e n d l ;

66

1 2 3 4 5 6 7 8 9

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

V i s i t i n g SgVarRef ’ g o ’ ’ V i s i t i n g SgVarRef ’ s t o p ∗∗∗ R e f e r e n c e t o a symbol ’ s t o p ’ f o u n d . ∗∗∗ 0 x 7 f 6 b 6 a d a 1 0 7 8 : SgVarRefExp At / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e t r a v e r s a l S ’ V i s i t i n g SgVarRef ’ g o ’ f o u n d . ∗∗∗ ∗∗∗ R e f e r e n c e t o a symbol ’ g o 0 x 7 f 6 b 6 a d a 1 0 1 0 : SgVarRefExp At / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e t r a v e r s a l S

Figure 7.28: Output code showing the result of short-circuiting the traversal.

7.3. MEMORY POOL TRAVERSALS

7.3

67

Memory Pool Traversals

Allocation of IR nodes in ROSE is made more efficient through the use of specialized allocators implemented at member function new operators for each class of the IR in Sage III. Such specialized memory allocators avoid significant fragmentation of memory, provide more efficient packing of memory, improve performance of allocation of memory and IR node access, and additionally provide a secondary mechanism to accessing all the IR nodes. Each IR node has a memory pool which is an STL vector of blocks (a fixed or variable sized array of contiguously stored IR nodes). The three types of traversals are: 1. ROSE Memory Pool Visit Traversal This traversal is similar to the one provided by the SimpleProcessing Class (using the visit() function and no inherited or synthesized attributes). 2. Classic Object-Oriented Visitor Pattern for Memory Pool This is a classic object-oriented visitor pattern. 3. IR node type traversal, visits one type of IR node for all IR types in the AST. This is useful for building specialized tools.

7.3.1

ROSE Memory Pool Visit Traversal

Figure 7.29 shows the source code for a translator which traverses the memory pool containing the AST. At each node the visit() function is called using only the input information represented by the current node. Note that using this simple traversal no context information is available to the visit function. All the IR nodes for a given memory pool are iterated over at one time. The order of the traversal of the different memory pools is random but fixed. Thus the order of the traversal of the IR nodes is in no way connected to the structure of the AST (unlike the previous non-memory pool traversals that were very much tied to the structure of the AST and which matched the structure of the original input source code being compiled).

68

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

#i n c l u d e ” r o s e . h” // ROSE V i s i t T r a v e r s a l ( s i m i l a r i n t e r f a c e a s Markus ’ s v i s i t t r a v e r s a l ) // i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t h as no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s RoseVisitor : public ROSE VisitTraversal { public : int counter ; v o i d v i s i t ( SgNode∗ node ) ; RoseVisitor ()

: c o u n t e r ( 0 ) {}

};

v o i d R o s e V i s i t o r : : v i s i t ( SgNode∗ node ) { // p r i n t f ( ” r o s e V i s i t o r : : v i s i t : c o u n t e r %4d node = %s \n ” , c o u n t e r , node−>c l a s s n a m e ( ) . c s t r ( ) ) ; c o u n t e r ++; } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // ROSE v i s i t t r a v e r s a l RoseVisitor v i s i t o r ; v i s i t o r . traverseMemoryPool ( ) ; p r i n t f ( ” Number o f IR n o d e s i n AST = %d \n ” , v i s i t o r . c o u n t e r ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 7.29: Example source showing simple visit traversal over the memory pools.

1

Number o f IR n o d e s i n AST = 16919

Figure 7.30: Output of input file to the visitor traversal over the memory pool.

7.3. MEMORY POOL TRAVERSALS

7.3.2

69

Classic Object-Oriented Visitor Pattern for Memory Pool

Figure 7.31 shows the source code for a translator which traverses the memory pools containing the AST. At each node the visit() function is called using only the input information represented by the current node. Note that using this simple traversal no context information is available to the visit function. The traversal order is the same as in the 7.29. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

#i n c l u d e ” r o s e . h” // C l a s s i c V i s i t o r P a t t e r n i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t h as no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s C l a s s i c V i s i t o r : p u b l i c ROSE VisitorPattern { public : // O v e r r i d e v i r t u r a l f u n c t i o n d e f i n e d i n b a s e c l a s s void v i s i t ( SgGlobal ∗ g l o b a l S c o p e ) { p r i n t f ( ” Found t h e S g G l o b a l IR node \n ” ) ; } void v i s i t ( SgFunctionDeclaration ∗ functionDeclaration ) { p r i n t f ( ” Found a S g F u n c t i o n D e c l a r a t i o n IR node \n ” ) ; } v o i d v i s i t ( SgTypeInt ∗ i n t T y p e ) { p r i n t f ( ” Found a SgTypeInt IR node \n ” ) ; } v o i d v i s i t ( SgTypeDouble ∗ doubleType ) { p r i n t f ( ” Found a SgTypeDouble IR node \n ” ) ; } };

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // C l a s s i c v i s i t o r p a t t e r n o v e r t h e memory p o o l o f IR n o d e s ClassicVisitor visitor A ; traverseMemoryPoolVisitorPattern ( v i s i t o r A ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 7.31: Example source showing simple visitor pattern.

70

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

a SgTypeInt IR node a SgTypeDouble IR node t h e S g G l o b a l IR node t h e S g G l o b a l IR node a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration

IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR

node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node

7.3. MEMORY POOL TRAVERSALS

7.3.3

71

ROSE IR Type Traversal (uses Memory Pools)

Figure 7.33 shows the source code for a translator which traverses only one type of IR node using the memory pool containing the AST. This traversal is useful for building specialized tools (often tools which only call static functions on each type of IR node). This example shows the use of an alternative traversal which traverses a representative of each type or IR node just one, but only if it exists in the AST (memory pools). This sort of traversal is useful for building tools that need only operate on static member functions of the IR nodes or need only sample one of each type or IR node present in the AST. this specific example also appears in: ROSE/src/midend/astDiagnostics/AstStatistics.C. The user’s use of the traversal is the same as for other ROSE AST traversals except that the ROSE VisitTraversal::traverseRepresentativeIRnodes() member function is called instead of ROSE VisitTraversal::traverseMemoryPool(). This mechanism can be used to generate more complete reports of the memory consumption of the AST, which is reported on if -rose:verbose 2 is used. Figure 7.35 shows a partial snapshot of current IR node frequency and memory consumption for a moderate 40,000 line source code file (one file calling a number of header files), sorted by memory consumption. The AST contains approximately 280K IR nodes. Note that the Sg File Info IR nodes is most frequent and consumes the greatest amount of memory, this reflects our bias toward preserving significant information about the mapping of language constructs back to the positions in the source file to support a rich set of source-to-source functionality. Note: more complete information about the memory use of the AST in in the ROSE User Manual appendix.

72

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

// T h i s example c o d e shows t h e t r a v e r s a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ;

o f IR t y p e s n o t a v a i l a b l e u s i n g t h e o t h e r t r a v e r s a l mechanism .

// CPP Macro t o implement c a s e f o r e a c h IR node ( we c o u l d a l t e r n a t i v e l y u s e a v i s i t o r p a t t e r n and a f u n c t i o n t #d e f i n e IR NODE VISIT CASE (X) \ c a s e V ##X : \ { \ X∗ c a s t N o d e = i s##X( node ) ; \ i n t numberOfNodes = castNode−>numberOfNodes ( ) ; \ i n t memoryFootprint = castNode−>memoryUsage ( ) ; \ p r i n t f ( ” c o u n t = %7d , memory u s e = %7d b y t e s , node name = %s \n ” , numberOfNodes , memoryFootprint , break ; \ } c l a s s RoseIRnodeVisitor : public ROSE VisitTraversal { public : int counter ; v o i d v i s i t ( SgNode∗ node ) ; R o s e I R n o d e V i s i t o r ( ) : c o u n t e r ( 0 ) {} }; void { // // //

RoseIRnodeVisitor : : v i s i t

( SgNode∗ node )

Usin g a c l a s s i c v i s i t o r p a t t e r n s h o u l d a v o i d a l l t h i s c a s t i n g , but e a c h f u n c t i o n must be c r e a t e d s e p a r a t e l y ( s o i t i s wash i f we want t o do a l l IR nodes , a s we do h e r e ) . s w i t c h ( node−>v a r i a n t T ( ) ) { IR NODE VISIT CASE ( S g F i l e I n f o ) IR NODE VISIT CASE ( S g P a r t i a l F u n c t i o n T y p e ) IR NODE VISIT CASE ( SgFunctionType ) IR NODE VISIT CASE ( S g P o i n t e r T y p e ) IR NODE VISIT CASE ( S g F u n c t i o n D e c l a r a t i o n ) IR NODE VISIT CASE ( SgFunctionSymbol ) IR NODE VISIT CASE ( SgSymbolTable ) IR NODE VISIT CASE ( S g I n i t i a l i z e d N a m e ) IR NODE VISIT CASE ( S g S t o r a g e M o d i f i e r ) IR NODE VISIT CASE ( SgFor Stat emen t ) IR NODE VISIT CASE ( S g F o r I n i t S t a t e m e n t ) IR NODE VISIT CASE ( S g C t o r I n i t i a l i z e r L i s t ) IR NODE VISIT CASE ( S g I f S t m t ) IR NODE VISIT CASE ( SgExprStatement ) IR NODE VISIT CASE ( S g T e m p l a t e D e c l a r a t i o n ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n D e c l ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n D e f n ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n M e m b e r F u n c t i o n D e c l ) IR NODE VISIT CASE ( SgClassSymbol ) IR NODE VISIT CASE ( SgTemplateSymbol ) IR NODE VISIT CASE ( SgMemberFunctionSymbol ) default : {

#i f 0 p r i n t f ( ” Case n o t h a n d l e d : %s \n ” , node−>c l a s s n a m e ( ) . c s t r ( ) ) ; #e n d i f } } }

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // ROSE v i s i t t r a v e r s a l SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // ROSE v i s i t t r a v e r s a l RoseIRnodeVisitor v i s i t o r ; v i s i t o r . traverseRepresentativeIRnodes ( ) ; p r i n t f ( ” Number o f t y p e s o f IR n o d e s ( a f t e r

b u i l d i n g AST) = %d \n ” , v i s i t o r . c o u n t e r ) ;

#i f 1 // IR n o d e s s t a t i s t i c s i f ( p r o j e c t −>g e t v e r b o s e ( ) > 1 ) c o u t << A s t N o d e S t a t i s t i c s : : I R n o d e U s a g e S t a t i s t i c s ( ) ; #e n d i f i n t errorCode = 0 ;

7.3. MEMORY POOL TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

73

count = 3 4 , memory u s e = 5168 b y t e s , node name = SgSymbolTable count = 1 4 7 9 , memory u s e = 59160 b y t e s , node name = S g S t o r a g e M o d i f i e r count = 1 0 9 5 5 , memory u s e = 964040 b y t e s , node name = S g F i l e I n f o No r e p r e s e n t a t i v e f o r S g U n p a r s e I n f o f o u n d i n memory p o o l s No r e p r e s e n t a t i v e f o r S g P a r t i a l F u n c t i o n T y p e f o u n d i n memory p o o l s count = 1 1 6 , memory u s e = 16704 b y t e s , node name = SgFunctionType count = 1 6 , memory u s e = 1920 b y t e s , node name = S g P o i n t e r T y p e count = 2 , memory u s e = 592 b y t e s , node name = SgF orSta teme nt count = 2 , memory u s e = 224 b y t e s , node name = S g F o r I n i t S t a t e m e n t count = 4 , memory u s e = 2400 b y t e s , node name = S g C t o r I n i t i a l i z e r L i s t count = 1 , memory u s e = 304 b y t e s , node name = S g I f S t m t No r e p r e s e n t a t i v e f o r S g D e c l a r a t i o n S c o p e f o u n d i n memory p o o l s count = 9 , memory u s e = 864 b y t e s , node name = SgExprStatement count = 2 , memory u s e = 1600 b y t e s , node name = S g T e m p l a t e I n s t a n t i a t i o n D e c l count = 1 , memory u s e = 304 b y t e s , node name = S g T e m p l a t e I n s t a n t i a t i o n D e f n count = 2 , memory u s e = 2192 b y t e s , node name = S g T e m p l a t e I n s t a n t i a t i o n M e m b e r F u n c t i o n D e c l count = 4 3 6 , memory u s e = 436000 b y t e s , node name = S g F u n c t i o n D e c l a r a t i o n count = 1 , memory u s e = 48 b y t e s , node name = SgClassSymbol count = 2 , memory u s e = 96 b y t e s , node name = SgMemberFunctionSymbol count = 4 3 4 , memory u s e = 20832 b y t e s , node name = SgFunctionSymbol count = 1 4 7 9 , memory u s e = 485112 b y t e s , node name = S g I n i t i a l i z e d N a m e Number o f t y p e s o f IR n o d e s ( a f t e r b u i l d i n g AST) = 0

Figure 7.34: Output of input file to the IR Type traversal over the memory pool. AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST

Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory

Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool

Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics:

numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes

= 114081 memory consumption = 5019564 node = Sg_File_Info = 31403 memory consumption = 628060 node = SgTypedefSeq = 14254 memory consumption = 285080 node = SgStorageModifier = 14254 memory consumption = 1140320 node = SgInitializedName = 8458 memory consumption = 169160 node = SgFunctionParameterTypeList = 7868 memory consumption = 1101520 node = SgModifierType = 7657 memory consumption = 398164 node = SgClassType = 7507 memory consumption = 2071932 node = SgClassDeclaration = 7060 memory consumption = 282400 node = SgTemplateArgument = 6024 memory consumption = 385536 node = SgPartialFunctionType = 5985 memory consumption = 1388520 node = SgFunctionParameterList = 4505 memory consumption = 1477640 node = SgTemplateInstantiationDecl = 3697 memory consumption = 162668 node = SgReferenceType = 3270 memory consumption = 758640 node = SgCtorInitializerList = 3178 memory consumption = 76272 node = SgMemberFunctionSymbol = 2713 memory consumption = 119372 node = SgPointerType = 2688 memory consumption = 161280 node = SgThrowOp = 2503 memory consumption = 60072 node = SgFunctionSymbol = 2434 memory consumption = 107096 node = SgFunctionTypeSymbol = 2418 memory consumption = 831792 node = SgFunctionDeclaration = 2304 memory consumption = 55296 node = SgVariableSymbol = 2298 memory consumption = 101112 node = SgVarRefExp = 2195 memory consumption = 114140 node = SgSymbolTable = 2072 memory consumption = 721056 node = SgMemberFunctionDeclaration = 1668 memory consumption = 400320 node = SgVariableDeclaration = 1667 memory consumption = 393412 node = SgVariableDefinition = 1579 memory consumption = 101056 node = SgMemberFunctionType = 1301 memory consumption = 31224 node = SgTemplateSymbol = 1300 memory consumption = 364000 node = SgTemplateDeclaration = 1198 memory consumption = 455240 node = SgTemplateInstantiationMemberFunctionDecl = 1129 memory consumption = 54192 node = SgIntVal = 1092 memory consumption = 56784 node = SgAssignInitializer = 1006 memory consumption = 52312 node = SgExpressionRoot

Truncated results presented ...

Figure 7.35: Example of output using -rose:verbose 2 (memory use report for AST).

74

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

Chapter 8

Graph Processing Tutorial 8.1

Traversal Tutorial

ROSE can collect and analyze paths in both source and binary CFGs. At moment it doesn’t attempt to save paths because if you save them directly the space necessary is extremely large, as paths grow 2n with successive if statements and even faster when for loops are involved. Currently a path can only cannot complete the same loop twice. However it is possible for a graph such that 1 -¿ 2 , 2-¿3, 3-¿1, 3-¿5, has paths, 1,2,3,1,2,3,5 and 1,2,3,5 because the loop 1,2,3,1 is not repeated. The tutorial example works as such:

75

76

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

CHAPTER 8. GRAPH PROCESSING TUTORIAL

#include #include //#i n c l u d e #include #include #include ” SgGraphTemplate . h” #include ” g r a p h P r o c e s s i n g . h”

#include ” staticCFG . h” #include ” i n t e r p r o c e d u r a l C F G . h” /∗ T e s t i n g t h e graph t r a v e r s a l mechanism now implementing i n A s t P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g / #include #include using namespace s t d ; using namespace b o o s t ;

/∗ You need t o use myGraph t y p e h e r e b e c a u s e t h e c o n v e r s i o n o f StaticCFG : : InterproceduralCFG or StaticCFG : :CFG i n a b o o s t form . The SgGraphTemplate . h f i l e h a n d l e s t h i s c o n v e r s i o n and myGraph i s s p e c i f i c t o t h a t f i l e ∗/ typedef myGraph CFGforT ;

/∗ Your b a s i c v i s i t o r t r a v e r s a l s u b c l a s s e d from SgGraphTraversal on t h e CFGforT t e m p l a t e as d e f i n e d above ∗/ c l a s s v i s i t o r T r a v e r s a l : public S g G r a p h T r a v e r s a l { public : int paths ; /∗ This i s t h e f u n c t i o n run by t h e a l g o r i t h m on e v e r y path , VertexID i s a t y p e implemented i n SgGraphTe void a n a l y z e P a t h ( v e c t o r & pth ) ; }; /∗ d e f i n i n g t h e a n a l y z e P a t h f u n c t i o n . This s i m p l y c o u n t s p a t h s as s h o u l d be o b v i o u s . Again , VertexID i s d e f i n e void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r & pth ) { p a t h s ++; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; /∗ F i r s t you need t o produce t h e p r o j e c t f i l e ∗/ SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j != NULL ) ; /∗ G e t t i n g t h e Function D e c l a r a t i o n and D e f i n i t i o n f o r p r o d u c i n g t h e graph ∗/ S g F u n c t i o n D e c l a r a t i o n ∗ mainDefDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j ) ; S g F u n c t i o n D e f i n i t i o n ∗ mainDef = mainDefDecl−>g e t d e f i n i t i o n ( ) ; /∗ I n s t a n t i a t i n g t h e v i s i t o r T r a v e r s a l ∗/ v i s i t o r T r a v e r s a l ∗ v i s = new v i s i t o r T r a v e r s a l ( ) ; /∗ This c r e a t e s t h e StaticCFG : : InterproceduralCFG o b j e c t t o be c o n v e r t e d t o a b o o s t graph ∗/ StaticCFG : : I n t e r p r o c e d u r a l C F G c f g ( mainDef ) ; stringstream ss ; S g I n c i d e n c e D i r e c t e d G r a p h ∗ g = new S g I n c i d e n c e D i r e c t e d G r a p h ( ) ; /∗ We g o t t h e n e c e s s a r y i n t e r n a l S g I n c i d e n c e D i r e c t e d G r a p h from t h e c f g ∗/ g = c f g . getGraph ( ) ; myGraph∗ mg = new myGraph ( ) ; /∗ C o n v e r t i n g t h e c f g t o a b o o s t graph ∗/ mg = i n s t a n t i a t e G r a p h ( g , c f g , mainDef ) ; /∗ S e t i n t e r n a l v a r i a b l e s ∗/ v i s −>p a t h s = 0 ; /∗ i n v o k i n g t h e t r a v e r s a l , t h e f i r s t argument i s t h e graph , t h e second i s t r u e i f you do not want bounds , f a l s e i f you do , t h e t h i r d and f o u r t h arguments a r e s t a r t i n g and s t o p p i n g v e r t i c e s r e s p e c t i v e l y , i f you ar e not bounding s i m p l y i n s e r t 0 . F i n a l l y t h e l a s t argument i s c u r r e n t l y d e p r e c a t e d ∗/ v i s −>c o n s t r u c t P a t h A n a l y z e r (mg , true , 0 , 0 , true ) ; s t d : : c o u t << ” f i n i s h e d ” << s t d : : e n d l ; s t d : : c o u t << ” p a t h s : ” << v i s −>p a t h s << s t d : : e n d l ; delete v i s ; }

Figure 8.1: Source CFG Traversal Example

8.1. TRAVERSAL TUTORIAL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

77

#include #include #include //#i n c l u d e ” interproceduralCFG . h” #include #include /∗ These ar e n e c e s s a r y f o r any b i n a r y T r a v e r s a l ∗/ #include ” g r a p h P r o c e s s i n g . h” #include ” B i n a r y C o n t r o l F l o w . h” #include ” B i n a r y L o a d e r . h” /∗ T e s t i n g t h e graph t r a v e r s a l mechanism now implementing i n g r a p h P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g /) ∗/ using namespace s t d ; using namespace b o o s t ; /∗ These s h o u l d j u s t be c o p i e d v e r b a t i m ∗/ typedef b o o s t : : g r a p h t r a i t s : : v e r t e x d e s c r i p t o r V e r t e x ; /∗∗< Graph v e r t e x t y p e . ∗/ typedef b o o s t : : g r a p h t r a i t s : : e d g e d e s c r i p t o r Edge ; /∗∗< Graph edg e t y p e . ∗/

/∗ We f i r s t make a v i s i t o r T r a v e r s a l , s u b c l a s s e d from SgGraphTraversal t e m p l a t e d on t h e B i n a r y A n a l y s i s : ControlFlow : : Graph which i s implemented as a b o o s t graph ∗/

class v i s it o r T r av e r s a l {

: public S g G r a p h T r a v e r s a l

public : long i n t p t h s ; long i n t t l t n o d e s ; /∗ This needs t o be i n any v i s i t o r T r a v e r s a l , i t i s t h e f u n c t i o n t h a t w i l l be run on e v e r y p a t h by t h e graph p a t h a n a l y s i s a l g o r i t h m , n o t i c e t h e V e r t e x t y p e i s from t h e above t y p e d e f s ∗/ v i r t u a l void a n a l y z e P a t h ( v e c t o r & pth ) ; }; /∗ This i s a v e r y s i m p l e i n c a r n a t i o n , i t j u s t c o u n t s p a t h s ∗/ void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r & pth ) { p t h s ++; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; /∗ Parse t h e b i n a r y f i l e ∗/ SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; s t d : : v e c t o r i n t e r p s = S a g e I n t e r f a c e : : querySubTree( p r o j e c t ) ; i f ( i n t e r p s . empty ( ) ) { f p r i n t f ( s t d e r r , ” no b i n a r y i n t e r p r e t a t i o n s f o u n d \n” ) ; exit (1); } /∗ C a l c u l a t e p l a i n o l d CFG. ∗/ BinaryAnalysis : : ControlFlow c f g a n a l y z e r ; B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph∗ c f g = new B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph ; c f g a n a l y z e r . b u i l d b l o c k c f g f r o m a s t ( i n t e r p s . back ( ) , ∗ c f g ) ; s t d : : o f s t r e a m mf ; mf . open ( ” a n a l y s i s . d o t ” ) ; /∗ D e c l a r i n g t h e v i s i t o r T r a v e r s a l ∗/ v i s i t o r T r a v e r s a l ∗ v i s = new v i s i t o r T r a v e r s a l ; /∗ S e t t i n g i n t e r n a l v a r i a b l e s ∗/ v i s −>t l t n o d e s = 0 ; v i s −>p t h s = 0 ;

/∗ v i s i t o r T r a v e r s a l has 5 arguments , t h e f i r s t i s t h e ambient CFG, t h e second i d e n t i f i e s w h e t h e r or not you a re bounding t h e graph , t h a t i s , w h e t h e r you want a l l your p a t h s t o s t a r t a t one s p e c i f i c node and end a t a n o t h e r s p e c i f i c node , t h e f o u r t h and f i f t h would be s t a r t and end i f t h e graph were bounded . S i n c e t h e y aren ’ t you can s i m p l y i n p u t 0 , f o r t h e moment t h e f i n a l argument i s d e p r e c a t e d , t h o u g h i t ’ s p u r p o s e was t o t e l l t h e progra t h a t your a n a l y s i s f u n c t i o n was t h r e a d s a f e , t h a t i s t h a t openMP c o u l d run i t w i t h o u t h a v i n g a c r i t i c a l command . C u r r e n t l y a c r i t i c a l i s a l w a y s used ∗/ v i s −>c o n s t r u c t P a t h A n a l y z e r ( c f g , true , 0 , 0 , f a l s e ) ;

78

CHAPTER 8. GRAPH PROCESSING TUTORIAL

Chapter 9

Scopes of Declarations The scope of an IR node may be either stored explicitly in the IR node or obtained through computation through its parent information in the AST. Figure X shows an example where the variable definition for a variable is the scope of namespace X. The declaration for variable a is in the namespace X. In a more common way, the function foo is a member function of B with a declaration appearing in class B, but with a function definition in global scope. namespace X{ extern int a; } int X::a = 0; class B { void foo(); }; void B::foo() {}

In C++, using name qualification the scope of a declaration can be independent of it structural location in the AST. The get parent() member function (available on most IR nodes) communicates the structural information of the original source code (also represented in the AST). The scope information must at times be stored explicitly when it can not be interpreted structurally. The example in this chapter show how to find the scope of each C++ construct . Note that SgExpression IR nodes can take their scope from that of the statement where they are found. SgStatement and SgInitializedName IR nodes are the interesting IR nodes from the point of scope. The SgInitializedName and all SgStatement IR nodes have a member function get scope() which returns the scope of the associated IR nodes. The example code in this chapter traverses the AST and reports the scope of any SgInitializedName and all SgStatement IR nodes. It is intended to provide a simple intuition about what the scope can be expected to be in an application. The example code is also useful as a simple means of exploring the scopes of any other input application.

9.1

Input For Examples Showing Scope Information

Figure 9.1 shows the input example code form this tutorial example. 79

80

1 2 3 4 5 6 7 8 9 10 11 12

CHAPTER 9. SCOPES OF DECLARATIONS

i n t xyz ; void foo ( i n t { int y ; for ( int { int z = } }

x)

i =0; i < 1 0 ;

i ++)

z; 42;

Figure 9.1: Example source code used as input to program in codes used in this chapter.

9.2

Generating the code representing any IR node

The following code traverses each IR node and for a SgInitializedName of SgStatement outputs the scope information. The input code is shown in figure 9.1; the output of this code is shown in figure 9.3.

9.2. GENERATING THE CODE REPRESENTING ANY IR NODE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

81

// T h i s example shows t h e s c o p e o f e a c h s t a t e m e n t and name ( v a r i a b l e names , b a s e c l a s s names ,

etc . ) .

#i n c l u d e ” r o s e . h” class {

visitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { // There a r e t h r e e t y p e s i r IR n o d e s t h a t can be q u e r i e d f o r s c o p e : // − SgStatement , and // − SgInitializedName SgStatement ∗ s t a t e m e n t = i s S g S t a t e m e n t ( n ) ; i f ( s t a t e m e n t != NULL) { S g S c o p e S t a t e m e n t ∗ s c o p e = s t a t e m e n t −>g e t s c o p e ( ) ; ROSE ASSERT( s c o p e != NULL ) ; p r i n t f ( ” Sg Stat eme nt = %12p = %30 s has s c o p e = %12p = %s ( t o t a l number = %d ) \n ” , s t a t e m e n t , s t a t e m e n t −>c l a s s n a m e ( ) . c s t r ( ) , s c o p e , s c o p e −>c l a s s n a m e ( ) . c s t r ( ) , ( i n t ) s c o p e −>numberOfNodes ( ) ) ; } SgInitializedName ∗ initializedName = isSgInitializedName (n ) ; i f ( i n i t i a l i z e d N a m e != NULL) { S g S c o p e S t a t e m e n t ∗ s c o p e = i n i t i a l i z e d N a m e −>g e t s c o p e ( ) ; ROSE ASSERT( s c o p e != NULL ) ; p r i n t f ( ” S g I n i t i a l i z e d N a m e = %12p = %30 s has s c o p e = %12p = %s ( t o t a l number = %d ) \ n ” , i n i t i a l i z e d N a m e , i n i t i a l i z e d N a m e −>get name ( ) . s t r ( ) , s c o p e , s c o p e −>c l a s s n a m e ( ) . c s t r ( ) , ( i n t ) s c o p e −>numberOfNodes ( ) ) ; } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; p r i n t f ( ” Number o f s c o p e s ( S g S c o p e S t a t e m e n t ) = %d \n ” , ( i n t ) S g S c o p e S t a t e m e n t : : numberOfNodes ( ) ) ; p r i n t f ( ” Number o f s c o p e s ( S g B a s i c B l o c k ) = %d \n ” , ( i n t ) S g B a s i c B l o c k : : numberOfNodes ( ) ) ; #i f 0 p r i n t f ( ” \ n\n ” ) ; p r i n t f ( ”Now o u t p u t a l l t h e s y m b o l s i n e a c h symbol t a b l e \n ” ) ; S a g e I n t e r f a c e : : outputLocalSymbolTables ( p r o j e c t ) ; p r i n t f ( ” \ n\n ” ) ; #e n d i f return 0; }

Figure 9.2: Example source code showing how to get scope information for each IR node.

82

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

CHAPTER 9. SCOPES OF DECLARATIONS

SgStatement = 0 x7f3045d22120 = SgGlobal SgStatement = 0 x7f3038dce010 = SgVariableDeclaration SgInitializedName = 0 x7f303987ec20 = xyz has s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t a l number = 0 ) SgStatement = 0 x7f3039614790 = SgFunctionDeclaration SgStatement = 0 x7f303998b6b8 = SgFunctionParameterList SgInitializedName = 0 x7f303987eeb0 = x has s c o p e = 0 x 7 f 3 0 3 8 b 8 e 0 1 0 = S g F u n c t i o n D e f i n i t i o n ( t o t a l number = SgStatement = 0 x7f3038b8e010 = SgFunctionDefinition SgStatement = 0 x7f3038c23010 = SgBasicBlock SgStatement = 0 x7f3038dce2a8 = SgVariableDeclaration SgInitializedName = 0 x7f303987eff8 = y has s c o p e = 0 x 7 f 3 0 3 8 c 2 3 0 1 0 = S g B a s i c B l o c k ( t o t a l number = 0 ) SgStatemen t = 0 x7f3038afd010 = SgF orSta teme nt SgStatemen t = 0 x7f3038ac6080 = SgForInitStatement SgStatemen t = 0 x7f3038dce540 = SgVariableDeclaration SgInitializedName = 0 x7f303987f140 = i has s c o p e = 0 x 7 f 3 0 3 8 a f d 0 1 0 = SgF orSta teme nt ( t o t a l number = 0 ) SgStatemen t = 0 x7f30389c7010 = SgExprStatement SgStatemen t = 0 x7f3038c23128 = SgBasicBlock SgStatemen t = 0 x7f3038dce7d8 = SgVariableDeclaration SgInitializedName = 0 x7f303987f288 = z has s c o p e = 0 x 7 f 3 0 3 8 c 2 3 1 2 8 = S g B a s i c B l o c k ( t o t a l number = 0 ) SgStatemen t = 0 x7f30389c7070 = SgExprStatement Number o f s c o p e s ( S g S c o p e S t a t e m e n t ) = 0 Number o f s c o p e s ( S g B a s i c B l o c k ) = 2

has s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t ha s s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t

has s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t h as s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t

0) h as s c o p e = 0 x 7 f 3 0 4 5 d 2 2 1 2 0 = S g G l o b a l ( t o t has s c o p e = 0 x 7 f 3 0 3 8 b 8 e 0 1 0 = S g F u n c t i o n D e ha s s c o p e = 0 x 7 f 3 0 3 8 c 2 3 0 1 0 = S g B a s i c B l o c k

has s c o p e = 0 x 7 f 3 0 3 8 c 2 3 0 1 0 = S g B a s i c B l o c k has s c o p e = 0 x 7 f 3 0 3 8 a f d 0 1 0 = SgForStatemen ha s s c o p e = 0 x 7 f 3 0 3 8 a f d 0 1 0 = SgForStatemen

has s c o p e = 0 x 7 f 3 0 3 8 a f d 0 1 0 = SgForStatemen has s c o p e = 0 x 7 f 3 0 3 8 a f d 0 1 0 = SgForStatemen has s c o p e = 0 x 7 f 3 0 3 8 c 2 3 1 2 8 = S g B a s i c B l o c k

has s c o p e = 0 x 7 f 3 0 3 8 c 2 3 1 2 8 = S g B a s i c B l o c k

Figure 9.3: Output of input code using scopeInformation.C

Chapter 10

AST Query This chapter presents a mechanism for simple queries on the AST. Such queries are typically a single line of code, instead of the class that must be declared and defined when using the traversal mechanism. While the traversal mechanism is more sophisticated and more powerful, the AST Query mechanism is particularly simple to use.

10.1

Simple Queries on the AST

This section demonstrates a simple query on the AST. The program in figure 10.1 calls an internal ROSE Query Library. Queries of the AST using the query library are particularly simple and often are useful as nested queries within more complex analysis. More information of the ROSE AST Query Library is available within ROSE User Manual. Using the input program in figure 10.2 the translator processes the code and generates the output in figure 10.3.

10.2

Nested Query

This section demonstrates a nested AST query, showing how to use composition in the construction of more elaborate queries from simple ones. The number of traversals of the AST can be reduced by using nested queries. Nested queries permits queries on the result from a NodeQuery. Another advantage is that nested (combined) queries can be formed to query for information without writing new query, the nested query is a new query. The program in figure 10.4 calls an internal ROSE Query Library. Two different queries are performed to find all access functions within the AST. The first query is nested, the returned list from a query is used in a traversal, and the second query queries the AST for the same nodes. Using the input program in figure 10.5 the translator processes the code and generates the output in figure 10.6.

83

FIXME: Put an example of composition of AST queries into the example input code.

84

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

CHAPTER 10. AST QUERY

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ;

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r

int counter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration ∗ f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n (∗ i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // DQ ( 3 / 5 / 2 0 0 6 ) : Only o u t p u t t h e non−c o m p i l e r g e n e r a t e d IR n o d e s i f ( ( ∗ i )−> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) { // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( ” F u n c t i o n #%2d name i s %s a t l i n e %d \n ” , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n −>get name ( ) . s t r ( ) , f u n c t i o n D e c l a r a t i o n −> g e t f i l e i n f o ()−> g e t l i n e ( ) ) ; } else { // Output s o m e t h i n g about t h e c o m p i l e r −g e n e r a t e d b u i l t i n f u n c t i o n s p r i n t f ( ” Compiler−g e n e r a t e d ( b u i l t i n ) f u n c t i o n #%2d name i s %s \n ” , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n −>get name ( ) . s t r ( ) ) ; } } // Note : Show c o m p o s i t i o n o f AST q u e r i e s return 0; }

Figure 10.1: Example source code for translator to read an input program and generate a list of functions in the AST (queryLibraryExample.C).

10.2. NESTED QUERY

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

85

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 10.2: Example source code used as input to program in figure 10.1 (queryLibraryExample.C).

86

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d Compiler−g e n e r a t e d

CHAPTER 10. AST QUERY

( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (

builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin

) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) )

function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function

# 0 # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82

name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name

is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is

builtin atan2 builtin atan2f builtin atan2l builtin abs builtin copysign builtin copysignf builtin copysignl builtin fmod builtin fmodf builtin fmodl builtin frexp builtin frexpf builtin frexpl builtin inf builtin inff builtin infl builtin ldexp builtin ldexpf builtin ldexpl builtin modf builtin modff builtin modfl builtin powi builtin powif builtin powil builtin pow builtin acos builtin acosf builtin acosl builtin acosh builtin acoshf builtin acoshl builtin asin builtin asinf builtin asinl builtin asinh builtin asinhf builtin asinhl builtin atan builtin atanf builtin atanl builtin atanh builtin atanhf builtin atanhl builtin cbrt builtin cbrtf builtin cbrtl builtin ceil builtin ceilf builtin ceill builtin cos builtin cosf builtin cosh builtin coshf builtin coshl builtin cosl builtin erf builtin erff builtin erfl builtin erfc builtin erfcf builtin erfcl builtin exp builtin expf builtin expl builtin exp2 builtin exp2f builtin exp2l builtin expm1 builtin expm1f builtin expm1l builtin fdim builtin fdimf builtin fdiml builtin floor builtin floorf builtin floorl builtin fma builtin fmaf builtin fmal builtin fmax builtin fmaxf builtin fmaxl

10.2. NESTED QUERY

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

87

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ;

// F u n c t i o n q u e r y S o l v e r A c c e s s F u n c t i o n s ( ) // f i n d a c c e s s f u n c t i o n s ( f u n c t i o n name s t a r t s w i t h ” g e t ” o r ” s e t ” ) NodeQuerySynthesizedAttributeType q u e r y S o l v e r A c c e s s F u n c t i o n s ( SgNode ∗ astNode ) { ROSE ASSERT ( astNode != 0 ) ; NodeQuerySynthesizedAttributeType returnNodeList ; S g F u n c t i o n D e c l a r a t i o n ∗ f u n c D e c l = i s S g F u n c t i o n D e c l a r a t i o n ( astNode ) ; if

( f u n c D e c l != NULL) { s t r i n g functionName = f u n c D e c l −>get name ( ) . s t r ( ) ; i f ( ( functionName . l e n g t h ( ) >= 4 ) && ( ( functionName . s u b s t r ( 0 , 4 ) == ” g e t ” ) r e t u r n N o d e L i s t . p u s h b a c k ( astNode ) ; }

||

( functionName . s u b s t r ( 0 , 4 ) == ” s e t

return returnNodeList ; } // F u n c t i o n p r i n t F u n c t i o n D e c l a r a t i o n L i s t w i l l p r i n t a l l f u n c t i o n names i n t h e l i s t v o i d p r i n t F u n c t i o n D e c l a r a t i o n L i s t ( R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t ) { int counter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L i s t . end ( ) ; { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration ∗ f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n (∗ i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( ” f u n c t i o n name #%d i s %s a t l i n e %d \n ” , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n −>get name ( ) . s t r ( ) , f u n c t i o n D e c l a r a t i o n −> g e t f i l e i n f o ()−> g e t l i n e ( ) ) ; } } i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST and f i n d // ( f u n c t i o n name s t a r t s w i t h ” g e t ” o r ” s e t ” )

all

access

functions

// B u i l d l i s t u s i n g a q u e r y o f t h e whole AST R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n ) ; // B u i l d l i s t u s i n g n e s t e d Q u e r i e s ( o p e r a t i n g on r e t u r n r e s u l t o f p r e v i o u s q u e r y ) R o s e S T L C o n t a i n e r a c c e s s F u n c t i o n s L i s t ; a c c e s s F u n c t i o n s L i s t = NodeQuery : : q u e r y N o d e L i s t ( f u n c t i o n D e c l a r a t i o n L i s t ,& q u e r y S o l v e r A c c e s s F u n c t i o n s ) ; printFunctionDeclarationList ( accessFunctionsList ); // A l t e r n a t i v e form o f same q u e r y b u i l d i n g t h e l i s t u s i n g a q u e r y o f t h e whole AST a c c e s s F u n c t i o n s L i s t = NodeQuery : : querySubTree ( p r o j e c t ,& q u e r y S o l v e r A c c e s s F u n c t i o n s ) ; printFunctionDeclarationList ( accessFunctionsList ); // Another way t o q u e r y f o r c o l l e c t i o n s o f IR n o d e s V a r i a n t V e c t o r vv1 = V S g C l a s s D e f i n i t i o n ; s t d : : c o u t << ”Number o f c l a s s d e f i n i t i o n s i n t h e memory p o o l

i s : ” << NodeQuery : : queryMemoryPool ( vv1 ) . s i z e ( ) << s t d :

// Another way t o q u e r y f o r c o l l e c t i o n s o f m u l t i p l e IR n o d e s . // V a r i a n t V e c t o r ( V SgType ) i s i n t e r n a l l y expanded t o a l l IR n o d e s d e r i v e d from SgType . V a r i a n t V e c t o r vv2 = V a r i a n t V e c t o r ( V S g C l a s s D e f i n i t i o n ) + V a r i a n t V e c t o r ( V SgType ) ; s t d : : c o u t << ”Number o f c l a s s d e f i n i t i o n s AND t y p e s i n t h e memory p o o l i s : ” << NodeQuery : : queryMemoryPool ( vv2 ) . s i z e // Note : Show c o m p o s i t i o n o f AST q u e r i e s return 0; }

88

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

CHAPTER 10. AST QUERY

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 10.5: Example source code used as input to program in figure 10.4 (nestedQueryExample.C). 1 2 3 4 5 6 7 8 9 10 11 12 13 14

f u n c t i o n name #0 i s g e t f o o f u n c t i o n name #1 i s s e t f o o f u n c t i o n name #2 i s g e t f o o f u n c t i o n name #3 i s s e t f o o f u n c t i o n name #4 i s g e t f o o f u n c t i o n name #5 i s s e t f o o f u n c t i o n name #0 i s g e t f o o f u n c t i o n name #1 i s s e t f o o f u n c t i o n name #2 i s g e t f o o f u n c t i o n name #3 i s s e t f o o f u n c t i o n name #4 i s g e t f o o f u n c t i o n name #5 i s s e t f o o Number o f c l a s s d e f i n i t i o n s Number o f c l a s s d e f i n i t i o n s

a t l i n e 10 a t l i n e 11 at l i n e 0 at l i n e 0 a t l i n e 28 a t l i n e 29 a t l i n e 10 a t l i n e 11 at l i n e 0 at l i n e 0 a t l i n e 28 a t l i n e 29 i n t h e memory p o o l i s : 3 AND t y p e s i n t h e memory p o o l

i s : 175

Figure 10.6: Output of input file to the AST query processor (nestedQueryExample.C).

Chapter 11

AST File I/O Figure 11.1 shows an example of how to use the AST File I/O mechanism. This chapter presents an example translator to write out an AST to a file and then read it back in.

11.1

Source Code for File I/O

Figure 11.1 shows an example translator which reads an input application, forms the AST, writes out the AST to a file, then deletes the AST and reads the AST from the previously written file. The input code is shown in figure 11.2, the output of this code is shown in figure 11.3.

11.2

Input to Demonstrate File I/O

Figure 11.2 shows the example input used for demonstration of the AST file I/O. In this case we are reusing the example used in the inlining example.

11.3

Output from File I/O

Figure 11.3 shows the output from the example file I/O tutorial example.

11.4

Final Code After Passing Through File I/O

Figure 11.4 shows the same file as the input demonstrating that the file I/O didn’t change the resulting generated code. Much more sophisticated tests are applied internally to verify the correctness of the AST after AST file I/O.

89

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

CHAPTER 11. AST FILE I/O

// Example d e m o n s t r a t i n g f u n c t i o n

i n l i n i n g ( maximal i n l i n i n g , up t o p r e s e t number o f

inlinings ).

#i n c l u d e ” r o s e . h” u s i n g namespace s t d ; // T h i s i s a f u n c t i o n i n Qing ’ s AST i n t e r f a c e v o i d F i x S g P r o j e c t ( S g P r o j e c t& p r o j ) ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // DQ ( 7 / 2 0 / 2 0 0 4 ) : Added i n t e r n a l c o n s i s t a n c y AstTests : : runAllTests ( p r o j e c t ) ;

t e s t s on AST

b o o l modifiedAST = t r u e ; int count = 0; // I n l i n e one c a l l a t a t i m e u n t i l do { modifiedAST = f a l s e ;

a l l have been i n l i n e d .

Loops on r e c u r s i v e c o d e .

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) // Loop o v e r a l l f u n c t i o n c a l l s // f o r ( l i s t : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; i != f u n c t i o n C a l l L i s t . end ( ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; w h i l e ( modifiedAST == f a l s e && i != f u n c t i o n C a l l L i s t . end ( ) ) { SgFunctionCallExp ∗ f u n c t i o n C a l l = isSgFunctionCallExp (∗ i ) ; ROSE ASSERT( f u n c t i o n C a l l != NULL ) ; #i f 0 f u n c t i o n C a l l −> g e t f i l e i n f o ()−> d i s p l a y ( ” i n l i n i n g

f unc t ion at fu nc ti on

call ”);

#e n d i f #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

#e n d i f // Not a l l f u n c t i o n c a l l s can be i n l i n e d i n C++, s o r e p o r t bool s u c e s s f u l l y I n l i n e d = doInline ( functionCall ) ;

if

successful .

#i f 0 p r i n t f ( ” s u c e s s f u l l y I n l i n e d = %s \n ” , s u c e s s f u l l y I n l i n e d ? ” t r u e ” : ” f a l s e ” ) ; #e n d i f #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

#e n d i f if

( s u c e s s f u l l y I n l i n e d == t r u e ) { // As s o o n a s t h e AST i s m o d i f i e d recompute t h e l i s t o f f u n c t i o n // c a l l s ( and r e s t a r t t h e i t e r a t i o n s o v e r t h e m o d i f i e d l i s t ) modifiedAST = t r u e ; } else { modifiedAST = f a l s e ; }

// I n c r e m e n t t h e i ++; }

list

iterator

// Q u i t e when we have c e a s e d t o do any i n l i n e t r a n s f o r m a t i o n s // and o n l y do a p r e d e f i n e d number o f i n l i n e t r a n s f o r m a t i o n s c o u n t++; } w h i l e ( modifiedAST == t r u e && c o u n t < 1 0 ) ; // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

// C a l l f u n c t i o n t o p o s t p r o c e s s t h e AST and f i x u p symbol t a b l e s FixSgProject (∗ p r o j e c t ) ; #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g c o n s i s t a n c y . checkTransformedFlagsVisitor ( project ) ;

i ++)

11.4. FINAL CODE AFTER PASSING THROUGH FILE I/O

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah ’ s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t ”x” v o i d incrementX ( ) { x++; } int foo () { int a = 0; while ( a < 5) { ++a ; } return a + 3; } i n t main ( i n t , c h a r ∗ ∗ ) { // Two t r i v a l f u n c t i o n incrementX ( ) ; incrementX ( ) ;

calls

to i n l i n e

// Something more i n t e r e s t i n g t o i n l i n e // f o r ( ; f o o ( ) < 7 ; ) int i = 0; f o r ( ; f o o ( ) < 7 ; i ++) { x++; } return x ; }

Figure 11.2: Example source code used as input to demonstrate the AST file I/O support.

Figure 11.3: Output of input code after inlining transformations.

91

92

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

CHAPTER 11. AST FILE I/O

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah ’ s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t ”x” v o i d incrementX ( ) { x++; } int foo () { int a 0 = 0; while ( a 0 < 5){ ++a 0 ; } return a 0 + 3; } i n t main ( i n t , c h a r ∗ ∗ ) { x++; x++; // Something more i n t e r e s t i n g t o i n l i n e // f o r ( ; f o o ( ) < 7 ; ) int i 0 = 0; f o r ( ; t r u e ; i 0 ++) { int a 2 = 0; while ( a 2 < 5){ ++a 2 ; } int rose temp 7 1 = a 2 + 3; bool rose temp 3 = ( bool )( rose temp i f (! rose temp 3 ) { break ; } else { } x++; } return x ; }

7

1 < 7);

Figure 11.4: Output of input code after file I/O.

Chapter 12

Debugging Techniques There are numerous methods ROSE provides to help debug the development of specialized source-to-source translators. This section shows some of the techniques for getting information from IR nodes and displaying it. More information about generation of specialized AST graphs to support debugging can be found in chapter 5 and custom graph generation in section 28.

12.1

Input For Examples Showing Debugging Techniques

Figure 12.1 shows the input code used for the example translators that report useful debugging information in this chapter. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

// Example program showing m a t r i x m u l t i p l y // ( f o r u s e w i t h l o o p o p t i m i z a t i o n t u t o r i a l example ) #d e f i n e N 50 i n t main ( ) { int i , j , k ; d o u b l e a [N ] [ N] , b [ N ] [ N ] , c [ N ] [ N ] ; f o r ( i = 0 ; i <= N−1; i +=1) { f o r ( j = 0 ; j <= N−1; j +=1) { f o r ( k = 0 ; k <= N−1; k+=1) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } } return 0; }

Figure 12.1: Example source code used as input to program in codes showing debugging techniques shown in this section.

93

94

12.2

CHAPTER 12. DEBUGGING TECHNIQUES

Generating the code from any IR node

Any IR node may be converted to the string that represents its subtree within the AST. If it is a type, then the string will be the value of the type; if it is a statement, the value will be the source code associated with that statement, including any sub-statements. To support the generation for strings from IR nodes we use the unparseToString() member function. This function strips comments and preprocessor control structure. The resulting string is useful for both debugging and when forming larger strings associated with the specification of transformations using the string-based rewrite mechanism. Using ROSE, IR nodes may be converted to strings, and strings converted to AST fragments of IR nodes. Note that unparsing associated with generating source code for the backend vendor compiler is more than just calling the unparseToString member function, since it introduces comments, preprocessor control structure and formating. Figure 12.2 shows a translator which generates a string for a number of predefined IR nodes. Figure 12.1 shows the sample input code and figure 12.5 shows the output from the translator when using the example input application.

12.3

Displaying the source code position of any IR node

This example shows how to obtain information about the position of any IR node relative to where it appeared in the original source code. New IR nodes (or subtrees) that are added to the AST as part of a transformation will be marked as part of a transformation and have no position in the source code. Shared IR nodes (as generated by the AST merge mechanism are marked as shared explicitly (other IR nodes that are shared by definition don’t have a SgFileInfo object and are thus not marked explicitly as shared. The example translator to output the source code position is shown in figure 12.4. Using the input code in figure 12.1 the output code is shown in figure 12.5.

12.3. DISPLAYING THE SOURCE CODE POSITION OF ANY IR NODE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

95

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; int main ( i n t a r g c , c h a r ∗ a r g v [ ] { ios : : sync with stdio ();

) // Syncs C++ and C I /O s u b s y s t e m s !

// I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; if

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( ” I n p r e p r o c e s s o r . C : main ( ) \n ” ) ;

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // AST d i a g n o s t i c t e s t s A s t T e s t s : : r u n A l l T e s t s ( c o n s t c a s t ( p r o j e c t ) ) ; // t e s t s t a t i s t i c s i f ( p r o j e c t −>g e t v e r b o s e ( ) > 1 ) { c o u t << A s t N o d e S t a t i s t i c s : : t r a v e r s a l S t a t i s t i c s ( p r o j e c t ) ; c o u t << A s t N o d e S t a t i s t i c s : : I R n o d e U s a g e S t a t i s t i c s ( ) ; } ( p r o j e c t −>g e t v e r b o s e ( ) > 0 ) p r i n t f ( ” G e n e r a t e t h e p d f o u t p u t o f t h e SAGE I I I AST \n ” ) ; generatePDF ( ∗ p r o j e c t ) ; if

( p r o j e c t −>g e t v e r b o s e ( ) > 0 ) p r i n t f ( ” G e n e r a t e t h e DOT o u t p u t o f t h e SAGE I I I AST \n ” ) ; generateDOT ( ∗ p r o j e c t ) ; if

R o s e S T L C o n t a i n e r n o d e L i s t ; // n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgType , NodeQuery : : E x t r a c t T y p e s ) ; n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgForStatement ) ; p r i n t f ( ” \ n n o d e L i s t . s i z e ( ) = %zu \n ” , n o d e L i s t . s i z e ( ) ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { p r i n t f ( ” Query node = %p = %s = %s \n ” , ∗ i , ( ∗ i )−> s a g e c l a s s n a m e ( ) , ( ∗ i )−>u n p a r s e T o S t r i n g ( ) . c s t r ( ) ) ; i ++; } return 0; }

Figure 12.2: Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node.

96

1 2 3 4 5

CHAPTER 12. DEBUGGING TECHNIQUES

nodeList . s i z e () = 3 Query node = 0 x 7 f e 5 6 d 3 3 f 0 1 0 = Sg ForSt atem ent = f o r ( i = 0 ; i <= 50 − 1 ; i += 1 ) { f o r ( j = 0 ; j <= 50 − 1 ; j += 1 ) { f Query node = 0 x 7 f e 5 6 d 3 3 f 1 3 8 = Sg ForSt atem ent = f o r ( j = 0 ; j <= 50 − 1 ; j += 1 ) { f o r ( k = 0 ; k <= 50 − 1 ; k += 1 ) { c Query node = 0 x 7 f e 5 6 d 3 3 f 2 6 0 = Sg ForSt atem ent = f o r ( k = 0 ; k <= 50 − 1 ; k += 1 ) { c [ i ] [ j ] = c [ i ] [ j ] + a [ i ] [ k ] ∗ b [

Figure 12.3: Output of input code using debuggingIRnodeToString.C 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; int main ( i n t a r g c , c h a r ∗ a r g v [ ] { ROSE INITIALIZE ; if

)

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( ” I n p r e p r o c e s s o r . C : main ( ) \n ” ) ;

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; R o s e S T L C o n t a i n e r n o d e L i s t ; n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgForStatement ) ; p r i n t f ( ” \ n n o d e L i s t . s i z e ( ) = %zu \n ” , n o d e L i s t . s i z e ( ) ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { S g F i l e I n f o & f i l e I n f o = ∗ ( ( ∗ i )−> g e t f i l e i n f o ( ) ) ; p r i n t f ( ” Query node = %p = %s i n %s \n −−−−− a t l i n e %d on column %d \n ” , ∗ i , ( ∗ i )−> s a g e c l a s s n a m e ( ) , f i l e I n f o . g e t f i l e n a m e ( ) , f i l e I n f o . get line () , f i l e I n f o . get col ()); i ++; } if

( p r o j e c t −>g e t v e r b o s e ( ) > 0 ) p r i n t f ( ” C a l l i n g t h e backend ( ) \n ” ) ;

return 0; }

Figure 12.4: Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node. 1 2 3 4 5 6 7 8

nodeList . s i z e () = 3 Query node = 0 x 7 f d f 6 3 8 e f 0 1 0 −−−−− a t l i n e 11 on column Query node = 0 x 7 f d f 6 3 8 e f 1 3 8 −−−−− a t l i n e 13 on column Query node = 0 x 7 f d f 6 3 8 e f 2 6 0 −−−−− a t l i n e 15 on column

= S gFor State ment i n / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs 6 = S gFor State ment i n / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs 11 = S gFor State ment i n / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs 16

hared IR nodes (as generated by the AST merge mechanism are mar

Figure 12.5: Output of input code using debuggingSourceCodePositionInformation.C

Part II

Complex Types

This part elaborates some details for handling complex types in ROSE.

97

Chapter 13

Type and Declaration Modifiers Most languages support the general concept of modifiers to types, declarations, etc. The keyword volatile for example is a modifier to the type where it is used in a declaration. Searching for the modifiers for types and declarations,however, can be confusing. They are often not where one would expect, and most often because of corner cases in the language that force them to be handled in specific ways. This example tutorial code is a demonstration of a how to access the volatile modifier used in the declaration of types for variables. We demonstrate that the modifier is not present in the SgVariableDeclaration or the SgVariableDefinitoon, but is located in the SgModifierType used to wrap the type returned from the SgInitializedName (the variable in the variable declaration).

13.1

Input For Example Showing use of Volatile type modifier

Figure 13.1 shows the example input used for demonstration of test for the volatile type modifier.

1 2 3 4 5 6 7 8 9 10

// I n p u t example o f u s e o f ” v o l a t i l e ” t y p e m o d i f i e r volatile

int a ,∗b ;

void foo () { for ( volatile { } }

i n t y = 0 ; y < 1 0 ; y++)

Figure 13.1: Example source code used as input to program in codes used in this chapter.

99

100

CHAPTER 13. TYPE AND DECLARATION MODIFIERS

13.2

Generating the code representing the seeded bug

Figure 13.2 shows a code that traverses each IR node and for and SgInitializedName IR node checks its type. The input code is shown in figure 13.1, the output of this code is shown in figure 13.3. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

#i n c l u d e

” r o s e . h”

using

namespace

class {

visitorTraversal public : void

std ; :

public

AstSimpleProcessing

v i s i t ( SgNode ∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode ∗ n ) { // The ” v o l a t i l e ” m a d i f i e r i s i n t h e t y p e o f t h e S g I n i t i a l i z e d N a m e SgInitializedName∗ initializedName = isSgInitializedName (n ) ; i f ( i n i t i a l i z e d N a m e != NULL) { p r i n t f ( ” Found a S g I n i t i a l i z e d N a m e = %s \n ” , i n i t i a l i z e d N a m e −>g e t n a m e ( ) . s t r ( ) ) ; SgType ∗ t y p e = i n i t i a l i z e d N a m e −>g e t t y p e ( ) ; p r i n t f (” i n i t i a l i z e d N a m e : t y p e = %p = %s \n ” , t y p e , t y p e−>c l a s s n a m e ( ) . c s t r ( ) ) ; SgModifierType ∗ modifierType = isSgModifierType ( type ) ; i f ( m o d i f i e r T y p e != NULL) { b o o l i s V o l a t i l e = m o d i f i e r T y p e −>g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V o l a t i l e ( ) ; p r i n t f (” i n i t i a l i z e d N a m e : S g M o d i f i e r T y p e : i s V o l a t i l e = %s \n ” , ( i s V o l a t i l e == t r u e ) ? ” t r u e ” } S g M o d i f i e r N o d e s ∗ m o d i f i e r N o d e s = t y p e−>g e t m o d i f i e r s ( ) ; p r i n t f (” i n i t i a l i z e d N a m e : m o d i f i e r N o d e s = %p \n ” , m o d i f i e r N o d e s ) ; i f ( m o d i f i e r N o d e s != NULL) { S g M o d i f i e r T y p e P t r V e c t o r m o d i f i e r L i s t = m o d i f i e r N o d e s −>g e t n o d e s ( ) ; f o r ( S g M o d i f i e r T y p e P t r V e c t o r : : i t e r a t o r i = m o d i f i e r L i s t . b e g i n ( ) ; i != m o d i f i e r L i s t . end ( ) ; { p r i n t f ( ” i n i t i a l i z e d N a m e : m o d i f i e r s : i = %s \n ” , ( ∗ i )−> c l a s s n a m e ( ) . c s t r ( ) ) ; } }

:

” false ”);

i ++)

}

// Note t h a t t h e ” v o l a t i l e ” m a d i f i e r i s n o t i n t h e S g V a r i a b l e D e c l a r a t i o n n o r t h e S g V a r i a b l e D e f i n i t i o n SgVariableDeclaration∗ variableDeclaration = isSgVariableDeclaration (n ) ; i f ( v a r i a b l e D e c l a r a t i o n != NULL) { b o o l i s V o l a t i l e = v a r i a b l e D e c l a r a t i o n −>g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V o l a t p r i n t f ( ” S g V a r i a b l e D e c l a r a t i o n : i s V o l a t i l e = %s \n ” , ( i s V o l a t i l e == t r u e ) ? ” t r u e ” : ” f a l s e ” ) ; S g V a r i a b l e D e f i n i t i o n ∗ v a r i a b l e D e f i n i t i o n = v a r i a b l e D e c l a r a t i o n −>g e t d e f i n i t i o n ( ) ; // p r i n t f ( ” v a r i a b l e D e f i n i t i o n = %p \n ” , v a r i a b l e D e f i n i t i o n ) ; i f ( v a r i a b l e D e f i n i t i o n != NULL) { b o o l i s V o l a t i l e = v a r i a b l e D e f i n i t i o n −>g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V p r i n t f ( ” S g V a r i a b l e D e f i n i t i o n : i s V o l a t i l e = %s \n ” , ( i s V o l a t i l e == t r u e ) ? ” t r u e ” : ” f a l s e ” ) ; } } } // must h a v e a r g c and a r g v h e r e ! ! i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ; SgProject

∗ project = frontend

( argc ,

See

Rose : : i n i t i a l i z e

argv ) ;

visitorTraversal myvisitor ; myvisitor . traverseInputFiles ( project , preorder ) ; return

backend ( p r o j e c t ) ;

}

Figure 13.2: Example source code showing how to detect volatile modifier.

13.2. GENERATING THE CODE REPRESENTING THE SEEDED BUG

1 2 3 4 5 6 7 8 9

// I n p u t example o f u s e o f ” v o l a t i l e ” t y p e m o d i f i e r volatile int a ; v o l a t i l e i n t ∗b ; void foo () { for ( volatile } }

i n t y = 0 ; y < 1 0 ; y++) {

Figure 13.3: Output of input code using volatileTypeModifier.C

101

102

CHAPTER 13. TYPE AND DECLARATION MODIFIERS

Chapter 14

Function Parameter Types The analysis of functions often requires the query of the function types. This tutorial example shows how to obtain the function parameter types for any function. Note that functions also have a type which is based on their signature, a combination of their return type and functions parameter types. Any functions sharing the same return type and function parameter types have the same function type (the function type, a SgFunctionType IR node, will be shared between such functions). Figure 14.1 shows a translator which reads an application (shown in figure 14.2) and outputs information about the function parameter types for each function, shown in figure 14.3. This information includes the order of the function declaration in the global scope, and name of the function, and the types of each parameter declared in the function declaration. Note that there are a number of builtin functions defined as part of the GNU g++ and gcc compatibility and these are output as well. These are marked as compiler generated functions within ROSE. The code shows how to differentiate between the two different types. Notice also that instantiated template functions are classified as compiler generated.

103

104

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

CHAPTER 14. FUNCTION PARAMETER TYPES

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ;

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r

int functionCounter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration ∗ f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n (∗ i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // DQ ( 3 / 5 / 2 0 0 6 ) : Only o u t p u t t h e non−c o m p i l e r g e n e r a t e d IR n o d e s i f ( ( ∗ i )−> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) { S g F u n c t i o n P a r a m e t e r L i s t ∗ f u n c t i o n P a r a m e t e r s = f u n c t i o n D e c l a r a t i o n −>g e t p a r a m e t e r L i s t ( ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( ” Non−c o m p i l e r g e n e r a t e d f u n c t i o n name #%3d i s %s \n ” , f u n c t i o n C o u n t e r ++, f u n c t i o n D e c l a r a t

S g I n i t i a l i z e d N a m e P t r L i s t & p a r a m e t e r L i s t = f u n c t i o n P a r a m e t e r s −>g e t a r g s ( ) ; i n t parameterCounter = 0 ; f o r ( S g I n i t i a l i z e d N a m e P t r L i s t : : i t e r a t o r j = p a r a m e t e r L i s t . b e g i n ( ) ; j != p a r a m e t e r L i s t . end ( ) ; j + { SgType∗ parameterType = ( ∗ j )−> g e t t y p e ( ) ; p r i n t f (” parameterType #%2d = %s \n ” , p a r a m e t e r C o u n t e r ++,parameterType−>u n p a r s e T o S t r i n g ( }

} else { p r i n t f ( ” C o m p i l e r g e n e r a t e d f u n c t i o n name #%3d i s %s \n ” , f u n c t i o n C o u n t e r ++, f u n c t i o n D e c l a r a t i o n − } } return 0; }

Figure 14.1: Example source code showing how to get type information from function parameters.

105

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] { foo (42); foo (3.14159265);

)

t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 14.2: Example source code used as input to typeInfoFromFunctionParameters.C.

106

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler

CHAPTER 14. FUNCTION PARAMETER TYPES

generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated

function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function

name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82

is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is

builtin atan2 builtin atan2f builtin atan2l builtin abs builtin copysign builtin copysignf builtin copysignl builtin fmod builtin fmodf builtin fmodl builtin frexp builtin frexpf builtin frexpl builtin inf builtin inff builtin infl builtin ldexp builtin ldexpf builtin ldexpl builtin modf builtin modff builtin modfl builtin powi builtin powif builtin powil builtin pow builtin acos builtin acosf builtin acosl builtin acosh builtin acoshf builtin acoshl builtin asin builtin asinf builtin asinl builtin asinh builtin asinhf builtin asinhl builtin atan builtin atanf builtin atanl builtin atanh builtin atanhf builtin atanhl builtin cbrt builtin cbrtf builtin cbrtl builtin ceil builtin ceilf builtin ceill builtin cos builtin cosf builtin cosh builtin coshf builtin coshl builtin cosl builtin erf builtin erff builtin erfl builtin erfc builtin erfcf builtin erfcl builtin exp builtin expf builtin expl builtin exp2 builtin exp2f builtin exp2l builtin expm1 builtin expm1f builtin expm1l builtin fdim builtin fdimf builtin fdiml builtin floor builtin floorf builtin floorl builtin fma builtin fmaf builtin fmal builtin fmax builtin fmaxf builtin fmaxl

Chapter 15

Resolving Overloaded Functions Figure 15.1 shows a translator which reads an application and reposts on the mapping between function calls and function declarations. This is trivial since all overloaded function resolution is done within the frontend and so need not be computed (this is because all type resolution is done in the frontend and stored in the AST explicitly). Other compiler infrastructures often require this to be figured out from the AST, when type resolution is unavailable, and while not too hard for C, this is particularly complex for C++ (due to overloading and type promotion within function arguments). Figure 15.2 shows the input code used to get the translator. Figure 15.3 shows the resulting output.

107

108

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

CHAPTER 15. RESOLVING OVERLOADED FUNCTIONS

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) ; int functionCounter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; { SgFunctionCallExp ∗ functionCallExp = isSgFunctionCallExp (∗ i ) ; ROSE ASSERT( f u n c t i o n C a l l E x p != NULL ) ;

i != f u n c t i o n C a l l L i s t . end ( ) ;

i ++

S g E x p r e s s i o n ∗ f u n c t i o n E x p r e s s i o n = f u n c t i o n C a l l E x p −>g e t f u n c t i o n ( ) ; ROSE ASSERT( f u n c t i o n E x p r e s s i o n != NULL ) ; SgFunctionRefExp ∗ f u n c t i o n R e f E x p = i s S g F u n c t i o n R e f E x p ( f u n c t i o n E x p r e s s i o n ) ; SgFunctionSymbol ∗ f u n c t i o n S y m b o l = NULL ; i f ( f u n c t i o n R e f E x p != NULL) { // Case o f non−member f u n c t i o n f u n c t i o n S y m b o l = f u n c t i o n R e f E x p −>g e t s y m b o l ( ) ; } else { // Case o f member f u n c t i o n ( h i d d e n i n r h s o f b i n a r y d o t o p e r a t o r e x p r e s s i o n ) SgDotExp∗ dotExp = isSgDotExp ( f u n c t i o n E x p r e s s i o n ) ; ROSE ASSERT( dotExp != NULL ) ; f u n c t i o n E x p r e s s i o n = dotExp−>g e t r h s o p e r a n d ( ) ; SgMemberFunctionRefExp ∗ memberFunctionRefExp = isSgMemberFunctionRefExp ( f u n c t i o n E x p r e s s i o n ) ; ROSE ASSERT( memberFunctionRefExp != NULL ) ; f u n c t i o n S y m b o l = memberFunctionRefExp−>g e t s y m b o l ( ) ; } ROSE ASSERT( f u n c t i o n S y m b o l != NULL ) ; S g F u n c t i o n D e c l a r a t i o n ∗ f u n c t i o n D e c l a r a t i o n = f u n c t i o n S y m b o l −>g e t d e c l a r a t i o n ( ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // Output mapping o f f u n c t i o n c a l l s t o f u n c t i o n d e c l a r a t i o n s p r i n t f ( ” L o c a t i o n o f f u n c t i o n c a l l #%d a t l i n e %d r e s o l v e d by o v e r l o a d e d f u n c t i o n d e c l a r e d a t l i n e f u n c t i o n C o u n t e r ++, f u n c t i o n C a l l E x p −> g e t f i l e i n f o ()−> g e t l i n e ( ) , f u n c t i o n D e c l a r a t i o n −> g e t f i l e i n f o ()−> g e t l i n e ( ) ) ; } return 0; }

Figure 15.1: Example source code showing mapping of function calls to overloaded function declarations.

109

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// Added t o a l l o w non− t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 15.2: Example source code used as input to resolveOverloadedFunction.C.

1 2 3 4

Location Location Location Location

of of of of

function function function function

call call call call

#0 #1 #2 #3

at at at at

line line line line

29 30 33 34

resolved resolved resolved resolved

by by by by

overloaded overloaded overloaded overloaded

function function function function

Figure 15.3: Output of input to resolveOverloadedFunction.C.

declared declared declared declared

at at at at

line line line line

14 0 0 0

110

CHAPTER 15. RESOLVING OVERLOADED FUNCTIONS

Chapter 16

Template Parameter Extraction Figure 16.1 shows a translator which reads an application and gathers a list of loop nests. At the end of the traversal it reports information about each instantiated template, including the template arguments. Figure 16.2 shows the input code used to get the translator. Figure 16.3 shows the resulting output.

111

112

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

CHAPTER 16. TEMPLATE PARAMETER EXTRACTION

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r t e m p l a t e I n s t a n t i a t i o n D e c l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g T e m p l a t e I n s t a n t i a t i o n D e c l ) ; i n t classTemplateCounter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = t e m p l a t e I n s t a n t i a t i o n D e c l L i s t . b e g i n ( ) ; i != t e m p l a t e I n s t a n t i a t i o n D e c l L i s t . end ( ) ; i ++) { SgTemplateInstantiationDecl ∗ instantiatedTemplateClass = i s S g T e m p l a t e I n s t a n t i a t i o n D e c l (∗ i ) ; ROSE ASSERT( i n s t a n t i a t e d T e m p l a t e C l a s s != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( ” C l a s s name #%d i s %s \n ” , c l a s s T e m p l a t e C o u n t e r ++, i n s t a n t i a t e d T e m p l a t e C l a s s −>g e t t e m p l a t e N a m e ( ) . s t r ( ) ) ;

c o n s t SgTemplateArgumentPtrList& t e m p l a t e P a r a m e t e r L i s t = i n s t a n t i a t e d T e m p l a t e C l a s s −>g e t t e m p l a t e A r g u m i n t parameterCounter = 0 ; f o r ( SgTemplateArgumentPtrList : : c o n s t i t e r a t o r j = t e m p l a t e P a r a m e t e r L i s t . b e g i n ( ) ; j != t e m p l a t e P a r a m e t e r L i s t . end ( ) ; j ++) { p r i n t f (” TemplateArgument #%d = %s \n ” , p a r a m e t e r C o u n t e r ++,(∗ j )−>u n p a r s e T o S t r i n g ( ) . c s t r ( ) ) ; } } return 0; }

Figure 16.1: Example source code used to extract template parameter information.

113

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; };

i n t main ( ) { t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s I n t ; t e m p l a t e C l a s s i n s t a n t i a t e d C l a s s F l o a t ; t e m p l a t e C l a s s > i n s t a n t i a t e d C l a s s N e s t e d C h a r ; f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 16.2: Example source code used as input to templateParameter.C.

1 2 3 4 5 6 7 8

C l a s s name #0 i s t e m p l a t e C l a s s TemplateArgument #0 = c h a r C l a s s name #1 i s t e m p l a t e C l a s s TemplateArgument #0 = i n t C l a s s name #2 i s t e m p l a t e C l a s s TemplateArgument #0 = f l o a t C l a s s name #3 i s t e m p l a t e C l a s s TemplateArgument #0 = t e m p l a t e C l a s s < c h a r >

Figure 16.3: Output of input to templateParameter.C.

114

CHAPTER 16. TEMPLATE PARAMETER EXTRACTION

Chapter 17

Template Support This chapter is specific to demonstrating the C++ template support in ROSE. This section is not an introduction to the general subject of C++ templates. ROSE provides special handling for C++ templates because template instantiation must be controlled by the compiler. Templates that require instantiation are instantiated by ROSE and can be seen in the traversal of the AST (and transformed). Any templates that can be instantiated by the backend compiler and are not transformed are not output within the code generation phase.

17.1

Example Template Code #1

This section presents figure 17.4, a simple C++ source code using a template. It is used as a basis for showing how template instantiations are handled within ROSE. 1 2 3 4 5 6 7 8 9 10 11 12

t e m p l a t e class X { public : void foo ( ) ; }; X x ; v o i d X : : f o o ( ) { }

Figure 17.1: Example source code showing use of a C++ template.

17.2

Example Template Code #2

This section presents figure 17.4, a simple C++ source code using a template function. It is used as a basis for showing how template instantiations are handled within ROSE.

115

FIXME: Provide a list of when templates are generated internally in the AST and when template instantiations are output.

116

1 2 3 4 5 6 7 8 9 10 11

CHAPTER 17. TEMPLATE SUPPORT

t e m p l a t e < typename T > class X { public : void foo ( ) ; }; c l a s s X< i n t > x ; t e m p l a t e <> v o i d X< i n t > : : f o o ( ) { }

Figure 17.2: Example source code after processing using identityTranslator (shown in figure 2.1).

1 2 3 4 5 6 7 8 9 10 11 12 13

// t e m p l a t e f u n c t i o n t e m p l a t e void foo ( T t ) { } // S p e c i a l i z a t i o n from u s e r t e m p l a t e <> v o i d f o o ( i n t x ) {} i n t main ( ) { foo ( 1 ) ; }

Figure 17.3: Example source code showing use of a C++ template.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

// t e m p l a t e f u n c t i o n t e m p l a t e < typename T > void foo ( T t ) { } // S p e c i a l i z a t i o n from u s e r t e m p l a t e <> v o i d f o o < i n t { } i n t main ( ) { : : foo< i n t }

> ( int x)

> (1);

Figure 17.4: Example source code after processing using identityTranslator (shown in figure 2.1).

Part III

Program Analyses

This part exemplifies the use of existing ROSE analyses and how to build customized analyses using ROSE.

117

Chapter 18

Generic Dataflow Analysis Framework This chapter summarizes the basic considerations behind the ROSE dataflow framework as well as how its API can be used to implement inter- and intra-procedural dataflow analyses. It is oriented towards potential users of the framework.

18.1

Basics of DataFlowAnalysis

Dataflow analysis is a technique for determining an applications possible states at various points in a program. It works as a fixed-point iteration over a space of possible facts about each node in the applications Control Flow Graph (CFG). The algorithm starts with no information about each node and iterates by accumulating all the constraints on the application’s state at each node until it reaches a fixed point where no additional constraints can be discovered. The designer of a given dataflow analysis must specify an abstract representation of the set of all possible application states that maintains the relevant details of the state (e.g. whether a variable has a constant value or the linear relationships between variable pairs), while igoring the rest. For example, a state abstraction for the constant propagation analysis may have three different values: the special symbol ⊥ if the variable is uninitialized, a numeric value if this is the only value the variable may hold at a given CFG node or > if it may have more than one value. More sophisticated abstractions represent application state using polyhedral constraints or predicate logic. Further, the designer must specify a “transfer” function that maps the abstract state before any application operation to the state after it. For example, if before statement i + + it is known that i == n then after the statement it is known that i − 1 == n. To deal with control flow the designer also specifies a meet function that conjoins the abstract states along multiple control paths. For example, if at the end of the if branch of a conditional it is known that i=5 and at the end of the else branch i<10, the strongest fact that is true immediately after both branches of the conditional is 5 ≤ i <10. The set of possible abstract states must form a lattice, which is a partial order where for any pair of elements there exists a unique least upper bound. Intuitively, states that are lower in the partial order represent fewer constraints on the application state and higher states represent 119

120

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.1: Example of a constant propagation analysis.

more constraints. The special state ⊥ corresponds to the least state in the partial order, where the application has done nothing to constrain its state (e.g. all variables are uninitialized). The meet function must guarantee the uniqueness of the least upper bound and the transfer function must be monotonic (if A≤B then transfer(A)≤transfer(B)). The dataflow fixed-point iteration ensures that the abstract state of every CFG node rises monotonically as it incorporates information about more possible application behaviors. When the analysis reaches a fixed point, the abstract state at each CFG node corresponds to the tightest set of constraints that are can be specified by the abstraction about the application state at that location. For an intuition about how dataflow analyses work, 18.1 presents an example of a constant propagation analysis. The CFG is on the left and the table on the right shows the fixed-point solution of the abstract state immediately after each node. At each node the abstract application state records for each variable one of the following values: (i) ⊥, which indicates that the variable is uninitialized, (ii) a specific constant value if the variable may only have this value at node n or (iii) > which indicates that the variable may have more than one value (i.e., is not representable as a single constant).It shows that immediately after node A it is known that i=0 and similarly after node B, i=0 and j=5. The same is true after node C since it has no side-effects and after the assignment in node D, the state changes to i=2, j=5. When the two conditional branches meet, the abstract state is the union of the states on both branches: the strongest assertions that are true of both states. Since j has the same value and i has two different values, the abstract state after node E is i=>, j=5. ?? presents an example with a more complex abstraction: conjunction of linear relationships between variables. At node B the dataflow analysis computes that i=0 and j=5. When this state is propagated through the loop, the analysis discovers that after the first iteration i=4 and j=5. It then computes the meet of i=0 ∧ j=5 and i=4 ∧ j=5, the facts along both paths. Since this abstraction represents linear relationships, the union finds the tightest linear relationships that are true of both input states. It thus infers that i=0 (mod 4), i=0 (mod 5) (i is divisible by 4 and j by 5) and that 5i=4j-5. When this state is propagated again through the body of the loop, these assertions are discovered to be the invariants of this loop and become the fixed-point solution after node C. If they were not invariants, the algorithm would iterate until invariants were found or it reached the abstract state > which means that no linear constraints are known. Further, since the conditional j<100 is also linear, j<100 is recorded in the states of the nodes inside the loop and j ≥ 100 is recorded at node F after the loop.

18.2. ROSE DATAFLOW FRAMEWORK

121

Figure 18.2: Example of a dataflow analysis with abstraction of affine constraints.

18.2

ROSE Dataflow Framework

ROSE provides a framework for implementing dataflow analyses. It allows users to specify their dataflow analysis by implement the standard dataflow components: (i) an abstraction of the applications state, (ii) a transfer function that specifies the effects of code on the application state and (iii) a meet operator that combines multiple possible abstract states into one. These are implemented by extending base classes provided by the framework and implementing key virtual methods that correspond to the above functionality. The framework then solves the dataflow equations using the user-provided classes and saves the results at each CFG node. This section describes the functionality provided by the framework and how it can be used to implement analyses. 18.1 summarizes the functionality provided by the framework.

18.2.1

Call and Control-Flow Graphs

The ROSE dataflow analysis framework operates on top of the ROSE Call Graph (CG) and Virtual Control-Flow Graph (VCFG). The CG documents the caller/callee relationships between application functions. The VCFG connects SgNodes in the applications AST to identify the possible execution orders between them. The VCFG is dynamic in that instead of being computing once for the entire application, it computes the outgoing and incoming edges of a given SgNode fresh every time this information is needed. This makes the VCFG very flexible because it automatically responds to changes in the AST with no need for complex adjustments to the graph.

18.2.2

Analyses

ROSE supports both inter-and intra-procedural analyses. Users implement basic, non-dataflow analyses by extending the IntraProceduralAnalysis and InterProceduralAnalysis classes. Intra analyses iterate over the CFG of each function, and inter analyses apply intra analyses to individual

122

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK Class Analysis

Purpose Implement Simple CFG passes

Intra Procedural Dataflow

Implement intraprocedural dataflow iteration Implement the inter-procedural dataflow Generic interface for abstractions of the application’s state Stores dataflow information of each CFG node

Inter Procedural Dataflow Lattice

Nodestate

AstInterface

Transforms the CFG

Interface Classes IntraProceduralAnalysis InterProceduralAnalysis Classes IntraFWDataflow IntraBWDataflow

User Responsibilities Extend the classes and implement their runAnalysis and transfer methods Extend classes and implement genInitState and transfer methods

Classes - Context Insensitive InterProcedural Dataflow Methods initialize copy meetUpdate operator== str

Execute on a given instance of IntraDataflow

Methods setLatticeAbove getLatticeAbove deleteLatticeAbove setFact getFact deleteFacts Methods insertBeforeUsing CommaOp, insertAfterUsing CommaOp, replaceWithPattern

Call methods to access dataflow information

Extend the Lattice class and implement interface methods

Call methods

Table 18.1: The Functionality of the Dataflow Interface functions. To implement an analysis an application developer must derive a class from the IntraProceduralAnalysis and/or InterProceduralAnalysis classes and implement the runAnalysis method. Classes UnstructuredPassInterAnalysis and UnstructuredPassIntraAnalysis Figure 18.2.3 provide examples of simple analyses. UnstructuredPassInterAnalysis takes as an argument a reference to an InterProceduralAnalysis and iterates once through all functions. It applies the runAnalysis method of the intra analysis to each function. UnstructuredPassIntraAnalysis iterates once through all the CFG nodes in the given function, applying its visit method to each node. These analyses can be used to implement simple passes through the applications CFG and serve as the foundation of the dataflow analysis framework. For example, src/simpleAnalyses/saveDotAnalysis.C and src/simpleAnalyses/printAnalysisStates.C are examples of simple one-pass analyses. saveDotAnalysis prints the applications CFG as a DOT file and printAnalysisStates prints the dataflow

1 2 3 4 5 6 7 8 9 10

c l a s s UnstructuredPassInterAnalysis : virtual public InterProceduralAnalysis { U n s t r u c t u r e d P a s s I n t e r A n a l y s i s ( I n t r a P r o c e d u r a l A n a l y s i s& i n t r a A n a l y s i s ) void runAnalysis ( ) ; }; c l a s s UnstructuredPassIntraAnalysis : virtual public IntraProceduralAnalysis { b o o l r u n A n a l y s i s ( c o n s t Function& func , NodeState ∗ s t a t e ) ; v i r t u a l v o i d v i s i t ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e )=0; }; Figure 18.3: Example of simple analyses

18.2. ROSE DATAFLOW FRAMEWORK

123

Figure 18.4: Each variable’s lattice for constant-propagation analysis states of all CFG nodes in the application, which is useful for debugging.

18.2.3

Dataflow

To implement a dataflow analysis in ROSE users must first extend the Lattice class to create an abstraction of the applications state that will be used by the analysis. Lattices implement methods such as meet, equality, ordering and operators that allow the Lattice to be moved from one lexical scope to another (e.g. from a caller function to the callee). Users then create an intra-procedural analysis by extending the IntraFWDataflow to create a forward analysis and from IntraBWDataflow to create a backward analysis. Within this class they must implement a function that returns the default abstract state of any given node at the start of the analysis. Further, they implement a transfer function that maps the applications abstract state from before a given CFG node to the state that results from the execution of the nodes expression or statement. Finally, users combine the intra-procedural analysis that they have developed with an inter-procedural analysis of their choice. This analysis will apply the intra-procedural analysis the user has implemented to the applications functions and resolve the effects of function calls on the applications abstract state, utilizing the users own state abstraction. For a concrete example, consider how the classical constant-propagation analysis is implemented using ROSE. This analysis uses a simple abstraction of application state, where the abstract state of each variable may be a value the lattice shown in 18.4 The code in below shows a class that implements this lattice. This class derives from the FiniteLattice class because the distance between the smallest and largest value in the lattice is finite. Similar functionality is provided for infinite lattices. Its state (lines 4-15) consists of its current level in the lattice as well as its value if the level is valKnown. Since the type of value is long, this abstraction can only represent integral constants. Further, the class has a special uninitialized level that means that the object has not yet been used as part of a dataflow analysis. This class implements methods meetUpdate (lines 42-65) and the equality operator (lines 68-74) to provide the basic semantics of a lattice. meetUpdate computes least upper bound of the constraints in this lattice object and another one, storing the results in this object.If both lattices have the same state, the meet is equal to this state and if they have different states, the meet is > since the variable represented by the lattice may be set to multiple values on different execution paths. The equality operator determines whether two lattice objects have the same information content. Further, the class implements utility methods that help the dataflow framework manipulate it. The initialize method (lines 24-27) ensures the object is ready to be used. Further, two copy methods (lines 30-38) make it easy to clone lattice objects. Finally, an str method (lines 82-88)

124

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

simplifies analysis debugging by printing the abstract states at CFG nodes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

c l a s s constPropLat : p u b l i c F i n i t e L a t t i c e { // The d i f f e r e n t l e v e l s o f t h i s o b j e c t s l a t t i c e t y p e d e f enum { u n i n i t i a l i z e d =0, // This o b j e c t i s u n i n i t i a l i z e d bottom =1, // No c o n s t r a i n s on t h i s o b j e c t s v a l u e a r e known valKnown=2, // The v a l u e o f t h e v a r i a b l e i s known ( one a s s i g n m e n t s e e n ) top=3 // This v a r i a b l e may have more than one v a l u e } latticeLevels ; // The l e v e l o f t h i s o b j e c t w i t h i n i t s l a t t i c e latticeLevels level ; // The v a l u e o f t h e v a r i a b l e ( i f l e v e l == valKnown ) long value ; n o d e C o n s t L a t t i c e ( ) { l e v e l=u n i n i t i a l i z e d ; } n o d e C o n s t L a t t i c e ( c o n s t n o d e C o n s t L a t t i c e& t h a t ) : v a l u e ( t h a t . v a l u e ) , l e v e l ( t h a t . l e v e l ) {} // I n i t i a l i z e s t h i s L a t t i c e t o i t s d e f a u l t s t a t e , // i f i t i s not a l r e a d y i n i t i a l i z e d void i n i t i a l i z e ( ) { i f ( l e v e l == u n i n i t i a l i z e d ) l e v e l=bottom ; } // Returns a copy o f t h i s l a t t i c e L a t t i c e ∗ copy ( ) c o n s t { r e t u r n new n o d e C o n s t L a t t i c e ( ∗ t h i s ) ; } // O v e r w r i t e s t h e s t a t e o f t h i s L a t t i c e with t h a t o f t h a t L a t t i c e v o i d copy ( L a t t i c e ∗ t h a t ) { n o d e C o n s t L a t t i c e ∗ t h a t = d y n a m i c c a s t ( t h a t a r g ) ; v a l u e = that −>v a l u e ; l e v e l = that−>l e v e l ; } // Computes t h e meet o f t h i s and t h a t and s a v e s t h e r e s u l t i n t h i s // r e t u r n s t r u e i f t h i s c a u s e s t h i s t o change and f a l s e o t h e r w i s e b o o l meetUpdate ( L a t t i c e ∗ t h a t ) { // Record t h i s o b j e c t s o r i g i n a l s t a t e t o e n a b l e change d e t e c t i o n

18.2. ROSE DATAFLOW FRAMEWORK 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

125

unsigned long oldValue = value ; l a t t i c e L e v e l s oldLevel = l e v e l ; // Cast t h a t i n t o a n o d e C o n s t L a t t i c e and a b o r t i f t h i s i s not p o s s i b l e n o d e C o n s t L a t t i c e ∗ t h a t = d y n a m i c c a s t ( t h a t a r g ) ; ROSE ASSERT( t h a t ) ; // I f t h a t i s a t a h i g h e r l a t t i c e l e v e l than t h i s , t h e v a r i a b l e must have // m u l t i p l e p o s s i b l e v a l u e on d i f f e r e n t e x e c u t i o n p a t h s i f ( that−>l e v e l > l e v e l ) l e v e l = top ; // I f both a r e a t t h e same l e v e l e l s e i f ( that−>l e v e l == l e v e l ) { // I f l a t t i c e s c o r r e s p o n d t o d i f f e r e n t v a l u e s o f t h e v a r i a b l e i f ( l e v e l == valKnown && v a l u e != that −>v a l u e ) l e v e l = top ; // The union o f both t h e s e f a c t s i s top } // Otherwise , t h i s l a t t i c e d o e s n t change // Return whether t h i s o b j e c t was m o d i f i e d r e t u r n ( oldValID != valID ) | | ( o l d L e v e l != l e v e l ) ; } // E q u a l i t y Operator b o o l o p e r a t o r==( L a t t i c e ∗ t h a t a r g ) { // Cast t h a t i n t o a n o d e C o n s t L a t t i c e and a b o r t i f t h i s i s not p o s s i b l e n o d e C o n s t L a t t i c e ∗ t h a t = d y n a m i c c a s t ( t h a t a r g ) ; ROSE ASSERT( t h a t ) ; r e t u r n l e v e l==that −>l e v e l && ( l e v e l !=valKnown | | v a l u e==that−>v a l u e ) ; } // Returns a s t r i n g r e p r e s e n t a t i o n o f t h i s o b j e c t ( t h i s f u n c t i o n i s // r e q u i r e d t o s i m p l i f y debugging ) s t r i n g s t r ( s t r i n g i n d e n t =””) { } // S e t s t h e s t a t e o f t h i s l a t t i c e t o t h e g i v e n v a l u e . Returns t r u e i f t h i s // c a u s e s t h e l a t t i c e ’ s s t a t e t o change , f a l s e o t h e r w i s e bool set ( long value ) { b o o l m o d i f i e d = t h i s −>l e v e l != valKnown | | t h i s −>v a l u e != v a l u e ; t h i s −> v a l u e = v a l u e ; l e v e l = valKnown ; return modified ; } };

126

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

The second step in implementing constant propagation is to provide a class that implements the dataflow analysis itself. This is done by extending the IntraFWDataflow class, which implements forward intra-procedural analyses and implementing the genInitState and transfer methods, described below. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

c l a s s c o n s t P r o p A n a l y s i s : p u b l i c IntraFWDataflow { c o n s t P r o p A n a l y s i s ( ) : IntraFWDataflow ( ) { } // G e n e r a t e s t h e i n i t i a l l a t t i c e s t a t e f o r t h e g i v e n d a t a f l o w node , i n t h e // g i v e n f u n c t i o n , with t h e g i v e n NodeState v o i d g e n I n i t S t a t e ( c o n s t Function& func , c o n s t DataflowNode& n , c o n s t NodeState& s t a t e , v e c t o r & i n i t L a t t i c e s , v e c t o r & i n i t F a c t s ) ; // The t r a n s f e r f u n c t i o n t h a t i s a p p l i e d t o e v e r y node i n t h e CFG // n − The d a t a f l o w node t h a t i s b e i n g p r o c e s s e d // s t a t e − The NodeState o b j e c t t h a t d e s c r i b e s t h e s t a t e o f t h e node , a s // e s t a b l i s h e d by e a r l i e r a n a l y s i s p a s s e s // d f I n f o − The L a t t i c e s t h a t t h i s t r a n s f e r f u n c t i o n o p e r a t e s on . The // f u n c t i o n t a k e s t h e s e l a t t i c e s a s i n p u t and o v e r w r i t e s them with // the r e s u l t of the t r a n s f e r . // Returns t r u e i f any o f t h e i n p u t l a t t i c e s changed a s a r e s u l t o f t h e // t r a n s f e r f u n c t i o n and f a l s e o t h e r w i s e . b o o l t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r & d f I n f o ) ; } The constPropAnalysis implementation of method genInitState creates a lattice (lines 5-7) that maintains the initial abstract state of the application at CFG node n. This lattice is an instance of the utility class FiniteVarsExprsProductLattice, which creates one copy of constPropLat for each variable that is live at node n. Since it is a product of lattices, this class is also a lattice with well-defined meet and equality operators based on the operators of its constituent lattices. The dataflow framework provides an identical class for infinite lattices as well as a generic ProductLattice class for arbitrary products of lattices. The function then adds (line 4) the lattice to vector initLattices, which is read by the dataflow framework. This function can also specify one or more facts that the framework will maintain at each node. These facts are not subject to dataflow iteration and can be used to maintain information that is useful independently of the current dataflow state.

1 2 3 4 5

v o i d c o n s t P r o p A n a l y s i s : : g e n I n i t S t a t e ( c o n s t Function& func , c o n s t DataflowNode& n , c o n s t NodeState& s t a t e , v e c t o r & i n i t L a t t i c e s , v e c t o r & i n i t F a c t s ) { i n i t L a t t i c e s . push back ( new F i n i t e V a r s E x p r s P r o d u c t L a t t i c e ( t r u e , f a l s e ,

18.2. ROSE DATAFLOW FRAMEWORK 6 7 8

127 new constPropLat ( ) , NULL, func , n , s t a t e ) ) ;

} The transfer method maps the abstract state before the CFG node n to the state that results from its execution. It begins by accessing the applications abstract state above node n from the dfInfo argument (lines 6-7) . This is the vector of lattices created by genInitState for node n. It can also be obtained from the state object, which maintains the state of the lattices both below and above each node, as well as the facts at each node. The function then initializes all the constPropLats in the product lattice (10-13) and advances to analyze the effects of the current node on the abstract state. Lines 16-127 show how the transfer function operates on different types of SgNodes. This code leverages a key feature of how ROSE represents the applications structure. Since ROSE focuses on source-to-source transformations that minimize the changes in the applications source code, all analyses must work on the original AST and cannot perform large normalization passes such as transforming the application into SSA form. Since it is difficult to implement complex analyses on top of the AST, we have developed an “on-demand” normalization that significantly simplifies the analysis development without changing the AST. Working with AST is difficult because AST sub-trees that describe the structure of expressions are complex and difficult to parse (e.g. consider analyzing all the side-effects of a=b=foo(c=d)). As such, our framework treats every SgExpression that does not correspond to an actual memory object as if it produces a temporary object that is read by its parent SgExpression. For example, in the SgExpression a=(b=c*5+d), SgIntVal 5 produces a temporary variable that is consumed by SgMultiplyOp c*5. SgVarRefExp c produces a real application variable, which is also consumed by the SgMultiplyOp. The SgMultiplyOp in turn produces a temporary variable that is consumed by SgAddOp c*5+d, which produces a temporary variable that is consumed by SgAssignOp b=c*5+d, the result of which is consumed by SgAssignOp a=(b=c*5+d). The use of these temporary variables makes it possible for user analyses to focus on just the effects of individual AST nodes without having to analyze sub-trees of the AST. Section 18.2.4 discusses how this on-demand normalization in maintained when updating to the AST. The effects of integral constants (e.g. SgIntVal or SgLongLongIntVal) are transferred on lines 16-31. On line 20 the analysis calls function SgExpr2Var to convert the SgExpression into a varID, which is an abstract representation of the memory object (either a real or temporary variable) denoted by the SgExpression. On line 21 it queries the FiniteVarsExprsProductLattice prod with this varID to get the constPropLat associated with this memory object. If this variable is live (a nonNULL lattice is returned), on lines 26-28 it sets the lattice object to be at level valKnown and sets the value to be equal to the constant represented by the SgExpression. The same logic is used for non-integral constants on lines 33-40. However, since our abstraction cannot represent such constants, their lattices are set to >. Lines 44-60 manage assignments and the lattices of the left-hand-side expression and the assignment SgAssignOp itself are set to be equal to the lattice of the right-hand-side expression. The code for variable declaration (lines 63-81) and initialization (lines 84-98) are similar in that the lattice of the right-hand-side is copied to the lattice of the initialized variable. Finally, lines 101-127 focus on arithmetic operations. If the lattices of the left- and right-hand-side expressions are both at levels valKnown,the operation is performed immediately by the analysis on their statically known values and the result is stored in the lattices of the left-hand-side expression and the SgExpression itself. Finally,

128

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

on line 129 the function returns the modified variable, which keeps track of whether the state of the downstream lattices has changed. Since these lattices are inputs to other SgExpressions, this informs the dataflow framework whether it needs to analyze how these lattices are transferred by those expressions.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

b o o l c o n s t P r o p A n a l y s i s : : t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r & d f I n f o ) { b o o l m o d i f i e d=f a l s e ; // Get t h e l a t t i c e o b j e c t F i n i t e V a r s E x p r s P r o d u c t L a t t i c e ∗ prodLat = d y n a m i c c a s t (∗( d f I n f o . b e g i n ( ) ) ) ; // Make s u r e t h a t a l l t h e non−c o n s t a n t L a t t i c e s a r e i n i t i a l i z e d c o n s t v e c t o r & l a t t i c e s = prodLat−>g e t L a t t i c e s ( ) ; f o r ( v e c t o r :: c o n s t i t e r a t o r i t = l a t t i c e s . b e g i n ( ) ; i t != l a t t i c e s . end ( ) ; i t ++) ( d y n a m i c c a s t (∗ i t ))−> i n i t i a l i z e ( ) ; // I n t e g r a l Numeric C o n s t a n t s i f ( isSgLongLongIntVal ( n . getNode ( ) ) || // Other t y p e s o f i n t e g r a l c o n s t a n t s ...) { // Memory o b j e c t and l a t t i c e o f t h e e x p r e s s i o n s r e s u l t varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; constPropLat ∗ r e s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; // I f t h e r e s u l t e x p r e s s i o n i s l i v e i f ( resLat ) { i f ( isSgLongLongIntVal ( n . getNode ( ) ) ) m o d i f i e d = r e s L a t −>s e t ( isSgLongLongIntVal ( n . getNode ())−> g e t v a l u e ( ) ) | | modified ; // Same f o r o t h e r t y p e s o f i n t e g r a l c o n s t a n t s ... } // Non−i n t e g r a l C o n s t a n t s } e l s e i f ( isSgValueExp ( n . getNode ( ) ) ) { // Memory o b j e c t and l a t t i c e o f t h e e x p r e s s i o n s r e s u l t varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; constPropLat ∗ r e s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; // I f t h e r e s u l t e x p r e s s i o n i s l i v e , s e t i t t o top s i n c e we o n l y work // with i n t e g r a l c o n s t a n t s i f ( r e s L a t ) m o d i f i e d = r e s L a t −>setTop ( ) | | m o d i f i e d ;

18.2. ROSE DATAFLOW FRAMEWORK 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86

129

// P l a i n a s s i g n m e n t : l h s = r h s } e l s e i f ( isSgAssignOp ( n . getNode ( ) ) ) { // Memory o b j e c t s denoted by t h e e x p r e s s i o n s l e f t − and r i g h t −hand // s i d e s a s w e l l a s t h e SgAssignOp i t s e l f varID l h s = SgExpr2Var ( isSgAssignOp ( n . getNode ())−> g e t l h s o p e r a n d ( ) ) ; varID r h s = SgExpr2Var ( isSgAssignOp ( n . getNode ())−> g e t r h s o p e r a n d ( ) ) ; varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; // The l a t t i c e s a s s o c i a t e d t h e t h r e e memory o b j e c t s constPropLat ∗ r e s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; constPropLat ∗ l h s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( l h s ) ) ; constPropLat ∗ r h s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r h s ) ) ; // I f t h e l h s and/ o r t h e SgAssignOp a r e l i v e , copy l a t t i c e from t h e r h s i f ( l h s L a t ) { l h s L a t −>copy ( r h s L a t ) ; m o d i f i e d = t r u e ; } i f ( r e s L a t ) { r e s L a t −>copy ( r h s L a t ) ; m o d i f i e d = t r u e ; } // V a r i a b l e D e c l a r a t i o n } e l s e i f ( i s S g I n i t i a l i z e d N a m e ( n . getNode ( ) ) ) { varID var ( i s S g I n i t i a l i z e d N a m e ( n . getNode ( ) ) ) ; constPropLat ∗ varLat = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( var ) ) ; // I f t h i s v a r i a b l e i s l i v e i f ( varLat ) { // I f t h e r e was no i n i t i a l i z e r , i n i t i a l i z e i t s l a t t i c e t o Bottom i f ( initName−> g e t i n i t i a l i z e r ()==NULL) m o d i f i e d = varLat−>s e t B o t ( ) | | m o d i f i e d ; // Otherwise , copy t h e l a t t i c e o f t h e i n i t i a l i z e r t o t h e v a r i a b l e else { varID i n i t = SgExpr2Var ( i s S g I n i t i a l i z e d N a m e ( n . getNode ())−> g e t i n i t i a l i z e r ( ) ) ; ConstPropLat ∗ i n i t L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( i n i t ) ) ; i f ( i n i t L a t ) { varLat−>copy ( i n i t L a t ) ; m o d i f i e d = t r u e ; } } } // I n i t i a l i z e r f o r a v a r i a b l e } e l s e i f ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ( ) ) ) { // Memory o b j e c t s o f t h e i n i t i a l i z e d v a r i a b l e and t h e // i n i t i a l i z a t i o n e x p r e s s i o n

130 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK varID r e s = SgExpr2Var ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ( ) ) ) ; varID asgn = SgExpr2Var ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ())−> g e t o p e r a n d ( ) ) ; // The l a t t i c e s a s s o c i a t e d both memory o b j e c t s constPropLat ∗ r e s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; constPropLat ∗ asgnLat = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( asgn ) ) ; // I f t h e v a r i a b l e i s l i v e , copy l a t t i c e from t h e a s s i g n m e n t i f ( r e s L a t ) { r e s L a t −>copy ( asgnLat ) ; m o d i f i e d = t r u e ; }

// += A r i t h m e t i c O p e r a t i o n } e l s e i f ( i s S g P l u s A s s i g n O p ( n . getNode ( ) ) ) { // Memory o b j e c t s denoted by t h e e x p r e s s i o n s l e f t − and r i g h t −hand // s i d e s a s w e l l a s t h e SgAssignOp i t s e l f varID l h s = SgExpr2Var ( isSgAssignOp ( n . getNode ())−> g e t l h s o p e r a n d ( ) ) ; varID r h s = SgExpr2Var ( isSgAssignOp ( n . getNode ())−> g e t r h s o p e r a n d ( ) ) ; varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; // The l a t t i c e s a s s o c i a t e d t h e t h r e e memory o b j e c t s constPropLat ∗ r e s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; constPropLat ∗ l h s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( l h s ) ) ; constPropLat ∗ r h s L a t = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r h s ) ) ; // I f t h e l h s and/ o r t h e SgAssignOp a r e l i v e and we know both t h e i r // v a l u e s o f t h e v a l u e o f t h e r h s e x p r e s s i o n , s e t t h e i r l a t t i c e t o be t h e // sum o f t h e two . i f ( l h s L a t && l h s L a t −>l e v e l==constPropLat : : valKnown && rhsLat−>l e v e l==constPropLat : : valKnown ) { m o d i f i e d = l h s L a t −>s e t ( l h s L a t −>v a l u e + rhsLat−>v a l u e ) | | m o d i f i e d ; } i f ( r e s L a t && r e s L a t −>l e v e l==constPropLat : : valKnown && rhsLat−>l e v e l==constPropLat : : valKnown ) { m o d i f i e d = r e s L a t −>s e t ( r e s L a t −>v a l u e + rhsLat−>v a l u e ) | | m o d i f i e d ; } } // Same f o r o t h e r a r i t h m e t i c o p e r a t i o n s ... return modified ; } Once the intra-procedural analysis has been implemented, it can be executed on the application

18.2. ROSE DATAFLOW FRAMEWORK

131

by combining it with an inter-procedural analysis. Currently two such analyses are implemented. ContextInsensitiveInterProceduralDataflow implements a basic context-insensitive analysis that propagates abstract state from callers to callees but does not differentiate between different call sites of the same function. As such, it is sensitive to inter-procedural data flows but can be imprecise because it takes into account control flows that are actually impossible, such as entering a function from one call site but returning to another. The code below provides an example of how this analysis is used to create an inter-procedural constant propagation analysis. The dataflow framework is initialized on line 6 and the applications call graph is built on lines 9-11. The intra-procedural analysis object is created on line 17 and the context-insensitive inter-procedural analysis is created on line 20. The user passes into its constructor references to their intra-procedural analysis and the call graph. Finally, on line 23 the user applies the full inter-procedural analysis to the entire application.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

i n t main ( i n t argc , c h a r ∗ argv [ ] ) { // B u i l d t h e AST used by ROSE S g P r o j e c t ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // I n i t i a l i z e t h e ROSE d a t a f l o w framework initAnalysis ( project ); // B u i l d t h e c a l l graph C a l l G r a p h B u i l d e r cgb ( p r o j e c t ) ; cgb . b u i l d C a l l G r a p h ( ) ; S g I n c i d e n c e D i r e c t e d G r a p h ∗ graph = cgb . getGraph ( ) ; // S e t t h e debug l e v e l t o p r i n t t h e p r o g r e s s o f t h e d a t a f l o w a n a l y s i s analysisDebugLevel = 1; // C r e a t e t h e i n t r a −p r o c e d u r a l c o n s t a n t p r o p a g a t i o n a n a l y s i s c o n s t P r o p A n a l y s i s cp ( p r o j e c t ) ; // C r e a t e t h e i n t e r −p r o c e d u r a l a n a l y s i s f o r i n t r a −a n a l y s i s cp C o n t e x t I n s e n s i t i v e I n t e r P r o c e d u r a l D a t a f l o w i n t e r c p (&cp , graph ) ; // Run i n t e r −p r o c e d u r a l c o n s t a n t p r o p a g a t i o n on t h e e n t i r e a p p l i c a t i o n in te r c p . runAnalysis ( ) ; }

To simplify debugging the framework also provides the UnstructuredPassInterDataflow analysis, which simply applies the users intra-procedural analysis on each function within the application. While this produces globally incorrect results, it simplifies debugging analyses on individual functions.

132

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

18.2.4

Transferring Information Between Analyses

Since in practice users need to implement multiple analyses where one depends on the results of another, the ROSE dataflow framework maintains the results of all analyses at each CFG nodes and makes it easy for analyses to access this data. The lattices and facts of a given CFG node are stored in its associated NodeState object. The data produced by an analysis can be retrieved by using its pointer, as shown in the example below. This code shows analysis exAnalysis, which takes in its constructor a pointer to the constPropAnalysis described above (lines 4-5). Inside its transfer function this analysis calls the getLatticeBelow method of its argument state (instance of the NodeState class) to get the lattice associated with constPropAnalysis that has index 0 (lines 11-13). It then gets the constPropLat of any variable it cares about and make analysis decisions based on what is statically known about its state. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

c l a s s exAnalysis { // C l a s s m a i n t a i n s a p o i n t e r t o t h e c o n s t a n t p r o p a g a t i o n a n a l y s i s t o make // i t p o s s i b l e t o a c c e s s i t s r e s u l t s c o n s t P r o p A n a l y s i s& c p A n a l y s i s ; e x A n a l y s i s ( c o n s t P r o p A n a l y s i s ∗ c p A n a l y s i s ) : c p A n a l y s i s ( c p A n a l y s i s ) {} b o o l t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r & d f I n f o ) { // Get t h e L a t t i c e s computed by t h e c o n s t a n t p r o p a g a t i o n a n a l y s i s f o r t h e // c u r r e n t CFG node F i n i t e V a r s E x p r s P r o d u c t L a t t i c e ∗ prodLat = d y n a m i c c a s t ( s t a t e −>g e t L a t t i c e B e l o w ( c p A n a l y s i s , 0 ) ) ; // Some a p p l i c a t i o n v a r i a b l e o f i n t e r e s t varID var = . . . ; // The constPropLat o f t h i s v a r i a b l e constPropLat varCPLat = d y n a m i c c a s t ( prodLat−>g e t V a r L a t t i c e ( r e s ) ) ; // Analyze d i f f e r e n t l y depending on what i s known about t h e // v a r i a b l e s v a l u e i f ( varCPLat ) i f ( varCPLat−>l e v e l == constPropLat : : bottom ) { ... } e l s e i f ( varCPLat−>l e v e l == constPropLat : : valKnown ) { ... } e l s e i f ( varCPLat−>l e v e l == constPropLat : : top ) { ... } } ...

18.2. ROSE DATAFLOW FRAMEWORK 34

133

}; The code below shows the full functionality of the NodeState class. . Lines 5-16 show the functions to set, get and delete the lattices above and below the associated CFG node. Lines 20-26 provide the same functionality for facts. The str method on line 20 returns a string representation of the lattices and facts associated with the CFG node, which is very useful for debugging. Lines 37-50 show the objects static methods. The getNodeState method on line 37 returns the NodeState object of a given CFG node. Since the ROSE virtual CFG can have multiple CFG nodes for the same AST node, this method requires an additional index argument to identify the node in question. Finally, method copyLattices aEQa and related methods (lines 39-50) copy lattice information from above a CFG node to below it and vice versa, from one node to another or from one analysis at a given node to another analysis at the same node.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

c l a s s NodeState { // S e t s t h e l a t t i c e s above / below t h i s node f o r t h e g i v e n a n a l y s i s t o t h e // g i v e n l a t t i c e v e c t o r v o i d s e t L a t t i c e A b o v e ( c o n s t A n a l y s i s ∗ a n a l y s i s , v e c t o r & l a t t i c e s ) ; v o i d s e t L a t t i c e B e l o w ( c o n s t A n a l y s i s ∗ a n a l y s i s , v e c t o r & l a t t i c e s ) ; // Returns t h e l a t t i c e l a t t i c e N a m e g e n e r a t e d by t h e g i v e n a n a l y s i s from // above / below t h e node Lattice ∗ getLatticeAbove ( const Analysis ∗ analysis , int latticeName ) const ; Lattice ∗ getLatticeBelow ( const Analysis ∗ analysis , int latticeName ) const ; // D e l e t e s a l l l a t t i c e s above / below t h i s node t h a t a r e a s s o c i a t e d with t h e // g i v e n a n a l y s i s void deleteLatticeAbove ( const Analysis ∗ a n a l y s i s ) ; void deleteLatticeBelow ( const Analysis ∗ a n a l y s i s ) ; // S e t s t h e f a c t s a t t h i s node f o r t h e g i v e n a n a l y s i s t o t h e g i v e n // f a c t v e c t o r v o i d s e t F a c t s ( c o n s t A n a l y s i s ∗ a n a l y s i s , c o n s t v e c t o r & newFacts ) ; // Returns t h e g i v e n f a c t , owned by t h e g i v e n a n a l y s i s NodeFact ∗ g e t F a c t ( c o n s t A n a l y s i s ∗ a n a l y s i s , i n t factName ) c o n s t ; // D e l e t e s a l l f a c t s a t t h i s node a s s o c i a t e d with t h e g i v e n a n a l y s i s void deleteFacts ( const Analysis ∗ a n a l y s i s ) ; // Returns a s t r i n g r e p r e s e n t a t i o n o f a l l t h e l a t t i c e s and f a c t s // a s s o c i a t e d with t h e CFG node s t r i n g s t r ( Analysis ∗ analysis , s t r i n g indent ) const ; // −−− S t a t i c Methods −−− // Returns t h e NodeState o b j e c t a s s o c i a t e d with t h e g i v e n d a t a f l o w node

134

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.5: Example of Transformation on the CFG 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

// i n d e x i s used when m u l t i p l e NodeState o b j e c t s a r e a s s o c i a t e d with a // g i v e n node // ( ex : SgFunctionCallExp has 3 NodeStates : entry , f u n c t i o n body , e x i t ) s t a t i c NodeState ∗ g e t N o d e S t a t e ( c o n s t DataflowNode& n , i n t i n d e x =0); // C o p i e s from ’ s above l a t t i c e s f o r t h e g i v e n a n a l y s i s t o to ’ s above // l a t t i c e s f o r t h e same a n a l y s i s s t a t i c v o i d c o p y L a t t i c e s a E Q a ( A n a l y s i s ∗ a n a l y s i s , NodeState& to , c o n s t NodeState& from ) ; // C o p i e s from ’ s above l a t t i c e s f o r a n a l y s i s A t o to ’ s above l a t t i c e s f o r // a n a l y s i s B s t a t i c v o i d c o p y L a t t i c e s a E Q a ( A n a l y s i s ∗ a n a l y s i s A , NodeState& to , A n a l y s i s ∗ a n a l y s i s B , c o n s t NodeState& from ) ; // S i m i l a r methods f o r c o p y i n g i n d i f f e r e n t p e r m u t a t i o n s ... };

18.2.5

CFG Transformations

ROSE makes it easy to modify the applications AST as a result of dataflow analyses. The dataflow framework maintains an on-demand normal form, where analyses can focus on the actions of individual SgNodes and ignore how they are arranged within the AST. ROSE maintains this abstraction by providing an API that inserts new SgExpressions into the applications CFG, making all the needed changes in the AST to make sure that the correct control flow is maintained. To get the intuition of this functionality consider the expression foo()=(bar()+baz()). Suppose the user has decided based on the dataflow state before the SgAddOp + that it they want to add a call to function print immediately before it. From the perspective of the CFG, this is a simple and well-defined operation, as shown at the top of Figure 4. The side-effects of the calls to bar and baz must complete before the call to print and the side-effects of print must complete before the execution of the + operation. The call to foo is not well-ordered relative print or the other

18.2. ROSE DATAFLOW FRAMEWORK

135

Figure 18.6: Example of the Transformation on the Source Code operations by the structure of the CFG. Unfortunately, it is difficult to implement these semantics in the context of the AST because (i) there is no way to add a function call to an SgAddOp and (ii) because in C++ the sequence points required by the above semantics (some side-effects much complete before others) are provided by a few specific constructs such as statement boundaries and the comma operator. As such, the transformation requires the complex set of AST changes shown in the Figure 5. We must create temporary variables to hold the results of the calls to bar and baz. We then transform the original SgAddOp into a longer SgCommaOpExp, where we first call bar and baz, saving their results into the temporary variables, then call print and finally perform the addition. The result of the addition is the result of the entire comma expression, so this transformation correctly enforces the semantics of the desired transformation. ROSE provides the three functions to make it easy to insert expressions into the CFG. Functions insertBeforeUsingCommaOp and insertAfterUsingCommaOp insert SgExpressions before or after existing SgExpressions using a generalization of the transformation described in Figure 18.6. 1 // I n s e r t an e x p r e s s i o n ( new exp ) b e f o r e a n o t h e r e x p r e s s i o n ( a n c h o r e x p ) has 2 // p o s s i b l e s i d e e f f e c t s , w i t h o u t c h a n g i n g t h e o r i g i n a l s e m a n t i c s . This i s 3 // a c h i e v e d by u s i n g a comma o p e r a t o r : ( new exp , a n c h o r e x p ) . The comma 4 // o p e r a t o r i s r e t u r n e d . 5 SgCommaOpExp ∗ insertBeforeUsingCommaOp ( S g E x p r e s s i o n ∗ new exp , 6 SgExpression ∗ anchor exp ) ; 7 // I n s e r t an e x p r e s s i o n ( new exp ) a f t e r a n o t h e r e x p r e s s i o n ( a n c h o r e x p ) has 8 // p o s s i b l e s i d e e f f e c t s , w i t h o u t c h a n g i n g t h e o r i g i n a l s e m a n t i c s . This i s 9 // done by u s i n g two comma o p e r a t o r s : 10 // type T1 ; . . . ( ( T1 = anchor exp , new exp ) , T1) ) . . . , 11 // where T1 i s a temp v a r i a b l e s a v i n g t h e p o s s i b l e s i d e e f f e c t o f a n c h o r e x p . 12 // The top l e v e l comma op exp i s r e t u r n e d . The r e f e r e n c e t o T1 i n T1 = 13 // a n c h o r e x p i s saved i n t e m p r e f . 14 SgCommaOpExp ∗ insertAfterUsingCommaOp ( S g E x p r e s s i o n ∗ new exp , 15 S g E x p r e s s i o n ∗ anchor exp , SgStatement ∗∗ t e m p d e c l = NULL, 16 SgVarRefExp ∗∗ t e m p r e f = NULL ) ; Function replaceWithPattern (Figure 18.2.5) replaces one SgExpression with another. However, since the original expression may still be valuable, it allows the original expression to be

136

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.7: Code Replacement Transformation included at one or more locations inside the new expression that contain nodes of type SgVariantExpression. 1 2 3 4

// Re p la c e an anchor node with a s p e c i f i e d p a t t e r n s u b t r e e with o p t i o n a l // S g V a r i a n t E x p r e s s i o n . A l l S g V a r i a n t E x p r e s s i o n i n t h e p a t t e r n w i l l be // r e p l a c e d with c o p i e s o f t h e anchor node . SgNode∗ r e p l a c e W i t h P a t t e r n ( SgNode ∗ anchor , SgNode∗ n e w p a t t e r n ) ; An example of this transformation is shown Figure 18.7, where the original code is the same as in the example above and the new pattern expression is a single SgMultOp where the arguments are both SgVariantExpressions. The result of the transformation is that the original SgAddOp is replaced with a multiplication the arguments of which are copies of the SgAddOp: (bar()+baz())*(bar()+baz()).

Chapter 19

Recognizing Loops Figures 19.1 and 19.2 show a translator which reads an application and gathers a list of loop nests. At the end of the traversal it reports information about each loop nest, including the function where it occurred and the depth of the loop nest. FIXME: This example program Using this translator we can compile the code shown in figure 19.3. The output is shown in is unfinished. It will output a list of objects representing figure 19.4. information about perfectly nested loops.

137

138

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

CHAPTER 19. RECOGNIZING LOOPS

// ROSE i s // r o s e . C : #i n c l u d e class {

a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an e x a m p l e p r e p r o c e s s o r b u i l t Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e

w i t h ROSE .

” r o s e . h”

InheritedAttribute public : int

loopNestDepth ;

InheritedAttribute InheritedAttribute

( ) : loopNestDepth ( 0 ) {}; ( const InheritedAttribute & X )

{};

}; class {

SynthesizedAttribute public : SynthesizedAttribute ()

{};

}; class {

Traversal

:

public

SgTopDownBottomUpProcessing

public : // F u n c t i o n s r e q u i r e d InheritedAttribute evaluateInheritedAttribute SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute );

(

SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute , SubTreeSynthesizedAttributes synthesizedAttributeList

);

}; InheritedAttribute Traversal : : evaluateInheritedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute ) { s w i t c h ( a s t N o d e−>v a r i a n t T ( ) ) { c a s e V SgForStatement : { p r i n t f ( ” Found a S g F o r S t a t e m e n t \n ” ) ; //

This l o o p i s one d e e p p e r than t h e depth i n h e r i t e d A t t r i b u t e . l o o p N e s t D e p t h ++;

of

the

parent ’ s

inherited

attribute

break ; } default : { // g++ n e e d s }

a

block

here

} return

inheritedAttribute ;

} SynthesizedAttribute Traversal : : evaluateSynthesizedAttribute ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute ,

Figure 19.1: Example source code showing loop recognition (part 1).

139

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

SubTreeSynthesizedAttributes

synthesizedAttributeList

)

{ SynthesizedAttribute

returnAttribute ;

s w i t c h ( a s t N o d e−>v a r i a n t T ( ) ) { c a s e V SgForStatement : { break ; } default : { // g++ n e e d s }

a

block

here

} return

returnAttribute ;

}

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

See

Rose : : i n i t i a l i z e

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; //

Build the i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute ;

//

C a l l t h e t r a v e r s a l s t a r t i n g a t t h e s a g e P r o j e c t node o f t h e AST myTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t , i n h e r i t e d A t t r i b u t e ) ;

Traversal

return

myTraversal ;

0;

}

Figure 19.2: Example source code showing loop recognition (part 2).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

i n t main ( ) { int x [ 4 ] ; int y [ 4 ] [ 4 ] ; f o r ( i n t i =0; i < 4 ; { x [ i ] = 7; }

i ++)

f o r ( i n t i =0; i < 4 ; i ++) { f o r ( i n t j =0; j < 4 ; j ++) { y [ i ] [ j ] = 42; } } return 0; }

Figure 19.3: Example source code used as input to loop recognition processor.

140

1 2 3

CHAPTER 19. RECOGNIZING LOOPS

Found a S gForS tate ment Found a S gForS tate ment Found a S gForS tate ment

Figure 19.4: Output of input to loop recognition processor.

Chapter 20

Virtual CFG The ROSE virtual control flow graph interface provides a higher level of detail than ROSE’s other control flow graph interfaces. It expresses control flow even within expressions, and handles shortcircuited logical and conditional operators properly1 . The interface is referred to as “virtual” because no explicit graph is ever created: only the particular CFG nodes and edges used in a given program ever exist. CFG nodes and edges are value classes (they are copied around by value, reducing the need for explicit memory management). A CFG node consists of two components: an AST node pointer, and an index of a particular CFG node within that AST node. There can be several CFG nodes corresponding to a given AST node, and thus the AST node pointers cannot be used to index CFG nodes. The particular index values for the different AST node types are explained in Section 20.1.

20.1

CFGNode Index values

To facilitate traversal and represent sufficient details, each eligible ROSE AST node (expression, statement and SgInitializedName) has several corresponding CFGNodes in virtual CFG. These CFGNodes have indices from 0 to n. CFGNode of index value of 0 is used to represent the beginning CFG node for an AST node, while the CFGNode of index n is the end CFGNode for the AST node. The beginning node represents the point in the control flow immediately before the construct starts to execute, and the ending node represents the point immediately after the construct has finished executing. Note that these two nodes do not dominate the other CFG nodes in the construct due to goto statements and labels. Reimplementation of S gNode::cfgIndexForEnd() returns the index value for n of each eligible SgNode type. See source file src/frontend/SageIII/virtualCFG/memberFunctions.C for valid index values for each type of eligible SgNode. 1 It assumes operands of expressions are computed in left-to-right order, unlike the actual language semantics, however.

141

142

CHAPTER 20. VIRTUAL CFG

20.2

Important functions

The main body of the virtual CFG interface is in virtualCFG.h; the source code is in src/frontend/SageIII/virtualCFG/ and is linked into librose . The filtered CFG interface explained below is in filteredCFG.h, and functions for converting the CFG to a graph in Dot format are in cfgToDot.h. Two functions provide the basic way of converting from AST nodes to CFG nodes. Each SgNode has two methods, cfgForBeginning() and cfgForEnd(), to generate the corresponding beginning and end CFG nodes, respectively. These functions require that the AST node is either an expression, a statement, or a SgInitializedName.

20.2.1

Node methods

• CFGNode(SgNode∗ node, unsigned int index): Build a CFG node from the given AST node and index. Valid index values are in Section 20.1. • toString(): Produce a string representing the information in the node. • toStringForDebugging(): Similar, but with more internal debugging information. • id(): A C identifier representing the node. • getNode(): Get the underlying AST node. • getIndex(): Get the index (as explained in Section 20.1) for this CFG node within its underlying AST node. • outEdges(): Return a vector of outgoing CFG edges from this node. This function internally calls cfgOutEdges(unsigned int idx) to generate out edges for each CFGNode of a given index value. • inEdges(): Return a vector of CFG edges coming into this node (note that the sources and targets of the edges are not reversed, and so each in edge has its target as the current node). This function internally calls cfgInEdges(unsigned int idx) to generate in edges for each CFGNode of a given index value. • isInteresting (): See Section 20.6.1. • Nodes are also comparable using the operators ==, !=, and <.

20.2.2

Edge methods

• toString(): Produce a string representing the information in the node. • toStringForDebugging(): Similar, but with more internal debugging information. • id(): A C identifier representing the node. • source(): The starting CFG node for this edge. • target (): The ending CFG node for this edge.

20.3. DRAWING A GRAPH OF THE CFG

143

• condition(): When there are multiple CFG edges from the same starting node, each of them is taken under certain conditions. The condition() method returns the condition, of type EdgeConditionKind. The possible return values are: – eckUnconditional: An edge that is always taken. – eckTrue: True case of a two-way branch (either an if statement or a loop – eckFalse: False case of a two-way branch – eckCaseLabel: Case label in a switch statement (key is given by caseLabel()) – eckDefault: Default label of a switch statement – eckDoConditionPassed: Enter Fortran do loop body – eckDoConditionFailed: Fortran do loop finished – eckForallIndicesInRange: Start testing forall mask – eckForallIndicesNotInRange: End of forall loop – eckComputedGotoCaseLabel: Case in computed goto – number needs to be computed separately – eckArithmeticIfLess: Edge for the arithmetic if expression being less than zero – eckArithmeticIfEqual: Edge for the arithmetic if expression being equal to zero – eckArithmeticIfGreater: Edge for the arithmetic if expression being greater than zero • caseLabel(): For an edge with condition eckCaseLabel, an expression representing the key for the case label. • computedGotoCaseIndex(): The index of this edge’s case within a Fortran computed goto (an edge of kind eckComputedGotoCaseLabel). • conditionBasedOn(): The test expression or switch expression that is tested by this edge. • scopesBeingExited(), scopesBeingEntered(): Variables leaving and entering scope during this edge. This information has not been extensively verified, and should not be relied upon. • Edges can also be compared using the operators == and !=. They are not ordered to avoid dependencies on pointer comparison on different computers.

20.3

Drawing a graph of the CFG

Fig. 20.3 shows a translator to dump full (debug) virtual control flow graphs for all functions within input source files. It also dumps a simplified version (interesting) version of virtual control flow graphs. A standalone tool named virtualCFG is installed under ROSE INSTALL TREE/bin for users to generate both debug and interesting dot files of virtual CFGs. The example input code is given in Fig. 20.3. Debug and interesting virtualCFG of function main() are shown in Fig. 20.3 and Fig. 20.4, respectively. Debug and interesting virtualCFG of function testIf () are shown in Fig. 20.5 and Fig. 20.6, respectively.

144

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

CHAPTER 20. VIRTUAL CFG

// Example t r a n s l a t o r t o g e n e r a t e d o t #i n c l u d e ” r o s e . h” #i n c l u d e u s i n g namespace s t d ; u s i n g namespace Rose ;

files

of

virtual

c o n t r o l flow graphs

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ s a g e P r o j e c t = f r o n t e n d ( argc , argv ) ; // P r o c e s s a l l f u n c t i o n d e f i n i t i o n b o d i e s f o r v i r t u a l c o n t r o l f l o w graph g e n e r a t i o n R o s e S T L C o n t a i n e r f u n c t i o n s = NodeQuery : : querySubTree ( s a g e P r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; f o r ( R o s e S T L C o n t a i n e r : : c o n s t i t e r a t o r i = f u n c t i o n s . b e g i n ( ) ; i != f u n c t i o n s . end ( ) ; ++i ) { SgFunctionDefinition ∗ proc = i s S g F u n c t i o n D e f i n i t i o n (∗ i ) ; ROSE ASSERT ( p r o c != NULL ) ; s t r i n g f i l e N a m e= S t r i n g U t i l i t y : : s t ri p Pa t hF r om F i le N am e ( proc−> g e t f i l e i n f o ()−> g e t f i l e n a m e S t r i n g ( ) ) ; s t r i n g dotFileName1=f i l e N a m e +”.”+ proc−>g e t d e c l a r a t i o n ()−> get name ( ) +”. debug . d o t ” ; s t r i n g dotFileName2=f i l e N a m e +”.”+ proc−>g e t d e c l a r a t i o n ()−> get name ( ) +”. i n t e r e s t i n g . d ot ” ; // Dump o u t t h e f u l l CFG, i n c l u d i n g b o o k k e e p i n g n o d e s VirtualCFG : : cfgToDotForDebugging ( proc , dotFileName1 ) ; // Dump o u t o n l y t h o s e n o d e s which a r e ” i n t e r e s t i n g ” f o r VirtualCFG : : i n t e r e s t i n g C f g T o D o t ( proc , dotFileName2 ) ;

analyses

} return 0; }

Figure 20.1: Example source code showing visualization of virtual control flow graph.

20.3. DRAWING A GRAPH OF THE CFG

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

145

#i n c l u d e #i n c l u d e < s t r i n g . h> #i n c l u d e < a s s e r t . h> size t i ; char b u f f e r [ 1 0 ] ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { f o r ( i =0; i < s t r l e n ( a r g v [ 1 ] ) ; { b u f f e r [ i ] = argv [ 1 ] [ i ] ; } return 0; }

i ++)

int t e s t I f ( int i ) { int rt ; i f ( i %2 ==0) r t =0; else r t =1; return rt ; }

Figure 20.2: Example source code used as input to build virtual control graphs. As we can see in Fig. 20.5, the debug dot graph has all CFGNodes for each eligible SgNode. For example, there are three CFGNodes for SgIfStmt, with indices from 0 to 2. The interesting CFGNode of SgIfStmt has solid line oval while non-essential CFGNodes have dashed line ovals in the graph. The caption of each node has a format of @linenumber:CFGNode-index-value. It is obvious from the graph that SgIfStmt’s interesting CFGNode has an index value of 1. In comparison, Fig. 20.6 only shows interesting CFGNodes with solid line ovals. Again, the captions tells line numbers and CFGNode’s index values for each CFGNode.

146

CHAPTER 20. VIRTUAL CFG

Start(::main) 0x7f99f64b9010 @line=8, col=1 :idx=0

main_parameter_list_ 0x7f99f7ba4d78 @line=7, col=1 :idx=0

initialized_name_argc 0x7f99f661f500 argc :idx=0

main_parameter_list_ 0x7f99f7ba4d78 @line=7, col=1 :idx=1

initialized_name_argv 0x7f99f661f648 argv :idx=0

main_parameter_list_ 0x7f99f7ba4d78 @line=7, col=1 :idx=2

After parameters(::main) 0x7f99f64b9010 @line=8, col=1 :idx=1

After pre-initialization(::main) 0x7f99f64b9010 @line=8, col=1 :idx=2

0x7f99f654e010 0x7f99f654e010 @line=8, col=1 :idx=0

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=0

SgForInitStatement 0x7f99f63f1080 @line=9, col=8 :idx=0

SgExprStatement 0x7f99f6358010 @line=9, col=8 :idx=0

SgAssignOp_undef_name 0x7f99f6387010 @line=9, col=8 :idx=0

var_ref_of_i 0x7f99f63be010 @line=9, col=8 :idx=0

SgAssignOp_undef_name 0x7f99f6387010 @line=9, col=8 :idx=1

SgCastExp_undef_name 0x7f99f679b100 @line=0, col=0 :idx=0

integer_value_exp_0 0x7f99f6ccc350 @line=9, col=10 :idx=0

integer_value_exp_0 0x7f99f6ccc350 @line=9, col=10 :idx=1

SgCastExp_undef_name 0x7f99f679b100 @line=0, col=0 :idx=1

SgAssignOp_undef_name 0x7f99f6387010 @line=9, col=8 :idx=2

SgExprStatement 0x7f99f6358010 @line=9, col=8 :idx=1

SgForInitStatement 0x7f99f63f1080 @line=9, col=8 :idx=1

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=1

SgExprStatement 0x7f99f6358070 @line=9, col=13 :idx=0

SgLessThanOp_undef_name 0x7f99f6249010 @line=9, col=13 :idx=0

var_ref_of_i 0x7f99f63be078 @line=9, col=13 :idx=0

SgLessThanOp_undef_name 0x7f99f6249010 @line=9, col=13 :idx=1

function_call_function_ref_strlen 0x7f99f6280010 @line=9, col=17 :idx=0

function_ref_strlen 0x7f99f6321010 @line=9, col=17 :idx=0

function_call_function_ref_strlen 0x7f99f6280010 @line=9, col=17 :idx=1

expr_list_exp_SgCastExp_undef_name 0x7f99f62b7010 @line=0, col=0 :idx=0

SgCastExp_undef_name 0x7f99f679b178 @line=0, col=0 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea010 @line=9, col=24 :idx=0

var_ref_of_argv 0x7f99f63be0e0 @line=9, col=24 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea010 @line=9, col=24 :idx=1

integer_value_exp_1 0x7f99f6ccc3b8 @line=9, col=29 :idx=0

integer_value_exp_1 0x7f99f6ccc3b8 @line=9, col=29 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea010 @line=9, col=24 :idx=2

SgCastExp_undef_name 0x7f99f679b178 @line=0, col=0 :idx=1

expr_list_exp_SgCastExp_undef_name 0x7f99f62b7010 @line=0, col=0 :idx=1

function_call_function_ref_strlen 0x7f99f6280010 @line=9, col=17 :idx=2

function_call_function_ref_strlen 0x7f99f6280010 @line=9, col=17 :idx=3

SgLessThanOp_undef_name 0x7f99f6249010 @line=9, col=13 :idx=2

SgExprStatement 0x7f99f6358070 @line=9, col=13 :idx=1

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=2 key(true)

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=4

SgExprStatement 0x7f99f63580d0 @line=11, col=5 :idx=0

0x7f99f654e010 0x7f99f654e010 @line=8, col=1 :idx=1

SgAssignOp_undef_name 0x7f99f6387080 @line=11, col=5 :idx=0

SgReturnStmt 0x7f99f61e7010 @line=13, col=3 :idx=0

array_ref_of_var_ref_of_buffer_at_var_ref_of_i 0x7f99f62ea080 @line=11, col=5 :idx=0

integer_value_exp_0 0x7f99f6ccc488 @line=13, col=10 :idx=0

var_ref_of_buffer 0x7f99f63be1b0 @line=11, col=5 :idx=0

array_ref_of_var_ref_of_buffer_at_var_ref_of_i 0x7f99f62ea080 @line=11, col=5 :idx=1

key(false)

0x7f99f654e128 0x7f99f654e128 @line=10, col=3 :idx=0

integer_value_exp_0 0x7f99f6ccc488 @line=13, col=10 :idx=1

0x7f99f654e010 0x7f99f654e010 @line=8, col=1 :idx=2

var_ref_of_i 0x7f99f63be218 @line=11, col=12 :idx=0

SgReturnStmt 0x7f99f61e7010 @line=13, col=3 :idx=1

End(::main) 0x7f99f64b9010 @line=8, col=1 :idx=3

array_ref_of_var_ref_of_buffer_at_var_ref_of_i 0x7f99f62ea080 @line=11, col=5 :idx=2

SgAssignOp_undef_name 0x7f99f6387080 @line=11, col=5 :idx=1

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i 0x7f99f62ea160 @line=11, col=17 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea0f0 @line=11, col=17 :idx=0

var_ref_of_argv 0x7f99f63be280 @line=11, col=17 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea0f0 @line=11, col=17 :idx=1

integer_value_exp_1 0x7f99f6ccc420 @line=11, col=22 :idx=0

integer_value_exp_1 0x7f99f6ccc420 @line=11, col=22 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea0f0 @line=11, col=17 :idx=2

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i 0x7f99f62ea160 @line=11, col=17 :idx=1

var_ref_of_i 0x7f99f63be2e8 @line=11, col=25 :idx=0

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i 0x7f99f62ea160 @line=11, col=17 :idx=2

SgAssignOp_undef_name 0x7f99f6387080 @line=11, col=5 :idx=2

SgExprStatement 0x7f99f63580d0 @line=11, col=5 :idx=1

0x7f99f654e128 0x7f99f654e128 @line=10, col=3 :idx=1

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=3

SgPlusPlusOp_undef_name 0x7f99f6216010 @line=9, col=34 :idx=0

var_ref_of_i 0x7f99f63be148 @line=9, col=34 :idx=0

SgPlusPlusOp_undef_name 0x7f99f6216010 @line=9, col=34 :idx=1

Figure 20.3: The debug virtual control flow graph for function main() shows all virtual CFG nodes and edges

20.3. DRAWING A GRAPH OF THE CFG

147

Start(::main) 0x7f99f64b9010 @line=8, col=1 :idx=0

initialized_name_argc 0x7f99f661f500 argc :idx=0

initialized_name_argv 0x7f99f661f648 argv :idx=0

main_parameter_list_ 0x7f99f7ba4d78 @line=7, col=1 :idx=2

var_ref_of_i 0x7f99f63be010 @line=9, col=8 :idx=0

integer_value_exp_0 0x7f99f6ccc350 @line=9, col=10 :idx=1

SgCastExp_undef_name 0x7f99f679b100 @line=0, col=0 :idx=1

SgAssignOp_undef_name 0x7f99f6387010 @line=9, col=8 :idx=2

SgExprStatement 0x7f99f6358010 @line=9, col=8 :idx=1

SgForInitStatement 0x7f99f63f1080 @line=9, col=8 :idx=1

var_ref_of_i 0x7f99f63be078 @line=9, col=13 :idx=0

function_ref_strlen 0x7f99f6321010 @line=9, col=17 :idx=0

var_ref_of_argv 0x7f99f63be0e0 @line=9, col=24 :idx=0

integer_value_exp_1 0x7f99f6ccc3b8 @line=9, col=29 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea010 @line=9, col=24 :idx=2

SgCastExp_undef_name 0x7f99f679b178 @line=0, col=0 :idx=1

expr_list_exp_SgCastExp_undef_name 0x7f99f62b7010 @line=0, col=0 :idx=1

function_call_function_ref_strlen 0x7f99f6280010 @line=9, col=17 :idx=3

SgLessThanOp_undef_name 0x7f99f6249010 @line=9, col=13 :idx=2

SgExprStatement 0x7f99f6358070 @line=9, col=13 :idx=1

0x7f99f6428010 0x7f99f6428010 @line=9, col=3 :idx=2 key(true) var_ref_of_buffer 0x7f99f63be1b0 @line=11, col=5 :idx=0

var_ref_of_i 0x7f99f63be218 @line=11, col=12 :idx=0

array_ref_of_var_ref_of_buffer_at_var_ref_of_i 0x7f99f62ea080 @line=11, col=5 :idx=2

key(false) integer_value_exp_0 0x7f99f6ccc488 @line=13, col=10 :idx=1

SgReturnStmt 0x7f99f61e7010 @line=13, col=3 :idx=1

End(::main) 0x7f99f64b9010 @line=8, col=1 :idx=3

var_ref_of_argv 0x7f99f63be280 @line=11, col=17 :idx=0

integer_value_exp_1 0x7f99f6ccc420 @line=11, col=22 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 0x7f99f62ea0f0 @line=11, col=17 :idx=2

var_ref_of_i 0x7f99f63be2e8 @line=11, col=25 :idx=0

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i 0x7f99f62ea160 @line=11, col=17 :idx=2

SgAssignOp_undef_name 0x7f99f6387080 @line=11, col=5 :idx=2

SgExprStatement 0x7f99f63580d0 @line=11, col=5 :idx=1

var_ref_of_i 0x7f99f63be148 @line=9, col=34 :idx=0

SgPlusPlusOp_undef_name 0x7f99f6216010 @line=9, col=34 :idx=1

Figure 20.4: The virtual control flow graph for function main() shows only interesting virtual CFG nodes and edges. Each CFGNode’s caption tells associated source line number and CFGNode index value (@line-num:index-value)

148

CHAPTER 20. VIRTUAL CFG

Start(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=0

testIf_parameter_list_ 0x7f99f7ba5228 @line=16, col=1 :idx=0

initialized_name_i 0x7f99f661f8d8 i :idx=0

testIf_parameter_list_ 0x7f99f7ba5228 @line=16, col=1 :idx=1

After parameters(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=1

After pre-initialization(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=2

0x7f99f654e240 0x7f99f654e240 @line=17, col=1 :idx=0

_variable_declaration_rt 0x7f99f6b460e0 @line=18, col=3 :idx=0

initialized_name_rt 0x7f99f661fa20 rt :idx=0

_variable_declaration_rt 0x7f99f6b460e0 @line=18, col=3 :idx=1

0x7f99f654e240 0x7f99f654e240 @line=17, col=1 :idx=1

0x7f99f6152010 0x7f99f6152010 @line=19, col=3 :idx=0

SgExprStatement 0x7f99f6358130 @line=19, col=7 :idx=0

SgEqualityOp_undef_name 0x7f99f60e4010 @line=19, col=7 :idx=0

SgModOp_undef_name 0x7f99f611b010 @line=19, col=7 :idx=0

var_ref_of_i 0x7f99f63be350 @line=19, col=7 :idx=0

SgModOp_undef_name 0x7f99f611b010 @line=19, col=7 :idx=1

integer_value_exp_2 0x7f99f6ccc4f0 @line=19, col=9 :idx=0

integer_value_exp_2 0x7f99f6ccc4f0 @line=19, col=9 :idx=1

SgModOp_undef_name 0x7f99f611b010 @line=19, col=7 :idx=2

SgEqualityOp_undef_name 0x7f99f60e4010 @line=19, col=7 :idx=1

integer_value_exp_0 0x7f99f6ccc558 @line=19, col=13 :idx=0

integer_value_exp_0 0x7f99f6ccc558 @line=19, col=13 :idx=1

SgEqualityOp_undef_name 0x7f99f60e4010 @line=19, col=7 :idx=2

SgExprStatement 0x7f99f6358130 @line=19, col=7 :idx=1

0x7f99f6152010 0x7f99f6152010 @line=19, col=3 :idx=1 key(true) SgExprStatement 0x7f99f6358190 @line=20, col=5 :idx=0

key(false) SgExprStatement 0x7f99f63581f0 @line=22, col=5 :idx=0

SgAssignOp_undef_name 0x7f99f63870f0 @line=20, col=5 :idx=0

SgAssignOp_undef_name 0x7f99f6387160 @line=22, col=5 :idx=0

var_ref_of_rt 0x7f99f63be3b8 @line=20, col=5 :idx=0

var_ref_of_rt 0x7f99f63be420 @line=22, col=5 :idx=0

SgAssignOp_undef_name 0x7f99f63870f0 @line=20, col=5 :idx=1

SgAssignOp_undef_name 0x7f99f6387160 @line=22, col=5 :idx=1

integer_value_exp_0 0x7f99f6ccc5c0 @line=20, col=9 :idx=0

integer_value_exp_1 0x7f99f6ccc628 @line=22, col=9 :idx=0

integer_value_exp_0 0x7f99f6ccc5c0 @line=20, col=9 :idx=1

integer_value_exp_1 0x7f99f6ccc628 @line=22, col=9 :idx=1

SgAssignOp_undef_name 0x7f99f63870f0 @line=20, col=5 :idx=2

SgAssignOp_undef_name 0x7f99f6387160 @line=22, col=5 :idx=2

SgExprStatement 0x7f99f6358190 @line=20, col=5 :idx=1

SgExprStatement 0x7f99f63581f0 @line=22, col=5 :idx=1

0x7f99f6152010 0x7f99f6152010 @line=19, col=3 :idx=2

0x7f99f654e240 0x7f99f654e240 @line=17, col=1 :idx=2

SgReturnStmt 0x7f99f61e7070 @line=24, col=3 :idx=0

var_ref_of_rt 0x7f99f63be488 @line=24, col=10 :idx=0

SgReturnStmt 0x7f99f61e7070 @line=24, col=3 :idx=1

0x7f99f654e240 0x7f99f654e240 @line=17, col=1 :idx=3

End(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=3

Figure 20.5: The debug virtual control flow graph for function testIf() shows all virtual CFG nodes and edges

20.3. DRAWING A GRAPH OF THE CFG

149

Start(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=0

initialized_name_i 0x7f99f661f8d8 i :idx=0

testIf_parameter_list_ 0x7f99f7ba5228 @line=16, col=1 :idx=1

initialized_name_rt 0x7f99f661fa20 rt :idx=0

_variable_declaration_rt 0x7f99f6b460e0 @line=18, col=3 :idx=1

var_ref_of_i 0x7f99f63be350 @line=19, col=7 :idx=0

integer_value_exp_2 0x7f99f6ccc4f0 @line=19, col=9 :idx=1

SgModOp_undef_name 0x7f99f611b010 @line=19, col=7 :idx=2

integer_value_exp_0 0x7f99f6ccc558 @line=19, col=13 :idx=1

SgEqualityOp_undef_name 0x7f99f60e4010 @line=19, col=7 :idx=2

SgExprStatement 0x7f99f6358130 @line=19, col=7 :idx=1

0x7f99f6152010 0x7f99f6152010 @line=19, col=3 :idx=1 key(true) var_ref_of_rt 0x7f99f63be3b8 @line=20, col=5 :idx=0

key(false) var_ref_of_rt 0x7f99f63be420 @line=22, col=5 :idx=0

integer_value_exp_0 0x7f99f6ccc5c0 @line=20, col=9 :idx=1

integer_value_exp_1 0x7f99f6ccc628 @line=22, col=9 :idx=1

SgAssignOp_undef_name 0x7f99f63870f0 @line=20, col=5 :idx=2

SgAssignOp_undef_name 0x7f99f6387160 @line=22, col=5 :idx=2

SgExprStatement 0x7f99f6358190 @line=20, col=5 :idx=1

SgExprStatement 0x7f99f63581f0 @line=22, col=5 :idx=1

var_ref_of_rt 0x7f99f63be488 @line=24, col=10 :idx=0

SgReturnStmt 0x7f99f61e7070 @line=24, col=3 :idx=1

End(::testIf) 0x7f99f64b9140 @line=17, col=1 :idx=3

Figure 20.6: The virtual control flow graph for function testIf() shows only interesting virtual CFG nodes and edges. Each CFGNode’s caption tells associated source line number and CFGNode index value (@line-num:index-value)

150

20.4

CHAPTER 20. VIRTUAL CFG

Robustness to AST changes

Control flow graph nodes and edges can be kept (i.e., are not invalidated) in many cases when the underlying AST changes. However, there are some limitations to this capability. Changing the AST node that is pointed to by a given CFG node is not safe. CFG nodes for deleted AST nodes are of course invalid, as are those pointing to AST nodes whose parent pointers become invalid.

20.5

Limitations

Although workable for intraprocedural analysis of C code, the virtual CFG code has several limitations for other languages and uses.

20.5.1

Fortran support

The virtual control flow graph includes support for many Fortran constructs, but that support is fairly limited and not well tested. It is not recommended for production use.

20.5.2

Exception handling

The virtual CFG interface does not support control flow due to exceptions or the setjmp/longjmp constructs. It does, however, support break, continue, goto, and early returns from functions.

20.5.3

Interprocedural control flow analysis

In virtual CFGs, interprocedural control flow analysis is disabled by default. It can be enabled by setting the parameter virtualInteproceduralControlFlowGraphs in SgNode::cfgInEdges, SgNode::cfgOutEdges, and their subclasses’ definitions. Interprocedural edges are labeled with the eckInterprocedural EdgeConditionKind. In cases where the flow of control cannot be determined statically (calls of virtual functions, function pointers, or functors), the interprocedural control flow graph contains all possible options. In keeping with the ‘virtual’ nature of ROSE’s control flow graphs, the set of options is computed on-the-fly; therefore, changes to the AST will be reflected in subsequent interaction with the control flow graph.

20.6

Node filtering

FIXME

20.6.1

“Interesting” node filter

To simplify the virtual CFG, non-essential CFGNodes, such as the beginning and the end CFGNodes for each AST node, can be filtered out. Each eligible SgNode type has a most important CFGNode out of its all CFGNodes. The interesting CFGNode’s index value for each Node type is

20.7. STATIC CFG

151

returned by calling the derived implementation of virtual bool SgNode::cfgIsIndexInteresting(int idx).

20.6.2

Arbitrary filtering

20.7

Static CFG

Since a virtual CFG does not produce any real graph, it is quite inefficient to traverse a virtual CFG frequently. It is necessary to build a static CFG which may improve the performance of some specific operations. A SgGraph object (actually, it’s a SgIncidenceDirectedGraph object) is created to store the static CFG. Each node in the graph is a SgGraphNode object. In a virtual CFG, each node contains two members: node and index. A SgGraphNode already holds a pointer to SgNode, and we have to add the other property “index” to our SgGraphNode. This can be done by adding the corresponding attribute to SgGraphNode.

20.7.1

Class methods

• CFG(): The default constructor. • CFG(SgNode∗ node, bool isFiltered = false): Initialize a static CFG with the start node to build from and a flag indicating if the CFG is a full or filtered one. • setStart (SgNode∗ node): Set the start node for building a static CFG. Note that the argument has to be an object of any of the following classes: SgProject, SgStatement, SgExpression, and SgInitializedName. If a SgProject object is passed in, several graphs are built for every function definition. • isFilteredCFG(): Return the isFiltered flag. • setFiltered (bool flag ): Set the isFiltered flag. • buildCFG(): Build a full or filtered CFG according to the isFiltered flag. • buildFullCFG(): Build a full CFG for debugging. • buildFilteredCFG(): Build a filtered CFG which only contains interesting nodes. • getOutEdges(SgGraphNode∗ node): Return a vector of outgoing CFG edges (SgDirectedGraphEdge objects) from the given node. • getInEdges(SgGraphNode∗ node): Return a vector of CFG edges coming into the given node. • cfgForBeginning(SgNode∗ node): Return the CFG node for just before this AST node. • cfgForEnd(SgNode∗ node): Return the CFG node for just after this AST node. • getIndex(SgGraphNode∗ node): Return the index of the given CFG node.

152

CHAPTER 20. VIRTUAL CFG • cfgToDot(SgNode∗ node, const std::string& filename): Generate a DOT file for the current CFG. Note that the start node to be drawn can be indicated which is not necessary to be the start node of the CFG.

20.7.2

Drawing a graph of the CFG

Figure 20.7.2 shows a translator to dump full (debug) static control flow graphs for all functions within input source files. It also dumps a simplified version (interesting) version of static control flow graphs. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

// Example t r a n s l a t o r t o g e n e r a t e d o t #i n c l u d e ” r o s e . h” #i n c l u d e u s i n g namespace s t d ; u s i n g namespace Rose ;

files

of

static

c o n t r o l flow graphs

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ s a g e P r o j e c t = f r o n t e n d ( argc , argv ) ; // P r o c e s s a l l f u n c t i o n d e f i n i t i o n b o d i e s f o r s t a t i c c o n t r o l f l o w graph g e n e r a t i o n R o s e S T L C o n t a i n e r f u n c t i o n s = NodeQuery : : querySubTree ( s a g e P r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; f o r ( R o s e S T L C o n t a i n e r : : c o n s t i t e r a t o r i = f u n c t i o n s . b e g i n ( ) ; i != f u n c t i o n s . end ( ) ; ++i ) { SgFunctionDefinition ∗ proc = i s S g F u n c t i o n D e f i n i t i o n (∗ i ) ; ROSE ASSERT ( p r o c != NULL ) ; s t r i n g f i l e N a m e= S t r i n g U t i l i t y : : s t ri p Pa t hF r om F i le N am e ( proc−> g e t f i l e i n f o ()−> g e t f i l e n a m e S t r i n g ( ) ) ; s t r i n g dotFileName1=f i l e N a m e +”.”+ proc−>g e t d e c l a r a t i o n ()−> get name ( ) +”. debug . d o t ” ; s t r i n g dotFileName2=f i l e N a m e +”.”+ proc−>g e t d e c l a r a t i o n ()−> get name ( ) +”. i n t e r e s t i n g . d ot ” ; StaticCFG : : CFG c f g ( p r o c ) ; // Dump o u t t h e f u l l CFG, i n c l u d i n g b o o k k e e p i n g n o d e s c f g . buildFullCFG ( ) ; c f g . cfgToDot ( proc , dotFileName1 ) ; // Dump o u t o n l y t h o s e n o d e s which a r e ” i n t e r e s t i n g ” f o r cf g . buildFilteredCFG ( ) ; c f g . cfgToDot ( proc , dotFileName2 ) ;

analyses

} return 0; }

Figure 20.7: Example source code showing visualization of static control flow graph. The example input code is given in Fig. 20.3. Debug and interesting static CFG are shown in Fig. 20.5 and Fig. 20.6, respectively.

20.8

Static, Interprocedural CFGs

ROSE supports construction of interprocedural control flow graphs using the InterproceduralCFG class, a subclass of StaticCFG. Like the StaticCFG, the InterproceduralCFG can be con-

20.8. STATIC, INTERPROCEDURAL CFGS

153

structed from any SgNode that affects control flow. If an InterproceduralCFG is constructed from a given node, it will contain all possible paths of execution from that point. Edges between procedures will be labelled with the ‘eckInterprocedural’ EdgeConditionKind. In cases where a function call cannot be statically resolved to a function definition, the InterproceduralCFG includes edges from the call node to all possible definitions, which are determined by ROSE’s CallGraph.

154

CHAPTER 20. VIRTUAL CFG

Chapter 21

Generating Control Flow Graphs The control flow of a program is broken into basic blocks as nodes with control flow forming edges between the basic blocks. Thus the control flow forms a graph which often labeled edges (true and false), and basic blocks representing sequentially executed code. This chapter presents the Control Flow Graph (CFG) and the ROSE application code for generating such graphs for any function in an input code. The CFG forms a fundamental building block for more complex forms of program analysis. Figure 21.1 shows the code required to generate the control flow graph for each function of an application. Using the input code shown in figure 21.2 the first function’s control flow graph is shown in figure 21.3. Figure 21.3 shows the control flow graph for the function in the input code in figure 21.2.

155

156

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

CHAPTER 21. GENERATING CONTROL FLOW GRAPHS

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e

” r o s e . h” ”CFGImpl . h” ” GraphDotOutput . h” ” pr eC on tr olF lo wG ra ph . h” ”CommandOptions . h”

u s i n g namespace s t d ; // Use t h e ControlFlowGraph i s d e f i n e d i n both PRE // and t h e D o m i n a t o r T r e e s A n d D o m i n a n c e F r o n t i e r s namespaces . // We want t o u s e t h e one i n t h e PRE namespace . u s i n g namespace PRE ; class {

visitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { SgFunctionDeclaration ∗ functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { S g F u n c t i o n D e f i n i t i o n ∗ f u n c t i o n D e f i n i t i o n = f u n c t i o n D e c l a r a t i o n −>g e t d e f i n i t i o n ( ) ; i f ( f u n c t i o n D e f i n i t i o n != NULL) { S g B a s i c B l o c k ∗ f u n c t i o n B o d y = f u n c t i o n D e f i n i t i o n −>g e t b o d y ( ) ; ROSE ASSERT( f u n c t i o n B o d y != NULL ) ; ControlFlowGraph c o n t r o l f l o w ; // The CFG can o n l y be c a l l e d on a f u n c t i o n d e f i n i t i o n ( a t p r e s e n t ) makeCfg ( f u n c t i o n D e f i n i t i o n , c o n t r o l f l o w ) ; s t r i n g f i l e N a m e = f u n c t i o n D e c l a r a t i o n −>get name ( ) . s t r ( ) ; f i l e N a m e += ” . d o t ” ; ofstream d o t f i l e ( fileName . c s t r ( ) ) ; printCfgAsDot ( d o t f i l e , c o n t r o l f l o w ) ; } } } i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; CmdOptions : : G e t I n s t a n c e ()−> S e t O p t i o n s ( a r g c , a r g v ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 21.1: Example source code showing visualization of control flow graph.

157

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

#i n c l u d e #i n c l u d e < s t r i n g . h> #i n c l u d e < a s s e r t . h> size t i ; char b u f f e r [ 1 0 ] ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { f o r ( i =0; i < s t r l e n ( a r g v [ 1 ] ) ; { b u f f e r [ i ] = argv [ 1 ] [ i ] ; } return 0; }

i ++)

int t e s t I f ( int i ) { int rt ; i f ( i %2 ==0) r t =0; else r t =1; return rt ; }

Figure 21.2: Example source code used as input to build control flow graph.

1 i = 0; in SgForInitStatement

4 i < strlen(argv[1]) in SgForStatement

2 return 0;

0

3 buffer[i] = argv[1][i];

5 i++ in SgBasicBlock

Figure 21.3: Control flow graph for function in input code file: inputCode 1.C.

158

CHAPTER 21. GENERATING CONTROL FLOW GRAPHS

Chapter 22

Graph Processing Tutorial 22.1

Traversal Tutorial

ROSE can collect and analyze paths in both source and binary CFGs. At moment it doesn’t attempt to save paths because if you save them directly the space necessary is extremely large, as paths grow 2n with successive if statements and even faster when for loops are involved. Currently a path can only cannot complete the same loop twice. However it is possible for a graph such that 1 -¿ 2 , 2-¿3, 3-¿1, 3-¿5, has paths, 1,2,3,1,2,3,5 and 1,2,3,5 because the loop 1,2,3,1 is not repeated. The tutorial example works as such:

159

160

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

CHAPTER 22. GRAPH PROCESSING TUTORIAL

#include #include //#i n c l u d e #include #include #include ” SgGraphTemplate . h” #include ” g r a p h P r o c e s s i n g . h”

#include ” staticCFG . h” #include ” i n t e r p r o c e d u r a l C F G . h” /∗ T e s t i n g t h e graph t r a v e r s a l mechanism now implementing i n A s t P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g / #include #include using namespace s t d ; using namespace b o o s t ;

/∗ You need t o use myGraph t y p e h e r e b e c a u s e t h e c o n v e r s i o n o f StaticCFG : : InterproceduralCFG or StaticCFG : :CFG i n a b o o s t form . The SgGraphTemplate . h f i l e h a n d l e s t h i s c o n v e r s i o n and myGraph i s s p e c i f i c t o t h a t f i l e ∗/ typedef myGraph CFGforT ;

/∗ Your b a s i c v i s i t o r t r a v e r s a l s u b c l a s s e d from SgGraphTraversal on t h e CFGforT t e m p l a t e as d e f i n e d above ∗/ c l a s s v i s i t o r T r a v e r s a l : public S g G r a p h T r a v e r s a l { public : int paths ; /∗ This i s t h e f u n c t i o n run by t h e a l g o r i t h m on e v e r y path , VertexID i s a t y p e implemented i n SgGraphTe void a n a l y z e P a t h ( v e c t o r & pth ) ; }; /∗ d e f i n i n g t h e a n a l y z e P a t h f u n c t i o n . This s i m p l y c o u n t s p a t h s as s h o u l d be o b v i o u s . Again , VertexID i s d e f i n e void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r & pth ) { p a t h s ++; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; /∗ F i r s t you need t o produce t h e p r o j e c t f i l e ∗/ SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j != NULL ) ; /∗ G e t t i n g t h e Function D e c l a r a t i o n and D e f i n i t i o n f o r p r o d u c i n g t h e graph ∗/ S g F u n c t i o n D e c l a r a t i o n ∗ mainDefDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j ) ; S g F u n c t i o n D e f i n i t i o n ∗ mainDef = mainDefDecl−>g e t d e f i n i t i o n ( ) ; /∗ I n s t a n t i a t i n g t h e v i s i t o r T r a v e r s a l ∗/ v i s i t o r T r a v e r s a l ∗ v i s = new v i s i t o r T r a v e r s a l ( ) ; /∗ This c r e a t e s t h e StaticCFG : : InterproceduralCFG o b j e c t t o be c o n v e r t e d t o a b o o s t graph ∗/ StaticCFG : : I n t e r p r o c e d u r a l C F G c f g ( mainDef ) ; stringstream ss ; S g I n c i d e n c e D i r e c t e d G r a p h ∗ g = new S g I n c i d e n c e D i r e c t e d G r a p h ( ) ; /∗ We g o t t h e n e c e s s a r y i n t e r n a l S g I n c i d e n c e D i r e c t e d G r a p h from t h e c f g ∗/ g = c f g . getGraph ( ) ; myGraph∗ mg = new myGraph ( ) ; /∗ C o n v e r t i n g t h e c f g t o a b o o s t graph ∗/ mg = i n s t a n t i a t e G r a p h ( g , c f g , mainDef ) ; /∗ S e t i n t e r n a l v a r i a b l e s ∗/ v i s −>p a t h s = 0 ; /∗ i n v o k i n g t h e t r a v e r s a l , t h e f i r s t argument i s t h e graph , t h e second i s t r u e i f you do not want bounds , f a l s e i f you do , t h e t h i r d and f o u r t h arguments a r e s t a r t i n g and s t o p p i n g v e r t i c e s r e s p e c t i v e l y , i f you ar e not bounding s i m p l y i n s e r t 0 . F i n a l l y t h e l a s t argument i s c u r r e n t l y d e p r e c a t e d ∗/ v i s −>c o n s t r u c t P a t h A n a l y z e r (mg , true , 0 , 0 , true ) ; s t d : : c o u t << ” f i n i s h e d ” << s t d : : e n d l ; s t d : : c o u t << ” p a t h s : ” << v i s −>p a t h s << s t d : : e n d l ; delete v i s ; }

Figure 22.1: Source CFG Traversal Example

22.1. TRAVERSAL TUTORIAL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

161

#include #include #include //#i n c l u d e ” interproceduralCFG . h” #include #include /∗ These ar e n e c e s s a r y f o r any b i n a r y T r a v e r s a l ∗/ #include ” g r a p h P r o c e s s i n g . h” #include ” B i n a r y C o n t r o l F l o w . h” #include ” B i n a r y L o a d e r . h” /∗ T e s t i n g t h e graph t r a v e r s a l mechanism now implementing i n g r a p h P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g /) ∗/ using namespace s t d ; using namespace b o o s t ; /∗ These s h o u l d j u s t be c o p i e d v e r b a t i m ∗/ typedef b o o s t : : g r a p h t r a i t s : : v e r t e x d e s c r i p t o r V e r t e x ; /∗∗< Graph v e r t e x t y p e . ∗/ typedef b o o s t : : g r a p h t r a i t s : : e d g e d e s c r i p t o r Edge ; /∗∗< Graph edg e t y p e . ∗/

/∗ We f i r s t make a v i s i t o r T r a v e r s a l , s u b c l a s s e d from SgGraphTraversal t e m p l a t e d on t h e B i n a r y A n a l y s i s : ControlFlow : : Graph which i s implemented as a b o o s t graph ∗/

class v i s it o r T r av e r s a l {

: public S g G r a p h T r a v e r s a l

public : long i n t p t h s ; long i n t t l t n o d e s ; /∗ This needs t o be i n any v i s i t o r T r a v e r s a l , i t i s t h e f u n c t i o n t h a t w i l l be run on e v e r y p a t h by t h e graph p a t h a n a l y s i s a l g o r i t h m , n o t i c e t h e V e r t e x t y p e i s from t h e above t y p e d e f s ∗/ v i r t u a l void a n a l y z e P a t h ( v e c t o r & pth ) ; }; /∗ This i s a v e r y s i m p l e i n c a r n a t i o n , i t j u s t c o u n t s p a t h s ∗/ void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r & pth ) { p t h s ++; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; /∗ Parse t h e b i n a r y f i l e ∗/ SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; s t d : : v e c t o r i n t e r p s = S a g e I n t e r f a c e : : querySubTree( p r o j e c t ) ; i f ( i n t e r p s . empty ( ) ) { f p r i n t f ( s t d e r r , ” no b i n a r y i n t e r p r e t a t i o n s f o u n d \n” ) ; exit (1); } /∗ C a l c u l a t e p l a i n o l d CFG. ∗/ BinaryAnalysis : : ControlFlow c f g a n a l y z e r ; B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph∗ c f g = new B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph ; c f g a n a l y z e r . b u i l d b l o c k c f g f r o m a s t ( i n t e r p s . back ( ) , ∗ c f g ) ; s t d : : o f s t r e a m mf ; mf . open ( ” a n a l y s i s . d o t ” ) ; /∗ D e c l a r i n g t h e v i s i t o r T r a v e r s a l ∗/ v i s i t o r T r a v e r s a l ∗ v i s = new v i s i t o r T r a v e r s a l ; /∗ S e t t i n g i n t e r n a l v a r i a b l e s ∗/ v i s −>t l t n o d e s = 0 ; v i s −>p t h s = 0 ;

/∗ v i s i t o r T r a v e r s a l has 5 arguments , t h e f i r s t i s t h e ambient CFG, t h e second i d e n t i f i e s w h e t h e r or not you a re bounding t h e graph , t h a t i s , w h e t h e r you want a l l your p a t h s t o s t a r t a t one s p e c i f i c node and end a t a n o t h e r s p e c i f i c node , t h e f o u r t h and f i f t h would be s t a r t and end i f t h e graph were bounded . S i n c e t h e y aren ’ t you can s i m p l y i n p u t 0 , f o r t h e moment t h e f i n a l argument i s d e p r e c a t e d , t h o u g h i t ’ s p u r p o s e was t o t e l l t h e progra t h a t your a n a l y s i s f u n c t i o n was t h r e a d s a f e , t h a t i s t h a t openMP c o u l d run i t w i t h o u t h a v i n g a c r i t i c a l command . C u r r e n t l y a c r i t i c a l i s a l w a y s used ∗/ v i s −>c o n s t r u c t P a t h A n a l y z e r ( c f g , true , 0 , 0 , f a l s e ) ;

162

CHAPTER 22. GRAPH PROCESSING TUTORIAL

Chapter 23

Dataflow Analysis The dataflow analysis in Rose is based on the control flow graph (CFG). One type of dataflow analysis is called def-use analysis, which is explained next.

23.1

Def-Use Analysis

The definition-usage (def-use) analysis allows to query the definition and usage for each control flow node (CFN). Any statement or expression within ROSE is represented as a sequence of CFN’s. For instance, the CFG for the following program 1 2 3 4 5

i n t main ( ) { int x = 9; x = x + 1; }

Figure 23.1: Example input code. is illustrated in Figure 23.4.

23.1.1

Def-use Example implementation

Fig. 23.2 shows an example program of how the def-use analysis is called. It generates a dot graph showing def/use information within a control flow graph. It also outputs reaching definition information for each variable references of an input code. This program (named as defuseAnalysis) is installed under ROSE INST/bin as a standalone tool for users to experiment the def/use analysis of ROSE. Figure 23.3 shows the screen output of the code(Fig. 23.2) running on the input code(Fig. 23.1). Each variable reference in the input code has at least one reaching definition node. The associated definition statement is also printed out. 163

164

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

CHAPTER 23. DATAFLOW ANALYSIS

#i n c l u d e ” r o s e . h” #i n c l u d e ” D e f U s e A n a l y s i s . h” #i n c l u d e #i n c l u d e u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; v e c t o r a r g v L i s t ( argv , a r g v + a r g c ) ; SgProject ∗ project = frontend ( argvList ) ; // C a l l t h e Def−Use A n a l y s i s DFAnalysis ∗ d e f u s e = new D e f U s e A n a l y s i s ( p r o j e c t ) ; b o o l debug = f a l s e ; d e f u s e −>run ( debug ) ; // Output d e f u s e a n a l y s i s r e s u l t s i n t o a d o t f i l e d e f u s e −>dfaToDOT ( ) ; // Find a l l v a r i a b l e r e f e r e n c e s N o d e Q u e r y S y n t h e s i z e d A t t r i b u t e T y p e v a r s = NodeQuery : : querySubTree ( p r o j e c t , V SgVarRefExp ) ; NodeQuerySynthesizedAttributeType : : c o n s t i t e r a t o r i = vars . begin ( ) ; f o r ( ; i != v a r s . end ();++ i ) { SgVarRefExp ∗ v a r R e f = isSgVarRefExp ( ∗ i ) ; S g I n i t i a l i z e d N a m e ∗ initName = i s S g I n i t i a l i z e d N a m e ( varRef−>g e t s y m b o l ()−> g e t d e c l a r a t i o n ( ) ) ; s t d : : s t r i n g name = initName−>g e t q u a l i f i e d n a m e ( ) . s t r ( ) ; // Find r e a c h i n g d e f i n i t i o n o f initName a t t h e c o n t r o l f l o w node v a r R e f v e c t o r v e c = d e f u s e −>g e t D e f F o r ( varRef , initName ) ; ROSE ASSERT ( v e c . s i z e ( ) >0 ) ; // e a c h v a r i a b l e r e f e r e n c e must have a d e f i n i t i o n somewhere // Output e a c h r e a c h i n g d e f i n i t i o n node and t h e c o r r e s p o n d i n g s t a t e m e n t . s t d : : cout<<”−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−”<u n p a r s e T o S t r i n g ( ) << ” @ l i n e ” << varRef−> g e t f i l e i n f o ()−> g e t l i n e ()<<”:”<< varRef−> g e t f i l e i n f o ()−> g e t c o l ( ) << s t d : : e n d l ; f o r ( s i z e t j =0; j c l a s s n a m e ()<<” ”<u n p a r s e T o S t r i n g ()<<” @ l i n e ”< g e t f i l e i n f o ()−> g e t l i n e ( ) <<”:”<< d e f s t m t −> g e t f i l e i n f o ()−> g e t c o l ( ) <
Figure 23.2: Example source code using def use analysis

23.1.2

Accessing the Def-Use Results

For each CFN in the CFG, the definition and usage for variable references can be determined with the public function calls: vector getDefFor(SgNode*, SgInitializedName*) vector getUseFor(SgNode*, SgInitializedName*) where SgNode* represents any control flow node and SgInitializedName any variable (being used or defined at that CFN). The result is a vector of possible CFN’s that either define

23.1. DEF-USE ANALYSIS

1 2 3 4 5 6 7 8

165

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− 1 d e f i n i t i o n entry / e n t r i e s f o r x @ l i n e 4:3 S g A s s i g n I n i t i a l i z e r 0 x7fa4be1f0010 int x = 9; @ line 3:3 −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− 1 d e f i n i t i o n entry / e n t r i e s f o r x @ l i n e 4:7 S g A s s i g n I n i t i a l i z e r 0 x7fa4be1f0010 int x = 9; @ line 3:3

Figure 23.3: Output of the program (getDefFor) or use (getUseFor) a specific variable. Figure 23.4 shows how the variable x is being declared and defined in CFN’s between node 1 and 6. Note that the definition is annotated along the edge. For instance at node 6, the edge reads (6) DEF: x (3) = 5. This means that variable x was declared at CFN 3 but defined at CFN 5. The second statement x=x+1 is represented by CFN’s from 7 to 12. One can see in the figure that x is being re-defined at CFN 11. However, the definition of x within the same statement happens at CFN 8. Hence, the definition of the right hand side x in the statement is at CFN 5 : (8) DEF: x (3) = 5. Another usage of the def-use analysis is to determine which variables actually are defined at each CFN. The following function allows to query a CFN for all its variables (SgInitializedNames) and the positions those variables are defined (SgNode): std::multimap getDefMultiMapFor(SgNode*) std::multimap getUseMultiMapFor(SgNode*) All public functions are described in DefuseAnalysis.h. To use the def-use analysis, one needs to create an object of the class DefUseAnalysis and execute the run function. After that, the described functions above help to evaluate definition and usage for each CFN.

166

CHAPTER 23. DATAFLOW ANALYSIS

::main : ( 2 ) - [] 0x7fa4be4bd010 @line=2, col=1 :idx=0 DEF: x ( 4 ) - 12 : ( 3 ) - [] 0x7fa4bf0536b8 @line=1, col=1 :idx=0

: ( 4 ) - [] initVar : x0x7fa4bef46c20 x :idx=0 DEF: x ( 4 ) - 4 : ( 5 ) - [] 0x7fa4be223010 @line=3, col=11 :idx=1 DEF: x ( 4 ) - 4 : ( 6 ) - [] 0x7fa4be1f0010 @line=0, col=0 :idx=1 DEF: x ( 4 ) - 6 : ( 7 ) - [] varDecl : x,0x7fa4be378010 @line=3, col=3 :idx=1 DEF: x ( 4 ) - 6 : ( 8 ) - [] varRef : x0x7fa4be1bd010 @line=4, col=3 :idx=0 DEF: x ( 4 ) - 6 : ( 9 ) - [] varRef : x0x7fa4be1bd078 @line=4, col=7 :idx=0 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 10 ) - [] 0x7fa4be223078 @line=4, col=11 :idx=1 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 11 ) - [] 0x7fa4be186010 @line=4, col=7 :idx=2 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 12 ) - [] 0x7fa4be14f010 @line=4, col=3 :idx=2 DEF: x ( 4 ) - 12 : ( 13 ) - [] 0x7fa4be120010 @line=4, col=3 :idx=1 DEF: x ( 4 ) - 12 ::main : ( 2 ) - [] 0x7fa4be4bd010 @line=2, col=1 :idx=3

Figure 23.4: Def-Use graph for example program.

23.2. LIVENESS ANALYSIS

23.2

167

Liveness Analysis

Liveness analysis is a classic data flow analysis performed by compilers to calculate for each program point the variables that may be potentially read before their next write (re-definition). A variable is live at a point in a program’s execution path if it holds a value that may be needed in the future. Fig. 23.5 shows an example program of how the liveness analysis is called in a ROSE-based translator. It generates a dot graph showing def/use information within a control flow graph, alone with live-in and live-out variables. This program (named as livenessAnalysis) is installed under ROSE INST/bin as a standalone tool for users to experiment the liveness analysis of ROSE. Figure 23.7 shows control flow graph with live variable information for the code(Fig. 23.5) running on the input code(Fig. 23.6).

23.2.1

Access live variables

After calling liveness analysis, one can access live-in and live-out variables from a translator based on the virtual control flow graph node. Figure 23.8 shows an example function which retrieves the live-in and live-out variables for a for loop. The code accesses the CFG node (showing in Figure 23.7) of a for statement and retrieve live-in variables of the true edge’s target node as the live-in variables of the loop body. Similarly, the live-out variables of the loop are obtained by getting the live-in variables of the node right after the loop (target node of the false edge).

168

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

CHAPTER 23. DATAFLOW ANALYSIS

#i n c l u d e ” r o s e . h” #i n c l u d e #i n c l u d e #i n c l u d e u s i n g namespace s t d ; u s i n g namespace Rose ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; v e c t o r a r g v L i s t ( argv , a r g v + a r g c ) ; SgProject ∗ project = frontend ( argvList ) ; i f ( p r o j e c t −> g e t f i l e L i s t ( ) . s i z e ( ) ==0) return 0; // P r e p a r e t h e Def−Use A n a l y s i s DFAnalysis ∗ d e f u s e = new D e f U s e A n a l y s i s ( p r o j e c t ) ; b o o l debug = f a l s e ; d e f u s e −>run ( debug ) ; i f ( debug ) d e f u s e −>dfaToDOT ( ) ; // P r e p a r e l i v e n e s s a n a l y s i s L i v e n e s s A n a l y s i s ∗ l i v = new L i v e n e s s A n a l y s i s ( debug , ( D e f U s e A n a l y s i s ∗ ) d e f u s e ) ; ROSE ASSERT ( l i v != NULL ) ; // Find a l l f u n c t i o n d e f i n i t i o n s R o s e S T L C o n t a i n e r n o d e L i s t= NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; s t d : : v e c t o r > d f a F u n c t i o n s ; R o s e S T L C o n t a i n e r : : c o n s t i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; b o o l abortme= f a l s e ; f o r ( ; i != n o d e L i s t . end ();++ i ) { SgFunctionDefinition ∗ func = i s S g F u n c t i o n D e f i n i t i o n (∗ i ) ; // run l i v e n e s s a n a l y s i s FilteredCFGNode r e m s o u r c e = l i v −>run ( f u n c , abortme ) ; i f ( abortme ) { c e r r <<”E r r o r : L i v e n e s s a n a l y s i s i s ABORTING . ” << e n d l ; ROSE ASSERT( f a l s e ) ; } i f ( r e m s o u r c e . getNode ( ) ! =NULL) dfaFunctions . push back ( rem source ) ; } S g F i l e P t r L i s t f i l e l i s t = p r o j e c t −> g e t f i l e L i s t ( ) ; s t d : : s t r i n g f i r s t F i l e N a m e = S t r i n g U t i l i t y : : st r ip P at h Fr o m Fi l eN a me ( f i l e l i s t [0] − > g e t F i l e N a m e ( ) ) ; s t d : : s t r i n g f i l e N a m e = f i r s t F i l e N a m e +” l i v e n e s s . d o t ” ; std : : ofstream f s ( fileName . c s t r ( ) ) ; dfaToDot ( f s , s t r i n g ( ” v a r ” ) , d f a F u n c t i o n s , ( D e f U s e A n a l y s i s ∗ ) d e f u s e , l i v ) ; fs . close (); return 0; }

Figure 23.5: Example source code using liveness analysis

23.2. LIVENESS ANALYSIS

1 2 3 4 5 6 7 8 9 10 11 12 13 14

169

int a [100]; void foo2 () { int i ; i n t tmp ; tmp = 1 0 ; f o r ( i =0; i <100; i ++) { a [ i ] = tmp ; tmp =a [ i ]+ i ; } a[0] = 1 ; }

Figure 23.6: Example input code.

170

CHAPTER 23. DATAFLOW ANALYSIS

::foo2 : ( 3 ) - [] 0x7ff917870010 @line=4, col=1 :idx=0 out : : ( 4 ) - [] 0x7ff918021528 @line=3, col=1 :idx=0 out : : ( 5 ) - [] initVar : i0x7ff917f0db98 i :idx=0 out : DEF: i ( 5 ) - 5 : ( 6 ) - [] varDecl : i,0x7ff917ab02a8 @line=5, col=3 :idx=1 out : : ( 7 ) - [] initVar : tmp0x7ff917f0dce0 tmp :idx=0 out : DEF: tmp ( 7 ) - 7 : ( 8 ) - [] varDecl : tmp,0x7ff917ab0540 @line=6, col=3 :idx=1 out : : ( 9 ) - [] varRef : tmp0x7ff91783d010 @line=7, col=3 :idx=0 out : : ( 10 ) - [] 0x7ff917c40078 @line=7, col=9 :idx=1 out : : ( 11 ) - [] 0x7ff917806010 @line=7, col=3 :idx=2 out : tmp, DEF: tmp ( 7 ) - 11 : ( 12 ) - [] 0x7ff9177d7010 @line=7, col=3 :idx=1 out : tmp, : ( 13 ) - [] varRef : i0x7ff91783d078 @line=8, col=8 :idx=0

: ( 34 ) - [] 0x7ff917637010 @line=11, col=10 :idx=2

out : tmp,

out : i,

: ( 14 ) - [] 0x7ff917c400e0 @line=8, col=10 :idx=1

: ( 35 ) - [] 0x7ff917806160 @line=11, col=5 :idx=2 out : tmp,i, DEF: tmp ( 7 ) - 35

out : tmp,

: ( 15 ) - [] 0x7ff917806080 @line=8, col=8 :idx=2

: ( 36 ) - [] 0x7ff9177d7190 @line=11, col=5 :idx=1

out : tmp,i, DEF: i ( 5 ) - 15

out : tmp,i,

: ( 16 ) - [] 0x7ff9177d7070 @line=8, col=8 :idx=1

: ( 37 ) - [] varRef : i0x7ff91783d148 @line=8, col=18 :idx=0 out : tmp, USE: i ( 5 ) - 37

out : tmp,i,

: ( 17 ) - [] 0x7ff91770f080 @line=8, col=8 :idx=1

: ( 38 ) - [] 0x7ff9176a5010 @line=8, col=18 :idx=1 out : tmp,i, DEF: i ( 5 ) - 38

out : tmp,i,

: ( 18 ) - [] varRef : i0x7ff91783d0e0 @line=8, col=12 :idx=0 out : tmp,i, USE: i ( 5 ) - 18 : ( 19 ) - [] 0x7ff917c40148 @line=8, col=14 :idx=1 out : tmp,i, : ( 20 ) - [] 0x7ff9176d8010 @line=8, col=12 :idx=2 out : tmp,i, : ( 21 ) - [] 0x7ff9177d70d0 @line=8, col=12 :idx=1 out : tmp,i, : ( 22 ) - [] 0x7ff917746010 @line=8, col=3 :idx=2 out : tmp,i,

out : tmp,i,

: ( 23 ) - [] varRef : a0x7ff91783d1b0 @line=10, col=5 :idx=0

: ( 39 ) - [] varRef : a0x7ff91783d488 @line=13, col=4 :idx=0

out : tmp,i,

out : i, USE: i ( 5 ) - 33

out :

: ( 24 ) - [] varRef : i0x7ff91783d218 @line=10, col=7 :idx=0

: ( 40 ) - [] 0x7ff917c401b0 @line=13, col=6 :idx=1

out : tmp,i, USE: i ( 5 ) - 24

out :

: ( 25 ) - [] 0x7ff91766e010 @line=10, col=5 :idx=2

: ( 41 ) - [] 0x7ff91766e0f0 @line=13, col=4 :idx=2

out : tmp,i,

out :

: ( 26 ) - [] varRef : tmp0x7ff91783d280 @line=10, col=12 :idx=0

: ( 42 ) - [] 0x7ff917c40218 @line=13, col=11 :idx=1

out : i, USE: tmp ( 7 ) - 26

out :

: ( 27 ) - [] 0x7ff9178060f0 @line=10, col=5 :idx=2

: ( 43 ) - [] 0x7ff9178061d0 @line=13, col=4 :idx=2

out : i,a, DEF: ::a ( 2 ) - 27

out : DEF: ::a ( 2 ) - 43

: ( 28 ) - [] 0x7ff9177d7130 @line=10, col=5 :idx=1

: ( 44 ) - [] 0x7ff9177d71f0 @line=13, col=4 :idx=1

out : i,a,

out :

: ( 29 ) - [] varRef : tmp0x7ff91783d2e8 @line=11, col=5 :idx=0

::foo2 : ( 3 ) - [] 0x7ff917870010 @line=4, col=1 :idx=3 out : i,a, : ( 30 ) - [] varRef : a0x7ff91783d350 @line=11, col=10 :idx=0 out : i, USE: ::a ( 2 ) - 30 : ( 31 ) - [] varRef : i0x7ff91783d3b8 @line=11, col=12 :idx=0 out : i, USE: i ( 5 ) - 31 : ( 32 ) - [] 0x7ff91766e080 @line=11, col=10 :idx=2 out : i, : ( 33 ) - [] varRef : i0x7ff91783d420 @line=11, col=15 :idx=0

Figure 23.7: Control flow graph annotated with live variables for example program.

23.2. LIVENESS ANALYSIS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

171

v o i d g e t L i v e V a r i a b l e s ( L i v e n e s s A n a l y s i s ∗ l i v , SgF orSta teme nt ∗ f o r s t m t ) { ROSE ASSERT( l i v != NULL ) ; ROSE ASSERT( f o r s t m t != NULL ) ; s t d : : v e c t o r l i v e I n s , l i v e O u t s ; // For SgForStatement , v i r t u a l CFG node which i s i n t e r e s t i n g has an i n d e x number o f 2 , // a s shown i n i t s d o t graph ’ s node c a p t i o n . // ” @ 8 : 2” means t h i s node i s f o r a f o r s t a t e m e n t a t s o u r c e l i n e 8 , w i t h an i n d e x 2 . CFGNode c f g n o d e ( f o r s t m t , 2 ) ; FilteredCFGNode f i l t e r n o d e= FilteredCFGNode ( c f g n o d e ) ; // Check e d g e s v e c t o r > o u t e d g e s = f i l t e r n o d e . out Edges ( ) ; ROSE ASSERT( o u t e d g e s . s i z e ( ) == 2 ); v e c t o r > : : i t e r a t o r i t e r = o u t e d g e s . b e g i n ( ) ; f o r ( ; i t e r != o u t e d g e s . end ( ) ; i t e r ++) { FilteredCFGEdge < I s D F A F i l t e r > e d g e= ∗ i t e r ; // one t r u e e d g e g o i n g i n t o t h e l o o p body // x . Live−i n ( l o o p ) = l i v e −i n ( f i r s t −stmt−i n −l o o p ) i f ( e d g e . c o n d i t i o n ()== eckTrue ) { SgNode∗ f i r s t n o d e= e d g e . t a r g e t ( ) . getNode ( ) ; l i v e I n s = l i v −>g e t I n ( f i r s t n o d e ) ; } // one f a l s e e d g e g o i n g o u t o f l o o p // x . l i v e −o u t ( l o o p ) = l i v e −i n ( f i r s t −stmt−a f t e r −l o o p ) e l s e i f ( e d g e . c o n d i t i o n ()== e c k F a l s e ) { SgNode∗ f i r s t n o d e= e d g e . t a r g e t ( ) . getNode ( ) ; l i v e O u t s 0 = l i v −>g e t I n ( f i r s t n o d e ) ; } else { c e r r <<”Unexpected CFG o u t e d g e t y p e f o r SgForStmt!”<< e n d l ; ROSE ASSERT( f a l s e ) ; } } // end f o r ( e d g e s ) }

Figure 23.8: Example code retrieving live variables based on virtual control flow graph

172

CHAPTER 23. DATAFLOW ANALYSIS

Chapter 24

Generating the Call Graph (CG) The formal definition of a call graph is: ’A diagram that identifies the modules in a system or computer program and shows which modules call one another.’ IEEE A call graph shows all function call paths of an arbitrary code. These paths are found by following all function calls in a function, where a function in the graph is represented by a node and each possible function call by an edge (arrow). To make a call graph this process is redone for every called function until all edges are followed and there are no ungraphed functions. ROSE has an in-build mechanism for generating call graphs. ROSE provides support for generating call graphs, as defined in src/midend/programAnalysis/CallGraphAnalysis/CallGraph.h. Figure 24 shows the code required to generate the call graph for each function of an application. Using the input code shown in figure 24 the first function’s call graph is shown in figure 24.3. A standalone tool named buildCallGraph is installed under ROSE INSTALL/bin so users can use it to generate call graphs in dot format.

173

174

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

CHAPTER 24. GENERATING THE CALL GRAPH (CG)

#i n c l u d e ” r o s e . h” #i n c l u d e #i n c l u d e u s i n g namespace s t d ; u s i n g namespace Rose ; u s i n g namespace Rose : : S t r i n g U t i l i t y ; // A F u n c t i o n o b j e c t u s e d a s a p r e d i c a t e t h a t d e t e r m i n e s which f u n c t i o n s a r e // t o be r e p r e s e n t e d i n t h e c a l l graph . s t r u c t k e e p F u n c t i o n : p u b l i c u n a r y f u n c t i o n { bool operator ( ) ( SgFunctionDeclaration ∗ funcDecl ) { bool returnValue = true ; ROSE ASSERT( f u n c D e c l != NULL ) ; s t r i n g f i l e n a m e = f u n c D e c l −> g e t f i l e i n f o ()−> g e t f i l e n a m e ( ) ; s t d : : s t r i n g func name = f u n c D e c l −>get name ( ) . g e t S t r i n g ( ) ; s t r i n g s t r i p p e d f i l e n a m e = s tr i pP a th F r om F il e Na m e ( f i l e n a m e ) ; // s t r i n g : : s i z e t y p e l o c ; // F i l t e r o u t f u n c t i o n s from t h e ROSE p r e i n c l u d e h e a d e r f i l e i f ( f i l e n a m e . f i n d ( ” r o s e e d g r e q u i r e d m a c r o s a n d f u n c t i o n s ” ) ! = s t r i n g : : npos ) returnValue = f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s e l s e i f ( f u n c D e c l −> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ()== t r u e ) r e t u r n V a l u e= f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s e l s e i f ( f u n c D e c l −> g e t f i l e i n f o ()−> i s F r o n t e n d S p e c i f i c ()== t r u e ) r e t u r n V a l u e= f a l s e ; // f i l t e r o u t o t h e r b u i l t i n f u n c t i o n s ” ,0)== 0 ) ; // e l s e i f ( func name . f i n d ( ” // returnValue = f a l s e ; IO feof , etc . // I O g e t c I O p u t c // l o c = func name . f i n d ( ” I O ” , 0 ) ; // i f ( l o c == 0 ) r e t u r n V a l u e = f a l s e ; // s k i p f u n c t i o n s from s t a n d a r d s y s t e m h e a d e r s // TODO Need more r i g i d c h e c k else if ( s t r i p p e d f i l e n a m e==s t r i n g ( ” s t d i o . h ” ) | | s t r i p p e d f i l e n a m e==s t r i n g ( ” l i b i o . h ” ) | | s t r i p p e d f i l e n a m e==s t r i n g ( ” math . h ” ) | | s t r i p p e d f i l e n a m e==s t r i n g ( ” t i m e . h ” ) | | s t r i p p e d f i l e n a m e==s t r i n g ( ” s e l e c t . h ” ) | | s t r i p p e d f i l e n a m e==s t r i n g ( ” m a t h c a l l s . h ” ) ) r e t u r n V a l u e= f a l s e ; i f ( returnValue ) cout <<”Debug:”<< func name << ” from f i l e :”<< s t r i p p e d f i l e n a m e <<” Keep : ”<
( p r o j e c t −> g e t f i l e L i s t ( ) . s i z e ( ) >=1) // C o n s t r u c t a c a l l Graph C a l l G r a p h B u i l d e r CGBuilder ( p r o j e c t ) ; CGBuilder . b u i l d C a l l G r a p h ( k e e p F u n c t i o n ( ) ) ; CGBuilder . b u i l d C a l l G r a p h ( b u i l t i n F i l t e r ( ) ) ;

//

// Output t o a d o t f i l e AstDOTGeneration d o t g e n ; S g F i l e P t r L i s t f i l e l i s t = p r o j e c t −> g e t f i l e L i s t ( ) ; s t d : : s t r i n g f i r s t F i l e N a m e = S t r i n g U t i l i t y : : s t ri p Pa t hF r o mF i le N am e ( f i l e l i s t [0] − > g e t F i l e N a m e ( ) ) ; d o t g e n . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( CGBuilder . getGraph ( ) , f i r s t F i l e N a m e +” c a l l G r a p h . do t ” ) ; } r e t u r n 0 ; // backend ( p r o j e c t ) ; }

Figure 24.1: Example source code showing visualization of call graph.

175

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

// s i m p l e ( class A { public : int int int }; void void void void

foo1 foo2 foo3 foo4

t r i v i a l ) example c o d e u s e d t o d e m o n s t r a t e t h e

f1 () { return 0;} f 2 ( ) { p f = &A : : f 1 ; (A : : ∗ p f ) ( ) ;

c a l l graph g e n e r a t i o n

r e t u r n ( t h i s −>∗p f ) ( ) ; }

(); (); (); ();

void foo1 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo2 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo3 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo4 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } i n t main ( ) { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; return 0; }

Figure 24.2: Example source code used as input to build call graph.

176

CHAPTER 24. GENERATING THE CALL GRAPH (CG)

::A::f2 child_count:0 0x14dc410

::A::f1 child_count:0 0x14dc3a0

::main child_count:0 0x14dc330

::foo1 child_count:0 0x14dc170

::foo2 child_count:0 0x14dc1e0

::foo4 child_count:0 0x14dc2c0

::foo3 child_count:0 0x14dc250

Figure 24.3: Call graph for function in input code file: inputCode BuildCG.C.

Chapter 25

Dataflow Analysis based Virtual Function Analysis C++ Virtual function provides polymorphism to the developer but makes it difficult for compilers to do optimizations. Virtual functions are usually resolved at runtime from the vtable. It’s very difficult for a compiler to know which functions will be called at compile time. ROSE provides a flow sensitive dataflow analysis based approach to cut down the set of possible function calls. The code for Virtual Function Analysis is located in src/midend/programAnalysis/VirtualFunctionAnalysis/VirtualFunctionAnalysis.h. It also provides a mechanism to resolve any function calls. It’s a whole program analysis and supposed to be expensive. It memorizes all the resolved function calls for any call site, so that subsequent calls are resolved faster. Figure 25.1 shows the code required to generate the pruned call graph. Using the input code shown in figure 25 Call Graph Analysis generates call graph shown in figure 25.3. Executing dataflow analysis to resolve virtual function calls resulted in the figure 25.4.

177

178

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75

CHAPTER 25. DATAFLOW ANALYSIS BASED VIRTUAL FUNCTION ANALYSIS

#i n c l u d e ” s a g e 3 b a s i c . h” #i n c l u d e #i n c l u d e #i n c l u d e ” V i r t u a l F u n c t i o n A n a l y s i s . h” u s i n g namespace b o o s t ;

u s i n g namespace s t d ; u s i n g namespace b o o s t ; // A F u n c t i o n o b j e c t u s e d a s a p r e d i c a t e t h a t d e t e r m i n e s which f u n c t i o n s a r e // t o be r e p r e s e n t e d i n t h e c a l l graph . // L i a o 1 / 2 3 / 2 0 1 3 . I t t u r n s o u t t h e r e i s a n o t h e r F u n c t i o n F i l t e r u s e d i n s r c / midend / p r o g r a m A n a l y s i s / V i r t u a l F u n c // We have t o make two f i l t e r s c o n s i s t e n t o r t h e r e w i l l be mismatch ! s t r u c t k e e p F u n c t i o n : p u b l i c u n a r y f u n c t i o n { public : bool operator ( ) ( SgFunctionDeclaration ∗ funcDecl ){ bool returnValue = true ; ROSE ASSERT( f u n c D e c l != NULL ) ; s t r i n g f i l e n a m e = f u n c D e c l −> g e t f i l e i n f o ()−> g e t f i l e n a m e ( ) ; // F i l t e r o u t f u n c t i o n s from t h e ROSE p r e i n c l u d e h e a d e r f i l e i f ( f i l e n a m e . f i n d ( ” r o s e e d g r e q u i r e d m a c r o s a n d f u n c t i o n s ” ) ! = s t r i n g : : npos ) returnValue = f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s i f ( f u n c D e c l −> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ()== t r u e ) r e t u r n V a l u e= f a l s e ; #i f 0

// F i l t e r o u t p r o t o t y p e s when d e f i n i n g f u n c t i o n d e c l a r a t i o n s e x i s t a t t h e same t i m e // T h i s i s now n e c e s s a r y s i n c e we a l w a y s g e n e r a t e t h e f i r s t n o n d e f i n i n g d e c l a r a t i o n i n ROSE u s i n g EDG 4 . // We c a n n o t do t h i s s i n c e t h e c a l l graph g e n e r a t o r a l w a y s t r i e s t o u s e f i r s t n o n d e f i n i n g d e c l whenenve // T h i s non−d e f i n i n g d e c l f i l t e r w i l l e s s e n t i a l l y z e r o o u t t h e c a l l graph . // L i a o 1 / 2 3 / 2 0 1 3 i f ( f u n c D e c l −>g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL) i f ( f u n c D e c l −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == f u n c D e c l ) returnValue = f a l s e ; #e n d i f return returnValue ; } }; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; CallGraphBuilder b u i l d e r ( p r o j e c t ) ; b u i l d e r . buildCallGraph ( keepFunction ( ) ) ; // G e n e r a t e c a l l graph i n d o t f o r m a t AstDOTGeneration d o t g e n ; d o t g e n . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( b u i l d e r . getGraph ( ) , ” o r i g i n a l c a l l g r a p h . d o t ” ) ; S g F u n c t i o n D e c l a r a t i o n ∗ mainDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j e c t ) ; i f ( mainDecl == NULL) { s t d : : c e r r << ”Can ’ t e x e c u t e V i r t u a l F u n c t i o n A n a l y s i s w i t h o u t main f u n c t i o n \n ” ; return 0; } V i r t u a l F u n c t i o n A n a l y s i s ∗ a n a l = new V i r t u a l F u n c t i o n A n a l y s i s ( p r o j e c t ) ; a n a l −>run ( ) ; a n a l −>p r u n e C a l l G r a p h ( b u i l d e r ) ; AstDOTGeneration d o t g e n 2 ; d o t g e n 2 . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( b u i l d e r . getGraph ( ) , ” p r u n e d c a l l g r a p h . d o t ” ) ; d e l e t e anal ;

return 0; }

Figure 25.1: Source code to perform virtual function analysis

179

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

c l a s s Animal { public : v i r t u a l v o i d s h o u t ( ) {} }; c l a s s dog : p u b l i c Animal { public : v i r t u a l v o i d s h o u t ( ) {} }; c l a s s t e r r i e r : p u b l i c dog { public : v i r t u a l v o i d s h o u t ( ) {} }; class yterrier : public t e r r i e r { public : v i r t u a l v o i d s h o u t ( ) {} }; i n t main ( v o i d ) { Animal ∗∗p , ∗∗ q ; dog ∗x , d ; t e r r i e r ∗y ; y = new y t e r r i e r ; x = &d ; p = ( Animal ∗∗)&x ; q = p; ∗p = y ; x−>s h o u t ( ) ; return 0; }

Figure 25.2: Example source code used as input for Virtual Function Analysis.

::dog::shout child_count:0 0xeec1e0

::terrier::shout child_count:0 0xeec250

::main child_count:0 0xeec100

::Animal::shout child_count:0 0xeec170

::yterrier::shout child_count:0 0xeec2c0

::dog::dog child_count:0 0xeec330

::yterrier::yterrier child_count:0 0xeec3a0

Figure 25.3: Call graph generated by Call Graph Analysis for input code in inputCode vfa.C.

180

CHAPTER 25. DATAFLOW ANALYSIS BASED VIRTUAL FUNCTION ANALYSIS

::dog::shout child_count:0 0x23101e0

::terrier::shout child_count:0 0x2310250

::main child_count:0 0x2310100

::Animal::shout child_count:0 0x2310170

::yterrier::shout child_count:0 0x23102c0

::dog::dog child_count:0 0x2310330

::yterrier::yterrier child_count:0 0x23103a0

Figure 25.4: Call graph resulted from Virtual Function Analysis for input code in inputCode vfa.C.

Chapter 26

Generating the Class Hierarchy Graph 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

#i n c l u d e ” r o s e . h” #i n c l u d e ” CallGraph . h” #i n c l u d e #d e f i n e f o r e a c h BOOST FOREACH u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // C o n s t r u c t c l a s s h i e r a r c h y graph ClassHierarchyWrapper h i e r ( p r o j e c t ) ; // D i s p l a y t h e a n c e s t o r s o f e a c h c l a s s v e c t o r a l l C l a s s e s = S a g e I n t e r f a c e : : querySubTree( p r o j e c t , foreach ( SgClassDefinition ∗ classDef , a ll C la s s es ) { p r i n t f ( ” \ n%s s u b c l a s s e s : ” , c l a s s D e f −>g e t d e c l a r a t i o n ()−> get name ( ) . s t r ( ) ) ; foreach ( SgClassDefinition ∗ subclass , hier . getSubclasses ( classDef )) { p r i n t f (”% s , ” , s u b c l a s s −>g e t d e c l a r a t i o n ()−> get name ( ) . s t r ( ) ) ; } } return 0; }

Figure 26.1: Example source code showing visualization of class hierarchy graph. For C++, because of multiple inheritance, a class hierarchy graph is a directed graph with pointers from a class to a superclass. A superclass is a class which does not inherit from any other class. A class may inherit from a superclass by inheriting from another class which does 181

V SgClassDefinitio

182

CHAPTER 26. GENERATING THE CLASS HIERARCHY GRAPH

rather than by a direct inheritance. Figure 26 shows the code required to generate the class hierarchy graph for each class of an application. Using the input code shown in figure 26 the first function’s call graph is shown in figure 26.3. 1 2 3 4 5

c l a s s A{ } ; class B :

p u b l i c A{ } ;

class C :

p u b l i c B{ } ;

Figure 26.2: Example source code used as input to build class hierarchy graph. Figure 26.3 shows the class hierarchy graph for the classes in the input code in figure 26.

183

Figure 26.3: Class hierarchy graph in input code file: inputCode ClassHierarchyGraph.C.

184

CHAPTER 26. GENERATING THE CLASS HIERARCHY GRAPH

Chapter 27

Database Support This chapter is specific to support in ROSE for persistent storage. ROSE uses the SQLite database and makes it simple to store data in the database for retrieval in later phases of processing large multiple file projects. FIXME: Need more information here.

27.1

ROSE DB Support for Persistent Analysis

This section presents figure 27.3, a simple C++ source code using a template. It is used as a basis for showing how template instantiations are handled within ROSE. An example translator using a database connection to store function information is shown in Fig.27.1 and Fig.27.2. The output by the translator operating on the C++ source code is shown in Fig. 27.4.

27.2

Call Graph for Multi-file Application

This section shows an example of the use of the ROSE Database mechanism where information is stored after processing each file as part of generating the call graph for a project consisting of multiple files. The separate files are show in figures 27.3 and ??. These files are processed using the translator in figure ?? to generate the final project call graph shown in figure ??.

27.3

Class Hierarchy Graph

This section presents a translator in figure ??, to generate the class hierarchy graph of the example shown in figure ??. The input is a multi-file application show in figure ?? and figure ??. This example is incomplete.

185

FIXME: This example still needs to be implemented to use the new ROSE call graph generator.

FIXME: This example is still incomplete.

186

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

CHAPTER 27. DATABASE SUPPORT

// Example ROSE T r a n s l a t o r : u s e d f o r

t e s t i n g ROSE i n f r a s t r u c t u r e

#i n c l u d e ” r o s e . h” u s i n g namespace s t d ; u s i n g namespace Rose ; // DQ ( 9 / 9 / 2 0 0 5 ) : Don ’ t i n c l u d e t h e d a t a b a s e by d e f a u l t // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL #i n c l u d e ” GlobalDatabaseConnectionMYSQL . h” #e n d i f i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL // B u i l d t h e Data b a s e G l o b a l D a t a b a s e C o n n e c t i o n ∗gDB ; gDB = new G l o b a l D a t a b a s e C o n n e c t i o n ( ” functionNameDataBase ” ) ; gDB−> i n i t i a l i z e ( ) ; s t r i n g command = ” ” ; command = command + ”CREATE TABLE F u n c t i o n s ( name TEXT, c o u n t e r

);”;

Query ∗q = gDB−>getQuery ( ) ; q−>s e t ( command ) ; q−>e x e c u t e ( ) ; if

( q−>s u c c e s s ( ) != 0 ) c o u t << ” E r r o r c r e a t i n g schema : ” << q−>e r r o r ( ) << ”\n ” ; // A l t e r n a t i v e s y n t a x , but d o e s no t p e r m i t a c c e s s t o e r r o r m e s s a g e s and e x i t c o d e s // gDB−>e x e c u t e ( command . c s t r ( ) ) ; #e n d i f // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // Run i n t e r n a l c o n s i s t e n c y t e s t s on AST AstTests : : runAllTests ( p r o j e c t ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n ) ; int counter = 0; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L i s t . end ( ) ; i ++) { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l // t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration ∗ f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n (∗ i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; SgName func name = f u n c t i o n D e c l a r a t i o n −>get name ( ) ; // S k i p b u i l t i n f u n c t i o n s f o r s h o r t e r output , L i a o 4 / 2 8 / 2 0 0 8 i f ( func name . g e t S t r i n g ( ) . f i n d ( ” b u i l t i n ” ,0)==0) continue ;

Figure 27.1: Example translator (part 1) using database connection to store function names.

27.3. CLASS HIERARCHY GRAPH

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

187

// o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( ” f u n c t i o n name #%d i s %s a t l i n e %d \n ” , c o u n t e r ++,func name . s t r ( ) , f u n c t i o n D e c l a r a t i o n −> g e t f i l e i n f o ()−> g e t l i n e ( ) ) ; s t r i n g functionName = f u n c t i o n D e c l a r a t i o n −>g e t q u a l i f i e d n a m e ( ) . s t r ( ) ; // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL command = ”INSERT INTO F u n c t i o n s v a l u e s ( \ ” ” + functionName + ” \ ” , ” + S t r i n g U t i l i t y : : numberToString ( c o u n t e r ) + ” ) ; ” ; // A l t e r n a t i v e i n t e r f a c e // q−>s e t ( command ) ; // c o u t << ” E x e c u t i n g : ” << q−>p r e v i e w ( ) << ”\n ” ; // q−>e x e c u t e ( ) ; gDB−>e x e c u t e ( command . c s t r ( ) ) ; #e n d i f } // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL command = ”SELECT ∗ from F u n c t i o n s ; ” ; // A l t e r n a t i v e I n t e r f a c e ( u s i n g q u e r y o b j e c t s ) // q << command ; q−>s e t ( command ) ; c o u t << ” E x e c u t i n g : ” << q−>p r e v i e w ( ) << ”\n ” ; // e x e c u t e and r e t u r n r e s u l t ( a l t e r n a t i v e u s a g e : ”gDB−>s e l e c t ( ) ” ) R e s u l t ∗ r e s = q−>s t o r e ( ) ; i f ( q−>s u c c e s s ( ) != 0 ) c o u t << ” E r r o r r e a d i n g v a l u e s : ” << q−>e r r o r ( ) << ”\n ” ; else { // Read t h e t a b l e r e t u r n e d from t h e q u e r y // r e s −>s h o w R e s u l t ( ) ; f o r ( R e s u l t : : i t e r a t o r i = r e s −>b e g i n ( ) ; i != r e s −>end ( ) ; i++ ) { // A l t e r n a t i v e s y n t a x i s p o s s i b l e : ”Row r = ∗ i ; ” s t r i n g functionName = ( ∗ i ) [ 0 ] . g e t s t r i n g ( ) ; i n t counter = (∗ i ) [ 1 ] ; p r i n t f ( ” functionName = %s c o u n t e r = %d \n ” , functionName . c s t r ( ) , c o u n t e r ) ; } } gDB−>shutdown ( ) ; #e l s e p r i n t f ( ” Program c o m p i l e d w i t h o u t d a t a b a s e c o n n e c t i o n s u p p o r t ( add u s i n g ROSE c o n f i g u r e o p t i o n ) \n ” ) ; #e n d i f return 0; }

Figure 27.2: Example translator (part 2) using database connection to store function names.

188

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

CHAPTER 27. DATABASE SUPPORT

// T h i s example c o d e i s u s e d t o r e c o r d names o f f u n c t i o n s i n t o t h e d a t a b a s e . class A { public : virtual int f1 () = 0; v i r t u a l i n t f 2 ( ) {} int f3 ( ) ; virtual int f4 ( ) ; }; int A: : f3 () { f1 ( ) ; i n t A : : f 4 ( ) {}

return f3 ( ) ; }

class B : public A { public : virtual int f1 ( ) ; v i r t u a l i n t f 2 ( ) {} }; i n t B : : f 1 ( ) {} class C : public A { public : v i r t u a l i n t f 1 ( ) {} i n t f 3 ( ) {} }; class D : public B { public : v i r t u a l i n t f 2 ( ) {} }; class E : public D { public : virtual int f1 () { return 5; } }; class G : public E { public : virtual int f1 ( ) ; }; i n t G : : f 1 ( ) {} class F : public D { public : v i r t u a l i n t f 1 ( ) {} virtual int f2 () { return 5;} int f3 () { return 2;} }; class H : public C { public : v i r t u a l i n t f 1 ( ) {} v i r t u a l i n t f 2 ( ) {} i n t f 3 ( ) {} };

Figure 27.3: Example source code used as input to database example.

27.3. CLASS HIERARCHY GRAPH

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function

name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name

#0 i s #1 i s #2 i s #3 i s #4 i s #5 i s #6 i s #7 i s #8 i s #9 i s #10 i s #11 i s #12 i s #13 i s #14 i s #15 i s #16 i s #17 i s #18 i s #19 i s #20 i s #21 i s #22 i s #23 i s #24 i s #25 i s #26 i s #27 i s #28 i s #29 i s #30 i s #31 i s #32 i s #33 i s #34 i s #35 i s #36 i s #37 i s #38 i s #39 i s #40 i s #41 i s #42 i s #43 i s #44 i s #45 i s #46 i s #47 i s #48 i s #49 i s #50 i s #51 i s #52 i s #53 i s #54 i s #55 i s #56 i s #57 i s #58 i s #59 i s #60 i s #61 i s #62 i s #63 i s #64 i s #65 i s #66 i s #67 i s #68 i s #69 i s #70 i s #71 i s #72 i s #73 i s #74 i s #75 i s #76 i s #77 i s #78 i s #79 i s #80 i s #81 i s #82 i s

s y n c f e t c h a n d a d d at l i n e 0 s y n c f e t c h a n d a d d 1 at l i n e 0 s y n c f e t c h a n d a d d 2 at l i n e 0 s y n c f e t c h a n d a d d 4 at l i n e 0 s y n c f e t c h a n d a d d 8 at l i n e 0 s y n c f e t c h a n d a d d 1 6 at l i n e 0 s y n c f e t c h a n d s u b at l i n e 0 s y n c f e t c h a n d s u b 1 at l i n e 0 s y n c f e t c h a n d s u b 2 at l i n e 0 s y n c f e t c h a n d s u b 4 at l i n e 0 s y n c f e t c h a n d s u b 8 at l i n e 0 s y n c f e t c h a n d s u b 1 6 at l i n e 0 s y n c f e t c h a n d o r at l i n e 0 s y n c f e t c h a n d o r 1 at l i n e 0 s y n c f e t c h a n d o r 2 at l i n e 0 s y n c f e t c h a n d o r 4 at l i n e 0 s y n c f e t c h a n d o r 8 at l i n e 0 s y n c f e t c h a n d o r 1 6 at l i n e 0 s y n c f e t c h a n d a n d at l i n e 0 s y n c f e t c h a n d a n d 1 at l i n e 0 s y n c f e t c h a n d a n d 2 at l i n e 0 s y n c f e t c h a n d a n d 4 at l i n e 0 s y n c f e t c h a n d a n d 8 at l i n e 0 s y n c f e t c h a n d a n d 1 6 at l i n e 0 s y n c f e t c h a n d x o r at l i n e 0 s y n c f e t c h a n d x o r 1 at l i n e 0 s y n c f e t c h a n d x o r 2 at l i n e 0 s y n c f e t c h a n d x o r 4 at l i n e 0 s y n c f e t c h a n d x o r 8 at l i n e 0 s y n c f e t c h a n d x o r 1 6 at l i n e 0 s y n c a d d a n d f e t c h at l i n e 0 s y n c a d d a n d f e t c h 1 at l i n e 0 s y n c a d d a n d f e t c h 2 at l i n e 0 s y n c a d d a n d f e t c h 4 at l i n e 0 s y n c a d d a n d f e t c h 8 at l i n e 0 s y n c a d d a n d f e t c h 1 6 at l i n e 0 s y n c s u b a n d f e t c h at l i n e 0 s y n c s u b a n d f e t c h 1 at l i n e 0 s y n c s u b a n d f e t c h 2 at l i n e 0 s y n c s u b a n d f e t c h 4 at l i n e 0 s y n c s u b a n d f e t c h 8 at l i n e 0 s y n c s u b a n d f e t c h 1 6 at l i n e 0 s y n c o r a n d f e t c h at l i n e 0 s y n c o r a n d f e t c h 1 at l i n e 0 s y n c o r a n d f e t c h 2 at l i n e 0 s y n c o r a n d f e t c h 4 at l i n e 0 s y n c o r a n d f e t c h 8 at l i n e 0 s y n c o r a n d f e t c h 1 6 at l i n e 0 s y n c a n d a n d f e t c h at l i n e 0 s y n c a n d a n d f e t c h 1 at l i n e 0 s y n c a n d a n d f e t c h 2 at l i n e 0 s y n c a n d a n d f e t c h 4 at l i n e 0 s y n c a n d a n d f e t c h 8 at l i n e 0 s y n c a n d a n d f e t c h 1 6 at l i n e 0 s y n c x o r a n d f e t c h at l i n e 0 s y n c x o r a n d f e t c h 1 at l i n e 0 s y n c x o r a n d f e t c h 2 at l i n e 0 s y n c x o r a n d f e t c h 4 at l i n e 0 s y n c x o r a n d f e t c h 8 at l i n e 0 s y n c x o r a n d f e t c h 1 6 at l i n e 0 sync bool compare and swap at l i n e 0 sync val compare and swap at l i n e 0 sync val compare and swap 1 at l i n e 0 sync val compare and swap 2 at l i n e 0 sync val compare and swap 4 at l i n e 0 sync val compare and swap 8 at l i n e 0 sync val compare and swap 16 at l i n e 0 s y n c l o c k t e s t a n d s e t 1 at l i n e 0 s y n c l o c k t e s t a n d s e t 2 at l i n e 0 s y n c l o c k t e s t a n d s e t 4 at l i n e 0 s y n c l o c k t e s t a n d s e t 8 at l i n e 0 s y n c l o c k t e s t a n d s e t 1 6 at l i n e 0 s y n c l o c k r e l e a s e at l i n e 0 s y n c l o c k r e l e a s e 1 at l i n e 0 s y n c l o c k r e l e a s e 2 at l i n e 0 s y n c l o c k r e l e a s e 4 at l i n e 0 s y n c l o c k r e l e a s e 8 at l i n e 0 s y n c l o c k r e l e a s e 1 6 at l i n e 0 sync swap at l i n e 0 sync swap 1 at l i n e 0 sync swap 2 at l i n e 0 sync swap 4 at l i n e 0 sync swap 8 at l i n e 0

189

190

CHAPTER 27. DATABASE SUPPORT

Chapter 28

Building Custom Graphs What To Learn From This Example This example shows how to generate custom graphs using SgGraph class. Rose provides a collection type SgGraph to store a graph. Two specific graphs are also provided which are derived from SgGraph: SgIncidenceDirectedGraph and SgIncidenceUndirectedGraph. Nodes and edges in a SgGraph are represented by SgGraphNode and SgGraphEdge separately. A SgGraph is built by adding SgGraphNodes and SgGraphEdges using its member function addNode and addEdge. You can get all nodes and edges of a SgGraph by calling its functions computeNodeSet and computeEdgeSet separately. More interfaces of SgGraph and its subclasses can be found in doxygen of Rose. Since SgGraph is for Rose use, each node in it holds a pointer to SgNode, which is the default attribute of a SgGraphNode. If you want to add more attributes inside, you can use SgGraphNode’s member function addNewAttribute by providing a name and an AstAttribute object to add a new attribute to a node. Normally, you have to build your own attribute class which should be derived from class AstAttribute. At least three attribute classes are provided by ROSE: AstRegExAttribute, AstTextAttribute, and MetricAttribute. For more information about them, please refer to Rose’s doxygen.

191

192

CHAPTER 28. BUILDING CUSTOM GRAPHS

Part IV

Program Transformations and Optimizations

This part gives examples of building source-to-source program transformations and optimizations.

193

Chapter 29

Generating Unique Names for Declarations There are many instances where a unique name must be generated for either a function or variable declaration. ROSE defines a mechanism to make the generation of unique names from all SgDeclarationStatment IR nodes and the SgInitializedName IR node. This simplifies ROSEbased applications that require this sort of mechanism. Our experience has found that a significant number of tools require such a mechanism and that its correct implementation can have subtle points. The specific translator described in this chapter traverses an AST and outputs the unique names that can be generated for each declaration showing the use of the unique name generation mechanism. This tool is intended as an example of how to generate unique names using ROSE. Not all IR nodes can be used to generate a unique name. The generated names are unique under the following rules: 1. Any two generated names are the same if the declarations are the same. Declaration can be the same across files or within the same file. Declarations that are the same can have different location in the same file (be represented multiple times) or be in different files. Language constructs that are the same must follow the One-time Definition Rule (ODR) across files. 2. Declarations in different unnamed scopes (e.g. for loop bodies) will generate different names. 3. Names are the same when generated by different ROSE tools. Pointer values could be used to generate unique names of all IR nodes, but this would work only within a single invocation of the ROSE based tool. Generated names are not based on internal pointer values and are thus insensitive to pointer values. Generated names of the same declaration are thus the same even if generated from different tools. This allows multiple ROSE tools to inter-operate. This unique name generation mechanism is only applicable to specific IR nodes, specifically: • SgInitializedName 195

196

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS • SgDeclarationStatement IR nodes: – Obvious IR nodes supported: ∗ SgClassDeclaration ∗ SgFunctionDeclaration ∗ SgEnumDeclaration ∗ SgNamespaceDeclarationStatement ∗ SgTypedefDeclaration – Less obvious IR nodes not supported (support for these would not make sense): ∗ SgAsmStmt ∗ SgCtorInitializerList ∗ SgFunctionParameterList ∗ SgNamespaceAliasDeclarationStatement ∗ SgPragmaDeclaration ∗ SgTemplateDeclaration (can this have a mangled name?) ∗ SgTemplateInstantiationDirectiveStatement ∗ SgUsingDeclarationStatement ∗ SgUsingDirectiveStatement ∗ SgVariableDeclaration Note that the SgVariableDeclaration contains a list of SgInitializedName nodes and the mangled names are best queried from each SgInitializedName instead of the SgVariableDeclaration. ∗ SgVariableDefinition • Un-named scopes A number of scopes are un-names and so there is an opportunity to generate non-unique names from declarations in such scopes. To fix this we generate names for each un-named scope to guarantee uniqueness. Nodes handled are: – – – –

SgForStatement SgBasicBlock SgIfStmt get the complete list ...

Other language constructs can generate unique names as well, but their name could be invalid after certain transformation that move it structurally within the generated source code.

29.1

Example Code Showing Generation of Unique Names

29.2

Input For Examples Showing Unique Name Generation for Variables

Figure 29.1, shows an example translator demonstrating the generation of unique names from declarations in the AST. For each SgInitializedName we generate the mangled name. Figure 29.2

29.3. EXAMPLE OUTPUT SHOWING UNIQUE VARIABLE NAMES

197

shows the input code and figure 29.3 shows the generated output from the translator (the mangled names from the AST associated with the input application).

29.3

Example Output Showing Unique Variable Names

29.4

Input For Examples Showing Unique Name Generation for Functions

Figure 29.1, shows an example translator demonstrating the generation of unique names from declarations in the AST. For each SgInitializedName we generate the mangled name. Figure 29.4 shows the input code and figure 29.5 shows the generated output from the translator (the mangled names from the AST associated with the input application).

29.5

Example Output Showing Unique Function Names

198

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

// T h i s example shows t h e g e n e r a t i o n o f u n i q u e names from d e c l a r a t i o n s . // // // // // //

Mangled name demo T h i s t r a n s l a t o r q u e r i e s t h e AST f o r a l l S g I n i t i a l i z e d N a m e s and S g F u n c t i o n D e c l a r a t i o n s , and f o r e a c h one p r i n t s ( a ) t h e s o u r c e l o c a t i o n , ( b ) t h e s o u r c e name o f t h e o b j e c t , and ( c ) t h e mangled name .

#i n c l u d e u s i n g namespace s t d ; // R e t u r n s a S g F i l e I n f o o b j e c t a s a d i s p l a y −f r i e n d l y s t a t i c st r i n g toString ( const S g F i l e I n f o ∗ info ) { ostringstream i n f o s t r ; i f ( info ) i n f o s t r << ’ [ ’ << i n f o −>g e t r a w f i l e n a m e ( ) << ” : ” << i n f o −>g e t r a w l i n e ( ) << ’ ] ’ ; return i n f o s t r . str ( ) ; }

string ,

”[ source : l i n e ] ” .

// D i s p l a y s l o c a t i o n and mangled name o f an S g I n i t i a l i z e d N a m e o b j e c t . s t a t i c v o i d p r i n t I n i t i a l i z e d N a m e ( c o n s t SgNode∗ node ) { c o n s t S g I n i t i a l i z e d N a m e ∗ name = i s S g I n i t i a l i z e d N a m e ( node ) ; ROSE ASSERT ( name != NULL ) ; if

( name−> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) c o u t // << t o S t r i n g ( name−> g e t f i l e i n f o ( ) ) // << ” ” << name−>get name ( ) . s t r ( ) << ” −−> ” << name−>g e t m a n g l e d n a m e ( ) . s t r ( ) << e n d l ;

} // D i s p l a y s l o c a t i o n and mangled name o f an S g F u n c t i o n D e c l a r a t i o n o b j e c t . s t a t i c v o i d p r i n t F u n c t i o n D e c l a r a t i o n ( c o n s t SgNode∗ node ) { c o n s t S g F u n c t i o n D e c l a r a t i o n ∗ d e c l = i s S g F u n c t i o n D e c l a r a t i o n ( node ) ; ROSE ASSERT ( d e c l != NULL ) ; if

( d e c l −> g e t f i l e i n f o ()−> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) c o u t // << t o S t r i n g ( d e c l −>g e t s t a r t O f C o n s t r u c t ( ) ) // << ” ” << d e c l −>g e t q u a l i f i e d n a m e ( ) . s t r ( ) << ” −−> ” << d e c l −>g e t m a n g l e d n a m e ( ) . s t r ( ) << e n d l ;

} i n t main ( i n t a r g c , c h a r ∗∗ a r g v ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; c o u t << e n d l << ”∗∗∗∗∗ BEGIN i n i t i a l i z e d names ∗∗∗∗∗” << e n d l ; R o s e S T L C o n t a i n e r i n i t n a m e s = NodeQuery : : querySubTree ( p r o j , V S g I n i t i a l i z e d N a m e ) ; f o r e a c h ( i n i t n a m e s . b e g i n ( ) , i n i t n a m e s . end ( ) , p r i n t I n i t i a l i z e d N a m e ) ; c o u t << ”∗∗∗∗∗ END i n i t i a l i z e d names ∗∗∗∗∗” << e n d l ; c o u t << e n d l << ”∗∗∗∗∗ BEGIN f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗” << e n d l ; R o s e S T L C o n t a i n e r f u n c d e c l s = NodeQuery : : querySubTree ( p r o j , V S g F u n c t i o n D e c l a r a t i o n ) ; f o r e a c h ( f u n c d e c l s . b e g i n ( ) , f u n c d e c l s . end ( ) , p r i n t F u n c t i o n D e c l a r a t i o n ) ; c o u t << ”∗∗∗∗∗ END f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗” << e n d l ; r e t u r n backend ( p r o j ) ; }

Figure 29.1: Example source code showing the output of mangled name. The string represents the code associated with the subtree of the target IR node.

29.5. EXAMPLE OUTPUT SHOWING UNIQUE FUNCTION NAMES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

// I n p u t

file

199

to t e s t mangling o f S g I n i t i a l i z e d N a m e o b j e c t s .

int x ; // G l o b a l c l a s s class A { private : int x ; // N e s t e d c l a s s class B { private : int x ; public : void foo ( i n t x arg ) { i n t x ; } }; }; t e m p l a t e void f o o (T x a r g ) { T x; f o r ( x = 0 ; x < 1 0 ; x++) { T x = 0; do { // L o c a l c l a s s class A { private : // N e s t e d c l a s s class B { T x; }; public : v o i d f o o (T x ) {} }; T x = 0; } while (x > 0 ) ; do { T x = 0; } while (x > 0 ) ; // N e s t e d s c o p e { T x = 0; } } } t e m p l a t e v o i d f o o ( i n t x ) ; t e m p l a t e v o i d f o o ( d o u b l e x ) ; v o i d b ar ( v o i d ) { f o r ( i n t x = 0 ; x != 0 ; x++) f o r ( i n t x = 0 ; x != 0 ; x++) f o r ( l o n g x = 0 ; x != 0 ; x++) ; try { f o r ( i n t x = 0 ; x != 0 ; x++) ; } c a t c h ( i n t ) {} c a t c h ( c h a r x ) {} }

Figure 29.2: Example source code used as input to program in codes showing debugging techniques shown in this section.

200

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

NOTE:

Found

∗∗∗∗∗ x −−> x −−> x −−> x arg x −−> x arg x −−> x −−> x −−> x −−> x −−> x −−> x −−> x −−> x −−> x −−> x −−> ∗∗∗∗∗

BEGIN i n i t i a l i z e d names ∗∗∗∗∗ x scope x class A class A scope class B scope x −−> L6R L7R ARG1 L7R scope SgSS2 scope x L6R −−> f o o Fb v Gb T0x7f3063191010 Fe L1R ARG1 foo Fb v Gb T0x7f3063191010 Fe L1R scope SgSS2 scope x foo Fb v Gb T0x7f3063191010 Fe L1R scope SgSS2 scope L8R L9R ARG1 Fb v Gb T0x7f3063191010 Fe L1R scope SgSS2 scope foo foo Fb v Gb T0x7f3063191010 Fe L1R scope SgSS2 scope Fb v Gb T0x7f3063191010 Fe L1R scope SgSS2 scope foo bar Fb v Gb Fe L10R scope SgSS2 scope SgSS3 scope Fb v Gb Fe L10R scope SgSS2 scope SgSS3 scope bar bar Fb v Gb Fe L10R scope SgSS2 scope SgSS3 scope bar Fb v Gb Fe L10R scope SgSS2 scope SgSS6 scope bar Fb v Gb Fe L10R scope SgSS2 scope SgSS10 scope END i n i t i a l i z e d names ∗∗∗∗∗

input

type

to

convert template struct secondary ()

∗∗∗∗∗ BEGIN f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗ : : A : : B : : f o o −−> L6R L7R : : f o o −−> f o o Fb v Gb T0x7f3063191010 A : : f o o −−> L8R L9R : : b a r −−> b a r Fb v Gb Fe L10R ∗∗∗∗∗ END f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗

Fe

marked

a s p−>v a r i a n t . c l a s s s t r u c t u n i o n . i s t e m p l a t e c l a s s == f a l s e

SgSS6

scope

SgSS5

SgSS6 SgSS6 SgSS6 x SgSS4 SgSS4 SgSS7 x

scope scope scope

SgSS5 SgSS5 SgSS5

scope scope scope

scope

x

scope scope scope

SgSS4 SgSS8 SgSS9

x SgSS5 x

L1R

Figure 29.3: Output of input code using generatingUniqueNamesFromDeclaration.C

scope

x

scope scope scope

x

29.5. EXAMPLE OUTPUT SHOWING UNIQUE FUNCTION NAMES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

// I n p u t

long long long long long long {

file

201

to t e s t mangling o f S g F u n c t i o n D e c l a r a t i o n o b j e c t s .

foobar ( ) ; foobar ( int foobar ( int foobar ( int foobar ( int foobar ( int

); y); x); x = 0); xyz )

r e t u r n xyz ; } char foobarChar ( char ) ; char foobarChar ( char c ) ; // I n p u t

file

to t e s t mangling o f S g F u n c t i o n D e c l a r a t i o n o b j e c t s .

typedef int value0 t ; typedef value0 t value t ; namespace N { typedef struct { int a ; } s t ; c l a s s A { p u b l i c : A ( ) {} v i r t u a l v o i d f o o ( i n t ) {} } ; c l a s s B { p u b l i c : B ( ) {} v o i d f o o ( v a l u e t ) c o n s t {} } ; c l a s s C : p u b l i c A { p u b l i c : C ( ) {} v o i d f o o ( i n t ) {} v o i d f o o ( c o n s t v o i d f o o ( c o n s t s t ∗ ) {} }

s t &) {} } ;

typedef N: : s t s2 t ; void foo ( v a l u e t ) ; v o i d f o o ( s 2 t ) {} v o i d f o o ( f l o a t x [ ] ) {} void foo ( value t , s 2 t ) ; t e m p l a t e v o i d f o o (T) {} namespace P { typedef long double t y p e t ; namespace Q { t e m p l a t e v o i d f o o (T) {} class R { public : R ( ) {} t e m p l a t e v o i d f o o (T) {} v o i d f o o (P : : t y p e t ) {} t e m p l a t e i n t f o o (T) { r e t u r n x ; } }; } } t e m p l a t e i n t f o o (T) { r e t u r n x ; } template template template template

void void void void

f o o ( c h a r ) ; f o o ( c o n s t v a l u e t P : : Q : : f o o ( l o n g ) ; P : : Q : : R : : f o o ( v a l u e t ) ;

∗);

Figure 29.4: Example source code used as input to program in codes showing debugging techniques shown in this section.

202

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

∗∗∗∗∗ BEGIN i n i t i a l i z e d names ∗∗∗∗∗ x y z −−> f o o b a r Fb l Gb i Fe L9R ARG1 a −−> N scope class anonymous 0x18e18e0 scope a −−> L10R L11R ARG1 −−> L12R L13R ARG1 −−> L14R L15R ARG1 −−> f o o Fb v Gb L7R Fe L16R ARG1 x −−> L17R L18R ARG1 Fb v Gb T0x7f3f7aaaa010 Fe L19R ARG1 −−> f o o −−> L20R L21R ARG1 −−> L22R L23R ARG1 −−> L24R L25R ARG1 −−> L26R L27R ARG1 −−> f o o Fb i Gb T0x7f3f7aaaa290 Fe L28R ARG1 ∗∗∗∗∗ END i n i t i a l i z e d names ∗∗∗∗∗ ∗∗∗∗∗ BEGIN f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗ : : f o o b a r −−> f o o b a r Fb l Gb Fe L29R : : f o o b a r −−> f o o b a r Fb l Gb i Fe L9R : : f o o b a r −−> f o o b a r Fb l Gb i Fe L9R Fb l Gb i Fe L9R : : f o o b a r −−> f o o b a r : : f o o b a r −−> f o o b a r Fb l Gb i Fe L9R Fb l Gb i Fe L9R : : f o o b a r −−> f o o b a r : : f o o b a r C h a r −−> f o o b a r C h a r Fb c Gb c Fe L30R Fb c Gb c Fe L30R : : f o o b a r C h a r −−> f o o b a r C h a r : : N : : A : : A −−> L31R L32R : : N : : A : : f o o −−> L33R L34R : : N : : B : : B −−> L35R L36R : : N : : B : : f o o −−> L10R L11R : : N : : C : : C −−> L37R L38R : : N : : C : : f o o −−> L39R L40R : : N : : C : : f o o −−> L12R L13R : : N : : f o o −−> L14R L15R : : f o o −−> f o o Fb v Gb L6R Fe L41R : : f o o −−> f o o Fb v Gb L7R Fe L16R : : f o o −−> L17R L18R : : f o o −−> f o o Fb v Gb L6R sep L7R Fe L42R : : f o o −−> f o o Fb v Gb T0x7f3f7aaaa010 Fe L19R : : P : : Q : : f o o −−> L20R L21R : : P : : Q : : R : : R −−> L43R L44R : : P : : Q : : R : : f o o −−> L22R L23R : : P : : Q : : R : : f o o −−> L24R L25R : : P : : Q : : R : : f o o −−> L26R L27R : : f o o −−> f o o Fb i Gb T0x7f3f7aaaa290 Fe L28R ∗∗∗∗∗ END f u n c t i o n d e c l a r a t i o n s ∗∗∗∗∗

Figure 29.5: Output of input code using generatingUniqueNamesFromDeclaration.C

Chapter 30

Command-line Processing Within Translators ROSE includes mechanism to simplify the processing of command-line arguments so that translators using ROSE can trivially replace compilers within makefiles. This example shows some of the many command-line handling options within ROSE and the ways in which customized options may be added for specific translators.

30.1

Commandline Selection of Files

Overview This example shows the optional processing of specific files selected after the call to the frontend to build the project. First the SgProject if build and then the files are selected for processing via ROSE or the backend compiler directly. This example demonstrates the separation of the construction of a SgProject with valid SgFile objects for each file on the command line, but with an empty SgGlobal scope, and the call to the frontend, called for each SgFile in a separate loop over all the SgFile objects.

203

204

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

CHAPTER 30. COMMAND-LINE PROCESSING WITHIN TRANSLATORS

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; u s i n g namespace Rose ; int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; R o s e S T L C o n t a i n e r l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( ” P r e p r o c e s s o r ( b e f o r e ) : a r g v = \n%s \n ” , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // Remove c e r t a i n s o r t s o f o p t i o n s from t h e command l i n e C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l ,” − edg : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l ,”−−edg : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l ,” − e d g p a r a m e t e r : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l ,”−− e d g p a r a m e t e r : ” ) ; // Add a t e s t f o r a custom command l i n e o p t i o n int integerOptionForVerbose = 0; i f ( C o m m a n d l i n e P r o c e s s i n g : : i s O p t i o n W i t h P a r a m e t e r ( l ,” − m y T r a n s l a t o r : ” , ” ( v | v e r b o s e ) ” , i n t e g e r O p t i o n F o r V e r b o s { p r i n t f ( ” Turning on my t r a n s l a t o r ’ s v e r b o s e mode ( s e t t o %d ) \n ” , i n t e g e r O p t i o n F o r V e r b o s e ) ; } // Adding a new command l i n e p a r a m e t e r ( f o r mechanisms i n ROSE t h a t t a k e command l i n e s ) // p r i n t f ( ” a r g c = %zu \n ” , l . s i z e ( ) ) ; // l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( ” l . s i z e ( ) = %zu \n ” , l . s i z e ( ) ) ; p r i n t f ( ” P r e p r o c e s s o r ( a f t e r ) : a r g v = \n%s \n ” , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // // // //

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; G e n e r a t e t h e s o u r c e c o d e and c o m p i l e u s i n g t h e vendor ’ s c o m p i l e r r e t u r n backend ( p r o j e c t ) ;

// B u i l d t h e AST, frontend ( l ) ; return 0; }

g e n e r a t e t h e s o u r c e c o d e and c a l l

t h e backend c o m p i l e r

...

Figure 30.1: Example source code showing simple command-line processing within ROSE translator.

1 2 3 4 5 6

P r e p r o c e s s o r ( b e f o r e ) : argv = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / b u i l d t r e e / t u t o r i a l / . l i b s / l t −c o m m a n d l i n e Turning on my t r a n s l a t o r ’ s v e r b o s e mode ( s e t t o 4 2 ) l . size () = 4 P r e p r o c e s s o r ( a f t e r ) : argv = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / b u i l d t r e e / t u t o r i a l / . l i b s / l t −c o m m a n d l i n e

Figure 30.2: Output of input code using commandlineProcessing.C

30.1. COMMANDLINE SELECTION OF FILES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

205

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; u s i n g namespace Rose ; int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; R o s e S T L C o n t a i n e r l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( ” P r e p r o c e s s o r ( b e f o r e ) : a r g v = \n%s \n ” , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // Remove c e r t a i n s o r t s o f o p t i o n s from t h e command l i n e C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l ,” − edg : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l ,”−−edg : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l ,” − e d g p a r a m e t e r : ” ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l ,”−− e d g p a r a m e t e r : ” ) ; // Add a t e s t f o r a custom command l i n e o p t i o n int integerOptionForVerbose = 0; i f ( C o m m a n d l i n e P r o c e s s i n g : : i s O p t i o n W i t h P a r a m e t e r ( l ,” − m y T r a n s l a t o r : ” , ” ( v | v e r b o s e ) ” , i n t e g e r O p t i o n F o r V e r b o s e , t r u e ) ) { p r i n t f ( ” Turning on my t r a n s l a t o r ’ s v e r b o s e mode ( s e t t o %d ) \n ” , i n t e g e r O p t i o n F o r V e r b o s e ) ; } // Adding a new command l i n e p a r a m e t e r ( f o r mechanisms i n ROSE t h a t t a k e command l i n e s ) // p r i n t f ( ” a r g c = %zu \n ” , l . s i z e ( ) ) ; // l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( ” l . s i z e ( ) = %zu \n ” , l . s i z e ( ) ) ; p r i n t f ( ” P r e p r o c e s s o r ( a f t e r ) : a r g v = \n%s \n ” , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // // // //

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; G e n e r a t e t h e s o u r c e c o d e and c o m p i l e u s i n g t h e vendor ’ s c o m p i l e r r e t u r n backend ( p r o j e c t ) ;

// B u i l d t h e AST, frontend ( l ) ; return 0; }

g e n e r a t e t h e s o u r c e c o d e and c a l l

t h e backend c o m p i l e r

...

Figure 30.3: Example source code showing simple command-line processing within ROSE translator.

1 2 3 4 5 6

P r e p r o c e s s o r ( b e f o r e ) : argv = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / b u i l d t r e e / t u t o r i a l / . l i b s / l t −c o m m a n d l i n e P r o c e s s i n g − Turning on my t r a n s l a t o r ’ s v e r b o s e mode ( s e t t o 4 2 ) l . size () = 4 P r e p r o c e s s o r ( a f t e r ) : argv = / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / b u i l d t r e e / t u t o r i a l / . l i b s / l t −c o m m a n d l i n e P r o c e s s i n g −

Figure 30.4: Output of input code using commandlineProcessing.C

206

CHAPTER 30. COMMAND-LINE PROCESSING WITHIN TRANSLATORS

Chapter 31

Tailoring The Code Generation Format Figure 31.1 shows an example of how to use the mechanisms in ROSE to tailor the format and style of the generated code. This chapter presents an example translator that modifies the formatting of the code that is generated within ROSE. The details of functionality are hidden from the user and a high level interface is provided that permits key parameters to be specified. This example will be made more sophisticated later, for now it just modifies the indentation of nested code blocks (from 2 spaces/block to 5 spaces/block).

31.1

Source Code for Example that Tailors the Code Generation

Figure 31.1 shows an example translator which calls the inliner mechanism. The code is designed to only inline up to ten functions. the list of function calls is recomputed after any function call is successfully inlined. The input code is shown in figure 31.2, the output of this code is shown in figure 31.3.

31.2

Input to Demonstrate Tailoring the Code Generation

Figure 31.2 shows the example input used for demonstration of how to control the formatting of generated code.

31.3

Final Code After Tailoring the Code Generation

Figure 31.3 shows the results from changes to the formatting of generated code.

207

208

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

CHAPTER 31. TAILORING THE CODE GENERATION FORMAT

// T h i s example w i l l be made more s o p h i s t i c a t e d l a t e r , f o r now i t j u s t // m o d i f i e s t h e i n d e n t a t i o n o f n e s t e d c o d e b l o c k s ( from 2 s p a c e s / b l o c k // t o 5 s p a c e s / b l o c k ) . #i n c l u d e ” r o s e . h” #i n c l u d e ” unparseFormatHelp . h” c l a s s CustomCodeFormat : p u b l i c UnparseFormatHelp { public : CustomCodeFormat ( ) ; ˜ CustomCodeFormat ( ) ; virtual virtual

i n t g e t L i n e ( SgLocatedNode ∗ , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) ; i n t g e t C o l ( SgLocatedNode ∗ , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) ;

// r e t u r n t h e v a l u e f o r i n d e n t a t i o n o f c o d e ( p a r t o f c o n t r o l o v e r s t y l e ) v i r t u a l i n t tabIndent ( ) ; // r e t u r n t h e v a l u e f o r where l i n e wrapping s t a r t s ( p a r t o f c o n t r o l o v e r s t y l e ) v i r t u a l i n t maxLineLength ( ) ; private : int defaultLineLength ; int defaultIndentation ; };

CustomCodeFormat : : CustomCodeFormat ( ) { // d e f a u l t v a l u e s h e r e ! defaultLineLength = 20; defaultIndentation = 5; } CustomCodeFormat : : ˜ CustomCodeFormat ( ) {} // r e t u r n : > 0 : s t a r t new l i n e s ; == 0 : u s e same l i n e ; < 0 : d e f a u l t int CustomCodeFormat : : g e t L i n e ( SgLocatedNode ∗ , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) { // Use d e f a u l t mechanism t o s e l e c t t h e l i n e where t o o u t p u t g e n e r a t e d c o d e r e t u r n −1; } // r e t u r n s t a r t i n g column . i f < 0 , u s e d e f a u l t int CustomCodeFormat : : g e t C o l ( SgLocatedNode ∗ , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) { // Use d e f a u l t mechanism t o s e l e c t t h e column where t o o u t p u t g e n e r a t e d c o d e r e t u r n −1; } int CustomCodeFormat : : t a b I n d e n t ( ) { // Modify t h e i n d e n t a t i o n o f t h e g e n e r a t e d c o d e ( t r i v a l example o f return defaultIndentation ; }

t a i l o r i n g code g e n e r a t i o n )

int CustomCodeFormat : : maxLineLength ( ) { return defaultLineLength ; } i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; CustomCodeFormat∗ f o r m a t C o n t r o l = new CustomCodeFormat ( ) ; r e t u r n backend ( p r o j e c t , f o r m a t C o n t r o l ) ; }

31.3. FINAL CODE AFTER TAILORING THE CODE GENERATION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

e x t e r n i n t min ( i n t

, int

209

);

v o i d dgemm( d o u b l e ∗a , d o u b l e ∗b , d o u b l e ∗ c , i n t n ) { int var 1 ; int var 0 ; int i ; int j ; int k ; v a r 1 <= −1 + n ; v a r 1 += 1 6 ) { for ( var 1 = 0; v a r 0 <= −1 + n ; v a r 0 += 1 6 ) { for ( var 0 = 0; f o r ( i = 0 ; i <= −1 + n ; i += 1 ) { f o r ( k = v a r 1 ; k <= min(−1 + n , v a r 1 + 1 5 ) ; k += 1 ) { i n t dummy 1 = k ∗ n + i ; f o r ( j = v a r 0 ; j <= min ( n + −16 , v a r 0 ) ; j += 1 6 ) { int var 2 = ( j ); c [ j ∗ n + i ] = c [ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] } f o r ( ; j <= min(−1 + n , v a r 0 + 1 5 ) ; j += 1 ) { c [ j ∗ n + i ] = c [ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n } } } } } }

+ k]; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ;

+ k];

Figure 31.2: Example source code used as input to program to the tailor the code generation.

210

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

CHAPTER 31. TAILORING THE CODE GENERATION FORMAT

e x t e r n i n t min ( i n t

, int

);

v o i d dgemm( d o u b l e ∗a , d o u b l e ∗b , d o u b l e ∗ c , i n t n ) { int var 1 ; int var 0 ; int i ; int j ; int k ; v a r 1 <= − 1 + n ; v a r 1 += 1 6 ) { for ( var 1 = 0; v a r 0 <= − 1 + n ; v a r 0 += 1 6 ) { for ( var 0 = 0; f o r ( i = 0 ; i <= − 1 + n ; i += 1 ) { f o r ( k = v a r 1 ; k <= min(− 1 + n , v a r 1 + 1 5 ) ; k += i n t dummy 1 = k ∗ n + i ; f o r ( j = v a r 0 ; j <= min ( n + − 1 6 , v a r 0 ) ; j += 1 6 ) int var 2 = j ; c [ j ∗ n + i ] = c[ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] var 2 = 1 + var 2 ; c [ var 2 ∗ n + i ] = c [ var 2 ∗ n + i ] + a [ k ∗ n + i ] } f o r ( ; j <= min(− 1 + n , v a r 0 + 1 5 ) ; j += 1 ) { c [ j ∗ n + i ] = c[ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n } } } } } }

1) { { + k]; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ; ∗ b [ var 2 ∗ n + k ] ;

+ k];

Figure 31.3: Output of input code after changing the format of the generated code.

Chapter 32

AST Construction AST construction is a fundamental operation needed for building ROSE source-to-source translators. Several levels of interfaces are available in ROSE for users to build AST from scratch. High level interfaces are recommended to use whenever possible for their simplicity. Low level interfaces can give users the maximum freedom to manipulate some details in AST trees. This chapter uses several examples to demonstrate how to create AST fragments for common language constructs (such as variable declarations, functions, function calls, etc.) and how to insert them into an existing AST tree. More examples of constructing AST using high level interfaces can be found at rose/tests/nonsmoke/functional/roseTests/astInterfaceTests. The source files of the high level interfaces are located in rose/src/frontend/SageIII/sageInterface.

32.1

Variable Declarations

What To Learn Two examples are given to show how to construct a SAGE III AST subtree for a variable declaration and its insertion into the existing AST tree. • Example 1. Building a variable declaration using the high level AST construction and manipulation interfaces defined in namespace SageBuilder and SageInterface. Figure 32.1 shows the high level construction of an AST fragment (a variable declaration) and its insertion into the AST at the top of each block. buildVariableDeclaration() takes the name and type to build a variable declaration node. prependStatement() inserts the declaration at the top of a basic block node. Details for parent and scope pointers, symbol tables, source file position information and so on are handled transparently. • Example 2. Building the variable declaration using low level member functions of SAGE III node classes. Figure 32.2 shows the low level construction of the same AST fragment (for the same variable declaration) and its insertion into the AST at the top of each block. SgNode constructors and their member functions are used. Side effects for scope, parent pointers and symbol tables have to be handled by programmers explicitly. Figure 32.3 shows the input code used to test the translator. Figure 32.4 shows the resulting output. 211

212

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

CHAPTER 32. AST CONSTRUCTION

// S a g e B u i l d e r c o n t a i n s a l l h i g h l e v e l buildXXX ( ) f u n c t i o n s , // s u c h a s b u i l d V a r i a b l e D e c l a r a t i o n ( ) , b u i l d L a b e l S t a t e m e n t ( ) e t c . // S a g e I n t e r f a c e c o n t a i n s h i g h l e v e l AST m a n i p u l a t i o n and u t i l i t y f u n c t i o n s , // e . g . appendStatement ( ) , l o o k u p F u n c t i o n S y m b o l I n P a r e n t S c o p e s ( ) e t c . #i n c l u d e ” r o s e . h” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; c l a s s SimpleInstrumentation : public SgSimpleProcessing { public : v o i d v i s i t ( SgNode ∗ astNode ) ; }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ astNode ) { S g B a s i c B l o c k ∗ b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { SgVariableDeclaration ∗ variableDeclaration = b u i l d V a r i a b l e D e c l a r a t i o n (” newVariable ” , buildIntType prependStatement ( v a r i a b l e D e c l a r a t i o n , block ) ; } }

());

int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project ,

preorder ) ;

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.1: AST construction and insertion for a variable using the high level interfaces

32.1. VARIABLE DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

// // // // //

213

ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e a t t h e to p and bottom o f e a c h b a s i c b l o c k . Member f u n c t i o n s o f SAGE I I I AST node c l a s s e s a r e d i r e c t l y u s e d . So a l l d e t a i l s f o r S g F i l e I n f o , s c o p e , p a r e n t , symbol t a b l e s have t o be e x p l i c i t l y h a n d l e d .

#i n c l u d e ” r o s e . h” class {

SimpleInstrumentation : public : void

visit

public SgSimpleProcessing

( SgNode∗ astNode ) ;

}; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode∗ astNode ) { S g B a s i c B l o c k ∗ b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { // Mark t h i s a s a t r a n s f o r m a t i o n ( r e q u i r e d ) Sg File Info ∗ sourceLocation = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; ROSE ASSERT( s o u r c e L o c a t i o n != NULL ) ; SgType∗ t y p e = new SgTypeInt ( ) ; ROSE ASSERT( t y p e != NULL ) ; SgName name = ” n e w V a r i a b l e ” ; S g V a r i a b l e D e c l a r a t i o n ∗ v a r i a b l e D e c l a r a t i o n = new S g V a r i a b l e D e c l a r a t i o n ( s o u r c e L o c a t i o n , name , t y p e ) ; ROSE ASSERT( v a r i a b l e D e c l a r a t i o n != NULL ) ; S g I n i t i a l i z e d N a m e ∗ i n i t i a l i z e d N a m e = ∗ ( v a r i a b l e D e c l a r a t i o n −>g e t v a r i a b l e s ( ) . b e g i n ( ) ) ; i n i t i a l i z e d N a m e −> s e t f i l e i n f o ( S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) ) ; // DQ ( 6 / 1 8 / 2 0 0 7 ) : The u n p a r s e r r e q u i r e s t h a t t h e s c o p e be s e t ( f o r name q u a l i f i c a t i o n t o work ) . i n i t i a l i z e d N a m e −>s e t s c o p e ( b l o c k ) ; // L i a o ( 2 / 1 3 / 2 0 0 8 ) : A s t T e s t s r e q u i r e s t h i s t o be s e t v a r i a b l e D e c l a r a t i o n −>s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( v a r i a b l e D e c l a r a t i o n ) ; ROSE ASSERT( b l o c k −>g e t s t a t e m e n t s ( ) . s i z e ( ) > 0 ) ; b l o c k −>g e t s t a t e m e n t s ( ) . i n s e r t ( b l o c k −>g e t s t a t e m e n t s ( ) . b e g i n ( ) , v a r i a b l e D e c l a r a t i o n ) ; v a r i a b l e D e c l a r a t i o n −>s e t p a r e n t ( b l o c k ) ; // Add a symbol t o t h e s y b o l t a b l e f o r t h e new v a r i a b l e S g V a r i a b l e S y m b o l ∗ v a r i a b l e S y m b o l = new S g V a r i a b l e S y m b o l ( i n i t i a l i z e d N a m e ) ; b l o c k −>i n s e r t s y m b o l ( name , v a r i a b l e S y m b o l ) ; } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project ,

preorder

);

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.2: Example source code to read an input program and add a new variable declaration at the top of each block.

214

1 2 3 4 5 6 7 8 9

CHAPTER 32. AST CONSTRUCTION

i n t main ( ) { f o r ( i n t i =0; i < 4 ; { int x ; }

i ++)

return 0; }

Figure 32.3: Example source code used as input to the translators adding new variable.

1 2 3 4 5 6 7 8 9 10

i n t main ( ) { i n t newVariable ; for ( int i = 0; i < 4; i n t newVariable ; int x ; } return 0; }

i ++) {

Figure 32.4: Output of input to the translators adding new variable.

32.2. EXPRESSIONS

32.2

215

Expressions

Figure 32.5 shows a translator using the high level AST builder interface to add an assignment statement right before the last statement in a main() function. Figure 32.6 shows the input code used to test the translator. Figure 32.7 shows the resulting output. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

// E x p r e s s i o n s can be b u i l t u s i n g both bottomup ( recommended ) and topdown o r d e r s . // Bottomup : b u i l d o p e r a n d s f i r s t , o p e r a t i o n l a t e r // Topdown : b u i l d o p e r a t i o n f i r s t , s e t o p e r a n d s l a t e r on . #i n c l u d e ” r o s e . h” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // go t o t h e f u n c t i o n body S g F u n c t i o n D e c l a r a t i o n ∗ mainFunc= f i n d M a i n ( p r o j e c t ) ; S g B a s i c B l o c k ∗ body= mainFunc−>g e t d e f i n i t i o n ()−> g e t b o d y ( ) ; p u s h S c o p e S t a c k ( body ) ; // bottomup : b u i l d o p e r a n d s f i r s t , c r e a t e e x p r e s s i o n l a t e r on // d o u b l e r e s u l t = 2 ∗ ( 1 − gama ∗ gama ) ; SgExpression ∗ i n i t e x p = buildMultiplyOp ( buildDoubleVal ( 2 . 0 ) , buildSubtractOp ( buildDoubleVal ( 1 . 0 ) , b u i l d M u l t i p l y O p ( buildVarRefExp ( ” gama ” ) , buildVarRefExp ( ” gama ” ) ))); S g V a r i a b l e D e c l a r a t i o n ∗ d e c l = b u i l d V a r i a b l e D e c l a r a t i o n (” r e s u l t ” , buildDoubleType ( ) , b u i l d A s s i g n I n i t i a l i z e r ( i n i t e x p ) ) ; S gStatement ∗ l a s t s t m t = g e t L a s t S t a t e m e n t ( t o p S c o p e S t a c k ( ) ) ; insertStatementBefore ( laststmt , decl ) ; // topdown : b u i l d e x p r e s s i o n f i r s t , s e t o p e r a n d s l a t e r on // d o u b l e r e s u l t 2 = a l p h a ∗ b e t a ; SgExpression ∗ i n i t e x p 2 = buildMultiplyOp ( ) ; se tL hsOperand ( i n i t e x p 2 , buildVarRefExp ( ” a l p h a ” ) ) ; setRhsOperand ( i n i t e x p 2 , buildVarRefExp ( ” b e t a ” ) ) ; S g V a r i a b l e D e c l a r a t i o n ∗ d e c l 2 = b u i l d V a r i a b l e D e c l a r a t i o n (” r e s u l t 2 ” , buildDoubleType ( ) , b u i l d A s s i g n I n i t i a l i z e r ( i n i t e x p 2 ) ) ; l a s t s t m t = getLastStatement ( topScopeStack ( ) ) ; insertStatementBefore ( laststmt , decl2 ) ; popScopeStack ( ) ; AstTests : : runAllTests ( p r o j e c t ) ; // i n v o k e backend c o m p i l e r t o g e n e r a t e o b j e c t / b i n a r y r e t u r n backend ( p r o j e c t ) ;

files

}

Figure 32.5: Example translator to add expressions

216

1 2 3 4 5 6 7 8

CHAPTER 32. AST CONSTRUCTION

i n t main ( ) { d o u b l e a l p h a= 0 . 5 ; double beta = 0 . 1 ; d o u b l e gama = 0 . 7 ; return 0; }

Figure 32.6: Example source code used as input

1 2 3 4 5 6 7 8 9 10

i n t main ( ) { double alpha = 0 . 5 ; double beta = 0 . 1 ; d o u b l e gama = 0 . 7 ; d o u b l e r e s u l t = 2 . 0 0 0 0 0 ∗ ( 1 . 0 0 0 0 0 − gama ∗ gama ) ; double r e s u l t 2 = alpha ∗ beta ; return 0; }

Figure 32.7: Output of the input

32.3. ASSIGNMENT STATEMENTS

32.3

217

Assignment Statements

Figure 32.8 shows a translator using the high level AST builder interface to add an assignment statement right before the last statement in a main() function. Figure 32.9 shows the input code used to test the translator. Figure 32.10 shows the resulting output. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

// S a g e B u i l d e r c o n t a i n s a l l h i g h l e v e l buildXXX ( ) f u n c t i o n s , // s u c h a s b u i l d V a r i a b l e D e c l a r a t i o n ( ) , b u i l d L a b e l S t a t e m e n t ( ) e t c . // S a g e I n t e r f a c e c o n t a i n s h i g h l e v e l AST m a n i p u l a t i o n and u t i l i t y f u n c t i o n s , // e . g . appendStatement ( ) , l o o k u p F u n c t i o n S y m b o l I n P a r e n t S c o p e s ( ) e t c . #i n c l u d e ” r o s e . h” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // go t o t h e f u n c t i o n body o f main ( ) // and push i t t o t h e s c o p e s t a c k S g F u n c t i o n D e c l a r a t i o n ∗ mainFunc= f i n d M a i n ( p r o j e c t ) ; S g B a s i c B l o c k ∗ body= mainFunc−>g e t d e f i n i t i o n ()−> g e t b o d y ( ) ; p u s h S c o p e S t a c k ( body ) ; // b u i l d a v a r i a b l e a s s i g n m e n t s t a t e m e n t : i =9; // buildVarRefExp ( s t r i n g varName ) w i l l a u t o m a t i c a l l y s e a r c h f o r a matching v a r i a b l e symbol s t a r t i n g // from t h e c u r r e n t s c o p e t o t h e g l o b a l s c o p e . SgExprStatement ∗ a s s i g n S t m t = b u i l d A s s i g n S t a t e m e n t ( buildVarRefExp ( ” i ” ) , b u i l d I n t V a l ( 9 ) ) ; // i n s e r t i t b e f o r e t h e l a s t r e t u r n s t a t e m e n t S gStatement ∗ l a s t S t m t = g e t L a s t S t a t e m e n t ( t o p S c o p e S t a c k ( ) ) ; insertStatementBefore ( lastStmt , assignStmt ) ; popScopeStack ( ) ; // A s t T e s t s e n s u r e s t h e r e i s no d a n g l i n g SgVarRefExp w i t h o u t a mathing symbol AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.8: Example source code to add an assignment statement

1 2 3 4 5

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { int i ; return 0; }

Figure 32.9: Example source code used as input

218

1 2 3 4 5 6 7

CHAPTER 32. AST CONSTRUCTION

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { int i ; i = 9; return 0; }

Figure 32.10: Output of the input

32.4. FUNCTIONS

32.4

219

Functions

This section shows how to add a function at the top of a global scope in a file. Again, examples for both high level and low level constructions of AST are given. • Figure 32.11 shows the high level construction of a defining function (a function with a function body). Scope information is passed to builder functions explicitly when it is needed. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

// T h i s e x a m p l e sh ow s how t o c o n s t r u c t a d e f i n i n g // u s i n g h i g h l e v e l AST c o n s t r u c t i o n i n t e r f a c e s . // #i n c l u d e ” r o s e . h ” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; class {

SimpleInstrumentation public : void

visit

(

:

public

function

( with

a

function

body )

SgSimpleProcessing

SgNode ∗ a s t N o d e

);

}; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ a s t N o d e ) { SgGlobal ∗ g l o b a l S c o p e = i s S g G l o b a l ( astNode ) ; i f ( g l o b a l S c o p e != NULL) { // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // C r e a t e a p a r a m e t e r l i s t w i t h a p a r a m e t e r // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgName v a r 1 n a m e = ” v a r n a m e ” ; SgReferenceType ∗ r e f t y p e = buildReferenceType ( buildIntType ( ) ) ; S g I n i t i a l i z e d N a m e ∗ v a r 1 i n i t n a m e = b u i l d I n i t i a l i z e d N a m e ( var1 name , r e f t y p e ) ; SgFunctionParameterList∗ parameterList = buildFunctionParameterList ( ) ; appendArg ( p a r a m e t e r L i s t , v a r 1 i n i t n a m e ) ; // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // C r e a t e a d e f i n i n g f u n c t i o n D e c l a r a t i o n ( w i t h a f u n c t i o n body ) // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgName f u n c n a m e = ” my function ”; SgFunctionDeclaration ∗ func = buildDefiningFunctionDeclaration ( func name , b u i l d I n t T y p e ( ) , p a r a m e t e r L i s t , g l o b a l S c o p e ) ; = f u n c −>g e t d e f i n i t i o n ()−> g e t b o d y ( ) ; SgBasicBlock∗ func body // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // I n s e r t a s t a t e m e n t i n t h e f u n c t i o n body // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgVarRefExp ∗ v a r r e f = b u i l d V a r R e f E x p ( v a r 1 n a m e , f u n c b o d y ) ; SgPlusPlusOp ∗ p p e x p r e s s i o n = b u i l d P l u s P l u s O p ( v a r r e f ) ; SgExprStatement ∗ new stmt = buildExprStatement ( p p e x p r e s s i o n ) ; //

i n s e r t a s t a t e m e n t i n t o t h e f u n c t i o n body p r e p e n d S t a t e m e n t ( n ew s tm t , f u n c b o d y ) ; prependStatement ( func , g l o b a l S c o p e ) ;

} } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

See

Rose : : i n i t i a l i z e

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project ,

preorder

);

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.11: Addition of function to global scope using high level interfaces

220

CHAPTER 32. AST CONSTRUCTION • Figure 32.12 shows almost the same high level construction of the defining function, but with an additional scope stack. Scope information is passed to builder functions implicitly when it is needed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

// T h i s e x a m p l e s h ow s how t o c o n s t r u c t a d e f i n i n g f u n c t i o n ( w i t h // u s i n g h i g h l e v e l AST c o n s t r u c t i o n i n t e r f a c e s . // A s c o p e s t a c k i s u s e d t o p a s s s c o p e i n f o r m a t i o n i m p l i c i t l y t o #i n c l u d e ” r o s e . h ” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

See

function

body )

builder

functions

Rose : : i n i t i a l i z e

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SgGlobal ∗ globalScope = g e t F i r s t G l o b a l S c o p e // push g l o b a l s c o p e i n t o s t a c k pushScopeStack ( isSgScopeStatement

a

some

( project );

( globalScope ) ) ;

//

Create a parameter l i s t with a parameter SgName v a r 1 n a m e = ” v a r n a m e ” ; SgReferenceType ∗ r e f t y p e = buildReferenceType ( buildIntType ( ) ) ; S g I n i t i a l i z e d N a m e ∗ v a r 1 i n i t n a m e = b u i l d I n i t i a l i z e d N a m e ( var1 name , r e f t y p e ) ; SgFunctionParameterList∗ parameterList = buildFunctionParameterList ( ) ; appendArg ( p a r a m e t e r L i s t , v a r 1 i n i t n a m e ) ;

//

C r e a t e a d e f i n i n g f u n c t i o n D e c l a r a t i o n ( w i t h a f u n c t i o n body ) SgName f u n c n a m e = ” my function ”; SgFunctionDeclaration ∗ func = buildDefiningFunctionDeclaration ( func name , b u i l d I n t T y p e ( ) , p a r a m e t e r L i s t ) ; SgBasicBlock∗ func body = f u n c −>g e t d e f i n i t i o n ()−> g e t b o d y ( ) ;

//

push f u n c t i o n body s c o p e i n t o s t a c k pushScopeStack ( isSgScopeStatement ( func body ) ) ;

//

b u i l d a s t a t e m e n t i n t h e f u n c t i o n body SgVarRefExp ∗ v a r r e f = b u i l d V a r R e f E x p ( v a r 1 n a m e ) ; SgPlusPlusOp ∗ p p e x p r e s s i o n = b u i l d P l u s P l u s O p ( v a r r e f ) ; SgExprStatement ∗ new stmt = buildExprStatement ( p p e x p r e s s i o n ) ;

//

i n s e r t a statement int o the f u n c t i on appendStatement ( new stmt ) ; pop f u n c t i o n body o f f t h e s t a c k popScopeStack ( ) ;

body

i n s e r t the f u n c ti o n d e c l a r a t i o n prependStatement ( func ) ; popScopeStack ( ) ;

the

//

//

into

scope

at

the

top

of

the

scope

stack

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.12: Addition of function to global scope using high level interfaces and a scope stack • The low level construction of the AST fragment of the same function declaration and its insertion is separated into two portions and shown in two figures (Figure 32.13 and Figure 32.14 ). Figure 32.27 and Figure 32.28 give the input code and output result for the translators above.

32.4. FUNCTIONS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

221

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t s ho ws t h e d e s i g n o f a t r a n s f o r m a t i o n // a t t h e t o p o f t h e s o u r c e f i l e . #i n c l u d e

is to

an e x a m p l e instrument

preprocessor s o u r c e code ,

b u i l t w i t h ROSE . p l a c i n g s o u r c e code

” r o s e . h”

#d e f i n e TRANSFORMATION FILE INFO S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) class {

SimpleInstrumentation public : void

visit

(

:

public

SgSimpleProcessing

SgNode ∗ a s t N o d e

);

}; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ a s t N o d e ) { SgGlobal ∗ g l o b a l S c o p e = i s S g G l o b a l ( astNode ) ; i f ( g l o b a l S c o p e != NULL) { // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // C r e a t e t h e f u n c t i o n D e c l a r a t i o n // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgType ∗ f u n c r e t u r n t y p e = new S g T y p e I n t ( ) ; SgName f u n c n a m e = ” my function ”; SgFunctionType ∗ f u n c t y p e = new S g F u n c t i o n T y p e ( f u n c r e t u r n t y p e , f a l s e ) ; SgFunctionDeclaration ∗ func = new S g F u n c t i o n D e c l a r a t i o n ( TRANSFORMATION FILE INFO , f u n c n a m e , SgFunctionDefinition ∗ func def = new S g F u n c t i o n D e f i n i t i o n ( TRANSFORMATION FILE INFO , f u n c ) ; SgBasicBlock ∗ func body = new S g B a s i c B l o c k (TRANSFORMATION FILE INFO ) ; // //

s e t t h e end s o u r c e p o s i t i o n a s t r a n s f o r m a t i o n g e n e r a t e d s i n c e the c o n s t r u c t o r s only s e t the beginning source p o s i t i o n f u n c −>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; f u n c −>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( f u n c ) ;

by

func type ) ;

default

f u n c d e f −>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; f u n c d e f −>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( f u n c d e f ) ; f u n c b o d y −>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; f u n c b o d y −>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( f u n c b o d y ) ; // //

Sets func Sets func

t h e body i n t o t h e d e f i n i t i o n d e f −>s e t b o d y ( f u n c b o d y ) ; the d e f i n t i o n ’ s parent to the d e f −>s e t p a r e n t ( f u n c ) ;

declaration

// DQ ( 9 / 8 / 2 0 0 7 ) : F i x u p t h e d e f i n i n g and non−d e f i n i n g d e c l a r a t i o n s ROSE ASSERT( f u n c −>g e t d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; f u n c −>s e t d e f i n i n g D e c l a r a t i o n ( f u n c ) ; ROSE ASSERT( f u n c −>g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; ROSE ASSERT( f u n c −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != f u n c ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : We h a v e n o t b u i l d a non−d e f i n i n g d e c l a r a t i o n , s o ROSE ASSERT( f u n c −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ;

this

should

be NULL .

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add f u n c t i o n s y m b o l t o g l o b a l s c o p e ! // p r i n t f ( ” F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r f u n c t i o n = %p = %s \n ” , g l o b a l S c o p e , g l o b a l S c o p e −>c l a s s n a m e ( ) . c s t r ( ) , f u S g F u n c t i o n S y m b o l ∗ f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( f u n c ) ; g l o b a l S c o p e −>i n s e r t s y m b o l ( f u n c −>g e t n a m e ( ) , f u n c t i o n S y m b o l ) ; ROSE ASSERT( g l o b a l S c o p e −>l o o k u p f u n c t i o n s y m b o l ( f u n c −>g e t n a m e ( ) ) != NULL ) ; // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // C r e a t e t h e I n i t i a l i z e d N a m e f o r a p a r a m e t e r w i t h i n t h e p a r a m e t e r l i s t // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgName v a r 1 n a m e = ” v a r n a m e ” ; SgTypeInt ∗ v a r 1 t y p e = new S g T y p e I n t ( ) ; S g R e f e r e n c e T y p e ∗ r e f t y p e = new S g R e f e r e n c e T y p e ( v a r 1 t y p e ) ; S g I n i t i a l i z e r ∗ v a r 1 i n i t i a l i z e r = NULL ; S g I n i t i a l i z e d N a m e ∗ v a r 1 i n i t n a m e = new S g I n i t i a l i z e d N a m e ( v a r 1 n a m e , v a r 1 i n i t n a m e −> s e t f i l e i n f o (TRANSFORMATION FILE INFO ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : We now t e s t t h i s , s o i t v a r 1 i n i t n a m e −>s e t s c o p e ( f u n c d e f ) ;

has

to

be

set

ref type ,

v a r 1 i n i t i a l i z e r , NULL ) ;

explicitly .

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add v a r i a b l e s y m b o l t o g l o b a l s c o p e ! // p r i n t f ( ” F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r S g I n i t i a l i z e d N a m e = %p = %s \n ” , g l o b a l S c o p e , g l o b a l S c o p e −>c l a s s n a m e ( ) . c S g V a r i a b l e S y m b o l ∗ v a r s y m b o l = new S g V a r i a b l e S y m b o l ( v a r 1 i n i t n a m e ) ; f u n c d e f −>i n s e r t s y m b o l ( v a r 1 i n i t n a m e −>g e t n a m e ( ) , v a r s y m b o l ) ;

Figure 32.13: Example source code shows addition of function to global scope (part 1).

222 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

CHAPTER 32. AST CONSTRUCTION ROSE ASSERT( f u n c d e f −>l o o k u p v a r i a b l e s y m b o l ( v a r 1 i n i t n a m e −>g e t n a m e ( ) ) ROSE ASSERT( v a r 1 i n i t n a m e −>g e t s y m b o l f r o m s y m b o l t a b l e ( ) != NULL ) ; // Done

constructing

the

InitializedName

!= NULL ) ;

variable

//

I n s e r t argument i n f u n c t i o n p a r a m e t e r l i s t ROSE ASSERT( f u n c != NULL ) ; // S g F i l e I n f o ∗ p a r a m e t e r L i s t F i l e I n f o = new S g F i l e I n f o ( ) ; // S g F i l e I n f o ∗ p a r a m e t e r L i s t F i l e I n f o = S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) ; S g F u n c t i o n P a r a m e t e r L i s t ∗ p a r a m e t e r L i s t = new S g F u n c t i o n P a r a m e t e r L i s t (TRANSFORMATION FILE INFO ) ; ROSE ASSERT( p a r a m e t e r L i s t != NULL ) ; p a r a m e t e r L i s t −>s e t d e f i n i n g D e c l a r a t i o n (NULL ) ; p a r a m e t e r L i s t −>s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( p a r a m e t e r L i s t ) ; f u n c −>s e t p a r a m e t e r L i s t ( p a r a m e t e r L i s t ) ; ROSE ASSERT( f u n c −>g e t p a r a m e t e r L i s t ( ) != NULL ) ; f u n c −>g e t p a r a m e t e r L i s t ()−> a p p e n d a r g ( v a r 1 i n i t n a m e ) ; // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // I n s e r t a s t a t e m e n t i n t h e f u n c t i o n body // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗

#i f

// //

c r e a t e a VarRefExp S g V a r i a b l e S y m b o l ∗ v a r s y m b o l = new S g V a r i a b l e S y m b o l ( v a r 1 i n i t n a m e ) ; SgVarRefExp ∗ v a r r e f = new SgVarRefExp ( TRANSFORMATION FILE INFO , v a r s y m b o l ) ; v a r r e f −>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; v a r r e f −>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( v a r r e f ) ;

//

c r e a t e a ++ e x p r e s s i o n , 0 f o r p r e f i x ++ S g P l u s P l u s O p ∗ p p e x p r e s s i o n = new S g P l u s P l u s O p ( TRANSFORMATION FILE INFO , v a r r e f , 0 ) ; p p e x p r e s s i o n −>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; p p e x p r e s s i o n −>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( p p e x p r e s s i o n ) ; markLhsValues ( p p e x p r e s s i o n ) ;

//

c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t ∗ n e w s t m t = new S g E x p r S t a t e m e n t ( TRANSFORMATION FILE INFO , p p e x p r e s s i o n ) ; n ew st mt−>s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; n ew st mt−>g e t e n d O f C o n s t r u c t ()−> s e t p a r e n t ( n e w s t m t ) ;

0 // DQ ( 9 / 8 / 2 0 0 7 ) : T h i s i s no l o n g e r r e q u i r e d , // c r e a t e an e x p r e s s i o n t y p e S g T y p e I n t ∗ e x p r t y p e = new S g T y p e I n t ( ) ; //

SgExpressionRoot

is

not

longer

used

in

t h e ROSE IR .

c r e a t e an e x p r e s s i o n r o o t S g E x p r e s s i o n R o o t ∗ e x p r r o o t = new S g E x p r e s s i o n R o o t ( TRANSFORMATION FILE INFO , p p e x p r e s s i o n , e x p r t y p e ) ; e x p r r o o t −>s e t p a r e n t ( n e w s t m t ) ;

// DQ ( 1 1 / 8 / 2 0 0 6 ) : M o d i f i e d t o r e f l e c t u s e n ew st mt−>s e t e x p r e s s i o n ( e x p r r o o t ) ;

of

SgExpression

instead

of

SgExpressionRoot

p p e x p r e s s i o n −>s e t p a r e n t ( n ew st mt−>g e t e x p r e s s i o n ( ) ) ; #e n d i f p p e x p r e s s i o n −>s e t p a r e n t ( n e w s t m t ) ; //

i n s e r t a s t a t e m e n t i n t o t h e f u n c t i o n body f u n c b o d y −>p r e p e n d s t a t e m e n t ( n e w s t m t ) ;

//

s e t t i n g the parent e x p l i c i t l y i s f u n c −>s e t p a r e n t ( g l o b a l S c o p e ) ;

// //

s c o p e s o f s t a t m e n t s must be s e t e x p l i c i t l y s i n c e w i t h i n C++ t h e y a r e n o t t o be t h e same a s t h a t i n d i c a t e d by t h e p a r e n t ( s e e ChangeLog f o r S p r i n g f u n c −>s e t s c o p e ( g l o b a l S c o p e ) ;

not

required

since

it

would be

done

w i t h i n AST p o s t −p r o c e s s i n g

guaranteed 2005).

// DQ ( 6 / 1 5 / 2 0 1 3 ) : T h e r e s h o u l d be a non−d e f i n i n g d e c l a r a t i o n ( b u t we n e e d t o // ROSE ASSERT( f u n c −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; i f ( f u n c −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL) { p r i n t f ( ”WARNING: f u n c −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL f o r }

migrate

this

case

f u n c = %p = %s

of

example

// ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // I n s e r t t h e f u n c t i o n d e c l a r a t i o n i n t h e c o d e // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ g l o b a l S c o p e −>p r e p e n d d e c l a r a t i o n ( f u n c ) ; // //

R e q u i r e d p o s t p r o c e s s i n g o f AST r e q u i r e d temporaryAstFixes ( globalScope ) ; AstPostProcessing ( globalScope ) ;

to

set

parent

pointers

and

fixup

template

names ,

Figure 32.14: Example source code shows addition of function to global scope (part 2).

etc .

to

instead

use

t h e AST

( allowed

for

tutorial

32.4. FUNCTIONS

1 2 3 4 5 6 7 8 9 10

i n t main ( ) { f o r ( i n t i =0; i < 4 ; { int x ; }

223

i ++)

return 0; }

Figure 32.15: Example source code used as input to translator adding new function.

1 2 3 4 5 6 7 8 9 10 11 12 13

i n t m y f u n c t i o n ( i n t &var name ) { ++var name ; } i n t main ( ) { for ( int i = 0; int x ; } return 0; }

i < 4;

i ++) {

Figure 32.16: Output of input to translator adding new function.

: TODO: This tutorial vel AST manipulation. d have a more concise sing SageInterface and SageBuilder functions.

224

CHAPTER 32. AST CONSTRUCTION

32.5

Function Calls

Adding functions calls is a typical task for instrumentation translator. • Figure 32.17 shows the use of the AST string based rewrite mechanism to add function calls to the top and bottom of each block within the AST. • Figure 32.18 shows the use of the AST builder interface to do the same instrumentation work. Another example shows how to add a function call at the end of each function body. A utility function, instrumentEndOfFunction(), from SageInterface name space is used. The interface tries to locate all return statements of a target function and rewriting return expressions with side effects, if there are any. Figure 32.19 shows the translator code. Figure 32.20 shows the input code. The instrumented code is shown in Figure 32.21.

32.6

Creating a ’struct’ for Global Variables

This is an example written to support the Charm++ tool. This translator extracts global variables from the program and builds a structure to hold them. The support is part of a number of requirements associated with using Charm++ and AMPI. Figure 32.22 shows repackaging of global variables within an application into a struct. All reference to the global variables are also transformed to reference the original variable indirectly through the structure. This processing is part of preprocessing to use Charm++. This example shows the low level handling directly at the level of the IR.

32.6. CREATING A ’STRUCT’ FOR GLOBAL VARIABLES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54

225

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e // a t t h e to p and bottome o f e a c h b a s i c b l o c k . #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; class {

SimpleInstrumentation : public : void

visit

public SgSimpleProcessing

( SgNode∗ astNode ) ;

}; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode∗ astNode ) { S g B a s i c B l o c k ∗ b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { c o n s t u n s i g n e d i n t SIZE OF BLOCK = 1 ; i f ( b l o c k −>g e t s t a t e m e n t s ( ) . s i z e ( ) > SIZE OF BLOCK ) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e s t r i n g codeAtTopOfBlock = ” v o i d myTimerFunctionStart ( ) ; myTimerFunctionStart ( ) ; ” ; s t r i n g codeAtBottomOfBlock = ” v o i d myTimerFunctionEnd ( ) ; myTimerFunctionEnd ( ) ; ” ; // I n s e r t new c o d e i n t o t h e s c o p e r e p r e s e n t e d by t h e s t a t e m e n t ( a p p l i e s t o S g S c o p e S t a t e m e n t s ) MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ; // I n s e r t t h e new c o d e a t t h e t o p and bottom o f t h e s c o p e r e p r e s e n t e d by b l o c k M i d d l e L e v e l R e w r i t e : : i n s e r t ( b l o c k , codeAtTopOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : TopOfCurrentScope ) ; M i d d l e L e v e l R e w r i t e : : i n s e r t ( b l o c k , codeAtBottomOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : BottomOfCurrentScope ) ; } } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project ,

preorder

);

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.17: Example source code to instrument any input program.

226

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

CHAPTER 32. AST CONSTRUCTION

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e // a t t h e t o p and bottom o f e a c h b a s i c b l o c k . #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; u s i n g namespace S a g e I n t e r f a c e ; u s i n g namespace S a g e B u i l d e r ; c l a s s SimpleInstrumentation : public SgSimpleProcessing { public : v o i d v i s i t ( SgNode ∗ astNode ) ; }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ astNode ) { S g B a s i c B l o c k ∗ b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { c o n s t u n s i g n e d i n t SIZE OF BLOCK = 1 ; i f ( b l o c k −>g e t s t a t e m e n t s ( ) . s i z e ( ) > SIZE OF BLOCK ) { SgName name1 ( ” myTimerFunctionStart ” ) ; // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e SgFunctionDeclaration ∗ decl 1 = buildNondefiningFunctionDeclaration ( name1 , b u i l d V o i d T y p e ( ) , b u i l d F u n c t i o n P a r a m e t e r L i s t ( ) , b l o c k ) ; ( ( d e c l 1 −>g e t d e c l a r a t i o n M o d i f i e r ( ) ) . g e t s t o r a g e M o d i f i e r ( ) ) . s e t E x t e r n ( ) ; SgExprStatement ∗ c a l l S t m t 1 = b u i l d F u n c t i o n C a l l S t m t ( name1 , b u i l d V o i d T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , b l o c k ) ; prependStatement ( callStmt 1 , block ) ; prependStatement ( d e c l 1 , block ) ; SgName name2 ( ” myTimerFunctionEnd ” ) ; // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e SgFunctionDeclaration ∗ decl 2 = buildNondefiningFunctionDeclaration ( name2 , b u i l d V o i d T y p e ( ) , b u i l d F u n c t i o n P a r a m e t e r L i s t ( ) , b l o c k ) ; ( ( d e c l 2 −>g e t d e c l a r a t i o n M o d i f i e r ( ) ) . g e t s t o r a g e M o d i f i e r ( ) ) . s e t E x t e r n ( ) ; SgExprStatement ∗ c a l l S t m t 2 = b u i l d F u n c t i o n C a l l S t m t ( name2 , b u i l d V o i d T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , b l o c k ) ; appendStatement ( d e c l 2 , b l o c k ) ; appendStatement ( c a l l S t m t 2 , b l o c k ) ; } } } int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project ,

preorder ) ;

AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.18: Example source code using the high level interfaces

32.6. CREATING A ’STRUCT’ FOR GLOBAL VARIABLES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

/∗! \ b r i e f test instrumentation right ∗/ #i n c l u d e ” r o s e . h” #i n c l u d e u s i n g namespace S a g e I n t e r f a c e ; u s i n g namespace S a g e B u i l d e r ;

b e f o r e t h e end o f a f u n c t i o n

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // Find a l l f u n c t i o n d e f i n i t i o n s we want t o i n s t r u m e n t s t d : : v e c t o r f u n c D e f L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; s t d : : v e c t o r : : i t e r a t o r i t e r ; f o r ( i t e r = f u n c D e f L i s t . b e g i n ( ) ; i t e r != f u n c D e f L i s t . end ( ) ; i t e r ++) { SgFunctionDefinition ∗ c u r d e f = i s S g F u n c t i o n D e f i n i t i o n (∗ i t e r ) ; ROSE ASSERT( c u r d e f ) ; S g B a s i c B l o c k ∗ body = c u r d e f −>g e t b o d y ( ) ; // B u i l d t h e c a l l s t a t e m e n t f o r e a c h p l a c e SgExprStatement ∗ c a l l S t m t 1 = b u i l d F u n c t i o n C a l l S t m t ( ” c a l l 1 ” , b u i l d I n t T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , body ) ; // i n s t r u m e n t t h e f u n c t i o n i n t i= i n s t r u m e n t E n d O f F u n c t i o n ( c u r d e f −>g e t d e c l a r a t i o n ( ) , c a l l S t m t 1 ) ; s t d : : cout <<”I n s t r u m e n t e d ”<
// end o f i n s t r u m e n t a t i o n AstTests : : runAllTests ( p r o j e c t ) ; // t r a n s l a t i o n o n l y p r o j e c t −>u n p a r s e ( ) ; return 0;

}

Figure 32.19: Example source code instrumenting end of functions

227

228

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

CHAPTER 32. AST CONSTRUCTION

/∗ Example c o d e : ∗ a f u n c t i o n with m u l t i p l e r e t u r n s ∗ some r e t u r n s have e x p r e s s i o n s w i t h s i d e ∗ a f u n c t i o n w i t h o u t any r e t u r n ∗/ extern int foo ( ) ; extern int c a l l 1 ( ) ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { i f ( a r g c >1) return foo ( ) ; else return foo ( ) ; return 0; }

effects

v o i d b ar ( ) { int i ; }

Figure 32.20: Example input code of the instrumenting translator for end of functions.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

/∗ Example c o d e : ∗ a f u n c t i o n with m u l t i p l e r e t u r n s ∗ some r e t u r n s have e x p r e s s i o n s w i t h s i d e ∗ a f u n c t i o n w i t h o u t any r e t u r n ∗/ extern int foo ( ) ; extern int c a l l 1 ( ) ;

effects

i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { i f ( argc > 1) { int rose temp 1 = foo ( ) ; call1 (); return rose temp 1 ; } else { int rose temp 2 = foo ( ) ; call1 (); return rose temp 2 ; } call1 (); return 0; } v o i d b ar ( ) { int i ; call1 (); }

Figure 32.21: Output of instrumenting translator for end of functions.

32.6. CREATING A ’STRUCT’ FOR GLOBAL VARIABLES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t s ho ws t h e d e s i g n o f a t r a n s f o r m a t i o n #i n c l u d e using

is to

229

an e x a m p l e p r e p r o c e s s o r b u i l t w i t h ROSE . do a t r a n s f o r m a t i o n s p e c i f i c f o r Charm++.

” r o s e . h”

namespace

std ;

R o s e S T L C o n t a i n e r buildListOfGlobalVariables ( SgSourceFile∗ { // T h i s f u n c t i o n b u i l d s a l i s t o f g l o b a l a s s e r t ( f i l e != NULL ) ;

file

)

variables

( from a

SgFile ) .

R o s e S T L C o n t a i n e r g l o b a l V a r i a b l e L i s t ; S g G l o b a l ∗ g l o b a l S c o p e = f i l e −>g e t g l o b a l S c o p e ( ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r i = g l o b a l S c o p e −>g e t d e c l a r a t i o n s ( ) . b e g i n ( ) ; w h i l e ( i != g l o b a l S c o p e −>g e t d e c l a r a t i o n s ( ) . end ( ) ) { S g V a r i a b l e D e c l a r a t i o n ∗ v a r i a b l e D e c l a r a t i o n = i s S g V a r i a b l e D e c l a r a t i o n (∗ i ) ; i f ( v a r i a b l e D e c l a r a t i o n != NULL) { R o s e S T L C o n t a i n e r & v a r i a b l e L i s t = v a r i a b l e D e c l a r a t i o n −>g e t v a r i a b l e s ( ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r v a r = v a r i a b l e L i s t . b e g i n ( ) ; w h i l e ( v a r != v a r i a b l e L i s t . end ( ) ) { g l o b a l V a r i a b l e L i s t . push back (∗ var ) ; v a r ++; } } i ++; } return

globalVariableList ;

} // T h i s f u n c t i o n i s n o t u s e d , b u t i s u s e f u l f o r // g e n e r a t i n g t h e l i s t o f a l l g l o b a l v a r i a b l e s R o s e S T L C o n t a i n e r buildListOfGlobalVariables ( SgProject∗ project ) { // T h i s f u n c t i o n b u i l d s a l i s t o f g l o b a l v a r i a b l e s

( from a

SgProject ) .

R o s e S T L C o n t a i n e r g l o b a l V a r i a b l e L i s t ; c o n s t S g F i l e P t r L i s t& f i l e L i s t = p r o j e c t −> g e t f i l e L i s t ( ) ; SgFilePtrList : : c o n s t i t e r a t o r f i l e = f i l e L i s t . begin ( ) ; // Loop o v e r t h e // when m u l t i p l e w h i l e ( f i l e != { Rose STL

f i l e s in the p r o j e c t ( multiple f i l e s e x i s t s o u r c e f i l e s a r e p l a c e d on t h e command l i n e ) . f i l e L i s t . end ( ) ) C o n t a i n e r f i l e G l o b a l V a r i a b l e L i s t = b u i l d L i s t O f G l o b a l V a r i a b l e s ( i s S g S o u r c e F i l e ( ∗ f i l e ) ) ;

// DQ ( 9 / 2 6 / 2 0 0 7 ) : Moved f r o m s t d : : l i s t t o s t d : : v e c t o r // g l o b a l V a r i a b l e L i s t . merge ( f i l e G l o b a l V a r i a b l e L i s t ) ; g l o b a l V a r i a b l e L i s t . i n s e r t ( g l o b a l V a r i a b l e L i s t . b e g i n ( ) , f i l e G l o b a l V a r i a b l e L i s t . b e g i n ( ) , f i l e G l o b a l V a r i a b l e L i s t . end ( ) ) ; f i l e ++; } return

globalVariableList ;

} R o s e S T L C o n t a i n e r b u i l d L i s t O f V a r i a b l e R e f e r e n c e E x p r e s s i o n s U s i n g G l o b a l V a r i a b l e s ( SgNode ∗ node ) { // T h i s f u n c t i o n b u i l d s a l i s t o f ” u s e s ” o f v a r i a b l e s ( SgVarRefExp IR n o d e s ) //

return variable R o s e S T L C o n t a i n e r g l o b a l V a r i a b l e U s e L i s t ;

//

l i s t o f a l l v a r i a b l e s ( t h e n s e l e c t o u t t h e g l o b a l v a r i a b l e s by R o s e S T L C o n t a i n e r n o d e L i s t = NodeQuery : : q u e r y S u b T r e e

testing ( node ,

within

t h e AST .

the scope ) V SgVarRefExp

);

R o s e S T L C o n t a i n e r : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { SgVarRefExp ∗ v a r i a b l e R e f e r e n c e E x p r e s s i o n = i s S g V a r R e f E x p ( ∗ i ) ;

Figure 32.22: Example source code shows repackaging of global variables to a struct (part 1).

230

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

CHAPTER 32. AST CONSTRUCTION

assert ( variableReferenceExpression

!= NULL ) ;

a s s e r t ( v a r i a b l e R e f e r e n c e E x p r e s s i o n −>g e t s y m b o l ( ) != NULL ) ; a s s e r t ( v a r i a b l e R e f e r e n c e E x p r e s s i o n −>g e t s y m b o l ()−> g e t d e c l a r a t i o n ( ) != NULL ) ; a s s e r t ( v a r i a b l e R e f e r e n c e E x p r e s s i o n −>g e t s y m b o l ()−> g e t d e c l a r a t i o n ()−> g e t s c o p e ( )

!= NULL ) ;

// Note t h a t v a r i a b l e R e f e r e n c e E x p r e s s i o n −>g e t s y m b o l ()−> g e t d e c l a r a t i o n ( ) r e t u r n s t h e // S g I n i t i a l i z e d N a m e ( n o t t h e S g V a r i a b l e D e c l a r a t i o n w h e r e i t was d e c l a r e d ) ! S g I n i t i a l i z e d N a m e ∗ v a r i a b l e = v a r i a b l e R e f e r e n c e E x p r e s s i o n −>g e t s y m b o l ()−> g e t d e c l a r a t i o n ( ) ; SgScopeStatement ∗

v a r i a b l e S c o p e = v a r i a b l e −>g e t s c o p e ( ) ;

// Check i f t h i s i s a v a r i a b l e d e c l a r e d i n g l o b a l s c o p e , i f s o , t h e n s a v e i f ( i s S g G l o b a l ( v a r i a b l e S c o p e ) != NULL) { g l o b a l V a r i a b l e U s e L i s t . push back ( v a r i a b l e R e f e r e n c e E x p r e s s i o n ) ; } i ++; } return

it

globalVariableUseList ;

}

SgClassDeclaration∗ b u i l d C l a s s D e c l a r a t i o n A n d D e f i n i t i o n ( s t r i n g name , S g S c o p e S t a t e m e n t ∗ { // T h i s f u n c t i o n b u i l d s a c l a s s d e c l a r a t i o n and d e f i n i t i o n // ( b o t h t h e d e f i n i n g and n o n d e f i n i n g d e c l a r a t i o n s a s r e q u i r e d ) .

scope )

//

B u i l d a f i l e i n f o o b j e c t marked a s a t r a n s f o r m a t i o n Sg File Info ∗ f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ;

//

This i s the c l a s s d e f i n i t i o n ( the f i l e I n f o SgClassDefinition∗ classDefinition = new a s s e r t ( c l a s s D e f i n i t i o n != NULL ) ;

//

S e t t h e end o f c o n s t r u c t e x p l i c t l y ( w h e r e n o t a c l a s s D e f i n i t i o n −>s e t e n d O f C o n s t r u c t ( f i l e I n f o ) ;

//

This i s the d e f i n i n g d e c l a r a t i o n f o r the c l a s s ( with a r e f e r e n c e to the c l a s s d e f i n i t i o n ) S g C l a s s D e c l a r a t i o n ∗ c l a s s D e c l a r a t i o n = new S g C l a s s D e c l a r a t i o n ( f i l e I n f o , name . c s t r ( ) , S g C l a s s D e c l a r a t i o n : : e s t r u c t , NULL, c l a s s D a s s e r t ( c l a s s D e c l a r a t i o n != NULL ) ;

//

Set the d e f i n i n g d e c l a r a t i o n in the d e f i n i n g d e c l a r a t i o n ! c l a s s D e c l a r a t i o n −>s e t d e f i n i n g D e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ;

//

S e t t h e non d e f i n i n g d e c l a r a t i o n i n t h e d e f i n i n g d e c l a r a t i o n ( b o t h a r e r e q u i r e d ) S g C l a s s D e c l a r a t i o n ∗ n o n d e f i n i n g C l a s s D e c l a r a t i o n = new S g C l a s s D e c l a r a t i o n ( f i l e I n f o , name . c s t r ( ) , S g C l a s s D e c l a r a t i o n : : e s t r u c t , a s s e r t ( c l a s s D e c l a r a t i o n != NULL ) ; n o n d e f i n i n g C l a s s D e c l a r a t i o n −>s e t s c o p e ( s c o p e ) ; n o n d e f i n i n g C l a s s D e c l a r a t i o n −>s e t t y p e ( S g C l a s s T y p e : : c r e a t e T y p e ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ) ;

//

S e t t h e i n t e r n a l r e f e r e n c e t o t h e non−d e f i n i n g d e c l a r a t i o n c l a s s D e c l a r a t i o n −>s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ; c l a s s D e c l a r a t i o n −>s e t t y p e ( n o n d e f i n i n g C l a s s D e c l a r a t i o n −>g e t t y p e ( ) ) ;

//

S e t t h e d e f i n i n g and no−d e f i n i n g d e c l a r a t i o n s i n t h e non−d e f i n i n g c l a s s d e c l a r a t i o n ! n o n d e f i n i n g C l a s s D e c l a r a t i o n −>s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ; n o n d e f i n i n g C l a s s D e c l a r a t i o n −>s e t d e f i n i n g D e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ;

//

Set the nondefining d e c l a r a t i o n as a forward n o n d e f i n i n g C l a s s D e c l a r a t i o n −>s e t F o r w a r d ( ) ;

i s the p o s i t i o n of the opening SgClassDefinition ( f i l e I n f o );

transformation

s e t t h e s c o p e e x p l i c i t l y ( name q u a l i f i c a t i o n c l a s s D e c l a r a t i o n −>s e t s c o p e ( s c o p e ) ;

is

the

location

of

the

closing

brace )

declaration !

// Don ’ t f o r g e t t h e s e t t h e d e c l a r a t i o n i n t h e d e f i n i t i o n c l a s s D e f i n i t i o n −>s e t d e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ; //

this

brace )

tricks

can

( IR node

imply

it

constructors

is

not

are

always

s i d e −e f f e c t

the

parent

free !)!

IR node ! )

// some e r r o r c h e c k i n g a s s e r t ( c l a s s D e c l a r a t i o n −>g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; a s s e r t ( c l a s s D e c l a r a t i o n −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; a s s e r t ( c l a s s D e c l a r a t i o n −>g e t d e f i n i t i o n ( ) != NULL ) ;

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add f u n c t i o n s y m b o l t o g l o b a l s c o p e ! p r i n t f ( ” F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r c l a s s = %p = %s \n ” , s c o p e , s c o p e −>c l a s s n a m e ( ) . c s t r ( ) , c l a s s D e c l a r S g C l a s s S y m b o l ∗ c l a s s S y m b o l = new S g C l a s s S y m b o l ( c l a s s D e c l a r a t i o n ) ; s c o p e −>i n s e r t s y m b o l ( c l a s s D e c l a r a t i o n −>g e t n a m e ( ) , c l a s s S y m b o l ) ;

Figure 32.23: Example source code shows repackaging of global variables to a struct (part 2).

32.6. CREATING A ’STRUCT’ FOR GLOBAL VARIABLES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

ROSE ASSERT( s c o p e −>l o o k u p c l a s s s y m b o l ( c l a s s D e c l a r a t i o n −>g e t n a m e ( ) ) return

231

!= NULL ) ;

classDeclaration ;

}

SgVariableSymbol ∗ putGlobalVariablesIntoClass { // T h i s f u n c t i o n i t e r a t e s SgVariableSymbol ∗ for { //

( R o s e S T L C o n t a i n e r & g l o b a l V a r i a b l e s , over

the

list

of

global

variables

and

inserts

them

into

the

SgClassDeclaration∗ iput

class

classDeclaration

)

definition

g l o b a l C l a s s V a r i a b l e S y m b o l = NULL ;

( R o s e S T L C o n t a i n e r : : i t e r a t o r

var = g l o b a l V a r i a b l e s . begin ( ) ;

var

!=

g l o b a l V a r i a b l e s . end ( ) ;

v a r++)

p r i n t f ( ” Appending g l o b a l v a r i a b l e = %s t o new g l o b a l V a r i a b l e C o n t a i n e r \n ” , ( ∗ v a r)−>g e t n a m e ( ) . s t r ( ) ) ; S g V a r i a b l e D e c l a r a t i o n ∗ g l o b a l V a r i a b l e D e c l a r a t i o n = i s S g V a r i a b l e D e c l a r a t i o n ( ( ∗ v a r)−> g e t p a r e n t ( ) ) ; a s s e r t ( g l o b a l V a r i a b l e D e c l a r a t i o n != NULL ) ;

// Get t h e g l o b a l s c o p e f r o m t h e g l o b a l v a r i a b l e d i r e c t l y S g G l o b a l ∗ g l o b a l S c o p e = i s S g G l o b a l ( g l o b a l V a r i a b l e D e c l a r a t i o n −>g e t s c o p e ( ) ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ; if

( v a r == g l o b a l V a r i a b l e s . b e g i n ( ) ) { // T h i s i s t h e f i r s t t i m e i n t h i s l o o p , r e p l a c e t h e f i r s t g l o b a l v a r i a b l e w i t h // t h e c l a s s d e c l a r a t i o n / d e f i n i t i o n c o n t a i n i n g a l l t h e g l o b a l v a r i a b l e s ! // Note t h a t i n i t i a l i z e r s i n t h e g l o b a l v a r i a b l e d e c l a r a t i o n s r e q u i r e m o d i f i c a t i o n // o f t h e p r e i n i t i a l i z a t i o n l i s t i n t h e c l a s s ’ s c o n s t r u c t o r ! I am i g n o r i n g t h i s f o r g l o b a l S c o p e −>r e p l a c e s t a t e m e n t ( g l o b a l V a r i a b l e D e c l a r a t i o n , c l a s s D e c l a r a t i o n ) ; //

now !

B u i l d s o u r c e p o s i t i o n i n f o r m a i t o n ( marked a s t r a n s f o r m a t i o n ) Sg File Info ∗ f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ;

// Add t h e v a r i a b l e o f t h e c l a s s t y p e t o t h e g l o b a l s c o p e ! S g C l a s s T y p e ∗ v a r i a b l e T y p e = new S g C l a s s T y p e ( c l a s s D e c l a r a t i o n −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) ) ; a s s e r t ( v a r i a b l e T y p e != NULL ) ; S g V a r i a b l e D e c l a r a t i o n ∗ v a r i a b l e D e c l a r a t i o n = new S g V a r i a b l e D e c l a r a t i o n ( f i l e I n f o , ” A M P I g l o b a l s ” , v a r i a b l e T y p e ) ; a s s e r t ( v a r i a b l e D e c l a r a t i o n != NULL ) ; g l o b a l S c o p e −>i n s e r t s t a t e m e n t ( c l a s s D e c l a r a t i o n , v a r i a b l e D e c l a r a t i o n , f a l s e ) ; a s s e r t ( v a r i a b l e D e c l a r a t i o n −>g e t v a r i a b l e s ( ) . empty ( ) == f a l s e ) ; S g I n i t i a l i z e d N a m e ∗ v a r i a b l e N a m e = ∗ ( v a r i a b l e D e c l a r a t i o n −>g e t v a r i a b l e s ( ) . b e g i n ( ) ) ; a s s e r t ( v a r i a b l e N a m e != NULL ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : Need t o s e t t h e s c o p e o f v a r i a b l e N a m e −>s e t s c o p e ( g l o b a l S c o p e ) ; //

t h e new

variable .

build the return value g l o b a l C l a s s V a r i a b l e S y m b o l = new S g V a r i a b l e S y m b o l ( v a r i a b l e N a m e ) ;

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add t h e s y m b o l t o t h e g l o b a l s c o p e ( new t e s t i n g r e q u i r e s t h i s ) . g l o b a l S c o p e −>i n s e r t s y m b o l ( v a r i a b l e N a m e −>g e t n a m e ( ) , g l o b a l C l a s s V a r i a b l e S y m b o l ) ; ROSE ASSERT( g l o b a l S c o p e −>l o o k u p v a r i a b l e s y m b o l ( v a r i a b l e N a m e −>g e t n a m e ( ) ) != NULL ) ; } else { // f o r a l l o t h e r i t e r a t i o n s o f t h i s l o o p . . . // remove v a r i a b l e d e c l a r a t i o n f r o m t h e g l o b a l s c o p e g l o b a l S c o p e −>r e m o v e s t a t e m e n t ( g l o b a l V a r i a b l e D e c l a r a t i o n ) ; } // add t h e v a r i a b l e d e c l a r a t i o n t o t h e c l a s s d e f i n i t i o n c l a s s D e c l a r a t i o n −>g e t d e f i n i t i o n ()−>append member ( g l o b a l V a r i a b l e D e c l a r a t i o n ) ; } return

globalClassVariableSymbol ;

}

void f i x u p R e f e r e n c e s T o G l o b a l V a r i a b l e s ( R o s e S T L C o n t a i n e r & v a r i a b l e R e f e r e n c e L i s t , S g V a r i a b l e S y m b o l ∗ g l o b a l C l a s s V a r i a b l e S y m b o l ) { // Now f i x u p t h e SgVarRefExp t o r e f e r e n c e t h e g l o b a l v a r i a b l e s t h r o u g h a s t r u c t f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r v a r = v a r i a b l e R e f e r e n c e L i s t . b e g i n ( ) ; v a r != v a r i a b l e R e f e r e n c e L i s t . end ( ) ; v a r++) { a s s e r t ( ∗ v a r != NULL ) ;

Figure 32.24: Example source code shows repackaging of global variables to a struct (part 3).

232

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

CHAPTER 32. AST CONSTRUCTION

//

printf

(” Variable

reference

f o r %s \n ” , ( ∗ v a r)−> g e t s y m b o l ()−> g e t d e c l a r a t i o n ()−> g e t n a m e ( ) . s t r ( ) ) ;

SgNode ∗ p a r e n t = ( ∗ v a r)−> g e t p a r e n t ( ) ; a s s e r t ( p a r e n t != NULL ) ; //

I f t h i s i s n o t an e x p r e s s i o n t h e n i s l i k e l y a m e a n i n g l e s s SgExpression∗ parentExpression = isSgExpression ( parent ) ; a s s e r t ( p a r e n t E x p r e s s i o n != NULL ) ; the

reference

through

the

global

class

variable

statement

such

as

(” x ; ” )

( ” x ” −−> ” A M P I g l o b a l s . x ” )

//

Build

//

B u i l d s o u r c e p o s i t i o n i n f o r m a i t o n ( marked a s t r a n s f o r m a t i o n ) Sg File Info ∗ f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ;

//

Build ” AMPI globals ” S g E x p r e s s i o n ∗ l h s = new SgVarRefExp ( f i l e I n f o , g l o b a l C l a s s V a r i a b l e S y m b o l ) ; a s s e r t ( l h s != NULL ) ; // B u i l d ” A M P I g l o b a l s . x ” f r o m ” x ” SgDotExp∗ g l o b a l V a r i a b l e R e f e r e n c e = new SgDotExp ( f i l e I n f o , l h s , ∗ v a r ) ; a s s e r t ( g l o b a l V a r i a b l e R e f e r e n c e != NULL ) ; if

( p a r e n t E x p r e s s i o n != NULL) { // I n t r o d u c e r e f e r e n c e t o ∗ v a r

through

the

data

structure

//

case of binary operator SgUnaryOp∗ u n a r y O p e r a t o r = isSgUnaryOp ( p a r e n t E x p r e s s i o n ) ; i f ( u n a r y O p e r a t o r != NULL) { u n a r y O p e r a t o r −>s e t o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } else { // c a s e o f b i n a r y o p e r a t o r SgBinaryOp ∗ b i n a r y O p e r a t o r = i s S g B i n a r y O p ( p a r e n t E x p r e s s i o n ) ; i f ( b i n a r y O p e r a t o r != NULL) { // f i g u r e o u t i f t h e ∗ v a r i s on t h e l h s o r t h e r h s i f ( b i n a r y O p e r a t o r −>g e t l h s o p e r a n d ( ) == ∗ v a r ) { b i n a r y O p e r a t o r −>s e t l h s o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } else { a s s e r t ( b i n a r y O p e r a t o r −>g e t r h s o p e r a n d ( ) == ∗ v a r ) ; b i n a r y O p e r a t o r −>s e t r h s o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } } else { // i g n o r e t h e s e c a s e s f o r now ! s w i t c h ( p a r e n t E x p r e s s i o n −>v a r i a n t T ( ) ) { // Where t h e v a r i a b l e a p p e r s i n t h e f u n c t i o n argument l i s t t h e p a r e n t i s a S g E x p r L i s t E x p c a s e V SgExprListExp : { p r i n t f ( ” S o r r y n o t i m p l e m e n t e d , c a s e o f g l o b a l v a r i a b l e i n f u n c t i o n argument l i s t assert ( false ); break ; } case V S g I n i t i a l i z e r : c a s e V SgRefExp : c a s e V SgVarArgOp : default : { p r i n t f (” Error : d e f a u l t reached in switch p a r e n t E x p r e s s i o n = %p = %s \n ” , p a r e n t E x p r e s s i o n , p a r e n t E x p r e s s i o n −>c l a s s n a m e ( ) . c s t r ( ) ) ; assert ( false ); } } } } } } } #d e f i n e OUTPUT NAMES OF GLOBAL VARIABLES 0 #d e f i n e OUTPUT NAMES OF GLOBAL VARIABLE REFERENCES 0 void

Figure 32.25: Example source code shows repackaging of global variables to a struct (part 4).

...

\n ” ) ;

32.6. CREATING A ’STRUCT’ FOR GLOBAL VARIABLES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

transformGlobalVariablesToUseStruct { a s s e r t ( f i l e != NULL ) ; //

(

SgSourceFile

∗file

233

)

T h e s e a r e t h e g l o b a l v a r i a b l e s i n t h e i n p u t program ( p r o v i d e d a s h e l p f u l i n f o r m a t i o n ) R o s e S T L C o n t a i n e r g l o b a l V a r i a b l e s = b u i l d L i s t O f G l o b a l V a r i a b l e s ( f i l e ) ;

#i f

OUTPUT NAMES OF GLOBAL VARIABLES p r i n t f (” g l o b a l v a r i a b l e s ( declared in g l o b a l scope ) : f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r { p r i n t f (” %s \n ” , ( ∗ v a r)−>g e t n a m e ( ) . s t r ( ) ) ; } p r i n t f (”\n ” ) ; #e n d i f

\n ” ) ; var = g l o b a l V a r i a b l e s . begin ( ) ;

ignoring

all

other

var

!=

g l o b a l V a r i a b l e s . end ( ) ;

//

get the g l o b a l scope within the f i r s t f i l e ( c u r r e n t l y S g G l o b a l ∗ g l o b a l S c o p e = f i l e −>g e t g l o b a l S c o p e ( ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ;

//

Build the c l a s s d e c l a r a t i o n SgClassDeclaration ∗ c l a s s D e c l a r a t i o n = buildClassDeclarationAndDefinition (” AMPI globals t ” , globalScope ) ;

v a r++)

files )

// Put t h e g l o b a l v a r i a b l e s i n t o t h e c l a s s SgVariableSymbol ∗ globalClassVariableSymbol = p u t G l o b a l V a r i a b l e s I n t o C l a s s ( g l o b a l V a r i a b l e s , c l a s s D e c l a r a t i o n ) ; // //

T h e i r a s s o c i a t e d s y m b o l s w i l l be l o c a t e d w i t h i n t h e p r o j e c t ’ s AST ( where they o c c u r i n v a r i a b l e r e f e r e n c e e x p r e s s i o n s ) . R o s e S T L C o n t a i n e r v a r i a b l e R e f e r e n c e L i s t = b u i l d L i s t O f V a r i a b l e R e f e r e n c e E x p r e s s i o n s U s i n g G l o b a l V a r i a b l e s ( f i l e ) ;

#i f

OUTPUT NAMES OF GLOBAL VARIABLE REFERENCES p r i n t f ( ” g l o b a l v a r i a b l e s a p p e a r i n g i n t h e a p p l i c a t i o n : \n ” ) ; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r v a r = v a r i a b l e R e f e r e n c e L i s t . b e g i n ( ) ; { p r i n t f (” %s \n ” , ( ∗ v a r)−> g e t s y m b o l ()−> g e t d e c l a r a t i o n ()−> g e t n a m e ( ) . s t r ( ) ) ; } p r i n t f (”\n ” ) ; #e n d i f //

var

!=

v a r i a b l e R e f e r e n c e L i s t . end ( ) ;

F i x u p a l l r e f e r e n c e s t o g l o b a l v a r i a b l e t o a c c e s s t h e v a r i a b l e t h r o u g h t h e c l a s s ( ” x ” −−> ” A M P I g l o b a l s . x ” ) fixupReferencesToGlobalVariables ( variableReferenceList , globalClassVariableSymbol ) ;

}

void transformGlobalVariablesToUseStruct ( SgProject ∗ project ) { // C a l l t h e t r a n s f o r m a t i o n o f e a c h f i l e ( t h e r e a r e m u l t i p l e S g F i l e // o b j e c t s when m u l t i p l e f i l e s a r e s p e c f i e d on t h e command l i n e ! ) . a s s e r t ( p r o j e c t != NULL ) ; c o n s t S g F i l e P t r L i s t& f i l e L i s t = p r o j e c t −> g e t f i l e L i s t ( ) ; SgFilePtrList : : c o n s t i t e r a t o r f i l e = f i l e L i s t . begin ( ) ; w h i l e ( f i l e != f i l e L i s t . end ( ) ) { t r a n s f o r m G l o b a l V a r i a b l e s T o U s e S t r u c t ( i s S g S o u r c e F i l e (∗ f i l e ) ) ; f i l e ++; } } // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // MAIN PROGRAM // ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ int main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; //

B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; a s s e r t ( p r o j e c t != NULL ) ;

//

transform a p p l i c a t i o n as r e q u i r e d transformGlobalVariablesToUseStruct ( project ) ;

// Code g e n e r a t i o n p h a s e ( w r i t e r e t u r n backend ( p r o j e c t ) ; }

o u t new

application

” r o s e
file

name>”)

Figure 32.26: Example source code shows repackaging of global variables to a struct (part 5).

v a r++)

234

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

CHAPTER 32. AST CONSTRUCTION

int x ; int y ; long z ; float pressure ; i n t main ( ) { int a = 0; int b = 0; floa t density = 1.0; x++; b++; x = a + y; return 0; }

Figure 32.27: Example source code used as input to translator adding new function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

s t r u c t AMPI globals t { int x ; int y ; long z ; float pressure ; } ; s t r u c t AMPI globals t AMPI globals ; i n t main ( ) { int a = 0; int b = 0; floa t density = 1.0; AMPI g l o b a l s . x++; b++; AMPI g l o b a l s . x = a + A M P I g l o b a l s . y ; return 0; }

Figure 32.28: Output of input to translator adding new function.

Chapter 33

Parser Building Blocks It is often needed to write a small parser to parse customized source code annotations in forms of C/C++ pragmas or Fortran comments. The parser is responsible for recognizing keywords, variables, expressions in the annotations and storing the recognized information in AST, often in a form of persistent AstAttribute. ROSE provides a set of helper functions which can be used to create such a simple parser using recursive descent parsing1 . These functions are collected in the namespace named AstFromString, documented under the Namespace List of ROSE’s online Doxygen web references. A suggested workflow to build your own pragma (or comment) parser is: • input: define the grammar of your pragma, including keywords, directives and optional clauses. Borrowing grammars of OpenMP is a good idea. • output: define your own AstAttribute in order to store any possible pragma information generated by your parser. The attribute data structure should store the AST subtrees generated by a parser. The Attribute itself should be attached to relevant statement node (pragma declarations or others) in the original AST. • parser: write your recursive descent pragma parser by leveraging the functions defined in rose sourcetree/src/frontend/SageIII/astFromString/AstFromString.h. The parsing results will be used to generate your attribute which should be attached to your pragma declaration statement (or a following statement for a Fortran comment). • use of parsing results: in another phase, write your traversal to recognize the attached attribute to do whatever you plan to do with the pragma information. A full example is provided under rose/projects/pragmaParsing to demonstrate the entire workflow of using parser building blocks (helper functions within AstFromString) to create a parser to parse user-defined pragmas. We will use this example in this tutorial. 1 Description of basic recursive descent parsing techniques can be found at http://en.wikipedia.org/wiki/ Recursive_descent_parser

235

236

33.1

CHAPTER 33. PARSER BUILDING BLOCKS

Grammar Examples

The first step of building a parser is to formally define the grammar of input strings. The online Doxygen web reference lists helper functions from the AstFromString namespace, with detailed information about the underneath grammars they try to recognize. These grammar can be used as example about how to prepare grammars. For example bool afs match cast expression () follows the grammar like: cast expression : ’(’ type name ’)’ cast expression | unary expression, which means a cast expression is either a unary expression or another cast expression prepended with a type name enclosed in a pair of parenthesis. Note that the grammar has a rule with a right recursion here (cast expression : ’(’ type name ’)’ cast expression). The grammars should try to avoid left recursion (e.g., result : result something else ), which may leads infinite recursive calls in your parser. Again, a helper function in AstFromString often implements a grammar. Please take a look at some of them to see how grammars are written to facilitate building recursive descent parsers. The pragma in the pragmaParsing project has the following grammar (documented in rose/projects/pragmaParsing/hcpragma.h): ---------- grammar begin ----------% ’string’ means literal string to be matched % | means alternation hcc_pragma = ’#pragma’ hc_simple_part | hc_cuda_part hc_simple_part = ’hc’ ’entry’| ’suspendable’ | ’entry suspendable’ | ’suspendable entry’ hc_cuda_part = ’CUDA’ kernel_part| place_part kernel_part = ’kernel’ % place could be an expression % the grammar uses assignment_expression instead of expression to disallow comma expressions % (list of expressions connected with comma) e.g. exp1, exp2 will be parsed to be (ex1, exp2) % otherwise. % place_part = assignment_expression autodim_part | dim_part % autodim([, , , ]) % [ ] means optional % , means ’,’ to be simple % assignment_expression is used to avoid parsing exp1, exp2, exp3 to be one single comma % expression ((exp1,exp2),exp3) autodim_part = ’autodim’ ’(’ assignment_expression [, assignment_expression [, assignment_expression [, assignment_expression ] ] ] ’)’ % dim(blocksPerGrid, threadsPerBlock[, shared_size]) dim_part = ’dim’ ’(’ assignment_expression , assignment_expression ,

33.2. ASTATTRIBUTE TO STORE RESULTS [ , assignment_expression ]

237 ’)’

The example grammar allows a list of expressions inside a pragma. A tricky part here is that C allows single comma expression like ((exp1,exp2),exp3). We use assignment expression to avoid parsing exp1, exp2, exp3 to be one such single comma expression. The point here is that the terms in the grammar have to be accurately mapped to formal C grammar terms. Some familiarity of formal C grammar terms, as shown at http://www.antlr.org/grammar/ 1153358328744/C.g, is required since helper functions have names matching the formal terms in C grammar. The assignment expressions, not expressions, are what we care about in this particular simple grammar.

33.2

AstAttribute to Store results

Once the grammar is defined with terms matching helper functions of AstFromString, a data structure is needed to store the results of parsing. It is recommended to create your data structure by inheriting AstAttribute, which can be attached to any AST nodes with location information. As in the pragmaParsing project, we define a few classes as the following: class HC_PragmaAttribute: public AstAttribute { public: SgNode * node; enum hcpragma_enum pragma_type; ... }; class HC_CUDA_PragmaAttribute: public HC_PragmaAttribute { public: SgExpression* place_exp; ... }; class HC_CUDA_autodim_PragmaAttribute: public HC_CUDA_PragmaAttribute { public: SgExpression* dim1_exp; SgExpression* dim2_exp; SgExpression* dim3_exp; SgExpression* shared_size_exp; ... };

The point is that the class is inherited from AstAttribute and it contains fields to store all terms defined in the grammar.

238

CHAPTER 33. PARSER BUILDING BLOCKS

33.3

The AstFromString Namespace

AstFromString has a few namespace scope variables, such as: • char ∗ c char: this indicates the current position of the input string being parsed. • SgNode∗ c sgnode: a SgNode variable storing the current anchor AST node, which servers as a start point to find enclosing scopes for resolving identifiers/symbols discovered by a parser. • SgNode ∗ c parsed node: a SgNode variable storing the root of an AST subtree generated by the parser. In general, your parser should initialize c char with the first position of an input string to be parsed. It should also set c sgnode to be the pragma statement when you parse pragma strings, or the immediate following statement when you parse Fortran comments. The results often are stored in c parsed node. Your parser should timely check the results and filling your AstAttribute structure with the AST substrees for identifiers, constants, expressions, etc. Helper functions within AstFromString include functions to parse identifiers, types, substrings, expressions. AST pieces will be generated automatically as needed. So users can focus on building their grammar and parser without doing repetitive chores of parsing common language constructs. Take bool afs match assignment expression() as an example, this function will try to match an expression that satisfies a grammar rule like: assignment expression : lvalue assignment operator assignment expression | conditional expression . If a successful match is found, the function returns true. In the meantime, the function builds an AST subtree to represent the matched expression and stores the subtree into the variable SgNode∗ c sgnode.

33.4

Write your parsers using parser building blocks

rose/src/frontend/SageIII/astFromString/AstFromString.cpp has the implementation of all parser building blocks (helper functions) for a wide range of grammar rules. They can serve as examples about how to hand-write additional functions to recognize your own grammars. For example, to implement a simple grammar rule like type qualifier : ’const’ | ’ volatile ’, we have the following helper function: /* type_qualifier : ’const’ | ’volatile’ ; */ bool afs_match_type_qualifier() { bool result = false;

33.4. WRITE YOUR PARSERS USING PARSER BUILDING BLOCKS

239

const char* old_char = c_char; if (afs_match_substr("const")) { c_parsed_node = buildConstVolatileModifier (SgConstVolatileModifier::e_const); result = true; } else if (afs_match_substr("volatile")) { c_parsed_node = buildConstVolatileModifier (SgConstVolatileModifier::e_volatile); result = true; } if (result == false) c_char = old_char; return result; } Please note that the function above tries to undo any side effects if the parsing fails. If successful, the parsed result will be stored into c parsed node. Here is another example with right recursion: /* Grammar is cast_expression : ’(’ type_name ’)’ cast_expression | unary_expression ; */ bool afs_match_cast_expression() { bool result = false; const char* old_char = c_char; if (afs_match_unary_expression()) { if (isSgExpression(c_parsed_node)) result = true; } else if (afs_match_char(’(’)) { if (afs_match_type_name()) { SgType* t = isSgType(c_parsed_node); assert (t!= NULL); if (afs_match_char(’)’)) { if (afs_match_cast_expression()) { SgExpression* operand = isSgExpression(c_parsed_node);

240

CHAPTER 33. PARSER BUILDING BLOCKS c_parsed_node = buildCastExp(operand, t); result = true; // must set this!! } else { c_char = old_char; } } else { c_char = old_char; } } else { c_char = old_char; result = false; } } if (result == false) return result;

c_char = old_char;

} sourcetree/projects/pragmaParsing/hcpragma.C gives a full example of how to use helper functions inside your own parsing functions.

33.5

Limitations

Currently, the parser building blocks support C only. Expressions parsing functions are ready to be used by users. Statement parsing is still ongoing work.

Chapter 34

Handling Comments, Preprocessor Directives, And Adding Arbitrary Text to Generated Code What To Learn From These Examples Learn how to access existing comments and CPP directives and modify programs to include new ones. Where such comments can be automated they can be used to explain transformations or for more complex transformations using other tools designed to read the generated comments. Also included is how to add arbitrary text to the generated code (often useful for embedded system programming to support back-end compiler specific functionality). This chapter deals with comments and preprocessor directives. These are often dropped from compiler infrastructures and ignored by make tools. ROSE takes great care to preserve all comments and preprocessor directives. Where they exist in the input code we save them (note that EDG drops them from their AST) and weave them back into the AST. Note that #pragma is not a CPP directive and is part of the C and C++ grammar, thus represented explicitly in the AST (see SgPragmaDeclaration).

34.1

How to Access Comments and Preprocessor Directives

Comments and CPP directives are treated identically within ROSE and are saved as special preprocessor attributes to IR nodes within the AST. Not all IR nodes can have these specific type of attributes, only SgLocatedNodes can be associated with such preprocessor attributes. The more general persistent attribute mechanism within ROSE is separate from this preprocessor attribute mechanism and is available on a wider selection of IR nodes. 241

242CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

34.1.1

Source Code Showing How to Access Comments and Preprocessor Directives

Figure 34.1 shows an example translator which access the comments and preprocessor directives on each statement. Note that in this example the AST is traversed twice, first header files are ignored and then the full AST (including header files) are traversed (generated additional comments). The input code is shown in figure 34.2, the output of this code is shown in figure 34.3 for the source file only. Figure 34.4 shows the same input code processed to output comments and preprocessor directives assembled from the source file and all header files.

34.1.2

Input to example showing how to access comments and CPP directives

Figure 34.2 shows the example input used for demonstration of how to collect comments and CPP directives.

34.1.3

Comments and CPP Directives collected from source file (skipping headers)

Figure 34.3 shows the results from the collection of comments and CPP directives within the input source file only (without -rose:collectAllCommentsAndDirectives).

34.1.4

Comments and CPP Directives collected from source file and all header files

Figure 34.4 shows the results from the collection of comments and CPP directives within the input source file and all headers (with -rose:collectAllCommentsAndDirectives).

34.2

Collecting #define C Preprocessor Directives

This example shows how to collect the #define directives as a list for later processing.

34.2.1

Source Code Showing How to Collect #define Directives

Figure 34.5 shows an example translator which access the comments and preprocessor directives on each statement. Note that in this example the AST is traversed twice, first header files are ignored and then the full AST (including header files) are traversed (generated additional comments). The input code is shown in figure 34.6, the output of this code is shown in Figure 34.7 shows the same input code processed to output comments and preprocessor directives assembled from the source file and all header files.

34.3. AUTOMATED GENERATION OF COMMENTS

34.2.2

243

Input to example showing how to access comments and CPP directives

Figure 34.6 shows the example input used for demonstration of how to collect comments and CPP directives.

34.2.3

Comments and CPP Directives collected from source file and all header files

Figure 34.7 shows the results from the collection of comments and CPP directives within the input source file and all headers (with -rose:collectAllCommentsAndDirectives).

34.3

Automated Generation of Comments

Figure 34.8 shows an example of how to introduce comments into the AST which will then show up in the generated source code. The purpose for this is generally to add comments to where transformations are introduced. If the code is read by the use the generated comments can be useful in identifying, marking, and/or explaining the transformation. This chapter presents an example translator which just introduces a comment at the top of each function. The comment includes the name of the function and indicates that the comment is automatically generated. Where appropriate such techniques could be used to automate the generation of documentation templates in source code that would be further filled in by the used. In this case the automatically generated templates would be put into the generated source code and a patch formed between the generated source and the original source. The patch could be easily inspected and applied to the original source code to place the documentation templates into the original source. The skeleton of the documentation in the source code could been be filled in by the use. The template would have all relevant information obtained by analysis (function parameters, system functions used, security information, side-effects, anything that could come from an analysis of the source code using ROSE).

34.3.1

Source Code Showing Automated Comment Generation

Figure 34.8 shows an example translator which calls the mechanism to add a comment to the IR node representing a function declaration (SgFunctionDeclaration). The input code is shown in figure 34.9, the output of this code is shown in figure 34.10.

34.3.2

Input to Automated Addition of Comments

Figure 34.9 shows the example input used for demonstration of an automated commenting.

34.3.3

Final Code After Automatically Adding Comments

Figure 34.10 shows the results from the addition of comments to the generated source code.

244CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

34.4

Addition of Arbitrary Text to Unparsed Code Generation

This section is different from the comment generation (section 34.3) because it is more flexible and does not introduce any formatting. It also does not use the same internal mechanism, this mechanism supports the addition of new strings or the replacement of the IR node (where the string is attached) with the new string. It is fundamentally lower level and a more powerful mechanism to support generation of tailored output where more than comments, CPP directives, or AST transformation are required. It is also much more dangerous to use. This mechanism is expected to be used rarely and sparingly since no analysis of the AST is likely to leverage this mechanism and search for code that introduced as a transformation here. Code introduced using this mechanism is for the most part unanalyzable since it would have to be reparsed in the context of the location in the AST were it is attached. (Technically this is possible and is the subject of the existing ROSE AST Rewrite mechanism, but that is a different subject). Figure 34.11 shows an example of how to introduce arbitrary text into the AST for output by the unparser which will then show up in the generated source code. The purpose for this is generally to add backend compiler or tool specific code generation which don’t map to any formal language constructs and so cannot be represented in the AST. However, since most tools that require specialized annotations read them as comments, the mechanism in the previous section 34.3 may be more appropriate. It is because this is not always that case that we have provide this more general mechanism (often useful for embedded system compilers).

34.4.1

Source Code Showing Automated Arbitrary Text Generation

Figure 34.11 shows an example translator which calls the mechanism to add a arbitrary text to the IR node representing a function declaration (SgFunctionDeclaration). The input code is shown in figure 34.12, the output of this code is shown in figure 34.13.

34.4.2

Input to Automated Addition of Arbitrary Text

Figure 34.12 shows the example input used for demonstration of the automated introduction of text via the unparser.

34.4.3

Final Code After Automatically Adding Arbitrary Text

Figure 34.13 shows the results from the addition of arbitrary text to the generated source code.

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

245

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; // C l a s s d e c l a r a t i o n c l a s s v i s i t o r T r a v e r s a l : public AstSimpleProcessing { public : v i r t u a l v o i d v i s i t ( SgNode∗ n ) ; }; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { // On e a c h node l o o k f o r any comments o f CPP d i r e c t i v e s SgLocatedNode ∗ l o c a t e d N o d e = i s S g L o c a t e d N o d e ( n ) ; i f ( l o c a t e d N o d e != NULL) { A t t a c h e d P r e p r o c e s s i n g I n f o T y p e ∗ comments = l o c a t e d N o d e −>g e t A t t a c h e d P r e p r o c e s s i n g I n f o ( ) ; if

( comments != NULL) { p r i n t f ( ” Found a t t a c h e d comments ( t o IR node a t %p o f t y p e : %s ) : \n ” , l o c a t e d N o d e , l o c a t e d N o d e −>c l a s s n a m e ( ) int counter = 0; AttachedPreprocessingInfoType : : i t e r a t o r i ; f o r ( i = comments−>b e g i n ( ) ; i != comments−>end ( ) ; i ++) { p r i n t f (” Attached Comment #%d i n f i l e %s ( r e l a t i v e P o s i t i o n=%s ) : c l a s s i f i c a t i o n %s : \ n%s \n ” , c o u n t e r ++,(∗ i )−> g e t f i l e i n f o ()−> g e t f i l e n a m e S t r i n g ( ) . c s t r ( ) , ( ( ∗ i )−> g e t R e l a t i v e P o s i t i o n ( ) == P r e p r o c e s s i n g I n f o : : b e f o r e ) ? ” b e f o r e ” : ” a f t e r ” , P r e p r o c e s s i n g I n f o : : d i r e c t i v e T y p e N a m e ( ( ∗ i )−>g e t T y p e O f D i r e c t i v e ( ) ) . c s t r ( ) , ( ∗ i )−> g e t S t r i n g ( ) . c s t r ( ) ) ; } } else { p r i n t f ( ” No a t t a c h e d comments ( a t %p o f t y p e : %s ) : \n ” , l o c a t e d N o d e , l o c a t e d N o d e −>s a g e c l a s s n a m e ( ) ) ; }

} } i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // // // // //

C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST T r a v e r s e a l l h e a d e r f i l e s and s o u r c e f i l e ( t h e −r o s e : c o l l e c t A l l C o m m e n t s A n d D i r e c t i v e s commandline o p t i o n c o n t r o l s i f comments and CPP d i r e c t i v e s a r e s e p a r a t e l y e x t r a c t e d from h e a d e r f i l e s ) . exampleTraversal . t r a v e r s e ( project , preorder ) ; exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0;

}

Figure 34.1: Example source code showing how to access comments.

246CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14

// #i n c l u d e #d e f i n e SOURCE CODE BEFORE INCLUDE A #d e f i n e SOURCE CODE BEFORE INCLUDE B #i n c l u d e #d e f i n e SOURCE CODE AFTER INCLUDE A #d e f i n e SOURCE CODE AFTER INCLUDE B // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e i n t main ( ) { return 0; }

Figure 34.2: Example source code used as input to collection of comments and CPP directives.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

No a t t a c h e d comments ( a t 0 x 7 f c 1 a 1 1 7 2 1 2 0 o f t y p e : S g G l o b a l ) : Found a t t a c h e d comments ( t o IR node a t 0 x 7 f c 1 9 4 a 6 4 7 9 0 o f t y p e : S g F u n c t i o n D e c l a r a t i o n ) : Attached Comment #0 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u // #i n c l u d e

Attached Comment #1 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u #d e f i n e SOURCE CODE BEFORE INCLUDE A

Attached Comment #2 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u #d e f i n e SOURCE CODE BEFORE INCLUDE B

Attached Comment #3 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u #i n c l u d e

Attached Comment #4 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u #d e f i n e SOURCE CODE AFTER INCLUDE A

Attached Comment #5 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u #d e f i n e SOURCE CODE AFTER INCLUDE B

Attached Comment #6 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e No No No No No

attached attached attached attached attached

comments comments comments comments comments

( at ( at ( at ( at ( at

0 x7fc194ddb6b8 0 x7fc193fde010 0 x7fc194073010 0 x7fc193f7c010 0 x7fc193fab010

of of of of of

type : type : type : type : type :

SgFunctionParameterList ) : SgFunctionDefinition ) : SgBasicBlock ) : SgReturnStmt ) : SgIntVal ) :

Figure 34.3: Output from collection of comments and CPP directives on the input source file only.

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

247

No a t t a c h e d comments ( a t 0 x 7 f 0 f c 7 3 6 3 1 2 0 o f t y p e : S g G l o b a l ) : Found a t t a c h e d comments ( t o IR node a t 0 x 7 f 0 f b a c 5 4 7 9 0 o f t y p e : S g F u n c t i o n D e c l a r a t i o n ) : Attached Comment #0 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t // #i n c l u d e

Attached Comment #1 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t #d e f i n e SOURCE CODE BEFORE INCLUDE A

Attached Comment #2 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t #d e f i n e SOURCE CODE BEFORE INCLUDE B

Attached Comment #3 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t #i n c l u d e

Attached Comment #4 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t #d e f i n e SOURCE CODE AFTER INCLUDE A

Attached Comment #5 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t #d e f i n e SOURCE CODE AFTER INCLUDE B

Attached Comment #6 i n f i l e / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e No No No No No

attached attached attached attached attached

comments comments comments comments comments

( at ( at ( at ( at ( at

0 x7f0fbafcb6b8 0 x7f0fba1ce010 0 x7f0fba263010 0 x7f0fba16c010 0 x7f0fba19b010

of of of of of

type : type : type : type : type :

SgFunctionParameterList ) : SgFunctionDefinition ) : SgBasicBlock ) : SgReturnStmt ) : SgIntVal ) :

Figure 34.4: Output from collection of comments and CPP directives on the input source file and all header files.

248CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; // B u i l d a s y n t h e s i z e d a t t r i b u t e f o r t h e t r e e t r a v e r s a l class SynthesizedAttribute { public : // L i s t o f #d e f i n e d i r e c t i v e s ( s a v e t h e P r e p r o c e s s i n g I n f o o b j e c t s // s o t h a t we have a l l t h e s o u r c e c o d e p o s i t i o n i n f o r m a t i o n ) . l i s t

a c c u m u l a t e d L i s t ; void display ( ) const ; }; void SynthesizedAttribute : : display () const { l i s t

: : c o n s t i t e r a t o r i = a c c u m u l a t e d L i s t . b e g i n ( ) ; w h i l e ( i != a c c u m u l a t e d L i s t . end ( ) ) { p r i n t f ( ”CPP d e f i n e d i r e c t i v e = %s \n ” , ( ∗ i )−> g e t S t r i n g ( ) . c s t r ( ) ) ; i ++; } } class {

visitorTraversal public : // v i r t u a l virtual

:

p u b l i c AstBottomUpProcessing

f u n c t i o n must be d e f i n e d SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode∗ n , S y n t h e s i z e d A t t r i b u t e s L i s t c h i l d A t t r i b u t e s

);

}; SynthesizedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode∗ n , { SynthesizedAttribute localResult ;

SynthesizedAttributesList

childAttributes )

// p r i n t f ( ” I n e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( n = %p = %s ) \n ” , n , n−>c l a s s n a m e ( ) . c s t r ( ) ) ;

// B u i l d t h e l i s t from c h i l d r e n ( i n r e v e r s e o r d e r t o p r e s e r v e t h e f i n a l o r d e r i n g ) f o r ( S y n t h e s i z e d A t t r i b u t e s L i s t : : r e v e r s e i t e r a t o r c h i l d = c h i l d A t t r i b u t e s . r b e g i n ( ) ; c h i l d != c h i l d A t t r i b u t { l o c a l R e s u l t . a c c u m u l a t e d L i s t . s p l i c e ( l o c a l R e s u l t . a c c u m u l a t e d L i s t . b e g i n ( ) , c h i l d −>a c c u m u l a t e d L i s t ) ; } // Add i n t h e i n f o r m a t i o n from t h e c u r r e n t node SgLocatedNode ∗ l o c a t e d N o d e = i s S g L o c a t e d N o d e ( n ) ; i f ( l o c a t e d N o d e != NULL) { A t t a c h e d P r e p r o c e s s i n g I n f o T y p e ∗ c o m m e n t s A n d D i r e c t i v e s = l o c a t e d N o d e −>g e t A t t a c h e d P r e p r o c e s s i n g I n f o ( ) ; if

( c o m m e n t s A n d D i r e c t i v e s != NULL) { // p r i n t f ( ” Found a t t a c h e d comments ( t o IR node a t %p o f t y p e : %s ) : \n ” , l o c a t e d N o d e , l o c a t e d N o d e −>c // i n t c o u n t e r = 0 ;

// Use a r e v e r s e i t e r a t o r s o t h a t we p r e s e r v e t h e o r d e r when u s i n g p u s h f r o n t t o add e a c h d i r e c t i v AttachedPreprocessingInfoType : : r e v e r s e i t e r a t o r i ; f o r ( i = commentsAndDirectives−>r b e g i n ( ) ; i != commentsAndDirectives−>r e n d ( ) ; i ++) { // The d i f f e r e n t c l a s s i f i c a t i o n s o f comments and d i r e c t i v e s a r e i n ROSE/ s r c / f r o n t e n d / S a g e I I I / i f ( ( ∗ i )−>g e t T y p e O f D i r e c t i v e ( ) == P r e p r o c e s s i n g I n f o : : C p r e p r o c e s s o r D e f i n e D e c l a r a t i o n ) { #i f 0 p r i n t f (” Attached Comment #%d i n f i l e %s ( r e l a t i v e P o s i t i o n=%s ) : c l a s s i f i c a t c o u n t e r ++,(∗ i )−> g e t f i l e i n f o ()−> g e t f i l e n a m e S t r i n g ( ) . c s t r ( ) , ( ( ∗ i )−> g e t R e l a t i v e P o s i t i o n ( ) == P r e p r o c e s s i n g I n f o : : b e f o r e ) ? ” b e f o r e ” : ” a f t e r ” , P r e p r o c e s s i n g I n f o : : d i r e c t i v e T y p e N a m e ( ( ∗ i )−>g e t T y p e O f D i r e c t i v e ( ) ) . c s t r ( ) , ( ∗ i )−> g e t S t r i n g ( ) . c s t r ( ) ) ; #e n d i f // u s e p u s h f r o n t ( ) t o end up w i t h s o u r c e o r d e r i n g o f l o c a l R e s u l t . accumulatedList . p u s h f r o n t (∗ i ) ; } } } } // p r i n t f ( ” l o c a l R e s u l t a f t e r a d d i n g c u r r e n t node i n f o \n ” ) ; // l o c a l R e s u l t . d i s p l a y ( ) ;

final

list

of

directives

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

1 2 3 4

249

#d e f i n e JUST A MACRO j u s t a m a c r o #d e f i n e ANOTHER MACRO a n o t h e r m a c r o

Figure 34.6: Example source code used as input to collection of comments and CPP directives.

250CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

CPP d e f i n e

d i r e c t i v e = #d e f i n e SKIP ROSE BUILTIN DECLARATIONS

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

DBL MIN EXP

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT LEAST16 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

ATOMIC ACQUIRE 2

CPP d e f i n e

d i r e c t i v e = #d e f i n e

FLT MIN

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT LEAST8 TYPE

CPP d e f i n e

d i r e c t i v e = #d e f i n e

INTMAX C ( c ) c ## L

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CHAR BIT

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT8 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

WINT MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

ORDER LITTLE ENDIAN

CPP d e f i n e

d i r e c t i v e = #d e f i n e

SIZE MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

WCHAR MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC HAVE SYNC COMPARE AND SWAP 1 1

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC HAVE SYNC COMPARE AND SWAP 2 1

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC HAVE SYNC COMPARE AND SWAP 4 1

CPP d e f i n e

d i r e c t i v e = #d e f i n e

DBL DENORM MIN

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC HAVE SYNC COMPARE AND SWAP 8 1

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC ATOMIC CHAR LOCK FREE 2

CPP d e f i n e

d i r e c t i v e = #d e f i n e

FLT EVAL METHOD

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT FAST64 MAX

1 8 4 4 6 7 4 4 0 7 3 7 0 9 5 5 1 6 1 5UL

CPP d e f i n e

d i r e c t i v e = #d e f i n e

SIG ATOMIC TYPE

int

CPP d e f i n e

d i r e c t i v e = #d e f i n e

DBL MIN 10 EXP

CPP d e f i n e

d i r e c t i v e = #d e f i n e

FINITE MATH ONLY

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT FAST8 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

DEC64 MAX EXP

restrict

unix

( −1021) 65535

1 . 1 7 5 4 9 4 3 5 0 8 2 2 2 8 7 5 0 7 9 7 e −38F unsigned char

8 255 4 2 9 4 9 6 7 2 9 5U 1234

1 8 4 4 6 7 4 4 0 7 3 7 0 9 5 5 1 6 1 5UL 2147483647

d o u b l e ( 4 . 9 4 0 6 5 6 4 5 8 4 1 2 4 6 5 4 4 1 7 7 e −324L )

0

1

GCC ATOMIC CHAR32 T LOCK FREE 2 x86 64 1

( −307) 0 255 385

CPP d e f i n e

d i r e c t i v e = #d e f i n e

INT8 C ( c ) c

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT LEAST64 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

SHRT MAX

32767

CPP d e f i n e

d i r e c t i v e = #d e f i n e

LDBL MAX

1 . 1 8 9 7 3 1 4 9 5 3 5 7 2 3 1 7 6 5 0 2 e +4932L

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT LEAST8 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

GCC ATOMIC BOOL LOCK FREE 2

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINTMAX TYPE

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

CPP d e f i n e

d i r e c t i v e = #d e f i n e

UINT32 MAX

CPP d e f i n e

d i r e c t i v e = #d e f i n e

LDBL MAX EXP

1 8 4 4 6 7 4 4 0 7 3 7 0 9 5 5 1 6 1 5UL

255

long unsigned i n t

linux 1 DEC32 EPSILON

1E−6DF

unix 1 4 2 9 4 9 6 7 2 9 5U 16384

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

251

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; class {

visitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { SgFunctionDeclaration ∗ functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { s t r i n g comment = s t r i n g ( ” Auto−comment f u n c t i o n name : ” ) + f u n c t i o n D e c l a r a t i o n −>get name ( ) . s t r ( ) + ” i s now a commented f u n c t i o n ” ; // Note t h a t t h i s f u n c t i o n w i l l add t h e ”//” o r ”/∗ ∗/” comment s y n t a x a s r e q u i r e d S a g e I n t e r f a c e : : attachComment ( f u n c t i o n D e c l a r a t i o n , comment ) ; }

f o r C o r C++, o r F o r t r a n .

SgValueExp ∗ valueExp = i s S g V a l u e E x p ( n ) ; i f ( valueExp != NULL) { // Check i f t h e r e i s an e x p r e s s i o n t r e e from t h e o r i g i n a l u n f o l d e d e x p r e s s i o n . // T h i s i s a t r i v i a l example o u f t h e o u t p u t o f an a n a l y s i s r e s u l t . s t r i n g comment = s t r i n g ( ” Auto−comment v a l u e : ” ) + ( ( valueExp−>g e t o r i g i n a l E x p r e s s i o n T r e e ( ) != NULL) ? ” t h i s I S a c o n s t a n t f o l d e d v a l u e ” : ” t h i s i s NOT a c o n s t a n t f o l d e d v a l u e ” ) ; S a g e I n t e r f a c e : : attachComment ( valueExp , comment ) ; } } // T y p i c a l main f u n c t i o n f o r ROSE t r a n s l a t o r i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 34.8: Example source code showing how automate comments.

252CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8

int foo () { int x = 2; x += 3 + 4 ; return x ; }

Figure 34.9: Example source code used as input to automate generation of comments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

// Auto−comment f u n c t i o n name : int foo () { int x = // Auto−comment v a l u e : 2; x += // Auto−comment v a l u e : 3 + // Auto−comment v a l u e : 4; return x ; }

foo

i s now a commented f u n c t i o n

this

i s NOT a c o n s t a n t f o l d e d v a l u e

this

i s NOT a c o n s t a n t f o l d e d v a l u e

this

i s NOT a c o n s t a n t f o l d e d v a l u e

Figure 34.10: Output of input code after automating generation of comments.

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

253

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e ” r o s e . h” u s i n g namespace s t d ; class {

visitorTraversal public : v i r t u a l void

:

public AstSimpleProcessing

v i s i t ( SgNode∗ n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { SgFunctionDeclaration ∗ functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { // T h i s i s an example o f a XYZ t o o l s p e c i f i c a n n o t a t i o n s t r i n g c o m p i l e r S p e c i f i c D i r e c t i v e = ”\n# i f XYZ TOOL \n \” b u i l t i n \”\ n#e n d i f \n ” ; S a g e I n t e r f a c e : : addTextForUnparser ( f u n c t i o n D e c l a r a t i o n , c o m p i l e r S p e c i f i c D i r e c t i v e , A s t U n p a r s e A t t r i b u t e : : e b e f o r e ) ; } SgValueExp ∗ valueExp = i s S g V a l u e E x p ( n ) ; i f ( valueExp != NULL) { // Add a backend s p e c i f i c c o m p i l e r d i r e c t i v e s t r i n g c o m p i l e r S p e c i f i c D i r e c t i v e = ”\n# i f CRAY \n c r a y s p e c i f i c a t t r i b u t e \n#e n d i f \n ” ; S a g e I n t e r f a c e : : addTextForUnparser ( valueExp , c o m p i l e r S p e c i f i c D i r e c t i v e , A s t U n p a r s e A t t r i b u t e : : e b e f o r e ) ; } } // T y p i c a l main f u n c t i o n f o r ROSE t r a n s l a t o r i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e AST u s e d by ROSE SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 34.11: Example source code showing how automate the introduction of arbitrary text. 1 2 3 4 5 6

int foo () { int x = 42; return x ; }

Figure 34.12: Example source code used as input to automate generation of arbitrary text.

254CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13

# i f XYZ TOOL ” builtin ” #e n d i f int foo () { int x = # i f CRAY cray specific attribute #e n d i f 42; return x ; }

Figure 34.13: Output of input code after automating generation of arbitrary text.

Chapter 35

Partial Redundancy Elimination (PRE) Figure 35.1 shows an example of how to call the Partial Redundancy Elimination (PRE) implemented by Jeremiah Willcock. This transformation is useful for cleaning up code generated from other transformations (used in Qing’s loop optimizations).

35.1

Source Code for example using PRE

Figure 35.1 shows an example translator which calls the PRE mechanism. The input code is shown in figure 35.2, the output of this code is shown in figure 35.3. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

// Example t r a n s l a t o r d e m o n t r a t i n g P a r t i a l Redundancy E l i m i n a t i o n (PRE ) . #i n c l u d e ” r o s e . h” #i n c l u d e ”CommandOptions . h” i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . s t d : : v e c t o r l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; CmdOptions : : G e t I n s t a n c e ()−> S e t O p t i o n s ( a r g c , a r g v ) ; SgProject ∗ project = frontend ( l ) ; PRE : : p a r t i a l R e d u n d a n c y E l i m i n a t i o n ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 35.1: Example source code showing how use Partial Redundancy Elimination (PRE).

255

256

35.2

CHAPTER 35. PARTIAL REDUNDANCY ELIMINATION (PRE)

Input to Example Demonstrating PRE

Figure 35.2 shows the example input used for demonstration of Partial Redundancy Elimination (PRE) transformation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

// Program , b a s e d on example i n Knoop e t a l ( ” Optimal c o d e motion : t h e o r y and // p r a c t i c e ” , ACM TOPLAS 1 6 ( 4 ) , 1 9 9 4 , pp . 1117 −1155 , a s c i t e d i n P a l e r i e t a l // ( s e e p r e . C ) ) , c o n v e r t e d t o C++ i n t unknown ( ) ; // ROSE bug :

i n c l u d i n g body ” r e t u r n 0 ; ” h e r e doesn ’ t work

void foo () { i n t a , b , c , x , y , z , w; ( unknown ( ) ) { y = a + b; a = c; // Added by J e r e m i a h W i l l c o c k t o t e s t w = a + b; a = b; x = a + b; w = a + b; a = c; // End o f added p a r t x = a + b; } if

l o c a l PRE

( unknown ( ) ) { w h i l e ( unknown ( ) ) {y = a + b ; } } e l s e i f ( unknown ( ) ) { w h i l e ( unknown ( ) ) {} i f ( unknown ( ) ) {y = a + b ; } e l s e { g o t o L9 ; } // FIXME : t h e PRE c o d e c r a s h e s } else { g o t o L10 ; } if

if

t h i s isn ’ t in a block

z = a + b; a = c; L9 : x = a + b ; L10 :

0 ; // ROSE bug : u s i n g r e t u r n ; h e r e doesn ’ t work

} i n t unknown ( ) { 0 ; // Works around ROSE bug return 0; } i n t main ( i n t , c h a r ∗ ∗ ) { foo ( ) ; return 0; }

Figure 35.2: Example source code used as input to program to the Partial Redundancy Elimination (PRE) transformation.

35.3. FINAL CODE AFTER PRE TRANSFORMATION

35.3

Final Code After PRE Transformation

Figure 35.3 shows the results from the use of PRE on an the example input code.

257

258

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74

CHAPTER 35. PARTIAL REDUNDANCY ELIMINATION (PRE)

// Program , b a s e d on example i n Knoop e t a l ( ” Optimal c o d e motion : t h e o r y and // p r a c t i c e ” , ACM TOPLAS 1 6 ( 4 ) , 1 9 9 4 , pp . 1117 −1155 , a s c i t e d i n P a l e r i e t a l // ( s e e p r e . C ) ) , c o n v e r t e d t o C++ // ROSE bug : i n c l u d i n g body ” r e t u r n 0 ; ” h e r e doesn ’ t work i n t unknown ( ) ; void foo () { // P a r t i a l redundancy e l i m i n a t i o n : c a c h e v a r 1 i s a c a c h e o f a + b int cachevar 1 ; int a ; int b; int c ; int x ; int y ; int z ; i n t w; i f ( ( unknown ( ) ) ) { y = a + b; a = c; // Added by J e r e m i a h W i l l c o c k t o t e s t l o c a l PRE w = a + b; a = b; cachevar 1 = a + b ; x = cachevar 1 ; w = cachevar 1 ; a = c; // End o f added p a r t x = a + b; } else { } i f ( ( unknown ( ) ) ) { cachevar 1 = a + b ; w h i l e ( ( unknown ( ) ) ) { y = cachevar 1 ; } } e l s e i f ( ( unknown ( ) ) ) { w h i l e ( ( unknown ( ) ) ) { } // FIXME : t h e PRE c o d e c r a s h e s i f t h i s i s n ’ t i n a b l o c k i f ( ( unknown ( ) ) ) { cachevar 1 = a + b ; y = cachevar 1 ; } else { g o t o L9 ; } } else { g o t o L10 ; } z = cachevar 1 ; a = c; L9 : x = a + b; // ROSE bug : u s i n g r e t u r n ; h e r e doesn ’ t work L10 : 0; } i n t unknown ( ) { // Works around ROSE bug 0; return 0; } i n t main ( i n t { foo ( ) ; return 0; }

, char ∗∗)

Figure 35.3: Output of input code after Partial Redundancy Elimination (PRE) transformation.

Chapter 36

Calling the Inliner Figure 36.1 shows an example of how to use the inline mechanism. This chapter presents an example translator to to inlining of function calls where they are called. Such transformations are quite complex in a number of cases (one case is shown in the input code; a function call in a for loop conditional test). The details of functionality are hidden from the user and a high level interface is provided.

36.1

Source Code for Inliner

Figure 36.1 shows an example translator which calls the inliner mechanism. The code is designed to only inline up to ten functions. the list of function calls is recomputed after any function call is successfully inlined. The input code is shown in figure 36.2, the output of this code is shown in figure 36.3.

36.2

Input to Demonstrate Function Inlining

Figure 36.2 shows the example input used for demonstration of an inlining transformation.

36.3

Final Code After Function Inlining

Figure 36.3 shows the results from the inlining of three function calls. The first two function calls are the same, and trivial. The second function call appears in the test of a for loop and is more complex.

259

260

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

CHAPTER 36. CALLING THE INLINER

// Example d e m o n s t r a t i n g f u n c t i o n

i n l i n i n g ( maximal i n l i n i n g , up t o p r e s e t number o f

inlinings ).

#i n c l u d e ” r o s e . h” u s i n g namespace s t d ; // T h i s i s a f u n c t i o n i n Qing ’ s AST i n t e r f a c e v o i d F i x S g P r o j e c t ( S g P r o j e c t& p r o j ) ; i n t main ( i n t a r g c , c h a r ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . S e e Rose : : i n i t i a l i z e ROSE INITIALIZE ; // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // DQ ( 7 / 2 0 / 2 0 0 4 ) : Added i n t e r n a l c o n s i s t a n c y AstTests : : runAllTests ( p r o j e c t ) ;

t e s t s on AST

b o o l modifiedAST = t r u e ; int count = 0; // I n l i n e one c a l l a t a t i m e u n t i l do { modifiedAST = f a l s e ;

a l l have been i n l i n e d .

Loops on r e c u r s i v e c o d e .

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) // Loop o v e r a l l f u n c t i o n c a l l s // f o r ( l i s t : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; i != f u n c t i o n C a l l L i s t . end ( ) ; R o s e S T L C o n t a i n e r : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; w h i l e ( modifiedAST == f a l s e && i != f u n c t i o n C a l l L i s t . end ( ) ) { SgFunctionCallExp ∗ f u n c t i o n C a l l = isSgFunctionCallExp (∗ i ) ; ROSE ASSERT( f u n c t i o n C a l l != NULL ) ; #i f 0 f u n c t i o n C a l l −> g e t f i l e i n f o ()−> d i s p l a y ( ” i n l i n i n g

f unc t ion at fu nc ti on

call ”);

#e n d i f #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

#e n d i f // Not a l l f u n c t i o n c a l l s can be i n l i n e d i n C++, s o r e p o r t bool s u c e s s f u l l y I n l i n e d = doInline ( functionCall ) ;

if

successful .

#i f 0 p r i n t f ( ” s u c e s s f u l l y I n l i n e d = %s \n ” , s u c e s s f u l l y I n l i n e d ? ” t r u e ” : ” f a l s e ” ) ; #e n d i f #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

#e n d i f if

( s u c e s s f u l l y I n l i n e d == t r u e ) { // As s o o n a s t h e AST i s m o d i f i e d recompute t h e l i s t o f f u n c t i o n // c a l l s ( and r e s t a r t t h e i t e r a t i o n s o v e r t h e m o d i f i e d l i s t ) modifiedAST = t r u e ; } else { modifiedAST = f a l s e ; }

// I n c r e m e n t t h e i ++; }

list

iterator

// Q u i t e when we have c e a s e d t o do any i n l i n e t r a n s f o r m a t i o n s // and o n l y do a p r e d e f i n e d number o f i n l i n e t r a n s f o r m a t i o n s c o u n t++; } w h i l e ( modifiedAST == t r u e && c o u n t < 1 0 ) ; // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g checkTransformedFlagsVisitor ( project ) ;

consistancy .

// C a l l f u n c t i o n t o p o s t p r o c e s s t h e AST and f i x u p symbol t a b l e s FixSgProject (∗ p r o j e c t ) ; #i f 0 // DQ ( 4 / 6 / 2 0 1 5 ) : Adding c h e c k f o r i s T r a n s f o r m e d f l a g c o n s i s t a n c y . checkTransformedFlagsVisitor ( project ) ;

i ++)

36.3. FINAL CODE AFTER FUNCTION INLINING

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

261

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah ’ s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t ”x” v o i d incrementX ( ) { x++; } int foo () { int a = 0; while ( a < 5) { ++a ; } return a + 3; } i n t main ( i n t , c h a r ∗ ∗ ) { // Two t r i v a l f u n c t i o n incrementX ( ) ; incrementX ( ) ;

calls

to i n l i n e

// Something more i n t e r e s t i n g t o i n l i n e // f o r ( ; f o o ( ) < 7 ; ) int i = 0; f o r ( ; f o o ( ) < 7 ; i ++) { x++; } return x ; }

Figure 36.2: Example source code used as input to program to the inlining transformation.

262

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

CHAPTER 36. CALLING THE INLINER

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah ’ s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t ”x” v o i d incrementX ( ) { x++; } int foo () { int a 0 = 0; while ( a 0 < 5){ ++a 0 ; } return a 0 + 3; } i n t main ( i n t , c h a r ∗ ∗ ) { x++; x++; // Something more i n t e r e s t i n g t o i n l i n e // f o r ( ; f o o ( ) < 7 ; ) int i 0 = 0; f o r ( ; t r u e ; i 0 ++) { int a 2 = 0; while ( a 2 < 5){ ++a 2 ; } int rose temp 7 1 = a 2 + 3; bool rose temp 3 = ( bool )( rose temp i f (! rose temp 3 ) { break ; } else { } x++; } return x ; }

7

1 < 7);

Figure 36.3: Output of input code after inlining transformations.

Chapter 37

Using the AST Outliner Outlining is the process of replacing a block of consecutive statements with a function call to a new function containing those statements. Conceptually, outlining the inverse of inlining (Chapter 36). This chapter shows how to use the basic experimental outliner implementation included in the ROSE projects directory. There are two basic ways to use the outliner. The first is a “user-level” method, in which you may use a special pragma to mark outline targets in the input program, and then call a high-level driver routine to process these pragmas. You can also use command line option to specify outlining targets using abstract handle strings (detailed in Chapter 46). The second method is to call “low-level” outlining routines that operate directly on AST nodes. After a brief example of what the outliner can do and a discussion of its limits (Sections 37.1–37.2), we discuss each of these methods in Sections 37.3 and 37.5, respectively.

37.1

An Outlining Example

Figure 37.1 shows a small program with a pragma marking the outline target, a nested for loop, and Figure 37.2 shows the result. The outliner extracts the loop and inserts it into the body of a new function, and inserts a call to that function. The outlined code’s input and output variables are wrapped up as parameters to this function. We make the following observations about this output. Placement and forward declarations. The function itself is placed, by default, at the end of the input file to guarantee that it has access to all of the same declarations that were available at the outline target site. The outliner inserts any necessary forward declarations as well, including any necessary friend declarations if the outline target appeared in a class member function. Calling convention. The outliner generates a C-callable function (extern ‘‘C’’, with pointer arguments). This design choice is motivated by our need to use the outliner to extract code into external, dynamically loadable library modules. 263

264

37.2

CHAPTER 37. USING THE AST OUTLINER

Limitations of the Outliner

The main limitation of the outliner implementation is that it can only outline single SgStatement nodes. However, since an SgStatement node may be a block (i.e., an SgBasicBlock node), a “single statement” may actually comprise a sequence of complex statements. The rationale for restricting to single SgStatement nodes is to avoid subtly changing the program’s semantics when outlining code. Consider the following example, in which we wish to outline the middle 3 lines of executable code. 2 4 6

int x = 5 ; // START o u t l i n i n g h e r e . foo (x ) ; Object y ( x ) ; y . foo ( ) ; // STOP o u t l i n i n g h e r e . y . bar ( ) ; This example raises a number of issues. How should an outliner handle the declaration of y, which constructs an object in local scope? It cannot just cut-and-paste the declaration of y to the body of the new outlined function because that will change its scope and lifetime, rendering the call to y.bar() impossible. Additionally, it may be unsafe to move the declaration of y so that it precedes the outlined region because the constructor call may have side-effects that could affect the execution of foo(x). It is possible to heap-allocate y inside the body of the

5

10

15

20

25

namespace N { class A { i n t f o o ( void ) const { return 7 ; } i n t b a r ( void ) const { return f o o ( ) / 2 ; } public : i n t b i z ( void ) const { int r e s u l t = 0 ; #pragma r o s e o u t l i n e f o r ( i n t i = 1 ; i <= f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= b a r ( ) ; j ++) r e s u l t += i ∗ j ; return r e s u l t ; } }; } extern ”C” i n t p r i n t f ( const char ∗ fmt , i n t main ( ) { N: :A x ; p r i n t f ( ”%d\n” , x . b i z return 0 ; }

( ) ) ; // P r i n t s

...);

’168 ’

Figure 37.1: inputCode OutlineLoop.cc: Sample input program. The #pragma directive marks the nested for loop for outlining.

37.2. LIMITATIONS OF THE OUTLINER

s t a t i c void O U T namespace N { 5

10

15

20

25

30

1

( int ∗ r e s u l t p

12969

class A { public : f r i e n d void : : O U T

1

265 , const void ∗ t h i s

12969 ( int ∗ r e s u l t p , const void ∗ t h i s p t r

p

ptr

p

);

);

private : i n l i n e i n t f o o ( ) const { return 7 ; } i n l i n e i n t b ar ( ) const { return ( t h i s)−> f o o ( ) / 2 ; } public : i n l i n e i n t b i z ( ) const { // //A d e c l a r a t i o n f o r t h i s p o i n t e r = this ; const c l a s s A ∗ t h i s p t r int r e s u l t = 0 ; (& r e s u l t , & t h i s p t r OUT 1 12969 return r e s u l t ; } } ; } extern ”C” { i n t p r i n t f ( const char ∗ fmt , . . . ) ; }

);

35

40

45

50

i n t main ( ) { class N: :A x ; // P r i n t s ’ 1 6 8 ’ p r i n t f ( ”%d\n” , ( x . b i z return 0 ; }

()));

( int ∗ r e s u l t p , const void ∗ t h i s p t r p s t a t i c void O U T 1 1 2 9 6 9 { ); i n t &r e s u l t = ∗ ( ( i n t ∗ ) r e s u l t p = ∗ ( ( const c l a s s N : : A ∗ ∗ ) t h i s p t r const c l a s s N : : A ∗ & t h i s p t r f o r ( i n t i = 1 ; i <= t h i s p t r −>f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= t h i s p t r −>b a r ( ) ; j ++) r e s u l t += i ∗ j ; }

)

p

);

Figure 37.2: rose outlined-inputCode OutlineLoop.cc: The nested for loop of Figure 37.1 has been outlined.

outlined function so that it can be returned to the caller and later freed, but it is not clear if changing y from a stack-allocated variable to a heap-allocated one will always be acceptable, particularly if the developer of the original program has, for instance, implemented a customized memory allocator. Restricting outlining to well-defined SgStatement objects avoids these issues. It is possible to build a “higher-level” outliner that extends the outliner’s basic infrastructure to handle these and other issues.

266

CHAPTER 37. USING THE AST OUTLINER

The outliner cannot outline all possible SgStatement nodes. However, the outliner interface provides a routine, outliner::isOutlineable(s), for testing whether an SgStatement object s is known to satisfy the outliner’s preconditions (see Section 37.5 for details).

37.3

User-Directed Outlining via Pragmas

Figure 37.3 shows the basic translator, outline, that produces Figure 37.2 from Figure 37.1. This translator extends the identity translator with an include directive on line 5 of Figure 37.3, and a call to the outliner on line 16. All outliner routines live in the Outliner namespace. Here, the call to Outliner::outlineAll (proj) on line 16 traverses the AST, looks for #pragma rose outline directives, outlines the SgStatement objects to which each pragma is attached, and returns the number of outlined objects. A slightly lower-level outlining primitive. The Outliner::outlineAll() routine is a wrapper around calls to a simpler routine, Outliner::outline(), that operates on pragmas: Outliner : : Result

O u t l i n e r : : o u t l i n e ( SgPragmaDeclaration ∗ s ) ;

Given a pragma statement AST node s, this routine checks if s is a rose outline directive, and if so, outlines the statement with which s is associated. It returns a Outliner::Result object, which is simply a structure that points to (a) the newly generated outlined function and (b) the statement corresponding to the new function call (i.e., the outlined function call-site). See Outliner.hh or the ROSE Programmer’s Reference for more details. The Outliner::outlineAll() wrapper. The advantage of using the wrapper instead of the lower-level primitive is that the wrapper processes the pragmas in an order that ensures the outlining can be performed correctly in-place. This order is neither a preorder nor a postorder traversal, but in fact a “reverse” preorder traversal; refer to the wrapper’s documentation for an explanation.

37.4

Outlining via Abstract Handles

The ROSE AST outliner also allows users to specify outlining targets using abstract handles (details are given in Chapter 46) without relying on planting pragmas into the source code. For the translator (e.g. named outline) built from the source shown in Figure 37.3, it accepts a command line option in a form of -rose:outline:abstract handle handle string. The outline program is able to locate a language construct matching the handle string within an input source file and then outline the construct. For example, a handle string ”ForStatement” will tell the outliner to outline the for loop at source position line 12. Another handle, ”FunctionDeclaration::ForStatement” indicates that the outlining target is the second loop within a function named initializer. Figure 37.5 shows the outlining results using the first handle(”ForStatement”) from an input source file (shown in Figure 37.4). Figure 37.6 shows the results using the second handle string for the same input.

37.5. CALLING OUTLINER DIRECTLY ON AST NODES

37.5

267

Calling Outliner Directly on AST Nodes

The preceding examples rely on the outliner’s #pragma interface to identify outline targets. In this section, we show how to call the outliner directly on SgStatement nodes from within your translator. Figure 37.7 shows an example translator that finds all if statements and outlines them. A sample input appears in Figure 37.8, with the corresponding output shown in Figure 37.9. Notice that valid preprocessor control structure is accounted for and preserved in the output. The translator has two distinct phases. The first phase selects all outlineable if-statements, using the CollectOutlineableIfs helper class. This class produces a list that stores the targets in an order appropriate for outlining them in-place. The second phase iterates over the list of statements and outlines each one. The rest of this section explains these phases, as well as various aspects of the sample input and output.

37.5.1

Selecting the outlineable if statements

Line 45 of Figure 37.7 builds a list, ifs (declared on line 44), of outlineable if-statements. The helper class, CollectOutlineableIfs in lines 12–35, implements a traversal to build this list. Notice that a node is inserted into the target list only if it satisfies the outliner’s preconditions; this check is the call to Outliner::isOutlineable() on line 28. The function Outliner::isOutlineable() also accepts an optional second boolean parameter (not shown). When this parameter is true and the statement cannot be outlined, the check will print an explanatory message to standard error. Such messages are useful for discovering why the outliner will not outline a particular statement. The default value of this parameter is false.

37.5.2

Properly ordering statements for in-place outlining

Each call to Outliner::outline(*i) on line 50 of Figure 37.7 outlines a target if-statement *i in if targets. However, in order for these statements to be outlined in-place, it is essential to outline the statements in the proper order. The postorder traversal implemented by the helper class, CollectOutlineableIfs, produces the correct ordering. To see why, consider the following example code: 2 4 6 8

i f ( a ) // [ 1 ] { i f ( b ) f o o ( ) ; // [ 2 ] } e l s e i f ( c ) // [ 3 ] { i f ( d ) bar ( ) ; // [ 4 ] } The corresponding AST is (roughly) / /

SgIfStmt:[1] \ \

268

CHAPTER 37. USING THE AST OUTLINER

SgIfStmt:[2]

SgIfStmt:[3] | SgIfStmt:[4]

The postorder traversal—2, 4, 3, 1—ensures that child if-statements are outlined before their parents.

37.6

Outliner’s Preprocessing Phase

Internally, the outliner implementation itself has two distinct phases. The first is a preprocessing phase, in which an arbitrary outlineable target is placed into a canonical form that is relatively simple to extract. The second phase then creates the outlined function, replacing the original target with a call to the outlined function. It is possible to run just the preprocessing phase, which is useful for understanding or even debugging the outliner implementation. To call just the preprocessor, simply replace a call to Outliner::outlineAll(s) or Outliner::outline(s) with a call to Outliner::preprocessAll(s) or Outliner::preprocess(s), respectively. The translator in Figure 37.10 modifies the translator in Figure 37.3 in this way to create a preprocessingonly translator. The preprocessing phase consists of a sequence of initial analyses and transformations that the outliner performs in order to put the outline target into a particular canonical form. Roughly speaking, this form is an enclosing SgBasicBlock node, possibly preceded or followed by start-up and tear-down code. Running just the preprocessing phase on Figure 37.1 produces the output in Figure 37.11. In this example, the original loop is now enclosed in two additional SgBasicBlocks (Figure 37.11, lines 24–35), the outermost of which contains a declaration that shadows the object’s this pointer, replacing all local references to this with the new shadow pointer. In this case, this initial transformation is used by the main underlying outliner implementation to explicitly identify all references to the possibly implicit references to this. The preprocessing phase is more interesting in the presence of non-local control flow outside the outline target. Consider Figure 37.12, in which the outline target contains two break statements, which require jumping to a regions of code outside the target. We show the preprocessed code in Figure 37.13. The original non-local jumps are first transformed into assignments to a flag, EXIT TAKEN (lines 18–20 and 26–29), and then relocated to a subsequent block of code (lines 38–53) with their execution controlled by the value of the flag. The final outlined result appears in Figure 37.14; the initial preprocessing simplifies this final step of extracting the outline target.

37.6. OUTLINER’S PREPROCESSING PHASE

269

// The ” p u r p o s e ” as i t a p p e a r s i n t h e man page , u n c a p i t a l i z e d and a s i n g l e , s h o r t , s t a t i c const char ∗ p u r p o s e = ” T h i s t o o l o u t l i n e s c o d e s e g m e n t s t o f u n c t i o n s ” ;

5

10

15

line .

s t a t i c const char ∗ d e s c r i p t i o n = ” Outlining i s the p r o c e s s of r e p l a c i n g a block of c o n s e c u t i v e ” ” s t a t e m e n t s w i t h a f u n c t i o n c a l l t o a new f u n c t i o n c o n t a i n i n g t h o s e s t a t e m e n t s . ” ” Conceptually , o u t l i n i n g i s the i n v e r s e of i n l i n i n g . ” ; #include #include #include #include



// // // //

must from from from

be f i r s t ROSE i n c l u d e ROSE ROSE ROSE

#include #include #include using namespace Rose ; using namespace Rose : : D i a g n o s t i c s ;

// t h e ROSE team i s m i g r a t i n g e v e r y t h i n g t o t h i s namespace // f o r mlog , INFO, WARN, ERROR, e t c .

D i a g n o s t i c s : : F a c i l i t y mlog ;

// most ROSE t o o l s use a d i a g n o s t i c s l o g g i n g

20 facility

25

// S w i t c h e s f o r t h i s t o o l . Tools w i t h l o t s o f s w i t c h e s w i l l p r o b a b l y want t h e s e t o be i n some S e t t i n g s s t r u c t m i r r o r i n g t h // approach used by some a n a l y s e s t h a t have l o t s o f s e t t i n g s . So we ’ l l do t h a t h e r e t o o even t h o u g h i t l o o k s funny . struct S e t t i n g s { bool s h o w O u t l i n e r S e t t i n g s ; // s h o u l d we show t h e o u t l i n e r s e t t i n g s i n s t e a d o f running i t ? bool u s e O l d P a r s e r ; // c a l l t h e o l d O u t l i n e r command−l i n e p a r s e r

30

Settings () : s h o w O u t l i n e r S e t t i n g s ( f a l s e ) , u s e O l d P a r s e r ( f a l s e ) {} } settings ;

35

// Parse t h e t o o l ’ s command−l i n e , p r o c e s s i n g o n l y t h o s e s w i t c h e s r e c o g n i z e d by Sawyer . Then r e t u r n t h e non−p a r s e d s w i t c h e s // t h e n e x t s t a g e o f p a r s i n g . We have t h r e e more s t a g e s t h a t need t o p r o c e s s t h e command−l i n e : O u t l i n e r ( t h e o l d approach // f r o n t e n d ( ) , and t h e backend c o m p i l e r . None o f t h e s e e x c e p t t h e backend c o m p i l e r can i s s u e e r r o r messages a b o u t m i s s p e l // s w i t c h e s b e c a u s e t h e f i r s t t h r e e must assume t h a t an u n r e c o g n i z e d s w i t c h i s i n t e n d e d f o r a l a t e r s t a g e . s t a t i c s t d : : v e c t o r parseCommandLine ( i n t a r g c , char ∗ a r g v [ ] ) { using namespace Sawyer : : CommandLine ;

40

45

// Use Rose : : CommandLine t o c r e a t e a c o n s i s t e n t p a r s e r among a l l t o o l s . I f you want a t o o l ’ s p a r s e r t o be d i f f e r e n t // t h e n e i t h e r c r e a t e one y o u r s e l f , or modify t h e p a r s e r p r o p e r t i e s a f t e r c r e a t e P a r s e r r e t u r n s . The c r e a t e E m p t y P a r s e r // c r e a t e s a p a r s e r t h a t assumes a l l u n r e c o g n i z e d s w i t c h e s a r e i n t e n d e d f o r a l a t e r s t a g e . I f t h e r e ar e no l a t e r s t a g e // t h e n use createEmptyParser i n s t e a d or e l s e u s e r s w i l l n e v e r s e e e r r o r messages f o r m i s s p e l l e d s w i t c h e s . P a r s e r p = Rose : : CommandLine : : c r e a t e E m p t y P a r s e r S t a g e ( p u r p o s e , d e s c r i p t i o n ) ; p . doc ( ” S y n o p s i s ” , ” @prop{programName} @v{ s w i t c h e s } @v{ f i l e s } . . . ” ) ; #i f 1 // DEBUGGING [ Robb P Matzke 2016−09−27] p . l o n g P r e f i x ( ”−” ) ; #e n d i f

50

// User e r r o r s ( what few w i l l be r e p o r t e d s i n c e t h i s i s o n l y a f i r s t −s t a g e p a r s e r ) s h o u l d be s e n t t o s t a n d a r d e r r o r i n // o f r a i s i n g an e x c e p t i o n . Programmer e r r o r s s t i l l c a u s e e x c e p t i o n s . p . e r r o r S t r e a m ( : : mlog [FATAL ] ) ; 55

60

65

70

// A l l ROSE t o o l s have some s w i t c h e s i n common , such as −−v e r s i o n , −V, −−h e l p , −h , −?, −−l o g , −L , −−t h r e a d s , e t c . We // i n c l u d e them f i r s t so t h e y appear near t h e t o p o f t h e documentation . The g e n e r i c S w i t c h e s r e t u r n s a // Sawyer : : CommandLine : : SwitchGroup , which t h i s t o o l c o u l d e x t e n d by a d d i n g a d d i t i o n a l s w i t c h e s . This c o u l d have been done // i n s i d e c r e a t e P a r s e r , b u t i t t u r n s o u t t h a t many t o o l s l i k e t o e x t e n d or re−o r d e r t h i s group o f s w i t c h e s , which i s // s i m p l e r t h i s way . p . w i t h ( Rose : : CommandLine : : g e n e r i c S w i t c h e s ( ) ) ;

// E v e n t u a l l y , i f we change f r o n t e n d so we can q u e r y what s w i t c h e s i t knows about , we c o u l d i n s e r t them i n t o our p a r s // t h i s p o i n t . The f r o n t e n d c o u l d r e p o r t a l l known s w i t c h e s ( s o r t o f how t h i n g s a r e o r g a n i z e d one ) or we c o u l d q u e r y // t h o s e g r o u p s o f f r o n t e n d s w i t c h e s t h a t t h i s t o o l i s i n t e r e s t e d i n ( e . g . , I don ’ t know i f t h e o u t l i n e r needs Fortran // s w i t c h e s ) . #i f 0 // [ Robb P Matzke 2016−09−27] p . w i t h ( C o m m a n d l i n e P r o c e s s i n g : : f r o n t e n d A l l S w i t c h e s ( ) ) ; // or s i m i l a r #e n d i f // The O u t l i n e r has some s w i t c h e s o f i t s own , so i n c l u d e them n e x t . These s w i t c h e s w i l l a u t o m a t i c a l l y a d j u s t t h e O u t l i n e r // s e t t i n g s . S i n c e t h e o u t l i n e r i s implemented as a namespace r a t h e r than a c l a s s , There can // be o n l y one i n s t a n c e o f an o u t l i n e r p e r t o o l , w h e t h e r t h e t o o l u s e s an o u t l i n e r // as p a r t o f some o t h e r a n a l y s i s . p . w i t h ( O u t l i n e r : : commandLineSwitches ( ) ) ;

it ’s essentially a singlton . directly ( like

t h i s one ) or i n d i r e

75 // F i n a l l y , a t o o l sometimes has i t s own s p e c i f i c SwitchGroup t o o l ( ” Tool− s p e c i f i c s w i t c h e s ” ) ;

s e t t i n g s , so we demo t h a t h e r e w i t h a c o u p l e made−up s w i t c h e s .

270

5

10

15

20

CHAPTER 37. USING THE AST OUTLINER

#d e f i n e MSIZE 500 i n t n ,m, m i t s ; double t o l , r e l a x = 1 . 0 , a l p h a = 0 . 0 5 4 3 ; double u [ MSIZE ] [ MSIZE ] , f [ MSIZE ] [ MSIZE ] , u o l d [ MSIZE ] [ MSIZE ] ; double dx , dy ; void i n i t i a l i z e ( ) { i n t i , j , xx , yy ; dx = 2 . 0 / ( n − 1 ) ; dy = 2 . 0 / (m− 1 ) ; f o r ( i =0; i
Figure 37.4: inputCode OutlineLoop2.c: Sample input program without pragmas.

37.6. OUTLINER’S PREPROCESSING PHASE

5

10

15

20

#d e f i n e MSIZE 500 int n ; i n t m; int mits ; double t o l ; double r e l a x = 1 . 0 ; double a l p h a = 0 . 0 5 4 3 ; double u [ 5 0 0 ] [ 5 0 0 ] ; double f [ 5 0 0 ] [ 5 0 0 ] ; double u o l d [ 5 0 0 ] [ 5 0 0 ] ; double dx ; double dy ; s t a t i c void O U T 1 1 0 9 0 5

( int ∗ i p

271

, int ∗ j p

, int ∗ xxp

, int ∗ yyp

);

, int ∗ j p

, int ∗ xxp

, int ∗ yyp

)

void i n i t i a l i z e ( ) { int i ; int j ; i n t xx ; i n t yy ; dx = 2 . 0 / ( n − 1 ) ; dy = 2 . 0 / (m − 1 ) ; O U T 1 1 0 9 0 5 (& i ,& j ,&xx ,& yy ) ; }

25

30

35

s t a t i c void O U T 1 1 0 9 0 5 { = 0; ∗ip for ( ∗ i p for ( ∗ j p = 0; ∗jp ∗ xxp = ( ( i n t )( − = ( ( i n t )( − ∗ yyp u[ ∗ip ][ ∗jp ] = ][ ∗jp ] = f [ ∗ip ∗ xxp )) ∗ (1.0 − ( ∗ yyp ∗ xxp )) − 2.0 ∗ (1.0 − ( } }

( int ∗ i p

< n ; ( ∗ i p )++) < m; ( ∗ j p )++) { 1 . 0 + dx ∗ ( ∗ i p − 1 . 0 + dy ∗ ( ∗ j p − 0.0; − 1.0 ∗ alpha ∗ ( 1 . 0 ∗ ∗ yyp )) − 2.0 ∗ ∗ yyp ∗ ∗ yyp ) ) ;

1))); 1))); − ( ∗ xxp ∗ (1.0 − ( ∗ xxp



Figure 37.5: rose inputCode OutlineLoop2.c: The loop at line 12 of Figure 37.12 has been outlined.

272

5

10

15

20

25

30

35

#d e f i n e MSIZE 500 int n ; i n t m; int mits ; double t o l ; double r e l a x = 1 . 0 ; double a l p h a = 0 . 0 5 4 3 ; double u [ 5 0 0 ] [ 5 0 0 ] ; double f [ 5 0 0 ] [ 5 0 0 ] ; double u o l d [ 5 0 0 ] [ 5 0 0 ] ; double dx ; double dy ; s t a t i c void O U T 1 1 0 9 0 5

CHAPTER 37. USING THE AST OUTLINER

( int i , int ∗ j p

, int ∗ xxp

, int ∗ yyp

);

void i n i t i a l i z e ( ) { int i ; int j ; i n t xx ; i n t yy ; dx = 2 . 0 / ( n − 1 ) ; dy = 2 . 0 / (m − 1 ) ; f o r ( i = 0 ; i < n ; i ++) O U T 1 1 0 9 0 5 ( i ,& j ,&xx ,& yy ) ; } s t a t i c void O U T 1 1 0 9 0 5 ( i n t i , i n t ∗ j p , int ∗ xxp , int ∗ yyp ) { = 0; ∗jp < m; ( ∗ j p )++) { for ( ∗ j p ∗ xxp = ( ( i n t )( − 1 . 0 + dx ∗ ( i − 1 ) ) ) ; = ( ( i n t )( − 1 . 0 + dy ∗ ( ∗ j p − 1))); ∗ yyp u[ i ][ ∗jp ] = 0.0; ] = − 1.0 ∗ alpha ∗ ( 1 . 0 − ( ∗ xxp ∗ ∗ xxp )) ∗ (1.0 − ( ∗ yyp ∗ f [ i ][ ∗jp ∗ yyp )) − 2.0 ∗ (1.0 − ( ∗ xxp ∗ ∗ xxp )) − 2.0 ∗ (1.0 − ( ∗ yyp ∗ ∗ yyp ) ) ; } }

Figure 37.6: rose inputCode OutlineLoop2b.c: The 2nd loop within a function named initializefrom Figure 37.12 has been outlined.

37.6. OUTLINER’S PREPROCESSING PHASE

5

// o u t l i n e I f s . cc : C a l l s O u t l i n e r d i r e c t l y t o o u t l i n e #include #include #include #include < l i s t >

273

if

statements .

#include using namespace s t d ; 10

15

// T r a v e r s a l t o g a t h e r a l l o u t l i n e a b l e S g I f S t m t nodes . c l a s s C o l l e c t O u t l i n e a b l e I f s : public A s t S i m p l e P r o c e s s i n g { public : // Container o f l i s t s t a t e m e n t s i n ‘ ‘ o u t l i n e a b l e ’ ’ o r d e r . typedef l i s t I f L i s t t ; // C a l l t h i s r o u t i n e t o g a t h e r t h e o u t l i n e t a r g e t s . s t a t i c void c o l l e c t ( S g P r o j e c t ∗ p , I f L i s t t & f i n a l ) { CollectOutlineableIfs collector ( final ); c o l l e c t o r . traverseInputFiles (p , postorder ) ; }

20

25

v i r t u a l void v i s i t ( SgNode∗ n ) { SgIfStmt ∗ s = isSgIfStmt (n ) ; i f ( Outliner : : isOutlineable ( s )) f i n a l t a r g e t s . push back ( s ) ; }

30

35

40

private : ( f i n a l ) {} C o l l e c t O u t l i n e a b l e I f s ( I f L i s t t& f i n a l ) : f i n a l t a r g e t s I f L i s t t & f i n a l t a r g e t s ; // F i n a l l i s t o f o u t l i n e t a r g e t s . }; //=================================================================== i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j ) ;

45 #i f 1 // B u i l d a s e t o f o u t l i n e a b l e i f s t a t e m e n t s . CollectOutlineableIfs : : IfList t i f s ; C o l l e c t O u t l i n e a b l e I f s : : c o l l e c t ( proj , i f s ) ; 50

55

// O u t l i n e them a l l . for ( C o l l e c t O u t l i n e a b l e I f s : : I f L i s t t : : i t e r a t o r i = i f s . begin ( ) ; i != i f s . end ( ) ; ++i ) Outliner : : o u t l i n e (∗ i ) ; #e l s e p r i n t f ( ” S k i p p i n g o u t l i n i n g due t o r e c e n t move from s t d : : l i s t t o s t d : : v e c t o r i n ROSE \n” ) ; #e n d i f // Unparse return backend ( p r o j ) ;

60 }

Figure 37.7: outlineIfs.cc: A lower-level outlining translator, which calls Outliner::outline() directly on SgStatement nodes. This particular translator outlines all SgIfStmt nodes.

274

CHAPTER 37. USING THE AST OUTLINER

#include using namespace s t d ; 5

10

15

20

25

30

35

#d e f i n e LEAP YEAR 0 i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { f o r ( i n t i = 1 ; i < a r g c ; ++i ) { s t r i n g month ( a r g v [ i ] ) ; s i z e t days = 0 ; i f ( month == ” January ” | | month == ”March” | | month == ”May” | | month == ” J u l y ” | | month == ” August ” | | month == ” Oc t ob e r ” | | month == ” December ” ) days = 3 1 ; #i f LEAP YEAR e l s e i f ( month == ” Fe b r ua r y ” ) days = 2 9 ; #e l s e e l s e i f ( month == ” Fe b r ua r y ” ) days = 2 8 ; #e n d i f e l s e i f ( month == ” A p r i l ” | | month == ” June ” | | month == ” September ” | | month == ” November ” ) days = 3 0 ; c o u t << a r g v [ i ] << ” ” << days << e n d l ; } return 0 ; }

Figure 37.8: inputCode Ifs.cc: Sample input program, without explicit outline targets specified using #pragma rose outline, as in Figures 37.1 and 37.12.

37.6. OUTLINER’S PREPROCESSING PHASE

5

10

15

20

25

30

35

40

45

50

#include using namespace s t d ; #d e f i n e LEAP YEAR 0 s t a t i c void O U T 1 1 0 0 9 8 s t a t i c void O U T 2 1 0 0 9 8 s t a t i c void O U T 3 1 0 0 9 8

( void ∗ monthp ( void ∗ monthp ( void ∗ monthp

, s i z e t ∗ daysp , s i z e t ∗ daysp , s i z e t ∗ daysp

275

); ); );

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { f o r ( i n t i = 1 ; i < a r g c ; ++i ) { s t d : : s t r i n g month ( a r g v [ i ] ) ; s i z e t days = 0 ; O U T 3 1 0 0 9 8 (&month ,& days ) ; ( ( ( cout<

)

s t a t i c void O U T 3 1 0 0 9 8 ( void ∗ monthp , s i z e t ∗ d a y s p ) { s t d : : s t r i n g &month = ∗ ( ( s t d : : s t r i n g ∗ ) monthp ) ; s i z e t &days = ∗ ( ( s i z e t ∗ ) d a y s p ) ; i f ( month==” January ” | | month==”March” | | month==”May” | | month==” J u l y ” | | month==” August ” | | month==” October ” | | month= days = 3 1 ; else #i f LEAP YEAR #e l s e { #e n d i f /∗ #i f LEAP YEAR . . . #e l s e ∗/ O U T 2 1 0 0 9 8 (&month ,& days ) ; } }

Figure 37.9: rose inputCode Ifs.cc: Figure 37.8, after outlining using the translator in Figure 37.7.

276

CHAPTER 37. USING THE AST OUTLINER

// o u t l i n e P r e p r o c . cc : Shows t h e o u t l i n e r ’ s p r e p r o c e s s o r −o n l y phase . #include #include 5

#include using namespace s t d ;

10

int main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j ) ;

15

20

#i f 1 c e r r << ” [ Running o u t l i n e r ’ s p r e p r o c e s s i n g p h a s e o n l y . . . ] ” << e n d l ; s i z e t count = O u t l i n e r : : p r e p r o c e s s A l l ( p r o j ) ; c e r r << ” [ P r o c e s s e d ” << c o u n t << ” o u t l i n e d i r e c t i v e s . ] ” << e n d l ; #e l s e p r i n t f ( ” S k i p p i n g o u t l i n i n g due t o r e c e n t move from s t d : : l i s t t o s t d : : v e c t o r i n ROSE \n” ) ; #e n d i f

25 c e r r << ” [ U n p a r s i n g . . . ] ” << e n d l ; return backend ( p r o j ) ; }

Figure 37.10: outlinePreproc.cc: The basic translator of Figure 37.3, modified to execute the Outliner’s preprocessing phase only. In particular, the original call to Outliner::outlineAll() has been replaced by a call to Outliner::preprocessAll().

37.6. OUTLINER’S PREPROCESSING PHASE

277

namespace N {

5

class A {

10

private : i n l i n e i n t f o o ( ) const { return 7 ; }

15

i n l i n e i n t b ar ( ) const { return ( t h i s)−> f o o ( ) / 2 ; }

20

25

30

35

40

45

public : i n l i n e i n t b i z ( ) const { // //A d e c l a r a t i o n f o r t h i s p o i n t e r = this ; const c l a s s A ∗ t h i s p t r int r e s u l t = 0 ; #pragma r o s e o u t l i n e { f o r ( i n t i = 1 ; i <= t h i s p t r −>f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= t h i s p t r −>b a r ( ) ; j ++) r e s u l t += i ∗ j ; } return r e s u l t ; } } ; } extern ”C” { i n t p r i n t f ( const char ∗ fmt , } i n t main ( ) { class N: :A x ; // P r i n t s ’ 1 6 8 ’ p r i n t f ( ”%d\n” , ( x . b i z return 0 ; }

...);

()));

Figure 37.11: rose outlined pp-inputCode OutlineLoop.cc: Figure 37.1 after outline preprocessing only, i.e., specifying -rose:outline:preproc-only as an option to the translator of Figure 37.3.

278

CHAPTER 37. USING THE AST OUTLINER

#include

5

10

15

20

s i z e t f a c t o r i a l ( s i z e t n) { s i z e t i = 1; s i z e t r = 1; while ( 1 ) { #pragma r o s e o u t l i n e i f ( i <= 1 ) break ; // Non−l o c a l jump #1 e l s e i f ( i >= n ) break ; // Non−l o c a l jump #2 else r ∗= ++i ; } return r ; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { s t d : : c o u t << ” 7 ! == ” << f a c t o r i a l return 0 ; }

( 7 ) << s t d : : e n d l ; // P r i n t s 5040

Figure 37.12: inputCode OutlineNonLocalJumps.cc: Sample input program, with an outlining target that contains two non-local jumps (here, break statements).

37.6. OUTLINER’S PREPROCESSING PHASE

279

#include

5

10

15

20

25

30

35

40

45

50

size t factorial { s i z e t i = 1; s i z e t r = 1; while ( 1 ) {

( s i z e t n)

#pragma r o s e o u t l i n e { i n t EXIT TAKEN = 0 ; { i f ( i <= 1 ) { EXIT TAKEN = 1 ; goto NON LOCAL EXIT ; } e l s e i f ( i >= n ) { EXIT TAKEN = 2 ; goto NON LOCAL EXIT ; } else r ∗= ++i ; NON LOCAL EXIT : ; ; } i f ( EXIT TAKEN == 1 ) { // Non−l o c a l jump #1 break ; } else { i f ( EXIT TAKEN == 2 ) { // Non−l o c a l jump #2 break ; } else { } } } } return r ; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // P r i n t s 5040 ( ( s t d : : c o u t << ” 7 ! == ” ) << f a c t o r i a l return 0 ; }

( 7 ) ) << s t d : : e n d l ;

Figure 37.13: rose outlined pp-inputCode OutlineNonLocalJumps.cc: The non-local jump example of Figure 37.12 after outliner preprocessing, but before the actual outlining. The non-local jump is handled by an additional flag, EXIT TAKEN , which indicates what non-local jump is to be taken.

280

CHAPTER 37. USING THE AST OUTLINER

#include s t a t i c void O U T 1 1 3 8 7 6

5

10

15

20

25

30

35

40

( s i z e t ∗ np , s i z e t ∗ i p i n t ∗EXIT TAKEN p ) ;

s i z e t f a c t o r i a l ( s i z e t n) { s i z e t i = 1; s i z e t r = 1; while ( 1 ) { { i n t EXIT TAKEN = 0 ; OUT 1 13876 (&n , &i , &r , &EXIT TAKEN i f ( EXIT TAKEN == 1 ) { // Non−l o c a l jump #1 break ; } else { i f ( EXIT TAKEN == 2 ) { // Non−l o c a l jump #2 break ; } else { } } } } return r ; } i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // P r i n t s 5040 ( ( s t d : : c o u t << ” 7 ! == ” ) << f a c t o r i a l return 0 ; } s t a t i c void O U T

1

13876

,

s i z e t ∗ rp

,

s i z e t ∗ rp

,

);

( 7 ) ) << s t d : : e n d l ;

( s i z e t ∗ np , s i z e t ∗ i p i n t ∗EXIT TAKEN p )

,

{ 45

50

55

60

s i z e t & n = ∗(( s i z e t ∗) np s i z e t & i = ∗(( s i z e t ∗) i p s i z e t & r = ∗(( s i z e t ∗) r p i n t &EXIT TAKEN = ∗ ( ( i n t ∗ ) i f ( i <= 1 ) { EXIT TAKEN = 1 ; goto NON LOCAL EXIT ; } e l s e i f ( i >= n ) { EXIT TAKEN = 2 ; goto NON LOCAL EXIT ; } else r ∗= ++i ; NON LOCAL EXIT : ; ; }

); ); ); EXIT TAKEN p

);

Figure 37.14: rose outlined-inputCode OutlineNonLocalJumps.cc: Figure 37.12 after outlining.

Chapter 38

Loop Optimization This section is specific to loop optimization and show several tutorial examples using the optimization mechanisms within ROSE.

38.1

Example Loop Optimizer

Simple example translator showing use of pre-defined loop optimizations. Figure 38.1 shows the code required to call some loop optimizations within ROSE. The translator that we build for this tutorial is simple and takes the following command line options to control which optimizations are done.

FIXME: We might want to reference Qing’s work explicitly since this is really just showing off here work. FIXME: We are not running performance tests within this tutorial, but perhaps we could later.

-ic1 :loop interchange for more reuses -bk1/2/3 :block outer/inner/all loops -fs1/2 :single/multi-level loop fusion for more reuses -cp :copy array -fs0 : loop fission -splitloop: loop splitting -unroll [locond] [nvar] : loop unrolling -bs : break up statements in loops -annot : Read annotation from a file which defines side effects of functions -arracc : Use special function to denote array access (the special function can be replaced with macros after transformation). This option is for circumventing complex subscript expressions for linearized multi-dimensional arrays. -opt : The level of loop optimizations to apply (By default, only the outermost level is optimized). -ta : Max number of nodes to split for transitive dependence analysis (to limit the overhead of transitive dep. analysis) -clsize : set cache line size in evaluating spatial locality (affect decisions in applying loop optimizations) -reuse_dist : set maximum distance of reuse that can exploit cache (used to evaluate 281

282

CHAPTER 38. LOOP OPTIMIZATION temporal locality of loops)

38.1. EXAMPLE LOOP OPTIMIZER

2

// LoopProcessor : // Assume no a l i a s i n g // apply loop opt to the bodies of

283

all

function definitions

4 // ===================================== 6 #include ” r o s e . h” 8 10

#include
#include ” L o o p T r a n s f o r m I n t e r f a c e . h” #include ”CommandOptions . h”

12 using namespace s t d ; 14 16 18

int main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

20 22 24

v e c t o r a r g v L i s t ( argv , a r g v + a r g c ) ; CmdOptions : : G e t I n s t a n c e ()−> S e t O p t i o n s ( a r g v L i s t ) ; AssumeNoAlias a l i a s I n f o ; LoopTransformInterface : : cmdline configure ( argvList ) ; L o o p T r a n s f o r m I n t e r f a c e : : s e t a l i a s I n f o (& a l i a s I n f o ) ;

26 S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g v L i s t ) ; 28 30 32 34

// Loop o v e r t h e number o f f i l e s i n t h e p r o j e c t i n t f i l e n u m = p r o j e c t −>n u m b e r O f F i l e s ( ) ; f o r ( i n t i = 0 ; i < f i l e n u m ; ++i ) { S g S o u r c e F i l e ∗ f i l e = i s S g S o u r c e F i l e ( p r o j e c t −> g e t f i l e L i s t ( ) [ i ] ) ; S g G l o b a l ∗ r o o t = f i l e −>g e t g l o b a l S c o p e ( ) ; S g D e c l a r a t i o n S t a t e m e n t P t r L i s t& d e c l L i s t = r o o t −>g e t d e c l a r a t i o n s

();

36 38 40 42 44

// Loop o v e r t h e d e c l a r a t i o n i n t h e g l o b a l s c o p e o f each f i l e f o r ( S g D e c l a r a t i o n S t a t e m e n t P t r L i s t : : i t e r a t o r p = d e c l L i s t . b e g i n ( ) ; p != d e c l L i s t . end ( ) ; ++p ) { SgFunctionDeclaration ∗ func = i s S g F u n c t i o n D e c l a r a t i o n (∗p ) ; i f ( f u n c == NULL) continue ; S g F u n c t i o n D e f i n i t i o n ∗ d e f n = f u n c−>g e t d e f i n i t i o n ( ) ; i f ( d e f n == NULL) continue ;

46 S g B a s i c B l o c k ∗ s t m t s = d e f n−>g e t b o d y ( ) ; A s t I n t e r f a c e I m p l faImpl ( stmts ) ;

48 50

// This w i l l do as much f u s i o n as p o s s i b l e ( f i n e r g r a i n e d // c o n t r o l o v e r l o o p o p t i m i z a t i o n s u s e s a d i f f e r e n t i n t e r f a c e ) . L o o p T r a n s f o r m I n t e r f a c e : : T r a n s f o r m T r a v e r s e ( f a I m p l , AstNodePtrImpl ( s t m t s ) ) ;

52 54

// JJW 10−29−2007 A d j u s t f o r i t e r a t o r i n v a l i d a t i o n and p o s s i b l e // i n s e r t e d s t a t e m e n t s p = s t d : : f i n d ( d e c l L i s t . b e g i n ( ) , d e c l L i s t . end ( ) , f u n c ) ; a s s e r t ( p != d e c l L i s t . end ( ) ) ; }

56 58 } 60 62

// Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; }

t h e vendor ’ s c o m p i l e r

Figure 38.1: Example source code showing use of loop optimization mechanisms.

284

CHAPTER 38. LOOP OPTIMIZATION

38.2

Matrix Multiply Example

Using the matrix multiply example code shown in figure 38.2, we run the loop optimizer in figure 38.1 and generate the code shown in figure 38.3.

2

// Example program showing m a t r i x m u l t i p l y // ( f o r use w i t h l o o p o p t i m i z a t i o n t u t o r i a l example )

4

#d e f i n e N 50

6 8

i n t main ( ) { int i , j , k ; double a [ N ] [ N ] , b [ N ] [ N ] , c [ N ] [ N ] ;

10 f o r ( i = 0 ; i <= N−1; i +=1) { f o r ( j = 0 ; j <= N−1; j +=1) { f o r ( k = 0 ; k <= N−1; k+=1) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } }

12 14 16 18 20 22

return 0 ; }

Figure 38.2: Example source code used as input to loop optimization processor.

38.2. MATRIX MULTIPLY EXAMPLE

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

285

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } // Example program showing m a t r i x m u l t i p l y // ( f o r use w i t h l o o p o p t i m i z a t i o n t u t o r i a l example ) #d e f i n e N 50 i n t main ( ) { int i ; int j ; int k ; double a [ 5 0 ] [ 5 0 ] ; double b [ 5 0 ] [ 5 0 ] ; double c [ 5 0 ] [ 5 0 ] ; var 0 ; int int var 1 ; v a r 1 <= 4 9 ; v a r 1 += 1 6 ) { for ( v a r 1 = 0 ; for ( v a r 0 = 0 ; v a r 0 <= 4 9 ; v a r 0 += 1 6 ) { f o r ( k = 0 ; k <= 4 9 ; k += 1 ) { f o r ( i = v a r 1 ; i <= min2 ( 4 9 , v a r 1 + 1 5 ) ; i += 1 ) { f o r ( j = v a r 0 ; j <= min2 ( 4 9 , v a r 0 + 1 5 ) ; j += 1 ) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } } } } return 0 ; }

Figure 38.3: Output of loop optimization processor showing matrix multiply optimization (using options: -bk1 -fs0).

286

CHAPTER 38. LOOP OPTIMIZATION

38.3

Loop Fusion Example

Using the loop fusion example code shown in figure 38.4, we run the loop optimizer in figure 38.1 and generate the code shown in figure 38.5.

2 main ( ) { 4 int x [ 3 0 ] ,

i;

6 for ( i = x[2 ∗ i } for ( i = x[2 ∗ i }

8 10 12 14

1 ; i <= 1 0 ; i += 1 ) { ] = x[2 ∗ i + 1] + 2; 1 ; i <= 1 0 ; i += 1 ) { + 3] = x[2 ∗ i ] + i ;

}

Figure 38.4: Example source code used as input to loop optimization processor.

2 4 6 8 10 12 14 16 18

i n t main ( ) { int x [ 3 0 ] ; int i ; f o r ( i = 1 ; i <= 1 1 ; i += 1 ) { i f ( i <= 1 0 ) { x[2 ∗ i ] = x[2 ∗ i + 1] + 2; } else { } i f ( i >= 2 ) { x [ 2 ∗ (−1 + i ) + 3 ] = x [ 2 ∗ (−1 + i ) ] + (−1 + i ) ; } else { } } }

Figure 38.5: Output of loop optimization processor showing loop fusion (using options: -fs2).

38.4

Example Loop Processor (LoopProcessor.C)

This section contains a more detail translator which uses the command-line for input of specific loop processing options and is more sophisticated than the previous translator used to handle the previous two examples. Figure 38.6 shows the code required to call the loop optimizations within ROSE. The translator that we build for this tutorial is simple and takes command line parameters to control which optimizations are done.

38.4. EXAMPLE LOOP PROCESSOR (LOOPPROCESSOR.C)

2 4

287

#include ” r o s e . h” #include #include ” p r e . h” #include ” f i n i t e D i f f e r e n c i n g . h”

6 8

// DQ ( 1 / 2 / 2 0 0 8 ) : I t h i n k t h i s // #i n c l u d e ” c o p y u n p a r s e r . h”

i s no l o n g e r used !

10

16

#include #include #include #include #include #include

18

using namespace s t d ;

20

#i f d e f USE OMEGA #include

12 14

” r e w r i t e . h”


22 24 26 28 30

32 34 36 38 40 42

extern D e p T e s t S t a t i s t i c s D e p S t a t s ; #e n d i f extern bool DebugAnnot ( ) ; extern void F i x F i l e I n f o ( SgNode∗ n ) ; c l a s s UnparseFormatHelp ; class UnparseDelegate ; void u n p a r s e P r o j e c t ( S g P r o j e c t ∗ p r o j e c t , UnparseFormatHelp ∗ u n p a r s e H e l p /∗= NULL∗/ , U n p a r s e D e l e g a t e ∗ r e p l /∗= NULL ∗/ ) ; void P r i n t U s a g e ( char ∗ name ) { c e r r << name << ” ” << ”” << ” \n” ; c e r r << ”−g o b j : g e n e r a t e o b j e c t f i l e \n” ; c e r r << ”−o r i g : copy non−m o d i f i e d s t a t e m e n t s from o r i g i n a l f i l e \n” ; c e r r << ”− s p l i t l o o p : a p p l y i n g l o o p s p l i t t i n g t o remove c o n d i t i o n a l s i n s i d e c e r r << ReadAnnotation : : g e t i n s t ()−> O p t i o n S t r i n g ( ) << e n d l ; c e r r << ”−p r e : a p p l y p a r t i a l redundancy e l i m i n a t i o n \n” ; c e r r << ”−f d : a p p l y f i n i t e d i f f e r e n c i n g t o a r r a y i n d e x e x p r e s s i o n s \n” ; LoopTransformInterface : : PrintTransformUsage ( c e r r ) ; }

l o o p s \n” ;

Figure 38.6: Detailed example source code showing use of loop optimization mechanisms (loopProcessor.C part 1).

288

2 4

CHAPTER 38. LOOP OPTIMIZATION

bool GenerateObj ( ) { return CmdOptions : : G e t I n s t a n c e ()−>HasOption ( ”−g o b j ” ) ; }

6 8 10 12 14 16 18 20 22 24 26 28 30 32 34

int main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; if

( a r g c <= 1 ) { PrintUsage ( argv [ 0 ] ) ; return −1;

} v e c t o r a r g v L i s t ( argv , a r g v + a r g c ) ; CmdOptions : : G e t I n s t a n c e ()−> S e t O p t i o n s ( a r g v L i s t ) ; AssumeNoAlias a l i a s I n f o ; LoopTransformInterface : : cmdline configure ( argvList ) ; L o o p T r a n s f o r m I n t e r f a c e : : s e t a l i a s I n f o (& a l i a s I n f o ) ; #i f d e f USE OMEGA D e p S t a t s . SetFileName ( b u f f e r . s t r ( ) ) ; #e n d i f OperatorSideEffectAnnotation ∗ funcInfo = OperatorSideEffectAnnotation : : g e t i n s t ( ) ; f u n c I n f o −>r e g i s t e r a n n o t ( ) ; ReadAnnotation : : g e t i n s t ()−> r e a d ( ) ; i f ( DebugAnnot ( ) ) f u n c I n f o −>Dump ( ) ; LoopTransformInterface : : s e t s i d e E f f e c t I n f o ( funcInfo ) ; S g P r o j e c t ∗ p r o j e c t = new S g P r o j e c t ( a r g v L i s t ) ;

36 38 40 42

i n t f i l e n u m = p r o j e c t −>n u m b e r O f F i l e s ( ) ; f o r ( i n t i = 0 ; i < f i l e n u m ; ++i ) { // S g F i l e &s a g e F i l e = s a g e P r o j e c t −> g e t f i l e ( i ) ; // S g G l o b a l ∗ r o o t = s a g e F i l e . g e t r o o t ( ) ; S g S o u r c e F i l e ∗ f i l e = i s S g S o u r c e F i l e ( p r o j e c t −> g e t f i l e L i s t ( ) [ i ] ) ; S g G l o b a l ∗ r o o t = f i l e −>g e t g l o b a l S c o p e ( ) ;

Figure 38.7: loopProcessor.C source code (Part 2).

38.5. MATRIX MULTIPLICATION EXAMPLE (MM.C)

38.5

289

Matrix Multiplication Example (mm.C)

Using the matrix multiplication example code shown in figure 38.8, we run the loop optimizer in figure 38.6 and generate the code shown in figure 38.9.

2

#d e f i n e N 50

4

void p r i n t m a t r i x ( double x [ ] [ N ] ) ; void i n i t m a t r i x ( double x [ ] [ N] , double s ) ;

6 8 10

main ( ) { int i , j , k ; double a [N ] [ N] , b [ N ] [ N ] , c [ N ] [ N ] ;

12

double s ; s = 235.0; initmatrix (a , s ); s = 321.0; initmatrix (b , s ) ;

14 16 18

printmatrix (a ) ; printmatrix (b ) ; f o r ( i = 0 ; i <= N−1; i +=1) { f o r ( j = 0 ; j <= N−1; j +=1) { f o r ( k = 0 ; k <= N−1; k+=1) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } }

20 22 24 26 28

printmatrix ( c ) ; }

Figure 38.8: Example source code used as input to loopProcessor, show in figure 38.6.

290

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38

CHAPTER 38. LOOP OPTIMIZATION

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } #d e f i n e N 50 void p r i n t m a t r i x ( double x [ ] [ 5 0 ] ) ; void i n i t m a t r i x ( double x [ ] [ 5 0 ] , double s ) ; i n t main ( ) { int i ; int j ; int k ; double a [ 5 0 ] [ 5 0 ] ; double b [ 5 0 ] [ 5 0 ] ; double c [ 5 0 ] [ 5 0 ] ; double s ; var 0 ; int int var 1 ; s = 235.0; initmatrix (a , s ); s = 321.0; initmatrix (b , s ) ; printmatrix (a ) ; printmatrix (b ) ; v a r 1 <= 4 9 ; v a r 1 += 1 6 ) { for ( v a r 1 = 0 ; v a r 0 <= 4 9 ; v a r 0 += 1 6 ) { for ( v a r 0 = 0 ; f o r ( k = 0 ; k <= 4 9 ; k += 1 ) { f o r ( i = v a r 1 ; i <= min2 ( 4 9 , v a r 1 + 1 5 ) ; i += 1 ) { f o r ( j = v a r 0 ; j <= min2 ( 4 9 , v a r 0 + 1 5 ) ; j += 1 ) { c [ i ][ j ] = c[ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } } } } printmatrix ( c ) ; }

Figure 38.9: Output of loopProcessor using input from figure 38.8 (using options: -bk1 -fs0).

38.6. MATRIX MULTIPLICATION EXAMPLE USING LINEARIZED MATRICES (DGEMM.C)291

38.6

Matrix Multiplication Example Using Linearized Matrices (dgemm.C)

Using the matrix multiplication example code shown in figure 38.10, we run the loop optimizer in figure 38.6 and generate the code shown in figure 38.11.

2

// Function p r o t o t y p e void dgemm( double ∗a , double ∗b , double ∗ c , i n t n ) ;

4 6 8 10

// Function d e f i n i t i o n void dgemm( double ∗a , double ∗b , double ∗ c , i n t n ) { int i , j , k ; // i n t n ;

12

f o r ( k =0;k
14 16 18 }

Figure 38.10: Example source code used as input to loopProcessor, show in figure 38.6.

292

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46

CHAPTER 38. LOOP OPTIMIZATION

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } // Function p r o t o t y p e void dgemm( double ∗a , double ∗b , double ∗ c , i n t n ) ; // Function d e f i n i t i o n void dgemm( double ∗a , double ∗b , double ∗ c , i n t n ) { int i ; int j ; int k ; var 0 ; int var 1 ; int v a r 1 <= −1 + n ; v a r 1 += 1 6 ) { for ( v a r 1 = 0 ; for ( v a r 0 = 0 ; v a r 0 <= −1 + n ; v a r 0 += 1 6 ) { f o r ( i = 0 ; i <= −1 + n ; i += 1 ) { f o r ( k = v a r 1 ; k <= min2(−1 + n , v a r 1 + 1 5 ) ; k += 1 ) { f o r ( j = v a r 0 ; j <= min2 ( n + −16 , v a r 0 ) ; j += 1 6 ) { c [ j ∗ n + i ] = c [ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n + k ]; c [(1 + j ) ∗ n + i ] = c [(1 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(1 + c [(2 + j ) ∗ n + i ] = c [(2 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(2 + c [(3 + j ) ∗ n + i ] = c [(3 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(3 + c [(4 + j ) ∗ n + i ] = c [(4 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(4 + c [(5 + j ) ∗ n + i ] = c [(5 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(5 + c [(6 + j ) ∗ n + i ] = c [(6 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(6 + c [(7 + j ) ∗ n + i ] = c [(7 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(7 + c [(8 + j ) ∗ n + i ] = c [(8 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(8 + c [(9 + j ) ∗ n + i ] = c [(9 + j ) ∗ n + i ] + a [k ∗ n + i ] ∗ b[(9 + c [(10 + j ) ∗ n + i ] = c [(10 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(10 c [(11 + j ) ∗ n + i ] = c [(11 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(11 c [(12 + j ) ∗ n + i ] = c [(12 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(12 c [(13 + j ) ∗ n + i ] = c [(13 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(13 c [(14 + j ) ∗ n + i ] = c [(14 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(14 c [(15 + j ) ∗ n + i ] = c [(15 + j ) ∗ n + i ] + a [ k ∗ n + i ] ∗ b[(15 } f o r ( ; j <= min2(−1 + n , 1 5 + v a r 0 ) ; j += 1 ) { c [ j ∗ n + i ] = c [ j ∗ n + i ] + a[k ∗ n + i ] ∗ b[ j ∗ n + k ]; } } } } } }

j) j) j) j) j) j) j) j) j) + + + + + +

∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; ∗ n + k]; j) ∗ n + k]; j) ∗ n + k]; j) ∗ n + k]; j) ∗ n + k]; j) ∗ n + k]; j) ∗ n + k];

Figure 38.11: Output of loopProcessor using input from figure 38.10 (using options: -bk1 -unroll nvar 16).

38.7. LU FACTORIZATION EXAMPLE (LUFAC.C)

38.7

293

LU Factorization Example (lufac.C)

Using the LU factorization example code shown in figure 38.12, we run the loop optimizer in figure 38.6 and generate the code shown in figure 38.13. double abs ( double x ) { i f

( x < 0 ) return −x ; e l s e return x ; }

2 4

#d e f i n e n 50 void p r i n t m a t r i x ( double x [ ] [ n ] ) ; void i n i t m a t r i x ( double x [ ] [ n ] , double s ) ;

6 8

main ( i n t a r g c , char ∗ a r g v [ ] ) { int p [ n ] , i , j , k ; double a [ n ] [ n ] , mu, t ;

10 12 14 16 18 20 22

initmatrix (a , 5.0); printmatrix (a ) ; f o r ( k = 0 ; k<=n −2; k+=1) { p[k] = k; mu = abs ( a [ k ] [ k ] ) ; f o r ( i = k +1; i <= n −1; i +=1) { i f (mu < abs ( a [ i ] [ k ] ) ) { mu = abs ( a [ i ] [ k ] ) ; p[k] = i ; } } f o r ( j = k ; j <= n −1; j +=1) { t = a[k][ j ]; a[k ][ j ] = a[p[k ] ] [ j ]; a[p[k ] ] [ j ] = t ; }

24 26 28

f o r ( i = k +1; i <= n −1; i +=1) { a [ i ] [ k ] = a [ i ] [ k]/ a [ k ] [ k ] ; } f o r ( j = k +1; j <=n −1; j +=1) { f o r ( i = k +1; i <=n −1; i +=1) { a [ i ] [ j ] = a [ i ] [ j ] − a [ i ] [ k]∗ a [ k ] [ j ] ; } }

30 32 34 36 38 40

} printmatrix (a ) ; }

Figure 38.12: Example source code used as input to loopProcessor, show in figure 38.6.

294

2 4 6 8 10

CHAPTER 38. LOOP OPTIMIZATION

double abs ( double x ) { i f ( x < 0) return −x ; else return x ; } #d e f i n e n 50 void p r i n t m a t r i x ( double x [ ] [ 5 0 ] ) ; void i n i t m a t r i x ( double x [ ] [ 5 0 ] , double s ) ;

12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { int p [ 5 0 ] ; int i ; int j ; int k ; double a [ 5 0 ] [ 5 0 ] ; double mu ; double t ; initmatrix (a , 5 . 0 ) ; printmatrix (a ) ; f o r ( k = 0 ; k <= 4 8 ; k += 1 ) { p[k] = k; mu = a bs ( a [ k ] [ k ] ) ; f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { i f (mu < abs ( a [ i ] [ k ] ) ) { mu = ab s ( a [ i ] [ k ] ) ; p[k] = i ; } else { } } f o r ( j = k ; j <= 4 9 ; j += 1 ) { t = a[k][ j ]; a[k ][ j ] = a[p[k ] ] [ j ]; a[p[k ] ] [ j ] = t ; } f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { a[ i ][k] = a[ i ][k] / a[k][k]; } f o r ( j = 1 + k ; j <= 4 9 ; j += 1 ) { f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { a[ i ][ j ] = a[ i ][ j ] − a[ i ][k] ∗ a[k][ j ]; } } } printmatrix (a ) ; }

Figure 38.13: Output of loopProcessor using input from figure 38.12 (using options: -bk1 -fs0 -splitloop -annotation).

38.8. LOOP FUSION EXAMPLE (TRIDVPK.C)

38.8

295

Loop Fusion Example (tridvpk.C)

Using the loop fusion example code shown in figure 38.14, we run the loop optimizer in figure 38.6 and generate the code shown in figure 38.15. #d e f i n e n 100 2 double a [ n ] , b [ n ] , c [ n ] , d [ n ] , e [ n ] ; double t o t [ n ] [ n ] ; double dux [ n ] [ n ] [ n ] , duy [ n ] [ n ] [ n ] , duz [ n ] [ n ] [ n ] ;

4 6 8 10 12

main ( ) { int i , j , k ; f o r ( j =0; j <=n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) duz [ i ] [ j ] [ 0 ] = duz [ i ] [ j ] [ 0 ] ∗ b [ 0 ] ; f o r ( k =1; k<=n −2; k+=1) f o r ( j = 0 ; j <= n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) duz [ i ] [ j ] [ k ] = ( duz [ i ] [ j ] [ k]−a [ k ] ∗ duz [ i ] [ j ] [ k −1 ] )∗ b [ k ] ;

14 16 18

f o r ( j =0; j <=n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) tot [ i ] [ j ] = 0;

20 22

f o r ( k =0; k<=n −2; k+=1) f o r ( j =0; j <=n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) t o t [ i ] [ j ] = t o t [ i ] [ j ] + d [ k ] ∗ duz [ i ] [ j ] [ k ] ;

24 26

f o r ( j =0; j <=n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) duz [ i ] [ j ] [ n −1] = ( duz [ i ] [ j ] [ n −1] − t o t [ i ] [ j ] ) ∗ b [ n − 1 ] ;

28 30

f o r ( j =0; j <=n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) duz [ i ] [ j ] [ n−2]=duz [ i ] [ j ] [ n −2] − e [ n −2]∗ duz [ i ] [ j ] [ n − 1 ] ;

32 34

f o r ( k=n −3; k>=0; k+=−1) f o r ( j = 0 ; j <= n −1; j +=1) f o r ( i =0; i <=n −1; i +=1) duz [ i ] [ j ] [ k ] = duz [ i ] [ j ] [ k ] − c [ k ] ∗ duz [ i ] [ j ] [ k +1] − e [ k ] ∗ duz [ i ] [ j ] [ n − 1 ] ;

36 38 40 }

Figure 38.14: Example source code used as input to loopProcessor, show in figure 38.6.

296

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36

CHAPTER 38. LOOP OPTIMIZATION

#d e f i n e n 100 double a [ 1 0 0 ] ; double b [ 1 0 0 ] ; double c [ 1 0 0 ] ; double d [ 1 0 0 ] ; double e [ 1 0 0 ] ; double t o t [ 1 0 0 ] [ 1 0 0 ] ; double dux [ 1 0 0 ] [ 1 0 0 ] [ 1 0 0 ] ; double duy [ 1 0 0 ] [ 1 0 0 ] [ 1 0 0 ] ; double duz [ 1 0 0 ] [ 1 0 0 ] [ 1 0 0 ] ; i n t main ( ) { int i ; int j ; int k ; f o r ( i = 0 ; i <= 9 9 ; i += 1 ) { f o r ( j = 0 ; j <= 9 9 ; j += 1 ) { duz [ i ] [ j ] [ 0 ] = duz [ i ] [ j ] [ 0 ] ∗ b [ 0 ] ; tot [ i ] [ j ] = 0; f o r ( k = 0 ; k <= 9 8 ; k += 1 ) { i f ( k >= 1 ) { duz [ i ] [ j ] [ k ] = ( duz [ i ] [ j ] [ k ] − a [ k ] ∗ duz [ i } else { } t o t [ i ] [ j ] = t o t [ i ] [ j ] + d [ k ] ∗ duz [ i ] [ j ] [ k ] ; } duz [ i ] [ j ] [ 1 0 0 − 1 ] = ( duz [ i ] [ j ] [ 1 0 0 − 1 ] − t o t [ duz [ i ] [ j ] [ 1 0 0 − 2 ] = duz [ i ] [ j ] [ 1 0 0 − 2 ] − e [ 1 0 0 f o r ( k = 9 7 ; k >= 0 ; k += −1) { duz [ i ] [ j ] [ k ] = duz [ i ] [ j ] [ k ] − c [ k ] ∗ duz [ i ] [ j } } } }

] [ j ] [ k − 1]) ∗ b[k ] ;

i ] [ j ] ) ∗ b[100 − 1 ] ; − 2 ] ∗ duz [ i ] [ j ] [ 1 0 0 − 1 ] ; ] [ k + 1 ] − e [ k ] ∗ duz [ i ] [ j ] [ 1 0 0 − 1 ] ;

Figure 38.15: Output of loopProcessor input from figure 38.14 (using options: -fs2 -ic1 -opt 1 ).

Chapter 39

Parameterized Code Translation This chapter gives examples of using ROSE’s high level loop translation interfaces to perform parameterized loop transformations, including loop unrolling, interchanging and tiling. The motivation is to give users the maximized flexibility to orchestrate code transformations on the targets they want, the order they want, and the parameters they want. One typical application scenario is to support generating desired code variants for empirical tuning. The ROSE internal interfaces (declared within the SageInterface namespace) to call loop transformations are: • bool loopUnrolling (SgForStatement *loop, size t unrolling factor): This function needs two parameters: one for the loop to be unrolled and the other for the unrolling factor. • bool loopInterchange (SgForStatement *loop, size t depth, size t lexicoOrder): The loop interchange function has three parameters, the first one to specify a loop which starts a perfectly-nested loop and is to be interchanged, the 2nd for the depth of the loop nest to be interchanged, and finally the lexicographical order for the permutation. • bool loopTiling (SgForStatement *loopNest, size t targetLevel, size t tileSize) The loop tiling interface needs to know the loop nest to be tiled, which loop level to tile, and the tile size for the level. For efficiency concerns, those functions only perform the specified translations without doing any legitimacy check. It is up to the users to make sure the transformations won’t generate wrong code. We will soon provide interfaces to do the eligibility check for each transformation. We also provide standalone executable programs (loopUnrolling,loopInterchange, and loopTiling under ROSE INSTALL/bin) for the transformations mentioned above so users can directly use them via command lines and abstract handles (explained in Chapter 46) to orchestrate transformations they want.

39.1

Loop Unrolling

Figure 39.1 gives an example input code for loopUnrolling. An example command line to invoke loop unrolling on the example can look like the following: 297

298

2 4 6 8 10 12

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

int a [ 1 0 0 ] [ 1 0 0 ] ; i n t main ( void ) { int j ; f o r ( i n t i =0; i <100; i ++) f o r ( j =0; j <100; j ++) { i n t k =3; a [ i ] [ j ]= i+j+k ; } return 0 ; }

Figure 39.1: Example source code used as input to loopUnrolling # unroll a for statement 5 times. The loop is a statement at line 6 within # an input file. loopUnrolling -c inputloopUnrolling.C \ -rose:loopunroll:abstract_handle "Statement" -rose:loopunroll:factor 5 Two kinds of output can be expected after loop unrolling. One (Shown in Figure 39.2) is the case that the loop iteration count is known at compile-time and can be evenly divisible by the unrolling factor. The other case (Shown in Figure 39.3 is when the divisibility is unknown and a fringe loop has to be generated to run possible leftover iterations.

39.1. LOOP UNROLLING

299

int a [ 1 0 0 ] [ 1 0 0 ] ; 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

int main ( ) { int j ; f o r ( i n t i = 0 ; i < 1 0 0 ; i ++) { f o r ( j = 0 ; j <= 9 9 ; j += 5 ) { int k = 3 ; a[ i ][ j ] = i + j + k; { int k = 3 ; a [ i ] [ j + 1] = i + ( j + } { int k = 3 ; a [ i ] [ j + 2] = i + ( j + } { int k = 3 ; a [ i ] [ j + 3] = i + ( j + } { int k = 3 ; a [ i ] [ j + 4] = i + ( j + } } } return 0 ; }

1) + k ;

2) + k ;

3) + k ;

4) + k ;

Figure 39.2: Output for a unrolling factor which can divide the iteration space evenly

300

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

int a [ 1 0 0 ] [ 1 0 0 ] ; 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

int main ( ) { int j ; f o r ( i n t i = 0 ; i < 1 0 0 ; i ++) { // i t e r c o u n t = ( ub−l b +1)%s t e p ==0?(ub−l b +1)/ s t e p : ( ub−l b +1)/ s t e p +1; // f r i n g e = i t e r c o u n t%u n r o l l f a c t o r==0 ? 0 : u n r o l l f a c t o r ∗ s t e p int lu fringe 1 = 3; f o r ( j = 0 ; j <= 99 − l u f r i n g e 1 ; j += 3 ) { int k = 3 ; a[ i ][ j ] = i + j + k; { int k = 3 ; a [ i ] [ j + 1 ] = i + ( j + 1) + k ; } { int k = 3 ; a [ i ] [ j + 2 ] = i + ( j + 2) + k ; } } f o r ( ; j <= 9 9 ; j += 1 ) { int k = 3 ; a[ i ][ j ] = i + j + k; } } return 0 ; }

Figure 39.3: Output for the case when divisibility is unknown at compile-time

39.2. LOOP INTERCHANGE

39.2

301

Loop Interchange

Figure 39.4 gives an example input code for loopInterchange.

2 4 6 8 10

void O U T 1 6 1 1 9 ( i n t r i , double ∗ rp , i n t s t e n c i l s i z e , i n t hypre { i n t s i , i i , j j , kk ; // t h e f o l l o w i n g 4− l e v e l l o o p n e s t i s t o be i n t e r c h a n g e d f o r ( s i = 0 ; s i < s t e n c i l s i z e ; s i ++) f o r ( kk = 0 ; kk < hypre m ; kk++) f o r ( j j = 0 ; j j < hypre m ; j j ++) f o r ( i i = 0 ; i i < hypre m ; i i ++) rp [ ( r i + i i ) + j j + kk ] −= Ap 0 [ i i + j j + kk ] ; }

m , const double ∗ Ap 0 )

Figure 39.4: Example source code used as input to loopInterchange An example command line to invoke loop interchange: # interchange a loop nest starting from the first loop within the input file, # interchange depth is 4 and # the lexicographical order is 1 (swap the innermost two levels) loopInterchange -c inputloopInterchange.C -rose:loopInterchange:abstract_handle \ "ForStatement" -rose:loopInterchange:depth 4 \ -rose:loopInterchange:order 1 Figure 39.5 shows the output.

2 4 6 8 10 12 14

void O U T 1 6 1 1 9 ( i n t r i , double ∗ rp , i n t s t e n c i l s i z e , i n t hypre { int s i ; int i i ; int j j ; i n t kk ; // t h e f o l l o w i n g 4− l e v e l l o o p n e s t i s t o be i n t e r c h a n g e d f o r ( s i = 0 ; s i < s t e n c i l s i z e ; s i ++) f o r ( kk = 0 ; kk < hypre m ; kk++) f o r ( i i = 0 ; i i < hypre m ; i i ++) f o r ( j j = 0 ; j j < hypre m ; j j ++) rp [ r i + i i + j j + kk ] −= Ap 0 [ i i + j j + kk ] ; }

Figure 39.5: Output for loop interchange

m , const double ∗ Ap 0 )

302

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

39.3

Loop Tiling

Figure 39.6 gives an example input code for loopTiling.

2

#d e f i n e N 100 int i , j , k ; double a [ N ] [ N ] , b [ N ] [ N ] , c [ N ] [ N ] ;

4 6 8 10 12

i n t main ( ) { f o r ( i = 0 ; i < N ; i ++) f o r ( j = 0 ; j < N ; j ++) f o r ( k = 0 ; k < N ; k++) c [ i ] [ j ]= c [ i ] [ j ]+ a [ i ] [ k ] ∗ b [ k ] [ j ] ; return 0 ; }

Figure 39.6: Example source code used as input to loopTiling An example command line to invoke loop tiling: # Tile the loop with a depth of 3 within the first loop of the input file # tile size is 5 loopTiling -c inputloopTiling.C -rose:loopTiling:abstract_handle \ "ForStatement" -rose:loopTiling:depth 3 -rose:loopTiling:tilesize 5 Figure 39.7 shows the output.

2 4 6

#d e f i n e N 100 int i ; int j ; int k ; double a [ 1 0 0 ] [ 1 0 0 ] ; double b [ 1 0 0 ] [ 1 0 0 ] ; double c [ 1 0 0 ] [ 1 0 0 ] ;

8 10 12 14 16 18 20

i n t main ( ) { lt var k ; int for ( l t v a r k = 0 ; l t v a r k <= 9 9 ; l t v a r k += 5 ) { f o r ( i = 0 ; i < 1 0 0 ; i ++) f o r ( j = 0 ; j < 1 0 0 ; j ++) f o r ( k = l t v a r k ; k <= ( ( ( 9 9 < ( l t v a r k + 5 − 1 ) ) ? 9 9 c[ i ][ j ] = c [ i ][ j ] + a[ i ][ k] ∗ b[k ][ j ]; } } return 0 ; }

Figure 39.7: Output for loop tiling

: ( l t v a r k + 5 − 1 ) ) ) ; k += 1 ) {

Part V

Correctness Checking

Tutorials of using ROSE to help program correctness checking or debugging.

303

Chapter 40

Code Coverage This translator is part of ongoing collaboration with IBM on the support of code coverage analysis tools for C, C++ and F90 applications. the subject of code coverage is much more complex than this example code would cover. The following web site: http://www.bullseye.com/coverage.html contains more information and is the source for the descriptions below. Code coverage can include: • Statement Coverage This measure reports whether each executable statement is encountered. • Decision Coverage This measure reports whether boolean expressions tested in control structures (such as the if-statement and while-statement) evaluated to both true and false. The entire boolean expression is considered one true-or-false predicate regardless of whether it contains logicaland or logical-or operators. Additionally, this measure includes coverage of switch-statement cases, exception handlers, and interrupt handlers. • Condition Coverage Condition coverage reports the true or false outcome of each boolean sub-expression, separated by logical-and and logical-or if they occur. Condition coverage measures the subexpressions independently of each other. • Multiple Condition Coverage Multiple condition coverage reports whether every possible combination of boolean subexpressions occurs. As with condition coverage, the sub-expressions are separated by logical-and and logical-or, when present. The test cases required for full multiple condition coverage of a condition are given by the logical operator truth table for the condition. • Condition/Decision Coverage Condition/Decision Coverage is a hybrid measure composed by the union of condition coverage and decision coverage. This measure was created at Boeing and is required for aviation software by RCTA/DO-178B. 305

306

CHAPTER 40. CODE COVERAGE • Modified Condition/Decision Coverage This measure requires enough test cases to verify every condition can affect the result of its encompassing decision. • Path Coverage This measure reports whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit. • Function Coverage This measure reports whether you invoked each function or procedure. It is useful during preliminary testing to assure at least some coverage in all areas of the software. Broad, shallow testing finds gross deficiencies in a test suite quickly. • Call Coverage This measure reports whether you executed each function call. The hypothesis is that faults commonly occur in interfaces between modules. • Linear Code Sequence and Jump (LCSAJ) Coverage This variation of path coverage considers only sub-paths that can easily be represented in the program source code, without requiring a flow graph. An LCSAJ is a sequence of source code lines executed in sequence. This ”linear” sequence can contain decisions as long as the control flow actually continues from one line to the next at run-time. Sub-paths are constructed by concatenating LCSAJs. Researchers refer to the coverage ratio of paths of length n LCSAJs as the test effectiveness ratio (TER) n+2. • Data Flow Coverage This variation of path coverage considers only the sub-paths from variable assignments to subsequent references of the variables. • Object Code Branch Coverage This measure reports whether each machine language conditional branch instruction both took the branch and fell through. • Loop Coverage This measure reports whether you executed each loop body zero times, exactly once, and more than once (consecutively). For do-while loops, loop coverage reports whether you executed the body exactly once, and more than once. The valuable aspect of this measure is determining whether while-loops and for-loops execute more than once, information not reported by others measure. • Race Coverage This measure reports whether multiple threads execute the same code at the same time. It helps detect failure to synchronize access to resources. It is useful for testing multi-threaded programs such as in an operating system. • Relational Operator Coverage This measure reports whether boundary situations occur with relational operators (¡, ¡=, ¿, ¿=). The hypothesis is that boundary test cases find off-by-one errors and mistaken uses of wrong relational operators such as ¡ instead of ¡=.

307 • Weak Mutation Coverage This measure is similar to relational operator coverage but much more general [Howden1982]. It reports whether test cases occur which would expose the use of wrong operators and also wrong operands. It works by reporting coverage of conditions derived by substituting (mutating) the program’s expressions with alternate operators, such as ”-” substituted for ”+”, and with alternate variables substituted. • Table Coverage This measure indicates whether each entry in a particular array has been referenced. This is useful for programs that are controlled by a finite state machine. The rest of this text must be changed to refer to the code coverage example within ROSE/tutorial. Figure 40 shows the low level construction of a more complex AST fragment (a function declaration) and its insertion into the AST at the top of each block. Note that the code does not handle symbol table issues, yet. Building a function in global scope.

308

2

CHAPTER 40. CODE COVERAGE

/ / ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t s h o w s t h e d e s i g n o f a t r a n s f o r m a t i o n // a t t h e t o p and b o t t o m e o f e a c h b a s i c b l o c k .

is to

an e x a m p l e instrument

preprocessor s o u r c e code ,

b u i l t w i t h ROSE . p l a c i n g source code

4 #i n c l u d e

” r o s e . h”

6 8 10

u s i n g namespace s t d ; u s i n g namespace Rose ; /∗ Design of t h i s code . I n p u t s : s o u r c e code ( f i l e . C) Outputs : instrumented source

12

code

( rose

f i l e . C and

file .o)

14 P r o p e r t i e s of instrumented source code : 1) added d e c l a r a t i o n f o r coverage s u p p o r t f u n c t i o n ( e i t h e r forward f u n c t i o n d e c l a r a t i o n or a #i n c l u d e to i n c l u d e a header f i l e ) . 2) Each f u n c t i o n i n t h e s o u r c e program i s i n s t r u m e n t e d t o i n c l u d e a c a l l t o t h e coverage support function /

16 18 20

∗/

22 24 26 28 30 32 34 36

// G l o b a l v a r i a b l e s so t h a t t h e g l o b a l f u n c t i o n d e c l a r a t i o n can be r e u s e d t o b u i l d / / e a c h f u n c t i o n c a l l e x p r e s s i o n i n t h e AST t r a v e r s a l t o i n s t r u m e n t a l l f u n c t i o n s . S g F u n c t i o n D e c l a r a t i o n ∗ g l o b a l F u n c t i o n D e c l a r a t i o n = NULL ; SgFunctionType ∗ globalFunctionType = NULL ; S g F u n c t i o n S y m b o l ∗ f u n c t i o n S y m b o l = NULL ; / / S i m p l e ROSE t r a v e r s a l c l a s s : T h i s a l l o w s u s t o v i s i t a l l t h e // new c o d e t o i n s t r u m e n t / r e c o r d t h e i r u s e . c l a s s S i m p l e I n s t r u m e n t a t i o n : public S g S i m p l e P r o c e s s i n g { public : // r e q u i r e d v i s i t f u n c t i o n t o d e f i n e what i s t o be done void v i s i t ( SgNode ∗ a s t N o d e ) ; };

functions

and

add

38 40 42 44 46

// Code t o b u i l d f u n c t i o n d e c l a r a t i o n : T h i s d e c l a r e s Shmuel ’ s f u n c t i o n c a l l w h i c h // w i l l be i n s e r t e d ( as a f u n c t i o n c a l l ) i n t o e a c h f u n c t i o n b o d y o f t h e i n p u t // a p p l i c a t i o n . void b u i l d F u n c t i o n D e c l a r a t i o n ( S g P r o j e c t ∗ p r o j e c t ) { / / ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ // C r e a t e t h e f u n c t i o n D e c l a r a t i o n / / ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗

48 // 50 52

S g G l o b a l ∗ g l o b a l S c o p e = p r o j e c t −> g e t f i l e ( 0 ) . g e t r o o t ( ) ; S g S o u r c e F i l e ∗ s o u r c e F i l e = i s S g S o u r c e F i l e ( p r o j e c t −> g e t f i l e L i s t ( ) [ 0 ] ) ; ROSE ASSERT ( s o u r c e F i l e != NULL ) ; S g G l o b a l ∗ g l o b a l S c o p e = s o u r c e F i l e −>g e t g l o b a l S c o p e ( ) ; ROSE ASSERT ( g l o b a l S c o p e != NULL ) ;

54 Sg File Info ∗ file info SgType ∗ f u n c t i o n r e t u r n t y p e

56 58

SgName f u n c t i o n n a m e = ” coverageTraceFunc1 ” ; SgFunctionType ∗ f u n c t i o n t y p e = new S g F u n c t i o n T y p e ( f u n c t i o n r e t u r n t y p e , f a l s e ) ; S g F u n c t i o n D e c l a r a t i o n ∗ f u n c t i o n D e c l a r a t i o n = new S g F u n c t i o n D e c l a r a t i o n ( f i l e i n f o , f u n c t i o n n a m e ,

60 62 64 66 68

= Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; = new SgTypeVoid ( ) ;

function type );

/ / DQ ( 9 / 8 / 2 0 0 7 ) : F i x u p t h e d e f i n i n g a n d n o n− d e f i n i n g d e c l a r a t i o n s ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n −>g e t d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; f u n c t i o n D e c l a r a t i o n −>s e t d e f i n i n g D e c l a r a t i o n ( f u n c t i o n D e c l a r a t i o n ) ; ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n −>g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != f u n c t i o n D e c l a r a t i o n ) ; / / DQ ( 9 / 8 / 2 0 0 7 ) : We h a v e n o t b u i l d a n o n− d e f i n i n g d e c l a r a t i o n , s o t h i s s h o u l d b e ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n −>g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ;

NULL .

70 72 74

/ / DQ ( 9 / 8 / 2 0 0 7 ) : N e e d t o a d d f u n c t i o n s y m b o l t o g l o b a l s c o p e ! p r i n t f ( ” F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r f u n c t i o n = %p = %s \n ” , g l o b a l S c o p e , g l o b a l S c o p e −>c l a s s n a m e ( ) . c s t f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( f u n c t i o n D e c l a r a t i o n ) ; g l o b a l S c o p e −>i n s e r t s y m b o l ( f u n c t i o n D e c l a r a t i o n −>g e t n a m e ( ) , f u n c t i o n S y m b o l ) ; ROSE ASSERT ( g l o b a l S c o p e −>l o o k u p f u n c t i o n s y m b o l ( f u n c t i o n D e c l a r a t i o n −>g e t n a m e ( ) ) != NULL ) ;

76 78 80

// // //

∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ Create the InitializedName for a parameter within the parameter l i s t ∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗∗ SgName v a r 1 n a m e = ” t e x t S t r i n g ” ;

Figure 40.1: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 1).

309

SgTypeChar ∗ v a r 1 t y p e SgPointerType ∗ p o i n t e r t y p e SgInitializer ∗ var1 initializer SgInitializedName ∗var1 init name v a r 1 i n i t n a m e −> s e t f i l e i n f o ( S g

2 4 6 8

//

10

= new SgTypeChar ( ) ; = new S g P o i n t e r T y p e ( v a r 1 t y p e ) ; = NULL ; = new S g I n i t i a l i z e d N a m e ( v a r 1 n a m e , p o i n t e r t y p e , v a r 1 i n i t i a l i z e r , NULL ) ; File Info : : generateDefaultFileInfoForTransformationNode ( ) ) ;

I n s e r t argument in f u n c t i o n parameter l i s t ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n −>g e t p a r a m e t e r L i s t ( )

!= NULL ) ;

ROSE ASSERT( f u n c t i o n D e c l a r a t i o n −>g e t p a r a m e t e r L i s t ( ) != NULL ) ; f u n c t i o n D e c l a r a t i o n −>g e t p a r a m e t e r L i s t ()−> a p p e n d a r g ( v a r 1 i n i t n a m e ) ;

12 14 //

S e t t h e p a r e n t n o d e i n t h e AST ( t h i s c o u l d b e f u n c t i o n D e c l a r a t i o n −>s e t p a r e n t ( g l o b a l S c o p e ) ;

// //

Set t h e scope e x p l i c i t l y ( s i n c e i t c o u l d be d i f f e r e n t from t h e p a r e n t ?) T h i s c a n ’ t b e d o n e b y t h e A s t P o s t P r o c e s s i n g ( u n l e s s we r e l a x s om e c o n s t r a i n t s ) f u n c t i o n D e c l a r a t i o n −>s e t s c o p e ( g l o b a l S c o p e ) ;

//

I f i t i s not a forward d e c l a r a t i o n then the unparser w i l l f u n c t i o n D e c l a r a t i o n −>s e t F o r w a r d ( ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n −>i s F o r w a r d ( ) == t r u e ) ;

//

M a r k f u n c t i o n a s e x t e r n ”C” f u n c t i o n D e c l a r a t i o n −>g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t s t o r a g e M o d i f i e r ( ) . s e t E x t e r n ( ) ; f u n c t i o n D e c l a r a t i o n −>s e t l i n k a g e ( ”C” ) ; // T h i s mechanism c o u l d be i m p r o v e d !

16 18 20 22 24 26 28 30

done

by

the

AstPostProcessing

skip

the

”;”

at

the

end

( need

to

fix

this

better )

globalFunctionType = function type ; globalFunctionDeclaration = functionDeclaration ;

32 //

Add f u n c t i o n d e c l a r a t i o n t o g l o b a l s c o p e ! g l o b a l S c o p e −>p r e p e n d d e c l a r a t i o n ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ;

// // //

f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ; A l l any m o d i f i c a t i o n s t o b e f i x e d up ( p a r e n t s e t c ) A s t P o s t P r o c e s s i n g ( p r o j e c t ) ; // T h i s i s n o t a l l o w e d and s h o u l d be f i x e d ! AstPostProcessing ( globalScope ) ;

34 36 38 40 } 42 44 46 48 50 52 54 56

#i f 0 / / DQ ( 1 2 / 1 / 2 0 0 5 ) : T h i s v e r s i o n o f // i n s t u m e n t a t i o n a t e a c h f u n c t i o n / / A t IBM w e m o d i f i e d t h i s t o b e a

the v i s i t function handles the speci al case of ( a f u n c t i o n c a l l at the top of each f u n c t i o n ) . version which instrumented every b lock .

void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ a s t N o d e ) { S g F u n c t i o n D e c l a r a t i o n ∗ f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n ( astNode ) ; SgFunctionDefinition∗ functionDefinition = f u n c t i o n D e c l a r a t i o n != NULL ? f u n c t i o n D e c l a r a t i o n −>g e t d e f i n i t i o n ( ) i f ( f u n c t i o n D e c l a r a t i o n != NULL && f u n c t i o n D e f i n i t i o n != NULL) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e s t r i n g f u n c t i o n N a m e = f u n c t i o n D e c l a r a t i o n −>g e t n a m e ( ) . s t r ( ) ; s t r i n g fileName = f u n c t i o n D e c l a r a t i o n −> g e t f i l e i n f o ()−> g e t f i l e n a m e ( ) ;

: NULL ;

58 60

// //

B u i l d a s o u r c e f i l e l o c a t i o n o b j e c t ( f o r c o n s t r u c t i o n o f e a c h IR n o d e ) N o t e t h a t we s h o u l d n o t b e s h a r i n g t h e s a m e S g F i l e I n f o o b j e c t i n m u l t i p l e IR n o d e s . Sg File Info ∗ f i l e i n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ;

62 SgFunctionSymbol ∗ functionSymbol = new S g F u n c t i o n S y m b o l ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ; S g F u n c t i o n R e f E x p ∗ f u n c t i o n R e f E x p r e s s i o n = new S g F u n c t i o n R e f E x p ( f i l e i n f o , f u n c t i o n S y m b o l , g l o b a l F u n c t i o n T y p e ) ; SgExprListExp ∗ e x p r e s s i o n L i s t = new S g E x p r L i s t E x p ( f i l e i n f o ) ;

64 66

s t r i n g c o n v e r ag e F u n c t i on I n p u t = functionName + s t r i n g ( ” i n ” ) + fileName ; S g S t r i n g V a l ∗ f u n c t i o n N a m e S t r i n g V a l u e = new S g S t r i n g V a l ( f i l e i n f o , ( char ∗ ) c o n v e r a g e F u n c t i o n I n p u t . c s t r ( ) ) ; e x p r e s s i o n L i s t −>a p p e n d e x p r e s s i o n ( f u n c t i o n N a m e S t r i n g V a l u e ) ; SgFunctionCallExp ∗ functionCallExp = new S g F u n c t i o n C a l l E x p ( f i l e i n f o , f u n c t i o n R e f E x p r e s s i o n , e x p r e s s i o n L i s t , g l o b a l F u n c t i o n T y p e ) ;

68 70 72

//

c r e a t e an e x p r e s s i o n t y p e SgTypeVoid ∗ e x p r t y p e = new SgTypeVoid ( ) ;

76

// //

c r e a t e an e x p r e s s i o n r o o t SgExpressionRoot ∗ expr root

78

//

c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t ∗ n e w s t m t = new S g E x p r S t a t e m e n t ( f i l e i n f o , f u n c t i o n C a l l E x p ) ;

74 = new

SgExpressionRoot ( f i l e

info , functionCallExp , expr type );

Figure 40.2: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 2).

310

CHAPTER 40. CODE COVERAGE // // //

2

e x p r r o o t −> s e t p a r e n t ( n e w s t m t ) ; n e w s t m t −> s e t e x p r e s s i o n r o o t ( e x p r r o o t ) ; f u n c t i o n C a l l E x p −> s e t p a r e n t ( n e w s t m t −> g e t

expression

root ());

4 f u n c t i o n C a l l E x p −>s e t p a r e n t ( n e w s t m t ) ; 6 //

i n s e r t a statement i n t o the f u n c t i o n body f u n c t i o n D e f i n i t i o n −>g e t b o d y ()−> p r e p e n d s t a t e m e n t ( n e w s t m t ) ;

// // //

T h i s s h o w s t h e a l t e r n a t i v e u s e o f t h e ROSE However , t h e p e r f o r m a n c e i s n o t as good as b u i l d i n g t h e IR n o d e s ) .

//

string string

c o d e A t T o p O f B l o c k = ” v o i d p r i n t f ( c h a r ∗ ) ; p r i n t f ( \ ” FUNCTION NAME i n FILE NAME \\ n \ ” ) ; ” ; co d eA tT o pO fB l oc k = ” c o v e r a g e T r a c e F u n c 1 ( \ ”FUNCTION NAME i n FILE NAME \ ” ) ; ” ;

string string

functionTarget fileTarget

8 10

#i f

0

12

R e w r i t e Mechanism t o do t h e same t h i n g ! t h e v e r s i o n w r i t t e n a b o v e ( more d i r e c t l y

14 16 18

= ”FUNCTION NAME” ; = ”FILE NAME” ;

20 c od eA t To pO f Bl oc k . r e p l a c e ( co de A tT op O fB lo c k . f i n d ( f u n c t i o n T a r g e t ) , f u n c t i o n T a r g e t . s i z e ( ) , f u n c t i o n N a m e ) ; c o de At T op Of B lo ck . r e p l a c e ( co d eA tT o pO fB l oc k . f i n d ( f i l e T a r g e t ) , f i l e T a r g e t . s i z e ( ) , f i l e N a m e ) ;

22 24

( ” c o d e A t T o p O f B l o c k = %s \ n ” , c o d e A t T o p O f B l o c k . c s t r ( ) ) ; ( ”%s i n %s \n ” , f u n c t i o n N a m e . c s t r ( ) , f i l e N a m e . c s t r ( ) ) ;

//

printf printf

//

I n s e r t new c o d e i n t o t h e s c o p e r e p r e s e n t e d b y t h e s t a t e m e n t ( a p p l i e s t o S g S c o p e S t a t e m e n t s ) MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ;

26 28

S g B a s i c B l o c k ∗ f u n c t i o n B o d y = f u n c t i o n D e f i n i t i o n −>g e t b o d y ( ) ; ROSE ASSERT( f u n c t i o n B o d y != NULL ) ;

30 32 // 34

I n s e r t t h e new c o d e a t t h e t o p o f t h e s c o p e r e p r e s e n t e d b y b l o c k M i d d l e L e v e l R e w r i t e : : i n s e r t ( f u n c t i o n B o d y , codeAtTopOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : T o p O f C u r r e n t S c o p e ) ;

#e n d i f }

36 38 40 42 44 46 48

} #e n d i f void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode ∗ a s t N o d e ) { S g B a s i c B l o c k ∗ b l o c k = NULL ; b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s S g F i l e I n f o ∗ f i l e I n f o = b l o c k −> g e t f i l e i n f o ( ) ; s t r i n g f i l e N a m e = f i l e I n f o −>g e t f i l e n a m e ( ) ; i n t lineNum = f i l e I n f o −>g e t l i n e ( ) ;

50

of

these

functions

link

time

/ / B u i l d a s o u r c e f i l e l o c a t i o n o b j e c t ( f o r c o n s t r u c t i o n o f e a c h IR n o d e ) / / N o t e t h a t we s h o u l d n o t b e s h a r i n g t h e s a m e S g F i l e I n f o o b j e c t i n m u l t i p l e IR n o d e s . Sg File Info ∗ newCallfileInfo = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ;

52 54

ROSE ASSERT( f u n c t i o n S y m b o l != NULL ) ; S g F u n c t i o n R e f E x p ∗ f u n c t i o n R e f E x p r e s s i o n = new S g F u n c t i o n R e f E x p ( n e w C a l l f i l e I n f o , f u n c t i o n S y m b o l , g l o b a l F u n c t i o n T y p e ) ; SgExprListExp ∗ e x p r e s s i o n L i s t = new S g E x p r L i s t E x p ( n e w C a l l f i l e I n f o ) ;

56 58

s t r i n g c o d e L o c a t i o n = f i l e N a m e + ” ” + S t r i n g U t i l i t y : : n u m b e r T o S t r i n g ( lineNum ) ; S g S t r i n g V a l ∗ f u n c t i o n N a m e S t r i n g V a l u e = new S g S t r i n g V a l ( n e w C a l l f i l e I n f o , ( char ∗ ) c o d e L o c a t i o n . c s t r ( ) ) ; e x p r e s s i o n L i s t −>a p p e n d e x p r e s s i o n ( f u n c t i o n N a m e S t r i n g V a l u e ) ; SgFunctionCallExp ∗ functionCallExp = new S g F u n c t i o n C a l l E x p ( n e w C a l l f i l e I n f o , f u n c t i o n R e f E x p r e s s i o n , e x p r e s s i o n L i s t , g l o b a l F u

60 62

66

// // // //

c r e a t e an e x p r e s s i o n t y p e S g T y p e V o i d ∗ e x p r t y p e = new S g T y p e V o i d ( ) ; c r e a t e an e x p r e s s i o n r o o t S g E x p r e s s i o n R o o t ∗ e x p r r o o t = new S g E x p r e s s i o n R o o t ( n e w C a l l f i l e I n f o , f u n c t i o n C a l l E x p , e x p r t y p e ) ;

68

//

c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t ∗ n e w s t m t = new S g E x p r S t a t e m e n t ( n e w C a l l f i l e I n f o , f u n c t i o n C a l l E x p ) ;

// // //

e x p r r o o t −> s e t p a r e n t ( n e w s t m t ) ; n e w s t m t −> s e t e x p r e s s i o n ( e x p r r o o t ) ; f u n c t i o n C a l l E x p −> s e t p a r e n t ( n e w s t m t −> g e t e x p r e s s i o n ( ) ) ; f u n c t i o n C a l l E x p −>s e t p a r e n t ( n e w s t m t ) ;

//

insert a statement into the function b l o c k −>p r e p e n d s t a t e m e n t ( n e w s t m t ) ;

64

70 72 74 76

body

78 } 80

}

Figure 40.3: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 3).

311

2 4 6 8 10 12 14 16 18

void f o o ( ) { // Should i f ( true ) int x } else { int x } }

d e t e c t t h a t f o o IS c a l l e d { = 3;

= 4;

void f o o b a r ( ) { i n t y =4; switch ( y ) { case 1 : // h e l l o w o r l d break ; case 2 :

20 case 3 : 22 default : { // }

24 26 28

} // Should d e t e c t t h a t f o o b a r i s NOT c a l l e d }

30 32 34 36

i n t main ( ) { i f ( true ) { } foo ( ) ; return 0 ; }

Figure 40.4: Example source code used as input to translator adding new function.

312

CHAPTER 40. CODE COVERAGE

extern ”C” void c o v e r a g e T r a c e F u n c 1 ( char ∗ t e x t S t r i n g ) ; 2 4 6 8 10 12 14

void f o o ( ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCode // Should d e t e c t t h a t f o o IS c a l l e d i f ( true ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCo int x = 3 ; } else { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCo int x = 4 ; } }

16 18 20 22 24 26 28 30 32 34

void f o o b a r ( ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCode int y = 4 ; switch ( y ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCo case 1 : // h e l l o w o r l d break ; case 2 : ; case 3 : ; default : { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputC // } } // Should d e t e c t t h a t f o o b a r i s NOT c a l l e d }

36 38 40 42 44

i n t main ( ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCode i f ( true ) { c o v e r a g e T r a c e F u n c 1 ( ” / e x p o r t /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / inputCo } foo ( ) ; return 0 ; }

Figure 40.5: Output of input to translator adding new function.

Chapter 41

Bug Seeding Bug seeding is a technique used to construct example codes from existing codes which can be used to evaluate tools for the finding bugs in randomly selected existing applications. The idea is to seed an existing application with a known number of known bugs and evaluate the bug finding or security tool based on the percentage of the number of bugs found by the tool. If the bug finding tool can identify all known bugs then there can be some confidence that the tool detects all bugs of that type used to seed the application. This example tutorial code is a demonstration of a more complete technique being developed in collaboration with NIST to evaluate tools for finding security flaws in applications. It will in the future be a basis for testing of tools built using ROSE, specifically Compass, but the techniques are not in any way specific to ROSE or Compass.

41.1

Input For Examples Showing Bug Seeding

Figure ?? shows the example input used for demonstration of bug seeding as a transformation.

2 4

void f o o b a r ( ) { // S t a t i c a r r a y d e c l a r a t i o n float array [ 1 0 ] ;

6 array [ 0 ] = 0; 8 f o r ( i n t i =0; i < 5 ; { array [ i ] = 0; }

10 12

i ++)

}

Figure 41.1: Example source code used as input to program in codes used in this chapter.

313

314

41.2

CHAPTER 41. BUG SEEDING

Generating the code representing the seeded bug

Figure 41.2 shows a code that traverses each IR node and for and modifies array reference index expressions to be out of bounds. The input code is shown in figure 41.1, the output of this code is shown in figure 41.3.

41.2. GENERATING THE CODE REPRESENTING THE SEEDED BUG

2

// // //

This example demonstrates the of bug ( b u f f e r o v e r f l o w ) i n t o t e s t bug f i n d i n g t o o l s .

seeding of a any e x i s t i n g

315

s p e c i f i c type a p p l i c a t i o n to

4 6

#i n c l u d e ” r o s e . h ” u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ;

8 namespace S e e d B u g s A r r a y I n d e x i n g { 10 12

class {

14 16 18

};

20

class {

22

32 34 36 38

:

public

{}

SgTopDownProcessing

(

};

26

30

BugSeeding

public : InheritedAttribute evaluateInheritedAttribute SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute );

24

28

InheritedAttribute public : bool i s L o o p ; bool i s V u l n e r a b i l i t y ; I n h e r i t e d A t t r i b u t e ( ) : i s L o o p ( f a l s e ) , i s V u l n e r a b i l i t y ( f a l s e ) {} I n h e r i t e d A t t r i b u t e ( c o n s t I n h e r i t e d A t t r i b u t e & X) : i s L o o p (X . i s L o o p ) , i s V u l n e r a b i l i t y (X . i s V u l n e r a b i l i t y )

InheritedAttribute BugSeeding : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode ∗ a s t N o d e , InheritedAttribute inheritedAttribute ) { / / U s e t h i s i f we o n l y w a n t t o s e e d b u g s i n l o o p s bool i s L o o p = i n h e r i t e d A t t r i b u t e . i s L o o p || ( i s S g F o r S t a t e m e n t ( a s t N o d e ) != NULL) | | ( i s S g W h i l e S t m t ( a s t N o d e ) != NULL) || ( i s S g D o W h i l e S t m t ( a s t N o d e ) != NULL ) ; / / Add F o r t r a n s u p p o r t i s L o o p = i s L o o p | | ( i s S g F o r t r a n D o ( a s t N o d e ) != NULL ) ;

40 //

Mark f u t u r e n o e s i n t h i s s u b t r e e a s i n h e r i t e d A t t r i b u t e . isLoop = isLoop ;

//

To t e s t t h i s o n s i m p l e c o d e s , bool a p p l y E v e r y W h e r e = t r u e ;

42 44

being

optionally

part

of

a

loop

allow

it

to

be

applied

everywhere

46 if 48 50 52 54

( i s L o o p == t r u e | | a p p l y E v e r y W h e r e == t r u e ) { / / The i n h e r i t e d a t t r i b u t e i s t r u e i f f we a r e i n s i d e a l o o p a n d t h i s SgPntrArrRefExp ∗ a r r a y R e f e r e n c e = isSgPntrArrRefExp ( astNode ) ; i f ( a r r a y R e f e r e n c e != NULL) { // Mark a s a v u l n e r a b i l i t y i n h e r i t e d A t t r i b u t e . i s V u l n e r a b i l i t y = true ;

56

//

58 60 62 64 66 // 68

is

a

SgPntrArrRefExp .

Now c h a n g e t h e a r r a y i n d e x ( t o s e e d t h e b u f f e r o v e r f l o w b u g ) SgVarRefExp ∗ a r r a y V a r R e f = i s S g V a r R e f E x p ( a r r a y R e f e r e n c e −>g e t l h s o p e r a n d ( ) ) ; ROSE ASSERT( a r r a y V a r R e f != NULL ) ; ROSE ASSERT( a r r a y V a r R e f −>g e t s y m b o l ( ) != NULL ) ; S g I n i t i a l i z e d N a m e ∗ arrayName = i s S g I n i t i a l i z e d N a m e ( a r r a y V a r R e f −>g e t s y m b o l ()−> g e t d e c l a r a t i o n ( ) ) ; ROSE ASSERT( arrayName != NULL ) ; SgArrayType ∗ a r r a y T y p e = i s S g A r r a y T y p e ( arrayName−>g e t t y p e ( ) ) ; ROSE ASSERT( a r r a y T y p e != NULL ) ; S g E x p r e s s i o n ∗ a r r a y S i z e = a r r a y T y p e−>g e t i n d e x ( ) ; SgTreeCopy c o p y H e l p ; Make a c o p y o f t h e e x p r e s s i o n u s e d t o h o l d t h e a r r a y s i z e i n t h e a r r a y d e c l a r a t i o n . S g E x p r e s s i o n ∗ a r r a y S i z e C o p y = i s S g E x p r e s s i o n ( a r r a y S i z e −>c o p y ( c o p y H e l p ) ) ; ROSE ASSERT( a r r a y S i z e C o p y != NULL ) ;

70 //

This i s the e x i s t i n g index e x p r e s s i o n S g E x p r e s s i o n ∗ i n d e x E x p r e s s i o n = a r r a y R e f e r e n c e −>g e t r h s o p e r a n d ( ) ; ROSE ASSERT( i n d e x E x p r e s s i o n != NULL ) ;

//

B u i l d a n e w e x p r e s s i o n : ” a r r a y [ n ] ” −−> ” a r r a y [ n+ a r r a y S i z e C o p y ] ” , w h e r e t h e a r r a y S i z e C o p y S g E x p r e s s i o n ∗ n e w I n d e x E x p r e s s i o n = buildAddOp ( i n d e x E x p r e s s i o n , a r r a y S i z e C o p y ) ;

78

//

S u b s t i t u t e t h e new e x p r e s s i o n f o r t h e o l d e x p r e s s i o n a r r a y R e f e r e n c e −>s e t r h s o p e r a n d ( n e w I n d e x E x p r e s s i o n ) ;

80

}

72 74 76

} 82 return

inheritedAttribute ;

}

84 } 86 88 90

int main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

See

Rose : : i n i t i a l i z e

92 SgProject ∗ p r o j e c t = f r o n t e n d ( argc , ROSE ASSERT ( p r o j e c t != NULL ) ;

94 96

argv ) ;

SeedBugsArrayIndexing : : BugSeeding t r e e T r a v e r s a l ; SeedBugsArrayIndexing : : I n h e r i t e d A t t r i b u t e i n h e r i t e d A t t r i b u t e ;

98 treeTraversal . traverseInputFiles

( project , inheritedAttribute ) ;

100 //

Running i n t e r n a l t e s t s ( o p t i o n a l ) AstTests : : runAllTests ( p r o j e c t ) ;

104

//

Output return

106

}

102

t h e new backend

code seeded ( project );

with

a

specific

form

of

bug .

Figure 41.2: Example source code showing how to seed bugs.

is

a

size

of

” array ”

316

2 4 6 8 10

CHAPTER 41. BUG SEEDING

void f o o b a r ( ) { // S t a t i c a r r a y d e c l a r a t i o n float array [ 1 0 ] ; array [0 + 10] = 0; f o r ( i n t i = 0 ; i < 5 ; i ++) { array [ i + 10] = 0; } }

Figure 41.3: Output of input code using seedBugsExample arrayIndexing.C

Part VI

Binary Support

Tutorials of using ROSE to handle binary executable files.

317

Chapter 42

Instruction Semantics The Instruction Semantics layer in ROSE can be used to “evaluate” instructions and is controlled by a policy that defines the details of what “evaluate” means. For instance, given the following “xor” instruction, the instruction semantics classes specify that the value of the “eax” and “edx” registers are read, those two 32-bit values are xor’d together, and the 32-bit result is then written to the “eax” register. The policy defines what a 32-bit value is (it could be an integer, some representation of a constant, etc), how it is read and written to the registers, and how to compute an xor. xor eax, edx ROSE has a collection instruction semantic classes, one for each architecture. It also has a small collection of policies. This chapter briefly describes a policy that tracks constant values. See the doxygen documentation for namespace Rose::BinaryAnalysis::InstructionSemantics2 for details.

319

320

CHAPTER 42. INSTRUCTION SEMANTICS

Chapter 43

Binary Analysis See doxgyen “Getting started with binary analysis” chapter.

43.1

DataFlow Analysis

Based on the control flow many forms of dataflow analysis may be performed. Dataflow analyses available are:

43.1.1

Def-Use Analysis

Definition-Usage is one way to compute dataflow information about a binary program.

43.1.2

Variable Analysis

This analysis helps to detect different types within a binary. Currently, we use this analysis to detect interrupt calls and their parameters together with the def-use analysis. This allows us to track back the value of parameters to the calls, such as eax and therefore determine whether a interrupt call is for instance a write or read. Another feature is the buffer overflow analysis. By traversing the CFG, we can detect buffer overflows.

43.2

Dynamic Analysis

ROSE has support for dynamic analysis and for mixing of dynamic and static analysis using the Intel Pin framework. This optional support in ROSE requires a configure option (--with-IntelPin=< path>). The path in the configure option is the absolute path to the top level directory of the location of the Intel Pin distribution. This support for Intel Pin has only been tested on a 64bit Linux system using the most recent distribution of Intel Pin (version 2.6). Note: The dwarf support in ROSE is currently incompatable with the dwarf support in Intel Pin. A message in the configuration of ROSE will detect if both support for Dwarf and Intel Pin are both specified and exit with an error message that they are incompatable options. 321

322

CHAPTER 43. BINARY ANALYSIS

See tutorial/intelPin directory for examples using static and dynamic analysis. These example will be improved in the future, at the moment they just call the generation of the binary AST. Note: We have added a fix to Intel Pin pin.H file: // DQ (3/9/2009): Avoid letting "using" declarations into header files. #ifndef REQUIRE_PIN_TO_USE_NAMESPACES using namespace LEVEL_PINCLIENT; #endif so that the namespace of Intel Pin would not be a problem for ROSE. The development team have suggested that they may fix their use of ”using” declarations for namespaces in their header files. Also note that the path must be absolute since it will be the prefix for the pin executable to be run in the internal tests and anything else might be a problem if the path does not contain the current directory (”.”). Or, perhaps we should test for this in the future. Note 2: Linking to libdwarf.a is a special problem. Both ROSE and Intel Pin use libdwarf.a and both build shred libraries that link to the static version of the library (libdwarf.a). This is a problem im building Pin Tools since both the PinTool and librose.so will use a statically linked dwarf library (internally). This causes the first use of dwarf to fail, because there are then two versions of the same library in place. The solution is to force at most one static version of the library and let the other one be a shared library. Alternatively both the Pin tool and librose.so can be built using the shared version of dwarf (libdwarf.so). There is a makefile rule in libdwarf to build the shared version of the library, but the default is to only build the static library (libdwarf.a), so use make make libdwarf.so to build the shared library. So we allow ROSE to link to the libdwarf.a (statically), which is how ROSE has always worked (this may be revisited in the future). And we force the Pin tool to link using the shared dwarf library (libdwarf.so). Note: The specification of the location of libdwarf.so in the Intel Pin directory structure is problematic using rpath, so for the case of using the Intel Pin package with ROSE please set the LD LIBRARY PATH explicitly (a better solution using rpath may be made available in the future).

43.3

Analysis and Transformations on Binaries

This section demonstrates how the binary can be analyized and transformed via operations on the AST. In this tutorial example we will recognize sequences of NOP (No OPeration) instructions (both single byte and less common multi-byte NOPs) and replace them in the binary with as few multi-byte NOP instructions as required to overwrite the identified NOP sequence (also called a nop sled). In the following subsections we will demonstrate three example codes that work together to demonstrate aspects of the binary analysis and transforamation using ROSE. All of these codes are located in the directory tutorial/binaryAnalysis of the ROSE distribution. We show specifically: 1. How to insert NOP sequences into binaries via source-to-source transformations. The tutorial example will demonstrate the trival insertion of random length nop sequences into

43.3. ANALYSIS AND TRANSFORMATIONS ON BINARIES

323

all functions of an input source code application. The purpose of this is to really just generate arbitrary test codes upon which to use in sperately demonstrating (and testing) the binary analysis support and the binary transformation support (next). This example is shown in figure 43.1. The file name is: 2. How to identify sequences of NOP instructions in the binary (nop sleds). The tutorial example show how to identify arbitrary NOP sequences in the binary. Note that our approach looks only for single instructions that have no side-effect operations and thus qualify as a NOP, and specifically does not look at collections of multiple statements that taken togehter have no side-effect and thus could be considered a NOP sequence. Our test on each instruciton is isolated to the SageInterface::isNOP(SgAsmInstruction*) function. Initially this function only detects NOP instructions that are single and multibyte Intel x86 suggested NOP instructions. The catagory of NOP instructions that have no side-effects is broader than this and will be addressed more completely in the future. This example is shown in figures 43.2, 43.3, and 43.4. 3. How transformations on the binary executable are done to support rewriting all identified NOP sequences as Intel x86 suggested NOP instructions. Importantly, this tutorial example of an AST rewrite does not change the size of any part of the final executable. This example is shown in figure 43.5.

43.3.1

Source-to-source transformations to introduce NOPs

This tutorial example (see figure 43.1) is a source-to-source trnasformation that is used to generate test codes for the binary NOP sequence detection example and the NOP sequence transformation example (the next two tutorial examples). This tutorial uses C language asm statements that are inserted into each function in the source code, the resulting generated source code (with asm statements) is then compiled to generate inputs to use in our detection and transformation of NOP sequences. Although it is possible, the backend compiler (at least the one we are using), does not optimize away these asm statements that represent NOP instructions. In general all C and C++ compilers turn off optimizations upon encountering the asm language construct, so it is easy to use this approach to build binaries from arbitrary source code that have a range of properties. Seeding the source code with such properties causes the binary to have the similar properties. We include this example in the tutorial because it is a cute example of how we can combine source code analsys with binary analysis (in this case we are only supporting testing of the binary analysis). Much more interesting example of the connection of source code and binary analysis are available.

43.3.2

Detection of NOP sequences in the binary AST

The tutorial example (see figures 43.2, 43.3, and 43.4) shows the detection of NOP sequences and is separated into three figures (the header file, the source code, and the main program). The header file and source file will be reused in the next tutorial example to demonstrate the transformation of the NOP sequences that this tutorial identifies. The transformation will be done on the AST (and a new modified executable generated by unparsing the AST).

324

2

CHAPTER 43. BINARY ANALYSIS

// This t r a n s l a t o r do es a source−to−s o u r c e t r a n s f o r m a t i o n on // t h e i n p u t s o u r c e code t o i n t r o d u c e asm s t a t e m e n t s o f NOP // i n s t r u c t i o n s a t t h e t o p o f each f u n c t i o n .

4 #include ” r o s e . h” 6 8

using namespace s t d ; using namespace S a g e I n t e r f a c e ; using namespace S a g e B u i l d e r ;

10 12 14 16 18 20 22 24

c l a s s NopTransform : public S g S i m p l e P r o c e s s i n g { public : NopTransform ( ) { s r a n d ( t i m e (NULL) ) ; } void v i s i t ( SgNode∗ n ) ; }; void NopTransform : : v i s i t ( SgNode∗ n ) { SgFunctionDefinition ∗ functionDefinition = isSgFunctionDefinition (n ) ; i f ( f u n c t i o n D e f i n i t i o n != NULL) { // I n t r o d u c e NOP’ s i n t o f u n c t i o n d e f i n i t i o n S g B a s i c B l o c k ∗ f u n c t i o n B o d y = f u n c t i o n D e f i n i t i o n −>g e t b o d y ( ) ; ROSE ASSERT( f u n c t i o n B o d y != NULL ) ;

26 // Use a m u l t i −b y t e NOP j u s t f o r added fun . // SgAsmStmt∗ nopStatement = buildAsmStatement (” nop ” ) ;

28

// Generate a s i n g l e random m u l t i −b y t e NOP (1−9 b y t e s l o n g i n c l u d i n g no NOP, when n == 0) i n t n = rand ( ) % 1 0 ; i f (n > 0) { p r i n t f ( ” I n t r o d u c i n g a m u l t i −b y t e NOP i n s t r u c t i o n ( n = %d ) a t t h e t o p o f f u n c t i o n %s \n” , n , f u n c SgAsmStmt∗ nopStatement = b u i l d M u l t i b y t e N o p S t a t e m e n t ( n ) ;

30 32 34 36

// Add t o t h e f r o n t o t h e l i s t o f s t a t e m e n t s i n t h e f u n c t i o n body p r e p e n d S t a t e m e n t ( nopStatement , f u n c t i o n B o d y ) ; }

38 }

40 } 42 44 46 48

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; // Generate t h e ROSE AST. SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ;

50 52 54

// AST c o n s i s t e n c y t e s t s ( o p t i o n a l f o r u s e r s , b u t t h i s e n f o r c e s more o f our t e s t s ) AstTests : : runAllTests ( p r o j e c t ) ; NopTransform t ; t . traverse ( project , preorder ) ;

56 58

// r e g e n e r a t e t h e o r i g i n a l e x e c u t a b l e . return backend ( p r o j e c t ) ; }

Figure 43.1: Source-to-source transformation to introduce NOP assemble instructions in the generated binary executable.

43.3. ANALYSIS AND TRANSFORMATIONS ON BINARIES

2 4 6 8

325

c l a s s C o u n t T r a v e r s a l : public S g S i m p l e P r o c e s s i n g { public : // L o c a l Accumulator A t t r i b u t e int count ; bool p r e v i o u s I n s t r u c t i o n W a s N o p ; SgAsmInstruction ∗ nopSequenceStart ; s t d : : v e c t o r > n o p S e q u e n c e s ;

10 C o u n t T r a v e r s a l ( ) : c o u n t ( 0 ) , p r e v i o u s I n s t r u c t i o n W a s N o p ( f a l s e ) {} void v i s i t ( SgNode∗ n ) ;

12 };

Figure 43.2: Header file for the traversal used to identify the NOP sequences in the binary executable. Using a simple preorder traversal over the AST we identify independent sequences of NOP instructions. This traversal save the sequence so that it can be used in both the tutorial example that detects the NOP sequences and also the tutorial example that will transform them to be more compact multi-byte NOP sequences that will use the suggested Intel x86 multi-byte NOP instructions. In this example, the sequence of a NOP sequence is saved as the address of the starting NOP instruction and the number of instructions in the sequence. The function SageInterface::isNOP(SgAsmInstruction*) is used to test of an instruction is a NOP. Presently this test only identifies single or multi-byte NOP instrucitons that have been suggested for use as single and multi-byte NOP instructions by Intel. Other instruction may semantically be equivalent to a NOP instruction and they are not identified in this initial work. Later work may include this capability and for this purpose we have isolated out the SageInterface::isNOP(SgAsmInstruction*). Note also that sequences of instructions may be semantically equivalent to a NOP sequence and we also do not attempt to identify these (separate interface functions in the SageInterface namespace have been defined to support this work; these functions are presently unimplemented.

43.3.3

Transformations on the NOP sequences in the binary AST

Using the results from the previously presented traversal to identify NOP sequences, we demonstrate the transformation to change these locations of the binary (in the AST) and then regenerate the executable to have a different representation. The new representation is a more clear (obvious within manual binary analysis) and likely more compressed representation using only the suggested Intel x86 multi-byte NOP.

43.3.4

Conclusions

FIXME: Not sure if the

The identification and transformation have been deomonstrated on the AST and can be ex- resulting executable will use the assembler on each instruction in pressed in the binary binary by calling the backend() function in ROSE; which will regenerate the generated binary from the the executable. The capability to regenerate the executable will not always result in a properly unparsing within ROSE. This might require some more work. formed executable that can be executed, this depends upon the transformations done and does So at the moment the AST is ROSE does not have specialized support for this beyond regnerating the binary executable from transformed and I have to look the AST. In the case of this NOP transformation we have not changed the size of any part into if the binary sees the effect of the transformation in the AST.

326

CHAPTER 43. BINARY ANALYSIS

of the binary so any relative offsets need not be fixed up. Assumeing we have not accedentaly interpreted data that had values that matched parts of the opcodes that were transformed, then resulting binary executable should execute without problems. This transformation however makes clear how critical it is that data not be interpreted as instructions (which could randomly be interpreted as NOPs in the case of this tutorial example).

43.3. ANALYSIS AND TRANSFORMATIONS ON BINARIES

327

#include ” r o s e . h” 2 4

#include ” s a g e I n t e r f a c e A s m . h” #include ” d e t e c t N o p S e q u e n c e s T r a v e r s a l . h” #include ” s t r i n g i f y . h”

6 8

using namespace s t d ; using namespace Rose ; using namespace S a g e I n t e r f a c e ;

10 12 14 16 18 20 22 24 26 28

void C o u n t T r a v e r s a l : : v i s i t ( SgNode∗ n ) { SgAsmInstruction ∗ asmInstruction = isSgAsmInstruction (n ) ; i f ( a s m I n s t r u c t i o n != NULL) { // Use t h e new i n t e r f a c e s u p p o r t f o r t h i s ( t h i s d e t e c t s a l l m u l t i −b y t e nop i n s t r u c t i o n s ) . i f ( S a g e I n t e r f a c e : : isNOP ( a s m I n s t r u c t i o n ) == true ) { i f ( p r e v i o u s I n s t r u c t i o n W a s N o p == true ) { // Increment t h e l e n g t h o f t h e i d e n t i f i e d NOP s e q u e n c e c o u n t++; } else { count = 1 ; // Record t h e s t a r t i n g a d d r e s s o f t h e NOP s e q u e n c e nopSequenceStart = asmInstruction ; }

30 p r e v i o u s I n s t r u c t i o n W a s N o p = true ; } else { i f ( count > 0) { // Report t h e s e q u e n c e when we have d e t e c t e d t h e end o f t h e s e q u e n c e . SgAsmFunction ∗ f u n c t i o n D e c l a r a t i o n = getAsmFunction ( a s m I n s t r u c t i o n ) ; p r i n t f ( ” R e p o r t i n g NOP s e q u e n c e o f l e n g t h %3d a t a d d r e s s %zu i n f u n c t i o n %s ( r e a s o n f o r t h i s b e i n g a count , n o p S e q u e n c e S t a r t −>g e t a d d r e s s ( ) , f u n c t i o n D e c l a r a t i o n −>get name ( ) . c s t r ( ) , f u n c t i o n D e c l a r a t i o n −>g e t r e a s o n ( ) , s t r i n g i f y S g A s m F u n c t i o n F u n c t i o n R e a s o n ( f u n c t i o n D e c l a r a t i o n −>g e t r e a s o n ( ) ) . c s t r ( ) ) ;

32 34 36 38 40 42 44

n o p S e q u e n c e s . p u s h b a c k ( p a i r ( n o p S e q u e n c e S t a r t , c o u n t ) ) ;

46

SgAsmBlock ∗ b l o c k = isSgAsmBlock ( n o p S e q u e n c e S t a r t −>g e t p a r e n t ( ) ) ; ROSE ASSERT( b l o c k != NULL ) ; S g A s m S t a t e m e n t P t r L i s t & l = b l o c k −>g e t s t a t e m e n t L i s t ( ) ;

48

// Now i t e r a t e o v e r t h e nop i n s t r u c t i o n s i n t h e s e q u e n c e and r e p o r t t h e l e n g h t o f each ( can be m u l t i −b y S g A s m S t a t e m e n t P t r L i s t : : i t e r a t o r i = f i n d ( l . b e g i n ( ) , l . end ( ) , n o p S e q u e n c e S t a r t ) ; ROSE ASSERT( i != l . end ( ) ) ; int counter = 0 ; while ( ( ∗ i != a s m I n s t r u c t i o n ) && ( i != l . end ( ) ) ) { p r i n t f ( ”−−− NOP #%2d i s l e n g t h = %2d \n” , c o u n t e r ++,( i n t ) i s S g A s m I n s t r u c t i o n ( ∗ i )−> g e t r a w b y t e s ( i ++; } }

50 52 54 56 58 60

count = 0 ; previousInstructionWasNop = false ;

62 } }

64 }

Figure 43.3: Example code to identify the NOP sequences in the binary executable.

328

2 4 6 8

// // // // // //

CHAPTER 43. BINARY ANALYSIS

This example ROSE t r a n s l a t o r j u s t do es an a n a l y s i s o f t h e i n p u t b i n a r y . The e x i s t e n c e o f NOP s e q u e n c e s a r e d e t e c t e d and r e p o r t e d . For each NOP i n t h e s e q u e n c e t h e s i z e o f t h e NOP i n s t r u c t i o n i s r e p o r t e d . Note t h a t a l l m u l t i −b y t e NOP i n s t r u c t i o n s a r e d e t e c t e d and so t h e r e p o r t e d s i z e o f each i n s t r u c t i o n i n t h e NOP s e q u e n c e can v a r y . I n t e l m u l t i −b y t e NOP i n s t r u c t i o n s can be 1−9 b y t e s l o n g .

#include ” r o s e . h” #include ” d e t e c t N o p S e q u e n c e s T r a v e r s a l . h”

10 12 14 16

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; // Generate t h e ROSE AST. SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ;

18 20 22

// AST c o n s i s t e n c y t e s t s ( o p t i o n a l f o r u s e r s , b u t t h i s e n f o r c e s more o f our t e s t s ) AstTests : : runAllTests ( p r o j e c t ) ; CountTraversal t ; t . traverse ( project , preorder ) ;

24 26

// r e g e n e r a t e t h e o r i g i n a l e x e c u t a b l e . return backend ( p r o j e c t ) ; }

Figure 43.4: Main program using the traversal to identify the NOP sequences in the binary executable.

43.3. ANALYSIS AND TRANSFORMATIONS ON BINARIES

2

// //

T h i s t u t o r i a l e x a m p l e s h o w how t o i n t r o d u c e t r a n s f o r m a t i o n s on t h e i n p u t b i n a r y e x e c t u t a b l e .

4

#i n c l u d e

” r o s e . h”

6

#i n c l u d e

” s a g e I n t e r f a c e A s m . h”

8

#i n c l u d e

” d e t e c t N o p S e q u e n c e s T r a v e r s a l . h”

10 12

329

using using using using

namespace s t d ; namespace Rose ; namespace S a g e I n t e r f a c e ; namespace S a g e B u i l d e r A s m ;

14 16

class {

18

NopReplacementTraversal

:

public

SgSimpleProcessing

public : // L o c a l A c c u m u l a t o r A t t r i b u t e s t d : : v e c t o r > & n o p S e q u e n c e s ; s t d : : v e c t o r > : : i t e r a t o r l i s t I t e r a t o r ; SgAsmStatementPtrList d e l e t e L i s t ;

20 22

N o p R e p l a c e m e n t T r a v e r s a l ( s t d : : v e c t o r > & X) { l i s t I t e r a t o r = nopSequences . begin ( ) ; }

24 26

:

n o p S e q u e n c e s (X)

28 void 30

visit

(

SgNode ∗ n

);

};

32 34 36 38 40

void N o p R e p l a c e m e n t T r a v e r s a l : : v i s i t ( SgNode ∗ n ) { SgAsmInstruction∗ asmInstruction = isSgAsmInstruction (n ) ; i f ( a s m I n s t r u c t i o n != NULL && a s m I n s t r u c t i o n == l i s t I t e r a t o r −> f i r s t ) { / / T h i s i s t h e i n s t r u c t i o n i n t h e AST t h a t m a t c h e s t h e s t a r t o f o n e SgAsmBlock ∗ b l o c k = i s S g A s m B l o c k ( a s m I n s t r u c t i o n −>g e t p a r e n t ( ) ) ; i n t n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s = l i s t I t e r a t o r −>s e c o n d ;

42 44

printf

NOP

sequences .

( ” n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s = %d \n ” , n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ) ;

S g A s m B l o c k ∗ b l o c k = i s S g A s m B l o c k ( a s m I n s t r u c t i o n −> g e t p a r e n t ( ) ) ; ROSE ASSERT( b l o c k != NULL ) ; S g A s m S t a t e m e n t P t r L i s t & l = b l o c k −>g e t s t a t e m e n t L i s t ( ) ;

//

Now i t e r a t e o v e r t h e n o p i n s t r u c t i o n s i n t h e s e q u e n c e a n d r e p o r t t h e l e n g h t o f e a c h ( c a n b e m u l t i − b y t e S g A s m S t a t e m e n t P t r L i s t : : i t e r a t o r i = f i n d ( l . b e g i n ( ) , l . end ( ) , a s m I n s t r u c t i o n ) ; ROSE ASSERT( i != l . end ( ) ) ; int n o p s l e d s i z e = 0; f o r ( i n t j = 0 ; j < n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ; j ++) { ROSE ASSERT( i != l . end ( ) ) ; p r i n t f ( ”−−− NOP #%2d i s l e n g t h = %2d \n ” , j , ( i n t ) i s S g A s m I n s t r u c t i o n ( ∗ i )−> g e t r a w b y t e s ( ) . s i z e ( ) ) ; n o p s l e d s i z e += ( i n t ) i s S g A s m I n s t r u c t i o n ( ∗ i )−> g e t r a w b y t e s ( ) . s i z e ( ) ; i ++; }

// //

From t h e s i z e o f t h e NOP s l e d , c o m p u t e h o w many m u l t i − b y t e T h i s c o d e i s s p e c i f i c t o x 8 6 s u p p o r t i n g m u l t i − b y t e NOPs i n c o n s t i n t MAX SIZE MULTIBYTE NOP = 9 ; i n t numberOfNopSizeN [ MAX SIZE MULTIBYTE NOP + 1 ] ;

//

0 th element of numberOfNopSizeN i s not used . numberOfNopSizeN [ 0 ] = 0 ; f o r ( i n t i = MAX SIZE MULTIBYTE NOP ; i > 0 ; i −−) { // i n t n u m b e r O f N o p S i z e 9 = n o p s l e d s i z e / 9 ; numberOfNopSizeN [ i ] = n o p s l e d s i z e / i ; n o p s l e d s i z e −= numberOfNopSizeN [ i ] ∗ i ; i f ( numberOfNopSizeN [ i ] > 0 ) p r i n t f ( ” numberOfNopSizeN [%d ] = %d \n ” , i , numberOfNopSizeN [ i ] ) ; }

// //

Now r e w r i t e t h e AST t o u s e t h e n u m b e r o f m u l t i − b y t e NOPS s p e c i f i e d i n t h e n u m b e r O f N o p S i z e N a r r a y f o r e a c h s i z e o f m u l t i − b y t e NOP .

50 52 54 56 58 60 62

the

//

46 48

of

printf

64

nop

instructions ).

( ” n o p s l e d s i z e = %d \n ” , n o p s l e d s i z e ) ; NOPs w e w i l l w a n t t o u s e s i z e d 1−9 b y t e s l o n g .

to

cover

the

same

space

in

the

binary .

66 68 70 72 74 76 78 80 printf

( ” Rewrite

t h e AST h e r e !

\n ” ) ;

82 // 84 86 88 90 92

I g n o r e t h e 0 t h e l e m e n t o f n u m b e r O f N o p S i z e N ( s i n c e a 0 l e n g t h NOP d o e s n o t m a k e s e n s e ) . f o r ( i n t i = 1 ; i <= MAX SIZE MULTIBYTE NOP ; i ++) { / / B u i l d t h i s many b y t e s o f t h i s s i z e f o r ( i n t j = 0 ; j < numberOfNopSizeN [ i ] ; j ++) { / / We w a n t t o b u i l d a ( b i n a r y AST n o d e ) S g A s m I n s t r u c t i o n o b j e c t i n s t e a d o f a ( s o u r c e // SgAsmStmt∗ n o p S t a t e m e n t = b u i l d M u l t i b y t e N o p S t a t e m e n t ( i ) ; / / S g A s m I n s t r u c t i o n ∗ m u l t i B y t e N o p = m a k e I n s t r u c t i o n ( x 8 6 n o p , ” n o p ” , modrm ) ; // S g A s m I n s t r u c t i o n ∗ m u l t i B y t e N o p I n s t r u c t i o n = b u i l d M u l t i b y t e N o p I n s t r u c t i o n ( i ) ; SgAsmInstruction∗ multiByteNopInstruction = buildX86MultibyteNopInstruction ( i ) ;

code

AST

node )

SgAsmStmt .

94 // //

96

Add t o t h e f r o n t o t h e l i s t o f s t a t e m e n t s i n t h e f u n c t i o n b o d y prependStatement ( nopStatement , bl ock ) ; i n s e r t I n s t r u c t i o n B e f o r e ( /∗ t a r g e t ∗/ a s m I n s t r u c t i o n , /∗ n e w i n s t r u c t i o n ∗/

multiByteNopInstruction ) ;

}

98 } 100 // 102 104 106

Now i t e r a t e o v e r t h e n o p i n s t r u c t i o n s i n t h e s e q u e n c e a n d r e p o r t t h e l e n g h t o f e a c h ( c a n b e m u l t i − b y t e n o p i n s t r u c t i o n s ) . i = f i n d ( l . b e g i n ( ) , l . end ( ) , a s m I n s t r u c t i o n ) ; ROSE ASSERT( i != l . end ( ) ) ; f o r ( i n t j = 0 ; j < n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ; j ++) { ROSE ASSERT( i != l . end ( ) ) ; / / p r i n t f ( ” D e l e t i n g o r i g i n a l NOP i n s t r u c t i o n #%2d i s l e n g t h = %2 d \ n ” , j , ( i n t ) i s S g A s m I n s t r u c t i o n ( ∗ i )−> g e t r a w b y t e s ( ) . s i z e ( ) ) ;

108 110

// //

Removing t h e o r i g i n a l r e m o v e S t a t e m e n t (∗ i ) ;

NOP

instruction .

330

CHAPTER 43. BINARY ANALYSIS

Chapter 44

Binary Construction ROSE is normally used in such a way that a file (source code or binary) is parsed to construct an AST, then operations are performed on the AST, and the modified AST is unparsed to create a new source or binary file. However, it is also possible to construct an AST explicitly without parsing and then use that AST to generate the output. The AST construction interface for binary files was designed so that working files could be created simply, while still providing methods to control the finer details of the resulting file. The example in this chapter shows how to construct a statically linked ELF executable containing a small “.text” section that simply causes the process to exit with a specific non-zero value.

44.1

Constructors

The AST node constructors are designed to construct the tree from the root down, and thus generally take the parent node as an argument. Nodes that refer to other nodes as prerequisites also take those prerequisites as arguments. For instance, an ELF Relocation Section is a child of the ELF File Header but also needs an ELF Symbol Table and therefore takes both objects as constructor arguments.

44.2

Read-Only Data Members

When two or more file formats have a similar notion then that notion is represented in a base class. However, part of the information may continue to survive in the specialized class. In these situations modifications to the specilized data will probably be overridden by the generic values from the base class. For instance, all formats have a notion of byte order which is represented in the base class SgAsmGenericHeader as little- or big-endian (an enumeration constant). The ELF specification provides an 8-bit unsigned field to store the byte order and therefore has potentially more than two possibilities. Any value assigned to the ELF-specific byte order will likely be overwritten by the generic byte order before the AST is unparsed. A similar situation arises with section offsets, sizes, memory mappings, permissions, etc. The SgAsmGenericSection class represents ELF Sections, ELF Segments, PE Objects, and other 331

332

CHAPTER 44. BINARY CONSTRUCTION

contiguous regions of a file and has methods for obtaining/setting these values. In addition, many of the formats have some sort of table that describes these sections and which also contains similar information (e.g., the ELF Segment Table, a.k.a., the ELF Program Header Table). As above, the generic representation of these notions (stored in SgAsmGenericSection) override the format-specific values (stored in SgAsmElfSegmentEntry). ROSETTA doesn’t make a distinction between data members that can be user-modified and data members that should be modified only by the parser. Therefore it is up to the user to be aware that certain data members will have their values computed or copied from other locations in the AST during the unparsing phase.

44.3

Constructing the Executable File Container

All executable files are stored as children of an SgAsmGenericFile node. The children are file format headers (SgAsmGenericHeader) such as an ELF File Header (SgAsmElfFileHeader). This design allows a single executable file to potentially contain more than one executable and is used by formats like Windows-PE where the file contains a DOS File Header as well as a PE File Header. For the purposes of this example the SgAsmGenericFile node will serve as the root of the AST and therefore we do not need to specify a parent in the constructor argument list. SgAsmGenericFile *ef = new SgAsmGenericFile;

44.4

Constructing the ELF File Header

The ELF File Header is the first thing in the file, always at offset zero. File headers are always children of an SgAsmGenericFile which is specified as the constructor argument. The section constructors (a file header is a kind of section) always create the new section so it begins at the current end-of-file and contains at least one byte. This ensures that each section has a unique starting address, which will be important when file memory is actually allocated and sections need to be moved around–the allocator needs to know the relative positions of the sections in order to correctly relocate them. If we were parsing an ELF file we would usually use ROSE’s frontend() method. However, one can also parse the file by first constructing the SgAsmElfFileHeader and then invoking its parse() method, which parses the ELF File Header and everything that can be reached from that header. We use the typical 0x400000 as the virtual address of the main LOAD segment, which occupies the first part of the file up through the end of the “.text” section (see below). ELF File Headers don’t actually store a base address, so instead of assigning one to the SgAsmElfFileHeader we’ll leave the header’s base address at the default zero and add base va explicitly whenever we need to. SgAsmElfFileHeader *fhdr = new SgAsmElfFileHeader(ef); fhdr->get_exec_format()->set_word_size(8); /* default is 32-bit; we want 64-bit */ fhdr->set_isa(SgAsmExecutableFileFormat::ISA_X8664_Family); /* instruction set architecture; default rose_addr_t base_va = 0x400000; /* base virtual address */

44.5. CONSTRUCTING THE ELF SEGMENT TABLE

44.5

333

Constructing the ELF Segment Table

ELF executable files always have an ELF Segment Table (also called the ELF Program Header Table), which usually appears immediately after the ELF File Header. The ELF Segment Table describes contiguous regions of the file that should be memory mapped by the loader. ELF Segments don’t have names–names are imparted to the segment by virtue of the segment also being described by the ELF Section Table, which we’ll create later. Being a contiguous region of the file, an ELF Segment Table (SgAsmElfSegmentTable) is derived from SgAsmGenericSection. All non-header sections have a header as a parent, which we supply as an argument to the constructor. Since all ELF Segments will be children of the ELF File Header rather than children of the ELF Segment Table, we could define the ELF Segment Table at the end rather than here. But defining it now is an easy way to get it located in its usuall location immediately after the ELF File Header. SgAsmElfSegmentTable *segtab = new SgAsmElfSegmentTable(fhdr);

44.6

Constructing the .text Section

ROSE doesn’t treat a “.text” section as being anything particularly special–it’s just a regular SgAsmElfSection, which derives from SgAsmGenericSection. However, in this example, we want to make sure that our “.text” section gets initialized with some instructions. The easiest way to do that is to specialize SgAsmElfSection and override or augment a few of the virtual functions. We need to override two functions. First, the calculate sizes() function should return the size we need to store the instructions. We’ll treat the instructions as an array of entries each entry being one byte of the instruction stream. In other words, each “entry” is one byte in length consisting of one required byte and no optional bytes. We need to also override the unparse() method since the base class will just fill the “.text” section with zeros. The SgAsmGenericSection::write method we use will write the instructions starting at the first byte of the section. Finally, we need to augment the reallocate() method. This method is reponsible for allocating space in the file for the section and performing any other necessary pre-unparsing actions. We don’t need to allocate space since the base class’s method will take care of that in conjuction with our version of calculate sizes(), but we do need to set a special ELF flag (SHF ALLOC) in the ELF Segment Table entry for this section. There’s a few ways to accomplish this. We do it this way because the ELF Section Table Entry is not created until later and we want to demonstrate how to keep all .text-related code in close proximity. class TextSection : public SgAsmElfSection { public: TextSection(SgAsmElfFileHeader *fhdr, size_t ins_size, const unsigned char *ins_bytes) : SgAsmElfSection(fhdr), ins_size(ins_size), ins_bytes(ins_bytes) {} virtual rose_addr_t calculate_sizes(size_t *entsize, size_t *required, size_t *optional, size_t *entcount) if (entsize) *entsize = 1; /* an "entry" is one byte of instruction */ if (required) *required = 1; /* each "entry" is also stored in one byte of the file */ if (optional) *optional = 0; /* there is no extra data per instruction byte */

334

CHAPTER 44. BINARY CONSTRUCTION

if (entcount) *entcount = ins_size; /* number of "entries" is the total instruction b return ins_size; /* return value is section size required */ } virtual bool reallocate() { bool retval = SgAsmElfSection::reallocate(); /* returns true if size or position of any sectio SgAsmElfSectionTableEntry *ste = get_section_entry(); ste->set_sh_flags(ste->get_sh_flags() | 0x02); /* set the SHF_ALLOC bit */ return retval; } virtual void unparse(std::ostream &f) const { write(f, 0, ins_size, ins_bytes); /* Write the instructions at offset zero in secti } size_t ins_size; const unsigned char *ins_bytes; }; The section constructors and reallocators don’t worry about alignment issues–they always allocate from the next available byte. However, instructions typically need to satisfy some alignment constraints. We can easily adjust the file offset chosen by the constructor, but we also need to tell the reallocator about the alignment constraint. Even if we didn’t ever resize the “.text” section the reallocator could be called for some other section in such a way that it needs to move the “.text” section to a new file offset. For the purpose of this tutorial we want to be very picky about the location of the ELF Segment Table. We want it to immediately follow the ELF File Header without any intervening bytes of padding. At the current time, the ELF File Header has a size of one byte and will eventually be reallocated. When we reallocate the header the subsequent sections will need to be shifted to higher file offsets. When this happens, the allocator shifts them all by the same amount taking care to satisfy all alignment constraints, which means that an alignment constraint of byte bytes on the “.text” section will induce a similar alignment on the ELF Segment Table. Since we don’t want that, the best practice is to call reallocate() now, before we create the “.text” section.

ef->reallocate(); /* Give existing sections a chance to all static const unsigned char instructions[] = {0xb8, 0x01, 0x00, 0x00, 0x00, 0xbb, 0x56, 0x00, 0x00, 0x SgAsmElfSection *text = new TextSection(fhdr, NELMTS(instructions), instructions); text->set_purpose(SgAsmGenericSection::SP_PROGRAM); /* Program-supplied text/data/etc. */ text->set_offset(alignUp(text->get_offset(), 4)); /* Align on an 8-byte boundary */ text->set_file_alignment(4); /* Tell reallocator about alignment const text->set_mapped_alignment(4); /* Alignment constraint for memory mappin text->set_mapped_rva(base_va+text->get_offset()); /* Mapped address is based on file offset text->set_mapped_size(text->get_size()); /* Mapped size is same as file size */ text->set_mapped_rperm(true); /* Readable */ text->set_mapped_wperm(false); /* Not writable */ text->set_mapped_xperm(true); /* Executable */ At this point the text section doesn’t have a name. We want to name it “.text” and we want those characters to eventually be stored in the ELF file in a string table which we’ll

44.7. CONSTRUCTING A LOAD SEGMENT

335

provide later. In ELF, section names are represented by the section’s entry in the ELF Section Table as an offset into an ELF String Table for a NUL-terminated ASCII string. ROSE manages strings using the SgAsmGenericString class, which has two subclasses: one for strings that aren’t stored in the executable file (SgAsmBasicString) and one for strings that are stored in the file (SgAsmStoredString). Both are capable of string an std::string value and querying its byte offset (although SgAsmBasicString::get offset() will always return SgAsmGenericString::unallocated). Since we haven’t added the “.text” section to the ELF Section Table yet the new section has an SgAsmBasicString name. We can assign a string to the name now and the string will be allocated in the ELF file when we’ve provided further information. text->get_name()->set_string(".text"); The ELF File Header needs to know the virtual address at which to start running the program. In ROSE, virtual addresses can be attached to a specific section so that if the section is ever moved the address is automatically updated. Some formats allow more than one entry address which is why the method is called add entry rva() rather than set entry rva(). ELF, however, only allows one entry address. rose_rva_t entry_rva(text->get_mapped_rva(), text); fhdr->add_entry_rva(entry_rva);

44.7

Constructing a LOAD Segment

ELF Segments define parts of an executable file that should be mapped into memory by the loader. A program will typically have a LOAD segment that begins at the first byte of the file and continues through the last instruction (in our case, the end of the “.text” section) and which is mapped to virtual address 0x400000. We’ve already created the ELF Segment Table, so all we need to do now is create an ELF Segment and add it to the ELF Segment Table. ELF Segments, like ELF Sections, are represented by SgAsmElfSection. An SgAsmElfSection is an ELF Section if it has an entry in the ELF Section Table, and/or it’s an ELF Segment if it has an entry in the ELF Segment Table. The methods get section entry() and get segment entry() retrieve the actual entries in those tables. Recall that the constructor creates new sections located at the current end-of-file and containing one byte. Our LOAD segment needs to have a different offset and size. SgAsmElfSection *seg1 = new SgAsmElfSection(fhdr); seg1->get_name()->set_string("LOAD"); seg1->set_offset(0); seg1->set_size(text->get_offset() + text->get_size()); seg1->set_mapped_rva(base_va); seg1->set_mapped_size(seg1->get_size()); seg1->set_mapped_rperm(true); seg1->set_mapped_wperm(false); seg1->set_mapped_xperm(true); segtab->add_section(seg1);

/* /* /* /* /* /* /* /* /* /*

ELF Segments are represented by SgAsmElfSection Segment names aren’t saved (but useful for debug Starts at beginning of file */ Extends to end of .text section */ Typically mapped by loader to this memory addres Make mapped size match size in the file */ Readable */ Not writable */ Executable */ Add definition to ELF Segment Table */

336

44.8

CHAPTER 44. BINARY CONSTRUCTION

Constructing a PAX Segment

This documentation shows how to construct a generic ELF Segment, giving it a particular file offset and size. ELF Segments don’t have names stored in the file, but we can assign a name to the AST node to aid in debugging–it just won’t be written out. When parsing an ELF file, segment names are generated based on the type stored in the entry of the ELF Segment Table. For a PAX segment we want this type to be PT PAX FLAGS (the default is PT LOAD).

SgAsmElfSection *pax = new SgAsmElfSection(fhdr); pax->get_name()->set_string("PAX Flags"); /* Name just for debugging */ pax->set_offset(0); /* Override address to be at zero rather pax->set_size(0); /* Override size to be zero rather than o segtab->add_section(pax); /* Add definition to ELF Segment Table */ pax->get_segment_entry()->set_type(SgAsmElfSegmentTableEntry::PT_PAX_FLAGS);

44.9

Constructing a String Table

An ELF String Table always corresponds to a single ELF Section of class SgAsmElfStringSection and thus you’ll often see the term “ELF String Section” used interchangeably with “ELF String Table” even though they’re two unique but closely tied classes internally. When the ELF String Section is created a corresponding ELF String Table is also created under the covers. Since string tables manage their own memory in reponse to the strings they contain, one should never adjust the size of the ELF String Section (it’s actually fine to enlarge the section and the new space will become free space in the string table). ELF files typically have multiple string tables so that section names are in a different section than symbol names, etc. In this tutorial we’ll create the section names string table, typically called “.shstrtab”, but use it for all string storage. SgAsmElfStringSection *shstrtab = new SgAsmElfStringSection(fhdr); shstrtab->get_name()->set_string(".shstrtab");

44.10

Constructing an ELF Section Table

We do this last because we want the ELF Section Table to appear at the end of the file and this is the easiest way to achieve that. There’s really not much one needs to do to create the ELF Section Table other than provide the ELF File Header as a parent and supply a string table. The string table we created above isn’t activated until we assign it to the ELF Section Table. The first SgAsmElfStringSection added to the SgAsmElfSectionTable becomes the string table for storing section names. It is permissible to add other sections to the table before adding the string table. SgAsmElfSectionTable *sectab = new SgAsmElfSectionTable(fhdr); sectab->add_section(text); /* Add the .text section */ sectab->add_section(shstrtab); /* Add the string table to store section

44.11. ALLOCATING SPACE

44.11

337

Allocating Space

Prior to calling unparse(), we need to make sure that all sections have a chance to allocate space for themselves, and perform any other operations necessary. It’s not always possible to determine sizes at an earlier time, and most constructors would have declared sizes of only one byte. The reallocate() method is defined in the SgAsmGenericFile class since it operates over the entire collection of sections simultaneously. In other words, if a section needs to grow then all the sections located after it in the file need to be shifted to higher file offsets. ef->reallocate(); The reallocate() method has a shortcoming (as of 2008-12-19) in that it might not correctly update memory mappings in the case when the mapping for a section is inferred from the mapping of a containing segment. This can happen in our example since the “.text” section’s memory mapping is a function of the LOAD Segment mapping. The work-around is to adjust mappings for these situations and then call reallocate() one final time. This final reallocate() call won’t move any sections, but should always be the last thing to call before unparsing() (it gives sections a chance to update data dependencies which is not possible during unparse() due to its const nature).

text->set_mapped_rva(seg1->get_mapped_rva()+(text->get_offset()-seg1->get_offset())); ef->reallocate(); /*won’t resize or move things this time since we didn’t modify much since the last call to re

44.12

Produce a Debugging Dump

A debugging dump can be made with the following code. This dump will not be identical to the one produced by parsing and dumping the resulting file since we never parsed a file (a dump contains some information that’s parser-specific). ef->dump(stdout); SgAsmGenericSectionPtrList all = ef->get_sections(true); for (size_t i=0; idump(stdout, " ", -1); }

44.13

Produce the Executable File

The executable file is produced by unparsing the AST. std::ofstream f("a.out"); ef->unparse(f); Note that the resulting file will not be created with execute permission–that must be added manually.

338

CHAPTER 44. BINARY CONSTRUCTION

Chapter 45

Dwarf Debug Support DWARF is a widely used, standardized debugging data format. DWARF was originally designed along with ELF, although it is independent of object file formats. The name is a pun on ”ELF” that has no official meaning but ”may be an acronym for ’Debug With Attributed Record Format’”. See Wikipedia for more information about the Dwarf debug format. This chapter presents the support in ROSE for Dwarf 3 debug information; its representation in the AST and its use in binary analysis tools. This work is part of general work to support as much information as possible about binaries. In the following sections we use a small example (see figure 45.1) that demonstrates various features of Dwarf. The source code of our binary example is:

2 4 6

// Test code t o d e m o n s t r a t i o n o f dwarf s u p p o r t . // Designed t o be s m a l l b e c a u s e Dwarf i s v e r b o s e . namespace e xa m pl e n a m e s p a c e { int a ; };

8 10 12 14 16 18

i n t main ( ) { int x = 42; // Loops ar e not r e c o g n i s e d i n Dwarf . . . f o r ( i n t i = 0 ; i < 1 0 ; i ++) { x = ( x % 2) ? x ∗ 2 : x + 1 ; }

20

return 0 ; }

Figure 45.1: Example source code used to generate Dwarf AST for analysis. Much larger binaries can be analyzed, but such larger binary executables are more difficult to present (in this tutorial). 339

XME: This need to be to correctly return sets nd sets of addresses in be a second interface.

340

45.1

CHAPTER 45. DWARF DEBUG SUPPORT

ROSE AST of Dwarf IR nodes

ROSE tools process the binary into an AST that is used to represent all the information in the binary executable. Figure 45.2 shows the subset of that AST (which includes the rest of the binary file format and the disassembled instructions) that is specific to Dwarf. A command line option (rose:visualize dwarf only) is used to restrict the generated dot file visualization to just the Dwarf information. This option is used in combination with -rose:read executable file format only to process only the binary file format (thus skipping the instruction disassembly).

45.2

Source Position to Instruction Address Mapping

One of the valuable parts of Dwarf is the mapping between the source lines and the instruction addresses at a statement level (provided in the .debug line section). Even though Dwarf does not represent all statements in the source code, the mapping is significantly finer granularity than that provided at only the function level by the symbol table (which identifies the functions by instruction address, but not the source file line numbers; the later requires source code analysis). The example code in 45.3 shows the mapping between the source code lines and the instruction addresses. Output from the compilation of this test code and running it with the example input results in the output shown in figure 45.4. This output shows the binary executables instruction address range (binary compiled on Linux x86 system), the range of lines of source code used by the binary executable, the mapping of a source code range of line numbers to the instruction addresses, and the mapping of a range of instruction addresses to the source code line numbers.

45.2. SOURCE POSITION TO INSTRUCTION ADDRESS MAPPING

Figure 45.2: Dwarf AST (subset of ROSE binary AST).

341

342

CHAPTER 45. DWARF DEBUG SUPPORT

#include ” r o s e . h” 2 4 6

int main ( i n t a r g c , char ∗∗ a r g v ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

8 10 12

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // This i s c o n t r o l e d by u s i n g t h e −−with−dwarf c o n f i g u r e command l i n e o p t i o n . #i f d e f ROSE HAVE LIBDWARF

14 16 18

// The i n p u t f i l e i s t h e b i n a r y f i l e . . . i n t b i n a r y f i l e i d = p r o j e c t −> g e t f i l e L i s t ()[0] − > g e t f i l e i n f o ()−> g e t f i l e i d ( ) ; // Increment t o g e t t h e n e x t f i l e i d ( f o r t h e s o u r c e f i l e , i n s t e a d o f t h e b i n a r y int s o u r c e f i l e i d = b i n a r y f i l e i d + 1 ;

file )

20 22 24

s t d : : s t r i n g b i n a r y F i l e n a m e = S g F i l e I n f o : : getFilenameFromID ( b i n a r y f i l e i d ) ; p r i n t f ( ” f i l e i d = %d b i n a r y F i l e n a m e = %s \n” , b i n a r y f i l e i d , b i n a r y F i l e n a m e . c s t r ( ) ) ; s t d : : s t r i n g s o u r c e F i l e n a m e = S g F i l e I n f o : : getFilenameFromID ( s o u r c e f i l e i d ) ; p r i n t f ( ” f i l e i d = %d s o u r c e F i l e n a m e = %s \n” , s o u r c e f i l e i d , s o u r c e F i l e n a m e . c s t r ( ) ) ;

26 28

// Compute t h e s o u r c e l i n e range from t h e i n s t r u c t i o n s i n t h e b i n a r y e x e c u t a b l e s t d : : p a i r s o u r c e F i l e R a n g e ; s o u r c e F i l e R a n g e = SgAsmDwarfLineList : : sourceCodeRange ( s o u r c e f i l e i d ) ;

30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64

p r i n t f ( ” \ n S o u r c e f i l e l i n e number r a n g e f o r : \ n −− f i l e = %s ( i d = %d ) \ n −− [ ( l i n e=%d , sourceFilename . c s t r ( ) , s o u r c e f i l e i d , s o u r c e F i l e R a n g e . f i r s t . f i r s t , s o u r c e F i l e R a n g e . f i r s t . second , sourceFileRange . second . f i r s t , sourceFileRange . second . second ) ;

// Compute t h e b i n a r y e x e c u t a b l e i n s t r u c t i o n a d d r e s s range s t d : : p a i r a d d r e s s R a n g e = SgAsmDwarfLineList : : i n s t r u c t i o n R a n g e ( ) ; p r i n t f ( ” \ nBinary i n s t r u c t i o n a d d r e s s r a n g e = ( 0 x%l x , 0x%l x ) \n” , a d d r e s s R a n g e . f i r s t , a d d r e s s R a n g e . s e c o n d ) ; i n t minLine = s o u r c e F i l e R a n g e . f i r s t . f i r s t ; i n t maxLine = s o u r c e F i l e R a n g e . s e c o n d . f i r s t ; i n t columnNumber = −1; p r i n t f ( ” \ n I n s t r u c t i o n a d d r e s s e s computed from s o u r c e p o s i t i o n s : \n” ) ; // I t e r a t e o v e r l i n e numbers t o map bac k t o i n s t r u c t i o n a d d r e s s e s f o r ( i n t lineNumber = minLine − 2 ; lineNumber <= maxLine + 2 ; lineNumber++) { // Out o f range v a l u e s g e n e r a t e t h e n e x t a d d r e s s or NULL. F i l e I d L i n e C o l u m n F i l e P o s i t i o n s ( s o u r c e f i l e i d , s t d : : p a i r (lineNumber , columnNumber ) ) ; u i n t 6 4 t i n s t r u c t i o n A d d r e s s = SgAsmDwarfLineList : : so u rc e C od e T oA d dr e s s ( s ) ; printf (” s ou r c eC o d eT o A dd r es s(%d,%d,%d ) = 0x%l x \n” , s . f i r s t , s . s e c o n d . f i r s t , s . s e c o n d . second , i n s t r u c } u i n t 6 4 t minInstructionAddress = addressRange . f i r s t ;

// I t e r a t e o v e r t h e a d d r e s s e s o f t h e b i n a r y and compute t h e s o u r c e code l i n e numbers ( l i m i t t h e range so t h e p r i n t f ( ” \ n S o u r c e l i n e s computed from a d d r e s s r a n g e ( t r u n c a t e d t o keep o u t p u t s h o r t ) : \n” ) ; f o r ( u i n t 6 4 t a d d r e s s = m i n I n s t r u c t i o n A d d r e s s − 1 ; a d d r e s s < m i n I n s t r u c t i o n A d d r e s s + 2 5 ; a d d r e s s++) { F i l e I d L i n e C o l u m n F i l e P o s i t i o n s map = SgAsmDwarfLineList : : a d d r e s s T o S o u r c e C o d e ( a d d r e s s ) ; printf (” a d d r e s s T o S o u r c e C o d e : a d d r e s s 0x%l x = (%d,%d,%d ) \n” , a d d r e s s , s map . f i r s t , s map . s e c o n d . f i r s } #e l s e p r i n t f ( ” \n\nROSE must be c o n f i g u r e d w i t h −−with−d w a r f= t o u s e Dwarf s u p p o r t . \n\n” ) ; #e n d i f

66 p r i n t f ( ” \ nProgram Terminated Normally ! \n\n” ) ; 68 70

c o l=%d ) , ( l i n e=%d ,

// S k i p c a l l t o backend s i n c e t h i s return backend ( p r o j e c t ) ; }

is

j u s t an a n a l y s i s .

Figure 45.3: Example source code (typical for reading in a binary or source file).

45.2. SOURCE POSITION TO INSTRUCTION ADDRESS MAPPING

2 ROSE must be c o n f i g u r e d w i t h −−with−d w a r f= t o u s e Dwarf s u p p o r t . 4 6

Program Terminated Normally !

Figure 45.4: Example source code (typical for reading in a binary or source file).

343

344

CHAPTER 45. DWARF DEBUG SUPPORT

Part VII

Interacting with Other Tools

How to build interoperable tools using ROSE.

345

Chapter 46

Abstract Handles to Language Constructs This chapter describes a reference design and its corresponding implementation for supporting abstract handles to language constructs in source code and optimization phases. It can be used to facilitate the interoperability between compilers and tools. We define an abstract handle as a representation for a unique language construct in a specific program. Interoperability between tools is achieved by writing out the abstract handles as strings and reading them within other tools to build the equivalent abstract handle. 1 The idea is to define identifiers for unique statements, loops, functions, and other language constructs in source code. Given the diverse user requirements, an ideal specification should include multiple forms to specify a language construct. Currently, we are interested in the following forms for specifying language constructs: • Source file position information including path, filename, line and column number etc. GNU standard source position from http://www.gnu.org/prep/standards/html_node/ Errors.html presents some examples. • Global or local numbering of specified language construct in source file (e.g. 2nd ”do” loop in a global scope). The file is itself specified using an abstract handle (typically generated from the file name). • Global or local names of constructs. Some language constructs, such as files, function definitions and namespace, have names which can be used as their handle within a context. • Language-specific label mechanisms. These include named constructs in Fortran, numbered labels in Fortran, and statement labels in C and C++, etc. 1 Abstract Handles are not appropriate for program analysis since they are not intended to be used to capture the full structure of a program. Instead, Abstract Handles represent references to language constructs in a program, capturing only a program’s local structure; intended to support interoperability between source based tools (including compilers). We don’t advise the use of abstract handles in an aggressive way to construct an alternative intermediate representation (IR) for a full program.

347

348

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

In addition to human-readable forms, compilers and tools can generate internal IDs for language constructs. It is up to compiler/tool developers to provide a way to convert their internal representations into human-readable formats. Abstract Handles can have any of the human-readable or machine-generated forms. A handle can be used alone or combined with other handles to specify a language construct. A handle can also be converted from one form to another (e.g. from a compiler specific form to an human readable form relative to the source position; filename, line number, etc.). Abstract handles can have different lifetimes depending on their use and implementation. An abstract handle might be required to be persistent if it is used to reference a language construct that would be optimized over multiple executions of one or more different tools. Where as an abstract-handle might be internally generated only for purposes of optimizations used in a single execution (e.g. optimization within a compiler).

46.1

Use Case

A typical use can for Abstract Handles might be for a performance tool to identify a collection of loops in functions that are computationally intensive and to construct Abstract Handles that refer to this specific loops. Then pass the Abstract Handles to a second tool that might analyze the source code and/or the binary executable to evaluate if the computational costs are reasonable or if optimizations might be possible. The specific goal of the Abstract Handles is to support these sorts of uses within autotuning using diverse tools used and/or developed as part of autotuning research within the DOE SciDAC PERI project.

46.2

Syntax

A possible grammar for abstract handles could be: /* a handle is a single handle item or a link of them separated by ::, or other delimiters */ handle ::= handle_item | handle ’::’ handle_item /* Each handle item consists of construct_type and a specifier. Or it can be a compiler generated id of any forms. */ handle_item ::= construct_type specifier | compiler_generated_handle /* Construct types are implementation dependent. An implementation can support a subset of legal constructs or all of them. We define a minimum set of common construct type names here and will grow this list as needed. */ construct_type ::= Project|SourceFile|FunctionDeclaration|ForStatement|... /* A specifier is used to locate a particular construct

46.3. EXAMPLES

349

e.g: */ specifier::= ’<’ specifier_type ’,’ specifier_value ’>’ /* tokens for specifier types could be name, position,numbering, label, etc. specifier type is necessary to avoid ambiguity for specifier values, because a same value could be interpreted in different specifier types otherwise */ specifier_type::= name | position | numbering | label /* Possible values for a specifier */ specifier_value::= string_lit|int_lit|position_value| label_value /*A label could be either integer or string */ label_value::= int_lit | string_lit /* Start and end source line and column information e.g.: 13.5-55.4, 13, 13.5 , 13.5-55 */ position_value:: = line_number[ ’.’ column_number][ ’-’ line_number[ ’.’ column_number]] /* Integer value: one or more digits */ int_lit ::= [0-9]+ /* String value: start with a letter, followed by zero or more letters or digits */ string_lit ::= [a-z][a-z0-9]*

46.3

Examples

We give some examples of language handles using the syntax mentioned above. Canonical AST’s node type names are used as the construct type names. Other implementations can use their own construct type names. • A file handle consisting of only one handle item: SourceFile • A function handle using a named handle item, combined with a parent handle using a name also: SourceFile::FunctionDeclaration

350

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS • A function handle using source position(A function starting at line 12, column 1 till line 30, column 1 within a file): SourceFile::FunctionDeclaration

• A function handle using numbering(The first function definition in a file): SourceFile::FunctionDeclaration • A return statement using source position (A return statement at line 100): SourceFile::ReturnStatement • A loop using numbering information (The second loop in function main()): SourceFile::FunctionDeclaration:: ForStatement • A nested loop using numbering information (The first loop inside the second loop in function main()): SourceFile::FunctionDeclaration:: ForStatement::ForStatement

46.4

Reference Implementation

Abstract Handles are fundamentally compiler and tool independent, however to clarify the concepts, provide meaningful examples, a working reference implementation we have provided a reference implementation in ROSE. The source files are located in src/midend/abstractHandle in the ROSE distribution. A generic interface (abstract handle.h and abstract handle.cpp) provides data structures and operations for manipulating abstract handles using source file positions, numbering, or names. Any compilers and tools can have their own implementations using the same interface.

46.4.1

Connecting to ROSE

A ROSE adapter (roseAdapter.h and roseAdapter.cpp) using the interface is provided as a concrete implementation for the maximum capability of the implementation (within a sourceto-source compiler). Figure 46.1 shows the code (using ROSE) to generate abstract handles for loops in an input source file (as in Figure 46.2). Abstract handle constructors generate handles from abstract nodes, which are implemented using ROSE AST nodes. Source position is used by default to generate a handle item. Names or numbering are used instead when source position information is not available. The Constructor can also be used to generate a handle item using a specified handle type (numbering handles in the example). Figure 46.3 is the output showing the generated handles for the loops.

46.4. REFERENCE IMPLEMENTATION

2 4 6 8 10 12

351

/∗ Example code t o g e n e r a t e a b s t r a c t h a n d l e s f o r l a n g u a g e c o n s t r u c t s by Liao , ∗/ #include #include #include #include #include

10/6/2008 ” r o s e . h” ” a b s t r a c t h a n d l e . h” ” r o s e A d a p t e r . h” < s t r i n g . h>

using namespace s t d ; using namespace A b s t r a c t H a n d l e ;

14 16 18 20 22 24 26

// a g l o b a l h a n d l e f o r t h e c u r r e n t f i l e s t a t i c a b s t r a c t h a n d l e ∗ f i l e h a n d l e = NULL ; c l a s s v i s i t o r T r a v e r s a l : public A s t S i m p l e P r o c e s s i n g { protected : v i r t u a l void v i s i t ( SgNode∗ n ) ; }; void v i s i t o r T r a v e r s a l : : v i s i t ( SgNode∗ n ) { S gForStatement ∗ f o r l o o p = i s S g F o r S t a t e m e n t ( n ) ;

28

if {

( forloop ) cout<<” C r e a t i n g h a n d l e s f o r a l o o p c o n s t r u c t . . . ”<
30 32 34

// Create an a b s t r a c t h a n d l e from t h e a b s t r a c t node // Using s o u r c e p o s i t i o n s p e c i f i e r s by d e f a u l t a b s t r a c t h a n d l e ∗ a h a n d l e = new a b s t r a c t h a n d l e ( anode ) ; cout<
t o S t r i n g ()<< e n d l ;

36 38

// Create h a n d l e s b a s e d on numbering s p e c i f i e r s w i t h i n t h e f i l e a b s t r a c t h a n d l e ∗ b h a n d l e = new a b s t r a c t h a n d l e ( anode , e numb ering , f i l e h a n d l e ) ; cout<t o S t r i n g ()<< e n d l <
40 }

42 } 44 46 48

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

50

SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ;

52

// Generate a f i l e h a n d l e a b s t r a c t n o d e ∗ f i l e n o d e = b u i l d r o s e N o d e ( ( p r o j e c t −> g e t f i l e L i s t ( ) ) [ 0 ] ) ; f i l e h a n d l e = new a b s t r a c t h a n d l e ( f i l e n o d e ) ;

54 56 58 60 62

// Generate h a n d l e s f o r l a n g u a g e c o n s t r u c t s visitorTraversal myvisitor ; myvisitor . traverseInputFiles ( project , preorder ) ; // Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; }

t h e vendor ’ s c o m p i l e r

Figure 46.1: Example 1: Generated handles for loops: using constructors with or without a specified handle type.

352

2 4 6 8 10

/∗ t e s t i n p u t f o r g e n e r a t e d a b s t r a c t h a n d l e s ∗/ int a [ 1 0 0 ] [ 1 0 0 ] [ 1 0 0 ] ; void f o o ( ) { int i , j , k ; f o r ( i =0; i ++; i <100) f o r ( j =0; j ++; j <100) f o r ( k =0; k++;k <100) a [ i ] [ j ] [ k]= i+j+k ;

12

f o r ( i =0; i ++; i <100) f o r ( j =0; j ++; j <100) f o r ( k =0; k++;k <100) a [ i ] [ j ] [ k ]+=5;

14 16

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

}

Figure 46.2: Example 1: Example source code with some loops, used as input.

46.4. REFERENCE IMPLEMENTATION

2 4 6

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

8 10 12 14

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

16 18 20 22

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

24 26 28 30

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

32 34 36 38

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

40 42 44 46

Creating handles for a loop construct . . . P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : ForStatement

Figure 46.3: Example 1: Abstract handles generated for loops.

353

354

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

A second example (shown in Figure 46.4) demonstrates how to create handles using userspecified strings representing handle items for language constructs within a source file (shown in Figure 46.5). This is particularly useful to grab internal language constructs from handles provided by external software tools. The output of the example is given in Figure 46.6.

2 4 6 8 10 12 14

/∗ Example code t o g e n e r a t e l a n g u a g e h a n d l e s from i n p u t s t r i n g s a b o u t ∗ source p o s i t i o n information ∗ numbering i n f o r m a t i o n by Liao , ∗/ #include #include #include #include #include

10/9/2008 ” r o s e . h” < s t r i n g . h> ” a b s t r a c t h a n d l e . h” ” r o s e A d a p t e r . h”

using namespace s t d ; using namespace A b s t r a c t H a n d l e ;

16 18 20

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ; SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ;

22 24 26 28 30 32 34 36

// Generate a f i l e h a n d l e from t h e f i r s t f i l e o f t h e p r o j e c t a b s t r a c t n o d e ∗ f i l e n o d e= b u i l d r o s e N o d e ( ( p r o j e c t −> g e t f i l e L i s t ( ) ) [ 0 ] ) ; a b s t r a c t h a n d l e ∗ h a n d l e 0 = new a b s t r a c t h a n d l e ( f i l e n o d e ) ; cout<<” C r e a t e d a f i l e h a n d l e : \ n”<t o S t r i n g ()<< e n d l <” ; a b s t r a c t h a n d l e ∗ h a n d l e 1 = new a b s t r a c t h a n d l e ( h a n d l e 0 , i n p u t 1 ) ; cout<<” C r e a t e d a h a n d l e : \ n”<t o S t r i n g ()<< e n d l <getNode()−> t o S t r i n g ()<< e n d l < ” ; a b s t r a c t h a n d l e ∗ h a n d l e 2 = new a b s t r a c t h a n d l e ( h a n d l e 0 , i n p u t ) ;

38 40 42

cout<<” C r e a t e d a h a n d l e : \ n”<t o S t r i n g ()<< e n d l <getNode()−> t o S t r i n g ()<< e n d l < ” ) ;

44 46 48 50

cout<<” C r e a t e d a h a n d l e : \ n”< t o S t r i n g ()<< e n d l ; // Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; }

t h e vendor ’ s c o m p i l e r

Figure 46.4: Example 2: Generated handles from strings representing handle items.

46.4. REFERENCE IMPLEMENTATION

2 4 6 8 10

void b ar ( i n t x ) ; namespace s p a c e 1 { class A { public : void b ar ( i n t x ) ; void f o o ( ) ; }; }

Figure 46.5: Example 2: Source code with some language constructs.

2 4 6 8

Created a f i l e handle : P r o j e c t : : F i l e L i s t : : S o u r c e F i l e Created a handle : P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : N a m e s p a c e D e c l a r a t i o n S t a t e m e n t

10 12 14 16

I t points to : namespace s p a c e 1 { c l a s s A { public : void b a r ( i n t x ) ; void f o o ( ) ; } ; } Created a handle : P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : C l a s s D e c l a r a t i o n



18 20 22 24 26 28

I t points to : c l a s s A { public : void b a r ( i n t x ) ; void f o o ( ) ; } ; Created a handle : P r o j e c t : : F i l e L i s t : : S o u r c e F i l e : : C l a s s D e c l a r a t i o n

: : M e m b e r F u n c t i o n D e c l a r a t i o n I t points to : public : void f o o ( ) ;

Figure 46.6: Example 2: Handles generated from string and their language constructs.

355

356

46.4.2

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

Connecting to External Tools

A third example is provided to demonstrate how to use the abstract interface with any other tools, which may have less features in terms of supported language constructs and their correlations compared to a compiler. Assume a tool operating on some simple for-loops within an arbitrary source file (the input file is not shown in this example). Such a tool might have an internal data structure representing loops; such as that in given in Figure 46.7. We will show how the tool specific data structure for loops can be used to generate abstract handles and output as strings that can be used by other tools which use abstract handles (which would generate the abstract handles by reading the strings).

2 4 6 8 10 12 14 16

/∗ ∗ A toy loop data s t r u c t u r e demonstrating a t h i n c l i e n t of a b s t r a c t handles : ∗ A s i m p l e s t l o o p t o o l which k e e p s a t r e e o f l o o p s i n a f i l e ∗/ #i f n d e f my loop INCLUDED #d e f i n e my loop INCLUDED #include #include c l a s s MyLoop { public : std : : s t r i n g sourceFileName ; s i z e t line number ; s t d : : v e c t o r c h i l d r e n ; MyLoop∗ p a r e n t ; };

18 #e n d i f

Figure 46.7: Example 3: A simple data structure used to represent a loop in an arbitrary tool. An adapter (loopAdapter.h and loopAdapter.cpp) using the proposed abstract handle interface is given in src/midend/abstractHandle. It provides a concrete implementation for the interface for the simple loops and adds a node to support file nodes (Compared to a full-featured IR for a compiler, the file node is an additional detail for tools without data structures to support files). The test program is given in Figure 46.8. Again, it creates a top level file handle first. Then a loop handle (loop handle1) is created within the file handle using its relative numbering information. The loop handle2 is created from from its string format using file position information (using GNU standard file position syntax). The loop handle3 uses its relative numbering information within loop handle1. The output of the program is shown in Figure 46.9. It demonstrates the generated strings to represent the abstract handles in the arbitrary code operated upon by the tool. Interoperability is achieved by another tool reading in the generated string representation to generate an abstract handle to the same source code language construct.

46.4. REFERENCE IMPLEMENTATION

2 4 6 8

#include #include #include #include #include #include

357

” a b s t r a c t h a n d l e . h” ” myloop . h” ” l o o p A d a p t e r . h”

using namespace s t d ; using namespace A b s t r a c t H a n d l e ;

10 12 14 16

i n t main ( ) { //−−−−−−−−−−−−−P r e p a r i n g t h e i n t e r n a l l o o p r e p r e s e n t a t i o n −−−−−−−−− // d e c l a r e and i n i t i a l i z e a l i s t o f l o o p s u s i n g MyLoop // The l o o p t o o l s h o u l d be a b l e t o g e n e r a t e i t s r e p r e s e n t a t i o n from // s o u r c e code somehow . We f i l l i t up manually h e r e . v e c t o r l o o p s ;

18 MyLoop l o o p 1 , l o o p 2 , l o o p 3 ; l o o p 1 . s o u r c e F i l e N a m e=” f i l e 1 . c ” ; loop1 . line number = 7; l o o p 1 . p a r e n t = NULL ; l o o p 2 . s o u r c e F i l e N a m e=” f i l e 1 . c ” ; loop2 . line number = 8; l o o p 2 . p a r e n t=&l o o p 1 ; l o o p 1 . c h i l d r e n . p u s h b a c k (& l o o p 2 ) ; l o o p 3 . s o u r c e F i l e N a m e=” f i l e 1 . c ” ; loop3 . line number = 12; l o o p 3 . p a r e n t=NULL ; l o o p s . p u s h b a c k (& l o o p 1 ) ; l o o p s . p u s h b a c k (& l o o p 3 ) ;

20 22 24 26 28 30 32

//−−−−−−−−−−−−−−−−−− u s i n g a b s t r a c t h a n d l e s −−−−−−−−−−−−− // Generate t h e a b s t r a c t h a n d l e for the source f i l e f i l e N o d e ∗ f i l e n o d e = new f i l e N o d e ( ” f i l e 1 . c ” ) ; f i l e n o d e −>setMLoops ( l o o p s ) ; a b s t r a c t h a n d l e ∗ f i l e h a n d l e = new a b s t r a c t h a n d l e ( f i l e n o d e ) ; cout<<” C r e a t e d a f i l e h a n d l e : ”<t o S t r i n g ()<< e n d l ;

34 36 38 40

// Create a l o o p h a n d l e w i t h i n t h e f i l e u s i n g numbering i n f o . a b s t r a c t n o d e ∗ l o o p n o d e 1= new loopNode (& l o o p 1 ) ; a b s t r a c t h a n d l e ∗ l o o p h a n d l e 1= new a b s t r a c t h a n d l e ( l o o p n o d e 1 , e numb ering , f i l e h a n d l e ) ; cout<<” C r e a t e d a l o o p h a n d l e : ”<t o S t r i n g ()<< e n d l ;

42 44

// Create a n o t h e r l o o p h a n d l e w i t h i n a f i l e u s i n g i t s s o u r c e p o s i t i o n i n f o r m a t i o n s t r i n g i n p u t 1 ( ” ForStatement

” ) ; a b s t r a c t h a n d l e ∗ l o o p h a n d l e 2= new a b s t r a c t h a n d l e ( f i l e h a n d l e , i n p u t 1 ) ; cout<<” C r e a t e d a l o o p h a n d l e : ”<t o S t r i n g ()<< e n d l ;

46 48 50

// Create y e t a n o t h e r l o o p h a n d l e w i t h i n a l o o p u s i n g i t s r e l a t i v e numbering i n f o r m a t i o n s t r i n g i n p u t 2 ( ” ForStatement ” ) ; a b s t r a c t h a n d l e ∗ l o o p h a n d l e 3= new a b s t r a c t h a n d l e ( l o o p h a n d l e 1 , i n p u t 2 ) ; cout<<” C r e a t e d a l o o p h a n d l e : ”<t o S t r i n g ()<< e n d l ;

52 54

return 0 ; 56

}

Figure 46.8: Example 3: A test program for simple loops’ abstract handles.

358

2 4 6 8

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

bash − 3 . 0 0 : . / testMyLoop Created a f i l e handle : S o u r c e F i l e Created a loop handle : S o u r c e F i l e : : ForStatement Created a loop handle : S o u r c e F i l e : : ForStatement

Created a loop handle : S o u r c e F i l e : : ForStatement : : ForStatement

Figure 46.9: Example 3: Output of the test program for simple loops’ abstract handles (as strings).

46.5. SUMMARY

46.5

359

Summary

Abstract handles are low level mechanisms to support multiple tools to exchange references to source code. Several examples are used to present the different features of abstract handles. Importantly, the specification of abstract handles is tool independent. A reference implementation is provided and is publically available within the ROSE compiler framework. We encourage debate on the pros and cons of this concept to support interoperability of tools which must pass references to source code between them. This work is expected to a small piece of the infrastructure to suport autotuning research.

360

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

Chapter 47

ROSE-HPCToolKit Interface ROSE-HPCToolKit is designed to read in performance data generated by HPCToolkit (and recently GNU gprof) and annotate ROSE AST with performance metrics. It is included in the ROSE distribution and enabled by default if an existing installation of the Gnome XML library, libxml2 (http://xmlsoft.org) can be detected by ROSE’s configure script. Or it can be enabled explicitly by specifying the --enable-rosehpct option when running configure. The HPCToolkit (http://www.hipersoft.rice.edu/hpctoolkit) is a set of tools for analyzing the dynamic performance behavior of applications. It includes a tool that instruments a program’s binary, in order to observe CPU hardware counters during execution; additional post-processing tools attribute the observed data to statements in the original source code. HPCToolkit stores this data and the source attributions in XML files. In this chapter, we give an overview of simple interfaces in ROSE that can read this data and attach it to the AST. GNU gprof is a basic but easy to use profiling tool. It produces an execution profile of applications. gprof’s output is a flat profile for each function by default, which is not very interesting to us. We use a line-by-line output with full file path information generated by using option -l -L with gprof.

47.1

An HPCToolkit Example Run

Consider the sample source program shown in Figures 47.1–47.2. This program takes an integer n on the command line, and has a number of loops whose flop and memory-operation complexity are either Θ(n) or Θ(n2 ). For this example, we would expect the loop nest at line 56, which has O(n2 ) cost, to be the most expensive loop in the program for large n. Suppose we use the HPCToolkit to profile this program, collecting cycle counts and floatingpoint instructions.1 HPCToolkit will generate one XML file for each metric. A schema specifying the format of these XML files appears in Figure 47.3. In essence, this schema specifies that the XML file will contain a structured, abstract representation of the program in terms of abstract program entities such as “modules,” “procedures,” “loops,” and “statements.” Each of these entities may have line number information and a “metric” value. 1 In

this example, HPCToolkit uses the PAPI to read CPU counters (http://icl.cs.utk.edu/papi).

361

362

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

(Refer to the HPCToolkit documentation for more information.) This schema is always the first part of an HPCToolkit-generated XML profile data file. We ran HPCToolkit on the program in Figures 47.1–47.2, and collected cycle and flop counter data. The actual XML files storing this data appear in Figures 47.4 and 47.5. By convention, these metrics are named according to their PAPI symbolic name, as shown on line 67 in both Figures 47.4 and 47.5. According to the cycle data on line 90 of Figure 47.4, the most expensive statement in profiled.c is line 62 of Figure 47.1, as expected.

47.1. AN HPCTOOLKIT EXAMPLE RUN

363

#include #include < s t d l i b . h> #include < s t r i n g . h> 5

10

15

20

25

30

35

typedef double v a l t ; static val t ∗ gen ( s i z e t n ) { v a l t ∗ x = ( v a l t ∗) malloc ( sizeof ( v a l t ) ∗ n ) ; i f ( x != NULL) { size t i ; f o r ( i = 0 ; i < n ; i ++) x [ i ] = ( val t )1.0 / n; } return x ; } static val t d o t ( s i z e t n , const v a l t ∗ x , const v a l t ∗ y ) { size t i ; v a l t sum ; f o r ( i = 0 , sum = 0 . 0 ; i < n ; i ++) sum += x [ i ] ∗ y [ i ] ; return sum ; } static s i z e t max ( s i z e t n , const v a l t ∗ x ) { size t i ; s i z e t i max ; v a l t v max ; if

40

( n <= 0 ) return 0 ;

v max = x [ 0 ] ; i max = 0 ; f o r ( i = 1 ; i < n ; i ++) i f ( x [ i ] > v max ) { v max = x [ i ] ; i max = i ; } return i max ;

45

} 50

55

60

s t a t i c void mv ( s i z e t n , const v a l t ∗ A, const v a l t ∗ x , v a l t ∗ y ) { size t j ; memset ( y , 0 , s i z e o f ( v a l t ) ∗ n ) ; f o r ( j = 0 ; j < n ; j ++) { const v a l t ∗ Ap ; register v a l t xj = x [ j ] ; size t i ; f o r ( i = 0 , Ap = A + j ∗n ; i < n ; i ++, Ap++) y [ i ] += Ap [ 0 ] ∗ x j ; } }

Figure 47.1: profiled.c (part 1 of 2): Sample input program, profiled using the HPCToolkit.

364

70

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

int main ( i n t a r g c , char ∗ a r g v [ ] ) { size t n; val t∗ x; val t∗ y; v a l t ∗ A; size t i ;

75

/∗ o u t p u t s ∗/ v a l t sum ; s i z e t i max ; v a l t y max ;

80

if

( a r g c != 2 ) { f p r i n t f ( s t d e r r , ” u s a g e : %s \n” , a r g v [ 0 ] ) ; return 1 ;

} n = a t o i ( argv [ 1 ] ) ; i f ( n <= 0 ) return 1 ;

85

A = gen ( n ∗ n ) ; x = gen ( n ) ; y = gen ( n ) ;

90

(A == NULL | | x == NULL | | y == NULL)

if {

f p r i n t f ( s t d e r r , ” ∗∗∗ Out o f memory ∗∗∗\ n” ) ; return 1 ;

95 }

sum = 0 ; f o r ( i = 0 ; i < n ; i ++) sum += d o t ( n , x , y ) ; mv ( n , A, x , y ) ; i max = max ( n , y ) ; y max = y [ i max ] ;

100

105 p r i n t f ( ”%g %l u %g \n” , sum , ( unsigned long ) i max , y max ) ; return 0 ; } 110 /∗ e o f ∗/

Figure 47.2: profiled.c (part 2 of 2): Sample input program, profiled using the HPCToolkit.

47.1. AN HPCTOOLKIT EXAMPLE RUN

5

10

15

20

25

30

35

40

45

50

55

60

365

< !DOCTYPE PROFILE [ < !ELEMENT PROFILE (PROFILEHDR, PROFILEPARAMS, PROFILESCOPETREE)> < !ATTLIST PROFILE version CDATA #REQUIRED> < !ELEMENT PROFILEHDR (#PCDATA)> < !ELEMENT PROFILEPARAMS (TARGET, METRICS)> < !ELEMENT TARGET EMPTY> < !ATTLIST TARGET name CDATA #REQUIRED> < !ELEMENT METRICS (METRIC)+> < !ELEMENT METRIC EMPTY> < !ATTLIST METRIC shortName CDATA #REQUIRED nativeName CDATA #REQUIRED period CDATA #REQUIRED units CDATA #IMPLIED displayName CDATA #IMPLIED display ( t r u e | f a l s e ) #IMPLIED> < !ELEMENT PROFILESCOPETREE (PGM) ∗> < !−− This i s e s s e n t i a l l y t h e PGM d t d w i t h M e l e m e n t added . −−> < !ELEMENT PGM (G|LM| F |M)+> < !ATTLIST PGM n CDATA #REQUIRED> < !−− Groups c r e a t e a r b i t r a r y s e t s o f o t h e r e l e m e n t s e x c e p t PGM. −−> < !ELEMENT G (G|LM| F | P | L | S |M) ∗> < !ATTLIST G n CDATA #IMPLIED> < !−− Runtime l o a d modules f o r PGM ( e . g . , DSOs, e x e ) −−> < !ELEMENT LM (G| F |M) ∗> < !ATTLIST LM n CDATA #REQUIRED> < !−− F i l e s c o n t a i n p r o c e d u r e s and s o u r c e l i n e i n f o −−> < !ELEMENT F (G| P | L | S |M) ∗> < !ATTLIST F n CDATA #REQUIRED> < !−− Procedures c o n t a i n s o u r c e l i n e i n f o n : p r o c e s s e d name ; l n : l i n k name −−> < !ELEMENT P (G| L | S |M) ∗> < !ATTLIST P n CDATA #REQUIRED l n CDATA #IMPLIED b CDATA #IMPLIED e CDATA #IMPLIED> < !−− Loops −−> < !ELEMENT L (G| L | S |M) ∗> < !ATTLIST L b CDATA #IMPLIED e CDATA #IMPLIED> < !−− Statement / Statement range −−> < !ELEMENT S (M) ∗> < !ATTLIST S b CDATA #REQUIRED e CDATA #IMPLIED i d CDATA #IMPLIED> < !ELEMENT M EMPTY> < !ATTLIST M n CDATA #REQUIRED v CDATA #REQUIRED> ]>

Figure 47.3: XML schema for HPCToolkit data files: This schema, prepended to each of the HPCToolkit-generated XML files, describes the format of the profiling data. This particular schema was generated by HPCToolkit 1.0.4.

366

65

70

75

80

85

90

95

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE



Figure 47.4: PAPI TOT CYC.xml: Sample cycle counts observed during profiling, generated from running the HPCToolkit on profiled.c (Figures 47.1–47.2.) These lines would appear after the schema shown in Figure 47.3.

47.1. AN HPCTOOLKIT EXAMPLE RUN

65

70

75

80

85

90

367



Figure 47.5: PAPI FP OPS.xml: Sample flop counts observed during profiling, generated from running the HPCToolkit on profiled.c (Figures 47.1–47.2.) These lines would appear after the schema shown in Figure 47.3.

368

47.2

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

Attaching HPCToolkit Data to the ROSE AST

To attach the data of Figures 47.4 and 47.5 to the AST, we augment a basic ROSE translator with two additional calls, as shown in Figure 47.6, lines 47–48 and 54. We describe these calls below.

47.2.1

Calling ROSE-HPCT

We must first include rosehpct/rosehpct.hh, as shown on line 6 of Figure 47.6. All ROSE-HPCT routines and intermediate data structures reside in the RoseHPCT namespace. Next, lines 47–48 of Figure 47.6 store the contents of the raw XML file into an intermediate data structure of type RoseHPCT::ProgramTreeList t. The RoseHPCT::loadProfilingFiles() routine processes command-line arguments, extracting ROSE-HPCT-specific options that specify the files. We discuss these options in Section 47.4. Line 54 of Figure 47.6, attaches the intermediate profile data structure to the ROSE AST. The RoseHPCT::attachMetrics() routine creates new persistent attributes that store the counter data.2 The attributes are named using the metric name taken from the XML file (see lines 67 of Figures 47.4–47.5); in this example, the attributes are named PAPI TOT CYC and PAPI FP OPS. Following the conventions of persistent attribute mechanism as described in Chapter 7, the attributes themselves are of type RoseHPCT::MetricAttr, which derives from the AstAttribute type.

47.2.2

Retrieving the attribute values

We retrieve the attribute values as described in Chapter 7. In particular, given a located node with cycle and flop attribute values, the printFlopRate() routine defined in lines 11–42 of Figure 47.6 prints the source position, AST node type, and estimated flops per cycle. We call printFlopRate() for each expression statement (SgExpressionStmt), for-initializer (SgForInitStatement), and for-statement (SgForStatement) in lines 59–66 of Figure 47.6. The output is shown in Figure 47.7. Inspecting the output carefully, you may notice seeming discrepancies between the values shown and the values that appear in the XML files, or other values which seem unintuitive. We explain how these values are derived in Section 47.2.3. This example dumps the AST as a PDF file, as shown on line 68 of Figure 47.6. You can inspect this file to confirm where attributes have been attached. We show an example of such a page in Figure 47.8. This page is the SgExprStatement node representing the sum-accumulate on line 26 of Figure 47.1.

47.2.3

Metric propagation

The example program in Figure 47.6 dumps metric values at each expression statement, forinitializer, and for-statement, but the input XML files in Figure 47.4–47.5 only attribute the profile data to “statements” that are not loop constructs. (The XML tags refer to statements, intended to be “simple” non-scoping executable statements; a separate tag 2 The last parameter to RoseHPCT::attachMetrics() is a boolean that, when true, enables verbose (debugging) messages to standard error.

47.3. WORKING WITH GNU GPROF

369

would refer to a loop.) Since the XML file specifies statements only by source line number, RoseHPCT::attachMetrics() attributes measurements to AST nodes in a heuristic way. For example, lines 78–80 of Figure 47.4 indicate that all executions of the “simple statements” of line 25 of the original source (Figure 47.1) accounted for 65534 observed cycles, and that line 26 accounted for an additional 65534 cycles. In the AST, there are multiple “statement” and expression nodes that occur on line 25; indeed, Figure 47.7 lists 4 such statements. The ROSE-HPCT modules uses a heuristic which only assigns metric values to non-scoping nodes derived from SgStatement. When multiple SgStatement nodes occur at a particular source line, ROSE-HPCT simply attaches the metric to each of them. But only one of them will be used for propagating metrics to parent scopes. How is the measurement of 65534 cycles attributed to all of the AST nodes corresponding to line 25 of Figure 47.1? Indeed, line 25 actually “contains” four different SgStatement nodes: an SgForStatement representing the whole loop on lines 25–26, an SgForInitStatement (initializer), and two SgExprStatements (one which is a child of the SgForInitStatement, and another for the for-loop’s test expression). The loop’s increment is stored in the SgForStatement node as an SgExpression, not an SgStatement. The SgForStatement node is a scoping statement, and so it “receives” none of the 65534 cycles. Since the increment is not a statement and one of the SgExprStatements is a child of the initializer, this leaves only two direct descendants of the SgForStatement—the initializer and the test expression statement—among which to divide the 65534 cycles. Thus, each receives 32767 cycles. The initializer’s SgExprStatement child gets the same 32767 as its parent, since the two nodes are equivalent (see first two cases of Figure 47.7). For the entire loop on lines 25–26 of Figure 47.1, the original XML files attribute 65534 cycles to line 25, and another 65534 cycles to line 26 (see Figure 47.4). Moreover, the XML files do not attribute any costs to this loop via an explicit tag. Thus, the best we can infer is that the entire for-statement’s costs is the sum of its immediate child costs; in this case, 131068 cycles. The RoseHPCT::attachMetrics() routine will heuristically accumulate and propagate metrics in this way to assign higher-level scopes approximate costs. The RoseHPCT::attachMetrics() routine automatically propagates metric values through parent scopes. A given metric attribute, RoseHPCT::MetricAttr* x, is “derived” through propagation if x->isDerived() returns true. In fact, if you call x->toString() to obtain a string representation of the metric’s value, two asterisks will be appended to the string as a visual indicator that the metric is derived. We called RoseHPCT::MetricAttr::toString() on lines 27 and 29 of Figure 47.6, and all of the SgForStatement nodes appearing in the output in Figure 47.7 are marked as derived. Alternatively, you cann call RoseHPCT::attachMetricsRaw(), rather than calling RoseHPCT::attachMetrics(). The “raw” routine takes the same arguments but only attaches the raw data, i.e., without attempting to propagate metric values through parent scopes.

47.3

Working with GNU gprof

ROSE-HPCT can also accept the line-by-line profiling output generated by GNU gprof. Currently, we only use the self seconds associated with each line and attach them to ROSE AST as AST attributes named WALLCLK. A typical session to generate compatible gprof profiling file for ROSE-HPCT is given below: [liao@codes]$ gcc -g seq-pi.c -pg

370

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

[liao@codes]$ ./a.out [liao@codes]$ gprof -l -L a.out gmon.out &>profile.result -l tells gprof to output line-by-line profiling information and -L causes gprof to output full file path information. An excerpt of an output file looks like the following: Flat profile: Each sample counts as 0.01 seconds. % cumulative self self time seconds seconds calls Ts/call 38.20 8.84 8.84 36.43 17.27 8.43 11.00 19.82 2.54 5.66 21.12 1.31 3.93 22.04 0.91 3.24 22.79 0.75 0.95 23.00 0.22 0.50 23.12 0.12 0.09 23.14 0.02 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00

47.4

total Ts/call

0.00 0.00 0.00 0.00

name jacobi (/home/liao6/temp/jacobi.c:193 @ 804899c) jacobi (/home/liao6/temp/jacobi.c:196 @ 8048a3f) jacobi (/home/liao6/temp/jacobi.c:188 @ 804893e) jacobi (/home/liao6/temp/jacobi.c:187 @ 8048968) jacobi (/home/liao6/temp/jacobi.c:197 @ 8048a71) jacobi (/home/liao6/temp/jacobi.c:191 @ 8048a7f) jacobi (/home/liao6/temp/jacobi.c:186 @ 8048976) jacobi (/home/liao6/temp/jacobi.c:187 @ 8048935) jacobi (/home/liao6/temp/jacobi.c:190 @ 8048a94) driver (/home/liao6/temp/jacobi.c:91 @ 8048660) error_check (/home/liao6/temp/jacobi.c:220 @ 8048b7c) initialize (/home/liao6/temp/jacobi.c:116 @ 8048722) jacobi (/home/liao6/temp/jacobi.c:160 @ 8048892)

Command-line options

The call to RoseHPCT::loadProfilingFiles() on line 49 of Figure 47.6 processes and extracts ROSE-HPCT-specific command-line options. To generate the output in this chapter, we invoked Figure 47.6 with the following command-line:

./attachMetrics \ -rose:hpctprof /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial/roseH -rose:hpctprof /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial/roseH -rose:hpcteqpath .=/export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial/r -c /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial/roseHPCT/profiled The main option is -rose:hpct:prof , which specifies the HPCToolkit-generated XML file containing metric data. Here, we use this option twice to specify the names of the cycle and flop data files (Figures 47.4–47.5). To accept gprof output file, please use another option -rose:gprof:linebyline . This option cannot be used with -rose:hpct:prof currently. We need the other option, -rose:hpct:eqpath
=, to specify how paths in the HPCToolkit XML files can be mapped to file paths in the ROSE AST. This option allows users to generate performance files on one machine and analyze the results on another machine. In this example, the XML files specify the source file as, “./profiled.c” (line 73 of Figures 47.4 and 47.5); the “eqpath” command-line option above remaps the relative path “.” to an absolute path as it would appear in the ROSE AST. Another example is to use the same performance file even after the original source tree is moved to another location. ROSE-HPCT can still correctly match performance data if the root source paths are given as -rose:hpct:eqpath =.

47.4. COMMAND-LINE OPTIONS

371

Yet another option -rose:hpct:enable debug is provided to display runtime debugging information such as metrics reading, attaching, and propagating. It also adds performance metrics into the ROSE output source file as source comments as shown below. Users can examine the source comments to make sure performance metrics are attached and propagated properly. As we can see, ROSE-HPCT attaches each performance metric to each matching statement. If there are multiple statements showing in the same line, the same metric will be attached to each of them. The metric propagation step will only propagate one of them to upper-level language constructs to ensure the correctness. /* ROSE-HPCT propagated metrics WALLCLK:18.95[SgForStatement at 0xb7beb218] */ for ( /* ROSE-HPCT raw data: Statement WALLCLK:0.02@File jacobi.c 190-0 -> SgForInitStatement 0x94e8d08 at 190 */ i = 1; /* ROSE-HPCT raw data: Statement WALLCLK:0.02@File jacobi.c 190-0 -> SgExprStatement 0x94516d8 at 190 */ i < (n - 1); i++) /* ROSE-HPCT propagated metrics WALLCLK:18.93[SgForStatement at 0xb7beb29c] */ for ( /* ROSE-HPCT raw data: Statement WALLCLK:0.75@File jacobi.c 191-0 -> SgForInitStatement 0x94e8d38 at 191 */ j = 1; /* ROSE-HPCT raw data: Statement WALLCLK:0.75@File jacobi.c 191-0 -> SgExprStatement 0x9451728 at 191 */ j < (m - 1); j++) /* ROSE-HPCT propagated metrics WALLCLK:18.18[SgBasicBlock at 0x93f60b4] */ { /* ROSE-HPCT raw data: Statement WALLCLK:8.84@File jacobi.c 193-0 -> SgExprStatement 0x9451750 at 193 */ resid = (((((ax * (((( *uold)[i - 1])[j]) + ((( *uold)[i + 1])[j]))) + (ay * (((( *uold)[i])[j - 1]) + ((( *uold)[i])[j + 1])))) + (b * ((( *uold)[i])[j]))) - ((( *f)[i])[j])) / b); /* ROSE-HPCT raw data: Statement WALLCLK:8.43@File jacobi.c 196-0 -> SgExprStatement 0x9451778 at 196 */ (( *u)[i])[j] = (((( *uold)[i])[j]) - (omega * resid)); /* ROSE-HPCT raw data: Statement WALLCLK:0.91@File jacobi.c 197-0 -> SgExprStatement 0x94517a0 at 197 */ error = (error + (resid * resid)); }

372

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

// a t t a c h M e t r i c s . cc −− Sample t r a n s l a t o r showing how t o a t t a c h // HPCToolkit d a t a t o t h e ROSE AST.

5

#include #include #include #include

” r o s e . h”

using namespace s t d ; 10

15

20

// P r i n t s t h e e s t i m a t e d f l o p s / c y c l e a t a l o c a t e d node . s t a t i c void p r i n t F l o p R a t e ( const SgNode∗ n ) { const SgLocatedNode ∗ n l o c = i s S g L o c a t e d N o d e ( n ) ; if ( n loc && n l o c −>a t t r i b u t e E x i s t s ( ”PAPI TOT CYC” ) && n l o c −>a t t r i b u t e E x i s t s ( ”PAPI FP OPS” ) ) { // E x t r a c t a t t r i b u t e s . const RoseHPCT : : M e t r i c A t t r ∗ c y c l e s a t t r = dynamic cast ( n−>g e t A t t r i b u t e ( ”PAPI TOT CYC” ) ) ; const RoseHPCT : : M e t r i c A t t r ∗ f l o p s a t t r = dynamic cast ( n−>g e t A t t r i b u t e ( ”PAPI FP OPS” ) ) ; ROSE ASSERT ( c y c l e s a t t r && f l o p s a t t r ) ;

25 // Get double string double string

30

t h e m e t r i c v a l u e s , as d o u b l e s and s t r i n g s . c y c l e s = c y c l e s a t t r −>g e t V a l u e ( ) ; c y c l e s s = const cast ( c y c l e s a t t r )−> t o S t r i n g ( ) ; f l o p s = f l o p s a t t r −>g e t V a l u e ( ) ; f l o p s s = const cast ( f l o p s a t t r )−> t o S t r i n g ( ) ;

// P r i n t node p o i n t e r / t y p e , parent , e s t i m a t e d f l o p r a t e , and s o u r c e p o s i t i o n . const SgNode∗ n p a r = n l o c −>g e t p a r e n t ( ) ; c o u t << ( const void ∗ ) n l o c << ”:<” << n l o c −>c l a s s n a m e ( ) << ”>” << ” ( Par=” << ( const void ∗ ) n p a r << ”:<” << n p a r −>c l a s s n a m e ( ) << ”>)” << ” = ( ” << f l o p s s << ” f l o p s ) ” << ” / ( ” << c y c l e s s << ” cy ) ” << ” ˜= ” << f l o p s / c y c l e s << ” f l o p s / cy ” << e n d l [ ” << n l o c −>g e t s t a r t O f C o n s t r u c t ()−> g e t r a w f i l e n a m e ( ) << ” << ’ : ’ << n l o c −>g e t s t a r t O f C o n s t r u c t ()−> g e t r a w l i n e ( ) << ” ] ” << e n d l ;

35

40 } } 45

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

50

v e c t o r a r g v L i s t ( argv , a r g v+a r g c ) ; c e r r << ” [ Loading HPCToolkit o r Gprof p r o f i l i n g d a t a . . . ] ” << e n d l ; RoseHPCT : : P r o g r a m T r e e L i s t t p r o f i l e s = RoseHPCT : : l o a d P r o f i l i n g F i l e s ( a r g v L i s t ) ;

55

c e r r << ” [ B u i l d i n g t h e ROSE AST . . . ] ” << e n d l ; SgProject ∗ proj = frontend ( argvList ) ; c e r r << ” [ A t t a c h i n g m e t r i c s t o t h e AST . . . ] ” << e n d l ; RoseHPCT : : a t t a c h M e t r i c s ( p r o f i l e s , p r o j , p r o j −>g e t v e r b o s e ( ) > 0 ) ;

60 c e r r << ” [ E s t i m a t i n g f l o p e x e c u t i o n r a t e s . . . ] ” << e n d l ; typedef R o s e S T L C o n t a i n e r N o d e L i s t t ;

65

N o d e L i s t t e s t m t s = NodeQuery : : querySubTree ( p r o j , V SgExprStatement ) ; f o r e a c h ( e s t m t s . b e g i n ( ) , e s t m t s . end ( ) , p r i n t F l o p R a t e ) ; N o d e L i s t t f o r i n i t s = NodeQuery : : querySubTree ( p r o j , V S g F o r I n i t S t a t e m e n t ) ; f o r e a c h ( f o r i n i t s . b e g i n ( ) , f o r i n i t s . end ( ) , p r i n t F l o p R a t e ) ;

70

N o d e L i s t t f o r s = NodeQuery : : querySubTree ( p r o j , V SgForStatement ) ; f o r e a c h ( f o r s . b e g i n ( ) , f o r s . end ( ) , p r i n t F l o p R a t e ) ; c e r r << ” [ Dumping a PDF . . . ] ” << e n d l ; generatePDF ( ∗ p r o j ) ;

75

80

// DQ ( 1 / 2 / 2 0 0 8 ) : This o u t p u t a p p e a r s t o have p r o v i d e d enough s y n c h r o n i z a t i o n i n // t h e o u t p u t t o a l l o w −j 3 2 t o work . S i n c e I can ’ t debug t h e problem f u r t h e r f o r // now I w i l l l e a v e i t . c e r r << ” [ C a l l i n g backend . . . ] ” << e n d l ; return backend ( p r o j ) ; }

47.4. COMMAND-LINE OPTIONS

373

Figure 47.7: Sample output, when running attachMetrics.cc (Figure 47.6) with the XML inputs in Figures 47.4–47.5. Here, we only show the output sent to standard output (i.e., cout and not cerr).

pointer:0x7f51a6128190 SgExprStatement /export/tmp.rose-mgr/jenkins/edg4x/workspace/release-ROSE-docs-weekly/tutorial/roseHPCT/profiled.c 25:8 IsTransformation:0 IsOutputInCodeGeneration:0 Click here to go to the parent node

SgNode* p_parent : 0x7f51a60600f0 bool p_isModified : 0 bool p_containsTransformation : 0 $CLASSNAME* p_freepointer : 0xffffffffffffffff Sg_File_Info* p_startOfConstruct : 0x7f51a6244d78 Sg_File_Info* p_endOfConstruct : 0x7f51a6244dd0 AttachedPreprocessingInfoType* p_attachedPreprocessingInfoPtr : 0 AstAttributeMechanism* p_attributeMechanism : 0x133d4d0 bool p_containsTransformationToSurroundingWhitespace : 0 SgLabelRefExp* p_numeric_label : 0 int p_source_sequence_value : -1 SgExpression* p_expression : 0x7f51a5f1e010

Figure 47.8: Sample PDF showing attributes. AstAttributes: tree_depth: 9

374

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

Chapter 48

TAU Instrumentation Tau is a performance analysis tool from University of Oregon. They have mechanisms for automating instrumentation of the source code’s text file directly, but these can be problematic in the present of macros. We present an example of instrumentation combined with code generation to provide a more robust means of instrumentation for source code. This work is preliminary and depends upon two separate mechanisms for the rewrite of the AST (one high level using strings and one low level representing a more direct handling of the AST at the IR level).

48.1

Input For Examples Showing Information using Tau

Figure 48.1 shows the example input used for demonstration of Tau instrumentation.

48.2

Generating the code representing any IR node

Figure 48.2 shows a code that traverses each IR node and for a SgInitializedName of SgStatement output the scope information. The input code is shown in figure 48.2, the output of this code is shown in figure 48.3.

375

376

2 4 6 8

CHAPTER 48. TAU INSTRUMENTATION

// #i n c l u d e // #i n c l u d e < s t d l i b . h> double f o o ( double x ) { double t h e V a l u e = x ; t h e V a l u e∗= x ; return t h e V a l u e ; }

10 12 14

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { int j , i ; double t S q u a r e d , t ; t = 1.0;

16 tSquared = t ∗ t ; 18 i = 1000; f o r ( j =1; j < i ; j += 2 ) { t S q u a r e d += 1 . 0 ; t S q u a r e d += f o o ( 2 . 2 ) ; }

20 22 24 26

return 0 ; }

Figure 48.1: Example source code used as input to program in codes used in this chapter.

48.2. GENERATING THE CODE REPRESENTING ANY IR NODE

377

// Demonstration o f i n s t r u m e n t a t i o n u s i n g t h e TAU performance m o n i t o r i n g t o o l s ( U n i v e r s i t y o f Oregon ) 2 #include ” r o s e . h” 4 using namespace s t d ; 6 #d e f i n e NEW FILE INFO S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) 8 10 12

SgClassDeclaration ∗ getProfilerClassDeclaration ( SgProject ∗ project ) { // Note t h a t i t would be more e l e g a n t t o l o o k t h i s up i n t h e Symbol t a b l e ( do t h i s n e x t ) S g C l a s s D e c l a r a t i o n ∗ r e t u r n C l a s s D e c l a r a t i o n = NULL ; R o s e S T L C o n t a i n e r c l a s s D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g C l a s s D e c l a r a t i o n ) ; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = c l a s s D e c l a r a t i o n L i s t . b e g i n ( ) ; i != c l a s s D e c l a r a t i o n L i s t . end ( ) ; { // Need t o c a s t ∗ i from SgNode t o a t l e a s t a SgStatement SgClassDeclaration ∗ c l a s s D e c l a r a t i o n = i s S g C l a s s D e c l a r a t i o n (∗ i ) ; ROSE ASSERT ( c l a s s D e c l a r a t i o n != NULL ) ;

14 16 18 20

// p r i n t f (” In g e t P r o f i l e r C l a s s D e c l a r a t i o n ( ) : c l a s s D e c l a r a t i o n −>get name ( ) = %s \n ” , c l a s s D e c l a r a t i o n −>get name ( ) . s

22 24

if

( c l a s s D e c l a r a t i o n −>get name ( ) == ” P r o f i l e r ” ) returnClassDeclaration = classDeclaration ;

}

26 28 30

i ++)

ROSE ASSERT( r e t u r n C l a s s D e c l a r a t i o n != NULL ) ; return r e t u r n C l a s s D e c l a r a t i o n ; }

32 34 36 38

int main ( i n t a r g c , char ∗ a r g v [ ] ) { // This t e s t code t e s t s t h e AST r e w r i t e mechanism t o add TAU I n s t r u m e n t i o n t o t h e AST. // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

40 SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; 42 44 46

// Output t h e s o u r c e code // generatePDF (∗ p r o j e c t ) ;

file

( as r e p r e s e n t e d by t h e SAGE AST) as a PDF f i l e

( w i t h bookmarks )

// Output t h e s o u r c e code // generateDOT (∗ p r o j e c t ) ;

file

( as r e p r e s e n t e d by t h e SAGE AST) as a DOT f i l e

( graph )

48 50 52 54

// Allow ROSE t r a n s l a t o r o p t i o n s t o i n f l u e n c e i f we t r a n s f o r m t h e AST i f ( p r o j e c t −>g e t s k i p t r a n s f o r m a t i o n ( ) == f a l s e ) { // NOTE: There can be m u l t i p l e f i l e s on t h e command l i n e and each f i l e

has a g l o b a l s c o p e

MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ; M i d d l e L e v e l R e w r i t e : : PlacementPositionEnum l o c a t i o n I n S c o p e = M i d L e v e l C o l l e c t i o n T y p e d e f s : : TopOfCurrentScope ;

56 // Add a TAU i n c l u d e d i r e c t i v e t o t h e t o p o f t h e g l o b a l s c o p e R o s e S T L C o n t a i n e r g l o b a l S c o p e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgGlobal ) ; f o r ( R o s e S T L C o n t a i n e r : : i t e r a t o r i = g l o b a l S c o p e L i s t . b e g i n ( ) ; i != g l o b a l S c o p e L i s t . end ( ) ; { // Need t o c a s t ∗ i from SgNode t o a t l e a s t a SgStatement SgGlobal ∗ g l o b a l S c o p e = i s S g G l o b a l (∗ i ) ; ROSE ASSERT ( g l o b a l S c o p e != NULL ) ;

58 60 62

i ++)

64

// TAU do es not seem t o c o m p i l e u s i n g EDG or g++ ( need t o s o r t t h i s o u t w i t h Brian ) // M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e ,”# d e f i n e PROFILING ON \n#i n c l u d e \n ” , scope , l o c a t i o n I n S c o p e ) ; // M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e ,”# i n c l u d e \n ” , scope , l o c a t i o n I n S c o p e ) ; M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e , ”#d e f i n e PROFILING ON 1 \n#d e f i n e TAU STDCXXLIB 1 \n#i n c l u d e }

66 68 70 72 74

#i f 1 // Now g e t t h e c l a s s d e c l a r a t i o n r e p r e s e n t i n g t h e TAU t y p e w i t h which t o b u i l d v a r i a b l e d e c l a r a t i o n s SgClassDeclaration ∗ tauClassDeclaration = getProfilerClassDeclaration ( project ) ; ROSE ASSERT( t a u C l a s s D e c l a r a t i o n != NULL ) ; SgClassType ∗ tauType = t a u C l a s s D e c l a r a t i o n −>g e t t y p e ( ) ; ROSE ASSERT( tauType != NULL ) ;

76 78

// Get a c o n s t r u c t o r t o use w i t h t h e v a r i a b l e d e c l a r a t i o n ( anyone w i l l due f o r code g e n e r a t i o n ) SgMemberFunctionDeclaration ∗ memberFunctionDeclaration = S a g e I n t e r f a c e : : g e t D e f a u l t C o n s t r u c t o r ( t a u C l a s s D e c l a r a t i o ROSE ASSERT( m e m b e r F u n c t i o n D e c l a r a t i o n != NULL ) ;

80 82

// Add t h e i n s t r u m e n t a t i o n t o each f u n c t i o n R o s e S T L C o n t a i n e r f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n )

378

CHAPTER 48. TAU INSTRUMENTATION

Test F a i l e d !

Figure 48.3: Output of input code using tauInstrumenter.C

Chapter 49

The Haskell Interface ROSE’s Haskell interface allows the user to analyse and transform the Sage III IR from Haskell, a statically typed pure functional programming language. See http://www.haskell.org/. The interface exposes almost all Sage III APIs to Haskell, allowing the user to call whichever APIs are required. The interface also supports an AST traversal mechanism inspired by Haskell’s scrap your boilerplate design pattern. The Haskell interface also provides a convenient mechanism for a user to rapidly experiment with the ROSE IR. GHC’s command-line interpreter ghci can be used to explore the IR interactively by invoking API methods at will. The Haskell interface relies on the Glasgow Haskell Compiler (GHC). It is auto-configured so long as the GHC binaries are in your $PATH. If not, you will need to supply the path to the binaries at configure time with the option --with-haskell=bindir, where bindir is the path to GHC’s bin directory. After installation, ROSE is available as a standard Haskell package named rose. This means that you can supply the flag -package rose to the Haskell compiler in order to make the extension available for use. To understand the usage of the interface, it is crucial to grasp how the concept of monads works in Haskell. For a useful tutorial on monads, the reader is referred to the “All About Monads” tutorial found at http://www.haskell.org/all about monads/. The simplest Haskell-based ROSE program is the identity translator, whose code is listed in Figure 49.1. module Main where 2 4 6 8

import ROSE import System main = do p r o j e c t <− f r o n t e n d =<< getArgs exitWith =<< backend p r o j e c t

Figure 49.1: Haskell version of identity translator.

379

380

49.1

CHAPTER 49. THE HASKELL INTERFACE

Traversals

As previously mentioned, the traversal mechanism is inspired by the scrap-your-boilerplate pattern. Our implementation of the scrap-your-boilerplate pattern provides both transformation and query traversals. A transformation traversal applies a global transformation to a tree by applying a given function to each tree node, whereas a query traversal derives a result from a tree using a function that produces a result from a node together with a combinator which combines the results from several nodes (for example in a summation query, the combinator may be the addition function). In order to carry out a traversal, two steps are necessary. Firstly one must build a type extension, a type-generic function built from one or more type-specific functions. Secondly one must employ a generic traversal combinator which applies the type extension throughout the program. In our interface type extensions for transformations are built using the functions mkMn, which builds a type extension from a type-specific function, and extMn, which extends an existing type extension with a type-specific function. Likewise mkMqn and extMqn for queries. These functions perform static and dynamic type checking such that they will only call the type-specific functions when it is safe to do so. The two generic traversal combinators are everywhereMc and everythingMc. They take two arguments: the type extension and the tree to be traversed. everywhereMc returns the transformed tree, and everythingMc the result of the query. Tying everything together, Figure 49.2 shows an example of a simple constant folding transformation.

49.2

Further Reading

Reference documentation for the interface is available on ROSE’s website at: http://www.rosecompiler.org/ROSE HaskellAPI/

49.2. FURTHER READING

module Main where 2 4 6 8

import import import import import import import

Data . Maybe C o n t r o l .Monad System Data . DataMc ROSE ROSE . Sage3 Time

10 12 14

simplifyAddOp : : SgAddOp ( ) −> IO ( S g E x p r e s s i o n ( ) ) simplifyAddOp = s i m p l i f y (+) s i m p l i f y S u b t r a c t O p : : SgSubtractOp ( ) −> IO ( S g E x p r e s s i o n ( ) ) s i m p l i f y S u b t r a c t O p = s i m p l i f y ( −)

16 18 20

s i m p l i f y M u l t i p l y O p : : SgMultiplyOp ( ) −> IO ( S g E x p r e s s i o n ( ) ) simplifyMultiplyOp = simplify (∗) s i m p l i f y D i v i d e O p : : SgDivideOp ( ) −> IO ( S g E x p r e s s i o n ( ) ) s i m p l i f y D i v i d e O p = s i m p l i f y div

22 24 26 28 30 32 34 36 38 40 42 44 46 48

s i m p l i f y op n | n == n u l l S g N o d e = return ( u p S g E x p r e s s i o n n ) | otherwise = do l h s <− binaryOpGetLhsOperand n r h s <− binaryOpGetRhsOperand n l h s I n t <− i s S g I n t V a l l h s r h s I n t <− i s S g I n t V a l r h s i f i s J u s t l h s I n t && i s J u s t r h s I n t then do l h s V a l <− i n t V a l G e t V a l u e ( fromJust l h s I n t ) r h s V a l <− i n t V a l G e t V a l u e ( fromJust r h s I n t ) l e t sum = l h s V a l ‘ op ‘ r h s V a l f i <− s g N u l l F i l e liftM u p S g E x p r e s s i o n ( newIntVal f i sum ( show sum) ) else return ( u p S g E x p r e s s i o n n ) main : : IO ( ) main = do t i m e 1 <− getClockTime p r j <− f r o n t e n d =<< getArgs t i m e 2 <− getClockTime putStrLn ( ” Frontend t o o k ” ++ show ( d i f f C l o c k T i m e s t i m e 2 t i m e 1 ) ) everywhereMc (mkMn simplifyAddOp ‘ extMn ‘ s i m p l i f y S u b t r a c t O p ‘ extMn ‘ s i m p l i f y M u l t i p l y O p ‘ extMn ‘ s i m p l i f y D i v i d e O p ) p r j t i m e 3 <− getClockTime putStrLn ( ” T r a v e r s a l t o o k ” ++ show ( d i f f C l o c k T i m e s t i m e 3 t i m e 2 ) ) exitWith =<< backend p r j

Figure 49.2: Haskell version of constant folding transformation.

381

382

CHAPTER 49. THE HASKELL INTERFACE

Part VIII

Parallelism

Topics relevant to shared or distributed parallel computing using ROSE.

383

Chapter 50

Shared-Memory Parallel Traversals Besides the traversal classes introduced in Chapter 7, ROSE also includes classes to run multithreaded traversals to make use of multi-CPU systems with shared memory (such as typical multicore desktop systems). These shared memory traversals are like the combined traversal classes in that they run several small traversal classes simultaneously; the difference is that here different visit functions may be executed concurrently on different CPUs, while the combined traversals always execute visit functions sequentially. Because of this similarity, the public interface for the parallel traversals is a superset of the combined traversal interface. For each Ast*Processing class there is an AstSharedMemoryParallel*Processing class that provides an interface for adding traversals to its internal list, getting a reference to the internal list, and for starting the combined traversal. The traverse() method performs the same combined traversal as in the corresponding AstCombined*Processing class, and the new traverseInParallel() method (with the same signature as traverse()) must be used to start a parallel traversal. (We currently do not provide traverseWithinFileInParallel() and traverseInputFilesInParallel() that would be needed to make the parallel processing classes a fully-featured drop-in replacement for other classes.) A example of how to use the parallel traversal classes is given in Figure 50.1 (note the similarity to Figure 7.24 on page 62). A group of traversal objects is executed first in combined mode and then in parallel threads. It is the user’s responsibility to make sure that the actions executed in the parallel traversal are thread-safe. File or terminal I/O may produce unexpected results if several threads attempt to write to the same stream at once. Allocation of dynamic memory (including the use of ROSE or standard C++ library calls that allocate memory) may defeat the purpose of multi-threading as such calls will typically be serialized by the library. Two member functions in each AstSharedMemoryParallel*Processing class are available to tune the performance of the parallel traversals. The first is void set numberOfThreads(size t threads) which can be used to set the number of parallel threads. The default value for this parameter is 2. Our experiments suggest that even on systems with more than two CPUs, running more than two traversal threads in parallel does not typically increase performance 385

386

CHAPTER 50. SHARED-MEMORY PARALLEL TRAVERSALS

#include 2 #i f d e f

// Does u s e r want ROSE t o be t h r e a d −aware ?

REENTRANT

4 6 8

c l a s s NodeTypeTraversal : public A s t S i m p l e P r o c e s s i n g { public : NodeTypeTraversal (enum VariantT v a r i a n t , s t d : : s t r i n g typeName ) : myVariant ( v a r i a n t ) , typeName ( typeName ) { }

10 12 14 16 18 20 22 24 26 28 30

protected : v i r t u a l void v i s i t ( SgNode ∗ node ) { i f ( node−>v a r i a n t T ( ) == myVariant ) { s t d : : c o u t << ”Found ” << typeName ; i f ( SgLocatedNode ∗ l o c = i s S g L o c a t e d N o d e ( node ) ) { S g F i l e I n f o ∗ f i = l o c −>g e t s t a r t O f C o n s t r u c t ( ) ; i f ( f i −>i s C o m p i l e r G e n e r a t e d ( ) ) { s t d : : c o u t << ” : c o m p i l e r g e n e r a t e d ” ; } else { s t d : : c o u t << ” : ” << f i −>g e t f i l e n a m e S t r i n g ( ) << ” : ” << f i −>g e t l i n e ( ) ; } } s t d : : c o u t << s t d : : e n d l ; } } private : enum VariantT myVariant ; s t d : : s t r i n g typeName ; };

32 34

i n t main ( i n t a r g c , char ∗∗ a r g v ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . See Rose : : i n i t i a l i z e ROSE INITIALIZE ;

36 SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; 38 s t d : : c o u t << ” combined e x e c u t i o n o f t r a v e r s a l s ” << s t d : : e n d l ; AstSharedMemoryParallelSimpleProcessing p a r a l l e l T r a v e r s a l ( 5 ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l (new NodeTypeTraversal ( V SgForStatement , ” f o r l o o p ” ) ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l (new NodeTypeTraversal ( V SgIntVal , ” i n t c o n s t a n t ” ) ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l (new NodeTypeTraversal ( V S g V a r i a b l e D e c l a r a t i o n , ” v a r i a b l e parallelTraversal . traverse ( project , preorder ) ; s t d : : c o u t << s t d : : e n d l ;

40 42 44

declaration ” ));

46 s t d : : c o u t << ” s h a r e d −memory p a r a l l e l e x e c u t i o n o f t r a v e r s a l s ” << s t d : : e n d l ; parallelTraversal . traverseInParallel ( project , preorder ) ; return 0 ;

48 50

}

52

#e l s e

54

i n t main ( ) { s t d : : c o u t <<” p a r a l l e l return 0 ; }

56

traversal

i s not supported i n t h i s

c o n f i g u r a t i o n ( u s e r d o e s n o t want ROSE t o be t h r e a

58 #e n d i f

Figure 50.1: Example source showing the shared-memory parallel execution of traversals. because the memory bandwidth is saturated. The second function is void set synchronizationWindowSize(size t windowSize). This

387

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44

combined e x e c u t i o n o f t r a v e r s a l s Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found f o r l o o p : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : c o m p i l e r g e n e r a t e d Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found f o r l o o p : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e

s h a r e d −memory p a r a l l e l e x e c u t i o n o f t r a v e r s a l s Found f o r l o o p : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l Found v a r i a b l e d e c l a r a t i o n F o u n d i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r 17 Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 8 . C: 1 7 Found Found f o r l o o p : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e :41 Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found v a r i a b l e d e c l a r a t i o n F o u n d : i n t c o n s t a n t / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 8 . C: 3 4 Found v a r i a b l e d e c l a r a t i o n F o u n d : i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t Found i n t c o n s t a n t 2 2 : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 8 . C: 4 1 Found v a r i a b l e d e c l a r a t i o n F o u n d : c o m p i l e r g e n e r a t e d i n t c o n s t a n t : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m Found i n t c o n s t a n t F o u n d v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r Found v a r i a b l e d e c l a r a t i o n : / export /tmp . r o s e −mgr/ j e n k i n s / edg4x / w o r k s p a c e / r e l e a s e −ROSE−docs−w e e k l y / t u t o r i a l / i n p u t C o d e E x a m 46

Figure 50.2: Output of input file to the shared-memory parallel traversals. Output may be garbled depending on the multi-threaded behavior of the underlying I/O libraries. sets a parameter that corresponds to the size of a ‘window’ of AST nodes that the parallel threads use to synchronize. The value is, in effect, the number of AST nodes that are visited by each thread before synchronizing. Smaller values may in theory result in more locality and therefore better cache utilization at the expense of more time spent waiting for other threads. In practice, synchronization overhead appears to dominate caching effects, so making this parameter too small inhibits performance. The default value is 100000; any large values will result in comparable execution times.

388

CHAPTER 50. SHARED-MEMORY PARALLEL TRAVERSALS

Chapter 51

Distributed-Memory Parallel Traversals ROSE provides an experimental distributed-memory AST traversal mechanism meant for very large scale program analysis. It allows you to distribute expensive program analyses among a distributed-memory system consisting of many processors; this can be a cluster or a network of workstations. Different processes in the distributed system will get different parts of the AST to analyze: Each process is assigned a number of defining function declarations in the AST, and a method implemented by the user is invoked on each of these. The parts of the AST outside of function definitions are shared among all processes, but there is no guarantee that all function definitions are visible to all processes. The distributed memory analysis framework uses the MPI message passing library for communicating attributes among processes. You will need an implementation of MPI to be able to build and run programs using distributed memory traversals; consult your documentation on how to run MPI programs. (This is often done using a program named mpirun, mpiexecute, or similar.) Distributed memory analyses are performed in three phases: 1. A top-down traversal (the ‘pre-traversal’) specified by the user runs on the shared AST outside of function definitions. The inherited attributes this traversal computes for defining function declaration nodes in the AST are saved by the framework for use in the next phase. 2. For every defining function declaration, the user-provided analyzeSubtree() method is invoked; these calls run concurrently, but on different function declarations, on all processors. It takes as arguments the AST node for the function declaration and the inherited attribute computed for this node by the pre-traversal. Within analyzeSubtree() any analysis features provided by ROSE can be used. This method returns the value that will be used as the synthesized attribute for this function declaration in the bottom-up traversal (the ‘post-traversal’). However, unlike normal bottom-up traversals, the synthesized attribute is not simply copied in memory as the AST is distributed. The user must therefore provide the methods serializeAttribute() and deserializeAttribute(). These compute a serialized 389

390

CHAPTER 51. DISTRIBUTED-MEMORY PARALLEL TRAVERSALS representation of a synthesized attribute, and convert such a representation back to the user’s synthesized attribute type, respectively. A serialized attribute is a pair of an integer specifying the size of the attribute in bytes and a pointer to a region of memory of that size that will be copied byte by byte across the distributed system’s communication network. Attributes from different parts of the AST may have different sizes. As serialization of attributes will often involve dynamic memory allocation, the user can also implement the deleteSerializedAttribute() method to such dynamic memory after the serialized data has been copied to the communication subsystem’s internal buffer. Within the analyzeSubtree() method the methods numberOfProcesses() and myID() can be called. These return the total number of concurrent processes, and an integer uniquely identifying the currently running process, respectively. The ID ranges from 0 to one less than the number of processes, but has no semantics other than that it is different for each process.

3. Finally, a bottom-up traversal is run on the shared AST outside of function definitions. The values returned by the distributed analyzers in the previous phase are used as synthesized attributes for function definition nodes in this traversal. After the bottom-up traversal has finished, the getFinalResults() method can be invoked to obtain the final synthesized attribute. The isRootProcess() method returns true on exactly one designated process and can be used to perform output, program transformations, or other tasks that are not meant to be run on each processor. Figure 51.1 gives a complete example of how to use the distributed memory analysis framework. It implements a trivial analysis that determines for each function declaration at what depth in the AST it can be found and what its name is. Figure 51.2 shows the output produced by this program when running using four processors on some input files.

391

2 4

// This i s a s m a l l example o f how t o use t h e d i s t r i b u t e d memory t r a v e r s a l mechanism . I t computes a l i s t o f f u n c t i o n // d e f i n i t i o n s i n a program and o u t p u t s t h e i r names , t h e i r d e p t h i n t h e AST, and t h e ID o f t h e p r o c e s s t h a t found i t . #include #include ” D i s t r i b u t e d M e m o r y A n a l y s i s . h”

6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36

// The pre−t r a v e r s a l runs b e f o r e t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s and i s used t o p r o p a g a t e c o n t e x t i n f o r m a t i o n down // t o t h e i n d i v i d u a l f u n c t i o n d e f i n i t i o n s i n t h e AST. Here , i t j u s t computes t h e d e p t h o f nodes i n t h e AST. c l a s s F u n c t i o n N a m e s P r e T r a v e r s a l : public AstTopDownProcessing { protected : i n t e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode ∗ , i n t depth ) { return depth + 1 ; } }; // The p o s t −t r a v e r s a l runs a f t e r t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s and i s used t o c o l l e c t t h e i n f o r m a t i o n i t // computed . Here , t h e s y n t h e s i z e d a t t r i b u t e s computed by t h e d i s t r i b u t e d a n a l y s i s a r e s t r i n g s r e p r e s e n t i n g i n f o r m a t i o n // a b o u t f u n c t i o n s . These s t r i n g s a r e c o n c a t e n a t e d by t h e p o s t −t r a v e r s a l ( and i n t e r l e a v e d w i t h n e w l i n e s where n e c e s s a r y ) . c l a s s F u n c t i o n N a m e s P o s t T r a v e r s a l : public AstBottomUpProcessing { protected : s t d : : s t r i n g e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode ∗ node , S y n t h e s i z e d A t t r i b u t e s L i s t s y n A t t r i b u t e s ) { std : : s t r i n g r e s u l t = ”” ; SynthesizedAttributesList : : iterator s ; f o r ( s = s y n A t t r i b u t e s . b e g i n ( ) ; s != s y n A t t r i b u t e s . end ( ) ; ++s ) { s t d : : s t r i n g &s t r = ∗ s ; r e s u l t += s t r ; i f ( s t r . s i z e ( ) > 0 && s t r [ s t r . s i z e ( ) − 1 ] != ’ \n ’ ) r e s u l t += ” \n” ; } return r e s u l t ; }

38

std : : s t r i n g defaultSynthesizedAttribute () { return ” ” ; }

40 42

};

44

// This i s t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s . The D i s t r i b u t e d M e m o r y T r a v e r s a l b a s e c l a s s i s a t e m p l a t e t a k i n g an // i n h e r i t e d and a s y n t h e s i z e d a t t r i b u t e t y p e as t e m p l a t e p ar am e te rs ; t h e s e a r e t h e same t y p e s used by t h e pre− and // p o s t −t r a v e r s a l s . c l a s s FunctionNames : public D i s t r i b u t e d M e m o r y T r a v e r s a l { protected : // The a n a l y z e S u b t r e e ( ) method i s c a l l e d f o r e v e r y d e f i n i n g f u n c t i o n d e c l a r a t i o n i n t h e AST. I t s second argument i s t h e // i n h e r i t e d a t t r i b u t e computed f o r t h i s node by t h e pre−t r a v e r s a l , t h e v a l u e i t r e t u r n s becomes t h e s y n t h e s i z e d // a t t r i b u t e used by t h e p o s t −t r a v e r s a l . s t d : : s t r i n g a n a l y z e S u b t r e e ( S g F u n c t i o n D e c l a r a t i o n ∗ f u n c D e c l , i n t depth ) { s t d : : s t r i n g funcName = f u n c D e c l −>get name ( ) . s t r ( ) ; std : : stringstream s ; s << ” p r o c e s s ” << myID ( ) << ” : a t depth ” << depth << ” : f u n c t i o n ” << funcName ; return s . s t r ( ) ; }

46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82

// The u s e r must implement t h i s method t o pack a s y n t h e s i z e d a t t r i b u t e ( a s t r i n g i n t h i s c a s e ) i n t o an a r r a y o f b y t e s // f o r communication . The f i r s t component o f t h e p a i r i s t h e number o f b y t e s i n t h e b u f f e r . s t d : : p a i r s e r i a l i z e A t t r i b u t e ( s t d : : s t r i n g a t t r i b u t e ) const { int len = a t t r i b u t e . s i z e ( ) + 1 ; char ∗ s t r = s t r d u p ( a t t r i b u t e . c s t r ( ) ) ; return s t d : : m a k e p a i r ( l e n , s t r ) ; } // This method must be implemented t o c o n v e r t t h e s e r i a l i z e d d a t a t o t h e a p p l i c a t i o n ’ s s y n t h e s i z e d s t d : : s t r i n g d e s e r i a l i z e A t t r i b u t e ( s t d : : p a i r s e r i a l i z e d A t t r i b u t e ) const { return s t d : : s t r i n g ( ( const char ∗ ) s e r i a l i z e d A t t r i b u t e . s e c o n d ) ; }

a t t r i b u t e type .

// This method i s o p t i o n a l ( t h e d e f a u l t i m p l e m e n t a t i o n i s empty ) . I t s j o b i s t o f r e e memory t h a t may have been // a l l o c a t e d by t h e s e r i a l i z e A t t r i b u t e ( ) method . void d e l e t e S e r i a l i z e d A t t r i b u t e ( s t d : : p a i r s e r i a l i z e d A t t r i b u t e ) const { std : : f r e e ( s e r i a l i z e d A t t r i b u t e . second ) ; } };

392

CHAPTER 51. DISTRIBUTED-MEMORY PARALLEL TRAVERSALS

----- found the following functions: -----process 0: at depth 3: function il process 0: at depth 5: function head process 0: at depth 5: function eq process 1: at depth 3: function headhead process 1: at depth 3: function List process 1: at depth 3: function find process 1: at depth 3: function head process 2: at depth 3: function operator!= process 2: at depth 3: function find process 2: at depth 3: function head process 2: at depth 3: function fib process 3: at depth 3: function xform process 3: at depth 3: function func process 3: at depth 3: function f process 3: at depth 3: function g process 3: at depth 3: function deref -------------------------------------------

Figure 51.2: Example output of a distributed-memory analysis running on four processors.

Chapter 52

Parallel Checker This Chapter is about the project DistributedMemoryAnalysisCompass, which runs Compass Checkers in Parallel, i.e. shared, combined and distributed.

52.1

Different Implementations

The project contains the following files: • parallel functionBased ASTBalance contains the original implementation, which is based on an AST traversal that is balanced based on the number of nodes in each function. Then the functions are distributed over all processors. It contains as well the original interfaces to the shared and combined traversal work. • parallel file compass distributed on the granularity level of files. • parallel functionBased dynamicBalance is the implementation of dynamically scheduling functions across processors. In addition, this algorithm weights the functions first and then sorts them in descending order according to their weight. • parallel compass performs dynamic scheduling based on nodes. The nodes are weighted and then sorted. This algorithm allows the greatest scalability.

52.2

Running through PSUB

The following represents a typical script to run parallel compass on 64 processors using CXX Grammer. CXX Grammar is a binary ROSE AST representation of a previously parsed program. We specify 65 processors because processor 0 does only communication and no computation. Furthermore, we ask for 17 nodes of which each has 8 processors giving us a total of 136 possible processes. We only need 65 but still want to use this configuration. This will average out our 65 processes over 17 nodes, resulting in about 4 processors per node. This trick is used because the AST loaded into memory takes about 400 MB per process. We end up with 1600MB per node. 393

394

CHAPTER 52. PARALLEL CHECKER

#!/bin/bash # Sample LCRM script to be submitted with psub #PSUB -r ncxx65 # sets job name (limit of 7 characters) #PSUB -b nameofbank # sets bank account #PSUB -ln 17 # == defines the amount of nodes needed #PSUB -o ~/log.log #PSUB -e ~/log.err #PSUB -tM 0:05 # Max time 5 min runtime #PSUB -x # export current env var settings #PSUB -nr # do NOT rerun job after system reboot #PSUB -ro # send output log directly to file #PSUB -re # send err log directly to file #PSUB -mb # send email at execution start #PSUB -me # send email at execution finish #PSUB -c zeus #PSUB # no more psub commands # job commands start here set echo echo LCRM job id = $PSUB_JOBID cd ~/new-src/build-rose/projects/DistributedMemoryAnalysisCompass/ srun -n 65 ./parallel_compass -load ~/CXX_Grammar.ast echo "ALL DONE" There are a few tricks that could be considered. Prioritization is based on the amount of time and nodes requested. If less time is specified, it is more likely that a job runs very soon, as processing time becomes available. To submit the job above, use psub file-name. To check the job in the queue, use squeue and to cancel the job use mjobctl -c job-number.

Chapter 53

Reduction Recognition Figures 53.1 shows a translator which finds the first loop of a main function and recognizes reduction operations and variables within the loop. A reduction recognition algorithm (ReductionRecognition()) is implemented in the SageInterface namespace and follows the C/C++ reduction restrictions defined in the OpenMP 3.0 specification.

2 4

// T e s t r e d u c t i o n r e c o g n i t i o n #i n c l u d e ” r o s e . h ” #i n c l u d e #i n c l u d e

6

u s i n g namespace s t d ;

8

i n t main ( i n t a r g c , char ∗ a r g v [ ] ) { // I n i t i a l i z e and c h e c k c o m p a t i b i l i t y . ROSE INITIALIZE ;

10

See

Rose : : i n i t i a l i z e

12 SgProject ∗ p r o j e c t = f r o n t e n d ( argc , argv ) ; // Find main ( ) f u n c t i o n SgFunctionDeclaration ∗ func = S a g e I n t e r f a c e : : findMain ( p r o j e c t ) ; ROSE ASSERT( f u n c != NULL ) ; S g B a s i c B l o c k ∗ body = f u n c −>g e t d e f i n i t i o n ()−> g e t b o d y ( ) ;

14 16 18

// Find t h e f i r s t l o o p R o s e S T L C o n t a i n e r n o d e l i s t = NodeQuery : : q u e r y S u b T r e e ( body , V S g F o r S t a t e m e n t ) ; SgForStatement ∗ loop = isSgForStatement ( ∗ ( n o d e l i s t . begin ( ) ) ) ; ROSE ASSERT( l o o p != NULL ) ;

20 22 24 26

// //

28

// C o l l e c t r e d u c t i o n v a r i a b l e s and o p e r a t i o n s std : : set < s t d : : p a i r > r e d u c t i o n s ; std : : set < s t d : : p a i r >:: c o n s t i t e r a t o r i t e r ; s t d : : s e t < s t d : : p a i r > r e d u c t i o n s ; s t d : : s e t < s t d : : p a i r > : : c o n s t i t e r a t o r S a g e I n t e r f a c e : : ReductionRecognition ( loop , r e d u c t i o n s ) ;

iter ;

30 // Show t h e r e s u l t s c o u t<<” R e d u c t i o n r e c o g n i t i o n r e s u l t s : ”< i t e m = ∗ i t e r ; c o u t<<” \ t v a r i a b l e : ”<u n p a r s e T o S t r i n g ()<<” \ t o p e r a t i o n : ”<
32 34 36 38

return 40

backend ( p r o j e c t ) ;

}

Figure 53.1: Example source code showing reduction recognition. Using this translator we can compile the code shown in figure 53.2. The output is shown in figure 53.3. 395

396

2 4 6 8 10 12 14 16

CHAPTER 53. REDUCTION RECOGNITION

i n t a [ 1 0 0 ] , sum ; int main ( ) { i n t i , sum2 , yy , z z ; sum = 0 ; f o r ( i =0; i <100; i ++) { i n t xx ; a [ i ]= i ; sum = a [ i ]+ sum ; xx++; yy =0; yy−−; z z∗=a [ i ] ; } return 0 ; }

Figure 53.2: Example source code used as input to loop reduction recognition processor.

2

Reduction r e c o g n i t i o n r e s u l t s : v a r i a b l e : sum o p e r a t i o n :+ v a r i a b l e : zz operation :∗

Figure 53.3: Output of input to reduction recognition processor.

Part IX

Tutorial Summary

Summary of the ROSE tutorials.

397

Chapter 54

Tutorial Wrap-up This tutorial has shown the construction and simple manipulation of the AST as part of the construction of the source-to-source translators using ROSE. Much more complex translators are possible using ROSE, but they are not such that they present well as part of a tutorial with short example programs. The remaining chapters of the tutorial include examples of translators built using ROSE as part of active collaborations with external research groups.

399

FIXME: Reference the User Manual, HTML Doxygen generated documentation, unresolved issues, etc. Reference other work currently using ROSE (ANL, Cornell in the future), academic collaborations.

400

CHAPTER 54. TUTORIAL WRAP-UP

Appendix This appendix includes information useful in supporting the ROSE Tutorial.

54.1

Location of To Do List

This was an older location for the Tutorial Tod List. We now keep the Tod list in the ROSE/docs/testDoxygen/ProjectToDoList.docs in the section called: ROSE Tutorial Todo List.

54.2

Abstract Grammar

In this section we show an abstract grammar for the ROSE AST. The grammar generates the set of all ASTs. On the left hand side of a production we have a non-terminal that corresponds to an inner node of the class hierarchy. On the right hand side of a production we have either one non-terminal or one terminal. The terminal corresponds to a leaf-node where the children of the respective node are listed as double-colon separated pairs, consisting of an access name (= name for get function) and a name that directly corresponds to the class of the child. Details like pointers are hidden. The asterisk shows where lists of children (containers) are used in the ROSE AST. For each terminal, a name followed by ’(’ and ’)’, a variant exists in ROSE with the prefix V_ that can be obtained by using the function variantT() on a node. Note, that concrete classes of AST nodes directly correspond to terminals and base classes to non-terminals. START:SgNode SgNode : SgSupport | SgLocatedNode | SgSymbol ; SgSupport : SgName() | SgSymbolTable() | SgInitializedName ( initptr:SgInitializer ) | SgFile ( root:SgGlobal ( declarations:SgDeclarationStatement* ) ) | SgProject ( fileList:SgFile ( root:SgGlobal ( declarations:SgDeclarationStatement* ) ) ) | SgOptions() | SgBaseClass ( base_class:SgClassDeclaration ) | SgTemplateParameter ( expression:SgExpression, defaultExpressionParameter:SgExpression, 401

402

|

| | | ;

APPENDIX templateDeclaration:SgTemplateDeclaration(), defaultTemplateDeclarationParameter:SgTemplateDeclaration() ) SgTemplateArgument ( expression:SgExpression, templateInstantiation:SgTemplateInstantiationDecl ( definition:SgClassDefinition ) ) SgFunctionParameterTypeList() SgAttribute SgModifier

SgAttribute : SgPragma() | SgBitAttribute ; SgBitAttribute : SgFuncDecl_attr() | SgClassDecl_attr() ; SgModifier : SgModifierNodes() | SgConstVolatileModifier() | SgStorageModifier() | SgAccessModifier() | SgFunctionModifier() | SgUPC_AccessModifier() | SgSpecialFunctionModifier() | SgElaboratedTypeModifier() | SgLinkageModifier() | SgBaseClassModifier() | SgDeclarationModifier() ; SgLocatedNode : SgStatement | SgExpression ; SgStatement : SgExprStatement ( expression_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgLabelStatement() | SgCaseOptionStmt ( key_root:SgExpressionRoot ( operand_i:SgExpression ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgTryStmt ( body:SgBasicBlock ( statements:SgStatement* ), catch_statement_seq_root:SgCatchStatementSeq ( catch_statement_seq:SgStatement* ) ) | SgDefaultOptionStmt ( body:SgBasicBlock ( statements:SgStatement* ) ) | SgBreakStmt() | SgContinueStmt() | SgReturnStmt ( expression_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgGotoStatement()

54.2. ABSTRACT GRAMMAR | | | | | | ;

403

SgSpawnStmt ( the_func_root:SgExpressionRoot ( operand_i:SgExpression ) ) SgForInitStatement ( init_stmt:SgStatement* ) SgCatchStatementSeq ( catch_statement_seq:SgStatement* ) SgClinkageStartStatement() SgDeclarationStatement SgScopeStatement

SgDeclarationStatement : SgVariableDeclaration ( variables:SgInitializedName ( initptr:SgInitializer ) ) | SgVariableDefinition ( vardefn:SgInitializedName ( initptr:SgInitializer ), bitfield:SgUnsignedLongVal() ) | SgEnumDeclaration() | SgAsmStmt ( expr_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgTemplateDeclaration() | SgNamespaceDeclarationStatement ( definition:SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) ) | SgNamespaceAliasDeclarationStatement() | SgUsingDirectiveStatement() | SgUsingDeclarationStatement() | SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ) | SgCtorInitializerList ( ctors:SgInitializedName ( initptr:SgInitializer ) ) | SgPragmaDeclaration ( pragma:SgPragma() ) | SgClassDeclaration | SgFunctionDeclaration ; SgClassDeclaration : SgTemplateInstantiationDecl ( definition:SgClassDefinition ) ; SgFunctionDeclaration : SgTemplateInstantiationFunctionDecl ( parameterList:SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ), definition:SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ) ) | SgMemberFunctionDeclaration ; SgMemberFunctionDeclaration : SgTemplateInstantiationMemberFunctionDecl ( parameterList:SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ), definition:SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ), CtorInitializerList:SgCtorInitializerList

404

APPENDIX

( ctors:SgInitializedName ( initptr:SgInitializer ) ) ; SgScopeStatement : SgGlobal ( declarations:SgDeclarationStatement* ) | SgBasicBlock ( statements:SgStatement* ) | SgIfStmt ( conditional:SgStatement, true_body:SgBasicBlock ( statements:SgStatement* ), false_body:SgBasicBlock ( statements:SgStatement* ) ) | SgForStatement ( for_init_stmt:SgForInitStatement ( init_stmt:SgStatement* ), test_expr_root:SgExpressionRoot ( operand_i:SgExpression ), increment_expr_root:SgExpressionRoot ( operand_i:SgExpression ), loop_body:SgBasicBlock ( statements:SgStatement* ) ) | SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ) | SgWhileStmt ( condition:SgStatement, body:SgBasicBlock ( statements:SgStatement* ) ) | SgDoWhileStmt ( condition:SgStatement, body:SgBasicBlock ( statements:SgStatement* ) ) | SgSwitchStatement ( item_selector_root:SgExpressionRoot ( operand_i:SgExpression ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgCatchOptionStmt ( condition:SgVariableDeclaration ( variables:SgInitializedName ( initptr:SgInitializer ) ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) | SgClassDefinition ; SgClassDefinition : SgTemplateInstantiationDefn ( members:SgDeclarationStatement* ) ; SgExpression : SgExprListExp ( expressions:SgExpression* ) | SgVarRefExp() | SgClassNameRefExp() | SgFunctionRefExp() | SgMemberFunctionRefExp() | SgFunctionCallExp ( function:SgExpression, args:SgExprListExp ( expressions:SgExpression* ) ) | SgSizeOfOp ( operand_expr:SgExpression ) | SgConditionalExp ( conditional_exp:SgExpression, true_exp:SgExpression, false_exp:SgExpression ) | SgNewExp ( placement_args:SgExprListExp ( expressions:SgExpression* ), constructor_args:SgConstructorInitializer( args:SgExprListExp(expressions:SgExpression*) ), builtin_args:SgExpression ) | SgDeleteExp ( variable:SgExpression ) | SgThisExp()

54.2. ABSTRACT GRAMMAR | | | | | | | | | | ;

405

SgRefExp() SgVarArgStartOp ( lhs_operand:SgExpression, rhs_operand:SgExpression ) SgVarArgOp ( operand_expr:SgExpression ) SgVarArgEndOp ( operand_expr:SgExpression ) SgVarArgCopyOp ( lhs_operand:SgExpression, rhs_operand:SgExpression ) SgVarArgStartOneOperandOp ( operand_expr:SgExpression ) SgInitializer SgValueExp SgBinaryOp SgUnaryOp

SgInitializer : SgAggregateInitializer ( initializers:SgExprListExp ( expressions:SgExpression* ) ) | SgConstructorInitializer ( args:SgExprListExp ( expressions:SgExpression* ) ) | SgAssignInitializer ( operand_i:SgExpression ) ; SgValueExp : SgBoolValExp() | SgStringVal() | SgShortVal() | SgCharVal() | SgUnsignedCharVal() | SgWcharVal() | SgUnsignedShortVal() | SgIntVal() | SgEnumVal() | SgUnsignedIntVal() | SgLongIntVal() | SgLongLongIntVal() | SgUnsignedLongLongIntVal() | SgUnsignedLongVal() | SgFloatVal() | SgDoubleVal() | SgLongDoubleVal() ; SgBinaryOp : SgArrowExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgDotExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgDotStarOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgArrowStarOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgEqualityOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgLessThanOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgGreaterThanOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgNotEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgLessOrEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression )

406 | | | | | | | | | | | | | | | | | | | | | | | | | | | | ;

APPENDIX SgGreaterOrEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAddOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgSubtractOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMultiplyOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgDivideOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgIntegerDivideOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgModOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAndOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgOrOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitXorOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitAndOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitOrOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgCommaOpExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgLshiftOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgRshiftOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgPntrArrRefExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgScopeOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgPlusAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMinusAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAndAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgIorAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMultAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgDivAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgModAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgXorAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgLshiftAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgRshiftAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression )

SgUnaryOp : SgExpressionRoot ( operand_i:SgExpression ) | SgMinusOp ( operand_i:SgExpression ) | SgUnaryAddOp ( operand_i:SgExpression ) | SgNotOp ( operand_i:SgExpression ) | SgPointerDerefExp ( operand_i:SgExpression ) | SgAddressOfOp ( operand_i:SgExpression ) | SgMinusMinusOp ( operand_i:SgExpression ) | SgPlusPlusOp ( operand_i:SgExpression ) | SgBitComplementOp ( operand_i:SgExpression ) | SgCastExp ( operand_i:SgExpression ) | SgThrowOp ( operand_i:SgExpression ) ; SgSymbol : SgVariableSymbol() | SgClassSymbol ( declaration:SgClassDeclaration ) | SgTemplateSymbol ( declaration:SgTemplateDeclaration() )

54.2. ABSTRACT GRAMMAR

407

| | | | |

SgEnumSymbol ( declaration:SgEnumDeclaration() ) SgEnumFieldSymbol() SgLabelSymbol ( declaration:SgLabelStatement() ) SgDefaultSymbol() SgNamespaceSymbol ( declaration:SgNamespaceDeclarationStatement ( definition:SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) ) ) | SgFunctionSymbol ; SgFunctionSymbol : SgMemberFunctionSymbol ( declaration:SgFunctionDeclaration ) ; SgPartialFunctionType : SgPartialFunctionModifierType ( ref_to:SgReferenceType, ptr_to:SgPointerType, modifiers:SgModifierNodes(), typedefs:SgTypedefSeq, return_type:SgType, orig_return_type:SgType ) ; This grammar was generated with GRATO, a grammar transformation tool, written by Markus Schordan. The input is a representation generated by ROSETTA. Several other versions of the grammar can be generated as well, such as eliminating nested tree nodes by introducing auxiliary non-terminals, introducing base types as non-terminals etc. Additionally from that grammar we can also generate grammars that can be used with yacc/bison, Coco, and other attribute grammar tools, as well as tree grammar based tools such as burg (requires a transformation to a binary tree).

408

APPENDIX

Glossary We define terms used in the ROSE manual which might otherwise be unclear. • AST Abstract Syntax Tree. A very basic understanding of an AST is the entry level into ROSE. • Attribute User defined information (objects) associated with IR nodes. Forms of attributes include: accumulator, inherited, persistent, and synthesized. Both inherited and synthesized attributes are managed automatically on the stack within a traversal. Accumulator attributes are typically something semantically equivalent to a global variable (often a static data member of a class). Persistent attributes are explicitly added to the AST and are managed directly by the user. As a result, they can persist across multiple traversals of the AST. Persistent attributes are also saved in the binary file I/O, but only if the user provides the attribute specific pack() and unpack() virtual member functions. See the ROSE User Manual for more information, and the ROSE Tutorial for examples. • CFG As used in ROSE, this is the Control Flow Graph, not Context Free Grammar or anything else. • EDG Edison Design Group (the commercial company that produces the C and C++ front-end that is used in ROSE). • IR Intermediate Representation (IR). The IR is the set of classes defined within SAGE III that allow an AST to be built to define any application in C, C++, and Fortran application. • Query (as in AST Query) Operations on the AST that return answers to questions posed about the content or context in the AST. • ROSE A project that covers both research in optimization and a specific infrastructure for handling large scale C, C++, and Fortran applications. • Rosetta A tool (written by the ROSE team) used within ROSE to automate the generation of the SAGE III IR. • SAGE++ and SAGE II An older object-oriented IR upon which the API of SAGE III IR is based. • Semantic Information What abstractions mean (short answer). (This might be better as a description of what kind of semantic information ROSE could take advantage, not a definition.) 409

FIXME: Define the following terms: IR node, Inherited Attribute, Synthesized Attribute, Accumulator Attribute, AST Traversal

410

GLOSSARY • Telescoping Languages A research area that defines a process to generate domainspecific languages from a general purpose languages. • Transformation The process of automating the editing (either reconfiguration, addition, or deletion; or some combination) of input application parts to build a new application. In the context of ROSE, all transformations are source-to-source. • Translator An executable program (in our context built using ROSE) that performs source-to-source translation on an existing input application source to generate a second (generated) source code file. The second (generated) source code is then typically provided as input to a vendor provided compiler (which generates object code or an executable program). • Traversal The process of operating on the AST in some order (usually pre-order, postorder, out of order [randomly], depending on the traversal that is used). The ROSE user builds a traversal from base classes that do the traversal and execute a function, or a number of functions, provided by the user.

Loading...

ROSE Tutorial - ROSE compiler infrastructure

ROSE Tutorial: A Tool for Building Source-to-Source Translators Draft Tutorial (version 0.9.10.151) Daniel Quinlan, Markus Schordan, Richard Vuduc, Qi...

2MB Sizes 0 Downloads 0 Views

Recommend Documents

Rational Rose Tutorial
Available at student lab. ○ Can download full version from ... ROSE = Rational Object Oriented Software Engineering. â

Freehold Acquisition - Rose and Rose Solicitors
Contact us for a Consultation or phone us on 020 8974 7490. Leasehold enfranchisement (also known as Collective Enfranch

Batu Barelang Rose atau Batam Rose
Jun 12, 2017 - Pdf perkembangan modern dunia islam mirip sebagai halnya barat (nasution dunia islam juga timbul pikiran

Rose International
Lead meetings with business and IT to develop understanding of business functions, application roles/entitlements, ident

Huidobro's Rose
Naturaleza, ello prueba que no amáis ni la Naturaleza ni el Arte. —Vicente Huidobro, “Creacionismo” 739 ..... sin

GREEN ROSE
May 3, 2012 - Berikut ini adalah beberapa cara sederhana membuat minyak sari kita sendiri. Minyak ini disarikan dari aro

St. Rose of Lima Church - St. Rose of Lima Brooklyn
Mar 11, 2018 - SATURDAY, March 10th. 9:00. Purgatorial Society (March) Marion. Romano, Leonel Hevia, Maria Teresa Merida

Origami rose tessellation instructions
Anson trivial and undiscovered substitutes for laporan pendahuluan stroke pdf their Glissades or avowedly personifies. S

Rose in Bloom
May 9, 2013 - Salah satunya adalah melalui penambahbaikan pengajaran guru dalam bilik darjah dan penambahbaikan dari seg

Spanish - Hendrickson Rose Publishing
Ahora usted puede tener 35 populares cuadros bíblicos, mapas y líneas de tiempo de Rose Publishing en un libro de espi

#5 Parte Dos: Buscaminas! Decisin atltica | Hentai 8,137 hide | Safe House